File windows doesn't display image

[Stevef48]

I've discovered a number of images that don't appear as thumbnails in the file window. As you can see from the screenshot, the camera and type of image varies.
Double-clicking on the thumbnail the viewer correctly shows the image. The ones I've checked display perfectly in the Windows 10 Photos app. WIC reports no problems with the raw images and tells me that no codec is required for JPEG.

The problem is:-
I can fix the jpegs by opening them in Photos, making a minor amendment and saving the changed file. However, I can't find them easily, because the only problem is seeing them in IMatch. Other photo browsers display the images. I have zipped sample images and their WIC reports, but can't upload the 54.1mb zip file, so I've put it on Dropbox https://www.dropbox.com/s/bqj5ujomosiz6fy/2016-04-April.zip?dl=0

Apologies for raising yet another issue,
Steve

[herman]

I have seen this phenomenon as well when I ingested a number (100 or so) of scanned photos last Sunday, all in .tif format.
I could correct it by conducting a forced rescan of the relevant folder.
As it was a one-off occurrence for me and as logging was set to "normal" I did not bother to report it yet....

[ubacher]

Happens to me also (although not lately). Like Herman I just force-rescan the files and that fixes it.

[Mario]

Reboot the PC. Then do a forced update by selecting the files and pressing Shift+Ctrl+F5.
You did not include an IMatch log file in your 50MB ZIP so I have no idea if and which problems IMatch ran into when processing these files.
All images process fine on my system and also on a fresh Windows 10 install with only the default WIC codecs installed.

Note that proprietary RAW files like RW2 can always be troublesome. Does SONY provide a WIC codec to process these proprietary files?

[Stevef48]

The forced update seems to have fixed the problem, without rebooting. I didn't include a log, because looking through previous debug logs I didn't see anything about the processing of these images.
I have attached a debug log, in case it helps. IMatch rescanned a folder whilst it was running and I now have thumbnails for DSC_6583 NEF & JPG. RW2 files are Panasonic raw. I'll see whether they have a codec.
Thanks
Steve

[Mario]

This is the log from the successful rescan?
The log from the session where you experienced this problem would be helpful. If the WIC codec failed or there was another reason for this, the log may contain warning or error entries.

[Stevef48]

There are many photos without thumbnails, both JPEG and various types of RAW. I keep coming across more of them, but can't see a pattern.
One folder has 1228 images, 29 without thumbnails. Photos either side of the blank images, taken with the same camera, have thumbnails.
Forcing rescan seems to be the answer, it has worked every time, although a normal rescan doesn't.

I'm not sure how to provide a debug log that shows anything relevant. Do you think this might work?:-
a) Find images;
b) Navigate to a different folder;
c) Start Debug Logging;
d) Close, then restart IMatch;
e) Navigate to the original folder;
f) Force rescan.

Steve

[Mario]

If IMatch runs into issues when processing files, it logs them at that time.
So the log file after you have rescanned files and noticed some files without a proper thumbnails would be the one that contains the info we need.

Virus checker has an exception for IMatch?

Is your system 'maxed out' when IMatch runs (disk or processor utilization nearby 100% in Windows Task Manager)?
Do you use a notebook, normal PC or workstation-grade PC for IMatch?

If you can reproduce this at will, you could try to reduce the number of threads IMatch uses when ingesting data.
See Process Control (Advanced Setting)
When you set the processes to 2 or 4 (if your CPU has more than 4 cores) and the problem vanishes, it is somehow caused by 'stress'...

[Stevef48]

If IMatch runs into issues when processing files, it logs them at that time.
So the log file after you have rescanned files and noticed some files without a proper thumbnails would be the one that contains the info we need. The attached Log won't show this. I forced update of all files.

Virus checker has an exception for IMatch?I think so. I changed from AVG to McAfee and it tells me that it has 'boosted' IMatch, so presumably it isn't interfering.

Is your system 'maxed out' when IMatch runs (disk or processor utilization nearby 100% in Windows Task Manager)? No
Do you use a notebook, normal PC or workstation-grade PC for IMatch? I'm using a Dell G3 15 Laptop, 16GB Ram, dedicated Graphics, 512gb NME SSD and 2tb MSATA SSD

If you can reproduce this at will, you could try to reduce the number of threads IMatch uses when ingesting data. I'm not sure that I understand. Images that have been rescanned with Force Update seem to always appear correctly. It isn't easy to find images without thumbnails, it means manually scrolling through 1,000s of images until I find a folder that contains them. After forcing updates the file modified date hasn't changed, so presumably IMatch doesn't modify these files? I have removed the preference to preserve dates.
See Process Control (Advanced Setting)
When you set the processes to 2 or 4 (if your CPU has more than 4 cores) and the problem vanishes, it is somehow caused by 'stress'... Do you think that after changing this IMatch will display the thumbnails, without a rescan?

[Stevef48]

Changing number of threads to 2,or 4, then doing a normal rescan did not generate thumbnails.
The attached LOG contains everything from the previous log, as IMatch is still running. It seems that IMatch doesn't generate a thumbnail if an image has no changes.
Steve

[Mario]

I see many messages like

Only thumbnail or small preview extracted. Falling back to ptc RAW processing for RW2 files.
See the log file for the names of corresponding files.

This means that the WIC codec for RW2 installed on your system cannot handle the files or only extract a small preview that is less than the minimum size configured under Edit > Preferences > Cache.

IMatch has to load the file again, falling back to the LibRaw processing pipeline. This seems to happen for all(most) RW2 files.
These constant failures may cause something to break. Either the WIC subsystem or maybe LibRaw or IMatch. I cannot see an actual reason.

Don't you have a WIC codec installed which understands the proprietary RW2 variant you are using? You can check with WIC Diagnostics

If your camera vendor does not provide a WIC codec which supports the file format of your cameras, maybe switch IMatch to always use LibRaw under Edit > Preferences > Application: Prefer photools.com RAW processing. This would speed things up because IMatch does not longer use WIC codecs, and if the codec or the constant fallback processing for each of your RAW files is the problem, it should be fixed.

Panasonic should equip you for free with a working WIC codec. If they don't, well, bugger. Money not well spent.

You can install the latest WIC codecs from Microsoft which have been reported to handle more RAW flavors.

https://www.microsoft.com/en-us/p/raw-image-extension/9nctdw2w1bh8?activetab=pivot:regionofsystemrequirementstab

[plastikman]

Personal rant: This whole WIC situation is a mess. Why do I need to install manufacturer-specific RAW codec drivers to begin with? macOS never asks me to install a driver for my RAW files, it just works. Microsoft should handle this gracefully in the background instead of making life hard for users (which they always like to do... e.g. aborted windows upgrades because of some compatibility thing X or conflict Y). Adobe doesn't even offer a DNG WIC codec so I am dependent on one from Microsoft that doesn't even work correctly (the image rotation problem). There is many thing I like about Windows but this is definitely not one of them.

[David_H]

Personal rant: This whole WIC situation is a mess. Why do I need to install manufacturer-specific RAW codec drivers to begin with?

Because only the manufacturers know what the contents of THEIR PROPRIETARY files means.

macOS never asks me to install a driver for my RAW files, it just works. Microsoft should handle this gracefully in the background instead of making life hard for users (which they always like to do... e.g. aborted windows upgrades because of some compatibility thing X or conflict Y).

Until, of course, it doesn't work and then you are wholly dependent on Apple (or Microsoft) to provide an update (what's that? You want to use a CR3 file? Better upgrade your OS. Oh. Doesn't work on your current hardware. Oh well; we've got some new shiny for you).

Microsoft came up with the almost perfect idea of weaning everyone out of proprietary SDKs which vendors used to have to use (or reverse engineer the format) to interact with raw files by providing a relatively simple plugin system that anyone could write a codec for to process a raw file in a standard way - and it almost worked, except people stopped bothering to write the codecs; so they had to provide their own and then utilising the libraw ones to keep up - but if you don't like how they work, you can use a different one (if it exists).

[Jingo]

Personal rant: This whole WIC situation is a mess. Why do I need to install manufacturer-specific RAW codec drivers to begin with? macOS never asks me to install a driver for my RAW files, it just works. Microsoft should handle this gracefully in the background instead of making life hard for users (which they always like to do... e.g. aborted windows upgrades because of some compatibility thing X or conflict Y). Adobe doesn't even offer a DNG WIC codec so I am dependent on one from Microsoft that doesn't even work correctly (the image rotation problem). There is many thing I like about Windows but this is definitely not one of them.

My "solution" - convert ALL images using supported RAW editors to TIFF or JPG and only deal with these formats.  Works in 99.9% of software without issue and with batch processes, it only take a few clicks to get things converted... even if no edits are made to the RAW file.

[Stevef48]

This means that the WIC codec for RW2 installed on your system cannot handle the files or only extract a small preview that is less than the minimum size configured under Edit > Preferences > Cache. I installed the latest Microsoft WIC at your suggestion a couple of days ago. Maybe there are too many WIC Codecs for the same file types.

IMatch has to load the file again, falling back to the LibRaw processing pipeline. This seems to happen for all(most) RW2 files.
These constant failures may cause something to break. Either the WIC subsystem or maybe LibRaw or IMatch. I cannot see an actual reason.

Don't you have a WIC codec installed which understands the proprietary RW2 variant you are using? You can check with WIC Diagnostics I've run this several times for different formats and the response is always that the format is fully supported by the codecs.

If your camera vendor does not provide a WIC codec which supports the file format of your cameras, maybe switch IMatch to always use LibRaw under Edit > Preferences > Application: Prefer photools.com RAW processing. This would speed things up because IMatch does not longer use WIC codecs, and if the codec or the constant fallback processing for each of your RAW files is the problem, it should be fixed. As I said, every file tested so far has passed the WIC test

Panasonic should equip you for free with a working WIC codec. If they don't, well, bugger. Money not well spent. They only appear to have a 32 bit codec

You can install the latest WIC codecs from Microsoft which have been reported to handle more RAW flavors.

https://www.microsoft.com/en-us/p/raw-image-extension/9nctdw2w1bh8?activetab=pivot:regionofsystemrequirementstab Done!

I think I'll find out how to delete the old Windows Codec, maybe having more than one is causing trouble???

[Mario]

I think I'll find out how to delete the old Windows Codec, maybe having more than one is causing trouble???

This looks fine. The new Microsoft Raw Image Decoder seem to handle your RW2 files just fine.

[Mario]

Personal rant: This whole WIC situation is a mess. Why do I need to install manufacturer-specific RAW codec drivers to begin with? macOS never asks me to install a driver for my RAW files, it just works. Microsoft should handle this gracefully in the background instead of making life hard for users (which they always like to do... e.g. aborted windows upgrades because of some compatibility thing X or conflict Y). Adobe doesn't even offer a DNG WIC codec so I am dependent on one from Microsoft that doesn't even work correctly (the image rotation problem). There is many thing I like about Windows but this is definitely not one of them.

You are mistaken.

With WIC Microsoft came up with a golden way to get rid of all the proprietary RAW format mess. And the encryption companies like Nikon and some others added to prevent reverse-engineering. The hiding of metadata in proprietary maker notes. Even moving standard EXIF data inside maker notes to make them harder to access. Or encrypting (!) while balance and lens information to make it hard and even illegal in some countries to access it. A big mess, (deliberately) created by camera vendors.

WIC is basically the same as the printer driver subsystem in Windows. Microsoft provides the infrastructure and the printer vendor (camera vendor) provides the driver(s) for one or more printers (WIC codecs for cameras).

The camera vendors know their cameras best and can create a WIC codecs which produce superior renditions of our RAW files.
Optimally, using the same code used by the camera itself to produce the preview rendition. Yay!

WIC codecs also allows companies access the RAW Bayer data in a standardized way, without revealing anything about the RAW format or metadata storage or maybe patented technologies used in the RAW format. This is for companies which create RAW processing software.

So, all a camera vendor needs to do is let one of their programmers maintain a WIC codec.
Making additions occasionally when new cameras or firmware versions come out which produce yet another RAW format variant.
And then provide their paying customers with a download of that WIC codec or send it to Microsoft for automatic inclusion in Windows (like printer drivers).

But, noooooo....

Even companies like Canon which initially provided WIC codecs no longer do. This means especially you, Canon.
For cost-cutting reasons. In order to favor their own branded "RAW" software. Or whatever.

(Nikon still does provide a codec which can read anyhting from the D1H to the Z7. Thanks).

So, Microsoft started to fill the gaps. But they don't make money from WIC codecs. And I don't know if they get any information from camera vendors. Looking at the sketchy support for RAW format variants in the default WIC codecs in Windows, I pretty much doubt it. So they probably depend on reverse-engineering RAW formats. Which costs extra money and may even be illegal for them.

Open source authors like Dave Coffin (dcraw) had to jump in. Or now the LibRaw project, which is a fork of Dave's dcraw, after he lost interest.
Or the developer of the Fast PictureViewer codecs, which were excellent. And still work well for many cameras.
But Axel no longer maintains them, he now works for Microsoft ;-)

Microsoft based their new Microsoft Camera Raw Decoder codecs on LibRaw, which either means they are trying to get away as cheap as possible.
Or sends some money into the project to make it live very long, adding support for all new RAW formats camera vendors produce.

But, frankly, the camera vendors don't do their job. Maintaining a WIC codec may cost a few 10K US$ per year. Probably as much as two or three magazine ads.
If they are too cheap to invest in that and let their customers stand in the rain, they may lose these customers.

My biggest problem with all that is that if often boils up in IMatch.
"My RAW does not show". "The rotation is wrong". "The colors are wrong".

Heck, all I want for IMatch is to make some function calls into WIC to produce a suitable rendition of a RAW file. With OKish colors, the correct orientation.
This should be a no-brainer and just work.

All that's needed is the camera vendor you give your money gives you a working WIC codec. Or provides Microsoft with all the information needed to include support for their RAW formats by default in Windows. Why is this so hard?

[Stevef48]

I feel your pain.
A partial solution therefore is to exclude my RAW images from IMatch, but I stress that's only a partial solution, because:-
a) Many of the images which had no thumbnail were JPEGs;
b) As I said earlier only a small percentage of RAW files in a folder do not generate a thumbnail, the others generated by the same camera are fine.

I'll try forcing IMatch to use LibRaw. It won't help with JPEGs, because WIC Diagnosis shows these don't use codecs, but anything that speeds up IMatch is worth a try.

Watch this space.
Steve

[Stevef48]

Would buying FastPictureViewer Codec make any difference?

Steve

[Mario]

They are no longer maintained. The developer now works for Microsoft.

IMatch ingests JPEG files via an internal JPEG standard implementation. And it loads JPEGs from the cache using the standard WIC JPEG codec in Windows.

[claudermilk]

A slight, related, aside. Does anyone know a method or tool that shows you what WIC codecs you have installed are? I strongly suspect I am having some crash issues because of multiple codecs installed. I would love to clear things out and only have the current Microsoft libraw-based on installed.

[Mario]

The WIC diagnostcis in IMatch shows the installed WIC codecs.
And you should see the install packages (except for the built-in codecs) in the control panel, Add/Remove apps.