Fuji RAF from X-T2 and FPV Codec Pack

[tschilp]

I have an issue with RAW files from my Fuji X-T2, latest firmware 2.11 and with latest FastPictureViwéwer Codec Pack 3.8.0.97: IMatch is no longer able to read the RAF files. It shows the file with a size of 0x0 pixel and no picture.

The debug log file says:
Error loading D:\Fotos\Neue Bilder\_DSF0645.RAF with error 1 'No error reported.'  'PTPIImage.cpp(288)'
EUQH::Load(1.1) failed for D:\Fotos\Neue Bilder\_DSF0645.RAF (8/'No error reported.') with 0 x 0

I guess it is an issue with the FPV Codec Pack. The WIC diagnostics says:
Testing file 'D:\Fotos\Neue Bilder\_DSF0645.RAF'
   Thumbnail: Codec 'Fuji Raw Decoder (FastPictureViewer Codec Pack)'
      () 160x104 pixel in 0 ms.
   Preview: Codec 'Fuji Raw Decoder (FastPictureViewer Codec Pack)'
      () 1920x1280 pixel in 485 ms.
   Full resolution: Codec 'Fuji Raw Decoder (FastPictureViewer Codec Pack)'
      () 21079x6032 pixel in 16 ms.

The resolution for "Full resolution" is stupid ... It should be ~ 6000 x 4000.

Does anybody else have some similar issue?
@Mario: Am I on the right track? Is the FPV the issue?

With files from my older X-E1 everything is OK!

Update:
I just installed IMatch 2017.9.6 (64-bit)

Pictures a loaded now, but only in "Preview" resolution. Size although is given as 21079 x 6032 pixel.
Messages from log file:
PTPIP::Load failed with No error reported.  'PTPIP.cpp(2863)'
Loaded FullRes at 0x0
Failed to load large size, falling back to preview  'PTPIImage.cpp(292)'
ImageInfo: 'Format: Fuji Raw Decoder (FastPictureViewer Codec Pack)
    Embedded Color Profile: Adobe RGB (1998)'
EUQH::Load(1) of D:\Fotos\Neue Bilder\TMP\_DSF0645.RAF with 1920 x 1280 (O: 21079 x 6032) in 547ms

Regards
Tim

[Mario]

By default, IMatch creates cache images from the embedded preview in the RAW. See Edit > Preferences > Cache.

1920 x 1280 is a standard HD preview image, sounds about correct.
Some cameras embed previews in that size, not a 100% preview.
And then IMatch uses that.

If you want to force IMatch to use the full RAW for creating cache images, configure a minimum preview size of 2000 pixel in the cache settings. Then select the files and press Shift+Ctrl+F5, force update to re-generate the cache images from the full RAW. This may be 2 to 20 times slower than using the embedded preview and the results may be different from what you see in camera (not IMatch's fault, just how RAW works).

Maybe find a setting in your camera to embed a 100% preview image - if your camera is capable. This would allow you to use the embedded preview in IMatch for speed and accuracy.

[tschilp]

I don't know, if the cache image is the problem?

IMatch does not recognize the correct resolution of 6.000 x 4.000, but gives 21079 x 6032 as resolution like FastPictureViewer Codec. The viewer displays only HD with 1920 x 1280 for the new files from the X-T2 in 100% zoom. Images from the X-E1 are displayed in full resolution of 4.952 x 3.288 with the same settings. Also after doing a Forced Update. I will send the log file by personal message or e-mail.

LR btw. reads all file correct and also exiftool gives correct resolution. But FastPictureViewer not. And FPV is used by IMatch, according to WIC diagnostics.

[Mario]

IMatch does not touch your files. It processes them only through the WIC subsystem.
As the WIC diagnosis shows, there is something strange happening.  I doubt that your files are really 21079 x 6032 pixel.

1. Please contact Fuji about an updated WIC codec for your files.
I think Fujy does not bother to provide a WIC codec for their proprietary RAW files (anymore). Cost savings, you know...

2. Send a sample file to the FPV developers so they can analyze the file and see what the problem is.
3. Send a sample to Microsoft so they can update the built-in WIC codec for Fuji RAW formats in Windows 10.

Once you have a WIC codec installed which understands the RAF format your camera is using, IMatch and all other applications which support the official WIC standard will work.
It seems that every Fuji model uses a different proprietary format now. This will become a nightmare for Fuji users in a couple of years. Or they have to give into a life-long subscription for some Adobe software...

LR btw. reads all file correc

Yes, Adobe's market force is so strong that camera vendors surely supply them with information, under the table.
Adobe has a large team of developers in India which only reverse-engineer RAW formats. I cannot do the same.

[tschilp]

OK, you proofed my fear ... 

My guess was also, that these are bugs by FPV/Fuji (and Microsoft).

• I already contacted the FPV developers yesterday evening.
• In the past there was a WIC codec from Fuji, but I did not find it again. Seems to be forgotten ...

Thanks for your help and the confirmation of my findings. Unfortunately this is bad. I am really happy with the camera itself 

[Mario]

I am really happy with the camera itself

Surely. But the camera is only part the package. How's the support? Compatibility? Warranty? Camera vendor-supported codecs for Windows, Mac, Linux?

I would never buy a camera for which now WIC codec exists. When I buy the next body I will look into cameras which output DNG as their primary RAW format.  Even my smart phone does!
There is nothing in the NEF, RAF, CR2, SRF formats a DNG could not do. But DNG is an open and documented file format, nothing you can lock your customers in with.

I've recently learned that Canon stopped providing a WIC codec, in favor or their proprietary software (Digital Photo Professional software).
No WIC codec anymore, you have to install their software and then export the file to make it usable in other applications. Bugger. Bad for Canon users. Canon may save a man day or two in their development department by this. And apparently they did not inform Microsoft of the changes in their latest CR2 variants, because the built-in Canon WIC codec in Windows does not handle all variants...


All the vendors are pulling the net tighter, with proprietary formats, proprietary software, solutions which only work with a subscription or which force the use of some sort of cloud.
Trying to bind customers to their product worlds, making it hard or even impossible to freely choose the software you want.

I don't like this at all.  This will all fall on our feet badly in a few years time.

But users are to uninformed or just don't care as long as it's 'easy'. This is how vendors like Google etc. get the masses. Make it easy to give up your privacy and the control over your data, get something 'free' in return. Sigh...

[sinus]

I would never buy a camera for which now WIC codec exists. When I buy the next body I will look into cameras which output DNG as their primary RAW format. 

Maybe you know this article, it is quite interesting:

https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng

[Mario]

Consider dealing with your proprietary RAW files in 10 years time - when probably the only software supporting it something than runs in the Adobe cloud and requires you to pay for every month...

Besides, I'm not in the habit to 'go back' to old RAWs and process them again, years later.
Once used, I almost never look at the RAW again. Most of my output are composites in Photoshop.

[cthomas]

Consider dealing with your proprietary RAW files in 10 years time - when probably the only software supporting it something than runs in the Adobe cloud and requires you to pay for every month...

Besides, I'm not in the habit to 'go back' to old RAWs and process them again, years later.
Once used, I almost never look at the RAW again. Most of my output are composites in Photoshop.

This is all very scary!!!!!!!!    I wonder if we can really do any thing about this? Should I convert all of my RAWs(.CR2) to .TIFF or .PNG?

[Mario]

This is all very scary!!!!!!!!    I wonder if we can really do any thing about this? Should I convert all of my RAWs(.CR2) to .TIFF or .PNG?

Most people out there don't think about this because they are not aware of the consequences.
Libraries etc. forbid using RAW for long-term storage for good reasons.

If you are aware of the potential dangers of using proprietary RAW formats, you are good. Keeping a TIFF copy of your most important files is a good idea. TIFF exists for, what, 30 years now, is highly standardized and the base format for many long-term archival standards.

[tschilp]

With having these issues now and the comment from Mario, I remember one "idea" I had in the past:
Converting all my RAW files into DNG!

It slipped out of my mind, but I need to consider it again!

Fortunately it still can be done free and easy with the DNG converter from Adobe.
Refer to this article: https://helpx.adobe.com/de/photoshop/digital-negative.html

Software is Adobe  , but as Mario stated, the format is free! 

With the DNG files we are back at the codec (for windows) to load and display them. There is/was a codec from Adobe, but I was not able to find it at the offical page.  Fortunately there is an alternative link supplied in this blog (german):
https://www.fotoworkshop-stuttgart.de/dng-windows-anzeigen-ohne-lightroom/

But be aware: Pictures may look different from the original RAW. At least for my Fuji RAF the converter applies some camera profile which makes a difference. So at least more rework afterwards is needed ...

[Mario]

But be aware: Pictures may look different from the original RAW. At least for my Fuji RAF the converter applies some camera profile which makes a difference. So at least more rework afterwards is needed ...

Then your software has a problem. Software producing DNG files is supposed to embed an 'as intended' preview - which allows other software to display the DNG file exactly as you see it in that product.

The interpretation and rendering of the RAW data is of course highly volatile and depends on the software you use. Each software uses different algorithms and optimizations.
A WIC codec which extracts the RAW data will thus produce a totally different file than Lr or whatever RAW processor you use.

From the longevity standardpoint, DNG is worse than TIFF. Although based on TIFF (like most RAW formats) DNG has seen several, sometimes, breaking changes. Adobe adds to DNG whenever they need something for one of their products, and then all other software has to catch up. The last of these changes where the introduction of special cache images which Adobe needed for their Lightroom product.

[tschilp]

Maybe you know this article, it is quite interesting:

https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng

Just read it. Thanks sinus!

But what to do now and what format to use ... ?

If you are aware of the potential dangers of using proprietary RAW formats, you are good. Keeping a TIFF copy of your most important files is a good idea. TIFF exists for, what, 30 years now, is highly standardized and the base format for many long-term archival standards.

Looks, as I need to fall back to TIFF ...

[lnh]

I have the exact same camera, firmware and FPV codec pack. IMatch renders the (compressed) RAF files fine and faster under the 64 bit version. I believe the size reports as the embedded preview size correctly, but the whole expected resolution is present when processing the files in Capture One Pro. System is Windows 10 Pro.

[sinus]

Consider dealing with your proprietary RAW files in 10 years time - when probably the only software supporting it something than runs in the Adobe cloud and requires you to pay for every month...

Besides, I'm not in the habit to 'go back' to old RAWs and process them again, years later.
Once used, I almost never look at the RAW again. Most of my output are composites in Photoshop.

This discussion is always here. I have read a lot in the last years for RAW and against RAW.
But the same is guilty for and against DNG.

There is not a really "best solution", from my point of view.

The best maybe is use TIFF, like Mario wrote.
But reading a RAW in 10 years, I think, will not be a problem. Or a digital problem. I mean, it is nowadays normal, to look, what is going on.
Operations system changes, software changes, formats also.

I mean, even IMatch cannot go over this. I cannot use the new newest version of IMatch with Window XP, for example (I believe). Hence I must look for a solution.
In this case I have to update my OS, I think, and it is worth doing so.   

The same for example for Word. I cannot read an external word-doc with my old Word. Hence I must update ... or I am (mostly) out of the game.

If I have RAWs, and I have still a converter, a software to edit and store them, so I will be able to do this also in 10 years. Except I buy new stuff and throw all old software and hardware away.

I have an old scanner (UMAX Powerlook 3000), what was quite exepensive (once 10'000 Euros). This scanner is not more state of the art, as it was, but it works still.
But I have to use an old SCSI-cable-connection. And the new windows-systems does not support it also  (no drivers).

Hence I use it on an old computer (I think 15 years or so) and it works only with this old computer.

What I want say, we have to spot always a bit the market, to see, what is going on.

[Jingo]

I know there is no complete solution.. but this is why I create and only catalog high resolution JPG images now from my RAW files.  As Mario stated, I rarely (if ever) go back to my original RAW files and re-export them ... so, I consider my RAW files true negatives.. and my resulting JPG images as the "prints".  JPG's are fast, small and universal... I don't need to worry about versioning or embedded previews.

I've gone the Versioning route (NEF+JPG), DNG route (embedding a full resolution image) and the JPG route with IMatch.. and the easiest and fastest by far has been the JPG method.

Hope this helps in some way!

[sinus]

I know there is no complete solution.. but this is why I create and only catalog high resolution JPG images now from my RAW files.  As Mario stated, I rarely (if ever) go back to my original RAW files and re-export them ... so, I consider my RAW files true negatives.. and my resulting JPG images as the "prints".  JPG's are fast, small and universal... I don't need to worry about versioning or embedded previews.

I've gone the Versioning route (NEF+JPG), DNG route (embedding a full resolution image) and the JPG route with IMatch.. and the easiest and fastest by far has been the JPG method.

Hope this helps in some way!

Thanks.
Yep, I use jpgs also a lot. But only to say it, since we talked here also from "some years in the future", Mario pointed out once (some years ago  ), that also the format jpg has some restrictions with licence and law, so even this very widely format is not as sure as tiff is.

But I do not know a lot about this, wanted only mention it.

[lnh]

Seems like no format is ever sacred. JPG will likely move to the dust bin in several years now that new iPhones have replaced it with HEIC. HEIC isn't an Apple invention, but they are the first to implement it in a massive way. I'd expect other mobile phone vendors to follow, and you can't discount the market power of smartphones to set the agenda for the photo industry. In the not to distant future, JPGs won't be part of the casual user's workflow. On the video front, GoPro has moved the higher resolution modes in their new action cam to H.265/HEVC. None of this is to say JPG or MP4 is going away tomorrow, but the introduction of a new "container" format scares me as a new way to get frustrated with codecs all over again.

[sinus]

Seems like no format is ever sacred. JPG will likely move to the dust bin in several years now that new iPhones have replaced it with HEIC. HEIC isn't an Apple invention, but they are the first to implement it in a massive way. I'd expect other mobile phone vendors to follow, and you can't discount the market power of smartphones to set the agenda for the photo industry. In the not to distant future, JPGs won't be part of the casual user's workflow. On the video front, GoPro has moved the higher resolution modes in their new action cam to H.265/HEVC. None of this is to say JPG or MP4 is going away tomorrow, but the introduction of a new "container" format scares me as a new way to get frustrated with codecs all over again.

Yep, I guess, HEIF and HEVC will come in the future. They offer some interesting things of course, and I would think, that in the future this will be the "new jpg".
But of course it is a bit of "looking in the stars", we do not know it now really.

But poor Mario and other software-programer will have to deal with it. 

[lnh]

Yep, I guess, HEIF and HEVC will come in the future. They offer some interesting things of course, and I would think, that in the future this will be the "new jpg".
But of course it is a bit of "looking in the stars", we do not know it now really.

But poor Mario and other software-programer will have to deal with it. 

I just hope the new container format doesn't have two consequences...

1) The confusion that comes with today's container formats which lead everyone from casual users to experts to spend countless hours trying to figure out why that file with that known file extension won't play or be read by some random software you're using.

2) It is forbidden for any executable code to sit within the container. We have more than enough attack surfaces for malware, etc without adding another one. Of course the non-presence of executable code doesn't guarantee anything. If I remember correctly, Tavis Ormandy jail broke an early iPhone by having it only display a particular TIF file.

[Carlo Didier]

I would never buy a camera for which now WIC codec exists. When I buy the next body I will look into cameras which output DNG as their primary RAW format. 

Maybe you know this article, it is quite interesting:

https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng

I have been converting all my raw files from various camera makes (Konica-Minolta, Sony, Nikon, Olympus, Fuji, ...) to DNG for years now and never ever had the slightest problem.
The DNGs from RAF coming from the X-T2 display perfectly correctly in iMatch and give the correct metadata information, whereas the RAF files don't (size given as 1920x1080 for example).

For me, the biggest advantage, in an Adobe workflow with ACR or Lightroom, is the embedded full size preview that reflects my settings in ACR/Lightroom and which I can even quickly extract from the DNG if needed (using exiftools for example). And iMatch picks that up too, so the images display in iMatch "as developed".

As to the argument about backups having to re-backup the whole file if only some bytes of metadata have changed, I can only say that that isn't a problem with modern backup software which is intelligent enough to only backup the parts of files that have changed, not the whole files. If I had to backup nearly 5GB of iMatch database each time I change a category assignment on a few photos, that would be ridiculous too, wouldn't it?

[Jingo]

I would never buy a camera for which now WIC codec exists. When I buy the next body I will look into cameras which output DNG as their primary RAW format. 

Maybe you know this article, it is quite interesting:

https://photographylife.com/why-i-no-longer-convert-raw-files-to-dng

I have been converting all my raw files from various camera makes (Konica-Minolta, Sony, Nikon, Olympus, Fuji, ...) to DNG for years now and never ever had the slightest problem.
The DNGs from RAF coming from the X-T2 display perfectly correctly in iMatch and give the correct metadata information, whereas the RAF files don't (size given as 1920x1080 for example).

For me, the biggest advantage, in an Adobe workflow with ACR or Lightroom, is the embedded full size preview that reflects my settings in ACR/Lightroom and which I can even quickly extract from the DNG if needed (using exiftools for example). And iMatch picks that up too, so the images display in iMatch "as developed".

As to the argument about backups having to re-backup the whole file if only some bytes of metadata have changed, I can only say that that isn't a problem with modern backup software which is intelligent enough to only backup the parts of files that have changed, not the whole files. If I had to backup nearly 5GB of iMatch database each time I change a category assignment on a few photos, that would be ridiculous too, wouldn't it?

As mentioned, I went this route as well..  a big disadvantage though - if you embed your metadata into your images are huge backup times... since DNG files with large embedded previews can create very large files - any metadata changes that are written back to the DNG require backup and that can be time consuming depending on your system.  Once I moved from DNG's over to JPG only.. my system backups went from 5 hours down to 20 minutes.  Not a major deal since these were done overnight anyway.. but something to consider.

[Mario]

This is a no-problem with modern 'maging' backup software (Macrium Reflect, TrueImage). When using diff or incremental backups, they only backup the changed portions of the file.

This means that changing 10 bytes in a 100 GB file results only in 512 bytes or 4KB in the backup (depending on whether it works with sectors or clusters).
Only when you use file-based backup software, the 100GB need to be backed up again. But that's old-fashioned.

[Jingo]

Interesting... but - using these methods.. you are locking into a software vendor as well?  ie: you must use their software to recover the files and you cannot see them using your OS?  Perhaps that is not an issue any longer - but I always preferred being able to access my backups using the OS for instant recovery - without relying upon a proprietary compression.

[Carlo Didier]

As mentioned, I went this route as well..  a big disadvantage though - if you embed your metadata into your images are huge backup times... since DNG files with large embedded previews can create very large files - any metadata changes that are written back to the DNG require backup and that can be time consuming depending on your system.  Once I moved from DNG's over to JPG only.. my system backups went from 5 hours down to 20 minutes.  Not a major deal since these were done overnight anyway.. but something to consider.

Apparently you didn't read what I wrote about backups:

As to the argument about backups having to re-backup the whole file if only some bytes of metadata have changed, I can only say that that isn't a problem with modern backup software which is intelligent enough to only backup the parts of files that have changed, not the whole files. If I had to backup nearly 5GB of iMatch database each time I change a category assignment on a few photos, that would be ridiculous too, wouldn't it?

[Carlo Didier]

Interesting... but - using these methods.. you are locking into a software vendor as well?  ie: you must use their software to recover the files and you cannot see them using your OS?  Perhaps that is not an issue any longer - but I always preferred being able to access my backups using the OS for instant recovery - without relying upon a proprietary compression.

Well, if you want to call it a backup and not just a copy, you need some sort of software to manage it. Backup software, as Mario wrote, allows to make incremental backups where only differences are backed up (originally whole files that had changed, nowadays only the changed parts of files), which in turn allows you to have more frequent backups (like every hour) and keep them longer (for years) without needing too much space.

[sinus]

Carlo
if you have the time, it would be interesting for me, if you (roughly) could write, how your workflow is with a Nikon, what produces Raws.

Just out of curious, because some years ago I tended some time to go your route.
And if it turns out, that this is maybe the better way, then I can always change.

[lnh]

Getting back to the Fuji RAF's...

If you're a user of most Fuji X cameras, it's likely you're using one with a X-Trans based sensor rather than the more prevalent Bayer array. There are endless arguments about X-Trans, but the bottom line is not all RAW converters are created equal and that is particularly true of RAFs from X-Trans especially through the demosaicing stage.  Iridient X-Transformer is considered by many the gold standard, and the only thing this does is make RAFs into (large) DNGs. I'll use it for an occasional image, but normally rely on Capture One which does a very good job with RAFs and has a very good workflow. I know few people ever go back and redevelop a RAW file, but there is no denying that these RAFs have benefited from advancements in RAW converters and baking a file into a DNG will deny you of those advancements. I guess you could always embed your RAF completely inside a DNG, but then very large files become ultra enormous.

[Carlo Didier]

There are endless arguments about X-Trans, ...

Right, but I researched a lot (Google and Youtube are your best friends) and for me, ACR (i.e. Lightroom, latest version) do a perfectly adequate job now.
I haven't had my X-T2 for long now, but from my first experiences, I have no reason to switch to any other RAW converter or abandon the DNG workflow. But that's for me. Any anyone must find his own Raw heaven. And I'm converting to DNG with Bridge/ACR, not with Iridient (which would largely complicate and slow down my workflow).

[Carlo Didier]

if you have the time, it would be interesting for me, if you (roughly) could write, how your workflow is with a Nikon, what produces Raws.

Markus, it's been the same for years now, be it for Konica-Minolta 7D, Sony A700, Nikon D600, Olympus E-M10 II or Fuji X-T2 (and several others that were not important):
- in case I have recoeded one, copy GPX file to PC
- read images from card with Breeze Downloader Pro which combines them with the coordinates from the GPX file(s) and renames them to a first naming convention
- browse/edit/cull/rate with Bridge and ACR
- rename the remaining to the final naming convention (may also be done later in iMatch instead)
- convert to DNG and save those in the final folder(s)
- iMatch detects and adds new images automatically
- add categories to images in iMatch (I don't use keywords and I don't write categories to the files)
- where applicable, later I stitch panoramas, HDRs, etc mostly in ACR, sometimes in Photoshop. Those get named so that my event scripts (which will never exist in IM2017!) in iMatch can automatically detect them as panos, HDRs, etc and set the corresponding categories on the composite images and on their parts and create stacks with the composite image on top. The scripts also set categories for the file formats (landscape, portrait, pano, square, ...) according to MY criteria for the ratios. They do some other stuff that mostly cannot be done in IM2017 anymore, at least not automatically.

My whole workflow is based on my naming convention and I'll never change that, not with 80000 images that conform to it, because I would either have to run two naming conventions in parallel or rename all 80000 images and get a hell of a time to find old images through their old names when I need them. It's all been working fine for years now, so why change it? I miss nothing that could force me to go that way.

[sinus]

Thanks, Carlo,

for your detailed workflow.
I will "study" it deeper a bit later, but you are correct of course, why change, if it works fine!?