Very slow writeback of metadata

[Martin P Wilson]

I have just bought iMatch 2017 and imported my archive (~90k images) - took 7 hours. Fair enough, its a lot of data. I subsequently compacted the database.

I have found some images that do not have create dates so they sort inappropriately. I selected 8, modest jpgs no variations or the like, and edited the Date Created. I then clicked write back of metadata for selected files. It took over 20minutes with no other applications running.

I am using Windows 10 on a desktop with 16GB ram, AMD fx-8150 8 core processor and latest iMatch 2017 64bit, dowloaded yesterday.

I am assuming I am doing something wrong or my set up is flawed, I can't believe that is how it works! Any suggestions please?

I have tinkered with iMatch 5 in the p ast but not used it seriously.

I have just noticed that despite all that they they are still not showing the data I entered (a date in 2003) or they are showing todays date and time! I notice that there is a warning message saying that the metdata had not been written back. When I do it singly and click on the pen icon in the file window it takes minutes as well,

[Mario]

I then clicked write back of metadata for selected files. It took over 20minutes with no other applications running.

There is surely something badly wrong. ExifTool can write-back JPEGs at a rate of at least one file per second.
7 hours for only 90K files is also really, really slow. Unless you process massive RAW files over a slow network or something...

A typical cause for such a behavior is badly corrupted metadata.
Another thing that springs to mind is some sort of read/write-back 'loop' created by the meta in the file, the metadata import settings.

If your files already have metadata and the metadata is not properly synchronized between IPTC and XMP or your keyword import settings don't match your existing file contents, this may cause an infinite read > write-back > read > write-back > ... loop.


First thing:

1. Go to the Help menu > Support and enable debug logging.
2. Go to Edit > Preferences > Background Processing and make sure "Write back changes to metadata immediately" is turned off.

Now restart IMatch.

- Open the Info & Activity Panel to see if IMatch is still working on an import or write-back. Let me know what you see there (screen shot of the panel).
- Select some of your files and then press <Shift>+<Ctrl>+<F5> and then choose "Force Reload". This will reload all metadata in the file into the database.
- Make some changes (e.g. rating & label, maybe a title) and then click on the pen icon to write back the files.

Afterwards go to Help > Support > Copy Log File.
Attach the (zipped) log file.

Also, if possible, attach a zipped "problem" JPEG file or upload it somewhere I can download it. Or send it to me via email (with a link back to this thread).

[jch2103]

Just a general note in addition to the specifics Mario has outlined:

IMatch is one of the few programs that manages metadata effectively. If you've been using other programs, especially if your collection goes back a number of years, it's not unusual for your metadata to have a number of issues including the synchronization problem Mario noted. After I started using IMatch, it took me some work to fully clean up otherwise hidden metadata problems. The good news is that IMatch is the best tool I know to do this kind of cleanup.

[Martin P Wilson]

Thanks Mario

Weird it seemed to work OK this time! Just a momentary pause and the data was written, apparently correctly.

For 10-15 years I have mostly used PhotoMechanic for keywording etc and Capture 1 Pro for RAW conversion etc. I have tinkered with LR 4-6 but only as backup to C1Pro.

One of the reasons for going with iMatch was to tidy up my database and make it more manageable. My other tools are not really DAM.

I would just add that I have since added a couple of other folders (5K images) and it took 15minutes even with debug logging. They were all smallish jpgs but it suggests that my initial import did take too long, it seems 4-4.5 hours should have been nearer the mark even allowing for it mainly being of raw files.

[Mario]

Looks good to me. The slowest write-back was 2s, most write-backs take 0.1 to 0.3 seconds.

[Martin P Wilson]

Thanks Mario,
Any thoughts as to why it may have been slow? The metadata slow write I suppose could have been due to something running in the background after the iMatch restarting after compacting database?

[Mario]

I have no clue, sorry. With a log file from that session we could determine what was 'slow' but out of thin air, I cannot even guess. IMatch is fast with everything it does - generally.

[Martin P Wilson]

Seems to be running OK now. Oh well, I will just be happy that it seems to be resolved, however it happened!

Thanks for your help.

[Martin P Wilson]

May have spoken a bit too soon.

I had a lot of outstanding metadata write backs (15k) apparently to synchronise Subject, Keywords, HierarchicalSubject, saw the previous thread on the matter. Ran writeback over night and have around 4,400 left. I ran a handful and it seemed to work, the pen icon disappeared and a few seconds later reappeared!  Some disaappeared after 2-3 attempts but most not. I am working with a filtered batch.

Also I notice I now have a lot of CR2 raw files that do not show a thumbnail, that did last night. Some, not all will show in viewer. Also many do not reognise the prientation so are not showing in portrait format as they should. Even though images from same batch are fine.

I have just done it again with full logging. Tried it on both 10 .CR2 files and 8 .JPG files - same resuslts. Apparently writes them, pen icon disappers and a few seconds later the pen icon reappears.

[Martin P Wilson]

After Help hung up I had to close without waiting for program to respond. When I went back in I only had 199 waiting for metadata writeback so I did them. The pen icon vanished and then reappeared again.

Switched off logging and closed iMatch. Came back in and it was down to 100 waiting for writeback! I will see what happens if I try again! I can only assume that the writeback happens in the background for a long time after the dialog box closes.

Closed it again and opened it up and I am back to 199 awaiting metadata writes.

[Mario]

Check out https://www.photools.com/community/index.php?topic=4102.msg27513#msg27513 and https://www.photools.com/community/index.php?topic=2624.msg16925#msg16925 for potential reasons related to CR2 files.

Which apps did you use to create / update these files before?  Do they contain XMP data?
Do you have a sample file we can look at? CR2 and .XMP sidecar please.

You are writing about issues with synchronizing metadata, which usually indicates that your existing metadata is a mess.

Your log file shows no write-backs to .CR2 files, only .JPEG files are written. In 0.2 seconds or so, each.
No warnings or errors in the log file. Nothing indicating IMatch having a problem. Is this the log file from the session where you had to shut-down IMatch (Yikes!) manually?

[Martin P Wilson]

I will look at tthose threads

I guess it is possible that my metadata is messed up but I have had no problems anywhere else. Generally I use PhotoMechanic for keywording and all metadata and Capture 1 Pro for RAW conversion. It is extremely rare that I edit any metadata in C1, I go back to PhotoMechanic and resynch before raw conversion. I have had no issues with metadata until iMatch which of course may be because it is more comprehensive/ diligent. These problem are mostly from 2009 so I may have created a LR catalog that included them but did not really use it ( had a period where I had issues with C1 and needed to use LR as an interim solution).

Why do some .cr2 files not show thumbnails or if they do sometimes they show incorrect orientation?

IJ030002.cr2 - does not write metadata, nor does it show thumbnail but it will show in viewer
IJ270047.cr2 - does not writ emetadata, shows a thumbnail but oriented as landscape not portrait format.

[Mario]

I cannot say more without having at least one of your CR2 files here. Photomechanic and C1 both have sometimes unique ways to deal with metadata, and I'm not sure that both products apply the Metadata Working Group rules for properly synchronizing metadata between EXIF, IPTC, XMP and GPS. And if not this may causes issues you first run your files through a real DAM which does.

If IMatch cannot read a thumbnail from a file, the file may be corrupted. Or one of 50 other reasons.
Try to force a refresh using <Shift>+<Ctrl>+<F5> => Force Update.

Frankly, you seem to have more problems than 50 other users together.
I would need to see more data.

I need a log file from a session where you actually had problems with writing back. Make sure you enable debug logging under Help > Support.

Also, when you write back, open the Output Panel (View > Panels > Output) and switch to the ExifTool tab.
Then click the pen.
IMatch will display the ExifTool output in the panel, including all errors and warnings issued while writing the CR2 or XMP file.
Copy/Paste that into a text file and attach.

Send me a sample CR2 file which causes the problems, together with the .XMP file with the same name from the same folder, or upload somewhere.
Looking at your actual files will tell me what the problem is, most likely.

[Martin P Wilson]

I too am hoping this is one-off teething issues as I have had no issues with my other software.

I was a programmer before I went into Project Management/ IT Director roles so have some sympathy with the challenges. I don't usually have this level of bother getting to grips with new software...

I have sent a link to the cr2 files via support email.

[Martin P Wilson]

I have uploaded alog file to the download folder link I sent.

[Mario]

Great. Give me a few days. I currently have  a backlog of over 60 emails from yesterday and today...

[Martin P Wilson]

Thanks, that suits me as I am tied up on other things as well.

[Martin P Wilson]

I have done some testing of my own and removing the embedded XMP seems to do the trick allowing writeback of metadata, resolves both the lack of thumbnail and the incorrect orientation of thumbnails.

I am going slowly batch by batch to check I don't lose any metada especially descriptions and headlines.

It is why I wanted to use iMatch, I could not readily do this with the other tools at my disposal.