Something seriously wrong with what I am doing here!

[eyemack]

I'm a new user of IMatch, and there is clearly something I am doing that is very wrong.
I go through this cycle every day.
1) I notice that files have the message as per attached 'this file has unwritten metadata...'.
2) I then select Commands>Meta Write-back>For all pending files. Usually says thousands of files are pending. I click on OK. Then after some time all looks good.
3) then OneDrive sync goes mad as all these files look modified and have to be re-uploaded.  (That is a side issue though and not relevant to IM.)
4) then, sometime later, I re-launch IM, it says 'Importing Metadata...' of the thousands of files - as per attached.  That completes, and then all the flags re-appear on the files and I'm back to 1) again!
What on earth am I doing wrong here?

[jch2103]

Welcome to IMatch! 

My guess is that you have conflicting IPTC and XMP metadata, based on your 2nd screenshot. See https://www.photools.com/help/imatch/md_for_beginners.htm?dl=hid-43 for an overview of this issue. See in particular

Code Select Expand

In this case it usually helps to manually remove the legacy IPTC keywords from the image once in the ExifTool Command Processor. Then let IMatch write-back, which produces a nice and clean and synchronized set of keywords in all three sets.
The Metadata Analyst app included in IMatch checks for out-of-sync keywords. If you experience any issues, run it on the file causing the problem to check for this and many other problems.
Let us know if you have more questions. 

[eyemack]

Thank you. I'll have a look in the morning, but it is largely jpg files.

[PandDLong]

You can't have OneDrive synching folders that are "managed" by iMatch.  It causes all kinds of issues - perhaps this one.

I started out with OneDrive synching the folders and I many "can't update metadata" errors when writing-back.  Mario pointed out the help documentation says to not have any background processes trying to sync or backup those folders - my miss (there is so much to read...)

Hope that is helpful.

Michael

[PandDLong]

I just read John's reply.  That also looks probable as a source of problems.

Good luck.

Michael

[thrinn]

Is it possible that OneDrive modifies your files? In this case, IMatch would identify them correctly as "updated". Then you perform a write-back. OneDrive sees that the files have changed and synchronizes them again. This could lead to a loop.

I found this post OneDrive randomly re-creates pictures, some metadata is lost in the process - if this really happens it would explain what you experience, even in the case of JPGs.

[Mario]

My first bet would be mismatching keywords in legacy IPTC and XMP which cannot be resolved automatically.

You can run the Metadata Analyst on the file in your screen shot. It will tell you if the keywords differ. If they do and IMatch cannot resolve the difference itself due to the mapping settings and thesaurus you use, you will end up in a write-back loop.

The easiest way to deal with this is to remove the legacy IPTC information from your files - unless you really need them for a client or a service you use. IIM IPTC is legacy, having been replaced by XMP 20 years ago. XMP is better in any way. IMatch has imported the legacy IPTC data into XMP automatically.

Do this:

Select a number of files and make sure you have written them back at least once.
This ensures that the files contain XMP metadata.

Now run the ExifTool Command Processor from the Tools menu and select the "Delete legacy IPTC (IIM) metadata" preset and run it with <F9>.

Close the ExifTool Command Processor. IMatch will rescan the modified files automatically.
If the files still show as pending, write back once.

[eyemack]

Thanks to you all for the replies.  Sorry, about late response, but these processes take a long time to complete and assess what's going on.
So, I followed Mario's advice and ran the ExifTool Command Processor, deleting legacy IPTC metadata.  Although I'm not sure where that would have come from.  My library is all from Lightroom.
Anyway, this hasn't worked. Still going through the exact same cycle as I posted originally.  I Writeback Metadata (to thousands of files), OneDrive syncs, I go back into IM and there are thousands (always different number) more with 'pending metadata'.
This really is driving me crazy!
I shold also say, that IM is almost always chugging away 'Reading Metadata' in the bottom right corner.

[Mario]

The screen shot in your post shows clearly that IMatch want to write XMP and legacy IPTC keywords. If this repeats, the cause is always out-of-sync legacy IPTC and XMP keywords. Deleting the legacy IPTC data should have fixed this since there can no longer be out-of-sync keywords since there is no legacy IPTC data anymore.

Can you reproduce this problem with files stored in a folder on your hard disk?
Just copy a folder with files from OneDrive to your C:\disk (e.g. c:\images) and then add it to your database from there.
This rules out any interference from OneDrive .

You might have to delete the legacy IPTC data again for these copied files to be sure.
Tip: you can check if this was successful by running the ExifTool Command Processor with the "List Metadata" preset for a file and then searching for [IPTC].

If there is a pending write-back, write back. This should work fine, because legacy IPTC data no longer interferes.

If this works (it should), we know that this has something to do with OneDrive.
Maybe OneDrive is restoring your files after IMatch has written them?
Maybe OneDrive is messing your files up by modifying metadata?

I have some folders with images and other data in OneDrive for testing purposes.
Some of the images have legacy IPTC data. I see no pending write-backs. When I add some keywords in IMatch, they are written  fine and the pen does not come back.

[eyemack]

OK, so I don't know if it was a good thing to do, but here is what I tried.
I copied a folder of 19 images, half of which had the issue in IM as we have discussed, to a non-OneDrive folder and created a new temp database with just that folder in it.
I did the 'deleting legacy IPTC data' and a 'pending write-back'.
At the moment, all is good. No recurring issues.

[eyemack]

To follow up, I just noticed your check to see if legacy ITPC data has correctly been removed, and the answer seems to be no, it hasn't as loads for those 19 images show in the result panel.  See attached.

[eyemack]

To follow-up on the follow-up.  I ran the 'delete legacy data' again, and this time all IPTC was gone from the results.  However, a minute or two later the files got modified, the write-back pencil came back on all them, they were all write-back pending, and there was IPTC data showing in the 'ListMetaData' command window, etc. etc., so looks like it is happening even in non OneDrive folders.

[Mario]

To follow up, I just noticed your check to see if legacy ITPC data has correctly been removed, and the answer seems to be no, it hasn't 

If ExifTool fails to write the files or to remove legacy IPTC data, it will report errors and warnings in the ExifTool Command Processor. Did you check that?

However, a minute or two later the files got modified,

How did the files get modified? IMatch will rescan the files automatically when you close the ExifTool Command Processor. Since IPTC metadata was removed, the files should be pending. Is this what you mean? But for only 19 files, this will be virtually instant, not somehow happening a minute or more later.

You should see no IPTC tags in the wite-back pen tooltip, correct?
IMatch does not create new IPTC legacy data, it only maintains existing legacy IPTC data when the target file already has.

Can you run the Metadata Analyst one of these problem files, and if errors or warnings are reported, use the BIG GREEN button at the top to copy them into the Windows clipboard and paste them here.

[eyemack]

OK, so here is what is happening now.

1) All 19 jpg files (in temp folder/database) showing Write-back pencil with 'List of tags to write – IPTC::ApplicationRecord\Keywords' in pop-up
2) Run Exif Tool Command Processor on the 19 images to 'delete legacy IPTC metadata'. It says '19 image files unchanged'
3) Then run 'Write-back Metadata', dialogue says '19 files pending'.  Completes OK.
4) Write-back pencils all gone from files.
5) I go back and run ExifTools Command with 'List Metadata' and there are IPTC references throughout.  See attached IPTC.jpg.
6) Run Metadata Analyst and get this (and as per attachment analyst.jpg):
   Metadata Analyst Results. Version 2023.8.6. 4/12/2024 4:26:07 PM
   File analyzed: C:\Temp Photos Library\IMG_1937.JPG
   Errors: 0
   Warnings: 9
   Warning: [Legacy IPTC] Character Set Encoding: unspecified.
   Warning: [Detailed Validation] [minor] Non-standard IFD0 tag 0x4749 RatingPercent
   Warning: [Detailed Validation] [minor] Non-standard ExifIFD tag 0xea1d OffsetSchema
   Warning: [Detailed Validation] ExifIFD tag 0xa432 LensInfo requires ExifVersion 0230 or higher
   Warning: [Detailed Validation] ExifIFD tag 0xa434 LensModel requires ExifVersion 0230 or higher
   Warning: [Detailed Validation] ExifIFD tag 0x9010 OffsetTime requires ExifVersion 0231 or higher
   Warning: [Detailed Validation] ExifIFD tag 0x9012 OffsetTimeDigitized requires ExifVersion 0231 or higher
   Warning: [Detailed Validation] ExifIFD tag 0x9011 OffsetTimeOriginal requires ExifVersion 0231 or higher
   Warning: [Detailed Validation] ExifIFD tag 0xa430 OwnerName requires ExifVersion 0230 or higher
7) There is IPTC data showing in the text above and the attached graphic.
If I run the 'delete legacy IPTC metadata' again, it runs OK, but then, after some time, the Write-back pencil icons re-appear on all files, and we're back to square one.
I really don't know what is happening!

[Mario]

Run Exif Tool Command Processor on the 19 images to 'delete legacy IPTC metadata'. It says '19 image files unchanged'

This would mean that there was no legacy IPTC data in the files and so ExifTool had nothing to do?

The ExifTool Command Processor (ECP) output in your image, is that for one file or did you select multiple files?
Because I see multiple IPTC records there, with different keywords.

Can you send me one or two of the files (with a link back to this thread) to support email address?

The MDA shows that legacy IPTC data was found in the file. Did you run the MDA before or after deleting the IPTC data with ExifTool?

Maybe I can tell you more when I have some of these troublesome files here in the lab.
This is something new. Re-appearing IPTC data? Strange.

[eyemack]

ECP output was all 19 images.
MDA was after deleting IPTC data with ExifTool.
I've sent the files to you by email.
Thanks again.

[PandDLong]

I had the situation where the legacy keywords were persistent - even after running the ECP preset - and of course, one legacy tag means the whole block gets recreated at write-back time.  And I was caught in a cycle.

https://www.photools.com/community/index.php/topic,13997.msg99163.html#msg99163

My solution was to have a metadata template that specifically deleted the legacy keywords and then I had a 2 step process:

1. Run the ECP preset
2. Apply my metadata template

Then everything was clean and I had no further issues. 

Michael

[eyemack]

Is the template something you can share?

[eyemack]

Although Mario hasn't got back on this yet, the more I see here, the more I think the issue is down to the ExifTools Command preset not deleting the IPTC data, just like in the thread linked to above.

[Mario]

Thanks for sending the files.

I probably did not mention this (? my bad) but you have to do a forced update (Select files, pres <Shift>+<Ctrl>+<F5> and then select Force Update) to solve this after deleting the legacy IPTC data.

Because, if you import a file with IPTC data into the database, the database contains that IPTC data. So IMatch "knows" that the file has legacy IPTC and it will consider that when a rewrite of any kind is triggered.

You've send two files. The file with the pool has legacy IPTC data, the other file has none.
I've ran the ECP with the Delete legacy IPTC data preset and verified that all legacy IPTC data is gone.

After closing the ECP, IMatch reloads the file since it has been modified. IMatch still thinks that the file has legacy IPTC data (the situation where an entire IPTC record is stripped in an external tool is not really expected) and "reproduces" the IPTC:Keywords tags by internally mapping from XMP:subject to IPTC:keywords. I shall make a note and look into this for some possible improvement at some point in time.

When I now select the file and do a forced rescan, IMatch wipes all metadata and other data for the file from the database and replaces it completely, like it would for a new file. No trace of the previous legacy IPTC data remains.

Now, adding a keyword produces the expected result:



Only XMP data is marked for write-back.

What you have to do is:

Select the problem files in a File Window.
Run the ECP with the Delete legacy IPTC data preset.
Close the ECP.
Press Shift+Ctrl+F5 and use the "Force Update" command.

[eyemack]

Thanks Mario, I'll give that a go.
However, I'm a little confused that you say the 2nd image doesn't show IPTC data. It does for me. See attached.

[Mario]

For my version of the file, neither the MDA nor ExifTool report any legacy IPTC data for the file.
All I see is:

Code Select Expand

[XMP-microsoft] Last Keyword IPTC             : Holidays, Cyprus
[Photoshop]    IPTC Digest                    : 1bc3c8304a9b3287cc79e91bdfca276d
That's for the file straight from the email saved to disk.

[eyemack]

Still not working.
These are the steps I am doing (on the temporary 19 file database):
1) select all files
2) run Meta Writeback. 19 files are processed.
3) run ECP with delete legacy IPTC metadata preset. Window shows 19 files updated. Close ECP. No pencil icon showing on files.
4) do a 'force update'.
5) then, all pencil icons are back, showing as per attached.

[eyemack]

Your last post is interesting.
If I run MDA on the file I sent you, then like you, I see no IPTC data.
However, if I run it on the file I copied it from, then I do!

[eyemack]

OK, so sorry to bombard you with posts here, but it is when IM auto updates, or a 'force update' is done that the pencils come back.  If I (quickly!) try and do a MDA on a file before the update occurs, then the IPTC data is not there.  But, whenever the update occurs, the pencil is back and so is the IPTC data.

[Mario]

I don't understand why you get different results for the same files.
The workflow I have provided will work. I did it with the pool file, no IPTC data remaining in the file and after the forced rescan, no legacy IPTC data remains in the database. You can verify this via the "Browser" layout in the Metadata panel. It will show IPTC:Keywords initially, and this tag will be gone after the forced rescan.

[Mario]

Maybe give IMatch a few seconds to do the initial rescan after closing the ECP. That the pen comes back in this case is normal.
Then do a forced rescan of all files. Then there will be no legacy IPTC data in the file or the database.

[eyemack]

Sorry Mario, it's just not working. My post #22 steps are reproducable every time with the same IPTC Keyword caption as in the attached above.
I don't know if it's something you do, but I would be happy to screen share for you to see.

[eyemack]

Maybe give IMatch a few seconds to do the initial rescan after closing the ECP. That the pen comes back in this case is normal.
Then do a forced rescan of all files. Then there will be no legacy IPTC data in the file or the database.

Tried that as well, but no joy.  Pencil with IPTC::Application Record/Keywords just keeps coming back or doesn't go away following force update even after IM does its inital scan.

[Mario]

Try setting all protection options except rating to off under Edit > Preferences > Metadata 2.
Then repeat.

[eyemack]

Nope, still the same pencil with IPTC::Application Record/Keywords message. Just in a loop here and not getting anywhere!
As I said above, if you could spare a few minutes to share my screen to see what's going on, that would be great.
I'm trying to make this work, but at the monent, IM isn't usable for me.  Let's hope we can resolve this.
I have seen another few posts with the same issue in a search for 'exiftools not deleting IPTC metdata'.

[Mario]

I've sent you the two images back.
The one with the house had no legacy IPTC data, but I've added a keyword and a description.
The pool image had legacy IPTC data removed by ECP, then I've added a keyword and a description.
Both files still have no IPTC data.

I did this twice, once with my regular database and again with a fresh database created only for this purpose. In both cases it worked as described above (remove legacy IPTC data, force rescan, legacy IPTC data gone).

Just to be extra sure, I've installed a fresh copy of Windows 11 in a virtual machine, installed IMatch 2023..8.5 (current trial version) trial, created a database and indexed the folder with your original files. Just in case something really weird is going on.

After indexing the images, I verified that the pool image has legacy IPTC data and the house image has none. Both with the MDA and ECP.

1. Removed the legacy IPTC data from the pool image.

2. Did a forced rescan.

3. Pool image shows pen and wants to write back IPTC:keywords

4. Disabled metadata protection for unwritten metadata (Edit > Preferences > Metadata 2)
If this is on, IMatch will not wipe IPTC:Keywords since it is marked as modified.

5. Forced a rescan of the file again.
Legacy IPTC data is now gone from the database and the file shows no pending write-back.

In all three tests, I get the same and expected results.
Did you do a forced rescan after disabling protection?

Now I've also tried to reproduce your original problem with the file.
Restored the original pool image from backup. With legacy IPTC data.
Added the keyword IMatch in IMatch.
Write back the file.

The pen vanished.
The file contains a consistent set of keywords in XMP and IPTC:

Code Select Expand

[IPTC]          Keywords                       : Holidays, Holidays|Cyprus, IMatch
[XMP-dc]        Subject                        : Holidays, Holidays|Cyprus, IMatch
[XMP-lr]        Hierarchical Subject           : Holidays, Holidays|Cyprus, IMatch
so removing the legacy IPTC data is actually not required at all.
At least not for the sample image I have here.

It may be needed if your other files contain unresolvable differences in keywords between legacy IPTC and XMP.

[eyemack]

Firstly, I'm not on the trial version.  I was so impressed by IMatch that I bought a licence a few days ago! I'm on 2023.8.6.
Anyway, we are making progress. The 19 image database is now working OK.  With the Metadata 2 optin set and a force update all is looking good. No IPTC metadata showing in a MDA run.
I'm going to try it now on my full database. Fingers crossed!
I'll report back.
Thanks again.

[PandDLong]

Is the template something you can share?

Certainly.

Here is a screenshot - it is just the one line in the template. 

Sounds like you are making progress so you may not need it.

Michael

[PandDLong]

For my version of the file, neither the MDA nor ExifTool report any legacy IPTC data for the file.
All I see is:

Code Select Expand

[XMP-microsoft] Last Keyword IPTC            : Holidays, Cyprus
[Photoshop]    IPTC Digest                    : 1bc3c8304a9b3287cc79e91bdfca276d
That's for the file straight from the email saved to disk.

I think the IPTC Digest tag is problematic - as I understand the namespace standards it is a hash value of the legacy IPTC data - perhaps this generates the action to recreate legacy data.

As per Phil of Exiftool it is a problematic tag and should be deleted. 

It may be part of the problem - if this isn't solved with all of the other advice.

Michael

[Mario]

The OP has solved the problem. After disabling metadata protection, a forced rescan of the file wiped all legacy IPTC data from the database.

[eyemack]

So, reporting back.
It's taken a long time, as these processes - on only 15000 images - take soooo long. I hate to think how long they would take if my database was hundreds of thousands!
After 6-7 hours of processing yesterday on the main dabase, the IPTC issue seems to have been solved.  However, a new, similar issue has now arisen.
This is a shortened version of what has happened since yesterday's post.
1) Using the main database seemed to go ok. Process was horrible with tasks, 'reading and writing metadata', 'updating and adding files', 'forcing updates', etc. taking hours. So when I went to bed all seemed to be calm and sorted.
2) This morning I loaded IMatch and a whole process of 'adding and updating files' started up again.  This again took a long time.
3) Then a whole bunch of pencils appeared. All with the caption XMP::dc\Subject. (See attached.)
4) So, then did the usual Write-back for pending files. About 3,200 were written to.  Then we were back to the painful 'adding and updating files' process as above and all the time this takes.
5) Then OneDrive resyncs, etc.
6) This process happens every time I load up IM and I have to go through the same cycle over and over. Just a few minutes ago, there were about 1200 files with the pencil and the process starts again. 
Apart from the enormous amount of my time this is taking up, there is about 40-50Gb of data being added to my daily incremental backups for all these changes. I appreciate that Mario has also spent a considerable amount of time on this too, but I'm reaching saturation point on all this.
Any ideas on the recent issue?

[Mario]

It is unusual that only XMP:subject needs to be written, since this tag is linked to XMP::lr hierarchicalSubject, and both tags are always updated by IMatch at the same time (e.g. when you make changes in the Keywords Panel).

So, unless you manipulate this tag directly somehow (behind the back of IMatch) (do you?), there is no situation in which only this tag needs to be written. It's always XMP hierarchicalSubject and  Subject. At least I don't know a situation where this would not happen.

What is also unusual is why you have to start whatever process over and over.
Usually this works like this:

1. Images are added to the database.
IMatch produces rich XMP data from native EXIF, GPS and legacy IPTC data found in the file and stores it in the database.
This causes the file to become pending for write-back.

2. User writes the file back once.

Done.

That's it.
Only if there are badly out-of-sync keywords in legacy IPTC that contradicts the keyword in XMP (also influenced on your IPTC import options, keyword flattening preferences and thesaurus), the pen may come back and user-action is required. But that's actually quite rare.

Not sure if you storing files on OneDrive causes all this. If OneDrive changes the file timestamps or even modifies the metadata in the files "for your protectino", IMatch will detect that the files have changed and re-import them. I have files on OneDrive but I have never seen this happen.

Maybe copy a folder of your images to your C: disk and create a new database and index only the test set of files. See if this now behaves like IMatch behaves for every other user.

[eyemack]

Thanks Mario.
No, I'm not manipulating anything in IMatch or behind its back! All I want is to get this working.
I am beginning to suspect that OneDrive has something to do with this.  I have turned off all syncing and IMatch seems to be behaving OK.
I will test this out for this afternoon, and if everything is OK, I will restart OD syncing and see if the issue comes back.
I'll report back later.

[eyemack]

So, everything is working well this afternoon. No background processes churning away all the time.
I'll switch OneDrive sync back later and report back.

[eyemack]

All messed up now. Thousands of XMP:subject pencils back and going through the whole cycle as previously described.
Clearly, this is all being caused by OneDrive syncing.
What can be done about this?  I need to be able to access my Pictures folder from all devices.  This has never been an issue in the past.

[Mario]

This would insinuate that OneDrive is not only modifying the "last modified" timestamp in the file system (causing IMatch to rescan the file) but also somehow magically adding a XMP flat keyword? Never heard about that before.

Why would OneDrive synching cause the last modified time stamp on the PC that contains the IMatch database change? I assume you sync from that PC to others? When you sync from other PCs to the PC with the IMatch database and OneDrive changes the "last modified" timestamp on disk, IMatch will rescan the files. It has to.

IMatch internally only works with hierarchical keywords. And when a hierarchical keyword is changed, it copies hierarchical keywords into XMP::subject to sync them in the database. But IMatch never modifies XMP::subject all by itself. Unless the user explicitly changes this tag.

Do you have configured a Metadata Template that runs when IMatch rescans a file? Does this template set XMP::subject?
That would the only logical explanation from my place.

This is all very obscure. Never heard about this before.
Maybe clear with Microsoft first what OneDrive is actually doing. Documentation of the workings of OneDrive is sparse.

[eyemack]

I don't have any kind of template. I am doing nothing except watching all these processes constantly chugging away.
I have created a test folder with about 20 of the problematic images. Btw, there is only about 2,000 with the issue and about 12,000 without.
I am syncing the test folder to OneDrive to try and see what is going on.
ATM, I feel that OD is downloading these files on sync rather than uploading the revised by IM ones. Hence the older ones are being updated by IM and the issues begin. 
All incredibly frustrating.

[thrinn]

Just a shot in the dark: Do you have by any chance something like "Automatic tagging" activated in OneDrive?

[Mario]

Oh, jeez. Good point.

Found this: https://support.microsoft.com/en-us/office/organize-and-find-photos-in-onedrive-6a9b0298-f504-4992-af0e-45e2f270afc9

Maybe OneDrive is really modifying the metadata in your images without you knowing? Adding keywords to XMP::subject?

[eyemack]

Well, looking at one image in ECP shows this:
[XMP-dc]        Subject                         : Holidays, Holidays|Devon
[XMP-lr]        Hierarchical Subject            : Holidays, Holidays|Devon

@thrinn Can't see any setting like 'automatic tagging'.

[eyemack]

So, the test folder containing about 20 of the problematic images is behaving correctly.  Syncing fine with OD, and IM is happy too.
What does that tell us?
I'm thinking what If I copy the entire contents of my main photos folder, delete it from OD and start again. Thoughts?

[thrinn]

@thrinn Can't see any setting like 'automatic tagging'.

You will see it (only) if you open OneDrive on the web. Then go to "Settings" (icon at top right), "Options", "Photos". You should see a setting like Photo Tagging. (I have the German version of OneDrive, so I have to guess at the exact English terms).

[eyemack]

@thrinn Can't see any setting like 'automatic tagging'.

You will see it (only) if you open OneDrive on the web. Then go to "Settings" (icon at top right), "Options", "Photos". You should see a setting like Photo Tagging. (I have the German version of OneDrive, so I have to guess at the exact English terms).

OK, so thanks for that.  I'll try switching that off.

[eyemack]

So, there seems to be good news here.
I moved my main photo folder to another drive and away from OneDrive - hence it was deleted from OneDrive, I  sorted out all the pencils in IM, and had a stable situation there.
I then - with OneDrive sync off - moved the folder back to the main Pictures system folder, but called it a different name just in case there were any legacy things kicking around in OneDrive.
I loaded up IM, and the database was still behaving nicely.  So, I restarted the OneDrive sync, it eventually uploaded all 100 Gb of photos, and so far all is working well with IMatch.
I will report back tomorrow on whether this was a temporary thing or not!