Unchanged metadata (XMP face region) needs write-back ?

[Jean-Maetso]

Hi all,
When I datect and recognize faces, IMatch create a rectangle which is stored in the database and is exported as XMP-MWG metadata. As a consequence, like any other metadata changed, IMatch displays the pen icon in the File Window to specify that this metadata (MWG:Regions\RegionInfo) should be written-back to the file.

When I click on the pen icon, the file is updated (i can check this with ExifTool and -XMP-MWG-rs:all option). As expected.

But, if I reload the metadata (CTRL + SHFT + F5, 3rd option), the pen icon appears again, as if it has changed. Which is wrong, no action has been done on the image. As a matter of fact, if I write back the MD again and check with Exif Tool, I see the tags are unchanged. And if relaod again, the pen icon comes back...

Why can't I get rid off the pen icon ? For any other MD, once the metadata has been written to the files, and as long it is not changed, it does come up again when you refresh (CT + Shft + F5) the file....

Any idea around ? Thanks !
Best,
JM

[axel.hennig]

Sounds like "metadata mess". I think we need an example file where this happens and the exact steps how to reproduce.

[Mario]

When I click on the pen icon, the file is updated (i can check this with ExifTool and -XMP-MWG-rs:all option). As expected.

IMatch automatically re-imports metadata after a write-back.

Why do you run a forced Metadata Reload, which wipes all metadata from the database and re-imports everything as "fresh". This very likely will create "new" timestamps, digests etc., causing the file to become pending.

Check the tooltip of the pen to see which tags IMatch wants to write.
Check the The ExifTool Panel after the write-back and see if there are any warnings or errors reported.
It is always helpful to include the IMatch log file from the session where you encountered an issue (see log file)

For general tips about how to fix metadata mess, including IMatch having to write back multiple times to set things straight or IMatch running into a write-back loop that can only be fixed by the user, see Metadata Problems and Pitfalls

[Jean-Maetso]

In fact this happens to me with any image.
If there is a face on the image, the tag "MWG-Region" always looks as pending write-back. And for any other tag, the reload from the file to the database does not make the tag "pending". Provided it has not been changed in the meantime of course.
I will post example and log file in a next message

[Jean-Maetso]

I created a database from scratch, added two images (one without face, on which i will play on the "title" tag ; and one with face, on which we will have the "region" tag).
I wrote MD in the file, the icon pen disappears.
I reload the MD, icon pen re-appears on the image having a face, not re-appears on the image having no face.

So in one case, the "region" tag is reimported, unchanged, but shows up as pending ; in the other case, the "title" tag is reimported, unchanged, and icon does not show up. 

Attached are screen shots and log file.
Thank you,

[Jean-Maetso]

And the two remaining screenshots, actually showing the problem

[Mario]

If there is a face on the image, the tag "MWG-Region" always looks as pending write-back.

That's expected. IMatch creates a face region from the fache and updates the XMP region tags, some related timestamps probably etc.

[Mario]

I don't understand. A normal rescan should do nothing, since the file "last modified" timestamp on the disk has not changed. Your request to rescan the file will be ignored. Or do you do a forced rescan or forced metadata reload.

What are you actually trying to achieve? IMatch automatically reloads metadata after write-back, bringing the database up-to-date. Why do you trigger a rescan at all?

[Jean-Maetso]

Hi Mario,

I agree that a normal rescan should do nothing. Because IMatch has the option "preserve date/time of original file", the wrinting of metadata does not show any changes to the timestamp disk point of view. So the file appears unchanged to the rescan engine.

In my example however I have forced the rescan by hitting CTRL+SHFT+F5, 3rd option (well, 2nd option "full update" will lead similar stuff).

Why ?

In some situations, I manually reload or manually fully update some files (for instance after modification by 3rd party apps). To do this, I may select a series of contiguous files in the File Window: some files actually need a reload, some others not... but for simplicity I select all of them. Etc. These are the reasons which could lead some of my pictures to go from time to time through a MD relaoding.

I am not trying to achieve anything special. I just kown that I type CTRL+SHFT+F5.

Of course, I agree that there is no criticity nor damage to the DB. It is rather a question of user experience. the annoying point is that, usually, the pen icon means "hey ! you have some changes that are waiting to be put into the files". In the case of the "RegionInfo" tag, this is not true anymore... The pen icon appears but no changes has to be written, the info is already in the file. And sometimes, there are changes anyway. I do not want to move my mouse over each pen icon to discover the list of tags that are pending and deciding if there are changes pending at 100% (ex, a "title" tag) or not.

To my mind, IMatch should check whether the reimported regions are identical to those already in the DB, and prevent the pen icon from showing up if they are so.

Thanks,
JM

[axel.hennig]

I could replicate this.

Maybe it is because the "xmp:InstanceID" is updated when the "XMP-mwg-rs:RegionInfo" is written. Maybe both of them are linked to each other within exiftool?

[Mario]

Because IMatch has the option "preserve date/time of original file",

This should never be used. It is a historic feature I keep around for a few users who rely on the "last modified" timestamp of files for archival purposes. Enabling this feature can severely mess up things, from change detection to backups missing modified files.

In some situations, I manually reload or manually fully update some files (for instance after modification by 3rd party apps).

You don't need to do this. IMatch detects files modified in other applications automatically based on the "last modified" file timestamp maintained in the Windows file system. The only situation where you would need to manually force a rescan is when other applications modify files but don't allow Windows to change the "last modified" time stamp - but that would be really rare.

When you force a rescan or force a metadata reload, IMatch wipes the database contents for the file (or all metadata) and starts from zero. This means that face regions are re-imported, timestamps are re-created, checksums and XMP document/instance Id are re-created etc.

whether the reimported regions are identical to those already in the DB,

A forced (!) rescan or metadata reload wipes all previously existing data from the database. That's the purpose of these special commands. IMatch has no previous data to compare anything to in this case.

IMatch sets the pen when at least one tag value has been created or modified during the import. Which is very likely when you force a rescan or metadata reload. Workflow data like instance/document ids, parts of XMPmm, timestamps etc. are likely to be created or set during a forced import. A forced metadata import or a forced file rescan is very different from a regular file re-import after write-back or a file re-import after IMatch has detected that a file has changed on disk.

[Jean-Maetso]

I understand the risk of having the timestamp unchanged while you actually change the file...
But I believe the option "Preserve date/time of original file" is set to "Yes" by default ? Which is not in favor of the above...

[Jean-Maetso]

A forced (!) rescan or metadata reload wipes all previously existing data from the database. That's the purpose of these special commands. IMatch has no previous data to compare anything to in this case.

IMatch sets the pen when at least one tag value has been created or modified during the import. Which is very likely when you force a rescan or metadata reload. Workflow data like instance/document ids, parts of XMPmm, timestamps etc. are likely to be created or set during a forced import. A forced metadata import or a forced file rescan is very different from a regular file re-import after write-back or a file re-import after IMatch has detected that a file has changed on disk.

Hi Mario,

1-
You say that it "wipes previous MD in the DB". Sounds clear. But you also say "sets the pen if a tag was modified during the import". Sounds contradictory.
What does it mean "modified during the import" ?

To my understanding (and agreeing that all metadat will be wiped then replaced), there is a check before deleting the metadata: for instance, if there is a "Title" tag, both in the DB and in the file, and if I reload the file, then the "Title" in the database is erased, the "Title" embedded in the file is re-imported. However the pen does not show up, because obviously (and intelligently), IMatch has detected that the replacing value is identical to the one that is to be erased.

2-
You say that very likely some tags are modified when I force a reload. I would object that for the most frequent tags it is not the case: if there are embedded in a file, you do not modify the file, and you re-import, I cannot see any changes in the file (by inspecting with ExifTool). Maybe some more exotic tags are refreshed each time one reads them, but I do not really understand why. 
As a matter of fact, re-importing a file can be done even if the file is read-only, right ? The process of reloading metadata only read the file and write the database, I believe. 

3-
You say that "A forced metadata import or a forced file rescan is very different from a regular file re-import after write-back or a file re-import after IMatch has detected that a file has changed on disk."
Can you please help us to understand the difference between the scenarios ? To my eyes, it was the same: accessing the file in read-only mode, obtain the metadata and other file properties (system timestamps...) and propagate these data into the database. Maybe there are a few subtleties I would be happy to understand.

Thanks,
JM

[Mario]

But you also say "sets the pen if a tag was modified during the import". Sounds contradictory.
What does it mean "modified during the import" ?

Timestamps, checksums, management entities like documentId, instanceId in the XMPmm namespace etc. It all depends on which metadata exists in the file.

To my understanding (and agreeing that all metadat will be wiped then replaced), there is a check before

There are no "checks before". The sole purpose of a forced rescan / metadata reload is a clean slate. IMatch clears all (meta)data, then re-imports the file.

Metadata is very, very complex and you should not do forced rescans or metadata reloads just for the sake of it. These special commands exists only to solve specific problems caused by other applications (like, changing a file but resetting the last modified timestamp on disk) or to forcefully wipe and reload metadata after the user has changed metadata-related options in IMatch.

IMatch has no history, no data to compare the imported data to when you force a rescan or metadata reload.
It does not know that it re-imports a file because of a write-back or because the file timestamp has changed. All the code that runs for files coming into the database is triggered, including creating timestamps from EXIF or other sources, filling photools.com tags, starting a document management workflow by creating XMPmm tags, re-importing XMP regions, running face recognition when enabled etc.

[Jean-Maetso]

Thank you Mario, your explanations are clear. As always.
I am almost done (satisfied) with the subject...

In order to better adapt and fine-tune my future workflows, could you please confirm (or not) the following:

1- Should the option "Preserve date/time of original file" (in Edit | Preferences | Metadata 2) be set to "NO" ? (nota bene: it looks to me that it is "YES" by default)

2-The "Forced MD reload" is a special command for specific problems and you recommand to use it very carefully and cleverly. Do you recommand the same care for "Full update" ? (I guess the answer is YES because I understand "Full update" doing more stuff than "Reload Metadata")

3-I understand that the forced reload command has no "history". But do you confirm that when a basic tag (like "Title") is reimported immediately after write-back, IMatch considers it as "unchanged" and therefore, does not show the pen icon ? (I have never seen the pen icon re-appearing after re-importing a "Title" tag...)

4-Do you confirm that when a "forced reload" is needed (sometimes...), it could make sense even if the file is read-only ? In other words, do you confirm that accessing the file in read-only mode does not prevent IMatch to correctly get and write the metadata in the DB ?

5-Do you confirm that after a write-back, IMatch reload the metadata ? If I understand correctly this means that: (1) we have some tag values in the DB, for instance the user has specified "Title" or detected faces; (2) the user press ALT+SHFT+S to embed these infos into the files; (3) ExifTool does the nice jobs and updates the file; (4) then IMatch wipes all MD for the files ; (5) IMatch recollects the MD as they have been formatted and written by ET; (6) IMatch puts them in the DB, thus ensuring a clean state. Do you confirm that steps (4), (5) and (6) are actually performed at each write-back operation ? Or maybe step (4) does not occur because we are in the situation of re-importing after write-back, not forcing re-import ?

Thanks again,
JM

[Mario]

1. The default is off and this option is only visible when the user has enabled the Expert Mode.

2. A forced update or metadata reload is usually never needed, except for the reasons I've explained. Which are rarely hit.

3. A re-import after a write back differs from a re-import when a file has changed and is processed differently. Importing a new file or running a forced reload is something else entirely.

4. A read-only file does not prevent ExifTool or IMatch from reading metadata from a file.

5. Writing back metadata will produce "new" metadata like digests, timestamps, data mapped from XMP to EXIF, legacy IPTC, GPS etc. This is done by ExifTool and ExifTool scripts IMatch runs during write-back. IMatch reloads the metadata afterwards to ensure the data in the database matches what's in the file.

See Metadata for Beginners and Metadata Write-back for related information.

[Jean-Maetso]

OK, thank you.

Only Point 3 let me not 100% clear.

3. A re-import after a write back differs from a re-import when a file has changed and is processed differently. Importing a new file or running a forced reload is something else entirely.

My question is, in terms of user experience and workflows, how should I handle these "differences" in processing ?

Example, if a file remain binary-wise unchanged but its timestamp changes on the disk (eg, replace the file by a basic copy/paste in Windows), the file will be rescan automatically ; so not manual reload nor forced update. But so why some metadata do appear as changed/pending (pen icon) and some others appear as unchanged/not-pending (no icon) ? This is what I find frustrating as a user... with the constraint of moving my mouse over every pen icon to see what is pending.

[Mario]

Example, if a file remain binary-wise unchanged but its timestamp changes on the disk (eg, replace the file by a basic copy/paste in Windows), the file will be rescan automatically

IMatch will rescan the file as "changed". If and which tags appear as changed depends on the data in the database and the data in the file, metadata settings you have made and many other factors.

Usually nothing of this is of interest to users, IMatch takes care for it. When IMatch marks a file as pending, it has reasons for this.

If you, for whatever reason, replace a file with a binary identical copy of itself (why?) and the last modified timestamp in the file system changes, IMatch detects the file and re-imports it. Under normal circumstances, no metadata should be set to pending.

I've just tried that. Copied a file to a new folder (outside of IMatch), waited a few seconds then used a PowerShell command to update the last modified timestamp on disk. Copied the file back, replacing the file in the database.

After a few seconds IMatch marks the folder as modified and then re-imports the replaced file because the timestamp is newer than the one recorded in the database. No metadata was set as pending, as expected.

When you force a rescan or metadata reload, things are different of course.

[Jean-Maetso]

Thank you Mario for taking the time to clarifying this in details. This will be helpful to me for tuning my workflows accordingly.
Best,
JM

[mlavicka]

For consistency, I believe the pen being set on reading in region info is the same as in this thread: https://www.photools.com/community/index.php/topic,15034.msg105350.html#msg105350

Where it was discussed as an issue with the order of operation, not just related to timestamps.
From Mario: This behavior is by design. When it imports face regions, IMatch no longer "knows" that the file on disk may have XMP face regions. When XMP face regions are found, the existing PersonInImage data is dropped and, at some later time, recreated. Changing this behavior would be quite complex and "expensive".

I assume that is still the case.