Another metadata loop question - MWG RegionInfo

[Redinstead]

New user and I hope I'm not missing something obvious, but I'm a previous Digikam user (still recent) and am switching over. I have previously used it for face identification and I'm having an issue with photos showing the yellow pencil, and wanting to write RegionInfo. That's all fine and I expect things like that initially, however when I wrote it on a test photo and then reread the metadata for that photo the pencil appears again, wanting to write the RegionInfo over again.

The writing doesn't appear to fail (Exiftool output from output panel below) but something is making it want to write again. I assumed that when I wrote it the database and file would be consistent. My gut is telling me it's to do with Digikam having created the regions, but I did strip all files of XMP:MP metadata so that it wasn't getting in the way. Is there any other diagnostics I can do to work out the issue? I am guessing I can strip all my face IDs and start again but would rather not if it can be helped. Maybe I'm misunderstanding reloading the metadata - I just assumed it would show everything in sync.

ADMIN: Removed 500 lines of text and added them as an attachment.

[Mario]

Welcome to the community.
DO NOT DUMP 500 lines of text data in your posts. This makes your post unreadable, fills the community search engine with junk and may considered to be rude by some.

ZIP the text file and attach it If you don't see the Other Options link below the post editor, click on Preview once to see them, including the option to attach files.

[Redinstead]

Apologies - I don't think I realised how much I was pasting. Absolutely should have been a file

[Mario]

You already sent a sample image and I will look into it.
Please understand that I get many "My metadata not working" email per week (usually not an IMatch problem, just ordinary metadata mess) and I process these often time-consuming emails in the order I receive them. There is always a stack of stuff to look into.

IMatch imports existing face regions as explained here: Working with XMP Face Regions but that does not cause new regions to be created. Do your images contain XMP face regions
Or do you have automatic face recognition enabled (Edit menu > Preferences > Application) and IMatch creates the regions?
The region info causes reoccurring write backs is something I have not seen before.

Run the Metadata Analyst one one of the problem files to learn more.
I have never just Digikam and I don't know which metadata it produces and how standard-compliant it is.
I will know more when I have looked at your sample image.

[Redinstead]

Digikam writes to XMP-mwg, yes. Unfortunately also to XMP-mp, which is impossible to turn off. I stripped the files of the XMP-mp metadata in the hope that I might avoid any issues.

Automatic face recognition is off, and the Metadata Analyst has zero errors and two warnings.

Warning: [System] File has unwritten metadata (pending write-back).<br/>The metadata loaded from the image and the data in the database may not match.
Warning: [Legacy IPTC] Character Set Encoding: unspecified.

Thank you - there's every chance I've botched something so I appreciate your time.

[Redinstead]

Just for some further info if it helps, I did comparison of  a file a couple of times pre and post writing the metadata back. It seems like it is constantly changing the region area with each write, so the data is being written - it's just not clear why it wants to tweak it on every reload.

I thought maybe there was a mismatch in the size of the photos vs the expected size causing the shift, but checked this file again and got following info from exiftool which suggests not.

[Composite]     ImageSize                       : 3648x2736
[XMP-mwg-rs]    RegionAppliedToDimensionsH      : 2736.000000
[XMP-mwg-rs]    RegionAppliedToDimensionsUnit   : pixel
[XMP-mwg-rs]    RegionAppliedToDimensionsW      : 3648.000000

You cannot view this attachment.You cannot view this attachment.

Also, Drive link to another photo in case anyone wants to see if my selfie behaves the same for them on metadata reload. Taken away from home and I'm ok with face showing

https://drive.google.com/file/d/1JuHTigKME7OwYNtf-h-5KeEDfbzj0Vzr/view?usp=sharing

[Redinstead]

I should say my current thinking is that this is a floating point calculation mismatch bewtween Exiftool and iMatch database, and if so I'm not unduly concerned. If this is looking likely from the previous information then it's probably not worth doing any depth work on the provided photos - more than happy for that time to be given to more pressing issues people may have.

[Mario]

Regions are expressed as floating point numbers. Both the anchor point and the width/height use values in the range 0..1, relative to the image size. In XMP metadata, these floating point numbers are written as text, rounded to whatever precision the creating application deemed sufficient. So there is always a change that there is some rounding error when importing XMP face regions. 

I have downloaded the file you have sent via email yesterday.
It contains this XMP region:

Code Select Expand

[XMP-mwg-rs]    Region Applied To Dimensions H  : 3840.000000
[XMP-mwg-rs]    Region Applied To Dimensions Unit: pixel
[XMP-mwg-rs]    Region Applied To Dimensions W  : 2880.000000
[XMP-mwg-rs]    Region Area H                  : 0.524219
[XMP-mwg-rs]    Region Area Unit                : normalized
[XMP-mwg-rs]    Region Area W                  : 0.524306
[XMP-mwg-rs]    Region Area X                  : 0.452778
[XMP-mwg-rs]    Region Area Y                  : 0.512891
[XMP-mwg-rs]    Region Name                    : Me
[XMP-mwg-rs]    Region Type                    : Face

IMatch imports this and creates a face annotation in the database. The face region created during the import process reports a change, thus requiring a write-back. Most likely some rounding difference somewhere. 

I write back.
The pen vanishes and does not come back. Database contents and file metadata contents are in sync.

I've used a stock IMatch installation with all default settings for Edit > Preferences > Metadata 2 and the default face-related settings in Edit > Preferences > Application on my notebook.

This is the behavior I expected. But not the behavior you see.

I've repeated the test with the second image from today. Same result. Import, face annotations are created and the file is marked as pending because of RegionInfo.

Writing back removes the pen. OK.

What could be the difference causing this to fail on your PC? I have no idea.

A reoccurring pen is not uncommon when new users have to clean up metadata mess produced by other applications, especially out-of-sync keywords. But I have not seen this, I believe, caused by RegionInfo. Once IMatch has produced the region info XMP structure from the face annotation and written it, it is in sync with the database.

Can some other users please import the image and check if they also experience the effect that the pen comes back after write-back, with RegionInfo in the tooltip?

I've did the test on my developer PC and notebook, which I keep on all the default settings IMatch ships with. It works here on both systems.

[Redinstead]

Just to clarify - it comes back when I reload metadata (albeit not if I do a normal rescan). Day-to-day I wouldn't have the need to reload metadata once the pencil vanishes, but I was reloading it on a group of files and that's when I noticed that all my files with regioninfo would show the pencil icon again, despite having previously been written to.

I think there is something happening in this reread of the metadata causing iMatch to want to write again with slightly different values, but if there is nothing wrong with my file once written I'm not really concerned.

Initially I was thinking something was structurally wrong with my files and it made me think the write wasn't "taking". After experimenting and seeing the drift last night each time I realised they were being written to ok.

I'm now not sure about the rounding since it's a drift of around 3%, but so long as over time the regions don't drift away from the face itself I'm not worried. It's an oddity but my main concern, as a new user, was that something was "wrong" with the metadata in my files and I may need to remove it all. That doesn't seem to be the case

[Mario]

Just to clarify - it comes back when I reload metadata (albeit not if I do a normal rescan).

If you force a metadata reload (why?) all metadata is wiped from the database and everything is re.-imported, creating new face annotations etc. This means that RegionInfo will be marked for write again, of course.

You usually only ever need to force a reload of metadata when you have changed metadata settings that affects how IMatch imports metadata.

[Redinstead]

I was really trialing out the functionality of different parts of the software on test photos for familiarisation purposes, so was doing some actions that I likely wouldn't do under normal circumstances. That's when I noticed it happening.

My assumption was that the metadata reload information would be read back as it was written with the previous write, and match the database - much like keywords, gps, etc don't mark for a rewrite after reloading metadata.

It seems I have had a fundamental misunderstanding of that, and not realise that this triggers a new region to be created rather than using the areas already in the metadata

[Mario]

It uses the regions in the XMP. But internal calculations may still mark the freshly imported region as modified, maybe the record was completed or there are small derivations due to rounding. This will only be triggered when you force a metadata reload, which you usually never do.

[Redinstead]

Yes, it seems to tweak the region area internally whenever it reads the region information, regardless of whether iMatch asked Exiftool to write it or not. That's the part that was causing me to think something was wrong with the files or the writing process.

Knowing all that, you're right - it's not something I'll do regularly at all and it's not something I'll worry about. I hope it's not been a completely ridiculous thing to clarify though - on the surface it seems unexpected.

Hopefully if anyone else is confused by this in the future then they'll manage to wade through my misunderstandings here and realise there is nothing to be worried about.

Thanks for your time.     

[mlavicka]

I believe the pen being set on reading in region info is the same as in this thread: https://www.photools.com/community/index.php/topic,15034.msg105350.html#msg105350

Where it was discussed as an issue with the order of operation.

From Mario: This behavior is by design. When it imports face regions, IMatch no longer "knows" that the file on disk may have XMP face regions. When XMP face regions are found, the existing PersonInImage data is dropped and, at some later time, recreated. Changing this behavior would be quite complex and "expensive".

Is this still the case?

[Mario]

Yes. No need to change this, it works fine.

[Tveloso]

I believe the pen being set on reading in region info is the same as in this thread: https://www.photools.com/community/index.php/topic,15034.msg105350.html#msg105350

Where it was discussed as an issue with the order of operation.

From Mario: This behavior is by design. When it imports face regions, IMatch no longer "knows" that the file on disk may have XMP face regions. When XMP face regions are found, the existing PersonInImage data is dropped and, at some later time, recreated. Changing this behavior would be quite complex and "expensive".

Is this still the case?

It looks like Mario was referring specifically to the PersonInImage tag there, which is not a part of MWG Regions (though the RegionName tag, contains the same information).  The discussion here, is about the MWG Regions tags becoming pending for Write-Back following a Forced Update.

Very much analogous to that, is adding a file that has been "fully processed" in IMatch, to a new IMatch Database.  This is something that I had been considering as an approach for using IMA.  The post referenced above:

https://www.photools.com/community/index.php/topic,15034.0.html

...led me to realize that the approach I had been considering for using IMA, would not work because of this "Write-Back loop" (I'll ask about the approach I was considering in another topic).

I think Redinstead has hit upon what is causing MWG Region Info to create a "Write-Back loop", even for IMatch-only created regions.

For a file containing MWG Regions created only by IMatch, I would expect that if such a file (prepared by IMatch, with a full complement of proper Metadata) were to be indexed into another IMatch Database, there should be no reason for anything to become pending for writeback following ingest there.  Yet the MWG Region Info structure does become pending for such a file.  I performed these two tests:

1.) Using a File that already had Region Info
First I used a newly indexed iPhone Photo, which already contained Face Regions:

    You cannot view this attachment.

...and used the ECP's List Metadata preset, to verify that those high-precision values were what Apple had delivered.

(Both of the Face Annotations resulting from the regions in this file became Manual Face Annotations when it was indexed in IMatch.  As discussed in other topics here, many of the Face Annotations created by importing iPhone Face Regions are very tight on the face, sometimes covering just one eye, and the nose.  So the first thing I usually do with a new batch of iPhone files is to run Face Detection - Ctrl+M,F - with the Ignore images with existing face annotations option unchecked, so as to discard the iPhone-created regions, and have IMatch create new ones.)

I then sent the file to the Viewer, pressed F6, and selected the Remove existing face annotations and run face detection again option, and IMatch detected one of the two faces, and assigned the correct person.  The other face was in profile, so it's understandable that it was not detected.  I drew a Manual Face Annotation for that one.

A Write-Back then resulted in these Face Regions (note that the pixel dimensions went from integer to floating point):

    You cannot view this attachment.

...and the ECP showed that is in fact what the file contained:

Code Select Expand

[XMP-mwg-rs]        - Region Applied To Dimensions H  : 3024.000000
[XMP-mwg-rs]        - Region Applied To Dimensions Unit: pixel
[XMP-mwg-rs]        - Region Applied To Dimensions W  : 4032.000000
[XMP-mwg-rs]        - Region Area H                   : 0.068667, 0.121197
[XMP-mwg-rs]        - Region Area Unit                : normalized, normalized
[XMP-mwg-rs]        - Region Area W                   : 0.052000, 0.095692
[XMP-mwg-rs]        - Region Area X                   : 0.467500, 0.416531
[XMP-mwg-rs]        - Region Area Y                   : 0.138333, 0.784584

At this point, everything is fine, and it should usually be the end of the story.

But to simulate indexing the file into a different IMatch Database, I removed the file from the Database:

    You cannot view this attachment.

...and re-scanned the folder.  I imagine that this is essentially the same as a Forced Update.

The resulting Region values looked identical following the new ingest, but the File was now pending for Write-Back for MWG Region Info.  After doing that write back, the Region Info then changed to this:

    You cannot view this attachment.

...(the "real face" remained unchanged, but the Manual Face Annotation had different values now for Area-W and Area-X).  The ECP showed that those new values were now in the file:

Code Select Expand

[XMP-mwg-rs]        - Region Applied To Dimensions H  : 4032.000000
[XMP-mwg-rs]        - Region Applied To Dimensions Unit: pixel
[XMP-mwg-rs]        - Region Applied To Dimensions W  : 3024.000000
[XMP-mwg-rs]        - Region Area H                   : 0.068667, 0.121197
[XMP-mwg-rs]        - Region Area Unit                : normalized, normalized
[XMP-mwg-rs]        - Region Area W                   : 0.052000, 0.103772
[XMP-mwg-rs]        - Region Area X                   : 0.467500, 0.420571
[XMP-mwg-rs]        - Region Area Y                   : 0.138333, 0.784584

I took a few more turns at removing the file from the database, indexing it again (to see MWG Region Info become pending), and performing the required Write-Back.  I found that Area-W and Area-X were each growing by one of two alternating values (Area-W grew by the exact same two alternating values, and Area-X grew by a different set of effectively the same two values as well - just a slight difference in one of the low-order digits once in a while):

    You cannot view this attachment.

So it seemed that IMatch was continually "moving" the Face Annotation with each ingest of the same file.  The Face Annotation could be seen changing shape (becoming more rectangular) with each ingest/writeback operation.

[Tveloso]

...(continued from previous post, due to attachment limits)

2.) Using a File with no MWG Regions
Next I did the same thing with a file that contained no face regions at all (taken with a Canon EOS Rebel T7i).

I first opened the file in the Viewer, and pressed F6.  IMatch detected the single face in the image, and assigned the correct person.

The following tags became pending for Writeback as a result of adding that face (the Person Record had added a Keyword to the file):

    You cannot view this attachment.

...and following the Write-Back, the file contained this Region Info:

    You cannot view this attachment.

I then removed the file from the database, and re-scanned the folder.  The file came back, now pending Write-Back for MWG Regions (with the region values still the same in the MD Panel).  I performed that write back, and this time, after the file was read back in, post-writeback, the Region values did not change (presumably because this was a "real face" - which also had not changed in the iPhone file either).

I performed a few more "remove from db / re-scan / write-back" cycles, and there was no "upward drift" in the Area-W and Area-X values, as was seen with the Manual Annotation in the iPhone photo (again, probably because this was happening only for Manual Face Annotations, and this was a "real face").  But the file still always came back in a Write-back pending state (for MWG Regions), when it was re-added to the Database (just as the iPhone photo had done).

I did this again with another file from the same camera, where I knew I'd have one real face, and one manual one.  Sure enough, one of the two faces was detected (and the correct person assigned), and the other face was not (it was almost entirely obscured), so I drew a Manual Face Annotation for that one.  Now after the Write-Back to establish the initial Face Regions, doing a few "remove from db / re-scan / write-back" cycles did see the Manual Face changing with each iteration.  This time though, it was the Area-H and Area-Y values that were changing.

So it seems that Manual Face Annotations are continually altered with each ingest into an IMatch Database.  But this is a separate issue I think.  Even when no change takes place to the Face Regions in a newly indexed file, that file still becomes pending for write back, for MWG Regions, when the expectation is that it should not.  IMatch has already produced a full set of proper Metadata for the File, so I would expect that indexing it again (into another IMatch Db) should not cause any changes to take place.

Might it be that the internal representation of the Region values, within the IMatch Face Annotation, is somehow always different (a different precision, etc.), from what is ultimately written to the XMP Face Regions?...(and this is what causes the pending Writeback condition following ingest - whether or not that "value drift" behavior happens)?

[Redinstead]

As a very novice user, I just wanted to say thank you for the investigative work there - whatever the outcome ends up being.

It would fit perfectly with my observations of drift (even though I went round in circles with myself to get there) given that all my face detections had been done previously by another application. These would presumably be treated the same as a manual face annotation in the background and so exhibit this behaviour as well.

[Mario]

Are the files stills pending after the write back? If not, it works as designed.

[Tveloso]

Are the files stills pending after the write back? If not, it works as designed.

No the files do not come back as pending once the Write-Back is done. 

But I would expect to be able to take a file that IMatch has "completed", and add it to another IMatch Database, without any need for Write-Back there.

When we remove a file from the database, and re-scan to bring it back in, it will always become pending for Write-Back (for MWG Regions), and depending on the Keyword settings in Edit > Preferences > Metadata, also Hierarchical Keywords.

[Mario]

But I would expect to be able to take a file that IMatch has "completed", and add it to another IMatch Database, without any need for Write-Back there.

As I just wrote in another thread, new metadata and timestamps may be created when you add the file to another database, metadata or other settings may differ in the other database, causing write-back.

When XMP regions are imported and IMatch produces face annotations from them, it also produces a new XMP regioninfo record, based on the numbers and data stored in the annotation, their order (which may differ from the imported order), the persons detected, the date and time etc. This will cause the file to become pending. This is not something that users usually consider, since adding an image to IMatch usually causes new metadata to be generated during the import.

The rather rare use-case of managing the same image in two IMatch databases is slightly different. It may not create new metadata (except maybe timestamps) but it will drop and re-create face annotations that may exist. And IMatch creates XMP face regions fresh from these annotations, not caring if there are already XMP regions in the file that have been imported or not. This avoids issues caused by rounding (often a thing with XMP floating point storage), digests, timestamps etc.

I have never considered this to be a "problem" or something to spend any time on. This behavior is by design.
When you often write back images in one database, and then import the same images into another database and these images contain XMP fade regions, this is something you will have to live with. I pretty much doubt that this is a problem many users will ever face.

[mlavicka]

I think that when you look at all the reasons as to why someone would like to scan a previously "fully processed" image into iMatch without it causing a pending writeback, you would see that it is not just a "rare use case".  It would be great if we could use iMatch to identify regions and persons in a way that we can save them in a file, and not cause a writeback if ever they are read in again.   Right now, as a work around, my solution is to clear the write back queue after ingestion. Obviously, you have to make sure that the only thing in the write back queue are these case types before doing it.

Thanks to Tveloso for doing the deep dive into how this behavior affects the region info.

[Mario]

It would be great if we could use iMatch to identify regions and persons in a way that we can save them in a file, and not cause a writeback i

You can do that, e.g. using the Text Exporter or the Copy Data app. Let us know if you need help with that.

If you import an image into IMatch, the XMP region, and usually many other metadata tags, will have to be written. Once.
And this is how things stay, unless you force a metadata reload for some reason. Or unless you add the same image to another IMatch database for whatever reason.

What is your workflow? Why do you add the same image to multiple databases? Why do you force IMatch to reload metadata and then consider the one time write-back a problem?

As always, feel free to add a feature request in the feature request board.
If a substantial number of users has the same workflow as you and sees this write-back as a problem that needs solving and likes your feature request, I will consider what can be done for one of the next releases.

Until then, since metadata management is mindbogglingly complex already, I'll keep things simple.

[mlavicka]

I think you may have misunderstood me.  When I said "save them in a file", I meant save the identified persons (and associated regions) in the metadata of that image file.  Then we would consider that file "fully processed" and would not want to have the write back pending set if we need read or scan in the image file again.  As to why we would need to scan the file in again.  There are many use cases that have been covered in other posts.  It could be for a new database, or a rescan in an existing database.  I think we are just looking to have metadata preserved the same way it is for other descriptive metadata that is part of the file.

[Mario]

I meant save the identified persons (and associated regions) in the metadata of that image file.  Then we would consider that file "fully processed" and would not want to have the write back pending set if we need read or scan in the image file again.  A

This is exactly what IMatch does. When you create face annotations in IMatch and write back, IMatch produces standard-compliant face regions. Only one write-back is needed.

When you add files containing XMP face regions to IMatch, IMatch creates face annotations from these regions. When you write back, IMatch creates XMP face regions from the annotations, same as above.

IMatch makes no difference between fade annotations created by IMatch itself or face annotations created by importing existing XMP regions. If annotations have been created, IMatch marks the entire XMP Region block and IPTC PersonInImage as pending for write-back. This behavior is by design.

I can only repeat myself again. It will be a lot of work to change this, which means it would be expensive to change this. And for whom? Two or three users (my estimate). I'm not against improving and enhancing IMatch, but what I have learned from the feature request board is that most requests only get one or two likes. Requests that affect many users get many likes quickly, or a discussion is spawned. This is what helps me to understand that users want.

And there is always the chance that the database you import into has different XMP region import settings, difference face thumbnail settings, different order settings, different settings related to PersonsInImage etc. All this is considered during import and applied to the face annotations created and the metadata associated with the images. This is one of the reasons why creating annotations (by IMatch or during import) results in a new XMP region record being created. It covers all bases and ensures proper sync between the full-precision face regions in the database and the data stored in XMP face region and PersonsInImage.

Changing this, adding extra checks just to tell whether or not XMP region must really be written / and/or PersonInImage for the rather rare use case of users forcefully reloading metadata or managing the same images in multiple databases is just not enough. IMatch works this way since at least 2020 and the vast majority of users just seems to get along with that fine.

Make a feature request. If this is a "problem" more than a handful users have and wants solved, I will look into it. Until then, metadata handling is complex enough as it is and I try not to add more and more complexity on top.