Sublocation problem when using metadata templates

[JohnZeman]

Recently I started having problems applying locations to my photos.  I use a metadata template to apply the location and that in itself seems to be ok.

The problem happens when I use that same metadata template to fill in the sublocation field.  For the sublocation I'm using the

{File.MD.XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation\0}

Tag.  IMatch fills in the sublocation field not only with the value I want it to fill in, it also adds the value of the location tag.

The below screenshot shows the problem.  The sublocation field should only have Retail House after the metadata template is applied, not Thistle Down Nursery; Retail House.

I think this all started with one of the latest versions of IMatch.

[Mario]

This tag is part of the repeatable structured tag XMP::iptcExt\LocationCreated.
Which tags precisely do you show in your custom Metadata Panel layout for location and sublocation? Can't tell from the screen shot.

What do you see when you check in the "6 IPTC Location" Metadata Panel layout, which shows the structured tags and their contents?

Maybe try to set XMP::iptcCore\Location\Location in your Metadata Template. This tag is mirrored to "Composite\MWG-Location\Location" and "XMP::iptcExt\LocationShownSublocation\LocationShownSublocation".

I need to reproduce this to learn what's going on in this case.
Structured tags are complex, but support for them was required and is new to IMatch 2023.
There still might be some issues to solve, depending on how users manipulate elements of structured tags individually.

[JohnZeman]

Below are the tags I'm using for location and sublocation.

{File.MD.Composite\MWG-Location\Location\0}

{File.MD.XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation\0}

Somewhere along the line I must have deleted the 6 IPTC Location metadata layout because it's not in my list of layouts.

Your suggestion to add the location and sublocation with a metadata template appeared to solve the problem.  That is until I did a metadata writeback.  When the writeback finished my sublocation field had 2 identical entries separated by a semicolon ; (See attached screenshot).

I only have a problem when I use a location with a metadata template.  If I enter all of the location metadata template with a metadata template all is fine.

[Mario]

Somewhere along the line I must have deleted the 6 IPTC Location metadata layout because it's not in my list of layouts.

This layout was added in 2023. Maybe re-import the system layouts again and merge with your custom layouts. The 2023 release notes have instructions for that.

That is until I did a metadata writeback.  When the writeback finished my sublocation field had 2 identical entries separated by a semicolon ; (See attached screenshot).

That's one of the miracle issues with ExifTool I had to fight when dealing with structured tags, mapping between Composite, IPTC, IPTC Ext and Photoshop XMP tags which all store one or multiple locations, derived from the structured IPTCExt::LocationShown and/ or IPTCExit::LocationCreated tags.
I'm still getting green and sick when thinking about it and many of my new gray hairs stem from that process.

I will look into what your Metadata Template is doing when I've worked down "My Metadata is not working" email queue a bit. The heap got to big and I have to spend some days working on it.

[JohnZeman]

I can only try to imagine what your workload must be like Mario.  Thanks for giving this some thought but don't hurry on my part because I'm getting along ok.  I just manually correct the sublocation errors when they happen.

[Mario]

I've tried to reproduce this using the following steps:

1. Create a MD template that fills IPTCExt::LocationCreatedSublocation with "IMatch Location"

2. Apply the template to a file that has no location data.
Result: Only the IPTCExt tag is filled, Composite\MWG-Location\Location remains empty.

3. Write-back.
The term "IMatch Location" appears only in the IPTCExt XMP tag as it should.

4. Force a re-import. Same result.

5. Do the same to a file which already has location data.
The IPTCExit tag is filled with "IMatch Location", the composite location tag remains unchanged.

6. Write back
The IPTCExt SubLocation reverts to the original, Composite unchanged as before.
Not for all files, though...

This is not the expected behavior but it stems from how IMatch deals with structured tags like LocationShown and LocationCreated. IMatch makes the individual components of these structured tags available for easy usage in e.g. variables or the Metadata Panel. But in these cases the user works, basically, with a copy of the actual value that is stored in the structured tag.

Whenever the user manipulates a part of a structured tag directly, IMatch has to perform special routines to make the magic work behind the curtain. This is why all new IMatch installations (and users who've followed the instructions in the release notes) now preferable use the "6. IPTC Location" MD Panel layout for viewing and modifying location data.

This layout not only shows all elements of the structure, but also can handle repeatable values (e.g. multiple locations covered by the same image or different translations for a place recorded as separate "records" for the structured tag.

Setting a part of a structured tag (like SubLocation that is part of IPTCExt::LocationCreated) via a Metadata Template seems to somehow circumvent what IMatch is doing. It's perfectly "legal" to do this of course and should work.

I need to debug this and see what the problem is. Hopefully this is not again one of those metadata-related issues which take me days to figure out and fix. So much time and code went into finally making structured tags work the way the IPTC standard requires...

[Mario]

The reverting of the location data was caused by ExifTool having an issue writing to the file. Stupid me!

I've tried my tests a number of times on different files but could not reproduce the problem you are experiencing.

The Composite Location (and IPTCCore Location) is linked to the LocationShown\SubLocation tag, not LocationCreated.
When I create a Metadata Template for LocationShown\SubLocation and apply it, all 3 tags (IPTCExt, Composite\MWG\Location and IPTCCore Location) are updated.

When I point the mouse cursor at the write-back pen of the file in the File Window, I see that all 3 tags need updating.

When I apply the metadata template that sets LocationCreated\SubLocation, only this tag is modified and the File Window pen icon shows only this tag needing a write-back.

Composite\MWG\Location is not changed when running the MD template and their during the write-back.

When I create a MD template that sets Composite\MWG\Location and apply it, 3 tags are changed. And the changes persist the write-back.

Is there anything else I can try?
Is setting the tag the only change your MD template performs?
Maybe this depends on the files you work with?
What happens if you create a fresh image in an editor, import it into IMatch and apply your MD template? Same effect?

[JohnZeman]

Mario thank you for taking so much time to try to get to the bottom of my head banger problem.
 
I just spent the last 4 hours trying to find answers to your questions and my problem but the only conclusion I'm coming to is I need to change the tag I'm using for my sublocations.
 
As long as I'm using
 
{File.MD.XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation\0}
 
For sublocations I have problems using IMatch locations with the problem usually being duplication of the metadata in the sublocation field after a writeback. 
 
So as an experiment I temporarily used the
 
{File.MD.XMP::xmp\Label\Label\0}
 
Tag for sublocations and everything works correctly when I use that tag with an IMatch location.
 
Oh by the way the tests I did today were with 5 new CR3 photos taken yesterday and the results were consistent with what I mentioned up above.  As long as I'm using the LocationCreatedSublocation tag for my sublocations I have problems after doing a writeback.
 
If I don't use an IMatch location with my metadata template so the template writes all of metadata including GPS coordinates, etc. then I have no problems.  It's only when I use an IMatch location with a metadata template when I have problems and even then the problems only occur when the sublocation tag is LocationCreatedSublocation.
 
I'm interested in your thoughts on this and I'm going to do some more testing but right now it's looking like I'll be changing the tag I use for sublocations.  Just configured a quick data-driven category to do a count on the number of images I'll need to change, at least it's a doable number, less than 40,000.
 
 

[sinus]

Hi

To be honest, I do not understand all here.
But today I have realised, that I have in my hierarchical keywords double entries.
Created from shown and created - tags.

Because I knew, that IMatch has changes the created-show-system in GPS, I checked my preferences and yep, there we are. 
I had created both, shown and created - variables to go into keywords.
Hence I took away my shown-variables and have now only still my created variable in the field.

And now all works again fine. 
The case, that I fill out different tags for shown and created is seldom, hence I have only to think in these cases to fill out the hierarchical keys "manually". Because it is seldom, I simply let this like it is. 

@John, I have no clue, could this help in your case, but I thought, well, I write it down. 

@Mario, ich hatte schon eine (nein, zwei ) andere meldung hier gepostet, aber dann schnell realisiert, dass im Attachement mein google api - key sichtbar war und habe einach diese post gelöscht, und deshalb hier diese hier geschrieben. Bin wohl noch nicht ganz wach ... phew, muss wohl aufpassen heute 

[Mario]

I'm not sure I understand this.

If I don't use an IMatch location with my metadata template so the template writes all of metadata including GPS coordinates, etc. then I have no problems.  It's only when I use an IMatch location with a metadata template when I have problems and even then the problems only occur when the sublocation tag is LocationCreatedSublocation.

Can you please export your metadata template, ZIP and attach it?
Edit Metadata Templates. Select the template and then click on the "Export" button at the bottom.

I did not use locations in my test. I used a hard-coded text to fill the "XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation\"in my tests. Maybe I can reproduce it using your template and a CR3 file.

[JohnZeman]

Good news for both of Markus, you just solved a duplicate metadata problem and so have I by changing the tag I was using for sublocations from
 
{File.MD.XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation\0}
 
To
 
{File.MD.XMP::iptcCore\Scene\Scene\0}
 
Once I made that change, my duplicate metadata problems disappeared. 
 
Mario as you requested I have zipped and attached one of the metadata templates I had been using.

[sinus]

John, great!

I did not know, but I hoped, that it could help you or at least give you another possibility to check.
I am very glad, that you could solve your problem, my dear friend.
Very good!   

[Mario]

Hi, John
you've exported all templates. My instructions were incomplete, sorry.

Which of the many templates to you use to produce the problem? Any that has SubLocationShown?

I've tried "Grod ... 80 Field". It sets all the tags, but there is no duplication, ever after a write-back.

[JohnZeman]

It didn't seem to matter which template I used Mario.  As long as SubLocationShown was the tag I used for sublocations the duplication problem occurred but not until after I did a metadata writeback.

As far as I'm concerned the problem is solved for me now.  Changing the tag I use for sublocations from SubLocationShown to iptcCore\Scene solved the duplication problem. 

I'll be leaving town for a couple days shortly so if you respond with a question or instructions for me it will likely be Thursday before I'll be able to respond.

[Mario]

I would really like to find a way to reproduce this here. Especially if there is a potential bug lurking in structured tags.
I have some bad memories about implementing this, fighting with ExifTool and metadata standards and the complexity of it all.

[JohnZeman]

I'm back from my overnight trip now Mario and I took a number of new CR3 photos.  After importing those new photos into IMatch and renaming them I made a temporary change to my Family:Bleed Home metadata template so it uses the SubLocationShown tag for the sublocation again.  Then I applied the Family: Bleed Home metadata template to the photos and the attached screenshot shows the results after doing a writeback.

As you can see this time the metadata assigned to the standard Composite Location tag was also copied to the SubLocationShown tag.

I've also exported the single Family:Bleed Home metadata template I just used to create the results in the attached screenshot.

[Mario]

This is quite mysterious.
I've applied your template and only "XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation" was changed (in the database and during write-back). I've used a JPG and CR3 file. The JPEG has embedded XMP and the CR3 has an XMP sidecar file.

IMatch does not do any special processing / mapping for XMP::iptcExt\LocationCreated*, only for XMP::iptcExt\LocationShown* (to map between various locations in XMP and IPTC).

Composite tags are "special" ExifTool tags it makes up from various source tags.
IMatch imports Composite tags "as delivered" and stores them in the database.
Probably this is some side-effect or another tag that makes ExifTool double the tag value or something...?
Or it is somehow related to your specific images and/or database. No idea, really.
Maybe you can send me one of the CR3 files to support email address (with a link back to this topic) and I apply your template to this file to see if this makes a difference.

I have some comments in the source code and configuration file to not link specific tags because it "ExifTool duplicates tag values during writing". I remember having a hard time figuring that all out when I improved support for linked tags and location tags for IMatch 2023. There is also a mile-long thread about this topic and how best to deal with LocationShown and LocationCreated.

[JohnZeman]

Maybe you can send me one of the CR3 files to support email address (with a link back to this topic) and I apply your template to this file to see if this makes a difference.

Mario I just sent you the link to a OneDrive zipped file of one of my CR3 photos straight out of my camera.  I also sent you an identical email in case the first link doesn't come through.

[Mario]

Hi, John

I've downloaded the ZIP, extracted the CR3 file and added it to a database.
Then I ran your Family Bleed Home Metadata Template which sets Headline, Rating and LocationCreated\SubLocation.
Headline and Rating are only applied if the target is empty, SubLocation is always applied.

After running the template, the headline has been updated, the rating has not been set to 3 (the CR3 file contains a hard-coded XMP::rating=0) and "Temp Location" has been set for "XMP::iptcExt\LocationCreatedSublocation\LocationCreatedSublocation".
Composite\MWG-Location\Location is still empty.

I write back. 
Inspecting the result arguments sent by IMatch to ExifTool shows:

Code Select Expand

-XMP-iptcExt:LocationCreated=[{Sublocation=Temp SubLocation}]
which sets the correct tag. No other tag is set to "Temp Sublocation"!
Looking at the embedded metadata in the ExifTool Command Processor shows nothing unusual and no "Temp Sublocation" anywhere. Looking at the XMP sidecar file now shows

Code Select Expand

[XMP-iptcExt]  Location Created Sublocation    : Temp SubLocation
which is correct. The Composite\MWG-Location\Location tag is still empty in the Metadata Panel.

I force a metadata reload and then a full reload. No change. Perfect result.

Then I look again at your metadata template and I notice that the option "Location: Automatic - use existing GPS data in image" is enabled. But the CR3 has no GPS data (except a GPS version tag, for reasons only Canon knows), so this will not make any difference.

I apply the metadata template several times and write-back several times, just in case. No change. Perfect result.
Created a new database and repeated all steps. Same result.

I'm flabbergasted and maybe even slightly bamboozled!

What is the difference between our two setups?
I've used the same file and the same metadata template, but I don't see the problem you can reproduce so easily.

Out of ideas. For now.

[JohnZeman]

The location being set to automatic was a temporary mistake I'd made on that one template.  I've since changed it to Family Bleed Home but as you noted that doesn't seem to have any effect on the problem I was having.

I don't know what else I could do or send you to help narrow down the problem Mario.  Fortunately for me changing the tag I use for sublocations to iptcCore\Scene solves my problem but I know it doesn't solve the mystery for you.

[Mario]

You may want to change how you set the Rating in your template.

You use the "only if empty" mode, but that fails since your CR3 files contain a hard-coded rating of "none" (0) and IMatch imports that.

Use this variable to set the rating instead:

Code Select Expand

{File.MD.rating|cast:int;numcomp:eq,0,3,{File.MD.rating|cast:int}}
This checks if the existing rating is 0 and if this is true, it sets the rating to 3, else it keeps the existing rating.

[JohnZeman]

Mario I will make that change to each one of my metadata templates that sets the rating, thank you!

[Mario]

I don't know why Canon does this. The only XMP field in the file is XMP::xmp:Rating=0...