Write-Back problems

[jmhdassen]

I am running into a weird issue with write-back.
Either I do not understand how it is supposed to work, or there is a problem.

I have loaded lots of images into IMatch all coming from a raw processor. There is already some metadata in XMP files and these load fine into IMatch. But since my Raw Processor has very primitive and out of date metadata handling I use IMatch to correct my metadata. I am here mainly concerned with location data.
Now I have the following happening:

My initial city and state data loads into the XMP::photoshop tags. IMatch also fills the corresponding locationShown tags. That is fine.
I do a write-back and an XMP file is created.

Now let's say I made a mistake initially and I put for photoshop\State 'Kwanza-Sul', but I find that it really should be 'Bengo Province'. So I update that in the Metadata Editor and it seems fine initially.
Then I do a write-back. Now my photoshop\State jumps back to 'Kwanza-Sul' after automatic rescan. Not nice. It is not really expected behavior to have the database changed right after you actually write out the database contents.

So I figured that must be happening, because the LocationShownProvinceState was still 'Kwanza-Sul'. So I changed BOTH to 'Bengo Province'. Save to database goes ok. But if I write-back then suddenly the State changes to: 'Kwanza-Sul, Bengo Province'. So I get both the new and the old one. So I figured it must come from the XMP file.

If I edit the XMP file and delete the whole <Iptc4xmpExt:LocationShownProvinceState> section then suddenly it works fine. But obviously I do not want to manually edit all XMP files.

So I figured, let's delete the XMP file altogether. I expected that all metadata would be in the database, so a fresh XMP would be written. Wrong again. After write back I have now lost ALL metadata except the State data. I even lost all the date fields  So this is totally unacceptable.

So then  figured I just replace 'Kwanza-Sul' with 'Bengo Province' in the XMP files using the sed command. So no more Kwanza-Sul anywhere. So what happens now after write-back ? I get 'Bengo Province, Bengo Province' for state information (double). Again not good.

Next try: I put the state and LocationShownProvinceState fields in the metadata editor to blank. After write-back now, it seems fine giving 'Bengo Province' in both tags. So obviously it takes that from the XMP file, not from the database.
But this is not a practical way of working.
Exactly the same happens with City and maybe also with Location and country, but I have not encountered this myself.

I am totally confused as to what the interaction is between database and XMP file. I would have expected the database to be the 'Master' and to be reflected in the Metadata Editor. But that does not seem to be the case.

I would expect a write-back to write out metadata from the database to the XMP file and the image. But that is also not the case. If the image is rescanned after the write-back, the metadata changes and some data is actually coming from the "old" XMP file.

Could anybody enlighten me as to what is actually happening during write-back and scan

Note that I have the following settings in Metadata 2: Protect unwritten metadata/existing XMP/Rating and Label all set to Yes.
IMatch version 2020.6.2.
Maybe there is again some magic setting that I missed??

What I would really like is this:
Use Metadata Editor to update the database. Write-back should write an up-to-date XMP file. Personally I do not care if the data goes into the JPG, but I do not mind if it does. Rescan should be prioritized from XMP file.

Cheers, Jozef Dassen

[Mario]

Write-back should write an up-to-date XMP file.

This is exacly what write-back does. Write back writes metadata from the database into the XMP record and synchronizes it with existing legacy and EXIF and native GPS data.
See Metadata Write-back
See Metadata Storage

IMatch offers a variety of debugging and helper features to diagnose and solve metadata problems.
For example, it uses the ExifTool Output Panel to show you exactly which tags are written, and where.
The log file logs any problems encountered during reading or writing metadata.
The The Metadata Analyst diagnoses your image files and sidecars and shows us if there are problems.

You did not state how you changed your metadata manually, which tags you updated manually.
You did not say which other applications you have used with these files.
If the metadata is in sync, if the files have native EXIF/IPTC/GPS data.
Things like that. All this is important and may be the source of the problems you are seeing.

Please do so, include the log file from a session where you did a write back, the contents of the ExifTool output panel after a write back (ATTACH as text, not just copy&paste into your reply).
Also run the Metadata Analyst app on one of the problem files and copy the results with the GREEN BUTTON into your reply.
Upload one of your image files so we can see what it contains. if you have no cloud space, you can send your file to support email address. PLEASE include a link to this topic, because I get up to 60 emails per day. 7 days a week.

This is the minimum we need to analyze this further.

[Mario]

The sample JPEG contains standard EXIF and a legacy IPTC (IM3) record.

You also have an XMP file for a JPEG file. This is unusual. According to the XMP standard, JPEG files *MUST* use embedded XMP metadata. Not sidecar files.

IMatch by default reads the XMP file for your JPEG when populating the master XMP record it maintains in the database.
But it of course only updates the XMP record inside the JPEG during write-back. This is in accordance with the XMP specification and industry standards.
The XMP sidecar file is superfluous afterwards and can be removed (unless you have a RAW file in the same folder which needs it).

Metadata Analyst reports several mismatches between EXIF/IPTC data embedded in the JPEG and the XMP record in the separate XMP sidecar file.
The creator tag in the XMP-dc namespace contains (apparently) nonsense characters: ..          , ..
There are also some issues with the EXIF record and the IFD data record in the JPEG. Nothing too serious.


1.  I use default settings for metadata in IMatch.
In the Metadata Panel, I change the contents of the tag {Composite\MWG::State\State\0} (State/Province) from Jawa Barat to Jawa Barat IMATCH.
I let IMatch write-back the file.

+ The JPEG has now a proper embedded rich XMP record. EXIF and legacy IPTC metadata have been synchronized properly.

+ The Jawa Barat IMATCH modification was written to these tags (synchronizing between XMP and legacy IPTC in your file):
[IPTC]          Province-State                  : Jawa Barat IMATCH
[XMP-iptcExt]   Location Shown Province State   : Jawa Barat IMATCH
[XMP-photoshop] State                           : Jawa Barat IMATCH

The data shows up in the Metadata Panel and Keyword panel. The State/Provice shows the correct Jawa Barat IMATCH value.

2.  I now fix the corrupted IPTC by-line /artist by entering IMATCH/AUTHOR in the Author tag in the Metadata Panel.
I write back again.

Metadata Analyst now no longer shows the warning. The new value for Author was correctly propagated to these tags:
[IFD0]          Artist                          : IMATCH/AUTHOR
[IPTC]          By-line                         : IMATCH/AUTHOR
[XMP-dc]        Creator                         : IMATCH/AUTHOR
[XMP-tiff]      Artist                          : IMATCH/AUTHOR
[XMP-xmpDM]     Artist                          : IMATCH/AUTHOR

3. I run a forced rescan. The metadata remains as entered by me - in the file and the MD/KW panels.

Looks all perfect to me.

[jmhdassen]

OK, so you use MWG tags to update location. I used photoshop tags.
This partially solves the problem. But things look FAR from perfect here....
For one thing: I have the "old" version of JPG, which I sent, there I can now update the state through MWG\State tag as you did.
I made a new version of the JPG with the latest version of Capture One and there the behaviour is different. If I change MWG\State from Java Barat to Jawa Barat, then photoshop\State and locationShownState change properly to Jawa Barat, but the MWG\State reverts back to Java Barat after write-back.

Too me it all seems terribly confusing, messy and unpredictable. I will try make a new set of files for the different cases, but it will take some time. There are so many issues.... It would help if you could publish a flow diagram of interaction between JPG, XMP file and database during write-back.

In my opinion a JPG file is what it is. People use a JPG file to view an image. There are many cameras and programs that create JPG and obviously not all follow metadata standards to the letter. If the image can be seen, the JPG is "good". Users have no control over how metadata is embedded.

Therefore, myself, I NEVER rely on embedded metadata. I trust only XMP file. For me that is the ultimate reference, beside a database (but a database is local to the application. I do not want to touch that). To communicate metadata to consumer apps, I rely on XMP files.
So needless to say that XMP sidecar file is not superfluous, never mind what the standard says. The standard says the XMP data should be embedded. It does not forbid a sidecar file. If IMatch successfully embeds metadata then that is fine, but if it can not, for me that is not an error. I still want a clean XMP file. For you IMatch seems to be the end destination for for images. For me it is a node in a workflow chain.

Getting my metadata in good shape looks to be much more complicated than I thought it would be, even with IMatch.

Regards, Jozef Dassen

[Mario]

OK, so you use MWG tags to update location. I used photoshop tags.

No. I use the standard tags IMatch displays for location in the Default layout in the Metadata Panel.
You cannot just pick random tags, update them and then expect all will work out as you think it will work. Metadata is much more complex. Which is why IMatch shields you from it.

Many tags must be synchronized between XMP, EXIF, IPTC and GPS during write-back. Many tags are linked and changing one tag requires to change other tags.
And if you change the wrong tags, your changes may be overwritten when IMatch synchronizes XMP into EXIF, IPTC and GPS during write-back.
I have provided links to the relevant information yesterday already.

Adding legacy IPTC IIM data, as in your case (sample file), complicates things further. Because legacy IPTC does not know a thing about character sets, has severe field limits and more. Which is why many users (including myself) have got rid of legacy IPTC years ago.

For one thing: I have the "old" version of JPG, which I sent, there I can now update the state through MWG\State tag as you did.

So I have wasted my time in testing a "wrong" file?

Too me it all seems terribly confusing, messy and unpredictable.

It seems to work pretty well for most users. IMatch does a great job in shielding you from metadata complexity and makes it just work.
Just update metadata with the features provided by IMatch. Use the Metadata Panel. Use the Keyword Panel. Use the Map Panel to set and change GPS data.
IMatch then automatically takes care that metadata is properly written and synchronized.

C1 is known for its mediocre metadata handling and as far as I know, all C1 users add and edit their metadata in IMatch for that reason.

Users have no control over how metadata is embedded.

Wrong. There are established standards (EXIF is 30 years old, XMP almost 20).
The EXIF and XMP metadata standards make it very clear, where and how metadata has to be created, modified and written.
Cameras usually embed proper EXIF data. Some newer models even a rudimentary XMP record.

If you let arbitrary software wrangle the metadata in your JPEG files, you should re-think your workflow.
Much damage has been done that way.
Usually users recognize the sad state of their metadata for the first time when they look at the files with ExifTool, or more comfortable, in IMatch.

Therefore, myself, I NEVER rely on embedded metadata.

I would think about this again. Embedded metadata is super-important. Because it keeps the data where it belongs - in your files.
Metadata embedded by IMatch can be used in Windows Explorer, Photoshop, Affinity products all XMP-aware applications on all platforms. Independent from IMatch.

Don't make your life (or mine) hard by not sticking to standards or trying to roll out your very own metadata management schema.
Others have tried and regretted this. Stick to the standards, use embedded metadata where required by the standards, use sidecar files where required by the standards.
IMatch automatically takes care for all that with its default settings. Don't work against that by not doing things like using XMP sidecar files for JPEG or similar. This does not work.

To recap:

Use the tools IMatch provides to create and edit metadata.
Don't pick and update arbitrary metadata tags just because their names sound right.
In general, only update XMP tags.
Use the layouts provided in the Metadata Panel. Use the Keyword Panel. Use the Map Panel. Use reverse geo-coding to fill in location data.
Stick to the standard tags IMatch defines (The Tag Selector).

More is usually not needed in typical professional or amateur workflows.

Use the Metadata Analyst to validate your files and to check for metadata issues.

Metadata is complex. If you for some reason need to update specific tags, make sure you have studied the XMP and EXIF / IPTC / GPS standards to understand which tags to update, which tags are linked in some way, and which tags have to be synchronized during write-back between XMP, legacy IPTC, EXIF, IFD and GPS.

[Mario]

The user provided another sample JPEG file via email, and a XMP sidecar file.

I have looked at the JPEG and it contains an embedded XMP record created by C1.
The XMP sidecar file contains a different set of keywords, different MWG location data and does not list the software that created it.

Changing the MWG files reported by the user in IMatch again caused a correct update of the XMP metadata in the JPEG.
IMatch does not update the XMP sidecar file for JPG files, as required per XMP standard.
Working OK.

Deleting the XMP sidecar file causes no problems and restores the proper default and ensures that there must be only one source of truth for metadata. And for JPEG files, this is the embedded XMP record.
Keeping separate XMP sidecar file for JPG images around, using different software which either updates the embedded XMP record or the sidecar, fiddling with advanced XMP metadata options to somehow force IMatch to break established standards etc. are not purposeful. Definitely not recommended. Stick to the standards and everything works just fine.

[jmhdassen]

I guess the upshot is that you do not want to read my problems properly and that you are trying to tell me to adapt my workflow to how you think it should be. Saying you were use the wrong JPG is totally ridiculous. It was JPG created by an OLD version of Capture One. I would expect a Catalog to handle old as well as new JPG. I can not reprocess all files every time a new version of Capture One comes out.

I was hoping IMatch was a TOOL and not a straight-jacket.

You have just not answered any of my issues and all you can say is "everything works as it is supposed to be".
Totally unhelpful I am afraid. You have not explained ANYTHING. Just told me I have the wrong files, I use the wrong tags, don't follow standards etc. But never you explain why and there is no proper guideline on how to update Metadata properly.

I guess I go back to complete developing my own nodejs based Catalog. At least that will be more flexible and platform independent.
I regret to have wasted so much time with IMATCH.

Bye, Jozef Dassen

[Mario]

You have not explained ANYTHING.

I have analyzed your both files. And everything works as it should.
I have explained my tests, the state of your two sample images, the metadata mess they contain and the results I get in both this thread and also via email.
There is not much else I can do.

I was hoping IMatch was a TOOL and not a straight-jacket.

IMatch excels when it comes to metadata processing (thanks to ExifTool) and you may not find another DAM which gives you that amount of freedom, standard compliance and versatility.

I guess I go back to complete developing my own nodejs based Catalog.

Whatever works for you is good.

IMatch works great for many, many users in 70 countries. It may not work for you.

[Carlo Didier]

I guess the upshot is that you do not want to read my problems properly and that you are trying to tell me to adapt my workflow to how you think it should be. Saying you were use the wrong JPG is totally ridiculous. It was JPG created by an OLD version of Capture One. I would expect a Catalog to handle old as well as new JPG. I can not reprocess all files every time a new version of Capture One comes out.

I was hoping IMatch was a TOOL and not a straight-jacket.

You have just not answered any of my issues and all you can say is "everything works as it is supposed to be".
Totally unhelpful I am afraid. You have not explained ANYTHING. Just told me I have the wrong files, I use the wrong tags, don't follow standards etc. But never you explain why and there is no proper guideline on how to update Metadata properly.

I guess I go back to complete developing my own nodejs based Catalog. At least that will be more flexible and platform independent.
I regret to have wasted so much time with IMATCH.

Bye, Jozef Dassen

Please stay cool. Mario just explained what the standards are.
Standards are made to simplify interoperability.

If you deliberately work against the standards, whether by your own direct actions or by using software which does not respect the standards (not Marios standards, but those from the creators of XMP for example!), that is your choice but it is then also not the fault of Mario or iMatch if it doesn't work correctly.

If you want to use a hammer by holding it the wrong way and then you hurt yourself, it's not the fault of the hammer ...

[sinus]

I regret to have wasted so much time with IMATCH.

Bye, Jozef Dassen

I think, time is never wasted.
Your time with darktable, C1 or your own js-system was also not wasted, I think.

We have also a bit to lern a software, like you had to lern more about your Nikon Z7, you could also not stay in the old Nikormat-times, or could you?

You can also not lern, say Photoshop or InDesign, in a very short time. Or, what is also possible, you use really only a fraction of it.

You could read about Metadata for example here:
https://www.photools.com/help/imatch/#base_work_md.htm

But it can be, that you did already and have more questions. Then this forum is a good thing for this.
From my point of view Mario has explained quite a lot and he has checked your files quite quickly. (try this with other software  ).

If you did not like the answers, simply ask more.

[herman]

[...]
Therefore, myself, I NEVER rely on embedded metadata. I trust only XMP file. For me that is the ultimate reference, beside a database (but a database is local to the application. I do not want to touch that). To communicate metadata to consumer apps, I rely on XMP files.
So needless to say that XMP sidecar file is not superfluous, never mind what the standard says.
[...]

I am a little bit late  to the party, but there may be a way to do what you want to do.

When I understand you correctly you want to have your added / changed / whatever XMP metadata in an XMP sidecar, accepting the risk that metadata in the JPG gets out-of-sync with what is in the sidecar.

I think this can be achieved by forcing IMatch to use XMP sidecar files for the JPG file format.
This setting can be found under Edit - Preferences - Metadata 2 - Configure File Formats - scroll down to JPG and change the settings there accordingly (see attached screenshot).

As has been said before in this topic, this is a highly non-standard-shoot-yourself-in-the-foot configuration, but it can be done and it may meet your requirements.
Please realize that, when you run into unexpected side-effects, you may be on your own.

Having said that, I do the same thing for DNG files.
My current cameras shoot DNG as their native raw format, so I prefer to treat DNG as another raw file format, regardless of what standards specify.
It works for me, but I don't use keywords, geolocation or other information that has to be written into the image file. When I use star ratings they get written into the XMP sidecar (but of course other applications may or may not read that info).

[jmhdassen]

As may be clear from above discussion, I have been having a lot of problems with metadata updates. I am mainly concerned about Location data here. Since the people who created the Standards had probably very limited international knowledge, they gave us only country/state/city/location which is not enough for most countries. As a result especially State and City become very ambiguous. In an effort to clean up 20 years of images I had to change State and City a lot.
It seemed to give almost random results in IMatch. Some were OK, others not. I tried all the different settings of Metadata2 with not much luck.

But now finally I got the solution.

It would seem that something goes wrong during writeback when LocationShown data exists in the XMP file and I try to update the photoshop location tags in the Metadata Editor. After I deleted LocationShownCity and LocationShownProvinceState from ALL my XMP files, suddenly everything works perfectly. I already did 15000 writeback without a single error.

Just 2 simple commands did the trick:
     sed -i '/LocationShownCity/d' 20*/*xmp
     sed -i '/LocationShownProvinceState/d' 20*/*xmp

Jozef Dassen

[jmhdassen]

Thank you Herman,

Your answer is to the point and works. I do get the XMP sidecar all the time.
But as I was trying to explain there is another problem. If exiftool for some reason fails to write all the XMP metadata, then writeback/rescan will modify the database again. So then I have a correct XMP file which is out of sync with a incorrect database.

So hopefully Mario will be able to fix the two scenarios under which exiftool fails to write the correct data.

Thanks, Jozef Dassen

[herman]

[...]
So hopefully Mario will be able to fix the two scenarios under which exiftool fails to write the correct data.

I would suspect that there are two conflicting sources of data, one inside your JPG and one in the sidecar.
As has been suggested in another topic clearing the IPTC info in the JPG may solve this situation.