Another baffling metadata question

Started by Panther, September 07, 2016, 07:31:15 PM

Previous topic - Next topic

Panther

OK, just when I thought I had this figured out I've encountered another baffling issue with date-related metadata tags.

I'm trying to figure out how to track and be able to change two particular metadata date-related tags - specifically, the ones that are affected when you use the command menu to modify time/date (Commands/Image/Modify EXIF Date and Time).

After some trial and error, I thought I had it nailed, and it works as expected on 2 of my databases (my testbed, and my wife's production one).  Specifically, the one tag shown in the metadata panel mouse-over as "XMP::photoshop\DateCreated\DateCreated\0" is being changed when you tell the Commands/Image/Modify EXIF Date and Time box to change the "Original" date (see first attachment), and the tag shown in the metadata panel mouse-over as "XMP::xmp\CreateDate\CreateDate\0" is being changed when you tell the Commands/Image/Modify EXIF Date and Time box to change the "Digitized" date (see second attachment).

In those two databases, these two tags change properly, independently of each other, depending on which of the tick boxes (Original versus Digitized date) is selected in the Commands/Image/Modify EXIF Date and Time box.  So far, so good, but here's where it gets weird.

In my third database (basically a duplicate copy of the structure of my wife's production database, just with different files/images in it), which has the metadata panel tags set up exactly the same way as the other 2 databases, the behavior of the Commands/Image/Modify EXIF Date and Time is different - in that database, BOTH of these tags change together when the "original" date box is ticked, and neither tag is changed when the "digitized" date box is ticked.

I set up the metadata panels the same way in all 3 databases and have checked the mouse-over descriptions as well as the descriptions you see when you "edit" the metadata panels and all 3 databases say they are using this same pair of tags.  I didn't consciously change anything (in terms of setting up any metadata templates or translations or anything like that) between these 3 databases, so why are 2 working one way and the third working differently?

Is there some sort of setting I might have accidentally activated that would equate these two tags in the third database while leaving them separate/independent in the other two databases?


Mario

The Modify EXIF Date & Time command is a very special command and should only be used when really needed - to fix wrong dates and times in files.
Usually you change the date and time of files via via the Metadata Panel.

The confusing names are not of my doing. XMP and EXIF use different names for the same thing, and Adobe came up with yet other names, as well as ExifTool. Created and Digitized are the best names IMHO because a file may be digitized on a different day than it was created (consider a photo taken in 1950 and later scanned (digitized) in 2016).

Tip: You can see the instructions IMatch sends to ExifTool when you use this command in the ExifTool Output Panel (<F9>,<O>).
IMatch updates not only the EXIF dates but also synchronizes IPTC and XMP timestamps as required by the MWG standard. If there is a problem with the existing data in your test files, the results may be confusing.

Check what IMatch is writing and check what's already in the file.
You can also use the ExifTool Command Processor to see the raw metadata in the file, before IMatch performs the mapping between EXIF and IPTC into XMP during the import of the metadata. What you see in the Metadata Panel is XMP metadata, produced from EXIF and IPTC in your files. IPTC date and time overrides EXIF date and time (!!) because it is considered as added by humans and thus overrides EXIF timestamps written by the camera. Check it out.

All EXIF, IPTC and XMP timestamps must be synchronized, which causes additional grief because EXIF does not support time zones, but IPTC and XMP do.  See the corresponding info in the help for the Edit > Preferences > Metadata and > Metadata 2 dialog boxes.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Panther

Mario - thanks as always for the prompt and thorough response.

I actually am using the Modify EXIF Date & Time command "when really needed - to fix wrong dates and times in files" - it started when we found out that one of our cameras had not been set to the right time zone for a particular vacation trip and I was trying to change all those times for the pictures taken by that camera by subtracting an hour.  That process worked perfectly in two of my databases, but for some reason it is not working in the third one, which has the metadata panels set up exactly the same way as the two in which that command is working properly.

I can see the metadata in the files by looking at the metadata panel, and I can see that this command changes the metadata properly in the two databases but does not do so in the third one.

And it's not any confusion about the names that is bothering me - what's bothering me is that telling that command to change only one of the two tags works correctly in 2 of the databases, but telling that command to change only one of the two tags changes both tags  (or neither tag) in the third database, when iMatch is telling me I'm using the same two tags in all 3 databases.

iMatch is telling me that the tags being used in each metadata panel are the same in all 3 databases.  Whatever may be going on with the data in the background (and I will check through all the things you mentioned later tonight when I get back from work), if this command doesn't work consistently when all the same tags are being used it's going to be hard to rely on it (and unfortunately I think we're going to have a lot of pictures from trips where one or more of the cameras were out of synch with each other as far as times go since we weren't really thinking about ever using the metadata when we were taking those trips).


Mario

The Modify Exif Date and Time always works the same. It does not depend on metadata options or anything.
Check what's in the file (ET Command Processor) and if IMatch shows the same data in the MD panel.  If not, there is a problem with mapping EXIF / IPTC to XMP. In this case, check for IPTC data contained in the image. It will override EXIF data on import!
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Panther

Thanks for the reply.  It's obvious I'm going to have to go to metadata school to figure this out (currently I have no idea what "ET Command Processor" is or how to "check for IPTC data contained in the image") - looks like I have a lot of reading to do in the help file and maybe elsewhere, so delving deeper into this will probably have to wait for the weekend.

Based on the last few responses and some additional tests, it's pretty clear to me that I haven't yet found the right 2 tags for what I'm trying to do here - there are still too many situations arising where sometimes these tags are being treated independently of each other and sometimes they are being treated as if they were actually the same tag or as if they were depending upon the contents of some other tag that I'm not aware of or in control of yet. 

I found another situation tonight in my wife's production database (one where the Modify EXIF Date & Time command has been properly changing these tags separately/independently as expected) where when I manually change the date/time in one of these two tags and write that new info back to the metadata by clicking the pencil, iMatch changes both tags (even though the mouse-over on the pencil says it is only going to write back one tag).  But when I tried that same sequence of events on my testbed database (the other one where the Modify EXIF Date & Time command has been properly changing these tags separately/independently as expected), iMatch only changed the one tag I was intending to change and left the second tag alone (as I would have expected it to do).

There has to be some way to find and use the right 2 tags that will function independently of each other and be of high enough priority in whatever algorithm is being used to rationalize or synchronize between all these competing/overlapping tags so as not to be over-written unless the user specifically asks for them to be, and hopefully the answer is lurking somewhere in the help file just waiting for me to find (and understand) it.

Mario

There are only created and digitized tags relevant for XMP. And the modified tag maybe. These are the only tags for which the MWG has defined mapping rules between EXIF, IPTC and XMP. And IMatch does this. Changing one of the tags has never an impact on the other tag (created / digitized), Match never treats them the same.

ET Command Processor means "ExifTool Command Processor" and is a feature of IMatch. If you don't know about EXIF and IPTC metadata, you should not change this data. Best stick to the Metadata Panel and change the dates and times there. Usually the "Modify EXIF date and time" command just works and is fully integrated into IMatch. If this does not work, something in the metadata in your files is wrong, and this is when it gets complicated and requires additional know-how, using the ECP to check for broken data, maybe manually delete legacy and damaged IPTC data and the like.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Panther

Quote from: Mario on September 09, 2016, 08:42:19 AM
There are only created and digitized tags relevant for XMP. And the modified tag maybe. These are the only tags for which the MWG has defined mapping rules between EXIF, IPTC and XMP. And IMatch does this. Changing one of the tags has never an impact on the other tag (created / digitized), Match never treats them the same.

ET Command Processor means "ExifTool Command Processor" and is a feature of IMatch. If you don't know about EXIF and IPTC metadata, you should not change this data. Best stick to the Metadata Panel and change the dates and times there. Usually the "Modify EXIF date and time" command just works and is fully integrated into IMatch. If this does not work, something in the metadata in your files is wrong, and this is when it gets complicated and requires additional know-how, using the ECP to check for broken data, maybe manually delete legacy and damaged IPTC data and the like.

Thanks for the additional info, though it doesn't do anything to raise my confidence level.  I've never messed with any of the metadata myself, except through the iMatch metadata panel, so if this odd, unpredictable behavior I'm seeing with iMatch changing tags when I don't want them changed is due to "something in the metadata in your files is wrong" instead of just me not having identified the right two tags to use yet, then I'm afraid my desire to use and rely on the metadata is doomed to failure from the start.  These files are coming from probably 15-20 different cameras, and maybe 6 or 8 different scanners over the last 20+ years, and there are tens of thousands of them, so the chances of there being some multitudes of ways there could be "something in the metadata in your files is wrong" is probably huge, and the chances of being able to manually, individually figure out what those problems are and fixing them are slim to none.

I suspect that I have seen a few files where the metadata must indeed be faulty - when first added to the database and selected in iMatch, there is no date/time info shown in the metadata panel for either of these two tags.  But when I try to add any metadata (e.g., the author) and write it back to the file, iMatch puts the current date/time into these two tags (even though mousing over the pencil says iMatch is only going to write the author tag).  While this is also undesirable from my standpoint (I would have preferred that the dates just stay empty), I had assumed that once iMatch tried to write any metadata it was discovering the "error" of missing date info and perhaps was required to fix it to avoid blowing up somehow, and I figured I could work around that.  And while undesirable to some degree, there aren't that many files that I've seen so far with no date/time info at all, so this little bit of strangeness is probably manageable.

But these situations where the tags have been populated before I start working with them, but the Modify EXIF Date & Time command sometimes changes the tags independently and sometimes changes them together (when I didn't ask it to), or where manually changing one tag in the metadata panel sometimes changes only that tag and sometimes changes both tags, just make the whole concept of relying on these metadata tags unreliable.

Maybe when I read up on this stuff this weekend I'll find that there's some way to have iMatch find, display and fix large groups of files that it thinks have damaged metadata so I can do this somehow in batches as I bring the files into the database, and perhaps keep this issue from getting completely out of hand.  When it works, the timeline view and other aspects of how iMatch lets you use the metadata info are very cool and really make me want to be able to rely on using it, but so far the results in practice have been way too erratic to instill any confidence about the metadata.

Mario

That's just a fallback by ExifTool. If a file has neither created nor digitized, it uses the "last modified". This is how it should be.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

jch2103

#8
Quote from: Panther on September 09, 2016, 04:37:19 PM
These files are coming from probably 15-20 different cameras, and maybe 6 or 8 different scanners over the last 20+ years, and there are tens of thousands of them, so the chances of there being some multitudes of ways there could be "something in the metadata in your files is wrong" is probably huge, and the chances of being able to manually, individually figure out what those problems are and fixing them are slim to none.

Given the history and origin of your files, it's all but inevitable that there will be issues with metadata. Having been through some similar issues, I'd recommend a stepwise approach to identifying and fixing the problem files. Unfortunately, since they could be messed up in a myriad of ways this may not be a simple process.

IMatch's data-driven categories may be a good place to start. See the Help and the example data-driven categories if you haven't used them before. Attached are screenshots of two data-driven categories that might help or at least be a start.

1. EXIF error. This one can help identify files that have an EXIF error and need to be individually addressed. When I started my cleanup of old files, I had a number of these.

2. No date assigned. This one could be a basis for looking for files that have or are missing particular time/date tags. In my case, I was looking for scanned images that had never had an assigned XMP::photoshop\DateCreated\DateCreated\0, so that I could add the date the original image was taken. I was lazy when I set this up, so it separately shows files taken in the 21st century (year 2000 and later) and files taken before that time (I don't have any files that pre-date the year 1000  ;)). The group I was looking for was 'Other'; i.e., files without any assigned dates. You may need to set up several of this kind of data-driven category, depending on the issues hidden in your files.

All this may seem like a daunting task, but IMatch is the only product I know if that gives you a full set of tools for cleaning up years of accumulated problems. Good luck! Let us know if you have questions, etc.
John

Panther

Thanks for posting those tips and examples.  I tried to replicate the first one and create a data-driven category to detect/collect files with errors, but I can't get the program to accept the tag you used as shown in your attached picture ("Extra\Error\Error\0") as a valid tag, so it won't let me set the "Enabled" line to "yes" or complete/save the category.

I tried poking around in the metadata tag selector screen to see if I could find any "extra" or "error" related tags, but wasn't able to find any yet.  Any idea what's going wrong here?

jch2103

#10
See this thread: https://www.photools.com/community/index.php?topic=1152.msg13023#msg13023
However, I'm having problems getting data-driven categories to do what Mario suggested.

Also see this message https://www.photools.com/community/index.php?topic=1152.msg6928#msg6928 further up in the thread for a more direct method. I set up some batch files that I've attached. You'll need to edit the picture path and also either put these in the folder containing ExifTool or set a path variable pointing to ExifTool.

I suspect there have been some changes in IMatch metadata handling and tag filtering since 2014...
John

Panther

#11
Thanks for the additional info.  I'm not sure why only some of the groups of tags appear on the dropdown of the tag selection screen, which was leading me to think that there were no "extra" tags, but when I just typed "extra" into that group field they showed up and I found the "Extra\Error" tag that was shown in the picture for that data driven category.  I chose that one and it let me turn "enable" to "yes" and save the category.

It doesn't seem to think any of my testbed files have metadata errors, which may be true (there are only about 100 images and they pretty much all came from my iPhone within the last couple of years), so I'll have to run some tests on some of the suspect files I encountered in my other databases and see if I get any hits with this approach.

[EDIT] Well, not sure but I suspect this approach isn't really working - I added a little over 2,000 of the files from my main database to my testbed, and neither the data driven category nor the value filter described in one of those threads posted about above report any files with metadata errors.  I suppose that might be possible but doesn't seem likely.  When I get more time tomorrow I'll have to try to track down the specific files that we've had issues with over the last few days and see if I can make them show up in these error category/searches.

Mario

#12
QuoteI'm not sure why only some of the groups of tags appear on the dropdown
This is a convenience feature. IMatch remembers the last used groups and presets the group field with frequently used groups to aid new users. There are currently almost 900 (!) groups and it would make no sense to list them all in this drop down.

The Extra\Error tag is filled by ExifTool when it detects problems when writing back to images (mostly). IMatch explicitly imports this Extra tag (as one of only a few) to manage the display of the yellow "warning" icon in the file window, indicating write-back problems and to show the message returned by ExifTool in the tooltip.

This is not a reliable indicator for general problems with files. In the rare cases where IMatch/ExifTool fail to properly standardize and map metadata between EXIF/IPTC/XMP it was usually broken/incomplete IPTC data or a really damaged metadata record. It's hard to find such issues automatically, at least without writing some ExifTool batch files or maybe an IMatch script which looks at the metadata checks for typical problems. But these cases are so rare, it's just not worth the time, sorry.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Panther

Mario - again, thanks for the response and information.  That all makes sense to me, I think.

Given that trying to use that tag to find errors wasn't working out, I went back into the production database where I was seeing the problem with the "Modify EXIF Date & Time" command changing both tags when I only told it to change one tag.  I remembered one of the files this was happening on, and added a copy of that file to my testbed database (where the Modify EXIF Date & Time command has been working just fine).  Sure enough, when I use the Modify EXIF Date & Time command on that file in my testbed database, it changes both tags, even while still changing only the desired tag on other files in that database. 

I copied the exif output screen contents from both a successful run of the same Modify EXIF Date & Time command on one file (where it only changed the one desired tag) and an erroneous run on the suspect file (where it changed both tags) into text files and have attached them to this post.  I can't make any sense out of them, and they both report errors of some kind, but maybe they will shed some light on what's happening here.

The suspect file itself is a 17 MB .tif file that is too big, even using 7-zip's ultra 7z compression settings, to attach here, but I'd be happy to send it to you somehow if it would help to inspect the file itself.

Mario

#14
IMatch is only updating one date/time stamp: (in your "both" tags file):

-XMP-photoshop:DateCreated+=0000:00:00 01:00:00

-EXIF:DateTimeOriginal+=0000:00:00 01:00:00

-IPTC:DateCreated+=0000:00:00 01:00:00
-IPTC:TimeCreated+=0000:00:00 01:00:00


so IMatch is performing correctly. If you experience that both dates are changed in the file, this is not caused by what IMatch is writing. You can send the file to my support email address (https://www.photools.com/support/) or just provide it via your cloud storage.

Did you try to just strip the legacy IPTC data from the file as I suggested near the top of this thread? If this solves the problem we can all save time.
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

Panther

I'm not sure exactly what you said in the earlier posts about stripping the IPTC data (much of what you said was still over my head at this point), and I was trying to heed your advice about not messing with metadata other than through the metadata panel if I didn't understand what I was doing, but I'll play around with it some more later today when I get some time and see if I can strip out the IPTC data and see if that changes things.

I can watch in the metadata panel as the operation is progressing and it clearly shows both metadata tags being changed when I run that command on the suspect file, and only one tag being changed when I run it on the other files, and I'm not doing anything else but using iMatch at the time, so I don't understand how "this is not caused by what IMatch is writing", and I don't know why the exif output info doesn't show the fact that both tags have been changed when they clearly have been, but I'll keep digging.

If I can't figure out what's going on with the IPTC data in the file I'll send you the file or a link to it.

Thanks for taking the time to help!

Mario

#16
When you say that both the created and digitized timestamps (not modified, this is normal to change!) are changed when IMatch only writes one timestamp, the problem is the data in the file. Without having the file here I cannot say more.

Analyzing broken metadata in files is tedious and does not benefit more than one user usually. Please allow for several days after I have received your file. I'm still rolling out the new web site, the new community server, the current IMatch AnywhereTM beta versions and I'm also preparing the IMatch 5.7 update. Busy bee, me.

-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook

jch2103

Quote from: Panther on September 10, 2016, 06:16:58 PM
I'm not sure exactly what you said in the earlier posts about stripping the IPTC data (much of what you said was still over my head at this point), and I was trying to heed your advice about not messing with metadata other than through the metadata panel if I didn't understand what I was doing, but I'll play around with it some more later today when I get some time and see if I can strip out the IPTC data and see if that changes things.

See https://www.photools.com/community/index.php?topic=5183.msg35959#msg35959 for more on stripping IPTC data. Doing this resolved a number of problems I was having with metadata/files.
John

Panther

#18
Thanks!

Well, now I've learned to use the ExifTool Command Processor in iMatch to list all the metadata and to strip out the IPTC metadata from a file, and that part of the process seems to work (listing all the metadata after stripping the IPTC data shows no IPTC data in the file), so that's good.

But unfortunately that doesn't seem to have solved the problem/mystery - the Modify EXIF Date & Time command is still changing both of these tags on this suspect file when it's been told only to change one of the tags and when it properly changes only the one correct tag for other files in this same database.

I hate to take up any more of Mario's time with this, but my wife is understandably reluctant to invest the time/effort to put her tens of thousands of images into this thing while we still don't know why these date tags are acting flaky like this or how many times we're likely to run into similar problems or how to fix them if they do crop up.

Mario - I'll go ahead and send you the file or a link to the file (the original version before I started messing with it)  - I understand it may take you a while and that's no problem. 

I copied the output of the list-all-metadata run I did on this original version file copy with the ExifTool Command Processor and pasted it into a text file and attached it to this post, in case anybody can maybe spot something if they feel like a challenge :)


jch2103

It's hard to tell what's going on regarding your attached list-all-metadate file without the image file. I do see that the ECP output includes IPTC data, so the ECP appears to have been run before you removed the IPTC data.

I assume part of your workflow and your objectives involves scanned images and the need to give them an original date and a (separate) digitized date (to simplify metadata nomenclature). At least that's the process I've gone through with scanned images.

In these kinds of cases, I've found it most useful to
1. group selected images,
2. delete the legacy IPTC data using the ECP and then
3. use a metadata panel to add/update the 'original' date/time (XMP::photoshop\DateCreated\DateCreated\0) and either add/update the scanned/digitized date/time (XMP::xmp\CreateDate\CreateDate\0) or leave it as is.

By the way, in my post https://www.photools.com/community/index.php?topic=5861.msg40486#msg40486 I mentioned a data-driven category for No Date Assigned. I forgot to mention that I've set the color coding (via category Properties) to highlight files with missing dates in the File window. That makes it much easier to spot and work with images that need attention.

Also, as I mentioned in another post, once you have the 'original' date/time (XMP::photoshop\DateCreated\DateCreated\0) set, the Timeline view will show you files in the correct date/time sequence.

As Mario mentioned previously, 'Commands/Image/Modify EXIF Date and Time...' is best suited to making corrections for things like incorrectly set camera time/date.
John

Panther

#20
Thanks!

Yes, that list of all metadata was before I did anything to the file (like strip the IPTC).

When I was perusing the list of all the metadata in that file I was struck by one thing in particular, and that was the traces of the name of the old program I was trying to use before I found out about iMatch in some of the tags, such as the following:

[XMP-tiff]      Software                        : Elements Organizer 8.0

(In fact, it was the crashing of that program and the horrible customer service/support from Adobe that drove me to look for something else and led me to find iMatch).

On a hunch, I searched my production database for all files with that value in that tag and found about 600 of them - and all of the half dozen or so of those files I tested displayed erratic date tag behavior.  I tested about a half dozen of the files without those Elements traces, and the date tag behavior in those worked just fine.  So, I'm still not sure exactly what is broken in those files or how to fix it, but I suspect I've probably located the root cause and a way to find the problematic files at least, and it makes me hopeful that the vast bulk of my files and all of my wife's files (which never came near the Elements program) might be OK in this regard.

I still need to figure out how to fix these 600 or so probably-broken files, but at least that's shaping up to be a more manageable problem.

Thanks for all the tips and thread links - always nice to learn something along the way to getting something done.

Oh, and yeah, as far as my work flow/intended process goes, (1) I do expect to use the "Commands/Image/Modify EXIF Date and Time" process only to change/correct the date/time info in pictures taken by digital cameras that weren't set to the right time when they were taking the pictures, and (2) for the scanned images I don't really care (at this point anyway - maybe I'll find a reason to care about them later) about the digitized/scanned date but I do need to be able to set the original date to a different date reflecting the real date of the scanned picture or document and to have that original date not be changed/impacted in weird ways that I don't expect.


Mario

I have checked the file you've sent. It starts with the following date and time info:

[ExifIFD]       Date/Time Original              : 2016:05:22 23:26:47
[IFD0]          Modify Date                     : 2016:09:10 01:53:50
[IPTC]          Date Created                    : 2016:05:22
[IPTC]          Time Created                    : 23:26:47-04:00
[XMP-exif]      Date/Time Original              : 2016:05:22 22:26:47
[XMP-xmp]       Modify Date                     : 2016:09:10 01:53:50-04:00


(I used the "All Dates" preset in the ExifTool Command Processor to see the data in the file).

Now I changed the Original Date and Time relative by + 1 day  and +1 hour using te Commands > Image > Modify Exif Date and Time command. The result looks correct, no digitized date has been added, only the existing info has been updated and synchronized (my time zone is UTC+2).

[ExifIFD]       Date/Time Original              : 2016:05:24 00:26:47
[IFD0]          Modify Date                     : 2016:09:12 19:00:01
[IPTC]          Date Created                    : 2016:05:24
[IPTC]          Time Created                    : 00:26:47-04:00
[XMP-exif]      Date/Time Original              : 2016:05:22 22:26:47
[XMP-xmp]       Modify Date                     : 2016:09:12 19:00:01+02:00
-- Mario
IMatch Developer
Forum Administrator
http://www.photools.com  -  Contact & Support - Follow me on 𝕏 - Like photools.com on Facebook