Author Topic: [OFFICIAL] - FEEDBACK WANTED  (Read 828 times)

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
[OFFICIAL] - FEEDBACK WANTED
« on: March 02, 2019, 02:51:35 PM »
After (again) spending valuable time analyzing a metadata problem reported by a user,
're-had' the idea (for the umpteenth time) to write an app which detects and reports common metadata problems.

The new app is named the Metadata Analyst.

This is how the app works:

+ The user focuses a 'problem' file in the active File Window and runs the app
+ The reads some IMatch settings (to learn how the user has configured the various metadata options)
+ The app extracts the metadata from the focused file and also from the XMP sidecar file if one exists
+ The app now runs various tests against the metadata to find issues

Results

The app displays the results in a human-readably form.
This includes the results of the tests, all problems found, the full metadata extracted from the file(s) and the relevant IMatch settings

The app has a feature to save all results to a plain text file.
This makes it convenient for the user to attach the results to a community post or send it to me for further analysis.

Where You can Help

I have the app already working (roughly) and over the next couple of weeks I will write the tests for common metadata issues, as time permits.

This is where I NEED YOUR HELP:

- Did you run into metadata issues in the past?
- If so, what was the problem and how was it diagnosed and solved?
- Please add a description in a reply to this post. if possible include a link to the corresponding community thread.

The more problem cases we can identify, the better. This will make it easier for users and me in the future.

Example:

The support case which triggered all this was caused by a NEF file with embedded XMP data. This XMP data contained date and time information different from the EXIF and IPTC date and time information.

Since IMatch by default considers embedded XMP data more important than data in sidecar files (there is an option for that), IMatch used the "wrong" timestamps. And the user wondered where this wrong data came from, and why his changes reverted back to the original values after write-back.

This all became clear after the user has uploaded the files, I have downloaded them and then did an analysis with ExifTool.
Not terribly hard to do, but time-consuming for both the user and me.

This is something I hope to solve with the Metadata Analyst app.
It should find issues like that automatically and report them. Then either the user can understand the problem and solve it her/hisself, or at least we have much better information to work with when supporting the user.

Post your problem cases below. The more, the better.

This is a quick preview of how the Metadata Analyst will report information and problems:


« Last Edit: March 04, 2019, 07:52:25 PM by Mario »

Aubrey

  • Super Hero
  • ****
  • Posts: 1011
  • IMatch user since June 2008
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #1 on: March 02, 2019, 09:10:15 PM »
I don't have a particular case but have spent many hours with exif tool removing this and that over the years. Really not wanting to bother the community with my personal woes.

I have a series of exif commands which work most of the time.

A tool to perhaps diagnose better will be useful.

For example yesterday I had a file which was fine in IMatch but when looked at in windows explorer was had wrong rotation. After running it through windows photoviewer and rotating, it was fine in explorer, and before and also after forcing F5 crtl shft in IMatch it was also fine.....??

Aubrey
« Last Edit: March 02, 2019, 09:18:51 PM by Aubrey »

jch2103

  • Super Hero
  • ****
  • Posts: 1619
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #2 on: March 02, 2019, 10:42:17 PM »
Very timely...

I'm currently running into a problem with images processed in Lightroom from NEF to JPG. After processing, I find that I get an endless update loop after adding keywords. If I run ECP with 'Delete IPTC Metadata in file' (-overwrite_original_in_place -IPTC:All= {Files}) and then write one more time, the problem goes away.

LR isn't my normal raw processor (waiting for DxO PhotoLab update for my Nikon Z6, due this coming Wednesday), so there may be a LR setting I missed that would prevent this.
John

JohnZeman

  • Global Moderator
  • *****
  • Posts: 1123
  • I'm too damn old to act my age.
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #3 on: March 03, 2019, 02:11:58 AM »
Like Aubrey I currently don't have a need for this but I have in the past and I'm sure I will in the future so this will be a nice tool to have.

Right now I'm in the middle of a massive total reorganization of half of my database.  I've added all of the original raw files from my Lightroom database to IMatch and am preparing for a migration from Adobe to Affinity Photo for raw processing.  This is turning out to be a multi 24 hour day project but I can already tell it's going to be so much better once it's done.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1199
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #4 on: March 03, 2019, 08:49:52 AM »
The only problem I had was to get rid of keywords. I couldn't find out how and when they got in the files and I had a hard time getting rid of them with exiftool.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #5 on: March 03, 2019, 09:32:28 AM »
The only problem I had was to get rid of keywords. I couldn't find out how and when they got in the files and I had a hard time getting rid of them with exiftool.
Which kind of keywords?

Legacy IPTC keywords?
XMP flat keywords (dc:Subject=
XMP hierarchical keywords?

Usually IMatch just synchronizes all of these on write-back.
The typical problem is IPTC keywords which don#t match the XMP keywords and which cause issues on re-import.
Or the wrong settings made by the user under Edit > Preferences > Metadata for flattening XMP hierarchical keywords which cause issues...

ben

  • Sr. Member
  • **
  • Posts: 405
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #6 on: March 03, 2019, 01:24:25 PM »
My problems are mainly related to wrong date/time information.

One big problem, that i usually don't recognize immediately is:
When iMatch cannot make up a reasonable date/time information it falls back to the Windows-LastModified date. This is always bad, since then sorting according to the date doesn't work any more.

So, the testcase for this great app would be:
"Find all files that don't contain a reasonable date/time information and iMatch has to fall back to the Windows-LastModified date"

Ben

Jingo

  • Hero Member
  • ***
  • Posts: 907
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #7 on: March 03, 2019, 01:53:05 PM »
Same here... luckily - no major metadata woes in all the years I have been keywording.. besides some date issues... seems dates are one of the key issue makers with Exif, IPTC and XMP... many many fields that all sound and look similar and often containing different data sets.  Looking forward to seeing the APP though.... mostly from a coding perspective!  ;)

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #8 on: March 03, 2019, 02:05:55 PM »
Quote
Looking forward to seeing the APP though.... mostly from a coding perspective!  ;)

Nothing magical happening. As I wrote in the corresponding geek thread in the App board, the hard work is done by ExifTool and the cool Match processRun function which delivers the metadata in JSON format right to the app. I have created a way to run the tests that is easily extensible - because I expect to add more test cases frequently.

As so often, creating the user interface required the most work. It's easier now that IMatch by default includes the awesome Vue.js framework. Which does of of of what the massive frameworks like Angular or React do, but with a super-simple tooling and super-easy integration into IMatch apps.

Vue makes things like form processing, event handling and routing in IMatch apps a lot easier. Its a win even for simple one page apps (which are the majority) and it will be indispensable for the more complex apps we'll see this year.

I'm now collecting as many test cases as possible, trying to put them into code.

Winfried

  • Full Member
  • **
  • Posts: 107
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #9 on: March 03, 2019, 04:01:27 PM »
I see sometimes problems with exif-data-fields that should have been updated by the photo-editor during processing.
example:
{File.MD.XMP::exif\ColorSpace\ColorSpace\0}  Farbraum   sRGB
If this file is converted to AdobeRGB or eciRGB the icc-profile is embedded, but the exif-flag is not altered.

Or a photo is cropped and becomes a portrait instead of an landscape orientation.
Sometimes the corresponding exif-flags are not altered.
This can happen if all metadata from the input image are 1:1 copied to the output image.
This keeps all metadata, but some data are no longer correct.

Winfried




Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #10 on: March 03, 2019, 04:07:31 PM »
Quote
This can happen if all metadata from the input image are 1:1 copied to the output image.

This is why I warn from copying EXIF data. EXIF was never designed to be copied from one image to another.
Your image editor should take care for this.

EXIF color space can have only two 'official' values: 1=sRGB and -1=unknown. There is no value for AdobeRGB or ProPhoto or whatever. This is what the embedded color profile is for. Most applications ignore the EXIF colorSpace tag.

The EXIF orientation tag (if this is what you means) refers to the physical orientation of the image data in the file, and how it must be rotated for display.
if you crop a file ans save it back, your image editor must write this tag, because it most likely orients the data in 0 (neutral) orientation.

Not sure if the IMatch Metadata Analyst should do anything here and what?.
These values usually don't cause problems in IMatch. The Analyst shows the EXIF orientation, though.
« Last Edit: March 03, 2019, 04:30:45 PM by Mario »

sinus

  • Global Moderator
  • *****
  • Posts: 3499
  • IMatch-User since 2001 (IMatch 3.6)
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #11 on: March 04, 2019, 08:50:50 AM »
This will be for sure a good app!

My most problems have been with keywords and date.

With keywords I had (have) the problem, if there is a space after a pipe(not sure, if it calls so) (senkrechter Strich), the write-back pencil does never goes away.
If I delete this space in the keywords ({File.MD.XMP::Lightroom\hierarchicalSubject\HierarchicalSubject\0}) then after run the pencil, all is ok.

With dates from old files, including scans, there is often a mess. I do not know, where exactly write the correct date, so that IMatch does correctly sync them all.
In such cases it would be the best, if I had ONE field, where I could write the correct date, and IMatch would do the rest.
I do this know since quite a long time with the filename, I write there the correct date-time, and for such files I let write IMatch the same date-time into some fields with the metadata-template.

I put this correct date (in my case from the file-name) into the fields:
{File.MD.XMP::exif\DateTimeOriginal\DateTimeOriginal\0}
{File.MD.XMP::photoshop\DateCreated\DateCreated\0}


Maybe I should copy the correct date into other fields too, I do not know




Best wishes from Switzerland! :-)
Markus

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #12 on: March 04, 2019, 09:17:59 AM »
Thanks for your feedback. I will investigate what the problem with the extra blank in hierarchical file names could be. No idea.
Which "flattening" options to you use when mapping hierarchical keywords into XMP and IPTC keywords (do your files still have legacy IPTC data?)
Show me Edit > Preferences > Metadata.

Quote
In such cases it would be the best, if I had ONE field, where I could write the correct date, and IMatch would do the rest.

Just update the created / date subject created date and time tags in the IMatch Metadata Panel (Default layout).
These XMP timestamps are synchronized by ExifTool into both EXIF and legacy IPTC. This is usually nothing that causes any trouble. Unless you update other timestamps or you prevent IMatch from synchronizing metadata or your files contain invalidate metadata etc.

sinus

  • Global Moderator
  • *****
  • Posts: 3499
  • IMatch-User since 2001 (IMatch 3.6)
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #13 on: March 04, 2019, 10:29:54 AM »
Thanks, Mario

Here are the peferences of my metadata.
Best wishes from Switzerland! :-)
Markus

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #14 on: March 04, 2019, 10:55:38 AM »
This may be it. You write each part of the hierarchical keyword separately (non-standard). And this means that you will end up with keywords with a leading blank, but the corresponding hierarchical keyword may have none => mismatch between keywords, reappearing with every write-back => infinite loop.

I'm not sure it would be safe to strip leading and trailing blanks from keywords when writing back...? Because a user may actually want or need them. I need to analyze where the blank is lost when re-importing the keywords and mapping them back into hierarchical keywords. Not a really urgent problem, though.

sinus

  • Global Moderator
  • *****
  • Posts: 3499
  • IMatch-User since 2001 (IMatch 3.6)
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #15 on: March 04, 2019, 11:03:50 AM »
This may be it. You write each part of the hierarchical keyword separately (non-standard). And this means that you will end up with keywords with a leading blank, but the corresponding hierarchical keyword may have none => mismatch between keywords, reappearing with every write-back => infinite loop.

I'm not sure it would be safe to strip leading and trailing blanks from keywords when writing back...? Because a user may actually want or need them. I need to analyze where the blank is lost when re-importing the keywords and mapping them back into hierarchical keywords. Not a really urgent problem, though.

Ah, thanks.
Hence it will be best do change my preferences?
Sorry, I am not sure, where I should change somehing.

If you could say me, what to change, I will do it for sure.  :)
Best wishes from Switzerland! :-)
Markus

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #16 on: March 04, 2019, 01:14:36 PM »
For now, just avoid adding extra spaces in after the | in your keywords.
Tip: You should be able to find all files with this problem using the file window search bar. Just search for |<space> where <space> means a real blank.
Then fix them manually or use the Metadata Mechanic.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1199
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #17 on: March 04, 2019, 02:15:29 PM »
Which kind of keywords?

I think they were IPTC. And I never had hierarchical keywords either.

sinus

  • Global Moderator
  • *****
  • Posts: 3499
  • IMatch-User since 2001 (IMatch 3.6)
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #18 on: March 04, 2019, 02:31:18 PM »
For now, just avoid adding extra spaces in after the | in your keywords.
Tip: You should be able to find all files with this problem using the file window search bar. Just search for |<space> where <space> means a real blank.
Then fix them manually or use the Metadata Mechanic.

Thanks, Mario, I will do so.
Best wishes from Switzerland! :-)
Markus

Tveloso

  • New Members
  • *
  • Posts: 17
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #19 on: March 05, 2019, 12:48:44 AM »
Another test case might be to report when any of the existing TimeStamps in a file is earlier than the one IMatch has used for the File Date & Time. 

This is very much along the same lines as what Ben has discussed above...but even if IMatch has not had to fall back to the System Creation Date, the "proper" Date found may still be incorrect.

I have a number of Videos taken with my iPhone, where the Date in IMatch is later than when the Video was actually recorded (and it appears to be the date that the file was transferred from my phone to my PC).  For example, one file contains these TimeStamps:

Code: [Select]
[System]        File Modification Date/Time     : 2018:06:16 16:41:26-04:00
[System]        File Access Date/Time           : 2018:06:18 14:19:05-04:00
[System]        File Creation Date/Time         : 2018:06:16 16:41:26-04:00
[QuickTime]     Create Date                     : 2018:06:18 18:17:02
[QuickTime]     Modify Date                     : 2018:06:18 18:17:13
[Track1]        Track Create Date               : 2018:06:18 18:17:02
[Track1]        Track Modify Date               : 2018:06:18 18:17:13
[Track1]        Media Create Date               : 2018:06:18 18:17:02
[Track1]        Media Modify Date               : 2018:06:18 18:17:13
[Track2]        Track Create Date               : 2018:06:18 18:17:02
[Track2]        Track Modify Date               : 2018:06:18 18:17:13
[Track2]        Media Create Date               : 2018:06:18 18:17:02
[Track2]        Media Modify Date               : 2018:06:18 18:17:13
[Track3]        Track Create Date               : 2018:06:18 18:17:02
[Track3]        Track Modify Date               : 2018:06:18 18:17:13
[Track3]        Media Create Date               : 2018:06:18 18:17:02
[Track3]        Media Modify Date               : 2018:06:18 18:17:13
[Track4]        Track Create Date               : 2018:06:18 18:17:02
[Track4]        Track Modify Date               : 2018:06:18 18:17:13
[Track4]        Media Create Date               : 2018:06:18 18:17:02
[Track4]        Media Modify Date               : 2018:06:18 18:17:13
[QuickTime]     Creation Date                   : 2018:03:13 20:43:30-04:00

...and IMatch shows that June date at 6:00 PM...but it's actually that very last March date that's the correct one.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #20 on: March 05, 2019, 08:49:22 AM »
You file has about a dozend proprietary Apple QuickTime timestamps. Nothing really reliable or standardized.

IMatch picks the timestamps as explained in How IMatch uses Date and Time Information, because this method has proved to be the most reliable for the various video container formats based on MP4 with embedded QuickTime metadata.

IMatch uses [QuickTime]Create Date when it exists and not [QuickTime]Creation Date - which you consider "more right" for the files created with your device.
If none of these timestamp exists, IMatch uses the track timestamp of the earliest track.

Why your Apple device records different dates for "create date" and "creation date" - I don't know. And IMatch can neither.
Maybe you can contact Apple support and ask them? I could not find anything definite in the QuickTime documentation, but this is normal.
This would be useful to know, e.g. under which condition which value should be used.

Can you create a small sample video with the same metadata and send it to me via support email address?
I can then use it for my test and sample library.

When I would change IMatch to favor "Creation Date" instead of the proven "Create Date" I will get reports from other users with files where "Create Date" is 'more right'.

Hard to make this work for everybody.
You can handle this with an IMatch Metadata Template that copies the right date and time information into the XMP created/subject created tags IMatch uses. This will also update the IMatch FileDateTime standard attribute and where IMatch positions the file on the timeline.

I've just checked some of my more recent sample MOV files (iPhone X). Apple apparently uses XMP. Or some software used to produce the MOV file does - the Software and Creator XMP tags are not filled. But In two of the files I've checked the "created" QuickTime timestamps are different than the "Created" XMP timestamps...? More test cases to check in the MD Analyst.
« Last Edit: March 05, 2019, 09:03:06 AM by Mario »

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #21 on: March 05, 2019, 04:03:47 PM »
@Sinus

The problem with your hierarchical keywords containing an extra blank after the | is caused by the way ExifTool handles keywords when mapping between XMP and IPTC. IMatch writes the blanks into the file, but they go amiss when ET maps the keywords.

And this then causes a mismatch on re-import between the IPTC keywords and the hierarchical keywords (which still have the space) and this is the reason for the infinite reoccurring write-back pen.

Just make sure you don't add additional blanks. I've explained above how you can quickly find and fix files with this problem.

sinus

  • Global Moderator
  • *****
  • Posts: 3499
  • IMatch-User since 2001 (IMatch 3.6)
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #22 on: March 05, 2019, 07:41:32 PM »
Thanks really, Mario!!!

Helps me a lot!
And once again shows your super support!
Fabulous.  :D :)
Best wishes from Switzerland! :-)
Markus

Tveloso

  • New Members
  • *
  • Posts: 17
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #23 on: March 06, 2019, 05:15:36 AM »
Mario, I have sent some sample iPhone videos to the support email address.

As you said, changing IMatch to favor the [QuickTime] Creation Date tag, will probably break this for other (maybe most) users…and the date discrepancy I'm experiencing may be a function of how I copy the photos from my phone (connected via a USB Cable). 

As I mentioned in my email, my wife uses iCloud Photo Library (which is something that hasn't worked for me, and that I abandoned trying to get fixed).  I suspect that most Apple users do it that way too...and it may well be that files sent to the PC via that method, have better Date tags.  I'll check her PC in the morning and will report back...

This type of messiness is the very thing that the Metadata Analyst will help us track down...

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #24 on: March 06, 2019, 09:35:21 AM »
I have checked my sample libraries and also download some sample Apple MOV/MP4 files yesterday.

Quite many of these show different values in Created and CreateData. Why? Ask Apple.
Some also contain embedded rudimentary XMP data (with a software tag that refers to an Apple device as the producer) - and the XMP data has 'created' timestamps which are again different from both the QuickTime timestamps. This is pure nonsense. How to tell which to use? And that's just Apple. I will have to look at videos produced on Android devices from several vendors.

Another thing I've found. Some Apple videos have GPS data - in the QuickTime section. Not really documented, they just merge GPS coordinates into a string, together with the Altitude (sometimes). Some videos even have multiple sets of GPS coordinates, in different languages (?). QT can handle multiple languages, but why GPS data in multiple languages? Ask Apple.

And, funny, when the movies also have an XMP record produced by the Apple device (as per creator tag), it does not contain the GPS data. So the XMP record exists, is rudimentary, does not really reflect the actual data in the file and even has sometimes contradicting date and time information. Why? Ask Apple.

This is not good for users.
It may work OKish when you create your photos/videos on an Apple device, then upload into the Apple cloud and process and view only with Appel software.
But when users dare to leave Apple's 'walled garden' and move files to another cloud service or even back under their own control, they may face some challenges. Which is intentional of course. All the big vendors want your data and money for your entire life.
« Last Edit: March 06, 2019, 09:38:25 AM by Mario »

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #25 on: March 06, 2019, 12:20:29 PM »
THANKS

for all users who already have reported problems. Keep them coming!

The Metadata Analyst app framework is now finished and I have already written over 40 tests.
I think I have covered everything reported here, including useful info about EXIF color., extra blanks in keywords, etc.

The app can do more than I initially planned. e.g. it now supports the File Window selection and allows you to navigate easily between files you want to analyze. It even shows a thumbnail of the analyzed file, for easier orientation.

Over the next months I'll finish this powerful app by thinking up and adding more tests.
This will be part of the feature set I have planned for IMatch 2020, because it uses some things only available in the IMatch 2020 engine and IMWS version.

ubacher

  • Super Hero
  • ****
  • Posts: 1967
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #26 on: March 06, 2019, 01:03:02 PM »
Thinking about troubleshooting metadata I came to the conclusion that knowing what should be, and
knowing what transformations are being done as Imatch processes metadata is the top hindrance.

A comparison of the metadata before/after ("diff") would be a good tool to learn about the transformations.

One problem:
The metadata of a newly ingested file is not the same as it was in the original ("out of the camera") file
since Imatch already makes corrections. Would it help if one could also compare the metadata from an external
file
to the metadata once ingested? This would show any initial metadata transformation gone awry.
It would also show if some metadata is present in the file but not included by Imatch. (Preferences-> Metadata2-> Tag manager)

Maybe you have already planned such a DIFF functionality incorporated - just wanted to mention it.

ben

  • Sr. Member
  • **
  • Posts: 405
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #27 on: March 06, 2019, 01:14:10 PM »
Very cool to see your progress.
Will it be possible to search for files that have a certain error?

If I find one file with errors with the new app, it might be very likely to have more files with this error in the database.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #28 on: March 06, 2019, 01:40:52 PM »
Would it help if one could also compare the metadata from an externalfile to the metadata once ingested?

Way to complicated I suppose. The Analyst analyzes the data in the file (and sidecar) and lists problems. It also analyzes some data in the database, e.g. to find keyword problems.
IMatch does not modify data when ingesting, it ingests the data as ExifTool outputs it. This is the same you see in the Metadata Analyst. The rich XMP record IMatch uses is created by ExifTool and by running some argument files provided by Phil and me. These are also used when the MD Analyst ingest the data. So you see what IMatch sees.

It would also show if some metadata is present in the file but not included by Imatch. (Preferences-> Metadata2-> Tag manager)

ExifTool currently supports about 14,000 metadata tags. IMatch by default imports over 1,000 tags by default. The tags excluded by default are mostly useless for humans, because they contain technical data, obscure maker notes, individual brush strokes recorded by Photoshop or Lr and similar fluff which you'll never need.

If you really want to figure out if your files contain some information you would like to see in IMatch, you can just look at the data shown in raw form by the MD Analyst. Switch the Metadata Panel in IMatch into the Browser layout. This shows you all data IMatch has imported. Easy to do, and when you find some maker note or obscure tag you want to enable in the Tag Manager, this is the way to do it.

You can do the same now by running the "List Metadata" preset in the ExifTool Command Processor.
There is no real standard for how vendors name the tags, or how ExifTool names the tags (remember: most of these tags are undocumented by the camera vendors). So looking at the ECP output and trying to find out the tag that holds the info you want is the only way. I doubt that this is of interest for many users.


Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #29 on: March 06, 2019, 01:46:12 PM »
Very cool to see your progress.
Will it be possible to search for files that have a certain error?

If I find one file with errors with the new app, it might be very likely to have more files with this error in the database.

Analyzing a file takes several seconds. The output is a JSON data record with several hundred information items. It would impossible (and probably useless) to run the analyst against all files in your database. The tool has been designed to highlight 'issues' in a file and then rely on human intelligence to fix the problem.

Most of the time this will be used to provide detailed information to me when a user reports yet another metadata problem. Much easier than to always explain how the ECP works, what a log file is, to ask for a sample image etc.

The MD Analyst is not a magic wand to fix metadata problems automatically.

If you identify a problem with the metadata in a file with the MD Analyst, there is usually a simple way to find files with the same issue using the Filter Panel or the File Window Search bar.

Many problems are caused by bad / out-of-sync legacy IPTC records or out of sync data. Solution is to delete the old IPTC record. Easy to do, even for hundreds of files.
More subtle problems usually require individual file-by-file inspection and manual repair.

The Metadata Mechanic is a very underused but extremely powerful tool for fixing problems. I have used it in several projects now and it is great.
« Last Edit: March 06, 2019, 01:50:08 PM by Mario »

Tveloso

  • New Members
  • *
  • Posts: 17
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #30 on: March 06, 2019, 03:56:38 PM »
Mario, you're quite right about the messiness in the video files that Apple is producing.  I wondered of files delivered via iCloud Photo Library might be "more correct", but I found that to not be the case.  The Videos on my wife's PC (she uses iCloud) are in much the same state as mine.

My wife's photos are delivered to her PC as four separate files (*.jpg, *.HEIC, *.MOV, and *_HEVC.MOV), and I have send you a sample photo (all four files for it) for your reference.

For the [QuickTime] Creation Date vs. [QuickTime] Create Date question, it's looking like (in my case at least), the Creation Date is the correct one (and matches the [ExifIFD] Date/Time Original from the stills):

.jpg  . . . : [ExifIFD]    Date/Time Original  : 2019:02:11 13:17:14

.HEIC . . . : [ExifIFD]    Date/Time Original  : 2019:02:11 13:17:14

.MOV  . . . : [QuickTime]  Creation Date       : 2019:02:11 13:17:13-05:00
              [QuickTime]  Create Date         : 2019:02:12 01:52:01
                   
_HEVC.MOV . : [QuickTime]  Creation Date       : 2019:02:11 13:17:13-05:00
              [QuickTime]  Create Date         : 2019:02:11 18:17:14


…so I’m wondering if IMatch should now favor the Creation Tag over Created?  Perhaps this all happened when Apple started using that HEIC/HEVC format?...

On the other hand, it could be that older files might have the oposite condition (I’ll check for that), so making this change would maybe break more files than it would fix (should older files that are now correct – but have incorrect dates lurking in the Creation or Track tags – then be re scanned after such an update)…and I’m sure you have much more to consider than just this one case.

You’ve said before that Metadata in Video is even more of a mess than in Photos…I see what you mean.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #31 on: March 20, 2019, 01:29:20 PM »
Mario, you're quite right about the messiness in the video files that Apple is producing.  I wondered of files delivered via iCloud Photo Library might be "more correct", but I found that to not be the case.  The Videos on my wife's PC (she uses iCloud) are in much the same state as mine
(...)

There is n way to handle this automatically. I have now a couple of sample QuickTime movies with both CreateDate and CreationDate and both have different values. It is reasonably to assume that the Apple device software, clould products and software produces are developed by different teams and that Apple has no process to ensure metadata consistency between application/service borders. Not untypical these days, not only for Apple. Sad.

I see no other solution than to add a new metadata option for this (Edit > Preferences > Metadata 2: Favor CreationDate over CreateDate).

Another problem is that ExifTool dynamically creates tags based on the varying data in the file (the streams in the video). And some Apple files I have have a CreatedData tag in the standard QuickTime metadata, but CreationDate only in one of the streams. This is impossible to handle automatically because what Apple writes here depends on the individual file. In such cases, only a custom Metadata Template or the MD Mechanic can help (to set the XMP created/subject created tags from the best possible value as defined by the user).
« Last Edit: March 20, 2019, 02:04:53 PM by Mario »

lbo

  • Full Member
  • **
  • Posts: 191
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #32 on: April 04, 2019, 09:42:25 AM »
maybe this reply is a bit late but I want to get my setup right:

You write each part of the hierarchical keyword separately (non-standard).

since my setup looks similar to what sinus posted one month ago: Which specific setting of is wrong, IOW causes each part of the hierarchical keyword written separately?

Oliver

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #33 on: April 04, 2019, 09:46:57 AM »
This thread has nothing to do with keywords. Its about various timestamps in QuickTime metadata.

Check your keyword settings under Edit > Preferences > Metadata. See Metadata

lbo

  • Full Member
  • **
  • Posts: 191
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #34 on: April 04, 2019, 12:29:51 PM »
This thread has nothing to do with keywords. Its about various timestamps in QuickTime metadata.

wrong, you can even see it in the quote I gave.

Again:

sinus posted his settings here: https://www.photools.com/community/index.php?topic=8859.msg62309#msg62309

and in the next message you told him that he uses a non-standard setting, I quote again your words:

You write each part of the hierarchical keyword separately (non-standard).

Since I currently have the same setting as sinus, I want to know the correct "standard" setting you suggest instead of sinus' settings.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21666
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #35 on: April 04, 2019, 01:58:38 PM »
This thread is me asking about feedback for a new app I have written for IMatch 2020.

It usually is counter-productive to hijack threads with the topic: [OFFICIAL - FEEEDBACK WANTED)
Please don't expect me to re-read such long threads with cross-posts from all sorts. This thread is 10 screen pages long.

Just ask your question in a new thread like other users do.

You asking about a problem with keywords you have?
What are your keyword settings under Edit > Preferences > Metadata??
What are your hierarchical keywords, and the resulting XMP and legacy IPTC keywords (if applicable)?
What are problems with keywords do you have?
What settings do you use under Edit > Preferences > Metadata 2
I recall you fiddling with the per-file format metadata settings as well. This also may cause unexpected side effects.


« Last Edit: April 04, 2019, 02:04:45 PM by Mario »

Aubrey

  • Super Hero
  • ****
  • Posts: 1011
  • IMatch user since June 2008
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #36 on: April 04, 2019, 08:30:52 PM »
This thread is me asking about feedback for a new app I have written for IMatch 2020.

Agreed: this is the only purpose of this thread.

Just ask your question in a new thread like other users do.

Agreed - and make sure that the subject is relevant: then this allows other users who have a similar issue at a later date to be able perform a constructive search to resolve their problem.

Aubrey.

lbo

  • Full Member
  • **
  • Posts: 191
Re: [OFFICIAL] - FEEDBACK WANTED
« Reply #37 on: April 05, 2019, 10:50:05 AM »
It usually is counter-productive to hijack threads with the topic: [OFFICIAL - FEEEDBACK WANTED)

Just for the records: I did not hijack this thread.

I just cited your posting from this thread where you blamed sinus for using "non-standard" settings which I can't understand.

So sorry if the content was off-topic, but it was not me starting it.

I will open a new thread about this.

Oliver