A Personal Perspective on Metadata Issues in Migrating to IMatch 5

[Ferdinand]

1.  XMP as the Primary Metadata Store

In the beginning the main metadata entry tool in IMatch 3.x was IPTC.  The editor was highly functional.  Then came XMP, but the entry tool was not so user friendly.  And IPTC was still easiest, because IMatch could populate XMP from IPTC, but not the other way around. 

In IMatch 5 XMP is now the main metadata entry tool, provided that Metadata Working Group compliance (MWG) is turned on in Metadata2 preferences.  So much so, that under MWG mode you can't edit IPTC fields in the metadata panel.  Instead you edit the XMP field and the information will be populated in the corresponding IPTC field, again subject to your Metadata2 XMP Export preferences.

2. Where to store metadata?

-  To write-back metadata or not?

In the default install of IMatch 5, automatic write-back of metadata (to the image file or the XMP sidecar) is off.  If you want the metadata available outside of IMatch you'll need to trigger write-back.  This can be done manually from the menus or using the pencil icons on the thumbs, or you can make it automatic (in the Background Processing tab in preferences) before you start inputting metadata.  But before you either trigger it manually or turn it on automatically you need to understand the implications of doing so, particularly with MWG compliance turned on.

-  MWG compliance implications
 
The default install has MWG compliance enabled.  Under this option, XMP is editable and IPTC is not. Also IPTC, which is *always* stored in the image, will be automatically populated from XMP, but this won't actually happen until write-back is triggered.  If you don't want this to happen then you will either need to turn MWG compliance off and then also turn off "Synchronise XMP with EXIF / IPTC / etc", or to make the image file read-only.  This is true for both RAW and non-RAW.  The help file provides more information on how to configure IMatch for various workflows (Configuring IMatch | Metadata 2 | Example Configurations).

-  Sidecars or not - RAW?

A related question is where to store XMP metadata - in the image file or in a sidecar?  Many people prefer to use XMP sidecars for RAW.  In fact many people hold this view with religious fervour.  Most other software (other than Nikon software) expects metadata for RAW files to be in XMP sidecars.  The default install option is to "Use existing XMP sidecar file, else use Mode 1 (Write to all supported formats)".  So if you want to exclusively use XMP sidecars for XMP metadata for RAW files, you will either need to (i) ensure that you have existing sidecars, or (ii) change this option, probably to option 5 to "Always write to XMP sidecars" or (iii) make your RAW files read-only, which will force IMatch and Exiftool to generate sidecars.  Sticking with XMP sidecars for RAW minimises the risk of inconsistency between embedded and sidecar XMP as a result of editing XMP in other software, again with Nikon as an exception.  Even just rating images in another application can result in an inconsistency.

-  Sidecars or not - Standard Formats?

While using XMP sidecars for RAW files is common, the opposite is true for so-called standard formats.  Adobe ignores XMP sidecars for these files, and so the default option to write XMP to standard formats makes sense if you want compatibility with Adobe applications.  As with RAW files, using embedded XMP for standard formats will avoid the possibility  conflict between embedded and sidecar XMP if you edit XMP in an  Adobe application.

-  3.6 settings

Given where you want to store XMP in IMatch 5, the question is where is it currently stored in IMatch 3.6?  In my view, IMatch 3.6 XMP preferences are hard to understand.  I had some settings that worked for me, but caused issues in IMatch 5.  My advice is to check where it is stored IMatch 3.6 as preparation for migration.  You might be surprised.  You might need to do some cleaning up, as I did.  I have written an IMatch 3.6 script that assists by putting those categories that you want to migrate to keywords in the right places in advance of the migration.
 
3.  Categories vs Keywords?

-   Which is better?

A long thread https://www.photools.com/community/index.php?topic=235.0 has been written on this already.  My view is that you use keywords if and when:
.  you want the keywords embedded in files (or in XMP sidecars, depending on your settings)
.  You don't plan on doing much reorganisation of the keywords hierarchy, since to do so would involve a lot of rewriting to files
.  You wish to use the Thesaurus to assign images to keywords

and in my view you use regular categories if and when:
.  you don't want the categories embedded in files (or in XMP sidecars, depending on your settings)
.  You want to ability to quickly and easily reorganise the categories hierarchy

The above-quoted thread lists some other advantages of regular categories. 

The simple trick is to remember that keywords may look a lot like categories, in the category view and the category panel, and you can perform some operations on them like you would with categories and use them in some areas like categories, but they're not categories - they are the keywords in your file.  My expectation is that there are limits to how far you can use them like categories, although I confess that I don't know where those limits are exactly.


-    Use of Scripts

It is still possible to do what was done in IMatch 3.6, and use scripts to write categories to image files as keywords.  I've written such a script which I may release in due course, and Mario has hinted that at some stage in the future he may include native capacity in IMatch 5 to do this.  The downside is that it's not automatic, and you will get double entries.  That is, you will get the same entries both in @Keywords and regular categories.

-  Use of the Thesaurus

The thesaurus is a lookup table and not directly linked to keywords.  Changing the *contents* of the thesaurus will therefore not change keywords or other metadata in your files.  I.e.  You can't change keywords or metadata just by editing the thesaurus.  It's just a look-up table.  But if you use those little check-boxes in the thesaurus to assign an image to a level in the thesaurus, then typically you will change keywords etc.  So you use the thesaurus to *assign* keywords, including deleting and reassigning, rather than editing existing keywords.

4. Misc

I haven't mentioned IPTC supplemental categories.  It was common in IMatch 3.6 to use this field to back up the category structure.  IMatch 5 can import them, but really you shouldn't need to do this - at the moment you can export your category structure from IMatch 3.6 and import it into IMatch 5 including file assignments.  There will be a database converter towards the end of the beta which will migrate them. I propose to wipe IPTC Supplemental Cats as part of a migration clean-up.

DNG is treated by Adobe software as a standard format, and so does IMatch 5.  In reality they can be a RAW file direct from a camera or a derived file.  If you use DNG you will need to think very carefully about your metadata preferences in light of the above discussion.


[Edits]
10 July - added a couple of sentences about the extent to which you can use categories like keywords
9 July - Corrected a mistaken statement that formula categories and @builder could not use keywords - they can.  Also general editorial changes for clarity.

[ChrisMatch]

Thanks for taking the time to write this down.

[ChrisMatch]

Hi Ferdinand

First I wondered why you say:

3.  Categories vs Keywords?
-   Which is better?
My view is that you use keywords if and when:
.  You don't need to make use of category formulas, data-driven categories, and @Builder

Now I think you meant it like: @Builder IS a Categorie (a special one) - so if you use Keywords ONLY, than you don't use @Builder - right?

But one could also think this means: If you decide to go with Keywords, ýou decide at the same time against using Categories. But of course those can be combined. One can use Keywords for the main organization of the images and at the same time build special 'Queries' using the @Builder or formulas (which themself can use Keywords).

I only wanted to clarify this in case I miss a point.

Thanks
Chris

[Ferdinand]

You are right and I was mistaken.  You can use keywords in formula categories and @Builder.  I have edited the top post in this thread accordingly.

Even though this is a "personal perspective", I am happy to consider any other comments.

[Photon]

Hello Ferdinand,

thank you very much for this valuable summary. This is to my humble opinion the most important information for all users who like to start with structuring image collections.
(see also https://www.photools.com/community/index.php?topic=235.0)
And yes, a couple of decisions are very personal: How you trust a database, how you treat backups, how you share/export metadata, how you protect assets, ...?

Coming from another DAM tool to Imatch, I was very happy that my keywords embedded in XMP sidecars could be imported relatively easy to IMatch v3.6. With this I created related categories in IMatch. For me the lack of hierarchies or the proprietary solution for hierarchies with keywords in various DAMs is/was never a problem. This because the restructuring of hierarchies with IMatch goes very fast as long as the imported keywords are existing and unique.

The only thing I like to have now vice versa with v3.6 or v5.0 is the ability to export my categories and/or keywords back to XMP sidecars. For me it is still not fully clear how this can be done automatically. E.g. via automatic category-keyword-sidecar synchronization? So the key question with v5.0 beta is: What does a v3.6 IM user has to do in v3.6 or/and final v5.0 to realize a good category-keyword-sidecar workflow? The decision is certainly dependant on the tools and scripts which are existing or possible in final version  of v5.0. Since I am not (yet) a good IM script programmer I patiently test and wait and gratefully read contributions like yours.

An issue quite similar to categories/keywords is the modificatoin of date-time of various media files. Modification of date-time in IMatch and/or external software is possible and the import/export does require some decisions in IMatch v3.6 and v5.0. May be the date-time topic requires a similar, but separate topic like this one started from Ferdinand?

Regards and thanks, Martin

[Mario]

The only think I like to have now vice versa with v3.6 or v5.0 is the ability to export my categories back to XMP sidecars.

The IMatch category concept is unique. There is no concept like it in XMP. If you really need to persist your categories in XMP, you must choose one of the available XMP fields and somehow store the categories in there. But you cannot store formulas or Alias categories or suchlike easily to text form.

And when you only want to store 'simple' categories in XMP, why don't you use the @Keywords category which has been designed exactly for this purpose, and automatically maps between hierarchical XMP keywords and IMatch categories? Re-inventing the wheel with yet another "how can I store my IMatch categories" is maybe not a good idea.


@Keyword category: All child categories are automatically stored in XMP (and optionally IPTC) and automatically synched with keywords.
Can be slow to write (file updates on disk!) but also visible in other applications.

All other categories: Stored safely in your IMatch database. Very fast to process, especially move and copy. Formulas. Data-driven. Alias. Ex- and importable via an open XML format.

This gives you a lot of options without the need to come up with yet another custom concept to persist categories in XMP or other metadata.

[clpratt]

I used to store my hierarchical IMatch categories in IPTC Supplemental Categories (now deprecated). This could be then be imported back into IMatch to recreate the categories tree if necessary. As far as I understand this is not now possible with IMatch 5.

I consider the category hierarchy to be an extremely valuable aid to DAM and really wish somebody at XMP headquarters would take up the idea as a universal aid to any future photo storage applications and implement a change to allow it storage space in XMP.
Surely hierarchical category records are preferential to flat categories/keywords in many cases and I cannot see why their importance hasn't been more widely appreciated and acted on.

I seemed to remember there was talk of a special "IMatch namespace" in XMP which would have allowed storage of IMatch Hierarchical categories.
What happened to this idea?

[Ferdinand]

If you really need to persist your categories in XMP, you must choose one of the available XMP fields and somehow store the categories in there.

Could be done using a script.  But that still raises the question of where in XMP?  As I understand it, IDImager had its own XMP namespace.  Tricky. 

I guess you could do the equivalent of what we used to do in IMatch 3.6, and store them in the XMP equivalent of Supplemental Categories.

But don't look at me, I'm not convinced it's necessary, esp if you have a robust backup strategy.

[Ferdinand]

I used to store my hierarchical IMatch categories in IPTC Supplemental Categories (now deprecated). This could be then be imported back into IMatch to recreate the categories tree if necessary. As far as I understand this is not now possible with IMatch 5.

Yes it is.  Look at "Preferences | Metadata".

Surely hierarchical category records are preferential to flat categories/keywords in many cases and I cannot see why their importance hasn't been more widely appreciated and acted on.

Because it's an undocumented (by Adobe) addition to their XMP standard.

I seemed to remember there was talk of a special "IMatch namespace" in XMP which would have allowed storage of IMatch Hierarchical categories.  What happened to this idea?

Mario will probably reply, but I think that his view is likely to be that hierarchical keywords is sufficient

[ianrr]

Thank you very much Ferdinand for taking the time to do this   ....  most timely indeed.

[Mario]

I used to store my hierarchical IMatch categories in IPTC Supplemental Categories (now deprecated). This could be then be imported back into IMatch to recreate the categories tree if necessary. As far as I understand this is not now possible with IMatch 5.

You can import them into a data-driven category in IMatch 5. See the help for data-driven categories for details on how to produce levels by splitting metadata at delimiters like .

I seemed to remember there was talk of a special "IMatch namespace" in XMP which would have allowed storage of IMatch Hierarchical categories.
What happened to this idea?

Many strings and pitfalls are attached to custom XMP namespaces. So far I'd rather go without introducing one. The data would be useless for other software. And there are better ways in IMatch to backup IMatch-specific data - starting from regular database backups to category and Attribute exports.

[DigPeter]

Thanks Ferdinand - I have been out of the loop for some time, just able to look in on the various fora, so this is a useful reviser for me. 

[dcb]

The results of my migration test have been to store almost all of my existing IM3.6 categories as hierarchical keywords for IM5 which then show up in the special keywords category. I have a script that takes the category in IM3.6 and writes it using ExifTool to the image file. I know I could export then import the category tree. My approach is one more of migration from IM3.6 to IM5 rather than an upgrade.

Some examples:

* Where.Oceania.Australia.Victoria.Bendigo.Kangaroo Flat.Swimming Pool becomes Where|Oceania|Australia|Victoria|Bendigo|Kangaroo Flat|Swimming Pool
* Event.Birthday.David - 40th becomes Event.Birthday|David - 40th
* When.2000.January.26 becomes When|2000|January|26 (I am likely to drop this tree once I get better sorted with the timeline)

As keywords these values are 'synced' to my files when I write-back metadata. If I use the more standard fields for country, state, location, etc., I need to script a conversion in both directions, AND handle locations longer than the 4-5 standard fields allow, AND keep track of what files changed. With keyword hierarchies all I need to do is rearrange the keyword tree and save when I feel like it. I can also take advantage of the keywords for faster data entry (faster than navigating a category tree).

Categories are then something different. The 'Source' metadata field contains the physical location of a scanned photo or video tape in the form "Photo Album|Family|01" or "Video|Reel 2006-001". I use a dynamic category to split the values on the '|' and create a tree to view. This category is then a representation of the metadata in another form. I'll also have categories for bringing family members together without the need to keep any data in a file other than who's in it.

[sinus]

This is a very interesting thread.
To be honest, I personally do not yet think about migration from 3 to 5 IM, because what I first want to do, is understand IM5 properly 
It is not that easy and I do think about a changed workflow (including filenaming, Keywords, Categories, Attributes, Metadata, bookmarks and so on ...).

Also I want first "play" with the xmp-sidecars, because I am afraid, that I have to go this line.

When I am ready with my workflow, THEN I will take the time to think, how I could best migrate my old DB or/and files from IM3.

I am quite sure, that I will be able to "bring" all informations from 3 into the new 5, if this will be by a script or simply by "working" one or two days.

But Ferdinand is right, we have to think about some things, how we want handle this and that.

[dcb]

This is a very interesting thread.
To be honest, I personally do not yet think about migration from 3 to 5 IM, because what I first want to do, is understand IM5 properly 
It is not that easy and I do think about a changed workflow (including filenaming, Keywords, Categories, Attributes, Metadata, bookmarks and so on ...).

You might find migration is a necessary part of understanding IM5. I've learnt a lot about IM5 and IM3.6 during the process. Still testing various items and ways of doing things.

[sinus]

This is a very interesting thread.
To be honest, I personally do not yet think about migration from 3 to 5 IM, because what I first want to do, is understand IM5 properly 
It is not that easy and I do think about a changed workflow (including filenaming, Keywords, Categories, Attributes, Metadata, bookmarks and so on ...).

You might find migration is a necessary part of understanding IM5. I've learnt a lot about IM5 and IM3.6 during the process. Still testing various items and ways of doing things.

Thanks for your post. You are right, we can lern a lot during migration.
But I dare to say, that I do understand 3.6 very good, I know, what I can do and what not to export some stuff in it. Maybe I am wrong, but I think, 3.6 is not the "problem" for me ... but still IM5 simply because it is still quite new and because it is much more mighty than 3.6.

[dcb]

Sinus, that I know. I've seen many if your posts in the old forum. Same for me in many ways. However the learning was about how I thought rather than how I did something. And that's been invaluable.

[Ferdinand]

The point of my post at the start of this thread is that you need to plan the migration from 3.6->5.0.  That planning needs to be based on a good understanding of the two programs. 

Some clean-up of metadata may be needed.  It could be done before or after migration.  My preference is to do it before.

[DigPeter]

Some clean-up of metadata may be needed.  It could be done before or after migration.  My preference is to do it before.

I think that is right.  Changes to the IM3.6 database are likely to be easier/quicker than doing it in IM5, where all changes must be written to metadata.  For instance I have recently decided as part of this clean up process to change a node category from, say, 'A' to 'B'.  This is automatically applied through the database in IM3.6 to the affected files and will be then picked up by whatever migration procedure is used - in my case Ferdinand's migraton script.

[Mario]

I think that is right.  Changes to the IM3.6 database are likely to be easier/quicker than doing it in IM5, where all changes must be written to metadata.

I don't understand this. When you change IPTC, EXIF or XMP in IMatch 3, it also has to be written to the database, and the file on disk. IMatch 5 does not differ here. When you just change category assignments, IMatch 3 and IMatch 5 are also identical. Only when you change keywords in your files by manipulating the @Keywords category, IMatch 5 will not only update the database, but also mirror the changes made to keywords in the file on disk.

[cytochrome]

I don't see the  problem in migrating from 3.6 to 5.0, went smooth and trouble free (for once I followed the manual ).

Maybe I am naive (problems will show up) or my workflow is too simplistic. I am no pro, my photos are for me, family, friends, the local officials in the mountain valley I live in.

What I do:

- write simple standard metadata to RAW files (the data is transmitted to JPG by the raw converter)
- the rest is in IMatch categories. No keywords apart from the occasional one when I look at my pictures in an external viewer and want to make a "quick note" for something I probably missed when cataloging in Imatch.

With such a workflow I know that all of my "clients", even the less computer savvy  have easy access to title, description, and where, which is what matter to them. And the rest which is in the categories is hidden.

Since I write to RAW I don't use keywords in place of categories. Writing to RAW is safe (in my opinion and experience) if one uses the manufacturer software (CNx and ViewNx for me) and IMatch and PhotoMechanic (don't know if the latter uses Exiftool, but it has been very reliable). It is safe I think because I write only standard IPTC/XMP fields, they have place holders, everything (??) seems well organized.

With keywords, especially hierarchical keywords, great blocks of text may be written into the raw, en route for disaster. The only way to handle this safely is XMP sidecar. Which I try to get rid off.

So again, at least with such a simple workflow, migration is simple too, I confess I did not plan anything, just copied the categories and used the same files than for IM3.6 (yes I have backups).

Francis

[DigPeter]

I think that is right.  Changes to the IM3.6 database are likely to be easier/quicker than doing it in IM5, where all changes must be written to metadata.

I don't understand this. When you change IPTC, EXIF or XMP in IMatch 3, it also has to be written to the database, and the file on disk. IMatch 5 does not differ here. When you just change category assignments, IMatch 3 and IMatch 5 are also identical. Only when you change keywords in your files by manipulating the @Keywords category, IMatch 5 will not only update the database, but also mirror the changes made to keywords in the file on disk.

I am not changing IPTC etc in IM3, just the name of the category in Category View.  Changing IPTC etc is not necessary at this stage.  When a change is made to an @keywords category in IM5, it has to be written to the metadata (otherwise there is no point in using IM5, as the system is predicated on using detadata).  If a change involves many files (12000 in the case I illustrated above), it takes a long time in IM5.  For the purpose of reorganising the IM3.6 database in preparation for the eventual migration to IM5, changing the category name is immediately visible in the categories panel below the file thumbnail.  I can then use Ferdinand's script, or your migration script when you release it, to migrate to IM5.   

[dcb]

I think that is right.  Changes to the IM3.6 database are likely to be easier/quicker than doing it in IM5, where all changes must be written to metadata.

I don't understand this. When you change IPTC, EXIF or XMP in IMatch 3, it also has to be written to the database, and the file on disk. IMatch 5 does not differ here. When you just change category assignments, IMatch 3 and IMatch 5 are also identical. Only when you change keywords in your files by manipulating the @Keywords category, IMatch 5 will not only update the database, but also mirror the changes made to keywords in the file on disk.

My view is that changes are easier in IM3.6 for me at the moment because IM5 is beta. As I play with something and decide how I want to reorganise it's easier to do it in IM3.6 in preparation for migration to IM5 when I move to a final database. I might have a 1,000  image sample set in IM5 and 22,000 images in IM3.6. I don't want to waste energy on a large list of "When I get to IM5 change image 0000345.jpg to category X". IM5 is faster and easier but you have to remember many of us are using both and treating beta as beta. Last night I copied across several thousand images from IM3.6 for further testing and let IM5 do its magic. That was much easier than IM3.6 ever was for updates.

[Richard]

At this point I am wondering if I will import my 3.6 database when the option becomes available. I am taking some risk that my efforts will not be lost but by organizing my files in IMatch 5 I am both testing and getting a much better organization for my files. I can accomplish more and do it faster and easier in IMatch 5. Plus I am learning more by doing.

[Mario]

I am not changing IPTC etc in IM3, just the name of the category in Category View.  Changing IPTC etc is not necessary at this stage.  When a change is made to an @keywords category in IM5, it has to be written to the metadata (otherwise there is no point in using IM5, as the system is predicated on using detadata).  If a change involves many files (12000 in the case I illustrated above), it takes a long time in IM5.  For the purpose of reorganising the IM3.6 database in preparation for the eventual migration to IM5, changing the category name is immediately visible in the categories panel below the file thumbnail.  I can then use Ferdinand's script, or your migration script when you release it, to migrate to IM5.   

You are mixing two concepts here: Categories and keywords. When you work with regular categories in IMatch 5, changes are as quick (or quicker) as in IMatch 3. Categories are sored inside the database and work almost instantaneously in IMatch 5 and IMatch 3.

However, when you change keywords in your files, either via the Keywords Panel or the special @Keywords category hierarchy, you are manipulating metadata in the image file. These changes of course have to be written back to the image files at some point. This is like you select 12,000 files in IMatch 3, open the IPTC editor, change some keywords, and press OK.

What further slows down the write-back, if you have enable it, is that IMatch 5 (different than IMatch 3) also migrates changes made to XMP (e.g. your keyword changes) back into IPTC (and EXIF) when the data is written. This ensures optimal metadata quality and implements the Metadata Workgroup guidelines, which makes the metadata in your files more secure and future proof. Doing that for 12,000 files is quite a task and will keep your system busy for many hours.

This is why you have the choice in IMatch 5. You can either use keywords, and persist all that information in the XMP and IPTC records stored in the file. This makes your categories/keywords accessible by other applications as well, but it's slow.

Or, you use IMatch categories, which are not only much smarter and powerful than simple keywords, but are also blazing fast. These categories are stored in the IMatch database only, and are not visible to other applications.

Or, you combine both methods. Whatever works best for you, IMatch 5 can do it.

[Ferdinand]

You are mixing two concepts here: Categories and keywords.

IMHO Peter is mixing them intentionally.  In IMatch 3.6 he is handling his keywords as categories using his own taxonomy.  In IMatch 5 he wants to migrate them to keywords.  So he is saying that if you're going to make that change, and if you're thinking of rearranging the hierarchy, then it's best to do so before you write the categories as keywords in the file.  Now it's true that the migration could be done in either 3.6 or 5.0 -  I have a script that works in both programs  - but the point remains that it's going to be easier to reorganise and then write to keywords that to write to keywords and then reorganise.

[Ferdinand]

1.  Writing to RAW is safe (in my opinion and experience) if one uses the manufacturer software (CNx and ViewNx for me) and IMatch and PhotoMechanic (don't know if the latter uses Exiftool, but it has been very reliable).

2.  With keywords, especially hierarchical keywords, great blocks of text may be written into the raw, en route for disaster. The only way to handle this safely is XMP sidecar.

My reading of this is that there is a contradiction between these two statements?

[DigPeter]

You are mixing two concepts here: Categories and keywords.

IMHO Peter is mixing them intentionally.  In IMatch 3.6 he is handling his keywords as categories using his own taxonomy.  In IMatch 5 he wants to migrate them to keywords.  So he is saying that if you're going to make that change, and if you're thinking of rearranging the hierarchy, then it's best to do so before you write the categories as keywords in the file.  Now it's true that the migration could be done in either 3.6 or 5.0 -  I have a script that works in both programs  - but the point remains that it's going to be easier to reorganise and then write to keywords that to write to keywords and then reorganise.

Thanks Ferdinand - I could not express it better    That is my point. 

@Mario. 

You are mixing two concepts here: Categories and keywords. When you work with regular categories in IMatch 5, changes are as quick (or quicker) as in IMatch 3. Categories are sored inside the database and work almost instantaneously in IMatch 5 and IMatch 3

After all this time I am very aware of the differences.  I am NOT changing keywords in IM3.6 - only the categories in the database, preparatory to migration to IM5, where all the super stuff that you have engineered will happen; all in one operation, rather than two, if the reorganisation is done after migration.

[DigPeter]

At this point I am wondering if I will import my 3.6 database when the option becomes available. I am taking some risk that my efforts will not be lost but by organizing my files in IMatch 5 I am both testing and getting a much better organization for my files. I can accomplish more and do it faster and easier in IMatch 5. Plus I am learning more by doing.

This a "horses for courses" business.  Each person will do what suits them best.  But I think there is a danger in "rushing your fences" at this stage of the Beta, unless you have the IM3.6 version of the database and files fully backed-up.

[cytochrome]

1.  Writing to RAW is safe (in my opinion and experience) if one uses the manufacturer software (CNx and ViewNx for me) and IMatch and PhotoMechanic (don't know if the latter uses Exiftool, but it has been very reliable).

2.  With keywords, especially hierarchical keywords, great blocks of text may be written into the raw, en route for disaster. The only way to handle this safely is XMP sidecar.

My reading of this is that there is a contradiction between these two statements?

The second sentence is a caveat and justification why I do not use keywords.

In my experience writing to raw is safe as long as 1) one uses the manufacturer' software, Imatch, PhotoMechanic or Exiftool per se, 2) one writes standard IPTC/XMP. The rest stays in the IM DB.

Writing to raw big chunks of data like possibly produced by complex keywords structures or the use of Adobe software may be problematic.  I refrain from it, and also I think my categories are of no interest to anybody but me.

So there is a basic choice between keeping it lean (metadata in file + categories in DB) and no xmp, or using xmp sidecars that have other potential but also their own problems (becoming bloated, accessed/read/written by other software).

I read what I can on this forum about keywords and have not found a good reason to use them (I understand others have different requirement). I don't see what keywords can do that categories cannot, and as Mario points out, they are almost of instant use, also in IM5 which is slow for using in file data via exiftool.

There is a good argument in favor of "all-in-xmp" which is safety. If IM DB gets corrupted I will loose a day work (only one daily backup...) which would be a pain.  Of course if ever I decide to switch to another cataloger I will have to pass through xmp. The day has not come..

Francis

[BenAW]

If IM DB gets corrupted I will loose a day work (only one daily backup...) which would be a pain.

I use a comparable setup as you seem to have. Keep it simple 
While assigning categories I make a regular Export of the Categories, like every 15-20 minutes.
Takes only a few seconds and could save your day.

[Ferdinand]

In my experience writing to raw is safe as long as 1) one uses the manufacturer' software, Imatch, PhotoMechanic or Exiftool per se, 2) one writes standard IPTC/XMP. The rest stays in the IM DB.

Writing to raw big chunks of data like possibly produced by complex keywords structures or the use of Adobe software may be problematic.  I refrain from it, and also I think my categories are of no interest to anybody but me.

I guess my view is that keywords are likely to be a small chunks of data compared to all the other metadata that you might write.  If you're worried about writing large chunks of data then you wouldn't do it at all.  Writing most things but not keywords isn't going to save much IMHO.

Of course I agree that not everything should be / needs to be in keywords, and I am still debating myself what to put in keywords and what to leave in categories.   But I'm not worried about file bloat from keywords.

[cytochrome]

If IM DB gets corrupted I will loose a day work (only one daily backup...) which would be a pain.

I use a comparable setup as you seem to have. Keep it simple 
While assigning categories I make a regular Export of the Categories, like every 15-20 minutes.
Takes only a few seconds and could save your day.

Ah.. hadn't thought about that, thanks Ben, will try it now.

My daily backup is via SyncBackSE, incremental backup of everything (files, settings, DB and so on), good but takes time...

Francis

[cytochrome]

[... Writing most things but not keywords isn't going to save much IMHO.

Of course I agree that not everything should be / needs to be in keywords, and I am still debating myself what to put in keywords and what to leave in categories.   But I'm not worried about file bloat from keywords.

But I don't write "most things", standard IPTC (less keywords) is enough if IMatch handles the rest. I have seen people using very complex cat/keywords schemes (not to speak of the taxonomists), with hundreds of terms. This will not fit in the normal placeholder of a Tiff-EP container. Of course the Raw can be extended, Nikon does it all the time, but then they "own" the NEF format and probably know what they do.

In fine, I try to avoid external xmp, and to play it safe I stay with IPTC in RAW and the rest in IMatch. I realize it may sound a bit stubborn..

Francis

[chips]

OK - I'll admit to being totally lost with 5. I've relied on 3 for years but migrating to a new computer 5 beta came along at a handy time to build a new database (3 had accrued a lot of duplicates over the years due to my sloppy ticking of boxes no doubt).  I have a year's worth of pictures as a tester - I can see the data beside individual thumbnails - I cannot see how to run a search term for a word.  My captions etc are written in Photo-Mechanic ( and therefore as IPTC ?).  After getting 3 set up with import IPTC and import EXIF it worked fine.  All I want and all I ever foresee wanting is a database searchable for a Name, Event or Date quickly (roughly 100,000) images.  I have no desire to spend hours learning how to work this thing.  Mario suggests reading the help - I have but not all of it.  Mario is an IT expert and understands all those words - I am a photographer who wants a database with the capabilities mentioned.  Categories and hierarchies and slide shows are of no interest - so an idiot's guide is needed.  Maybe just for me but I doubt it.

[Mario]

All I want and all I ever foresee wanting is a database searchable for a Name, Event or Date quickly (roughly 100,000) images.

Where did you store that info? Name, Event, Date?

When you look at the Metadata Panel in IMatch, do you see the information?
The Default layout shows the most frequently used metadata tags, like description, headline, copyright, dates and times, GPS data, ...

A) The quickest way for you is to type the name you're looking for in the search bar above the file window. I explained that in my email from yesterday.
Just bring the files you want to search in into the file window (a folder, a category or even the entire database). This gives you the scope to work in and the filter bar will search in it.



B) The typical workflow in IMatch to find files with a specific word in, for example, the IPTC caption would be:

1. Open the filter panel  <F9>,<L>  (which means press F9 and quickly afterwards press L)
2. If the Value Filter is not visible, select it from the Gear button in the toolbar of the filter panel
3. In the drop-down box of the value filter, select the metadata tag containing the information you are interested in (Here: Description)

The filter then shows all different values for that tag in the current scope which can be a folder, a category, all files on a drive, media or even the entire database.



To see only the files with a given value, check the box in front of the values.

If you have to many values, or you want to search across multiple tags or even all tags, use the Metadata Search filter (enable it via the Gear button). This filter allows you to search all or some fields with easy search operations like "Contains".

To quickly find files taken at a specific date use the Date and Time filer, or switch to the Timeline View.


IMatch 5 is similar to IMatch, but radically different in other aspects. Filters are one of the new areas, but they are easy to use and powerful. Like I wrote in my email from last night, it's worth the time to read the topic on filters. Searching was yesterday. IMatch 5 now filters.

[attachment deleted by admin]

[chips]

Thank you Mario - I do understand that this is second nature to you:  which is why you need a resident numpty,  Cheers

[chips]

Brilliant Mario - a little graphic makes sense of it.  I have been looking under "Search" in the top menu bar.   For whatever reason the little window eluded me.  Perhaps a more prominent colour or something : I doubt I'm the only person who when wishing to search will go back to the drop down where it used to be in 3.   For the record the info is recorded in Photo Mechanic categories : "caption" translates into "Description"  - date captured equates to date created so all is gut!   I shall now add some other years and see how it goes.  One thing: a search using date will not accept the format  15/04/2006   (we put day first then month)  : it will allow 15/04 or 2006 - the former is usable but not ideal.  You'll no doubt tell me there's a button I haven't pushed or box I failed to tick.  Many thanks - quickest response on the planet.  Cheers.

[BenAW]

Did you notice the Timeline? (Top left between Categories and Collections)

[Mario]

I doubt I'm the only person who when wishing to search will go back to the drop down where it used to be in 3.

The help is your friend. IMatch is new so new users need to refer to the help more often.
When I type Search in the help, I get all the links: Search Menu, Search Bar, Filter, Scripting, Search Engine,...

One thing: a search using date will not accept the format  15/04/2006   (we put day first then month)  : it will allow 15/04 or 2006 - the former is usable but not ideal.

Do you refer to the Date filter? And there the "to - from" mode?
Or do you search for dates in the file window search bar? If so you need to use the native format in which ExifTool formats dates (ISO standard). The search bar does not transform the text you enter for your search, or knows a thing about dates...
For searching for dates, the Date filter is much better. Or go to the Timeline View - no searching needed.

[chips]

I readily accept that I'm trying to migrate to 5 with as little effort as possible.  Everybody does a deal with 'time spent learning' versus 'time saved using'.   The older I get (don't ask) the more the balance swings toward 'less in: more out'.  I take on board the necessity of using standard date strings and searching in the right place and that's me almost done.  I'll search and see if "quick copy" is still here and then tackle a niggle I could actually live with - but not happily.  My captions (from Photo Mechanic) appear in Metadata when in "Browser" mode under IPTC Application Record> Caption-Abstract  but not as "description" or anything else in the tidier "Default" mode.  They are labelled "description" in Photoshop and I have monkeyed around in Metadata tags to no effect.  As I say I could live with this especially if i did not have to scroll down past a huge amount of data each time but the display always reverts to top line with each new file. 

[Mario]

I take on board the necessity of using standard date strings and searching

Try using the Timeline View instead of searching.

and see if "quick copy" is still here

Check out the Favorites Panel (<F9>,<F>). This panel maintains an automatic list of the last used folders ("Folder History"), and you can use this list to copy/move files to the last recently used folders. You can also drag and drop folders to this panel to create a favorite for them. This way you have your most frequently used folders always available. Favorites in IMatch work very similar to favorites/bookmarks in your browser.

My captions (from Photo Mechanic) appear in Metadata when in "Browser" mode under IPTC Application Record> Caption-Abstract

So PM has written classic IPTC data.

IMatch 5 is by default configured to import existing IPTC data into the corresponding XMP metadata field (in this case: Description) automatically so the data shows up right away in the Default Metadata Panel layout.

Did you perhaps create a new database with IMatch 5.0.108 or do you use a database created before? If so, did you switch the language to Neutral under Edit > Preferences > Metadata?

You can also send me one of your files so I can have a look at the metadata.

[cytochrome]

Hello Chips,

I am also a PM user and have no problem at finding back what I enter in the PM Stationary Pad. For example what I write into Description/Caption appears in IM5 in Description ({File.MD.XMP::dc\description\Description\0}).

Likewise Headline goes to Headline ({File.MD.XMP::photoshop\Headline\Headline\0})  in IM5.

I have configured my personal Metadata panel which shows exactly what I enter in PM. To ingest images, rename, and write simple IPTC/XMP PM is a breeze, on my old PC it takes around 8 seconds to write the metadata to 150 NEF...

Francis

[chips]

Thanks yet again - proof if proof were needed that when it comes to trying testers I am your man.  I'd read that advice about "neutral" language and I had looked several times for that setting - but being an old guy I looked for Neutral with an "n" which falls between "m" and "o" - you see where this is going don't you ?   I never found "neutral" because I was searching logically: so the language stayed at "en" where I found it.  "Neutral" as the first word in an alphabetical list - a step too far.  Of course now I've found it - more or less by accident - and all is hunkity dory .  So is there any reason why the language box does not default to "neutral" ? - I'm as sure as I can be that I didn't change anything to make it "en".  It's all falling into place now - and the speed of constructing a database is fantastic.  With iM3 it was necessary to allow at least a day - with iM5 (and a more capable machine) - minutes.  Night all.

[JohnZeman]

So is there any reason why the language box does not default to "neutral" ? - I'm as sure as I can be that I didn't change anything to make it "en".  It's all falling into place now - and the speed of constructing a database is fantastic.  With iM3 it was necessary to allow at least a day - with iM5 (and a more capable machine) - minutes.  Night all.

Hi Chips,

It's my understanding that any new databases created with build 110 and later will have the default language set to Neutral.

[MyMatch]

I read
-  MWG compliance implications
and
-  Sidecars or not - RAW?
several times, also the Help page about this topic.

But i still don´t understand, what to configure to make sure that IMatch writes nothing to my image-files, RAW or not.
Please note, that i have no interest or need for IPTC!

Currently, i have:

Background Processing:
    Write-back changes to metadata immediately: DISABLED

Metadata 2:
    Metadata Working Group Compliance: Yes
    Protect unwritten metadata: Yes
    Protect Rating and Label: Yes
    Allow to create XMP files (disabled): Yes (and i dont understand why there is "disabled" written)
    Keep existing XMP: Yes

I did not touch "Tag Manager" or "Configure File Formats ..."

I understood from Ferdinand´s posting, that using the MWGC may result in "some XMP" changes creating IPTC changes, which then will be written to my files - not automatically, but maybe later when i for some reason enable write-back.

I dont want any IPTC data or any other thing to be writen to my files - with the exception of XMP sidecar files.

So, i think that need to disable the MWGC and also edit any and all of the File extensions that are listed as "IPTC Read/Write" or "EXIF Read/Write" in the drop-down menu, uncheck "Use default settings" and then change any and all "Yes" to "Write ..." to "No".

Right?

:-O

[pajaro]

I read
-  MWG compliance implications
and
-  Sidecars or not - RAW?
several times, also the Help page about this topic.

But i still don´t understand, what to configure to make sure that IMatch writes nothing to my image-files, RAW or not.
Please note, that i have no interest or need for IPTC!

I configured IMatch to avoid any modification of original files - everything is written in sidecar files. I do not remember the settings. I will check them when I get back to my computer and will let you know.

[Mario]

@MyMatch

By making all these changes you are basically breaking the metadata backbone of IMatch. When IMatch updates XMP data that has to be mapped to EXIF data (which lives in the original image) but it cannot write the changes, you will have two sets of different EXIF data. One in the XMP record and one in the original EXIF data in your image file. The same is true when you work with GPS data. You will end up with one set of GPS data in XMP, but no or different GPS data in your image file.

Please always state in the future when you report anything related to metadata that you have deliberately and willingly made all these changes.

I don't even know (or consider) which side effects may come from your changes. So many more or less subtle problems can come from this, from your  metadata being incompatible with other applications to different metadata showing up in other applications, depending on whether it supports EXIF and/or XMP. Don't come to me in a few years time just to tell me that Lr or whatever software you then use complains about the metadata, does not see the changes you've made to XMP only or even refuses to load the file.

Note that the options to avoid synchronizing metadata between XMP and the EXIF/GPS/IPTC data in your image file are expert options, designed to be used by people who know a lot about metadata, their tool chain and why or why not they should deliberately break the synchronizing between XMP and other metadata In general, this is a really, really bad idea. Your own risk.

The default settings in IMatch ensure that your metadata is perfectly synchronized between XMP and other formats, that the metadata in your files is compliant to the rules and recommendations of the Metadata Working Group and that the metadata in your files is compatible with all major applications out there.

You are deliberately breaking all that for whatever reason. Bad idea.

ExifTool has been used to write billions and billions of images. It's used by millions of users out there, even big outlets like Flickr use it to process metadata. If you are so paranoid about updating your original files, why don't keep a pristine copy of the original file somewhere on a read-only media at a safe place, and then just let IMatch do it's job.

[ubacher]

Not sure if this has been mentioned before - but setting all your files to read-only will of course protect your files.

But of course you can not make any changes to metadata which already exists in your files else you get
the discrepancies Mario mentioned. Actually, a yellow pencil will be a warning that you have changed
some metadata which exists (i.e. is stored) in the file (and which will be changed when Imatch reads in the file again).