Database diagnosis warnings

[rolandgifford]

Virtually every time that I run a Database Diagnosis I get the following two warnings in the log file:

Checking Metabase:
      Warning: Found 977 rows in rel_rel for non-existing master files. Removed.
      Warning: Found 492 rows in rel_stack for non-existing files. Removed.
Completed.

The numbers are different each time of course.

When I'm running these diagnoses is it usually during a phase where I'm doing some reasonably heavy duty maintenance.

Recently I have added 30,000 images which I have geocoded to a number of GPX tracks, culled to about 3,000 images which I'm adding Keywords to before the next phase.

Alongside that I'm changing the bird species Keyword on every bird image (about 50,000 - I'm half way through) due to the change from IOC to Avilist taxonomies.

Should I be concerned about these warnings?
What can I do to avoid them happening?

No log file as I have no idea what action is causing these errors and I'm changing lots of images in each batch of work so any log would presumably be large.

[Mario]

These are related to the Relation Manager in IMatch, which handles masters and versions. It seems masters/versions were removed or become no longer masters or versions, but the internal structures were not updated accordingly.

This is not harmful, easily fixed by the diagnosis, but should not happen. Do you have any steps or operations which cause this? Maybe do what you did before, to a small number of files, and run the diagnosis to see if the warnings are back. This way we can drill down on what causes this.

[rolandgifford]

I'm not doing anything at all which should affect master/version links. I unfortunately don't know how to reproduce this as there aren't always these errors. They happen when I'm doing bulk repetative changes and I suspect getting somewhat ahead of IMatch background catch-up processes.

There is a different master/version problem which may be linked. Again, doesn't always happen so not reliably reproducible. Details as below:

There is nothing significant about this example. I have thousands of these pairs being updated

I'm changing a keyword between these two

@Keywords|IOC Taxonomy|New World Warblers - Parulidae|Mniotilta|Black-and-white Warbler
@Keywords|IOC Taxonomy|New World Warblers - Parulidae|Mniotilta|Black-and-white Warbler

I have a filter enabled to hide buddy and rejected files. My buddy rules are identical to my relation rules.

In this case the Categories panel shows that there are 27 images, 7 of which are shown

Select the Keyword in the categories Panel
Select all images - Ctrl-A
Click the Keyword - F2 to allow edit - Ctrl-C to copy to clipboard
Click the input area in the Keywords panel
Ctrl-V which shows the IOC and Avilist Keywords as both are in my Thesaurus
Select the Avilist one to add it
Ctrl-left click on the IOC Keyword to delete it
Green tick to update

Keyword count drops to 20, no images displayed

Command Metadata write-back all pending, there are 7
Count still 20
Command Metadata write-back all pending, there are 12
Count still 20
Command Metadata write-back all pending, there are 12
Count still 20
Command Metadata write-back all pending, there are 12
Count still 20
Look at collections and pending write-back count is zero
Command Metadata write-back all pending, there are none
Count still 20 - nothing displayed
Turn off my filter and 20 images are shown
Ctrl-A to select them all and they are assigned to the Avilist Keyword, not the IOC Keyword
Database Diagnosis no errors or warnings

I get round this simply by turning off my filter, selecting and updating all 27 images rather than relying on propagation.

I haven't found a way to rebuild the incorrect Keyword count in the Categories window and usually simply delete it.

Debug log attached

[Mario]

If this is an unrelated problem, please open a new thread.
Else we're discussing different issues with overlap, which makes things unmanageable very fast. Especially with posts spanning 4 screen pages of details (I'm working mobile currently) 

and I suspect getting somewhat ahead of IMatch background catch-up processes.

If you change relations or add masters/versions or modify metadata while IMatch is processing relations in the background, you can actually work against the system and cause harm. This is very often the case when a specific problem affects one user only. Not even IMatch can foresee everything or handle every fringe case.

[rolandgifford]

If this is an unrelated problem, please open a new thread.

It was intended as additional detail showing an example where mater/versions are not working correctly on my system. If doing it that way doesn't help, fair enough. I'll open a new thread.

If you change relations or add masters/versions or modify metadata while IMatch is processing relations in the background, you can actually work against the system and cause harm. This is very often the case when a specific problem affects one user only. Not even IMatch can foresee everything or handle every fringe case.

I'm not changing masters/relations as such but I am making changes to masters which propagate and I am deleting masters (which then delete buddies with the same rules) lots of, very quick as the changes are frequently mechanical rather than requiring thought.

It may well be that I'm working faster than IMatch can keep up with and that is the root of the diagnostics warnings. It isn't caused by me adding images which have master/version links added/changed.

I unfortunately don't have steps to cause these warnings and will accept that they don't matter and are fixed by diagnosis.

[Mario]

Maybe slow down a bit, keep an eye on the Info & Activity panel to see if IMatch is idle. Adding, removing, deleting, moving masters and/or versions while IMatch is still processing the files with the previous layout and setup can work against the system and, in the worst case, even cause de-synching. Versioning is complex, and relies a lot on file system calls in Windows which are unpredictable in their timing. Windows may even send IMatch a file moved event while it still reports the file as existing in another thread where IMatch is currently checking for potential versions.

Keep things simple. Versioning is an advanced feature and you can shot yourself in the foot when you try it hard enough. Propagation, checking for versions and masters is performed in background tasks with an inherently unpredictable order of operations. When you change files or file attributes relevant for versioning while IMatch is currently processing relations and propagation for these files in the background, you can definitely shoot yourself in your foot.

This is often specific to a particular setup, workflow, database user behavior and operations carried out and not something that can be "fixed". This is why there are only very rare reports of things like this, and, usually, tied to one user only.

Something you just have let IMatch finish background processing. The only alternative for IMatch would be to force-block the user interface to avoid any potentially harmful user interaction. But that would make the entire "processing things in the background" mode of operations useless. For all users, not only for users who happen to have a workflow and setup that is occasionally prone to produce the effects you describe in this post or in a apparently similar post from today.

[rolandgifford]

Slowing down a bit is a bit tricky 

Thanks for the other comments

There are 33 #sl entries in the debug log from changing and writing back a single changed keyword to 7 masters (plus 20 versions). I've no idea if any of those are significant.

I try to reduce the load on IMatch when making this sort of change (I sometimes go through a purge/cleanup of WHERE Keywords created by reverse geocoding) by turning my filter off as I'm updating every image with that Keyword and filtering doesn't make it faster. It could be that IMatch still does propagation etc in the background but doesn't need to make any changes. I usually wait till Task manager says that IMatch is idle before closing down.

[Mario]

IMatch just reports #sl when an operation is unusually slow. The larger your database becomes, the more #sl entries will be logged unless your computer is very fast.

Every time metadata changes, all data-driven categories, many formula-based categories become invalid and must be recalculated. When they are visible somewhere or used in something volatile, this may mean IMatch has to do it immediately, and probably many times.

Combine that with a filter that also becomes invalid every time something in the database changes. The filter must reload, may have to wait for categories or collections being calculated, the filter then applies, the File Window must reload.

When you filter, with the Categories panel open or a Categories filter in the Filter panel open and you also write-back, which writes data to files, then re-imports the metadata, maybe propagates data to other files, then re-imports their metadata, invalidating categories, collections, caches, cached filter results several times a second, you can truly bring the system down. On top of all you maybe add file relations, where IMatch needs to use very slow file system operations like reading directory info... 

There is only so much computing and disk transfer capacity available, and IMatch is basically a demanding database server with a user interface.

[sybersitizen]

When you filter, with the Categories panel open or a Categories filter in the Filter panel open and you also write-back, which writes data to files, then re-imports the metadata, maybe propagates data to other files, then re-imports their metadata, invalidating categories, collections, caches, cached filter results several times a second, you can truly bring the system down.

If you happen to have many files that are awaiting write-back and they're located in the same folder structure, I imagine that selecting them in the Media & Folders view with no other filtering visible would require the least additional processing. Is that correct?

[Mario]

Yes. The more things needs to update (in the UI), the more busy it will become. Filters can be expensive.
The more things in the UI IMatch must update while processing things in the background, the more performance will be needed.

If this causes issues, reducing the UI, e.g. by disabling filters, switching to the Media & Folders View, hiding the Filter Panel and disabling filters will be helpful.

All of this also depends on the size of the database. If you manage 50K assets, things will work fine all the time. If you manage 500,000 assets, things will take longer and more oomph must be supplied by the hardware.

I test IMatch with a database of almost 1,000,000 assets, and there are definitely situations where I have to wait a bit for things to settle on my 5 year old PC.