Delete rejected images from viewer leaves "spaces"

[Joe Austin]

In the last few days I have noticed that frequently when I hit the delete key to mark an image for deletion in the viewer, then exit the viewer, select 'delete' and approve the deletion of x # of images,  the deletion process seems to take longer than it used to and often leaves 'spaces' in the file window which look like an image should be there  (i.e. the space is selectable and shows icons when you hover)  but no thumbnail is present, all metadata is blank, and if you try to open/or delete the space in the viewer you get a message the the image(s) are off-line.

I assume this is the residue of a file deletion, but no file name is displayed in the "space" so I can't be sure.

[Mario]

Logfile?

[Joe Austin]

This is happening when loading the viewer with files from a category.    The 'space' seem to still count in the category totals.

[Mario]

A screen shot perhaps so we can see what the spaces look like?
The log file shows a ton of errors from Windows WIC, failing to load cache images in JPG format. Very unusual.
Windows WIC is really unhappy on your system. Too many error messages to actually count. Search the log file for W> to see more information.

[Joe Austin]

The only way I've found so far to remove the spaces is to select them and do a 'remove from database'

[JohnZeman]

I've never seen this happen, do those spaces disappear when you do an F5 Refresh/Reload Information in Category View?

[Mario]

This is what I would try first, too.

The empty panels don't even show a file name. This is what I would expect to see intermediately, while the File Window reloads many times due to images being removed from the category currently visible. But it should not "stick." Never saw that before.

As I wrote, the log file contains a ton (490!) warnings produced while IMatch is trying to ingest files using Windows WIC. Creating cache files for images on drive H:
Maybe what you see is a side effect of these many errors. Apparently, Windows cannot handle the ORF files and other RAW formats (?) you are using. Switch to LibRaw (photools.com RAW processing enabled in Application settings) and then force an update of the folder to create thumbnails and cache images.

[axel.hennig]

Tried to reproduce this, but wasn't able to do.

[Joe Austin]

I've never seen this happen, do those spaces disappear when you do an F5 Refresh/Reload Information in Category View?

No, category refresh doesn't remove them.

[Joe Austin]

Interesting,  CTL-SHF-F5 does not produce a menu when one of the spaces is selected,  and "Go to folder"  seems to go to a random folder that does not contain images similar to the others in that category.

I have gone through all of the folders from my recent shoot (the ones containing the images from which the categories where this has occurred were built)  and no 'spaces' exist, they exist only in the categories containing images from those folders, and only after deleting files via the viewer that were loaded from selections within those categories.

This is a new development.  I don't recall ever having seen it before.  It may be worth noting that the only change to Imatch that I made prior to this shoot was upgrading to 2025.1.14

[Mario]

The current version of IMatch is 2025.2, released about 4 weeks ago. Please try to always use the latest version.
Run a database diagnosis? Any problems reported?

When the goto folder command works, why does the File Window not even show a file name and just an empty panel. What have these files in common?

When you say IMatch jumps to a random file, is the folder correct? I mean, the folder the file was in before you deleted it?

As I said, I have never seen this, no other user ever reported something similar or can repro it. Which means that, so far, this effect only happens with your database and on your system.

Is this one of the default File Window layouts or one you have created yourself? If so, how did you configure it?

Keep IMatch set to debug logging (Help menu > Support) and when this happens again use Help menu > Copy Logfile to freeze the current log file. ZIP and attach.

[Joe Austin]

The current version of IMatch is 2025.2, released about 4 weeks ago. Please try to always use the latest version.
Run a database diagnosis? Any problems reported?

When the goto folder command works, why does the File Window not even show a file name and just an empty panel. What have these files in common?
  I don't understand this question.  I'm just saying that running the goto command on a "space" within a category sends Imatch to an apparently unrelated folder.  The folders themselves have no spaces.



When you say IMatch jumps to a random file, is the folder correct? I mean, the folder the file was in before you deleted it?
 In the cases I've looked at the folder does not appear to be correct.

As I said, I have never seen this, no other user ever reported something similar or can repro it. Which means that, so far, this effect only happens with your database and on your system.

Is this one of the default File Window layouts or one you have created yourself? If so, how did you configure it?
 This is the [Default] layout.

Keep IMatch set to debug logging (Help menu > Support) and when this happens again use Help menu > Copy Logfile to freeze the current log file. ZIP and attach. 

Should I set to debug and keep using WIC?  Or should I debug while running with Photools processing?


Database diagnosis was successful, but did produce a log with warnings referring to categories referring to non-existing files.....

[Joe Austin]

Diagnosis log...    dx appears to have eliminated the spaces.    Looks like residue of the deletion process.

[Joe Austin]

have switched to photools raw processing and done a couple of editing sessions.   No "spaces", but it has taken a long time (5-6 minutes) to delete ~40 files from a viewer selection of about 60.

Another curious thing I have seen recently is, at the end of the wait for the deleting process to end I see a stack of "Background tasks have completed/ processing queue is empty" messages that then quickly disappear.   There used to be only one.

[Mario]

The diagnosis reports:

Warning: Category references non existing files. Correcting.
For 18 files. This is what the empty panels were.

Now we need to find out when this happens, exactly. Keep debug logging on and do whatever you do to produce this problem. You <Del> files in the Viewer and then close the Viewer and select "Delete Files"? You start the Viewer from a category when this happens or also when you do the same in a folder?

No "spaces", but it has taken a long time (5-6 minutes) to delete ~40 files from a viewer selection of about 60.

This has nothing to do with WIC/LibRaw.

What I see in the log file:

Code Select Expand

04.16 08:33:59+    0 [2738] 05  M> >  0 CIMDXWnd::LoadFile  'V:\develop\IMatch5\src\IMatchNG\IMDXWnd.cpp(776)'
04.16 08:33:59+  47 [2738] 02  I> C:\ProgramData\photools.com\IMatch6\previewcache\9CF67E58-7C27-4DF1-96BF-C4022922F9A4\20\207765.jpg loaded in 156ms. Result: 2
04.16 08:34:38+38922 [2738] 05  M>  <  0 [38969ms #sl] CIMDXWnd::LoadFile


I irritating. IMatch loads an image from the cache into the Viewer or Quick View Panel. Loading the cache image takes only 150m, but the window reports a total load time or almost 40 (!) seconds. I have no idea why. There were several LoadFile operations before, taking only 60ms or 80ms. Then, suddenly, it takes 40 seconds. Then things are back to normal, and then it suddenly takes again 40 seconds.

The log is not in debug logging mode so the information is minimal. Switch to debug logging while analyzing such issues.
No log entries for deleting files (?).

Do you have an exception for the folder containing the database? And for IMatch?

[Joe Austin]

I have done a few edit sessions (not large) since turning on photools raw processing and no 'spaces' since, but still taking a long time.

I have turned on debugging now and will continue.

I rarely edit from folders so all of these instances have been images selected within a category and sent to the viewer then marked for deletion and, when exiting the viewer, selecting 'Delete' and answering 'Y' to 'are you sure'.

I have always had an exception for imatch, not for the files.   Never had a problem like this before.   I have added an exception for the files.

[Joe Austin]

I just did an edit (removing ~60 of 90 images via the viewer) with debugging log and exception in place.   No spaces, but still took 5mins to regain control of the cursor.    This used to happen in about 30secs.

The deletion process also seems to go through several iterations likely generating the multiple 'queue processing complete' messages.

[Mario]

The database is performing real bad for a database with only 50K files. Although you have 50,000 categories, almost one category per file. Why is that?

The bad performance during deleting files is because IMatch has to wait 7 or 9 seconds for the database. But there is not much else going on so this is very strange. Have you excluded the folder (!) containing the database in your virus checker?
Unexplained delays (without anything logged) are a typical sign for an on-access virus checker blocking the database after IMatch has written something for a scan...

[Joe Austin]

I do use a lot of categories catagorizing by shoot and species, but I can't see why it would be 50k.   Does that number include keyword categories?

I did another edit session (deleting 70 of 110 images) after checking that exclusions were in place for database, program and image folders.   Took 8 mins.   Log attached.

[Mario]

I do use a lot of categories catagorizing by shoot and species, but I can't see why it would be 50k.   Does that number include keyword categories?

This number is the total number of categories in the database. I cannot say which kind of categories there are. But it is a very high number for such a small database. Unless you have 50,000 different keywords...?

Two warnings about IMatch being unable to produce cache images for two ORF files:

_AAA8580.ORF
_AAA8779.ORF

16 ms for removing a file from the database. For 3 files.
Then, 7812 ms second for removing the fourth file from the database, because it could not get a transaction in time.
This means that some other part of IMatch was keeping the database very busy.
Were you ingesting files or updating folders at the same time? The two warnings about the ORF files would indicate that.

Some files are deleted in fast time, then the Viewer is opened again, many cache images are loaded. Cache images are created. Then another file is deleted in < 100 ms.
Then the last file is deleted, and this takes 7 seconds. Where you re-opening the Viewer while IMatch was still deleting files? I cannot tell such details from the logs.

[Joe Austin]

I do use a lot of categories catagorizing by shoot and species, but I can't see why it would be 50k.  Does that number include keyword categories?

This number is the total number of categories in the database. I cannot say which kind of categories there are. But it is a very high number for such a small database. Unless you have 50,000 different keywords...?
  I paged through my keyword categories at 52 lines per page and came up with 3100 keywords.  In addition to that I would have estimated maybe another 5000 in trip/species categories.  I have no idea how I could have 50,000

Two warnings about IMatch being unable to produce cache images for two ORF files:

_AAA8580.ORF
_AAA8779.ORF

I examined both of these files in Imatch and Photoshop/ACR and noticed nothing unusual.



16 ms for removing a file from the database. For 3 files.
Then, 7812 ms second for removing the fourth file from the database, because it could not get a transaction in time.
This means that some other part of IMatch was keeping the database very busy.
Were you ingesting files or updating folders at the same time? The two warnings about the ORF files would indicate that.
No, I was keeping the process as pristine as possible.


Some files are deleted in fast time, then the Viewer is opened again, many cache images are loaded. Cache images are created. Then another file is deleted in < 100 ms.
Then the last file is deleted, and this takes 7 seconds. Where you re-opening the Viewer while IMatch was still deleting files? I cannot tell such details from the logs.
No, just watching the file window and the background task indicators.

[Mario]

I examined both of these files in Imatch and Photoshop/ACR and noticed nothing unusual.

Run WIC diagnosis in IMatch for these two files: WIC Diagnostics
Adobe is no measure to judge Microsoft Windows WIC or LibRaw. That Adobe has no issues does not mean that LibRaw or Windows WIC has issues.

You are using undocumented, proprietary RAW formats from a company who provides no developer support. Adobe may have the incentive and funding to support the RAW formats you are using, but Microsoft may not care and the LibRaw project is an open source project run by volunteers.

I can only tell you that IMatch reported problems reading these two files. The WIC diagnostics will tell you what the problem is.

Tip: Contact your camera vendor and demand that they support the Windows WIC standard and provide a WIC codec which can read all your proprietary RAW files. That would be a good thing. You surely don't want to depend on Adobe to be the only company being able to process your RAW files, right?

I can only tell you what I see in the logs. RAW files not being processed due to errors. File Delete operations running normally unless they suddenly take 1000 times as long. I don't have your PC here in the lab, I don't have your database or images. It's all just looking at the issues one user reports. Trying to help with what I've got.

[Joe Austin]

WIC dx for both.

Looks like it reports no WIC codec for this format, but I have thousands in this format and the problem is a recent one.

[Mario]

Windows WIC does not handle the ORF variant you use.
But LibRaw does. Make sure Edit > Preferences > Application: Prefer photools.com RAW processing is enabled. This tells IMatch to use LibRaw over WIC.

If LibRaw fails to, sometimes, process some ORF files, it could be a "stress" issue (system not behaving under prolonged load), especially when you process large batches of files. In that case, try to switch to a lower performance profile (E > P > Application) like "Balances" or even "Low".

Regarding the delete operation performing normally (< 200 ms per file) and then suddenly taking 7 seconds while the database system reports several seconds wait time to open a transaction - so far we have no information that would indicate a reason for this. The log shows the repeated attempts to get a transaction, but not much else, especially not what else in IMatch could be using the database that heavily in that moment. Nothing is logged.

The only idea I have is that you change your behavior, aka what you are normally doing when this happens.
Close the Viewer, select "Yes" to delete the files. Then do nothing while IMatch is deleting. Does the same happen again?
What if you, instead of deleting the files, let IMatch mark them as "rejected". And then delete the rejected files. Same issue?

Always keep debug logging enabled. Maybe changing the workflow slightly reveals additional details about why the delete sometimes is so slow and what's causing it.

[Joe Austin]

Windows WIC does not handle the ORF variant you use.
But LibRaw does. Make sure Edit > Preferences > Application: Prefer photools.com RAW processing is enabled. This tells IMatch to use LibRaw over WIC.  I've been using LibRaw since you suggested it earlier.



If LibRaw fails to, sometimes, process some ORF files, it could be a "stress" issue (system not behaving under prolonged load), especially when you process large batches of files. In that case, try to switch to a lower performance profile (E > P > Application) like "Balances" or even "Low".  I have been using the default setting.  So 'Low' performance might provide faster results?

Regarding the delete operation performing normally (< 200 ms per file) and then suddenly taking 7 seconds while the database system reports several seconds wait time to open a transaction - so far we have no information that would indicate a reason for this. The log shows the repeated attempts to get a transaction, but not much else, especially not what else in IMatch could be using the database that heavily in that moment. Nothing is logged.

The only idea I have is that you change your behavior, aka what you are normally doing when this happens.
Close the Viewer, select "Yes" to delete the files. Then do nothing while IMatch is deleting. Does the same happen again?  That is what I have been doing all along.

What if you, instead of deleting the files, let IMatch mark them as "rejected". And then delete the rejected files. Same issue? Tried that yesterday, no change.

Always keep debug logging enabled. Maybe changing the workflow slightly reveals additional details about why the delete sometimes is so slow and what's causing it.
 
I have done some testing and found that this delete slowness seems to only occur when deleting from categories, and not when deleting from folders.    I have done some projecting and can't see how I could have more than about 8000 categories at the most.  50,000 sounds like a problem.    Is there some way I can produce a readable list of all the categories currently in the database?

[Mario]

Please don't use red text color when replying. It's feels similar to shouting and should only be used for really specific purposes.

Here's how to use quotes sensibly:

a) select text to quote
b) press the Quote selected text button

So 'Low' performance might provide faster results?

Low performance is the setting that makes IMatch use the least system resources. Not recommended for normal operation. It's just a test to see if the occasional inability of LibRaw to process your ORF files caused by some sort of stress issue on your PC. If PCs are maxed out for a long time and cannot handle it, strange things can happen.

I have done some testing and found that this delete slowness seems to only occur when deleting from categories, and not when deleting from folders.

Was this always the case in your log files? Because it in the log I've seen, 3 delete operations ran normally, and the 4. ran super-slow. What kind of category were you viewing? A @Keywords category? Data-driven? Formula-based? Manually assigned?

Deleting files will invalidate all categories and collections and IMatch has to recalculate them when they are visible somewhere or their counts are needed.

Looking at the log, I see many calls to ProcessTag, which means a data-driven category (or many) were updated. This can take a while to complete...

Is there some way I can produce a readable list of all the categories currently in the database?

The short stats in the log file just tell me:

148 folders
52181 files
51012 categories

The diagnosis log file lists all categories. 
First, maybe have a wee look at @Keywords and all categories you have created? Creating a data-driven category for, say, date and time, can easily produce as many categories as there are files in the database...

[Joe Austin]

Please don't use red text color when replying. It's feels similar to shouting and should only be used for really specific purposes.

Sorry, thought it would be easier to spot.

Low performance is the setting that makes IMatch use the least system resources. Not recommended for normal operation. It's just a test to see if the occasional inability of LibRaw to process your ORF files caused by some sort of stress issue on your PC. If PCs are maxed out for a long time and cannot handle it, strange things can happen.

Ok.  Just as a footnote, my pc is a I7-14700 with SSD drives so it shouldn't be much of a bottleneck  (and I never experienced these issues  with my i7-8700.

Was this always the case in your log files? Because it in the log I've seen, 3 delete operations ran normally, and the 4. ran super-slow. What kind of category were you viewing? A @Keywords category? Data-driven? Formula-based? Manually assigned?

All of the cases have been deletions from manually assigned categories.

The diagnosis log file lists all categories.
First, maybe have a wee look at @Keywords and all categories you have created? Creating a data-driven category for, say, date and time, can easily produce as many categories as there are files in the database

I have found the source of the excessive categories.  They are in the 'Admin'  area, principally Admin|Month  which appears to have tens of thousands of categories.  The formula is: XMP::xmp\CreateDate\CreateDate\0.    I don't remember creating this,  Is this a category supplied with Imatch and if so, in what version did it appear?

[Mario]

No, this is not a standard IMatch category.

I explicitly warn in the help about creating categories from full time stamps (including the time part down to seconds), because it will create a ton of categories in most cases. It's not a "problem", but it can bog down a system when such a category has to be updated often, like, one for each deleted file...

Maybe the problem is gone after you deleted this category?

Ok.  Just as a footnote, my pc is a I7-14700 with SSD drives s

That's a 20 core processor and IMatch will utilize all performance and efficiency cores for a prolonged time while indexing files. The more processor cores a system has, the more likely the storage system becomes the bottleneck. A 24+ core processor can max out even a .m2 SSD when IMatch is ingesting files and updates the database with 48 threads or more in parallel.

Not a problem per-sé, but, sometimes, users experience strange issues like WIC errors or a few files ending up without a thumbnail, or just some strange warnings in the log file or, mostly on laptops, IMatch being terminated by Windows (no DUMP file). I even had to implement my own temporary file naming schema, because the standard Windows function was overloaded and produced identical file names instead of random file names

The reason for such random effects could be basically anything. A overheating problem. Faulty RAM usually not used. A glitch in the firmware or the Matrix. Really hard to track down.

If the issue is reproducible, my first suggestion is usually to temporarily use a lower performance profile to reduce the maximum of CPU IMatch is allowed to consume.  To see if it's a kind of stress issue.

[Tveloso]

As it happens, I had this same warning in a Db Diagnostics I ran this morning:

    [033509] WorkBatch|2 Location
      Children:          0
      Assigned files:    490
      Calculated files:  490
      Warning: Category references non existing files. Correcting.

...which I'm pretty sure I haven't seen before.

I had also deleted a few files via the Viewer.

Last evening, I had 2 or 3 Viewer Sessions (where I was working on Faces), and deleted 2 Files upon exit from one Viewer session, and 1 File upon exit from another.  In my case, I was working from a Result Window (and not a Cagtegory).

After completing the Diagnostics this morning, that Category now reports 489 Files:

   

I imagine that, had I not run a Diagnostics, I might also have seen that "thumbnail space"?  I went through that category afterwards and did not see that.

I'll keep an eye out for this, but I imagine that it's not easily reproducible...I have marked files for deletion plenty of other times (via the Viewer) in the current version of IMatch (and clicked to Delete them upon exit), and this is the first time this has happened.

And since the Diagnostics corrects the condition, it's not really a problem.

[Mario]

I have not seen this so far. And the code removing deleted files from categories looks OK and is called when a file is deleted.
Let me know when you can find a condition or series of steps which produce this outcome.
To figure this out would mean to delete a file in the Viewer and then run the diagnosis. If the error does not happen, repeat until it happens...

[Joe Austin]

I've deleted the problem category and also tried "Low" performance but no significant change in the time it takes to delete, and in the last three tests of 100 deletions each I have had 60 "spaces"

I'll keep observing, but for now I'm out of ideas.

[Mario]

I have tried to force this somehow.
New folder with 1,000 images. Added this folder to a category.
Switched to the Category View and selected that category.
Also opened the Category Panel for maximum impact.

The Category View and Panel can be really "demanding", when forced to recalculate the counts for complex categories which e.g. use collections, variables, data-driven etc. This may "compete" with other operations which also use the database a lot, like, deleting files. Deleting a file invalidates all categories, and when you are in the Categories View and/or the Categories Panel is open, this will cause categories to recalculate immediately.

I could not reproduce this slow down yet. 
I see the "empty" panels shortly while the File Window is updating to reflect that files have been removed. But this takes maybe a second and then the deleted files are gone. New empty panels may show while the delete op continues. But after all 1,000 files have been deleted, the file window is empty. Or shows only the files I de-selected before deleting.

But my guess is that, especially with 50,000+ categories, at some point the category updates interfere so badly with the thousands of database operations caused by deleting files that performance slows down.

I have an idea for an optimization for the Categories Panel to temporarily delay updates while a batch file op is running (this is how I named this). I will try this out and if it improves things, I will include it in the next release.

[Joe Austin]

Thanks for the effort.   I did note that I removed all but 10000 of the categories but still have the problem.

[Joe Austin]

I have upgraded to 2025.2, but still experiencing the problem,  with .CR2 files and .ORF files.

I did see two other photools processes running in task manager : photools Imatch Chromiu...   

Their properties are identical (IMatchChromiumHelper.exe) but they don't seem to use any CPU time.

Any chance these could be a problem?

[Mario]

No. IMatch runs apps isolated in ChroimHelper instances.