IMatch unresponsive for hours taking lots of resources when exiting

[Damit]

I have not used IMatch for a while.  I went in to import some files and the program has been unresponsive updating and importing the photos for a long while. First it said it was going to take a few minutes, but then it turned to 51 hours. I tried to get into the database tools to compact the database, but now it is waiting for the background processes to complete. It already crashed and restarted once during this process but now it has been taking up 70-80% of the processor for 30-40 minutes, all while the IMatch UI is greyed out and unresponsive.

This is a recent i9 processor on an extremely fast computer with everything in SSDs locally. Unfortunately I cannot say this is new.  For some reason this program labors in importing files. Every time I do it crashes at stalls at least once and finally finishes importing when I restart. But this time it is taking forever.  Granted it is about 300-400 gb of info but it should not take this long. Please advise on how to get things up and running.

[JohnZeman]

I suspect Mario will want you to attach the log file when IMatch is in debug mode.  Even if IMatch freezes it should produce a log file.

[Mario]

Switch IMatch to debug logging via Help menu > Support.
Let it run for a while, indexing files.
Use Help > Support > Copy Logfile to copy the log file.
Close IMatch. Copy the log file now, so we can see what happens during shut-down.
See log file for detailed info.

This provides is with a minimum of information to work with.

Which virus checker do you use?
We've just had a similar case yesterday where the virus checker was the cause for all the troubles. Again...

[Damit]

I have updated, enabled debugging and am letting it run overnight. It seems to be stuck importing metadata from the same batch of files over and over. Sometimes it does it repetitively in the same session.  It will read the metadata, say it is updating files, and then starts reading the metadata of the same 2049 files again.  Every time I try to shut it down it hangs on finishing background processes for at least 20 minutes, mostly much longer. Hopefully this will resolve itself with hours of running time (though it has been running off and on for over 6 hours). I will report back with the log file attached tomorrow. I do not use anything other than windows defender and I have exceptions for iMatch. While I have had similar issues before, they did not persist and were resolved after 1 or 2 restarts.  This has not resolved after 4-5 restarts and an update.

[Mario]

Do not let it run over night. The log file will be probably huge and impossible for me to analyze.
What you describe sounds like a virus checker blocking IMatch or ExifTool from accessing files. Or the files contain so many errors that ExifTool gets stuck. All this will of course be logged.So, when you see this for a couple of minutes, make a copy of the logfile and ZIP and attach. I'm sure there will be many warnings and errors in the log file.

Again, my question about which virus checker you use. Important.

[Damit]

OK, I checked back and stopped the index and exited.  The file was huge, but I think you only need to check the end of it.  I can try again if this is not acceptable, but I hope you can make use of it.  I had to post a link to it, since it was 12.5 MB. As I wrote, I do not use anything other than windows defender (I just checked all my programs to make sure) and I have exceptions for iMatch, both the program (current version) and the database folder in Windows Defender (we have been through this before). I tried again completely turning defender off.  Nothing changed.  The behavior is this same.
Log File

[Mario]

Looking at the log file, I get these stats:

2771454 lines.
6 warnings.
0 errors.
92801 slow statements.

The warnings are normal, except for one warning that the folder scanner has found a modified folder in the Brand New hierarchy and is scanning and updating it. This is due to the new check in IMatch 2025 which works around the "new files in folder, but folder last modified not changed" Windows issue.

Then it gets bad. Literally everything on your system is slow. IMatch logs a #sl (slow statement) when it encounters something that takes longer than 5 seconds. And your log file has over 90,000 of these statements. Normal would be none or maybe a handful.

Your database has 280K files, which is a medium-sized database. On a PC with 32 cores and 128GB RAM this database should just fly. I use a database with this size on my notebook, with excellent performance!

The database is 'D:\IMatch\test3\IMatch Database-test3.imd5'. Is D: your fastest SSD/disk?

I don't understand why everything is so slow. I mean, right at the start I see log entries like

[131093ms #sl] PTMetabase::GetWriteBackFiles
[133687ms #sl] CIMCollection::InnerCalculate
[162078ms #sl] PTMetabase::ImportFileValues
[13985ms #sl] CIMEngineUpdateQueue::AddEntry

There are usually all routines which run in a few milliseconds or maybe 100ms. Not 13,000ms.
Adding an entry to the update queue usually does not even measure because it takes < 5ms. It's just a simple database add operation.

Your system is totally maxed out, indexing files. I see tens of thousands of entries like this

08.06 05:19:06+    0 [4EA4] 50  M>  <  0 [17016ms #sl] PTMetabase::Propagate
08.06 05:19:06+    0 [4EA4] 10  M>  <  0 [17094ms #sl] PTMetabase::ImportFileValues
08.06 05:19:06+    0 [4EA4] 10  M>  <  0 [17094ms #sl] PTMetabase::ImportFiles
08.06 05:19:06+    0 [94B4] 10  M>  <  0 [16875ms #sl] PTMetabase::ImportFiles

Everything "touching" the disk / database takes 13 to 20 seconds....

Either the system is maxed out beyond belief (but not a 32 core system) or the disk subsystem is performing so badly that the database system's performance slows to a crawl.

This is a typical sign of a on-access virus checker scanning the entire database after each write-operation, blocking IMatch for  the duration of the scan. This would also explain the very long but similar timed execution times for statements which access files or the database. Or both.

This also explains the slow shut-down. IMatch is processing files on all cores, and when you shut-down it completes/cancels the active threads. But when every thread is stuck for ~15 seconds, and IMatch runs many of these, it will take a really long time to shut-down everything completely. When IMatch is stuck in a operating system call, e.g. reading or writing files, it cannot terminate the thread until the operating system call finishes. And when the call is blocked by your virus checker...

Things to try:

1. Open Edit menu > Preferences > Application and set the performance profile to "Balanced" or "Low". 
This reduces the "stress" on your system.

2. Double-check that the IMatch executables
'C:\Program Files\photools.com\imatch6\IMatch2025x64.exe'
'C:\Program Files\photools.com\imatch6\exiftool.exe'
are excluded in your virus checker.

3. Make sure the database folder

'D:\IMatch\test3\'

is excluded in your virus checker.

[Damit]

Thank you, Mario, for your help! I really hope to get to the bottom of this. I am willing to provide any information you may need to figure out what is going on. BTW, I left the program running and it is still trying to import the files. This is distressing.

I will try your suggestions, but this is not the first time I use IMatch on this computer. I have been using it for over a year with this database adding small amounts of data here and there with very little issue. But perhaps it is only with data dumps this large that the problem becomes apparent?

The D drive is a 4TB WD sn850x nvme ssd, with over 1.5 TB in space, it is plenty fast.  The files are on a 3-Lexar MN790 nvme SSDs run in JBOD directly attached to the Motherboard. They should fly. My memory is 128 GB and barely ever touches 50%. and I have an i9-13900K to run it all.  I specifically built this computer to work on DAM, specifically IMatch, trying to eliminate hardware from ever being the problem, as I have had issues with stalling before, and I do not run anything else on it except a browser, word or excel.  I do all my other work on a server. I reinstalled IMatch and completely rebuilt this database on this computer carefully to avoid problems. So, it is particularly painfully that despite all my efforts I am still having issues.

The warnings are normal, except for one warning that the folder scanner has found a modified folder in the Brand New hierarchy and is scanning and updating it. This is due to the new check in IMatch 2025 which works around the "new files in folder, but folder last modified not changed" Windows issue.

This is what I was doing when everything went bad.  I added siles from my SD card to the "Brand New" folder and iMatch asked me to do a Shift=F5 rescan.  I did this and that is when everything began to stall.  Granted this could be due to the new files added.I have gotten ranges of 5 minnutes to over 100 hours for the files to be imported and updated.  None of these files have anything other than normal camera metadata attached to them.

I have checked again, and D:\IMatch is added as an exclusion in Windows Defender, as you have advised before. I also have the actual executable for 2023 and 2025 as exclusions and will add the main program folder parent directory as an exclusion as well. I have also completely turned off defender and the behavior does not change.  I don't know what else to do to rule out the Virus Checker, but I think I have.  I can send a program list if you wish to convince you I have no other antivirus and am willing to do whatever to rule out a virus checker. 

Either the system is maxed out beyond belief (but not a 32 core system) or the disk subsystem is performing so badly that the database system's performance slows to a crawl.

The system does max out, but only when iMatch is indexing. It can take up to 95% of my processor (under task manager: photools.com IMatch itself is taking over 96% at times, I have a word doc with screenshots showing this. The only Virus Check I see is Microsoft Defender Core Service at 3%, but even if I turn it off, and the percentage foes to 0%. The behavior continues. So, it is IMatch that is taking all that power to process, but what it is doing, only you may know. I wonder why would IMatch be taking so much processing power if a virus checker is checking a file? I could understand a lag, with IMatch idling, but the opposite is happening. If IMatch is not running, I barely hit 15% processing. Also, I set the performance to balanced and when I did the system hanged for about 10 minutes. Does a virus checker explain the program rescanning the same files over and over again, or all the other behavior? I sadly suspect this is an issue between IMatch and my files, but I cannot fathom a reason why. I hope I am wrong.

[Mario]

The default "Performance" performance mode is allowed to use up 80% of disk, cpu and memory, on the average. Peaks are normal.

Maybe your 32 core rig in this mode is too much and the disk cannot keep up. Some SSDs become a lot slower once their internal high-speed cache memory is exhausted and they must actually do work.

Try the Balanced or eve Low performance profiles to see if your system handles this badly.

Keep an exe on the Task Manager. If the disk keeps up, IMatch will use ~ 80% of all processor cores, with occasional 100% peaks. If the disk is maxed out or above 80%, IMatch will reduce core usage. Sometimes it even drops to one single thread, then it increasees threads, monitoring disk performance and database performance. Suddenly, transaction times shoot up again and IMatch reduces threads. But it cannot kill threads currently processing, so the situation takes a while to remedy.


This usually balances out well, getting things done as quickly as possible. On my 24 core workstation I create new 50,000 files test databases super fast.


I see in the log that IMatch dynamically adds and reduces threads, because the transaction duration of the database system suddenly goes up to 5 seconds or more. Not 20ms or 50ms. That's bad.

But when a simple "write one record to the database" suddenly takes 20 seconds, or importing metadata values for a single file takes 15 seconds, all bets are off. IMatch struggles badly when usually super-fast functions suddenly stall for 15 or 20 seconds.

Change the performance profile to Balanced or Low. This should be more than fast enough on your rig. A 32 core processor is server-grade or in a workstation, usually.

Keep an eye on the Performance tab in Windows Task Manager. I wonder what is maxed out, CPU or disk(s)?
During indexing, everything is usually I/O bound, cores waiting for the disk.
Unless the files have embedded face regions or IMatch is set to perform face recognition automatically. This will utilize all cores to the maximum allowed by the active performance profile.


IMatch reads from N: and writes to D: and C: (TEMP folder). This is usually the best scenario, different disks for database and images, TEMP folder on the fastest SSD.

[Damit]

I had already set the performance to balanced, but the problem was persisting.  I checked the performance monitor and I did notice that the C drive was being accessed close to 100% of the time for 10-20 second periods followed by 10-20 seconds of 5-10% active time.  This was cycling. While the access time was 100% the actual transfer rate of 10-12 MBps is much lower than the drive can handle. Even so, most of the traffic on the C drive was with the C:\$LogFile (NTFS Volume Log). Photools is not even in the top 10-15 of processes using the disk, and even when the disk use is high, nearly at 100%, it is reading and writing below 6-7,000 B/sec. The log file behavior seems to be very similar to that in my server.

The C drive is a 2TB samsung 990 Pro nvme.  This and the SN850x have large amounts of dram and SLC cache.  They have very fast Random I/O and transfer rates for long periods of time.  They are top of the class. I have hard disk sentinel and all my drives are reading 100% health. My hardware is not the problem. I assume we have ruled out an Antivirus?

When I switched to low performance the toll on the C drive went completely away, but that is because the system hanged. A may be important to note that it did not hang when I hit apply after making the change, it only hung when I clicked on "OK," just like it hangs when I close the database. Why would IMatch hang when I just make a change in processing power?
This did not happen before importing these last files. The photos I just imported are sony jpegs and ARW as well as GoPRo and iphone photos. Nothing exotic. Anyways, once iMatch stops hanging, the same exact pattern occurs with the disk maxing out for 10-15 secs and then dropping to almost 0% for another 10-15 seconds.  I experimented and set iMatch to max performance. Same thing happens. The program hangs for 5-10 minutes and when it comes back the same exact pattern.  The C drive is not tasked any more or less if the performance is set to low or max. As a matter of fact, while writing this, the C drive is now just covering at 1-3% Active time even though IMatch is still adding and updating files in Max performance mode (though the counter has not gone up in over 30 minutes). The only difference observed is that the CPU went from 10-15%, to 25%. This morning and afternoon I have not seen it get close to using 95% of the CPU, as it did last night for hours (I am not sure why the difference, the program is still importing 2868 files, so it should be similarly tasked). I think the most I have seen it reach today is 45%.

This is the third time in 3 years I have run into an unresponsive a problematic database on IMatch and had to rebuild everything. It takes so long. I built a new computer with all new components and moved to Windows 11 from 10 to try to eliminate problems and I am still having very similar issues of unresponsive databases that hang. This last time I tried to be as careful as possible. I do not think I am doing anything special.  I am just importing photos into the program.

[jch2103]

How much free v used space do you have on your c: drive? If it's limited, the OS could be spending time swapping files. Just a guess, though.

[Damit]

Nice guess.  I thought the same.  It has 708 GB left out of 2TB.  Plenty of space.

[Damit]

I cleared all files that need updating and that at least got the database stable.  The program shuts down normally and there is only one warning for a face missing a thumbnail. But this just delays the issue. Something is causing IMatch to stall when updating some of my files and since it is just a select number of files (the ones that I cancelled updating, I actually rescanned the new files with no issues).

[Mario]

Something is causing IMatch to stall when updating some of my files

By the looks of the log file provided and the 90,000+ "slow" warnings, it looks like more that all your files are effected by this, and also the database itself. When writing a single record to the database takes 5 to 15 seconds, something is really wrong.

Try the "Minimal" performance profile setting which limits IMatch to using only two threads. Processing two files in parallel should cause no load on your system at all. Does this change anything?

[fisketjon]

Are you using NordVPN? It has a "Threat Protection" that can really slow down things.

[Damit]

Are you using NordVPN? It has a "Threat Protection" that can really slow down things.

Thanks for trying to help, fisketjon! Nope.  I use Private Internet Access for VPN, but the VPN is not being used for imatch. I use split tunnel on only my browsers use the vpn.

Something is causing IMatch to stall when updating some of my files

By the looks of the log file provided and the 90,000+ "slow" warnings, it looks like more that all your files are effected by this, and also the database itself. When writing a single record to the database takes 5 to 15 seconds, something is really wrong.

Try the "Minimal" performance profile setting which limits IMatch to using only two threads. Processing two files in parallel should cause no load on your system at all. Does this change anything?

I don't know why, now that I stopped all pending metadata rewrites, that all the problems went away.  I could close out with no issues, the Database Diagnostics produce only one warning of a thumbnail mismatch, and I can update the new files with no problems. I have done a refresh of all the folders the last time I used IMatch a few months ago with no issues. Obviously, the problem lies with something that was being written back, but if the problem was so pervasive, I would think I would still have issues.  As far as I could tell the problem was with less than 6000 files.

I am going to try to update file by folder by folder to try to isolate the problem.  I wonder if my use of squared brackets in folder names may be causing an issue.  I am trying to eliminate this, as I have educated myself on the pitfalls of some characters in file and especially folder names.  J

I tried minimal to max and there was no change at all at either setting except the cpu usage which when from 5-10% to 35 % at max.  My computer and the disk were at speeds that they could easily handle.

[Mario]

IMatch usually balances itself pretty well. The ability to change the performance profile just exists for some rare situations, or to deal with the occasional PC that does not behave. 

But a case like yours, where basically everything that reads or writes data is 100 times or more slower than normal is pretty unique. I see things like this usually only when a virus checker really does not behave.

Using "special" characters like [] in file names can be a problem with some software. IMatch uses 3rd party software like ExifTool, FFMpeg, PDFLib and similar, it must communicate file names via HTML to apps {[] has a special meaning e.g. in JSON and in URLS) etc. Always best to stick to A-Z an 0-9 and maybe - _ in file names.