Very slow imports and missing thumbnails

[dcb]

I'm running 2025.4.4. In this version and the last I've noticed an incredible slow-down in import speeds. I'm not suggesting it's a bug per-se but thought this the best place to report it.

There are two things happening and I've not yet been able to determine why.

1. Imports used to be fast. Tonight I've imported just 32 cr3 files and it's taken several minutes. In this past this would have been no more than 30 seconds (guessing, but I know it's much, much slower now).

2. After the import most images are missing a thumbnail and need a force-update to create it.

Tonight it was .cr3 but I'm seeing the same behaviour with .dng, .jpg, .afphoto (Affinity Photo).

My anti-virus is Sophos Home. I've excluded both the IMatch folder in it's entirety and also my photo drive. I've even fully disabled it. 

Metadata write-back is perfectly fast.

Overall IMatch is very fast. It's just the initial import and thumbnail that's slow.  I'm running Windows 11, 32GB RAM, AMD Ryzen 5600 CPU. New device, less than 12 months old. Photos and database are on the same drive, a M.2 motherboard fitted SSD.

I don't know what to look at next. The problem only began with the version previous to the current. 

[Mario]

No logfile attached. Not much data to work with.
A log in debug mode (see log file) is the first thing we need to analyze this.

Photos and database are on the same drive.

Probably a starving problem with the SSD as the bottleneck?
Switch the performance profile to "Balanced" or "Low" and retry and then force a rescan of the 32 CR2 files to see if there is a change.

Which virus checker are you using and have you made exceptions for the database folder and IMatch2025x64.exe?

[dcb]

Thanks Mario. Here is some more information. I appreciate your help.

Setup

• Put IMatch into debug log mode. Closed IMatch
• Created new 10 images that have not been previously ingested. Each image is CR3, DNG and JPEG so 30 images totaling 1.13GB.
• Copied images to S:\memories\import\_SD Card folder.
• Confirmed Sophos Home virus checker has IMatch executable as an exception

Test

1. Start IMatch (Performance Mode)
2. Waited for files to ingest
3. Wrote back metadata
4. Reviewed log file (attached). Could find nothing other than a SLOWCAT

Thoughts
It it always subjective but I think that ran faster and more as I expected. Previously I had the imatch6 folder excluded from anti virus and this time it's just the IMatch2025x64.exe. This time, progress was much faster. Prior to the AV change, IMatch was freezing, and reporting times like 1.25 hours(!) to finish, though in reality it did not take that long.

I'm happy to leave this for now Mario, but can return if I notice any further slow downs.

[ubacher]

I have the same problem: Thumbnails don't show after import. Not always - but with 185 Raw files import on idle system it did.. Rescan fixes it.

[stefanjan2]

I also often have a problem with thumbnails not showing after import and have to rescan to fix.

[Mario]

ExifTool extracts metadata in between ~0.4 s and 1.6 s per file. Interestingly, JPG are not always faster than RAW files, even if the number of tags is about the same. On the average less than a second per file, which is OK. IMatch processes many files in parallel.

It takes ~ 30 ms to ~ 400ms to import the data into the database.
Write-back takes between 1 and 2 seconds per file, probably depending on current disk utilization.
The total time to write-back 30 files is about 7 seconds (IMatch writes many files in parallel). 
This all looks pretty normal to me.

What is puzzling it the rather long time it takes to create cache images for some files.

The file S:\memories\import\_SD Card\IMG_2965.CR3 is loaded in 100 ms, but it takes almost 30 seconds before the cache file creation is completed. The same happens for the file S:\memories\import\_SD Card\IMG_2968.CR3, loaded in 150 ms, but total cache file also takes almost 30 seconds. The 30 seconds are basically the time Windows WIC needs convert the already loaded RAW file into JPG and write it to disk.

The category 'social tagged versions' is reported several times as a slow cat (> 2 seconds update time). For a database with 40K files this should not happen, but IMatch was busy indexing, writing back and re-indexing files, and there is only so much CPU to go around. So this is also OK.

The modified folder is found at
06.29 09:31:23+  16 [C26C] 10  I> FolderSweeper: Folder [3550] 'S:\memories\import\_SD Card\' modified.

The last time metadata was imported (after the write back) was
06.29 09:33:57+    0 [E3F8] 10  M>  <  0 [375ms] PTMetabase::ImportFileValues

Then a bit of normal IMatch processing, apps running., background processing and then you close IMatch.

The only unexpectedly slow operation is writing the cache images. 
Everything happens on the same disk (S:). It contains the database, the images, and also the cache folder. This is the "worst" case for performance of course. Everything competes for disk I/O, so I assume the disk becomes the bottleneck. But a SSD in a one year old system should be able to handle that, I guess?

Did you only change the virus checker or also the performance profile?
When you keep the Windows Task Manager (Performance Tab) open while you repeat this test, is the S: disk shown as 100% utilized and/or are the processor cores 100% utilized.

[dcb]

Thanks for the analysis. It might take me a couple of days to repeat the test to check S:\ usage. From memory, when the problem was first noticed I checked CPU and that wasn't near 100% so I suspect, as you do, it may be the disk.

All I changed was the anti virus path. I did not change the performance profile from what it was originally.

I assume database, images and cache all separate is ideal. I do have two other SSDs. The main system disk and a second drive. The images need to stay where they are on S:\ for space, but I could split the others.

[dcb]

More testing.

Test 1

• Essentially the same as yesterday's test. Monitored S:\ in task manager. Max seen was 10%. There was a spike on the refresh where it temporarily hit 100%

Test 2

• Cleared files from IMatch.
• Moved cache to P:\temp\imatch cache, and copied all cache files across.

P:\ is a SATA SSD. 1-2% use during import.

• Ran import, metadata write
• Ran 2 x full rescan. On the first, S: hit 100% but to be honest I wasn't watching that window at the time of import. Could have been something else. Second rescan, only 1-2% use on S:.

[Mario]

Do you still experience slow imports after updating your virus checker definitions and moving the cache?
If you can move the database to an SDD that not also contains your images, you should see performance improvements.

The "slow" time to create a cache image is down to 5 to 8 seconds, instead of 30. Still not that good. I would expect to see more like 1 or 2 seconds, max., for typical current RAW files.

I also see that some threads had to wait a rather long time to get access to the database. Database and images on the same disk still? Performance profile set to "Performance"?

[dcb]

Do you still experience slow imports after updating your virus checker definitions and moving the cache?
If you can move the database to an SDD that not also contains your images, you should see performance improvements.

I think with the cache moved, the speed is much better.

More tests

Test 3 (database, images and cache separated)
- Performance mode (import, write metadata, full rescan)
- Database moved to C:\ as the only other SSD I have available. It's an older SSD.
- Cache on P:\
- Images on S:\
- Performance was worse than when I first raised the issue. C:\ at 100% for a long period of time. Overall app response was slow.

Test 4 (database, images and cache separated)
- Balanced mode (import, write metadata, full rescan)
- Database on C:\. Drive still getting hit, but not for as long or as consistently.
- Cache on P:\
- Images on S:\
- Performance was better than Test 3, but still poor. Overall app response was slow

Test 5 (database + images, separate cache)
- Balanced mode (import, write metadata, full rescan)
- Database on S:\ (where it was originally)
- Cache on P:\
- Images on S:\
- Performance was slower than yesterday due to balanced mode. Faster than when database on C:\. C:\ still hitting 100%

Test 6 (no log file)  (database + images, separate cache)
- Maximum performance mode mode (import, write metadata, full rescan)
- Database on S:\
- Cache on P:\
- Images on S:\
- No noticeable difference over performance mode. C:\ hitting 100%


I don't know if C:\ is the ultimate culprit. It's the Windows drive and app drive. Nothing much else happening so all disk activity safe to assume IMatch related, even if not IMatch caused.

Possibly I could install a second NVME drive on the motherboard.

CrystalDiskInfo reports:

C: 77% health (44,000 hours)
P: 88% health (33, 000 hours)
S: 99% health (2165 hours)

[Mario]

General rule as explained in the help: Database on the fastest SSD you have. If possible, images on a different disk. 
This is the best scenario. But most people have now only one SSD in their systems, and they are so fast that it does not really matter.

Making sure that the virus checker is not slowing things down (most frequent cause for slow performance) is important.

When images, database and cache are on S: /P: , the only things IMatch writes to C: is the log file (assuming your TEMP folder is on C:)  and short-lived XMP files written by ExifTool during the import merge.
How would writing a log file and small XMP files max out your C: disk?
If debug logging is enabled, a lot more data will be written and noting is cached. This produces more I/O. But it should still show merely a blip in performance monitor.