Slow Rescan on local SSD

[ERBRO]

Hello,

I added new files (about 3000) on my C: Drive outside of Imatch and proceed to a rescan (Shft+F5).

The rescan looks to me slow (about 2 to 3 hours based on attached print screenà

Is this a normal speed ? Ir not wht could affect the performance knowinf that the folders to rescan and the the imatch Database are on my C: drive.

Thank you for any recommendation or advice.

Best regards
Eric

[Mario]

3,000 files should take a couple minutes, tops. Unless the 3,000 files are all 10GB video files or something.

You did not include any information that is helpful, like the file format you use, your virus checker, if you have made exceptions for IMatch and of course the super-important IMatch log file from that session (see log file).

Because the log file would answer all that and tell us what takes how long.

[ERBRO]

Hello Mario,

These are quite small JPG Files (< 1 Mb/file).  I have activate the debugging log  and will provide you with it by mail later today.

Thank you in advance.

Best regards
Eric 

[Mario]

I had a look at your log file.
45,700 lines of text.

16 folders are added to the database.
At the start of the log file, I see about 10 warnings like this:

Code Select Expand

Error loading C:\Users\---.jpg with error 1 'File Sharing Error'
Error loading C:\Users\---.jpg with error 1 'Unknown image type.'
so something is fishy. The same images are loaded fine 6 minutes later! Strange.

Then IMatch does some folder rescanning and bookkeeping between 15:17:15 and 15:17:48, checking the folder tree. Lots of file system operations, rather slow by nature.

15:17:48 the first new file is added to the database and enqueued for processing. Many files added.
Since you probably have the folder(s) open in a File Window, the File Window reloads several times per second. It looks like a bazillion times. Not sure why the File Window is forced to reload so often. Which folders did you show? Options? Layout? Which panels were open?

I also see a ton of file relation operations and logs like

Code Select Expand

#FindRelated: 72 masters, 12 definitions, 0 links found. 9410401 files analyzed.
Do you use very complex file relation rules? Why is IMatch forced to analyze 9,410,401 files when checking for versions for only 72 master files? 10 million files to check, and that over and over again. It's fast, but it will still result in a lot of database activity.

Updating (rescanning, checking, relations) of the the folder C:\Users\ericb\Pictures\Imaginwoo\P12\ takes 188 seconds(!) alone, probably due to all file relation checks. Other folders take 200 seconds or even 700 seconds! 
You can find the names of these folders by searching for #sl and then backwards for AddOrUpdateFolderInner

This of course takes a lot of time.

15:29:05 (!) The first metadata is extracted from an image.Everything before was folder scanning, file relations updates, and tons of File Window reloads.

Metadata is extracted in 63 ms.

Again, the File Window is reloading, reloading, reloading....
Did you have the database node or a very deeeeep folder hierarchy selected and hierarchical display enabled? 

File processing times look OK, 200-300ms per file, sometimes 500, with many files being processing in parallel.

I think all the time required to rescan your folder hierarchies and resolve file relation is what makes this slow, not the actual file processing.

[ERBRO]

Hello Mario,

Thank a lot for this deep analysis.

I cross-checked the relation Definition and indeed almost all were flagged.  I have removed all except the one I really need. 

After I performed a test with a new folder and 7000 images and it was rescan in less the 5 minutes.


Thanks again.

Best regards
Eric