Batch processor - IMatch stalled for 15 minutes - Include Folder problem?

[bekesizl]

I created a Batch processor preset to convert HEIC files to JPEG.
I want to use it to convert iCloud imported photos in my catalog as a copy in my catalog.

I have defined the Output folder as a Year/Heic2JPG_YYYYMMDD subfolder, where the Year folder is indexed already by IMatch. I add the bottom level folder to the database.

When I run this preset on very few files (1-4 HEIC files), it converts the files and then "kills" IMatch for approximately 15 minutes. I cannot do anything and one core of the cpu is occupied by IMatch.

My first try was without "Add output folder" to a folder outside the database. But I wanted the converted files to get indexed automatically.
The second try was with include every folder level settings. This caused a duplicate folder display in the Media & Folders view. The Heic2JPG folder was listed two times with the same count of files and one got offline after deleting the other. Only an IMatch restart could solve this.

I wonder why these happened/happen and if it might have caused my database error case last week?
And if maybe they have something to do with my disappearing custom AI tags?

[Mario]

I would not use IMatch for that, unless the HEIC files have useful embedded previews.
IMatch uses Windows WIC to process HEIC and this has by no means the same workflow and quality as you would get when you use a proper RAW processor, which applies device-specific optimizations to the HEIC matrix data.

As always, we need at least the log file (see log file) of the IMatch session where you experienced the problem. We have basically nothing to work with.  When one core high, IMatch is doing something. Like, ingesting new files, creating cache images, updating categories, maybe writing back

Do you use versioning with metadata propagation? If so, show your rules and folder layout.
A bad version propagation may also wipe AI tags, just as a hint. Since no user has ever reported lost AI tags (or physical database damage several times), any hint and detail matters.

Do you have immediate write-back on?

Switch IMatch to debug logging before you try this again (Help menu > Support > ...) so we get more detailed log files.
ZIP and attach to your reply.
Then we can see what IMatch is doing and what takes so long. Could be anything.

[bekesizl]

I would not use IMatch for that, unless the HEIC files have useful embedded previews.
IMatch uses Windows WIC to process HEIC and this has by no means the same workflow and quality as you would get when you use a proper RAW processor, which applies device-specific optimizations to the HEIC matrix data.

Files are simple photos like a JPEG, not some "RAW" files. Otherwise I would convert them with XnView or XnConvert, but the image quality of the IMatch conversion is good enough.

Do you use versioning with metadata propagation? If so, show your rules and folder layout.
A bad version propagation may also wipe AI tags, just as a hint. Since no user has ever reported lost AI tags (or physical database damage several times), any hint and detail matters.

I use versioning and also some propagation, but only bookmarks, flags, ratings and pins is checked and categories without any category checked.

But the source folder is out of the scope of versioning.
It is the folder where the iCloud client downloads the photos from the cloud. I just want to put them as JPEG in another part of my library.

Do you have immediate write-back on?

No, immediate write-back is not enabled.

Switch IMatch to debug logging before you try this again (Help menu > Support > ...) so we get more detailed log files.
ZIP and attach to your reply.
Then we can see what IMatch is doing and what takes so long. Could be anything.

Attached is a debug log of this evening. I started exporting 15 HEIC files around 20:23 and left my computer. The log ends when I returned to it. 
It seemed like it did convert all files, then asked the question about metadata propagation where I answered no (I don't want to touch the files in the iCloud folder not knowing what happens to them). Then IMatch got itself in a "not responding" state where I left my computer after a minute or so.

Attached is also the Batch Export definition.

I thought about a workaround of exporting files to a folder that is already indexed and I could uncheck the add output folder to database.
But I don't think I configured something wrong in this export preset.

[Mario]

Let's see...

30 files are processed by the Batch Processor.
05.30 20:24:09 => Batch Processor starts

05.30 20:24:20 => last File is processed

05.30 20:24:20 => S:\fotok\Fotok_I_M_f\2025\heic2jpg_20250530\ is added to the database, files are added, collections and categories are updated, folders are scanned for file relations, etc.

05.30 20:24:21 => IMatch displays the prompt about pending write backs before it starts copying metadata to the output files.

05.30 20:24:25 => ExifTool is used by the BP to copy metadata from source files to output files

05.30 20:24:31 => The metadata copying operation is finished.

05.30 20:24:32 => FolderScanner is signaled (made to run). Multiple instances of this background thread are used to monitor folders indexed by the database for changes.

05.30 20:25:59 => The folder S:\fotok\Fotok_I_M_f\2025\heic2jpg_20250530\ is found to be modified and scanned for new or updated files.

05.30 20:27:47 => 4851 folders were checked by the background folder scanner in 0ms.

The FolderScanner signal and temp cache file tasks run repeatedly in internals. Nothing else happens.


05.30 20:46:17 => CIMRelationManager::UpdateRelations for 23068 files and 11763 links in 1306672 ms

05.30 20:46:20 => The Batch Processor result message is displayed. Relations are updated in the background.

05.30 20:46:21 => Files in S:\fotok\Fotok_I_M_f\2025\ are processed, metadata is imported etc.
05.30 20:46:24 => IMatch cannot find in the XMP face region of 'S:\fotok\Fotok_I_M_f\2025\heic2jpg_20250530\IMG_1175.jpg' and the XMP region has no label (typical Apple problem)

Many more files from the folder are imported. 198 files in total.

05.30 20:46:34 => The folder C:\Users\zolib\iCloudPhotos\Photos\ is checked for changes.
05.30 20:47:08 => The folder check is completed (as far as I can tell, no modified or new files are found, thousands of lines in the log file to wade through).

IMatch goes back to idle processing.

05.30 21:13:11 #FindRelated: 15982 masters, 14 definitions, 1296 links found. 600883362 files analyzed.
05.30 21:13:12 CIMRelationManager::UpdateRelations for 15982 files and 4791 links in 1563671 ms

05.30 21:13:12 => Metadata is imported from
C:\Users\zolib\iCloudPhotos\Photos\att.29hOSCAu8pAIWyba3lzIU1iw9c2dAF7tnLnYVCytEb0.jpg
C:\Users\zolib\iCloudPhotos\Photos\att.TGfchbyLobmFUNpvCz9gEtrHjA9KxlWwZ9aaaQa2iyo.jpg

and other files in that folder. The file names look strange, more like something the cloud service is doing, temporary files perhaps?

05.30 21:13:16 => The folder 'S:\fotok\Fotok_I_M_f\2025\2025_05_xx\' is reported as pending and checked.

More files like "C:\Users\zolib\iCloudPhotos\Photos\att.UWCgfGFnh2vNAV06eJUXn0S-ng5TBUzoeG3mYb7_HQQ.jpg" are processed and added to the database.

05.30 21:13:45  => Files named like "C:\Users\zolib\iCloudPhotos\Photos\IMG_1105.MOV" are processed, thumbnail extracted etc.

More files are processed / added / updated.

05.30 21:14:14 => IMatch logs from IDispatch errors returned by Windows WIC (failed to load a file)
05.30 21:14:14 => Failed to create a cache file for C:\Users\zolib\iCloudPhotos\Photos\IMG_8821.MOV'

05.30 21:30:07 The the log ends.

Things To Do For You

Why does the Relation Manager has to analyze  600,883,362 files to check for versions?
Why is the Relation Manager so slow? 21 Minutes for an update...?

99% of the runtime is operating system file system routines, where IMatch scans folders to find masters and versions. This means if many folders have to be scanned due to a misconfiguration, things will get really ugly quickly.

Check which folders your rules have to scan. Usually "master folder and one level down" is sufficient. Don't let IMatch search the entire database or deep hierarchy.

Also, if you index a "cloud folder" in your database, the cloud drive service may start to download files, or at least file names, when IMatch scans the folder for new or modified files. This may cause delays. Not sure what the Apple client does, Apple is always "special".

This is the first thing I would check for.

Check the file names I have listed to see if they make sense. Where does the xx folder come from?

You can see for yourself in your log file what IMatch is doing, which folders it processes, which files it adds etc.

[bekesizl]

Thank you very much for looking into this.
I will solve this job (of iCloud photo import) with a very different approach.

C:\Users\zolib\iCloudPhotos\Photos\att.TGfchbyLobmFUNpvCz9gEtrHjA9KxlWwZ9aaaQa2iyo.jpg

and other files in that folder. The file names look strange, more like something the cloud service is doing, temporary files perhaps?

File name is indeed from iCloud Photos. It is naming files saved in apps (e.g. Messenger) likes this.
It is OK, at least not an error.

Why does the Relation Manager has to analyze  600,883,362 files to check for versions?
Why is the Relation Manager so slow? 21 Minutes for an update...?

That is an eye opener for me and is probably caused by following version relation rule and some similar ones.

It was intended to leave out "C:\Users\zolib\iCloudPhotos\Photos" from the version relations.
iCloud is using filenames multiple times so my version relations were pairing old jpg and new heic with same name but completely different content.
I didn't want that and changed the folder part of the relation settings. Now I reverted it to Master Folder / specified folder only as it was originally.

Check the file names I have listed to see if they make sense. Where does the xx folder come from?

I use daily folders/event folders named like 2025_05_xx for every month for a simpler structure.

I will simply use an import folder indexed by IMatch and copy the new files with XnView to this folder. Then I can delete them from there or simply move them to the correct folders after renaming them.
This way I will have to track whats new, but at least in XnView I have a preview.
Or maybe I find a way with Syncovery for this copying of newer files.

You don't have to worry about this topic anymore.