- Welcome to photools.com Community.
Recent posts
#71
General / Re: IMatch freezes when readin...
Last post by Mario - June 29, 2025, 11:42:40 AMI see nothing unusual in the two (?) log files in the post from 28-06-2025, 22:52:53, except fomr ##WAIT logs where a background metadata write thread had to wait < 2 s before it could acquire a transaction. That's an indication for a busy system, but not harmful.
In the log attached to the first post, something really bad happened. IMatch logs
06.28 21:47:26+ 0 [797C] 10 M> < 0 [134000ms #sl] PTMetabase::SetFileValues
Over two (!) minutes for writing the data of one file to the database. This is usually done in 30 to 100 ms.
The same thread has imported metadata from many files before, ExifTool extracts about 400 tags per file in 0.5 to 1.5 s. All pretty normal. This continues until this operation is logged:
06.28 21:45:12+ 0 [797C] 10 M> > 0 PTMetabase::SetFileValues
After this operation is logged, we suddenly see 3 to 4 second "idle" times between each log entry. Only independent background threads log something every few seconds. All threads currently in the process of importing file values or updating the database are silent, stopped dead. Nothing is happening anymore. At this time, several threads running in parallel are trying to import metadata values extracted by ExifTool into the database. Which usually takes only ~20 ms.
Before this happens, I see that the time ExifTool needs to extract metadata from files goes up from ~400 ms to 1500 ms.
Also, the time for PTMetabase::ImportFiles goes up from 20-200 ms to 1500ms, but then falls back to 16 ms again, shortly before the "freeze" happens.
This looks like all disk write operations to the database suddenly stall. Like in a deadlock, where parallel threads lock each other infinitely. The shared resource between the threads is the database. All have imported metadata and now want to write data to the database. Which took ~20ms before (max once 200ms) but now takes 2 minutes?
At
06.28 21:47:26+ 0 [797C] 10 I> ##BLOCK: 31100 (797c) 'Engine.UPDQDelegate.Meta'
a thread trying to write metadata to the database bails out with a timeout trying to get a transaction. The other threads running in parallel doe the same, shifted by a few seconds. The typical transaction duration for setting metadata in the log file before was 20 ms, so no thread should run into any issue. Even if 5 threads come before the 6., they should complete in ~100-200 ms, which causes no issues at all for 6.
Instead all threads wait for the database/disk until they run into hard-coded timeouts and give up. None could get a transaction in the wait time.
The SetFileValues method sets the metadata for one file. It opens a transaction to ensure that either all or none database updates are performed, then processes and writes the metadata and closes the transaction again. The transaction is held by a RAII object that ensures that the transaction is released even if SetFileValues exits unexpectedly or raises an exception. The typical transaction time in the log before the deadlock was ~20ms.
SetFileValues is called every time metadata is imported or you change metadata in other ways (adding keywords, changing a rating, running a metadata template...). It is designed to be super-fast and reliable.
Still, at 06.28 21:45:13 all database writing stops and all background busy importing metadata are stalled until the bail out with a timeout 2 minutes later. All of them. Some threads are unlocked after 533 seconds (!)
This repeats over and over until the log file ends.
There are no write operations to the database which succeed, all end up running into timeouts.
Reading operations (folder scans etc.) work fine. So does writing to the log file. But no write operations to the database file work anymore.
The database and the images are on drive Z:; the log file is on drive C:
The only explanation I currently have (based on experience) for such a sudden problem is a virus checker, blocking write access to the database file. Or maybe Z: crapped out? Or was locked by the AV?
Update
Tried to reproduce this. Dumped 50 ARW files into a new folder in a 100K files test database.
Database, log file and image files are on the same SSD.
12 cores with 24 threads HT, performance profile "Performance".
Used Lr to create JPG versions. Setup a ARW file relation that propagates collections, rating, label, title, headline, description, hierarchical keywords hand all XMP IPTC and IPTC Ext tags.
Refreshed relations. 50 masters and 50 versions detected.
Changed rating, label for the masters. Added headline, description and hierarchical keywords with AutoTagger.
Selected the 50 masters and forced propagation with <F4>,<P>
Total runtime of the propagation was less than a minute. No warnings, no locks, no problems.
In the log attached to the first post, something really bad happened. IMatch logs
06.28 21:47:26+ 0 [797C] 10 M> < 0 [134000ms #sl] PTMetabase::SetFileValues
Over two (!) minutes for writing the data of one file to the database. This is usually done in 30 to 100 ms.
The same thread has imported metadata from many files before, ExifTool extracts about 400 tags per file in 0.5 to 1.5 s. All pretty normal. This continues until this operation is logged:
06.28 21:45:12+ 0 [797C] 10 M> > 0 PTMetabase::SetFileValues
After this operation is logged, we suddenly see 3 to 4 second "idle" times between each log entry. Only independent background threads log something every few seconds. All threads currently in the process of importing file values or updating the database are silent, stopped dead. Nothing is happening anymore. At this time, several threads running in parallel are trying to import metadata values extracted by ExifTool into the database. Which usually takes only ~20 ms.
Before this happens, I see that the time ExifTool needs to extract metadata from files goes up from ~400 ms to 1500 ms.
Also, the time for PTMetabase::ImportFiles goes up from 20-200 ms to 1500ms, but then falls back to 16 ms again, shortly before the "freeze" happens.
This looks like all disk write operations to the database suddenly stall. Like in a deadlock, where parallel threads lock each other infinitely. The shared resource between the threads is the database. All have imported metadata and now want to write data to the database. Which took ~20ms before (max once 200ms) but now takes 2 minutes?
At
06.28 21:47:26+ 0 [797C] 10 I> ##BLOCK: 31100 (797c) 'Engine.UPDQDelegate.Meta'
a thread trying to write metadata to the database bails out with a timeout trying to get a transaction. The other threads running in parallel doe the same, shifted by a few seconds. The typical transaction duration for setting metadata in the log file before was 20 ms, so no thread should run into any issue. Even if 5 threads come before the 6., they should complete in ~100-200 ms, which causes no issues at all for 6.
Instead all threads wait for the database/disk until they run into hard-coded timeouts and give up. None could get a transaction in the wait time.
The SetFileValues method sets the metadata for one file. It opens a transaction to ensure that either all or none database updates are performed, then processes and writes the metadata and closes the transaction again. The transaction is held by a RAII object that ensures that the transaction is released even if SetFileValues exits unexpectedly or raises an exception. The typical transaction time in the log before the deadlock was ~20ms.
SetFileValues is called every time metadata is imported or you change metadata in other ways (adding keywords, changing a rating, running a metadata template...). It is designed to be super-fast and reliable.
Still, at 06.28 21:45:13 all database writing stops and all background busy importing metadata are stalled until the bail out with a timeout 2 minutes later. All of them. Some threads are unlocked after 533 seconds (!)
This repeats over and over until the log file ends.
There are no write operations to the database which succeed, all end up running into timeouts.
Reading operations (folder scans etc.) work fine. So does writing to the log file. But no write operations to the database file work anymore.
The database and the images are on drive Z:; the log file is on drive C:
The only explanation I currently have (based on experience) for such a sudden problem is a virus checker, blocking write access to the database file. Or maybe Z: crapped out? Or was locked by the AV?
Update
Tried to reproduce this. Dumped 50 ARW files into a new folder in a 100K files test database.
Database, log file and image files are on the same SSD.
12 cores with 24 threads HT, performance profile "Performance".
Used Lr to create JPG versions. Setup a ARW file relation that propagates collections, rating, label, title, headline, description, hierarchical keywords hand all XMP IPTC and IPTC Ext tags.
Refreshed relations. 50 masters and 50 versions detected.
Changed rating, label for the masters. Added headline, description and hierarchical keywords with AutoTagger.
Selected the 50 masters and forced propagation with <F4>,<P>
Total runtime of the propagation was less than a minute. No warnings, no locks, no problems.
#72
General Discussion and Questions / Re: Send to viewer from Files ...
Last post by Mario - June 29, 2025, 10:05:35 AMDo you use versioning with visual proxy files?
Both the Quick View Panel and the Viewer use the unique file id (OID) to retrieve the cache image to display. There was never a reported case where the Quick View Panel and the Viewer showed different images. The only way I can see for this to happen is a visual proxy and the Viewer set to not show the proxy (the QVP always does).
Both the Quick View Panel and the Viewer use the unique file id (OID) to retrieve the cache image to display. There was never a reported case where the Quick View Panel and the Viewer showed different images. The only way I can see for this to happen is a visual proxy and the Viewer set to not show the proxy (the QVP always does).
#73
General Discussion and Questions / Send to viewer from Files with...
Last post by fisketjon - June 29, 2025, 01:28:47 AMQuickview showed the correct photo, but the viewer displayed a completely different photo. After I confirmed the faces in the photo shown in the viewer the photo that was displayed in the Unconfirmed Faces disappeared as if I had confirmed faces in that photo.
#74
General / Re: IMatch freezes when readin...
Last post by dcb - June 29, 2025, 12:51:26 AMHi, David from the other thread about slow import. For what it's worth, I've also experienced slowdown when doing forced propogations using F4,P. I'll get some testing done today and a log through to Mario.
#75
General / Re: IMatch freezes when readin...
Last post by thrinn - June 28, 2025, 10:52:53 PMAnother test: I switched back to the standard Performance Profile. Selected just 16 ARW files. Hid all panels with F9,Backspace. Pressed F4,P to propagate MD. Immediately, IMatch got stuck "Propagating".
Log file shows a lot of "Failed to get transaction or CS lock" messages:
After waiting about 15min, IMatch finished propagating. But 15min to propagate MD for 16 files from ARW to JPG?
At least, this time I did not get any ##BLOCK messages.
IMATCH6_LOG.zip
Log file shows a lot of "Failed to get transaction or CS lock" messages:
Code Select
06.28 22:37:32+ 63 [6464] 02 I> CIMCollection::InnerCalculate: Failed to get transaction or CS lock for 'Callout'After waiting about 15min, IMatch finished propagating. But 15min to propagate MD for 16 files from ARW to JPG?
At least, this time I did not get any ##BLOCK messages.
IMATCH6_LOG.zip
#76
General / IMatch freezes when reading me...
Last post by thrinn - June 28, 2025, 10:30:23 PMAfter installing the newest version, I suffer from frequent freezes where the UI is completely blocked. This happens in the "Reading Metadata" phase after a write-back, when IMatch re-reads some of the changed files. But I see it also when I try to explicitly propagate metadata (F4, P).
In the log file I see many ##BLOCK messages like below, often in combination with a "TranBegin wait loop" with a very long time (>130.000 ms):
I already dialled down the Performance Profile to "Low", just in case. I use Windows 11 with Defender, IMatch process and executable declared as exception, whole folder containing the database likewise, even the folder containing the photos. We are talking about ARW and JPG, no large TIFFs or videos or such.
I suspect that somehow the file relation rules interfere with the "Reading Metadata" step: Specifically, I shoot ARW+JPG, and I have a Buddy rule as well as a Version rule to link the JPG as buddy and version to the ARW master. Some Metadata is propagated, but this has never been a problem before. Included is some XMP metadata, some collections (Pins, Dots), and some categories.
The attached log file (incomplete, as I had to close IMatch the hard way) is from a session where I tried to propagate metadata for about 350 ARW files.
Is there any possibility to find out what exactly is blocking by what? At the moment, I have no idea where to look.
I saw one other bug report from today concerning slow import, but in my case, the initial metadata load when ingesting new files did not have any issues. Propagation and Write-back, however, caused the blocking.
IMATCH6_LOG.zip
In the log file I see many ##BLOCK messages like below, often in combination with a "TranBegin wait loop" with a very long time (>130.000 ms):
Code Select
06.28 22:05:15+ 1234 [71B4] 50 I> TranBegin wait loop: 600 134453ms.
06.28 22:05:15+ 0 [71B4] 10 I> ##BLOCK: 29108 (71b4) 'Engine.UPDQDelegate.Meta' TransBegin 0 ms from bool __cdecl CIMRelationManager::Propagate(const class std::vector<unsigned int,class std::allocator<unsigned int> > &,bool,class CBitVector *,const class CBitVector *,bool) (3647) lev: 0I already dialled down the Performance Profile to "Low", just in case. I use Windows 11 with Defender, IMatch process and executable declared as exception, whole folder containing the database likewise, even the folder containing the photos. We are talking about ARW and JPG, no large TIFFs or videos or such.
I suspect that somehow the file relation rules interfere with the "Reading Metadata" step: Specifically, I shoot ARW+JPG, and I have a Buddy rule as well as a Version rule to link the JPG as buddy and version to the ARW master. Some Metadata is propagated, but this has never been a problem before. Included is some XMP metadata, some collections (Pins, Dots), and some categories.
The attached log file (incomplete, as I had to close IMatch the hard way) is from a session where I tried to propagate metadata for about 350 ARW files.
Is there any possibility to find out what exactly is blocking by what? At the moment, I have no idea where to look.
I saw one other bug report from today concerning slow import, but in my case, the initial metadata load when ingesting new files did not have any issues. Propagation and Write-back, however, caused the blocking.
IMATCH6_LOG.zip
#77
General Discussion and Questions / Re: IMatch 2025.4.4 Released -...
Last post by Mario - June 28, 2025, 10:01:54 AMIMatch upgrades don't alter settings, unless I mean to. And in that case I document it in the release notes.
Upgrading the database system has nothing to do with settings.
1. No change related to orientation. Please provide a minimum of information, like file format,
where is the orientation wrong? File Window? Viewer? If you use RAW, WIC or photools.com RAW processing?
2. That's a File Window setting (hierarchical mode) I guess, which is stored per File Window. No change with that. I assume this is what you have "configured". It helps to provide more details, like what you actually have configured, which option or settings you have used etc. I can only guess otherwise.
3. "Not displayed at all". IMatch does not remove sub-folders from your database once added. It will by default not add new sub-folders found under folders in the database, but notify you to perform a rescan of the parent folder.
Adding folders to a database does not touch other folders. A folder is only marked for rescan when the last modified on disk timestamp is newer in the database, or when IMatch detects a file newer than the folder during the initial folder scan. May this be the case, you or some other application changing the folder timestamps or adding files without you knowing?
The log file from that session would contain more information, e.g. what you did, what IMatch found in the file system, if and which folders were considered as outdated.
The update is out for almost a week, and no reports from other users, no emails about similar problems. I use the same update on my 3 PCs without any issues.
Upgrading the database system has nothing to do with settings.
1. No change related to orientation. Please provide a minimum of information, like file format,
where is the orientation wrong? File Window? Viewer? If you use RAW, WIC or photools.com RAW processing?
2. That's a File Window setting (hierarchical mode) I guess, which is stored per File Window. No change with that. I assume this is what you have "configured". It helps to provide more details, like what you actually have configured, which option or settings you have used etc. I can only guess otherwise.
3. "Not displayed at all". IMatch does not remove sub-folders from your database once added. It will by default not add new sub-folders found under folders in the database, but notify you to perform a rescan of the parent folder.
Adding folders to a database does not touch other folders. A folder is only marked for rescan when the last modified on disk timestamp is newer in the database, or when IMatch detects a file newer than the folder during the initial folder scan. May this be the case, you or some other application changing the folder timestamps or adding files without you knowing?
The log file from that session would contain more information, e.g. what you did, what IMatch found in the file system, if and which folders were considered as outdated.
The update is out for almost a week, and no reports from other users, no emails about similar problems. I use the same update on my 3 PCs without any issues.
#78
General Discussion and Questions / Re: Propagating metadata to ve...
Last post by Mario - June 28, 2025, 09:56:22 AMWhat data do you propagate? I'm sure IMatch is not removing the ICC profile intentionally, but metadata is complex and propagating to much or to less can be an issue. Also, it might be a side effect of copying the metadata and the way ExifTool handles this. 100 different reasons. More information and details needed.
Made a quick test, using a DNG and a PSD with embedded Pro Photo ICC profile. Saved both images to a JPG. Made the JPG versions of the corresponding master. Used a standard propagation with rating/label/collections/All XMP IPTC and IPTC Core data (safe for propagation).
Wrote back the masters. Metadata correctly propagated, ICC profiles unchanged still Pro Photo in both masters and versions. Do not, for example, propagate EXIF metadata between files. Always dangerous.
Made a quick test, using a DNG and a PSD with embedded Pro Photo ICC profile. Saved both images to a JPG. Made the JPG versions of the corresponding master. Used a standard propagation with rating/label/collections/All XMP IPTC and IPTC Core data (safe for propagation).
Wrote back the masters. Metadata correctly propagated, ICC profiles unchanged still Pro Photo in both masters and versions. Do not, for example, propagate EXIF metadata between files. Always dangerous.
#79
General Discussion and Questions / IMatch 2025.4.4 Released - som...
Last post by Stenis - June 27, 2025, 10:26:29 PMThis version is supposed to make use of the latest version of the database you use in iMatch. I want to ask if that has been the resons to why some of my settings and a few other things have seemed to be altered by this version upgrade?
1. First, I noticed that some of my portrait pictures that I have spent some time correcting before now is in landscape again. Pretty disturbing - why is that happening?
2. Since I always like to see both my RAW and the JPEG-files that I always keep in one or several subdirectories I had that configured too but that was changed too so only the RAW was displayed. I never change that setting by myself.
3. Some of my newly applied subdirectories was not displayed at all. So, I had to remove and append those folders again (three in total) to make that happen and when I did the system started to mark ALL my folders for refresh/update and it took two hours before the close to 20 000 files where processed and back in usable condition again.
Is this a behavior to expect even in the future or is it some way to avoid that? I mean I often use to remove folders and append new ones without that is happening normally.
1. First, I noticed that some of my portrait pictures that I have spent some time correcting before now is in landscape again. Pretty disturbing - why is that happening?
2. Since I always like to see both my RAW and the JPEG-files that I always keep in one or several subdirectories I had that configured too but that was changed too so only the RAW was displayed. I never change that setting by myself.
3. Some of my newly applied subdirectories was not displayed at all. So, I had to remove and append those folders again (three in total) to make that happen and when I did the system started to mark ALL my folders for refresh/update and it took two hours before the close to 20 000 files where processed and back in usable condition again.
Is this a behavior to expect even in the future or is it some way to avoid that? I mean I often use to remove folders and append new ones without that is happening normally.
#80
General Discussion and Questions / Propagating metadata to versio...
Last post by Pawel - June 27, 2025, 09:36:53 PMSome of my JPG files have ProPhoto color profile embedded. When I want to replace JPG in the database with a new version I use workflow described in this post.
I noticed that as soon as I propagate metadata from the old files (the masters) to the new files (the versions) the colorspace information is removed from the new files - when I open them in Photoshop I get a message there is no color space/no color profile embedded. As soon as I assign the files ProPhoto colorspace and save them they are displayed correctly again.
Any idea what is wrong?
I noticed that as soon as I propagate metadata from the old files (the masters) to the new files (the versions) the colorspace information is removed from the new files - when I open them in Photoshop I get a message there is no color space/no color profile embedded. As soon as I assign the files ProPhoto colorspace and save them they are displayed correctly again.
Any idea what is wrong?