- Welcome to photools.com Community.
Recent posts
#21
General Discussion and Questions / Re: f(x) before category in pa...
Last post by axel.hennig - June 29, 2025, 10:17:26 PMIf you mean something like this
then this is a category-formula. Look into "Properties -> Formula" to see how it is calculated.
Or see here: Category Formulas
then this is a category-formula. Look into "Properties -> Formula" to see how it is calculated.
Or see here: Category Formulas
#22
General Discussion and Questions / f(x) before category in panels
Last post by Kucera - June 29, 2025, 09:44:28 PMHello,
my Categories and Filter panels show f(x) between the check-off box and the category name, just for some categories – what does it mean and how did it get there? Could not find anything in the online help or by searching here, sorry.
Regards, Emil
my Categories and Filter panels show f(x) between the check-off box and the category name, just for some categories – what does it mean and how did it get there? Could not find anything in the online help or by searching here, sorry.
Regards, Emil
#23
General Discussion and Questions / Re: Send to viewer from Files ...
Last post by Mario - June 29, 2025, 04:06:46 PMQuote from: fisketjon on June 29, 2025, 03:34:27 PMUnder Relation definitions Canon Versioning is set to Use this version "As Visual". Where can I see the Viewer setting you mention?Click <S> to open the Smart Menu in the Viewer and check the "Visual Proxy" tile.
See Visual Proxy in the Viewer help for details.
#24
General Discussion and Questions / Re: Send to viewer from Files ...
Last post by fisketjon - June 29, 2025, 03:34:27 PMUnder Relation definitions Canon Versioning is set to Use this version "As Visual". Where can I see the Viewer setting you mention?
#25
General / Re: IMatch freezes when readin...
Last post by Mario - June 29, 2025, 02:58:05 PMWe need to learn more to understand this. I've repeated my 50 RAW propagation test on my notebook and in a fresh Windows 11 container. The notebook was faster (newer SSD) and the virtualized Windows machine took almost 2 minutes (write-back via network bridge). But all completed, no hang. 100K file test database with plenty of categories. Tried to make this as similar as possible to your case.
#26
General / Re: IMatch freezes when readin...
Last post by thrinn - June 29, 2025, 02:07:35 PMThank you, Mario, for taking time to analyze this. I will try to track this behaviour down or at least find some influencing factors. Unfortunately I don't have much time today, so it might take some days until I can come back with more information.
Regarding hardware, I will double check, but both Z: (where the IMatch DB and my pictures are stored) and C: are SSDs. C: is a Kingston 1TB SSD, Z: is a Samsung 860 EVO 1 TB. Nothing fancy, but also not a bottleneck so far. I also had a look at the Windows Ressource Monitor during my tests. I could not see any SSD under heavy load.
I will first try to reproduce it with a test database. I don't want to make to many changes back and forth in my main DB.
Regarding hardware, I will double check, but both Z: (where the IMatch DB and my pictures are stored) and C: are SSDs. C: is a Kingston 1TB SSD, Z: is a Samsung 860 EVO 1 TB. Nothing fancy, but also not a bottleneck so far. I also had a look at the Windows Ressource Monitor during my tests. I could not see any SSD under heavy load.
I will first try to reproduce it with a test database. I don't want to make to many changes back and forth in my main DB.
#27
General / Re: Very slow imports and miss...
Last post by dcb - June 29, 2025, 12:51:57 PMThanks for the analysis. It might take me a couple of days to repeat the test to check S:\ usage. From memory, when the problem was first noticed I checked CPU and that wasn't near 100% so I suspect, as you do, it may be the disk.
All I changed was the anti virus path. I did not change the performance profile from what it was originally.
I assume database, images and cache all separate is ideal. I do have two other SSDs. The main system disk and a second drive. The images need to stay where they are on S:\ for space, but I could split the others.
All I changed was the anti virus path. I did not change the performance profile from what it was originally.
I assume database, images and cache all separate is ideal. I do have two other SSDs. The main system disk and a second drive. The images need to stay where they are on S:\ for space, but I could split the others.
#28
General / Re: IMatch freezes when readin...
Last post by Mario - June 29, 2025, 11:42:40 AMI see nothing unusual in the two (?) log files in the post from 28-06-2025, 22:52:53, except fomr ##WAIT logs where a background metadata write thread had to wait < 2 s before it could acquire a transaction. That's an indication for a busy system, but not harmful.
In the log attached to the first post, something really bad happened. IMatch logs
06.28 21:47:26+ 0 [797C] 10 M> < 0 [134000ms #sl] PTMetabase::SetFileValues
Over two (!) minutes for writing the data of one file to the database. This is usually done in 30 to 100 ms.
The same thread has imported metadata from many files before, ExifTool extracts about 400 tags per file in 0.5 to 1.5 s. All pretty normal. This continues until this operation is logged:
06.28 21:45:12+ 0 [797C] 10 M> > 0 PTMetabase::SetFileValues
After this operation is logged, we suddenly see 3 to 4 second "idle" times between each log entry. Only independent background threads log something every few seconds. All threads currently in the process of importing file values or updating the database are silent, stopped dead. Nothing is happening anymore. At this time, several threads running in parallel are trying to import metadata values extracted by ExifTool into the database. Which usually takes only ~20 ms.
Before this happens, I see that the time ExifTool needs to extract metadata from files goes up from ~400 ms to 1500 ms.
Also, the time for PTMetabase::ImportFiles goes up from 20-200 ms to 1500ms, but then falls back to 16 ms again, shortly before the "freeze" happens.
This looks like all disk write operations to the database suddenly stall. Like in a deadlock, where parallel threads lock each other infinitely. The shared resource between the threads is the database. All have imported metadata and now want to write data to the database. Which took ~20ms before (max once 200ms) but now takes 2 minutes?
At
06.28 21:47:26+ 0 [797C] 10 I> ##BLOCK: 31100 (797c) 'Engine.UPDQDelegate.Meta'
a thread trying to write metadata to the database bails out with a timeout trying to get a transaction. The other threads running in parallel doe the same, shifted by a few seconds. The typical transaction duration for setting metadata in the log file before was 20 ms, so no thread should run into any issue. Even if 5 threads come before the 6., they should complete in ~100-200 ms, which causes no issues at all for 6.
Instead all threads wait for the database/disk until they run into hard-coded timeouts and give up. None could get a transaction in the wait time.
The SetFileValues method sets the metadata for one file. It opens a transaction to ensure that either all or none database updates are performed, then processes and writes the metadata and closes the transaction again. The transaction is held by a RAII object that ensures that the transaction is released even if SetFileValues exits unexpectedly or raises an exception. The typical transaction time in the log before the deadlock was ~20ms.
SetFileValues is called every time metadata is imported or you change metadata in other ways (adding keywords, changing a rating, running a metadata template...). It is designed to be super-fast and reliable.
Still, at 06.28 21:45:13 all database writing stops and all background busy importing metadata are stalled until the bail out with a timeout 2 minutes later. All of them. Some threads are unlocked after 533 seconds (!)
This repeats over and over until the log file ends.
There are no write operations to the database which succeed, all end up running into timeouts.
Reading operations (folder scans etc.) work fine. So does writing to the log file. But no write operations to the database file work anymore.
The database and the images are on drive Z:; the log file is on drive C:
The only explanation I currently have (based on experience) for such a sudden problem is a virus checker, blocking write access to the database file. Or maybe Z: crapped out? Or was locked by the AV?
Update
Tried to reproduce this. Dumped 50 ARW files into a new folder in a 100K files test database.
Database, log file and image files are on the same SSD.
12 cores with 24 threads HT, performance profile "Performance".
Used Lr to create JPG versions. Setup a ARW file relation that propagates collections, rating, label, title, headline, description, hierarchical keywords hand all XMP IPTC and IPTC Ext tags.
Refreshed relations. 50 masters and 50 versions detected.
Changed rating, label for the masters. Added headline, description and hierarchical keywords with AutoTagger.
Selected the 50 masters and forced propagation with <F4>,<P>
Total runtime of the propagation was less than a minute. No warnings, no locks, no problems.
In the log attached to the first post, something really bad happened. IMatch logs
06.28 21:47:26+ 0 [797C] 10 M> < 0 [134000ms #sl] PTMetabase::SetFileValues
Over two (!) minutes for writing the data of one file to the database. This is usually done in 30 to 100 ms.
The same thread has imported metadata from many files before, ExifTool extracts about 400 tags per file in 0.5 to 1.5 s. All pretty normal. This continues until this operation is logged:
06.28 21:45:12+ 0 [797C] 10 M> > 0 PTMetabase::SetFileValues
After this operation is logged, we suddenly see 3 to 4 second "idle" times between each log entry. Only independent background threads log something every few seconds. All threads currently in the process of importing file values or updating the database are silent, stopped dead. Nothing is happening anymore. At this time, several threads running in parallel are trying to import metadata values extracted by ExifTool into the database. Which usually takes only ~20 ms.
Before this happens, I see that the time ExifTool needs to extract metadata from files goes up from ~400 ms to 1500 ms.
Also, the time for PTMetabase::ImportFiles goes up from 20-200 ms to 1500ms, but then falls back to 16 ms again, shortly before the "freeze" happens.
This looks like all disk write operations to the database suddenly stall. Like in a deadlock, where parallel threads lock each other infinitely. The shared resource between the threads is the database. All have imported metadata and now want to write data to the database. Which took ~20ms before (max once 200ms) but now takes 2 minutes?
At
06.28 21:47:26+ 0 [797C] 10 I> ##BLOCK: 31100 (797c) 'Engine.UPDQDelegate.Meta'
a thread trying to write metadata to the database bails out with a timeout trying to get a transaction. The other threads running in parallel doe the same, shifted by a few seconds. The typical transaction duration for setting metadata in the log file before was 20 ms, so no thread should run into any issue. Even if 5 threads come before the 6., they should complete in ~100-200 ms, which causes no issues at all for 6.
Instead all threads wait for the database/disk until they run into hard-coded timeouts and give up. None could get a transaction in the wait time.
The SetFileValues method sets the metadata for one file. It opens a transaction to ensure that either all or none database updates are performed, then processes and writes the metadata and closes the transaction again. The transaction is held by a RAII object that ensures that the transaction is released even if SetFileValues exits unexpectedly or raises an exception. The typical transaction time in the log before the deadlock was ~20ms.
SetFileValues is called every time metadata is imported or you change metadata in other ways (adding keywords, changing a rating, running a metadata template...). It is designed to be super-fast and reliable.
Still, at 06.28 21:45:13 all database writing stops and all background busy importing metadata are stalled until the bail out with a timeout 2 minutes later. All of them. Some threads are unlocked after 533 seconds (!)
This repeats over and over until the log file ends.
There are no write operations to the database which succeed, all end up running into timeouts.
Reading operations (folder scans etc.) work fine. So does writing to the log file. But no write operations to the database file work anymore.
The database and the images are on drive Z:; the log file is on drive C:
The only explanation I currently have (based on experience) for such a sudden problem is a virus checker, blocking write access to the database file. Or maybe Z: crapped out? Or was locked by the AV?
Update
Tried to reproduce this. Dumped 50 ARW files into a new folder in a 100K files test database.
Database, log file and image files are on the same SSD.
12 cores with 24 threads HT, performance profile "Performance".
Used Lr to create JPG versions. Setup a ARW file relation that propagates collections, rating, label, title, headline, description, hierarchical keywords hand all XMP IPTC and IPTC Ext tags.
Refreshed relations. 50 masters and 50 versions detected.
Changed rating, label for the masters. Added headline, description and hierarchical keywords with AutoTagger.
Selected the 50 masters and forced propagation with <F4>,<P>
Total runtime of the propagation was less than a minute. No warnings, no locks, no problems.
#29
General / Re: Very slow imports and miss...
Last post by Mario - June 29, 2025, 10:38:39 AMExifTool extracts metadata in between ~0.4 s and 1.6 s per file. Interestingly, JPG are not always faster than RAW files, even if the number of tags is about the same. On the average less than a second per file, which is OK. IMatch processes many files in parallel.
It takes ~ 30 ms to ~ 400ms to import the data into the database.
Write-back takes between 1 and 2 seconds per file, probably depending on current disk utilization.
The total time to write-back 30 files is about 7 seconds (IMatch writes many files in parallel).
This all looks pretty normal to me.
What is puzzling it the rather long time it takes to create cache images for some files.
The file S:\memories\import\_SD Card\IMG_2965.CR3 is loaded in 100 ms, but it takes almost 30 seconds before the cache file creation is completed. The same happens for the file S:\memories\import\_SD Card\IMG_2968.CR3, loaded in 150 ms, but total cache file also takes almost 30 seconds. The 30 seconds are basically the time Windows WIC needs convert the already loaded RAW file into JPG and write it to disk.
The category 'social tagged versions' is reported several times as a slow cat (> 2 seconds update time). For a database with 40K files this should not happen, but IMatch was busy indexing, writing back and re-indexing files, and there is only so much CPU to go around. So this is also OK.
The modified folder is found at
06.29 09:31:23+ 16 [C26C] 10 I> FolderSweeper: Folder [3550] 'S:\memories\import\_SD Card\' modified.
The last time metadata was imported (after the write back) was
06.29 09:33:57+ 0 [E3F8] 10 M> < 0 [375ms] PTMetabase::ImportFileValues
Then a bit of normal IMatch processing, apps running., background processing and then you close IMatch.
The only unexpectedly slow operation is writing the cache images.
Everything happens on the same disk (S:). It contains the database, the images, and also the cache folder. This is the "worst" case for performance of course. Everything competes for disk I/O, so I assume the disk becomes the bottleneck. But a SSD in a one year old system should be able to handle that, I guess?
Did you only change the virus checker or also the performance profile?
When you keep the Windows Task Manager (Performance Tab) open while you repeat this test, is the S: disk shown as 100% utilized and/or are the processor cores 100% utilized.
It takes ~ 30 ms to ~ 400ms to import the data into the database.
Write-back takes between 1 and 2 seconds per file, probably depending on current disk utilization.
The total time to write-back 30 files is about 7 seconds (IMatch writes many files in parallel).
This all looks pretty normal to me.
What is puzzling it the rather long time it takes to create cache images for some files.
The file S:\memories\import\_SD Card\IMG_2965.CR3 is loaded in 100 ms, but it takes almost 30 seconds before the cache file creation is completed. The same happens for the file S:\memories\import\_SD Card\IMG_2968.CR3, loaded in 150 ms, but total cache file also takes almost 30 seconds. The 30 seconds are basically the time Windows WIC needs convert the already loaded RAW file into JPG and write it to disk.
The category 'social tagged versions' is reported several times as a slow cat (> 2 seconds update time). For a database with 40K files this should not happen, but IMatch was busy indexing, writing back and re-indexing files, and there is only so much CPU to go around. So this is also OK.
The modified folder is found at
06.29 09:31:23+ 16 [C26C] 10 I> FolderSweeper: Folder [3550] 'S:\memories\import\_SD Card\' modified.
The last time metadata was imported (after the write back) was
06.29 09:33:57+ 0 [E3F8] 10 M> < 0 [375ms] PTMetabase::ImportFileValues
Then a bit of normal IMatch processing, apps running., background processing and then you close IMatch.
The only unexpectedly slow operation is writing the cache images.
Everything happens on the same disk (S:). It contains the database, the images, and also the cache folder. This is the "worst" case for performance of course. Everything competes for disk I/O, so I assume the disk becomes the bottleneck. But a SSD in a one year old system should be able to handle that, I guess?
Did you only change the virus checker or also the performance profile?
When you keep the Windows Task Manager (Performance Tab) open while you repeat this test, is the S: disk shown as 100% utilized and/or are the processor cores 100% utilized.
#30
General Discussion and Questions / Re: Send to viewer from Files ...
Last post by Mario - June 29, 2025, 10:05:35 AMDo you use versioning with visual proxy files?
Both the Quick View Panel and the Viewer use the unique file id (OID) to retrieve the cache image to display. There was never a reported case where the Quick View Panel and the Viewer showed different images. The only way I can see for this to happen is a visual proxy and the Viewer set to not show the proxy (the QVP always does).
Both the Quick View Panel and the Viewer use the unique file id (OID) to retrieve the cache image to display. There was never a reported case where the Quick View Panel and the Viewer showed different images. The only way I can see for this to happen is a visual proxy and the Viewer set to not show the proxy (the QVP always does).