Problem confirming uncomfirmed faces

[jln]

I have been checking the unconfirmed faces and correcting, confirming, or erasing the face recognition rectangle. On more than one occasion, this has led to iMatch crashing. I am attaching the log file from when I restarted the program.

[Mario]

The log file shows the dreaded (and rare)

CIMSQ: Physical database damage: database disk image is malformed

errors. The error is reported by the database system when IMatch tries to remove data from a database table. This happened while IMatch was adding new files to your database.

Do you get an error message when you open this database?
Did you recently encounter a power failure or similar event?
IMatch crashing does not cause database damage.

Please run a diagnosis immediately and keep your database backups ready in case the database file has become unusable.

[jln]

I do not get an error message when I start iMatch. IMatch seems to operate normally until it crashes.
I have had a couple of power failures in the past week.
I have tried running a diagnosis a couple of times. The diagnosis crashes. I do regularly use iMatch's backup. Unfortunately, that takes long enough that I often leave the computer with it running. I'm afraid that means the last backup iMatch took was likely of a corrupted database.

Fortunately, I have a program that makes a daily cloud backup of my files, so I can use that to recover the backup files from any day. Just to be certain, there are four files in the backup folder with extensions .imd5, .imsema .imd5-shm, and imd5-wal. Is that correct? As I look at the backups, sometimes the .imsema or the imd5-wal files have 0 bytes. Could that be a possible indicator of corruption?

A problem is to try to determine the backup of the last good database. Would it work to just go back a day at a time, run the diagnostics, and consider the database intact if they complete? Would I just replace the existing files with the backup files? I assume I will lose all work done since the replacement backup.

[Mario]

The diagnosis crashes.

That's also very rare. Does IMatch produce a DUMP file when this happens: The Debug Dump File
The diagnosis crashing is really rare (I don't recall a case for many years) and would also most likely indicate severe database damage. The diagnosis log file and the IMatch log file will contain additional details.

afraid that means the last backup iMatch took was likely of a corrupted database.

If IMatch does not shut-down completely, the database system leaves the -shm and -wal files behind. This allows the database system to detect that it was not shut-down completely and to roll-back (undo) all pending changes. The sema file is used by IMatch to lock databases. All of these files (the entire folder containing the database, actually) should be part of your daily backup routine. 

Would it work to just go back a day at a time, run the diagnostics, and consider the database intact if they complete? 

Yes. If the diagnosis runs without failure and the database system does not find any issues with the database, you#re good.
I have IMatch database files in daily operation which were created with IMatch 3, more than a decade ago.

[jln]

Things are still not working for me. I finally found a week-old set of database files that didn't immediately trigger a warning from iMatch that the database was corrupt. Once iMatch started, I ran the database diagnostic tool. It reported that the database was good, although there were a few warnings.

iMatch immediately opened a window indicating that it was downloading metadata. I generally had iMatch write metadata back to the files each session, so I suppose it was getting that metadata. I also saw messages on the bottom indicating that the file structure didn't correspond to that on the device and that I should do a refresh, which didn't surprise me. The window indicated that I could dismiss it, but I left it so as to complete the step as quickly as possible.

After a very long time (hours), iMatch crashed. I repeated this several times refreshing the database files with those from my backup. The same thing always happened.

The log file was massive, too big to attach. So I zipped it and included a couple of more .txt files created by iMatch that were in the same location in case they might be useful

[Mario]

In the BACKUP log from the session before, there are no database warnings. But IMatch has found many folders, e.g. "FolderSweeper: Folder [464] 'G:\Scanned\1984\P..." that have new or updated files and is scheduling them for rescanning.

IMatch 2025 introduced a change where it no longer relies on the unreliable "last modified" folder timestamp but, after opening a database, checks for modified folders and for folders which contain files newer than the folder (this happens when you copy a file into a folder). If IMatch detects this, it schedules the folders for a background rescan. In your case, it found quite a number of folders to process (names are in the BACKUP log file).

The backup log file ends when IMatch starts to close the database. No folders where rescanned yet, just scheduled. 

The normal log file schedules some additional folders, and then starts to processing files.
Many ExifTool logs many warnings about unrecognized maker notes in the Pixel 3 folder.., warnings about preview images, warnings about corrupted maker notes (probably edited with software which does corrupt maker notes), bad FPXR metadata, corrupted Canon maker notes,

many reports from Windows WIC about unrecognized RAW formats (you should switch to LibRaw by enabling "photools.com RAW processing" in the application settings (this is the default for new installations now).

IMatch imports XMP face regions from files in "G:\asheville..." and, suddenly, I see

09.02 20:59:53+  15 [FF78] 00  E> Critical database error: 1  'V:\develop\IMatch5\src\imsq\IMSQLite.cpp(3920)'
09.02 20:59:53+    0 [FF78] 00  E> CIMSQ: Physical database damage: 'database disk image is malformed'

•   'V:\develop\IMatch5\src\imsq\IMSQLite.cpp(231)'

when IMatch tries to insert metadata into the database. From now on, every attempt to write something into the database failes with the critical damage error. This database is gone. After several hundred error messages, the log ends.

From experience, I see this kind of database damage only when there is a power failure or Windows crash just at the wrong moment (database system has written data, Windows acknowledged it as written but fails to actually write it to the disk due to a the power failure or crash). Other reasons are databases stored on external disks with bad connections or on network drives when the network goes down.

The database system used by IMatch is extremely robust and also used by companies like Google, Apple, Microsoft, for that very reason. 

What kind of drive is C:?
What kind of computer do you use?
Which virus checker do you use?

What happens if you create a new database in a new folder and index, say, 5,000 to 10,000 of your files? Any problems?

[jln]

Drive C: is an SSD. The database is stored there. The photos, however, are stored on an external hard drive for additional space and security. If the machine fails, I may be easily able to transfer that drive to another machine. The primary antivirus I use is Malwarebytes. I also have Windows Defender (the version that comes with a subscription to Microsoft Office) that periodically scans my files. Do you have an opinion about those programs?

I will try creating another database. It seems like a lot of lost work.

[Mario]

Don't run two anti-virus systems side-by-side, that's a recipe for trouble.

Windows Explorer rarely gets in the way, but MalwareBytes has been reported as a source of trouble occasionally.
When a virus checker stops IMatch/the database system from writing to the database file, this will definitely damage the database.

Situations like this, where a database suddenly becomes damaged after a successful diagnosis run are really rare. Damaged databases are already rare.

You said you ran a diagnosis successfully. This includes (unless you hold <Ctrl> when starting the diagnosis) a complete check of the database file, where the database system scans for damage. If none is reported, the physical structure of the file on disk is valid. Then you run IMatch, and after writing to the database file a few times, the file on disk is damaged and the database system complains. This could definitely be a virus checker kicking in...

Did you follow the suggestion in the help system and created an "exception/exclusion" for the folder (!) containing the database? Make also make an exception for the "C:\Program Files\photools.com\imatch6\IMatch2025x64.exe" IMatch executable.

Since you apparently have a backup of the database that's OK (no errors reported in the diagnosis!?), try this to see if the database becomes corrupted again after making the exceptions.

Databases on internal disks like C: usually don't become corrupted. Maybe by a power failure, but not out of the blue sky a few minutes after successfully running a database diagnosis.

[jln]

I disabled MalwareBytes. Microsoft Defender Antivirus is now the only antivirus program on my system. I wasn't able to find in the help system about creating " "exception/exclusion" for the folder (!) containing the database? Make also make an exception for the "C:\Program Files\photools.com\imatch6\IMatch2025x64.exe." I did tell Microsoft Defender not to scan either  IMatch2025x64.exe or exiftools.exe. I also granted access to those programs to protected files. The folder with the database was not protected. I then reinstalled the database files from the backup and started iMatch.

The behavior was exactly the same as before. I immediately ran the diagnostics and no errors were found in the database. iMatch then began a very lengthy process of reading metadata. After several hours had passed it crashed. I am again including the log file.

Since the database is OK, could there be a problem processing one of the files? If I knew which one(s) I could try restoring it from backup. Or should I completely reinstall iMatch and exiftool?

[Mario]

Please remember to ZIP your log files. This brings down the 1 MB to 60 KB, which saves space on my server. Thanks.

IMatch was processing / ingesting files.

The last file processed was "G:\asheville, Lauren's wedding  July 2006\jpg\_DSC0080.jpg".

Then I see an "not an error" from the database system (actually, this is a 'generic' error) and then two error message from IMatch about a database problem while trying to store the data of a folder.  Then the log ends.

What happened in that session? Did IMatch crash, reporting information about a DUMP file?
Or did it just close (which would indicate that it was terminated by Windows because of an error in some external component like a WIC codec or a driver).

Thankfully, no error message about physical database damage this time!

Maybe your system is overloaded and becomes unstable after a while under prolonged load? Such issues may not show up under normal load, but can manifest when a software like IMatch utilizes all processor cores for a long time at nearly 100%.

Go to edit > Preferences > Application and set the performance profile to "Balanced" or "Low" and then repeat your test. If this runs though, your system is unstable under heavy prolonged load, causing all kinds of random issues.