Performance Testing

Dismayed · January 05, 2021, 04:29:04 AM

I'm testing iMatch in several possible configurations with my NAS (TrueNAS ZFS file system om Intel Xeon 6 core processor and 32 GB RAM):

1. Photos and iMatch database on NAS
2. Photos on NAS iMatch DB on local HDD
3. Both photos and iMatch DB on local drives

I initially ingested a folder of jpgs for scenario 1, which went fine. So I then added scans that are saved as tiffs (about 100). Ingestion seemed fine, but the txt log files in the folder seemed to clog up the works, so I killed it after close to an hour.

Next try was scenario 2, but I went straight for the tiff folder. Same thing - it seems stuck on a txt file, so I killed it after more than an hour.

I still need to test scenario 3, but I'd like to understand why txt files are creating a problem.

Mario · January 05, 2021, 09:11:19 AM

Always store the IMatch database on your fastest drive - this is critical for performance. See Creating the Database in the IMatch Help System for related information. Storing a database on a NAS is the worst-case for performance and may even lead to database corruption if the network and NAS are not 100% reliably and stable under stress.

Text files are usually never a problem. I have many in my test suite and also in my personal database.
Please run IMatch in debug logging mode (see log file) and then ZIP and attach the log file here after you've made your test. The log file will show us exactly what IMatch is doing and how long each operation takes. This allows us to provide better feedback and give you tips.

In 90% of all "IMatch is slow" cases, the installed virus checker is the problem. See IMPORTANT: Virus Checkers for more information.

sinus · January 05, 2021, 11:05:36 AM

I do have also a lot of txt-files and docs and so on ... never a problem.

Dismayed · January 05, 2021, 01:55:10 PM

Quote from: Mario on January 05, 2021, 09:11:19 AM
Always store the IMatch database on your fastest drive - this is critical for performance. See Creating the Database in the IMatch Help System for related information. Storing a database on a NAS is the worst-case for performance and may even lead to database corruption if the network and NAS are not 100% reliably and stable under stress.

Text files are usually never a problem. I have many in my test suite and also in my personal database.
Please run IMatch in debug logging mode (see log file) and then ZIP and attach the log file here after you've made your test. The log file will show us exactly what IMatch is doing and how long each operation takes. This allows us to provide better feedback and give you tips.

In 90% of all "IMatch is slow" cases, the installed virus checker is the problem. See IMPORTANT: Virus Checkers for more information.

I had performance issues both with the database on the NAS and with it on my local drive. The NAS is configured as Raid Z2, so it is fairly fast.

Perhaps my virus program, Norton Security, is the first place to look. It interfered with the install, so I paused virus check and reinstalled. There is no virus program on the NAS, but there is on the machine with iMatch installed. I'll see if I can get the virus check to leave iMatch be.

Dismayed · January 05, 2021, 04:39:51 PM

I'm now running a test with everything local. Ingestion of Nikon RAW files was fast, so now I'm copying over the folder with Tiffs to see if local ingestion works well. If yes, then I'll start adding in various NAS involvement. I may have to use the NAS as an archive/backup.

Mario · January 05, 2021, 04:46:55 PM

See also this article in the IMatch KB: Configuring Process Control for Slow Media (CD-ROM, DVD, NAS, ...)

Carlo Didier · January 05, 2021, 09:49:34 PM

I have a much slower NAS than yours and no problems. But my DB is on a local SSD.

Mario · January 05, 2021, 10:09:50 PM

The DB on the SSD is the proper way to do it.
Indexing files from a NAS is slower than from a local disk, but IMatch has to 'touch' the files on the NAS only once (during ingesting). All other operations then run on the local database on your SSD.
IMatch may need to access the NAS again when an image needs to be displayed (Viewer, Quick View Panel), but once the file is in the cache, IMatch does not need the NAS anymore.

Dismayed · January 06, 2021, 03:06:13 AM

Thank you for the advice. My testing will pick up next week because I need to help my younger son move back to college. I'll miss him, and his technical support for assembling and setting up the NAS (Math/Computer Science major). I will, however, read more about metadata strategies and about iMatch functionality. iMatch seems to be an impressive tool, but I've barely scratched the surface.

Carlo Didier · January 07, 2021, 12:19:22 PM

Quote from: Dismayed on January 06, 2021, 03:06:13 AM... I'll miss him, and his technical support for assembling and setting up the NAS (Math/Computer Science major)...

There's lots of possibilities for him to give you remote support nowadays. For me it's the reverse. I have to remotely support our daughter

Dismayed · January 12, 2021, 05:06:37 PM

Quote from: Carlo Didier on January 07, 2021, 12:19:22 PM
Quote from: Dismayed on January 06, 2021, 03:06:13 AM... I'll miss him, and his technical support for assembling and setting up the NAS (Math/Computer Science major)...
There's lots of possibilities for him to give you remote support nowadays. For me it's the reverse. I have to remotely support our daughter

He's starting a machine learning research project, so perhaps normal 'tech support' will be a nice break for him!