Fast import video

[javiavid]

Lately I'm trying to import a lot of video files (500-1000) and the import is very slow.
Is there any trick to speed up the process?

For example, I've changed the preferences to only generate 1 frame per video, turned off face recognition...

Except for the technical data of the file, duration, codec... they don't have any metadata like title, description, author...
Is it possible to deactivate something to ignore it and import them faster?

[Mario]

1. Define slow.
2. What doe 500-1000 mean? Runtime? File size in KB or MB or GB?
3. Where is the video stored? Local SSD or super-slow NAS?
4. Always attach the IMatch log file (log file) so we can see exactly what took how long. Enable the DEBUG mode and force a rescan of one of your videos.
5. IMatch processes videos like other files, including metadata extraction, checksum generation, thumbnail, cache file, visual query data, face recognition etc.
6. If you don't want videos being managed by IMatch, you can disable the file format.

[javiavid]

1.The comparison is between imatch and Premiere or Bridge.

I have looked at the number of files and there are more than 10,000, so I understand that it is normal that it takes so many hours.
I don't know how long it will take because it varies a lot from 27 hours to 4 hours.
I'll do a test when I'm done with a smaller number of files.

3.The files are on a RAID 0 of internal mechanical disks (300 MB/s aprox.)

4.When I do the test with less files I will send it to you.

5.ok

6.It's really what I want

[abgestumpft]

When you check my post here:
https://www.photools.com/community/index.php?topic=11882.0
you see that import time is mainly thumbnail generation + CRC calculation:
I had (on one HDD):
ca 10 sec to generate 10 thumbnails / video
ca. 3 sec to generate 1 thumbnail / video
+
1sec-10sec to calculate CRC (strongly depends on video size/length)

So if I would import 10.000 Videos with my setup with 1 Thumbnail/Video this would mean 10.000 * (3sec (preview thumbnail) + 5sec (average CRC)) = 80.000sec = 22h

When I tested Videos on SSD vs. HDD (with 10 thubmnails/video):
HDD: 10.09 11:16:10+    0 [12B30] 10  I> EUQH: Processed 19 files in 275265ms
SSD: 10.09 11:21:15+    0 [11928] 10  I> EUQH: Processed 19 files in 165062ms

On SSD the Thubmnail generation was similar to HDD.
But CRC calculation was much faster: 200ms - 1sec on SSD vs 1sec-10sec on HDD

So with you RAID0 setup you could be a little faster than my HDD setup.

[Mario]

1.The comparison is between imatch and Premiere or Bridge.

Apples and oranges.
Bridge is a file system browser and does virtually nothing with your video files.
IMatch is a DAM and does a lot with your video files. But all the work is done only once, so I don't see the problem...
Do you really sit in front of your PC while IMatch is ingesting 10,000 video files (which is an awful lot!), waiting for it to finish?

I don't know how long it will take because it varies a lot from 27 hours to 4 hours.

This is normal. The processing time depends on the file size and structure.
Maybe your system is overloaded and has to throttle? Dial down the parallel threads to 2 or 4, see Process Control (Advanced Setting)

Maybe the virus checker is slowing down IMatch. This is very often the case.

3.The files are on a RAID 0 of internal mechanical disks (300 MB/s aprox.)

If you use spinning disks for processing 10,000 video files, definitely dial down the number of threads IMatch uses. Regular disks cannot manage to many parallel requests and the seek times will bring the performance down a lot.

4.When I do the test with less files I will send it to you.

I currently have about 20 "Please test the metadata in these images, it's not what I expect" or "Here is my test database which demonstrates this very complex problem with feature X I'm having".
These emails may keep me busy for a days or even two weeks...

Please don't send more stuff if not really needed. Attach it to this thread instead. It really helps to have everything in one place.

Processing 10,000 video files is something I would let IMatch do over night. Then it's irrelevant how long it takes.
Virus Checker with an exception for IMatch.
Parallel threads maybe down to 4 (my workstation-grade PC has 12 cores).
Then let IMatch and FFMpeg rip.

Processing 10,000 video files is something you probably do only once.
Unless you produce 10,000 videos per month. In that case, a dedicated video DAM with dedicated high-end hardware or SaaS may be the better choice.

[javiavid]

Thank you both for your help.

I usually have 1500 weekly videos, my idea with these tests is to decide if I can use Imatch for video or not.
I'm not an expert in Imatch and I just wanted to ask for suggestions to be able to optimize the process.

[Mario]

Well. 1,500 videos per week is quite extraordinary. And if course depends on the size of the videos.

You did not answer my questions. You did not attach a log file. So, unless you do, all this is just guesswork and a waste of your and my time.