Author Topic: Data in IMD5 file: Complete file written at closing?  (Read 1163 times)

Rene Toepfer

  • Guest
Data in IMD5 file: Complete file written at closing?
« on: March 10, 2021, 07:12:02 AM »
My IMD5 file has a current size of appx 20.3 GiB, containing 355k files. If I revise data, like adding keywords, face recognition etc, will the whole IMD5 file be rewriten while closing or just added new/revised data at the end of the file?

sinus

  • Global Moderator
  • *****
  • Posts: 4707
  • IMatch-User since 2001 (IMatch 3.6)
Re: Data in IMD5 file: Complete file written at closing?
« Reply #1 on: March 10, 2021, 07:32:48 AM »
Wow, 355'000 files. Impressive.
I have only about 300'000.

The answer on your question I do not know.
Best wishes from Switzerland! :-)
Markus

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 31554
Re: Data in IMD5 file: Complete file written at closing?
« Reply #2 on: March 10, 2021, 10:08:10 AM »
The database system used by the IMatch engine for core disk storage is a relational database management system.
It has complex block allocation strategies, free block management, block reuse etc. It does never "rewrite" the entire database file.

New blocks are allocated at the "end" of the database file when needed (in bigger chunks), then used up as new data goes in. Blocks freed due to index updates or data being removed are reused.
This database system is also used by companies like Google, Apple, Microsoft, Adobe and others. For good reasons.

The only operation which rewrites the complete database file is the "Compact & Optimize" process. Which is supposed to.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 31554
Re: Data in IMD5 file: Complete file written at closing?
« Reply #3 on: March 10, 2021, 10:09:08 AM »
Wow, 355'000 files. Impressive.
I have only about 300'000.

My largest test database now has almost 900,000 files.
The largest database I see in telemetry has 2 million files.
The largest database I know of has about 4 - 5 million files (running on a dedicated high-end workstation in a research lab).

Rene Toepfer

  • Guest
Re: Data in IMD5 file: Complete file written at closing?
« Reply #4 on: March 10, 2021, 11:37:32 AM »
The database system used by the IMatch engine for core disk storage is a relational database management system.
It has complex block allocation strategies, free block management, block reuse etc. It does never "rewrite" the entire database file.

New blocks are allocated at the "end" of the database file when needed (in bigger chunks), then used up as new data goes in. Blocks freed due to index updates or data being removed are reused.
This database system is also used by companies like Google, Apple, Microsoft, Adobe and others. For good reasons.

The only operation which rewrites the complete database file is the "Compact & Optimize" process. Which is supposed to.


Thanks for this explanation. To be honest, it is as I have expected it.  :D


Wow, 355'000 files. Impressive.

Thanks, images since 1986.
I am really impressed how fast IMatch remains even with that much files and I can say it's the fastest DAM I have seen in my life.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 31554
Re: Data in IMD5 file: Complete file written at closing?
« Reply #5 on: March 10, 2021, 11:45:19 AM »
I am really impressed how fast IMatch remains even with that much files and I can say it's the fastest DAM I have seen in my life.

Very good. Spread the word! Let others know about IMatch.

Word-of-mouth is the best marketing I can afford.
Adobe spends about 150 million US$ per year on marketing and 'endorsement'. This buys them a lot of exposure everywhere and makes it very hard for small businesses to attract attention.