Author Topic: Possible caching problem when switching databases?  (Read 238 times)

thrinn

  • Hero Member
  • ***
  • Posts: 594
Possible caching problem when switching databases?
« on: October 19, 2019, 01:33:42 PM »
Hi Mario,
recently I stumbled upon a ton of warnings with "error 10020" when running a DB diagnostics. These warnings could be corrected by the diagnostics, a second run showed no warnings. So I thought it was some kind of one-time effect, and I was not able to reproduce this error or warning, so I let it go. But it happens again and again.
Code: [Select]
Checking files:
    Warning: Failed to load file [1] with error 10020
    Warning: Failed to load file [2] with error 10020

While I am not able to reproduce it precisely on will, it looks to me that it happens mainly when switching between databases without exiting IMatch in between.

  • Started IMatch and opened my "Test-03" DB. (This is my main test database).
  • Switched DB using "Database > Open Database" to T-01.imd5
  • Run diagnostics on T-01. See T-01_Run1_im5diag.txt. 173 warnings with error 10020. NOTE that this test db in fact only contains 1 folder with 2 files!
  • Run diagnostics again on T-01. No errors or warnings. Log T-01_Run2_im5diag.txt shows 1 folder and 2 files -> CORRECT!
  • Switched DB using "Database > Open Database" to T-02.imd5
  • Run diagnostics on T-02. This time I get no 10020 errors, but 7 warnings regarding categories referencing non-existing files plus a time line warning. See T-02_Run1_im5diag.txt.
  • Run diagnostics again on T-02. No more errors or warnings.
  • But the Timeline only shows 1 file. The T-02 DB contains 4 files. And 2 files are copies of the other 2.
  • Close and restart IMatch. Load T-02 again. "Rebuild Timeline". Now the timeline shows 4 files (CORRECT).
  • Switch to T-01.imd5 and run diagnostics (see T-01_Run3_im5diag.txt). Again Error 10020, for 2 files this time. Maybe because T-02, the DB that was open before, had 4 files, but T-01 only has 2?
T-01 and T-02 were both freshly created today with IMatch defaults (e.g. IMatch Standard and Workflow categories). Both contain only one folder (not the same one). T-01 contains 2 files, T-02 4 files.
Something strange is going on here...

Thorsten
Win 10 / 64, IMatch 2018, IMA

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Possible caching problem when switching databases?
« Reply #1 on: October 19, 2019, 02:03:12 PM »
Please send me the database which produces the errors for later diagnosis.
Looks like somehow an internal cache table did mot match the actual contents of the file table.

Did you have a crash? Transactions partially aborted because of timeouts in apps or an app running wild?
Database copied without copying auxiliary or journaling files?

It's hard to produce such issues. What did you do to your test database before this happened?

Running the diagnosis once may not be able to fix all problems caused by this, at least not always.
Running the diagnosis several times until no warnings are reported anymore usually helps.

You may also want to include the IMatch log file in debug mode next time. IMatch may log useful data to the log file.
IMatch may just ignore the cache until the diagnosis runs the next time.

Why the timeline does not work, I don't know. I would need a sample database with this problem to learn more.
Having 1,2,3 steps or a sample database with the problem before you did run the diagnosis will be very helpful.
I can then analyze it and maybe send a sample to the database vendor if required.

Note that diagnosis for such obscure issues can take a long time and thus has to wait until I find the time to spend some hours or days with this.

I have created maybe 20 test databases today (create fresh, add folders and filers etc.) while testing a new import/export feature. None of the databases had any errors (my test suite runs a diagnosis always as the first and last step).
« Last Edit: October 19, 2019, 02:12:11 PM by Mario »

thrinn

  • Hero Member
  • ***
  • Posts: 594
Re: Possible caching problem when switching databases?
« Reply #2 on: October 19, 2019, 03:33:37 PM »
Ok. I tried to reproduce it with Debug Logging switched on. I put the log files as well as the test databases I used in a Onedrive folder. Several subfolders represent the results after each test step.Both databases were freshly created today purely for testing purposes. I did nothing special between the separate diagnostics, only switching from Media & Folders to Timeline view to check the timeline file count. After loading a database, I waited until the initial activity (like loading attributes etc.) as shown in the status bar was finished. Then I called the database diagnostics.Here are the steps I performed.
  • Start IMatch. Load T-01. Immediately perform a diagnostics. Shows no errors. Close diagnostics. Exit IMatch. Means: T-01 should be a "clean" DB. Results see Step 1 subfolder.
  • Start IMatch. Load T-02. Immediately perform a diagnostics. Shows no errors. Close diagnostics. Exit IMatch. Means: T-02 should be a "clean" DB. Results see Step 2 subfolder.
  • Start IMatch. Load T-02 (because it has more files).
  • Switch to T-01 from within IMatch. Perform diagnostics. No errors.
  • Switch to T-02 from within IMatch. Perform diagnostics. No errors.
  • Switch to T-01 from within IMatch. Perform diagnostics. No errors.
  • Switch to T-02 from within IMatch. Perform diagnostics. NOW I get at least 2 timeline warnings. Close diagnostics. Exit IMatch. Results see Step 3 subfolder.
  • Start IMatch. Open T-02. Perform diagnostics. No errors.
  • Switch to T-01 from within IMatch. Perform diagnostics. NOW I get 2 10020 errors. Close diagnostics. Exit IMatch. Results see Step 4 subfolder.
You can download both test databases and the corresponding log files from https://1drv.ms/u/s!Au-siC5_q07IjVshl1Ug0OPF1wEn?e=wTvkbh
Thorsten
Win 10 / 64, IMatch 2018, IMA

thrinn

  • Hero Member
  • ***
  • Posts: 594
Re: Possible caching problem when switching databases?
« Reply #3 on: October 19, 2019, 10:32:48 PM »
Quote
Note that diagnosis for such obscure issues can take a long time and thus has to wait until I find the time to spend some hours or days with this.
I fully understand that. I can easily work around this problem by exiting IMatch and starting it again instead of switching databases from a running instance.

Quote
Did you have a crash? Transactions partially aborted because of timeouts in apps or an app running wild?
Database copied without copying auxiliary or journaling files?
No, no, and no. As it happens also with my 2 brand new test databases, it does not seem to depend on a specific database.

Quote
I have created maybe 20 test databases today (create fresh, add folders and filers etc.) while testing a new import/export feature. None of the databases had any errors (my test suite runs a diagnosis always as the first and last step).
But includes your test suite switching databases from inside a running IMatch instance? The problem does not seem to arise when I start IMatch and open a database, only by switching databases without closing IMatch in between.

Question:
Is it somehow possible to run a database diagnostics without automatic repairs? I did not find a possibility yet to check for this kind of problem without running the diagnostics - and as soon as I do that it will be repaired automatically. I tried to trick IMatch by opening the DB in read-only mode, but then it is not possible to run the diagnostics at all.
Thorsten
Win 10 / 64, IMatch 2018, IMA

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Possible caching problem when switching databases?
« Reply #4 on: November 07, 2019, 06:40:13 PM »
I have looked at your databases, thanks.

No errors are reported here for DB 2 taken from Step 3.
But, the diagnosis log shows that the db (Step 3) has only 2 files, while it has actually 4 files...
IMatch reports 4 files when loading the db and the database has four files. But your diagnosis log shows only 2 files?
Maybe some mix up? Or a strange effect?

And the timeline does not show all four files, just two. How did you manage that?

I have created two databases. Added four images to each (two with metadata, two without any metadata).
Run diagnosis. Swapped databases. Run diagnosis. 10 times at least. No error was ever reported by the diagnosis.
Timeline always contained all files. Two files with the correct (EXIF) timestamp and the two others with the "last modified" timestamp pulled from the file system.
Very strange. I must be missing a step or something...



Quote
Is it somehow possible to run a database diagnostics without automatic repairs?

Yes. So to speak.
Just make a copy of the database before the diagnosis.
If the diagnosis finds, fixes and reports a problem, you'll have the original database to start again.


thrinn

  • Hero Member
  • ***
  • Posts: 594
Re: Possible caching problem when switching databases?
« Reply #5 on: November 08, 2019, 07:06:25 AM »
Hi Mario,
thank you for taking a look. This is tricky. I still have not found a way to reproduce this behaviour consistently. Yesterday evening I was able to switch between both test databases multiple times without errors. Then, at once, I got the error again.

Quote
Yes. So to speak.
Just make a copy of the database before the diagnosis.
Stupid me! Should have thought of this myself - even if it is a bit cumbersome...

I will think about a way to automate some of the steps. For example, "snapshotting" the DB file and the log file with timestamps. As long as I have not found an exact sequence of steps to reproduce the problem without repeating switching databases back and forth, it is difficult to document each and every step.
I will report back to this thread if I am able to reproduce this with the same order of steps repeatedly.
Thorsten
Win 10 / 64, IMatch 2018, IMA

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Possible caching problem when switching databases?
« Reply #6 on: November 08, 2019, 02:39:26 PM »
It puzzles me that for one of your test IMatch reports 4 files (and this is indeed what's in DB 2) but the diagnosis only reports 2 files (this is how many files are in DB 1).
And that the timeline did pick up two of four files only, but immediately finds them when I use the "Refresh Timeline" command. I could not reproduce this either.

ubacher

  • Oldtimer
  • ****
  • Posts: 2053
Re: Possible caching problem when switching databases?
« Reply #7 on: November 10, 2019, 06:19:36 AM »
Quote
I stumbled upon a ton of warnings with "error 10020" when running a DB diagnostics. These warnings could be corrected by the diagnostics, a second run showed no warnings.

Yesterday I had problems with the filter function not working (Yes, I had also switched db).
Finally I ran diagnostics and got a large number of 10020 errors. Running the diagnostics a second time fixed this (and my filtering problem).

What is/are 10020 errors?


Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Possible caching problem when switching databases?
« Reply #8 on: November 10, 2019, 08:47:25 AM »
This seems unrelated to the issue discussed in this thread.
Tip: If in doubt, always open a new thread / bug report. Not append to an apparently 'similar' bug report.

The 10020 error code means that IMatch could not find an object (without a diagnosis log and IMatch log file I can only guess) - probably a file?
Referenced by a category, collection,timeline... or another object. The diagnosis log file reports details.

You write your own apps and maybe have to shut-down IMatch hard sometimes or deal with errors caused by your apps...?
I suggest that you run diagnosis more often, especially while you work on your apps, to figure out what in your workflow or your apps is causing this. And then implement changes as required. Or provide a step-by-step how to reproduce this problem.
« Last Edit: November 10, 2019, 09:01:06 AM by Mario »