Collections are gone, are lost ...

[sinus]

Hi Mario
As I pointed out in another thread, from time to time pins, flags or dots did and does vanish on my system, but I cannot give more information, I know only, that I cannot trust my collections.

But yesterday I used the newest version 5.1.12 and again flags and dots dispeared ghostly.
Unfortunately I had no log-file.

------------
But just NOW I did something with 112 images.
They had 3 cats and each a "flag on" (karierte Flagge) from my yesterday work.

Now I gave for the 112 images each a red dot.
And 1 of the 112 image I gave a green pin.

Because I have formula-cats they appeared correct (see left side of attachement):

112 images in cat "a Auswahlverfahren"

112 images in cat "a laufend nicht gewählt"
formula "@Collection[Flags|On]" AND ("@Collection[Dots|Red]" OR "@Collection[Dots|Green]")

1 image in cat "c laufend gewählt"
formula "@Collection[Flags|On]" AND "@Collection[Dots|Red]" AND "@Collection[Pins|Green]"

Then I made a screenshot and closed IMatch.
After 1 minute I reopend IMatch and ... the red dots are gone and the green flag is gone, hence the formula-cats, based on flags, dots and pins, are empty.

You can see this on the right side of the attachement.
Left is correct, before I closed IMatch and right is wrong after I reopend IMatch.

I attache also the log-file.

I hope, you can get some informations on my logfile.

(PS: if you wonder the difference from 751 and 753 images in the cat "A NEU importiert START", then these 2 files are the two screenshots).

[attachment deleted by admin]

[Mario]

I have tried to reproduce this problem over the weekend, hoping to find a reason and to fix it for the 5.1.12. But I never managed to lose collections.
I tried your steps with a fresh database with maybe 10,000 files. Setting collections, changing collections, opening and closing IMatch etc. Doing the same while IMatch was importing and writing back files etc. It never lost a collection.

Any specific steps which reliably reproduce this on your system? Your log file shows no errors or warnings at all. If IMatch had problems storing or loading a collection it would log it...

[sinus]

I have tried to reproduce this problem over the weekend, hoping to find a reason and to fix it for the 5.1.12. But I never managed to lose collections.
I tried your steps with a fresh database with maybe 10,000 files. Setting collections, changing collections, opening and closing IMatch etc. Doing the same while IMatch was importing and writing back files etc. It never lost a collection.

Any specific steps which reliably reproduce this on your system? Your log file shows no errors or warnings at all. If IMatch had problems storing or loading a collection it would log it...

Thanks, Mario.
I know, because you can not reproduce it, than you cannot solve such a tricky thing.
I am almost sure, that it will happen again, and I will try to look very good at it, if it occurs again.

Either I do something special stupid things or it is really a tricky, hidden "bug".

Let's look, if we can catch this moody kobold.

[Mario]

Either I do something special stupid things or it is really a tricky, hidden "bug".

I think this is rather what you do ad in which order. In which view (Folder, Categories, ...) are you when you work on collections? Do you switch between views? Do other things at the same time, e.g. assigning cats, moving files etc?

[sinus]

Either I do something special stupid things or it is really a tricky, hidden "bug".

I think this is rather what you do ad in which order. In which view (Folder, Categories, ...) are you when you work on collections? Do you switch between views? Do other things at the same time, e.g. assigning cats, moving files etc?

For example just now, some minutes ago.
I did before a diagnosis and optimize run: no error.

I close IMatch
I open IMatch, are in the Folder-view.
The large window pops up and does "adding and updating Files" (1698 files).
I dismiss the window.

This gives me for half a minute a white screen (normal here).
Then it is again ok.
I change to the categorie view.
I choose 3 files, give them a red dot. I select only one of this files, change the red dot to a green dot ... white screen, chrash, "IMatch will create a Dump-file" or so is written, I see no such a dump-file and close IMatch.

I open IM again ... of course the dots are gone.

Now IM is open and 3/4 of the updating is done, then I will try again to work with IM.

[sinus]

Newest thing, just now again:

I opened IMatch 5.
I work and I am ONLY in the cat-view.
I changed some formula in the cats, all went fine.
Then IMatch close.
IMatch open.

Fine.
In the cat "0 AKTUELLSTE NEU IMPORTIERT" are 92 images (correct). The formula is "flags on and Added today" (you can see this also in the attachement).

All if fine.
I change by clicking to the cat "2 im WORKFLOW", there are 3 images and I DELETE these images (Ctrl Del).
When IM if finnished (takes some seconds), wusch ... all 92 images in the cat "0 AKTUELLSTE NEU IMPORTIERT" are gone, vanished. Instead that there are still 92 image, it is written (0), means 0 images.

I am baffled and view the log and stores it.
Because the logstore-place is also indexed by IM, it takes a moment and suddenly IM shows now 3 images in the cat "0 AKTUELLSTE NEU IMPORTIERT".

So while the newest (logfile) would be correct, the other 2 I do not know, but all 92 images are gone.
I found them in the folder, but the flag (kariert) and the red dots are all gone from all 92 images.

Attached the logfile and the 2 images with the 92 images and then without, I touched for sure not the dots or flags, I changed nothing except I deleted 3 other files.

German:
Mario, ich weiss echt nicht, ob das Sinn macht da weiter zu forschen bei mir.
Zwar haben einige andere auch Probleme mit collection, aber offenbar nur wenige.
Ich kann auf jeden Fall so mit dieser DB unmöglich arbeiten, weil dauernd pins und dots und flags verschwinden. Dabei mache ich echt nichts besonderes.

Da ich mit IM5 nicht arbeiten kann (ich wollte jetzt mal diese 92 Bilder mit IM5 machen), muss ich mit IM3 arbeiten, und das geht natürlich wunderbar.
Deshalb frage ich mich, ob ich einfach diese DB in IM5 shreddern soll und bei nächster Gelegenheit wieder die ganze DB von IM3 konvertieren soll, das hatte ja eigentlich gut geklappt.

Mir ist es auch nicht recht, Dir dauernd solche Probleme zu senden, aber es ist halt nun mal leider so, dass ich vor allem mit den Collections Probleme hatte und habe.

Auf jeden Fall weiss ich, dass ich irgendwann mit IM5 richtig arbeiten kann und werde, und da freue ich mich drauf (hatte auch mal vorsorglich 2 Licenzen gekauft )








[attachment deleted by admin]

[sinus]

Mario,
this here could be a clue to you, but of course I do not know.
I was in the cat-view and jumped a bit here and there.

Then I changed the flag from 1 file and then I saw something curious.

MAIN-CAT "0 NEUE DATEIEN" (0/81)
 - "A HEUTE NEU IMPORTIERT" (0)
 - "B DIESE WOCHE NEU IMPORTIERT" (81)
 - "C NEU IMPORTIERT" (81)
 - "D OHNE DATEIEN IN AUSWAHL" (81)

So far, so good. All these cats are builded on the "flag on" (karierte Flagge), if the flag is on, they are assigned to one of the sub-cats, hence they must be shown also in the Main-Cat.

Now comes the curious thing.
I select on of the file and CHANGE the "flag on" to "flag off" (anstatt karierte Flagge neu rote Flagge).
In all sub-cats the number changes immediately to correct 80, but in the main-cat the (wrong) number 81 stays still there.
I thought, ok, maybe a short delay, clicked on other cats, changed even to the folder-view, back to cats and the wrong number 81 is still there, but only in the main-cat.

So I changed to the main-cat and oh, there is not only the wrong number, but the file with the red flag is also there! And this file should not be there!!

So this file is in a way "ghostly" there.

I changed now the flag, nothing happens.
Now I changed from this "ghostly" file the dot (from no dot to red dot) and voila, ... IMatch chrashed.

When I remember, this happens in the past also, but I was not sure and could not show.
But now, you can see, in my attachement, the clearly wrong file and the wrong number. This file with the red flag should not be there (in the sub-cats it is correctly gone), and hence if I add a dot for a file, what should not be there, for me it is logic, that IMatch chrashes.

I attache the cat-view and I attache also the log-file, backuped after the crash.
I hope, this helps you, would be great!






[attachment deleted by admin]

[Mario]

Do you have a DUMP file from the crash?

The "set dot" operation completed successfully.
Then IMatch loaded metadata for a file from the database.
The variable parser caused an exception (error) which can have dozens of reasons. This error was handled, but usually such errors are just symptoms for something more series. Shortly afterwards the log file ends.

Where do you use variables? In the file window layout? Maybe an App Panel? The ID plate? ...

[sinus]

Unfortunately I have only the logfile, no DUMP.

Yes, I use variables, here:

in the plate:
 Version: {Application.Version} space {Application.Workspace.Name}

in the file-Layout:
header 1 left: {File.Name|substr:0,14}<LineBreak/>{File.Name|substr:14,13}<LineBreak/>{File.Name|substr:27,50}

heade 1 right:
<Run FontSize='10pt'>{File.Modified|substr:0,5}.{File.Modified|substr:8,2}</Run><LineBreak/><Run FontSize='10pt'>{File.Name|substr:6,2}.{File.Name|substr:4,2}.{File.Name|substr:2,2}</Run><LineBreak/>{File.Ext}

I do not use (yet) apps, but the App "Category Dashboard (System)" was open, but this is natively from IMatch.

If you have an idea, what I could check, please let it know me.

THANKS!

[Mario]

The incorrect count in the parent category is caused by a glitch introduced by an optimization I have shipped with 5.1.12.

Before 5.1.12 IMatch always invalidated all categories when collections changed. This is unnecessary, because a change in a collection has no influence on most categories - only on categories which use formulas (and which may use collections in some way).

For 5.1.12 I added a feature which allows IMatch to invalidate only formula-driven categories, but keep the results for other categories in the cache. This improves performance a lot, and makes things run much smoother.

In addition to the categories with the formulas, all their parents up to @All need to be invalidated too of course. And this is where the glitch was (fixed for the upcoming 5.1.14). Under some conditions IMatch failed to invalidate the cache for the parent categories. So, if you change a collection, and you use categories which reference these collections in formulas, the parents of these categories, and their parents, recursively, may show outdated counts and contents.

This is a real bug, but not related to the crash. This is still open.

I also think, after spending hours and hours, identified a subtle problem where a collection update may have caused a rollback of the recent change in the database. This is a pretty rare situation but may have caused some trouble in various parts of IMatch - entirely depending on "where" the user is and how he updates collections.

I doubt that this is responsible for all files suddenly being removed from a collection, unless all the files where added to the collection with one operation, and then the obscure problem described above came into play and undid the change. But that would be reflected in the UI and also logged in the log file, and no "dots missing" bug report ever showed that specific behavior.

I can still not reproduce the vanishing collections problem.

[sinus]

Thanks, Mario, for your deep searching, I am glad, that at least you have one glitch.