Author Topic: [BBD] First file is missing from target folder after Renamer  (Read 496 times)

bekesizl

  • Jr. Member
  • *
  • Posts: 84
[BBD] First file is missing from target folder after Renamer
« on: March 14, 2021, 12:06:11 PM »
I am having a strange behaviour of the Renamer function.
My rule is displayed in the attached images.

The new files are copied to the folder: e:\Fotok_I_M\Canon_M50\Import\
After copying I rescan this folder, select all files.
The folder e:\Fotok_I_M\ is indexed by IMatch.
The files are CR3+JPG pairs. There are buddy and versioning rules defining JPG as a buddy file to the CR3.

The rule moves the files to a dated folder, e.g. e:\Fotok_I_M\Canon_M50\2021\2021_03_13

The strange thing is and it is so since a relative long time (I was a new user of IMatch 2020), that the first RAW file doesn't appear in the destionation folder in IMatch.
If I rescan the folder, the file gets imported.

In the rule I didn't set "add created folders to the database", as the whole root folder is already part of the database. The new folder also appears in the folder tree and all but the first raw files are displayed.

I can live with this behaviour, but maybe it is a small bug.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #1 on: March 14, 2021, 12:42:05 PM »
What does "the first RAW doss not appear" mean, exactly?
Do you process multiple RAW files (how many) with the Renamer and only the first one is not added to the database? And always the first one processed by the Renamer?

Please keep in mind that there are many places where things can go wrong - there is a complexity limit for IMatch and moving files also causes Windows to send "folder change", "File deleted", "File Created" or "File Renamed" "File Moves" events to IMatch, on which IMatch has to react and which may interfere. Especially since you not just Rename files but also move files between existing and indexed database folders, in combination with buddy files processing and renaming files at the same time and moving them into folders with dynamically created names based on variables. Very complicated. Maybe two Renamer presets which do the same in two steps would be safer.

I would at least need info on how many files and which files you process, the IMatch log file in debug mode from a session where this problem happened, your file version and buddy setup etc. to process this further.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #2 on: March 14, 2021, 02:29:36 PM »
It is probably more correct to say "the first file of the selection" is not getting displayed in the destination folder, which is in my case a RAW file.
It is totally reproducible on my computer. It happens every time I import new files.
I usually import files between 5-300. The error is not dependent on this count.
If I import files from two dates at once, then the first file of both dates is "missing".

It looks like the first element in a loop is not processed.

Here are my file relation rules. It was also the case with the older version of my rules.
https://www.photools.com/community/index.php?topic=10996.msg78383#msg78383

I attach a debug log.
I also attach 3 screenshots.
1) Shows the imported files in the "Import" folder (downloaded by Breeze Downloader Pro, CR3 read-only, JPG normal)
2) Shows the destination folder after the renamer was executed
3) Shows the destination folder, after a simple "Rescan now" was executed on the folder.
No file transfer on these files was executed in Windows. Mylio is not running in the background.

If I filter the log for warnings, I get this:
   Line 507: 03.14 13:54:02+    0 [2BDC] 01  W>      Spelling: Cannot find a dictionary for language 'en' or failed to load.  'V:\develop\IMatch5\src\IMatchNG\IMatch.cpp(3782)'
   Line 1698: 03.14 13:57:37+    0 [2BDC] 01  W>                     Renamer: File 'E:\Fotok_I_M\Canon_M50\Import\IMG_9933.JPG' (Dest: '', Copy: '') failed with code 200  'V:\develop\IMatch5\src\IMatchNG\Renamer.cpp(1714)'
   Line 1699: 03.14 13:57:37+    0 [2BDC] 01  W>                     Renamer: File 'E:\Fotok_I_M\Canon_M50\Import\IMG_9934.JPG' (Dest: '', Copy: '') failed with code 200  'V:\develop\IMatch5\src\IMatchNG\Renamer.cpp(1714)'
   Line 1700: 03.14 13:57:37+    0 [2BDC] 01  W>                     Renamer: File 'E:\Fotok_I_M\Canon_M50\Import\IMG_9935.JPG' (Dest: '', Copy: '') failed with code 200  'V:\develop\IMatch5\src\IMatchNG\Renamer.cpp(1714)'
   Line 2725: 03.14 14:10:51+    0 [2BDC] 01  W>                     Renamer: File 'E:\Fotok_I_M\Canon_M50\Import\IMG_9936.JPG' (Dest: '', Copy: '') failed with code 200  'V:\develop\IMatch5\src\IMatchNG\Renamer.cpp(1714)'
   Line 2726: 03.14 14:10:51+   15 [2BDC] 01  W>                     Renamer: File 'E:\Fotok_I_M\Canon_M50\Import\IMG_9937.JPG' (Dest: '', Copy: '') failed with code 200  'V:\develop\IMatch5\src\IMatchNG\Renamer.cpp(1714)'


These are the JPG buddy files of my CR3. They can't be moved, because they already got transfered with the Master file (I assume).

I will post 2 more screenshots, where I import 2 more RAW+JPG pairs.
In this case the first file is displayed without needing a manual rescan.

If you mean I should split this rule into two separate, I can accept it.
But it works nearly perfect in the current condition and from my experience, it could be some minor bug in the program.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #3 on: March 14, 2021, 02:31:40 PM »
The two more screenshots + file relation rules regarding RAW files.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #4 on: March 14, 2021, 02:46:13 PM »
Quote
If I import files from two dates at once, then the first file of both dates is "missing".

With import do you mean the process when IMatch is scanning/indexing a new or rescanning a folder already in the database?
This process does not use the Renamer.

Quote
These are the JPG buddy files of my CR3. They can't be moved, because they already got transfered with the Master file (I assume).

How can this happen? I though you were only trying to rename the CR3 files. Or do you mix CR3 and JPEG files in one Renamer step, despite the JPG files being buddy files and hence are automatically renamed?

Quote
It is probably more correct to say "the first file of the selection"

Do you mean that the first file of the selection you used the Renamer on is not added to the database? Or does it just not show in the File Window?
In the log file you have attached, which CR3 file is not added to the database?

I see also write-backs during import. Do you have immediate write-back enabled (Edit > Preferences > Background Processing). This is off by default.
Do you use versioning with metadata propagation?


bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #5 on: March 14, 2021, 03:13:29 PM »

Quote
If I import files from two dates at once, then the first file of both dates is "missing".
Quote
With import do you mean the process when IMatch is scanning/indexing a new or rescanning a folder already in the database?
This process does not use the Renamer.
Yes, I man automatic or manual rescanning of the "Import" folder.
I know that this doesn't use the Renamer, but if the "Import" folder contains files from two dates, then the first file of both dates will be "missing".

Quote
Quote
These are the JPG buddy files of my CR3. They can't be moved, because they already got transfered with the Master file (I assume).
How can this happen? I though you were only trying to rename the CR3 files. Or do you mix CR3 and JPEG files in one Renamer step, despite the JPG files being buddy files and hence are automatically renamed?
The selection includes both cr3 and jpg (out of laziness). As the version stack is not collapsed automatically at rescan, I sometimes forget to collapse it before renaming.

Quote
Quote
It is probably more correct to say "the first file of the selection"
Do you mean that the first file of the selection you used the Renamer on is not added to the database? Or does it just not show in the File Window?
In the log file you have attached, which CR3 file is not added to the database?
Yes, the first file (in my case IMG_9933.CR3) is not listed in the destionation folder in the File Window and the file count is -1 to what it should be. The file is definitely in the folder, if you check it with Explorer.
If I execute a rescan manually (a simple one from the context menu), this file appears and shows the signs of being analyzed by IMatch like not displaying the thumbnail, only a blurred/grayed out one.

Quote
I see also write-backs during import. Do you have immediate write-back enabled (Edit > Preferences > Background Processing). This is off by default.
Do you use versioning with metadata propagation?
Immediate write back is not enabled. I only write-back, when I want (or it is intended this way, see attached setting).
I use the CR3 files as read-only, but I assumed it was not dependent for the renamer.
If it is in connection with this (unsupported) setting, than I will find another way for this problem (or leave it like this).

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #6 on: March 14, 2021, 03:24:22 PM »
Additional information:
IMatch is wanting to write-back the preserved filename to the other files, but the one "missing" has nothing to write-back.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #7 on: March 15, 2021, 03:11:26 PM »
Additional information:
It happens also, when I toggle the Version stack with "T" in the Folder view in the "Import" folder.

sinus

  • Global Moderator
  • *****
  • Posts: 4298
  • IMatch-User since 2001 (IMatch 3.6)
Re: First file is missing from target folder after Renamer
« Reply #8 on: March 15, 2021, 06:07:14 PM »
Additional information:
IMatch is wanting to write-back the preserved filename to the other files, but the one "missing" has nothing to write-back.

Sorry, I cannot follow (a problem of my English, I guess).

I see only, that 3 of your thumbs are read only.
Can this be a source of the problem or has this nothing to say?
Best wishes from Switzerland! :-)
Markus

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #9 on: March 15, 2021, 10:21:12 PM »
The 3 read-only files are the 3 CR3 files I don't want to change.
I can't tell for sure if it is in connection with this fact, but the files get moved to the folder (all of them), but the first file in the list is not in the database after it was moved.
The other (also read-only) files are moved without a problem.

English isn't my native language, maybe I write my thoughts a bit too complicated. I don't use English very much, I am much better in German, but most of this forum is in English, so I try to formulate my posts in English.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #10 on: March 16, 2021, 09:01:49 AM »
This is all way to complicated for me.
You mix renaming files with moving files with versioning and buddy files. You throw in some files explicitly marked as read-only in the file system for good measure.
Maybe you are just trying to much. IMatch cannot plan for or support any possible combination of any possible situation with every possible combination of options.

If you can produce

- a folder layout I can re-create here
- file names you use in these folders) so I can re-create them here
- Renamer steps I can re-create
- File Relations you use I can re-create
- other instructions needed

which allow me to reproduce this behavior here, I will look into it.

I cannot spend a day trying to reproduce this obscure situation and then getting nothing. This is way to complex and probably no other user will ever run into this. Otherwise it would have been reported already.
And this may even depend on timing issues or some other computer-dependent variable nobody else can reproduce...

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #11 on: March 16, 2021, 02:50:35 PM »
OK. I made one more try and I created a test package for Mario.

I uploaded it to my Onedrive. Link: https://1drv.ms/u/s!AnQSer5eDlmQuLgq5SM0RjpJaKI9VA?e=QUvIWN

There are 3 CR3+JPG pairs, unrenamed and 3 already renamed, in a folder structure. There is also my empty test datase (just a copy, no pack&go).
I created a new database, checked only CR3 as format.
The File relation rules and Renamer rules are included as screenshots.
File relation rules are the default ones created by IMatch for CR3. I only enabled the Versioning rules.

There are the 6 files in the Import folder.
If you collapse the version stacks, select the 3 masters and execute the renamer, then 5 files will be displayed in the dated folder.
It doesn't matter, if the original files are read-only or not (I checked it on my system).

So it seems, that it is definitely a bug.
Or don't allow a combinded rename+move at the creation of Renamer rules.

Database used this root folder:
e:\IMatchTestDB\FotoLib
e:\IMatchTestDB\FotoLib\CanonM50\Import <- folder with the not renamed files
e:\IMatchTestDB\FotoLib\CanonM50\2021\2021_03_14 <- already renamed files
e:\IMatchTestDB\FotoLib\CanonM50\2021\2021_03_16 <- folder created by the Renamer

PS:
I tried a splitted rename+move approach (not included in the uploaded DB, but simply copying the original rule and deleting the other part).
The renaming part was OK.
The move part seems to have a problem, see screenshot in this attachment. The resulting name doens't contain the filename. Maybe the problem lies there.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #12 on: March 16, 2021, 03:24:21 PM »
You are just using a MoveFile, but nothing that produces or feeds in the original file name. Hence all the names you produce are the same.
Add an original file name step in front.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #13 on: March 16, 2021, 03:48:08 PM »
I corrected the "move only" Renamer with the original file name.
The result is the same. Destionation folder only displays 5 files (3+3 were moved). First CR3 is missing.

Repeated the test with non read-only CR3 files. Same result.

Repeated the test with non read-only CR3 files without renaming them. Same result.

Definitely a bug in the file moving part. At least as far as a folder structure like mine is used.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #14 on: March 30, 2021, 05:39:18 PM »
Is there any news on resolving this bug in IM2021?

Or will I have to change my folder structure to handle this better?

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #15 on: March 30, 2021, 05:52:53 PM »
I have not yet looked into this. This naming / folder schema is very complex and apparently no other user has ever encountered a similar issue.
The bug ticket is open and I will look into this at some point in time.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #16 on: April 02, 2021, 07:23:35 PM »
I made another try. Move files only. (Renamed before, written back all data).
This time target folder is starting with an absolute part. Same result.
Source: e:\Fotok_I_M\Canon_M50_Import
Destination: e:\Fotok_I_M\Canon_M50\Year\Day

thrinn

  • Hero Member
  • ***
  • Posts: 921
Re: First file is missing from target folder after Renamer
« Reply #17 on: April 02, 2021, 08:00:59 PM »
Maybe I am overlooking something, but you write that your source file is in the Canon_M50_Import folder, isn't it?
According to the attached screenshot, on the other hand, the source file is already in the Canon M50 folder, that is, the destination folder. For me it looks as if you called the renamer accidentally on the target file, not the source file. This would move the file onto itself - which can not work.
Thorsten
Win 10 / 64, IMatch 2018, IMA

sinus

  • Global Moderator
  • *****
  • Posts: 4298
  • IMatch-User since 2001 (IMatch 3.6)
Re: First file is missing from target folder after Renamer
« Reply #18 on: April 03, 2021, 09:25:36 AM »
Maybe I am overlooking something, but you write that your source file is in the Canon_M50_Import folder, isn't it?
According to the attached screenshot, on the other hand, the source file is already in the Canon M50 folder, that is, the destination folder. For me it looks as if you called the renamer accidentally on the target file, not the source file. This would move the file onto itself - which can not work.

If I look exactly, yep, that is the case. Good catch.

The source-file is not here:
Source: e:\Fotok_I_M\Canon_M50_Import
Best wishes from Switzerland! :-)
Markus

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #19 on: April 03, 2021, 02:16:51 PM »
It was already moved. Picture was only to display the rule.

I made the same try with a testdb, same relative folder structure. See photos. This time all files very not read-only.

How does your folder structure look like? Folder of the photos after moving the files and the import folder relative to it.
I don't think mine is an extremly difficult case, but it is strange, that I am the only one having this bug.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #20 on: April 04, 2021, 12:43:50 PM »
I could so far not reproduced this. I've used the same mask as you. And tried both new folders not part of the database, folders already in the database and adding the target folder automatically. Files with the same date and time, different dates and times.

When you use the preview button, are the file name as you expect them?

Maybe versioning or buddy file management involved somehow?

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #21 on: April 05, 2021, 04:32:21 PM »
I have it with every download from my camera.
It is also in a testdb, where only very few files are inside and created with the New command in IMatch.
(description in following post): https://www.photools.com/community/index.php?topic=11140.msg79448#msg79448

Here is a 7Z with my testfiles with the folder structure::
https://1drv.ms/u/s!AnQSer5eDlmQuLgq5SM0RjpJaKI9VA?e=QUvIWN
TestDB Pack&Go:
https://1drv.ms/u/s!AnQSer5eDlmQuMA9BHFascqj5oH9-Q?e=3Yn1S0

And yes, the preview button and the result window shows the correct path with all files.
The files get also moved there.
Only the first CR3 gets lost from the IMatch DB (until rescan).


Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #22 on: April 05, 2021, 04:41:47 PM »
Which settings do you use for the target folder (add, add leaf, ...) or is the folder already managed by the database?

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #23 on: April 06, 2021, 11:24:42 PM »
The parent folder of the target folder is already managed by the database.
The option "Add created folder to database" is set to "no".

Should I set it to "Bottom level only"?

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #24 on: April 07, 2021, 09:54:20 AM »
This is a very complex setup.

You move files with the Renamer to a folder which is not in the database, but one of its immediate parents is.
You explicitly forbid the Renamer to add the new folder to the database, so it won't and also not trigger a rescan. You basically move your files out of the database at this time.

Now, since the parent folder is in the database already, Windows sends IMatch notifications that something in the parent folder has changed.
A background thread in IMatch responds to this and starts scanning the parent folder for new files and optionally sub-folders (depends on your settings).
This can happen while the Renamer is still running!

Due to the overlap of the rename, move, buddy file processing, folder rescanning all happening at the same time, there is a potential for timing issues. Resulting possible in one or more files being moved to that folder during the time IMatch is scanning it. And then IMatch thinks that the folder is current, but some files have been sneaked behind the curtain into that folder.

The simple solution is to just set the option Add created folders to database to "Yes, all".
Please try this and see if the problem persists.
Here it did not.

bekesizl

  • Jr. Member
  • *
  • Posts: 84
Re: First file is missing from target folder after Renamer
« Reply #25 on: April 07, 2021, 03:54:30 PM »
What you write sounds completely logical.
Maybe it was working like this all the time, I only recently recognized this behaviour.

I changed the setting to add all folders to the database and it was working correctly on the test database.
I tested it with the combined rename+move rule.

So thank you for your help. You can close this bug report as "not a bug".

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 27956
Re: First file is missing from target folder after Renamer
« Reply #26 on: April 07, 2021, 04:19:12 PM »
Very good.
By using the "Add" option, IMatch adds the target folder to the database (if it is not already in the database) and also performs a rescan automatically when the list of files has been processed. This way, no file can be missed due to how Windows notifies IMatch about changes in the parent folder while the Renamer is running.