Author Topic: Folder add fails to complete; thumbnails grayed out and needing refresh  (Read 375 times)

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Mario -

Thanks for a great program - I appreciate your hard work!

However, in the past couple weeks I have had problems adding new folders to the database.  The add seems to complete successfully - the folder shows in the directory listing, and there are thumbnails for each of the images in the folder.  However, the process never completes - the thumbnails are grayed out, and have the blue 'need to be refreshed' icon.  The 'adding and updating folders' message in the lower right corner of the screen never goes away. Clicking on one of the images generally will display the preview in the 'quick view' window, but sometimes there are problems with that. 

I have changed the number of process threads to '1' in the process options, as I saw that solved someones similar problem, but that showed no change.  I have closed and reopened the database, I have run database diagnostics, I have rebooted the computer, but I have not found anything that helps.

I have attached a debug log file, and I hope you can tell something from that.  When I look at the end of that it appears that IMatch is idle and thinks there's nothing to do.  Please take a look and let me know of any actions I can take to try to resolve the issue.

Thanks - Dave



Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #1 on: October 15, 2019, 08:59:47 AM »
The log is clean, except that the the commercial image library IMatch uses reports "unknown image type" for some .xcf files.
These are GIMP images (I suppose) and IMatch may be unable to extract a preview. This depends on whether or not a "thumbnail handler" is installed for this format.

But the file processing completes normally, no crashes or hangs as far as I can see.
Then IMatch is idle for quite a while and then some files are loaded from the cache, probably for the Viewer or Quick View Panel.

Does this always affect the same files in the same folders?
Are only XCF files affected?
If other files are affected, which formats? RAW or normal image files?
Are the files stored on a local disk, external disk, network?

Reducing the number or processing threads massively reduces performance. I only advise this is a system becomes unstable under load or if images need to be processed which reside on slow media like remote NAS boxes connected via slow networks or Wi-Fi. Under normal conditions, IMatch handles this automatically.

Maybe just select the files in question and force a rescan using Shift+Ctrl+F5?

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #2 on: October 15, 2019, 03:44:42 PM »
Thanks, Mario, I'll gather more information and report back.

Both .NEF (Nikon RAW) and .JPG files are affected, the files reside on a local SSD drive.  The problem seems to be happening with all new folders I'm adding.

I'll Ctrl+Shift+F5 a set of images and gather the log file from that.

Thanks - Dave

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #3 on: October 15, 2019, 04:03:52 PM »
Background processing still enabled under Edit > Preferences > Background Processing?

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #4 on: October 15, 2019, 04:32:23 PM »
As far as I know I've never changed that setting, but I'll double check that.  Thanks - Dave

ubacher

  • Oldtimer
  • ****
  • Posts: 2053
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #5 on: October 15, 2019, 06:50:19 PM »
That sounds like a problem I was having quite some versions back.
The only thing that helped me was clearing the processing queues and then closing/reopening IM.
IM then processed the files that were "hung".

Eventually the problem went away (with some newer version) and I haven't had it since.

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #6 on: October 17, 2019, 04:38:21 AM »
ubacher - can you explain what you mean by 'clearing the processing queues?

Mario - I've made some progress, I sequestered the 4 GIMP .xcf files and reprocessed the folder they were in, and it completed - Yay!  I then removed from the database the 8 or 9 folders that seemed to be hung.  At this point attachment 1) shows the 'Adding and Updating files' message in the lower right corner of the screen even though all files were loaded at that point.  Note the little database icon that appears to the left of the message when activity is happening is not there.

I started adding back in the folders I was having trouble with (names ending in '115' down through '108' - I added them back in reverse order).  When I got to folder 111 things got hung up as shown in attachment 2).  I left it there for an hour or so, then clicked 'dismiss', only to find the folder was loaded and all the thumbnails were visible.  Odd, but I was happy.

When I got to folder 109, processing appeared to hang again. Attachments 3) and 4) show that the message in the lower right of the screen didn't change for over 3 hours.  I checked task manager and see that IMatch has a bunch of memory allocated, as shown in attachment 5).  I don't know if that's normal or not.  Attachment 6) shows the thumbnails in the folder are all grayed out, and the preview doesn't show for the image selected.

I ended IMatch and went back in.  Attachment 7) shows that IMatch starts processing files right away, but I find it odd that the number of files it thinks it is working on does not match the number of files in the folder.  Maybe that's a clue to something?

Attachment 8) shows the background process is still enabled.

I've also attached a couple log files that cover this experiment period.  I hope this information is helpful, please let me know if there is anything else I should try.  I'll attach the maximum of 5 attachments to this post and enter another post with the rest.

Thanks - Dave

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #7 on: October 17, 2019, 04:43:21 AM »
Here are the remaining attachments from my previous post.  -- Dave

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #8 on: October 17, 2019, 09:06:14 AM »
Database > Tools > Clear Processing Queues.

When IMatch suddenly stops processing files, what did you do before?
Do you work with IMatch while it is processing files in the background or do you work in other applications?
Did you make IMatch and the folder (!) containing your database an exception in your virus checker?

What happens when you just close and restart IMatch? Does it continue to process files?

ubacher

  • Oldtimer
  • ****
  • Posts: 2053
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #9 on: October 17, 2019, 05:15:34 PM »
In my case IM showed xx files in the processing queue but never finished processing the files.
Even after a restart it showed xx files in the queue but did not process them.
Only when I cleared the queue and then restarted Im it processed the rest of the files.

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #10 on: October 18, 2019, 04:22:53 AM »

re: Do you work with IMatch while it is processing files in the background or do you work in other applications? - Since it takes IMatch so long to load files, I generally switch to a different window and do something else.  Sometimes I just let it work and go have supper or something. Sometimes I dismiss the 'adding files' dialog and work on images while new ones are added in the background.

re: Did you make IMatch and the folder (!) containing your database an exception in your virus checker? - This was not the case before, but I have added them both in as exceptions now.

re: What happens when you just close and restart IMatch? Does it continue to process files? - yes, when I was having problems and I restart IMatch it recognizes that there is work to be done and starts in on it.

Latest update: This evening I took ubacher's advice and followed your instructions on clearing the processing queues.  It took quite a while for that to finish, but it completed successfully, and I was able to close IMatch without killing it off this time.  I reopened the program and the blue 'adding files' message in the lower right corner of the screen was finally gone, so that was good.  In the directory I was working in there were some files that had been modified, and the program recognized that and reloaded them successfully without me forcing a rescan, so that again is good.

Summary - as of now there are no issues - no thumbnails are grayed out, no files or folders are marked for rescan, no blue 'adding files' message is present.  I added a new folder with 10 jpg files and it loaded them all no problem. Ahh - feels so good! 

Thanks for the help Mario and ubacher!  I wish I knew why things started to go downhill, but I'm happy it seems to be solved.  I will provide an update to this thread if the issue resurfaces in the future, but I hope it doesn't!

Thanks - Dave

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #11 on: October 18, 2019, 09:01:00 PM »
Mario -

Well, I unfortunately have an update.  After my last post I loaded a folder with 10 jpg files and a subfolder with the corresponding .RAF (FUJI raw) files with no problem.  All OK at that point.

I then accessed 3 of the raw files and made some updates, overwriting the corresponding jpg files with the new output.  IMatch successfully recognized the jpg files needed updating - the thumbnails were grayed out and go the little blue 'needs to be updated' icon.  Unfortunately, that's as far as things went.  I left the computer on and went to bed, in case things were just slow.  However, this morning showed no further progress.

I then Ctrl+Sft+F5 rescanned the entire folder, forcing update.  All the thumbnails then went gray and got the blue icon.  Clicking on a thumbnail failed to display the image in the quick view window, which is black and displaying 'Loading' in the middle of the screen.  The blue message 'Adding and Updating Files' is back in the lower right corner of the screen.  I just had my wife check on it now 8 hours later and nothing further has occurred.

Can you please take a look at the attached log file and see if you can determine what the issue is?

Thanks - Dave

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 23189
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #12 on: October 19, 2019, 08:28:05 AM »
Which applications do you use?
This sounds as if the application is keeping the files locked, preventing Windows / IMatch from accessing them.

The log file has no warnings or errors. It shows that several files have been successfully loaded.
ExifTool has imported the metadata, IMatch has imported the files, processed the images etc. All looks well.

I, and I'm sure many other users, work in the same way you work. Editing files indexed in IMatch, replacing them, creating new files etc.
This must be something peculiar to your workflow, applications, virus checker or settings. Else the community would be full of similar posts.

Very hard to diagnose. I see that the background processing queue is signaled and processing entries after the last file has been imported (DJEF2963.JPG) in about 0.4 seconds.
Then it calculates the CRC checksum for the file DJEF2954.RAF but fails to load / open the image.
A bunch of RAW files has been checked for updates, but apparently only DJEF2954.RAF was modified.
No error message, though. Maybe IMatch is stuck in the WIC codec for RAF files on your system?

Your system seems to have multiple WIC codecs for RAF installed. Two from Microsoft (?) and one from Fuji. Maybe that's the problem.
Is the Fuji codec recent? I did not even know that Fuji offers their own WIC codec...
« Last Edit: October 19, 2019, 08:41:50 AM by Mario »

WoodnMetalGuy

  • New Members
  • *
  • Posts: 11
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #13 on: October 28, 2019, 04:13:06 PM »
Thanks for looking it over, Mario.  My installation has been working OK lately.  Sometimes IMatch doesn't seem to recognize right away when a file has been updated, but a 'rescan now' gets it updated.  I've added multiple new folders without issue.  Fuji jpgs and Nikon NEFs, no Fuji RAF files.

Applications generally running on my machine include Outlook and iTunes, plus all the system stuff that runs in the background. Your idea that something else is locking the files could certainly be the case, but I don't have high hopes of being able to track that down.

Odd about the WIC codecs for RAF files.  I've never done anything with them (not even installed them), so something must have done that its own.  I will have to do some research and perhaps clean that up.

Thanks - Dave

loweskid

  • Full Member
  • **
  • Posts: 111
Re: Folder add fails to complete; thumbnails grayed out and needing refresh
« Reply #14 on: October 28, 2019, 09:06:44 PM »
Do you use Adobe Bridge?  I had a similar problem in the past and I traced it to (I assume corrupted) Bridge files in the folders.  They are called something like *.bridgecache or *.bridgesort and I think they are normally hidden files so you may not see them unless you have 'show hidden files' and/or 'show system files' turned on in Windows.

I did a search in Windows Explorer for anything with 'bridge' in the file name and deleted them all - not had any similar problems since.