Author Topic: iMatch remaing in memory after Closing database and Exiting  (Read 143 times)

dkorman

  • Jr. Member
  • *
  • Posts: 63
Mario,

I use several iMatch databases on one machine, often closing one to open another for various processing.  I've noticed a pattern with the largest database (~250,000 files): when I Close this database, and then Exit iMatch (which causes the UI to close and then asks be about Pack & Go, which I decline), I cannot restart iMatch for quite some time (hours) because I receive a message that only one instance of iMatch can be run.  TaskManager shows that iMatch2019x64.exe and iMatchChromiumHelper.exe are still memory resident, and iMatch2019x64.exe is using 12% of my CPU (under TaskManager Details, but not Processes).  I can force end these resident processes and restart iMatch, but that's probably not a good idea - I just don't know what these processes are still doing.  Is it "safe" to terminate them?

I've tried Rescanning and Compressing the large database (both of which take some time), but that does not seem to affect iMatch remaining in memory.  This behavior does not occur every time.  Additionally, sometimes (but not always) when I close the larger database and open another, without exiting iMatch, iMatch will seem initially to open the new database correctly, but then the dreaded white line across the top of iMatch displays, and the application freezes (I've waited for hours, as a test); I'm forced to force-quit iMatch and restart, which then iMatch seems to work correctly.

Attached is a screen shot Task Manager iMatch entries, and the latest iMatch Log file (with a few edits for privacy) while iMatch is still memory resident.

Any ideas?




Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21859
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #1 on: May 14, 2019, 08:13:45 AM »
Apparently something is not working correctly on your system.
IMatch is shutting down normally but is then halting in one of the final steps.

Maybe a virus-checker. Or maybe some incompatible component installed. Or something else.
One of the 3rd party components IMatch is using being blocked, blocking IMatch in turn...

Do you use Dropbox or BoxCryptor or HP scanner drivers? I recall a similar case from a few years back which occurred in combination of IMatch and one of these on one PC...

Do you reboot your machine (I mean a real reboot, not only shutting down / restarting) regularly? If not, try a reboot first.
Then re-install IMatch via the "Repair" option in the Windows control panel.

Please understand that with such unique cases, coming up with solutions is always difficult.
Maybe try to close the database first via the "File" Database menu. Then shut-down IMatch and see if this makes a difference.

I open and close databases all the time (I work with about 10 different databases for testing) and I have not seen this.

Also, switch your installation to Debug logging via the Help > Support menu. This produces a more detailed log file, which might be helpful.
« Last Edit: May 14, 2019, 08:17:42 AM by Mario »

dkorman

  • Jr. Member
  • *
  • Posts: 63
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #2 on: May 14, 2019, 04:17:16 PM »
Mario,

A) I do have a virus checker (Symantec/Norton), but have not experienced this problem with any other program (that I am aware of).  Regarding iMatch components - I'd clearly have no idea.  I know that we previously had a problem with the PDF component, and the database in question does include PDF files, as well as other file formats.

B) Dropbox is installed and an HP scanner driver does exist.  I have noticed a problem in the past with the HP scanner driver (a system message about a memory fault), which no other program has ever reported. 

C) Rebooting: Maybe once/week.  What do mean by a "real reboot?" 
    How often do you reboot Windows 10?
    I will try the Windows Repair option and see if this makes any difference.

D) I have used the File menu options to Close the database and Exist iMatch, and this problem occurs.

E) I will place iMatch in Debug mode.  Does this affect your telemetry?

This problem has persisted across the last several iMatch updates.  I will try your suggestions and report back.

Thank you, as always, for your quick reply,
David

dkorman

  • Jr. Member
  • *
  • Posts: 63
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #3 on: May 18, 2019, 05:01:20 AM »
Mario,

After making the changes you recommended, and additionally installing the latest version of iMatch 2019.5.2, iMatch again remained in memory for slightly over 2 hours after exiting yesterday.  A log file with the last 500+ lines of the log file in Debug mode is attached.  You can see waiting in the log file lines:

05.17 19:53:38+    0 [2748] 50  M>       <  6 PTFileSystemMonitor::FlushQueue
05.17 22:07:05+8006469 [9DE8] 10  M>      <  5 [20997172ms #sl] PTDataGrouper::Execute
05.17 22:07:05+    0 [9DE8] 10  M>      >  5 PTDataGrouper::UpdateGroup  'v:\develop\imatch5\src\imengine\ptdatagrouper.cpp(1904)'

Here's the processing I did before the problem occurred:

 - iMatch was open with no database open
 - I copied (through Windows Explorer) >50 PDFs and EPUBs to a temporary folder that is included in the large iMatch database
 - I opened the large database in iMatch, highlighted the temporary folder (under Media & Folders), and had iMatch Rescan the folder; which iMatch completed
 - I selected all files in the temporary folder and dragged them to the Image Batch Processor (under Import & Export panel) to create 800x600 JPGs of the files' "first pages" (which I use in a process outside of iMatch); the process completed as expected
 - within iMatch, I moved the files from the temporary folders (for source files and JPGs) to other folders which are also part of this iMatch database

Everything seemed to work as expected.

 - I Closed the database, and then Exited iMatch (from the Database menu); iMatch asked if I wanted to create a backup, which I declined.
 - The iMatch UI closed
 - iMatch & iMatchChromium remained in memory for over two hours. 

Does this help?  Any other suggestions?

As an aside, the PDF processor, which previously had problems when not run single threaded (something you fixed last year) also seems to unnecessarily place its temporary files in the recycle bin, which are clearly not needed, and "clutters up" the recycle bin.

Thanks for looking into this,
David

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21859
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #4 on: May 18, 2019, 08:47:34 AM »
No idea. The only hint is

RefreshGroup-Completed: 'PDF Files' in 20997172 ms.

which seems that IMatch needs 5 (?) hours to update the category named PDF Files.
Maybe something was stuck badly. The log file closed normally afterwards which indicates a clean shut-down.
Maybe drop this category and then retry. Or, when you process PDF files, try a session without. If the external PDF processor or the Windows PDF thumbnail handler fail because they hit a broken or unsupported PDF file, they may hang, and this means that IMatch hangs as well.

PDF files and recycle bin....

IMatch uses an external tool to create thumbnails from PDF files. it uses intermediate PNG files in the standard TEMP folder and deletes these files again after processing. It uses standard Windows "Delete File" routines and this apparently moves the files into your recycle bin. Probably I have just forgotten to explicitly specify the parameter for using the recycle bin as false - it defaults to true for good reasons. I will check this for one of the next releases. Windows manages your recycle bin automatically and will delete old files. No harm done.

dkorman

  • Jr. Member
  • *
  • Posts: 63
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #5 on: May 22, 2019, 02:49:55 PM »
As you suggested, I removed the PDF category a few days ago, and ... voila, there has been no repeat of the "resident in memory after exit" problem.  Problem solved?  Not really - more like problem area identified.

Could a single "malformed"/"non-standard" PDF have been the cause?  If so, how do I find it? 

Could iMatch be configured to report "problematic" files?  In this case, I guess, it's not a matter of a third party processing returning an error code - it's more like the third party just not returning. 

Mario, thank you for identifying what seems to be the problem area in this case.  Could I be of any help to you in furthering trying to identify the precise cause, and if so, what would you like me to do?

David


Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 21859
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #6 on: May 22, 2019, 03:10:27 PM »
This is such a uncommon problem (apparently affecting on your installation / database). I have no special provisions in my code to monitor the external PDF preview extractor or the Windows function calls IMatch uses when the 3rd party PDF extractor returns an "not supported" error. I have never seen the "pdftopng.exe" stay in memory. I have seen the Adobe PDF thumbnail handler do that often (or allocate 4 GB or RAM and then crash), which is why I've switched to XPDF last year. No problems have ever been reported.

If the IMatch log file contains no traces when this happens, look in task manager for "pdftopng.exe" processes. This may be a hint.
If IMatch falls back to the Windows thumbnail routines, which then use whatever PDF handler is installed on your machine (usually Adobe Acrobat Reader), this is not visible anywhere, except IMatch just blocking dead in all threads which use that. But that would be visible in the log file, which way not the case in the log files() you have posted.

The only way to analyze this would be to organize your PDF files into groups, and process each group. When a group produces the problem, drill down on that group until the problem PDF file is found.

But that all has of course nothing to do with a "PDF Category". Processing/updating categories does not trigger PDF thumbnail extraction or anything. If you mean the default PDF Files category in the "IMatch Standard Categories" hierarchy, these are just data-driven categories. IMatch will update them on-demand.

Maybe one or more of your PDF files contains metadata which causes the data-driven category to "hang forever" or something?
To analyze this would require to add one PDF file at a time to a new database (or in small batches), then closing IMatch after each batch until it again fails to shut-down. Then the last group has the PDF file which causes the issue and this would allow for deeper analysis.

All very complicated, time-consuming, ... and if only one user is affected by this...

dkorman

  • Jr. Member
  • *
  • Posts: 63
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #7 on: May 22, 2019, 06:28:52 PM »
Mario,

As you know, most "bugs" are going to be "uncommon" - especially with a mature product like iMatch, on which I'm sure you do considerable testing prior to release, and with the responsiveness you display to the wonderful community of users.  As for this unusual iMatch behavior, as you mentioned, I think that it would be too much labor for me to recreate the issue in the manner you suggest - I'm just appreciative that you were able to identify the probable culprit and I have been able to work around it.  Thank you.

David

jch2103

  • Super Hero
  • ****
  • Posts: 1636
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #8 on: May 22, 2019, 07:56:34 PM »
This is a long shot: If by chance the PDF issues created an ExifTool error message, this could be used to identify problem files. For example, I have a data-driven category based on the tag Extra\Error\Error\0 that I've used to spot problem files (happily no such problem files at the moment).
John

dkorman

  • Jr. Member
  • *
  • Posts: 63
Re: iMatch remaing in memory after Closing database and Exiting
« Reply #9 on: May 22, 2019, 11:46:23 PM »
John,

Thank you for the suggestion. 

I created a category with your suggested tag in another database (not the one with the "staying resident in memory after exit" problem), and of almost 70,000 files, the ExifTool error tag returned 12 error files:

   2 PDFs with internal errors (which I was able to fix using GhostScript),
   1 XLS file with a format problem (fixed), and
   9 PDFs which iMatch reported within the category as "Error creating file": which I believe were path-filenames which were too long (journal articles) (and when shortened, seemed to be fixed). 

This was the only place that I saw this kind of error reporting in iMatch, and I think that it's a great addition (again, thanks to Mario for iMatch's flexibility).

When I have an opportunity, I'll add the category to the "problematic" database, attempt to fix any reported errors, and see if this affects the "residency" behavior.

Again, thank you for taking the time to help.

David