Author Topic: Global (persistent) script variables?  (Read 4536 times)

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Global (persistent) script variables?
« on: March 25, 2015, 10:54:23 AM »
Is there a way to have variables in scripts which are persistent?
What I want is this: Through the events, run a first script when iMatch is started which reads a text file and writes that information into an array.
On file created and file modified events, another script is triggered which has to use the information in that array.
Evidently, I don't want to read the above file for each of the triggered file new/changed events.

For those who are interested in the background: The text file would contain regular expressions which I could then use to assign files to certain categories, thus avoiding to have the regular expressions in calculated categories which take a long time to re-calculate. Instead of having iMatch re-calculate dozens of such categories all the time, these expressions would only be used by the event scripts when files are changed or added, thus giving me the power of regular expressions without most of the disadvantages when used in calculated categories.

Ferdinand

  • 100 years since I was shot and a war was started
  • Global Moderator
  • *****
  • Posts: 1670
Re: Global (persistent) script variables?
« Reply #1 on: March 25, 2015, 11:09:21 AM »
Global Attributes?

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #2 on: March 25, 2015, 11:12:52 AM »
Global Attributes?

I thought Attributes are per file? I need global variables in the scripts (i.e. which are common to any script I run).
I'm looking at the ScriptSettings class right now. That might just be what I'm looking for if there is no such thing as global script variables.

Ferdinand

  • 100 years since I was shot and a war was started
  • Global Moderator
  • *****
  • Posts: 1670
Re: Global (persistent) script variables?
« Reply #3 on: March 25, 2015, 11:38:20 AM »
There are per file attributes and global attributes.  See the help page on attributes.  I think properties in 3.6 were similar. 

I've not scripted global attributes, but if you look at the sample scripts, the one on attributes seems to have examples on how to manipulate global attributes.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #4 on: March 25, 2015, 11:42:13 AM »
To store script settings, the best way would be using the ScriptSettings class.
It allows you to store any number of settings in a script, in the same way IMatch stores settings. The settings are even stored in the imatch5.pts file, together with all other settings. See the help on ScriptSettingsfor details.

There is dedicated example script for the ScriptSettings, and many of the Import & Export scripts and Apps also use this class to persist any type of data between script runs.

You can also use User Variables or even Global Attributes. But the preferred method to store arbitrary 'settings' data in scripts is ScriptSettings.

Ferdinand

  • 100 years since I was shot and a war was started
  • Global Moderator
  • *****
  • Posts: 1670
Re: Global (persistent) script variables?
« Reply #5 on: March 25, 2015, 11:57:05 AM »
Clearly I'm rusty.  I don't script much any more, I just use a few of them fairly often.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #6 on: March 25, 2015, 01:15:05 PM »
Thanks Mario and Ferdinand!
I'll do some testing to see what suits my needs best, global attributes or ScriptSettings.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #7 on: March 25, 2015, 01:19:48 PM »
Global attributes are stored per-database, ScriptSettings are truly global (if you want them to be).

Ferdinand

  • 100 years since I was shot and a war was started
  • Global Moderator
  • *****
  • Posts: 1670
Re: Global (persistent) script variables?
« Reply #8 on: March 25, 2015, 02:33:27 PM »
Global attributes are stored per-database

But they could be exported and imported into another DB?

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #9 on: March 25, 2015, 05:10:12 PM »
Quote
But they could be exported and imported into another DB?

Sure. But it deems a bit to much work just to remember some settings in a script.

Ferdinand

  • 100 years since I was shot and a war was started
  • Global Moderator
  • *****
  • Posts: 1670
Re: Global (persistent) script variables?
« Reply #10 on: March 25, 2015, 11:28:51 PM »
Well, the OP was about storing some category information, so it's not just about script settings.  I just wanted to reassure Carlo that this could be migrated if he went the script settings route.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #11 on: March 26, 2015, 09:11:49 AM »
Well, the OP was about storing some category information, so it's not just about script settings.  I just wanted to reassure Carlo that this could be migrated if he went the script settings route.

I have decided to use global attributes. As I said, it will be used to assign files to events, based on their filenames in relation to start and end dates. The attributes will look something like the attached screenshot and the filenames always include the date (like "D20140618...").


[attachment deleted by admin]

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #12 on: March 26, 2015, 09:28:30 AM »
Would a hierarchy of 'event categories' not be an alternative? Having this info in a global Attribute Set is working, but you probably could 'do' a lot more when you manage this in categories.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #13 on: March 26, 2015, 12:09:22 PM »
Would a hierarchy of 'event categories' not be an alternative? Having this info in a global Attribute Set is working, but you probably could 'do' a lot more when you manage this in categories.

I actually had this in categories based on FileRegEx formulas which slowed iMatch down quite a lot. This should be much faster because the file added/modified event scripts just have to quickly compare dates from these Attributes with a single filename and then eventually assign the file to one category, instead of iMatch recalculating 50+ RegEx based categories each time.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #14 on: March 26, 2015, 02:25:33 PM »
Event scripts may down IMatch probably more than a few data-driven categories or a category using regular expressions.
If you have a database script, IMatch needs to fire events for many operations, the Basic language needs to process these events, route them via COM channels, check the script if a handler exists for a given event and then execute the code. I cannot estimate what this means in terms of performance for your database, so do some timings and check if IMatch slows down overall. It depends which events need to be handled etc.

Quote
iMatch recalculating 50+ RegEx based categories each time.

50 categories using regular expressions can be really fast, unless each category has to run the regexp for all files in the database every time. Usually you can limit the regular expression to sub-trees or sub-sections of files, which greatly improves performance. You can check the log file if it reports categories which take longer to calculate than 2s by searching for 'slowcat'.

Depending on the change frequency of your database, you may be able to run the regexp only once - and then to persist the result by assigning the files to the category and then removing the formula.

Whatever solution you find, have fun.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #15 on: March 26, 2015, 04:09:16 PM »
Event scripts may down IMatch probably more than a few data-driven categories or a category using regular expressions.
I noticed an extreme slowdown with the RegEx categories. The main problem is that iMatch wants to recalculate ALL those categories for many changes, even if they have no possible impact on them (like creating a new category somewhere in the category tree).

50 categories using regular expressions can be really fast, unless each category has to run the regexp for all files in the database every time.
That's what it has to do. And the number of those categories is constantly growing with a new one for each event.

Usually you can limit the regular expression to sub-trees or sub-sections of files
Not possible in my case as the concerned files may appear anywhere in my folder structure and the filename is the only sure reference I can depend on.

Depending on the change frequency of your database, you may be able to run the regexp only once - and then to persist the result by assigning the files to the category and then removing the formula.
But the automatic assignment is what I want. Otherwise I can just do it all manually. The only useful option would be to have a type of calculated categories that does not update automatically, but only on demand. Then I could run the update once, for example when I quit iMatch. I could also do that with my current script if it slows iMatch down too much.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #16 on: March 26, 2015, 04:30:27 PM »
The same effect can be achieved by a script which updates the categories (creates/removes categories, assigns files) by whatever criteria you need. You can run this script manually when needed, or when you open or close IMatch.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #17 on: March 26, 2015, 10:05:38 PM »
The same effect can be achieved by a script which updates the categories (creates/removes categories, assigns files) by whatever criteria you need. You can run this script manually when needed, or when you open or close IMatch.
That's what I meant with my last reply. If the load gets too heavy for the add/modified events, I'll run the script manually as needed.

Ferdinand

  • 100 years since I was shot and a war was started
  • Global Moderator
  • *****
  • Posts: 1670
Re: Global (persistent) script variables?
« Reply #18 on: March 26, 2015, 10:44:01 PM »
I can relate to all this.  In IMatch 3.6 I used a script and regex to manually assign images to categories based on the file name.  In IMatch 3.6 I had hope it would be automatic, and using regex categories it can be, but at a performance cost.  So it's back to the script.

What I have done in my database is export the regex categories (without file assignments).  If needed I can import them, use them and then delete them.

Or I can use a regex filter.  This doesn't seem to have the performance hit, since it's transitory, although it's not as convenient as a regex category would be.

If I had my time all over again I'd change my file naming schema so that the characters I'm trying to extract from the filename are always in the same position, and so it could be done without regex.  But the existing file names are now too deeply embedded in all my systems.  And this approach didn't seem necessary back in 3.6.

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #19 on: March 27, 2015, 07:52:50 AM »
Or I can use a regex filter.  This doesn't seem to have the performance hit, since it's transitory, although it's not as convenient as a regex category would be.

If I had my time all over again I'd change my file naming schema so that the characters I'm trying to extract from the filename are always in the same position, and so it could be done without regex.  But the existing file names are now too deeply embedded in all my systems.  And this approach didn't seem necessary back in 3.6.

In my case I want to assign files based on date ranges. Those dates are in the file names, but a RegEx to select a date range from that is relatively complicated and slow. Too complicated for the filter alternative and too slow for calculated categories.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 30082
Re: Global (persistent) script variables?
« Reply #20 on: March 27, 2015, 09:19:34 AM »
I've made a quick check on a 160,000 files database. A regular expression in a category formula

"@FileRegExp[^[a-d]]"

takes 180ms to find 8200 files. That's quite impressive for processing 160,000 file names. Of course when you have 50 of those and IMatch needs to calculate them all, it takes 180 x 50 = ~10 seconds.

IMatch invalidates categories on many occasions. Invalidating a category just marks it as 'dirty'. The next time the category contents are needed, it will be re--calculated. You can often avoid to many recalcs by hiding the category counts. If only the name of the category is displayed, it does not need to be re-calculated. If the counts show, it has to...

sinus

  • Global Moderator
  • *****
  • Posts: 4489
  • IMatch-User since 2001 (IMatch 3.6)
Re: Global (persistent) script variables?
« Reply #21 on: March 27, 2015, 09:21:35 AM »

In my case I want to assign files based on date ranges. Those dates are in the file names, but a RegEx to select a date range from that is relatively complicated and slow. Too complicated for the filter alternative and too slow for calculated categories.

Carlo, this is interesting. I have also the date and time (not seconds) in my filename, like

20110114-1635-146259-s-pri-simona-feller_a.nef

where obviously is:
2011 year
01 month
14 day
16 hour
35 minutes

Based on this I had in IM3 a script for do a range. I used this script quite often for searching.
I clicked on an image and then the scriped had the date from this seleced image for the source of the search.

In a box I could then choosing things like

- the same year
- the same month
- the same day
- the same hour

- one month after this date
- one week before and after this date
- 3 days before this date

... and so on.
I would like to write also such a script for here, in fact, I played with it already.

But I am sure, like I did in this old script, was a rough kind (Holzfäller), but it worked fine.
So if someone knows a good method to "play" with this date, this would be great.

For example, that we could choose another date, and the script (or IMatch) would find all image in this range.
My code was hardcoded (- 30 days and + 30 days ...), so I could not search for 45 days before, if I has not coded it.

But this was again a good thing for me, having filenames with a lot of information in it.

(For example, if I did 3 shooting in the same day, parallel, then this search worked not, because it dit not distinguish between the shootings, only between hours. But in my search I could combine day, hour and description, what is the last part of my filename, and so it worked perfectly)












Best wishes from Switzerland! :-)
Markus

Carlo Didier

  • Super Hero
  • ****
  • Posts: 1720
Re: Global (persistent) script variables?
« Reply #22 on: March 27, 2015, 11:23:58 AM »
Hello Markus,
with the calculated categories, I was using RegExes (quite complex ones), but in the script I will be able to simply convert the part of the file name into a date value and simply compare that to the bounderies in my events list, something like "if ((FileDate >= EventStart) and (FileDate <= EventEnd))". Much simpler than building a RegEx for each DateTime range.

sinus

  • Global Moderator
  • *****
  • Posts: 4489
  • IMatch-User since 2001 (IMatch 3.6)
Re: Global (persistent) script variables?
« Reply #23 on: March 27, 2015, 11:41:07 AM »
Hello Markus,
with the calculated categories, I was using RegExes (quite complex ones), but in the script I will be able to simply convert the part of the file name into a date value and simply compare that to the bounderies in my events list, something like "if ((FileDate >= EventStart) and (FileDate <= EventEnd))". Much simpler than building a RegEx for each DateTime range.

Hi Carlo
I see. I envy you, I would not be able to do a) complex Regex-terms and b) stuff in a script like bounderies and so on ...

Good luck for your project!
Best wishes from Switzerland! :-)
Markus