APP: Super Search (or the search it all app)

[Jingo]

Howdy all... just wanted to post an app that I worked on many moons ago that allows you to search the whole database for an item right from an app.  I created this because I often found myself filtering a set of data and then needing to search the entire database for other data - without disturbing the results I already had. 

The app isn't anything pretty - but gets the job done.  Been awhile since I wrote it but I just reviewed it, tweaked a few items and did some general testing and it seems to still work.

The app allows you to search the entire DB or the local scope (not really that useful since the scope is the focus of the normal search).  You can use either the normal AND, OR, AND NOT phrases or you can use the shortcuts +, - and | - the software will convert and search as such.  I dont' believe the () are working.. but I need to verify that.

The results will display in a results tab which is nice since your main tab is untouched.  As always, use with caution!  The folder within the zip can be placed inside your app user folder: C:\ProgramData\photools.com\imatch6\webroot\user\

Enjoy! - Andy.

Fancy Icon


The App window (not Fancy!)


The App in Action

[Mario]

Thanks for sharing 

[Jingo]

Happy to code and share... scripting in IM is FUN!   

[BanjoTom]

Really nice, Jingo!  On (limited) testing, this seems to work well and can be very helpful!  Thanks!   

[Jingo]

Really nice, Jingo!  On (limited) testing, this seems to work well and can be very helpful!  Thanks!   

Thx Tom - I was originally thinking of creating some sort of "builder" search app like you see in the MacOS Photos Smart Album dialog... where you can say:

Keywords          Contains (All)             BEAR; MOUNTAIN; SNOW             <AND>
Create Date       Is In The Range         11-12-2015 thru 11-19-2015     

With dropdown boxes for the choices: 1st column is metadata fields, attributes and general terms, 2nd column is ALL, AND, OR, Contains, etc... and 3rd column is free text, lookups to data (ie: Keywords in the database) and date chooser,   4th column allows for additional conditions to combine 2 or more rows together with AND/OR/NOT AND

But - this would take me a lot of time.  I think this type of "natural language" or "builder" search create would be valuable though so may re-visit when I have lots of free time and the programming itch hits!

[sinus]

The app allows you to search the entire DB or the local scope (not really that useful since the scope is the focus of the normal search).  You can use either the normal AND, OR, AND NOT phrases or you can use the shortcuts +, - and | - the software will convert and search as such.  I dont' believe the () are working.. but I need to verify that.

The results will display in a results tab which is nice since your main tab is untouched.  As always, use with caution!  The folder within the zip can be placed inside your app user folder: C:\ProgramData\photools.com\imatch6\webroot\user\

Enjoy! - Andy.

Hi Andy, really cool stuff!
Thanks a lot for this! 

After first impression, it is simply cool!

It finds the same files like IMatch, if I search the whole DB, but on another way, for me easier.

I am happy with this, like it is!

If you would "enhance" your app, I could imagine the following things:

- I have nearly 300'000 files to search. Hence, it takes some time, and hence if a search is running after press Search, it would be helpful, to see this, that a search is running.
Like with the native IMatch-search, I have not really a clue (except there a bit dimming), if a search running or not.
It would be cool, if a box would stay and informe me, that a search is runnung. If a search runs only for some seconds, this could be annoying, hence maybe it should be possible, to enable this or not.

- in the box (or other form of info) above would be a running time cool. My searches depends, sometimes only seconds, but often also til 200 seconds, hence it would be cool, if I would see, "the search runs now for 20 seconds, I await around 150 seconds, hence I can quickly create a coffee" 
And if the search is finnished, this box would say me, how many files has been found in what time.

- it is nice, that in the created result tab there is written the search-term and the number of result-file. What would be even nicer for me, if there would be in brackets the number of founded files. This could be instead the number of result window or both together [3/212] (this would mean window 3 with 212 founded files).

Jingo, do not get me wrong: your app is really cool and I will this use for sure.
The above "enhancements" are only proposals and I have no idea, is this difficult to code or not.
If you say, takes too long time to create something like this, I am happy with this, what you have created until now!

Thanks a lot for your work and even more for sharing it! 

[Jingo]

Thx Markus - love these ideas and I will go ahead and implement them today!  I'm glad this app is useful... been sitting in my app bin for a few moons now though I did add a bit of tweaking to it since I first wrote it.  Stay tuned for the updates! - Andy.

[sinus]

Thx Markus - love these ideas and I will go ahead and implement them today!  I'm glad this app is useful... been sitting in my app bin for a few moons now though I did add a bit of tweaking to it since I first wrote it.  Stay tuned for the updates! - Andy.

Hooh! Would be really cool, if you could use and add some of the ideas. 
Thanks in advance for thinking about it!

And, hey, Andy, yep, Congratulations also from me for your 1000 postings!

[Jingo]

Hi Markus - thanks for your suggestions!  I have implemented the following changes to the APP today (new zip attached):

1 - a new button will appear as the search is running.  This will provide some feedback that the script is actually working and searching for items.  Once complete, the button will disappear and the results will display in the results tab.

2 - the result tab header now displays the number of found files, the number of searched files and the search phrase

Currently, ALL metadata is searched... I can always add some specific keys to check if this is helpful.  ENJOY!


MARIO - is there any methods available to determine progress once an IM or IMWS request is made?  I know we wait for the promise or error... but while the request is going on - are there any system variables that can be checked to find things a) number of files searched  b) total results found so far?  This would make progress bars and results as they are found to be displayed a possibility.....  of course, it would require more resources as part of the services to store this data for retrieval...  just thinking out loud!

[BanjoTom]

I like the small improvements you've made.  They add a lot to this (already) very helpful app! 

[sinus]

Hi Markus - thanks for your suggestions!  I have implemented the following changes to the APP today (new zip attached):

1 - a new button will appear as the search is running.  This will provide some feedback that the script is actually working and searching for items.  Once complete, the button will disappear and the results will display in the results tab.

2 - the result tab header now displays the number of found files, the number of searched files and the search phrase

Currently, ALL metadata is searched... I can always add some specific keys to check if this is helpful.  ENJOY!

Hi Andy,
This is really very cool!!!
Thanks very much for creating/implementing this.

The new button shows me at least, that the app is running and that is very helpful.

Your Fancy Search - simply easy and powerful.
Very easy and quick to use: open the app, write the search - term, return, voila!

It searches the whole DB and all metadata including filenames!

The new info in the result-tabs are nice.

Because I can be sure, ALL metas and filenames are searched.
If found, I can see, how many files. If I want now drill down the files, I can simply and quickly use the search bar.
Both together is very effective, your app is the basic for this.

Since you asked:
Of course sometimes it would be helpful to search only for the most important metas, like in the search bar (e.g. only Headline, Description).
For me this would make only sense, if the app would stay that easy, like it is now.

For example, if in your app there would be buttons to choose only some tags or group tags, then this would be fine.
Because if I would not enable such buttons, simply all metas and names would searched.

Hence from my point of view it is not necessary, but if easy implemented, why not.

If you would like to "enhance" your app, I had some proposals, but since I love your app, like it is, I do not write them (but if you like, I can try to write, what I mean).

One enhancement, what would be nice:
If am an not wrong: if the search finds nothing (0), the search box disappears, but there is no message.
It would be nice, if there would be a message like "no files found" in a box.
Why?
My searches are running sometimes 30, 50 or 100 minutes, hence I go away from my computer. When I am back, and nothing is found, I am not more fully sure "hm, did I hit the search button or not?".

Andy, thanks for your work!

[loweskid]

I've just been trying out this app - brilliant..   Thanks for sharing.

[Mario]

MARIO - is there any methods available to determine progress once an IM or IMWS request is made?!

No. This has been optimized to the hilt and runs inside the IMatch engine. Raising events would slow things down massively. Pumping these evens over web sockets (so apps could react on them) would reduce the search speed massively.

I've made a quick check on my largest database (520,000 files).
Searching the entire database for pattern 'airplane' takes 14 seconds if the search cache is cold (empty).
This goes down to only 3(!) seconds for 520,000 files if the cache is hot (filled), which is the case from the second search request and until the database is modified. Not much user feedback needed for these times. A more typical 100,000 files database has search times of 3 (cold) and less than one second (hot cache).

A user should be able to watch an animated "working..." icon or something for these few seconds. You can of course do whatever you like in your app until the promise is resolved with the result.

Note: I here assume that you use either the group or the tag parameter in your query.
If you don't use them, the engine cannot optimize anything, no caching can work and a full search on the database is performed. For Marku's 300,000 file sample database this probably means that IMatch has to search about 100 million values every time your app runs. On disk. And Markus does not use SSD storage when I'm not mistaken.

I used a mix of about 20 frequently used tags for my test, including keywords, file name, title, description, headline, etc.
I used the v1/search/metadata endpoint.

[sinus]

MARIO - is there any methods available to determine progress once an IM or IMWS request is made?!

...

Note: I here assume that you use either the group or the tag parameter in your query.
If you don't use them, the engine cannot optimize anything, no caching can work and a full search on the database is performed. For Marku's 300,000 file sample database this probably means that IMatch has to search about 100 million values every time your app runs. On disk. And Markus does not use SSD storage when I'm not mistaken.

I used a mix of about 20 frequently used tags for my test, including keywords, file name, title, description, headline, etc.
I used the v1/search/metadata endpoint.

True, I have no SSD for the storage. 

I made a quick test:

1) search for janine
2) search for ccm
3) search for Bern AND Kaf

Start IMatch, first with the "Fancy Search" from Andy:
1) 7332 hits, 58 Seconds
2) 737 hits, 58 Seconds
3) 85 hits, 45 Seconds

ReStart IMatch, in "Media & Folder", Search Bar with search for everywhere:
1) 6693 hits, 66 Seconds
2) 735 hits, 8 Seconds
3) 85 hits, 8 Seconds

This means, like you wrote, Mario, that with Andy's App the search starts always from new.
Anyway, I like this search-app really.

As for the differences in the results of searches 1 and 2, I haven't done any research now (Andy's app finds more hits).

[Mario]

Search everywhere is the worst case for the search, because no optimizations can be made.
I'm not sure if you really need to find data in obscure EXIF tags or retired IPTC fields. Otherwise using one of the search sets in the Search bar will speed up your search considerably.

[Jingo]

Search everywhere is the worst case for the search, because no optimizations can be made.
I'm not sure if you really need to find data in obscure EXIF tags or retired IPTC fields. Otherwise using one of the search sets in the Search bar will speed up your search considerably.

Very true... I'm working on some modifications based on the feedback from Markus - though I am running into some troubles so it make take me a day or two.  My idea is similar to the Tag Info app - allow the user to type in a term (ie: title) and then a multi-select drop down would include all metadata tags with title in it.  Then, the user can select the tags they want which builds a list and then the search can use that list to pass as tags for the v1/search/metadata which I am also using.   The tag search results can be saved to the database so it can be pulled back in and edited/altered at a later time as well.

This will allow the search to be narrowed down to only certain fields.  I don't intend this to replace standard functionality... as mentioned, I wrote this because I liked to have quick access to result tabs without affecting the main tabs... and its' fun to program apps!

[Mario]

Another way to search several sets of files in IMatch would be:

Select the files in a File Window, then "Open selection in result window" command (Ctrl+G, R).
This gives you a new scope in a separate result window and you can use the search bar independently from the other File Windows.

This 'trick' is often new to users when I demonstrate it.
Probably not the same as your app, I just wanted to mention it for the sake of the community search engine.

[Mario]

Also, consider to warm up the search engine by running a search immediately when your app starts (while your code is still building the front-end, loading settings etc.).
This improves the experience for the user, because his first search will be much faster.

Tip: Set a Boolan variable _ready=false when the app starts and set _ready=true; when the promise of your background search is fulfilled. Enable the "Run Search" button in your app only when _ready is true.
For regular database the background search will be finished before the user sees the app so he will never notice.
If your user has a very large database and/or slow spinning disks he may need to wait for a few seconds for the "Run Search" feature to become available. but the first search will still be much faster.

[sinus]

Andy, only to say it again:

your app works simply fine and cool.
I work often with it.

open the app, write the search-term, click the search-button and I get a visible clue, that it is searching.

Thanks for that, I use it often. Very cool.

[Mario]

You search for what?

Why not selecting the Database node and do it with the regular File Window search bar? That's how I search my entire database.
Or I use the filter panel for more complex searches.

Or I use the File Finder App, which is especially useful during migration projects, when similar file names have to be found. Or I need to search for batches of files by uploading a list of file names. Very useful this app.

[Carlo Didier]

Why not selecting the Database node and do it with the regular File Window search bar? That's how I search my entire database.

Because that is very slow as iMatch first tries to display tens of thousends of files? That's the reason I try to avoid selecting the database node.

[sinus]

You search for what?

Why not selecting the Database node and do it with the regular File Window search bar? That's how I search my entire database.
Or I use the filter panel for more complex searches.

Or I use the File Finder App, which is especially useful during migration projects, when similar file names have to be found. Or I need to search for batches of files by uploading a list of file names. Very useful this app.

I search for a word or two somewhere in my files in the whole DB.

I use two search apps:

1) your File Finder App. It allows me, to search for filenames over the whole DB, it is the quickest possibility.
2) the fancy Search App from Andy (Jingo). The reason is also the same: it is the quickes possibility. Otherwise I have to go tho the Database node, scroll up, click on it and with 270.000 files this takes some time. Then I have to check, that I do search for sure in the correct tags and so on. And I can stay (for examples in a specific category), where I am.

I think, with these 2 apps I can search really good and I know, they searches over the whole DB, does not matter, where I am.

[Jingo]

Andy, only to say it again:

your app works simply fine and cool.
I work often with it.

open the app, write the search-term, click the search-button and I get a visible clue, that it is searching.

Thanks for that, I use it often. Very cool.

Thanks Markus... I'm very happy the search app works for you... I too use it quite often when searching the whole database.

[sinus]

Jingo, I know, I have written a lot above, but I am too lazy (and too busy) to reread again my own stuff 

Wanted again praise your app. For me it is still the most pleasant way to search the whole DB.
Hit a button, no need to change the directory, I can stay, where I am, cool. For this quick way is is not really important, that a search over the whole DB is of course slow (search everywhere).

Usually I hit the button, enter the value and ... get a coffee 

The most possible enhancement could be (and I guess, easy one), if the result is nothing (no catches), then it would be nice, if this wold be written like "no hits" or so.
But not really important, specially because during a search I see a "wheel" (what is cool, because then I see, the app is working).

I use your search quite often, except if I know exactly the search-value is in the filename, then I use the native "file-Finder" - App.

[Jingo]

Thanks Markus.. always nice to hear that a tool I've written provides some use for others as well.  When I have some time (!), I would like to revisit the search and not only make it faster, but add some simple items such as this to the program.  Thx again for the feedback.. and stay tuned!

[thrinn]

At least...

Just downloaded your Fancy Search App. Nice, fast and easy to use! Well done!

[Jingo]

Thx Thorsten... glad you like it!  I was thinking of adding some features to it over the next few weeks... so hopefully I'll be able to get to it!  Enjoy... Andy.

[sinus]

Thx Thorsten... glad you like it!  I was thinking of adding some features to it over the next few weeks... so hopefully I'll be able to get to it!  Enjoy... Andy.

Would be cool, to add some features.
I just now made a short test and to my astonishment your app was even quicker than the search with the file-window.
I searched over all and the search-expression was

interlaken OR Linda

I have in the DB 301.506 files, means quite a lot.
The founded results was 376 files.

With your "Fancy Search" it took 2 Minutes and 34 seconds (File Window search 12 seconds longer).

This is quite a long time, but first I have quite a big DB and the search-term was also a kind of regex.
The real beauty of your App is, that I can start wherever I want and the result will be in a result-window.
This is cool. If I add the time for a file-window-search, where I have first leave my place (what is not good) and go to a place with all images, then the time for this native search is even far longer.

Hence I like your App.
If I want search only for filename, well, in this case the native App "File Finder" is very, very quickly and not to beat.

If you fine one day the time to add some features, here are my thoughts, just a kind of "mindstorm":

1) displaying the time for the search would be nice, because then I could see "ah, 35 secondes past, this will still go some time".
2) if the result is 0, then some display of this result would be nice (like "no files are founded" or so)

3) To make the search quicker, maybe some kind of radio - buttons or something like this to choose, where to search. In the old days of visual basic I had once such a thing, I made then some choosing for me, what I could simply click on and the search-time of course benefitted of this.
And, btw, some kind of buttons, what I can see at once, is far better than some drop-down-fields, because I can see on one sight, what values are currently choosen.

I added, as I can remember, some points like

- search only in the filename
- search only in keywords
- search only in the headline, description (because I use this most and I guess, a lot of users also)
- search only headline, description, keywords
- search only in filename, headline, description, keywords
- serach over all (like you have it now)

But Andy, you know, this are only proposals and maybe some other features, what you have in mind, are fare better.
At least, I can only repeat, the most cool thing is, I can stay in IMatch, where I am, choose your app, hit a value and let run.
The results will be shown in a new window, voila, simply cool.

My 2cents.

[Mario]

Wow, 2 minutes. That is quite slow.

For a 700,000 files database, with

+ Search File Names
+ Search Keywords
+ Search Title/Desc/...
+ Advanced Search On
+ regexp off

searching for word AND other word  on the Database node:

takes about 20-24 seconds for the first search (search index cold, not loaded yet) and then 8-10 seconds for all following searches. I timed this with my smart phone ad-hoc, so times may be off a sec.

Your mileage will vary, depending on the performance of your computer and how much data really is in the database for the fields you search.
But 2 minutes is quite a lot.

[sinus]

Wow, 2 minutes. That is quite slow.
...

Thanks for your test-searching.
Yes, you are right, this is a long time here on my db.

Even more bad is, that if I be burried deep in a category or some subfolders, maybe "hide stacked files" on, I have to go first on a node with all files.
And this takes time, even with default file window layout.
And I have to unstack the file-window.

That is the reason for me, because I use the fancy search-app.
Simply very convenient, I stay where I am, open the app and here we go.

Like the very good and same convenient "File Finder-App".
But unfortunately this App cannot search for words in the metadata.

Luckily I have not to search very often, that is why the search-time bothers me not that much. 

[Mario]

I just selected the Database node in the 700,000 files database.

[Jingo]

Thx Thorsten... glad you like it!  I was thinking of adding some features to it over the next few weeks... so hopefully I'll be able to get to it!  Enjoy... Andy.

Would be cool, to add some features.
I just now made a short test and to my astonishment your app was even quicker than the search with the file-window.
I searched over all and the search-expression was

interlaken OR Linda

I have in the DB 301.506 files, means quite a lot.
The founded results was 376 files.

With your "Fancy Search" it took 2 Minutes and 34 seconds (File Window search 12 seconds longer).

This is quite a long time, but first I have quite a big DB and the search-term was also a kind of regex.
The real beauty of your App is, that I can start wherever I want and the result will be in a result-window.
This is cool. If I add the time for a file-window-search, where I have first leave my place (what is not good) and go to a place with all images, then the time for this native search is even far longer.

Hence I like your App.
If I want search only for filename, well, in this case the native App "File Finder" is very, very quickly and not to beat.

If you fine one day the time to add some features, here are my thoughts, just a kind of "mindstorm":

1) displaying the time for the search would be nice, because then I could see "ah, 35 secondes past, this will still go some time".
2) if the result is 0, then some display of this result would be nice (like "no files are founded" or so)

3) To make the search quicker, maybe some kind of radio - buttons or something like this to choose, where to search. In the old days of visual basic I had once such a thing, I made then some choosing for me, what I could simply click on and the search-time of course benefitted of this.
And, btw, some kind of buttons, what I can see at once, is far better than some drop-down-fields, because I can see on one sight, what values are currently choosen.

I added, as I can remember, some points like

- search only in the filename
- search only in keywords
- search only in the headline, description (because I use this most and I guess, a lot of users also)
- search only headline, description, keywords
- search only in filename, headline, description, keywords
- serach over all (like you have it now)

But Andy, you know, this are only proposals and maybe some other features, what you have in mind, are fare better.
At least, I can only repeat, the most cool thing is, I can stay in IMatch, where I am, choose your app, hit a value and let run.
The results will be shown in a new window, voila, simply cool.

My 2cents.

Thanks Markus - these are great ideas and it gives me a good push to get back into this again and make it even better!

The biggest plus for me is the apps ability to search without altering the display and being able to choose the scope.  Stay tuned for some changes... it may take a bit but now I'm interested in getting back into it!

[sinus]

....

Thanks Markus - these are great ideas and it gives me a good push to get back into this again and make it even better!

The biggest plus for me is the apps ability to search without altering the display and being able to choose the scope.  Stay tuned for some changes... it may take a bit but now I'm interested in getting back into it!

Andy, awesome, I have just realized now, when I read this old thread and my "wishes" ... finally you realized now all!
Very cool, indeed.

[BanjoTom]

The latest version of "Fancy Search" is awesome, amazing, and blazing fast!   It will surely be an indispensable tool for me...  THANKS, Andy!

[Jingo]

Thx Markus/Tom - funny to see these items were the same requests from earlier.. glad they could finally be fulfilled! 

Which leads me to some housekeeping in this thread - Fancy App has been enhanced!  Here are the changes:

1 - The ability to restrict your search to multiple fields across XMP groupings!
2 - The ability to load and save search fields across sessions... now, with 2 clicks, you can run a search across your favorite group of fields!
3 - A timing and results bar was added so you can see how long your search took and how many images were found/searched.
4 - The routine is MUCH faster now when restricting to any field or set of fields... of course, things speed up even more after the first search and the buffering magic within IMatch! 

My test searches have shown a COLD run of 0.48 seconds to run a search on 10 XMP fields across 87063 files!  I hope others are seeing similar results.. the key is to restrict down to a core set of fields.. even 30 or 40 of them will speed things up tremendously!  Without selecting key fields - the same search across ALL fields takes 28.67 seconds.. still not bad.. but 28 seconds FASTER if I only search say DC:Subject, Headline, Description, Location tags, etc.

Always happy to further enhance the routine if other suggestions are added.... to use the new features, please download the FancySearch zip file below and overwrite the files in your C:\ProgramData\photools.com\imatch6\webroot\user folder... the app will then reload and the new features will be available to you.

ENJOY!

Updated APP Screen:



Timing and Results bar will appear after the first search is run:

[ubacher]

Tried it - and had difficulty in selecting tags. Could you give us some examples how to get the tags
into your app. Thanks.

[Jingo]

What difficulties are you having?

To get tags, you just:

1- Type a search term like Subject into the "Restrict Tags to Search" text box and then
2- Hit the Find Tags button...
3- Select the tag or tags from the result list and
4- Click the Add Tag to List button. 

Rinse and repeat to select as many tags as you would like to search through... the tags appear in the Tags to be searched box below.  When your list is ready, just type a search term into the Metadata Search box and hit the Search button.

Once you have a completed list of tags, I recommend click the Save Search Tags button to remember the list for the next app use... that way, you can just Load the Search Tags, type in a search term and hit Search button.. easy!

Hope this helps!

[sinus]

ubacher,
at first I clicked at the very beginning on the "Find Tags"-button (what is in Jingos image above labeled as 2), and then nothing happend.
Maybe this happend also to you.

But you have first to entr a search-term for tags, like headline or city, and then it works super.

But Andy explained it very good above.

[Jingo]

Hmm.. I probably should be inactivating the buttons until the search terms/tag info is added.. maybe in the next version!