Recursive list of categories

[Carlo Didier]

I thought this should give me the list of all categories under the specified path, but it only lists the first level. It's the syntax from the webservices doc app.

"http://127.0.0.1:50519/v1/categories?auth_token=&path=WHY|EVENTS&recursive=true"

The doc says it only works when either the field "children" is requested or all fields. This returns all fields, so it should work.

[Mario]

Works here as designed. When I use one of the standard IMatch categories:

Code: [Select]

IMWS.get('v1/categories',{
    path: 'IMatch Standard Categories|Image Files',
    recursive: true
}).then(function(response) {
    console.log(JSON.stringify(response));
});

I get "Image Files", "Image Files|Lens", "Image Files|ISO" etc. plus all children, recursively. It's easy to see in the debugger output (see attachment). For the image I reduced the number of requested fields to id,path,children to make it readable. It usually makes no sense to request all category data. This consumes a lot of memory in the browser and bandwidth.

[Carlo Didier]

As shown in my post, I'm not using javascript  but a simple http request (from powershell, but that's irrelevant).

Try this on any category with subcategories:

Code: [Select]

((Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&path=WHY|EVENTS&recursive=true").Content | convertfrom-json).categories | ft name,path -AutoSizeIt will only return the starting category, ignoring the recursive option.

What works is this:

Code: [Select]

((Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&regexp=path,WHY|EVENTS&recursive=true").Content | convertfrom-json).categories | ft name,path -AutoSize
But the first version should also work the same way. Specifying the id instead of the path doesn't help either.

[Mario]

I only test JavaScript, Python, None.js. Maybe it's PowerShell that's the problem. Try CURL.

[David_H]

As shown in my post, I'm not using javascript  but a simple http request (from powershell, but that's irrelevant).

Try this on any category with subcategories:

Code: [Select]

((Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&path=WHY|EVENTS&recursive=true").Content | convertfrom-json).categories | ft name,path -AutoSizeIt will only return the starting category, ignoring the recursive option.

What works is this:

Code: [Select]

((Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&regexp=path,WHY|EVENTS&recursive=true").Content | convertfrom-json).categories | ft name,path -AutoSize
But the first version should also work the same way. Specifying the id instead of the path doesn't help either.

You need

Code: [Select]

((Invoke-WebRequest -Uri "http://127.0.0.1:49152/v1/categories?auth_token=&path=Events&recursive=true").Content | convertfrom-json).categories.children | ft name,path -AutoSize

Note the object path is categories.children not just categories - your second one works because your regex will match the category wherever it is (so long as its in WHY|EVENTS), but they'll all be reported as top level (hence showing up in categories, but not children).
Your query will also be considerably quicker if you add &fields=id,name,path,children otherwise you'll bring everything back...

[Mario]

curl also works

curl -o c:\temp\categories.json.gz "http://127.0.0.1:50519/v1/categories?path=IMatch%20Standard%20Categories%7CImage%20Files&fields=id%2Cpath%2Cchildren&recursive=true&auth_token=

produces a gzipped file named c:\temp\categories.json.gz, including all child categories recursively. Please double-check your path pattern and make sure that it actually works. Just unzip to get the results (IMWS is returning GZipped responses for all non-trivial responses).

[Mario]

To close this:

I have created a WHY|EVENTS category in my database with several children. I get the proper results using JavaScript, Python and curl.

[David_H]

Note the object path is categories.children not just categories - your second one works because your regex will match the category wherever it is (so long as its in WHY|EVENTS), but they'll all be reported as top level (hence showing up in categories, but not children).
Your query will also be considerably quicker if you add &fields=id,name,path,children otherwise you'll bring everything back...

You may also need to recurse the child categories you get back, so the powershell would look something like (note, my powershell is rubbish) :

Code: [Select]

function walkCategory($cat) {
 $cat
 foreach ($c in $cat.children) { walkCategory($c) }
}
#get all the categories
$cats = ((Invoke-WebRequest -Uri "http://127.0.0.1:49152/v1/categories?auth_token=&path=Events&recursive=true&fields=id,name,path,children").Content | convertfrom-json).categories

# now walk them
walkCategory $cats | ft name,path -AutoSize

Output:

Sports                                 Events|Sports
Netball                                Events|Sports|Netball
0 - Team Photos                        Events|Sports|Netball|0 - Team Photos
01 - Tournaments                       Events|Sports|Netball|01 - Tournaments


Mario - Carlo's getting the correct results (with the path query), he's retrieving a list of categories (X[]) back from IMWS, but he's only selecting the name and the path from the top level (x[0], x[1], etc); for the path query there will only be one response (x[0] and as you've shown, lots of children) which isn't being handled correctly - Carlo needs to process x[0].children[] (and potentially x[0].children[0].children[0].....). The regex query works because everything matches and comes back in the top level array x[].

[Carlo Didier]

I only test JavaScript, Python, None.js. Maybe it's PowerShell that's the problem. Try CURL.

Maybe, but this is not a Powershell thing, but an IMWS thing as it's the http request that doesn't work as expected (and obviously different than javascript which is at least an inconsistency). Even a simple request without any conversion, like

Code: [Select]

Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&path=WHY|EVENTS&recursive=true"won't work recursively. Obviously, Powershell can return the recursive list when IMWS returns it, as with the regex query.

From your documentation it should work as I used it, whether you tested it or not.
The workaround with CURL just complicates a simple thing unnecessarily (apart from the fact that I don't even know what it is and how it works).

[Carlo Didier]

You may also need to recurse the child categories you get back, so the powershell would look something like (note, my powershell is rubbish) :

Code: [Select]

function walkCategory($cat) {
 $cat
 foreach ($c in $cat.children) { walkCategory($c) }
}
#get all the categories
$cats = ((Invoke-WebRequest -Uri "http://127.0.0.1:49152/v1/categories?auth_token=&path=Events&recursive=true&fields=id,name,path,children").Content | convertfrom-json).categories

# now walk them
walkCategory $cats | ft name,path -AutoSize
and comes back in the top level array x[].

Well that should not be necessary as the "recurse" option in the query should do exactly that, right?

[David_H]

You may also need to recurse the child categories you get back, so the powershell would look something like (note, my powershell is rubbish) :

Code: [Select]

function walkCategory($cat) {
 $cat
 foreach ($c in $cat.children) { walkCategory($c) }
}
#get all the categories
$cats = ((Invoke-WebRequest -Uri "http://127.0.0.1:49152/v1/categories?auth_token=&path=Events&recursive=true&fields=id,name,path,children").Content | convertfrom-json).categories

# now walk them
walkCategory $cats | ft name,path -AutoSize
and comes back in the top level array x[].

Well that should not be necessary as the "recurse" option in the query should do exactly that, right?

No - thats the bit that makes the children field get populated (which is why it must have all fields returned or the children field; otherwise its not going to do anything). You don't get a flattened result which is what I think you were expecting.

[Mario]

IMWS is working perfectly correct. It returns a hierarchical JSON object, with recursively nested children objects as requested with the recursive=true option.

I took the time to paste two screen shots of the IMWS response in a debugger to show you how the structure returned by IMWS Is formatted. It's a perfectly correct JSON object. Check them out again.

Your PowerShell code is not dealing with this correctly as it seems. But that's nothing IMWS can fix.
You need to make your PowerSheel code handle recursion correctly when parsing the JSON response. It's not a flat list you are getting.

[Carlo Didier]

It's not a flat list you are getting.

Ah, that's the (undocumented) clue ... Out of habit from other situations, I interpreted "recursive" as just that, not as hierarchical. In every case I used something recursive until now, the result was a flat list of things from a recursive "search".

[Carlo Didier]

You don't get a flattened result which is what I think you were expecting.

Of course I expected that (out of habit). I supposed the "recurse" option would do the work for me. If I have to walk through the whole hierarchy myself anyway, I might as well leave the "recurse" option off. It's pretty much useless.

[David_H]

You don't get a flattened result which is what I think you were expecting.

Of course I expected that (out of habit). I supposed the "recurse" option would do the work for me. If I have to walk through the whole hierarchy myself anyway, I might as well leave the "recurse" option off. It's pretty much useless.

That depends on what you're bringing back though....

If you've got something that looks like :
Events
 | Sports
 | | Netball
 | | | Tournaments
 | | | Team Photos

And you just query Events (without recurse), you'll just get back Events (and Sports will be the ONLY child category WITH NO DESCENDANTS).
If you query it with recurse, you'll get the lot.
If you use the regex, you'll get the lot anyhow (and you might want to avoid the recurse there, otherwise you'll get the lot back in the children too....)

I generally just load the category tree once from the top (Id, name, path, children) at the start of my code and work from there; it all depends on what you're using it for though 

[Mario]

Ah, that's the (undocumented) clue ...

Don't make assumptions. Just look at the actual response you get from IMWS (or any other web service). This takes out all the guesswork and will save us all a lot of time.
If you just type your URL (from your initial post) into your browser and then use the built-in developer tools, you can see the response, analyze it, browse it etc. Very comfy, no programming required at all. Your browser even formats it nicely and shows you the formatted and the raw response. See my screen shots.

[Carlo Didier]

I generally just load the category tree once from the top (Id, name, path, children) at the start of my code and work from there; it all depends on what you're using it for though 

Me too. In the case at hand, the regex solution works for me (as long as there are no sublevel combinations that match "WHY|EVENTS" of course ... then I'd have to include the '^' at the beginning of the regex, oh what's the %... code for that again  ...)

[Carlo Didier]

Aaaaaarghhhh!

One more annoying inconsistency: although in the fields list it's named "description", in a regexp filter the SAME field has to be referenced as just "desc" ...

This would be logical but doesn't work:
Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&regexp=description,^D[0-9]{11}-D[0-9]{11}&fields=id,name,description,path,children&recursive=true"

But this works (an example in the IMWS doc shows this, that's how I found it out):
Invoke-WebRequest -Uri "http://127.0.0.1:50519/v1/categories?auth_token=&regexp=desc,^D[0-9]{11}-D[0-9]{11}&fields=id,name,description,path,children&recursive=true"

[Mario]

This is no inconsistency. The documentation is pretty clear:

A regular expression term. Either path,term (search term in path: path,beach.*) or desc,term (search term in description: desc,hello world). The syntax is the same as for the @Category[] category formula.

desc happens to have the same length as path, which is extra neat.

Don't make assumptions. Your mental model must not necessarily be reflected by the IMWS REST endpoint model which is not pretty much stable for almost two years.

[Carlo Didier]

This is no inconsistency. The documentation is pretty clear:

A regular expression term. Either path,term (search term in path: path,beach.*) or desc,term (search term in description: desc,hello world). The syntax is the same as for the @Category[] category formula.

desc happens to have the same length as path, which is extra neat.

Don't make assumptions. Your mental model must not necessarily be reflected by the IMWS REST endpoint model which is not pretty much stable for almost two years.

Sorry, but this is inconsistent, even if the doc says that's the way it is. And to have it called "desc" just because that neatly has the same length as "path" is just irrational. It's not logical, not intuitive and totally inconsistent. You could as well have called it "comment" then.

[start rant]But that's in line with the incomprehensible fact that things are named in all lowercase on one side and in camelCase on another side, even when they represent the exact same data (which wouldn't be a problem if the irrational case sensitivity was out of the way), so maybe there is consistency in the inconsistency here ...[end rant]

As for the doc, you might consider adding something to stress that regexp filters can only be used with the path or desc(ription) fields. It's a limitation that's not immediately clear, I think. To me it looked like those were just two examples but that I could also use it for example on the "name" field (or other fields for other objects than categories).

[Mario]

Sorry, but this is inconsistent, even if the doc says that's the way it is.

Then it's not. My design decision.

And to have it called "desc" just because that neatly has the same length as "path" is just irrational

Not really. And there is also a length limit for query strings, so keeping parameter names short leaves more room for payload.

But that's in line with the incomprehensible fact that things are named in all lowercase on one side and in camelCase on another side,

Also a design decision I've made two years ago. Using all lower-case parameters makes the API language agnostic and works best across all platforms and programming languages.
The returned JSON of course follows the camelCase notation that is standard for JSON, especially in web environments.
When IMWS would return XML data, it would use dashes instead of camelCase. Because that's also standard.

It's really not that hard. Just remember once that parameters are always lower case, and the returned JSON responses are standard.

I'm already implementing IMWS 2, which uses slightly different and 'current' (aka 2019) naming conventions. But that version will not be used in the IMatch desktop context so you'll probably never have to 'face' it.

[Carlo Didier]

Sometimes I wonder where you get those 48 hour days to not only maintain iMatch but also to constantly learn new stuff to integrate the latest technologies 

I'm right now at 11 hour workdays and have very little time to do anything else. Fortunately I'll get a break from Dec 20th to Jan 2nd, but from there it will be at least another 3 months of 11 hour work days