Author Topic: Unexpected behavior when formatting float variables  (Read 375 times)

Jean-Maetso

  • New Members
  • *
  • Posts: 26
Unexpected behavior when formatting float variables
« on: March 25, 2021, 07:28:51 PM »
Hi,
It might "not be a bug, just a feature" however it is clearly a kind of unexpected behavior.
No crash, not freeze, nothing in the log. I am using IMAtch 2017 on a 40,000 file DB.
Here is the trick:
  • I use the VarToy App to "see in direct" the content of variables. Typing the metadata variable for the aperture value of the camera, I enter : {File.MD.XMP::Exif\ApertureValue\ApertureValue\0|numformat:float,4,2}
    and the App displays 4.5 (four-dot-five) for my test picture
  • I create a data-driven category based on variables and enter as criteria the exact same variable: {File.MD.XMP::Exif\ApertureValue\ApertureValue\0|numformat:float,4,2}. The created categories are all formatted with comma decimal separator instead of point. In other words, the test picture is in category 4,5 (four-comma-five). Note the same formatting as the first case !!?

This is doubly confusing. All the more, how can I test the aperture value against another decimal number? How can I format something like {....Aperture|between:0,2.8,large aperture, normal aperture}? Because if I replace 2.8 by 2,8 in this code, the "between" function gest confused with all the commas...
Thanks.
JM

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #1 on: March 25, 2021, 07:47:50 PM »
Variables by default format numbers using the the number format configured for the current user. You may need to add a cast. Casting Variables

Note that support for IMatch 2017 has been discontinued years ago.

Jean-Maetso

  • New Members
  • *
  • Posts: 26
Re: Unexpected behavior when formatting float variables
« Reply #2 on: March 25, 2021, 09:09:26 PM »
The help says:
Quote
numformat:This function allows you to cast (convert) the contents of a variable into an int or float numerical data type
So I do not understand why you say I may need to add a cast... Numformat performs one already, ok?

The most surprising is the incoherence between the two cases I have described: with the very same formatting of the same variable of the same photo, the displayed decimal separator is not the same: I get a dot in the "VarToy App" and a comma in a data-driven category.... without changing anything anywhere else (of course)

I also noticed that Imatch 2017 is no longer supported... I am not complaining about that because I know your constraints. But I would be curious to know if the same behaviour is present in IMatch 2020 !

Best,

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #3 on: March 25, 2021, 09:13:46 PM »
Use the cast formatting function to inform IMatch that you are working with a floating point number. I have provided the link above.

Note that support for IMatch 2017 has been discontinued years ago.

Jean-Maetso

  • New Members
  • *
  • Posts: 26
Re: Unexpected behavior when formatting float variables
« Reply #4 on: March 25, 2021, 09:31:25 PM »
Nothing changed with the cast.  :(
In other words, with a test picture taken at F4,5:
The following string {File.MD.XMP::exif\ApertureValue\ApertureValue\0|cast:real}
  - if copied/pasted in the built-in App "VarToy", the output is 4.5 (with a dot)
  - if copied/pasted as the parameter of a data-driven category, the output is 4,5 (with a comma)

Once again, I would be curious to know if the phenomenon appears in the last version of IMatch.

Knowing that IMatch 2017 is no longer supported, should I move this post in a another part of the forum ?
Thanks !

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #5 on: March 25, 2021, 10:09:54 PM »
Sorry, it seems I'm not following. Very stressful days.
cast is to this problem, the output will of course be formatted using your local floating point format.

{File.MD.XMP::exif\ApertureValue\ApertureValue\0|numformat:float,2,2}

gives me the correct result in VarToy. Not in a data-driven category based on this variable. Strange.
Not sure what comes into effect here. Data-driven categories are complex.
I will look into this for a future update. Not sure this came ever up before so I have no idea at the moment. I need to test this when I work on the IMatch 2020 code base the next time.

In general: note that when you use ; or , in variables you might need to escape them with ~

Jean-Maetso

  • New Members
  • *
  • Posts: 26
Re: Unexpected behavior when formatting float variables
« Reply #6 on: March 26, 2021, 07:25:59 AM »
Don't be sorry Mario !
Thanks for checking this strange thing. Curious, isn't it ? And no doubt that data-driven category are complex stuff... they are so powerful !! No software provides even the beginning of it, seemingly.  :D

I have noticed the ~ escape (I had already used it for ; in other situations)

Have a very nice day,
Jim

sinus

  • Global Moderator
  • *****
  • Posts: 4342
  • IMatch-User since 2001 (IMatch 3.6)
Re: Unexpected behavior when formatting float variables
« Reply #7 on: March 26, 2021, 07:37:16 AM »
Don't be sorry Mario !
Thanks for checking this strange thing. Curious, isn't it ? And no doubt that data-driven category are complex stuff... they are so powerful !! No software provides even the beginning of it, seemingly.  :D

I have noticed the ~ escape (I had already used it for ; in other situations)

Have a very nice day,
Jim

This variable
{File.MD.XMP::exif\ApertureValue\ApertureValue\0|numformat:float,2,2}
gives me a value of one of my pictures of 6.30

And this here:
{File.MD.XMP::exif\ApertureValue\ApertureValue\0|cast:real}
a value of 6,30

Both in VarToy.
I can remember, always, if I had such problems, it had to do with the settings in Window, how to display numbers.
If I changed it, IMatch did also change.
Best wishes from Switzerland! :-)
Markus

Jean-Maetso

  • New Members
  • *
  • Posts: 26
Re: Unexpected behavior when formatting float variables
« Reply #8 on: March 26, 2021, 08:53:44 AM »
Dear Sinus,
Thank for taking the time to check on your system. Which version of IMatch are you using?

For VarToy: I confirm that I can obtain both formats (dot or comma) depending on the cast. NB: "cast:real" is not available in IMatch 2017.

What do you obtain by creating a data-driven category ? (tip: if you have a big DB and want to make it fast, put a well-chosen "discriminating" keyword/string (such a rare place or name...) in the field "Category filter" of the data-driven category dialog. This will restrict the scope on which IMatch create the data-driven category).

About your last point: yes, of course, I can change the global Windows settings. But I would like to avoid that... I am sufficiently confident that I can manage some turnaround in IMatch. And I think I will purchase the next version of IMatch... :-)

sinus

  • Global Moderator
  • *****
  • Posts: 4342
  • IMatch-User since 2001 (IMatch 3.6)
Re: Unexpected behavior when formatting float variables
« Reply #9 on: March 26, 2021, 12:26:45 PM »
Dear Sinus,
Thank for taking the time to check on your system. Which version of IMatch are you using?

I use version 2020.14.2
 
You can always use some replace-stuff, see my attachements.

sorry, the variable in the last attachement is cutted, here the variable:

{File.MD.XMP::exif\ApertureValue\ApertureValue\0|numformat:float,2,2;pereplace:,==.}
Best wishes from Switzerland! :-)
Markus

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #10 on: March 26, 2021, 01:43:52 PM »
The official aperture tag {File.MD.Composite\Exif-Aperture\Aperture\0}  or the corresponding shortcode {File.MD.aperture} both have the data type "unknown" and a value using a . and not a comma. Hard-coded. ExifTool fills the tag that way. IMatch treats them as text and does not reformat the value. So it just works.
I would recommend using this tag for your data-driven category. Can you try that?

The ApertureValue tag is marked a a floating point (real) variable by ExifTool, which makes IMatch format it using the decimal comma and thousand grouping symbol.

The cast:real function tells IMatch how to parse the variable value (as a floating point in this case). This is usually only useful when a tag is of unknown type, to make IMatch parse it in a specific format, e.g. treat text as an int or a floating point value. When casting to a floating point, the default numerical formatting for the current user is applied.
The documentation for cast is wrong, because it states that a cast would remove the 1000th grouping, which is not the case when the current locale uses a 1000th grouping symbol. I need to think about what is wrong, the result of the cast or the documentation. Changing this could break things for some users out there...

I don't have the IMatch 2017 source code anymore so I don't know if there is a difference in that old version.

The same is true for numformat. This function can be used to format a numerical value with a specific number of decimals. Many tags contain text in the form "123456.723872320" and numformat can be used to make this better suited for output. numformat also uses the locale and number format for the current user.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #11 on: March 26, 2021, 03:28:55 PM »
A {File.MD.XMP::exif\ApertureValue\ApertureValue\0|replace:~,==.}  should do the trick, replacing , with . before doing other operations like numcmp.

Note: While analyzing this I have identified an issue with parameter processing in IMWS. The == in the above variable could confuse the parser, thus causing the variable to fail when tested in the VarToy app.
Using the variable in IMatch features (e.g. for File Window layouts or data-driven categories) works correctly.


Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #12 on: March 26, 2021, 04:24:11 PM »
Another update: VarToy shows 5.0 and that is wrong.
When you use the variable in a data-driven category, a File Window Tip or layout etc., 5,0 is emitted when your local number format uses a decimal comma.

If you don't want to use your local number format, use replace to change the comma as needed.


It seems that one of the runtime or Windows routines used for formatting works differently when being used in the IMWS context. Intriguing. I will investigate further as time permits.

Update: Stupid me. This is the expected behavior, because IMWS of course always formats response data in a standardized locale. Which is usually correct, except for the particular case of VarToy and variable results using numbers with decimal comma / 1000th groupings. Irritating. Not sure if changing this would be worth the effort...
« Last Edit: March 26, 2021, 04:39:52 PM by Mario »

Jean-Maetso

  • New Members
  • *
  • Posts: 26
Re: Unexpected behavior when formatting float variables
« Reply #13 on: March 26, 2021, 09:29:27 PM »
Thanks for all these explanations & checks !
Using escape (with ~) or the replace/pereplace, I had my stuff done.
 8) 8)

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 28532
Re: Unexpected behavior when formatting float variables
« Reply #14 on: March 27, 2021, 08:30:42 AM »
I have introduced a new parameter for the IMWS endpoints dealing with variables for IMatch 2021.
On my side, this affects only VarToy, which is now consistent in how numerical variables are formatted with the rest of IMatch.