Author Topic: Face Detect: File, Photo, Face Orientations  (Read 418 times)

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Face Detect: File, Photo, Face Orientations
« on: May 24, 2020, 01:33:19 AM »
ADMIN: Attachments of persons removed. Please do not post images with recognizable persons in public forums. This will violate laws in many countries, including Germany, where this community is hosted.

I suspect the face detection algorithms have problems with file orientations and detection orientations – especially when the user changes the face detect to an angled box. File 20180811_203704.jpg is my example source.

File AliceRohrer01 shows a confirmed face as black for a file visible in the file viewer which appears as expected orientation.
In Face Manager, selecting the black face file, and doing View is shown in file AliceRohrer02. Note that the image is rotated 90 degrees from expected and the face detection box is not on the face.

Viewing the suspect file in the Viewer shows correct image orientation and the Face box on the face, albeit rotated. This is attached image AliceRohrer03.

I suspect I have problems with the system detecting and identifying this face due to confirmed faces which apparently are not really faces. This is the first problem.

This first problem then seems to create a bigger mess across the Face portions of the database. The second problem is that trying to fix the above lost confirmed other faces. Just prior to working on the above, I had all people with status No Unconfirmed Faces in my database. I had managed zero Unknown Persons. Then I went to work on the above face detection by deleting the face box and restoring. Somewhere after my playing, Alice had 21 Unconfirmed Faces (which were not her) and Unknown Persons category had 294 listed.

So, trying to fix one faceless entry for one person has affected the status of many people.

A Database Diagnostic at this point has 1 warning:
Checking Annotation Objects:
      Containers:  2,082
Completed.
Checking Event Graph: Events:
    0 events loaded.
Checking People Graph: Events:
PersonManager Stats:
    Files with faces: 3232
    Persons: 119
    Faces: 3232
    Confirmed: 2854
    Unconfirmed: 21
    Unassigned: 357
    Warning: Ignored face records with confirmed/train/ignored state or with an entity oid. Corrected.
    Files: 3232
    Faces: 10468
    Entities: 8748. Confirmed: 8748
    Entity keywords: 0.
    Vector data size: 7530KB
    Thumbnail data size: 122154KB
    Annotations without face data: 482KB
Completed.

After the Diagnostic repair, I again have all people with status No Unconfirmed Faces in my database. However, I now have 305 Unknown Persons (started with zero).

Attached are the screen shots referenced above.

I was afraid to downsize the actual problem image file for fear the metadata would get changed as related to any possible orientation information. Since the file is too big for the forums, the source can be found at: https://drive.google.com/open?id=1dirGnfxMbuDtaEyD55dPnZIDs7Nzb0np

IM v2020.5.4 in use.
« Last Edit: May 24, 2020, 09:25:58 AM by Mario »

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 25817
Re: Face Detect: File, Photo, Face Orientations
« Reply #1 on: May 24, 2020, 09:28:11 AM »
I'm not sure that I understand the problem? you are describing.

IMatch sends the images to the AI for detection and rotates the cache images before as needed.
If you manually change the face annotation rectangles in the Viewer (?) you may prevent the AI from detecting faces. Resizing the rectangle usually solves the problem. This is not always 100% 'logical' and just how AI technology works. The AI sees images different than you and figuring out what a face is from a mass of 40 million pixels is really tough.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #2 on: May 24, 2020, 01:20:53 PM »
There ar a couple of problems here that I see....
1) Pictures AliceRohrer02 and AliceRohrer03 show views of face detect boxes and don't agree on the location of the face. Perhaps there is bad source data in the image source file for height/width/rotation and image characteristics that cause bad location calculations in some face detect formula.

2) When working with issue above, many previously confirmed face names get lost (returns to "?" tag) and the diagnostics report a warning that gets fixed. The diagnostic fix does indeed fix some face references (new unconfirmed gets fixed to be confirmed), generally for the person being worked on. The diagnostic seems not to fix, perhaps does make worse, other face references (I keep getting more and more Unknown Persons). What I don't know is if the above file and face detect concern is indeed related to this confirmed/unconfirmed face issue or if this issue is just a fact of working in the database with faces. I have worked for hours to tag and confirm faces and get my Unknown Persons list to quantity zero - only to go off and work on problem above and return to hundreds of entries in my Unknown Persons list. Seeing an increase of unconfirmed faces in a person reference or an increase of Unknown Persons indicates to me that the database is damaged and Diagnostics need run again.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 25817
Re: Face Detect: File, Photo, Face Orientations
« Reply #3 on: May 24, 2020, 02:40:31 PM »
Since the images were deleted for privacy reasons, I cannot comment on this.

Where these files created by an Apply device, by chance?
Apple devices detect faces and record them in the XMP record, but they mess up the coordinates (Apple knows about this) and this causes the face rectangles to be wrong when IMatch imports the XMP metadata.

Quote
many previously confirmed face names get lost (returns to "?" tag

Under which condition would that happen. Confirmed faces are locked to the face and a no longer touched by the AI.
Do you use the latest IMatch version? Important.
Do you manually move around face annotations in the Viewer? Do you copy & paste faces?  Do you use the annotation gallery?

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #4 on: May 24, 2020, 04:59:15 PM »
According to the metadata the image was captured on a Samsung Galaxy Note-5 (SAMSUNG-SM-N920A). I have no idea if/what software may have changed/edited orientation information. I likely got the image sent to me via Email (not known for sure anymore). I may or may not have done orientation changes in IM or Photoshop using various tools or settings. So any photo manipulations are not well known.

The IM Viewer looks as expected with correct image orientation and face detect box on the face. The image face is angled and the detect box was likely manually rotated to the angle of the face. I'm playing with the system and only been working with the People tools for a couple days.

In the IM Face Manager the image is 90 degrees rotated to the left. It has a different shape face box and it is not on the face. The box is not angled adjusted but is square with the picture.

I have maybe three files of this photo session that I think do similar things. I don't know if the manual angle of the face detect box is significant to the problem. I don't know if maybe the saving of unique face box data is related to other files getting changed - perhaps a data container storage problem within the database.

It will take me some time to fix (again) my 303 files with Unknown Persons. Maybe I'll then play again with this file and keep track of what step sequence is in play and when I note database problems. A zip file of the sample image and my screen grabs will be too big for Email. (Thanks for your steps in identity protection of my erroneous public face posting). Shall I provide a ZIP file via my dropbox?

What are you calling the "image gallery"? The screen labeled Face Manager and called by the smile face icon? I am learning my way around the all/confirmed/unconfirmed/trained/rejected faces and tools.

I am using iMatch 2020.5.4.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #5 on: May 24, 2020, 09:43:31 PM »
I now suspect the Face detection problem is not a trigger for the Rejected/Unconfirmed database problems.

Attached is a log of my wondering through the Face Manager Unconfirmed/Rejected faces operations, while checking status and running diagnostics. There are some weird results. The results may be explained away if the face AI system is in play for each Reject or Unconfirm operation. If the AI is not involved then the conversion logic and numbers of faces converted just don't make sense in some conditions within my database. I hope Mario can follow my log and see my operations and concerns. Is there anything else I can do to help troubleshoot?

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 25817
Re: Face Detect: File, Photo, Face Orientations
« Reply #6 on: May 25, 2020, 11:40:20 AM »
Sorry, I don't open Office documents from the 'outside' for security reasons.
Is this an IMatch log file or something else? IMatch log files are plain text.

If you have an image and steps which produce the problem you are seeing, feel free to send the image and the repro steps to support email address.
I can then try to reproduce this behavior. I'm at this point not 100% sure if I understand the problems you are experiencing.

When you confirm or reject a face, this imports all other unconfirmed/unassigned face.
Because this may change how the AI sees the person and this means it re-checks all 'free' faces (unassigned or unconfirmed) to see if it can find matches for the person. Or if another person now matches better.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #7 on: May 25, 2020, 02:16:40 PM »
Attached is a -plain text- manual record of my steps through the Face Manager Unconfirmed/Rejected faces operations, while checking status and running diagnostics. There are some weird results. The results may possibly be explained away if the face AI system is in play for each Reject or Unconfirm operation - though I'm not fully convinced. If the AI is not involved then the conversion logic and numbers of faces converted just don't make sense in some conditions within my database. I've commented on what I did as well as where the results came back unexpected. The detail in the text file emphasizes my following overall concerns.

1) IM operators can certainly be allowed to do some stupid stuff in Face Manager (i.e. Reject a face that is really that person). But an IM operator should never be able to do something that damages the database and requires running a diagnostic to fix it. I have numerous sequences in the text of doing an operation and getting unexpected results. Running the diagnostic fixes a warning and that seems to fix my previous unexpected result.

2) I have cases near the end of the text where the Face Manager shows the spinning circle (working) and never terminates or comes to a conclusion. The processing is hung.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #8 on: June 12, 2020, 08:10:28 PM »
Trying to catch the database error sequence of the Unconfirmed/Reject operations. This time I have a log (unfortunately only in Normal Logging mode). Attached is my handwritten log of my steps. Unfortunately my attempts to repeat the database error failed while in Debug Logging. I'll make another attempt at catching the error. Also attached is the iMatch log file.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #9 on: June 12, 2020, 10:38:01 PM »
This post may have more complete data than my recent previous post. I have had some success in trapping an error sequence.

I have found a sequence, and made a repeat, of the face database corruption as is being fixed by Diagnostics. In this instance I am NOT doing Unconfirm/Reject operations. This seems a result of only a Face Detect and assignment operation which produces the wrong name applied to the face. I have attached full Diagnostic reports. I have attached a Debug level Log that covers two problem occurrences. I have attached my hand recorded sequence of steps.

Mario, I hope there is enough data to get you on track. I highly suspect this is all related to my unique elements, unique face data, and the organization of my particular database. Good Luck!

I'll try not to do anything in my database, especially related to faces, for a few days, in case you may need me to record more data or repeat or ... further debug.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #10 on: June 16, 2020, 02:38:08 PM »
Mario,
Any luck with the previous post? Is there anything I can do to help with the troubleshooting?

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 25817
Re: Face Detect: File, Photo, Face Orientations
« Reply #11 on: June 17, 2020, 05:44:28 PM »
I think this shall be covered by the bug fixed for the next release which deals with the "replace person of annotation" which could lead to unlinked annotations and diagnosis errors.
I have worked on that part (users copying & pasting, changing persons, dealing with the same person assigned wrongly to be similar faces, in the same image,...) and more. Lots of time-consuming detail work on my part.

Let me know if you can still reproduce this using your steps in the document. I could not.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #12 on: June 24, 2020, 10:24:21 PM »
This diagnostic error still appears in my test case after upgrade to IM v2020.6.2.

Checking Annotation Objects:
      Containers:  2,436
Completed.
Checking Event Graph: Events:
    0 events loaded.
Checking People Graph: Events:
PersonManager Stats:
    Files with faces: 2421
    Persons: 123
    Faces: 3930
    Confirmed: 3243
    Unconfirmed: 687
    Unassigned: 0
    Warning: Ignored face records with confirmed/train/ignored state or with an entity oid. Corrected.
    Files: 2421
    Faces: 3930
    Entities: 3929. Confirmed: 3243
    Entity keywords: 9676.
    Vector data size: 2843KB
    Thumbnail data size: 47923KB
    Annotations without face data: 159KB
Completed.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 25817
Re: Face Detect: File, Photo, Face Orientations
« Reply #13 on: June 24, 2020, 11:17:44 PM »
How do you make this diagnosis warning re-appear? I tried, but I cannot.
The exact steps you perform after the diagnosis to make the next diagnosis report the same error. This would be helpful.
« Last Edit: June 24, 2020, 11:22:48 PM by Mario »

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #14 on: June 25, 2020, 12:28:13 AM »
My reply #9 above (June 12, 4;38pm) is my best documented case. I can re-do it if you think the info might be different with this version of IM.

Unfortunately it took me quite a few hours of confirming faces and running diagnostics until I found this file and this situation that would produce the error. Face detect was working awesome through 4 new faces to the database, training, and picking them out, for about 260 files. Then I hit the problem file where 2 faces were correctly found, 1 face not detected, and 1 face was assigned the wrong name. I suspect it is the wrong name face (picked out correctly in many previous pictures) that is related to the problem but I certainly have no proof or even indecent data for my hunch. Perhaps the problem lies within the wrong name assigned to the face. I just don't know.

Even if I gave you the problem file and my complete database (~4.4GB backup file) I don't know if the access on your end would have enough similarities in the global face data that you could replay the error.

I have paused any work in my database regarding faces while we work on this problem. I've seen enough to expect more face work will change my ability to find a problem example file. I don't want to lose this test case. Let me know if there is some troubleshooting I can help with or if you think I can bundle some files for you to try on your end.

Mario

  • IMatch Developer
  • Administrator
  • *****
  • Posts: 25817
Re: Face Detect: File, Photo, Face Orientations
« Reply #15 on: June 29, 2020, 07:21:51 PM »
I've tried to follow your description from post #9.
After doing many things, you've

Found a file where Mark was identified as another person.
You changed the person assigned to the "Mark face" to Mark.
Two other faces in the same image remained unchanged (and unconfirmed).
No you run the diagnosis which reported one warning about an ignored face which has also train/confirm state. But you did not ignore a face, as far as I can tell from your description, correct?

You now re-run face detection for the same image, with the corrected Mark face. This removes the information about the face from the AI so it cannot longer know that this face belongs to Mark (if this was the only assigned Mark face so far). So it is normal that it assigns a person for which it has similar face data (Keith).
You write: Diagnostics = 1 Warning.

Both attached log files show a warning about an ignored face which is also marked as confirmed/rejected. But you did never ignore a face in the Viewer or Face Manager?

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #16 on: June 29, 2020, 08:45:09 PM »
my replies in Red - as best I can remember....

I've tried to follow your description from post #9.
After doing many things, you've

Found a file where Mark was identified as another person. Yes.
You changed the person assigned to the "Mark face" to Mark. No. I left it with the wrong name and checkmark visible.
Two other faces in the same image remained unchanged (and unconfirmed). AI must have been very confident in those faces. They were already confirmed (I did not have a checkmark to click). I left them alone. Faces were correctly named.
No you run the diagnosis which reported one warning about an ignored face which has also train/confirm state. But you did not ignore a face, as far as I can tell from your description, correct? I did not ignore a face in that file. I did not make any face modifications. I left it as produced by the F6 face detect.

You now re-run face detection for the same image, with the corrected Mark face. I did not correct Mark face. I deleted the whole Mark face reference box to remove the AI references that were wrong. This removes the information about the face from the AI so it cannot longer know that this face belongs to Mark (if this was the only assigned Mark face so far). Mark has been correctly named to face in a lot of other images in the folder. So it is normal that it assigns a person for which it has similar face data (Keith).
You write: Diagnostics = 1 Warning.

Both attached log files show a warning about an ignored face which is also marked as confirmed/rejected. But you did never ignore a face in the Viewer or Face Manager? Correct. I deleted instead of ignored. I tried to get People / Unconfirmed Faces to zero files.

I've just detected some more crazy actions I'll document and post in a bit.


StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #17 on: June 29, 2020, 10:24:21 PM »
Here is a slightly different set of steps and a new discovery.
- Ran Diagnostics and made sure they were clean.
- Opened the problem file (IN230234.jpg) in the viewer.
- The prior face assignments were visible. Deleted all three face assignments.
- Closed viewer.
- Diagnostics fixed one warning.
- Opened the problem file (IN230234.jpg) in the viewer. Face annotations are gone as expected (I deleted just a couple steps ago). I accessed the file from the Media & Folders view.
- Went to the People tab.
- The Unknown Persons entry shows 1 file (IN230234.jpg). When I open the file in the viewer, I find three faces detected. 2 faces are named correctly and have the checkmark confirmation available. Mark’s face is identified with a ? (question mark – in my previous examples he is identified with the wrong name). I make no changes and close the viewer.
- Diagnostics still pass.

!!! This is really strange. If I open the file in the viewer as coming from Media & Folders view, I see no faces detected. If I open the supposedly same file as coming from the People / Unknown Person view, the viewer shows annotated faces. Why is the viewer showing me different face info depending on from where I open the file?
If I open the file from Media & Folders, in the viewer, and see no face annotations, I can advance to another file in the folder and annotations are visible as expected. So this is not a viewer hide annotations setting. It seems file related.

- If I open the file (IN230234.jpg) from People / Unknown Faces, and in the viewer, delete all face entries, I can exit the viewer and then see People / Unknown Persons = 0 files.
- If I view the file from Media & Folders I see no annotations (expected).
- Diagnostics = no errors.

This last sequence, opening the file from Persons / Unknown Persons, and deleting all faces, may have fixed the errors and problems. From here, doing the following steps, I cannot recreate the database error, even doing sequences that previously were repeatable in the past couple of weeks.

- From Media & Folders I open the file (IN230234.jpg) in viewer. Run F6 Face Detect. Results in 2 correct faces named and with no checkmark to confirm (very confident AI). Mark’s face is labeled Keith and has a checkmark available. I make no changes. Close viewer.
- Diagnostics = 0 errors.
- People / Unknown persons = 0 files.
- Write Back Metadata on this one file.
- People / Unknown persons = 0 files.
- Diagnostics = 0 errors.
- So have I cleaned up something? I didn’t create the error on this pass. Let’s try the exact sequence from before.
- From Media & Folders, opened (IN230234.jpg) in viewer. Deleted the Keith annotation on Mark’s face. Exit viewer.
- People / Unknown Persons = 0 files.
- From Media & Folders, opened (IN230234.jpg) in viewer. F6 Face Detect / Remove Existing. Again: Results in 2 correct faces named and with no checkmark to confirm (very confident AI). Mark’s face is labeled Keith and has a checkmark available. I make no changes. Close viewer.
- People / Unknown Persons = 0 files.
- Diagnostics = 0 errors.
- It appears I have lost the ability to recreate the database error with this photo. No success with multiple varied attempts.

StanRohrer

  • Sr. Member
  • **
  • Posts: 279
Re: Face Detect: File, Photo, Face Orientations
« Reply #18 on: July 03, 2020, 10:06:59 PM »
Tried multiple ways to get the failures to return to my face data. Rescanned the folder (381 files) with the various Face Detect settings. Tried individual file edits. Worked from the Media & Folders view as well at the People view. Unable to recreate the errors. I'll quit holding this database and move on to more face work and see if I can find another case.