[BUG] Unknown inidividuals aren't being marked as unknown

[clowncracker]

Describe the bug
Unknown individuals are being marked as matches with low confidence instead of unknown. Example:

Random person walks to the camera, it's identified as casey with a 7.48% certainty. This should be identified as unknown.

Version of Double Take
1.13.1 in Home Assistant

Expected behavior
I expect anyone not matched to be considered unknown.

Screenshots
Screenshot of issue

Hardware

• OS: Home Assistant Core 2023.6.2

Additional context
Add any other context about the problem here.

[townsmcp]

@clowncracker please add your full DT configuration. Without it no one can help you

[tyanai]

I also having the same exact problem.

Whenever the image is unknown, I'm seeing this in the doubletake log:

23-06-25 16:10:08 info: done processing reolink_cam: 1687709372.470719-uovc1f in 35.49 sec
23-06-25 16:10:08 info: {
id: '1687709372.470719-uovc1f',
duration: 35.49,
timestamp: '2023-06-25T16:10:08.472Z',
attempts: 22,
camera: 'reolink_cam',
zones: [ 'Entrance_porch' ],
counts: { person: 0, match: 0, miss: 0, unknown: 0 },
matches: [],
misses: [],
unknowns: []
}

Frigate Config:

mqtt:
host: 192.168.1.200
user: admin
password: XXXX
go2rtc:
streams:
reolink_entrance:
- http://192.168.1.231/flv?port=1935&app=bcs&stream=channel0_main.bcs&user=admin&password=XXXX
- "ffmpeg:reolink#audio=opus"
reolink_entrance_sub:
- http://192.168.1.231/flv?port=1935&app=bcs&stream=channel0_ext.bcs&user=admin&password=XXXX

cameras:
reolink_cam:
ffmpeg:
inputs:
- path: rtsp://127.0.0.1:8554/reolink_entrance?video=copy&audio=aac
input_args: preset-rtsp-restream
roles:
- record
- path: rtsp://127.0.0.1:8554/reolink_entrance?video=copy
input_args: preset-rtsp-restream
roles:
- detect
motion:
mask:
- 1280,0,1280,720,699,720,569,398,744,82,620,0,258,108,0,583,0,0
zones:
Entrance_porch:
coordinates: 601,496,692,720,30,720,249,258,339,512
objects:
- person
Entrance_pathway:
coordinates: 483,124,648,483,333,495,228,142
objects:
- person
Entrance_before_gate:
coordinates: 266,146,789,172,642,44,380,69
objects:
- person
objects:
track:
- person
mask:
- 1280,0,1280,720,699,720,569,398,744,82,620,0,258,108,0,583,0,0
snapshots:
enabled: true
bounding_box: true
retain:
default: 2
required_zones:
- Entrance_porch
- Entrance_pathway
- Entrance_before_gate
mqtt:
timestamp: true
bounding_box: true
crop: true
quality: 100
height: 1500
record:
enabled: true
retain:
days: 7
mode: motion
events:
retain:
default: 14
mode: active_objects
objects:
dog: 2
car: 7

live:
stream_name: reolink_entrance
height: 1920
quality: 8
birdseye:
enabled: True
mode: continuous
ffmpeg:
output_args:
record: preset-record-generic-audio-aac

DoubleTake Config:

frigate:
url: http://192.168.1.200:5000
update_sub_labels: true

min_area: 0

labels:
- person

attempts:
latest: 22
snapshot: 0
mqtt: false
delay: 1.25

image:
height: 1500

mqtt:
host: 192.168.1.200
username: admin
password: Gazoz010
topics:
frigate: frigate/events
homeassistant: homeassistant
matches: double-take/matches
cameras: double-take/cameras

detectors:
compreface:
url: http://192.168.1.200:8000/
key: XXXX
det_prob_threshold: 0.7

telemetry: false

detect:
match:
save: true
base64: false
confidence: 80
purge: 168
min_area: 2000

unknown:
save: true
base64: false
confidence: 40
purge: 168
min_area: 0

[townsmcp]

@tyanai change your ‘unknown’ ‘confidence’ from 40 to 80. The issue is there is a gap between known (at 80%) and unknown at 40%. You need unknown to cross to known with no gap. For reference my DT config is the following (I use 85% confidence) and all unknowns are being marked as unknown:

# Double Take
# Learn more at https://github.com/jakowenko/double-take/#configuration



# "Snapshot" -- frame from frigate, saved then frigate detect an event (motion, for example)
# "Latest" -- frame from camera, on the moment then double-take trying to make an recognition (Usually, the difference between them is 1-2 seconds)
# "MQTT" -- The frame that the frigate transmitted with the event by MQTT protocol. It's usually equivalent to "snapshot", but depends on frigate settings

# If u use double-take in conjunction with frigate, the "snapshot" method will be enough

# Want either MQTT or Snapshot. MQTT does not include zone

home_assistant:
  url: http://192.168.1.198:8123
  token: %TOKEN_VALUE%

frigate:
  url: http://192.168.1.158:5000
  update_sub_labels: True
  # stop the processing loop if a match is found
  # if set to false all image attempts will be processed before determining the best match
  stop_on_match: True
  # ignore detected areas so small that face recognition would be difficult
  # quadrupling the min_area of the detector is a good start
  # does not apply to MQTT events
  #min_area: 1600

  # object labels that are allowed for facial recognition
  labels:
    - person
    
  attempts:
    # number of times double take will request a frigate latest.jpg for facial recognition
    latest: 0
    # number of times double take will request a frigate snapshot.jpg for facial recognition
    snapshot: 10
    # process frigate images from frigate/+/person/snapshot topics
    mqtt: false
    # add a delay expressed in seconds between each detection loop
 #   delay: 0
 
  # only process images from specific zones
  zones:
    - camera: doorbell
      zone: garden_doorbell
    - camera: car-port
      zone: driveway_carport
  
mqtt:
  host: 192.168.1.198
  username: %MQTT_USERNAME_VALUE%
  password: %MQTT_PASSWORD_VALUE%
  
detectors:

#If you want no match then increase det_prob_threshold: 80 a little at a time and re-test your matches… it will return nothing if it fails this test. I have been creeping this value up and it has really reduced the number of images accepted and false positives from blurry movement shots etc.

#detectors:
  #compreface:
    # minimum required confidence that a recognized face is actually a face
   # det_prob_threshold: 0.98
  compreface:
    url: http://192.168.1.158:8000
    # recognition api key
    key: %KEY_VALUE%
    # number of seconds before the request times out and is aborted
    timeout: 15
    # minimum required confidence that a recognized face is actually a face
    # value is between 0.0 and 1.0
    det_prob_threshold: 0.8
    # require opencv to find a face before processing with detector
    opencv_face_required: false
    # comma-separated slugs of face plugins
    # https://github.com/exadel-inc/CompreFace/blob/e1ee791cac6cd9b1d3a9ea2cb129c78cb00a9083/docs/Face-services-and-plugins.md)
    # face_plugins: mask,gender,age
    # only process images from specific cameras, if omitted then all cameras will be processed
    # cameras:
    #   - front-door
    #   - garage
#  aiserver:
#    url: http://192.168.1.158:32168
    # number of seconds before the request times out and is aborted
 #   timeout: 15
    # require opencv to find a face before processing with detector
#    opencv_face_required: false
telemetry: false


    
      
detect:
  match:
    # save match images
    save: true
    # include base64 encoded string in api results and mqtt messages
    # options: true, false, box
    base64: false
    # minimum confidence needed to consider a result a match
    confidence: 85
    # hours to keep match images until they are deleted
    purge: 24
    # minimum area in pixels to consider a result a match
    min_area: 2500
    #when using substream image, 3600

  unknown:
    # save unknown images
    save: true
    # include base64 encoded string in api results and mqtt messages
    # options: true, false, box
    base64: false
    # minimum confidence needed before classifying a name as unknown
    confidence: 85
    # hours to keep unknown images until they are deleted
    purge: 8
    # minimum area in pixels to keep an unknown result
    min_area: 64

Also, according to the log it is taking a very long time to reach a decision. The ‘attempts’ should be able to be reduced significantly. Might also pay to add the ‘stop_on_match’ to true?
Throwing a Google Coral into the mix would help Frigate out significantly, and hardware acceleration too. I am using a PCIe rather than USB Coral and inference went down from 200 to 7

[tyanai]

Thanks a lot! Added the stop_on_match and will probably purchase the Coral once I'm happpy with the setup

[tyanai]

Seems this is still not solving the issue.

I'm still getting "most" of the images captured by Frigate to be lost within DoubleTake even after changing confidence to 80.

Which makes me thinks maybe DoubleTake don't takes them as "Person"? Cause I see its always counts: { person: 0, match: 0, miss: 0, unknown: 0 },

Within my Frigate instance, everything is labeled as "Person".

From DoubleTake Configuration;
frigate:
url: http://192.168.1.158:5000
update_sub_labels: True
stop_on_match: True

labels:
- person

23-06-26 12:24:57 info: done processing reolink_cam: 1687782257.650865-8vahqx in 11.04 sec
23-06-26 12:24:57 info: {
id: '1687782257.650865-8vahqx',
duration: 11.04,
timestamp: '2023-06-26T12:24:57.020Z',
attempts: 22,
camera: 'reolink_cam',
zones: [ 'Entrance_pathway' ],
counts: { person: 0, match: 0, miss: 0, unknown: 0 },
matches: [],
misses: [],
unknowns: []
}
23-06-26 12:25:24 info: done processing reolink_cam: 1687782285.441066-37ntpv in 38.2 sec
23-06-26 12:25:24 info: {
id: '1687782285.441066-37ntpv',
duration: 38.2,
timestamp: '2023-06-26T12:25:24.184Z',
attempts: 22,
camera: 'reolink_cam',
zones: [],
counts: { person: 0, match: 0, miss: 0, unknown: 0 },
matches: [],
misses: [],
unknowns: []
}
23-06-26 12:27:06 info: processing reolink_cam: 1687782425.065121-nafiga
23-06-26 12:27:44 info: done processing reolink_cam: 1687782425.065121-nafiga in 38.35 sec
23-06-26 12:27:44 info: {
id: '1687782425.065121-nafiga',
duration: 38.35,
timestamp: '2023-06-26T12:27:44.365Z',
attempts: 22,
camera: 'reolink_cam',
zones: [ 'Entrance_pathway' ],
counts: { person: 0, match: 0, miss: 0, unknown: 0 },
matches: [],
misses: [],
unknowns: []
}

[townsmcp]

@tyanai its really hard to read your config on here as you haven’t encapsulated it in the ' marks (the left hand one on the keyboard ''', then next line add your code and final last line by itself add ''') that way we can see your indentation as well.
However, in your DT config, under Frigate section you don’t have the camera or zones defined?

[tyanai]

No, I don't have any camera zone defined, as I want at this stage to capture everything that is coming in for testing. I noticed that when I'm opening mqtt and latest I'm getting much more results and more unknown are detected, but just with snapshots its more challenged to detect unknown from some reason.

[townsmcp]

OK, but you have camera zones and required_zones defined in Frigate:

required_zones:
- Entrance_porch
- Entrance_pathway
- Entrance_before_gate

I would recommend adding all those zones into the DT config as well otherwise there is a gap between what DT should be doing and what it is doing as DT cant track back to the zone defined in Frigate

[tyanai]

Thanks, I actually tried to simplify the entire thing, so no zones within Frigate at all:

I see the image as an event in Frigate, and see that DT is getting the signal that Frigate took something, but it keeps failing:

23-06-27 11:40:55 info: processing reolink_cam: 1687866055.404717-zvscnu
23-06-27 11:40:56 info: processing reolink_cam: e68383e4-7d75-432d-a2da-da9773546011
23-06-27 11:40:56 info: done processing reolink_cam: e68383e4-7d75-432d-a2da-da9773546011 in 0.68 sec
23-06-27 11:40:56 info: {
  id: 'e68383e4-7d75-432d-a2da-da9773546011',
  duration: 0.68,
  timestamp: '2023-06-27T11:40:56.730Z',
  attempts: 1,
  camera: 'reolink_cam',
  zones: [],
  counts: { person: 0, match: 0, miss: 0, unknown: 0 },
  matches: [],
  misses: [],
  unknowns: []
}

Frigate

cameras:
  reolink_cam:
    ffmpeg:
      inputs:
        - path: rtsp://127.0.0.1:8554/reolink_entrance?video=copy&audio=aac
          input_args: preset-rtsp-restream
          roles:
            - record
        - path: rtsp://127.0.0.1:8554/reolink_entrance?video=copy
          input_args: preset-rtsp-restream
          roles:
            - detect
    motion:
      mask:
        - 2111,842,1239,0,2034,0,2560,0,2560,646
    objects:
      track:
        - person
      mask:
        - 1404,1920,1125,736,977,759,0,1920,0,0,1025,0,2034,0,2560,0,2560,1920
    snapshots:
      enabled: true
      bounding_box: true
      retain: 
        default: 2
    mqtt:
      timestamp: true
      bounding_box: true
      crop: true
      quality: 100
      height: 1920
record:
  enabled: true
  retain:
    days: 7
    mode: motion
  events:
    retain:
      default: 14
      mode: active_objects
      objects:
        dog: 2
        car: 7

DT

frigate:
  url: http://192.168.1.200:5000
  update_sub_labels: true
  min_area: 0
  stop_on_match: true
  labels:
    - person 
  attempts:
    latest: 22
    snapshot: 1
    mqtt: true
    delay: 1.25
  image:
    height: 1920

mqtt:
  host: 192.168.1.200
  username: admin
  password: XXXXX
  topics:
    frigate: frigate/events
    homeassistant: homeassistant
    matches: double-take/matches
    cameras: double-take/cameras

detectors:
  compreface:
    url: http://192.168.1.200:8000/
    key: 3528cc1d-b1ea-4916-aa97-488eb52a7a0b
    det_prob_threshold: 0.7
telemetry: false


  
detect:
  match:
    save: true
    base64: false
    confidence: 80
    purge: 168
    min_area: 2000

  unknown:
    save: true
    base64: false
    confidence: 80
    purge: 168
    min_area: 0

[townsmcp]

Erm just a thought, up snapshot from 1 in DT config.?I have it set to 10.