Face Tracker 0.7.2

[norihiro]

I have reduced Kp to have just smooth (and slow movement) if neccessary but I have noticed that the plugin keeps zooming in and out slightly at all times bc of head movement/turning? how can I avoid that?

Thank you for interested in this plugin.
My recommended workaround is to use Scale max property to limit the zoom.
In the group Tracking target location, please try these setting. With this workaround, zoom will be set manually.

• Set Zoom to large value such as 1.0.
• Set Scale max to small value such as 1.4 - 2.0. This property will control the amount of zoom.

I put your issue to my FAQ page. (I added another workaround idea but less sure it works.)

Regarding the shot from side, I think it should be a future implement. The face-detection algorithm in this plugin is weak to detect faces shot from side.

 

[mille_eintausend]

Wow! Thank you for the quick reply! Your suggestion/settings are looking really promising!

 

[Hermes Amorim]

norihiro submitted a new resource:

Face Tracker - face detection, face tracking



Read more about this resource...

Do you have a tutorial about Face Tracking plugin? I`m having some difficult to setup its parameters. Thanks

 

[chaunkidz]

I really enjoyed using this plugin. Great job!
Can you include an option to track a face of only a specific person?

 

[toplachi]

Another idea for a feature is the ability to switch to another scene if the tracked subject reach a certain portion on the screen :-)

 

[rusty1]

What a great plugin! I'm implementing it on a close-up video of someone who is seated. Even though he isn't walking around, his body naturally starts to lean left or right as he gets tired. When he does so, he starts to fall out of the frame. This plugin fixes that problem!

A feature that would be cool is the ability to turn off the tracker, but stay on the current output scale + position when you do. That would allow me to hit a button to momentarily turn the filter on, to re-center him, and then it would turn back off and save the new layout, until i need to adjust it again. This would allow me to only use the CPU required for face-tracking to fix the layout when I need to readjust it.

 

[GregPeatfield]

What a great plugin! I'm implementing it on a close-up video of someone who is seated. Even though he isn't walking around, his body naturally starts to lean left or right as he gets tired. When he does so, he starts to fall out of the frame. This plugin fixes that problem!

A feature that would be cool is the ability to turn off the tracker, but stay on the current output scale + position when you do. That would allow me to hit a button to momentarily turn the filter on, to re-center him, and then it would turn back off and save the new layout, until i need to adjust it again. This would allow me to only use the CPU required for face-tracking to fix the layout when I need to readjust it.

Can you create a hotkey or use something like a stream deck to enable/disable the filter?

 

[rusty1]

Can you create a hotkey or use something like a stream deck to enable/disable the filter?

Yes, but turning off the filter returns the scale and position to the default. I was talking about preserving it, like freezing the filter....but as i am playing with it more, i see that the CPU is not as bad as i had feared. so might not really be an issue after all.

 

[khaver]

This is a fantastic plugin, and I'd like to use it for my church's sermon recordings.

Let me explain how we record now. I have a 4k webcam, capturing a wide shot, attached to the same computer we run OBS from for recording and a presentation software for projecting songs lyrics and bible verses during the sermon. I have a single scene set up in OBS with a canvas size of 3840 x 2160. This canvas is split into quadrants with one quadrant the wide camera shot (so now 1920x1080). The second quadrant is a duplicate of the webcam but zoomed in 2x using the crop and resize handles in the preview window. The third quadrant is a display capture of the projector attached to the computer as a second monitor that the presentation software projects to. The fourth quadrant is a browser source showing a local html file that has the sermon title, date, and name of the speaker, in black text on a white background. This UHD MKV recording is then imported into DaVinci Resolve and edited using the different quadrants as if they where separate video recordings. This works extremely well because I can create a template in Resolve that composites this all together so I only have to edit between the wide and close shot using Resolve's multicam feature. With the mutlicam feature I usually switch to the wide shot whenever the speaker moves out of the frame of the close shot. It would be more flexible to be able to follow the speaker in the close shot using your plugin and only have to cut to the wide shot for effect, not necessity.

The problem I've run into with your plugin is trying to apply it to only one of the webcam quadrants. As you know, if you apply an effect to a source that's duplicated in a scene, the filter gets applied to both instances in the preview window. Usually you can get around this by putting one the the instances in a group and applying the filter to the group. I tried this with your plugin but it does not work.

Can you add the ability to use your plugin on a group? If this could be added, I could almost be able to do everything in OBS without even needing to edit in Resolve. Thanks

 

[khaver]

Here's a screen grab of a test of our recording.

And the final composite.

 

[norihiro]

This is a fantastic plugin, and I'd like to use it for my church's sermon recordings.

Let me explain how we record now. I have a 4k webcam, capturing a wide shot, attached to the same computer we run OBS from for recording and a presentation software for projecting songs lyrics and bible verses during the sermon. I have a single scene set up in OBS with a canvas size of 3840 x 2160. This canvas is split into quadrants with one quadrant the wide camera shot (so now 1920x1080). The second quadrant is a duplicate of the webcam but zoomed in 2x using the crop and resize handles in the preview window. The third quadrant is a display capture of the projector attached to the computer as a second monitor that the presentation software projects to. The fourth quadrant is a browser source showing a local html file that has the sermon title, date, and name of the speaker, in black text on a white background. This UHD MKV recording is then imported into DaVinci Resolve and edited using the different quadrants as if they where separate video recordings. This works extremely well because I can create a template in Resolve that composites this all together so I only have to edit between the wide and close shot using Resolve's multicam feature. With the mutlicam feature I usually switch to the wide shot whenever the speaker moves out of the frame of the close shot. It would be more flexible to be able to follow the speaker in the close shot using your plugin and only have to cut to the wide shot for effect, not necessity.

The problem I've run into with your plugin is trying to apply it to only one of the webcam quadrants. As you know, if you apply an effect to a source that's duplicated in a scene, the filter gets applied to both instances in the preview window. Usually you can get around this by putting one the the instances in a group and applying the filter to the group. I tried this with your plugin but it does not work.

Can you add the ability to use your plugin on a group? If this could be added, I could almost be able to do everything in OBS without even needing to edit in Resolve. Thanks

You can have a new scene that has the camera source which you want to apply the filter. Then, please add the filter to the scene instead of to the camera source.
You can put the scene to another scene.

 

[khaver]

Thank you, norihiro. That worked perfectly.

 

[norihiro]

norihiro updated Face Tracker with a new update entry:

Face Tracker 0.3.0

This is a release with some small new features.

• face-tracker: Add hotkeys to pause/resume tracking and reset tracking states. I'd like to thank @rusty1 and @GregPeatfield for the idea.
• PTZ: Add TCP connection to VISCA device. I'd like to thank @GregPeatfield and Jimcom US for providing support to debug this feature.

Read the rest of this update entry...

 

[khaver]

Feature suggestions.
1) Keep the face detector from looking in certain areas of the source frame by defining this area using left, right, top and bottom sliders to "crop" out the areas. When you check the "Stop tracking faces" box, you can see red crop lines, and they move as you adjust the sliders.

2) Some sort if setting that when several faces are detected, more weight is given to the last face detected, and/or more weight is given to the face closest to the center of the frame, and/or if another face is detected, it won't be selected if it's more than a settable distance from the current face. These will keep the tracking from bouncing from one face to the next.

3)A way to adjust the video levels (gain, brightness, contrast) of the frame sent to the detector algorithm. This way you can lower the chance of a face being detected in dark grainy backgrounds.

Thanks

 

[khaver]

I simulated 1) by modifying your face-tracker.effect file. In function PS_RGB2Y I changed it to:

float4 PS_RGB2Y(VertInOut vert_in) : TARGET
{
    float px = vert_in.uv.x;
    float py = vert_in.uv.y;
    float4 rgb = 0.0;
    if (px > 0.3 && px < 0.7 && py > 0.1 && py < 0.9) rgb = image.Sample(def_sampler, vert_in.uv);
    float y = 0.299 * rgb.x + 0.587 * rgb.y + 0.114 * rgb.z;
    if (y < 0.07) y = 0.0;
    return float4(y, 0.0, 0.0, 1.0);}

This cropped out 30% on the left and right side, and 10% at the top and bottom. Worked like a charm.
I added "if (y < 0.07) y = 0.0;" so any dark grainy backgrounds are set to black so no false faces in the dark areas will be detected.
Now if you could add controls in the plugin to set these and a checkbox that would display the frame being sent to the detector, I would be extremally happy.

Again, thank you for your plugin.

 

[khaver]

Clarification for my what I really want for the cropping out parts of the frame from the face detection. What I would like is to be able to block areas from the face detection algorithm but not the tracking. In my workaround above, the areas cropped are made black so no faces can be found, but this also means if the face moves into the cropped area, tracking will not happen. Is there a way to block areas from detection but continue to allow tracking into these areas? Your face-tracker.effect file basically sends a black and white representation of the frame in the red channel to your detection and tracking algorithms, but could you combine the cropped frame on the red channel with the unmodified frame on the green channel and the face detector would read the frame image from the red channel and the tracker would read the frame image from the green channel?
The new code would look something like this:

float4 PS_RGB2Y(VertInOut vert_in) : TARGET
{
    float px = vert_in.uv.x;
    float py = vert_in.uv.y;
    float4 face = 0.0;
    float4 track = image.Sample(def_sampler, vert_in.uv);
    if (px > 0.3 && px < 0.7 && py > 0.1 && py < 0.9) face = track;
    float r = 0.299 * face.x + 0.587 * face.y + 0.114 * face.z;
    float g = 0.299 * track.x + 0.587 * track.y + 0.114 * track.z;
    if (r < 0.07) r = 0.0;
    return float4(r, g, 0.0, 1.0);}

 

[Ossi_Wilson]

Hi! Love the plugin.

Minor problem which is probably due to settings but i can't figure out which one.

Here is a video of my problem: https://drive.google.com/file/d/1c6ufmGdrBeaJrYbqimNc3nU34j-gjFzw/view?usp=sharing
The sudden jump that the cam does 3 times during the "replay". This has happened on multiple occasions, but not always.

The picture is coming from a main front camera, not the one that is currently shown, just to be clear.

Can it be dueto it possbile detecting 2 faces? Even they the jump isn't into the direction of the other face?

Attached a picture of my settings.I believe im currently using 0.2.2

 

[norihiro]

@khader,
1)
Thank you for the suggestion. Currently both face-detection and correlation-track use the same frame. Currently only red channel is transferred from GPU to CPU to reduce the amount of the data.
Rather than making black, I prefer to crop the image before sending to the detector. If we can use the same amount of crop for both face-detection and correlation-track, we can crop GPU side. However, if correlation-track need the original image without cropping, we need to crop in CPU side. I'm thinking to crop the image at `face_detector_dlib::detect_main()`
2)
I'm aware of the problem but right now I don't have clear idea to solve it cleanly. As you commented, giving more weight to the closest face sounds more promising than what I previously tried.
3)
According to what you change in the effect file, I think it is sufficient to provide two properties (black level and white level). Gamma is not necessary, is it?

 

[norihiro]

The sudden jump that the cam does 3 times during the "replay". This has happened on multiple occasions, but not always.
Can it be dueto it possbile detecting 2 faces? Even they the jump isn't into the direction of the other face?

Thank you for providing the video. As you commented, I suspect the jump is caused by hopping two faces detected.
Currently the algorithm trying to zoom into the highest-scored face. It mainly depends on the size of the face but also frequently changes when the direction of the face changes.

 

[Ossi_Wilson]

Thank you for providing the video. As you commented, I suspect the jump is caused by hopping two faces detected.
Currently the algorithm trying to zoom into the highest-scored face. It mainly depends on the size of the face but also frequently changes when the direction of the face changes.

Thank you very much for the reply. This isnt that much of an issue so i will definately still continue using thje plugin. awesome work man!