Detect - Object Detection, Tracking built-in OBS v0.0.2

[royshilkrot]

royshilkrot submitted a new resource:

Detect - Object Detection, Tracking built-in OBS - Real-time Detect, Follow, Zoom or Blur/Mask over 80 categories of objects. No GPU required*

A plugin that allows you to detect many types of objects in any source, track them and apply masking.

If you like this work, which is given to you completely free of charge, please consider supporting it by sponsoring us on GitHub:

• https://github.com/sponsors/royshil
• https://github.com/sponsors/umireon

This work uses the great contributions from EdgeYOLO-ROS and...

Read more about this resource...

 

[Want To Learn]

Hi Roy
This sounds like an exciting plugin as I have used some of your others, but I don't know how to download and install this one for a Mac. I go to your download link and I don't see a Mac installer for it. Can you please post a link for me? Thanks!......just figured it out! Click on assets. Can't wait to try it. Thanks!

 

[royshilkrot]

Hi Roy
This sounds like an exciting plugin as I have used some of your others, but I don't know how to download and install this one for a Mac. I go to your download link and I don't see a Mac installer for it. Can you please post a link for me? Thanks!......just figured it out! Click on assets. Can't wait to try it. Thanks!

I'll sort this out.. thanks for flagging!

 

[Want To Learn]

Hi again Roy.
I installed it for my Mac as you already know. However, you seem to have two options that I don't have on my filter. I have detect, but not detect tracking, Is this because I'm using a Mac, but it's all there? Thanks.

 

[royshilkrot]

Hi again Roy.
I installed it for my Mac as you already know. However, you seem to have two options that I don't have on my filter. I have detect, but not detect tracking, Is this because I'm using a Mac, but it's all there? Thanks.

Detect is the filter of this plugin it will detect and track many kinds of objects in the video of the source you add it on.
Tracking is done by realtime detection.
Although I'm working on legit tracking which would be available soon.

 

[Want To Learn]

Hi again Roy.
I'm sure its something that Im doing wrong, but I can't seem to get the zoom feature to work at all. Any ideas? Also, thanks so much for all your work and effort! It's now in it's infancy, so I'm sure you're going to rock it soon!

 

[royshilkrot]

Hi again Roy.
I'm sure its something that Im doing wrong, but I can't seem to get the zoom feature to work at all. Any ideas? Also, thanks so much for all your work and effort! It's now in it's infancy, so I'm sure you're going to rock it soon!

The zoom will try to fit the object in the image - but if the object is too big then no zoom will occur
You can limit the detection to a single type of object, to get a smaller detection on screen

 

[Want To Learn]

Hi Roy,
I tried many things, but the zoom does not work. This detection plugin allows me to select objects, blur and stuff, but no zoom. I even used MacOS screen capture to zoom in on your youtube vid example and it doesn't work. It appears I have enough CPU.as Face tracker and every thing else that I use works, but don't know why detection doesn't zoom on my Mac . It's possible it's me, but I can't figure it out. Thanks!

 

[royshilkrot]

Hi Roy,
I tried many things, but the zoom does not work. This detection plugin allows me to select objects, blur and stuff, but no zoom. I even used MacOS screen capture to zoom in on your youtube vid example and it doesn't work. It appears I have enough CPU.as Face tracker and every thing else that I use works, but don't know why detection doesn't zoom on my Mac . It's possible it's me, but I can't figure it out. Thanks!

Ok I will have a look to search for a potential bug..

 

[Want To Learn]

Ok I will have a look to search for a potential bug..

Thanks.

 

[PaRaLLaX]

Hey I'm thinking about automating the intermission screen. Something like "if person is detected in webcam-source -> hide intermission screen; else -> show intermission screen" basically. Could this be achievable with your plugin? I'm using Streamer.Bot, so I could watch on file changes or react on WebSocket notifications etc if that helps?

Would be awesome :)

 

[royshilkrot]

Hey I'm thinking about automating the intermission screen. Something like "if person is detected in webcam-source -> hide intermission screen; else -> show intermission screen" basically. Could this be achievable with your plugin? I'm using Streamer.Bot, so I could watch on file changes or react on WebSocket notifications etc if that helps?

Would be awesome :)

That's a great idea! It could be implemented but I need to add output to the settings so you can trigger on it. Can you open an issue on GitHub so I would remember? Thanks!

 

[PaRaLLaX]

That's a great idea! It could be implemented but I need to add output to the settings so you can trigger on it. Can you open an issue on GitHub so I would remember? Thanks!

Awesome thanks a lot :)

Output whether person is detected or not (intermission screen automation) · Issue #4 · occ-ai/obs-detect

Hey, thank you for looking into it! I had the idea to automate the intermission screen via something like: "if person is detected -> Hide intermission Screen; else -> Show intermission screen" Sinc...

github.com

Here's the issue :)

 

[fabianlujan]

Hey Roy,
really awesome plugin!
Detection works really good.
I tried a handball match with 1 ball, 14 players and 2 officials. It detects everything!
The zoom feature do not work or I can't figure out.
Maskings are okay, they are all fine.

In my example, all players were detected with no major problem, only when persons are next to each other.
The sports ball somethings is not seen. Maybe because the nature speed of the game.

It could be awesome to isolate a player and track him individually. Also, maybe, the ball.

In this example, the camera pans left-right all the time. But with a fixed wide-angle camera, it could follow a specific player.

Great work

 

[royshilkrot]

Hey Roy,
really awesome plugin!
Detection works really good.
I tried a handball match with 1 ball, 14 players and 2 officials. It detects everything!
The zoom feature do not work or I can't figure out.
Maskings are okay, they are all fine.

In my example, all players were detected with no major problem, only when persons are next to each other.
The sports ball somethings is not seen. Maybe because the nature speed of the game.

It could be awesome to isolate a player and track him individually. Also, maybe, the ball.

In this example, the camera pans left-right all the time. But with a fixed wide-angle camera, it could follow a specific player.

Great work

@fabianlujan thanks!
you can try a bigger model which is higher resolution and would find objects better, but only try that if you have a GPU...

i will take your ideas into account.
will post an updated version soon!

 

[fabianlujan]

@fabianlujan thanks!
you can try a bigger model which is higher resolution and would find objects better, but only try that if you have a GPU...

i will take your ideas into account.
will post an updated version soon!

Sure Roy, I will check with a different camera set in order to improve detection.
thanks!

 

[Uruchima]

royshilkrot submitted a new resource:

Detect - Object Detection, Tracking built-in OBS - Real-time Detect, Follow, Zoom or Blur/Mask over 80 categories of objects. No GPU required*



Read more about this resource...

The plugin works amazing! I am looking for specific object selection for tracking. Just tried using it for a live event and I can't have it stay on 1 specific person at the moment. I'm sure that's coming soon!

 

[royshilkrot]

The plugin works amazing! I am looking for specific object selection for tracking. Just tried using it for a live event and I can't have it stay on 1 specific person at the moment. I'm sure that's coming soon!

Yes tracking and re-identification is not yet fully working. I'm working on it but currently busy with other plugins..

 

[nuckingfuts]

I also would really appreciate a multi-select object detect list. I'm attempting to utilize this with wifi streaming gopro's covering an area 15m x 35m to potentially track 1/10th scale R/C Cars. It's definitely detecting items but not having much luck with it for my objects even with confidence down to 0.3. I've tried all 3 models sizes, been up to 6 threads iirc. Attempted at 720p and 1080p over 2.4ghz and 5ghz trying to improve source quality. I've also tried with a directly connected gopro and a different directly connected webcam. Occasionally with the trucks(larger of the object size range) it would pick up momentarily at very close distances. I've tried setting the detect object to car and truck. In some camera locations I've left it to All but it picks up a bunch of items I don't need. Perhaps I'm misunderstanding some settings. This has so much potential for my use case. I'm sure some of it is just down to camera equipment but unfortunately the next ideological step up is a PTZ Camera with patrol/tracking and a bit out of my range right now. Just wanted to say thanks for giving us a chance to take streaming to a new level. Anyways, I need to +1 Fabian's idea about selecting an object to mask/track on the fly so if the commentators are focused on a specific player, or object and talking about it that would bring a little more focus to the end user. e.g, "hey this car is the leader!" In my case I am the operator and commentator. The objects are moving fairly fast so a single click to select item would be needed. If you'd like some reference video I can post/send links to the livestreams.

 

[royshilkrot]

I also would really appreciate a multi-select object detect list. I'm attempting to utilize this with wifi streaming gopro's covering an area 15m x 35m to potentially track 1/10th scale R/C Cars. It's definitely detecting items but not having much luck with it for my objects even with confidence down to 0.3. I've tried all 3 models sizes, been up to 6 threads iirc. Attempted at 720p and 1080p over 2.4ghz and 5ghz trying to improve source quality. I've also tried with a directly connected gopro and a different directly connected webcam. Occasionally with the trucks(larger of the object size range) it would pick up momentarily at very close distances. I've tried setting the detect object to car and truck. In some camera locations I've left it to All but it picks up a bunch of items I don't need. Perhaps I'm misunderstanding some settings. This has so much potential for my use case. I'm sure some of it is just down to camera equipment but unfortunately the next ideological step up is a PTZ Camera with patrol/tracking and a bit out of my range right now. Just wanted to say thanks for giving us a chance to take streaming to a new level. Anyways, I need to +1 Fabian's idea about selecting an object to mask/track on the fly so if the commentators are focused on a specific player, or object and talking about it that would bring a little more focus to the end user. e.g, "hey this car is the leader!" In my case I am the operator and commentator. The objects are moving fairly fast so a single click to select item would be needed. If you'd like some reference video I can post/send links to the livestreams.

Thanks for the suggestions
You're welcome to join our discord server and post anything you think would be useful
But I think the best tracking solution would be to train a model to find the objects you're interested in. Training is not simple... Doable though. Once you get a dataset then it's just matter if you have the hardware GPU etc.
I'll be working on the tracking and selection soon.