Machine learning automatic green screen

[simernes]

Good morning

I was reading about this article from NVIDIA https://blogs.nvidia.com/blog/2019/09/26/nvidia-rtx-broadcast-engine-twitch-livestream-ai/ talking about incorporating ai into OBS live streams. Just curious on how they will do it, and if it would be possible to develop a plugin for OBS that automatically detects the border of the person on camera so that you can blur out the background, or change it like with a green screen.

Has anyone heard of anyone working on something like that that would be possible to try out?

 

[gadicc]

I'm not an expert in this field but I thought I'd link in some relevant info:

BodyPix does real time body segmentation in the Browser via TensorFlow.js
https://github.com/tensorflow/tfjs-models/tree/master/body-pix

Presumably the same resnet model can be used in Tensorflow outside the browser,
otherwise Tensorflow's Deeplab has segmentation models (but I'm not sure if they work in realtime):
https://github.com/tensorflow/models/tree/master/research/deeplab

Tensorflow C++ API:
https://www.tensorflow.org/api_docs/cc

Not sure how it would compare with nVidia's proprietary code for their own chips, but Tensorflow is all open source and works in multiple environments.

 

[gadicc]

Oops I realized 10m after writing my message that the original article actually references the OBS author's talking specifically about RTX support in OBS, and now it's too late to delete or edit my message. Anyways, sounds like nVidia is the way to go here.