Custom filter for frame processing (not a shader)

[Cr33zz]

Hi, I'm working on fast neural style transfer project. This is basically a neural network capable of generating stylized images from any input image in real-time. I wanted to implement OBS Studio plugin so I can apply this stylization to any source. My neural network requires an RGB image as an input, so I need to grab the current source frame, push it through my network and set the stylized frame as an output of my effect/filter. I was googling for how to achieve that but most of the filters are implemented as shader-style effects which is not what I need. Any hints on how I could achieve that would be appreciated :)

 

[Lain]

You'd want an asynchronous video filter. Currently the only example of an asynchronous video filter is in plugins/obs-filters/async-delay-filter.c, which is just used to delay video data. But all the callbacks are there. Only problem is that async video filters will take in any format that is being output by the source. So you can get all the different YUV formats, or RGB, BGR, BGRA. Depends on what the source is outputting at the time.

Additionally, this only works on specific types of sources. Specifically, video capture device source, media source. It cannot be used on other types of sources because they're drawn via GPU and stored on VRAM, so the application of this type of filter is a bit limited.

 

[Cr33zz]

It worked and turned out to be pretty straightforward :) Thanks a lot for hints. For anyone interested here is the first pass of the plugin.
https://github.com/Cr33zz/obs-stylefilter/blob/master/src/obs-stylefilter.cpp

I implemented YUV420/422<->RGB conversions on my own. Is there any utility functionality already in OBS that I missed for format conversions?

Thanks!

 

[Lain]

Hey that's pretty neat. You're clearly pretty skilled to have figured it out and have done that already.

For color conversion, OBS just uses FFmpeg for CPU-based color conversions when they happen (which is usually never). There's a simplified FFmpeg wrapper of this conversion here: https://github.com/obsproject/obs-studio/blob/master/libobs/media-io/video-scaler.h -- other structures relevant to this are in video-io.h. However currently there's an issue with using it on directshow data, so I probably don't recommend it right now for the time being. In the near future there'll be an auto RGB conversion if an async filter requires it, but it's not merged to master right now.