Custom filter for frame processing (not a shader)

Cr33zz

New Member
Hi, I'm working on fast neural style transfer project. This is basically a neural network capable of generating stylized images from any input image in real-time. I wanted to implement OBS Studio plugin so I can apply this stylization to any source. My neural network requires an RGB image as an input, so I need to grab the current source frame, push it through my network and set the stylized frame as an output of my effect/filter. I was googling for how to achieve that but most of the filters are implemented as shader-style effects which is not what I need. Any hints on how I could achieve that would be appreciated :)
 

Jim

Jim
Developer
You'd want an asynchronous video filter. Currently the only example of an asynchronous video filter is in plugins/obs-filters/async-delay-filter.c, which is just used to delay video data. But all the callbacks are there. Only problem is that async video filters will take in any format that is being output by the source. So you can get all the different YUV formats, or RGB, BGR, BGRA. Depends on what the source is outputting at the time.

Additionally, this only works on specific types of sources. Specifically, video capture device source, media source. It cannot be used on other types of sources because they're drawn via GPU and stored on VRAM, so the application of this type of filter is a bit limited.
 

Jim

Jim
Developer
Hey that's pretty neat. You're clearly pretty skilled to have figured it out and have done that already.

For color conversion, OBS just uses FFmpeg for CPU-based color conversions when they happen (which is usually never). There's a simplified FFmpeg wrapper of this conversion here: https://github.com/obsproject/obs-studio/blob/master/libobs/media-io/video-scaler.h -- other structures relevant to this are in video-io.h. However currently there's an issue with using it on directshow data, so I probably don't recommend it right now for the time being. In the near future there'll be an auto RGB conversion if an async filter requires it, but it's not merged to master right now.
 
Top