Custom filter - a little help needed

[EmperrorAoyama]

Hi everyone,
I'm creating a custom filter with which I'd want to get current video frame (2D array of its pixels) and transform it in a way described by another 2D "transformation Array".

I read a lot of scripts and filters' codes for these that are already attached to OBS source, but I'm still having some basic level problems before I start development.

First thing:
Is there any basic documentation about any API provided for achieving such basic frame gathering actions and similiar things, especially for custom filter development?

Second thing:
In this filter for example
https://github.com/jp9000/obs-studio/blob/master/plugins/obs-filters/mask-filter.c

There is this part:
.type = OBS_SOURCE_TYPE_FILTER,
.output_flags = OBS_SOURCE_VIDEO,

I'm guessing it's purpose is to connect my custom methods with some obs filter manager. My question here is: are these ".type" ".output" possible options described somewhere? Or do you know in which script are these methods executed when they're attached as a filter in OBS studio?

 

[Lain]

Raw frame stored on RAM or texture to use with a shader? The mask filter is an effect (shader) filter that applies an additional shader to a source.

libobs/obs-source.h is most likely what you're looking for.

 

[EmperrorAoyama]

That's exactly the class I was looking for, didn't expect that it's attached straight to the obs-source without a mid-level "filter manager" layer.

And texture is probably what I'm looking for then.

How can I get the texture from the source then?

 

[Lain]

You don't technically get the texture from the source directly. You use an effect and it's automatically assigned to the effect as a parameter when the time comes so it can generate/process the filter chain correctly. Look at the other effect filters to see how to do it.

obs_source_process_filter_begin and obs_source_process_filter_end are the functions you'd use.

 

[EmperrorAoyama]

One additional question here:

Is it possible to create some image and set it's rgb values pixel by pixel basing on float rgb values I have?

I'd want to create such an image and then create gs_texture_t from it. Any clues on how could I achieve that?

 

[Lain]

You can create a texture with raw RGB data and pass it in to the effect shader as well. All such functions are in libobs/graphics/graphics.h. gs_texture_create. I don't quite understand the context of the question though, please explain what you're trying to do so I can advise what I would feel is the most ideal solution based upon the problem.

 

[EmperrorAoyama]

I'm having a big 2d array of x,y transformations and I'm trying to pass these data to my effect shader, So far I managed to upload it in a form of a png graphic containing these data stored as RGB.rg values (I translated the tranfsormations into uv equivalents).

The problem is I used java with some fast integer based png generation to generate this graphic and it was a masively lossy method. It was created for a demo though and at this part it worked.

I was also trying to pass this 2d array to my effect shader as a 1D array of data but it was still impossible due to it's length. Storing the data as png allowed me to do that easily and fast though.

If there's a easier and faster way, I'm open for suggestions as well.

 

[Lain]

Yea, you explained the transformation array in the original post to some extent, but I'm still not sure if I fully understand what you're overarching goal is.

 

[EmperrorAoyama]

What my shader is doing is for each uv position on a from-source-frame it's also taking the same uv positioned pixel from the transformation graphic and from it's rg values I'm getting the uv2 transformation values to determine where this source pixel should go in final view.

I'm trying to pass this 2d array of these "uv values" to my effect shader and as 2d textures are automaticaly supported for to-shader data passing, I decided to store these uvs in that format.

 

[Lain]

Forgive me, but I meant your overall goal you're trying to achieve. What sort of thing you're writing. I still don't understand based upon your description.

This is going to be a really really stupid analogy for what's going on here, but bear with me: let's say a man (a miner) is moving a mine cart along a track, another miner who sees that he's stopped mining comes up and asks him "what are you doing?", the guy says "what does it look like I'm doing, I'm moving a cart". The other guy says, "no no, I mean, what happened, what's going on?", the guy again responds, "you see, I push the cart and the wheels turn due to the momentum of the force I exert on the cart, and thus the cart moves". The other guy palms his face, wondering if he's talking to a fellow miner or neil degrasse tyson or something, closes his eyes and just pauses for a moment, then takes a deep breath, slowly opens his eyes again, and says "yes, but *why* are you doing it?", and the guy says "Oh! Why didn't you just ask? Because I'm finished mining all the minerals I could find in this section of the mine so I'm transporting them back out of the mine".

Don't worry you're fine, I apologize if I made it sound worse than it is, you're honestly fine -- but it is just a tiny bit awkward trying to understand the situation. So with that being said, what's your overall goal? What are you making? Is there any way to describe what sort of filter you're making? I can't quite grasp it based upon your actions yet.

Like a short description. Or even a title, title of your filter that describes what it does even. For example, we have "Mask filter", "Chroma key filter", "Color correction filter". What would yours be? What would be its short description? Something to where I can understand what you're trying to do.

 

[EmperrorAoyama]

Something kinda similiar to what ffmpeg's remap is doing, just with this custom solution as it was easier for me to start with :D

 

[Lain]

Ah, I'm beginning to understand now. "Remap filter". I guess it's not as easy to slap a name on that one. The best description I found in FFmpeg describes the remap filter as such: Target_frame[y][x] = Source_frame[ ymap[y][x] ][ [xmap[y][x] ];

I'm able to comprehend what it's doing based upon that little bit of pseudocode alone, I apologize if I didn't understand sooner. This would be very simple with a shader and a map texture. The output texel would just be looked up on the main texture with a float2 UV coordinate that's created from using the current actual UV coordinate for the texel on the map texture.

A pixel shader for what you're doing would be something like:

float4 Remap(VertInOut vert_in) : TARGET
{
    float2 uv = mapTexture.Sample(some_default_sampler, vert_in.uv).rg;
    return image.Sample(some_default_sampler, uv);
}

Would that be correct? I think I'm beginning to understand. Just create that mapping texture as you were describing, then use the filter functions to pass that texture as a parameter like that.

 

[Lain]

By the way, I saw you dropped by the #obs-dev chat, but you didn't stay long enough. If you want to talk real time feel free to stick around. I'd recommend an IRC client and just idling for a bit next time you have a question, I'll always try to answer when I see it.

 

[Lain]

By the way, to answer your question about how to create textures, you can call gs_texture_create_from_file if you have a file, or gs_texture_create if you have the RGB data right there.

EXPORT gs_texture_t *gs_texture_create(uint32_t width, uint32_t height,
		enum gs_color_format color_format, uint32_t levels,
		const uint8_t **data, uint32_t flags);

The first three parameters are self-explanatory. The 'levels' parameter is a parameter used in 3D rendering that allows you to set the number of total levels with mipmaps. 1 would just be a single texture with no mip-maps. 2 would be a texture with one half-sized mipmap, 3 would be a texture with two mipmaps, one half sized, one quarter sized. And so on. The 'data' parameter specifies the texture and sub-levels if any. As you see it's **, but that's because it's meant to pass a pointer array, so data[0] would be a pointer to the first level (the main texture), data[1] would be a pointer to the half-sized mapmap, 2 the quarter-sized mipmap, and so-on, if they exist. That texture will then be created with that RGB data you specify. 'flags' specifies texture flags, such as GS_DYNAMIC or GS_RENDER_TARGET, which I'm guessing you don't need.

If you just have a pointer to raw RGB data and don't want mipmaps, you'd have something like this:

uint8_t *my_texture_data = get_my_texture_data_or_something();
gs_texture_t *texture = gs_texture_create(400, 400, GS_RGBA,
		1, &my_texture_data, 0);

400x400 being a hypothetical size.

 

[EmperrorAoyama]

Yup, that is how my shader is working right now :) And I'm trying to create exactly such a map texture for it basing on the data I'm reading from file :)

The example you presented is exactly what I need, especially the desciption of **data which was confusing for me (didn't know I can include mip-maps there), I will try to implement it into the filter later on :)

I'll get back to the IRC in the afternoon and will have time to stay longer then, maybe I'll be able to catch you then :)

 

[RytoEX]

@EmperrorAoyama
Did you make any further progress with this? I'm curious to see it in action.

 

[desaintmartin]

One year later, I'm also interested into any progress that would have been made in this topic.
I'm trying to insert some PIP video in a 360 stream, and I would need custom "ffmpeg remap-like" feature with my own xy mapping file or a shader file to do this.