Question / Help Using one Video Source as Alpha for another?

[Ozzkuu]

It was difficult to pry open OBS plugin building in one hour, so I opted to overwrite an existing shader in OBS to accomplish mixed reality streaming for SteamVR in OBS.

Okay so if you want to use OBS to use the color of the top-right quadrant of a source as the alpha on the top-left quadrant of a source, simply replace the contents of any .effect file in obs-studio/data/obs-plugins/obs-filters/ with

uniform float4x4 ViewProj;
uniform texture2d image;

uniform float4 color;
uniform float contrast;
uniform float brightness;
uniform float gamma;

uniform float4 key_color;
uniform float similarity;
uniform float smoothness;

sampler_state textureSampler {
Filter = Linear;
AddressU = Clamp;
AddressV = Clamp;
};

struct VertData {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};



VertData VSDefault(VertData v_in)
{
VertData vert_out;
vert_out.pos = mul(float4(v_in.pos.xyz, 1.0), ViewProj);
vert_out.uv = v_in.uv;
return vert_out;
}



float4 OffsetPixel(float4 rgba, VertData v_in)
{
float2 shiftedUV = v_in.uv;
shiftedUV.x += 0.5f;
return image.Sample( textureSampler, shiftedUV ) * color;
}

float4 PSColorKeyTRAppliedTLAndCroppedTL( VertData v_in )
{
float4 rgba = image.Sample( textureSampler, v_in.uv ) * color;

if ( v_in.uv.x < 0.5f && v_in.uv.y < 0.5f )
{
float4 offsetRgba = OffsetPixel( rgba, v_in );
rgba.w = offsetRgba.x;
}
return rgba;

}

float4 PSColorKeyRGBA(VertData v_in) : TARGET
{
float4 rgba = PSColorKeyTRAppliedTLAndCroppedTL( v_in );
return rgba;
}



technique Draw
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSColorKeyRGBA(v_in);
}
}


and then use the filter of the same name inside of OBS. I recommend not writing over an effect file for a filter you are already using, and to back up your original effects files before overwriting one or more of them.

Thanks! I got the shader working after removing the unneeded multiplication by color on sampling.

I tried replacing the Sharpen filter today, with the code you gave with no luck.
Any other pointers you have?

The edited shader code that I got working:

uniform float4x4 ViewProj;
uniform texture2d image;

sampler_state textureSampler {
Filter = Linear;
AddressU = Clamp;
AddressV = Clamp;
};

struct VertData {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};

VertData VSDefault(VertData v_in)
{
VertData vert_out;
vert_out.pos = mul(float4(v_in.pos.xyz, 1.0), ViewProj);
vert_out.uv = v_in.uv;
return vert_out;
}

float4 OffsetPixel(float4 rgba, VertData v_in)
{
float2 shiftedUV = v_in.uv;
shiftedUV.x += 0.5f;
return image.Sample( textureSampler, shiftedUV );
}

float4 PSColorKeyTRAppliedTLAndCroppedTL( VertData v_in )
{
float4 rgba = image.Sample( textureSampler, v_in.uv );
if ( v_in.uv.x < 0.5f && v_in.uv.y < 0.5f )
{
float4 offsetRgba = OffsetPixel( rgba, v_in );
rgba.w = offsetRgba.x;
}
return rgba;
}

float4 PSColorKeyRGBA(VertData v_in) : TARGET
{
float4 rgba = PSColorKeyTRAppliedTLAndCroppedTL( v_in );
return rgba;
}

technique Draw
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSColorKeyRGBA(v_in);
}
}

 

[artinco]

It was difficult to pry open OBS plugin building in one hour, so I opted to overwrite an existing shader in OBS to accomplish mixed reality streaming for SteamVR in OBS.

Okay so if you want to use OBS to use the color of the top-right quadrant of a source as the alpha on the top-left quadrant of a source, simply replace the contents of any .effect file in obs-studio/data/obs-plugins/obs-filters/ with

uniform float4x4 ViewProj;
uniform texture2d image;

uniform float4 color;
uniform float contrast;
uniform float brightness;
uniform float gamma;

uniform float4 key_color;
uniform float similarity;
uniform float smoothness;

sampler_state textureSampler {
Filter = Linear;
AddressU = Clamp;
AddressV = Clamp;
};

struct VertData {
float4 pos : POSITION;
float2 uv : TEXCOORD0;
};



VertData VSDefault(VertData v_in)
{
VertData vert_out;
vert_out.pos = mul(float4(v_in.pos.xyz, 1.0), ViewProj);
vert_out.uv = v_in.uv;
return vert_out;
}



float4 OffsetPixel(float4 rgba, VertData v_in)
{
float2 shiftedUV = v_in.uv;
shiftedUV.x += 0.5f;
return image.Sample( textureSampler, shiftedUV ) * color;
}

float4 PSColorKeyTRAppliedTLAndCroppedTL( VertData v_in )
{
float4 rgba = image.Sample( textureSampler, v_in.uv ) * color;

if ( v_in.uv.x < 0.5f && v_in.uv.y < 0.5f )
{
float4 offsetRgba = OffsetPixel( rgba, v_in );
rgba.w = offsetRgba.x;
}
return rgba;

}

float4 PSColorKeyRGBA(VertData v_in) : TARGET
{
float4 rgba = PSColorKeyTRAppliedTLAndCroppedTL( v_in );
return rgba;
}



technique Draw
{
pass
{
vertex_shader = VSDefault(v_in);
pixel_shader = PSColorKeyRGBA(v_in);
}
}


and then use the filter of the same name inside of OBS. I recommend not writing over an effect file for a filter you are already using, and to back up your original effects files before overwriting one or more of them.

Thank you

 

[romibi]

In case anyone else is hesistant to use the LIV thing mentioned before and is also hesistant to replace an existing effect, I'm writing here the solution I found.
Because I'm not searching for this for Mixed Reality I didn't want to use that LIV thing. And as said I didn't want to overwrite an existing effect file. Therefore I experimented a long time with gifs as image mask.
But that has some constraints to filesize or loop lenght (not sure what.)

Finally the Effect-File text above in combination with https://obsproject.com/forum/resources/custom-effect.871/ allowed me to do something like this:


Setup:
1. Install Custom-Effect Plugin and create a new .effect text-file with content from above (doesn't matter where)
2. Create new Scene, where
- the top left corner is a video/capture source to display
- and the top right corner is the video/capture/browser source for the mask
(i used a countdown clock i've written myself in p5.js via browser source)
3. Apply Filter "Custom Effect" to scene and select .effect file created in step 1.
4. Apply Crop Filter and crop right and bottom to half width and height
5. Use Scene as source in another scene