[ Filter Request ] Remove Webcam Film Grain

dodgepong

dodgepong

Administrator
Community Helper
Usually the solution to grainy video is to improve your lighting conditions.

I'm not sure how good a live video denoiser could be in practice. Applying denoising to a single image is one thing, but applying denoising to 30 1080p images per second is another matter entirely.
 
calvinvalerian

calvinvalerian

New Member
dodgepong said:
Usually the solution to grainy video is to improve your lighting conditions.
Click to expand...
I'll try this suggestion as I was already planning on doing it. I really hope it's just lighting problem and not that my C922 was designed with noise.

dodgepong said:
I'm not sure how good a live video denoiser could be in practice. Applying denoising to a single image is one thing, but applying denoising to 30 1080p images per second is another matter entirely.
Click to expand...
Since COVID-19 pandemic everybody must stay home for quite a long time and I guess many more people will want to become youtuber as they don't have anything else to do right now, and adding denoising will definitely help since everybody's financial condition is at stagnant and buying high-end cameras might not be an option.
 
S

Shraeder

New Member
I know this thread is ages old by this point, but it would be nearly impossible to implement a feature like this for the following reasons:

1.) You cannot increase the resolution or remove noise from a still image or video without introducing distortion (or unintended artifacts). Improvements made to AI and video processing has gotten good, albeit still not perfect. Even Photoshop's effects aren't perfect on still images and can produce artifacts from time to time.

2.) It would require a significant amount of system resources to upscale or even add post processing to every single video frame as it's being streamed or recorded. That kind of feature is always at the bleeding edge of technology. It's one of the main reasons video games require some of the most powerful hardware in order to get decent frame rates. Every single frame, your graphics card and cpu work together to calculate lighting, geometry, texture, post processing (ie. filters).

3.) assuming you have the proper hardware and software support to do anything remotely useful to improve video quality on the fly, you could afford a better camera that wouldn't require a "denoiser" to begin with. You would need an absolute powerhouse of a GPU to do something like that, and a lot of decent GPUs that can do things like that in Blender to increase picture quality/remove noise cost upwards of $1000 and would have to be paired with a paid software. there's no way any sort of free open source software for recording your screen would offer that, considering that it would be a revolutionary piece of software.

P.S.
A lot of webcams will look grainy in low light conditions because they will automatically adjust their ISO, which is basically the frame time, to capture more light per frame. The upside is that you don't need to have perfect lighting all the time to make out people and objects, but the downside is that the image can look grainy because some pixels of the imaging sensor can be overexposed. The more light you have on yourself, the lower the ISO will be on the camera, thus improving image quality.
 
E

Eki Halkka

New Member
I've been toying around with the idea of a simple temporal noise reduction filter. I've dabbled with user-defined shaders, but writing proper plugins is beyond my expertise at this point - i'd write the plugin myself if i knew how. But here's some pseudo-code on how it would work, at it's simplest:

Create user-defined parameter "threshold"
Initiate a frame buffer "B"

Process frame
{
Load a video frame "A" from the source
For each pixel, calculate the difference between source "A" and buffer "B"
If the difference is above the threshold, use "A", else use "B"
Write the resulting image to buffer "B" (and output)
}

As far as i can tell, this should be quite easy on the processing resources.

I'd know how to code this as a user-defined shader, if i only knew a way to implement the "B" buffer. Perhaps it's not even possible in a shader. Unfortunately, i have not had as much time to spend on learning this as i would have liked to...

I recreated the idea in After Effects, and it seems to work pretty well when the threshold is set just above the noise level: in the areas of the image that do not move, noise is reduced by averaging multiple frames together, the areas that do move trigger the threshold and are used from the current frame of the source. They still have noise, but it's not really noticeable in moving subjects.

I did the tests using good quality source material (Prores 422 from BMCC and 100 Mbit mp4 from Sony A6300) - my aim is to get broadcast quality greenscreen results in obs*, and reducing the noise before keying greatly improves the semi-transparent areas - even a tiny amount of noise can be visible there. I'm not sure whether this approach would help much with lower quality sources, because the noise threshold is much higher, and raising it might cause visible "trailing" motion artifacts.

* I've written my own keyer as a user-defined shader. It works pretty well already, better than the built-in greenscreen key effect in some ways. Still not done with that though.
 
Last edited:
S

SolarSailor

New Member
Eki Halkka said:
I've been toying around with the idea of a simple temporal noise reduction filter.
Click to expand...
Exactly the same I've been thinking. I'm not sure I understand the attitudes of people above because temporal denoising isn't terribly hard to implement (there are many implementations on shadertoy to use, there's also hqdn3D which is a workhorse in video players), works pretty well in practice (video players, 3d rendering etc), and doesn't consume lots of processing power at all even on CPU, and the overhead is negligible when running on a GPU. The "just add more light" line of thinking isn't exactly productive because cheap webcams always have noise, even if you drop several times their cost into lighting. (which you should do anyway)
Two main reasons to have it:
  • Greatly improve the keying quality with a green screen as the slightest amount of noise tends to mess up with the results, even with a good camera
  • For users on a budget, it's a great way to improve the quality of cheap webcams with only cheap standard three-point lighting
The only obstacle, as you said, is inability to access to previous frames from a shader. This should probably be either posted as a feature/pull request to StreamFX/StreamFX Plus plugins github, or implemented as a separate plugin. Sadly, I'm not experienced enough to write one either.

That said, temporal denoising isn't the only denoise type that would be nice to have. Basic color denoising (within the same buffer) would improve the quality a lot already, and doesn't need access to previous frames. Additional deblocking finter is also an easy and great way to improve the quality of cheap webcams as they all use low-bandwidth USB 2.0 and are using horribly compressed streams; MJPEG artifacts are typically very visible, especially if you apply some color grading or sharpen the image afterwards.

(I'd like to see your keyer as well if you are interested in publishing it, especially if you use hue transfer to reduce the screen spilling!)
 
E

Eki Halkka

New Member
Yep, to all you wrote.

I've been toying around with spatial denoising too, but not yet with usable results...

My WIP shaders are here, some work, some don't, yet at least (i guess I should get into the whole GitHub thing, but drive it is for now):

Halsu_shaderfilter - Google Drive

drive.google.com drive.google.com

The latest keyer is Halsu_HybridKeyer_V014.

I actually made an overview video of an older version of the keyer, it's missing some features but all the essentials are there:

As far as the spill reduction part goes, I'm doing it in RGB space. I compare the green channel to a user-controlled mix of r and b channels, and if g is higher than the r/b mix, replace green with the r/b mix. Another option would be to use the maximum of r and b, but I like the control over the spill bias one gets with the mixing method. Maybe I'll add a checkbox to use this as an alternative method.

The keyer is also RGB, essentially just the maximum of rb minus g. I do some simple RGB preprocessing to improve the hue of the green before key, but better results could be obtained by rotating the green screen hue in HLS to pure green before keying. I'll probably implement this at some point.
 
R

recognitium

New Member
@Eki Halkka thanks a lot for sharing your WIP shaders / keyers. Has anybody translated them to GLSL ? It would be great to have versions that worked on linux. Also, there is obs-shaderfilter-plus now, seems little work to adapt it. I have done a little bit of trial and error, starting from zero knowledge on the subject, and haven't managed to get any productive result.
 
S

SCREAMING_SNAKE_CASE

New Member
I was able to get temporal denoising working in OBS by capturing my webcam with VLC through DirectShow, applying hqdn3d inside VLC, and piping it through NDI to OBS (which turned out to induce less latency and system load than the VLC video source).

Key takeaways:
  • hqdn3d works absolute wonders on lower end USB 2.0 webcams which stream terribly compressed and subsampled MJPEG garbage. Due to its temporal nature, it gives the appearance of deblocking and deringing without an actual filter, because of the lowered noise floor. So you can rely on sharpness filter in OBS instead of the trash built into the webcam itself, with much less worrying about compression artifacts becoming visible.
  • The temporal part of hqdn3d doesn't cause noticeable motion artifacts unless you are dancing before your camera. For a typical talking head scene, temporal denoising greatly improves the result. No excessive softness either.
  • Most noise seems to be chromatic. Luma denoising can be omitted for negligible quality loss.
  • Surprisingly, it doesn't hog the system too much at 1920x1080, 30fps.
As the users above mentioned, temporal denoising MASSIVELY improved my chroma keying quality. With a $50 chinese webcam from Aliexpress, I got the acceptable key under below average room lighting, with 99% of my hair preserved (something I never thought was possible with that kind of hardware). Huge thanks to @Eki Halkka , the shaders he posted above do an excellent job.

On quite a dated desktop system (i5-4670K @ 3.4GHz, GTX 970) the pipeline I described was using about 10% CPU and 20% GPU for OBS, plus ~16% CPU for VLC, all spread between the cores, running at 30fps and1920x1080 resolution. That's with a LUT, StreamFX color correction filter, @Eki Halkka 's hybrid keyer, and some Sharpness, all of this dumped into OBS virtual webcam output.

Worth mentioning that this is an ad hoc solution that does a lot of unnecessary processing and copying, applies the denoising to all channels, and does this on the CPU. With a native OBS plugin or a shader only applied to chroma, it must be possible to reduce the load further, so I totally agree on the usefulness of such a plugin.

(I don't even want to talk about what it took to beat VLC into submission with its directshow capture latency... ugh)
 
Last edited:
D

DefJeff

New Member
Eki Halkka said:
Yep, to all you wrote.

I've been toying around with spatial denoising too, but not yet with usable results...

My WIP shaders are here, some work, some don't, yet at least (i guess I should get into the whole GitHub thing, but drive it is for now):

Halsu_shaderfilter - Google Drive

drive.google.com drive.google.com

The latest keyer is Halsu_HybridKeyer_V014.

I actually made an overview video of an older version of the keyer, it's missing some features but all the essentials are there:

As far as the spill reduction part goes, I'm doing it in RGB space. I compare the green channel to a user-controlled mix of r and b channels, and if g is higher than the r/b mix, replace green with the r/b mix. Another option would be to use the maximum of r and b, but I like the control over the spill bias one gets with the mixing method. Maybe I'll add a checkbox to use this as an alternative method.

The keyer is also RGB, essentially just the maximum of rb minus g. I do some simple RGB preprocessing to improve the hue of the green before key, but better results could be obtained by rotating the green screen hue in HLS to pure green before keying. I'll probably implement this at some point.
Click to expand...
I know this is an old thread, but really wanting to use this with OBS v28 on MacOS. I have downgraded to OBS v27.1 to test this on Mac, but there is just a gray screen when I select the filter. My camera image does not even come through. This seems to be a great tool, I just cannot get it to work. Thanks in advance.
 

Attachments

  • Screen Shot 2022-10-26 at 3.14.41 PM.png
    Screen Shot 2022-10-26 at 3.14.41 PM.png
    62.1 KB · Views: 658
CurlyBlonde

CurlyBlonde

New Member
Eki Halkka said:
Yep, to all you wrote.

I've been toying around with spatial denoising too, but not yet with usable results...

My WIP shaders are here, some work, some don't, yet at least (i guess I should get into the whole GitHub thing, but drive it is for now):

Halsu_shaderfilter - Google Drive

drive.google.com drive.google.com

The latest keyer is Halsu_HybridKeyer_V014.

I actually made an overview video of an older version of the keyer, it's missing some features but all the essentials are there:

As far as the spill reduction part goes, I'm doing it in RGB space. I compare the green channel to a user-controlled mix of r and b channels, and if g is higher than the r/b mix, replace green with the r/b mix. Another option would be to use the maximum of r and b, but I like the control over the spill bias one gets with the mixing method. Maybe I'll add a checkbox to use this as an alternative method.

The keyer is also RGB, essentially just the maximum of rb minus g. I do some simple RGB preprocessing to improve the hue of the green before key, but better results could be obtained by rotating the green screen hue in HLS to pure green before keying. I'll probably implement this at some point.
Click to expand...

Please pardon my incompetence since I'm a novice, how does one apply / use your shaders in obs ?
I'm using Version 28.1.2 & I've found Shaderfilter Plus but upon selecting your denoising shader for example it gives this error :

Could not create the effect due to the following error:

device_pixelshader_create (D3D11): Compiler warnings/errors for C:/Users/meghi/Downloads/Halsu_noisereduction.shader (Pixel shader, technique Draw, pass 0):

C:\Users\meghi\Downloads\Halsu_noisereduction.shader (Pixel shader, technique Draw, pass 0)(10,12-26): error X3004: undeclared identifier 'render'



Pass (0) <> missing pixel shader!
Click to expand...
 
C

cyberjunk

New Member
Eki Halkka said:
Yep, to all you wrote.

I've been toying around with spatial denoising too, but not yet with usable results...

My WIP shaders are here, some work, some don't, yet at least (i guess I should get into the whole GitHub thing, but drive it is for now):

Halsu_shaderfilter - Google Drive

drive.google.com drive.google.com

The latest keyer is Halsu_HybridKeyer_V014.

I actually made an overview video of an older version of the keyer, it's missing some features but all the essentials are there:

As far as the spill reduction part goes, I'm doing it in RGB space. I compare the green channel to a user-controlled mix of r and b channels, and if g is higher than the r/b mix, replace green with the r/b mix. Another option would be to use the maximum of r and b, but I like the control over the spill bias one gets with the mixing method. Maybe I'll add a checkbox to use this as an alternative method.

The keyer is also RGB, essentially just the maximum of rb minus g. I do some simple RGB preprocessing to improve the hue of the green before key, but better results could be obtained by rotating the green screen hue in HLS to pure green before keying. I'll probably implement this at some point.
Click to expand...
is there any chance that you might be able to convert your keyer into the new format for the new version of the Shader Filter: ShaderFilter Plus: https://obsproject.com/forum/threads/obs-shaderfilter-plus.119594/
 
James Somerset

James Somerset

New Member
Eki Halkka said:
Yep, to all you wrote.

I've been toying around with spatial denoising too, but not yet with usable results...

My WIP shaders are here, some work, some don't, yet at least (i guess I should get into the whole GitHub thing, but drive it is for now):

Halsu_shaderfilter - Google Drive

drive.google.com drive.google.com

The latest keyer is Halsu_HybridKeyer_V014.

I actually made an overview video of an older version of the keyer, it's missing some features but all the essentials are there:

As far as the spill reduction part goes, I'm doing it in RGB space. I compare the green channel to a user-controlled mix of r and b channels, and if g is higher than the r/b mix, replace green with the r/b mix. Another option would be to use the maximum of r and b, but I like the control over the spill bias one gets with the mixing method. Maybe I'll add a checkbox to use this as an alternative method.

The keyer is also RGB, essentially just the maximum of rb minus g. I do some simple RGB preprocessing to improve the hue of the green before key, but better results could be obtained by rotating the green screen hue in HLS to pure green before keying. I'll probably implement this at some point.
Click to expand...
Hi @Eki Halkka,

Thank you so much for making this available. It's really valuable.

I've just downloaded your V022 version. You understand what all the settings do. I don't! So I tried changing them to see what happened but, with the numeric text box controls, I found it slow and fiddly. So, I did some Googling and found that it's possible to change the number boxes to sliders. See here: Exeldro's GitHub

In your code, I've taken every setting and tweaked their options, like this:

Code: 
uniform float Prekey_despill<
    string widget_type = "slider";
    float minimum = -1000.0;
    float maximum = 1000.0;
    float step = 1.0;
> = -1000.0;

In the OBS filter dialog, the numeric boxes become sliders, like this:

screenshot of Halsu HybridKeyer V022 with controls converted to sliders


Personally, I found this really helpful. With the slider controls, it's fast and easy to change the settings. As the changes are instantly visible, it's really helping me work out how to fine tune your filter.

I don't know enough about what effect each value has. It may be that having limits from -1000 to +1000 is too much for some settings. Until I understand this more, they are the limits I've used.

I'm documenting it here in case this is helpful to anyone else.

Thank you again @Eki Halkka—much appreciated.
 
E

Eki Halkka

New Member
James Somerset said:
In the OBS filter dialog, the numeric boxes become sliders, like this:
Click to expand...
Mmm... they already could use sliders, you just need to check the "use slider inputs" box.

But thanks for the heads up, it seems this can be forced within the code. Also, the slider range can be set properly using that same snippet. I do not think that was the case when I first read the docs. In my keyer, the range of every slider is the default -1000 to +1000 which is almost always manually converted to -1 to +1 range or similar. I guess I should read the docs again, in case I missed some other stuff ;-)

1696866596300.png
 
James Somerset

James Somerset

New Member
Hmmmm, that's odd: I don't see that option:

no sliders.png


Maybe I'm using a Shader plugin that your shader isn't designed for?

version.png


(That might also explain the other issues I'm having that I PMed you about!)

This is what I see unless I make the tweaks mentioned above:

num.png
 
G

geniusjy

New Member
Eki Halkka said:
I think i'm just using an older version of the shaderfilter. The checkbox maybe got removed when the feature to define sliders in-code was added?
Click to expand...
In the new versions of OBS, including 28, 29, and 30, as well as the plugin version 2.1.3, the slider functionality has indeed been removed. As someone without a coding background, it is challenging for me to implement slider functionality. The absence of slider functionality significantly increases the difficulty of operation. This plugin is excellent, and I feel fortunate to have encountered it. Thank you very much. I hope version 2.1.4 can fix the slider functionality.
 
T

TKK9000

New Member
Hey @Eki Halkka, thanks for your great work on the shaders :)
I would love to use your HybridKeyer but I can't get it to work on MacOS :(

I'm using obs-shaderfilter 2.1.3 and I get the following error (screenshot below)
Any suggestions how to proceed
 

Attachments

  • Screenshot 2023-11-20 at 16.35.50.png
    Screenshot 2023-11-20 at 16.35.50.png
    62.5 KB · Views: 104
You must log in or register to reply here.
Top