Yep, to all you wrote.
I've been toying around with spatial denoising too, but not yet with usable results...
My WIP shaders are here, some work, some don't, yet at least (i guess I should get into the whole GitHub thing, but drive it is for now):
drive.google.com
The latest keyer is Halsu_HybridKeyer_V014.
I actually made an overview video of an older version of the keyer, it's missing some features but all the essentials are there:
As far as the spill reduction part goes, I'm doing it in RGB space. I compare the green channel to a user-controlled mix of r and b channels, and if g is higher than the r/b mix, replace green with the r/b mix. Another option would be to use the maximum of r and b, but I like the control over the spill bias one gets with the mixing method. Maybe I'll add a checkbox to use this as an alternative method.
The keyer is also RGB, essentially just the maximum of rb minus g. I do some simple RGB preprocessing to improve the hue of the green before key, but better results could be obtained by rotating the green screen hue in HLS to pure green before keying. I'll probably implement this at some point.
