Encoding for very (very) high noise / motion in OBS

[Sphynx]

Hi, not sure if this is necessarily the best place to ask but I figure it's a good start.
I need some input from people who are very knowledgeable about encoding in general.

I stream / record analog "glitch art" and have a pretty involved chain that involves downscaling signals from my PC to go into an analog hardware chain and then it gets upscaled back to 1080p by a retrotink 5x and captured by an avermedia live gamer hd2.

I have a pretty decent understanding of encoder settings in general for "normal" streaming but when it comes to the most extreme edge cases in terms of noise and motion I'm having a hard time finding much documentation or discussion. Currently I just use a CBR with a constant bitrate of 6000 for streaming (twitch "limit") and local recordings are done at 10-12k. Encoding is done with NVENC on a 3080.

I guess my question is if you were hypothetically trying to encode pure static (worst case scenario?) is CBR with the highest bitrate your system/service can handle the best option to minimize artifacts with high noise/motion? I've researched most of the settings for NVENC and encoding modes but very few really talk about their particular impact on high noise and motion, most discussion, documentation and tutorials are really focused on "general quality" of fairly normal content instead of edge cases like mine. I haven't messed with CQP or VBR much but conceptually it seems like both would be worse in these kinds of scenarios.

I'm definitely trying to find things that will help the most when it comes to live streaming (with Twitch's incredibly low bitrate cap) since I prefer doing this video art stuff live just noodling. Recordings are mostly fine just going with a really high bitrate (rip storage space), but I only do them when I am working on something like a music video where I'm capturing and editing.

A few things I've already tried just for reference (feel free to skip this stuff since most of it is pretty basic troubleshooting)

Helped:

1. Streaming at 720p60 instead of the preferred 1080p60 - (this obviously has more impact when streaming with the low Twitch bitrate)
2. Local recording at very high CBR bitrates - Not a solution for streaming, works for local recordings with large file size (which is mostly fine)
3. Retrotink Post Processing aperture grille - My retrotink 5x has a bunch of post processing options to simulate various types of CRTs, PVMs and BVMs. I've found that a couple of the simulated CRT shadow masks and grilles really help encoding, while some make it much worse. I'm guessing this is because it's introducing a lot of stable pixels into the video that don't move and don't really change color (except for some simulated bloom).

Didn't Help:

1. Dropping the FPS to 30 - This didn't make as much difference as I thought it would and imo it also ruins the look of a lot of the analog NTSC glitches which are happening at 60hz. I guess it could potentially be an option for the most extreme cases.
2. Going above the Twitch bitrate limit - This did help, but I only tried this once or twice doing 1080p60 at 8k bitrate (i'm an affiliate btw not partner). It did look slightly better, and even though I had encoding resolution options while streaming I still had a lot of viewers that were having buffering issues even when set to low resolutions. I guess there is some sort of deal with Twitch's internal encoding for downscaling where having a high bitrate still causes issues with lower resolution encodes from their servers. Also I think Twitch would likely just throw errors and not push my stream through if I went much higher than 8k.
3. CPU Encoding - I rely on my 3080's NVENC for pretty much all my stream encoding. I know that quality can technically be better on CPU encoding but I'm running an i7-8700k overclocked, and even though the software I use for my video stuff isn't particularly CPU heavy, this is just not a great CPU for encoding. I do have a 2nd PC that I can encode with but the CPU is even worse and struggles to encode 1080p60 on CPU even when being used as a dedicated capture PC. I'm planning on upgrading CPU's soon, but I don't know if going to CPU encoding would actually help.
4. Messing with most NVENC settings - It's hard to do a like for like comparison with some of these. I haven't been able to find any info at all about how things like psychovisual tuning, look ahead, keyframe intervals and max b-frames specifically affect high noise and high motion content. For the most part everything I find about these is about general image quality and I've basically set everything to what most guides say is "highest quality".

Last thing for reference is my current NVENC settings for streaming (Recording is the same just with the bitrate cranked to 10-12k)

CBR - 6000Kbps
Keyframe Interval - 2
Preset - Max Quality
Profile - High
Look Ahead - On
Psycho Visual Tuning - On
Max B-Frames - 3

TLDR: What encoding mode / settings are best if you are trying to stream something very high noise/motion like static?

Thanks, sorry for the long post.

 

[sandrix]

1. Reducing the output resolution and fps, this is what allows you to save bitrate and improve the quality of the broadcast. All this is relevant for Twitch. You can use 1664x936, 1600x900, 1536x864 resolutions.
720p I do not recommend using, because. You lose a lot of image clarity. In short, there must be reasons for this.
2. If you are a twitch companion, then you can safely use 8000 kbps bitrate for streaming. If there are no problems with network frame drops.
3. If you are recording/broadcasting dynamic video, disable look-ahead, set 2 b-frames, enable Psycho Visual Tuning.
4. For low motion content. Increase the number of b-frames to 4, enable look-ahead and Psycho Visual Tuning. This will significantly improve the quality when the picture in the video is still.
5. For recording, you can compensate for everything with a high bitrate. If you need excessive quality, you can use the "Lossless" preset. But the video size can be huge. I usually just set the bitrate to CBR 50000 kbps for fast paced games and half as much for low motion content.
I have absolutely all the data about each parameter in the NVENC, x264 encoder, etc., but unfortunately I can't post it here due to moderation. I'll try something.

 

[sandrix]

Setting up OBS Studio to stream to Twitch via NVENC H.264 - In this guide you will find a lot of interesting things, but you will need a google translator
- Psycho-visual Tuning
- look-ahead
- NVIDIA Nvenc Obs Guide

 

[koala]

The only thing that can really improve quality if there is a high noise/high motion video to encode is more bitrate. High noise is killing the compression. There may be somewhat better or somewhat less good encoder settings, but in the end only more bitrate can make a real change.
However, you cannot use infinitely high bitrate, because your viewers don't have infinite download, and most streaming providers are limiting the bitrate.

As far as I see it, my recommendation is to look for a different kind of presentation of this stuff, so the bitrate requirement lowers. Show only smaller parts, picture in picture, not complete fullscreen images full of noise.

 

[Corduroy]

As far as I see it, my recommendation is to look for a different kind of presentation of this stuff, so the bitrate requirement lowers. Show only smaller parts, picture in picture, not complete fullscreen images full of noise.

I came here actually looking for a plugin that would do this. My problem is with FPS games with high foliage, making the image very mosaic. When I edit the videos, I export a lower bitrate version and use it as a backdrop for an overlay of the same video, but just the center, at its highest quality. Because all the action is when Im shooting, the center of the screen is the most important.
I wonder if a solution for streaming could be made like a center-weighted bitrate option.

 

[PaiSand]

I came here actually looking for a plugin that would do this. My problem is with FPS games with high foliage, making the image very mosaic. When I edit the videos, I export a lower bitrate version and use it as a backdrop for an overlay of the same video, but just the center, at its highest quality. Because all the action is when Im shooting, the center of the screen is the most important.
I wonder if a solution for streaming could be made like a center-weighted bitrate option.

That could be achived with AI, and all the power and resources it needs to do so.
A cheaper way is to have a 2 PC setup, so you game in one PC and stream/recod with the other, connecting both with a capture good card. This way you get better image quality and reduce the amount of resources used by the game and OBS.

 

[Sphynx]

Thanks for the suggestions. A few of the things I already knew but I'll try out disabling look-ahead and lowering the b-frames.

I definitely can stream at a higher bitrate it's just some of my viewers who don't have the greatest connections get a lot of buffering / connection errors, even on lower resolution encodes. I can experiment more though because I'm sure those people also have some issues just at 6k. I can recommend people turn off low latency mode and stuff.

The other alternative I know is just using youtube streaming instead of twitch which has higher encoding and resolution limits, but I have like 1/10th the followers on youtube and mainly just use it as a stream archive. I really wish twitch would take some time and money to upgrade their infrastructure a bit.

Anyway thanks, I'm open to any more suggestions if anyone has it. If anyone is curious the type of video I'm talking about here is a vod from a recent session. You can skip around and see how some stuff encodes fine and some is really rough, it depends on a lot of factors though.

This is 6k 720p60 https://www.twitch.tv/videos/1592333993
For the most part it looks fine outside some really noisy bits, I just wish I could do 1080p60 without it turning into a mess.

Part of me asking this too is just because I like to have VODs as archives of these live video jams I do to save local hard drive space, but I guess I should bite the bullet and get some huge externals drives just to dedicate to higher quality local recordings for this stuff if I really want to have decent quality archives.

 

[sandrix]

If you're a companion, as you stated, then you almost always get transcoded videos on twitch. This will allow your viewers to run your stream at a lower resolution and bitrate. You don't have to worry about video buffering in your viewers if you have transcoding. So you can set the bitrate to 8000 kbps. I have been using this value for many years, although many streamers do not think about it. I think you are being too picky, don't go too far.

 

[Corduroy]

That could be achived with AI, and all the power and resources it needs to do so.
A cheaper way is to have a 2 PC setup, so you game in one PC and stream/recod with the other, connecting both with a capture good card. This way you get better image quality and reduce the amount of resources used by the game and OBS.

The goal of that isnt to save performance, and it wouldnt save any performance anyways. Its so people who cant stream at high quality can pick and choice the "focal points" of higher quality video. With FPS gaming, usually the center is the most important.
The performance tax for this would be equivalent to hosting two streams and compositing one over the other. This can be done now by using two separate streaming softwares and overlaying a cropped sourced material over already streamed, and lower bitrate, material, and at a delay.
Edit: Blurs/mosaic overlays produce the same effect