NVENC file recording encoding of high quality 4K fails on RTX 3060

[goofyseeker3]

OBS claims encoder becomes overloaded when outputting 4K h.264 video file at high quality medium file size setting.
stream quality at 4K (6000kbit/s) and 4K to 1080p works fine. the OBS machine is encoding the video from a separate Xbox/PC.
when recording outputting 4K, the preview screen video runs very slow. I thought that RTX 3060 or any RTX 3000+ could handle
all kinds of 4K recording compression. so whats wrong or is RTX 3060 NVENC just slower. or does NVENC chip have a performance cap.
even recording windows desktop without any fast moving fps stuff, will get the "encoder is overloaded". what's wrong?

edit: windows task manager shows 97% video encode usage. are any other video encoder solutions (RTX or other) any better.
I think the high quality, medium file size recording setting will overload NVENC chip by default.
50 Mbps stream quality recordings should work but they only record at stream bit rate cap.

 

[goofyseeker3]

when encoding using hardware (amd, h.264) using integrated gpu, the recording quality drops to 600kbps for high quality and medium file size. and the output quality is terrible. dont know what affects that, obs settings or amd gpu internal workings.

 

[koala]

Nvenc on any RTX GPU is able to encode 4k 60 fps content fine. If it doesn't work for you, you set unusual/non default settings. However, you need to post a log for us to see what is happening.

 

[goofyseeker3]

according to nvidia nvenc performance table, it should be mostly fine and same for 3000 series and close to 4000 series performance. but I had to drop the quality to P4 from P7 to get the encoder to be below 100%. now its at 70%. note: default streaming to twitch at 6000kbps and similar will work. and now it works for recording at 35000kbps CBR at P4 quality. none of the not-same-as-stream defaults worked for me.

I have been able to encode 4K content (to 1080p) before just fine, now when outputting 4K quality content the problems started.

 

[goofyseeker3]

also: I use the elgato 4K X capture card in OBS using NDI through 4K capture utility stream link in 4K mode.

 

[koala]

Encoding 4k instead of 1080p requires 4 times the encoder resources (it's 4 times the amount of pixels), encoding with higher presets also requires significantly more resources, so find a balance with the parameters. There's nothing wrong or broken at your end. You just cannot expect infinite performance from nvenc.

 

[goofyseeker3]

I expected that nvenc would just fully encode 4k in all situations no questions asked. but that was clearly not the case.
4K encoding to 1080p works in all cases tho.

 

[goofyseeker3]

I wonder why nvidia and/or amd and intel, have not implement gpu encoding/decoding (as encoding/decoding chip mode), in addition to video encoder/decoder chips. or at least gpu encoding instead of nvenc/chip encoding should be an option in OBS. yep when you have the gpu doing nothing else than encoding. capturing/streaming from other source, like xbox or pc.

 

[goofyseeker3]

Solution: If you have an AMD GPU, discrete or integrated, which is not being otherwise used, like in a streaming pc,
then you can use AMD HW h.264 (AVC) at maximum quality 100Mbps CBR for 4K encoding.
Unlike Nvidia NVENC, it will not claim overloading of encoding, but shows high gpu utilization.

Why is the CBR bitrate capped to 100Mbps. it should be at least 200Mbps.
Also why the AMD HW encoder produces very small bitrate video files with poor quality
when using same CQP or file quality settings as for NVIDIA nvenc encoding.

 

[goofyseeker3]

why does OBS have 10% cpu usage while doing nothing, just to copy elgato 4K X video buffer to gpu memory?
This issue has not yet resolved. does the 4K to 4K encode with nvenc fail because OBS cannot copy 4K buffers around fast enough?
This situation is different from direct game capture on same computer, because the external pc needs to do the capture card transfer operation.

edit: yes the cpu 10% load is because the video capture device is active. checking deactivate when not showing makes cpu load about 0%.
if there is interference on the main OBS cpu thread from the video buffer copying with some encoding operations, then it might explain why it does not work. epoxvox said 4K to 4K at 60fps encoding should work easily even on older cards, like 1080.

 

[goofyseeker3]

Why is GPU (3D) at 20% when only showing active video capture device on OBS.
GPU (Copy) is at 5% when the video capture device is deactivated in OBS.

Is OBS limited to 1 CPU thread while using NVENC.

 

[goofyseeker3]

If OBS is limited to 1 CPU thread during NVENC, it would explain the 4K to 4K encoding jams.

Also windows task manager reports OpenShot gpu usage as being OBS gpu 3D usage. a bug.

 

[goofyseeker3]

whats with the simple settings high quality medium file size recording settings working at 70% NVENC HEVC vs advanced settings instantly hitting 100% encoder overload, 30Mbps CBR or CQ 15 NVENC HEVC both on RTX 3060. is the advanced settings bugged with something overloading by default vs simple settings.

 

[goofyseeker3]

finally using CQ level 23 and P5 preset at 4K to 4K no rescale output in advanced mode did produce the same 70% nvenc usage as the simple settings high quality medium file size profile. and does not overload encoding.

I wonder why CQ level 15 works with P5 without overloading but not with P2.