Bug Report NVENC Invalid Parameters with output resolution w > ~4096

[forbiddenera]

Hi,

Trying to record an NV surround setup, 3 screens (2x1680x1050, 1x1920x1080) setup in NV surround at 5164x1050 (inc. bezel correction)

Thought I'd try OBS since Geforce Experience/ShadowPlay doesn't want to record at that resolution (I can set to 1680x1050, hit record, set to 5164x1050 and it will continue recording but it is basically anamorphic, squeezing the entire play area down to 1680)

Guess the problem might actually be with NVENC, OBS will record up until I set the output resolution past about 4k on the width, the options presented being 3884x768 works and 4144x840 does not.

Using an RTX2060 w/8-core DP Xeon, 32-GB ram, Windows 10 1809, GeForce 425.31, OBS 23.1.0 64bit

Wikipedia says nvenc supports 8k on 20xx?

22:49:25.834: ------------------------------------------------ 22:49:32.680: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM) 22:49:32.718: --------------------------------- 22:49:32.718: [NVENC encoder: 'recording_h264'] settings: 22:49:32.718: rate_control: CBR 22:49:32.718: bitrate: 2500 22:49:32.718: cqp: 0 22:49:32.718: keyint: 250 22:49:32.718: preset: hq 22:49:32.718: profile: high 22:49:32.718: width: 5180 22:49:32.718: height: 1050 22:49:32.718: 2-pass: false 22:49:32.718: b-frames: 2 22:49:32.718: GPU: 0 22:49:32.718: 22:49:32.956: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library

 

[forbiddenera]

20:33:02.237: CPU Name: Intel(R) Xeon(R) CPU E5462 @ 2.80GHz 20:33:02.238: CPU Speed: 2793MHz 20:33:02.238: Physical Cores: 8, Logical Cores: 8 20:33:02.238: Physical Memory: 32758MB Total, 23828MB Free 20:33:02.238: Windows Version: 10.0 Build 17763 (revision: 437; 64-bit) 20:33:02.238: Running as administrator: false 20:33:02.238: Aero is Enabled (Aero is always on for windows 8 and above) 20:33:02.238: Windows 10 Gaming Features: 20:33:02.238: Game DVR: On 20:33:02.241: Sec. Software Status: 20:33:02.245: Windows Defender Antivirus: disabled (AV) 20:33:02.245: AVG Antivirus: enabled (AV) 20:33:02.246: Windows Firewall: enabled (FW) 20:33:02.247: Windows Defender Antivirus: disabled (ASW) 20:33:02.247: AVG Antivirus: enabled (ASW) 20:33:02.250: Current Date/Time: 2019-04-15, 20:33:02 20:33:02.250: Browser Hardware Acceleration: true 20:33:02.250: Portable mode: false 20:33:03.325: OBS 23.1.0 (64-bit, windows) 20:33:03.325: --------------------------------- 20:33:03.344: --------------------------------- 20:33:03.344: audio settings reset: 20:33:03.344: samples per sec: 44100 20:33:03.344: speakers: 2 20:33:03.348: --------------------------------- 20:33:03.348: Initializing D3D11... 20:33:03.348: Available Video Adapters: 20:33:03.354: Adapter 1: NVIDIA GeForce RTX 2060 20:33:03.354: Dedicated VRAM: 1975517184 20:33:03.354: Shared VRAM: 4289740800 20:33:03.355: output 1: pos={0, 0}, size={5180, 1050}, attached=true 20:33:03.360: Loading up D3D11 on adapter NVIDIA GeForce RTX 2060 (0) 20:33:03.469: D3D11 loaded successfully, feature level used: 45056

Tried a few other codec settings.. hevc_nvenc worked.. h264 wont, I read somewhere that 264 was updated to do 8k or something?

22:49:25.834: Settings changed (outputs) 22:49:25.834: ------------------------------------------------ 22:49:32.680: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM) 22:49:32.718: --------------------------------- 22:49:32.718: [NVENC encoder: 'recording_h264'] settings: 22:49:32.718: rate_control: CBR 22:49:32.718: bitrate: 2500 22:49:32.718: cqp: 0 22:49:32.718: keyint: 250 22:49:32.718: preset: hq 22:49:32.718: profile: high 22:49:32.718: width: 5180 22:49:32.718: height: 1050 22:49:32.718: 2-pass: false 22:49:32.718: b-frames: 2 22:49:32.718: GPU: 0 22:49:32.718: 22:49:32.956: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library 22:51:33.462: Switched to scene 'Scene' 22:51:33.468: Switched to regular Preview mode 22:51:33.468: ------------------------------------------------ 22:51:35.468: Switched to Preview/Program mode 22:51:35.468: ------------------------------------------------ 22:51:35.910: Switched to scene 'Scene' 22:51:35.913: Switched to regular Preview mode 22:51:35.913: ------------------------------------------------ 23:05:20.306: --------------------------------- 23:05:20.307: video settings reset: 23:05:20.307: base resolution: 5180x1050 23:05:20.307: output resolution: 2960x600 23:05:20.307: downscale filter: Bicubic 23:05:20.307: fps: 60/1 23:05:20.307: format: NV12 23:05:20.307: YUV mode: 601/Partial 23:05:20.307: NV12 texture support enabled 23:05:20.322: Settings changed (video) 23:05:20.322: ------------------------------------------------ 23:05:22.299: [jim-nvenc: 'recording_h264'] settings: 23:05:22.299: rate_control: CBR 23:05:22.299: bitrate: 2500 23:05:22.299: cqp: 20 23:05:22.299: keyint: 250 23:05:22.299: preset: hq 23:05:22.299: profile: high 23:05:22.299: width: 2960 23:05:22.299: height: 600 23:05:22.299: 2-pass: false 23:05:22.299: b-frames: 2 23:05:22.299: lookahead: false 23:05:22.299: psycho_aq: true 23:05:22.299: 23:05:22.324: --------------------------------- 23:05:22.325: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3 23:05:22.325: 23:05:22.356: ==== Recording Start =============================================== 23:05:22.356: [ffmpeg muxer: 'adv_file_output'] Writing file 'J:/gamerec/2019-04-15 23-05-22.mkv'... 23:05:28.542: [ffmpeg muxer: 'adv_file_output'] Output of file 'J:/gamerec/2019-04-15 23-05-22.mkv' stopped 23:05:28.542: Output 'adv_file_output': stopping 23:05:28.542: Output 'adv_file_output': Total frames output: 352 23:05:28.543: Output 'adv_file_output': Total drawn frames: 347 (371 attempted) 23:05:28.543: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 24 (6.5%) 23:05:28.544: Video stopped, number of skipped frames due to encoding lag: 28/367 (7.6%) 23:05:28.582: ==== Recording Stop ================================================ 23:05:49.696: --------------------------------- 23:05:49.696: video settings reset: 23:05:49.696: base resolution: 5180x1050 23:05:49.696: output resolution: 3884x786 23:05:49.696: downscale filter: Bicubic 23:05:49.696: fps: 60/1 23:05:49.696: format: NV12 23:05:49.696: YUV mode: 601/Partial 23:05:49.696: NV12 texture support enabled 23:05:49.709: Settings changed (video) 23:05:49.709: ------------------------------------------------ 23:05:51.471: [jim-nvenc: 'recording_h264'] settings: 23:05:51.471: rate_control: CBR 23:05:51.471: bitrate: 2500 23:05:51.471: cqp: 20 23:05:51.471: keyint: 250 23:05:51.471: preset: hq 23:05:51.471: profile: high 23:05:51.471: width: 3884 23:05:51.471: height: 786 23:05:51.471: 2-pass: false 23:05:51.471: b-frames: 2 23:05:51.471: lookahead: false 23:05:51.471: psycho_aq: true 23:05:51.471: 23:05:51.497: --------------------------------- 23:05:51.497: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3 23:05:51.497: 23:05:51.524: ==== Recording Start =============================================== 23:05:51.524: [ffmpeg muxer: 'adv_file_output'] Writing file 'J:/gamerec/2019-04-15 23-05-51.mkv'... 23:05:54.848: [ffmpeg muxer: 'adv_file_output'] Output of file 'J:/gamerec/2019-04-15 23-05-51.mkv' stopped 23:05:54.848: Output 'adv_file_output': stopping 23:05:54.848: Output 'adv_file_output': Total frames output: 187 23:05:54.848: Output 'adv_file_output': Total drawn frames: 199 23:05:54.849: ==== Recording Stop ================================================ 23:06:12.902: --------------------------------- 23:06:12.902: video settings reset: 23:06:12.902: base resolution: 5180x1050 23:06:12.902: output resolution: 4144x840 23:06:12.902: downscale filter: Bicubic 23:06:12.902: fps: 60/1 23:06:12.902: format: NV12 23:06:12.902: YUV mode: 601/Partial 23:06:12.902: NV12 texture support enabled 23:06:12.916: Settings changed (video) 23:06:12.916: ------------------------------------------------ 23:06:14.269: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM) 23:06:14.305: --------------------------------- 23:06:14.306: [NVENC encoder: 'recording_h264'] settings: 23:06:14.306: rate_control: CBR 23:06:14.306: bitrate: 2500 23:06:14.306: cqp: 0 23:06:14.306: keyint: 250 23:06:14.306: preset: hq 23:06:14.306: profile: high 23:06:14.306: width: 4144 23:06:14.306: height: 840 23:06:14.306: 2-pass: false 23:06:14.306: b-frames: 2 23:06:14.306: GPU: 0 23:06:14.306: 23:06:14.532: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library 23:00:38.908: Starting recording due to hotkey 23:45:23.026: Settings changed (outputs) 23:45:23.026: ------------------------------------------------ 23:45:24.558: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM) 23:45:24.593: --------------------------------- 23:45:24.593: [NVENC encoder: 'recording_h264'] settings: 23:45:24.593: rate_control: CBR 23:45:24.593: bitrate: 2500 23:45:24.593: cqp: 0 23:45:24.593: keyint: 250 23:45:24.593: preset: hq 23:45:24.593: profile: high 23:45:24.593: width: 4144 23:45:24.593: height: 840 23:45:24.593: 2-pass: false 23:45:24.593: b-frames: 2 23:45:24.593: GPU: 0 23:45:24.593: 23:45:24.843: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library 23:45:42.409: Settings changed (outputs) 23:45:42.409: ------------------------------------------------ 23:45:43.671: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM) 23:45:43.706: --------------------------------- 23:45:43.706: [NVENC encoder: 'recording_h264'] settings: 23:45:43.706: rate_control: CBR 23:45:43.706: bitrate: 2500 23:45:43.706: cqp: 0 23:45:43.706: keyint: 250 23:45:43.706: preset: hq 23:45:43.706: profile: high 23:45:43.706: width: 4144 23:45:43.706: height: 840 23:45:43.706: 2-pass: false 23:45:43.706: b-frames: 2 23:45:43.706: GPU: 0 23:45:43.706: 23:45:43.951: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library 23:46:08.245: Settings changed (outputs) 23:46:08.245: ------------------------------------------------ 23:46:09.605: --------------------------------- 23:46:09.605: [NVENC encoder: 'recording_h264'] settings: 23:46:09.605: rate_control: CBR 23:46:09.605: bitrate: 2500 23:46:09.605: cqp: 0 23:46:09.605: keyint: 250 23:46:09.605: preset: hq 23:46:09.605: profile: high 23:46:09.605: width: 3314 23:46:09.605: height: 672 23:46:09.605: 2-pass: false 23:46:09.605: b-frames: 2 23:46:09.605: GPU: 0 23:46:09.605: 23:46:10.002: --------------------------------- 23:46:10.003: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3 23:46:10.003: 23:46:10.029: ==== Recording Start =============================================== 23:46:10.029: [ffmpeg muxer: 'adv_file_output'] Writing file 'J:/gamerec/2019-04-15 23-46-09.mkv'... 23:46:11.616: [ffmpeg muxer: 'adv_file_output'] Output of file 'J:/gamerec/2019-04-15 23-46-09.mkv' stopped 23:46:11.616: Output 'adv_file_output': stopping 23:46:11.616: Output 'adv_file_output': Total frames output: 67 23:46:11.616: Output 'adv_file_output': Total drawn frames: 95 23:46:11.617: ==== Recording Stop ================================================ 23:46:11.697: Video stopped, number of skipped frames due to encoding lag: 29/82 (35.4%) 23:46:27.787: Settings changed (outputs) 23:46:27.787: ------------------------------------------------ 23:46:29.137: --------------------------------- 23:46:29.137: [NVENC encoder: 'recording_h264'] settings: 23:46:29.137: rate_control: CBR 23:46:29.137: bitrate: 2500 23:46:29.137: cqp: 0 23:46:29.137: keyint: 250 23:46:29.137: preset: hq 23:46:29.137: profile: high 23:46:29.137: width: 4144 23:46:29.137: height: 840 23:46:29.137: 2-pass: false 23:46:29.137: b-frames: 2 23:46:29.137: GPU: 0 23:46:29.137: 23:46:29.393: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library 23:48:31.074: Settings changed (outputs) 23:48:31.074: ------------------------------------------------ 23:48:32.564: ==== Recording Start =============================================== 23:48:36.101: Output 'adv_ffmpeg_output': stopping 23:48:36.102: Output 'adv_ffmpeg_output': Total frames output: 208 23:48:36.102: Output 'adv_ffmpeg_output': Total drawn frames: 235 23:48:36.102: ==== Recording Stop ================================================ 23:48:52.178: Settings changed (outputs) 23:48:52.178: ------------------------------------------------ 23:48:53.491: Failed to open video codec: Generic error in an external library 23:48:53.491: ffmpeg_data_init failed 23:48:53.491: ==== Recording Stop ================================================

 

[forbiddenera]

Ya, hevc/h265 works, h264 gives error.. is it going to be possible for me to stream only using hevc? I don't really care about recording if I'm going to edit and mux it myself before any distribution but if I wanted to livestream at >4k ..?

 

[forbiddenera]

h264 level 6.2 can do:

3,840×2,160@300.0 (16)
7,680×4,320@128.9 (5)
8,192×4,320@120.9 (5)

I'm thinking h264 may be more feasible than h265 for streaming letalone any realtime encoding.

 

[koala]

The maximum resolution for nvenc h.264 is 4096x4096. For nvenc h.265, the maximum resolution is 8192x8192.
See here: https://developer.nvidia.com/nvidia-video-codec-sdk#NVENCFeatures

h.265 isn't supported by streaming services, and it is available only with ffmpeg output of OBS, so for almost all cases you have to stick with 4096x4096 if you want to use hardware acceleration for streaming.
If you don't need to stream but simply produce > 8k footage for later upload, you can use ffmpeg output of OBS for h.265 and recode with x264 with an external app like Handbrake. Software encoder x264 doesn't have a resolution limit.

 

[waliiido]

How to get "libx265" listed in my recording settings? I have 3080ti and the latest OBS. I checked all settings.

 

[koala]

You can use hevc (which implements h.265), either available with the upcoming OBS 28 (currently beta for test) comfortably in advanced output mode available from here: https://obsproject.com/forum/list/test-builds.20/
or with the current stable 27.x you can use custom ffmpeg output as described here:

Recording of 3x 4k trading screens

Hi All, I would appreciate your thoughts. I am a trader and am needing a solution to record a day's worth of trading. I use 3 4k 43" screens with NVIDIA Quadro per desk. These show many graphs/charts of live market data (but no video content). At the end of the day I just want to extract and...

obsproject.com

 

[waliiido]

You can use hevc (which implements h.265), either available with the upcoming OBS 28 (currently beta for test) comfortably in advanced output mode available from here: https://obsproject.com/forum/list/test-builds.20/
or with the current stable 27.x you can use custom ffmpeg output as described here:

Recording of 3x 4k trading screens

Hi All, I would appreciate your thoughts. I am a trader and am needing a solution to record a day's worth of trading. I use 3 4k 43" screens with NVIDIA Quadro per desk. These show many graphs/charts of live market data (but no video content). At the end of the day I just want to extract and...

obsproject.com

Hi Koala, the setting you provided worked great, thanks. Unfortuounaly, the video size I recorded was choppy; I am limited by hardware, I think.
From your experience, what would be the no1 upgrade I should have for a smoother recording? What GPU? Here is a report:

 

[koala]

Physically remove the 1080 Ti and only use the 3080. The physical presence of a 2nd GPU in your mainboard divides the available 1x16 pci-express throughput to 2x8, one x8 for each GPU, so both only get half the bus throughput.

However, it's not sure this is the bottleneck for your case. I don't have any experience with resolutions that high. Try other presets in the encoder configuration, for example "performance" instead of "quality". This actually doesn't lower visual quality for recording as far as I know, it will just increase the file size slightly. For streaming, this would be bad, as it requires more bandwidth, but for recording disk space is irrelevant.

How performance hungry the preset is can be seen here: https://docs.nvidia.com/video-technologies/video-codec-sdk/nvenc-application-note/#nvenc-performance so it is worth to do test runs with every preset and see what performance can be achieved.

 

[FerretBomb]

Modern NVENC can generally support up to 2 4K60 (3840x2160) encoding streams simultaneously, maybe slightly more as a margin for error.

That's on-par with what @waleedovkh is seeing (able to run 7680x2160). 7680x3840 is nearly double that maximum by pixel-count, and absolutely will exceed what NVENC is able to encode.

He will need to swap over to CPU-based encoding, and likely invest in a multi-processor Threadripper or EPYC setup to be able to do it in real-time.

 

[waliiido]

Modern NVENC can generally support up to 2 4K60 (3840x2160) encoding streams simultaneously, maybe slightly more as a margin for error.

That's on-par with what @waleedovkh is seeing (able to run 7680x2160). 7680x3840 is nearly double that maximum by pixel-count, and absolutely will exceed what NVENC is able to encode.

He will need to swap over to CPU-based encoding, and likely invest in a multi-processor Threadripper or EPYC setup to be able to do it in real-time.

But NVENC is capable of handling up to 8192x8192 as far as I know; there is a bottleneck somewhere. Having my GPU @ %100 while recording is the limit I think. I just wonder if a 3090ti would solve this.

 

[FerretBomb]

But NVENC is capable of handling up to 8192x8192 as far as I know; there is a bottleneck somewhere. Having my GPU @ %100 while recording is the limit I think. I just wonder if a 3090ti would solve this.

It would not. The NVENC core is the same across all GPUs in the line, and is a separate section of the die from the 3D rendering side of the silicon.

Just because it will accept 8192x8192 as a valid resolution does not mean it is capable of encoding at that resolution smoothly. That's just bare standards-compliance with h.265/HEVC.
It might also be able to run an encode of that resolution at a very low framerate, no guarantees.

Again, the maximum that Turing will handle (smoothly) is just a bit north of 2x4K60 video streams.
Around 1 billion pixels per second; 3840x2160x2x60=995,328,000. It falls off shortly after that.
So a 8192x8192 feed MIGHT be able to be processed, IF it was 14-15fps.

If raw-pixel-count holds, it looks like your 7680x3840 setup MIGHT be able to encode smoothly somewhere in the 30-35fps range at most. As there was no logfile, I'm not sure what framerate you're trying to record at, but if it's 60 I'm entirely unsurprised.

 

[koala]

Having my GPU @ %100 while recording is the limit I think. I just wonder if a 3090ti would solve this.

Before you invest in different hardware, you need to make sure you actually narrowed down the issue. Windows task manager is able to show the load of the different GPU modules:

Here you can see how the "Video encode" went to to 26% after I started recording with OBS. This means the encoder still has free resources for another 2 encoding sessions and still get no issue with encoding. Or get 3-4 times the current amount of pixels. Your 100% GPU might very well be with 3D, which has nothing to do with video encoding but just displaying 3D images such as with a game.