Bug Report NVENC Invalid Parameters with output resolution w > ~4096

forbiddenera

New Member
Hi,

Trying to record an NV surround setup, 3 screens (2x1680x1050, 1x1920x1080) setup in NV surround at 5164x1050 (inc. bezel correction)

Thought I'd try OBS since Geforce Experience/ShadowPlay doesn't want to record at that resolution (I can set to 1680x1050, hit record, set to 5164x1050 and it will continue recording but it is basically anamorphic, squeezing the entire play area down to 1680)

Guess the problem might actually be with NVENC, OBS will record up until I set the output resolution past about 4k on the width, the options presented being 3884x768 works and 4144x840 does not.

Using an RTX2060 w/8-core DP Xeon, 32-GB ram, Windows 10 1809, GeForce 425.31, OBS 23.1.0 64bit

Wikipedia says nvenc supports 8k on 20xx?


22:49:25.834: ------------------------------------------------
22:49:32.680: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM)
22:49:32.718: ---------------------------------
22:49:32.718: [NVENC encoder: 'recording_h264'] settings:
22:49:32.718: rate_control: CBR
22:49:32.718: bitrate: 2500
22:49:32.718: cqp: 0
22:49:32.718: keyint: 250
22:49:32.718: preset: hq
22:49:32.718: profile: high
22:49:32.718: width: 5180
22:49:32.718: height: 1050
22:49:32.718: 2-pass: false
22:49:32.718: b-frames: 2
22:49:32.718: GPU: 0
22:49:32.718:
22:49:32.956: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
 

forbiddenera

New Member
20:33:02.237: CPU Name: Intel(R) Xeon(R) CPU E5462 @ 2.80GHz
20:33:02.238: CPU Speed: 2793MHz
20:33:02.238: Physical Cores: 8, Logical Cores: 8
20:33:02.238: Physical Memory: 32758MB Total, 23828MB Free
20:33:02.238: Windows Version: 10.0 Build 17763 (revision: 437; 64-bit)
20:33:02.238: Running as administrator: false
20:33:02.238: Aero is Enabled (Aero is always on for windows 8 and above)
20:33:02.238: Windows 10 Gaming Features:
20:33:02.238: Game DVR: On
20:33:02.241: Sec. Software Status:
20:33:02.245: Windows Defender Antivirus: disabled (AV)
20:33:02.245: AVG Antivirus: enabled (AV)
20:33:02.246: Windows Firewall: enabled (FW)
20:33:02.247: Windows Defender Antivirus: disabled (ASW)
20:33:02.247: AVG Antivirus: enabled (ASW)
20:33:02.250: Current Date/Time: 2019-04-15, 20:33:02
20:33:02.250: Browser Hardware Acceleration: true
20:33:02.250: Portable mode: false
20:33:03.325: OBS 23.1.0 (64-bit, windows)
20:33:03.325: ---------------------------------
20:33:03.344: ---------------------------------
20:33:03.344: audio settings reset:
20:33:03.344: samples per sec: 44100
20:33:03.344: speakers: 2
20:33:03.348: ---------------------------------
20:33:03.348: Initializing D3D11...
20:33:03.348: Available Video Adapters:
20:33:03.354: Adapter 1: NVIDIA GeForce RTX 2060
20:33:03.354: Dedicated VRAM: 1975517184
20:33:03.354: Shared VRAM: 4289740800
20:33:03.355: output 1: pos={0, 0}, size={5180, 1050}, attached=true
20:33:03.360: Loading up D3D11 on adapter NVIDIA GeForce RTX 2060 (0)
20:33:03.469: D3D11 loaded successfully, feature level used: 45056


Tried a few other codec settings.. hevc_nvenc worked.. h264 wont, I read somewhere that 264 was updated to do 8k or something?


22:49:25.834: Settings changed (outputs)
22:49:25.834: ------------------------------------------------
22:49:32.680: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM)
22:49:32.718: ---------------------------------
22:49:32.718: [NVENC encoder: 'recording_h264'] settings:
22:49:32.718: rate_control: CBR
22:49:32.718: bitrate: 2500
22:49:32.718: cqp: 0
22:49:32.718: keyint: 250
22:49:32.718: preset: hq
22:49:32.718: profile: high
22:49:32.718: width: 5180
22:49:32.718: height: 1050
22:49:32.718: 2-pass: false
22:49:32.718: b-frames: 2
22:49:32.718: GPU: 0
22:49:32.718:
22:49:32.956: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
22:51:33.462: Switched to scene 'Scene'
22:51:33.468: Switched to regular Preview mode
22:51:33.468: ------------------------------------------------
22:51:35.468: Switched to Preview/Program mode
22:51:35.468: ------------------------------------------------
22:51:35.910: Switched to scene 'Scene'
22:51:35.913: Switched to regular Preview mode
22:51:35.913: ------------------------------------------------
23:05:20.306: ---------------------------------
23:05:20.307: video settings reset:
23:05:20.307: base resolution: 5180x1050
23:05:20.307: output resolution: 2960x600
23:05:20.307: downscale filter: Bicubic
23:05:20.307: fps: 60/1
23:05:20.307: format: NV12
23:05:20.307: YUV mode: 601/Partial
23:05:20.307: NV12 texture support enabled
23:05:20.322: Settings changed (video)
23:05:20.322: ------------------------------------------------
23:05:22.299: [jim-nvenc: 'recording_h264'] settings:
23:05:22.299: rate_control: CBR
23:05:22.299: bitrate: 2500
23:05:22.299: cqp: 20
23:05:22.299: keyint: 250
23:05:22.299: preset: hq
23:05:22.299: profile: high
23:05:22.299: width: 2960
23:05:22.299: height: 600
23:05:22.299: 2-pass: false
23:05:22.299: b-frames: 2
23:05:22.299: lookahead: false
23:05:22.299: psycho_aq: true
23:05:22.299:
23:05:22.324: ---------------------------------
23:05:22.325: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3
23:05:22.325:
23:05:22.356: ==== Recording Start ===============================================
23:05:22.356: [ffmpeg muxer: 'adv_file_output'] Writing file 'J:/gamerec/2019-04-15 23-05-22.mkv'...
23:05:28.542: [ffmpeg muxer: 'adv_file_output'] Output of file 'J:/gamerec/2019-04-15 23-05-22.mkv' stopped
23:05:28.542: Output 'adv_file_output': stopping
23:05:28.542: Output 'adv_file_output': Total frames output: 352
23:05:28.543: Output 'adv_file_output': Total drawn frames: 347 (371 attempted)
23:05:28.543: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 24 (6.5%)
23:05:28.544: Video stopped, number of skipped frames due to encoding lag: 28/367 (7.6%)
23:05:28.582: ==== Recording Stop ================================================
23:05:49.696: ---------------------------------
23:05:49.696: video settings reset:
23:05:49.696: base resolution: 5180x1050
23:05:49.696: output resolution: 3884x786
23:05:49.696: downscale filter: Bicubic
23:05:49.696: fps: 60/1
23:05:49.696: format: NV12
23:05:49.696: YUV mode: 601/Partial
23:05:49.696: NV12 texture support enabled
23:05:49.709: Settings changed (video)
23:05:49.709: ------------------------------------------------
23:05:51.471: [jim-nvenc: 'recording_h264'] settings:
23:05:51.471: rate_control: CBR
23:05:51.471: bitrate: 2500
23:05:51.471: cqp: 20
23:05:51.471: keyint: 250
23:05:51.471: preset: hq
23:05:51.471: profile: high
23:05:51.471: width: 3884
23:05:51.471: height: 786
23:05:51.471: 2-pass: false
23:05:51.471: b-frames: 2
23:05:51.471: lookahead: false
23:05:51.471: psycho_aq: true
23:05:51.471:
23:05:51.497: ---------------------------------
23:05:51.497: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3
23:05:51.497:
23:05:51.524: ==== Recording Start ===============================================
23:05:51.524: [ffmpeg muxer: 'adv_file_output'] Writing file 'J:/gamerec/2019-04-15 23-05-51.mkv'...
23:05:54.848: [ffmpeg muxer: 'adv_file_output'] Output of file 'J:/gamerec/2019-04-15 23-05-51.mkv' stopped
23:05:54.848: Output 'adv_file_output': stopping
23:05:54.848: Output 'adv_file_output': Total frames output: 187
23:05:54.848: Output 'adv_file_output': Total drawn frames: 199
23:05:54.849: ==== Recording Stop ================================================
23:06:12.902: ---------------------------------
23:06:12.902: video settings reset:
23:06:12.902: base resolution: 5180x1050
23:06:12.902: output resolution: 4144x840
23:06:12.902: downscale filter: Bicubic
23:06:12.902: fps: 60/1
23:06:12.902: format: NV12
23:06:12.902: YUV mode: 601/Partial
23:06:12.902: NV12 texture support enabled
23:06:12.916: Settings changed (video)
23:06:12.916: ------------------------------------------------
23:06:14.269: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM)
23:06:14.305: ---------------------------------
23:06:14.306: [NVENC encoder: 'recording_h264'] settings:
23:06:14.306: rate_control: CBR
23:06:14.306: bitrate: 2500
23:06:14.306: cqp: 0
23:06:14.306: keyint: 250
23:06:14.306: preset: hq
23:06:14.306: profile: high
23:06:14.306: width: 4144
23:06:14.306: height: 840
23:06:14.306: 2-pass: false
23:06:14.306: b-frames: 2
23:06:14.306: GPU: 0
23:06:14.306:
23:06:14.532: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
23:00:38.908: Starting recording due to hotkey
23:45:23.026: Settings changed (outputs)
23:45:23.026: ------------------------------------------------
23:45:24.558: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM)
23:45:24.593: ---------------------------------
23:45:24.593: [NVENC encoder: 'recording_h264'] settings:
23:45:24.593: rate_control: CBR
23:45:24.593: bitrate: 2500
23:45:24.593: cqp: 0
23:45:24.593: keyint: 250
23:45:24.593: preset: hq
23:45:24.593: profile: high
23:45:24.593: width: 4144
23:45:24.593: height: 840
23:45:24.593: 2-pass: false
23:45:24.593: b-frames: 2
23:45:24.593: GPU: 0
23:45:24.593:
23:45:24.843: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
23:45:42.409: Settings changed (outputs)
23:45:42.409: ------------------------------------------------
23:45:43.671: [jim-nvenc: 'recording_h264'] init_encoder: nv.nvEncInitializeEncoder(enc->session, params) failed: 8 (NV_ENC_ERR_INVALID_PARAM)
23:45:43.706: ---------------------------------
23:45:43.706: [NVENC encoder: 'recording_h264'] settings:
23:45:43.706: rate_control: CBR
23:45:43.706: bitrate: 2500
23:45:43.706: cqp: 0
23:45:43.706: keyint: 250
23:45:43.706: preset: hq
23:45:43.706: profile: high
23:45:43.706: width: 4144
23:45:43.706: height: 840
23:45:43.706: 2-pass: false
23:45:43.706: b-frames: 2
23:45:43.706: GPU: 0
23:45:43.706:
23:45:43.951: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
23:46:08.245: Settings changed (outputs)
23:46:08.245: ------------------------------------------------
23:46:09.605: ---------------------------------
23:46:09.605: [NVENC encoder: 'recording_h264'] settings:
23:46:09.605: rate_control: CBR
23:46:09.605: bitrate: 2500
23:46:09.605: cqp: 0
23:46:09.605: keyint: 250
23:46:09.605: preset: hq
23:46:09.605: profile: high
23:46:09.605: width: 3314
23:46:09.605: height: 672
23:46:09.605: 2-pass: false
23:46:09.605: b-frames: 2
23:46:09.605: GPU: 0
23:46:09.605:
23:46:10.002: ---------------------------------
23:46:10.003: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: 3
23:46:10.003:
23:46:10.029: ==== Recording Start ===============================================
23:46:10.029: [ffmpeg muxer: 'adv_file_output'] Writing file 'J:/gamerec/2019-04-15 23-46-09.mkv'...
23:46:11.616: [ffmpeg muxer: 'adv_file_output'] Output of file 'J:/gamerec/2019-04-15 23-46-09.mkv' stopped
23:46:11.616: Output 'adv_file_output': stopping
23:46:11.616: Output 'adv_file_output': Total frames output: 67
23:46:11.616: Output 'adv_file_output': Total drawn frames: 95
23:46:11.617: ==== Recording Stop ================================================
23:46:11.697: Video stopped, number of skipped frames due to encoding lag: 29/82 (35.4%)
23:46:27.787: Settings changed (outputs)
23:46:27.787: ------------------------------------------------
23:46:29.137: ---------------------------------
23:46:29.137: [NVENC encoder: 'recording_h264'] settings:
23:46:29.137: rate_control: CBR
23:46:29.137: bitrate: 2500
23:46:29.137: cqp: 0
23:46:29.137: keyint: 250
23:46:29.137: preset: hq
23:46:29.137: profile: high
23:46:29.137: width: 4144
23:46:29.137: height: 840
23:46:29.137: 2-pass: false
23:46:29.137: b-frames: 2
23:46:29.137: GPU: 0
23:46:29.137:
23:46:29.393: [NVENC encoder: 'recording_h264'] Failed to open NVENC codec: Generic error in an external library
23:48:31.074: Settings changed (outputs)
23:48:31.074: ------------------------------------------------
23:48:32.564: ==== Recording Start ===============================================
23:48:36.101: Output 'adv_ffmpeg_output': stopping
23:48:36.102: Output 'adv_ffmpeg_output': Total frames output: 208
23:48:36.102: Output 'adv_ffmpeg_output': Total drawn frames: 235
23:48:36.102: ==== Recording Stop ================================================
23:48:52.178: Settings changed (outputs)
23:48:52.178: ------------------------------------------------
23:48:53.491: Failed to open video codec: Generic error in an external library
23:48:53.491: ffmpeg_data_init failed
23:48:53.491: ==== Recording Stop ================================================

 

forbiddenera

New Member
Ya, hevc/h265 works, h264 gives error.. is it going to be possible for me to stream only using hevc? I don't really care about recording if I'm going to edit and mux it myself before any distribution but if I wanted to livestream at >4k ..?
 

forbiddenera

New Member
h264 level 6.2 can do:

3,840×2,160@300.0 (16)
7,680×4,320@128.9 (5)
8,192×4,320@120.9 (5)

I'm thinking h264 may be more feasible than h265 for streaming letalone any realtime encoding.
 

koala

Active Member
The maximum resolution for nvenc h.264 is 4096x4096. For nvenc h.265, the maximum resolution is 8192x8192.
See here: https://developer.nvidia.com/nvidia-video-codec-sdk#NVENCFeatures

h.265 isn't supported by streaming services, and it is available only with ffmpeg output of OBS, so for almost all cases you have to stick with 4096x4096 if you want to use hardware acceleration for streaming.
If you don't need to stream but simply produce > 8k footage for later upload, you can use ffmpeg output of OBS for h.265 and recode with x264 with an external app like Handbrake. Software encoder x264 doesn't have a resolution limit.
 
Top