Question / Help NVENC Settings for HQ Local Recording

RedTail · Sep 17, 2014

Recently started playing around with Shadowplay and discovered I could record at up to 50,000kb/s without issue. Decided to try using the NVENC codec in OBS (for desktop recording and overlays), but I can't replicate that peformance or get anywhere close to it. To get a video without any frameskips, I have to go down to 10,000kb/s and disable CBR. OBS should be able to replicate Shadowplay's performance since it's using the same encoding method, so I figure it's gotta be something with my settings, right?

Here's what my log looks like when I'm recording at 50,000kb/s w/ CBR enabled:

Code:

10:42:47: CUDA loaded successfully
10:42:48: 1 CUDA capable devices found
10:42:48: [ GPU #0 - < GeForce GTX 660M > has Compute SM 3.0, NVENC Available ]
10:43:42: Open Broadcaster Software v0.635b - 32bit (´・ω・｀)
10:43:42: -------------------------------
10:43:42: CPU Name: Intel(R) Core(TM) i7-3630QM CPU @ 2.40GHz
10:43:42: CPU Speed: 2395MHz
10:43:42: Physical Memory:  4095MB Total, 4095MB Free
10:43:42: stepping id: 9, model 58, family 6, type 0, extmodel 1, extfamily 0, HTT 1, logical cores 8, total cores 4
10:43:42: monitor 1: pos={0, 0}, size={1920, 1080}
10:43:42: Windows Version: 6.2 Build 9200
10:43:42: Aero is Enabled
10:43:42: -------------------------------
10:43:42: OBS Modules:
10:43:42: Base Address  Module
10:43:42: 00270000  OBS.exe
10:43:42: 644D0000  OBSApi.dll
10:43:42: 648E0000  DShowPlugin.dll
10:43:42: 64810000  GraphicsCapture.dll
10:43:42: 647E0000  NoiseGate.dll
10:43:42: 64A20000  PSVPlugin.dll
10:43:42: ------------------------------------------
10:43:42: Adapter 1
10:43:42:  Video Adapter: NVIDIA GeForce GTX 660M
10:43:42:  Video Adapter Dedicated Video Memory: 2082271232
10:43:42:  Video Adapter Shared System Memory: 2147676160
10:43:42:  Video Adapter Output 1: pos={0, 0}, size={1920, 1080}, attached=true
10:43:42: ------------------------------------------
10:43:42: Adapter 2
10:43:42:  Video Adapter: NVIDIA GeForce GTX 660M
10:43:42:  Video Adapter Dedicated Video Memory: 2082271232
10:43:42:  Video Adapter Shared System Memory: 2147676160
10:43:42: =====Stream Start: 2014-09-17, 10:43:42===============================================
10:43:42:  Multithreaded optimizations: On
10:43:42:  Base resolution: 1920x1080
10:43:42:  Output resolution: 1920x1080
10:43:42: ------------------------------------------
10:43:42: Loading up D3D10 on NVIDIA GeForce GTX 660M (Adapter 1)...
10:43:42: ------------------------------------------
10:43:42: Audio Format: 48000 Hz
10:43:42: ------------------------------------------
10:43:42: Audio Channels: 2 Ch
10:43:42: Playback device Default
10:43:42: ------------------------------------------
10:43:42: Using desktop audio input: Speakers (Realtek High Definition Audio)
10:43:42: Global Audio time adjust: 0
10:43:42: ------------------------------------------
10:43:42: Audio Encoding: AAC
10:43:42:  bitrate: 128
10:43:42: Using graphics capture
10:43:42: Trying to hook process: emulator.exe
10:43:42: Scene buffering time set to 700
10:43:42: NVENC internal init finished successfully
10:43:43: NVENC supports 8 h264 presets
10:43:43: ------------------------------------------
10:43:43: Video Encoding: NVENC
10:43:43:  fps: 60
10:43:43:  width: 1920, height: 1080
10:43:43:  preset: hq
10:43:43:  profile: high
10:43:43:  level: autoselect
10:43:43:  keyint: 30
10:43:43:  CBR: yes
10:43:43:  CFR: yes
10:43:43:  max bitrate: 50000
10:43:43:  avg bitrate: 50000
10:43:43:  buffer size: 0
10:43:43: ------------------------------------------
10:43:43: SharedTexCapture hooked
10:47:23: FlushBufferedVideo: Flushing 46 packets over 750 ms
10:47:24: Total frames encoded: 13220, total frames duplicated: 572 (4.33%)
10:47:24: Number of frames skipped due to encoder lag: 260 (1.97%)
10:47:24: Total frames rendered: 12957, number of late frames: 14 (0.11%) (it's okay for some frames to be late)
10:47:24: Encoder closed
10:47:24: NVENC deinitialized
10:47:24:
10:47:24: Profiler time results:
10:47:24:
10:47:24: ==============================================================
10:47:24: video thread frame - [100%] [avg time: 0.896 ms] [children: 92.4%] [unaccounted: 7.59%]
10:47:24: | scene->Preprocess - [0.112%] [avg time: 0.001 ms]
10:47:24: | GPU download and conversion - [92.3%] [avg time: 0.827 ms] [children: 81.6%] [unaccounted: 10.7%]
10:47:24: | | flush - [30.2%] [avg time: 0.271 ms]
10:47:24: | | CopyResource - [5.58%] [avg time: 0.05 ms]
10:47:24: | | conversion to 4:2:0 - [45.8%] [avg time: 0.41 ms]
10:47:24: Convert444Threads - [100%] [avg time: 0.94 ms] [children: 99.1%] [unaccounted: 0.851%]
10:47:24: | Convert444toNV12 - [99.1%] [avg time: 0.932 ms]
10:47:24: encoder thread frame - [100%] [avg time: 11.383 ms] [children: 2.58%] [unaccounted: 97.4%]
10:47:24: | sending stuff out - [2.58%] [avg time: 0.294 ms]
10:47:24: ==============================================================
10:47:24:
10:47:24:
10:47:24: Profiler CPU results:
10:47:24:
10:47:24: ==============================================================
10:47:24: video thread frame - [cpu time: avg 0.227 ms, total 2953.13 ms] [avg calls per frame: 1]
10:47:24: | scene->Preprocess - [cpu time: avg 0.003 ms, total 46.875 ms] [avg calls per frame: 1]
10:47:24: | GPU download and conversion - [cpu time: avg 0.164 ms, total 2125 ms] [avg calls per frame: 1]
10:47:24: | | flush - [cpu time: avg 0.121 ms, total 1578.13 ms] [avg calls per frame: 1]
10:47:24: | | CopyResource - [cpu time: avg 0.012 ms, total 156.25 ms] [avg calls per frame: 1]
10:47:24: | | conversion to 4:2:0 - [cpu time: avg 0.009 ms, total 125 ms] [avg calls per frame: 1]
10:47:24: Convert444Threads - [cpu time: avg 0.931 ms, total 24046.9 ms] [avg calls per frame: 2]
10:47:24: | Convert444toNV12 - [cpu time: avg 0.922 ms, total 23812.5 ms] [avg calls per frame: 2]
10:47:24: encoder thread frame - [cpu time: avg 2.688 ms, total 35421.9 ms] [avg calls per frame: 1]
10:47:24: | sending stuff out - [cpu time: avg 0.145 ms, total 1921.88 ms] [avg calls per frame: 1]
10:47:24: ==============================================================
10:47:24:
10:47:24: =====Stream End: 2014-09-17, 10:47:24=================================================


Last game capture log:
2014-09-17, 10:43:42: we're booting up:
10:43:42: CaptureThread: attached to process emulator.exe
10:43:43: D3D9 Present
10:43:43: D3D9EndScene called
10:43:43: D3DPRESENT_PARAMETERS {
10:43:43:    BackBufferWidth: 1920
10:43:43:    BackBufferHeight: 1080
10:43:43:    BackBufferFormat: D3DFMT_X8R8G8B8
10:43:43:    BackBufferCount: 3
10:43:43:    MultiSampleType: D3DMULTISAMPLE_NONE
10:43:43:    MultiSampleQuality: 0
10:43:43:    SwapEffect: D3DSWAPEFFECT_DISCARD
10:43:43:    hDeviceWindow: 1640076
10:43:43:    Windowed: false
10:43:43:    EnableAutoDepthStencil: true
10:43:43:    AutoDepthStencilFormat: D3DFMT_D24S8
10:43:43:    Flags: None
10:43:43:    FullScreen_RefreshRateInHz: 59
10:43:43:    PresentationInterval: 2147483648
10:43:43: };
10:43:43: successfully set up d3d9 hooks
10:43:43: D3D9Present called
10:43:43: DoD3D9GPUHook: success
10:43:43: D3DSURFACE_DESC {
10:43:43:    Format: D3DFMT_X8R8G8B8
10:43:43:    Type: D3DRTYPE_SURFACE
10:43:43:    Usage: D3DUSAGE_RENDERTARGET
10:43:43:    Pool: D3DPOOL_DEFAULT
10:43:43:    MultiSampleType: D3DMULTISAMPLE_NONE
10:43:43:    MultiSampleQuality: 0
10:43:43:    Width: 1920
10:43:43:    Height: 1080
10:43:43: };
10:43:43: successfully capturing d3d9 frames via GPU
10:43:43: DXGI Present
10:43:43: (half life scientist) everything..  seems to be in order
10:47:24: ---------------------- Cleared D3D9 Capture ----------------------
10:47:24: D3D9EndScene called
10:47:24: D3D9Present called
10:47:25: D3D9Reset called
10:47:25: ---------------------- Cleared D3D9 Capture ----------------------
10:47:25: D3DPRESENT_PARAMETERS {
10:47:25:    BackBufferWidth: 496
10:47:25:    BackBufferHeight: 384
10:47:25:    BackBufferFormat: D3DFMT_X8R8G8B8
10:47:25:    BackBufferCount: 3
10:47:25:    MultiSampleType: D3DMULTISAMPLE_NONE
10:47:25:    MultiSampleQuality: 0
10:47:25:    SwapEffect: D3DSWAPEFFECT_DISCARD
10:47:25:    hDeviceWindow: 1640076
10:47:25:    Windowed: true
10:47:25:    EnableAutoDepthStencil: true
10:47:25:    AutoDepthStencilFormat: D3DFMT_D24S8
10:47:25:    Flags: None
10:47:25:    FullScreen_RefreshRateInHz: 0
10:47:25:    PresentationInterval: 2147483648
10:47:25: };
10:47:25: successfully set up d3d9 hooks
10:47:25: D3D9EndScene called
10:47:25: D3D9Present called
10:47:28: D3D9Reset called
10:47:28: ---------------------- Cleared D3D9 Capture ----------------------
10:47:28: D3DPRESENT_PARAMETERS {
10:47:28:    BackBufferWidth: 1920
10:47:28:    BackBufferHeight: 1080
10:47:28:    BackBufferFormat: D3DFMT_X8R8G8B8
10:47:28:    BackBufferCount: 3
10:47:28:    MultiSampleType: D3DMULTISAMPLE_NONE
10:47:28:    MultiSampleQuality: 0
10:47:28:    SwapEffect: D3DSWAPEFFECT_DISCARD
10:47:28:    hDeviceWindow: 1640076
10:47:28:    Windowed: false
10:47:28:    EnableAutoDepthStencil: true
10:47:28:    AutoDepthStencilFormat: D3DFMT_D24S8
10:47:28:    Flags: None
10:47:28:    FullScreen_RefreshRateInHz: 59
10:47:28:    PresentationInterval: 2147483648
10:47:28: };
10:47:28: successfully set up d3d9 hooks
10:47:28: D3D9EndScene called
10:47:28: D3D9Present called

BtbN · Sep 17, 2014

Disable cbr, set buffer size to 0 and quality to 10.
That's the best possible quality you can get.

achmetha · Sep 17, 2014

i use the "high quality low latency" preset and do not have issues. i see in your log you are using "high quality" only. try changing it to high quality low latency and see if it helps.

BtbN · Sep 17, 2014

llhq can't keep up at 1080p 60fps, which is why automatic falls back to hq.

Also, your keyint is needlessly low. Disable it entirely for local recordings. You are currently doing 2 keyframes per second, which affects the quality very badly.

Milo · Sep 20, 2014

how would one disable keyint entirely? is that what setting it to 0 does?

BtbN · Sep 20, 2014

Question / Help NVENC Settings for HQ Local Recording

RedTail

New Member

BtbN

Member

achmetha

Member

BtbN

Member

Milo

New Member

BtbN

Member