Question / Help Custom output to URL breaks NvEnc

[Cobra_Fast]

My friends and I have been experiencing this problem for a while, we're unsure when it started.
We're using a custom MKV-based streaming server for small scale streaming within our friend circle.
This is what my output settings look like: https://x.ezl.re/20180701_201414.png

The problem is, that nvenc produces grey frames and/or massive grey artifacts on low compexity scenes.
This is what the Rocket League menu screen looks like for example: https://x.ezl.re/20180701_201717.png the only things somewhat visible out of the grey mush are those actually changing (animated background objects, animated car paintjob, music infobox). Keyframes are rendered properly but then it immediately reverts to a grey screen where animated objects will slowly become visible as they change. During regular gameplay the grey gets "erased" more quickly as there is more movement in the picture.

No matter if I try nvenc_h264 or nvenc_hevc. Happens on 980Ti and 1060. Also tried to blindly fool around with options from https://gist.githubusercontent.com/...436556089d8bc8454a2fa6c3802128ccf8/hevc_nvenc but nothing helped.

The log looks uneventful:

20:13:07.000: ---------------------------------
20:13:07.000: video settings reset:
20:13:07.000: base resolution: 2560x1440
20:13:07.000: output resolution: 1280x720
20:13:07.000: downscale filter: Bicubic
20:13:07.000: fps: 60/1
20:13:07.000: format: NV12
20:13:07.000: YUV mode: 601/Partial
20:13:07.007: Settings changed (video)
20:13:07.007: ------------------------------------------------
20:16:17.969: Using muxer settings:
20:16:17.969: live=1
20:16:18.067: ==== Recording Start ===============================================
20:16:38.777: [game-capture: 'Spielaufnahme'] ----------------- d3d9 capture freed -----------------
20:16:43.915: [game-capture: 'Spielaufnahme'] d3d9 shared texture capture successful
20:16:43.918: [game-capture: 'Spielaufnahme'] shared texture capture successful
20:22:27.303: Output 'adv_ffmpeg_output': stopping
20:22:27.303: Output 'adv_ffmpeg_output': Total frames output: 22155
20:22:27.303: Output 'adv_ffmpeg_output': Total drawn frames: 22153 (22167 attempted)
20:22:27.303: Output 'adv_ffmpeg_output': Number of lagged frames due to rendering lag/stalls: 14 (0.1%)
20:22:27.304: ==== Recording Stop ================================================

Recording to file with NVENC, streaming to Twitch with NVENC, or streaming to the MKV-Server with x264 are not affected and work fine.

Setting key frames to 1 (so every frame is a keyframe) is a workaround but comes at a severe loss of quality.

Is there any option or parameter I am missing? Anything I'm doing wrong besides actually trying to use nvenc ;-) ?

 

[Cobra_Fast]

Anybody?

 

[Boildown]

Need your whole log file. Don't use the more adventurous settings until you get basic settings to work. So switch from High10 profile to High, change level to auto, use HQ or HP preset instead of LLHQ (I'd recommend this regardless, as HQ > LLHQ for quality and HP > LLHQ for performance). And instead of using the custom output use the build-in NVEnc.

You mention streaming but your settings in the jpg and the log clip look like recording.... need a log from your streaming attempt.

What resolution are you streaming at? 2560kbps might be ok for 720p, but its far too little for 1440p. Even for 720p, I was kind of expecting a higher bitrate... and why the weird number (2560 instead of 2500 or 3000 or something)? Did you come up with that yourself or are you following some guide?

The description to me sounds like non-i-frames aren't being encoded or decoded correctly. If I had to guess, I'd say its the 10-bit encoding. But that's strictly a guess.