Bug Report Lossless NVENC in 0.14.2: Failed to open NVENC codec: Invalid argument

NalaNosivad

Member
As the title says, I'm getting this error while testing out the lossless NVENC support added in 0.14.2. First of all, I have the latest updates for everything, etc., etc.. Running Windows 10, an i7 4790k, a GTX 970.

As for what happens: I choose the lossless rate control method as so: https://i.imgur.com/Tiv57vz.png

Then I go to hit the start recording button, and nothing happens. I tested every real combination of settings (including colour formats in advanced settings) all to no change. I've attached log files of all combinations I tried (I know, I missed the high performance reset, but that results in no difference).

If I had to guess what was going on, it seems like maybe OBS isn't using the lossless preset under the hood, or not passing on to ffmpeg to use the lossless preset resulting in some strange behaviour. The other rate control methods are fine.

I did also notice something else: to test i444 with NVENC, I set my colour format to i444, the rate control method to CQP, and the profile to high444p. What ends up happening in that case (regardless of rate control method) is the following: http://i.imgur.com/OQ7dY5p.png

A far more competent friend of mine suggested that could be due to a buffer handling the chroma values somewhere along the line expecting YUV420 and getting YUV444, which might explain the last half of the screen having no chroma value (and thus being green, since he believes a chroma of 0 results in green).

One way or the other, it most definitely doesn't appear to be a decoding issue with the video. No software displays it without the green bottom half, and even ffmpeg when transcoding it to something like utvideo renders the artifact in the output.

You can find the video itself here: https://www.dropbox.com/sh/wqboj9nbatrz1ft/AAAine0EHiqaqtZU_VWduZ1Ua?dl=0
Original mkv, and remuxed to mp4 for convenience.

Also for completeness, here's ffprobe's output of the file: http://pastebin.com/7fF4aVJj

Any input on either of these issues would be fantastic!

Edit:
I should add that I was keeping an eye on lossless NVENC support being added via commits, and was making my own builds at various stages before 0.14.2 was released. In one particular build, lossless NVENC did work, along with full-chroma i444 in advanced settings, and the hi444p profile. Unfortunately, even then, it still had the green lower half of the screen problem. The upper half did appear perfectly lossless and full chroma still, though.

Even more unfortunately is I no longer have the exact build or version of the source that had the working lossless to do more tests.
 

Attachments

  • nvenc_rgb_default_high444p.txt
    16.6 KB · Views: 44
  • nvenc_i444_default_high444p.txt
    14.9 KB · Views: 20
  • nvenc_i444_default_high.txt
    16.9 KB · Views: 21
  • nvenc_nv12_default_high444p.txt
    17.7 KB · Views: 17
  • nvenc_nv12_default_high.txt
    15.5 KB · Views: 17

NalaNosivad

Member
What is EncoderStressTest.exe? I like testing stuff :P
Can you post DL link please?
Ah, that's something I threw together in Unity one day incredibly lazily! It looks like this, and is about 45MB unpacked. Like I say though, incredibly lazily thrown together, not optimised for size or anything like that. Standard FPS controls, cursor locks to the window, hit escape to unlock it.

You can grab it here if you really want to!
 

Gol D. Ace

Member
Yes VS 2015 & 2013 Community both work for building Studio.
But you need an updated version of ffmpeg which isn't included yet in the build dependencies on fosshub.
 
Last edited:

Oliver43

New Member
Thank you!
Much appreciated, although I've downloaded ran but the record button still seems unresponsive. I've got a log file, not sure if it will help but worth uploading I guess.

Would this be still an issue with the build or system specific?


Ollie
http://youtube.com/olli43
 

Attachments

  • 2016-05-16 15-38-29.txt
    11.4 KB · Views: 49

Oliver43

New Member
Actually on further testing, I can record with nvenc as a codec just not with the lossless rate control.
Thanks,
 

Gol D. Ace

Member
According to Jim there is a newer ffmpeg build/version needed for this.
Nothing I can do myself right now.

So wait for the next patch.
I will delete that version so that nobody else tries it.
 

NalaNosivad

Member
Well, the good news is lossless NVENC works with the five teeny changes. (I wasn't exactly going to clone the entire thing again just for five lines)

The green lower half/chroma issue is still there, but that was to be expected. It also seems to refuse to encode lossless with anything but the i444 colour format selected (to be expected), and it forces the hi444p profile (again, as expected!).
 

NalaNosivad

Member
hey,
when will the fix been released?
waiting so long for lossless nvenc :3
thanks.
I mean I could upload the build it works in, but it wouldn't do any good with the half-green-screen chroma bug. Unless, of course, you only want half the screen to be usable. I had the idea of capturing at 1920x2160 to overcome the chroma bug. It works, but there are still some artifacts while decoding the video (in the "correct" upper half). They do definitely only seem to be decoding bugs with most software, though. Transcoding the file from the lossless NVENC-produces H.264 to something like huffyuv results in a perfect file.

So... if you build everything yourself and want a workaround for the chroma bug, just capture at 1920x2160 (or double the vertical resolution of whatever you intend to capture at... so it sucks if you want to do 4k), transcode to something else, and you should be good to go.

But yeah, I could put the build out, but I highly doubt it's going to be too long before the next minor patch anyway, so I doubt it's worth it.
 
so i can wait.
the chroma bug was already in classic obs (still today) but it doesn't show up when i load it to premiere.
you only see it in players (like vlc or mpc)
so worth a try?
thx
 

NalaNosivad

Member
Hey does your fix, fix the problem with obs studio and windows 10 dropping frames while using the nvenc option?
I don't recall having that problem myself (so I wouldn't know for sure if it did or not), but I seriously doubt it.

The only change between my build and 0.14.2 is this commit here.
 
Top