Question / Help Failed to open NVENC codec: Generic error in an external library (I HAVE TRIED EVERYTHING)

TheRealNoman

New Member
I have attached the log file. No matter what settings I change or what I do, I get this same error using Nvenc in OBS Studio.

I am running 64 bit OBS Studio, I have tried running it as administrator. I have run it with nothing else open and tried to record my desktop.

I have already reinstalled OBS studio multiple times in multiple different locations.
I have reinstalled my GPU drivers after cleaning them out using DDU.
I have changed every setting around in the output tab while in advanced mode.

Please someone help me fix this. I want to stream/record using Nvenc and OBS studio.
I want to stream with OBS studio specifically as it has all my scenes set up already, and my new GPU is coming in and I'm sure Nvenc encoding will not work even when I swap out the GPU as it seems to be an OBS Studio specific error.

NOTE: Nvenc encoding works with Streamlabs OBS for me. This leads me to believe my install of OBS Studio is broken, but reinstalling it in different locations has done nothing to stop this error.

If any of you help me fix this error, I will be insanely grateful.
Thanks in advance.
 

Attachments

  • 2019-11-16 02-42-45.txt
    10.4 KB · Views: 4,995

TheRealNoman

New Member
UPDATE | I fixed it

Changing the Color Format/Range didn't seem to fix it either.

I did a complete clean install of OBS Studio. I not only uninstalled it, but also deleted the folders of any installations of OBS Studio before reinstalling it, and this seemed to fix the problem.

@koala thanks for the reply
 

csmgj

New Member
UPDATE | I fixed it

Changing the Color Format/Range didn't seem to fix it either.

I did a complete clean install of OBS Studio. I not only uninstalled it, but also deleted the folders of any installations of OBS Studio before reinstalling it, and this seemed to fix the problem.

@koala thanks for the reply
After you reinstalled it, was "Encoding" still set to hardware? Before you reinstalled, would it work with "Encoding" set to "Software"?
 

dalenacio

New Member
I also had this problem. On another website, I found this can be caused by trying to use the GPU for encoding when the GPU doesn't support that encoding. Click "Settings" then select "Output". Change "Encoding" to "Software" under "Streaming" and "Recording."
This fixed it for me. Thanks!
 

DerkWehler

New Member
Same problem

In Settings->Advanced->Video, set Color Format to NV12 and Color Range to Partial.

Mine is already set like this. I tried changing the Encoder to "Software (x264)" (though I would prefer to use hardware). Sometimes it works, sometimes it doesn't. I tried "Software (x264) low CPU usage..." too, which does the same.

It seems like with Software, it works when I click "start recording", but when I use my hot key, it gives me the error. Yep, recurring: hot key gives codec error, button works (on software encoding).
 

tensai

New Member
I also had this problem. On another website, I found this can be caused by trying to use the GPU for encoding when the GPU doesn't support that encoding. Click "Settings" then select "Output". Change "Encoding" to "Software" under "Streaming" and "Recording."
?? I am running into these codec/encoder errors using nvenc and looking for a fix but I don't see "encoding" when I go to output?? running 25.0.8
 

tensai

New Member
Oh by software you mean x264, so using cpu to encode and avoiding nvenc altogether. Not really a solution though as some need to use nvenc since their cpu's can't handle the encoding. For me I have used nvenc without a problem for a long time so it's weird that this is happening all of a sudden.
 

Ken_SF

New Member
I had this problem on a seldom used Windows 7 Pro 64 Bit computer. The Settings-->output-->change to software encoding did not work for me. What did work: update my NVIDIA drivers. That was a problem too as I first had to install Java and subsequently the NVIDIA driver web page kept asking me to install over and over, even though I had the most current version.

I finally, opened the NVIDIA control panel, found the model of my current driver and manually downloaded the NVIDIA driver that seemed to be the best match. There was a compatibility check portion of the NVIDIA driver installation, so hopefully, that will protect someone if they select the wrong NVIDIA driver; I selected the right one.

After installing the NVIDIA drivers I rebooted, launched OBS (updated to most recent version today - 06/15/2020) and recording worked perfectly with hardware or software encoding selected in settings-->output.

I hope this helps someone else.

Thanks for the prior posts; they helped me figure out what to do.
 

domaxi198

New Member
I also had this problem. On another website, I found this can be caused by trying to use the GPU for encoding when the GPU doesn't support that encoding. Click "Settings" then select "Output". Change "Encoding" to "Software" under "Streaming" and "Recording."

Thanks for this solution, it worked for me.
 

braintalked

New Member
I also had this problem. On another website, I found this can be caused by trying to use the GPU for encoding when the GPU doesn't support that encoding. Click "Settings" then select "Output". Change "Encoding" to "Software" under "Streaming" and "Recording."
I had the same problem and you helped me fix it! Many thanks :-)
 

Awoolyx

New Member
Hey everyone!

Some of you guys used the solution of going with CPU Encoding (Software), but this isn't really great if you have an NVIDIA GPU.

Here's my fix if you are looking for NVENC:
In Settings > Output ensure that only have EITHER Streaming OR Recording set to Hardware Encoding (NVENC). Set the other one that you don't want to CPU/Software (x264). NVIDIA does not let you use NVENC twice/parallel.

If you want to stream and record, I'd recommend using NVENC for Recording and Software for Streaming, as recording has a much higher bitrate than streaming.

- Alyx
 

R1CH

Forum Admin
Developer
Recent drivers allow up to three NVENC sessions. As recording has practically unlimited bitrate, it's best to use NVENC on streaming so it can make the most of the limited bitrate.
 

Rutopia

New Member
I found that for me, it was scaling the video. Apparently I had it scale to an 8k resolution... (I was messing about with different scenes). Setting scaling to my screen resolution allowed recording with NVENC to continue.

Scaling can be found in Settings -> Output. It's under both Stream and Record. (freely translated from Dutch ;))

Hope it helps anyone else in the future.
 

oxkos

New Member
I have attached the log file. No matter what settings I change or what I do, I get this same error using Nvenc in OBS Studio.

I am running 64 bit OBS Studio, I have tried running it as administrator. I have run it with nothing else open and tried to record my desktop.

I have already reinstalled OBS studio multiple times in multiple different locations.
I have reinstalled my GPU drivers after cleaning them out using DDU.
I have changed every setting around in the output tab while in advanced mode.

Please someone help me fix this. I want to stream/record using Nvenc and OBS studio.
I want to stream with OBS studio specifically as it has all my scenes set up already, and my new GPU is coming in and I'm sure Nvenc encoding will not work even when I swap out the GPU as it seems to be an OBS Studio specific error.

NOTE: Nvenc encoding works with Streamlabs OBS for me. This leads me to believe my install of OBS Studio is broken, but reinstalling it in different locations has done nothing to stop this error.

If any of you help me fix this error, I will be insanely grateful.

Thanks in advance.

I had same problem and i have fixed the issue!
Here is a tutroail i have created for you guys to fix this problem : Video > https://www.youtube.com/watch?v=74Vp15x2nrU&feature=youtu.be
If you follow instructions in the video you will fix the issue 100%
Thanks

NP
 

devguy

New Member
My problem will show up randomly, or when I put my PC to sleep and after I will get this error. I have to re-boot for the problem to go away. I have a RTX 3080 card and when "NVIDIA NVEND H.264" works, it works amazing, by off loading processing from the CPU to the GPU!

I can't seem to find an explanation why this error just shows up randomly. I've got the latest NVidia driver for my Linux system. I wish the OBS team did better logging so they could look into this bug and determine what's causing it. Their dismissive attitude of, it's not an OBS bug is off-putting, how do they know? I was told I am not using their support version of Linux so they closed off my bug and told me to go on the forum.
 

elderbard

New Member
I also had this problem. On another website, I found this can be caused by trying to use the GPU for encoding when the GPU doesn't support that encoding. Click "Settings" then select "Output". Change "Encoding" to "Software" under "Streaming" and "Recording."
seems to have sorted my issue, so far. Thank you!
 

sharath

New Member
For me on windows 11 choosing graphic preference helped. It seems like by default it will be set to power saving mode.
1652165581841.png
 

Mr. n

New Member
For me on windows 11 choosing graphic preference helped. It seems like by default it will be set to power saving mode.
View attachment 83341
Thank you, this was the case for me (I'm on Windows 10, though)! Registered to say thanks :).

And it had been working a few minutes before... before I let windows update run. OBS was the only desktop app that I had actual preference in windows which GPU should it use, and windows update changed that preference. Beware!
 
Top