OBS not working with RTX 5090 (NV_ENC_ERR_INVALID_DEVICE)

T

Takibo

New Member
Hey!
I'm not entirely sure if this is going to be an isolated case or a bigger issue across other RTX 5090 Cards.
I'll gladly help find a solution for this & hopefully will help find a fix for other 50 Cards users as well.

I have spent over a week working around with OBS in an attempt to fix this, i'll list here below everything that was attempted as i tried to fix this issue.

Do note that, this issue does NOT occur with ANY other recording software, so the concerns for it being a hardware issue is highly unlikely, i am currently getting by with another recording software, as i do YouTube full time and couldn't afford to not find a solution ASAP.



First, the issue:

1. OBS crashes my recording midway through with the error: NV_ENC_ERR_INVALID_DEVICE
2. Crashes are not immediate, and may often take hours to appear (happen at random) ranging from as quick as 15 minutes into a recording to taking 8 hours to crash.


Second, useful information & all i've tested, etc:

1. This issue does NOT persist anywhere else (tested with Shadowplay, FFmpeg NVENC Rendering, Meld Studio)
2. This issue happens regardless of settings & 'Video Encoder' (anything NVENC err's out)
3. Stress testing does not accelerate nor does it deaccelerate the rate of these errors (tested increasing thermals, power draw, quality settings, no impact)
4. similar to #3, it does not matter whether im idle, or running benchmarks, the crashes happen all the same.
5. I have updated drivers, reinstalled utilizing DDU (multiple times), tested "studio drivers", also ran older drivers, nothing changed
6. I have attempted with rolling back OBS (and nvidia) no impact.
7. HAGS has been tested both off and on
8. High Priority has no impact
9. Running as admin has no impact
10. Power management has been tested
11. Have reseated the RTX 5090, also have changed the 16-pin power cable for another
12. Have tested limiting power draw, no impact
13. Matter of fact; thermals are always fine, power draw has standard fluctuations & CPU/SSD/RAM Behavior is always standard AFAIK.
14. Each and every ERROR gives me an "Event Viewer" Error: (Source: nvlddmkm / Event ID: 153) log attached
15. Does not seem to have VRAM or RAM leaks
16. There are likely more i've experimented with (its been over a week of straight experimenting) but this is about all i remember

I have attached here one of the various crash logs, from obs, and one from event viewers, there are a few others like it, but they always have the same readings so i wont fill you with many logs


For the time being i'll remain recording with another software, am looking forward to returning to OBS though..

Thanks!
 

Attachments

  • OBS LOG 2025-03-04 17-05-58.txt
    12.9 KB · Views: 14
  • event viewer crash.txt
    649 bytes · Views: 5
R

rockbottom

Active Member
You're not the only person having trouble with a 5090. Drivers don't seem to be all that great yet & are usually the cause of the encoder lockup. Have you tried running all your monitors @ 60HZ? Also, Nvidia released another driver on Wednesday (572.70), might be worth a shot.

Some other things, probably not contributing to the lock up but.
Disable Game DVR
Run all monitors @ 60HZ for testing
Set Color Range to Partial
Fix your bugged Scene
Use Preset P5
Set Keyframe Interval to 2 Seconds
Write to an mkv container while testing
 
T

Takibo

New Member
rockbottom said:
Sounds familiar?

Click to expand...
His issue seems to be different from mine, my load does not drop, matter of fact nothing seems out of place when the error occurs, games dont freeze, dont misbehave, dont drop in fps, it simply gives me the error on OBs and thats it, as if obs just lost communication or something witht the nvenc encoder
rockbottom said:
You're not the only person having trouble with a 5090. Drivers don't seem to be all that great yet & are usually the cause of the encoder lockup. Have you tried running all your monitors @ 60HZ? Also, Nvidia released another driver on Wednesday (572.70), might be worth a shot.

Some other things, probably not contributing to the lock up but.
Disable Game DVR
Run all monitors @ 60HZ for testing
Set Color Range to Partial
Fix your bugged Scene
Use Preset P5
Set Keyframe Interval to 2 Seconds
Write to an mkv container while testing
Click to expand...
Ill test the hz just for pure testing sakes and will return here later to share results! Thats about the only thing there i have not tested (also the color range, ill test that)

As for the drivers from yesterday, P5, game dvr, mkv, and keyframe interval all have been tested and none worked

Will drop results here with the 60hz and color range as soon as i can, im working on a few videos at the moment so ill probably run the experiment tomorrow!
 
R

rockbottom

Active Member
Many different issues over at that forum but hi refresh rates seem to be playing a part in some of the situations. Anyway, I'm still running the same version of OBS as you but I'm also running older hardware & 566.xx driver with no AV1 or issues. Looking at the release notes for the latest version of OBS, you may want to update since you're trying to encode AV1.
 
You must log in or register to reply here.
Top