Terrible Recording Quality/Replay Buffer Inconsistency (HELP!)

[Jackadoodles]

Hello, total OBS noob here.

I am looking for an alternative to NVIDIA's Shadowplay, which has proven to be pretty unreliable at times, and generally lower quality. I've heard many good things about OBS, but despite looking into just about every corner I can find, I can't seem to get it to work. I'm using very high-end hardware, which only adds to my confusion.

-RECORDINGS-
Test recordings I've made run at extremely low FPS (like "98-99% of frames are dropped" extremely low), and the recorded audio quality is significantly worse than the original. I've messed around with bitrates, file types, encoders and presets - but even when using the "highest" of what's available, I still get a low-quality result. OBS has told me that the encoder is struggling, but with no more than 1.4% of my CPU being used at a time, and a pretty heavy-duty GPU (more on that below), I'm not sure how that is possible.

-BUFFERS-
In addition, I just can't for the life of me get the Replay Buffer to work reliably. It just won't work. It doesn't matter if there's a game running or it's just my desktop display, clicking on the "Save Buffer" option or hitting my hotkey won't save a damn thing. The Replay Buffer is what is most important to me (more so than recordings), as I originally set my sights on OBS as an alternative to Shadowplay - however, if even my recordings are suffering such terrible quality, I'm not sure that the buffers would be any better if they even saved in the first place.

Those are the main issues I'm facing - low FPS, bad audio quality, and the Replay Buffer barely working.

-SPECS-
RTX 3090
Intel i9-10900KF
128GB RAM
Windows 10

-TARGET-
My target is to record/save buffers at 2560x1440 (as that is my native resolution) with at least 120 FPS (165 would be preferable though, as that is my monitor's refresh rate). Having recorded audio quality that is on par with the original audio is also very high priority for me. It may be naïve to think so, but I honestly figured these specs would be way past overkill (bordering genocide) for this sort of goal - you guys may have to tell me otherwise.

In any case, I am most likely (definitely) doing something wrong and/or missing something(s), so anything to help me reach my target would be much, much appreciated.

Log file attached here: https://obsproject.com/logs/VvYG148JEQtXOAmD

 

[Harold]

Using CBR as your rate control for recording, especially with only 2500kbit as your bitrate, is going to give you TERRIBLE picture quality.

Switch the recording encoder to nvenc, its rate control to CQP, the CQ level to 14, and the preset to "quality"

 

[Jackadoodles]

Using CBR as your rate control for recording, especially with only 2500kbit as your bitrate, is going to give you TERRIBLE picture quality.

Switch the recording encoder to nvenc, its rate control to CQP, the CQ level to 14, and the preset to "quality"

Thanks for the reply.

I've gone ahead and done that, and increased the amount of max B-frames to 4 instead of 2 - let me know if that's not recommended though.
Couple questions - is it recommended to have Look-Ahead and Psycho Visual Tuning enabled? How much does changing the CQ level affect quality and performance?

I'm still suffering from the nearly non-existent frames (and the other issues previously mentioned), but all help thus far is truly appreciated.

 

[Deleted member 121471]

Revert B-frames to 2, disable Lookahead and Psycho Visual Tuning then set Preset to "Quality".

Concerning CQP, lower values increase image quality at the cost of higher storage requirements. I prefer a range of 16-20, depending on what I'm recording. Unlike CBR, it'll adjust bitrate used depending on image motion/changes to maintain a constant quality.

Lastly, record at 60 FPS and cap your ingame FPS at a multiple of it, so 60/120/180, depending on what you can reasonably maintain ingame.

 

[Jackadoodles]

Revert B-frames to 2, disable Lookahead and Psycho Visual Tuning then set Preset to "Quality".

Concerning CQP, lower values increase image quality at the cost of higher storage requirements. I prefer a range of 16-20, depending on what I'm recording. Unlike CBR, it'll adjust bitrate used depending on image motion/changes to maintain a constant quality.

Lastly, record at 60 FPS and cap your ingame FPS at a multiple of it, so 60/120/180, depending on what you can reasonably maintain ingame.

Got it down, thanks. Is there a particular reason why you want to disable those two options?

In any case, even when recording at 60 FPS I am still getting the same terrible framerate.

 

[Deleted member 121471]

Those options and Max Quality leverage CUDA cores for negligible quality gain while potentially leading to rendering and encoding lag.

Once you change all the options mentioned in the replies offered thus far, post a log if you still have issues so we can check if there's anything we missed.

 

[Jackadoodles]

Those options and Max Quality leverage CUDA cores for negligible quality gain while potentially leading to rendering and encoding lag.

Once you change all the options mentioned in the replies offered thus far, post a log if you still have issues so we can check if there's anything we missed.

Understood.

Checking the log, the terrible frames I reported in my last comment was actually my fault - I reviewed the wrong video, which was recorded prior to the 60 FPS change - hence the bad framerate. My bad!

Viewing the correct video now, as well as running another test, confirms a smooth framerate. I really appreciate the help guys!

Here's the new log: https://obsproject.com/logs/LCQZ_ompNRm7CA37

With the framerate issue resolved, how might I go about improving the audio quality? Assuming that can even be done (OBS gives me a max sample rate of 48kHz for the stereo option and a bitrate of 320 for the actual tracks).

And a final question: why did my computer struggle to record at >60 FPS in the first place? Was it the resolution? I've read about people pushing 300 FPS at 1080p (which I understand to be really pushing it, but it's 300 FPS nonetheless).

 

[Jackadoodles]

I have no filters applied to my desktop audio in the mixer - that may be the issue, as it seems a bitrate of 320 is actually supposed to be quite high quality...

 

[Jackadoodles]

Any further help into these issues would be much appreciated.

 

[Deleted member 121471]

The sample rate has to be the same across all devices and OBS and the audio bitrate is just fine set at 160.

Concerning audio quality, using audio filters is your best bet.

Wiki - Filters Guide | OBS

This page has moved to https://obsproject.com/kb/filters-guide

obsproject.com

Your system was struggling because NVENC preset and additional options, if set on max and turned on, leverages CUDA cores to improve image quality marginally.

Lastly, my signature contains my opinion on >60FPS capturing.

 

[Jackadoodles]

My headphones run at a sample rate of 96k Hz, is it possible to make OBS run at that sample rate too? Or is it better to drop it to 48k? Or 44.1?

 

[Harold]

With obs running at 48k, having your headphones at an even-number multiple of 48k is better to deal with than the mismatch between 44.1 and 48k.

You can try running both at 48k if you want.

 

[Jackadoodles]

I've tried messing around with all the filters, after looking at the guide. While some do reduce background noise well, it often makes my mic's audio worse - with no filters, my mic sounds fine but background sound is still audible. I'll just have to work at it I guess.

As for the desktop audio, it still sounds pretty terrible. Whether it's a game or a YouTube video playing, the audio still sounds very compressed/generally worse when compared to the original output. Discord audio is somehow even worse, constantly peaking on top of everything said before.

I've switched every device to the same bitrate, have experimented with filters, reset the bitrates in the "Output" back to 160 (from 320) and yet I can't seem to get any improvement.

On top of that, the replay buffer is still super inconsistent. Not sure if I'm doing something wrong, but it'll outright refuse to save a buffer even if it's been properly loaded/set to capture the correct display and audio. But after resetting the buffer (which of course means that the thing I wanted to record is now lost), it works perfectly.

I don't know what to do, really. OBS has proven to be quite the unfriendly little program.

Not sure if it'll help, as I've restarted my computer a number of times in between experiments, but here's the log: https://obsproject.com/logs/GUUcMMdTWe4SO6Ad

 

[Jackadoodles]

I'm close to giving up on OBS. I can't even record at anything more than 60 FPS on it, and these other issues don't seem to have a solution.

 

[Jackadoodles]

Here's a log that contains an attempt to record a buffer at 120 FPS. 80% of the frames are dropped. I know it would be easier to just leave it at 60, but I'd really prefer the higher framerate - especially since I know my hardware can handle it. Considering said hardware, I don't get how OBS seems to be having so much trouble with both video AND audio like this.

I also tried to record lossless to see if anything might change, but it only turns out as a black screen with perfect audio in Microsoft's video player. Not sure what's up about that.

https://obsproject.com/logs/e9pa8gw9D66lHq8q