Question / Help Seemingly Impossible To Get Good Quality Video On High Spec PC

[Gabriel Balaich]

Let me start with my system specs:

Asus Tuf Mark 1 X299
Core i9 7900x (4.4GHz OC)
GTX 1080
GTX 1050ti
32GB DDR4
RME HDSPe AIO(Audio Card)
1tb Intel 600P M.2 Drive

Shadow play worked fairly well for the last year until it started recognizing my audio mixer as a game and will no longer let me record in actual games, or at least that is seemingly the problem. I decided to try OBS, easy peasy right? Apparently not, I had assumed with this beefy of a computer I could get something going.

So I began testing only to find my recordings were dropping frames like crazy, using just some standard settings. Read up a little bit and decided to give NVENC a try - similar if not the same results. This lead my stupid brain to believe I needed a capture card, I have a 3440x1440 100hrz ASUS display (PG348Q), so I knew I needed something that could do 4k to achieve 3440x1440 at 100 FPS.

Finally after much dismay I purchased a Magewell Pro Capture HDMI 4K Plus, hooked it up, opened Capture Express (Magewell software) and got some decent 4k video at 60fps using NVENC. FINALLY some progress I thought, however bit rate maxed at 16000, that seems a bit low, oh well lets get resolution figured out first. This was done by flashing the capture card with an EDID file, after a week of trying to figure that out I finally got the card to be recognized as 3440x1440 @ 100hrz. However Capture Express would resize the video from 21:9 to 16:9 warping it and the bit rate still only went to 16000, Well back to OBS I guess.

HMM OBS still dropping frames like crazy using NVENC and the capture card, how is this possible? Capture express was working fine. I thought the Capture Card had somehow fixed this already... Reloaded the capture card with it's old firmware to completely restore it and it's EDID file, now it was back to capturing 4k. I know capture express was at least able to capture 4k 60fps with NVENC so surely OBS can right? Apparently not... dropping frames everywhere.

Get online to do some research, apparently putting a capture card in the same system you are playing on does absolutely nothing. Why didn't I look into this more, this thing was like $800! Oh well it is still in it's return window. Back to OBS without the capture card I must be doing something wrong. Even with just 1920x1080 with a 2500 bit rate I can't capture 100FPS, how is this possible I see YouTube videos of people capturing lossless 4k using NVENC?

After some more research I found people buying second GPUs for dedicated encoding, maybe while I'm playing it's using too much of the GPU in game and there not enough left to encode. Just to test I started to record and GPU usage went up to 40% with no game open, what the heck? I though NVENC was controlled by a dedicated chip on the GPU supposedly unaffected by what else you are running... Oh well must have been wrong, I'll try a second GPU.|

I install an additional 1050ti for dedicated encoding - same results as before, can't even get 100fps at 1920x1080 100 bit with NVENC using GPU 1 (1050ti). Furthermore when I start to record, even without a game open my 1050ti reaches 60% usage when I start recording, what?

What on earth could be going on? So far my best recording have come from the capture card using it's software, really not terrible recordings but only 60FPS at a higher resolution than what I need with a really low bit rate, tons of artifacting and occasional screen tears... Why is this even working better than OBS if capture cards do basically nothing running in the same PC you are recording on?

If the solution to this is literally building a second PC than so be it, I'm already $1000 in with the capture card and GTX1050ti. But I want to be sure that is actually the solution, so much evidence online seems to point to the contrary but my experience says otherwise. It just doesn't make any sense to me that other people online could be getting 4k 60fps lossless using NVENC on a PC not nearly as powerful as mine. Heck when I'm recording x264 regardless of settings my CPU doesn't appear to be maxing out, neither do either of my GPUs when using NVENC, why would it drop FPS before using all hardware potential?

I could be a bit more detailed with my OBS settings but I'm using seemingly standard ones according to any tutorial I watch / read and didn't want to make the post any longer as I knew it would be a long one. Does anybody have any advice at all? I am completely stumped.

PS: used Rocket League for testing FPS and regardless of capture settings (OBS or Capture Express) my game was always running 3440x1440 @ 100FPS.

 

[Gabriel Balaich]

Update:

When it says it's using 40% of my GPU I am reading this in task manager and it is actually just the encoder if you drill in for more details. Jumps up to 40% before even opening a game and capturing nothing but a black screen and still odd that it would start dropping frames before getting passed 40-60%...

 

[TryHD]

You just want to record, so Shadowplay on the GTX1080 and audiacity for voice recording and you are done. You did waste a lot of money for nothing.

 

[Boildown]

He said Shadowplay wouldn't work because it focuses on his audio mixer instead of the game.

Get OBS working at 1080p60 first, using software only, no capture card. If you can't do that you won't be able to do the advanced stuff. This is testing, not a the final solution.

Trying to capture > 1080p60 is very bleeding edge, hardly anyone is doing it and posting technical details about it.

If you want actual settings help, you'll need to run 5 minute tests and post an OBS log file from that test. Make sure your tests are valid, for example, of actual high action gameplay, not just standing around with static screens. You want your system stressed when you test.

That Magewell card looks pretty sweet, I have a Datapath DVI-DL myself but that one looks significantly better, not the least of why being that it can do audio too. As you said though, without a second computer you shouldn't need a capture card.

 

[Gabriel Balaich]

Thank you for the replies, and yes shadow play unfortunately doesn't work. I do professional audio mixing / producing and my HDSPe / Fireface UC setup requires the software mixer to run and totally breaks Shadowplay. As for my voice recording, that runs on a separate computer and I capture my voice and discord in Adobe Audition. To be honest even when Shadowplay would run it gave me issues regularly, like out of sync audio and broken segments of the video. Was looking for something a bit more reliable and at a higher bitrate.

After further testing I was actually able to capture 2560x1080 @ 60FPS, it seemed to be pretty consistent, I never saw the FPS dip at least, I don't know if there is a section of the program that specifically shows dropped frames. I think the really interesting part is that the capture card in Capture Express using NVENC captured 4k 60FPS at 16000Kbps no sweat, I cannot seem to replicate this in OBS with or without the capture card. It almost makes me believe the capture card has a dedicated chip for some sort of additional processing.

I've found that changing the bitrate in Output Mode:Advanced>Recording>Standard does nothing for performance. If I try to record 2560x1080 @ 100fps Lossless I get around 70FPS, if I try the same res and frame rate at a 100 bitrate (CBR but other rate controls seem to make no difference) I still get around 70FPS. I would think changing the Bitrate would massively improve frame rate but that appears to be untrue, clearly one thing I've learned here is that I know almost nothing about this stuff. If I go any higher than 2560x1080 I consistantly get around 49 - 53 FPS regardless of bitrate.

I at least know that 4k 60FPS NV12 200Mbps-400Mbps is possible using the capture card in a separate PC due to this video. It's basically just a 40 minute vlog about getting the capture card to work but in the end he has it running in a second PC using NVENC from a 980TI capturing 4k 60FPS 200Mbps and up. However he mentioned he's using FFmpeg but also alluded to previously using OBS before wanting extra customization.

I just wonder if one of my BIOS settings meant for reducing audio latency for producing music is somehow screwing with my ability to do this stuff in one PC.

 

[Boildown]

The problem from an OBS support perspective is that you're in a place that hardly anyone goes. Trying to stream a non-gaming application, (sometimes) using a capture card in a single-PC solution, and doing it at over 1920x1080p60. And no log files, just theorycrafting. So the problem is, your case is so specific no one can theorycraft it with you.

One thing: There's practically zero correlation between bitrate and CPU or GPU usage. The preset used and the pixel rate (resolutionxframerate) has the biggest correlation.

 

[Gabriel Balaich]

It's true not many go this route, didn't mean to complicate things but I do intend to use this almost solely for game recording, I don't really plan on streaming just capturing game play videos for now.

I've seen several people talking about capturing 4k 60FPS on OBS using NVENC but I guess I don't actually see a lot of videos for it, but then again Shadowplay does it fine for many using NVENC. It only made sense to me that OBS would be able to do the same using NVENC, at least at the same bitrate. I'm fairly certain that the NVENC in 10 series was actually setup to theoretically encode 8k.

 

[Adrian Schofield]

Your Magewell probably has onboard encoding however this encoding power is not available to OBS. This is true of the Elgato HD 60 pro which only exposes it's encoding capability to Elgato's software. OBS has to use your GPU for encoding (or CPU) as you know. This would explain why things work in your Magewell software but not in OBS with a like for like configuration. This is why not many folks use a capture card in a single PC setup as it doesn't help much outside the capture cards software.

In theory NVENC and OBS should be able to get you 4K but I'm not exactly sure what it is you are trying to capture - if it's non gaming then desktop capture ought to work for you otherwise Windows capture might work it depends what you are capturing.

Also you never posted a log of the problem where you fail to capture 4K that may help us.

https://obsproject.com/forum/threads/please-post-a-log-with-your-issue-heres-how.23074/

 

[Gabriel Balaich]

Thank you for the reply, I believe the first message is a bit confusing, was trying to give as much detail as possible without making the post a novel.

Let me explain what my exact end goal is:

Record the gameplay on my PC at 3440x1440 100FPS with a bitrate 200Mbps+ preferebly with a 10-20 min replay buffer the majority of the time. (plan on upping ram to 64gb)
Record the gameplay on my Switch 1080p 60fps with bitrate 200Mbps+
Record the gameplay on my PS4 Pro 4k 60fps with bitrate 200Mbps+

The last 2 are novelties and I understand I will need to keep the capture card to record them, I occasionally record my desktop (as in out of games) but I feel like that will just work no problems.

Currently at work but will happily pass over any log file when I get home to do more testing.

I mostly play Rocket League on my PC, very fast pace and tons of moving objects so should be about as bad as it gets. I play some PUBG and MOBAs as well, so far my FPS hovers in the same windows I described earlier regardless of visuals though.

 

[Gabriel Balaich]

Here are 2 log files, both doing 3440x1440 in and out at 100 fps, however I changed the bitrate from 250000 to 100 in the second test. Like I was mentioning before the frame counter on obs seemed to have hovered between 49 - 53 even after changing the bitrate.

 

[sam686]

OBS-Studio logs shows render lag, not encoder lag. Your encoder is fine as it is.

As for render lag...

... I have a 3440x1440 100hrz ASUS display (PG348Q)...

OBS-Studio logs also shows as loading up on GTX 1080 ti, with 3 monitors.

Your monitor have g-sync, and screenshots of GPU usage in task manager is from "creator's update".
The problem might be g-sync, windows "Creators update", and multi monitor. All of this mostly doesn't work together yet. Try disabling g-sync.
https://www.google.com/search?q=creators+update+g-sync
edit: also there is a problem with DWM + multi monitor + different frame rate, all of that may v-sync to slower refresh rate monitor.
https://www.google.com/search?q=windows+dwm+multi+monitor+different+refresh+rate

Additionally, with each monitor plugged into different graphics card, you can try to change primary monitor to the other graphics card, launch OBS-studio, then switch back primary monitor. Works good with capture card. This slows down game capture, and monitor capture won't work on different graphics card.

Capture card works best with 2 computers. Streaming computer and gaming computer (or gaming console).

 

[Gabriel Balaich]

OBS-Studio logs shows render lag, not encoder lag. Your encoder is fine as it is.

As for render lag...

OBS-Studio logs also shows as loading up on GTX 1080 ti, with 3 monitors.

Your monitor have g-sync, and screenshots of GPU usage in task manager is from "creator's update".
The problem might be g-sync, windows "Creators update", and multi monitor. All of this mostly doesn't work together yet. Try disabling g-sync.
https://www.google.com/search?q=creators+update+g-sync
edit: also there is a problem with DWM + multi monitor + different frame rate, all of that may v-sync to slower refresh rate monitor.
https://www.google.com/search?q=windows+dwm+multi+monitor+different+refresh+rate

Additionally, with each monitor plugged into different graphics card, you can try to change primary monitor to the other graphics card, launch OBS-studio, then switch back primary monitor. Works good with capture card. This slows down game capture, and monitor capture won't work on different graphics card.

Capture card works best with 2 computers. Streaming computer and gaming computer (or gaming console).

Welp, that made all the difference.

After disabling G-Sync I saw my frames stick at 100FPS in OBS, however after previewing the video it was clear it was still not 100FPS, more like 25FPS very stuttery regardless of the log file saying I dropped next to no frames. The next time I disconnected all monitors except for the main one and left G-Sync off. I couldn't monitor OBS anymore as I only had one monitor but when I went to play the video back, sure enough, 3440x1440 100FPS. Some stuttering here and there, but very little. I assume this is because the capture isn't replicating V-Sync or something. After opening the log I was dropping less than half a percent of all frames as opposed to 36-40% before.

I guess my worst fears have been confirmed, building a second PC may be the most viable option here. I run 3 displays off my main computer and another 2 off a secondary computer, I use all of them at basically all times. Now the question is how do I monitor OBS on the second computer without buying another 3440x1440 100Hz monitor. Maybe I could emulate one and remote into the computer? Would running a 3440x1440 100hz display along side the capture card even work? Would it even matter, could I just run a 1080p monitor with no problems because the "monitor" being recorded was no longer a monitor but a capture card? Who knows, but I plan on finding out.

Thank you so much for the help, I'll update when I've got everything setup.

 

[sam686]

The monitor's resolution and refresh rate of streaming computer shouldn't matter. Maybe go with older windows 7 that allows turning off DWM/Aero that was forcing v-sync, or maybe a free, newest Linux OS that can shut off v-sync. The capture card link from your first post says "Compatible with Linux".

There are ways to transfer video files to your gaming computer with windows file sharing to play back to high refresh rate monitor running at 100Hz.