Fedora 38: Either getting really bad bitrate or really bad FPS. Also, am I even using my GPU?

I've been trying to figure out how to get my OBS properly working so that its acceptable for recording good quality clips and I don't really know what I'm doing.

In the first clip, I get somewhat okay FPS but the video looks abysmal. Even setting the bitrate to something like 12000 does nothing to change it.
It also results in stuttering of the game, which ends up being reflected in the video, although I'm not sure if it's visible in the clip.
This is just using simple output settings and x264 software encoding, indistinguishable quality.

The second clip has okay visual quality but terrible framerate. I set the output settings to advanced, and am using openh264, with 12000 bitrate
If i use x264 and a higher bitrate in this advanced settings, it's not as bad but there are still pretty frequent stutters.

If I use x264 I get bad bitrate and stutters. If i use openh264, i get horrendous framerate. So kind of stuck between a rock and a hard place.
Do these settings even use my GPU? I have mesa drivers installed (I think fedora even preinstalls them out of the box) but in the simple output settings, there's only software options available. That label isn't in the advanced output options, but I can't really tell. CPU usage goes to about 38% on average when recording, which makes me think I'm not.

CPU: Ryzen 5 2600
GPU: AMD RX-570 4GB
OS: Fedora 38
OBS version: 29.1.3 installed via RPM
Game is THUG2 with THUGPro mod.

LMK if there's any info needed that might help with sorting this out, thanks in advance.
 
Top