Fedora 41 - OpenH264 CPU usage low with very low output quality

[SaintIsaiah]

Hey everyone,

I've been using OBS for about 8 years now on Windows. I have recently made the switch to Linux, specifically Fedora 41, so that I am doing all of my work on Linux and just my gaming or Windows app development on a Windows VM via VFIO and Looking Glass.

I have Installed OBS and got the Looking Glass plugin working without any issues, but when reviewing the recording the quality is just terrible. Despite the my settings to ensure the bitrate is at a decent amount (30,000kbps) the video has lots of artifacts from high motion, the playback is very laggy (as if it's playing at 12-16 fps rather than 60 fps) and the CPU usage tops out around 6% according to OBS. Whenever I would record/stream on Windows I would just use x264, set the cpu preset to medium/slow with a decent bitrate and my recordings looked great, but on Linux the settings are quite different, with "x264" not being an option and only "OpenH264" being present and no cpu preset like with x264, only the profile and bitrate. I have to rely on my CPU to encode this as my RTX 4090 is used for passthrough on the Windows VM. So with all of this in mind, here are immediate questions I have to better understand this:

1. Is "x264" not an option on linux? Or is it something I need to compile and install myself due to licensing issues on Linux?
2. Is "OpenH264" using the CPU, or trying to use my iGPU to encode the recording?
3. Is "OpenH264" basically just as capable as "x264", but merely an open-source solution?
4. Are there specific tweaks both within OBS Studio and on my system I should be doing so that OBS will use enough CPU to keep up with the settings I have in place?

Below are the settings I was using for my recordings in Fedora. I tried this on both the dnf version and the flatpak version with no difference in quality.

My settings
Video Output: 2560x1440 @ 60fps (not scaled)
Recording Format: mkv
Video Encoder: OpenH264
Encoder Profile: Main
Encoder Bitrate: 30000kbps

My Specs
CPU: Ryzen 9 7950X3D (I have used isolcpus and nohz to statically reserve CCD0 cores/threads except Core 0 with 3D VCache for the Windows VM - all other cores/threads with Core 0 are available to and used by the Linux host)

RAM: 64GB (2 x 32GB) Corsair Vengeance DDR5 @ 6400mhz (32GB are utilized on the Windows VM, leaving 32GB for the Linux host)

GPU: Dell OEM RTX 4090 - Assigned to VFIO and reserved strictly for the Windows VM

OS: Fedora 41 (KDE variant) with Wayland.

I am using the Radeon iGPU to drive my monitors.

Other than OBS having a quality issue with the video output, I am having zero performance issues with my setup. Everything from gaming on the VM to working on large development projects in Linux all work without issue.

---

If anyone can help me identify what I can fix so that the recordings are along the same level of quality of the gaming, I would greatly appreciate it.

 

[SaintIsaiah]

Small update from some further testing:

I decided to go into the recording tab and use the "Custom Output (FFmpeg)" option. By setting my Container Output to mp4 and the encoder as "libx264", this resulted in a much higher quality output with less artifacts during high motion and no more laggy framerate. While this result is encouraging that it doesn't seem to be a major issue with my system, using this option is not ideal because I would still have to rely on "OpenH264" in the streaming options, which would result in the laggy framerate and the artifacts during high motion.

Is there a better option for me so that will work for both streaming and recording that will properly utilize my CPU and prevent problems with video quality?

 

[Tuna]

Fedora is very restrictive when it comes to non-free codec. Try installing the Flatpak version (from OBS, not Fedora.. which is another weirdness with Fedora..).

 

[SaintIsaiah]

Fedora is very restrictive when it comes to non-free codec. Try installing the Flatpak version (from OBS, not Fedora.. which is another weirdness with Fedora..).

Well would you look at that, the flatpak version from Flathub has x264 with all of the settings I was looking for.

Now the next question would be how can I move over my Looking Glass plugin into the Flatpak version? I tried copying over the file, but even with dolphin opened with the --sudo flag, it says I don't have permission. I'm using the non-immutable version of Fedora and I've never really used Flatpaks yet, so I'm not aware of how I can properly do this.

 

[Tuna]

It wouldd have to provide a Flatpak plugin. Everything else is a hack and may not work. Or you try to build your own native version of OBS..

 

[SaintIsaiah]

It wouldd have to provide a Flatpak plugin. Everything else is a hack and may not work. Or you try to build your own native version of OBS..

I just managed to stumble into the right answer to what I was looking for. To move the plugin into the Flatpak version, I did the following:

1. In OBS, I went to File-> Show Settings Folder
2. In the settings folder, I created a new folder called "plugins" and dropped the Looking Glass Client plugin into that folder.

To get the actual plugin to work correctly, because it reads the shared memory buffer that the VM creates to eliminate lag, I installed Flatseal and gave OBS access to the Virtualization and Shared Memory devices.

I still have some fine tweaking to do so the quality is exactly where I need it, but after running a couple test recording it's 1000% better and more in line with what I was expecting. Thanks for pointing me in the right direction Tuna! <3