Newly Added Vaapi AV1 Option Crashes OBS When Trying to Record

[BluesAdam]

Hi everyone,

First off, here is my system specs:

I am not well versed in terminal commands so if there is any other command I can run to give more detailed information about my system, please let me know and I will. So I have been using OBS Studio with "FFmpeg Vaapi AV1" encoder option with no issues. A couple days ago I got an update to OBS, which seemingly added another encoder, namely "Vaapi AV1." So I wanted to try it hoping maybe it could give better image quality or performance but after recording for a few seconds, OBS closes without an error or anything, it just suddenly disappears. Whatever footage it records is there, so I know it is succeeding in recording until the crash.

I am using game capture and tried running the game on XWayland and Wayland methods, tried enabling and disabling NTSync, which made no difference. Upon checking log files, I noticed it does not even mention OBS shutting down or any error at the end of the log. I have uploaded the latest log as an attachment. Please let me know if you need any more information and let me know how I can obtain that information. I am not good at using the terminal so do not know what type of commands I should be using to get information.

Thanks a lot in advance.

 

[BluesAdam]

So upon further testing, I think I understand what is wrong. First I realized Manguhud was causing issues. Because OBS would shutdown whenever Mangohud overlay came up. So I turned it off and managed to make longer recordings. First went on for 4 minutes before OBS crashing again. Last one I actually managed to record an entire match, 35 minutes long but the moment the match ended and game returned to main menu, OBS crashed.

So for some reason, the new Vaapi AV1 encoder is very touchy when it comes to screen changes. I don't know if it is the scene change detection algorithm, or if the game actually changes the window in some way when a game ends but, any big changes cause an OBS crash. I am not sure about this, but I think it crashed once when I minimized OBS window while checking things.

So it is very touchy. But whatever it could record, quality was noticeably worse compared to FFmpeg Vaapi AV1. On top of that CPU usage was higher by around 2% (2.5% for ffmpeg and 4.5% for vaapi.) It also used around 150mb more vram. So I suggest sticking with ffmpeg vaapi av1 encoder for now.

Attaching log file again, but similarly the log ends abruptly with no warning whatsoever.

 

[Tuna]

You are using a 3rd party VAAPI encoder. At the same time it is not using the latest available GStreamer version.

 

[BluesAdam]

You are using a 3rd party VAAPI encoder. At the same time it is not using the latest available GStreamer version.

Well, since it came with the latest OBS update, I thought it was official. After all I did not personally add it or anything. I think they will update it eventually though. Will give it another shot in future versions. Thanks for the heads up about the GStreamer version.

 

[Tuna]

P.S. The plugin is by me and it should be working.. at least that is my intention. But it would require a GDB backtrace to see what exactly is crashing.

 

[BluesAdam]

P.S. The plugin is by me and it should be working.. at least that is my intention. But it would require a GDB backtrace to see what exactly is crashing.

If there is anything I can do to help trace issues, just let me know. I am not well-versed in Linux so don't know much but will do anything I can.

 

[Tuna]

Not sure how Bazzite works, but here is the run down:

To help resolve your issue, we need a debug backtrace. Please follow these steps to get a backtrace using GDB (GNU Debugger):


If you're on a native package (like the Ubuntu PPA), skip to step 3.

1. Install requirements: flatpak install org.kde.Sdk/x86_64/6.6 com.obsproject.Studio.Debug
2. Run flatpak run --command=sh --devel com.obsproject.Studio
3. Run gdb obs
4. Once you are in the (gdb) prompt run set logging enabled on
5. In the (gdb) prompt run run
6. This will download a bunch of additional debug information on first run, just wait for it to finish
7. Once OBS starts, reproduce your crash
8. Once crashed OBS will freeze. In the (gdb) prompt run bt full
9. To get out of gdb run exit
   • If you are on Flatpak: run exit again to get out of the Flatpak

 

[BluesAdam]

Hey again!

In step 1 the command installs version 6.6 but when I run step 2 it says version 6.8 is not installed. So I modified the command on step one to install version 6.8 and the rest worked successfully. I hope this doesn't skew the debugger logs.

I have attached the log. The only thing I saw said a segmentation fault. Recording took 16 seconds to freeze, immediately during a scene change (game goes from character selection screen to hub area.)

I hope this helps you track the issue. As a note, following is the only launch parameter running so I can get game capture to work in OBS. I disabled all other launch parameters.

env OBS_VKCAPTURE=1 %command%

EDIT: Btw, do I need to uninstall what I installed with the command in step 1? I don't want to have two copies of OBS on the system. Please let me know :) Nevermind, uninstalled the leftovers from version 6.6 and all is good. Bazzite system updater was giving a warning about an outdated package, which came with the 6.6 version. Now all is good. I don't think I need to uninstall anything else.

 

[Tuna]

Thats interesting, it does happen in the gstreamer encoder. Would be interesting to know if it happens with the latest gstreamer version as well. Its weird that it happens after time x though. Perhaps it is also something with the driver that returns something unexpected which makes the gstreamer encoder crash.

 

[BluesAdam]

Thats interesting, it does happen in the gstreamer encoder. Would be interesting to know if it happens with the latest gstreamer version as well. Its weird that it happens after time x though. Perhaps it is also something with the driver that returns something unexpected which makes the gstreamer encoder crash.

Since Bazzite is practically immutable (creator of Bazzite says it is not immutable but image-based,) I don't know how to update Gstreamer. As always, if you can point me in the right direction, I'm not afraid to get my hands dirty.

 

[Tuna]

I have absolute no idea how bazzite works unfortunately.

 

[BluesAdam]

I have absolute no idea how bazzite works unfortunately.

Asked on Bazzite forums and they said we'll have to wait for Fedora to upgrade it and then for Bazzite to follow. Well, I'll keep checking your encoder in the future for sure. Thanks for sparing your time and attention and wish you the best luck!

 

[Tuna]

I'm on Fedora myself. GStreamer 1.26 has been available for a long time. IT even is at a 1.26.5 point release. So I guess Bazzite may be tracking an old Fedora version then.