Defending against injection

[[DE]Glen]

I work at Digital Extremes on Warframe and we've noticed that we get a lot of crashes in our launcher that are correlated with OBS being injected. These crashes are submitted anonymously so we don't actually know people experiencing the crash but we know that they account for almost half of our crashes in the launcher.

We also don't know that OBS is to blame but it's hard to ignore the fact that 19/20 of the minidumps I checked all had obs-studio-hook\graphics-hook64.dll injected.

The actual crash is in VkLayer_steam_fossilize64.dll which led us astray at first -- it's tricky because the Steam overlay doesn't usually show up for the launcher. It's almost as if it learns that the real game is a different process and you only get it injected into the launcher once. Anyway, all initial attempts to reproduce just with Steam got us nowhere.

When we noticed the correlation with OBS we messed around with adding various capture types and couldn't reproduce the problem -- it's almost as if you have to get the steam overlay *and* have OBS injected to get it to happen and we're kind of stuck.

It's a wild shot in the dark but we were thinking that since game capture injection doesn't actually work with our launcher maybe we could defend against injection somehow so that OBS knows better than to even try? Window capture works -- but game capture seems ill-advised and if we could discourage it we could check our crash stats to see if it makes a difference.

 

[R1CH]

The fact that the Steam Vulkan layer is present means something is triggering Vulkan to load and OBS includes a Vulkan layer for game capture, so it comes along for the ride. If the launcher isn't using Vulkan then I would investigate what is triggering Vulkan (and the associated OBS and Steam layers) to load.

As a brute force approach, you could set launcher environment variables on process start to disable the Steam Vulkan layer which seems to be the one crashing: DISABLE_VK_LAYER_VALVE_steam_fossilize_1=1. Note there is another layer, VK_LAYER_VALVE_steam_overlay that is the actual Steam overlay, Fossilize seems to be some kind of helper library, I am not sure what side effects may occur when disabling it.

 

[[DE]Glen]

We were already trying to opt-out with DISABLE_VK_LAYER_VALVE_steam_overlay_1 and weren't aware of this second interface; we pushed an update an hour ago with DISABLE_VK_LAYER_VALVE_steam_fossilize_1 as well and preliminary results look pretty great! Thanks for the tip!