Experimental zero-copy screen capture on Linux

w23

New Member
#1
I was having performance issues when streaming livecoding heavy GLSL raymarching and pathtracing shaders, so instead of optimizing my shaders I read about libdrm, KMS, DMA-BUF and EGL. As a result, I made a very experimental zero-copy screen capture OBS plugin for Linux based on DMA-BUF fds and EGLImages.
It does solve most of my performance woes, and you can see this screen capture method in action here: https://youtu.be/L2Y7_vBfWm8 or here https://www.twitch.tv/videos/389112056
Notice how even for heavy shaders it still maintains real-time fps. Vanilla OBS with XSHM would struggle capturing it and would barely keep ~10 fps, even though the shader itself can be as high as 40-60fps.

Background
There's this DRM infrastructure that is used to talk to GPUs in a vendor-agnostic way on Linux. Among other things it incorporates:
  • Kernel modesetting (KMS), a technology to enumerate and control video outputs, their video modes and framebuffers
  • DMA-BUF objects, basically handles to GPU-side memory, e.g. framebuffers and textures. Handles to these are file descriptors.
Then there's a EGL interface that can be used to manage OpenGL contexts and resources. It is analogous to GLX and WGL, but is newer, better, more portable and with more extensions.
Among these extensions we can find:
  • EGL_EXT_image_dma_buf_import, it allows creating EGLImage objects bound to existing DMA-BUF object (using its fd)
  • GL_OES_EGL_image, it allows binding GL textures to EGLImage objects. Note that while it is technically an OpenGL ES extenstion, it is exposed in Mesa implementation of desktop OpenGL, and works there just fine on opensource amdgpu and intel drivers.

I bet you can see that all of these fit together perfectly. Moreover, this can be used to capture not only X11, but everything else that is backed by KMS, including Wayland and even bare terminals!

One caveat though is that getting framebuffer DMA-BUF fd requires CAP_SYS_ADMIN, and running OBS with what basically amounts to root privileges is a YOLO practice.
However, you can remember such things as UNIX sockets, sendmsg() functions and SCM_RIGHTS flags. These can be used to transfer open file descriptors between processes. This also works on DMA-BUF fds, and so we can construct a small binary with `setcap cap_sys_admin+ep` with only purpose to get the framebuffer fd and transfer it to whoever.

Now we can make it work.

Overview
Necessary changes to OBS are:
  • Use EGL instead of GLX to initialize OpenGL context. This is a change to libobs-opengl library.
  • Create a new source plugin that creates a GL texture from EGLImage created from DMA-BUF fd read from UNIX socket.

You can find these changes in this branch: https://github.com/w23/obs-studio/tree/linux-libdrm-grab

A setuid/cap_sys_admin+ep utility is also needed to send the dma-buf fd.
It can be found in a separate repository: https://github.com/w23/drmtoy/tree/wip-drm-send-recv

How to

Now, as I got it to work only yesterday, this is highly experimental.
It is painful to set up at the time of writing.

  1. It is assumed that you have all the necessary development libraries and tools installed on your system
  2. Get drmtoy enum and drmsend
    Code:
    git clone https://github.com/w23/drmtoy.git
    cd drmtoy
    git checkout wip-drm-send-recv
    make enum drmsend
  3. Find the right framebuffer id for your screen
    Code:
    ./build/rel/enum
    It will output lots of lines. You're interested in the last few lines that will looks something like these:
    Code:
    count_fbs = 2
           0: 0x56
               width=3200 height=1800 pitch=12800 bpp=32 depth=24 handle=0
           1: 0x55
               width=256 height=256 pitch=1024 bpp=32 depth=32 handle=0
    Here you can see that there's a framebuffer 0x56 sized 3200x1800. This looks like the main screen.
    0x55 is 256x256 and doesn't look like anything to me
  4. (a) Run drmsend with elevated privileges. Replace 0x56 with your framebuffer id
    Code:
    sudo ./build/rel/drmsend 0x56 drmsend.sock &
    # chown socket so that your user can access it
    USER=$(whoami) sudo chown $USER drmsend.sock
    (b) Alternatively you can set the right caps on drmsend and run it under a regular user. Note that you'd need your fs to be mounted without nosuid flag.
    Code:
    sudo setcap cap_sys_admin+ep ./build/rel/drmsend
    sudo chown root ./build/rel/drmsend
    ./build/rel/drmsend 0x56 drmsend.sock &
  5. Get patched OBS
    Code:
    git clone https://github.com/w23/obs-studio.git
    cd obs-studio
    git checkout linux-libdrm-grab
  6. Build patched OBS
    Code:
    mkdir build
    cd build
    cmake .. -DUNIX_STRUCTURE=0 -DUSE_EGL=1 -GNinja
    ninja
  7. Run it.
    Note that -DUSE_EGL=1 forcefully replaces GLX with EGL, and none of GLX-dependent things are patched to support EGL. E.g. XSHM and Xcomp screen capture modules won't work, and the entire linux-capture plugin won't even load due to missing symbols.
    Code:
    cd rundir/RelWithDebInfo/bin/64bit/
    -p is for portable mode, for it to not mess with your existing obs configuration
    If you're feeling adventurous (like I am), back up your obs config and run it w/o -p flag
    Code:
    ./obs -p
  8. Now you can add "DMA-BUF source" to your scene as you would with any other regular source.
    In configuration dialog you need to specify drmsend socket:
    1. Click "Browse" and navigate to the directory with drmsend.sock, e.g. where you checked out drmtoy and ran build/rel/drmsend
    2. QFileDialog will be unhelpful enough to not show UNIX sockets, so you'd need to type your filename manually, e.g. drmsend.sock and press enter.
    3. Now you should have a preview of your framebuffer screen. Note that there's no cursor capture yet.
    4. VYGODA

Please test and enjoy!

Notes and questions
There are several things that I want to point out and discuss with experienced OBS developers.

  1. Current experimental implementation makes EGL vs GLX a compile-time choice using -DUSE_EGL=1 cmake argument. It is not that hard to make a separate libobs-opengl-egl.so plugin and make choosing between it and legacy GLX-based libobs-opengl.so a runtime choice based on user preference, like DX vs GL choice on Windows.
    XSHM, Xcomp and DMABUF sources will need to be able to detect GLX vs EGL at runtime.
    Is there a better way than `obs_get_video_info()` and then checking `graphics_module` name?
    There's also an issue of one of GLX or EGL not being available (e.g. on Wayland or older/weird X respectively) on some system.
    I haven't looked at how libglad works, but it will likely require splitting libglad into libglad-gl, libglag-glx and libglad-egl. Also, missing symbols will need to be handled gracefully by linux-capture plugin.
  2. linux-dmabuf plugin is a temporary thing. I feel it needs to be integrated into linux-capture w/ added xcursor support. But see note above about dynamic decision of EGL/GLX and symbols.
  3. dmabuf_source module requires EGLDisplay handle, which lives deep inside libobs-opengl/EGL plugin. Is there a recommended way to get handles internal to some graphics impl? I couldn't find one.
    Currently dmabuf_source includes graphics-internal.h, declares struct gl_platform itself (copied from gl-x11-egl.c), calls gs_get_context() and chases some pointers (see https://github.com/w23/obs-studio/b...c3cd0f0ba7/plugins/linux-dmabuf/dmabuf.c#L145). This is obviously not sustainable.
  4. Mode changes aren't handled. A the moment I have no idea how to do that, and how current implementation would behave in such case.
  5. I believe that sampling this DMA-BUF-backed texture will read framebuffer memory directly. No synchronization is implemented. This may or may not be a problem.
  6. Obviously this scheme with manually running enum and drmsend (from another repo!) is not very user-friendly. However, making it user-friendly in general case is hard.
    One feasibly-looking approach is:
    • integrate drmsend into obs repo
    • make drmsend perform framebuffer discovery
    • dmabuf_source would have picker for framebuffers in its config dialog
    • dmabuf_source would spawn drmsend with right arguments itself

    Later we could make drmsend even smarter:
    • make it not framebuffer-centric, but CRTC (monitor)-centric.
    • make it listen on mode change events
    • make dmabuf_source listen on drmsend events

    This would make it almost work for mode changes, but not quite. Xorg creates one huge framebuffer for multiple monitors, so mode changes would affect monitor positions and cropping coords. I don't think these rules are generalizable with OBS transformations.
  7. Privileged drmsend itself is too public morozov:
    • CAP_SYS_ADMIN is too much, we need to patch kernel with something like CAP_DRM_CAPTURE as a more fine-grained capability
    • leave control over which users in the system can capture screens into distro packagers hands (they might e.g. add video-capture group or whatever).
 
Last edited:

w23

New Member
#2
I've committed a change that allows user selection between GLX and EGL. So now one needs to go to Settings -> Advanced -> Video and select "OpenGL EGL" in order to enable DMABUF source.
linux-capture/XSHM works under both GLX and EGL.
linux-capture/Xcomposite crashes under EGL, so it is disabled in EGL mode for now.
 
#3
Very interesting, and good talking to you in Discord. I'm excited to see where this goes and hopefully it will lead to a PR for OBS.

So is XSHM more performant when using EGL?
 

w23

New Member
#4
Very interesting, and good talking to you in Discord. I'm excited to see where this goes and hopefully it will lead to a PR for OBS.
Thanks! it was also nice talking to you and other developers. I appreciate the help.

I'm slowly working towards making it a PR, and hope to be able to submit it by this weekend.

So is XSHM more performant when using EGL?
I haven't profiled that, but I'd expect XSHM under EGL should perform basically the same as under GLX. It is still essentially the same (1) instruct XCB/Xorg to copy image data from GPU into RAM, (2) upload this data into GL texture back on GPU.

I've also added drmsend util into my obs-studio fork, so now it is built as obs-drmsend binary as part of obs build process. Note that adding necessary capabilities is commented out for now, as it requires building w/ root. I will address this later, or just leave it into developers' or packager hands.

Next I plan to add libdrm framebuffer enumeration into dmabuf settings, but will likely have time for that only tomorrow.
 

w23

New Member
#5
I just pushed an update that adds a GUI selector for framebuffers. It works by calling obs-drmsend binary that enumerates libdrm resources and gets fds for all available framebuffers and sends these along with metadata to master obs process via unix socket.

obs-drmsend is integrated into the repo and build system, so there's no need to get the drmtoy repo or build anything manually anymore.

Updated instructions:
  1. Get the linux-libdrm-grab branch from https://github.com/w23/obs-studio
  2. Build it as usual: mkdir build && cd build && cmake .. -DUNIX_STRUCTURE=0 -GNinja && ninja. It should pick up EGL if you have it in your system.
  3. cd rundir/RelWithDebInfo/bin/64bit and manually assign obs-drmsend with CAP_SYS_ADMIN: sudo setcap cap_sys_admin+ep ./obs-drmsend. Make sure that you don't have nosuid bit on your filesystem from where you're running obs.
  4. Run obs while being in the same dir: ./obs -p. Go to Settings -> Advanced -> Video and select "OpenGL EGL".
  5. Restart OBS
  6. Add "DMABUF source"
  7. Property screen should appear where you can pick up your framebuffer.
  8. Have a great optimized stream!

Note that some paths are still hardcoded:
  • GPU is expected to be accessible at /dev/dri/card0
  • ./obs-drmsend is run from current directory
  • Unix socket will be at /tmp/drmsend.sock

There's a technical difficulty at getting the filename of the GPU used by current EGL context. It may not be accessible at all. Also, it might be possible to grab framebuffers from another GPU, but I don't have a way to test that.

What would be the right path for obs temp stuff?
 
#6
Depends on what exactly is the temp material...but if I had to direct you to one safe place, the user's home directory, maybe in the hidden obs-studio profile/settings folder:
~/.config/obs-studio

and maybe make a temp folder in there...but really tmp files in Linux-land are placed depending on what they are for.
 
Top