Question / Help Trying to create a window capture for any window except OBS fails with BadAlloc

[zepar]

My system is Manjaro, with Gnome 3.34

and anytime i want to create a window capture, terminal gives out the error message

error: xcompcap: glXCreatePixmap failed: BadAlloc (insufficient resources for operation)

for any window that i tried to record, except for OBS itself, which it will happily record

info: xcompcap: [window-capture: 'window capture'] update settings:
    title: OBS 24.0.1-1 (linux) - Profil: Unbenannt - Szenen: Unbenannt
    class: obs
    Bit depth: 24
    Found proper GLXFBConfig (in 9): yes

i know that i could stream stuff completely fine in gnome 3.32, but recently the update for 3.34 came out and i installed it on the same day, and today is the first time i actually tried to stream again, and this is the error
i also havent found anybody else talking about this yet, so im creating this thread here now

 

[Tuna]

Probably your desktop session is now running on Wayland (echo $XDG_SESSION_TYPE)? Only apps running on X11 backwards compatible mode (like OBS) can be captured with XSHM window capture. Native wayland windows cannot be captured with that plugin.

If you want this to work you need to start your desktop in an X11/Xorg session instead of Wayland.

 

[zepar]

Probably your desktop session is now running on Wayland (echo $XDG_SESSION_TYPE)? Only apps running on X11 backwards compatible mode (like OBS) can be captured with XSHM window capture. Native wayland windows cannot be captured with that plugin.

If you want this to work you need to start your desktop in an X11/Xorg session instead of Wayland.

no, im running on xorg11

 

[zepar]

on further investigation, a lot of programs are actually possible to record with no problem (firefox, thunderbird, spotify, team fortress 2), but certain windows are still completely impossible (rhythmbox, nautilus, gnome-terminal)
is it possible that im missing some dependency? pamac tells me i have every dependency installed

[zepar@zepar-pc ~]$ pamac info obs-studio
Name                     : obs-studio
Version                  : 24.0.1-1
Beschreibung             : Free, open source software for live streaming and recording
URL                      : https://obsproject.com
Lizenzen                 : GPL2
Paketquelle              : community
Installierte Größe       : 12,7 MB
Hängt ab von             : ffmpeg jansson libxinerama libxkbcommon-x11 mbedtls qt5-svg qt5-x11extras curl jack gtk-update-icon-cache
Optionale Abhängigkeiten : libfdk-aac: FDK AAC codec support [Installiert]
                           libxcomposite: XComposite capture support [Installiert]
                           libva-intel-driver: hardware encoding [Installiert]
                           libva-mesa-driver: hardware encoding [Installiert]
                           luajit: scripting support [Installiert]
                           python: scripting support [Installiert]
                           vlc: VLC Media Source support [Installiert]
Benötigt von             : obs-linuxbrowser-bin
Packer                   : Jonathan Steel <jsteel@archlinux.org>
Erstelldatum             : 22.09.2019
Installiert am           : 15.10.2019
Installationsgrund       : Ausdrücklich installiert

 

[Xiomaro]

I've had this same issue on Arch for over a month now. I can capture those same windows on KDE, XFCE, i3 and many other DEs but Gnome and some of its close relatives do exactly what you're describing.

I'm sorry this doesn't help but please update the thread if you find a solution. I've been searching for a while now and I've just switched away from Gnome for the time being. I'll also update if I find a solution.

 

[codiflow]

Hey guys,

I got the same problem trying to record two instances of Telegram in separate windows.

Maybe my information can help a bit further solve this problem:

System: Linux MYPC 5.5.6-1-MANJARO #1 SMP Mon Feb 24 09:24:51 UTC 2020 x86_64 GNU/Linux, Manjaro Linux
Compositor: X11
Graphics: GP104 [nVidia GeForce GTX 1070], nonfree driver
Desktop environment: XFCE
OBS Studio: 24.0.6-1 (64 bit)

What I did:

I opened two instances of Telegram with separate configuration folders (Command: /home/codiflow/Telegram/Telegram -many -workdir /home/codiflow/.TelegramDesktop_Account_1/)

Afterwards I started obs using terminal and added the first Xcomposite window source:

info: User added source 'Telegram 1' (xcomposite_input) to scene 'Szene'
error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
info: xcompcap: [window-capture: 'Telegram 1'] update settings:
title: xfce4-panel
class: xfce4-panel
Bit depth: 32
Found proper GLXFBConfig (in 9): yes

error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
info: xcompcap: [window-capture: 'Telegram 1'] update settings:
title: Telegram
class: Telegram
Bit depth: 24
Found proper GLXFBConfig (in 9): yes

So far everything went fine. I got the first Telegram window inside my screen and it gets recorded like it should.

Now I added the second Xcomposite window source getting another error message (orange). But only after I selected the second Telegram window from the dropdown menu:

info: User added source 'Telegram 2' (xcomposite_input) to scene 'Szene'
error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
info: xcompcap: [window-capture: 'Telegram 2'] update settings:
title: xfce4-panel
class: xfce4-panel
Bit depth: 32
Found proper GLXFBConfig (in 9): yes

error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
error: xcompcap: glXCreatePixmap failed: BadAlloc (insufficient resources for operation)
error: xcompcap: cleanup glXReleaseTexImageEXT failed: GLXBadPixmap
info: xcompcap: [window-capture: 'Telegram 1'] update settings:
title: Telegram
class: Telegram
Bit depth: 24
Found proper GLXFBConfig (in 9): yes

So far I can say it makes no difference if both Telegram windows use a different binary.

I will investigate further and report it back here.

 

[IcaroMuniz]

Any updates?
Having the same issue here on 20.04 with Nvidia 970 card.
As far as I can tell, the problem is mainly related to some gnome's default applications like Terminal, Gedit, Nautilus, etc.
LibreOffice's Writer and Impress, for instance, are working just fine.
Cheers!