Question / Help OBS xcomposite problem

[Waishon]

Hello,

I've a problem with adding xcomposite captures of my windows... If I add one i get a red screen in OBS.
This problem didn't exist if I use the nouveau driver, but with the GTX 660 and the Nvidia 304.125 driver.

Errors:

error: xcompcap: glXCreatePixmap failed: BadAlloc (insufficient resources for operation)

If I add Firefox:

error: glCopyImageSubDataNV failed, glGetError returned 0x502
error: device_copy_texture (GL) failed

Here are some usefull informations:
operating system: Elementary OS Freya
I7 3770
GTX 660
16GB Ram

OpenGL:

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GTX 660/PCIe/SSE2
OpenGL core profile version string: 4.2.0 NVIDIA 304.125
OpenGL core profile shading language version string: 4.20 NVIDIA via Cg compiler
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.2.0 NVIDIA 304.125
OpenGL shading language version string: 4.20 NVIDIA via Cg compiler
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:

 

[Maltahl]

Try the included Nvidia Driver 331 on Elementary OS. i use this with no problems other than known issues such as desynced audio.

Please be aware that Elementary OS Freya is a Beta and is not always reliable to do testing of other software on.
For proper testing do this on Stable releases of distros such as Ubuntu 14.04.

 

[Waishon]

Thanks, I will try it and hope that I dont kill my PC :D

Its not the official beta. Its an other unstable build, but with nouveau it works...

 

[termac]

Hi. I experience the same problem. When using xcomposite capture I get red windows in OBS with an error "error: xcompcap: glXCreatePixmap failed: BadAlloc (insufficient resources for operation".
I am using a GTX 970 with 343.36 Nvidia drivers. I got the compositor "compton" running.

I remember when searching for this problem before, there was a thread stating that compton would apply an alpha channel to every window, even if no transparency is used and that would cause problems with OBS.
I can not find that thread again, but does anyone know if this is still the case?

 

[ubuntuaddicted]

i can say that i use compton and don't have a problem capturing most windows. only an xfce4 terminal, that has a transparent layer so it's a red square. i've posted the bug report on mantis. i would advise everyone with the issue go add your situation here: https://obsproject.com/mantis/view.php?id=133

if an issue has enough people complain about it I would think it gets more attention but that's just a guess.

 

[sln9]

Hi, this is happening to me as well. I've added some details on Mantis. A 'funny' thing is that the first time I tried OBS there were no problem at all (I tried with Dota 2 and I could stream just fine). The next day I opened OBS to do it again and I found the "red window" problem. I've tried to remember if I changed something on my system but I have no idea...

 

[admalledd]

Able to reproduce most of the errors on my debug build, so a touch more info:

debug: [API][Error]{High}: GL_INVALID_OPERATION error generated. The source or destination texture is not complete.
error: glCopyImageSubDataNV failed, glGetError returned 0x502
error: device_copy_texture (GL) failed

• Forced nvidia xorg meta-mode setting "ForceFullCompositionPipeline" on/off had no effect.
  
• Other nvidia meta-mode options not able to be tested, they prevent my i3wm from even running :/ (none look like they might do anything anywho)
  
• Changing/Disabling transparency of xcfe4-terminal had no effect (might not remove alpha layer if so?).
• Running with or without compton (compositing manager) had no effect.

Spoiler: Block of wild mass guessing

From what I could debug (and feel free to guide me with more info on what to look at!) it might be something along these lines:

• p.tex, p.pixmap, p.glxpixmap and p.glxtex not actually having matching formats (but some how slipping through the sanity checks?)
• the Xcomposite pixmap becoming invalid and not being updated again. (missed event in XCompcap::processEvents()?)
• p.tex / p.pixmap alpha layer not set properly in source application (unlikely)
• p.tex / p.pixmap alpha layer not being passed from xserver (because we don't ask correctly?), causing invalid pixmap state
• source application is not rendering in xserver (eg minimized window) (had no effect having xfce4-terminal in background or foreground for me with i3wm. Hidden windows also rendered fine, like remmina, or firefox)

Alas this is mostly speculation because I do not quite understand as much as I would like.

For the glXCreatePixmap, I cannot reproduce "BadAlloc" that one (probably due to me using i3-gaps-next over some other WM?) but according to https://www.opengl.org/sdk/docs/man2/xhtml/glXCreatePixmap.xml the "BadAlloc" can come from either inability to alloc the memory, *OR* there is already a "GLXFBConfig" for that pixmap. I am going to admit that my understanding of what that means is basically zero though...

I can reproduce the "glXCreatePixmap failed: BadMatch (invalid parameter attributes)", however I am at a loss on what to even look at for debugging. My simple test of doing "xlock.resetError();" causes indentical 0x502 GL_INVALID_OPERATION Errors. Makes me wonder if they are more related than seems? Root cause something messed up with getting the textures/pixmaps? Again just some wild mass guessing.

Anywho, here is my system information in case that helps:

Spoiler: Big Block of sysinfo

ffmpeg from PPA, obs-studio is from "0.10.1-50-g92cae0c" (latest when testing)

glxinfo (although I don't think anymore that it is openGL's fault...):

OpenGL vendor string: NVIDIA Corporation
OpenGL renderer string: GeForce GTX 860M/PCIe/SSE2
OpenGL core profile version string: 4.4.0 NVIDIA 352.09
OpenGL core profile shading language version string: 4.40 NVIDIA via Cg compiler
OpenGL core profile context flags: (none)
OpenGL core profile profile mask: core profile
OpenGL core profile extensions:
OpenGL version string: 4.5.0 NVIDIA 352.09
OpenGL shading language version string: 4.50 NVIDIA
OpenGL context flags: (none)
OpenGL profile mask: (none)
OpenGL extensions:
OpenGL ES profile version string: OpenGL ES 3.1 NVIDIA 352.09
OpenGL ES profile shading language version string: OpenGL ES GLSL ES 3.10
OpenGL ES profile extensions:

xprop -id 14682712 #The xfce4-terminal that failed.

_NET_WM_STATE(ATOM) =
WM_STATE(WM_STATE):
  window state: Normal
  icon window: 0x0
WM_HINTS(WM_HINTS):
  Client accepts input or input focus: True
  Initial state is Normal State.
  bitmap id # to use for icon: 0xe00020
  bitmap id # of mask for icon: 0xe00023
  window id # of group leader: 0xe00001
XdndAware(ATOM) = BITMAP
_MOTIF_DRAG_RECEIVER_INFO(_MOTIF_DRAG_RECEIVER_INFO) = 0x6c, 0x0, 0x5, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x10, 0x0, 0x0, 0x0
_NET_WM_ICON(CARDINAL) =  #SNIPPED FOR SIZE

_NET_STARTUP_ID(UTF8_STRING) = "i3/i3-sensible-terminal/2032-3-admalledd-laptop_TIME175007"
WM_WINDOW_ROLE(STRING) = "xfce4-terminal-1434393212--1075693122"
_NET_WM_SYNC_REQUEST_COUNTER(CARDINAL) = 14682714
_NET_WM_WINDOW_TYPE(ATOM) = _NET_WM_WINDOW_TYPE_NORMAL
_NET_WM_USER_TIME(CARDINAL) = 10850552
_NET_WM_USER_TIME_WINDOW(WINDOW): window id # 0xe00a59
WM_CLIENT_LEADER(WINDOW): window id # 0xe00001
_NET_WM_PID(CARDINAL) = 2828
WM_LOCALE_NAME(STRING) = "en_US.UTF-8"
WM_CLIENT_MACHINE(STRING) = "admalledd-laptop"
WM_NORMAL_HINTS(WM_SIZE_HINTS):
  program specified minimum size: 48 by 51
  program specified resize increment: 8 by 15
  program specified base size: 16 by 21
  window gravity: NorthWest
WM_PROTOCOLS(ATOM): protocols  WM_DELETE_WINDOW, WM_TAKE_FOCUS, _NET_WM_PING, _NET_WM_SYNC_REQUEST
WM_CLASS(STRING) = "xfce4-terminal", "Xfce4-terminal"
WM_ICON_NAME(STRING) = "Terminal_testing"
_NET_WM_ICON_NAME(UTF8_STRING) = "Terminal_testing"
WM_NAME(STRING) = "Terminal_testing"
_NET_WM_NAME(UTF8_STRING) = "Terminal_testing"

xprop -id 35661739 #window ID of obs-studio itself, which works (no red box)

WM_STATE(WM_STATE):
     window state: Normal
     icon window: 0x0
_NET_WM_ICON(CARDINAL) =
XdndAware(ATOM) = BITMAP
_NET_WM_NAME(UTF8_STRING) = "OBS 0.10.1-50-g92cae0c-modified (linux)"
_MOTIF_WM_HINTS(_MOTIF_WM_HINTS) = 0x3, 0x3e, 0x7e, 0x0, 0x0
_NET_WM_WINDOW_TYPE(ATOM) = _NET_WM_WINDOW_TYPE_NORMAL
_XEMBED_INFO(_XEMBED_INFO) = 0x0, 0x1
WM_CLIENT_LEADER(WINDOW): window id # 0x2600002
WM_HINTS(WM_HINTS):
     Client accepts input or input focus: True
     Initial state is Normal State.
_NET_WM_PID(CARDINAL) = 18008
_NET_WM_SYNC_REQUEST_COUNTER(CARDINAL) = 39845897
WM_CLASS(STRING) = "obs", "obs"
WM_PROTOCOLS(ATOM): protocols  WM_DELETE_WINDOW, WM_TAKE_FOCUS, _NET_WM_PING, _NET_WM_SYNC_REQUEST
WM_NORMAL_HINTS(WM_SIZE_HINTS):
     user specified location: 1922, 20
     user specified size: 1916 by 1037
     program specified minimum size: 734 by 254
     window gravity: Static

"inxi -Fz" info for hardware stuff (snipped a bit)

System:  Host: admalledd-laptop Kernel: 3.13.0-24-generic x86_64 (64 bit) Desktop: i3-gaps-next Distro: Linux Mint 17 Qiana
Machine:  Mobo: ASUSTeK model: G551JM version: 1.0 Bios: American Megatrends version: G551JM.202 date: 07/17/2014
CPU:  Quad core Intel Core i7-4710HQ CPU (-HT-MCP-) cache: 6144 KB flags: (lm nx sse sse2 sse3 sse4_1 sse4_2 ssse3 vmx)
  Clock Speeds: 1: 800.00 MHz 2: 2501.00 MHz 3: 800.00 MHz 4: 800.00 MHz 5: 800.00 MHz 6: 800.00 MHz 7: 800.00 MHz 8: 1200.00 MHz
Graphics:  Card: Intel 4th Gen Core Processor Integrated Graphics Controller
  X.Org: 1.15.1 driver: nvidia Resolution: 1920x1080@60.1hz, 1920x1080@60.0hz, 1920x1080@60.0hz
  GLX Renderer: GeForce GTX 860M/PCIe/SSE2 GLX Version: 4.5.0 NVIDIA 352.09
Audio:  Card-1: Intel Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller driver: snd_hda_intel
  Card-2: Intel 8 Series/C220 Series Chipset High Definition Audio Controller driver: snd_hda_intel
  Sound: Advanced Linux Sound Architecture ver: k3.13.0-24-generic
Network:  Card-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller driver: r8169
  IF: eth0 state: up speed: 1000 Mbps duplex: full mac: <filter>
  Card-2: Intel Wireless 7260 driver: iwlwifi
  IF: wlan0 state: up mac: <filter>
Drives:  HDD Total Size: 500.1GB (54.7% used) 1: id: /dev/sda model: Samsung_SSD_850 size: 500.1GB
Partition: ID: / size: 459G used: 255G (59%) fs: ext4
RAID:  No RAID devices detected - /proc/mdstat and md_mod kernel raid module present
Sensors:  System Temperatures: cpu: 60.0C mobo: N/A gpu: 56C
  Fan Speeds (in rpm): cpu: N/A
Info:  Processes: 218 Uptime: 4:48 Memory: 3123.0/15492.2MB Client: Shell inxi: 1.8.4

TL;DR: xcomposite seems to have (at least) three related errors:

1. glXCreatePixmap failed: BadAlloc
2. glXCreatePixmap failed: BadMatch
3. device_copy_texture (GL) failed

If you error on "XCompositeNameWindowPixmap failed: BadMatch" your window is probably not active in your window manager and thus not rendering. (minimized or on another workspace or such)

I can repro some of the above, and have gdb (via qtcreator) just need guidance on how to get whatever info people need to quash this.

 

[NateDevv]

I am getting this problem on Debian 8, nvidia drivers as well.

 

[WayZHC]

I had the exact same problem in June. But somehow it just disappeared by itself and works perfectly now. I don't know if Nvidia drivers fixed it (i always update to the latest ones as soon as they release new drivers, 355.06 currently in).

So i have no idea what actually fixed it. If i ever get the same bug and manage to fix it, i'll report back.

 

[admalledd]

--snip--
So i have no idea what actually fixed it. If i ever get the same bug and manage to fix it, i'll report back.

Aww you got my hopes up... Mind providing some more system information? Or are you using the system in your signature at default settings? (aka, because sigs can change : ubuntu 15.04, gnome shell(?), gtx780)

I updated to the 355.06 drivers myself on my laptop, still no window capture. Info remains the same otherwise from my earlier post. Again I can reproduce easily with a debugger, if someone wants some more info let me know what to look at then...

 

[WayZHC]

Yes that's my rig. And it's official Ubuntu Gnome 15.04 so it comes with "ubuntu-gnome-desktop" (3.14) packages.

Now i get the bug again. Still haven't figured out what's the problem but i'm trying. It's seems pretty random. Nothing has changed since my last post.

When i launch obs from commandline, it gives this kind of error. Not sure if related.

error: xcompcap: XCompositeRedirectWindow failed: BadWindow (invalid Window parameter)

 

[admalledd]

The XCompositeRedirectWindow: BadWindow generally means either (1) you don't have compositing available (unlikely) or (2) obs-studio has lost which window it is you are trying to capture, try re-selecting it from the drop-down-list. There are a few other causes but IIRC those are the most common.