I'm not knowledgeable, but a screenshot might help. I am on MX Linux 19.4 using 26.1.2, I have to add "scene" (which seems to do nothing other than let me then add sources). I add a source (window capture xcomposite) and then choose the "YouTube - Google Chrome" from the "Window" dropdown.
If you can get a screenshot of your "Properties for WindowCapture(Xcomposite)" screen with the "Window" dropdown exposed, that might be informative. I assume you know the screen I'm talking about (with the four very wide "crop" fields)? Does it even show a Firefox option in the Window dropdown?
One other thing I've noticed: I have two virtual desktops in MX Linux. If the browser is on a different desktop than the one obs is on, it will be in the dropdown list of windows to choose. But, it won't show any window content in the preview. If you have an external monitor, maybe make sure OBS is on the same screen? If you have more than one virtual desktop, maybe change it to be one desktop to remove any possibility that's interfering?
I think I may be experiencing the same issue. For window capture (xcomposite), any browser (Firefox/Chrome) only displays as black. Any other windows, as well as other capture types (screen) seem to work as expected.