It doesn't take much to exceed OBS's audio capabilities, and you're right on the edge. There probably *is* a way to make OBS do what you want, but I think it'd be easier to manage if you switched to running your audio *outside* of OBS, and only gave it what you want to record.
You might start with one of these:
VoiceMeeter Banana, the Advanced Virtual Audio Mixer by V.Burel
vb-audio.com
VoiceMeeter Potato, the Ultimate Virtual Audio Mixer for Windows
vb-audio.com
Pick the one that fits what you're doing and likely will do in the future.
In addition to the physical connections, each of those also installs some virtual devices that behave the same way to any application, but they're connected to the virtual audio mixer instead. Now:
- Your mic connects to a physical input of the virtual mixer, NOT OBS!
- Your game connects to a virtual input of the mixer (a virtual speaker), NOT OBS!
- Your music connects to a different virtual input (a different virtual speaker), NOT OBS!
- Your headphones connect to a physical output of the virtual mixer, NOT OBS!
- OBS connects to a virtual output of the mixer (a virtual mic), AND THAT'S ALL!
- You might use two virtual outputs (virtual mics), just to have different processing in OBS that Voicemeeter doesn't do, but that's as far as you go with audio in OBS.
Use the routing buttons in the virtual audio mixer, to send each thing to where it needs to go, and not to where it shouldn't go.