Is it just a glitch? Does it fade down and back up? Drop out altogether?
This might not necessarily be your problem, but I try to have all of my audio as global sources, and keep that number to a minimum too. (Settings -> Audio) I really don't like audio sources in scenes if I can avoid it, because their transitions are always a hard cut. A global source stays regardless, which then allows me to use the
Advanced Scene Switcher to automate a fade, independently of the scene transition or triggered by the scene transition, so that it actually does fade instead of just cutting on/off.
If I need more than what will fit in OBS's global audio sources, then that's a good indication to me that I need to move it all into a DAW and have Adv. SS control the DAW somehow. OBS then, only receives the finished soundtrack as a single global source.