I am unable to reproduce your problem. Can you tell me which version of OBS you are using along with the setting you use? Then I can hopefully reproduce the problem and find a fix.for some reason it isn't.
It may possibly be that if you are using the "Force Mute Other Audio" option and switch on the mic, for example, and then go to move back by unmuting the desktop. Doing this won't switch back as you need to use the same output as you switch with.