Nah all that red speaker does it output audio form the capture to the desktop, muting it doesn't mute the audio from the actual capture device it just stop the audio being sent the desktop speakers.
This is your exact problem. GraphEdit outputs the audio to Desktop, which you have muted.
That speaker being muted, mutes all desktop sound. If you go into OBS, and set your desktop audio to disabled. Check and see that the capture card audio will still show a green bar if not using GraphEdit.
The capture card audio can only be used by either:
A) OBS by itself
or
B) GraphEdit by itself (by streaming the audio to desktop).
Not both at the same time, that is just how these directshow devices work.
So as soon as you tell GraphEdit to "Open the Audio Stream" to your Xi-Fi Sound Card, OBS can no longer use the stream itself.
This is 100% what your issue is, so this workaround may not be for you from what it sounds like.
Your only other option would be to use something like Virtual Audio Cable, have GraphEdit output the capture card audio to Line 1, then Windows Sound, use the "Listen to Device" option to your Xi-Fi Sound on Line 1, then set OBS to use Line 1 as desktop audio. That is the only way this will work and not include skype message sounds, etc. Otherwise, GraphEdit will not be able to do what you want it to do.