Any Output Capture sources you may have, are the same as the Desktop Audio, and can cause the same problem.
Or, you could not send anything to the Monitor:
Use the Monitor Off setting for everything there...which also means you'll get nothing from OBS in your headphones or speakers.