Audio Monitoring is for using a separate sound device so you can hear what a source sounds like.
By default though it will output to your default playback device, when using Monitor (Mute Output) the stream will still hear the audio because it is being played back on the default playback device through the Desktop Audio Source.
When using Monitor and Output you end up duplicating the audio because the source itself and Desktop Audio Source are both playing the same output.
When you correctly configure Audio Monitoring to use a sound device that is not your Default, that is when Monitor (Mute Output) will actually mute it for stream. It is important to use Monitor and Output when using a separate sound device unless you intentionally don't want the stream to hear it.
Monitor off just means do nothing, don't listen.
Audio Monitoring is generally not for the streamer, it is for the producer to monitor levels/sources on their own set of headphones. Think big time setups where the host/streamer is not by the PC.
It has limited use in smaller setups, the delay is far too high to be used as a psuedo sidetone to monitor your microphone, it can cause you to stutter, slur or jam your speech entirely