The audio monitoring device is used to select the audio output device that you can optionally send your audio input sources to.
So, for instance, if you have a USB microphone connected that you're using to record commentary, you can select headphones as the audio monitoring device, and then in the mixer, choose "monitor and output" for your microphone input.
Be warned that some Mac users (myself included) report this functionality as broken in the Mac version; for me, that setting displays only audio *inputs* rather than outputs, and as such cannot be used.
For info on recording desktop audio see this stickied post:
https://obsproject.com/forum/threads/how-to-capture-desktop-audio-on-mac.16491/