All three are just video package formats. They have the same videostream data in them. FLV can only support a single audio track, MKV and MP4 can support multiple. Other than that, they're the same quality. Remuxing just re-wraps them into another container.
The main problem with recording direct to MP4 is that if there is ANY issue during recording, even a minor hiccup, your entire recording will be rendered corrupted, and 100% un-recoverable by any means. A heap of digital garbage, only good for deleting. MP4 keeps its file data at the END of the file instead of the beginning. If that data isn't written out successfully, there's no way to play it back. MKV and FLV write that info out first, at the start of the file, so if there's a problem while recording, you'll lose at most a half-second or so.
The secondary problem with direct-recorded MP4s is that many major video editors (Premiere (Pro), Vegas, Resolve) can have issues importing, playing, or generally working with the MP4 files OBS puts out directly including crashing the editor in some extreme cases. Remuxed files do not have this issue.
So yeah.
There's no drawbacks to recording to MKV, while recording to MP4 has a bunch of risk of total loss of potentially un-repeatable moments.