You're a bit fast with your conclusions.
You're using nvenc as encoder. You tried to create a video file with an output resolution of 5760x1080. This didn't work, because nvenc supports resolutions only up to 4096. This is no "bug" in OBS, this is simply a technical limitation of nvenc.
To work around this limitation, you can either reduce the output resolution (which you already found out). This means of course you don't get the native resolution of your 3 monitors in your video file.
Or you can keep the output resolution and use a different encoder, for example x264 (software). In case you use a software encoder, you need to make sure your CPU has enough computing power to support your high resolution.
In either case, I recommend you review your decision to create a 48:9 aspect ratio video. Such a video can only be viewed on a similar monitor configuration, which is absolutely uncommon. Everybody with just 1 monitor will get enormous black bars at the top and bottom - 2/3 of the monitor will be black bars.