By using the custom output (ffmpeg) output, you're basically on your own. There are so many combinations of configuration options, nobody can say if the output will be valid or not. Generally, you work out ffmpeg parameters by consulting the ffmpeg documentation first and after that by trial and error.
In your case, I guess ffmpeg refuses to put h.264-encoded frames or aac-encoded audio frames into a mpegts transport stream, or the bitrate is too high. I once was successfully able to broadcast a obs output with mpegts container format by choosing mpeg2video as video encoder and mp2 as audio encoder.
It helps if you uncheck "Show all codecs", because this will remove all codecs from the drop down list that are incompatible with the container format.