When you have a problem you need to include a log file of the problem if possible.
However, I can probably guess that you simply have the bitrate too low for 1080p. If you're recording to file output only, set the bitrate to something like 8000/8000, and then to further increase quality, use the advanced x264 option: crf=18