Use the software x264 renderer, not the Apple Software renderer. The Apple Software renderer is for Macs with Intel GPUs, and your Mac Pro (same year as mine) does not have one. You have an Nvidia card, but NVENC is not supported on MacOS, so x264 software renderer is your only choice.