Cannot get advanced outputs to work

[Splashpoint]

Hi all,
I have OBS configured to record a 4K input via a Blackmagic Decklink card. I am currently trying to record 5.1 to AC3, which per the OBS instructions requires you to use "Custom Output (FFMpeg)" in Settings>Output>Recording>Type. However no matter what encoder I select (have tried primarily to use hevc_nvenc (hevc_amf) as suggested due to HDR and my NVIDIA card, the output is a stuttering mess.

Alternatively whenever I simply select "Standard" as the recording type in Settings>Output>Recording>Type it works perfectly and I get an extremely high quality, smooth HDR output, albeit one without AC3 sound since that requires the Custom Output for some reason.

Here is my log: https://obsproject.com/logs/duMxok8xyEnuQet9

From taking a look at that it seems like a significant number of frames are being dropped due to "encoding lag", which explains the stuttering output file I'm getting. I wonder if it's a problem with FFMpeg? NVIDIA? OBS settings?

Here are my "standard" settings (which work great except no AC3 audio since that requires "Custom Output"):

Here are the settings when I try to configure "Custom Output":

Video/color settings:

Any help is appreciated.

Thanks!

 

[Splashpoint]

One thing to note, when I tried NV12 as the color format using the above "Custom Output" settings (in order to get AC3 audio) I did get a playable video, but the colors/HDR were completely burned/washed out.