Your first setting is some encoder-specific thing I don't know, because I don't have a AMD card. Your second setting (settings->Advanced->Video->Renderer) deals with the renderer OBS paints its own dialogs and preview with. It doesn't deal with capture.
Regardless any of these settings, OBS is able to capture directx 12 and Vulkan games (like Doom) fine. This support is built in, always available, and there's nothing to configure. You just use a game source for this, and the renderer required to capture is auto-detected.