my first snarky thought was ... it depends (which it does)
but to be more helpful even though your question basically isn't answerable ..
you need to clarify as you request is the equivalent of 'what is the best car' and the answer is 'it depends on your budget and what you are trying to do with it'
For example, most streaming providers, due to screwy H.265 licensing, won't accept H.265 streaming input (FB, YT, etc will convert on their end for more efficient bandwidth usage).
For recording, H.265/HEVC tends to be more compressed but that takes CPU... so it depends on available resources. And more likely what you have/plan to use for video editing. What should drive your recording format is how you are going to edit/use the file (technically) later.
And then the more relevant aspect of what OBS supports (I'll let you look that up as finding that info at the Help -Wiki link above will do you good)
So, realize you are NOT asking a simple question.... it really does depend on technical specifics as to what to recommend.
Your situation is made worse as AMD wrote a terrible driver, which they don't support, for GPU encoding offload (unlike nVidia). There is a BETA OBS 3rd party plug-in trying to work around that... fwiw
so in the mean time, Realistically, you are safer/more likely to need to use CPU for real-time video encoding, and that basically means standard x64. My recommendation - use Simple / autoconfig to start and focus on content and delivery and OBS functions, before trying to get fancy with codecs. OBS is a powerful, flexible free tool that expects the end-user to do a fair amount of self-education to get the most out of it.
Be sure to read the Wiki Getting Started Guide.
I hope this helps.
