Where OBS isn't offering any explicit option, it's using the ffmpeg internal default. In my opinion, you're wasting your time. All those encoders and their settings are years old, are present in ffmpeg for years, and people had years of time to tune the all the parameters. The current defaults are the result from years of experience. They would not be the default, if they were not the best settings. So use the defaults. So don't use ffmpeg custom output but instead advanced output mode and use the amd encoder given there. It's the same encoder as in ffmpeg custom output with the only difference that those parameters are exposed that one might need to adjust, if necessary.
If you want to learn how parameter changes work out for a real video, don't use OBS for testing. Instead, create (with OBS) a lossless recording of some example, then use ffmpeg.exe directly to use the lossless example as source and try and vary parameters and recode the example to some compressed video. This way you have exactly the same video to compare for all your tests. You will learn that video will generally look worse if you change parameters from the default (except of course for bitrate, preset and quality parameters, which are of course exposed by OBS in advanced output mode).
There is no hidden setting that will magically double the quality. If such setting exists, it would be made default.