Just general observations
- AMD GPU encoding offload H.264 for streaming is well-reported to be pretty bad. unfortunately. Do a search in this forum on AMD GPU and encoding settings to see if there is anything to help you
- Your first log does NOT include a required recording or streaming session, so important details not present in that log, but your 2nd log has the details [someone knowledgeable on AMD encoder settings would have to comment, as not something I know about]. Importantly, that 2nd log does NOT show any dropped rendering or encoding frames. Be aware that the OBS log does NOT provide details on what all else is going on with your system.
Further, recognize that if you use OBS in Studio Mode you are forcing the OBS PC to do 2X the screen rendering. And you cause the PC to do more work, when you re-scale video (base canvas to output resolution) though then less data to process... so ymmv
And the OBS monitor would have the lowest priority, so even if it suffers, how is the resulting recording/stream (that which actually counts)?
If the issue is lower game performance, then in general you are dealing with the problem of a running a demanding workload (the game), possibly a bunch of other background stuff, and then adding on top of that real-time video encoding which is VERY computationally demanding. There are lots of bad advice out there on optimizing your Operating System, so beware, but there are important changes to default settings which can help. And of course being aware of what various background tasks are doing, and whether they may interfere.
I recommend monitoring hardware resource (CPU, GPU, RAM, Disk I/O, etc) utilization [for ex. using Task manager’s Performance tab and/or Resource Monitor] to see if your system is being maxed out with your settings. Recognize that such monitoring is very high level and doesn't get into detal on the CPU, nor PCIe bus data transfer, and potential bottlenecks in that realm.. but is a good starting point nonetheless (vs doing the equivalent of driving blind-folded). These articles may have some tips that apply to you
https://obsproject.com/wiki/General-Performance-and-Encoding-Issues and
https://obsproject.com/wiki/GPU-overload-issues