Your CPU/GPU should be able to handle reasonable workloads just fine
However, it is easy to configure OBS with unreasonable settings and drive a US$5->10K to overload. So, if you stick with OBS' default/recommended settings, that is a good place to start.
Are you by chance running a streamelements plugin? if yes, I'd start with ripping that PoS out (requires re-installing OBS if I recall correctly) and trying/testing again. Beware settings advice you may come across, as I've seen plenty that fail to provide context (and are therefore inappropriate in many/most cases), or simply outright stupid/wrong. So, if you tweaked settings, without knowing exactly why (and confirmed reasonableness of the suggestion), then I'd be wary. Also, beware gamer-oriented OBS plugins, that are often written with similar code quality as that of many games. Finally, there are some wonderful OBS plugins, as I mentioned earlier, that consume a LOT of CPU cycles, and can overload a PC if you aren't careful.
And even with a powerful PC, if you have lots of auto-start apps, or resource hogging background processes, even a powerful computer can be overwhelmed. Some OS optimization rarely hurts (if done properly). And then you have the possibility of a hardware issue (either an outright fault, or an overclocking/instability issue, which triggers under heavy load (like game play + real-time video encoding)
Are you using NVENC for video encoding? that should reduce load on the CPU (and using a separate circuit, won't impact game-play)
Though you may need to disable Psychovisual Tuning and Lookahead (Advanced video settings).. it depends (sorry not my area.. something to test)
Jun 9 2021
NVENC. Tools > Auto-Configuration Wizard
Look-ahead allows the encoder to dynamically select the number of B-Frames, between 0 and the number of B-Frames you specify. B-frames are great because they increase image quality, but they consume a lot of your available bitrate (you should use CQP for recording so bitrate is not an issue), so they reduce quality on high motion content. Look-ahead enables the best of both worlds. This feature is CUDA accelerated; toggle this off if your GPU utilization is high to ensure a smooth recording.
Psycho Visual Tuning enables the Rate Distortion Optimization in the encoder, which greatly optimizes the way you use bitrate (not sure exactly how this works with CQP, possibly same quality at lower bitrate), improving image quality on movement. This feature is also CUDA accelerated.
