first of all you need vsync always enabled in game, no exceptions - otherwise stuttering and frame skipping
now 144hz monitor - for smooth recording its best to play in 60hz so you have source material (game image) already in same frequency as desired recording, so set your desktop refresh to 60hz, obs to 60 in video tab and problem solved. yes you will sacrifice 144hz but playing in 144 and recording 60hz is problematic and rather never smooth as it should be. its a matter of priorities. if its freesync monitor you may need to disable it also
encoder - try VCE in simple output mode, but unfortunately its not best solution. im not familiar with it but its far from perfect judging from what ive seen on the forums. you will be much better with Quicksync in your cpu - also simple output mode, high or indistinguishable quality. bitrate will be auto managed by ICQ rate control (its ICQ=23 for high and 15 or 16 for indistinguishable )
https://obsproject.com/forum/resources/how-to-use-quicksync.82/
https://obsproject.com/forum/resources/custom-parameters-of-quicksync.104/
there is also amd vce fork of old OBS maintained by awesome guy
@jackun who dedicated his life to it if you really want to try VCE
https://obsproject.com/forum/threads/obs-branch-with-amd-vce-support.13996/ but old OBS lacks many features of studio and is not developed anymore besides jakun's work on vce afaik