Setting threads below the default is totally legit, I do it in Handbrake, and I do it in OBS, and it works to make sure some of the CPU is available for other things. You're not "depriving" OBS of anything, x264 simply uses what it is given and what it can use. In my actual testing, 12 threads is more than it can use. And actually (slightly) hurts performance on my i7 2600k that I use for encoding (also 4 core + hyperthreading as the OP's CPU). I generally set threads to between 8 and 10 and get fewer lagged frames than when I have it set to 11 or left at default of 12. The 1.5x rule wasn't intended for CPUs with so many cores and virtual cores, its a legacy setting from the days before so many virtual cores were common.
My 2600k is a stand-alone capture box, with a video capture card, that doesn't have to do anything else. I use the medium preset. It
is much improved quality over "very fast". It
is not the same as going from very slow to placebo as someone earlier suggested, it is far more worthwhile than that. There are important X264 features turned on in between there, but they are more CPU-costly.
That said, OP, you're going too far away from the presets before you've even figured out if your CPU can handle the presets available. You also have redundant settings in there. Just get rid of all of them and start with just a preset and your animation tune. Start with just a preset and once you verify it works, go one further. You want to find out which is the best preset your computer can handle, until you go one too far, then drop back to the one previous. Only then add things, and then add them one at a time.
And don't add things that are in the preset you're using by default, that just confuses the issue and makes it look like you don't know anything about the presets or settings you're using when you ask for help.
Since you're also playing a game, don't increase the process priority of OBS. Leave it at normal.
Here's another idea:
Do you need to live stream? In other words, is it bad if you stream delayed? If you want the absolute best quality, capture with a relatively easy setting at a high bitrate and full resolution and fps, but don't stream it, capture it to disk. Then when you're not playing, take Handbrake, and re-encode it at a constant bitrate just below what your connection can handle. You can safely go to town and use placebo if you want. Then use the
video source plugin to stream the file you create in Handbrake. You can't have a higher quality stream than that, it just won't be anything you're doing live.