OBS is always using some of your GPU to render the composited scene output, regardless if the encoder is active or not.
Running a game without vertical sync or a frame rate limiter will frequently cause performance issues with OBS because your GPU will be maxed out. Enable vsync or set a reasonable frame rate limit that your GPU can handle without hitting 100% usage. If that's not enough you may also need to turn down some of the video quality options in the game.
If none of that works, then you need to invest in a more powerful GPU that can keep up with what you're trying to do.
None of this is the cause of the preview stutter. I get it every game, whether Window, Display or Game capture, with or without FPS cap, at 30 or 60 FPS capture, VSync or not, and with every hint, tip or suggestion in place that anyone has ever thrown at this issue in over two years.
It isn't a hardware issue, it happens across every conceivable combination of hardware. It is a fundamental issue and a well-documented one.
One screen, multi-screen, one PC, multi-PC, GSync on or off, VSync on or off, SLI on or off, single card, multi-card, AMD, Intel or nVidia, Game Mode on or off, multiples of 30 Hz, 29.94 Hz, overclock or underclock, high, low, or real-time priority, CPU affinity, DPC latency, Windows 7, 8 or 10, fullscreen optimisations on or off, registry this, drivers that, Bios the other. Innumerable folk have sought to resolve the issue over the years and nobody has. Not one single person across the entire Internet has presented a solution.
I think it is time to formally address it.