Hopefully this isn't archived yet, but I found something interesting about the current version of OBS. I use a GTX 1060 3GB to record with and is using Windows 10 20H2 Pro for Workstations and the latest video driver. The stuttering occurs either immediately or about 5 minutes into a recording.
I took on a different best practice with the NVIDIA cards as a cluster of features are now added. As a best practice, I recommend the following:
-Turn off Threaded Optimizations (Globally)
-Turn off CUDA (Globally)
-Turn off Low Latency Mode (Globally)
-If you use G-Sync, change the application and requested states to Fixed Refresh Rate (Globally)
Select the OBS profile (if not, then add it) and then turn on the above features. You can't even record without CUDA on OBS, for example.
-Use the Nvidia Profile Inspector (google it or look for Orbmu2K's github for the latest) and find the OBS Studio profile. In the section called "Other", you will see CUDA P2 state. NVIDIA added this back in the day to put some power saving on the memory clocks to CUDA. You know that power saving is a must these days, so there is nothing wrong with that. However, OBS uses CUDA and this is one of the main reasons the preview becomes choppy, the P2 state will kick in.
Set the CUDA P2 state to Off (notice the list of applications that is recommended for the Off setting. OBS is one of them as well as some games).
This is my best practice for OBS. Without doing any of these, every application (Globally) will use CUDA, Low Latency, G-Sync, and Threaded Optimizations when they were never designed to. Unless the devs say so specifically, then don't use CUDA globally.