Some news at least for my kind of problem.
First of all, we can agree, that there are many different kinds of problems or sources of problems in this thread.
Most common: rendering lag that can be seen in the log file
At least this can be fixed 99% of the time by making sure that the GPU always has some power left for the scene rendering of OBS. So simply use a frame limit for the game and adjust in-game details accordingly, until the GPU load always stays below ~95%.
Then there are Windows 10 Game DVR and other recording tools like Shadowplay that should be disabled.
Next on the list is the mixed refreshrate multi-monitor setup, that windows often struggles with.
In most cases, you can only avoid it by not having any GPU accelerated programs opened on the second monitor (OBS with preview enabled, webbrowser with stream/video playing...).
Then there are also a potential PCIe bandwidth problem with rigs that use more than one graphics card and a CPU+Mainboard combination that will reduce the x16 PCIe slots to only x8, when a second GPU is installed. In these cases, it often works better with just one GPU and full x16 PCIe speed.
And of course, some games are simply behaving very odd, like Destiny 2.
It can only be captured (or at least when I played it) when played in window or borderless mode and game_capture is not working, so you need to use window_capture or screen_capture, which are way worse (performance impact) then game_capture. I think you also had to disable windows fullscreen optimizations for the .exe file.
In my case none of theses common causes are the case.
When I record/stream for example Monster Hunter World or Rocket League, I will see some stutter in the OBS preview, that will also be in the recording/stream. This stutter is not very pronounced..it looks more like some microstutter or frame timing problem. This stutter also disappears and comes back. Some times it takes more than 15minutes before it appears for one minute and then disappears again.
By the way, it can also be spottet, when I record/stream with only 30fps, but it's way harder to spot, as the whole video is not as fluid as with 60fps. So for testing, I went with 60fps, as the stutter can be spottet directly in the live preview of OBS and in the recording as well.
I monitor my GPU/CPU load and can definitely say, that neither CPU nor GPU were even hitting over 60% in Rocket League, when the micro stutter appears (it doesn't seem to be linked to a specific high or low load). It also appears when using NVENC.
Of course my OBS log file did not show rendering lag or encoding problem, otherwise I would have known, where I could try to solve the problem.
Some times in local recording it's hard to tell, if the stutter is really there, or if it's just VLC (a few times I noticed, that the stutter was not there, after I jumped back 3sec in the video and played it again, so in theses rare moments, it was just the Playback software).
Things that I already did to get rid of this problem:
- I uninstalled or closed Tools like MSI Afterburner+RTSS or Desktop Gadgets like CPU and GPU Monitor
- did try different fps limit methods (ingame, driver, RTSS and so on)
- ran the whole system without any Overclocking
- used the "Custom Resolution Utility" to make sure that both monitors are running at true 60Hz instead of 59.94Hz which Windows shows as 60Hz)
- created a new/empty OBS scene collection with just a game_capture source in it
- did a completely new OBS installation with empty profile
- used a low resolution base canvas + output canvas and the bilinear scaling instead of lanczos
- last year I even did a completely new/fresh Windows 10 installation to verify
During the whole process of trial and error, I also changed the game_capture options....at least I did try it with and without the "anti cheat compatibility" and with limiting fps during recording feature.
Nothing helped.
Today I tried again with some of those game_capture options (not really sure which ones I already tried last year).
Per default (at least in my profile) capture mouse cursor and the anti cheat compatibility hook were active.
Interestingly I got both live preview and recording completely smooth, after additionally enabling the allow transperancy option and the record overlays from 3rd party (Steam etc.) options.
At first I was sure, that this was just another coincidence and that the stutter will come back, after I play a little longer or close and start OBS again.
So I did test the whole thing again for two more times and the stutter did not come back. Maybe it will, but as I was able to reproduce the problem pretty fast and reliable in Monster Hunter World before, I am slightly positive, that this might be the fix for me.
Of course, if MHW will hit 98-100% GPU load (which is does some times during hunts), there will be a little stutter here and there, but this can be expected. In the base / camp however my GPU load stays between 65-85% GPU load and I was able to spot stutter there almost instantly after starting the game or at least after coming back from a hunt. This has not happened again, since I changed those two options in the game capture properties.
I'm not sure which one of those two additonal options has helped me (maybe it was "allow transparency", or enabling overlay recording), but it seems to work now.
If some one here has comparable circumstances on their stutter problems, give it a try.
edit: nevermind, after 30min of playing Rocket League, Stutter Session came back, still no dropped or skipped frames :-(