The framerate drops seem to have lessened, but the output file looks like garbage. I changed the settings back to record in 4K and it uses 65%(!!!!!!!) of my GPU! Is there going to be anything left for the game? You really shouldn't be using nvenc if you're at 4K and already GPU limited. My CPU only needs to load about 25% for the same task, and it's largely idle during 4K games.
The framerate still fluctuates, just not as bad as with software encoding. It perhaps goes up/down by 1 fps as opposed to 20. Sync seems to be broken between OBS and win 11. Perhaps using gpu encoding uses a different method that does not encounter the same bug. I had an older win 7 computer record the same game, at the same quality, at 4k, WITH SOFTWARE ENCODING, with no framerate issues.
I don't consider using 65% of my GPU for smooth-ish encoding a solution. That's not "barely noticeable."
The framerate can be perfectly smooth, a rock solid hard 60 fps. And then it just... stops being smooth. It's like sync just breaks and one or more frames per second miss the vsync timing. And then it just... fixes itself? It's like the computer decided to do an update in the middle of recording, except that it didn't. CPU and GPU usage is flat.
In game Factoio is attributing frame time delay to waiting on vsync. (Factorio has a very detailed frame time update chart that tells you exactly how many microseconds each element of the game took.) Normally something called "Flip" takes up most of the time. This is the effect of vsync, and it's the GPU waiting for the next frame to start. When playing without OBS, normally it's about 14ms, which makes sense because the frame normally does not take up much GPU time, which is getting close to the 16.66ms a frame has in it. This is Factorio after all.
But you open OBS and now Flip is around 10ms, meaning a good 4ms is being taken by OBS, without even recording at all. Just having it open is consuming a 25% of my GPU's power. That seems... wrong. This is a 3080ti. But then vsync seems to break mid recording. Where you go back to look at Flip and it's gone from ~10ms to around 0.250ms. According to Factorio this implies the GPU is getting maxed out. (basically no waiting had to occur between frames) If you alt-tab away from the game to do something such as write this post, and then tab back, Flip returns to it's pre-tanked value. You can watch it go back in real time. But the GPU wasn't being maxed out. I have a second monitor where I can monitor these things and the load has remained constant.
It's like OBS is hijacking sync and disabling it (or win 11 doing the same,) which causes it to start missing sync timings, causing the FPS to drop suddenly. This is classic vsync. If you're at 60 fps and you miss a frame, for that moment, now you're at 30 fps. If it happens several times a second you'll get an extremely rough number somewhere between 60 and 30. The critical thing here is that everything is fine so long as OBS isn't open.