lowering the GOP should not make it worse. did you try balanced preset? also play with host/dx11 thing (cant recall what its called :P)
All Tests done at 1920x1080 downscaled (Lancos) to 1280x720 at 60 fps. Nothing but OBS, Chrome and Steam running (in user space). Add about 500-600kb/s to still as there are full frames every now and then.
GOP, Preset: kbit(still) - kbit(moving)
Engine Type Host: (GOP 120+ causes High Encoding with 9% CPU usage)
030, Speed: 422kb/s - 6332kb/s
060, Speed: 411kb/s - 3028kb/s
120, Speed: 433kb/s - 2728kb/s
240, Speed: 436kb/s - 3052kb/s (BSoD: Hell yeah)
030, Balanced: 443kb/s - 6542kb/s
060, Balanced: 458kb/s - 4928kb/s
120, Balanced: 474kb/s - 4080kb/s
240, Balanced: 458kb/s - 3407kb/s
030, Quality: 551kb/s - 7269kb/s
060, Quality: 499kb/s - 4729kb/s
120, Quality: 573kb/s - 4066kb/s
240, Quality: 551kb/s - 3559kb/s
Engine Type DX11: (GOP 120+ causes High Encoding with 9% CPU usage)
030, Speed: 439kb/s - 5970kb/s
060, Speed: 445kb/s - 4898kb/s
120, Speed: 292kb/s - 4403kb/s
240, Speed: 315kb/s - 3203kb/s
030, Balanced: 289kb/s -
Edit: DX11 seems to really help and GOP 240 or higher too.
Perhaps the solution is not GOP but some other variable? It seems to try to keep some sort of arbitrary quality, though it's not clearly visible if it's CRF or CQP. I couldn't see a difference in the saved video files except on moving scenes.
Edit: Got a crashlog while testing!
Edit 2: GOP above 200 cause CPU spikes, GOP above 300 cause mini-freezing, GOP above 400 can crash driver, GOP above 500 just BSoDs
Edit 3: I managed to have it hit just below what x264 uses in bitrate: GOP 240, Quality Preset, VBR with peak set. CBR still doesn't work.