Updated Guide with info about OBS 24.0.3 new solution for high GPU encoding. Forget about complex fine tuning. You can now run OBS in priority mode to make sure your stream runs perfectly in any situation with NVENC. Just open OBS in admin mode and that will prioritize the stream over everything, ensuring you get max FPS in game with silky smooth 60 FPS streaming.
Updating the guide after the Windows 10 1903 release. For this new version of Windows, the new recommended settings are as follows:
Windows: Make sure you update to Windows 10 version 1903, and enable Game Mode. This version includes performance enhancements for streaming, as well as an updated Game Mode compatible with streaming.
GPU Utilization for 1440p and 4K streaming: When streaming higher resolutions, or using an asset that is higher resolution, if your GPU utilization is above 95% Windows will start prioritizing the game over everything, making your stream lag. This can also occur if you are streaming 1080p or below but have an asset that is 1440p or above, like a 4K webcam. To solve this, you can either reduce the game or asset resolution to 1080p, or make sure the game is using less than 90% of your GPU resources.
You can check your GPU utilization in the Windows Task Manager > Performance tab. To limit GPU utilization: cap FPS in-game, run the game in Borderless Windowed mode, reduce game graphics or resolution, or turn V-Sync on.
To reduce the resolution of your webcam, double click on the source in OBS and under Resolution select Custom and specify a resolution under 1080p.