I tried shadowplay and it won't be stuck, but the quality of shadowplay video is simply too bad. The quality at the same bit rate is much worse than the NVENC of OBS.
I used to play the same game with the same screen settings. OBS never stuck. Recently I updated win10 1809, updated the graphics driver, and slightly overclocked the CPU. OBS started to hang when the GPU was high. Later I went back. Turning the CPU frequency back to the default, there is still this problem.
The graphics card driver uses DDU to uninstall and reinstall multiple different versions, which is still the case.
But I remember that after just updating 1809, I recorded some game videos. OBS should not be stuck yet. After that, I updated the graphics driver and CPU overclocking. (But after turning these settings back, OBS is still stuck (╯‵ □ ') ╯ (┻━┻)
The resolution can not be reduced, the performance of the graphics card is not bad, the game quality has been adjusted to a very low level, the result is to increase the frame rate, GPU occupancy is still high.
I contacted NVIDIA SUPPORT in the past few days to ask how to limit the GPU usage of the game. They think this is a problem with OBS. It seems that NVIDIA SUPPORT does not know how to limit the GPU usage. They suggested that I contact the OBS SUPPORT feedback bug.
OBS is used so much, so easy to use, I have used it for so long, I can't believe that because the game uses GPU, it is normal for stuck. After all, many people use it to record games, I have never encountered it before, and Google No similar cases can be found.
There is no other solution than limiting GPU utilization. OBS can't encode frames it can't render. If it can't render the current frame, you get the last frame until a new frame is rendered.
If you can't decrease framerate, you can reduce resolution or other fidelity options to attempt to preserve GPU resources for OBS.
Limit the fps in the game to not cause such high gpu usage.