Your GPU is maxed out and OBS can't render scenes fast enough. Running a game without vertical sync or a frame rate limiter will frequently cause performance issues with OBS because your GPU will be maxed out. OBS requires a little GPU to render your scene.

Enable Vsync or set a reasonable frame rate limit that your GPU can handle without hitting 100% usage.

If that's not enough you may also need to turn down some of the video quality options in the game. If you are experiencing issues in general while using OBS, your GPU may be overloaded for the settings you are trying to use.

Please check our guide for ideas why this may be happening, and steps you can take to correct it: GPU Overload Issues.

Encoder overload may be related to your CPU or GPU being overloaded, depending on the encoder in question. If you are using a software encoder (x264) please see the CPU Overload Guide. If you are using a hardware encoder (AMF, QSV/Quicksync, NVENC) please see the GPU Overload Guide.