OBS has costs on the CPU and GPU. Ain't no such thing as a free lunch and all that.
Post an OBS log file from an at-least five minute encode of high action content. Maybe you can adjust some settings to make it use less resources, leaving more available for your game.
I tried reducing everything to lowest quality and still did not improve. Here is Dota 2 streaming at 720p:
https://www.twitch.tv/xiigage/v/64460666
I chose dota 2 because it is not too graphic intensive and you can see my FPS in the corner. I am getting around 100 fps in game, but the video is terribly choppy. On my hardware I should not be getting this, so something is amiss.
I added an OBS log and some pictures of my settings.
Sorry, that log is from the other day, here is a piece of the most recent stream (around 2 hours, playing overwatch and streaming at 480p):
20:34:27: =====Stream Start: 2016-05-04, 20:34:27===============================================
20:34:27: Multithreaded optimizations: On
20:34:27: Base resolution: 2560x1440
20:34:27: Output resolution: 852x480
20:34:27: ------------------------------------------
20:34:27: Loading up D3D10 on NVIDIA GeForce GTX 980 Ti (Adapter 1)...
20:34:28: ------------------------------------------
20:34:28: Audio Format: 48000 Hz
20:34:28: ------------------------------------------
20:34:28: Audio Channels: 2 Ch
20:34:28: Playback device Default
20:34:28: ------------------------------------------
20:34:28: Using desktop audio input: Headphones (2- High Definition Audio Device)
20:34:28: Global Audio time adjust: 0
20:34:28: ------------------------------------------
20:34:28: Using auxilary audio input: Microphone (2- High Definition Audio Device)
20:34:28: Mic time offset: 0
20:34:28: ------------------------------------------
20:34:28: Audio Encoding: AAC
20:34:28: bitrate: 128
20:34:28: Using graphics capture
20:34:28: Trying to hook process: Overwatch.exe
20:34:28: Scene buffering time set to 400
20:34:28: ------------------------------------------
20:34:28: Video Encoding: x264
20:34:28: fps: 30
20:34:28: width: 852, height: 480
20:34:28: preset: faster
20:34:28: profile: main
20:34:28: keyint: 60
20:34:28: CBR: yes
20:34:28: CFR: yes
20:34:28: max bitrate: 3500
20:34:28: buffer size: 3500
20:34:28: ------------------------------------------
20:34:28: SharedTexCapture hooked
20:34:29: Using RTMP service: Twitch
20:34:29: Server selection: rtmp://live-dfw.twitch.tv/app
20:34:29: Interface: Intel(R) Ethernet Connection I217-V (ethernet, 100 mbps)
20:34:30: Completed handshake with rtmp://live-dfw.twitch.tv/app in 156 ms.
20:34:30: SO_SNDBUF was at 65536
21:33:10: RTMPPublisher::SocketLoop: Stalled for 2268 ms to write 463649 bytes (buffer: 0 / 463872), unstable connection?
21:33:11: RTMPPublisher::SocketLoop: Stalled for 1660 ms to write 462541 bytes (buffer: 0 / 463872), unstable connection?
22:57:54: RTMPPublisher::SocketLoop: Stalled for 1968 ms to write 461332 bytes (buffer: 0 / 463872), unstable connection?
22:58:00: RTMPPublisher::SocketLoop: Stalled for 1761 ms to write 463739 bytes (buffer: 0 / 463872), unstable connection?
22:58:02: RTMPPublisher::SocketLoop: Stalled for 2306 ms to write 461415 bytes (buffer: 0 / 463872), unstable connection?
22:58:04: RTMPPublisher::SocketLoop: Stalled for 1597 ms to write 462648 bytes (buffer: 0 / 463872), unstable connection?
22:58:11: RTMPPublisher::SocketLoop: Stalled for 3302 ms to write 460936 bytes (buffer: 0 / 463872), unstable connection?
22:58:13: RTMPPublisher::SocketLoop: Stalled for 1643 ms to write 463857 bytes (buffer: 0 / 463872), unstable connection?
22:58:32: RTMPPublisher::SocketLoop: Stalled for 2636 ms to write 463341 bytes (buffer: 0 / 463872), unstable connection?
22:58:41: RTMPPublisher::SocketLoop: Stalled for 1644 ms to write 461118 bytes (buffer: 0 / 463872), unstable connection?
22:59:10: RTMPPublisher::SocketLoop: Stalled for 1738 ms to write 461997 bytes (buffer: 0 / 463872), unstable connection?
23:09:21: Total frames encoded: 278794, total frames duplicated: 55065 (19.75%)
23:09:21: Total frames rendered: 235010, number of late frames: 23295 (9.91%) (it's okay for some frames to be late)
23:09:21:
23:09:21: Profiler time results:
23:09:21:
23:09:21: ==============================================================
23:09:21: video thread frame - [100%] [avg time: 15.985 ms] [children: 92.7%] [unaccounted: 7.28%]
23:09:21: | scene->Preprocess - [0.00626%] [avg time: 0.001 ms]
23:09:21: | GPU download and conversion - [92.7%] [avg time: 14.821 ms] [children: 1.13%] [unaccounted: 91.6%]
23:09:21: | | flush - [1.06%] [avg time: 0.169 ms]
23:09:21: | | CopyResource - [0.05%] [avg time: 0.008 ms]
23:09:21: | | conversion to 4:2:0 - [0.025%] [avg time: 0.004 ms]
23:09:21: Convert444Threads - [100%] [avg time: 0.228 ms] [children: 97.8%] [unaccounted: 2.19%]
23:09:21: | Convert444toNV12 - [97.8%] [avg time: 0.223 ms]
23:09:21: encoder thread frame - [100%] [avg time: 0.521 ms] [children: 6.91%] [unaccounted: 93.1%]
23:09:21: | sending stuff out - [6.91%] [avg time: 0.036 ms]
23:09:21: ==============================================================
23:09:21:
23:09:21:
23:09:21: Profiler CPU results:
23:09:21:
23:09:21: ==============================================================
23:09:21: video thread frame - [cpu time: avg 0.225 ms, total 53015.6 ms] [avg calls per frame: 1]
23:09:21: | scene->Preprocess - [cpu time: avg 0 ms, total 156.25 ms] [avg calls per frame: 1]
23:09:21: | GPU download and conversion - [cpu time: avg 0.079 ms, total 18609.4 ms] [avg calls per frame: 1]
23:09:21: | | flush - [cpu time: avg 0.053 ms, total 12656.3 ms] [avg calls per frame: 1]
23:09:21: | | CopyResource - [cpu time: avg 0.007 ms, total 1875 ms] [avg calls per frame: 1]
23:09:21: | | conversion to 4:2:0 - [cpu time: avg 0.004 ms, total 1015.63 ms] [avg calls per frame: 1]
23:09:21: Convert444Threads - [cpu time: avg 0.27 ms, total 127094 ms] [avg calls per frame: 2]
23:09:21: | Convert444toNV12 - [cpu time: avg 0.268 ms, total 125969 ms] [avg calls per frame: 2]
23:09:21: encoder thread frame - [cpu time: avg 0.16 ms, total 44765.6 ms] [avg calls per frame: 1]
23:09:21: | sending stuff out - [cpu time: avg 0.029 ms, total 8359.38 ms] [avg calls per frame: 1]
23:09:21: ==============================================================
23:09:21:
23:09:21: =====Stream End: 2016-05-04, 23:09:21=================================================