Settle on one resolution and frame rate and don't bounce around all over the place changing stuff. I reccommend that when you do change settings to exit OBS and restart up again.
- For your hardware, stick to 720@60FPS max.
- Read Twitch guidelines and look at their recommended settings:
https://stream.twitch.tv/encoding/
Set your encoder to x264, rate control to CBR, bitrate to 3500, keyframe to 2, CPU usage to veryfast in the advanced Output tab of OBS.
Dont screw with anything else other than bitrate and CPU usage until you get this to run.
Bring up the View | Stats window and monitor your stream.
if you get encoding, rendering, or network drops, thats another set of problems that need to be dealt with.