Use the 64bit version instead of the 32bit version
Set your keyframe interval to 2
That framerate drop is going to happen because live video encoding with high compression takes a lot of processing power. (This is needed to stream at a lower bitrate without it looking like garbage, which is also why when recording you don't have to have great compression as long as the bitrate is high then it will look fine.) And with such a beefy GPU you're game is CPU limited, when you start streaming it uses a lot of your CPU taking away from the game.
But still 100 frames in a game is much more than playable. But changing the CPU preset to veryfast would get you a lot of frames back. Also capping your in game framerate helps a lot because it will leave extra CPU power to be used as needed.
Most people don't get this big of a frame hit since they are GPU limited, and their processor already has extra power that its not using.