With your setup for streaming:
x264 = Quality
NVENC = Performance
With your i7 4790k you can try streaming at 720p with x264 with download scale filter set to Lanczos and CPU preset for streaming set to very fast, using rate control of CBR with a bitrate of 2500 or 3500 as your internet connection allows; keyframe interval 2. Ideally at 30 fps you'll want a minimum bitrate of 2765 for a quality picture.
If you are having a massive performance hit (since you're playing on the same PC) switch to NVENC and use the default settings, with the keyframe and bitrates mentioned.
For recording at 1080p, with your setup, I would go with NVENC using rate control of VBR or CBR with a bitrate of 60,000; keyframe interval 2; preset default; and profile high.