Code:
[x264 encoder: 'streaming_h264'] settings:
18:11:51.796: rate_control: CBR
18:11:51.796: bitrate: 3500
18:11:51.796: buffer size: 3500
18:11:51.796: crf: 0
18:11:51.796: fps_num: 60
18:11:51.796: fps_den: 1
18:11:51.796: width: 2560
18:11:51.796: height: 1440
18:11:51.796: keyint: 120
Wait, are you trying to stream 2560x1440 and 1920x1080 at 60 FPS? That's not possible with i5-6400 & GTX 960, even 1080p60 is extremely intensive on a high-end hardware. If you are trying to record and stream at the same time, that's even more difficult.
Set the output (scaled) resolution to something like 720p and lower the CPU preset (choose lighter one, like Faster / Veryfast / Superfast).
You can record fine with the GTX 960, using the NVENC Encoder and acceptable settings (something like 1080p60 / 50,000 kbps and even higher).
(Tip: Don't rescale / downscale the resolution in the "Output" tab (next to the encoder settings). Set the output resolution in the "Video" settings tab and uncheck the "Rescale Output" box anywhere else)
Your GPU isn't bad, it's actually quite decent for recording and streaming. The CPU isn't very powerful though and you wouldn't be able to stream with very high quality, for example.
Make sure to keep your drivers up to date (and most importantly the GPU drivers). You can experiment with all kind of combinations and settings. Trial and error, experimenting and testeing is essential part of finding the 'best' settings for someone's needs.
