The settings seem correct and within acceptable ranges for a 1st Gen Maxwell card. The only thing I can think is that you have something else running that is calling an instance of the NVENC engine (e.g., Shadowplay, or some other recording software), since 1st Gen Maxwell cards can only run one NVENC instance. Just for kicks, have you tried setting the NVENC params to defaults? Is this a constant failure all the time, or does it fail most of the time or only sometimes?
The AMD encoder will not run on your system because you do not have a compatible AMD APU or GPU installed. You might be able to get away with running the x264 encoder in some cases, even with the older i7, unless your games are already using up most of the CPU.
