Please post a logfile from a recording attempt. There is a pinned thread at the top of the forum with instructions on how.
Reasons this could be happening:
1) You have NVENC selected, but your GPU does not support it
2) Your GPU drivers are out of date
We'd be able to tell more from a logfile from a recording attempt where the error popped up.