Sorry, I'm way too much a newbie to have good advice/specific regarding encoding options.
Though in general (no idea if this is even close to accurate), I'd think x.264 would use CPU vs some GPU offload using NVENC (assuming right drivers in place). I believe I've read on these forums that x.264 would yield slightly higher quality encoding vs NVENC... so it depends on which resource you are running short of, as to which to recommend to use.
What I don't know is if a 5yr old GTX 960M should be able to handle 720p60fps [when combined with your overlays]?
And when you say issue popping up again... assuming you are updating Windows, realize OS updates and/or GPU drivers could be cause of new behavior
With out better advice from others... my recommendation would be to drop to 30fps and see if that resolves encoding overload situation (hopefully it will). While at it, learn how to use Task Mgr/Resource Monitor if you don't already know. Then, if you are inclined, try changing encoder and see if you have the resources for your choices (ie CPU for x.264) and which you prefer. I'm sure there are optimization options... but which ones specifically to recommend for your situation is way beyond my knowledge level
