Open the stats window. It is shown in realtime.
For CUDA, it's not a matter of being powerful enough. It absolutely is. The problem is in how the resources are being allocated. If the only thing your GPU had to do was run OBS, then more than likely you would have absolutely no issue. However, you have this gigantic resource hog called a game that is fighting with OBS for all the resources it thinks are top priority.
Best analogy I can give is you have an entire table set with food. OBS is sitting at the table with his own plate of food (imagine this as the nvenc encoder). As long as all it needs is what's on its plate, then everything should run fine. But as soon as you need something that hasn't been already put on the plate, then you have to compete with everyone else at the table for dibs. Some games aren't greedy... others are gluttons that don't know the concept of playing fair.