I don't know why the NVENC wasn't working, judging from the log ('unknown error'), but x264 (which you tried using afterward) draws on the CPU's power -- not the graphics card. Do you have a powerful CPU? You're getting a lot of lagged and skipped frames due to OBS Studio failing to get the necessary cycles to encode video (CPU 'too slow').
If your CPU is powerful-enough to encode video in real time, are you running other CPU-intensive software (a CPU-intensive game?) with a higher CPU priority than OBS Studio? If so, that's another situation where OBS could be starved for cycles, even if the CPU is powerful: an application with a higher CPU priority will always get access to the cycles before one with a lower-priority.
If you're sure that your CPU is powerful enough, I recommend experimenting with closing all software other than OBS Studio and the game that you're trying to stream. Make sure that your scenes in OBS Studio don't have too many sources. Make sure that you're using the veryfast x264 CPU preset -- and, in OBS Studio's 'Advanced' settings, make sure that it's running at a higher-than-normal CPU priority.
If that doesn't work, all that I can assume (I'm no expert, especially without seeing your rig) is that your CPU just isn't beefy enough. I hope that's not the case, though, for your sake. Good luck!
