You're using Win11 insiders build which may include bugs and conflict with other programs. Please stay with the stable release. Or even better, with the last Windows stable and reliable OS, Win 10.
OBS (Open Broadcaster Software) is free and open source software for video recording and live streaming. Stream to Twitch, YouTube and many other providers or record your own videos with high quality H264 / AAC encoding.
obsproject.com
For the nvenc error you may want to try a clean reinstall:
If you're experiencing strange issues or errors that might be related to your GPU drivers, one of the easiest potential fixes is a clean driver installation. Follow the instructions below. 1. Download the latest version of Display Driver...
obsproject.com
Also, for recording its recommended to use CQP intead of CBR and mkv format instead of mp4.
