Ok sorry i missed one part in the log before
its still in this included
12:04:33.995: [jim-nvenc: 'recording_h264'] init_session: nv.nvEncOpenEncodeSessionEx(¶ms, &enc->session) failed: 10 (NV_ENC_ERR_OUT_OF_MEMORY)
that can have 2 reasons first like itsounds to less vram free for start encoding
another is a driver issue
you have the latest version of nvidia driver from nvidia website?