Open Settings → Advanced and set:
P010Rec. 2100 (PQ)Partial
Open your Game Capture source properties and set sRGB Color Space to Rec. 2100 (PQ).
Open Settings → Output → Recording. Use a 10-bit HEVC/H.265 or AV1 encoder; select the Main10 profile if it is available.