Given you have a Ryzen 7 1700x and a 1080ti, you should be able to record any game at max in-game graphic settings...
With your setup, you should be able to run your webcam output at 720p30fps with no issue, which will increase quality of the image.
As for recording output, try these settings:
Video:
Base Resolution: 1920x1080
Output Resolution: 1920x1080
Downscale Filter: Lanczos
FPS Value: 60
Output:
Output Mode: Advanced
-
Recording Tab -
Recording Format: flv
Encoder: x264
Rate Control: CRF
CRF: 8-15
Keyframe Interval: 0
CPU Preset: Veryfast or faster (Higher) for CPU intensive games, may be able to do slower (Lower) then veryfast for non CPU intensive games (Games that push your CPU, not GPU)
Profile: High
Tune: None or Personal Taste/Dependent upon recording scenario. See below link for a little more info on each preset:
https://superuser.com/questions/564402/explanation-of-x264-tune
x264 Options: Leave blank.
Note on CRF value: Anything below 10-12 will equal massive bitrate spikes for a single frame on occasions throughout the recording, as well as huge file output sizes for very little noticeable gain in quality. For example: Game recording of Battlefield 4 Large Conquest 64 players, ~30 minute video = @ Ultrafast preset @ CRF: 8, tune: film, average bitrate will be 185Mb+, single frame spikes of 750+Mb(!) ~30+ GB filesize.
Given you have a 1080ti, you could also look at using nVidia's Dynamic Super Resolution tech to down-sample from higher resolutions to push out more quality.
If you don't mind, can you upload a logfile with a recording in after trying those settings out, with info on file size output and your view on quality please. Would be handy for reference as I don't have your hardware (Great setup you have) and will assist if any problems arise from the settings that need troubleshooting.