If you get pixelation, you need one of three things. More bitrate, less framerate, or less resolution. If you cannot raise the bitrate due to a service cap (as in the case of streaming to Twitch), you need to lower the resolution or framerate. High-motion games will also require comparatively more bitrate (FPSes) than low-motion games (MOBAs).
There is no 'magic bullet' or 'best settings' that will get around this. You can try another streaming service that allows using more bitrate.
Beyond that, it is a balancing act that you will need to work out for yourself.
(As a side note, Twitch's maximum recommended is 6000kbps, but will allow higher rates up to around 8500kbps. It WILL cause increasing numbers of player and network errors for your viewers, and is a terrible idea unless you are a Partner, with guaranteed transcodes/quality options. It will severely hamper a new/small channel's growth due to sharply limiting the number of people who can watch smoothly, and is shooting yourself in the foot as a new caster, getting lost chasing numbers.)
I broadcast on twitch, I currently use this configuration, which is supposed to be the best for twitch in 2021, you have already seen it in 10 or 15 youtube videos this year
Output mode: Advanced
Frequency control: CBR
Bitrate: 6000
Keyframes: 2
Preset: Highest Quality
Profile: High
Look Ahead: YES
Visual Psycho: YES
Gpu: 0
Maximum B-frames: 2
FPS: 60
Scale filter: Bicubic 16, samples
Resolution: 1920x1080
But the one that OBS recommended to me was this:
Output mode: Simple
Encoder: Hardware (NVENC)
Bitrate: 6000
Audio Bitrate: 256
FPS: 60
Resolution: 1920x1080
Scale filter: Bicubic, I leave it the same as the previous settings that I had applied
I really don't know much about the subject, I don't know which one is better if bicubico, lanczos, for example also what difference is there in putting 60 FPS or 59.94 FPS ???
In the simple output mode, below there is an option that says * habilidat the advanced encoder configuration says Encoder Preset, with my settings which one would you choose? performance, max performance, quality, max quality etc?
And also what is the Custom Encoder Configuration for, what is it for, how does it work etc?