Use x264 instead of NVENC, but only, if your CPU is beefy enough to handle x264 presets like "faster", "fast" or medium".
Other than that, there is nothing you can do.
If the bitrate is fix / limited, you have only these options:
Decrease resolution = less details/sharpness but also less pixelation/artifacts
Decrease streaming framerate = same sharpness, less pixelation/artifacts but less smoothness
Increase encoder efficiency by choosing x264 with more CPU heavy presets.
But keep in mind, that switching from NVENC to x264@very fast will only give you very little improvement in quality and a lot more CPU load.
Going from x264@veryfast to x264@medium will increase CPU load extremely while only resulting in a small improvement of the image quality.
So bascially to achieve a very obvious improvement, you would need way more bitrate, but this can be limited by the streaming platform and even if it's not, you might loose many viewers, if your stream is not transcoded by the streaming service and you stream with 8.000kbit/s or even more.
Example of the meaning of transcoding and the problems without it:
Let's say you stream at 10.000kbit/s 1080p. If the Twitch server has enough resources left and you have >100 viewers, the Twitch server will transcode your stream to some lower resolutions, so the viewers can choose between 480p, 720p and original source quality.
If the streaming provider is not transcoding for you, the viewers can only watch your original stream quality. As many viewers are using slow Laptops, bad WiFi, mobile broadband or even a slow Internet connection in general, they will not be able to watch your high quality stream without constant stutter/buffering pauses.