As TryHD said, if you're not seeing pixelization in the preview or recordings, then the only thing that will be affecting the quality of the streamed output is the bitrate... and the frame complexity.
If you don't have much movement on screen, you will probably have a cleaner webcam image. But as soon as your scene starts including more information it needs to store in the limited bitrate, your webcam quality is going to suffer.
This is called "Interframe Compression". It's a normal thing, and
Tom Scott does a great job of explaining it.
There's several things you can do, to varying degrees of impact
- Increase your bitrate (highest impact)
- Increase your compression quality (lower impact, very diminishing returns past the "fast" preset on x264)
- Reduce your output resolution (higher impact -- gives more data per pixel, at the cost of overall image sharpness)
- Lower your framerate (very high impact -- fewer frames, gives more bitrate per frame, but at the cost of video smoothness)
- Increase your webcam size (depends -- this is more to cover the compression artifacting, as well as have less to compress from the more complex area)