Upload and Download see different treatment, as long as you have no dropped frames and your recordings on twich are smooth (once loaded), then twitch is getting your stream properly.
If you have dropped frames you should work on that. (edit: there's a guide somewhere what to check for that, using a different twitch server might be your best bet.)
If not, then it can still help to lower bitrate, or use a proxy to watch, or a different streaming service to broadcast/watch the stream. (hitbox is nice too)