It's only buffering while you are watching it? You're not getting dropped frames?
That is mostly an issue with your connection to Twitch. You aren't able to download your video from Twitch's servers fast enough. It won't happen to everyone who watches your stream. You can try streaming to a different server or lowering your bit rate if you're worried about it.