Seeing the number of people that recently had the exact same problem. Its twitch. You can try lowering your bitrate until noone buffers anymore, that would be the sweetspot. But some day its 3mbit, some day its less than 1mbit and the next week you can even watch 4mbit without a problem.
I got used to it and a lot of friends have too. Twitch sucks, their HLS change helped nothing but their wallets and I cant wait for hitbox etc to grow a bit bigger or maybe the return of Hashd. :)