I had the same problem. Nobody bothered to assist me though. I can stream with 8,000 bitrate using Shadowplay but trying less than 5,000 on OBS kills my internet. Can barely load google. I too was trying to stream on Mixer.
You bring up a great point and a question I once asked someone over at AMD that works on AMD AMF and he replied with a much more in-depth answer but here is the jist:
Shadowplay and Relive do not function the same as OBS, one advantage those features has is direct hardware access across their own API's or drivers that cannot be accessed outside of those services, it doesn't have to deal with all the overhead OBS does, and that's why you can do things like 2K or 4K 30-60FPS with Shadow/Relive with only a 1-2% hit in frames, versus what might be monstrously larger hits while using OBS, it's not OBS's fault, it is just the nature of the beast. Even though we have like NVENC and AMF ENCODER, while based on some of the same techniques and code, it is functionally different and works not as direct to the metal and not without overhead as the Native Shadowplay and Relive solutions.
Oops I just noticed you were talking about saturation of your connection differences in Shadow vs. OBS, there really shouldn't be a big difference in TCP overhead that I'm aware of, it could just be optimized code for offloading the traffic to the NIC or the TCP Stack that is maybe not available in OBS? Have you tried the "Enable New Networking Code" feature in Advanced Settings? This may help matters, for some it makes worse, I've noticed it really depends on your network hardware more than anything else for this feature, and sometimes drivers for GPU as it can have an effect on how effective the traffic handoffs can be, although I could be wrong, just something I've noticed.
