I'm testing similar things to you at the minute.
What is abundantly obvious is that as soon as you go past around 4500kbps, a portion of your viewers get 'buffering' issues and either complain to you and leave or (more commonly...) just leave. It's either because they are in a network that is shared with other people and devices, netflix etc, or they are walking around on 4G / LTE and the signal strength goes up and down. Or their hardware just can't handle it, they have too many browser tabs open, etc etc etc.
I've stumbled across a few people stating the benefits of 864p to resolve this issue and have nice high streaming quality for viewers so I have recommend you also try these two scenarios. This avoids the need for just going over the 4500kbps threshhold @ 6000kbps for 1080p.
1) The H264 AMD GPU Hardware method
VIDEO
Base Canvas 1920 x 1080
30FPS
No Downscaling settings here (see Streaming)
OUTPUT
-STREAMING
Apple H264 Hardware VT (AMD Graphics)
Downscale to 1536 x 864
Lancozos Sharpened Filter (32 or 36)
Streaming Bitrate 4301 kbps
Lancozos Sharpened downscaling is best for in-person video but not so good for fast paced gaming.
Audio Quality 320kbps if you want the highest audio quality but you can easily lower this to 192 or 256kbps
2) The x264 CPU method
You could try and use x.264 instead of Apple VT Hardware AMD on the Output. Apparently, the AMD Graphics are really only very good performers when recording above 10,000kbps but smaller bitrates required for streaming are not so great to look at.
To do this, you'd need to set everything up so there is no Downscaling and try some tweaks at the bottom of this post.
In order to give your audience a lower enough bitrate so that people can watch it streamed live (the max is around 4500kbps) I recommend the following:
VIDEO (BASE AND DOWNSCALE)
1536 x 864 Base
1536 x 864 Downscale
(ie No Downscaling)
30 Frames Per Second
STREAMING
ENCODER
x264 (your CPU)
No Downscaling
Stream Bitrate 4301kbps (this was calculated based on the link at the bottom of this page)
Keyframes 2
Profile High
Extra tweaks for Mac that take up CPU / RAM
- Ensure everything else is closed ESPECIALLY Chrome.
- Remove items from the desktop and put them somewhere else
- Remove as many Apps from the Dock
- Shrink the OBS screen down so that the Preview video is smaller (this uses up less CPU)
- Remove your Recent Items (Apple logo top left corner, RECENT ITEMS, CLEAR MENU
All of these numbers are based on this calculator.
Need help finding the best settings for your livestream? Use our handy livestream calculator to find the best settings for your internet plan.
www.chow-bryant.com
If the above works better in terms of video quality and frame rate for your viewers, you could consider using a separate login to your mac with the above set up (empty desktop, nothing in the Dock bar etc) so you can use your laptop generally as you would prefer to work when you are not streaming.
I've given you these options in the hope of finding a sweet spot as yours will be different to others and I've recently learned about 1536 x 864 x 30fps being great for viewing.
Let me know how you get on :)