OK mate try these settings that I have attached for you first and then work backwards until you get it working. I have a final solution which involves spending more money but will try to avoid that first.
Your CPU handles your graphics which means on top of running the whole system, isn't very powerful, but can work.
The ideal machine for OBS is Windows based and contains Nvidia graphics cards capable of NVENC. Doesn't have to be new, mine is a 2012 model. You are running 3 cameras via USB by the sounds of it, which is pushing it.
If you want your stream to be watchable you need to be running 720p minimum. On twitch you can get nice results with 1536 x 864 but youtube wants 720p or 1080p streaming.
To optimise the Mac to be ready for a stronger streaming performance there are some things you can do:
___
Extra tweaks for Mac that take up CPU / RAM
- Ensure everything else is closed ESPECIALLY Chrome.
- Remove items from the desktop and put them somewhere else, as the mac pre-loads them to a degree
- Remove as many Apps from the Dock, as the mac pre-loads these to a degree
- Shrink the OBS screen down so that the Preview video is much smaller but you can still see what you need (this uses up less CPU power)
- Remove your Recent Items (Apple logo top left corner, RECENT ITEMS, CLEAR MENU, as apple pre-loads these to a degree
- Take out anything on USB that is not needed. If you have a USB keyboard and mouse, and a bluetooth one, perhaps try the bluetooth ones to free up the USB slots.
Then go into every camera and change the setting on it to:
- resolution 1280x720p
- frame rate 30fps
- Colour Format 709 and Colour Range Partial.
- Change each Camera to Low Latency (instead of Normal Latency) and see if it works. Sometimes this makes it worse.
Note in the images for CPU Usage there are options such as VERY FAST, FASTER, and FAST - when you first encounter these options they might not make sense, so just know that VERY FAST is working the CPU harder, FASTER relieves it a little, and FAST can yield less jitters working the CPU less hard. I would imagine you need to use FASTER or FAST.
Upgrade options keeping the costs down
1. Keep the Mac you have, and buy a (used?) ATEM MINI or ATEM MINI PRO. Be aware you need cameras that output to HDMI on this as it it is a professional camera switcher. As far as I know an Atem Mini doesn't listen to USB webcams and stuff like that - it needs a DSLR or Mirrorless style camera with HDMI output. The switcher will take the strain of receiving the cameras and handling all of that information, and just send one signal to your iMac which will stream much better.
2. Buy a Windows laptop with Nvidia Nvenc graphics card (or find an old machine you have that happens to have Nvidia graphics). When you are gaming externally, some older machines combined with Nvidia perform exceptionally well like my Lenovo Thinkpad W540. Graphics cards found in old Dell / HP / Lenovo and similar laptops which work include Quattro K1100m, K1200m, K2000m, K2100m but there are many more. These cards outrun my iMac 2011 Intel I5 2.5ghz quad core with 16gb Ram and dedicated AMD Graphics and my Macbook Pro Mid 2015 Intel I7 with 16gb Ram and AMD Graphics.
I'm a bit of a believer that the iMacs work well when you have Atem Mini on them because they aren't doing the heavy lifting! Imacs with Intel graphics can also work well with one camera but 2 or 3 can be pushing it.
SETTINGS