This controls the CRF parameter in x264. Combined with the VBV system, this generally controls the CRF with which static scenes are encoded at. If you specify quality 10, static scenes will look better, but leave less buffer space available if there is sudden motion, resulting in pixelation. Using a quality of 5 or 6 will make static scenes slightly worse, but in the case of motion, quality will be improved. Experiment with this depending on your input, for FPS games and other high activity scenes you probably want this lower than 10. For fairly static games like MTG, FTL, turn based strategy, etc, you can stick with 10 without a problem.
Recommended: Varies, 5-10 depending on source.
Basically tells x264 to target this bitrate in kbps. The actual bitrate will vary depending on the complexity of the scene. Set this to around 80% of your upload speed. If you have high upload speed, keep in mind that your viewers will require an equivalent download speed to be able to view your stream, so you probably don't want to go over 3500kbps or so. Keep in mind if you stream at lower resolutions you will not need as high of a bitrate.
Recommended: 80% of your upload speed or 3500kbps (whichever is lower)
This ties in closely with the bitrate. x264 will encode everything into a fixed size buffer of this size. If there is a sudden scene change and high motion, it can fill up this buffer at a rate faster than the average max bitrate. Eg if you have a bitrate of 3000 and a buffer of 6000, x264 could decide that if a scene is complex enough, to use 5000 of the 6000 buffer at once. This makes your actual network bitrate quite spiky, which can lead to latency issues if queuing occurs on your modem or router. Raising this makes the stream bitrate more spiky but improves quality, reducing this makes the bitrate more tightly controlled and closer to the max bitrate setting. Don't set this too high as the client-side decode buffer will also need to be this big, and you have no control over that.
Recommended: Same as bitrate
This will set x264 into CBR mode. The "quality" setting will have no effect as the quality will be determined entirely by your bitrate, and x264 will try to ensure your stream goes out at close to the bitrate specified. This results in a less spiky (bandwidth wise). It will also cause both you and your viewers to consume more bandwidth as scenes with no motion at all will still use the specified bitrate unless "CBR Padding" is disabled.
CBR can help with viewer lag issues caused by variable bitrate as TCP is not well designed for variable bandwidth. The quality cost of CBR is very low and the issues it can fix make this very appealing, and it also eliminates the process of picking an appropriate quality value.