I don't think x264 provides for a truly constant bitrate in live streaming. But anyways:
Small buffer means the stream stays closer to the nominal bitrate you set. But then it can't compensate as well for complex scenes, so quality goes down.
Large buffer means that the stream can vary more from the nominal bitrate, increasing quality but potentially spiking the bitrate to levels that may interfere with game traffic or the ability of viewers to download the stream.
Personally I noticed my pings in Planetside 2 spiked when I streamed to Twitch at 3000 bitrate with 3000 buffer. So I kept the bitrate at 3000 but reduced the buffer to 2000, and all was then well. The quality seems higher than doing something like setting a bitrate of 2500 and buffer of 2500 (qualitatively, I didn't do any measuring).
I have not bothered to min/max this. Its possible that the best combination for me is something different, like 3500/1500 bitrate/buffer. It would take some testing that I probably won't bother to do unless my channel somehow becomes super popular (not gunna happen).