In general, if you allow a higher latency, you're giving the encoder more room to encode stuff. This increases picture quality for streaming, but it delays the stream slightly. However, this latency within the encoder is very small - a few frames only, usually much less than a second. Ordinary network latency due to buffering is way higher, up to multiple seconds. So the latency within the encoder is not that significant and you should use this potential to optimize the quality.