custom ffmpeg output. It's only available on recording.
Considering that youtube doesn't support it, the only reason to use it is to archive your recordings in a smaller format for use in editing later. If you're going to do that, just use handbrake to convert after you're done recording.