The "rescale output" option in the recording or streaming menu uses a CPU-based rescaler, while "Output (scaled) Resolution" uses a GPU-based rescaler.
Keep in mind the way frames are handled by OBS. First, you have a canvas. The canvas has the "Base (canvas) Resolution". You place the your sources on the canvas and they are mixed together and filters applied to get a picture. This all takes place on the GPU. The picture buffer of the canvas is kept on the graphics card (GPU). After the picture is composed, it is rescaled on the GPU to the "Output (Scaled) Resolution" and then downloaded (copied) to CPU memory.
From CPU memory, it is fed into the encoder (x264). You can have up to 2 encoders active: one for streaming, one for recording. If you use 2 encoders, the pictures are duplicated: one is fed into the streaming encoder, and the other into the recording encoder.
On each of the encoders, the picture may be rescaled a second time (if you configured the "rescale output").
The compressed frames from the encoders are then fed into the output facility. For streaming, they may be sent via tcp/ip to the rtmp server. For recording, they may be saved to harddisk.
It's possible to use the stream encoder for recording as well to reduce system load, or to exactly record what you stream. If you do this, you lose the possibility to set a different "Rescale Output" resolution for recording, because you have only one encoder running, thus only one possibility to rescale at the encoder-stage.
From that information it's clear which rescaling option you should use:
- If you produce only one output, rescale using "Output (Scaled) Resolution" only. Or don't rescale at all and set "Base (canvas) Resolution" to your final resolution. What you chose depends on the size of your sources.
- If you produce two outputs, one streaming and one recording, and you want both look the same with the same resolution, do the same and use the stream encoder for recording.
- If you produce two outputs, one streaming and one recording, and you want a different resolution for recording than for streaming, or you want a higher quality encoding for recording, configure different encoders for streaming and for recording. Set the "Output (Scaled) Resolution" to the bigger resolution of the two. For the smaller resolution of the two, set "rescale output" option of the encoder to your desired resolution. This way, both encoders produce different resolution with the minimum amount of rescalings.
An example for the most complex scenario: you have a game running at 2560x1440 and want to stream at 1280x720 and record at 1920x1080 at the same time.
Then you set "Base (canvas) Resolution" to 2560x1440 and place your game and additional sources in it. You set "Output (scaled) Resolution" to 1920x1080. You configure the stream encoder with CBR and bitrate of perhaps 5000. You configure the recording encoder with CRF/CQP and a quality of perhaps 18-20. For the stream encoder, you set "rescale output" to 1280x720, and for the recording encoder, you don't rescale output. To not overload your machine with this scenario, it is probably the best to use a hardware encoder (NVENC or Quicksync) for at least one of the encoders, preferable for the recording encoder, because it has the higher load because of the higher resolution and higher data rate.
If you instead keep "Output (scaled) Resolution" at 2560x1440 and "rescale output" the stream encoder to 1280x70 and "rescale output" the recording encoder to 1920x1080, you get lower quality of the recording and much higher resource/CPU utilization, because you transfer double the amount of data from GPU to CPU and you do 2 rescalings on the CPU instead of 1 rescaling on GPU and 1 rescaling on CPU. So don't do that.