A clear question, but somewhat difficult to answer and to understand, because the pixel aspect ratio is not 1:1 as in today's digital video processing. Pixel aspect ratio not 1:1 means a pixel is not a quadrat but actually a rectangle.
That means that if I tell you a resolution, it might look distorted due to the non-quadratic pixel aspect ratio, and you might say "you gave me the wrong resolution".
Depending on how your digitizer device works, it may be necessary to invert the distortion - or not.
According to
https://en.wikipedia.org/wiki/Standard-definition_television, you should start with 704x480 as resolution in your capture device. It might also be necessary to use 720x480 instead of 704x480, if you get a "full frame" from the digitizer.
This should be rescaled within OBS to 640x480 (or 654x480 resp.) to make the pixel aspect ratio quadratic. To achieve this, right-click your source->Transform->Edit transform and set the scaling options like this:
(set size to the original size as from the capture device, bounding box type to "stretch to bounds" and bounding box size to 640x480. This will squeeze the slightly too wide original image to the proper 640x480 size with clean 1:1 pixel ratio.
As very viable alternative, you can also just record directly what you digitize without rescaling (704x480) and perform rescaling with some postprocessing software. This is without using transform settings like above but do the corresponding thing within the postprocessing software of your choice.
To calibrate, find a recording where you see a big circle. Make sure the circle is a real circle and no ellipse in your final product.
Finally, if you digitize VHS recordings, the actual resolution on your tape is half of what I wrote above. It's only 352x240 (rescaled to 1:1 pixel ratio: 320x240). You might get better quality if you set the output resolution of OBS to 320x240 if you use the transform settings from above. If you intend to use your postprocessing software for undistorting, keep 720x480 within OBS and do the final rescale to 320x240 in your postprocessing software.
Why downscaling to half: Transmitted over air by the TV station was 720x480, but recorded on tape is only half of this. If you scan the tape with the original resolution 720x480/704x480, the image is filled with noise. This noise is much squeezed away if you downscale to the real resolution of the tape, so you get a cleaner picture with this, even if it is upscaled to your computer monitor with its 1920x1080 or whatever you have.
I hope this makes sense. You need much trial and error. Don't expect to get perfect results with your first recordings.