There is currently no way to change the shape of a webcam like that. You can crop the top/bottom/left/right, but you can't apply a mask. You will be able to do this in the OBS rewrite, though.
For now, people imitate round webcams by having an overlay with a round cutout. Not quite the same, but has a similar-ish effect.