You can mask a rectangular source to any form with the "Image Mask/Blend" filter. You add your (rectangular) webcam as source, then apply this filter. At the "path" property, you give the name of your circle mask you have to create. It should be a *.png file you prepare with any painting app.
Create an image that has a black background with a big white-filled circle where you want to see your webcam image. A full white circle on black background.
If you have a painting app that understands transparency, you can create a transparent background instead of black and choose the "Alpha Mask (Alpha Channel)" option in the filter instead of the default filter - this may be more intuitive, if you are familiar with painting with transparency.
But look closer to the video. There are 2 additional effects to the circle. There is no hard edge at the circle border. To the outside, the circle has a grey shadow, and to the inside the person is faded in from grey-ish to the real image.
Unfortunately, I'm not a painting expert, so my advice ends here. I can only say that instead of a white circle with a hard border, you can create a faded circle border with the "vignette" filter of many painting apps - this creates a circle that's becoming transparent to the outside. I have an idea of how to add the grey shadow to all this, but I doubt it's the way to do it right, so I will not send you on a fool's errand with that.
