The square form of highlighting

[Nederig]

I don't know how to describe this problem, in general:
I need to select a certain area that I want to blur, but I can only use a square form to select and most of it remains unselected. I don't want the selection to go beyond (in my case the monitor) I want to highlight a triangle or a circle, for example. Because if it goes out of bounds, my hands and other parts will be highlighted, which creates an unpresentable appearance

Does anyone know how to solve this problem? Maybe I haven't looked hard enough...

Take a look at the picture to better understand the situation:

 

[khaver]

You can use my Polymask filter. Add the source twice. Put the lower source in a group and blur the group. Put the top source in a group and add my Polymask filter to that group and adjust the points to match the outline of the monitor screen. Use the "Strength" and "Use color mask" settings to make it easier to place the corners. Use "Invert" if needed.

 

[Nederig]

You can use my Polymask filter. Add the source twice. Put the lower source in a group and blur the group. Put the top source in a group and add my Polymask filter to that group and adjust the points to match the outline of the monitor screen. Use the "Strength" and "Use color mask" settings to make it easier to place the corners. Use "Invert" if needed.

Thank you!

 

[Nederig]

You can use my Polymask filter. Add the source twice. Put the lower source in a group and blur the group. Put the top source in a group and add my Polymask filter to that group and adjust the points to match the outline of the monitor screen. Use the "Strength" and "Use color mask" settings to make it easier to place the corners. Use "Invert" if needed.

I have another question, when adding a second selection, for some reason it is not blurred, but remains colored
I tried to do this by adding another group, but it didn't work
How do I fix it?

s

 

[khaver]

The problem with using more than one Polymask filter (or any filter for that matter) is the input to the last filter takes the output of the filter below it. To make it work, OBS would need to somehow change the code for the way filters are processed. I think it would be better to create a new filter that blurs the area in or outside the mask itself. That way it wouldn't rely on a blurred version of the source behind it. This may be my next filter I work on. No ETA though.

 

[Suslik V]

Why not to draw the mask manually in any image editor? OBS has built-in Image Mask/Blend filter.

Edit:
In chroma keying thread you can find few examples of more advanced mask usage: https://obsproject.com/forum/threads/choosing-an-area-on-screen-for-chromakey.134874/