Hey so I have noticed, and maybe it is something that happened in recent OBS updates, but it seems like the edges on rotated masks are much more jagged than they were previously. For context I have some scenes where I have applied a rotated (-86 degrees) rectangular mask to an image source to create a diagonal edge on it. Previously, and my apologies I am not sure exactly when it started happening I just now noticed it, the diagonal edge would appear smooth in both the OBS preview and on recorded/streamed output. Now that diagonal line is jagged. I am recording at native 1080p so no downscale filtering is being applied.
Curiously though I created a static image mask with the same diagonal and applied it to the same source and the diagonal line is still jagged, which leads me to believe perhaps it is something that has changed to rendering inside OBS itself. I am on OBS v31.0.2.
I just thought I would bring this to your attention, and again, it may be something that has changed inside OBS itself as the built-in image mask with a similar mask applied to what I was creating with this plugin still appeared jagged. If anyone else has ran into similar issues and has any additional context that would be great. Thanks!