Sounds like the classic "feedback" effect, of capturing the same display that shows what's being captured. Audio can do that too, as an increasing screech or persistent echo.
Either way, the solution is to break that loop, which already happens when you no longer show the screen capture on the same screen. Immediately, the stream or recording will clear up.