I have the same issue as you.  What happens for me is that when I select my device for output, I get a small delay.  I figured this was normal since the OBS output monitor has a live delay.  But after some time, the delay becomes longer and longer.  If I just put the Audio Monitor on and wait....within about 10 minutes I can get up to a 3 second delay.  I put this in the bug reports some time ago, because i would love to use this plugin, but it is unusable.  I do a lot of routing to get into and out of OBS with VST's and other programs, but if there could be a solution that elminates all of that, it would be fantastic.