Audio desyncs on stream only, fixes itself, desyncs again

[cliophate]

Hi there,

yesterday I did my very first stream. I, of course, didn't try streaming once to Twitch before to check if all was well, and only checked everything by hitting record.
And on record, all was good.

But the moment I streamed my chat told me that my audio was out of sync. I fixed it momentarily by adding -700ms to the audio.
After a while, however, the audio was out of sync again. This time, however, since it was my first stream, most people left again (heheh, sad-emoji) and no one told me that it was out of sync.

After switching scenes, because I changed games and had a small break, when I came back, the audio was automagically in sync again, (according to the VOD) after a while it desynched again.

My setup is a bit particular though, and unfortunately, I can't get logs right now.
My setup is:

• Gaming-PC goes to Atem Mini via HDMI (max is 1080p60). Microphone (Elgato Wave:3, which kind of fakes a mixer) is hooked up to gaming PC and set to Listen via HDMI (and the device is the Nvidia Port thingaling) in the Windows audio settings.
• The camera goes to another HDMI port on the Atem Mini (1080p60 again)
• Both go via USB-C to the streaming-PC on which OBS is running.
• OBS recognizes one device which has my game, my audio and my camera. That's just how the Atem Mini works.

What I find particularly weird: It has NO issues when recording. At least, none that I have recognized, then again, I haven't yet recorded for more than 5 minutes (Might do that tonight). It only happens on stream and it kinda fixed itself when I was on a break. The scene I use for breaks doesn't have the Atem Mini listed as source.
When I came back, switched to a scene with the Atem Mini as a source, it was good for a while.

Attached some examples from the VOD:

Beginning when it had issues:

Twitch

Twitch is the world's leading video platform and community for gamers.

www.twitch.tv

I fixed it by setting the delay to -700: https://www.twitch.tv/videos/702292392?t=00h47m06s
Out of sync again:

Twitch

Twitch is the world's leading video platform and community for gamers.

www.twitch.tv

Automagically in sync: https://www.twitch.tv/videos/702292392?t=02h36m36s
Everything good is bad again:

And here we finally go! Hearthstone - How to survive in the Battlegrounds. - cliophate on Twitch

cliophate went live on Twitch. Catch up on their Hearthstone VOD now.

www.twitch.tv

 

[cliophate]

Here is a little update. I kept a recording running for over an hour and sometimes clapped to check the sync. And even the recording starts to desync after a while. So it clearly is not only a streaming issue.

 

[cliophate]

And one more update. It's not that only the recording and the stream are out of sync. But the more OBS is running, the worse it gets.
It's so bad right now after having it open for an hour, I can visually see the delay in the audio mixer when I clap into my hands.

So I clap, and the audio peaks a second or two later. So it's not a recording or streaming issue. It starts drifting the moment OBS is open.
All my Googling didn't do anything, I am seriously out of ideas.