So I found out one of the main reasons for the "audio desync" happening and why it doesn't happen to other people on Windows.
It's the Audio Filters system, and from what I could determine, it's an OBS issue on how it's accessing the Filters after one of the updates, after V27 through Windows (I don't know if a windows update did something about this as well).
I was trying to use Elements FX to route half of the VST2 Plugins that don't work in OBS (Lisp, the excellent free ToneBoosters which are better than Reaper, etc), and whenever I press record or such, it immediately desyncs the audio on the second PC. It works perfectly fine on the main PC, but the second PC get's desynced immediately with NDI due to a "hook" delay on the information starting.
When I removed the Element FX and the extra Filters, the desync issue stopped. Also, by removing specific Reaper plugins, I was able to get rid of also the random "start game and audio desyncs, so you have to start OBS AFTER starting the game" issues I was having since then by about 90% (as well the drift). This means half of the ReaJS tools are not useable.
Please note you need to REMOVE them, disabling the Filter with Eye Icon doesn't stop the desync/drift issues, I have no idea why this is still a thing with OBS.
This may also be an issue with Filters that do "Look Ahead" or "Pre Comps", even something small like 1MS is enough to cause issues I think.
I honestly think at this point something needs to be done about the Audio part of OBS, it doesn't support VST3, fine, but it's ridiculous that some excellent VST2 plugins can't use their UI in OBS (or they're broken), and you can't even close them unless you specifically press Close Plugin Interface, or how you need to use a DAW re-route to bypass a lot of the limitations OBS has in simply loading a plugin, even Adobe Premiere can load those.
I guess the solution is to outright setup a Voicemeter system and just set up your chain/filter from there and have OBS pick that up instead, I don't know how the latency will be, since I never did such a thing, but I'm going to assume a "static" 1.25MS desync is much better then whatever it is that OBS is trying to do half of the time.