Yes I'm on windows.
The only solutions I found were to set all audio levels to the same Sample Rate (48khz) and to disable "Use Device Stamps" for the Desktop Audio properties, I have no idea why but these things worked. Also, NDI doesn't like dedicated filters on your audio (they sometimes bug out) so if you have any kind of NDI Audio or ASIO filters/outputs, they might stop the second PC from picking it up via NDI. So either NDI Ouput filter should be used, or none at all (the usual filters like gates, etc. are fine).
As for the "performance", I don't know why, but it seems to be an OBS thing, I tried using some Media Sources (like videos) for when I was AFK on the stream, turns out OBS uses an abnormally large amount of processing power (almost 5-10% per media source), even when the media source isn't displayed or disabled. I tried a million things and it seems like an issue with OBS or a really weird way they decided to process stuff. I mean I don't know why OBS would need almost 30% of my processor if I have 2 videos in 2 different "Scenes" that are not active, but that could explain why NDI was having a heart attack on the primary PC.
NDI should only be costing you 2-3% processing power while active.
The only way your Primary PC would still be under the same load as if encoding is if it's Streaming, using a Plugin that does multi-streaming (Youtube + Twitch+ etc), you have many Media Sources (Videos, images, etc), you're Recording, you have plugins with fancy filters (like StreamFX) or are doing something fancy with the NDI filters (NDI Video + NDI Audio filters) to send data in a fancy way to your Secondary PC.