while JPEG compression does have some worst case scenarios, peaks that would suddenly go from 400-500 mbit/s to overloading a 1000 mbit/s network are very unlikely, I think it's more likely that there is an issue in the network and then unsent data packets pile up for a bit in the buffer (which is what the high buffer messages indicate) and are then sent in a burst when the network recovered. would be interesting to see a network usage curve from the time when this happens: does it first go down and then up (as described before, first congestion and then recovering in a burst), or first up and then down (this would indeed mean there's a peak which then causes follow-up issues)?
the fact that also NDI worked fine and then suddenly stopped working, and now Beam is also having issues, also points toward network problems, even when the error scenario does look different. although...does it? you said one of the PCs stopped working with NDI, and NDI doesn't have a manual connection mode, so when UDP Multicast discovery fails (this is what a source uses to find an active sender) NDI would indeed stop working, as there is no other way to fetch this feed. and with Beam you describe the same, discovery fails (Beam uses the same UDP Multicast discovery technology that NDI uses), just that with Beam you could make it work anyway, because it has a manual connection mode as a fallback. was that affecting the same PC in both cases?
also, yes, Beam with JPEG indeed would use significantly less bandwidth than NDI (maybe even half of it), if it was good enough for NDIs higher bandwidth needs before and now even fails for the smaller JPEG bandwidth needs, something's definitely broken there.
is any of the connections over Wifi? the reason why it's always recommended against using it for this kind of transmission is exactly that it can work for a year, then a neighbor sets up their new microwave at the other side of the wall where your router is, or also a new Wifi router that interfers with yours, and your network is now forever unstable. or for a week. or for each 2nd day. so using Wifi for this would be a huge gamble and if that's the case it would be pointless to continue debugging at this point, the only sensible course of action would be to go wired.
if it's already wired ethernet, then this could be a multitude of things. I would look at software first. software firewalls could cause both the UDP Multicast discovery failing and other instabilities, temporarily disable it on every PC involved and see whether discovery works again, if it does, you got your offender. ISPs in germany usually give out routers to their customers (instead of just plain modems), in that case extra software firewalls aren't necessary anyway. if one of the systems is a laptop that you sometimes use in networks outside of your home, you can configure the firewall to be disabled in private networks (your home network) and be enabled in others so that it still protects you there.
another piece of software that is known to cause a lot of issues like that is so called network optimizer software. many mainboard vendors like MSI or ASUS ("GameFirst" utility) provide them along with their mainboards and market them as being great for gaming, improving latency and so on, but mostly they cause a lot more trouble than their worth, and especially streamers should never use them. also some NIC vendors (Killer) provide such tools and also independent tools exist like cFosSpeed. whatever it is, if you have it, get rid of it. not only disable, completely uninstall, then reboot and hope it didn't also permanently mess up network settings that are not reverted upon uninstallation.
even more sneaky are certain security software suites, some mess with network traffic although they don't even look like they'd have anything to do with network and just call themselves something with "security". at least temporarily disable them, or better get rid of them altogether (Windows Defender is fine though, you can let that live).
looking at the surprisingly long list of network interfaces that you seem to have, do you have any kind of virtualization software installed, e.g. VirtualBox? this can sometimes cause issues with NDI, because NDI tries to use all network interfaces it finds, including virtual ones. for Beam make sure you always explicitly select the physical network interface you want to use.
if you ruled out software, then hardware is the next thing to look at. try to change network cables, if you don't have a spare one (usually a few accumulate over time, because every new router comes with one), swap those of 2 computers and see if the problem moves or changes.
if you're using your onboard LAN ports (like most people), think about getting a dedicated gigabit LAN card, even if it's only temporarily for ruling out that a specific LAN port is not the problem - you can get
decent ones for as low as 10 €.
and while testing all of these things to narrow down where the problems are coming from, it would be good to have something for testing your network that was meant for that use case, I myself use the command line tool iperf for this, but with jperf there is also a graphical interface:
https://wiki.ipfire.org/addons/iperf
