Frame drops / stuttering with more than 2 NDI sources 1080p30 in 1000 Mbit Network

[Grzegorz Bieniek]

Hi everybody,
i have a very special setup, but due to my Cameras i want it this way.
I have three computers running OBS with each one attached an DSLR in 1080p30 HDMI output as an source in OBS for recording the raw footage of the show. I then want to send from each computer, let them call NDI01- NDI03 the videofeed of the individual camera via NDI to my main streamig computer.
I need this step, because my DSLRs can not record in 1080p30 and send in 1080p30 but switch HDMI output to 720p30 when recording internally.
For each NDI stream i have around 90Mbit/sec inmy networkmonitor. so when i Start with one NDI stream my main Streaming computer gets 90 Mbit/sec in its networmonitor, when i start NDI02 it goes up to around 180 Mbit/sec. But when i now start with NDI03 the Networkmonitor at my main Streaming Computer caps at around 220Mbit/sec and all three Videostreams start to drop frames and stutter now and then. Does anyone has an idea why this might be the case?
If you need more clarification, i can add screenshots or othe documents you might seem helpful.
Thanks in advance.

 

[Lawrence_SoCal]

I need this step, because my DSLRs can not record in 1080p30 and send in 1080p30 but switch HDMI output to 720p30 when recording internally.

I think I understand. You can't use internal DSLR recording due to lower resolution when also using HDMI out. right?
If so, I still don't see why the 3 NDI/OBS PC setup you have.... unless there is something else??
With right PC setup and 3 HDMI capture cards/devices (desktop, possibly PCIe add-in cards to make sure not overloading bandwidth), or maybe single Thunderbolt 3 external device with 3 HDMI capture ports?? not sure.. not anything I've looked into
Basically, I don't get why 3 separate PCs, unless distance or other effects that preclude capturing all 3 cameras on 1 PC??
IF the issue is 1 PC is overloaded trying to handle HDMI capture from 3 cameras, switching that to NDI isn't really accomplishing anything that I'm aware of (could be wrong). NDI is just a common wrapper/protocol for transmitting video over Ethernet (equivalent, sort of, to iSCSI vs SCSI). Your primary PC still has to handle processing 3 separate incoming video streams... and then handle real-time video encoding after that... ouch... not trivial

For each NDI stream i have around 90Mbit/sec inmy networkmonitor. so when i Start with one NDI stream my main Streaming computer gets 90 Mbit/sec in its networmonitor, when i start NDI02 it goes up to around 180 Mbit/sec. But when i now start with NDI03 the Networkmonitor at my main Streaming Computer caps at around 220Mbit/sec and all three Videostreams start to drop frames and stutter now and then. Does anyone has an idea why this might be the case?

side/Curious - have you saved a large file on one of the NDI PCs and then done a file transfer to OBS main PC? can your network, and PCs sustain something closer to 1GbE traffic rates?

What PC hardware monitoring are you doing? Task Manager (Performance tab) and/or Resource Monitor?
I'm suspecting your PC CPU is overloaded trying to process/decode the 3 incoming NDI streams plus whatever else you are doing. but could be something else? Are you using OBS' Studio Mode (ie rendering video twice)?

So there is optimizing the receiving PC operation (from OS to OBS) and then adjusting (if possible) NDI traffic to minimize hardware requirements on the receiving end. Using the uncompressed NDI (ie not NDI HX) is bound to help as the decompressing simply takes more work. If optimized, there is getting a more powerful PC (and one designed for true throughput, in which case I'd steer away from consumer gear (which I always do anyway) and look at true workstation class machines). Low-end motherboard chipsets could be a bottleneck??

 

[Grzegorz Bieniek]

I think I understand. You can't use internal DSLR recording due to lower resolution when also using HDMI out. right?
If so, I still don't see why the 3 NDI/OBS PC setup you have.... unless there is something else??

It is excatlly this.
I want to be able to use my Raw footage for a recut after the liveshow if neccesary.

With right PC setup and 3 HDMI capture cards/devices (desktop, possibly PCIe add-in cards to make sure not overloading bandwidth), or maybe single Thunderbolt 3 external device with 3 HDMI capture ports?? not sure.. not anything I've looked into
Basically, I don't get why 3 separate PCs, unless distance or other effects that preclude capturing all 3 cameras on 1 PC??
IF the issue is 1 PC is overloaded trying to handle HDMI capture from 3 cameras, switching that to NDI isn't really accomplishing anything that I'm aware of (could be wrong). NDI is just a common wrapper/protocol for transmitting video over Ethernet (equivalent, sort of, to iSCSI vs SCSI). Your primary PC still has to handle processing 3 separate incoming video streams... and then handle real-time video encoding after that... ouch... not trivial

This is an interesting hint, which i Could try. Like open 4 Intances of OBS on my main computer, connecting the Cameras via Dongle, like at the moment and let 3 NDI Output instances of OBS running, which also record. And then locally sending the NDI output to the fourth instance for Streaming.
This should the machin handle without a problem.

side/Curious - have you saved a large file on one of the NDI PCs and then done a file transfer to OBS main PC? can your network, and PCs sustain something closer to 1GbE traffic rates?

What PC hardware monitoring are you doing? Task Manager (Performance tab) and/or Resource Monitor?
I'm suspecting your PC CPU is overloaded trying to process/decode the 3 incoming NDI streams plus whatever else you are doing. but could be something else? Are you using OBS' Studio Mode (ie rendering video twice)?

Yes i tried it. Copied the recording of one Camera via Network and the Transfer Speed was at around 800 Mbit/sec, du to the slow Harddrive in one Machine.
And in Task Manager (Performance tab) CPU and GPU are at around 40% CPU and GPU at 30% in "Video Encode" and 20% in 3D.
Yes i am using Studio Mode and also have a Program Window Fullscreen on a seperate Monitor.

So there is optimizing the receiving PC operation (from OS to OBS) and then adjusting (if possible) NDI traffic to minimize hardware requirements on the receiving end. Using the uncompressed NDI (ie not NDI HX) is bound to help as the decompressing simply takes more work. If optimized, there is getting a more powerful PC (and one designed for true throughput, in which case I'd steer away from consumer gear (which I always do anyway) and look at true workstation class machines). Low-end motherboard chipsets could be a bottleneck??

The Streaming Computer is an i7 gen 4 like 4600 or something like this with 24 Gig Ram, Geforce1650 /4GB Ram and internal graphics.
And i am constantly checking CPU, GPU and RAM usage and they are never higher than 70%.
I hope this helps and thanks a lot for your hints.

 

[Lawrence_SoCal]

ugh, 4th gen i7.... that is old (11 gen to be released supposedly this March) and real-time video encoding is computationally demanding. I'm surprised CPU load isn't higher... I'm still suspecting a resource bottleneck on that old of a PC (just not showing up in default monitoring options)

And now I get why separate recording per camera... and this isn't my area... but hopefully there is a better way to accomplish your goal, and someone else with share/advise regarding such

As a test step on an old, under-powered PC, try not using OBS Studio Mode as that creates a LOT of extra work to process 2 separate video outputs

 

[Grzegorz Bieniek]

ugh, 4th gen i7.... that is old (11 gen to be released supposedly this March) and real-time video encoding is computationally demanding. I'm surprised CPU load isn't higher... I'm still suspecting a resource bottleneck on that old of a PC (just not showing up in default monitoring options)

I will have a look in Resource Monitor next time i do my setup. At the moment i am redoing my office/studio so everything is packed up. But i am still quite sure, that it is not the CPU.
Espacially because i tried it also wit all 3 Cameras connected via HDMI->USB on the Machine and it was no problem. I just could not record each camera individualy, only the Stream.

And now I get why separate recording per camera... and this isn't my area... but hopefully there is a better way to accomplish your goal, and someone else with share/advise regarding such

As a test step on an old, under-powered PC, try not using OBS Studio Mode as that creates a LOT of extra work to process 2 separate video outputs

Will also try this and post everything i have discovered. Will take some time due to renovating.

 

[Grzegorz Bieniek]

side/Curious - have you saved a large file on one of the NDI PCs and then done a file transfer to OBS main PC? can your network, and PCs sustain something closer to 1GbE traffic rates?

I Just tested my network with my main computer and 2 of the NDI computers. I was sending a 16 GIG File to one of the NDI sources and receiving another 16GB file from the second NDI computer on my main Computer. It was sending and receiving at around 960-1000 Mbit/s. so i quess, that my problem has rather nothing to do with my network bandwidth.

 

[devin130]

I Just tested my network with my main computer and 2 of the NDI computers. I was sending a 16 GIG File to one of the NDI sources and receiving another 16GB file from the second NDI computer on my main Computer. It was sending and receiving at around 960-1000 Mbit/s. so i quess, that my problem has rather nothing to do with my network bandwidth.

I'm curious if your network can handle that type of transfer in parallel. Does your router/switch handle 1 gig out on each port, its seems like it may only handle 250Mb where you'd want around 300 going to your streaming pc.

 

[Grzegorz Bieniek]

I'm curious if your network can handle that type of transfer in parallel. Does your router/switch handle 1 gig out on each port, its seems like it may only handle 250Mb where you'd want around 300 going to your streaming pc.

Well as far as i could test with 3 computers in the same network, it worked fine. but i will setup al three computers and test it again with sending three files to my main machine, but concerning that i worked well, i should get around 330 MBit/sec with each computer.
Will test it and post. also I will try just the NDI tools feature to send 3 Streams through screen capturing.

 

[Grzegorz Bieniek]

I tested Yesterday my network and got 500 Mbit/sec in and out simultanously on my main PC while transfering some 14 GB testfiles from my notebooks to the main PC. so i quess it is not a network problem,because NDI network usage seems to cap out at around 220 Mbit/sec. I will try today to Stream my cameras via the original NDI tools, without the OBS Plugin and see if it might be some differnt behavior and report my findings.

 

[Grzegorz Bieniek]

Thanks @cyclemat , but this is in my current setup not possible, because the main streaming pc only has on network jack and no places to add an aditionall pci- network card. I tried the NDI tools "scan converter" yesterday to capture a full screen in 1080p30 and the network load was only at aound 50 Mbit/sec per Videofeed, compared to 120 Mbit/ sec, if I use the NDI tools Programm output in OBS. Only problem i encoutered is, that the NDI tools programm gives a higher load on my cpu, than the OBS plugin. So i have to fiddle a vit around again. Maybe i get one or 2 cheap capture cardas and a HDMI splitter, to split the signal of the cameras into 2 sources and use one source for recording and the other one for streaming.
Also i found some settings in the network switch concerning bandwidth and will fiddle with them around, maybe it is them. i have actually no clue, what i will do and what my changes will do, but it is worth trying. Thanks to everybody who pointed me to the networkswitch settings. it was a pain for me, to find the settings on my switch, lets hope i find it again.

 

[Tomasz Góral]

NDI use SSE3 for encoding and decoding, probably your processor is too old (too slow) for 3 signal via NDI.
NDI suggest for 1080p50 (1080p60) need bandwith 150 Mb/s (for High quality, in OBS you can choice quality). NewTek site

 

[Grzegorz Bieniek]

NDI use SSE3 for encoding and decoding, probably your processor is too old (too slow) for 3 signal via NDI.
NDI suggest for 1080p50 (1080p60) need bandwith 150 Mb/s (for High quality, in OBS you can choice quality). NewTek site

Thank You for pointing this website. I will try following tip: https://support.newtek.com/hc/en-us/articles/360043724494-Using-Access-Manager-to-address-NDI-issues
and i wanted to ask if settings done in this also have an impact , when i decide do use OBS as an output?

 

[Tomasz Góral]

You have TriCaster ?
OBS as NDI sender don't have any option.
NDI as input you can select Highest or Lowest or Audio only.

 

[Grzegorz Bieniek]

NDI use SSE3 for encoding and decoding, probably your processor is too old (too slow) for 3 signal via NDI.
NDI suggest for 1080p50 (1080p60) need bandwith 150 Mb/s (for High quality, in OBS you can choice quality). NewTek site

As i visited Intels website the processor supports Intel® SSE4.1, Intel® SSE4.2, Intel® AVX2
But i start getting to the point, that it might be a CPU bootleneck.
Thank you all so far with the help an also for teaching me, that there are usb to network adapters.....
Will have a day trying something next week and will tell you further results. so far nothing else helped.

 

[Grzegorz Bieniek]

Another thing is i might update my Stream Rig with this CPU :https://ark.intel.com/content/www/u...-processor-e5-2690-v3-30m-cache-2-60-ghz.html
After reading the tech specs, it seems, that this cpu does not support SSE3 , which would be a bad choice. but on the other hand i can not believe, that a processor from 2014 does not support a feature, which was introduced 2005.
Does anybody know something more?

 

[Grzegorz Bieniek]

I think i found a sollution. Today i hadd the chance to bring my recording setup to a friend with a beefier workstation and we tried my setup and it worked really fine. the main difference is, that his workstation has 2 ethernetports and it also was connected to the network with both ports. so i think it might be some limitation in my network setup, maybe depending on my network card. thank you everybody for all the hints.

 

[Lawrence_SoCal]

Possible, but unlikely. If I recall correctly, you have a 1Gb NIC, and you are doing anything to approach or exceed that. That workstation most likely has a much more powerful CPU and other components, all of which are making the difference. A modern single CPU/NIC PC should be able to handle 2 NDI streams, no need for a dual NIC workstation [which takes special settings to take advantage of on same network segment