Syncing Multiple H.264 Streams Automatically

[aquarat]

Hi

I've been using a SlingStudio system, which apears to be a locked-down Linux box with a bunch of ARM-based HDMI->H.264 video encoders which attach to cameras. The system works well but it's very restrictive.

I've been trying to replicate the SlingStudio system using OBS. The video sources I'm using are the SlingStudio itself (via RTMP) and several Raspberry Pis with HDMI inputs.

I'm using an NginX docker image (with recording enabled) to act as an intermediary between OBS and the my video sources. All video sources are RTMP.

My process for the Pi-based encoders is: Camera -> HDMI -> CSI -> Raw H.264 -> FFMPEG RTMP -> TCP/IP -> NGinX -> OBS

I've been trying different GOP sizes on the Pi's hardware, currently the delay is around 300ms for a 20 mbit/sec HD stream. The streams are reliable and run for hours on end without issue. I have tons of Pis and the HDMI bridge breakout boards are around $29, so this is a very cheap solution for me.

The streams are, unfortunately, out of sync by a variable amount. The difference between stream syncing appears to be established at the start of the streams but doesn't noticeably change for several hours. This means that currently I can apply the delay filter in OBS on a per stream basis to manually sync up the streams and they then remain synced for several hours.

I'm wondering if there's any way to get OBS to automatically sync the streams, perhaps using timecode? Or a different protocol? I've tried generating timestamps in ffmpeg using -fflags +genpts and use wall clock as timestamps, but this has no effect in OBS.

Can anyone suggest anything? I suspect this is a complex problem...

 

[aquarat]

I've confirmed that the wall-clock timestamp is present in the streams coming from the Pis (by getting ffmpeg to overlay the timestamp).

I'm now thinking that the syncing could be done using a Python script, along the lines of:
- Enumerate all sources with "xC" in the name (as an example)
- Get the latest timestamp amongst them
- For every source subtract that source's current timestamp from the largest timestamp
- Apply the delta as the async delay filter

Re-evaluate every 5 minutes and only apply changes if the sync is out by more than some threshold.

Getting the sources is easy, but I'm not sure how to get the latest timestamp for a source? It looks like a possibly usable timestamp forms part of obs_source_frame, but it appears that you can only get that as part of a filter effect or plugin.

 

[WildkatzKyr]

Did you ever solve this?

I'm looking at a similar issue with three of us streaming into RTMP on Raspberry Pi, then an OBS box receiving all three streams for showing split-screen game streams etc.

Would be very interested in hearing if you managed to solve this and get the pi to automatically sync up the streams using a timestamp.

 

[aquarat]

I didn't solve the issue per se, it sort of turned into a crazy deep dive into OBS and IP-based video switching in general, which, 6 months later, I've only just been able to solve sufficiently.

I've gone with NDI encoders over my own home-grown solution as the units I've gone for have a number of benefits over the Pi approach for my use case. I did manage to get the Raspberry Pi based approach working in terms of sync as a proof-of-concept.

I used a hacked-together version of YAVTA (yet another video test application) to manipulate the Pi's GPU/H.264 encoder into capturing and encoding video data from the CSI interface. I took the raw H.264 and piped it into ffmpeg. I got ffmpeg to timestamp the video stream with the "wall clock" time (which was synced on each Pi via ntpd) and then sent that out to an NginX instance and ultimately on to OBS using RTMP as a carrier. I've since discovered that RTMP is not a good carrier for low latency video and rather something like SRT would have been better. RTMP demuxers do a lot of caching.

The wall clock timestamps made their way into OBS, but this didn't cause OBS to sync the streams against each other - but what it did do is it allowed me to use a Python script to query the timestamps on the streams and calculate the offsets.

The result was that I could manually enter the sync deltas as an async delay filter in OBS and that synced all the streams predictably and reliably. In practice I found that once the streams were in sync they remained in sync. Reducing the GOP size on the Pi encoder also helped reduce the latency between the video sources and OBS (to about 300ms in my case). I could have probably gotten the latency lower by using SRT.

The quality produced by the Pi encoders at 20 mbit/sec for 1920x1080 @ 30p with a 3-frame GOP size was excellent. I may still use this approach in future for remote cameras.

After reading the OBS source code, I have a suspicion that if the wall clock times on the Pis were advanced by a few seconds that OBS would probably automatically sync the streams. I think this is the case because it looks like OBS chooses what frames to render based on whether their timestamp has elapsed... but I could be wrong.

 

[Filip S]

This is a nice subject - I was trying to sync two camera vidoa stream from two rpi4 using SRT and elgato camlink 1080p/25 frames on wifi - I can stop/restart the srt capture on my OBS and they are close to sync - I have not solved the problem - I would be cool to sync maybe on audio but I don't really know howto.

 

[Tomasz Góral]

I tested this solution, the rpi device sent 1080p25 signal via UDP, I received on my computer using ffplay but I marked the source as p30, as a result each frame was played as soon as it was sent and the signal from 4 rpi looked synchronized.
If you could set the frame rate as a parameter in Media Source, it could improve the timing.