Switch scenes in OBS using an audio or video source

[Rockma_Studio]

Good morning everyone, I wanted to ask if it’s possible to switch scenes based on an audio or video source. Let me explain better: I’m trying to set up a stream where my band and I perform a series of cover songs. What I’d like to have is a method, either within OBS or using external software, that allows me to switch scenes when the audio/video source we’re playing reaches a specific point. For example, when it’s time for a drum solo in the audio, OBS switches to the camera focusing on the drummer; when there’s a keyboard section, OBS switches to the camera focusing on the pianist. I hope I explained myself clearly.
Does anyone have any advice they can share?

 

[qhobbes]

If your pianist is using a MIDI keyboard, you could us a plugin such as obs-midi-mg or Free MIDIControl to change scenes. You could also use an Android/iOS device to change scenes and have your pianist do it (I'm assuming your pianist isn't playing all the time)l.

 

[Lawrence_SoCal]

qhobbes is referring to method for you/someone to trigger the OBS Studio Scene switch manually

and beware, I'm not a musician, though I do get to act as 'roadie' for my partner...

For a computer to watch/listen to a performance, and then act based upon that is doable, but, effectively would require some advanced programming to pull off. A computer has no idea, even if you input the sheet music, to know a solo vs other activity, and then desired camera shot to use (and/or manipulate a PTZ camera). Now, when you think autonomous vehicles, clearly there is machine learning/'artificial intelligence' that can accomplish the equivalent. However, you should be aware of the massive teams (hundreds/thousands of talented programmers and data analysts, outrageous sums of money, etc.. [getting 'close' to a solution is relatively easy with today's tech... getting reliable... that is really damn hard... ex Tesla FSD]
Another simpler option would be video matching, but that also quickly gets complicated [I've heard of one user using very simple video recognition (simple color) in a slide to trigger (using plugin Advanced Scene Switcher) scene changes [ie, put a very small color box in a portion of a slide that is cropped out of a capture. then plugin can still see entire source and act off content in it, and color matching is computationally easier than trying to read text in a video].

So, one option would be for you to provide the music to some software, and 'mark up' that music, such that when the computer picks up the audio, actions are taken based on your 'markings' [ie, you indicate when this sequence of music is played, do X]. but computers are really stupid. they will do EXACTLY what you tell them... meaning the music performed live would have to be EXACTLY (or very, very close) to what you 'marked' up.
Think big name live concert performances... and why are there are people as sound engineers, lighting, etc... because automating that, adjusting for live performances variances, etc... not really practical [costs WAY more to automate than to pay staff to execute, for years.. ie no ROI]

Now, I suspect there is some audio listening software that is more flexible than I'm aware of, and maybe really could tell where you were in a Cover song, even with live performance variations, impromptu bits, etc. How accurate/reliable that would be? no idea. But once you found that, and presuming that software offered 'triggers (ie perform a programmed action when a certain 'point' in a song hit), then you could send that trigger to other software (including OBS Studio, probably using websockets) to then execute what actions you programmed there. And if you had a PTZ camera, could trigger presets or other actions on that as well

so with all that... assuming no one else around other than band members performing... the question is whether there is one person who could be tasked with 'triggering' the changes... using either a MIDI trigger, or even a foot pedal or similar remote control pad [ex streamdeck or mobile device] ... there are other manual trigger options as well ... but in the low-cost approach, the switch will be manual. I'd recommend a direct trigger from a device... but thinking of the video slide example above... if you had a set camera perspective, and someone on stage changed a light/color card to correspond to a desired scene, that could work, with all the challenge of color matching in a variable lighting situation [ie, gets tricky/potentially unreliable].
The challenge will be.. .you are likely to start simple, then assuming a band member can multi-task (perform, and control livestream s/w at same time), you'll want to get more sophisticated ('cuz it will look better)... if that person is guitarist, gets to be a challenge if using a foot control ;^) but a phone or small tablet sized device on a mic stand... could easily have 6 -12 'trigger options' to select from?? [# depends primarily on person's eye sight and needed size of 'icon']

just some random thoughts... hope it triggers some ideas for you or others

Hopefully others will chime in

 

[AaronD]

If you play with a click track (and you don't collectively get off by an entire measure), then that will fix the timing so that you can just time it out. OBS has no knowledge of the music at all; it just knows the initial trigger, and times it out from there.

Expanding on that a little bit, if your click generator is actually a MIDI sequence, then not only can you put your click sounds in that, but you can also put OBS commands in there too. Maybe some unobtrusive patch if you have to hear it, or if you can have it at zero velocity or something like that, then you can do what you want. Either the MIDI plugin mentioned previously, or the Advanced Scene Switcher, can then receive and filter the MIDI data and respond to it.

Advanced Scene Switcher

This plugin will allow you to automate various tasks using "Macros". Macros consist of a list of conditions under which a list of actions will be performed. Examples and guides can be found in the wiki. Feel free to contribute! If you run...

obsproject.com

(I know there are lots of inaudible MIDI commands (MIDI Show Control - MSC - comes to mind), but it's probably much easier to use a piano roll for everything just to see it all together, and that more-or-less limits you to Note[On/Off].)

If you don't use a click, then the timing could be anything, and so you need to effectively "play the cameras" as if they were another instrument. Thinking that way, a pedalboard might be best, especially if it's set up to act as a keyboard as far as the computer is concerned. At that point, you just set up some hotkeys...

 

[ChristianS]

For example, when it’s time for a drum solo in the audio, OBS switches to the camera focusing on the drummer; when there’s a keyboard section, OBS switches to the camera focusing on the pianist. I hope I explained myself clearly.

Hi,

I would try the Elgato pedal. I do not have specific feedback with it, but generally Elgato devices work smoothly with OBS.
You should be able to assign shortcuts to each pedal, to switch manually (footly ?) between scenes.