Scripted OBS controller

[Paul van Dinther]

I decided to post here in the hope for some suggestions and feedback. I wanted to control OBS smoothly using variable inputs from a gamepad or joystick. I was unable to find what I was looking for so now I am working on an app that fills that need. This program (under construction) automatically senses the presence of input devices such as joysticks, gamepads and midi devices and calls the user script when a value changes. The user can scripts testing input values and sending commands containing some of the input values to OBS via the obs-websocket plugin. The result is a seamless smooth link between your controller hardware and OBS with limitless capabilities.

The software is 100% web based but installs as a PWA on your PC and can run without the internet once installed. So far I only run and test on a chrome browser on windows and it's rock-solid. I am still vague on how I implement the actual user scripting. Very keen on feedback and ideas.

Here is a screenshot of what I have so far. The UI is 100% interactive so you know exactly what input value comes in from where. The midi support is still a bit bare, I put that bit in today. The user script is called every time an input value changes. Analog inputs are de-jittered to keep the number of calls to OBS to a minimum.

This thing will let you use anything as a scene switcher while leaving your keyboard for typing. Use a slider to zoom in, another one for a transition. The sky is the limit. Well, actually the API for the obs-websocket library is but up to a point. A lot can be done before that api is called.

The USB controller shown is a little thing I made a few years ago based on a USB highres gamepad board BU0836A made by Leo Bodnar. All I did was wire a bunch of pots and switches onto it.

 

[Paul van Dinther]

Not much interest maybe but I am having fun and keep going :-)
Implemented Ace editor and a rudimentary console. The script shown does work but it will not be the release format. There is just too much code. It needs to be much cleaner.

Still keen on feedback. Today I was thinking that this should also work on a mobile device. In that case you can add graphic dials, sliders, switches and buttons. Make your own layout and your own script underneath.

 

[apaiva]

I am definetly interested! I am having a hard time fading audio sources and the ability to use my Korg NanoKontrol Studio would be a AMAZING!!

 

[southbysowhat]

Very interested, would love to be able to use an external device to ride audio levels. I've been playing around with this: http://75r.de/midikey2key/ but there doesn't seem to be any corresponding hotkey-assignable way to change audio levels

 

[Paul van Dinther]

Very interested, would love to be able to use an external device to ride audio levels. I've been playing around with this: http://75r.de/midikey2key/ but there doesn't seem to be any corresponding hotkey-assignable way to change audio levels

Awesome. It's going to be dirt easy with this app. Just plug in your midi device to your PC. It will be automatically detected and it will visualise on which channel, note and velocity signals come in. From there it is a simple matter of writing a script. Not sure yet but something like a two 3d arrays. One for the total known state and one containing the changes values for all the midi messages that arrive during one program cycle (animationFrame). That way every possible type of midi message can be scripted on. Something like:

const noteOn = '1001; //a program defined constant
const controlChange = '1011 ' //a program defined constant

let midiChannel = 0;
let control = 7;

if (midiControl(midiChannel, control) ){
       setVolume(currentScene.sources[0], midiControlNorm(midiChannel, control);
}

if (midiRaw(11, midiChannel, 64)){
       setVolume(currentScene.sources[1], midiRaw(11, midiChannel, 64) / 127);   //Div by 127 to get a value between 0 and 1
}

if (midiNoteOn(midiChannel, 65) ){
       setVolume(currentScene, midiNoteOnNorm(midiChannel, 65));   //Sets all sources of the scene to the same volume
}

The above are just ideas. It be fun to figure out the best way to make this as easy as possible without falling into restricted UI traps. I am very close to release an alpha version. I'd be delighted if you would give it a go.
Do note that I exclusively test the code on an Chrome browser on Windows 10.

 

[cignor_dappert]

Just suscribed for this :D
i'll gladly alpha test if you need, i'm on w10 aswell.

 

[Paul van Dinther]

Incredibly energised by this project. I have reliable midi data streaming into my app. The app runs in a browser so there is nothing to install. however, if desired chrome and Edge chrome (The Microsoft version) can install the webapp as a stand-alone PWA and run offline.
Firefox works as well but no MIDI.

As I was unsure what type of midi messages controllers usually broadcast I ended up installing TouchDAW on my phone. Via another program called rtpMIDI by Tobias Erichsen. I was able to send regular midi messages wirelessly to my PC which presented them to a virtual MIDI port which then in turn is dynamically picked up by my app.

Don't worry, you won't need to deal with any of this. I fully intent to build a UI similar to TouchDAW right here in the app which will run just fine on any touch device and MIDI won't even be needed in that case.

Just for fun I setup a scene in OBS with four scene items each on a quarter of the screen. The TouchDAW UI panel Shown superimposed on my app screenshot presents 4 sliders. Then I was able to script in my app for each slider which scene item to zoom in to fill the screen expanding from the appropriate corner. Quite a specific use case that would be hard to achieve any other way.

Below you see the app in action. Quite a crazy setup. All devices are automatically detected at startup or when you plug them in when the program is running. The top device is a USB Gamepad device. The second item is my Yamaha PSR S950 keyboard and the next two devices are virtual midi ports created by the rptMIDI program I have running. They are receiving MIDI messages from my phone running TouchDAW. It all just works! More work to do before I can let you guys test it. But time... that I have plenty of, during lockdown in New Zealand.

Super impressed, grateful and many thanks to Palakis for the foresight to build such an excellent plugin for OBS. I might be sending many more feature requests his way in the future. I am thinking media shuttle control or a way to send hotkey presses to OBS via the web socket.

 

[kineticscreen]

I would be very interested in this too. I have just been playing with OBS-Websocket-midi by @Twan which works perfectly, but unfortunately hasn't yet moved past v0.1, and can only change scenes. if it could also do source show/hide and change transitions then it would be perfect.

What is particularly impressive about it, especially for a first version, is that it seamlessly brings up the scenes of your OBS instance to select from. You should have a look at it for inspiration.

I'm currently working on controlling aspects of OBS on a PC via Qlab running on a Mac, and doing it via Midi seems like the easiest option. Will be very interested in giving your solution a go!

 

[Paul van Dinther]

I would be very interested in this too. I have just been playing with OBS-Websocket-midi by @Twan which works perfectly, but unfortunately hasn't yet moved past v0.1, and can only change scenes. if it could also do source show/hide and change transitions then it would be perfect.

What is particularly impressive about it, especially for a first version, is that it seamlessly brings up the scenes of your OBS instance to select from. You should have a look at it for inspiration.

I'm currently working on controlling aspects of OBS on a PC via Qlab running on a Mac, and doing it via Midi seems like the easiest option. Will be very interested in giving your solution a go!

I did install it just now. Very nice to popup some UI in response to a change event and attach an action to it. Looks like a web app as well wrapped in Electron. In my mind there are two issues the UI needs to solve. The first is to define what constitutes a trigger. This is usually the press of a single button, toggle of a switch, movement of a know or slider coming from one or several devices. But it can of course also be a combination of inputs. Do this with slider 1 but if the toggle switch is in the other position do something else.

But the program you linked made me realise that rather than having one script driving everything I need to break it up. Maybe replace the script panel with an "Action center". A button called "Add action" then detect the action input and run a wizard which constructs the initial script. In most cases it will be good enough. If not you can dive right in and do horrible things to ones brain.

if ((trigger(gp[0].B[1].pressed && gp[0].B[2].pressed) || trigger(mc[0].noteOn[64].value > 40)){
    let angle = gp[0].A[7] * 360;
    let fitXScale = videoInfo.baseWidth/currentScene.sources[0].source_cx;
    let fitYScale = videoInfo.baseHeight/currentScene.sources[0].source_cy
    let scaleInput = (gp[0].A[6]+1)/2;
    let scale = scaleInput*fitXScale*2;
    let x = ((videoInfo.baseWidth/2) - ((scale*videoInfo.baseWidth)/2))+gp[0].A[4] * videoInfo.baseWidth;
    let y = ((videoInfo.baseHeight/2) - ((scale*videoInfo.baseHeight)/2))+gp[0].A[2] * videoInfo.baseHeight;
    send('SetSceneItemProperties', {
        'item': currentScene.sources[0].name,
        'rotation': angle,
        'scale': {'x':scale, 'y':scale},
        'position': {'x': x, 'y': y}
    });
}

Implemented MIDI protocol today so we are not only gazing at byte arrays but also more meaning full data to work with right down to MIDI sysex. Hexample in the screenshot. Columns are not marked yet (channel, messagetypeId, messageTypeText, noteID or controllerID, value, Raw byte array)

Please keep the ideas coming.

 

[Paul van Dinther]

Midi is working too. I now have a midi controller feeding the script with sensible data and you can do literally everything with it.
MIDI is designed to be a stream of control data which is nice for music but we don't want to flood the browser with re-paints nor the OBS system with un-necessary update instructions. Like with a button, We only want to know if something changed and also if it went up or down. This keeps the CPU workload to an absolute minimum. Internally A complete map is maintained of all the midi values. Incoming midi messages are simply mapped into it and changes to the values trigger the stript.

Also nice clean references which I will also apply to the gamepad devices.
Checkout the code in the pic. See if it makes sense. I made the few helper functions to make the script cleaner.
"down" is a test to see if a midi value has gone down. There is also "up", "changed", "min" and "max" which will tell you when you reached the stops. In addition to the changed values, you can also interrogate any other midi value you want.

So: changed(midi.mc1.ch0.noteOn.key56) references mc1 (Midi Controller 1 which is the top keyboard) then looks on channel 0 to see if key54 has changed. you can also read the value midi.mc1.ch0.noteOn.val or as a normalised number between 0 and 1 midi.mc1.ch0.noteOn.norm

I am about to pass out. 5am. Did an all nighter. I still intend to put a friendlier frontend on but for some this program is just about useable. Let me know what you think.

 

[kineticscreen]

Very exciting to watch this come together! At what point will you have an online version that we can test?

Without wanting to completely blow out the scope of work, while you're doing this it might be worth looking at incorporating MSC (Midi Show Control) and even OSC network messages - both favoured forms of triggering cues in the music / theatre worlds!

 

[Paul van Dinther]

Very exciting to watch this come together! At what point will you have an online version that we can test?

Without wanting to completely blow out the scope of work, while you're doing this it might be worth looking at incorporating MSC (Midi Show Control) and even OSC network messages - both favoured forms of triggering cues in the music / theatre worlds!

I think I am only days away from putting it out there in it's current format. The UI will change dramatically over time but with the tests I have done, I am convinced that user ability to write scripts is essential to really put this thing to work.

I am limited to whatever I can receive inside a modern browser. No UDP. MSC looks totally doable. You can already receive sysEX byte arrays which you could interpret yourself in the user script. But I think I can add it to the protocol interpreter so it is easier to use. Specs on MSC vary. I was reading this page.

This app is heading more to the youtubers market and semi pro home studios.

 

[kineticscreen]

Sounds like a perfect market to be targetting this at! I'm currently doing a lot of experimenting with triggering OBS from Qlab, as well as using the latter to provide audio cues and video overlays via NDI, so being able to synchronise via Midi is really the dream.

And yes, I would strongly agree with making it as user friendly as possible.

 

[Paul van Dinther]

Loads of progress. What remains is testing for this version. I want to release this before I move on and build a different UI. But this thing is powerful. Hence the App name: "OBS Dominator" :-)

Cleaned up a million little details. Added a little feature that allows you to place the cursor in the editor. Check "Listen" and when you move a midi control or gamepad control or button the code reference is inserted into the script.

Improved several helper functions such as "changed" so you can pass multiple inputs to it. It produces cleaner user script code. Documentation will come once things settle down.

CSS is improved a lot. It is now possible to collapse controller panels to get irrelevant things out of the way. The app will now render properly on a mobile phone. This is important as you will see in the future. :-)

Here is an example of the script code as it works right now:

//  scene switcher
if (valDown(midi.mc0.ch0.noteOn.key54, pads.gp0.b6)){
    obs.setScene('Mix Scene');
    log('We set the Mix Scene');
}
if (valDown(midi.mc0.ch0.noteOn.key55, pads.gp0.b7)){
    obs.setScene('Web Cam Scene');
}
if (valDown(midi.mc0.ch0.noteOn.key56) || valDown(pads.gp0.b8)){
    obs.setScene('Silly Clip Scene');
}
if (valDown(midi.mc0.ch0.noteOn.key57) || valDown(pads.gp0.b9)){
    obs.setScene('Photo Scene');
}

//  source slider
let val = changed(midi.mc0.ch0.pitchBend, pads.gp0.a3, pads.gp0.b0);
if (val){
    expand('Web cam', val.map, 5, 0, 0, 1280);
}
if (val = changed(midi.mc0.ch1.pitchBend, pads.gp0.a5, pads.gp0.b1)){
    expand('Silly Clip', val.map, 6, 1280, 0, 1920);
}
if (changed(midi.mc0.ch2.pitchBend, pads.gp0.b2)){
    expand('Lockdown', changedVal.map, 10, 1280, 720, 4032);
}

//  volume mixer
if (changed(midi.mc0.ch4.pitchBend)){
    obs.setVolume(midi.mc0.ch4.pitchBend.map, 'Silly Clip');
}
if (changed(midi.mc0.ch5.pitchBend.map)){
    obs.setVolume(midi.mc0.ch5.pitchBend.map.map, 'Web cam');
}

//  You can extend the system by building your own effect functions
function expand(sourceName, value, alignment, xpos, ypos, sourceBaseWidth){
    obs.moveToTop(sourceName, 'Mix Scene');
    value = ((value / 2) + 0.5) * obs.videoInfo.baseWidth/sourceBaseWidth;
    obs.send('SetSceneItemProperties', {
        item: sourceName,
        scale: {
            x: value,
            y: value
        },
        position: {
            x: xpos,
            y: ypos,
            alignment: alignment
        },
        bounds: {
            type: 'OBS_BOUNDS_NONE',
        }
    });
}

I am want to do some more testing before I publish the alpha version but it's gonna be this weekend.

 

[Paul van Dinther]

Hey, just thought I let you know I am still working hard on my app. In order to overcome issues with web socket (the way my app talks to OBS via Web-Socket server) I am forced to pull some tricks to make the connection to ws:// possible from anywhere in your local network. Only connections to "localhost" which means on the same PC would work.

However, I have solved this by wrapping up my web the application in a windows application and the use of brand new WebView2 technology from Microsoft. For the techies... My windows app pulls the resources from my web-server via a secure https:// connection. The small http:// server that I build into the windows app then serves the resources to the WebView2 browser component and from that point network wide connections are possible. It all took a fair bit of R&D.

And since I was geeking out anyway, I decided to see what is involved in wrapping up the same web app into a native Android app. And that works nicely too as shown in the screenshot below. Kinda crazy to develop cross platform software to then wrap it in OS specific coats.
Mind you, this Android solution is unlikely to work for long. Google is closing off the security holes in their chrome browser that allow the ws:// connection to happen.

There is only one solution Palakis, your Web-Socket Server really needs to have wss:// implemented as an end point.

 

[kineticscreen]

Did this die?

 

[bramas]

If I'm not mistaken I did not found the url to the website, which looks great. In the meantime I developed quickly a working website that is very similar (but limited to scene switching).

https://bramas.pages.unistra.fr/obs-gamepad-control/

It is just a static website that connects to your local obs websocket plugin and to your gamepad. You can associate button to scene to quickly switch between scene using your gamepad.

 

[OldGrumpyGamer]

Definitely following this.

Got myself an AKAI APCmini that I'm hoping to use to control OBS. Can be done today with some tools, but I'd like one tool to rule them all. :)

 

[SpikeTurner]

Hey Paul,

Do you have a link for your program here? I'm working on a university undergrad project and would love to reference your design.

 

[Paul van Dinther]

I dusted of this old thing but I recon it still might be of some use. Its a controller interface that sits between game controllers and Midi devices. You can map any control input to functionality made available in OBS Websocket. So you need OBS web socket installed too. It works great with my AKAI APC mini. I'll work on it a bit more and make it more robust. It be nice to finish something. Everything works on the local machine. Beyond that you will run into issues that will require some network knowledge. Stull like reverse proxies and tunneling (I don't even know what that one means :-) ).

It's just a plain old web app. I renamed it to "OBS Control Room". Ditched the Windows EXE shell. It needs more spit and polish and definitely documentation. Come download the tiny 1Mb zip on Github. You can plonk it straight onto your web server or even your windows desktop. Just open index.html and it runs. No build required. It's pure javascript not fantasy script backed by 40 thousand other files.

Or just open the app here and if you want a local installed copy just install it like you do with PWA web apps.