Failed to accomplish work with Lua scripting

koala

Active Member
I have a bit of criticism about scripting with lua, because I failed to accomplish the task I set out to do and I am totally frustrated. I'd like to tell about this, because I feel there is room for improvement.

This was my first (and probably last) try to automate a workflow from within OBS. I'm familiar with programming and scripting, automating things with scripting is actually a major part of what I make a living for.

I thought it was a simple task I wanted to accomplish, and I was sure it was the right thing to do from within OBS.

Task:
Mark a rectangle section on the preview with some existing entity, for example with an empty group. This is easy and already there: just create an empty group, resize and position it.
When recording finished, I wanted to read the coordinates and size of that scene item, then externally call ffmpeg.exe to do something with the coordinates in the just finished recording file (I wanted to extract that part of the video and export it as animated gif)
This is an improvement for postprocessing workflows that regularly create animated gifs (or other exports) from some part of a scene. OBS doesn't support that natively, because you cannot just designate only a part of the canvas for output with with mouse as easy as it is to resize a source. Resizing the canvas isn't feasible: it's too tedious and you lose the big picture.

So it is this:
- create callback for "recording stopped"
- at callback, get marked coordinates and call a postprocessing tool with filename and coordinates

I chose Lua because of the simplicity. I assumed no more than 50, perhaps 100 lines of code, and certainly no more than 200.

I found the tools I was given tedious to use and almost undocumented. The only really documented things are the 5 generic callbacks a script is hooked into OBS.
It took hours to get me to the point where I had a tiny configuration GUI to select a scene and a source - the source I wanted to get the coordinates and size for later. Script size so far: about 5k and 150 lines of code.
Also included was the (still empty) callback that's called at recording end.

Then I searched the API for the file name of the recording, however I was unable to find it. First, I expected it somewhere in the "output" calldata that was given to the callback, but I was unable to extract the output_t item from the "output" value. The calldata_ptr() didn't work (type mismatch). There is a corresponding calldata_source() and calldata_sceneitem() to do this with source or sceneitem callbacks, or other calldata_xxx() for scalar values, but not for output_t. There is some generic calldata_cast(calldata, name, type) missing to get arbitrary types, not only void*.

Next thing was to ignore the calldata but just get the output object directly. I got the output object, but that was all. It was just an opaque object with no usable properties (for my purpose).
I googled a whole evening to find examples for how to find the properties of a file output, but nil. I scanned all the lua scripts in the resources section of the forum, but nobody solved this issue. The api documentation itself is bare. It's only listing function names but don't explain how to use it, let alone examples. The documentation doesn't tell how to use the api, it expects you know how to use it.

Next, without the filename, I wanted to just search for the newest media file in the recording output path. I found a directory scanning function, but the documented os_stat() to read the file timestamps (for sorting) simply wasn't there. Lua itself doesn't provide directory access.

In the end, I was only able to grab the coordinates and os.execute() some external program with that coordinates. Everything else has to be done externally: scan the directory for the newest file in the hope it is the last recording, then do something with it.

I was so frustrated I simply gave up at that point and deleted everything I made so far, because it would all be a big ugly hack only consisting of workarounds.

My main complaints:
  • the lua api is too lowlevel.
  • the lua api isn't documented. Instead, the C api is documented, but it is different to the lua api, because you use it differently in both languages.
  • there are no examples. Google shows usage examples, but no generic examples for every api function as it is common for api documentation.
  • the lua api is an incomplete port of the C api
  • the scripting api lacks basic essential functionality that is available to C functions but not to scripting. (or it isn't documented enough to enable me to find it)
  • there are blobs of opaque data that may contain useful information, but it's not available (or not documented enough to enable me to find it).
  • having to deal with types isn't appropriate for scripting languages like Lua. It's expected that types are handled by the runtime, hidden and implicit.
Google provided nice examples of how to manipulate image data with lua, but beyond that you're lost. Things you'd expect you can automate with scripting weren't possible. There is no higher level functionality available you expect with scripting.
There is a great example for how to provide high level access to low level functions: Autohotkey. You can even call Windows API functions directly, and you never have the feel you only mimic C with its pointers and buffer allocations.
But for what is the lua api good for? You can as well (and much more easily) just write C code and have the whole system available with it.

In essence, there is an automated handover to postprocessing tools missing. You cannot collect and save meta information about the circumstances of a recording and give this to a postprocessing tool in an automated way. Not even a filename is available.

I apologize for the rant.

Alex
 

upgradeQ

Member
There is documentation on scripting - https://obsproject.com/wiki/Getting-Started-With-OBS-Scripting
and on architecture https://obsproject.com/docs/ , you read those very carefully and practice along the way.
You cannot yet access raw audio/video via scripting (except shaders in Lua) so ignore details about them in the docs.
If you put more effort into reading docs, code, forum posts etc., (days or weeks) you'll get past the learning curve with working knowledge of internals.
Your search situation was really close to be resolved, instead of properties you should've been focusing on settings.
You can get a filename of the recording using this extension
with this snippet:
Lua:
local r = obs_frontend_get_recording_output()
local d = obs_output_get_settings(r)
print(obs_data_get_json(d))
obs_data_release(d)
obs_output_release(r)

This will show obs_data structure in json form, where you'll see "path" as current recording filename.

11:56:01.808: [FFmpeg aac encoder: 'simple_aac_recording'] bitrate: 192, channels: 2, channel_layout: 3
11:56:01.808:
11:56:01.816: ==== Recording Start ===============================================
11:56:01.816: [ffmpeg muxer: 'simple_file_output'] Writing file 'R:/my_path/2022-08-21 11-56-01.mkv'...
11:56:03.901: [Lua: console.lua] {"muxer_settings":"","path":"R:/my_path/2022-08-21 11-56-01.mkv"}
11:56:07.618: [Lua: console.lua] {"muxer_settings":"","path":"R:/my_path/2022-08-21 11-56-01.mkv"}
11:56:10.001: [Lua: console.lua] {"muxer_settings":"","path":"R:/my_path/2022-08-21 11-56-01.mkv"}
11:56:12.656: [ffmpeg muxer: 'simple_file_output'] Output of file 'R:/my_path/2022-08-21 11-56-01.mkv' stopped
11:56:12.656: Output 'simple_file_output': stopping
11:56:12.656: Output 'simple_file_output': Total frames output: 638
11:56:12.656: Output 'simple_file_output': Total drawn frames: 651
11:56:12.656: ==== Recording Stop ================================================
11:56:14.568: [Lua: console.lua] {"muxer_settings":"","path":"R:/my_path/2022-08-21 11-56-01.mkv"}
11:56:56.034: [Lua: console.lua] {"muxer_settings":"","path":"R:/my_path/2022-08-21 11-56-01.mkv"}

Note: there will be new filename once recording starts

Bonus:

Lua:
if obs_frontend_recording_active() then
local r = obs_frontend_get_recording_output()
if not t.sh then t.sh = obs_output_get_signal_handler(r) end
signal_handler_connect(t.sh, "file_changed", function(calldata) t.path_file = calldata_string(calldata, "next_file") end)
local d = obs_output_get_settings(r)
print(obs_data_get_json(d))
obs_data_release(d)
obs_output_release(r)
while obs_frontend_recording_active() do sleep(1.7) print(t.path_file) end end

This works on OBS Studio 28+ with split file recording feature and shows new file handle every 1.7 seconds.

09:19:52.266: ==== Recording Start ===============================================
09:19:52.266: [ffmpeg muxer: 'adv_file_output'] Writing file 'R:/my_path/2022-08-21 09-19-52.mkv'...
09:19:53.519: [Lua: console.lua] {"allow_overwrite":false,"allow_spaces":true,"directory":"R:/my_path","extension":"mkv","format":"%CCYY-%MM-%DD %hh-%mm-%ss","max_size_mb":30,"max_time_sec":0,"muxer_settings":"","path":"R:/my_path/2022-08-21 09-19-52.mkv","reset_timestamps":false,"split_file":true}
09:20:00.862: [ffmpeg muxer: 'adv_file_output'] Changing output file to 'R:/my_path/2022-08-21 09-20-00.mkv'
09:20:02.102: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:03.819: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:05.535: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:07.252: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:08.969: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:10.685: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:12.402: [Lua: console.lua] R:/my_path/2022-08-21 09-20-00.mkv
09:20:13.362: [ffmpeg muxer: 'adv_file_output'] Changing output file to 'R:/my_path/2022-08-21 09-20-13.mkv'
09:20:14.119: [Lua: console.lua] R:/my_path/2022-08-21 09-20-13.mkv
09:20:15.835: [Lua: console.lua] R:/my_path/2022-08-21 09-20-13.mkv
09:20:17.552: [Lua: console.lua] R:/my_path/2022-08-21 09-20-13.mkv
09:20:19.269: [Lua: console.lua] R:/my_path/2022-08-21 09-20-13.mkv
09:20:20.985: [Lua: console.lua] R:/my_path/2022-08-21 09-20-13.mkv
09:20:21.704: [ffmpeg muxer: 'adv_file_output'] Changing output file to 'R:/my_path/2022-08-21 09-20-21.mkv'
09:20:22.702: [Lua: console.lua] R:/my_path/2022-08-21 09-20-21.mkv
09:20:24.419: [Lua: console.lua] R:/my_path/2022-08-21 09-20-21.mkv
09:20:26.135: [Lua: console.lua] R:/my_path/2022-08-21 09-20-21.mkv
09:20:27.852: [Lua: console.lua] R:/my_path/2022-08-21 09-20-21.mkv
09:20:29.569: [Lua: console.lua] R:/my_path/2022-08-21 09-20-21.mkv
09:20:29.729: [ffmpeg muxer: 'adv_file_output'] Output of file 'R:/my_path/2022-08-21 09-20-21.mkv' stopped
09:20:29.729: Output 'adv_file_output': stopping
09:20:29.729: Output 'adv_file_output': Total frames output: 2235
09:20:29.730: Output 'adv_file_output': Total drawn frames: 2248
09:20:29.730: ==== Recording Stop ================================================
09:20:31.285: [Lua: console.lua] R:/my_path/2022-08-21 09-20-21.mkv


References:
https://github.com/obsproject/obs-s...43/plugins/obs-outputs/flv-output.c#L166-L170
https://github.com/obsproject/obs-studio/pull/5371/files
 

koala

Active Member
Thank you for your response. I will look into your examples (I visited most of them already, but they didn't got me there). However, my main issue is that the documentation is not explaining anything, and the API isn't at a level what you expect from a scripting language (binding at runtime and automated cleanup/destruction of handles).
To understand the API, you need to study the OBS source code.

What is missing, is a high level wrapper around the API, with classes and objects that structure the API in a natural way as well as can probably implement implicit reference allocation/deallocation, so you don't need to ever explicitly release anything.
I don't understand why other programmers don't complain about this, but I now understand why there are so few python or lua plugins. I expected a vast amount of them popping up after scripting became available years ago, but there is almost nothing, given the huge audience of OBS.

I resurrected my work and actually found the filename by own research - in the websocket plugin source code. I read somewhere the filename is available from websocket - that was the only hint that it is possible to get that information. I expected some hack to extract it from the depths of the OBS core, but was surprised that it is available openly as setting. Before that, I even found the "path" setting in the flv or muxer output plugin source, but I didn't realize this is a setting that is available to lua with the data API. For me, it looked as this was some internal private setting not available to the outside. This wasn't obvious at all.

The code is now this:

Lua:
-- get current recording filename from current recording output
function get_recording_filename()

    local output = obs.obs_frontend_get_recording_output()
    local settings = obs.obs_output_get_settings(output)
    local filename = obs.obs_data_get_string(settings, "path")
    obs.obs_data_release(settings)
    obs.obs_output_release(output)
    return filename
end

However, I don't feel I will write much more for OBS except my small postprocessing plugin. It's just too tedious.
Currently, it looks like this. One is the plugin, the other is the called batchfile that actually does the postprocessing.
Code can probably be made smaller.
 

Attachments

  • create-animated-gif.lua.txt
    11.1 KB · Views: 27
  • postprocess_from_obs.cmd.txt
    2.9 KB · Views: 22
Last edited:

koala

Active Member
I finally managed to get all things together, and released my script plugin. It's a helper for creating animated gifs or video snippets after a recording stopped. It will extract a section of the screen (OBS preview) and produce an animated gif out of it.
It's not yet approved by the moderators as resource, but you can get it from Github: https://github.com/tertius1/obs-lua-scripts

To get to the bottom of the API with lua, and to see what the API can do for me with lua and what it cannot, the following resources were the most useful for me:
Definitely the most useful resource in terms of productivity was obslua.lua as library in vscode. Currently, I'm looking into generating an updated one for OBS 28 and with better types.
 
Top