Tips and tricks for Lua scripts

E

equilibrier

New Member
By the way, I have on question for @bfxdev or @upgradeQ , in LUA there is a garbage collector right ? But as how I figured it out, it will only work for internal variables or specific lua data.
In case of OBS API which derivates from C/C++, I still need to free up resources...

Ok, if this is correct and maybe this will be the cause of one of my scripts actually hanging out OBS, i'm in the area of HOW, right now,
as I tried to use obs.bfree() on a variable that holds a string, and I'm stuck with an error saying that the original C function bfree expects a void* not a 'string'.
Should I make a cast in lua or something like that ?
Is it expected I free up strings (although it surely says that here: https://obsproject.com/docs/reference-frontend-api.html?highlight=replay buffer)
Is it bfree, the recommented practice to do this ? If yes, how should I use it, regarding to my above stated problem ?
and one last question: in LUA for OBS, I would only be concerned in freeing up lists and strings, as I read, right ? Lists are freed with
obs_frontend_source_list_free and strings with bfree/something else I don't know.

BTW, the two strings I tried to free with obs.bfree comes from these lines (it seems it is relevant since one seems to be using LUA functions and not obs (ported from C/C++) functions:
local nextcoll = readAll(<some file I have the name of the collection to be switched to...>)
local current_coll = obs.obs_frontend_get_current_scene_collection()
--[[...some code making the transitions...and then...]]--
obs.bfree(nextcoll)
obs.bfree(current_coll)


Thanks in advance for your answers :). Cheers !
 
E

equilibrier

New Member
Ok, I forgot to say, my readAll function is a custom function, not a lua standard function, and it includes this line: "local content = f:read("*all")" this is why I'm supposing I don't have to free up the "nextcoll" "pointer". Is this correct ?
 
U

upgradeQ

Member
equilibrier said:
Ok, I forgot to say, my readAll function is a custom function, not a lua standard function, and it includes this line: "local content = f:read("*all")" this is why I'm supposing I don't have to free up the "nextcoll" "pointer". Is this correct ?
Click to expand...
Does it include f:close() ?
equilibrier said:
one of my scripts actually hanging out OBS
Click to expand...
In which way ? OBS just stops executing or it shows error message or even does not start ?
 
F

FredJ

New Member
equilibrier said:
Hi, @vaggoscc , lua scripts are executed by default, it all deppends on how you configure the scripting defaults in lua, I mean, what function you give OBS to execute as "main" lua script function.
I'm a newbie in lua, but on a script I adapted, you have:

--- Loaded on startup
function script_load(settings)
print("Loading Next Scene script")
obs.timer_add(checkTransition, 500)
end

there are other default functions like
function script_save(settings)
function script_properties()
function script_update(settings)
function script_defaults(settings)
function script_description()

but these doesn't concern you right now, Moviebox in script_load function I'm using an obs lib timer to trigger the execution of a custom function (this can be yours), in my case, named "checkTransition", at every 500 ms (so, a repeated execution). I think if you just want to Moviebox Pro process a single function, once, you can write something like this:

function myfunc(myargs)
--[[...todo...]]--
end
function script_load(settings)
print("Loading Next Scene script")
myfunc(dummy)
end

I hope this helps ! Cheers and a new year with good thoughts ! Health to all, thanks for this thread !

PS: Maybe this tutorial would also help: https://dev.to/hectorleiva/start-to-write-plugins-for-obs-with-lua-1172
I'm finding it useful for beginners like us.
Click to expand...
I'm a beginner, this post is valuable to me, thanks for the clear explanation.
 
bfxdev

bfxdev

New Member
Wow I was a bit far away during the last weeks and we are now at "page-2" of this forum thread!

In the last weeks I worked on a "Getting started" page for scripting with a first tutorial. Please have a look at the pages Getting Started With OBS Scripting and Scripting Tutorial Source Shake. @upgradeQ you will see that I reference your tutorial (with translation) and your Scripting Cheatsheet.

I hope the pages will be helpful to clarify the most common questions, and show how to build a script from the beginning up to adding properties and improve robustness.

In addition, please be aware that the scripting room on the Discord server is very active. It is a quicker way to get support, especially for beginners.

Regarding your questions @equilibrier, as far as I remember I never had to use explicitly bfree for OBS scripting so far. But the situation is far from simple.

First example:
local nextcoll = readAll(<some file I have the name of the collection to be switched to...>)

Here the complete code is pure Lua no? The readAll function looks like on this page?
As @upgradeQ pointed out, a call to close() may be missing.

In any case you should not need to call some OBS function to free data that was allocated by pure Lua code. The garbage collector will take care of freeing the data as soon as there is no reference any more to it. For strings, in Python and Lua, even if the string is big you should not care about freeing it.

Second example:
local current_coll = obs.obs_frontend_get_current_scene_collection()

Here the API says to use bfree, but like many frontend functions, the Lua binding is implemented manually for Lua, outside of the automatic code generation by SWIG. The function returns a string of chars, not a "collection" as the name seems to imply.

In C the binding looks like:

C: 
static int get_current_scene_collection(lua_State *script)
{
    char *name = obs_frontend_get_current_scene_collection();
    lua_pushstring(script, name);
    bfree(name);
    return 1;
}

The code transforms the allocated area into a new Lua string with lua_pushstring, and then frees the block inside the binding, so here again you should not free it in Lua, only in C, contrarily to what the API says. The created Lua string will be garbage collected.

Hope this helps!
 
U

upgradeQ

Member
It's definitely looks smother in OBS preview rather than on that gif, wonder if there is a way to output gif in high FPS using OBS.
Regarding source shake scripts, I've tested a Lua one and there is a bug when you do this: repeatedly select shaked item, then drag , then deselect and eventually OBS will hang/freeze, on windows code error is - AppHangB1.
Will there be an information about how to retrieve graphics data back to obslua in second part of the guide?
Back to the topic, there is a tip for Lua filters hotkeys:
If you define a hotkey registering logic inside of an update and execute only one time - this will creates hotkeys bindings.
This works because if user interacts with properties, OBS calls to update. The previous method I've used was with timeout timers but it seems that there is a bug (hard to reproduce) with it. @bfxdev you might want to check out source code here where it's used.
Also a brief example to try:
Lua: 
print('restarted event loop')
while true do
  if t.pressed then
  print('yes')
  else print('no');sleep(2)
  end
sleep(0.01)
end
Attach a Console then run above code in it , assign a hotkeys and you will see the effect in the Script Log.
 
bfxdev

bfxdev

New Member
Hum, oh well, I was not expecting to have a bug report so quickly @upgradeQ ! Honestly I could not reproduce the bug, using the final version of the script (given as listing), keeping the hotkey pressed to shake continuously, then selecting, dragging, de-selecting the source many times. I'm using currently the official OBS 26.1.0 64 bits on Windows.

In the second part of the guide, no, I would love to have a simple way to access graphics data from Lua but as you may remember the only way I found to do that was to use a big block of FFI definition to access the texture_data member of the image. That would be way too complicated for a tutorial. This forum thread is a good place for such hacks! The only solution would be to improve the Lua and Python bindings, there is a lot to do.

Or did you find a way to do it simply so meanwhile?

The second tutorial is supposed to show how to create a source filter as video effects in Lua, like what StreamFX does as a plugin. I really would like to limit the scope to documented functions only. Using an image as a texture within a shader is easy, I hope it will be sufficient for the tutorial.

Wow quite crazy this console.lua ! I understand the pseudo multithreading, so you used coroutine to let Lua execute code with infinite loops like a thread. For the hotkeys they are registered in "update" but as well in the "load". Apparently it is not sufficient in "load"? Nice idea to display the log window with error().

I never tried to add a hotkey to a filter, good tip.
 
U

upgradeQ

Member
The shared memory might be a solution, right now I think its doable via plugin on Windows, on Python using
mmap , on LuaJIT writing something like this.Yet it only supports outputting in 250 ms.

load is not called upon initialization with custom Lua sources, only after OBS restarts.
Hotkeys can be also registered automatically in video_tick or video_render.
 
chrisforobs

chrisforobs

New Member
Mega64 said:
Of course, if you now want to add it to a scene, you have to make use of the obs_scene_t and obs_sceneitem_t structures and functions, but that is a whole other can of worms (I can explain it if somebody wants me to though).
Click to expand...

Hi @Mega64 Actually I would be interested in diving into understanding what you mentioned earlier. So I am sort of stuck at trying to make my own script to import overlays I designed outside of OBS (either .pngs, .webms, or .mp4 files). I could probably get away with this by just simply exporting a json file of the entire scene collection but I thought I could learn more about lua and how I can use scripting with OBS to make a nice little GUI to import my overlays for me.

Soooo far...after some research into the OBS documentation and going over some of the work that @bfxdev provided (btw thank you very much for sharing this bounty of information. The html usage inside LUA was very insightful) I was able to sort of scrap up together this small code. I was able to make a button and create a few functions but when I click on the button to create a new scene it just does nothing. I thought it would be fairly straight forward but it seems that I am still missing a larger piece of the puzzle after reading what @Mega64 described.

Here is my code:

Code: 
local logo="my image"

local description = [[
<hr/>
<p>Some descriptions
<hr/></p>]]


-- FUNCTIONS

obs = obslua

function script_description()
    return description
end

function add_scene()
    local source = create_source_object()
    local scene = obs.obs_scene_from_source(source)
    local sceneitem = obs.obs_scene_add(scene, source)

    obs.obs_source_release(source)
    return sceneitem

end

function create_source_object()

    local new_source = obs.obs_source_create("source_1", "Source 1", nil, nil)
    return new_source
end


function script_properties()
    local properties = obs.obs_properties_create()

    local create_new_scene_btn = obs.obs_properties_add_button(properties, "create_new_scene", "Create New Scene", add_scene)

    return properties

end

My main initial issues was that I was not passing any arguments into obs_scene_add() which requires two arguments, a scene object, and a source object. So I have a rough understanding what I needed to do from reading the documentation and some example codes I found around the internet but I am honestly stumped. I believe I created my source and scene objects correctly but honestly I do not know what I am doing.

Some more background: So I came across a script provided from a company called Nerd or Die(or Nod for short) that design overlay stream packages for Twitch Streamers. They basically include this script in their overlay packages where there is json formatted code inside the Lua script and they designed their script in a way that they could parse through the json and import their overlays for a user via their nice gui. I would like to do something similar without have to completely bite off the script they provided that I currently can easily read and copy code from, hence my rabbit hole into trying to learn lua. Any help would be greatly appreciated!
 
chrisforobs

chrisforobs

New Member
chrisforobs said:
Hi @Mega64 Actually I would be interested in diving into understanding what you mentioned earlier. So I am sort of stuck at trying to make my own script to import overlays I designed outside of OBS (either .pngs, .webms, or .mp4 files). I could probably get away with this by just simply exporting a json file of the entire scene collection but I thought I could learn more about lua and how I can use scripting with OBS to make a nice little GUI to import my overlays for me.

Soooo far...after some research into the OBS documentation and going over some of the work that @bfxdev provided (btw thank you very much for sharing this bounty of information. The html usage inside LUA was very insightful) I was able to sort of scrap up together this small code. I was able to make a button and create a few functions but when I click on the button to create a new scene it just does nothing. I thought it would be fairly straight forward but it seems that I am still missing a larger piece of the puzzle after reading what @Mega64 described.

Here is my code:

Code: 
local logo="my image"

local description = [[
<hr/>
<p>Some descriptions
<hr/></p>]]


-- FUNCTIONS

obs = obslua

function script_description()
    return description
end

function add_scene()
    local source = create_source_object()
    local scene = obs.obs_scene_from_source(source)
    local sceneitem = obs.obs_scene_add(scene, source)

    obs.obs_source_release(source)
    return sceneitem

end

function create_source_object()

    local new_source = obs.obs_source_create("source_1", "Source 1", nil, nil)
    return new_source
end


function script_properties()
    local properties = obs.obs_properties_create()

    local create_new_scene_btn = obs.obs_properties_add_button(properties, "create_new_scene", "Create New Scene", add_scene)

    return properties

end

My main initial issues was that I was not passing any arguments into obs_scene_add() which requires two arguments, a scene object, and a source object. So I have a rough understanding what I needed to do from reading the documentation and some example codes I found around the internet but I am honestly stumped. I believe I created my source and scene objects correctly but honestly I do not know what I am doing.

Some more background: So I came across a script provided from a company called Nerd or Die(or Nod for short) that design overlay stream packages for Twitch Streamers. They basically include this script in their overlay packages where there is json formatted code inside the Lua script and they designed their script in a way that they could parse through the json and import their overlays for a user via their nice gui. I would like to do something similar without have to completely bite off the script they provided that I currently can easily read and copy code from, hence my rabbit hole into trying to learn lua. Any help would be greatly appreciated!
Click to expand...


Actually please scratch this, so I was able to figure out how to simply create the scene which would look like the following:
Code: 
obs = obslua

function script_description()
    return description
end

function create_welcome_scene()

    local welcome_name = "Welcome"
    local welcome_scene = obs.obs_scene_create(welcome_name)
    return welcome_scene
end


function script_properties()
    local properties = obs.obs_properties_create()
    local create_new_scene_btn = obs.obs_properties_add_button(properties, "create_welcome_scene", "Import All Scenes", create_welcome_scene)
    return properties

end

I was able to create a new scene this way by passing the create_welcome_scene() function to the obs_properties_add_button(). Thanks to Matt on the Discord channel for helping me out.
 
bfxdev

bfxdev

New Member
Thanks for sharing @chrisforobs ! I saw your messages on Discord but Matt was quicker to answer.

ANNOUNCEMENT: I just finished a new tutorial on video filters in Lua, if you want to have a look!

The script relies on an HLSL effect file using a texture as a dithering pattern and a texture as color palette. It can produce such pictures:

1614753318685.png


1614753347941.png


1614753434714.png


Everything is explained in the very long tutorial.

Now I'm slowly coming back to trying to improve the bindings to Lua and Python.

Somebody on Discord asked how to export a picture from an OBS source and send it to some Docker image for further analysis (based on a neural network). In Python for example, it is possible to grab the picture from a source, have it in a texture in RAM (at least I suppose, using gs_stage_texture), but again, I cannot find a way to access the data for further processing by the CPU, and I would like to avoid using a hack with FFI. Calling gs_stagesurface_map crashes OBS.
 
bfxdev

bfxdev

New Member
I know two ways to access the data:
  • With FFI, similar to what is described in this post: https://obsproject.com/forum/threads/tips-and-tricks-for-lua-scripts.132256/#post-490983. As you can see it requires a lot of FFI code, and the objects are not compatible with the SWIG bindings.
  • Modification of the bindings, i.e. addition of "typemaps" in obspython.i or obslua.i to better manage buffers (SWIG default typemaps cannot understand pointers correctly) and re-compilation of OBS
So yes, it should be possible. If you are the only one to use the script, then try the FFI way. On my side I would like to use it on a larger scale so I'm trying to find a way to improve the bindings.
 
J

John_

New Member
Thanks for the repsonse!

Ideally I would like it to work out of the box with no modficiation to OBS, so the modfication goes a bit out the window.

Is your aim to modify OBS to include these - then attempt to merge into the main OBS release, or is it more a standalone project?
If it is a standalone project, is there somewhere where we can feature request adding in a binding along the lines of: obs.obs_get_texture_data(source)?
 
bfxdev

bfxdev

New Member
Short answer: the same kind of code is not possible in Lua or Python with OBS API functions only, as far as I know.

The problem here is the function gs_stagesurface_map
You will find the function used in the "screenshot" part of OBS you just mentioned (I bet it is used to render the preview in the filter properties dialog window) or in this screenshot plugin.

Actually, it is possible to grab the data and have it ready in a texture. That is what I'm trying to do now in Python (assuming source_name contains the name of the source to render):

Python: 
  source = obs.obs_get_source_by_name(source_name)
  if source:
    obs.obs_enter_graphics()

    source_width = obs.obs_source_get_width(source)
    source_height = obs.obs_source_get_height(source)

    render_texture = obs.gs_texrender_create(obs.GS_RGBA, obs.GS_ZS_NONE)
    stage_surface = obs.gs_stagesurface_create(source_width, source_height, obs.GS_RGBA)

    if obs.gs_texrender_begin(render_texture, source_width, source_height):

      clear_color = obs.vec4()
      obs.vec4_zero(clear_color)
      obs.gs_clear(obs.GS_CLEAR_COLOR, clear_color, 1, 0)

      obs.obs_source_video_render(source)

      obs.gs_texrender_end(render_texture)

      obs.gs_stage_texture(stage_surface, obs.gs_texrender_get_texture(render_texture))

      # Here the stage_surface variable should contain a screenshot of the source picture in RAM,
      #  but there is no way to access the bitmap data!!

    obs.gs_stagesurface_destroy(stage_surface)
    obs.gs_texrender_destroy(render_texture)

    obs.obs_leave_graphics()
    obs.obs_source_release(source)

The call to gs_clear may not be necessary here, but the gs_stage_texture function is the most important.

I understood recently the difference between image, texture, texrender and stagesurface (at least I hope I understood it):
  • A gs_image_file object can be read from disk or created from a string URI, but not written back to disk (unclear why there is no foreseen function, the ImageMagick library provides certainly many functions for that), and its bitmap data can be accessed through the undocumented member texture_data. In Lua, it is possible to access texture_data with FFI and a lot of definition code, but the gs_image_file objects needs to be as well initialized in FFI, such that at the end FFI creates objects that are not compatible with the rest of the script and the long set of C definitions makes it very sensitive to changes in the C code (this is no long-term solution).
  • A gs_texture_t object can be initialized e.g. through an image file and retrieved as gs_image_file.texture to be passed to a shader or used to draw sprites. A texture can be mapped to RAM with gs_texture_map but again not with Lua or Python, possibly with FFI. I understand it such that mapping a texture brings its data to the RAM (copy or just some bound buffer), but normally the data is in the GPU RAM (VRAM).
  • The undocumented texrender helper functions are useful to render to a texture in VRAM. The CPU would not be able to read or modify the data but if the rendered texture is used in a shader like in the Shader Filter plugin, then it does not matter too much.
  • Finally, the stagesurface functions can be used with a rendered texture. First the texture object is retrieved from the rendered texture with gs_texrender_get_texture, then it is "staged" with gs_stage_texture and finally mapped to memory with gs_stagesurface_map.
So gs_stage_texture copies a texture to a staging surface and copies it to RAM. It makes sense for our use-case. But once everything is available in memory, there seems to be no way to access the bitmap data! I did not find so far a way to use gs_stagesurface_map, and don't see another way except maybe FFI (and I would need to spend again a lot of time to design a working version with FFI, I prefer invest time in SWIG).

I spent already hours on that, and even opened an issue on the OBS repository about the use of gs_texture_map.

As you can see, the issue did not get a lot of attention (except by @upgradeQ who knows the problem very well!), and when I try to ask questions on Discord about the technical choices behind the bindings used in Lua and Python, I get no answer. In my opinion the SWIG bindings were added long ago and left out-of-the-box. Some functions were re-implemented manually (frontend, source info, callbacks) in plain C without trying to use the sophisticated mechanisms available in SWIG (typemaps). The set of functions is heterogeneous, not to mention its documentation.

--> Now my plan is to adapt the bindings, and for sure to propose a pull request to the main OBS repository if I get a working version. Currently, calling gs_stagesurface_map leads to a crash of OBS, so I'm quite confident that it would not raise regressions to some existing script if I modify the bindings.
 
J

John_

New Member
Thanks for the comprehensive response.

Forgive me if this is stupid, but wouldn't the easist solution be to add an optional arguement to the screenshot function that allows it to output to a lua/python friendly format instead of downloading to disk?

That way you wouldn't have to adapt any graphics bindings (except the output), and it would only require modification to a single file.

Feel free to tell me I'm stupid and missed something though!
 
bfxdev

bfxdev

New Member
Well there are certainly many ways to solve the issue! The majority of developers in the OBS community would certainly favor developing a real compiled plugin to get rid of the issue..

The screenshot function could be a way, but the function normally just saves a file. In addition it is in C++ and only exposed in the OBS API through a kind of source callback called by the function obs_frontend_take_source_screenshot.
I don't know if this would be possible, but in any case it does not look like an immediate modification. I cannot even trace the complete call chain to the Screenshot and Download functions of the C++ screenshot object from the OBS C API function.

That being said, it is good that you mentioned it because I never thought about using the screenshot function to retrieve picture data! Strangely, the API function is documented as returning void* but defined to return just void (the sign of a previous intent to return bitmap data?). In the header file you can see as well an undocumented function called obs_frontend_get_virtualcam_output , that returns an "output" object. Is it another way to get the data?


Maybe there is a misunderstanding about the term "bindings". What I mean is the way SWIG generates the "wrapper" code to make data conversion between Lua/Python and C, in a file called obsluaLUA_wrap.c attached to a previous post, generated during OBS compilation.

SWIG provides "typemaps" to re-define the code generated for each data conversion. Here is how far I am for the return values of gs_stagesurface_map, just changed in the file obspython.i, no other file was changed:

C-like: 
%typemap(arginit, noblock=1) uint32_t *OUTREF {
  $*1_type temp$argnum = 0;
  $1 = &temp$argnum;
}

%typemap(argout, noblock=1, doc="bytearray") uint32_t *OUTREF {
  $result = SWIG_Python_AppendOutput($result, SWIG_From_unsigned_SS_int((temp$argnum)));
}

%typemap(in, numinputs=0) uint32_t *OUTREF "// typemap(in)";

%typemap(arginit) uint8_t **OUTREF = uint32_t *OUTREF;
%typemap(in) uint8_t **OUTREF = uint32_t *OUTREF;

%typemap(argout, noblock=1) uint8_t **OUTREF {
  int size = gs_stagesurface_get_height(arg1)*(*arg3);
  $result = SWIG_Python_AppendOutput($result, PyByteArray_FromStringAndSize((const char *)temp$argnum, size));
}

bool gs_stagesurface_map(gs_stagesurf_t *stagesurf, uint8_t **OUTREF, uint32_t *OUTREF);

I don't want to go into the details of the code now (still ugly workarounds inside) but it seems to work. The function where the typemaps are applied is redefined with different names for the arguments (last line of the code). With these typemaps, the Python code looks like:

Python: 
res,data,linesize = obs.gs_stagesurface_map(stage_surface)

What a change! The typemaps transform the way the parameters are converted:
  • Pointers given as input parameters are considered optional and ignored if given
  • In addition to the returned boolean (normal return value of the function), a bytearray with the bitmap data and an integer containing linesize are returned
Transforming arguments passed by reference into additional return values is one way to solve the issue. In general, there is no notion of passing a pointer to a function in Python/Lua to fill the value of an integer, so it seems to be the classical way to return several values.

Another way would be to create a helper object, say "struct bitmap_buffer" that holds the different parameters and can be modified when passed by reference.

Another way would be to pass Python/Lua lists or arrays as arguments, and fill the first element with the expected value.

I'm still evaluating what would be the best solution. Do you have a preference?

To conclude, I prefer improving the typemaps because it is a very generic method. In many cases the bindings generated by SWIG are OK and we do not want to touch them. For some functions, typemaps can be changed and applied on each function that needs it (and the functions we mentioned are by far not the only ones that need an adapted typemap to be usable at all).

The risk of regression is minimized because we can choose which functions are affected in Python/Lua, and there is no change on the C side.
 
J

John_

New Member
I think that your python bindings are a lot cleaner than an FFI integration.

I was thinking of modifying the C code to accept a 2nd argument, a boolean: "should_download".

This would default to true, and if set to false instead would store the screenshot in RAM. It would then fire off a signal (using the global signal handler) to let you know the screenshot was ready, and you could then use the proc handler to get the data. This is similar to how replay buffers are handled:
local replay_buffer = obs.obs_frontend_get_replay_buffer_output()
local sh = obs.obs_output_get_signal_handler(replay_buffer)
obs.signal_handler_connect(sh, "saved", saved_function])

Then in the 'saved function':
local replay_buffer = obs.obs_frontend_get_replay_buffer_output()
local cd = obs.calldata_create()
local ph = obs.obs_output_get_proc_handler(replay_buffer)
obs.proc_handler_call(ph, "get_last_replay", cd)


It would basically be modifying the C++ code to enable/disable downloading and adding in the signal/proc handler events.

This way it would be a single .cpp file to update with changes to expose all the required functionality, it wouldn't break anything (as it's an optional argument) etc.

I don't know what output format would be best.
 
You must log in or register to reply here.
Top