Automatic Speedrunning Splits

T

tanoysinha

New Member
Brand new to OBS, not brand new to development.

I'd like to develop something that will help speedrunners (www.speedrunslive.com) log splits for their world record runs.

Currently, when people are speed running (trying to beat a video game as quickly as possible), they use programs like wsplit or Llanfair to record their progress against a world record time. Whenever the speed runner gets to a milestone in their game, say beating a boss or acquiring an item, they hit a button that causes the timer program to record a "split". The "split" is allows the runner, and anyone watching, to track the runner's progress compared to the world record time (or a personal best time).

I think it might be possible to automate the process of logging a split. Most split timings occur right around when a video game fades to black. It's fairly straightforward to register a still image to a fully black image and throw an event if the similarity between the two images is high enough. Using that approach, it might be possible to create a plugin that periodically registers the image on an OBS scene and throws an event if it sees that the image is fully black.

I'm just now starting the browse the source and determine if this is possible. Does anyone have any thoughts on a particular approach to achieving automated splits? Does the API support requesting the raw pixels of a given scene? Is there any documentation on starting a plugin using the API, or on the codebase at large that might help me out?

Thanks in advance!!

Tanoy
 
Bl00drav3n

Bl00drav3n

Member
There is no general way of telling what should be a split and what not - it's personal preference. A builtin timer would make more sense, but as you already said, there are very good programs like wsplit and Llanfair that can achieve the same thing and are not complicated to use with OBS.
 
T

tanoysinha

New Member
You might be right.

But you might not be right. It might be the case that issuing splits when the game faded to black would automate the process for most speed runners.

I believe it is an idea worth trying, and I have the development skills to work on it. Any help in understanding the code base more quickly would be wonderful.

Thanks!
 
Bl00drav3n

Bl00drav3n

Member
Okay so your two main problems are:
  • How do determine where to set splits and
  • how to process the screen quickly.

Assuming you just want to check for fade-outs, you would probably need to process a series of frames and look at some kind of brightness of the image. Maybe subsample the captured videoimage to a much smaller offscreen-texture, calculate brightness levels per pixel and map the resulting "brightness-map" back to client memory. Doing this every couple of frames you could then calculate the overall brightness of every frame and look if the value is decreasing over time, ultimately dropping to about zero. The tricky part is doing this with high performance, but the first few things could probably be done efficiently on the GPU while the CPU part runs asynchronously and updates only every couple of frames.

This was just a quick brainstorming, I have no idea if that's gonna work well.

As for how to determine where else to automatically set splits I have no clue though.
 
T

tanoysinha

New Member
Can I sample a set of the pixels on the screen and check if they are black? I could do that once every 60 frames and call it a day.

I can prevent triggering another split for some period of time to make sure I don't trigger a split each frame that the screen is black.
 
Jack0r

Jack0r

The Helping Squad
So in a cutscene you would probably get a split on each scene change, if they use fade to black. other games might show a black screen shortly when you open the menu, if this meets the second you check for black, you would get a split. Play a game like an old AvP and half of the levels are nearly pitch black when your light goes out for a second, next split. Play a game like the old Resident Evil, every door you open shortly switched to black, then showed a door animation, switched to black and back to the level :D
I would suggest you take a look at scene change detection in video programs. ITS HORRIBLE AND NEVER WORKS :)
 
A

AndehX

Member
A good idea imo, would be to take a screenshot from a game, where you want a split to occur, and then the program simply waits until it sees the same "screenshot" in the OBS window, when it does, it will make wsplit "split"
I speedrun Resident Evil 2, and this approach would work fantastic if possible, because as you all know, Resident Evil is full of still scenes, that could easily be compared to a "screenshot" to be used as a split.

Thats just my thoughts.
 
A

AndehX

Member
well thats where the "development" part comes in. Something I don't have the knowledge to do. Im sure someone here with the knowledge, can confirm weather its possible or not
 
K

Kharay

Member
Whereas automating splits is indeed something that needs to be looked into, I would caution about going about it by way of analyzing the visual data. The reason for this is simple -- change one setting in either OBS or your PC or the game, etc, etc, etc, etc. and the data would end up being different. My point being -- it is a very unstable, very fragile way of determining a split should occur or not.

In fact, as far as I'm concerned the most ideal way of going about automating splits would be to do memory analysis instead. As in, simply constantly snoop the memory of the emulator/game in question and trigger the split event based on specific parameters there. But, well, that is something for a different community to discuss, as it would not involve OBS at all. ;)
 
Krazy

Krazy

Town drunk
I'm not even sure how you would do that, though. Most serious speedrunners aren't using an emulator, they are using a capture card for their actual consoles. How are you going to read the console memory over a capture card?
 
K

Kharay

Member
You're not going to be able to do that, obviously. Automated splitting is a very tough subject to address either way. In fact, splitting in general is a very tough subject to touch on as manual splitting is highly inaccurate as well.

For emulators and PC games it would obviously work but for consoles it would indeed be an entirely separate subject to address but I still feel going with visual input to determine the splits is an even less desirable approach than simply manually splitting. I just don't see how it could ever be made to work reliably across all speedrunners' systems and related hardware.

You would end up having to 'teach' the splitting software which keyframe to look for and whereas this is entirely possible, as I said before -- change one thing and the keyframe may simply never even occur anymore or end up being different somehow and not triggering the split. So then you would need to teach this splitting software some manner of heuristics. To actually teach it what the content in that keyframe is, so it can adapt.

Well, have fun with that one. ;)
 
A

AndehX

Member
its entirely possible, in theory, when you setup your splits, you would have a screenshot of the game, when you want the split to occur, and then when the program reads, lets say, a 90% match to your screenshot, it would trigger the split.
Just some ideas to throw around.
 
K

Kharay

Member
Not saying it's not possible. I am saying it's not viable. First of all, there are numerous examples where black/dark frames would not be the keyframe we're looking for. And in those cases even 1 difference between runs -- health, missile count, weather, <insert random factors in here> will make comparing the 2 snapshots incredibly complex.

I would hate for people to spend a huge amount of time exploring this avenue when it should be obvious it is not ever going to work as beautifully as people would imagine it and would only end up costing vast amounts of time without ever even coming close to giving a good return value.
 
A

AndehX

Member
simple answer to the issue you highlighted there would be to only compare a certain area of the screenshot. like where a number changes, or a specific item appears.
 
K

Kharay

Member
So, now we're doing a heuristic analysis of a screen region? This I have to see in action. :P

Anyhow, my example was exclusive, not inclusive. As in -- you would need to omit that particular area, within the examples I mentioned. But I remain unconvinced. I don't believe this can be made to work reliably. Which is the first requirement in recording splits and keep track of records, that the data is reliable. Convenience, ease of use are of the least concern.
 
dodgepong

dodgepong

Administrator
Community Helper
If someone wants to make this plugin, go for it, but I highly doubt that the OBS team will create this. It sounds like an extraordinarily difficult problem that may or may not be reasonable or even feasible to accomplish, but if someone wants to take a crack at it, then by all means, go for it.
 
You must log in or register to reply here.
Top