Question / Help Programmatically detect when OBS crashes

dawggpie

New Member
Is there a way to programmatically detect when OBS has crashed and popped up the "Woops, OBS has crashed!" modal? The problem I'm having is that I run OBS programmatically via a node script to record game footage. Once in a while OBS will crash, but I don't have a good way to detecting when OBS is in this state. For my particular use case it would be better if the app crashed and closed, that way I could check if the exe is running and if it's not, restart it. Given the way the app currently works, is there any work around I could use to detect when it's in the crash state so that I can force kill the exe and restart it? Currently I've having to kill the app and restart it every time I want to record, just in the chance that it's in the crash state, which seems a bit inefficient. Thanks.
 

WizardCM

Forum Moderator
Community Helper
The better solution would be to figure out the crash so that it doesn't happen.

Go to:
Windows: %APPDATA%\obs-studio\crashes
Linux: ~/.config/obs-studio/crashes
Mac: ~/Library/Application Support/obs-studio/crashes

Upload the desired crash log, usually the latest, and we'll see what we can do.
 

dawggpie

New Member
That's a good idea! I'm using the obs-websocket plugin. Not sure if that has anthing to do with it. Thx.
 

Attachments

  • Crash 2019-01-19 18-56-00.txt
    41.5 KB · Views: 22
  • Crash 2019-01-19 18-57-35.txt
    40.9 KB · Views: 12
  • Crash 2019-01-19 19-20-33.txt
    41.5 KB · Views: 8
  • Crash 2019-01-19 19-42-27.txt
    41.5 KB · Views: 10
  • Crash 2019-01-19 20-23-57.txt
    40.8 KB · Views: 7
  • Crash 2019-01-19 20-26-28.txt
    40.8 KB · Views: 7
  • Crash 2019-01-19 21-02-33.txt
    40.8 KB · Views: 12
  • Crash 2019-01-19 21-19-42.txt
    40.8 KB · Views: 9
  • Crash 2019-01-19 22-13-48.txt
    41.5 KB · Views: 9
  • Crash 2019-01-19 23-31-35.txt
    40.9 KB · Views: 12

dawggpie

New Member
I believe that we're on the latest or very close to the latest Nvidia drivers. We're using gtx1080 cards. Is there something in the crash logs that indicates specifically a driver issue? Thanks.
 

dawggpie

New Member
Attached is the example of a crash log and it's related log file. Also attached is a log file of a session that worked as expected. The 2 logs look the same up until the point where it adjusts some audio buffering and tries to hook the display so sounds like you might be on the right track.
 

Attachments

  • Crash 2019-01-23 10-06-41.txt
    40.3 KB · Views: 15
  • 2019-01-23 10-05-15 (good).txt
    9.5 KB · Views: 17
  • 2019-01-23 10-07-17 (crash).txt
    5.4 KB · Views: 8

WizardCM

Forum Moderator
Community Helper
So, looking at the logs, here's what's interesting:
- You start/stop recordings quite often, and to do that a new websocket connection is opened. Any particular reason you don't use the same websocket connection to send new commands? That's kind of the whole point of a socket.
- I see you're using advanced ffmpeg output. It's possible that you have custom flags that are causing things to misbehave. Not sure about the best ways to troubleshoot this, but I'd start with something like Simple output mode and seeing if it still crashes.
- Recordings seem to be short and quick in this example - how long are they usually? It might be that obs-websocket doesn't respect the cooldown that OBS normally gives itself when stopping a recording so that it can be finalised.

Next, let's look at the timing of the crash:

Code:
10:06:40.668: [obs-websocket] new client connection from ::1:57125
10:06:40.772: Output 'adv_ffmpeg_output': stopping
10:06:40.772: Output 'adv_ffmpeg_output': Total frames output: 479
10:06:40.772: Output 'adv_ffmpeg_output': Total drawn frames: 490
10:06:40.772: ==== Recording Stop ================================================
~~~ Crash occurs somewhere here - Date/Time: 2019-01-23, 10:06:41
10:06:41.767: [obs-websocket] new client connection from ::1:57126
10:06:42.775: [obs-websocket] new client connection from ::1:57128

It seems to happen after the recording is stopped. This strengthens the possibility that it's related to the advanced ffmpeg output's configuration.
 

dawggpie

New Member
Thanks for the feedback. That's great input.

I'll have our dev rework the websocket code. Not sure why it was done that way initially.

I've changed the recording settings to use the "standard" configuration now. We're using the NVENC Nvidia hardware to do the encoding (actually testing out the new beta obs that allows the encoder to read the render buffer directly from the gpu memory rather than having to go through system dram). We weren't using any custom encoding flags. Most of the parameters in the config ini look the same, but, having said that, I haven't seen a crash since I swapped over to the "standard" setting, so something there may have been the issue.

That's correct, most of our recordings tend to be in the 15-60s range. We're extracting game highlight moments. This cooldown situation is interesting. It looks like obs-websocket kicks out "RecordingStopping" and "RecordingStopped" events. Any idea of the "RecordingStopped" event fires after the cooldown period? We've definitely into into situations where the outputted MP4 files are corrupted, I believe, because the mp4 header info hasn't been written properly. Ffmpeg will show the error "moov atom not found". I wonder is not respecting the cooldown period is the cause of this.
 

WizardCM

Forum Moderator
Community Helper
Thanks for the feedback. That's great input.

I'll have our dev rework the websocket code. Not sure why it was done that way initially.

I've changed the recording settings to use the "standard" configuration now. We're using the NVENC Nvidia hardware to do the encoding (actually testing out the new beta obs that allows the encoder to read the render buffer directly from the gpu memory rather than having to go through system dram). We weren't using any custom encoding flags. Most of the parameters in the config ini look the same, but, having said that, I haven't seen a crash since I swapped over to the "standard" setting, so something there may have been the issue.

That's correct, most of our recordings tend to be in the 15-60s range. We're extracting game highlight moments. This cooldown situation is interesting. It looks like obs-websocket kicks out "RecordingStopping" and "RecordingStopped" events. Any idea of the "RecordingStopped" event fires after the cooldown period? We've definitely into into situations where the outputted MP4 files are corrupted, I believe, because the mp4 header info hasn't been written properly. Ffmpeg will show the error "moov atom not found". I wonder is not respecting the cooldown period is the cause of this.

There is a known issue on the NVENC beta of OBS where stopping the recording sometimes breaks things (betas are a good way to test that sort of thing). I'd try it with a non-beta in the meantime.

If you're extracting game highlight moments, OBS does have a replay buffer - it saves the last x seconds (you choose in the settings) to a file. You simply turn on the buffer, then hit a hotkey (or trigger a function using obs-websocket) to save the last few seconds of the buffer to a file. Might be better suited than constantly restarting encoding.

My understanding is that yes, RecordingStopped should only be fired once the file has been fully saved. If you're running into corruption issues, I highly recommend not saving to MP4/MOV. FLV/MKV will do the job just as well, and you can always remux the recordings (OBS can do this for you, in the File menu, but the next version will also be able to do it automatically after every file save).
 

dawggpie

New Member
Ah, good to know regarding the beta issue. I'll keep that in mind. We've been running mainly the latest production build. That's the build that generated the crashes we were discussing above.

Check out www.gifyourgame.com. That's basically the use case we're targeting. We're generating game clips mostly from replay files generated by the games, so we're not actually able to use the replay buffer feature. Although, we could probably rework around that functionality. Is that more efficient to extract the recording clip that way vs starting/stopping multiple recordings?

Agreed, mp4 definitely isn't the most reliable container format to export to. So the next version would initially export to, say, FLV then automatically convert to another container like MP4? Could be useful. We do do some post processing on the videos using ffmpeg after the fact so really we should be having OBS export in FLV and converting to MP4 in our post processing.

Thanks for the continued support.
 
Top