Bug Report Encoding overload is tied to HDD speed

[linpao]

I used to always get a couple of hung frames while recording, but just accepted it as normal. Even though it didn't make much sense because the encoder (NVENC) wasn't overloaded in the slightest (~25% usage in task manager on a Turing card).

After trying to record to an HDD with very bad response times I realized what (I think) was actually happening: If OBS can't write the frame fast enough onto the recording device, that registers as an "encoder overload". The message isn't that great of course because it doesn't tell users where the actual problem is. So a separate message for this issue would be great.

What would be even greater however, is buffering the recording output and not halting if the device doesn't immediately react. Because almost any HDD can easily sustain even 100 or 200 mbit/s writes (that's only 25 MiB/s after all), but many HDDs won't be able to write every single frame within 16.6 ms.

If OBS already does that something seems to be broken about it, because recording to an SSD I get exactly 0 skipped frames due to encoder lag where I got like ~200 before (or up to 30% with a slow HDD (which would've still been able to sustain 200 mbit/s sequential writes without a problem though), making the recording basically unwatchable).

 

[linpao]

Is there any interest in fixing this? I guess these guys had the same problem:

https://obsproject.com/forum/threads/encoding-overload-nvenc.95330/

https://obsproject.com/forum/threads/issues-with-encoding-lag-nvenc.66516/

 

[Selphadur]

Can confirm encoder overload using HDD too (OBS 24.03).
I tried every option, and would get like 3% encoder overload, while encoder utilization sits at 15%.
Used CQP from 15 to 20 (tried VBR too at around 50-80mbps), 1080p60, 1080p, 1440p, old/new nvenc (and also x264 ultra fast!). Same results. Was messing around for like a day with no results.
Then tried saving to SSD with same settings and voila, overload around 0.3-0.5% (with uncapped fps at GPU usage ~90-95%!).
HDD is connected via USB 3.0, through dedicated adapter (desktop SATA hdd). Using HD Tune here are benchmark results of this HDD:
Minimum - 31.8 MB/s
Maximum - 135 MB/s
Average - 88.8 MB/s
Access Time - 22.6ms
Burst rate - 138.8 MB/s
As you can see it is more than enough to record even at around 200Mbps.
Could it be because of Access Time of 22ms?

My PC specs:
16gb ddr4 2933mhz
9600k @ 4.8ghz
rtx 2080
Nvme system disk
Win 10 1909 with all gaming stuff/modes off
144hz monitor

At first I thought it was a problem of uncapped fps. Capped fps at 60, turned vsync on. And guess what? Encoder overload became even bigger with around 5% (while encoder usage was barely 10-15%).
Presets were maximum with psycho and other stuff on. Or at quality with everything else off. Same results!

The problem is writing to HDD!
Either it is a problem how OBS writes data or idk.
I used a year or so ago to record to HDD and I do not recall such problems.

Oh and by the way, newest Nvidia drivers, tested with Ultra low latency, Low latency and Low latency off with absolutely same results! Also running OBS as admin with turning game mode to on, or running as admin without game mode turned on - same results as without running as admin!

 

[koala]

HDD is connected via USB 3.0

This will probably be the cause your issues, because removable external drives connected through USB use no write cache with Windows unless explicitly set up. If an app orders the operating system to write data to a drive that is not write cache enabled, the write operation only completes when the drive has really written the data to the spinning disk. Usually, this is really short, but sometimes the disk has to reposition its heads, or write other data, so the delay is such that it makes OBS lose a frame or two.
To enable write cache: https://www.maketecheasier.com/disable-disk-write-caching-windows10/
If you enable write cache this way, make sure you disconnect through the "Safely remove hardware" system tray function before you unplug the drive. If you don't do this, the file system might get corrupted up to total data loss and the files that were last written might become incomplete and broken.

 

[Selphadur]

I did enable write cache, and even turned off write cache buffer flushing (forgot to mention this in my previous post).
I tested without write cache, with cache enabled, with cache + turned off cache buffer flushing. All results were absolutely same. No change at all. Note that I am also using Ultimate Performace mode, which disables all power saving features.
Even tried recording to microSDHC UHS-I microSD card (read ~95MB/s, write ~20MB/s). The results were even worse, even though access time was 0.7ms. While using SSD (Sata or NVME) I get no more than 1% skipped frames, which actually most of the time is around 0-0.3%.
Probably issue lies either in how OBS records to external drives or how Windows actually handles these things.

 

[koala]

There might be some kind of hardware issue with your machine, or the USB bus makes USB-connected drives in general inferior for recording with a sustained data rate. I was always able to record flawlessly to internal spinning disk (SATA), although I never recorded directly to external drives. I use the "indistinguishable quality" quality setting, which produces a data rate of about 38-60 MByte/s with the stuff I record. I verified the disk I am recording to is able to always sustain a minimum data rate of about 80 MByte/s under all circumstances (maximum was 169).

The only thing I had to record to SSD was a lossless recording that required about 130 MByte/s.

However, I just remembered it right now, I always choose NAS-type HDs for my PC that were advertised for 24/7 usage and for surveillance systems with multiple parallel data streams. Not the ordinary desktop HDs.

On the other hand, it might have to do nothing at all with HD speed. You seem to work at the edge of the available computing speed of your machine. This makes tolerances very small, and even small delays within in one subsystem makes lags appear elsewhere. Usually, you should get 0% dropped frames ever. Don't try to get away with 0.5% lost frames - it's still at the edge. Turn down game settings, resolution, fps until you get 0 dropped frames. Then work out which settings you can raise again and still get 0% dropped frames. Then use the highest settings you found that still produced 0% dropped frames.

 

[Selphadur]

Well it is probably something with USB, or even with USB Sata adapter (it has it's own pcb, so maybe that's an issue?).
And regarding pegging my hardware to max limit, well I tried limiting at 60fps and ecoding lag was even worse (with GPU and CPU usage at around 50%, so it is probably not bandwidth related whatsoever).
I have no problems writing to SSD without any frame limiter pegging my GPU at 99% (using only Turing asics for encoding: no psycho, only quality mode instead max and etc). And those 0.3% are usually when loading new chunk let's say in Space Engineers (which is pegging CPU at ~99% at that moment). Usually nothing noticeable in the recording.
I came to a conclusion to just write to SSD in chunks (around 1 hour per chunk) and then remuxing to HDD when stopping after 1 hour. This should fix having limited SSD storage.
And yes, that HDD is like 5 or more years old. But if I remember correctly I used it internally (or maybe it was just another more recent HDD, can't remember).
There should not be any hardware related problems as i use Z390 motherboard from MSI and other components are pretty high end too. Only thing from hardware issues that may be related are probably with that internal SataHDD -> USB sata case/adapter, or an old HDD.
But good to know that with decent HDD (using NAS type or whatever) there are no issues.

 

[Sergux]

This will probably be the cause your issues, because removable external drives connected through USB use no write cache with Windows unless explicitly set up. If an app orders the operating system to write data to a drive that is not write cache enabled, the write operation only completes when the drive has really written the data to the spinning disk. Usually, this is really short, but sometimes the disk has to reposition its heads, or write other data, so the delay is such that it makes OBS lose a frame or two.
To enable write cache: https://www.maketecheasier.com/disable-disk-write-caching-windows10/
If you enable write cache this way, make sure you disconnect through the "Safely remove hardware" system tray function before you unplug the drive. If you don't do this, the file system might get corrupted up to total data loss and the files that were last written might become incomplete and broken.

Thank you it work for me :) its not perfect but obs doesnt overload and just lose some frames but that doesnt affect to the recorded video.