I also suffered the crashes of QSVhelper.exe at random times. With more demanding games like witcher 3 it happens quite often between 10 - 20 minutes.
So I had a closer look in the source code and identified, that OBS.exe terminates the QSVhelper.exe if it's not responding for 1s. There is IPC communication between this two processes, so at normal operation there should be communication even if there is nothing to encode and OBS.exe will terminate QSVhelper.exe only if it's really stalled. I tried to increase the timeout to 10s but this didn't help, no change of the behaviour.
This shows that QSVhelper.exe permantly hangs and there is no quick workaround.
After terminating QSVhelper.exe OBS.exe exits itself with a crash (function CrashError), there is no crashdump (function DumpError).
I downloaded a quite new Intel Media SDK and replaced the existing includes and libmfx with this but it doesn't helped either.
The question is, why does it only happen with WIN 7 and WIN 8 works without problems?
To identify the location where it hangs I first implemented more debug outputs for the Media SDK calls to the QSVHelper.log file.
With this additional logs I saw that SDK encode sync-operations permanently failed with MFX_ERR_DEVICE_FAILED. The only possibility to recover is to reset the Intel encoding device (Quick Sync Encoder). This is now implemented but not tested because I want to find the root cause for this error.
I studied the Intel Media SDk and found, that the quick sync device can operate in two modes, D3D9 and D3D11. In my case it operates always in D3D9 mode and I tried to force it to D3D11 mode, but it doesn't work because it uses special features of DirectX 11.1 which aren't available on Windows 7, only on Windows 8 onwards. And for D3D11 QSVhelper implemented a special D3D11 device and D3D11 frame allocator for the used frame buffers. For D3D9 there is no such device or frame allocator though it's available.
So here is the big difference between Win8 and Win7 in the implementation of the QSVhelper.
And I looked at the implementation for Win7 without the dedicated D3D11 device. There are only pointer to shared frame buffers between QSVhelper.exe and OBS.exe filled from OBS.exe handed over to Media SDK functions! In case of the D3D11 device this buffers are first copied to this device and after that handed over to Media SDK.
And in Media SDK documentation there is absolutely disadvised to implement the coding toolchain in more than one thread and here we hand over buffers possessed by 2 processes!
After identifying this problem I implemented a Direct3D9 device and a Direct3D9 frame allocator analogous to the Direct3D11 device and according to Intel Media SDK samples.
Since this optimization of QSVhelper.exe I could encode video for many hours on Win7 (Intel i7 4770, Games on Nvidia GTX 660ti, 16MB Ram) and the code to reset the QSV device in case of stalled operation never has to be executed...
I attached the Code patch and for those of you who couldn't compile OBS yourself the QSVhelper.exe executable (hope this isn't against the forum rules..)
PS: The QSVhelper.exe ist 32bit even on Win64 builds of OBS because it's known, that binding the 32bit libmfx.lib of Intel Media SDK provides more performance than 64bit bindings. So you can replace the existing QSVhelper.exe with the attached 32bit version on 32bit and 64bit builds of OBS.
So I had a closer look in the source code and identified, that OBS.exe terminates the QSVhelper.exe if it's not responding for 1s. There is IPC communication between this two processes, so at normal operation there should be communication even if there is nothing to encode and OBS.exe will terminate QSVhelper.exe only if it's really stalled. I tried to increase the timeout to 10s but this didn't help, no change of the behaviour.
This shows that QSVhelper.exe permantly hangs and there is no quick workaround.
After terminating QSVhelper.exe OBS.exe exits itself with a crash (function CrashError), there is no crashdump (function DumpError).
I downloaded a quite new Intel Media SDK and replaced the existing includes and libmfx with this but it doesn't helped either.
The question is, why does it only happen with WIN 7 and WIN 8 works without problems?
To identify the location where it hangs I first implemented more debug outputs for the Media SDK calls to the QSVHelper.log file.
With this additional logs I saw that SDK encode sync-operations permanently failed with MFX_ERR_DEVICE_FAILED. The only possibility to recover is to reset the Intel encoding device (Quick Sync Encoder). This is now implemented but not tested because I want to find the root cause for this error.
I studied the Intel Media SDk and found, that the quick sync device can operate in two modes, D3D9 and D3D11. In my case it operates always in D3D9 mode and I tried to force it to D3D11 mode, but it doesn't work because it uses special features of DirectX 11.1 which aren't available on Windows 7, only on Windows 8 onwards. And for D3D11 QSVhelper implemented a special D3D11 device and D3D11 frame allocator for the used frame buffers. For D3D9 there is no such device or frame allocator though it's available.
So here is the big difference between Win8 and Win7 in the implementation of the QSVhelper.
And I looked at the implementation for Win7 without the dedicated D3D11 device. There are only pointer to shared frame buffers between QSVhelper.exe and OBS.exe filled from OBS.exe handed over to Media SDK functions! In case of the D3D11 device this buffers are first copied to this device and after that handed over to Media SDK.
And in Media SDK documentation there is absolutely disadvised to implement the coding toolchain in more than one thread and here we hand over buffers possessed by 2 processes!
After identifying this problem I implemented a Direct3D9 device and a Direct3D9 frame allocator analogous to the Direct3D11 device and according to Intel Media SDK samples.
Since this optimization of QSVhelper.exe I could encode video for many hours on Win7 (Intel i7 4770, Games on Nvidia GTX 660ti, 16MB Ram) and the code to reset the QSV device in case of stalled operation never has to be executed...
I attached the Code patch and for those of you who couldn't compile OBS yourself the QSVhelper.exe executable (hope this isn't against the forum rules..)
PS: The QSVhelper.exe ist 32bit even on Win64 builds of OBS because it's known, that binding the 32bit libmfx.lib of Intel Media SDK provides more performance than 64bit bindings. So you can replace the existing QSVhelper.exe with the attached 32bit version on 32bit and 64bit builds of OBS.
