Custom Output (FFMPEG) needs a way to pass arguments and flags before input.

[RushingAlien]

Hello, loving OBS and thanks for all the devs' hard work.

Currently, there is a way to get any kind of codec supported by FFMPEG to work on OBS with the Custom Output (FFMPEG) option. However, I've encountered an issue.

VA-API encoders such as hevc_vaapi and h264_vaapi require that the flag/argument `-hwaccel vaapi -hwaccel_output_format vaapi` be passed before input `-i input.mkv` since VA-API encoders needs VAAPI formatted pixels. Currently in the Custom Output tab of OBS', there are no ways to pass these arguments cause it has to be set before input, having a way to do this will solve this issue and also allow us to choose GPU device `-hwaccel_device /dev/dri*` under the custom output tab which I don't see as possible right now since it needs to be passed before `-i` just like the flags mentioned above.

Because of this, I can't even use VAAPI h264 with the Custom Output option, I have to use the officially supported output methods.

typical usage of VA-API on FFMPEG CLI.
```
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i NVENC_Medium.mkv -c:v hevc_vaapi -profile:v main -b:v 5.5M codec-test/hevc-vaapi.mkv
```
```
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i NVENC_Medium.mkv -c:v h264_vaapi -profile:v main -b:v 5.5M codec-test/hevc-vaapi.mkv
```
performance compared to AMF (VA-API is AMD's open source driver, AMF is AMD's proprietary drivers :

VAAPI HEVC main profile ```
frame= 2500 fps=121 q=-0.0 Lsize= 33570kB time=00:00:49.98 bitrate=5502.2kbits/s speed=2.42x
```
AMF HEVC main profile```
frame= 2500 fps= 93 q=-0.0 Lsize= 33589kB time=00:00:49.98 bitrate=5505.2kbits/s speed=1.87x
``

these results also stay true (that VA-API is faster) with h264
VAAPI h264 main profile```
frame= 2500 fps=107 q=-0.0 Lsize= 33561kB time=00:00:49.98 bitrate=5500.7kbits/s speed=2.14x
```
AMF h264 main profile```
frame= 2500 fps= 92 q=-0.0 Lsize= 33564kB time=00:00:49.98 bitrate=5501.3kbits/s speed=1.85x
```

 

[RushingAlien]

Basically, VA-API encode via Custom Output (FFMPEG) unachievable due to not being able to pass required flags.

The same goes for GPU device selection via Custom Output (FFMPEG)

 

[RushingAlien]

I have also discovered that OBS still uses the nv12 pixel format when using the FFMPEG VAAPI encoder.

It would be prefferable that when the VAAPI encoder is used, that the VAAPI pixel format is used rather than nv12, for performance reasons.

This does mean, things like `-vf 'scale=w=1920:h=1080'` needs to be adjusted to `-vf 'scale_vaapi=w:1920=h:1080'`

```
ffmpeg -hwaccel vaapi -i NVENC_Medium.mkv -vf 'format=nv12,hwupload' -c:v hevc_vaapi -profile:v main -b:v 5.5M codec-test/hevc_vaapi1.mkv
``````
frame= 2500 fps= 84 q=-0.0 Lsize= 33570kB time=00:00:49.98 bitrate=5502.2kbits/s speed=1.69x
```vs
```
ffmpeg -hwaccel vaapi -hwaccel_output_format vaapi -i NVENC_Medium.mkv -vf 'format=vaapi,hwupload' -c:v hevc_vaapi -profile:v main -b:v 5.5M codec-test/hevc_vaapi4.mkv
``````
frame= 1015 fps=121 q=-0.0 Lsize= 13738kB time=00:00:20.28 bitrate=5549.1kbits/s speed=2.41x
```
as you can see there is a huge performance difference, this stays true with both h264_vaapi and hevc_vaapi.

Note that `-hwaccel_output_format` and `-hwaccel vaapi` are both required for `-vf 'format=vaapi,hwupload'` to work.

 

[RushingAlien]

hm.... looking at the code again, the post above is wrong