NVENC HEVC VBR settings

[DayGeckoArt]

As far as I remember, I recommended you make a reference video and use this reference as input for your encoding tests. So you have "the original" and the encoded version. That's the only thing what is required to use the psnr filter (or whatever other similarity filter is available from ffmpeg).
Create comparison:
ffmpeg -i "reference.avi" -i encoded.mp4 -filter_complex "psnr" -f null - 2> report.log
Then grep the last line from the log, that will look like this:
[Parsed_psnr_0 @ 0000024dec41b280] PSNR y:33.358496 u:39.906199 v:39.881322 average:34.662145 min:32.927402 max:37.659071

Use the "average" value for sorting. You need this, if you really want to identify what is "visually indistinguishable". You cannot rely on your eyes, you can only rely on comparison algorithms.

You can also use vmaf of course, but this is awfully slow.

My goal is visually indistinguishable or close to it but also efficient, so basically the idea is to rely on my eyes. There was one guy who wanted "lossless" and I would say that's a better case for analyzing the videos and taking eyeballs out of the equation.

But I finally did get around to trying FFMPEG and learned a couple things... Posting what I did because it may be useful to folks. I had to read several websites to figure out how to do this, it's surprisingly hard to find simple instructions.

First you need an FFMPEG compiled with NVENC which I found here https://www.gyan.dev/ffmpeg/builds

You can run with a command line or powershell but I set it up as a .BAT batch file placed in my Send To folder.
You get to it by typing shell:sendto in the Windows Explorer bar then copy the .BAT into that folder.

The .BAT :

echo processing %~n1%~x1 with NVDEC and NVENC ConstQP CQ 23

timeout 5

c:\ffmpeg\ffmpeg.exe -hwaccel nvdec -i %~n1%~x1 -c:v hevc_nvenc -rc constqp -qp 23 -pix_fmt yuv444p -c:a copy nvdec_nvenc_cq23_yuv444p_%~n1%~x1

timeout 5

So I just have to right click a file and then select that .BAT file in my Send To menu. Then the batch file gets passed the filename and location
%~n1 is the filename and %~x1 is the extension which gets used by ffmpeg and then to create the output file

-hwaccel nvdec sets FFMPEG to decode the source video using the hardware decoder. If you leave it out you get software decoding. For me the software decoding is actually faster, I get 2.2x realtime vs 1.7x

-c:v hvenc_nvenc sets it to use HEVC and NVENC, if you leave this out you just get your CPU compressing to x264

Then the -rc constqp and -qp 23 and -pix_fmt yuv444p are like my OBS parameters.

The -c:a copies the audio without re-encoding

nvdec_nvenc_cq23_yuv444p_%~n1%~x1 just creates a new file with my description plus the original filename and extension

I did some tests on both my gaming computer and my video encoding one using my super high bitrate drone video, and was surprised to find very different bitrate results. The log shows that FFMPEG sets a Q value which stands for quality, and on one computer it chooses 22.0 and the other it does 29.0. It's similar to the QP and CQ in that higher is lower quality. I can't find any method to set this manually! It may be different because one computer has an RTX 2060S gen 7 NVENC and the other has a Quadro T400 gen 6 NVENC.

Basically the NVENC_HVEC settings may be GPU specific or computer specific, so everybody has to do their own testing.

A good use of the Send To batch file method is re-encoding your existing videos to make them smaller, convert to 1080P, etc

 

[koala]

It's good that you start to standardize your tests, but I recommend to fully automate encoding with varying parameters. With ffmpeg.exe, this can be done with a batch file that fully automates this parameter variation. Collect all the video data in a csv an load the report into your favorite spreadsheet, where you can sort it better. I sorted the result by average psnr.

It would be great to use vmaf instead of psnr, but this takes so long I'm not willing to run that for just a demo. If you want to do it, you get the required AI model (usually missing from Windows ffmpeg.exe distributions) from the vmaf source: https://codeload.github.com/Netflix/vmaf/zip/refs/tags/v2.3.0

Spoiler: batchfile

setlocal ENABLEDELAYEDEXPANSION

set opt=-y -hide_banner -hwaccel cuda
set opt2=-an -vstats -vstats_file
rem set input=C:\Users\alex\AppData\Temp\lossless 2560x1440 2.avi
set input=C:\Users\alex\AppData\Temp\lossless 1920x1080.avi
set csv=report.csv

rem initialize report
echo filename,size mb,bitrate kb,psnr_avg,psnr_min,psnr_max > %csv%

set encoder=hevc_nvenc
set preset=p7
set profile=main
rem for vbr/constqp: upper limit = 200 mbit/s
set bitrate_limit=200m

call :encode "%profile%" "" "" ""

call :encode "%profile%" "cbr" "" 2500k
call :encode "%profile%" "cbr" "" 3500k
call :encode "%profile%" "cbr" "" 6000k
call :encode "%profile%" "cbr" "" 8000k

call :encode "%profile%" "vbr" "" ""

call :encode "%profile%" "vbr" 10 %bitrate_limit%
call :encode "%profile%" "vbr" 19 %bitrate_limit%
call :encode "%profile%" "vbr" 26 %bitrate_limit%
call :encode "%profile%" "vbr" 36 %bitrate_limit%

call :encode "%profile%" "constqp" "" ""
call :encode "%profile%" "constqp" 10 %bitrate_limit%
call :encode "%profile%" "constqp" 19 %bitrate_limit%
call :encode "%profile%" "constqp" 26 %bitrate_limit%
call :encode "%profile%" "constqp" 36 %bitrate_limit%

set encoder=h264_nvenc
set preset=p7
set profile=high

call :encode "%profile%" "" "" ""

call :encode "%profile%" "constqp" "" ""
call :encode "%profile%" "constqp" 10 %bitrate_limit%
call :encode "%profile%" "constqp" 19 %bitrate_limit%
call :encode "%profile%" "constqp" 26 %bitrate_limit%
call :encode "%profile%" "constqp" 36 %bitrate_limit%

call :encode "%profile%" "vbr" "" ""
call :encode "%profile%" "vbr" 10 %bitrate_limit%
call :encode "%profile%" "vbr" 19 %bitrate_limit%
call :encode "%profile%" "vbr" 26 %bitrate_limit%
call :encode "%profile%" "vbr" 36 %bitrate_limit%

call :encode "%profile%" "cbr" "" 2500k
call :encode "%profile%" "cbr" "" 3500k
call :encode "%profile%" "cbr" "" 6000k
call :encode "%profile%" "cbr" "" 8000k

:crf
set encoder=libx264
set preset=medium
set profile=high
set bitrate_limit=

call :encode "%profile%" "" "" ""

call :encode "%profile%" "crf" 10 ""
call :encode "%profile%" "crf" 19 ""
call :encode "%profile%" "crf" 26 ""
call :encode "%profile%" "crf" 36 ""

call :encode "%profile%" "cbr" "" 2500k
call :encode "%profile%" "cbr" "" 3500k
call :encode "%profile%" "cbr" "" 6000k
call :encode "%profile%" "cbr" "" 8000k

exit

rem profile=%1 rc=%2 qp/cq=%3 bitrate=%4
:encode

  set ps=%preset%
  set pf=%~1
  set rc=
  set q=
  set br=
  set filename=out-%encoder%-%ps%-%pf%

  rem handle default rc
  if "%~2"=="" (
    set filename=!filename!-default
  ) else (
    set rc=-rc %~2
    set filename=!filename!-%~2
  )

  rem handle qp parameter for constqp
  if "%~2"=="constqp" (
    if not "%~3"=="" (
      set q=-qp %~3
      set filename=!filename!-qp%~3
    )
  )

  rem handle cq parameter for vbr
  if "%~2"=="vbr" (
    if not "%~3"=="" (
      set q=-cq %~3
      set filename=!filename!-cq%~3
    )
  )

  rem different parameter handling for libx264
  if "%encoder%"=="libx264" (
    set rc=

    rem handle crf parameter for crf
    if "%~2"=="crf" (
      if not "%~3"=="" (
        set q=-crf %~3
        set br=
        set filename=!filename!-crf%~3
      )
    )

    rem no quality parameter for cbr
    if "%~2"=="cbr" (
      if not "%~3"=="" (
        set q=
      )
    )

  )

  rem handle bitrate
  if not "%~4"=="" (
    set br=-b:v %~4
    set filename=!filename!-br%~4
  )

  set out=%filename%.mp4
  set log=%filename%.log
  set log_psnr=%filename%-psnr.log

  if not exist "%out%" (
    ffmpeg %opt% -i "%input%" -c:v %encoder% %rc% %q% %br% -preset "%ps%" -profile:v "%pf%" %opt2% -vstats -vstats_file "%filename%_vstats.log" "%out%" > "%log%" 2>&1
    if errorlevel 1 exit
  )

  call :psnr "%out%" "%log_psnr%"
  call :report "%out%" "%log_psnr%"

exit /b


rem compute psnr between %1 and %input%
:psnr

  if not exist "%~2" (
rem       ffmpeg %opt% -i "%~1" -i "%input%" -filter_complex "[0:v]settb=AVTB[out0];[out0]setpts=PTS-STARTPTS[out0a];[out0a]select=gt(n\,3)[out0b];[1:v]settb=AVTB[out1];[out1]setpts=PTS-STARTPTS[out1a];[out1a]select=gt(n\,3)[out1b];[out0b][out1b]psnr" -f null -v info - 2>&1 | findstr "Parsed_psnr" > "%~2" 2>nul
    ffmpeg %opt% -i "%~1" -i "%input%" -filter_complex "[0:v]settb=AVTB[out0];[out0]setpts=PTS-STARTPTS[out0a];[1:v]settb=AVTB[out1];[out1]setpts=PTS-STARTPTS[out1a];[out0a][out1a]psnr" -f null -v info - 2>&1 | findstr "Parsed_psnr" > "%~2" 2>nul
    if errorlevel 1 exit
  )

exit /b


rem scan data and create report
:report

  rem filename, filesize, bitrate, psnr_avg, psnr_min, psnr_max
  echo filename=%~1

  FOR /F "usebackq" %%i IN (`ffprobe -v error -select_streams "v:0" -show_entries "stream=bit_rate" -of "default=noprint_wrappers=1:nokey=1" "%~1"`) DO set /a br=%%i/1000
  echo bitrate=%br%

  FOR /r %%f IN (%~1) DO set size=%%~zf
  set size=%size:~0,-6%
  echo size=%size%

  for /f "tokens=12,14,16 delims=: " %%f IN (%~2) do (
    set psnr_avg=%%f
    set psnr_min=%%g
    set psnr_max=%%h
  )
  echo psnr_avg=%psnr_avg%
  echo psnr_min=%psnr_min%
  echo psnr_max=%psnr_max%

  echo "%~1",%size%,%br%,%psnr_avg%,%psnr_min%,%psnr_max% >> %csv%

exit /b

Result: