Why is it so much harder to get good AMD encoder quality vs ReLive?

T

Thebigcheese

Member
I was having a heck of a time trying to get good quality (for recording purposes) using the AMD hardware encoder in OBS, whether it's h.265 or AV1, and really wanted a separate setup without my face cam, alerts, etc, so eventually I just moved over to using ReLive for the simplicity. Granted it's set to 70 mbps (I found this to be the sweet spot for essentially lossless quality), but the video quality is crystal clear and super sharp, perfect for my use case where I'm going to edit and re-encode to upload to YouTube. However, ReLive has some bugs... I record my mic to a separate track, but that audio file usually ends up being slightly longer (I can get around this by auto-syncing it in Resolve) and, more annoyingly, sometimes clips even though the mic is nowhere close to clipping. I have even confirmed that it's specific to ReLive by recording it with a DAW and with OBS at the same time as ReLive and there's no clipping in any of the other audio files, just ReLive. There's several dB of headroom (at least) when it's clipping in the ReLive audio file. It's not a total deal breaker, but it's pretty annoying, so I went through the hassle of setting up a second OBS instance specific for recording, but now I'm finding that I simply cannot get the quality in OBS to match the quality in ReLive. I've tried h.265 and AV1, I've tried CQP and CBR and HQCBR (with CBR settings also at 70 mbps), but it's always blurrier and noisier than just using ReLive. I have attached my latest log (ignore the overload parts, I saw that and adjusted) and a screenshot of my CQP settings for reference. I'm also using these parameters, though I am not sure how many of them are still relevant as all the information I can find conflicts each other:
MaxNumRefFrames=4
HighMotionQualityBoostEnable=1
BReferenceEnable=true
EnableVBAQ=false
EnablePreAnalysis=true
PASceneChangeDetectionEnable=false
PATemporalAQMode=1
PAHighMotionQualityBoostMode=1
PAFrameSadEnable=true
1716820167727.png

I would really love for either of these solutions to just work without compromises, but that doesn't seem to be the case. Unfortunately, my CPU is only a 12400, which is really not enough for encoding while gaming, and I use the QSV encoder for streaming, so I can't use it for recording. That really limits me to the AMD encoders for recording and as I said, it works well (clipping aside) with ReLive, so it seems like there ought to be a way to make this work. Is this just a problem with the OBS implementation? Any other advice?
 

Attachments

  • 2024-05-27 09-41-20.txt
    43.6 KB · Views: 35
R

rockbottom

Active Member
You log is incomplete but I can see you're not getting consistent performance. The first few encodes had no encoding lag, then you switched to the HQ Preset, some were OK then the encoding lag started even after you switched back to Quality.

Disable HAGS & run both monitors @ 120HZ. Some of your encodes the Profile is set to Main, try these for the test.

09:41:38.099: rate_control: CQP
09:41:38.099: bitrate: 2500
09:41:38.099: cqp: 15
09:41:38.099: keyint: 2 Seconds
09:41:38.099: preset: quality
09:41:38.099: profile: high
09:41:38.099: width: 1920
09:41:38.099: height: 1080
09:41:38.099: params: MaxNumRefFrames=4

I haven't used an AMF encoder in years so I'm not sure where these arguments are an OBS thing or something you set, but they are not working.
09:41:38.099: HighMotionQualityBoostEnable=1
09:41:38.099: BReferenceEnable=true
09:41:38.099: EnableVBAQ=false
09:41:38.099: EnablePreAnalysis=true
09:41:38.099: PASceneChangeDetectionEnable=false
09:41:38.099: PATemporalAQMode=1
09:41:38.099: PAHighMotionQualityBoostMode=1
09:41:38.099: PAFrameSadEnable=true
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property 'MaxNumRefFrames': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: HighMotionQualityBoostEnable': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: BReferenceEnable': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: EnableVBAQ': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: EnablePreAnalysis': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: PASceneChangeDetectionEnable': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: PATemporalAQMode': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: PAHighMotionQualityBoostMode': AMF_INVALID_ARG
09:41:38.099: [texture-amf-h265: 'advanced_video_recording'] Failed to set property '
09:41:38.099: PAFrameSadEnable': AMF_INVALID_ARG

These can be disabled as well
09:41:20.530: Game Bar: On
09:41:20.530: Game DVR: On

Audio is lagging too, sample-rates look OK. System may be struggling but lets see if there's any improvement with the changes above.
09:41:22.351: WASAPI: Device 'Analogue 1/2 (Audient iD14)' [48000 Hz] initialized (source: Desktop Audio)
09:41:22.355: WASAPI: Device 'Analogue 1/2 (Audient iD14)' [48000 Hz] initialized (source: Mic/Aux)
09:41:22.401: adding 42 milliseconds of audio buffering, total audio buffering is now 42 milliseconds (source: Mic/Aux)

09:54:05.343: Max audio buffering reached!
09:54:05.343: adding 917 milliseconds of audio buffering, total audio buffering is now 960 milliseconds (source: Desktop Audio)
09:54:05.343:
09:54:05.364: Source Desktop Audio audio is lagging (over by 30797.01 ms) at max audio buffering. Restarting source audio.

09:58:02.655: Source Desktop Audio audio is lagging (over by 29810.46 ms) at max audio buffering. Restarting source audio.
 
T

Thebigcheese

Member
As I said, ignore the couple in the middle where it was lagging, I was trying out different settings to see if it would look better and it just didn't work at all, so I changed the settings back around until it worked. I had also switched from HEVC to AV1 for the later tests and was definitely getting worse performance with that, so that could be what you're seeing there. Audient is on the latest driver. The issue I'm having isn't with skipping frames or anything, it's with the blurriness of the resulting video, so I'm not sure how changing to 120 Hz would help, especially since it works perfectly with ReLive. The issue is that even if I completely match my settings between ReLive and OBS, at least as best as I can, it still looks worse in OBS. I have removed the arguments, they were something I got from a YouTube video but as I said, information about "best settings" is contradictory and frequently wrong.

I've attached a new log after removing the arguments and adding the one you suggested. This one is just from a single recording pass and I don't think any of those errors are there anymore. I haven't done anything with HAGS yet only because I'm not aware of it having actually caused any issues for me, but if you think it would help with the quality here (or the audio clipping in ReLive, but I doubt that, I've tested so many options and it's always limited to just ReLive), I can try that next. However, I do seem to be getting better results now, maybe those non-functional arguments were somehow messing it up? Either way, I think this will get me what I want out of OBS.
 

Attachments

  • 2024-05-27 14-25-41.txt
    22.8 KB · Views: 32
R

rockbottom

Active Member
Nice! Timings look good now. On the quality side, you should set those monitors @ 120 when using OBS. Trust me, your recordings will look even better than they do now.

Try HAGS, it you don't have any issues or crashes let it go but if you notice any trouble, you know what to do.
 
T

Thebigcheese

Member
Thanks! I'm seeing some people say that MaxNumRefFrames=4 is the default now and therefore pointless, too, but I can mess with that on my own and see if it makes any difference. Certainly doesn't hurt anything this way. The file sizes are considerably larger than ReLive's recordings, but I've found a few spots where they are also very clearly superior quality now, so it is what it is.
 
T

Thebigcheese

Member
Last question, where should I be looking in Media Info to see that? I can't find any reference to MaxNumRefFrames whether I include that parameter or not.
 
R

rockbottom

Active Member
Change the view to Text, Tree or HTML

Video > Format settings &/or Format settings, Reference Frames. Should see 4 in there.
 
T

Thebigcheese

Member
Huh, neither file has any Format Settings listed, that field is simply not there (even with advanced mode enabled). Maybe the OBS encoder isn't including those tags? Oh well, I'll just leave it in, can't hurt anything.
 
R

rockbottom

Active Member
AMD encoder. I get the info with NVENC & I'm fairly certain QSV as well. I agree, leave it in as it is working & not failing like it was before.
 
You must log in or register to reply here.
Top