H.264 encoding overload, H.265 encoding works fine

M

mudspud

New Member
Hi, recently I started experiencing encoding overload with H.264 hardware encoding (AMD). In the past I was able to record games at 1440p 60fps with H.264, for some reason I am no longer able to do so, I cannot even record an empty scene without encoding lag unless I reduce the resolution or framerate. I cannot think of anything that could have changed for me to suddenly be unable to do something I was previously able to do with ease. I tried downgrading to a much older version of OBS and it did not fix anything.
I don't really know anything about how encoders work, but I think it is really strange that the H.265 encoder works without issue, when H.264 is unusable. Also, I cannot simply resolve my problem by switching to H.265 encoding as I cannot use that for streaming, and I also have to change it back to H.264 with a program such as VLC to be able to use the recordings in my editing software, which is a really slow process.
Log file
 
Suslik V

Suslik V

Active Member
Try to past the log-link into the OBS log analyzer: https://obsproject.com/tools/analyzer
There you may find some clues to the problem.
Because you are using hardware (HW) based encoder, then there is guide:

Encoding Performance Troubleshooting | OBS

OBS Knowledge Base. Learn best practices to solve encoding performance issues
obsproject.com obsproject.com
H.264 and H.265 are different encoders and cannot be compared directly - neither by output result nor by settings. Also, performance of the HW based encoder highly depends on video driver of the device.

Significant changes in the OBS code for AMD encoders started from OBS v28.
 
R

rockbottom

Active Member
Your monitor should be running in 8-bit color. HDR is enabled somewhere. If it's not enabled in Windows, your monitor may have an auto HDR feature &/or it's enabled in the AMD Control Panel.
22:24:36.306: name=LG ULTRAGEAR
22:24:36.306: pos={0, 0}
22:24:36.306: size={2560, 1440}
22:24:36.306: attached=true
22:24:36.306: refresh=144
22:24:36.306: bits_per_color=10
22:24:36.306: space=RGB_FULL_G22_NONE_P709

Color Range setting in OBS should be Partial, not Full.
22:24:36.415: YUV mode: Rec. 709/Full
 
R

rockbottom

Active Member
There should only be (1) Display, Game or Window capture per Scene. They can interfere with each other, some housekeeping is in order.
22:24:36.948: Loaded scenes:
22:24:36.948: - scene 'Scene':
22:24:36.948: - source: 'Display Capture' (monitor_capture)
22:24:36.948: - scene 'Fangames 600':
22:24:36.948: - source: 'input viewer' (window_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'Livesplit' (window_capture)
22:24:36.948: - source: 'decession: volume 6, obs -30' (text_gdiplus_v3)
22:24:36.948: - source: 'Display Capture 3' (monitor_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - scene 'Fangames 608':
22:24:36.948: - source: 'input viewer' (window_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'Deadsplit' (window_capture)
22:24:36.948: - source: 'Display Capture 2' (monitor_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'livesplit 608' (window_capture)
22:24:36.948: - scene 'Game':
22:24:36.948: - source: 'Gaming' (game_capture)
22:24:36.948: - scene 'Terraria':
22:24:36.948: - source: 'TML' (game_capture)
22:24:36.948: - source: 'Vanilla' (game_capture)
22:24:36.948: - source: 'Vanilla Audio' (wasapi_process_output_capture)
22:24:36.948: - scene 'I Wanna Maker speedrun':
22:24:36.948: - source: 'I Wanna Maker' (game_capture)
22:24:36.948: - source: 'input viewer' (window_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'livesplit' (window_capture)
22:24:36.948: - source: '2160x1216' (text_gdiplus_v2)
22:24:36.948: - source: 'Application Audio Capture (BETA)' (wasapi_process_output_capture)

Not nearly enough bits for 2k/60fps. Change your rate control to CQP for recording, B Frames can be zeroed too.
22:24:40.675: [texture-amf-h264: 'advanced_video_recording'] settings:
22:24:40.675: rate_control: CBR
22:24:40.675: bitrate: 2500
22:24:40.675: cqp: 20
22:24:40.675: keyint: 250
22:24:40.675: preset: quality
22:24:40.675: profile: high
22:24:40.675: b-frames: 3
22:24:40.675: width: 2560
22:24:40.675: height: 1440
22:24:40.675: params: (none)
22:24:40.676: ---------------------------------
22:24:40.676: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: stereo, track: 1
 
M

mudspud

New Member
rockbottom said:
There should only be (1) Display, Game or Window capture per Scene. They can interfere with each other, some housekeeping is in order.
22:24:36.948: Loaded scenes:
22:24:36.948: - scene 'Scene':
22:24:36.948: - source: 'Display Capture' (monitor_capture)
22:24:36.948: - scene 'Fangames 600':
22:24:36.948: - source: 'input viewer' (window_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'Livesplit' (window_capture)
22:24:36.948: - source: 'decession: volume 6, obs -30' (text_gdiplus_v3)
22:24:36.948: - source: 'Display Capture 3' (monitor_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - scene 'Fangames 608':
22:24:36.948: - source: 'input viewer' (window_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'Deadsplit' (window_capture)
22:24:36.948: - source: 'Display Capture 2' (monitor_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'livesplit 608' (window_capture)
22:24:36.948: - scene 'Game':
22:24:36.948: - source: 'Gaming' (game_capture)
22:24:36.948: - scene 'Terraria':
22:24:36.948: - source: 'TML' (game_capture)
22:24:36.948: - source: 'Vanilla' (game_capture)
22:24:36.948: - source: 'Vanilla Audio' (wasapi_process_output_capture)
22:24:36.948: - scene 'I Wanna Maker speedrun':
22:24:36.948: - source: 'I Wanna Maker' (game_capture)
22:24:36.948: - source: 'input viewer' (window_capture)
22:24:36.948: - filter: 'Crop/Pad' (crop_filter)
22:24:36.948: - source: 'livesplit' (window_capture)
22:24:36.948: - source: '2160x1216' (text_gdiplus_v2)
22:24:36.948: - source: 'Application Audio Capture (BETA)' (wasapi_process_output_capture)

Not nearly enough bits for 2k/60fps. Change your rate control to CQP for recording, B Frames can be zeroed too.
22:24:40.675: [texture-amf-h264: 'advanced_video_recording'] settings:
22:24:40.675: rate_control: CBR
22:24:40.675: bitrate: 2500
22:24:40.675: cqp: 20
22:24:40.675: keyint: 250
22:24:40.675: preset: quality
22:24:40.675: profile: high
22:24:40.675: b-frames: 3
22:24:40.675: width: 2560
22:24:40.675: height: 1440
22:24:40.675: params: (none)
22:24:40.676: ---------------------------------
22:24:40.676: [FFmpeg aac encoder: 'Track1'] bitrate: 160, channels: 2, channel_layout: stereo, track: 1
Click to expand...
Switching to CQP did the trick. Thanks very much for your help.
 
M

mudspud

New Member
rockbottom said:
GPU driver is suspect. Installed is 23.11.1 (November 2023), current available is 24.12.1.
Click to expand...
I downgraded my gpu driver to see if it would fix the problem as I had recently updated it before experiencing this issue, evidently I forgot to update it again afterwards.
 
R

rockbottom

Active Member
Read the release notes before updating. I do know that drivers 24.10.1 & 24.9.1 are bad, B Frame issues too. Not sure if 24.12.1 is gonna be the answer either.

  • Fixed Issues and Improvements​

    • Lower than expected performance may be observed while using llama.cpp. This may also impact Ollama and LM Studio.
    • Texture corruption may appear while playing The Crew™ Motorfest.
    • System crash may be observed when first launching SteamVR while using Valve Index® at 144Hz on Radeon™ RX 6000 series graphics products.
    • Intermittent application freeze may be observed while loading a save file in Monster Hunter: World.
    • Intermittent application crash may be observed while playing Fortnite on Radeon™ RX 5000 series and above series graphics products.
    • H.264 encoding may not work as expected while using OBS Studio with Twitch Enhanced Broadcasting.

Known Issues​

  • Intermittent driver timeout or crash may be observed while playing Warhammer 40,000: Space Marine 2 on some AMD Graphics Products, such as the AMD Ryzen™ AI 9 HX 370. Users experiencing this issue can enable Variable Graphics Memory in AMD Software: Adrenalin Edition as a temporary measure (AMD Software: Adrenalin Edition -> Performance -> Tuning -> Variable Graphics Memory).
  • Intermittent stutter may be observed while playing Marvel Rivals when AMD FidelityFX™ Super Resolution 3 frame generation is enabled. Users experiencing this issue are recommended to disable AMD FidelityFX™ Super Resolution 3 frame generation in the per-game settings as a temporary workaround.
  • After using the AMD Cleanup Utility for Windows®, the AMD Bug Report Tool may appear intermittently during a new driver install on AMD Ryzen™ 7000 and above series processors paired with AMD Graphics Products.
  • HEVC encoding may not work as expected while using OBS Studio with Twitch Enhanced Broadcasting.
  • Limitations for Windows Subsystem for Linux (WSL 2) support can be found here.
 
You must log in or register to reply here.
Top