As
@OG See OH Zee stated, your system is having a very hard time recording.
Unfortunately given your APU & you are using the iGPU, there is not much you can do unless you invest in a dedicated GPU.
A few tweaks for your settings:
Video tab:
output resolution: 1280x720 <--- Could lower again, will reduce quality though will help a lot for performance.
downscale filter: Bilinear <--- Could try Bicubic Filter, or even Lanczos Filter for a bit of quality increase, performance may decrease a lot.
fps: 30 <--- Will help your system a bit for performance.
Output tab, Recording:
Quality Preset: Speed <--- Balanced and Quality will decrease performance substantially for you.
Profile: Main <--- Should help a little in performance whilst does not seem to affect quality for me, maybe even lower this. Have not tested other values than High or Main.
Profile Level: Automatic
Rate Control Method: VBR_LAT
Target Bitrate: 50000 <--- May need to lower this for performance, bitrate is generally tied to your Hard Drive speed, the faster your hard drive the more bitrate you can push for increase in quality (From around 50000 the increase in quality becomes substantially less noticeable as you increase the bitrate) Reducing this will reduce load on your PC, will reduce quality exponentially under ~30000.
Peak Bitrate: 100000 <--- Maximum value available.
Minimum QP: 11 <--- Could try 0, will improve quality output though will tax your iGPU more.
Maximum QP: 51 <--- Could try 42, will improve quality output though will tax your iGPU more.
VBV Buffer: Automatic
VBV Buffer Strictness: 50.00
VBV Buffer Fullness: 100.00
Frame Skipping: Disabled <--- Could try enabling to see if performance improves, made no noticeable difference for me.
Enforce HRD Compatibility: Enabled <--- Did not seem to impact performance, image quality was slightly improved for some areas.
IDR Period: 60 <--- Always x2 your configured fps recording value.
B-Frames: 0 <--- Will have no effect/performance issues if higher than 0.
Deblocking Filter: Enabled <--- Does not seem to have an impact on performance, improves quality.
Scan Type: Progressive <--- Have not tested Interlaced, guessing Interlaced is not compatible with h264 encoder.
Motion Estimation: Half- & Quarter-Pixel <--- Could lower this to Quarter, Half or even None for quality reduction and performance increase.
Coding Type: Default <--- Could manually set this to CALVC, I noticed no difference in quality/performance from CABAC, same for file size output.
Wait For Task: Disabled <--- This absolutely killed performance for me when Enabled, was a stuttering mess.
Pre Analysis Pass: Enabled <--- Improves motion image a bit, did not notice performance impact.
VBAQ: Enabled <--- Seemed to improve quality a bit, did not notice a performance impact.
GOP Size: 15 <--- Always 1/2 your configured recording fps value.
GOP Alignment: Enabled <--- Noticed no difference when disabled.
Maximum Reference Frames: 4 <--- Could try lowering this, made the image a bit grainy for me, bit more blocky in fast motion footage. Did not seem to have a performance impact at lower values, have not tested higher values and guess it will have a performance impact.
Try OpenCL: Enabled, it may assist with offloading some computation to the integrated graphics card from the CPU itself.
Hopefully this helps you get more fluid recordings whilst maintaining as much quality as you can. You really need to test some of the setting changes yourself though as it really is system dependent and my system is an i5 2500 and r9 290x so your mileage may vary substantially for performance.
Can you please upload a logfile after you have attempted a few minutes worth of recording, this will help greatly in troubleshooting.