Question / Help Stuttering video in OBS

Your logfile shows a failure to hook into the Doom process, it does not show any actual recording/streaming success.

Try using OpenGL render mode in Doom in-game settings, the Vulkan API can't be hooked into as far as I am aware.
For Vulkan API capture you could try using Monitor capture, though recording/streaming performance of OBS will be reduced to a degree.
 
Can you upload a logfile of a recording/streaming attempt with a few minutes game play of WoW please, will be easier to troubleshoot.
 
https://gist.github.com/73f4e2ce622ee374048bebbaa7daf47e <<log file
My video was perfect yesterday but today every time I start a game the feed gets laggy but on my screen its fine and this is without streaming or recording??

https://gist.github.com/ae4515f4a33aede6f075847149a2b821 log file while skipping

In your logfile there is no record/stream attempt. One thing to mention is you have:
08:26:13.555: - source: 'Display Capture' (monitor_capture)
08:26:13.555: - source: 'Game Capture' (game_capture)
In the same scene, which is not recommended as you are doing two captures simultaneously. This will reduce performance. Game Capture is preferred and recommended as it directly hooks into the game process to access shared texture memory from the GPU, which is the most efficient way of grabbing frames to record/stream.

Also, do you mind making another thread as @TrickyHunter is the original poster.
 
@TrickyHunter, I really need a logfile of a recording/streaming session of WoW to better assist with troubleshooting for WoW specifically as it really is game dependent to a large degree (This is where making multiple profiles can be very beneficial as you can have multiple encoder settings for various games, dependent upon how resource intensive the game is)
As for the stuttering issue in Doom:
Do you have Multi-Adapter Compatibility ticked on in game capture settings? Can you try running with that off if you do.
Can you try to cap your fps to 61-63 (I use MSI AfterBurner, an alternative is EVGA Precision, I steer well clear of V-Sync as it induces noticeable input lag for me whenever I have used it in the past) and upload another logfile with a few minute recording attempt included in the logfile please.

Without logfiles showing a recording/streaming attempt can't really ascertain whether CPU or GPU lag is at play, for streaming might also be bandwidth issues, possibly connection quality issues to twitch servers, though your twitch stream seemed ok other than some frame drops from the parts I viewed.
 
Cheers for the video uploads,

A few quick questions:
Is your fps in-game capped or un-capped?
Is your fps always at 60/60+ or does it drop below (Generally play, not immediately after loading screen end) If it does, what is your minimum fps value when it does dip & for how long?
For the video upload of WoW, the part where you turned around and started running back does it stutter like that in-game for you as well?

15:35:00.220: Output 'adv_file_output': Number of skipped frames due to encoding lag: 80 (2.5%)
15:35:00.220: Output 'adv_file_output': Number of lagged frames due to rendering lag/stalls: 50 (1.5%)
The two above lines indicate you have a bit of struggles going on, the encoding lag is GPU encoder strain, the rendering lag is GPU strain. This could be attributed to mass texture 'pop' load in, not sure if that still is a thing of WoW as I haven't played it for years.

I think if you were to cap your fps in-game to 60-63 fps (Better to do 61-63 fps if you get noticeable screen tearing) and downscale in Output your recording resolution to 1920x1080@60fps and for streaming to 1280x720@60fps it may assist a bit.

Do you use AA in-game? If so can you lower that or turn it off completely, just to self-test to see if it assists.

15:19:09.992: output 1: pos={0, 0}, size={2560, 1080}, attached=true
15:19:09.992: output 2: pos={-1920, 0}, size={1920, 1080}, attached=true
Your logfile shows you have two monitors:
It may be the base resolution of your primary monitor that is giving you a bit of grief, though it is a very nice ultra-wide monitor you have there. Can you try disabling the 1080p monitor to see if it helps, if it doesn't can you disable your 21:9 monitor and test on the 1080p to see if that helps.

To note:
I believe the reason why OBS Studio is spamming your logfile with:
[game-capture: 'World of Warcraft'] cannot capture explorer.exe due to being blacklisted
is due to the fact you use Explorer as your default browser, if you were to switch to Chrome/Firefox or another browser, it would clear that logfile line up. I doubt it would improve performance though however it is really spamming your logfile, a couple of times every 2 to 3 seconds (Just helps to clear up your logfile a tad, low on priority overall)
 
First uploaded logfile:
First stream session (3.5k bitrate):
10:36:44.399: Output 'adv_stream': Total encoded frames: 117735
10:36:44.399: Output 'adv_stream': Total drawn frames: 117735
10:36:44.399: Output 'adv_stream': Number of skipped frames due to encoding lag: 1205 (1.0%)
10:36:44.399: Output 'adv_stream': Number of lagged frames due to rendering lag/stalls: 5651 (4.8%)
10:36:44.399: Output 'adv_stream': Number of dropped frames due to insufficient bandwidth/connection stalls: 149 (0.1%)
Second stream session (3.5k bitrate):
11:35:00.795: Output 'adv_stream': Total encoded frames: 209158
11:35:00.795: Output 'adv_stream': Total drawn frames: 209158
11:35:00.795: Output 'adv_stream': Number of skipped frames due to encoding lag: 2675 (1.3%)
11:35:00.795: Output 'adv_stream': Number of lagged frames due to rendering lag/stalls: 12813 (6.1%)
Second uploaded logfile:
First stream session (2k bitrate):
13:07:33.151: Output 'adv_stream': Total encoded frames: 1215
13:07:33.151: Output 'adv_stream': Total drawn frames: 1215
13:07:33.151: Output 'adv_stream': Number of lagged frames due to rendering lag/stalls: 8 (0.7%)
13:07:33.151: Output 'adv_stream': Number of dropped frames due to insufficient bandwidth/connection stalls: 497 (40.9%)
Second stream session (3.5k bitrate):
13:26:45.375: Output 'adv_stream': Total encoded frames: 67908
13:26:45.375: Output 'adv_stream': Total drawn frames: 67909
13:26:45.375: Output 'adv_stream': Number of lagged frames due to rendering lag/stalls: 305 (0.4%)
Ideally, you don't want to see any skipped or lagged frames, if you do then you want it to be 0.0% in ratio to number of frames encoded and rendered.

Other than loading/popping textures dropping your fps down a tad before it ramps up to fps cap again, there isn't much stuttering from what I can see in the playback of your stream. In-game options may have to be tweaked slightly to try and alleviate this. I haven't played WoW for a very long time so am not up to date on how the settings affect performance and cannot give you any recommended settings for picture quality, based on your PC setup.

I think for the video output (Downscaled) resolution you need to set to 1280x720, this would also reduce some load on your GPU.

Your bandwidth appears to be a little too saturated for your bitrate settings, dropping your bitrate will alleviate this.

However: In the two logs you uploaded there is a big difference where your bitrate for the second logfile you uploaded was set to 2k and you had ~40% dropped frames due to insufficient bandwidth, where the next streaming attempt your bitrate was set to 3.5k and you had no bandwidth issues at all (In the same logfile upload) Not sure what is going on there as the swing should be opposite if you did have a bandwidth issue due to too high bitrate, even then the dropped frames should not be ~40%. How is your internet connection?

Below is a link to a full guide by @NerdOrDie that is designed for OBS Studio users in regards to Twitch streaming:
https://www.youtube.com/playlist?list=PLT3Ure7_kYHwj8oT3AV-pZ4_r7yp6mDg-

Can you do a stream session for a few minutes, normal gameplay with the x264 (CPU) encoder and upload that logfile please, for a comparison.

Once we can get your setup to a point where you have smooth gameplay with zero/very little skipped/lagged/dropped frames shown via Twitch, we can start to try for more quality, which feasibly your PC setup can do.
 
First of all, many many thanks for the twitch VoD uploads, helps a lot as I don't stream at all and I haven't played WoW in years.

The x264 encoder really was a bust, quite surprised that your CPU couldn't handle the game and stream at the same time.

As for settings, the great thing is you don't have any bandwidth issues shown in your logfile with your bitrate at 3500. Maybe the last logfile uploads you just had some issues with your net on those days?

Can you try the following settings in a single stream attempt please (about 2.5 to 3 minutes long) incorporating a run like you did as well as combat with the Timeless Spellshaper (When you tested with OpenCL enabled, which again was a complete bust) or another mob that has a reasonable size health pool so the combat duration lasts around 20+ seconds:

Settings - Video tab -
Base Resolution: 1920x1080
Output Resolution 1920x1080 <--- Will allow a default for recording instead of streaming if you choose to do so. Will Force Rescale for streaming in Streaming tab.
Downscale Filter: Bilinear
FPS Value: 60

Settings - Output tab -

Output Mode: Advanced
- Output - Streaming tab -
- The following settings need View Mode: Master (Found at the bottom of the encoder settings list) -

Encoder: H264 Encoder (AMD Advanced Media Framework)
Enforce streaming service encoder settings: checked/ticked
Rescale Output: Checked, 1280x720
Preset: Twitch
Quality Preset: Balanced
Target Bitrate: 3000
Minimum QP: 11
Maximum QP: 51
VBV Buffer: Automatic
VBV Buffer Strictness: 50.00
VBV Buffer Fullness: 100.00
Frame Skipping: Enabled <--- Need to test in one stream attempt.
Enforce HRD Compatibility: Enabled
IDR Period: 120
B-Frames: 0
Pre-Analyis Pass: Enabled <--- May help give a better quality picture
VBAQ: Enabled <--- May give a better quality picture
GOP Size: 30
View Mode: Master

Hope these settings help!
 
Definitely is a lot smoother than before!

Unfortunately the frame analysis portion of your logfile returned some erroneous numbers so can't be used to determine performance issues (Have had this happened a number of times for my own, not a big deal, just means one has to fall back on quality & fluidity of the recording/stream)

Let's try to bump up the quality a tad:

Quality Preset: Quality
Target Bitrate: 3500

If no detriment to fluidity of the stream then try:
Downscale Filter: Lanczos

If all goes well with that then try:
Minimum QP: 0
Maximum QP: 42.

Just need a few minute test with an attached logfile for frame analysis once again. Need each change tested individually though.

Hope this helps!
 
Top