On a followup to this, I tried with OBS Studio, same result. It doesn't seem to matter how low I set the quality, the quality always seems similar and the stuttering always occurs.
19:16:52.225: CPU Name: AMD Phenom(tm) II X4 965 Processor
19:16:52.225: CPU Speed: 3684MHz
19:16:52.225: Physical Cores: 4, Logical Cores: 4
19:16:52.225: Physical Memory: 7130MB Total, 4304MB Free
19:16:52.225: Windows Version: 6.1 Build 7601 (revision: 18933)
19:16:52.225: Aero is Enabled
19:16:52.258: OBS 0.15.2 (64bit, windows)
19:16:52.258: ---------------------------------
19:16:52.259: ---------------------------------
19:16:52.259: audio settings reset:
19:16:52.259: samples per sec: 44100
19:16:52.259: speakers: 2
19:16:52.538: ---------------------------------
19:16:52.538: video settings reset:
19:16:52.538: base resolution: 1312x998
19:16:52.538: output resolution: 1048x798
19:16:52.538: fps: 30/1
19:16:52.538: format: NV12
19:16:52.539: ---------------------------------
19:16:52.539: Loading module: coreaudio-encoder.dll
19:16:52.540: [CoreAudio encoder]: Failed loading library 'CoreAudioToolbox.dll'
19:16:52.540: [CoreAudio encoder]: Couldn't load CoreAudio AAC encoder
19:16:52.540: ---------------------------------
19:16:52.541: Loading module: image-source.dll
19:16:52.541: ---------------------------------
19:16:52.554: Required module function 'obs_module_load' in module '../../obs-plugins/64bit/libcef.dll' not found, loading of module failed
19:16:52.554: ---------------------------------
19:16:52.555: Required module function 'obs_module_load' in module '../../obs-plugins/64bit/libEGL.dll' not found, loading of module failed
19:16:52.555: ---------------------------------
19:16:52.556: Required module function 'obs_module_load' in module '../../obs-plugins/64bit/libGLESv2.dll' not found, loading of module failed
19:16:52.556: ---------------------------------
19:16:52.556: Loading module: obs-browser.dll
19:16:52.556: Failed to load 'en-US' text for module: 'obs-browser.dll'
19:16:52.557: ---------------------------------
19:16:52.557: Loading module: obs-ffmpeg.dll
19:16:52.558: LoadLibrary failed for 'nvEncodeAPI64.dll', error: 126
19:16:52.559: ---------------------------------
19:16:52.559: Loading module: obs-filters.dll
19:16:52.559: ---------------------------------
19:16:52.560: Loading module: obs-outputs.dll
19:16:52.560: ---------------------------------
19:16:52.561: Loading module: obs-qsv11.dll
19:16:52.788: ---------------------------------
19:16:52.789: Loading module: obs-transitions.dll
19:16:52.789: ---------------------------------
19:16:52.789: Loading module: obs-x264.dll
19:16:52.789: ---------------------------------
19:16:52.790: Loading module: rtmp-services.dll
19:16:52.790: ---------------------------------
19:16:52.791: Loading module: text-freetype2.dll
19:16:52.794: ---------------------------------
19:16:52.794: Loading module: vlc-video.dll
19:16:52.794: Couldn't find VLC installation, VLC video source disabled
19:16:52.794: ---------------------------------
19:16:52.809: Loading module: win-capture.dll
19:16:53.160: ---------------------------------
19:16:53.160: Loading module: win-decklink.dll
19:16:53.166: No blackmagic support
19:16:53.166: Failed to start search for DeckLink devices
19:16:53.166: ---------------------------------
19:16:53.166: Loading module: win-dshow.dll
19:16:53.168: ---------------------------------
19:16:53.169: Loading module: win-mf.dll
19:16:53.169: [Media Foundation encoder]: plugin is disabled for performance reasons on Windows versions less than 8
19:16:53.300: ---------------------------------
19:16:53.301: Loading module: win-wasapi.dll
19:16:53.303: =====================================================================
19:16:53.304: output 'simple_stream' (rtmp_output) created
19:16:53.304: encoder 'simple_h264_stream' (obs_x264) created
19:16:53.304: AAC encoder bitrate mapping:
19:16:53.304: 32 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 64 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 96 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 128 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 160 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 192 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 224 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 256 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 288 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.304: 320 kbit/s: 'FFmpeg Default AAC Encoder' (ffmpeg_aac)
19:16:53.305: encoder 'simple_aac' (ffmpeg_aac) created
19:16:53.305: encoder 'simple_h264_recording' (obs_x264) created
19:16:53.305: encoder 'simple_aac_recording' (ffmpeg_aac) created
19:16:53.305: output 'simple_file_output' (ffmpeg_muxer) created
19:16:53.307: Service '' not found
19:16:53.307: service 'default_service' (rtmp_common) created
19:16:53.312: All scene data cleared
19:16:53.312: ------------------------------------------------
19:16:53.371: WASAPI: Device 'Speakers (Realtek High Definition Audio)' initialized
19:16:53.371: source 'Desktop Audio' (wasapi_output_capture) created
19:16:53.387: WASAPI: Device 'Stereo Mix (Realtek High Definition Audio)' initialized
19:16:53.387: source 'Mic/Aux' (wasapi_input_capture) created
19:16:53.387: source 'Game Capture' (game_capture) created
19:16:53.388: source 'retroarch' (scene) created
19:16:53.388: source 'Scene' (scene) created
19:16:53.416: == Profiler Results =============================
19:16:53.416: run_program_init: 1351.11 ms
19:16:53.416: ┣OBSApp::AppInit: 17.662 ms
19:16:53.416: ┃ ┗OBSApp::InitLocale: 3.955 ms
19:16:53.416: ┗OBSApp::OBSInit: 1306.86 ms
19:16:53.416: ┣obs_startup: 2.163 ms
19:16:53.416: ┗OBSBasic::OBSInit: 1157.6 ms
19:16:53.416: ┣OBSBasic::InitBasicConfig: 0.51 ms
19:16:53.416: ┣OBSBasic::ResetAudio: 0.251 ms
19:16:53.417: ┣OBSBasic::ResetVideo: 279.117 ms
19:16:53.417: ┣OBSBasic::InitOBSCallbacks: 0.022 ms
19:16:53.417: ┣OBSBasic::InitHotkeys: 0.057 ms
19:16:53.417: ┣obs_load_all_modules: 764.843 ms
19:16:53.417: ┃ ┣obs_init_module(coreaudio-encoder.dll): 0.433 ms
19:16:53.417: ┃ ┣obs_init_module(image-source.dll): 0.004 ms
19:16:53.417: ┃ ┣obs_init_module(obs-browser.dll): 0.068 ms
19:16:53.417: ┃ ┣obs_init_module(obs-ffmpeg.dll): 1.135 ms
19:16:53.417: ┃ ┣obs_init_module(obs-filters.dll): 0.009 ms
19:16:53.417: ┃ ┣obs_init_module(obs-outputs.dll): 0.003 ms
19:16:53.417: ┃ ┣obs_init_module(obs-qsv11.dll): 226.759 ms
19:16:53.417: ┃ ┣obs_init_module(obs-transitions.dll): 0.009 ms
19:16:53.417: ┃ ┣obs_init_module(obs-x264.dll): 0.002 ms
19:16:53.417: ┃ ┣obs_init_module(rtmp-services.dll): 0.282 ms
19:16:53.417: ┃ ┣obs_init_module(text-freetype2.dll): 2.315 ms
19:16:53.417: ┃ ┣obs_init_module(vlc-video.dll): 0.031 ms
19:16:53.417: ┃ ┣obs_init_module(win-capture.dll): 350.959 ms
19:16:53.417: ┃ ┣obs_init_module(win-decklink.dll): 5.309 ms
19:16:53.417: ┃ ┣obs_init_module(win-dshow.dll): 1.206 ms
19:16:53.417: ┃ ┣obs_init_module(win-mf.dll): 130.402 ms
19:16:53.417: ┃ ┣obs_init_module(win-wasapi.dll): 0.028 ms
19:16:53.417: ┃ ┗reset_win32_symbol_paths: 1.262 ms
19:16:53.417: ┣OBSBasic::ResetOutputs: 1.832 ms
19:16:53.417: ┣OBSBasic::CreateHotkeys: 0.919 ms
19:16:53.417: ┣OBSBasic::InitService: 0.407 ms
19:16:53.417: ┣OBSBasic::InitPrimitives: 3.367 ms
19:16:53.417: ┗OBSBasic::Load: 80.203 ms
19:16:53.417: =================================================
19:16:53.426: adding 46 milliseconds of audio buffering, total audio buffering is now 46 milliseconds
19:16:53.627: Update check: last known remote version is 0.15.2
19:17:04.334: rtmp-common.c: [initialize_output] Could not find service ''
19:17:04.335: rtmp-common.c: [initialize_output] Could not find service ''
19:17:04.336: rtmp-common.c: [initialize_output] Could not find service ''
19:17:04.362: rtmp-common.c: [initialize_output] Could not find service ''
19:17:06.176: rtmp-common.c: [initialize_output] Could not find service ''
19:17:14.478: rtmp-common.c: [initialize_output] Could not find service ''
19:17:16.818: rtmp-common.c: [initialize_output] Could not find service ''
19:17:19.203: rtmp-common.c: [initialize_output] Could not find service ''
19:17:35.402: rtmp-common.c: [initialize_output] Could not find service ''
19:17:37.158: rtmp-common.c: [initialize_output] Could not find service ''
19:17:41.124: rtmp-common.c: [initialize_output] Could not find service ''
19:17:50.453: rtmp-common.c: [initialize_output] Could not find service ''
19:17:54.743: rtmp-common.c: [initialize_output] Could not find service ''
19:18:02.003: rtmp-common.c: [initialize_output] Could not find service ''
19:18:02.859: rtmp-common.c: [initialize_output] Could not find service ''
