Yeah, some settings can reduce it. A higher framerate almost eliminated it completely in the preview, but I wasn't able to get smooth playback as any player I tried would skip frames all over the place
(MPCHC, XBMC, VLC).
Edit: Just popped in an Nvidia 8800GTX
(after uninstalling the ATI drivers of course). Even though it is far less powerful, it has the same exact stuttering (I tried in 2d to make sure). I forgot my GTX 470 has artifacts so I'm not going to bother with it. Next thing to try is a fresh install
(on a spare drive).
Edit2: Decided to play around with custom resolutions in the Nvidia panel. Created a custom resolution with a refresh rate of 59hz and set OBS to 640x480 59fps. Zero stuttering, absolutely flawless.
Went back to 60hz/60fps, stutters again. Tried 61hz/61fps, 59hz/60fps, 60hz/59fps, 61hz/59fps, 59hz/61fps, and they all stutter. Only 59hz/59fps is smooth.
Whatever this means, I do not know. But I do know that I'll be playing with 59hz a bit more.
Edit3: It worked on my 7950 too! I'm using this program for custom resolutions:
http://www.monitortests.com/forum/Thread-Custom-Resolution-Utility-CRU
My monitor was being detected at 59.950hz, I changed it to 60hz and tried again, but it didn't seem to make any difference. 59.000hz still works fine though.
I'm not sure how 59fps will play on Youtube, playback when at 60hz stutters quite a bit.
Edit4: Scratch that, 60hz works fine! Use the CRU reset-all to reset everything, then use CRU to change 59.950 to 60fps and reboot. Perfectly smooth now at 60hz/60fps! no noticable stuttering for ~10,000 frames.
New test video incoming. :)