You're lagging frames like crazy.
* Don't use monitor capture, especially not with 60fps. It's really bad. Use Window capture with Aero enabled, or better yet, use Game Capture if you can.
* Since you didn't post a video like I asked, seeing that you were using Monitor capture further confirms my suspicion that you were trying to capture a fullscreen game. Don't use Monitor Capture or Window capture to capture a fullscreen game.
* Lower your frame rate to 30fps until we can see that your system can handle it well enough for 60fps.
* Lower your bit rate to 3500, and raise your x264 preset to veryfast.
* It only sees 4 GB of RAM because you are using the 32bit version. 32bit programs can't utilize more than 4GB of RAM.
A good getting-started guide is here:
viewtopic.php?f=18&t=410
And you can find a good starting place for your settings here:
http://obsproject.com/estimator