First things first, update windows to the October 1809 update. The version you're on currently forces game mode to be enabled, which limits GPU usage in OBS. Once you've updated windows, turn game mode off by following the directions
here.
Once you've done that, attempt to record with modest settings. I'm not too familiar with AMF recording, so whichever settings you've had the best luck with there, try those first. If you are still having issues, then try limiting your in-game fps -- it should make an improvement as far as the ability to combat rendering lag.
One thing you should also definitely have open while you're adjusting settings is the Stats window -- this will tell you immediately whether you're encountering rendering lag or encoding lag. If it's rendering lag, then that means OBS doesn't have the GPU resources necessary to perform all the actions it needs, and usually means you need to back off your game's settings (this is why frame capping is the 1st step). If it's encoding lag, then the settings you've used for encoding are too much and you'll need to either use a less straining preset, or change encoders entirely (i.e. if you can't get AMF to run without issue, then try x264... or vice versa).
If you still can't get things recording smoothly, then as a last ditch troubleshooting step try lowering your output (scaled) resolution in the video settings to something very small like 640x360, and see if you can get a smooth recording there. If not, then that should 100% rule out any kind of encoding issues and it all comes down to OBS's ability to grab frames.