Use the Auto-Configuration Wizard. It was put in for situations just like this. It's up in the Tools menu, and it's really good. It will pick the correct encoder and settings to go with it.
Don't be surprised, however, when it picks setting that are a lot lower than what you want. It requires a lot of computer horsepower to encode video, especially using x264! Just looking at your specs, I would think Intel Quicksync would be the best option since your video card is not new enough to support NVENC.
Just for reference for using x264, the wizard recommended 720p @ 60fps on veryfast with my Core i7-4790K as a dedicated streaming PC! It wasn't until I went up to a dedicated i7-8700K that it went with 1080p, and still on veryfast preset. Ultimately, I dropped it back to 720p @ 60 fps so I could set the preset higher so I wouldn't limit my audience who don't have the bandwidth to watch a 1080p stream and because the game I play, Battlefield 1, has a LOT of detailed texture motion.
Sorry for the book, but don't give up--try the wizard and go from there. Good luck!!
-Beast