As you have Radeon you can't use NVENC, so you'll have to use x264. I recommend 720p60 (720p, 60 fps) or 1080p30 (1080p, 30 fps) in OBS. For 60 fps use the x264 preset 'veryfast', profile 'high' and Tune 'none'. For 30 fps you can try using the x264 preset 'fast'. Leave keyframe interval as 0 (auto).
If streaming to Twitch, tick 'Enforce streaming service encoder settings' which will prevent x264 using any encoder settings incompatible with Twitch's requirements (and Twitch can directly restream the 'source' quality, direct from you without needing to be reencoded).
If you have more than 10 Mbit/sec upload, for 1080p keep the encoding bitrate between 5000 to 6000 kbps (6 Mbit). You can send higher, but publicly they state 6 mbit/sec is their max for streams. If you stream at 720p, you can reduce the bit rate without any real picture quality impact - 3500 to 4500 kbit/sec.
Run upload speed tests to work out your maximum, you may find you can only stream at a lower bit rate.
If your upload is less than 10 mbit/sec, I'd recommend you only stream at 4 or 5 mbit/sec to avoid causing higher pings and congesting your internet connection (downloading also requires a reasonable amount of free upload bandwidth).
x265 (aka "HEVC") is far more CPU intensive to encode, and it doesn't yet enjoy universal device support, so use x264 for the time being. Twitch, YouTube and all the main streaming services all use x264 anyway.
You may also need to reduce graphical quality in-game to get the smoothest quality stream.
