using 100,000Mbps as the bitrate is sooooper overkill and lowering it will decrease the quality but it's already so high so the difference will barely be noticeable
try changing the recording format to mp4
if your fps only drops while recording, then it's simple - your hardware isn't good enough to record at the settings you're using
you have to lower your settings
the Ryzen 7 1700 isn't the best (but it's not that bad either) and the RX 580 is decent but the H264 and H265 encoder also aren't the best and combining that with the not so great CPU means that you will definitely have to lower your settings
that's all I can think of
try changing the recording format to mp4
if your fps only drops while recording, then it's simple - your hardware isn't good enough to record at the settings you're using
you have to lower your settings
the Ryzen 7 1700 isn't the best (but it's not that bad either) and the RX 580 is decent but the H264 and H265 encoder also aren't the best and combining that with the not so great CPU means that you will definitely have to lower your settings
that's all I can think of