hello. i'm late for a year to reply. maybe i can help you fix your RX 580 streaming setting setup.
i'm also have RX 580 but 8gb vram and using AMD encoder to stream my games. maybe you can change some settings in OBS like mine.
make sure you update your AMD driver and OBS to the latest version. mine currently AMD driver 20.9.2 and OBS 26.0.2.
note 1: RX 580 can't deliver quality over 1080p output resolution, GPU and stream will become lag, if you want to stream at 1080p only, option 1,2,3 is yours. otherwise, option 4,5,6 is quality with lower resolution.
note 2: if you don't want to mess up with advanced setting, only basic, go with option 1,4.
note 3: the only different here is SPEED preset and QUALITY preset. RX 580 with SPEED preset do support 1080p stream (no lag) but will give LOW QUALITY video for viewers. RX 580 with QUALITY preset do give viewers BETTER QUALITY video (not the best) but lower resolution.
note 4: bitrate is up to your internet capability. mine only goes to 10mbps so i pick 6mbps.
note 5: if you see any changes or warning error during streaming, tick box "enforce streaming service encoder settings" in advannced output (option 2,3,5,6).
option 1: 1080p60fps speed preset simple output.
View attachment 61824
View attachment 61825
option 2: 1080p60fps speed preset advanced output. basic mode.
View attachment 61826
View attachment 61825
option 3: 1080p60fps speed preset advanced output. master mode.
View attachment 61827
View attachment 61828
View attachment 61825
option 4: 864p60fps quality preset simple output.
View attachment 61830
View attachment 61831
option 5: 864p60fps quality preset advanced output. basic mode.
View attachment 61832
View attachment 61831
option 6: 864p60fps quality preset advanced output. master mode.
View attachment 61833
View attachment 61834
View attachment 61831