Fix - Xbox One settings > Devices & accessories > Xbox One Wireless Controller > Assign to someone > Guest
Basically for me I found that whenever i connected and disconnected the controller Party chat would get picked up on OBS for a brief second or two. By un-pairing it with your profile some how this allows party chat to come through and get recorded.
Hope this helps others, i drove me crazy and i've not seen this fix posted anywhere on my searches. I will be posting this in a couple of other OBS threads where it could help,