Are they hearing just themselves, or are they hearing your game audio through their headsets as well?
I don't have a PlayStation, so I don't use PS4 remote play, but I do stream Xbox games I play cooperatively with others, and in this case it's necessary to be careful how you isolate audio sources from one another so everybody hears only what they're supposed to. If I explain my setup perhaps you'll be able to see how it applies to yours.
I capture my Xbox's game audio and video through a capture card. That's the job that's done by the Remote Play application in your case.
In my console settings, I tell the Xbox to send all chat audio to my headset (a jack on my controller) and not to "speakers" or "both". This is to separate game audio from chat audio so I can control them separately. I assume the PS4 has a similar setting someplace.
I have a hardware adapter that attaches to my controller and provides me with separate headphone and microphone leads. This way I can connect the headphone lead to an input on my computer for capturing my party's audio. If you don't have another available hardware input, you can use a USB audio adapter for this-- ones made by sabrent work but there are many available.
This is the model I use:
https://www.amazon.com/Sabrent-External-Adapter-Windows-AU-MMSA/dp/B00IRVQ0F8
So if you've got a free analog input port you'll need only the USB audio dongle to send your microphone output to your PS4 controller. If you don't have a free analog input port, you'll need to use both ports-- one to send your mic to the controller, and one to get chat audio from the controller.
This is the one I use. It says it is compatible with PC, Xbox and PS4:
https://en-us.sennheiser.com/pcv-05
If the Audio Monitor function of OBS works for you then you're nearly done; just add Desktop audio to your scene (for game audio from Remote Play), plus the hardware input from the controller for chat audio (whether a USB dongle or built-in) and whatever microphone you normally use. In Advanced Audio setup, turn on monitoring for the sources, and in advanced settings choose the hardware output you want to monitor with (headphones of course).
The only thing we've missed out on here is how your party will hear you. You can achieve this with a a freebie app called LineIn. It lets you mirror any one audio device to any other audio device. In this case, you use it to mirror your microphone to the USB audio adapter that is connected to the microphone port of your controller.
https://rogueamoeba.com/freebies/
You can do this and a lot more with Rogue Amoeba's LoopBack (soundflower on steroids) program, but it's not free, so YMMV. SoundSource also does what LineIn does but also provides the volume control that LineIn does not.
If it's all set up properly, you can hear your party and they can hear you, and the stream can hear your game, you, and your party, and nobody is listening to echoes of themselves or hearing game audio come through in the party chat.
