First a check: why do you want to have Tiktok's dimensions as a guide within an apparently larger resolution? Why not set the output resolution of your stream to 576x1024?
But, explaining the workaround: in a graphical program create an image with the desired guides. In this case a png file (that file type supports transparancy) with dimensions 5
86x10
34px with 5px white borders and the inside of that being completely transparent. But obviously you can add crossed center lines or any other guides you like.
In OBS Studio add an
Image source to your scene containing this guides image. Do not scale the image or fit it to the output dimensions, just right click at the source and
Transform,
Center to screen. Now you can have guides visible while in Preview.
Before bringing the scene to Program, you have to hide the guides image. That can be done by the eye icon but I'd recommend assigning a hotkey for that (
Settings,
Hotkeys). Assign the same key to both show and hide the image source.
If you want to keep using the guides during your broadcast and you use one camera all the time, you can duplicate the camera scene and have one version with guides that you only use in Preview and one plain version of the scene for the actual broadcast (Program).
The downside... switching camera's becomes more cumbersome and error-prone.