I have almost an identical PC to yours (GTX 1080ti, 32GB Ram, i7-8700k), identical screen set up (one 2560 with g sync, one 1920) and I'm having the exact same problem as you.
Most games will record and stream at 1920x1080 completely fine, but whack them up to 2560 and it's a no-go, OBS drops frames and reports "Rendering Lag". I fixed this for a few games by limiting the in-game FPS, as many people have suggested. For example Fortnite is now capped at 120FPS and I can record and stream completely fine at 2560. However for other games, I had to stick them at 1920, which is fine for streaming, but does spoil the experience and the point of having the bigger screen.
Unfortunately Apex Legends, which came out two days ago, is not even working correctly at 1920, I'm getting OBS frame drops with that game regardless of how low I stick the graphical settings, and the only way that I've found to fix this is to turn my camera off, which is obviously not ideal for streaming, but fine for recording for YouTube (as I don't do Face Cam youtube videos).
SO, here are some bits of advice that I can give you (although I'm still waiting for a bigger fix for myself as I really want to stream Apex and the PC should be more than enough to do it).
1. Fully update windows 10 to the latest version (people have suggested going back to get rid of an update which caused this issue, but it's impractical and opens your PC up for viruses etc.)
2. Once updated, search for "Windows Game Bar" - Turn this off
3. Search for "Windows Game Mode" - Turn this off (This basically forces the PC to put all GPU power into a game, leaving none for OBS, so turning it off leaves a bit more free for OBS)
4. Whichever game you are trying to record and stream, add an in-game FPS limit to it, 120 works for some, but 60 is a safer bet.
5. OBS keeps scenes playing in the background to allow you to swap to them, you can right-click on webcam's, videos and audiotracks, and you can turn these to "disable when not in use", which will free up some power for OBS (although does make scene transitions look a bit less smooth unless you use studio mode to do it.
6. If you use a webcam, make it a lower resolution for you gaming scene, typically people only use a small image for their webcam whilst streaming, so having it at 1080 is pointless and using power that is unnecessary - you can keep it at 1080 for a full-screen scene, but make sure "disable when not in use" is ticked so that OBS is not processing this all the time.
These are all of the fixes that I am aware of currently, and they have all helped me slightly, however I am still digging for more information on how to get OBS working with Apex Legends so if anyone can add more to my list, please let me know.
