GTA5 has borderless window mode natively which allows for multitasking.
I get this on my end as well for a GTX 1060 6gig on select games that use all GPU horsepower, and I run the GPU in PCIe x16 3.0. When you put OBS on top it doesn't drop FPS (or what it's called lagged frames based on the log) even if the game in the background is using all of the GPU such as many Unreal Engine games. I'm thinking putting the window on top puts GPU priority on it which explains it. However, if you put OBS as 'always on top' there is no difference when you focus on the game.