OBS will always use some GPU resources, because it does source compositing and video rendering on the GPU. This is not x264-specific but a sign of just OBS doing its work. If you force OBS to run on "the other" GPU, you're forcing OBS to use less efficient capture methods and copying the raw picture data two times more through the pci-express bus.
In a one GPU setup, the data is transferred from GPU memory to CPU memory once: after compositing the video. In a two GPU setup where OBS is running on the other GPU, the video data has to be moved from GPU#1 memory to CPU memory, from CPU memory to GPU#2 memory for video composition, and a third time as composited video back to CPU memory again. Together with the halved pci-express bandwidth, this may create a bottleneck that results in micro or not so micro stutters and general sloppiness of the system.
The best option to deal with frame drops due to high GPU usage is to limit the frame rate of the game you're capturing, so OBS gets enough resources to composite the video - on that GPU. OBS is a very resource intensive application, you cannot expect your game running as if it is running exclusively.