Your GPU is struggling a bit at 2560x1440@60fps, not entirely sure as to why as I thought a 1080 should also be fine for 4k60fps recording as well. In saying that, I don't have a 1080 or an nVidia card so am not familiar with the performance of the nVidia encoder.
Some things to try for troubleshooting (Will need to test one at a time to see if they help, not all at once):
If you don't, cap your fps to 60 as it will decrease the load on your GPU.
Using High Quality, Low latency OR High Performance, Low Latency Preset
Keyframe Interval: 2
Using VBR instead of CQP - Bitrate of at least 50k
Note: To cap your fps I recommend using 3rd party software such as MSI AfterBurner or EVGA Precision, with a capped fps of 61-63, as if you get any screen tearing it will move the horizontal 'tear' to another location on the screen due to the slightly different refresh rate timing. Using V Sync can and most likely will induce input lag, which I can't stand, however if you don't notice it then it is fine to use though it can sometimes hurt performance as it forces your GPU to effectively idle momentarily to wait for your monitor to refresh the frame (Which is why it induces input lag)
Note: flv container format is better to use than mp4 as it allows for the footage to be played back/usable in the event of a crash of OBS Studio, your game, Windows OS, etc. One caveat of flv container format is it only allows for one audio track, not multiple.
