H.265 is a different (and newer) algorithm than H.264. H.265 also has some licensing baggage, so cheap new things might not have it either. AV1 is supposed to be better than both and legally free (or at least much cheaper), but it's so new that hardly anything supports it yet.
Anyway, you still have the old, well-known H.264. Use that.
For the overload when not playing anything, it's probably not by competition between game play and video encoding, because the encoding is done on a dedicated part of the chip that does nothing else anyway. If it's an older chip (or an old design sold for cheap), then it likely just can't handle some of the higher video settings or presets.
See if you can look up how many of what size video streams your specific card can ENcode, and don't exceed that size in OBS. Then play with the quality settings and preset levels (how hard it should try to be efficient, in terms of quality per bitrate) until it's both acceptable and stable. Or if you can't get an acceptable result reliably, then you need a different card.