There seems to be a glitch introduced in the GUI, one way or the other. Sometimes you can set matroska with hevc_nvenc without checking "show all encoders", and recordings actually are with hevc_nvenc, but if you open this configuration page after a restart of OBS, the settings are reverted to libx264. If you click enough between the checkbox and the audio+video encoder dropdown list, sometimes you are again able to select hevc_nvenc without the checkbox being checked.
Not consistent.
However, not opening a bug in github, since I'm not able to reproduce the exact click stream required to provoke this behavior.
To be on the safe side and not having your configuration accidentally reversed, you really need to make sure the checkbox is checked.