That's probably because the encoder is designed for real-world images that don't have the sharp edges that computer text does. It's certainly possible to make one that *is* designed for that, just like we do have for still images, but I don't know of one, let alone how to make it work in OBS.
In the meantime, either use an insanely high bitrate so it doesn't throw away that detail, or just live with it. Or you could magnify the section of interest before the encoding step, but that's another tool that you have to learn how to use. Some people do that though, and then go back to the overall view, and they make it work well.