Question / Help Jagged text edges in recording

[Steve Macintosh]

Hey,

I was testing out some settings and recording earlier. I used Limbo as I was waiting for other stuff to finish and still haven't tried that game. I noticed that whenever I recorded with OBS (both classic and studio) and using Game Capture and regardless of encoder. I would get jagged edges on the menu text. First thought was that it maybe was because of color format, so I've tried all of them but with no success. I used MPC-HC to play the recordings, but also double-checked in VLC and opened the video in Premiere. The text was jagged there as well.
After seeing this I also tried recording in ShadowPlay, DXTory and Bandicam. And there the effect wasn't present.

I took a screenshot from MPC-HC of the different programs/settings I used and made an album:
http://imgur.com/a/tRnYH

Make sure that you maximize the images to see the jagged effect. And then compare them to the shadowplay/dxtory screens where you see it's much smoother and how I see it when in-game.

Don't know if I'm missing something obvious here, but any help would be much appreciated. Let me know if you need more info from me.

EDIT: Tried with Bandicam as well and added screenshot to album.

 

[Suslik V]

Log? And you need to use same settings and same input when compare.

 

[Steve Macintosh]

Log? And you need to use same settings and same input when compare.

Well... No. Not really. You have to use the same settings and the same input if you are doing a comparison where you are trying to show why one is better than the other. In this case, I did use the same settings, and then a bunch of other ones to show that I get those jagged edges regardless of what encoder, regardless of what encoder mode and only in OBS. The input to compare is for example the "Resume" text, which is the same in all the screenshots. Having an identical input would change absolutely nothing. And the jagged edges are likes this throughout the game where there is text. For example "Press any key to start" has the same effect while in the other programs it doesn't. And I have tried, as far as I know, literally everything in OBS but still get that result. Spent countless hours on it yesterday trying to figure it out.

I don't have a log on this computer. But I can make one if needed. Which one of them would you want? NVENC? QuickSync? x264? Color format?

 

[Suslik V]

... Which one of them would you want? NVENC? QuickSync? x264? Color format?

Canvas and Output size.

 

[Steve Macintosh]

19:35:32.925: base resolution: 1920x1080
19:35:32.925: output resolution: 1920x1080

Attached log of a record I did just now.
Another album with more examples of how the texts are all rough: http://imgur.com/a/T50yy
(Again, they are only like this when I use OBS)

EDIT: And yes, using RGBA on this particular capture. But same result with NV12 / I420

 

[Suslik V]

OK. My log: https://gist.github.com/3e95305ed946886f620e4a6ba74638df (I've had a demo version of the Limbo the game, Steam).

1. Long time I can't get the picture (Transform's Size was zero o_O, log said "d3d9_shtex_init_shtex: Failed to create D3D11 texture (0x80070057): The parameter is incorrect."). Then I got the picture in preview (alt-tabbing) and start the capture. File - 2016-06-25 10-10-32.flv
2. Then I decided to switch off Steam Overlay completely (there was fps counter ON) and I restarted the game. I deleted and recreate a 'Game Capture' source too in Studio, just in case. Again, I got black screen for a while in a preview. After alt-tabbing few times I got the picture and start recording. File - 2016-06-25 10-13-15.flv
3. Because first two results gives me a stretched image (both in preview and inside the recorded movie) and switching OFF Steam's overlay do nothing to the picture itself, I decided to resize the 'Game Capture' via Transform>Edit Transform...>Size to 720 height. I fill some info by adding text layer and start recording. File - 2016-06-25 10-18-01.flv
4. While I got a good image, there is still black stripe at bottom of the movie. I decide to crop canvas Settings>Video>Base (Canvas) Resolution to 720 too (output as well). Some new info to the new text layer. Then recorded. There, I got strange 0 frames encoded message in my log O_o ("Output 'simple_file_output': Total encoded frames: 0") but file is OK! File - 2016-06-25 10-20-29.flv

So, where is your "jagged text"? I don't see it. My screens attached.

GomPlayer used to make this screens; then BMP to PNG compressed (all pics ~1.1 MB):


Edit: forgot to add, Settings>Video>Downscale Filter : Bicubic (Sharpened scaling, 16 samples) for all recordings.

 

[Steve Macintosh]

Hey Suslik, first of all thanks for taking the time and effort to try it out yourself and give me some feedback. I much appreciate that as I've been driving myself nuts over small details like this.
And while my real internet connection is down I just couldn't help myself but to check out your results right away using 3G.

So alright, yeah. First things first, after some googling yesterday late night I saw that Limbo actually has a native resolution of / is designed for 1280x720, and in my case running it in 1080p, it gets upscaled. So you not seeing any artifacts/jagged edges makes sense since your resolution already is quite close to Limbo's native resolution. I on the other hand am running on 1920x1080 and that's quite further away from the native resolution. I did however try to replicate your results, which turned out the same, so hopefully we're on the same page and it's not just my computer that's crazy (or me).

The issue I'm still having with this whole thing is that I would then be "forced" to play 720p to make it look decent, while 1080p on my desktop actually looks great. Add to the fact that DXTory/Shadowplay and Bandicam(of all apps) generate a video in 1080p that is pretty much identical to what I'm seeing on my monitor while playing. And that's what annoys me. Now I'm not saying that any of those programs are better than OBS, I've grown to love OBS and for the first time in maybe 20 years I actually want to be a part of a community/forum, that's why this bugs me so much. I want to see it through and find the "real" solution to this problem. Granted, this may be a very odd case, upscaled games and what not. Maybe Limbo is special in some ways, I don't know. Am I spending way more time fiddling with this than I will be playing/recording it? You bet.

One thing I did notice however, which I think is interesting. When using a canvas of 1920x1080 and you do the couple of alt-tabs to get the preview working, the preview is actually jagged as well. If Preview is a somewhat accurate representation of what will be recorded then it almost looks like it's missing some upscaling filter / antialiasing or a different type which the other apps I mentioned above do have. Could that be possible?

I began reading the OBS source yesterday but I didn't get to the finer details of the game capture hook/inject to see what it's actually doing, and now without proper net I guess I will have to pace myself in this matter.