OBS unable to match full RGB signal from capture card

[NathanF]

I have both a Nintendo Switch and Avermedia Live 4k capture card set to full RGB range. OBS outputs a washed out image with incorrect levels (the background should be full black) while Avermedia's RECentral4 captures look correct when using the same full/full settings:

Unless I'm doing something wrong with my settings it seems like OBS has a bug with capture card RGB levels. The only way to get correct captures seems to be using limited on both devices, which isn't ideal since I use passthrough to a PC monitor and may end up with an onscreen RGB mismatch while playing
(the reason I don't just use RECentral is that I need a replay buffer).

If the problem is user error on my end I'd be happy to know so that I can fix it:

 

[Suslik V]

things that needs to be checked twice (IMHO).

From the log I see HDR-capable monitor.

21:06:53.380:         bits_per_color=10

From the log I see "tonemapping" feature (this name usually means HDR to SDR conversion):

21:06:55.293: DShow: AVerMedia tonemapper enable=1

"...RGB signal from capture card" words from the header misleads me, because in the log - NV12 is set everywhere. I assume, that you trying to capture YUV with standard 4:2:0 chroma subsampling, not RGB. So, conversion from RGB to NV12 happens inside the capture device (or on driver level of it). If you have good old 8-bit monitor - try to plug it instead (I hope this disables any additional tonemapping).

If possible, please, upload to the device test images (for range and chroma subsampling) and try to capture them. For 8-bit videos (aka SDR) you can use https://obsproject.com/forum/resour...t-color-range-settings-guide-test-charts.442/

Edit: And where you see washed out colors: 1) In OBS Preview; 2) in media player?

 

[NathanF]

I see the washed out appearance in both the OBS preview window and my video editing software, Davinci Resolve (set to the default rec 709), and I also saw it when importing the clips to Hitfilm Express. The only place it does not appear washed out is when played in VLC- the background looks black there, which is strange because if I export the clips from Resolve first and then open them with VLC they do appear washed out.

I don't have a second monitor and I don't think it's possible to load images on the Switch without hacking, but I did try capturing from the Steam Deck. It should output full RGB by default, and when playing the "1920x1080_444_709_tv_10sec_10sec" video (using VLC from within Steam OS) it appeared washed out in OBS- in the OBS preview and the captured video itself. I've attached the direct capture of that here:

908.7 KB file on MEGA

mega.nz

I tried viewing the images in desktop mode but the Deck makes that very awkward, since the default screen is actually sideways and cloning it to a monitor results in a weird limit to screen size, but this is what scrolling through those looked like. It appeared the levels were correct in desktop mode? https://mega.nz/file/WFsX3KBC#iVpzQBoTv-oZH-uJMLiE3JrEtqI5b_JbTGCT1pFWF_0

I realize this isn't ideal but I'm not sure I have any other devices that can feed test patterns through the capture card. My blu-ray player can also view the patterns but copyright protection prevents me from getting any captures.

Here are the captures I took the screengrabs in the first post from-
RECentral: https://mega.nz/file/PJ91nLbA#mYsKL-LjA43KJugP8yOVxIu4n-73luIgpd2alf3RKMU
OBS: https://mega.nz/file/uAFBiYRT#We0bVwV6bn3DoeFkAupo9rJ8RofQUNg9doLiEN9agqU
I also attached the log from recording on Deck, in case anything was different there.

 

[NathanF]

One other thing to note- the card outputs a default image when no source is connected. Even this image is washed out with a gray background when setting the card to full RGB in OBS:

76.4 KB file on MEGA

mega.nz

 

[Suslik V]

Because you see washed out* image in OBS Preview, I assume you have wrong input for OBS. In other words your capture card outputs to DirectShow interface Limited(Partial/TV) range picture, and Video Capture Device source of OBS should be configured to accept Limited(Partial/TV) range, not Full.

In your first file ("Deck desktop 1920x1080_444_709_tv_10sec_10sec.mkv", 908 KiB) I see next issues:
1) I see upscale, that lowers final resolution of the image. I don't know where it was scaled up/down, maybe you shared wrong file? Reset Transformations of the OBS source, if there were any. Make sure that input/output resolution for the capture device is the same, and try again.
2) I see Limited (Partial/TV) range video without proper scaling. OBS expects Full range but input picture is TV range.

Again, are you sure that you need NV12 as output from your capture device (that feeds to OBS)? Maybe you are looking for BGR24, XRGB or something similar? Some capture cards can do RGB output up to 1920x1080@60 (about 356 MiBytes/sec of data).

Anyway, you need to ask manufacturer of the device how to configure output of your capture device, so it will output full range picture to DirectShow. By default, many YUV color planes (like NV12) has limited range colors. RGB, usually, full range but sometimes can has limited range too.

Edit: * washed out - here, it likely, "shaded" a lot.

 

[NathanF]

NV12 seemed to be the default in the advanced settings page, I didn't have a particular reason to choose it. I changed to XRGB and I think full RGB output looks correct now- the backgrounds of loading screens are full black at least- but without reliable ways to check patterns it's hard to say if the levels are correct. I'll try this for now and if the results continue to look OK I'll consider it resolved.

Should I also change the format from NV12 in the advanced settings to something else?

RECentral was using NV12 when it produced correct output at full RGB, so it seems to be possible to do with this card. These are the other options offered by RECentral:

OBS offers XRGB instead of RGB24... does that matter? Decode formats have been sort of a blind spot for me because I've never had to worry about them before:

The Steam Deck is a mess for this kind of testing- desktop PC mode was set to 1080p output but it's nearly impossible to view without scaling due to the weird framing issues that limit the image to a portion of the screen.
In the default Steam OS mode resolution is automatically set and because the card is 4k the output is set to 4k. I can try to force a manual resolution later when I get time and see if that works. Should I be playing the videos marked "tv" or "pc" for this?

Thanks for the help

 

[NathanF]

I did manage to capture rec 709 test patterns on the Switch YouTube app, which is about the jankiest way to do this because copyright protection only allows a few seconds before it kills the signal, but they seem to display the levels correctly when using XRGB. I exported this frame from Resolve:

The VLC player crushes the blacks so that no bars display, it definitely seems to have issues with its output. Every other editor and player I've tried (including the Windows Photo app) shows the levels correctly. Because VLC seems unreliable, I don't think any further Steam Deck tests would be useful, since I depended on that player there.

Based on what I'm seeing on Switch I think the issue is fixed, though. The native screen and my captures look very similar now, enough that the differences I'm seeing are probably due to the cheap LCD vs my monitor being OLED.

These are the raw clips if you want to check them, the patterns only animate near the end

File folder on MEGA

mega.nz

Thanks again for the advice

 

[Suslik V]

Should I be playing the videos marked "tv" or "pc"...?

If player capable of playing (properly) the Full range then it doesn't matter. Usually, range is scaled up/down on the fly by video player, and missing levels aliased between middle tones. For example, point that actually should be rgb (4, 4, 4) becomes rgb (3, 3, 3); rgb (244, 244, 244) becomes rgb (243, 243, 243) but this happens in zones that itself are hard to see without side-to-side comparison (0 to 16 are very dark points). And after compression with loss you may not notice the difference at all.

OBS offers XRGB instead of RGB24... does that matter?

No, it doesn't. It compatible. XRGB means alpha channel discarded. RGB24 means no alpha at all. Neither your capture card not OBS (v30 RC1) do not supports alpha for output.

Should I also change the format from NV12 in the advanced settings to something else?

No. In OBS this Advanced setting specifies format of the canvas and thus format of the main output. And there is no matter what sources were added to OBS.This is how OBS works. To combine pictures of NV12 sources (for example .mp4 movies) and RGB sources (.png images) OBS needs to convert them to same base. And this is done on the fly with OBS own shaders that are run by GPU (in other words OBS prepares textures and GPU combines them layer by layer, then final picture can be encoded with the chosen encoder). So, by setting Color Format in Advanced settings of OBS - you setting the main base (output, canvas - where all sources are combined). Many encoders/editors/online_services has its own restrictions, so improper setting there will end up in wrongly decoded file or bad performance (including "format/encoder not supported"). If you looking for "lossless" formats for master-copy creation of something unusual, then surely you should avoid any sub-sampled formats (NV12, I420 etc) and you should look for compatible encoders, compatible editors etc.

RECentral was using NV12 when it produced correct output at full...

The file you have posted ("RECentral Full.mp4", 4.10 MiB) is likely limited range video. It is hard to tell if source was full range or not, because properly scaled color range almost unnoticeable for me.