Necro-ing this thread because I could have used this advice myself today.
I really appreciate
@Suslik V 's answer, but the wording left a little to be desired, as well as some steps having been left out. I'm assuming English is not their first language, so please allow me to elaborate on the steps to make them easier to follow
First, we'll set things up in OBS Studio to make things a little easier on us down the road:
- Create a new scene named something like "Camera Only", and add your camera to it with no filters or anything else in the scene
- Create another scene, the one in which you will be using the camera and "selective" chroma key (e.g. "Chatting")
- Add two groups, "Group 1" and "Group 2", respectively
- Add the SCENE that has your camera by itself to this new scene, twice
- Drag the "Camera Only" scene into each of the two groups
- Add a source (image, video, game capture, etc) to the scene that you want to show in the "selective" chroma-key area, to the very bottom of the source list
Alright, now to make the two image masks:
- Switch over to the "Camera Only" scene and right click on the preview area, then click "Screenshot (Source)"
- A message will appear at the bottom left of OBS Studio saying that the screenshot was saved to "C:\user\blahblahblah" , which will be the location set for recordings in your OBS Studio settings (Settings > Output > Recording > Recording Path)
- Open your favorite image editing software (GIMP, Photoshop, etc), I use
GIMP in my example,
and a couple of my instructions are GIMP-specific since I don't know how to do it in other software
- Drag the screenshot you just saved from OBS Studio into the image editor
- Select the area of the screenshot in which you want the "selective" chroma key to be shown
- You can use whatever tool you like. For polygonal areas with straight sides, I prefer the "Path" tool, then "Selection from Path", but you can use the rectangular select, ellipse, lasso, etc.
- With the paint bucket tool, fill this selected area with
BLACK
- Invert the selection so that everything
BUT the black section is selected (In GIMP this is Ctrl+i)
- Fill the rest of the image with
WHITE
- Export this as something like "BlackHole" or "ImageMask1" or whatever will help you know that this is the one with the
BLACK chroma-key area
- You can export it as either jpg or png, it doesn't matter
- Now with the white area still selected, switch your bucket tool to BLACK and fill in this selection
- I know the entire image will be black, we'll fix that in a moment
- Invert the selection again, and BEFORE we fill the selection with white, we need to make it slightly bigger so that when we apply the masks and chroma-key within OBS Studio, there won't be a 1px border around the area
- In GIMP, click "Select > Grow" and increase the size of the selection by 2-5 pixels (no more than 5 pixels) then click "OK"
- NOW fill this new selection in with WHITE
- Export this new image as something like "WhiteHole" or "ImageMask2", similarly to the first
Now why do we need both of these masks? We'll find out when we apply them
Now comes the fun but confusing part--applying the masks and the chroma-key:
- In OBS Studio, return to our scene that has the two groups and the source that we want to show through the chroma key
- Right click on the first group, highest in the list, and click "Filters"
- in the bottom left click the Plus sign to add a new filter and select "Image Mask/Blend", you can leave the name as the default
- On the right, leave "Type" alone and click "Browse" next to "Path"
- Locate the "White Hole" file and double-click it to add it to the mask
- By default, applying a mask tells OBS Studio that anything in the white area needs to shine through to the output, and anything in black gets hidden. Thus, with this first mask, we're telling OBS Studio that only the area where we want to put our green screen is going to the output for this first group.
- Click the Plus sign again and add a "Chroma Key" filter, leaving the name alone again
- You'll likely have to fiddle with the sliders within the chroma key settings to make the required color transparent, but that's outside the scope of these instructions
- NOTE: In my example image, my chroma-key area is *blue*, even though "Key Color Type" is set to "Green". The options are "Green", "Blue", "Magenta", and "Custom".
Select whichever applies to your situation!
- Close the filter window for the first group
- Right click on the second group and click "Filters"
- Add another "Image Mask/Blend" filter, this time selecting the "Black Hole" image
- Now with the black and white areas switched, this mask is telling OBS Studio that everything BUT the chroma-key area needs to be sent to the output. Separating the masks like this, and then only applying the chroma-key filter to the first group, is what allows us to select a particular area to chroma-key, rather than the entire scene.
And Voila, that should be it! Chroma-key should now be applied only to a particular area of the view from your camera.