I just tried it with Display capture and that works. ... Something with Game capture ... It has worked in the past
The older methods of capturing all of what a specific device does - with a literal, pedantic understanding of that - are still more reliable than the newer ones that try to be more specific.
The Application Audio Capture also has problems, ranging from not picking it up at all, to technically getting the correct audio but inserting unusable crackling along with it.
So if you can use the more general methods, and control by other means what gets into that hose, it tends to work a lot better.
could it be base resolution and output resolution are different?
If those are different, then it'll scale from base to output. They don't have to match, and the scaler seems to be pretty well bulletproof. So that's not the problem.
Though I don't see why you would have them different except to trick something like YouTube into giving you more bits per pixel than it otherwise would. You're not adding more useful information, despite the higher number of pixels or bits.
And to scale to a smaller output size, also seems like a specialized case to me. Then you're throwing away information that you spent some CPU time to create and can't get back.