Don't choose Facebook in Settings->Stream. Facebook limits the output resolution. This is enforced by OBS even if you don't intend to stream. Either set any other streaming service or activate the "Ignore streaming service setting recommendations" checkbox in Settings->Stream.