Initially I tried to do the 4096 res I was streaming at the other night directly with OBS and I wanted to try with Dxtory.
I tried again making Dxtory scaling and output 1 the same res, and having the same res in custom in OBS. I still got the red box.
I then tried to put 1920 into the same fields and having the game in windowed mode as I usually would (stretching to top/bottom, left/right on one monitor on a Surround enabled desktop). There is still a red box.
I just reset to non Surround so that each monitor's app windows just maximize to that window. I wish Surround wouldn't expand all windows across all 3 screens. I thought Ultra Mon/Matrox's software that can be used by anyone was supposed to help with this but I can't find a way.
With non Surround and all fields set to 1920x1080, and the game in windowed mode, Dxtory showing it is working with the brown FPS running, I still get a red box in OBS.