These "overlay windows" are not the usual app or dialog windows. They cannot be explicitly found or selected for capture, they are internal. As I wrote: the only way to capture these kind of app is display capture.
By the way, window capture will never capture any popup menu or main menu, in almost no app. In almost all apps, these are created as tiny windows overlaid on top of the usual app surface. This is not specific to Photoshop. The only exception is when the app is "painting" the menu within the main window. Most games do it this way with their ingame menus.