If I understand correctly, you're referring to the
Scene Transitions dock that by default is located at the bottom right next to
Controls dock.
The gear wheel works on the item that is currently selected at the dropdown. So if the dropdown shows the built in
Cut transition, then there are no actions to perform on it (it cannot be renamed or deleted etc.) so clicking the gear wheel icon won't do anything.
To add a new transition, open the dropdown and select the desired
Add option. In older versions of OBS Studio use the plus icon. On these transitions you added, the gear wheel icon will work.
Version 27 and 25