You can fix this by either removing the offending OBS config file or by editing the file and removing the offending config option. Depends on your Windows skills. Just removing a file is easier.
Open Windows Explorer, go to directory %appdata%\obs-studio\basic\profiles
In this directory, you see one directory for each profile you have. Enter the directory of the profile that was active when you set the invalid value. Locate the file basic.ini. Check the file date to make sure you found the correct file: it must have the file date of your failed edit. Delete the file basic.ini. This way, you lose customizations of some video settings from that profile, but no scenes. You may have to do your customizations again.
Beware: in the obs-studio directory there are many basic.ini files, all of them with different purpose. Be sure to remove the correct file.
If you have a text editor on your system in addition to the default notepad app, instead of just deleting basic.ini, you may as well load the basic.ini into your editor, locate the [Video] section and remove all options from that section that start with FPS. Just remove them, they will be recreated automatically the next time you start OBS. Unfortunately, the default Windows notepad app isn't able to load the file correctly. You will have to use an editor that is capable of handling textfiles with unix-like linefeed characters.
