Start with a simple test. Pick... say F11 for start/stop stream. Try it while OBS has focus. If it works, then open notepad, but make it so you can still see OBS, and try it again.
Keep trying with each hotkey you've setup until you find the keys that don't work.