So far NVENC looks the Best when streaming directly from the PC (GTX 1080ti 9700k) But when using the Capture Card X264 at Fast Looks better then NVENC at Quality for streaming the PS5
This is a strange and probably a invalid/misleading observation, because you're comparing apples and oranges here. The input is different and independent from the output.
Your input is either the capture card (and your console), or game capture from the PC. As soon as a video is processed by OBS, regardless of the source, it is treated the same. The encoder is used for everything. It doesn't regard the origin of the source - it just encodes.
You might see a difference, if the resolution of your console is different to the resolution of your PC.
You're using nvenc on a GTX 1080 Ti, this will produce a quality roughly equal to x264 with the "veryfast" preset. With the "fast" preset, x264 is able to create better output. (this was verified by not doing visual comparisons but computation using the PSNR as criteria. In short: by comparing the raw video to the encoded version of the same video and computing the difference - the PSNR. The lower the difference, the better the quality). This is true regardless of the source - be it a capture card, be it game capture.
Nvenc on newer RTX cards will produce better quality. It is said that nvenc on RTX 2xxx will produce roughly the same quality as x264 medium to slow. RTX 3xxx cards with produce about the same as x264 slow.
You might perhaps experienced something different. In case you capture a game, this game and x264 are competing for the CPU. If there isn't enough CPU resources for both of them, the quality of the game (stutter) and the stream (lagged/lost frames) might suffer. With Nvenc, which is running on a dedicated circuit on the GPU, this doesn't happen.