I see the nvidia encoder says "new" on it. Is there a way to access the old one to see if that resolves it?
Short answer: yes, you can force the new NVENC encoder to fall back to the old encoder by checking the "Rescale output" box while in Advanced output mode, even if you don't actually change the scale. The UI won't show that it's using the old encoder, but the log file will show "[jim-nvenc] scaling enabled, falling back to ffmpeg".
Context: The "old" NVENC used FFmpeg to perform the encoding, which meant that encoding required copying rendered frames from OBS on VRAM back to RAM for FFmpeg to encode. The "new" NVENC (aka jim-nvenc) keeps rendered frames on the GPU before sending them to the GPU encoder, and is much more performant as a result. The reason the old encoder is even still around is because it does not have the ability to rescale the output on the encoder side. FFmpeg does have this ability, and people made use of it so that they can record at one resolution and stream on a different resolution. But generally speaking, for performance reasons, we advise you not to use the old NVENC.
I'm also going to try just switching to processor encoding, but feels like it would be a shame not to use the GPU I bought for explicitly this purpose.
I think that's a fair sentiment, though I would be curious if the issue is still present on software encoding. My current operating theory is still that there's something strange going on with one or more of your sources somewhere. The reason I think this is because, in your last log, the recording session that exhibited the problem showed "Number of lagged frames due to rendering lag/stalls: 15", which means one or more of your sources was running slowly and was not getting back to the OBS renderer fast enough to generate a new frame in time, which caused the hitching. This is confirmed by the profiler, which shows that at least one cycle of tick_sources took more than 7 seconds to complete ("max=7133.35 ms"). tick_sources is the section of code that basically says "hey sources, we've gone enough time to get another frame, so each of you go ahead and give me your next frame so I can render a single frame with all your new updates".
Now that I'm looking more closely at your sources list, if I had to make a guess at which source is the problematic one, I would blame it on the "cannon" source, not because of the virtual camera filter, but because it's pulling its frames from the EOS Webcam Utility. Generally speaking, when you're getting video frames from a nicer DSLR or mirrorless camera, it's better to use the camera's HDMI output and use an HDMI capture card. The EOS Webcam Utility instead grabs frames from the camera over USB (often USB 2.0), which can be much slower and less performant, and possibly lower quality.
The main problem on the Canon side is that, to my knowledge, a large number of their cameras do not provide clean HDMI out from their HDMI out ports. I'd need to know the model of your camera to know if a) it's using USB 2.0 or 3.0, and b) if clean HDMI output is an option for it.
A quicker way to confirm if that source is the problematic one would be to completely delete the "cannon" source from all your scenes and see if you can still reproduce the issue. Again, I recommend duplicating your scene collection before doing this, so that you don't have to set everything back up again.
