That CPU is ancient, as noted above. Also, you are low on RAM, even if that system is capable of doing GPU encoding offload.
And then there is the consideration of system performance impact if you are on a HDD instead of SSD
When gaming, take a look at the Operating System Performance Monitor (in Task Manager, or Resource Monitor) and take note of CPU and RAM utilization, as well as Disk I/O and average latency.
I'm suspecting your system is already bottlenecked... adding _any_ recording/streaming software would then only make overload situation worse.
With detailed knowledge you _might_ be able to heavy optimize (reduce load) at Operating System level, then cap game performance to leave a buffer to allow resource (CPU, RAM, GPU, etc) for Recording software to work... but even then, as noted above, you have one of the lowest end CPUs when released, and limited RAM... if you enjoy a technical challenge... good luck.
IF you can try/test/get for free some more RAM (assuming system supports more) and a SSD instead of HDD, and you have the time & inclination, could be an interesting exercise [this of course requires you are capable of working inside computer with damaging anything, and know how to setup an Operating System (OS) from scratch, or how to move an OS from one disk drive to another, setting boot flags, etc. But I would NOT spend money (more than 'pocket change') on such an old system
sorry to be the bearer of what I'm sure in disappointing news
