I'm not actually going to do this for probably a couple of years, but I'm looking for a "general feel" so I at least have some idea of where to go when I do get there:
Full-covering cosplay or theatrical costume, with cameras in the eyes and screens for the actor inside.
Now that I see some "3D cameras" on the market, I suspect I'll use one of those with a reasonable spacing. (not the super-wide ones for machine vision) That should eliminate the apparently-sensitive problem of aligning two separate cameras.
I thought briefly about a VR headset for the actor's display, but space is going to be at a premium here, and it looks to be a lot more compact for the same effect to use two separate retinal projectors. (not the "super-high resolution" kind of retinal, although that would be nice, but to project an image directly onto the retina instead of making you focus on something that is literally, physically present in front of you) This presents a *slight* problem in that the two projectors are probably two separate 2D displays instead of one 3D display like the VR headset is...and it brings back the alignment problem.
Also, I want to put some telemetry on it, like the main battery percentage, and maybe temperature as well (more for the actor's health than for the electronics), etc. So that's a live overlay, that needs to not be tiring on a 3D display. (whether a purpose-built display or cobbled together) I'm thinking to have it be an arbitrary graphics screen that gets keyed over the cameras, so that might be a 3rd logical display for a custom app, that feeds a "Display Capture" source in OBS???
(If that's the "primary" display, with a system menu, OBS's control screen, and all that, then it might be handy for other things too, as an internal PC desktop that is keyed over the cameras. Wireless mouse and keyboard, etc. Be careful with it, of course, as a completely-opaque keying will cause effective blindness in direct use...)
And the latency from the cameras has an absolute hard maximum, since it completely replaces the actor's vision. Telemetry can be delayed or have a slow update rate, but the signal path from cameras to eyes absolutely MUST be real-time and in sync.
In addition to the live 3D video + overlay, I also want to record all 8 tracks of audio (4 stereo pairs) from a single USB sound card. So the end result is a single 3D video file (guaranteed sync) that includes multitrack audio that can be post-processed into a stereo master.
(That "single USB sound card" is actually a custom DSP with tap points for the different signals that it's working with, hence the post-processing. Three stereo pairs of raw sources, and one "finished candidate" that is produced live in that DSP for the actor's benefit.)
---
So:
1. Can OBS do that? Take a 3D camera, key a 2D source onto it, then send each eye to a separate display, and record that with 8-channel audio from a single device, all in a single 3D video file?
(not 7.1, in case of any special processing in that mode, but 4 equal stereo pairs or even 8 equal mono's from one USB device)
1b. If not, what can?
2. What kind of hardware would I need to do that? I'm sure my "future proof" laptop can do it, given the right software, but it's huge and heavy. At the moment, there is a "Stereo Pi" project that tries to do part of this by exposing both of the camera ports on a Pi Compute Module, but the processing power just isn't there yet, even with the very welcome jump from the Pi 3 to the Pi 4. I've tried to capture 2D 1920x1080x30p in OBS on a Pi 4, using a HDMI -> USB 3 capture card, and it failed miserably.
So I need something better than a Pi 4. Preferably fanless, and powered from a single DC supply of some kind. (I don't expect to use any power-saving features despite running it on a battery, as it'll be processing live video literally all the time.) Does something like that exist yet?
Full-covering cosplay or theatrical costume, with cameras in the eyes and screens for the actor inside.
Now that I see some "3D cameras" on the market, I suspect I'll use one of those with a reasonable spacing. (not the super-wide ones for machine vision) That should eliminate the apparently-sensitive problem of aligning two separate cameras.
I thought briefly about a VR headset for the actor's display, but space is going to be at a premium here, and it looks to be a lot more compact for the same effect to use two separate retinal projectors. (not the "super-high resolution" kind of retinal, although that would be nice, but to project an image directly onto the retina instead of making you focus on something that is literally, physically present in front of you) This presents a *slight* problem in that the two projectors are probably two separate 2D displays instead of one 3D display like the VR headset is...and it brings back the alignment problem.
Also, I want to put some telemetry on it, like the main battery percentage, and maybe temperature as well (more for the actor's health than for the electronics), etc. So that's a live overlay, that needs to not be tiring on a 3D display. (whether a purpose-built display or cobbled together) I'm thinking to have it be an arbitrary graphics screen that gets keyed over the cameras, so that might be a 3rd logical display for a custom app, that feeds a "Display Capture" source in OBS???
(If that's the "primary" display, with a system menu, OBS's control screen, and all that, then it might be handy for other things too, as an internal PC desktop that is keyed over the cameras. Wireless mouse and keyboard, etc. Be careful with it, of course, as a completely-opaque keying will cause effective blindness in direct use...)
And the latency from the cameras has an absolute hard maximum, since it completely replaces the actor's vision. Telemetry can be delayed or have a slow update rate, but the signal path from cameras to eyes absolutely MUST be real-time and in sync.
In addition to the live 3D video + overlay, I also want to record all 8 tracks of audio (4 stereo pairs) from a single USB sound card. So the end result is a single 3D video file (guaranteed sync) that includes multitrack audio that can be post-processed into a stereo master.
(That "single USB sound card" is actually a custom DSP with tap points for the different signals that it's working with, hence the post-processing. Three stereo pairs of raw sources, and one "finished candidate" that is produced live in that DSP for the actor's benefit.)
---
So:
1. Can OBS do that? Take a 3D camera, key a 2D source onto it, then send each eye to a separate display, and record that with 8-channel audio from a single device, all in a single 3D video file?
(not 7.1, in case of any special processing in that mode, but 4 equal stereo pairs or even 8 equal mono's from one USB device)
1b. If not, what can?
2. What kind of hardware would I need to do that? I'm sure my "future proof" laptop can do it, given the right software, but it's huge and heavy. At the moment, there is a "Stereo Pi" project that tries to do part of this by exposing both of the camera ports on a Pi Compute Module, but the processing power just isn't there yet, even with the very welcome jump from the Pi 3 to the Pi 4. I've tried to capture 2D 1920x1080x30p in OBS on a Pi 4, using a HDMI -> USB 3 capture card, and it failed miserably.
So I need something better than a Pi 4. Preferably fanless, and powered from a single DC supply of some kind. (I don't expect to use any power-saving features despite running it on a battery, as it'll be processing live video literally all the time.) Does something like that exist yet?