3D capture in performance

Capturing 3D data from moving scenes is a hard problem in itself, but the harder problem is how to integrate it with a performance in real time. First you have to hide the 3D capture process from the audience, and then you need to present something related to the 3D capture in an engaging way.

Re-projecting coarse effects can be approximated with a multi-camera system, and fine details are dangerous due to system latency.

The real question is: what can you do with 3D data that you can't do with 2D images? The most interesting idea I have: you can cast shadows on a 3D form in a way you can't on a 2D form.