3D capture in performance

Capturing 3D data from moving scenes is a hard problem in itself, but the harder problem is how to integrate it with a performance in real time. First you have to hide the 3D capture process from the audience, and then you need to present something related to the 3D capture in an engaging way.

Re-projecting coarse effects can be approximated with a multi-camera system, and fine details are dangerous due to system latency.

The real question is: what can you do with 3D data that you can't do with 2D images? The most interesting idea I have: you can cast shadows on a 3D form in a way you can't on a 2D form.

Removing Image Noise

Gray-level images for foreground/background are often afflicted by fine noise when they are thresholded.

I've used a few different approaches to dealing with this:

• blurring before thresholding
• median filtering before thresholding
• morphological filtering after thresholding
• distance transform after thresholding, followed by thresholding again as a more continuous morphological filtering algorithm

Unfortunately, these all have the same problem: inner corners are rounded. These algorithms can't tell the difference between a black pixel in an inner corner and a black pixel that is just noise.

Perhaps what we should really be using is something like bilateral filtering (in Photoshop, "Surface Blur"). Bilateral filtering preserves sharp edges, while blurring large undefined regions. Unfortunately, bilateral filtering creates a sort of "glow" that still has issues with corners.

Maybe we need something more like "smart blur"? It seems to not propagate across edges...