This is really meant as the latest in a series I posted over irregular inte=
rvals since last summer, in which I'm going to try and explain some of the =
tricks used and call out some of the *real* issues that I'd like to tackle =
(like reverse "signal demasking" and reverse "layering")
These are the demos where some of the tricks are shown off. Bear in mind th=
at everything comes from single photos -- no triangulation, no built-in 3D =
models, etc. Lighting, coloring, texture, etc. are also inferred from the p=
hotos, not built-in or hard-coded.
 2D to 3D of a "2D to 3D" video. Trial #2.
We raid someone else's how-to-photoshop photo-to-3D site (it's a cottage in=
dustry these days) and turn a snapshot from *it* into a panning 3-D zoom-in=
.. It actually zooms inside the video frame.
 The Beast Stomp Rocks Chicago (literally)
The entire city scape of Chicago rocks in 3D. The cyborg voice in the backg=
round morphs from biological to fully robotic. No cityscape model. I left t=
he occlusion zones out in the open.
 Another Pretty Angel Smiles and Flirts
The face in the photo goes from a pout to a flirty smile, while slowly turn=
ing and then nods. The photo subject gives ironic twist to the designation =
"Artificial Reality" heralded at the end.
 Cosmetological Singularity, Preview 2
A panning flyby of Mount Rushmore as the dark ominous shadow of a Cosmetolo=
gical Singularity sweeps across the monument (like seen in Independence day=
), leaving behind hot women in its wake. Darth Ninja narrates. The "paralle=
l universe" shadow is a 3D version of a cross-fade.
 (An earlier take on  featuring also Chicago)
Distant panning fly-by over Chicago as 40 some-odd sprites turn about in mi=
d-sequence. Full lighting, shadows, etc. present.
A medley of prior runs showing off various methods .. and all the loose end=
s. The 2 zoom ins on Medusa are without and with interweaving 3D layers.
 Obama gets pied in the face and becomes Oprah
Oprah is a 3D sprite (she changes orientation as she flies by), flight path=
determined by cubic splines. Recoloring takes place in mid-flight.
 3-D "Wipe" Video Transition
The "wipe" effect, itself, takes place in 3-D. Several methods are demo'ed
So a few words on the ideas and methods...
(A) 2D to 3D:
Conversion to 3D normally requires 2 stages, as outlined in the opening of =
(1) Sampled estimates of depth (either by hand or by use of cues)
(2) Interpolation based on location and color (or other cues). The interpol=
ation method used "distance-based" weighting, with distance measured in bot=
h color space and pixel space.
Another general method may assume (as a first estimate) a generally spheroi=
dal shape for a selected image segment. There, the depth Z is made proporti=
onal to root(A^2 - R^2) where R is the distance to the edge of the image se=
gment and A the maximum value of R. Refined estimates could then be based o=
n shading, blur, etc. That was employed in ,  and parts of .
(B) Application: Layering & Sprites
Once you have 3D's you can assign them to layers -- but the layers themselv=
es being 3D. That was used in part of , ,  and  (and to a limit=
ed degree in the cross-fades done in ,  and ). 3D overlapping laye=
rs are used in .
Once you have zooming, you have motion. The sprites move in  (and  an=
A consequence of this is that you can also move face muscles. In , the m=
otion is done by turning selective parts into 3D and panning the viewpoint.=
The conversion to 3D was fully automated in  (as well as for the sprite=
s in  and ).
(C) Application: 3D versions of FXs: Cross-Fading, Wipe, Lighting/Shadows
As described above. Shadow-casting in  and  is done by a divide and c=
onquer strategy. Sprite-lighting done by self-shadow-casting.
(D) Recoloring & Equalization (& Re-Texturing)
A set of "recoloring" algorithms was devised that turn out to also provide =
quick and dirty solutions to other problems: lighting, fading-into-backgrou=
nd (the fog-over effect), limited texture-grafting, limited sheen lighting =
("specular reflection"), and colorizing black and white.
Both  and  have recoloring, which happened to pun the fading effect; =
Oprah recolors in  in mid-flight to become more white, like Obama.
The recoloring problem is not as well-defined as it appears because there a=
re *3* color dimensions. So, you're talking about rotations in color space,=
which may not be what you want.
The 3 algorithms used:
(1) Statistical matching against against a fixed set of principle component=
s (basically YCrCb simplified). This assumes each two components have littl=
e or no cross-correlation -- which is usually the case.
(2) Statistical matching which assigned principle components in descending =
order of covariance. This, however, can turn a red into a blue or something=
similar. But unlike (1) it will assign a best-fitting color to a monochrom=
(3) Non-parametric (histogram) with model (1).
Statistical methods were similarly used to infer texture, particular for th=
e rocky faces in  and .
(E) Inverse Problems
The last demo,  is a medley of demos meant to show off the dirty laundry=
and loose ends, plus some of the tricks. The 2 zoom ins to Medusa are with=
out and with 3D interweaving layering to show the difference. All 4 zooming=
shows highlight the *serious* need for solving the "reverse occlusion" pro=
blem, though some of this is resolved by the 3D interweaving layering done.
The music was pieced together in  and  also applied an inverse method=
algorithm -- here: reverse-sequencing individual elements out of another t=
rack and selectively resequencing them, but I won't say anything more about=
That's the inverse of the sound-mixing problem.
A similar problem for images is to peel away the layers *with the mixing co=
efficients* for an image -- even to remove fog and smoke into a separate la=
yer. In addition, there is the related problem of "looking behind" objects =
in a photo -- the "Reverse Occlusion" (or Image Reconstruction) problems.
For both of these I'm trying to find a way to do them as divide and conquer=
methods. In general it works this way:
(1) Decompose the signal (2D signal here) into 1/2 resolution + detail map.
(2) Apply the method at 1/2 resolution, recursively
(3) Scale it back up
(4) Use the detail map to refine and sharpen the results.
For image reconstruction, stage (4) is where patterns would also have to be=
propagated (e.g. tilings or regular features).