f



More experimentation with single photo 2D -> 3D & other inverse problems in 1D/2D/3D DSP. YouTube demos.

This is really meant as the latest in a series I posted over irregular inte=
rvals since last summer, in which I'm going to try and explain some of the =
tricks used and call out some of the *real* issues that I'd like to tackle =
(like reverse "signal demasking" and reverse "layering")

These are the demos where some of the tricks are shown off. Bear in mind th=
at everything comes from single photos -- no triangulation, no built-in 3D =
models, etc. Lighting, coloring, texture, etc. are also inferred from the p=
hotos, not built-in or hard-coded.

[0] 2D to 3D of a "2D to 3D" video. Trial #2.
https://www.youtube.com/watch?v=3DA0mp7hCn5ww
We raid someone else's how-to-photoshop photo-to-3D site (it's a cottage in=
dustry these days) and turn a snapshot from *it* into a panning 3-D zoom-in=
.. It actually zooms inside the video frame.

[1] The Beast Stomp Rocks Chicago (literally)
https://www.youtube.com/watch?v=3DJdN76XYvvg0
The entire city scape of Chicago rocks in 3D. The cyborg voice in the backg=
round morphs from biological to fully robotic. No cityscape model. I left t=
he occlusion zones out in the open.

[2] Another Pretty Angel Smiles and Flirts
https://www.youtube.com/watch?v=3DYKvYir1ldb4
The face in the photo goes from a pout to a flirty smile, while slowly turn=
ing and then nods. The photo subject gives ironic twist to the designation =
"Artificial Reality" heralded at the end.

[3] Cosmetological Singularity, Preview 2
https://www.youtube.com/watch?v=3DkjPbFEVVvWw
A panning flyby of Mount Rushmore as the dark ominous shadow of a Cosmetolo=
gical Singularity sweeps across the monument (like seen in Independence day=
), leaving behind hot women in its wake. Darth Ninja narrates. The "paralle=
l universe" shadow is a 3D version of a cross-fade.

[4] (An earlier take on [3] featuring also Chicago)
https://www.youtube.com/watch?v=3DAEXrCyPvIK0

[5] ChiSprite1
https://www.youtube.com/watch?v=3DRuja5HpTIkA
Distant panning fly-by over Chicago as 40 some-odd sprites turn about in mi=
d-sequence. Full lighting, shadows, etc. present.

[6] ChiMedley
https://www.youtube.com/watch?v=3D_moLRJNR_Po
A medley of prior runs showing off various methods .. and all the loose end=
s. The 2 zoom ins on Medusa are without and with interweaving 3D layers.

[7] Obama gets pied in the face and becomes Oprah
https://www.youtube.com/watch?v=3DzAkibbGJ3HM
Oprah is a 3D sprite (she changes orientation as she flies by), flight path=
 determined by cubic splines. Recoloring takes place in mid-flight.

[8] 3-D "Wipe" Video Transition
https://www.youtube.com/watch?v=3DRxHLnocl8mA
The "wipe" effect, itself, takes place in 3-D. Several methods are demo'ed

So a few words on the ideas and methods...

(A) 2D to 3D:
Conversion to 3D normally requires 2 stages, as outlined in the opening of =
[1].
(1) Sampled estimates of depth (either by hand or by use of cues)
(2) Interpolation based on location and color (or other cues). The interpol=
ation method used "distance-based" weighting, with distance measured in bot=
h color space and pixel space.

Another general method may assume (as a first estimate) a generally spheroi=
dal shape for a selected image segment. There, the depth Z is made proporti=
onal to root(A^2 - R^2) where R is the distance to the edge of the image se=
gment and A the maximum value of R. Refined estimates could then be based o=
n shading, blur, etc. That was employed in [2], [5] and parts of [6].

(B) Application: Layering & Sprites
Once you have 3D's you can assign them to layers -- but the layers themselv=
es being 3D. That was used in part of [2], [5], [6] and [7] (and to a limit=
ed degree in the cross-fades done in [3], [4] and [6]). 3D overlapping laye=
rs are used in [3].

Once you have zooming, you have motion. The sprites move in [5] (and [6] an=
d [2]).

A consequence of this is that you can also move face muscles. In [2], the m=
otion is done by turning selective parts into 3D and panning the viewpoint.=
 The conversion to 3D was fully automated in [2] (as well as for the sprite=
s in [5] and [6]).

(C) Application: 3D versions of FXs: Cross-Fading, Wipe, Lighting/Shadows
As described above. Shadow-casting in [5] and [6] is done by a divide and c=
onquer strategy. Sprite-lighting done by self-shadow-casting.

(D) Recoloring & Equalization (& Re-Texturing)
A set of "recoloring" algorithms was devised that turn out to also provide =
quick and dirty solutions to other problems: lighting, fading-into-backgrou=
nd (the fog-over effect), limited texture-grafting, limited sheen lighting =
("specular reflection"), and colorizing black and white.

Both [5] and [6] have recoloring, which happened to pun the fading effect; =
Oprah recolors in [7] in mid-flight to become more white, like Obama.

The recoloring problem is not as well-defined as it appears because there a=
re *3* color dimensions. So, you're talking about rotations in color space,=
 which may not be what you want.

The 3 algorithms used:
(1) Statistical matching against against a fixed set of principle component=
s (basically YCrCb simplified). This assumes each two components have littl=
e or no cross-correlation -- which is usually the case.
(2) Statistical matching which assigned principle components in descending =
order of covariance. This, however, can turn a red into a blue or something=
 similar. But unlike (1) it will assign a best-fitting color to a monochrom=
atic image.
(3) Non-parametric (histogram) with model (1).

Statistical methods were similarly used to infer texture, particular for th=
e rocky faces in [3] and [4].

(E) Inverse Problems
The last demo, [6] is a medley of demos meant to show off the dirty laundry=
 and loose ends, plus some of the tricks. The 2 zoom ins to Medusa are with=
out and with 3D interweaving layering to show the difference. All 4 zooming=
 shows highlight the *serious* need for solving the "reverse occlusion" pro=
blem, though some of this is resolved by the 3D interweaving layering done.

The music was pieced together in [3] and [4] also applied an inverse method=
 algorithm -- here: reverse-sequencing individual elements out of another t=
rack and selectively resequencing them, but I won't say anything more about=
 that here.

That's the inverse of the sound-mixing problem.

A similar problem for images is to peel away the layers *with the mixing co=
efficients* for an image -- even to remove fog and smoke into a separate la=
yer. In addition, there is the related problem of "looking behind" objects =
in a photo -- the "Reverse Occlusion" (or Image Reconstruction) problems.

For both of these I'm trying to find a way to do them as divide and conquer=
 methods. In general it works this way:
(1) Decompose the signal (2D signal here) into 1/2 resolution + detail map.
(2) Apply the method at 1/2 resolution, recursively
(3) Scale it back up
(4) Use the detail map to refine and sharpen the results.

For image reconstruction, stage (4) is where patterns would also have to be=
 propagated (e.g. tilings or regular features).
0
federation2005
9/10/2014 1:01:01 AM
comp.dsp 20331 articles. 1 followers. allnor (8509) is leader. Post Follow

0 Replies
1124 Views

Similar Articles

[PageSpeed] 50

Reply: