Keep It Real: a Window to Real Reality in Virtual Reality

1 Introduction

Traditional VR technologies usually emphasize creating an entirely immersive environment, visual inputs of the real realities are usually discouraged [Burdea and Coiffet, 2003; Bowman and McMahan, 2007]. Studies have attempted to introduce virtual mirrors to facilitate embodiment [Spanlang et al., 2014], rehabilitation [Sato et al., 2010] and education [Blum et al., 2012], but almost all of them reflects the virtual space (e.g. the body of the avatar in the virtual room). We demonstrated in our system, however, that the introduction of such a virtual window or mirror reflecting the real world can bridge the two realities in a surprisingly beneficial synergy: (1) introducing a virtual window or mirror which the user could activate or deactivate any time could encourage the user to actively attend to and seamlessly receive real-world feedbacks where virtual rendering was challenging; (2) projecting a geometrically accurate representation of users' image and reality into a movie could create an illusion that the user is presently sharing the same universe with the virtual characters - breaking the fourth wall.

1 The data and code to reproduce the video application can be downloaded at https://github.com/doerlbh/V2R

2 Projective Geometry

As in Figure 1, the projective geometry problem is outlined as follows: the user is the observer O who has a reflection of a virtual mirror image at O′ from a window or mirror on a virtual plane (e.g. a virtual wall); camera C with unknown distances covers the user in its field angle, and we wish to solve the perspective transform (the homography), from the camera view plane, (i) to the virtual camera view plane by the mirror, (ii) to the virtual plane where the mirror is located, and eventually (iii) to the image space where the observer sees. Solving (iii) is straightforward [Coxeter, 2003], but solving (i) and (iii) is nontrivial since depth estimation in a video and the camera at the same time is an innately chanllenging problem. In another word, there is not a unique homography for the projection of the mirror image, but a ray of solutions for each point. We tackled the issue by introducing an additional scaling factor to constrain the homography according to each pairs of image frame. The scheme follows Figure 1.

3 Virtual-to-Real (V2R) Mirrors orWindows

The V2R framework involves two main components: pattern matching (where to put our windows) and image warping.

The pattern matching problem is defined as a feature detection plus classification problem (in case of multiple windows or mirrors). Since we would like to apply it to real-time camera feed (as in AR and VR), there are two major difficulties: (1) a short response time requires the model to slim and fast; (2) noisy, blurring and shaking image frame in the video due to hand-held motion. Neural networks tend to be slower so we resort to the signal processing methods (such as filtering) to speed up the computation. For demonstration purpose, we chose the classical cross-shaped black-white circle as markers to benchmark our methods, and obtained robust performance with template matching, Harris and Hough operators [Brunelli, 2009] after applying Guassian and median filters.

Figure 3 summarizes the steps involved in the image warping: first, rectify the camera feed with respect to the virtual space; then, crop the image given a proper camera field angle; and finally project it the image space given the computed homography. This framework turns out to be fairly robust and flexible to apply to a wide range of applications. For instance, we can introduce a V2R mirror into a movie in every scene where a target painting appears, as in the example of the TV series "Friends". In VR, we can also make the postboard a V2R window whenever a button on the control bar is pressed.

Figure 2 confirmed the validity of our method: for instance, in the rightmost case, the virtual perspective is clearly looking down from above, and the warped image is indeed as if the observer was looking down and can only see from the mirror the bottom half of the body. V2R can be applied in realtime in a timely fashion (more example videos and images can accessed at https://github.com/doerlbh/V2R).

As an important concept of AI, the sense of presence can effectively engage the interaction between the human with the virtual agents in virtual environments. V2R aims to create such a synergy by bridging the two spaces seamlessly.

4 The Demonstration System

V2R provides extensive entertainment capacities and interactive possibilities, such as activating and deactivating the window given certain features or user commands (in VR), experiencing the thrilling emotion of being in the same room with the serial killer in a blockbuster horror movie only a mirror reflection away (in movies), blending your face with an ancient guru master through the secret portal of a crystal ball (in AR), priming customers to buy a product by projecting users' real world objects into a commercial (in online streaming ad). In the demonstration, we will provide a pool of interesting short videos, where the mirror location detection have been properly trained and tuned, such that the user can see themselves instantly within these movie clips (e.g. as in Figure 2).

5 Ongoing and Future Directions

Exciting next steps involve developing a wide spectrum of interactions around V2R and accompany intelligent modules such as gesture recognition (imagine creating a virtual window to the real world by drawing a square in the air with your hands), video conference with virtual windows to multiple camera feeds (literally as if in the same room), real-time rendering of virtual objects given the real-world camera feed (bring an apple from the real life to the virtual space) etc.. Another direction is to improve the projective geometry: these mirrors and windows projected from single-camera 2d images are currently pseudo-representations of what one should see from the exact computed point of view in the real world, as the single-camera setup make it hard to perform a 3d reconstruction. A useful extension of V2R is to introduce multiple cameras, such that the camera depth can be properly estimated along with a 3d projection of the details that could be revealed but hidden in 2d representation (e.g. an occlusion). These improvements should illicit highly interactive user experience and engage more users and research into the field of intelligent virtual reality interactions.