3D Studio

Omni-Directional (Free-View) 3D Studio

Current 2D display systems are restricted to view the entire scene from a single perspective besides such systems cannot represent the depth of the scene to the viewer. Free-view 3D studio enables the 3D visualization of the scene at all possible angles from 0o to 360o where the depth of the scene is presented to the viewer as if he/she is inside the scene.

In order to have 3D perception, left view is acquired from the left and right view is acquired from the right cameras and each view is directed to the corresponding eye of the viewer. Since it is infeasible to use a left and a right camera for each view point (0o-360o) inside the scene, most of the possible perspectives need to be interpolated from other view-points observable by the cameras.

The quality of interpolation(‘virtual views’) depends on the number of cameras, the quality of the hardware as well as robustness of the intermediate processing steps such as camera calibration, depth estimation and incorporated rendering algorithm.

There are several reasons to build up such a high-tech studio and the followings are some of them:

  • To capture the depth of the scene therefore to enable more efficient use of human visual system in multimedia applications(teleconferencing, movie production),
  • To avoid missing the details of a scene by using 360o of viewing angle,
  • To observe the scene more realistically,
  • To keep pace with the current popular 3D technologies such as MAX and Real 3D in movie theaters and enable them in commercial TV sets,
  • To build up a corner stone for the holographic TV technology that is currently being developed by 3DTV project of EU.

Of course there are still many reasons that can be written into the list. These are the ones that come to mind at first.

 

 

 

Description of the Research

 

The main objective of our research is to develop a camera/image processing system for acquiring high quality omni-directional 3D models of natural-non limited scenes and to achieve flexible and user-friendly 3D studio technology.

 

Secondary objectives aim for the speed and the streaming capabilities. The studio may have broader usage when depth estimation and rendering applications can be implemented in real time (CUDA coding may fit best). Additionally, the resulting 3D multi media may need to be transferred through a network (internet) which comes with the price of bandwidth restrictions to deal with.

 

 

Applications

         
- 3DTV

There are recently many improvements in cinema technology such as IMAX and Real 3D. 3D Movies (Avatar, Ice Age 3...) have been presented and been enjoyed by many people all over the world. The project aims the same improvements for off-shelve 3D displays at homes.

         
- Medical Applications

3D recording of a surgery will be beneficial for the education of medical students. Besides, with the incorporation of free-view, the students will be free to watch the surgery at any possible viewing point and won’t miss any details.

         
- Surveillance Applications  

3D technology can be well adapted to the recent surveillance applications. Multi-views from stereo cameras can be used to obtain super-resolution images which are useful in face-recognition
applications.

 

Research Directions

1.      Studio Setup

Stereo cameras will be used instead of mono cameras to have better depth estimates at each view-point. The number of stereo cameras necessary to have adequate free-view 3D reconstruction is an important task to find out. Also, the quality of the cameras and lenses to be used is also a crucial choice to be made.

2.      Camera Calibration  

There are two ways of calibrating the cameras; fixed and auto calibration. In this project, fixed calibration is used to estimate the intrinsic parameters of the cameras (lens distortion, pixel size…) and auto calibration is incorporated to estimate the extrinsic parameters (positions of cameras w.r.t each other).

3.      Depth (Correspondence) Estimation

The goal is to locate similar parts in all camera images, therefore to have an estimate of disparity between views which also lets us to have an estimate of depth inside the scene. The depth information is important for both 3D perception and rendering of the virtual views.

4.      Image Rendering

After evaluating the camera parameters and the depth estimate of the scene, light rays from the scene to the cameras can be reconstructed in reverse direction towards the scene, leading to a 3D model
hence; virtual views can be rendered as if they are observed from virtual cameras at that perspective.

 

People Involved

Gorkem Saygili

Laurens van der Maaten

Emile Hendriks

Marcel Reinders