A full sphere camera for point of view and VR photography

The camera assembly uses a pair of opposing motor driven Nikon cameras equipped with 220 degree fisheye lenses that simultaneously record the scene in all viewing directions. The camera assembly is quite rugged and compact enough that it can be hand or helmet carried on roller coasters, bikes, used inside aircraft cockpits, mounted outside of helicopters, etc.

Unmodified commercially available software is used to first assemble and convert the raw fisheye images into full sphere images and then extract a variety of views in any direction. First the two fisheye images are converted into an equidistant cylindrical projection and combined into a single image in Adobe Photoshop. This intermediate image is then texture mapped onto a 3D sphere, forming the conceptual equivalent of a "virtual Omnimax theater". The viewer is placed in the center of the texture mapped sphere where he can view in any direction. These views can be dynamically formed through the use of VRML.

Since the original publication of this paper in 1996, Helmut Dersch has developed a freely available software package that provides better and higher quality image processing than is described here. The use of his software "Panorama Tools" would be recommended as a higher quality alternative to the image processing described in this paper. The description here though should help to provide a good understanding of processing fisheye images into panoramic equivalents.
 
 

The ideal requirements for a VR camera system are:

1. Full sphere imaging -The camera should record a full sphere image that includes all possible viewing directions (4 pi steradians).
2. High resolution -The camera should record the scene to a detail level of what the human eye can resolve. For an unaided eye, this is approximately 1 arcminute. Film based wide angle high resolution imaging systems require large image areas on the film. Currently (in 1996), scanning panoramic and real or synthetic camera clusters are used to record wide angle high resolution imagery. If the VR camera is implemented in digital form, the camera would require one or more sensors and storage devices to capture and record the desired VR image. Approximately 20,000 picture elements are required for one equatorial ring of the image. A full sphere image sampled at a resolution of 66 nanosteradians per pixel (1 arcminute per pixel) produces 190,476,000 pixels per frame. If stored in a standard 24 bit per pixel format, the image would require 571 megabytes per frame. Both the image sensor and the storage device can be single or multiple devices.
3. Large storage capacity - A VR camera system should be capable of recording numerous images to provide a discrete approximation of time varying viewer positions. As each image could require over 500 megabytes per frame, a combination of good compression algorithms and large storage devices would be required.
4. POV capabilities - The Point Of View (POV) photography style attempts to place the camera viewpoint at the eyepoint of an actor or other position that is within a scene. The POV camera is within a scene, rather than viewing the scene from a remote location. The POV camera often needs to be rugged and compact as might be required for filming from a motorcycle helmet or skydiving helmet mount. People interested in POV imagery tend also to be interested in VR methods. As such, VR photography requirements are rapidly evolving into a superset of POV requirements. A new generation of VR cameras are starting to be built that are also compatible with the other requirements of POV photography.

Presently there are several vendors that provide VR camera systems. There are three classes of ultrawide angle cameras currently in use; scanning panoramic, synthetic cluster and real cluster. The scanning panoramic designs use a rotating lens to image onto a moving piece of film or a CCD sensor. Camera clusters take a series of images in different directions so as to record all desired viewing directions. By analogy to synthetic aperture radar, the cluster can be real or synthetic. A real cluster means that a separate camera is used for each viewing direction. A synthetic camera cluster means that a single camera is sequentially rotated to different viewing directions. Both film and CCD based cameras can be used. Currently, the film based systems have the advantage of higher resolution and portability although this will likely change in the near future. The film based systems generally either use a single scanning panoramic camera or a series of separate photographs that are stitched together to record a wide angle scene. Several of the more popular and interesting systems are briefly described here.

Apple Quick Time VR - The Apple Quick Time TV (QTVR) system uses either a single scanning panoramic camera shot or a series of still camera shots to record the scene in a circle around a common point. Scanning panoramic cameras commonly used include the Hulcherama,Roundshot and  Globuscope. Alternately, the QTVR system may use a still camera, typically a Nikon equipped with a 15 mm lens to record a series of images at different azimuth angles. A tripod and indexing device are often used to precisely position the camera for this series of images. If the still camera method (synthetic cluster) is used, the separate images are stitched together with a cylindrical mapping correction to form an image equivalent to that produced by a standard scanning panoramic camera. The cylindrical perspective image is then converted into the Apple QTVR file format for viewing with an Apple provided viewer.

Infinite Pictures (iMove) & Kaidan -- This pair of companies manufactures a full sphere imaging system using a full frame fisheye lens and a multiple angle stitching bracket. iMove is apparently also building full sphere video camera systems.

RealVR -- RealVR is a new dynamic viewing system developed by Live Picture. This system an extension of VRML 2.0. The RealVR viewer is freely available as a Netscape / Internet explorer plugin or as a standalone viewer. It is compatible with cylindrical and spherical image mappings (as used by the spherecam). A simple control file is needed to make the system work. Documentation on RealVR available. Using the RealVR viewer is described later in this paper.

Jeffrey Charles (Versacorp) has been building reflective hyperwide optics for many years. Similar optics are available commercially from others including, Tateyama , Be Here and Omnicam. Drexel University is experimenting with other hyperwide mirror systems.  Several other sites (1 , 2) include links to other hyperwide imaging systems.

Ipix - Ipix markets a system that can dynamically extract moderate angle views from any part of fisheye or full sphere image. Initially, their system used a standard Nikon camera equipped with an 8 mm f2.8 fisheye lens. After Nikon discontinued this lens, Ipix has been using lenses manufactured by Coastal Optical and using other Nikon fisheye lenses on the Coolpix 950 camera. In any case, the camera is mounted on a special tripod that insures that the camera is rotated precisely 180 degrees to maintain alignment of the aberrated lens entrance pupil between two adjacent exposures. If the rotation is not accurate, part of the scene will not be recorded. The fisheye image pair is combined with a proprietary software package into a form that is compatible with their viewer. Recently, Coastal Optical has started manufacturing a video equivalent of the Spherecam for IPIX using a pair of fisheye lenses. Within the last few years, Ipix has become quite litiguous and has fallen into disfavor with many photographers (sites explaining the controversy: 1, 2, 3, 4, more information can be found by searching the internet for "ipixfree").

PanoramIX- Another VR system.

Hemispherical imaging home page -- Scientific studies of plant canopies through the use of fisheye hemispherical photography. Also see Delta T Devices.
 

The camera assembly consists of a pair of opposite viewing motor driven Nikon F camera bodies. The camera bodies are rigidly attached to a machined aluminum skeletal frame. The cameras were stripped of viewfinders and battery packs so as to minimize the size and weight. Both cameras were equipped with Nikon 6 mm f5.6 220 degree fisheye lenses. This particular lens was chosen since it is very compact and provides a large 40 degree overlap region that greatly simplifies the later image splicing operation (Slater 1996). Unfortunately, this lens is no longer in production but can sometimes be found in the used camera marketplace. As the lens entrance pupils can not be coincident as in a synthetic cluster design, the large overlap region is particularly important when objects are close to the camera. A number of accessories were developed for the full sphere camera including a pistol grip, camera extension poles, a remote radio trigger system and a fill lighting system suitable for aircraft cockpit photography.
 

VR systems operate by extracting part of the image corresponding to the observer's current view port. The basic mathematical concepts behind the conversion of a cylindrical panoramic or spherical image into a view port are well described in books on map making ( Snyder 1989 , Snyder 1987), 3D computer graphics and spacecraft attitude determination (Wertz 1978).

The image processing for this system consists of 3 elements; stitching the two opposing fisheye images into a single 4 pi steradian image, texture mapping the 4 pi steradian image onto the surface of a sphere and then extracting the desired views from the center of the sphere. The method described here uses standard unmodified commercially available software. Better alternatives exist since this paper was originally written. An excellent software program that is freely available was written by Helmut Dersch and is now the software of choice for this application.
 

Step 1: Merge the pair of fisheye photographs into a single full sphere intermediate image.

1. Scan the two images into the computer. Use the same scanner settings for both images. The images should be centered in the frame.
2. Convert both images into a cylindrical format so that they can be merged. This can be done in Adobe Photoshop by using the polar to rectangular conversion filter. In cartographic terms, the fisheye image is an azimuthal equidistant projection. The Adobe Photoshop polar to rectangular conversion filter converts the fisheye image into an equidistant cylindrical (also known as an equirectangular) projection.
3. Increase the canvas size of one image vertically by the ratio of 360 to 220 (the ratio of the full sphere angle to the fisheye lens field of view), leaving the image at the top of the frame.
4. Rotate the second image 180 degrees and paste as a layer with a layer mask onto the first image.
5. Align the overlap region and blend the layers into a single composite image.

The two preceding images are a pair of 220 degree fisheye images (step 1). The next two images are the result of step 2:



These two images will be combined by rotating the image on the right by 180 degrees and overlapping.

The previous image combining process can be nicely automated in Photoshop 4.0 by using the Actions feature. The resulting image (called the intermediate image) corresponds to a full sphere image in an equidistant cylindrical projection. It is highly distorted but contains the necessary information from which other views can be extracted. The horizontal axis has 360 degrees of coverage, the vertical axis has 180 degrees of coverage. This projection is similar, but not identical to the central cylindrical projection used by a scanning panoramic camera. The difference is an arctangent function along the Y axis.

The intermediate image can be processed in a number of ways (Slater 1996). As one example, a rectangular to polar conversion (the inverse of the previously used polar to rectangular conversion) can be done to produce a fisheye image corresponding to a 360 degree field of view. As a second example, the intermediate image can be converted into the scanning panoramic format although, because of the arctangent function, the vertical field of view must be less than 90 degrees.
 
 

Step 2: Texture mapping the full sphere image onto a 3D sphere.

Step 3: View the texture mapped sphere

The above view is of the texture map applied to the outside of the sphere. This view is different than that of a 360 degree fisheye as shown in the next example. A slight processing misalignment produced the colored region around the spherical image.

All of the following images are obtained from virtual cameras located at the center of the texture mapped sphere. The next image was produced by a synthetic forward looking camera with a moderate wide angle lens (24 mm 35 mm still format)

The synthetic camera was rotated 80 degrees to the left for this next view. The field of view was similar to that of a very wide 12 mm lens on a 35 mm still format camera.

The following image is a forward looking rectilinear ultrawide angle view taken with the equivalent of a 3 mm lens on a 35 mm still format camera. There is no real equivalent to this lens.

Views can be extracted in any direction and orientation. The amount of available zoom is dependent upon the resolution of the texture map. As an extreme example, the following image shows a cylindrical perspective view with horizontal and vertical fields of view in excess of 360 degrees. Unlike a conventional scanning panoramic camera, this synthetic camera uses a f theta mapping on both axes to achieve the ultrawide vertical field of view. If a narrow horizontal slice were selected in any region other than the zenith or nadir, the resulting view would be similar to that of a conventional scanning panoramic camera.

The VRML description of the virtual Omnimax is straight forward:

#VRML V1.0 ascii

Separator {

Texture2 {filename "veg3.jpg"}

Sphere {radius 1.0}

}

RealSpace Inc. has developed an extension to VRML 2.0 that provides excellent dynamic viewing of the spherecam images. The process is quite simple: 1. Create a jpg file of the spherecam intermediate image, for example here "vegas.jpg". 2. Create the following file in notepad or a similar text editor. The file name should have an ivr extention.

#VRML V2.0 utf8

Vista {

type "SPHERE"

filename "vegas.jpg"

}

To dynamically view the jpeg image, just open this VRML file with Netscape, IE or directly with the RealVR viewer.
 

Describes several simple fisheye optical systems and proposes a lens design concept that provides a 4 pi steradian field of view.

Slater, D., Panoramic Photography with Fisheye Lenses, Panorama, Vol. 13, No. 2, 3, International Association of Panoramic Photographers, (1996)

Describes a camera system and digital processing system that can be used to create ultrawide angle point of view photographs.

Snyder, An Album of Map Projections, U. S. Geological Survey Paper 1453, U. S. Government Printing Office, Washington, DC (1989)

An excellent introduction to the theory of map projections. Highly recommended.

Snyder, Map Projections - A working Manual, U. S. Geological Survey Paper 1395, U. S. Government Printing Office, Washington, DC (1987)

A more detailed discussion of map projections.

Wertz, J., ed., Spacecraft Attitude Determination and Control, D. Reidel Publishing Co., Boston, MA (1978)

Includes a discussion of spherical trigonometry used in spherical image remapping.
 
 

Home