17.1 Virtual Reality

During the late 1980s and 1990s, virtual reality was touted as a new and emerging application that promised to revolutionize interactivity and man-computer interfaces. In fact, VR is much older than the 1980s, older or nearly as old as the entire computer graphics field itself.

In 1956, Morton Heilig began designing the first multi-sensory virtual experiences. Resembling one of today’s arcade machines, the Sensorama combined projected film, audio, vibration, wind, and odors, all designed to make the user feel as if he were actually in the film rather than simply watching it. Patented in 1961, the Sensorama placed the viewer in a one person theater and, for a quarter, the viewer could experience one of five two-minute 3D full color films with ancillary sensations of motion, sound, wind in the face and smells. The five “experiences” included a motorcycle ride through New York, a bicycle ride, a ride on a dune buggy, a helicopter ride over Century city in 1960 and a dance by a belly dancer. Since real-time computer graphics were many years away, the entire experience was prerecorded, and played back for the user.

Heilig also patented an idea for a device that some consider the first Head-Mounted Display (HMD). He first proposed the idea in 1960 and applied for a patent in 1962. It used wide field of view optics to view 3D photographic slides, and had stereo sound and an “odor generator”. He later proposed an idea for an immersive theater that would permit the projection of three dimensional images without requiring the viewer to wear special glasses or other devices. The audience would be seated in tiers and the seats would be connected with the film track to provide not only stereographic sound, but also the sensation of motion. Smells would be created by injecting various odors into the air conditioning system. Unfortunately, Heilig’s “full experience” theater was never built.

Comeau and Bryan, employees of Philco Corporation, constructed the first actual fabricated head-mounted display in 1961. Their system, called Headsight featured a single CRT element attached to the helmet and a magnetic tracking system to determine the direction of the head. The HMD was designed to be used with a remote controlled closed circuit video system for remotely viewing dangerous situations. While these devices contributed intellectual ideas for display and virtual experiences, the computer and image generation were yet to be integrated.

The field we now know as virtual reality (VR), a highly multidisciplinary field of computing, emerged from research on three-dimensional interactive graphics and vehicle simulation in the 1960s and 1970s. Not surprisingly, the development of the discipline can be traced to early work at MIT and Utah and none other than Ivan Sutherland.

Two of the necessary foundations of VR were being addressed at MIT by Larry Roberts and Sutherland, among others. The first necessary practical contributions included the research and development that allowed the CRT to serve as an affordable and effective device on which to create a computer generated image, and the interactive interfaces that showed that a user could interact with the CRT image to accomplish some desired task.

As we mentioned in Chapter 4, Roberts wrote the first algorithm to eliminate hidden or obscured surfaces from a perspective picture in 1963. His solutions to this and other related problems prompted attempts over the next decade to find faster algorithms for generating hidden surfaces. Among the important activities of Sutherland and his colleagues and students at the University of Utah were efforts to develop fast algorithms for removing hidden surfaces from 3D graphics images, a problem identified as a key computational bottleneck.

Students of the Utah program made two important contributions in this field, including an area search method by Warnock (1969) and a scan-line algorithm that was developed by Watkins (1970) and developed into a hardware system. One of the most important breakthroughs was Henri Gouraud’s development of a simple scheme for continuous shading (1971). Unlike polygonal shading, in which an entire polygon (a standard surface representation) was a single level of gray, Gouraud’s scheme involved interpolation between points on a surface to describe continuous shading across a single polygon, thus achieving a closer approximation of reality. The effect made a surface composed of discrete polygons appear to be continuous. This ability is essential in the process of generating the quality of visual images necessary to present a believable VR environment.

Each of these efforts provided part of the foundation for early attempts at addressing the concept of a virtual environment. The other part was the earlier work that resulted in the commercial development of the important head mounted display. The Ultimate Display was created by Sutherland in 1965. What made it so important was the fact that it had a stereoscopic display (one CRT element for each eye). The HMD had a mechanical tracking system, and later Sutherland experimented with an ultrasonic tracker. As was discussed in the National Academy of Sciences report Funding a Revolution: Government Support for Computing Research, the HMD was the central research component of the emerging field. The following text, from this report, outlines the important work in this area:

Work on head-mounted displays (HMDs) illustrates the synergy between the applications-focused environments of industry and government-funded (both military and civilian) projects and the fundamental research focus of university work that spills across disciplinary boundaries. Work on head-mounted displays benefited from extensive interaction and cross-fertilization of ideas among federally funded, mission-oriented military projects and contracts as well as private-sector initiatives. The players included NASA Ames, Armstrong Aerospace Medical Research Laboratory of the Air Force, Wright-Patterson Air Force Base, and, more recently, DOD programs on modeling and simulation, such as the Synthetic Theater of War program. Each of these projects generated a stream of published papers, technical reports, software (some of which became commercially available), computer-animated films, and even hardware that was accessible to other graphics researchers. Other important ideas for the head-mounted display came from Knowlton and Schroeder’s work at Bell Laboratories, the approach to real-time hidden-line solutions by the MAGI group, and the GE simulator project (Sutherland, 1968).

Early work on head-mounted displays took place at Bell Helicopter Company. Designed to be worn by pilots, the Bell display received input from a servo-controlled infrared camera, which was mounted on the bottom of a helicopter. The camera moved as the pilot’s head moved, and the pilot’s field of view was the same as the camera’s. This system was intended to give military helicopter pilots the capability to land at night in rough terrain. The helicopter experiments demonstrated that a human could become totally immersed in a remote environment through the eyes of a camera.

The power of this immersive technology was demonstrated in an example cited by Sutherland (1968). A camera was mounted on the roof of a building, with its field of view focused on two persons playing catch. The head-mounted display was worn by a viewer inside the building, who followed the motion of the ball, moving the camera by using head movements. Suddenly, the ball was thrown at the camera (on the roof), and the viewer (inside the building) ducked. When the camera panned the horizon, the viewer reported seeing a panoramic skyline. When the camera looked down to reveal that it was “standing” on a plank extended off the roof of the building, the viewer panicked!

In 1966, Ivan Sutherland moved from ARPA to Harvard University as an associate professor in applied mathematics. At ARPA, Sutherland had helped implement J.C.R. Licklider’s vision of human-computer interaction, and he returned to academe to pursue his own efforts to extend human capabilities. Sutherland and a student, Robert Sproull, turned the “remote reality” vision systems of the Bell Helicopter project into VR by replacing the camera with computer-generated images.(Other head-mounted display projects using a television camera system were undertaken by Philco in the early 1960s, as discussed by Ellis in 1996.) The first such computer environment was no more than a wire-frame room with the cardinal directions–north, south, east, and west–initialed on the walls. The viewer could “enter” the room by way of the “west” door and turn to look out windows in the other three directions. What was then called the head-mounted display later became known as VR.

Sutherland’s experiments built on the network of personal and professional contacts he had developed at MIT and ARPA. Funding for Sutherland’s project came from a variety of military, academic, and industry sources. The Central Intelligence Agency provided $80,000, and additional funding was provided by ARPA, the Office of Naval Research, and Bell Laboratories. Equipment was provided by Bell Helicopter. A PDP-1 computer was provided by the Air Force and an ultrasonic head-position acoustic sensor was provided by MIT Lincoln Laboratory, also under an ARPA contract.

Sutherland outlined a number of forms of interactive graphics that later became popular, including augmented reality, in which synthetic, computer-generated images are superimposed on a realistic image of a scene. He used this form of VR in attempting a practical medical application of the head-mounted display. The first published research project deploying the 3D display addressed problems of representing hemodynamic flow in models of prosthetic heart valves. The idea was to generate the results of calculations involving physical laws of fluid mechanics and a variety of numerical analysis techniques to generate a synthetic object that one could walk toward and move into or around (Greenfield, Harvey, Donald Vickers, Ivan Sutherland, Willem Kolff, et al. 1971. “Moving Computer Graphic Images Seen from Inside the Vascular System,” Transactions of the American Society of Artificial Internal Organs, 17:381-385. )

As Sutherland later recalled, there was clearly no chance of immediately realizing his initial vision for the head-mounted display. Still, he viewed the project as an important “attention focuser” that “defined a set of problems that motivated people for a number of years.” Even though VR was impossible at the time, it provided “a reason to go forward and push the technology as hard as you could. Spin-offs from that kind of pursuit are its greatest value.” 
(Ivan Sutherland in Virtual Reality Before It Had That Name, a videotaped lecture before the Bay Area Computer History Association.)

http://www.nap.edu/readingroom/books/far/contents.html

VR is one of those fields that Ivan Sutherland would christen “holy grails” – fields involving the synthesis of many separate, expensive, and risky lines of innovation in a future too far distant and with returns too unpredictable to justify the long-term investment.