Chapter 5: University research labs evolve

5.1 Cornell and NYIT

Cornell University

Mosaic of Director Don Greenberg made up of images of former students.
A hi-res image can be seen at
 http://www.realtimerendering.com/erich/DPG/2017/

Cornell University, in Ithaca, New York, hosts one of the country’s premiere laboratories contributing to the field of computer graphics imagery, the Program of Computer Graphics (PCG), founded in 1974 by the lab’s director Donald P. Greenberg. The work of Greenberg and his staff and students forms the foundation for many of the practical applications that computer graphics experts and practitioners now routinely use.

Greenberg studied architecture and engineering at Cornell and also at Columbia University. He worked for Severud Associates, an architecture and engineering firm, and was involved with the engineering design of the St. Louis Arch, Madison Square Garden, and other projects.

Images from “Cornell in Perspective

   

Like many of his contemporaries, the 1960s graphics pioneers in universities across the country, Greenberg saw how the complex engineering applications used by scientists could have an impact on the field of design and art. He received funding from the National Science Foundation to support the theoretical study and practical development of algorithms for the realistic display of images, particularly images of architectural environments.

Prior to 1973, much of the research in computer graphics at Cornell was conducted at a General Electric research facility in Syracuse. Greenberg and his students would travel to Syracuse to arrive as the normal workday ended, and leave when the machines were needed by GE again the following morning.

The Program of Computer Graphics received its first National Science Foundation grant in the fall of 1973, which enabled Greenberg to order the lab’s first computer graphics equipment. That equipment arrived early in January of 1974, and the first facility was set up in Rand Hall on the Cornell Campus. Greenberg published a paper at SIGGRAPH 77 describing the Cornell Lab.

The Cornell Box

The Cornell PCG is best known for pioneering work on realistic image synthesis, including the radiosity method for calculating direct and indirect illumination in synthetic scenes. The long-term goal of the lab was to develop physically-based lighting models and  perceptually based rendering procedures to produce images that were visually and measurably indistinguishable from real-world images. The work of the faculty and students has spawned much of the accurate lighting capabilities now present in commercial software. Roy Hall was instrumental in the development of the Abel Image Research raster system, and contributed to the Wavefront renderer (he also wrote the important book Illumination and Color in Computer Generated Imagery, published by Springer-Verlag in 1989.) The popular renderer Lightscape was also a result of Cornell research.

The Vermeer Studio

Since its founding the lab has researched and refined a framework for global illumination incorporating light reflection models, energy transport simulation, and visual display algorithms. The goal was to solve these computationally demanding simulations in real time using an experimental cluster of tightly coupled processors and specialized display hardware. They are achieving this goal by taking advantage of increased on-chip processing power, distributed processing using shared memory resources, and instructional-level parallelism of algorithms. The graphics research also involved three-dimensional modeling of very complex environments and new approaches for modeling architectural designs.

In 1991, the Program of Computer Graphics moved to the new Engineering and Theory Center Building across campus, now renamed Rhodes Hall as a tribute to Dr. Frank H.T. Rhodes, the former president of Cornell. Also in 1991, Cornell became one of five universities participating in the new National Science Foundation Science and Technology Center for Computer Graphics and Scientific Visualization (see Section 5 in this Chapter.) The list of Cornell alumni reads like a who’s who in graphics, and the research publications coming from the lab are some of the most significant contributions of any lab in the world.

Greenberg received the ACM-SIGGRAPH Coons award in 1987 for his lifetime contribution to computer graphics and interactive effects.

Movie 5.1 Cornell in Perspective


In 1971, Greenberg produced an early sophisticated computer graphics movie, Cornell in Perspective, using the General Electric Visual Simulation Laboratory.

 

Some of this material was taken from an article in Architecture Week about the graphics program at Cornell, and from the history of the Program of Computer Graphics on Cornell’s web site.

Cornell was one of 12 research labs showcased in an exhibition at SIGGRAPH 98 for the 25th anniversary of the conference. Their portion of the exhibition can be found at
http://www.graphics.cornell.edu/siggraph/1998images.html

Donald P. Greenberg, An Interdisciplinary Laboratory for Graphics Research and Applications, Proceedings of the Fourth Annual Conference on Computer Graphics, Interactive Techniques and Image Processing – SIGGRAPH, 1977

Cornell student Marc Levoy did his Bachelors and Masters work around 2D cartoon animation. His work led to a production system at Hanna Barbera Productions. As Levoy stated in an interview related to the Sands Award given for the best undergraduate research work in 1976:

“After the completion of my Master’s thesis, Don Greenberg and I tried to convince the Walt Disney’s feature animation group to incorporate computer graphics into their production process. Unfortunately, several of the “nine old men”, who had worked for Disney since the 1930’s, were still active, and they would have none if it. By the mid-1980’s, these gentlemen had all retired, and Pixar was able to convince Disney to explore the new technology. The system they co-developed was called CAPS (Computer Animation Production System). It ultimately won them an academy award in 1992 for its use in Beauty and the Beast.

Although we were unsuccessful at Disney, we managed to convince Hanna-Barbara[sic] Productions to employ our system in the television animation market. Hanna-Barbara was on the verge of closing down domestic production due to spiraling labor costs, and they saw our system as a way to forestall doing this. The animator’s union didn’t believe them, nor did they relish the prospect of computerization, leading to a bitter strike in 1982. In the end, the animators learned to use our system, which remained in production until 1996. At its peak in the late 80’s, one third of Hanna-Barbera’s yearly domestic production went through our system, including The Flintstones, Scooby Doo, and other shows. Ultimately, domestic labor costs become too high even for the computerized production line, and the the system was sold to James Wang Films of Taiwan. It has since been ported to PCs and is currently marketed under the name Animaster. The original technical team at Hanna-Barbera Productions consisted of myself, Bruce Wallace, Chris Odgers, Bennett Leeds, Jim Mahoney, Steve McDaniel, and Tim Victor. Our chief power user from the production side of the company was Ann Tucker. Interestingly, Ann subsequently went to work for Disney, becoming their chief power user as well.”

Levoy worked as director of the Hanna-Barbera Animation Laboratory from 1980 to 1983.

Gallery 5.1 Cornell PCG Images

 

New York Institute of Technology (NYIT)

Alexander Schure, a wealthy entrepreneur from New York, was interested in making a feature length movie, and after attempting the process with traditional animation, became enamored with and wanted to use computers to do it.

In 1953 Schure got into the business of providing veterans with correspondence courses, and in 1960 his school was accredited as New York Institute of Technology, or NYIT. Expanding his campus from NewYork City, he bought an old industrialist estate in Westbury on Long Island in 1963. His first movie attempt was in the early 1970s, a traditional animation project called Tubby the Tuba, which ultimately failed miserably.

Schure’s board of directors included members of the wealthy Rockefeller family, who also served on the board of the computer graphics company Evans and Sutherland in Utah. In 1974, Dr. Schure made a commitment to the computer film project by establishing the Computer Graphics Laboratory (CGL) at NYIT. The Rockefellers suggested that he visit E&S in Utah to get a sense of the kinds of technology that was available. He put together one of the most sophisticated studios of the time on the NYIT campus on Long Island, employing CGI equipment and state of the art computers that he obtained from E&S, but also top end analogue devices for doing special effects. He also was interested in top talent, and looked to the University of Utah to find this talent after having been exposed to the animated face and hand produced by Ed Catmull and Fred Parke. He hired Catmull (who had left Utah for the CAD company Applicon) and fellow Utah grad Malcolm Blanchard to run his facility, and they were soon joined by Alvy Ray Smith and David DiFrancesco, and later Lance Williams, Fred Parke, Garland Stern, and others, including many from Utah. He also attracted other technology experts and artists, including Jim Clark, Ralph Guggenheim, Ned Greene, Rebecca Allen and Ed Emshwiller, who was a Hugo award winning artist who had recently won a Guggenheim grant to make a film. Emshwiller worked with Stern, Williams and Smith and other NYIT staff to make the now famous film Sunstone in 1979.

Frame from Sunstone

The personnel at NYIT CGL were very prolific in the design of influential software during the period from 1975 to 1979, including the animation program Tween, the paint program Paint, the animation program SoftCel, and others. They also contributed to image techniques involving animation, fractals, morphing, image compositing, texture mapping (the famous Mip-Map approach), digital conversion of video signals, scanning of animated frames, and many other innovations.

The NYIT staff, most notably Gene Miller, Lance Williams and  Michael Chou, were responsible for an innovative rendering technique called “reflection mapping“, which was later used to provide realism to shiny objects in television and the movies (eg, Flight of the Navigator, Terminator, and The Abyss).

Michael Chou and his robot

One of the first such images (shown here) was done by Chou, showing him standing next to a shiny synthetic robot. NYIT published the technique at SIGGRAPH in 1982 and 1984, and in 1985 used the technique in a short video titled Interface.

Another innovation out of NYIT, arguably one of the most influential, was the alpha channel. This contribution made it possible to combine two images or parts of images, and to recognize the impact of transparency in the combination. As Alvy Ray Smith said in his 1995 technical memo, Alpha and the History of Digital Compositing

“Ed Catmull and I invented the notion of the integral alpha in the 1970s at New York Tech. This is the notion that opacity (or, equivalently, transparency) of an image is as fundamental as its color and should therefore be included as part of the image, not as a secondary accompaniment. To be very clear, we did not invent digital mattes or digital compositing. These were obvious digital adaptations of known analog techniques. We invented the notion of the alpha channel as a fundamental component of an image. We coined the term ‘alpha’ for the new channel. We called the resulting full-color pixel an ‘RGBA’ pixel.”

Alpha compositing was further developed in a 1984 paper by Tom Porter and Tom Duff at Lucasfilms.

Smith developed the 8-bit program Paint, which was sold to Ampex and became the main components of the Ampex AVA system. This system was used by artist LeRoy Niemann to do interactive painting during Superbowl XII in 1978. Paint and Smith’s later 24-bit Paint3 had a great deal of influence on subsequent systems, including the paint system at Lucasfilm and the CAPS system developed for Disney by Lucasfilm.

After Tubby the Tuba failed in the way that it did, many of the traditional animation staff left NYIT, and the emphasis on 3D moved forward. CGL worked on highly visible ads and television promotions for clients such as NOVA and ABC Sports.

Several videos from NYIT have become quite famous: Sunstone, Inside a Quark, by Ned Greene, and The Works, as well as some documentary work by Rebecca Allen. A great deal of effort at NYIT went into the development of the film “The Works“, which was written by Lance Williams[1]. For many reasons, including a lack of film-making expertise, it was never completed. Sequences from the work in progress still stand as some of the most astounding animated imagery of the time.

An article on The Works was featured in the trade magazine Computer Pictures in 1983

Many of the researchers at CGL were becoming disillusioned with Schure and the way he ran the facility, so they quietly looked for different opportunities elsewhere. The quality of the imagery, along with the other research and technology work at NYIT attracted the attention of George Lucas, who was interested in developing a CGI special effects facility at his company Lucasfilm. He sent one of his top people, Richard Edlund, to scope out what was going on at the Long Island lab. He soon after recruited much of the top talent from NYIT, including Catmull, and Smith and Guggenheim (who both took on a short stint at JPL) to start his division, which eventually was spun off as Pixar.

Scene from Kraftwerk music video – Rebecca Allen

Work continued after the departure of some of the originators of the CGL facility. Rebecca Allen, who came to NYIT from MIT, won an Emmy for work done for Walter Cronkite’s Universe, and she later did several music videos, including work for Kraftwerk and Peter Gabriel. The French company Thompson Digital Image, or TDI, bought the rights to use Garland Stern’s system BBOP, as did Omnibus in Canada.

Schure never got over the move of his top talent from NYIT, and he attempted to obtain patents on many of the innovative algorithms that were developed there by them. He was unsuccessful, in large part because he was so late in filing, because many had been published at SIGGRAPH and elsewhere, and because Microsoft and Pixar fought those attempts. Schure eventually succumbed to Altzheimer’s disease in 2009.

Ed Catmull and Lance Williams received the ACM-SIGGRAPH Coons award in 1993 and 2001 respectively, for lifetime contribution to computer graphics and interactive effects. Williams also received an Academy Award for Technical Achievement in 2002 for his influence in the field of computer generated animation and effects for motion pictures. Garland Stern received an Academy Award for Technical Achievement in 2001 for the Cel Paint Software System. Alvy Ray Smith received the ACM-SIGGRAPH Computer Graphics Achievement award in 1990 for an outstanding contribution to computer graphics and interactive effects. He also received an Academy Award for Technical Achievement in 1997 for his paint system.

A New York City office of NYIT’s CGL was established to market and sell the technology developed in Westbury. Also called CGL, it focused on the emerging television advertising and promotions industry. CGL sold hardware and licenses to the Images and Images II production systems to many other companies, including Omnibus in Toronto and JCGL in Japan.

Movie 5.2 Sunstone

Sunstone (1979) Ed Emshwiller (with Alvy Ray Smith, Lance Williams, Garland Stern) http://www.youtube.com/watch?v=tMW15OajuKc

 

Movie 5.3 Excerpt from The Works

 

Movie 5.4 The Works (1984)
http://www.youtube.com/watch?v=18OSLeWJVJQ

Movie 5.5 Kraftwerk Musique Non Stop music video
graphics by Rebecca Allen
http://www.dailymotion.com/video/x175jh_kraftwerk-music-non-stop_music

Movie 5.6 JCGL Demo Reel 1983
https://www.youtube.com/watch?v=_WlAnrdtGmg

 

Porter, Thomas, and Duff, Tom, Compositing Digital Images, Computer Graphics, Vol 18, No 3, Jul 1984, 253-259. SIGGRAPH’84 Conference Proceedings.

Smith, Alvy Ray, Image Compositing Fundamentals, Tech Memo 4, Microsoft, Jun 1995.

Smith, Alvy Ray, Alpha and the History of Digital Compositing, Tech Memo 7, Microsoft, Aug 1995.

An article on The Works was featured in the trade magazine Computer Pictures in 1983.

A great compilation of NYIT information can be found at Paul Heckbert’s site at http://www.cs.cmu.edu/~ph/nyit/.

Alvy Ray Smith wrote an article describing the Paint program as part of a survey of activities related to digital ink and paint called Digital Paint Systems, a Microsoft Technical Memo.

Alvy Ray Smith wrote an interesting account of he and Ed Emshwiller and the creation of Sunstone, now at the computer history site, and can be read at http://www.computerhistory.org/atchm/alvy-ed/

Heckbert wrote a personal account of the participation of the NYIT staff in the making of the IMAX movie “The Magic Egg“ for the 1984 SIGGRAPH conference in Minneapolis.

A 2007 NYIT Alumni article titled “Out of this World: Meet the NYIT scientists and alumni who amazed Silicon Valley, astonished Hollywood and engineered the 3-D computer animation universe” can be accessed here.

 

In 1989, Quantel, Ltd. filed suit in British courts accusing Spaceward, Ltd. of violating their claim to have invented digital “airbrushing” and digital compositing. Alvy Ray Smith testified as to his development of the concepts earlier than the claim of Quantel, but to no avail. Spaceward lost the case, and their product. Smith commented, in his paper Digital Paint Systems: An Anecdotal and Historical Overview, in 2001, that he had done this work while at NYIT in 1977: “To my mind, airbrushing and digital compositing are the same thing: One combines one image with another, using a third for transparency control.”

In 1996 a patent infringement case was again filed by Quantel Ltd. against Adobe Systems, Inc., accusing Adobe Photoshop’s “paint” feature of infringement. A jury ultimately found Quantel’s patents invalid because they covered basic software techniques developed in 1977 by Alvy Ray Smith, a computer graphics pioneer and co-founder of Pixar Animation Studios. Smith had sold his paint system to Ampex in the mid 1970s and briefly described his digital paint systems in “tutorial notes” distributed at conferences in 1978. The source code that implemented Smith’s paint systems was not itself readily accessible, yet it is fair to say modern computer graphics evolved, in large part, from Smith’s groundbreaking work.
From Taking Care of Business – Patent reform should promote innovation, not imitation, by Kelly C. Hunsaker. San Francisco Chronicle, August 30, 2005

This case was documented in an article in Computer Graphics in August, 1998, Computer graphics in court: the Adobe/Quantel case, by Dick Phillips.

The results of the case were outlined in a 1997 article, Adobe Defeats Infringement Claim, found at http://www.litstrat.com/Cases/ADOBEtest.pdf

 

 

 

Gallery 5.2 NYIT Image Collection

  1. Lance Williams passed away on August, 20, 2017. Hank Grebe, who worked with Williams at NYIT, eloquently paid tribute to him with a quote of the last paragraph of "The Works" script: "Beeper and Ipso gazed at the ship landing on the screen , and the boy and the robot and the vastness of the Works itself dwindled to tiny points on the earth, and the earth, to a tiny point among the stars, and the stars lost their identity in clouds of galaxies, and when the inconceivably vast clusters of the galaxies are indistinguishable in the milky swirl of the infinity of space, some people will still be wondering if the Universe is large enough to include a story as unlikely as this one." ↵

License

Icon for the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Computer Graphics and Computer Animation: A Retrospective Overview Copyright © 2017 by Wayne E. Carlson is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, except where otherwise noted.