James Cameron is a moviemaker with a passion for cutting-edge technology and risky projects. Before the opening of his new sci-fi movie, Avatar, pundits spoke of make-or-break scenarios for his unique brand of digital magic. ‘Break’ is no longer part of their vocabulary… By Anne Thompson
The 26 000 square-metre studio in Playa Vista, California, has a curious history as a launching pad for big, risky ideas. In the 1940s, Howard Hughes used the huge wooden aircraft hangar to construct the massive plywood H-4 Hercules seaplane – famously known as the . Two years ago, movie director James Cameron was in the Playa Vista studio at a crucial stage in his own big, risky project. He was viewing early footage from , the sci-fi epic he had been dreaming about since his early 20s. Cameron’s studio partner, Twentieth Century Fox, had already committed to a budget of R1,5 billion (the final cost is reportedly closer to R2,25 billion) on what promised to be the most technologically advanced work of cinema ever undertaken. But as Cameron looked into his computer monitor, he knew something had gone terribly wrong.
The film – although “film” seems to be an anachronistic term for such a digitally intense production – takes place on a moon called Pandora, which circles a distant planet. Jake Sully, a former Marine paralysed from the waist down during battle on Earth, has travelled to this lush, green world teeming with exotic, bioluminescent life to take part in the military’s avatar program. The human settlers are interested in mining Pandora’s resources, but can’t breathe its toxic atmosphere, so to help explore the moon and meet with the native Na’vi who live there, Sully has his consciousness linked with a genetically engineered 3 metre-tall human-alien hybrid.
Cameron wrote his first treatment for the movie in 1995 with the intention of pushing the boundaries of what was possible with cinematic digital effects. In his view, making would require blending live-action sequences and digitally captured performances in a three-dimensional, computer-generated world. Part action-adventure, part interstellar love story, the project was so ambitious that it took 10 more years before Cameron felt cinema technology had advanced to the point where was even possible.
The scene on Cameron’s screen at Playa Vista – an important turning point in the movie’s plot – showed Na’vi princess Neytiri, played by Zoë Saldana, as she first encounters Sully’s avatar in the jungles of Pandora. Everything in the forest is luminous. Glowing sprites float through Pandora’s atmosphere, landing on Sully as Neytiri determines if he can be trusted. Playing Sully is Sam Worthington, an Australian actor whom Cameron had plucked from obscurity to play the movie’s hero. Cameron was staring directly into Worthington’s face – or, rather, he was looking into the face of a digitally rendered Worthington as a creature with blue skin and large yellow eyes – but he might as well have been staring into a Kabuki mask.
The onscreen rendering of Worthington was supposed to be a sort of digital sleight of hand – a human character inhabiting an alien body so that he could blend into an alien world, played by a human actor inhabiting a digital body in a digital world. To make the whole thing work, Worthington’s performance, those subtle expressions that sell a character to the audience, had to come through the face of his avatar. But after millions of dollars of research and development, the avatar’s face was not only lifeless, it was downright creepy. It “scared the crap out of me”, Cameron recalls. “Horrible! It was dead, it was awful, it wasn’t Sam. God, I thought. We’ve done everything right and this is what it looks like?”
The reaction Cameron was feeling has a name. It’s called the uncanny valley, and it’s a problem for roboticists and animators alike. Audiences are especially sensitive to renderings of the human face, and the closer a digital creation gets to a photorealistic human, the higher expectations get. If you map human movements and expression to cute furry creatures that dance and sing like people, then audiences willingly suspend disbelief and go along with it. (Think of the penguins in Happy Feet.) But if you try to give a digital character a humanoid face, anything short of perfection can be uncanny – thus the term. Sometimes audience unease is to a character’s advantage; in The Lord of the Rings the creature Gollum was supposed to be unsettling. But Cameron was looking for empathy, and in the first footage, that’s not what he got.
Why is the computer-generated face of a blue, cat-eyed human-alien hybrid so important? Well, for one thing, lots of money is riding on it. But so, to an extent, is James Cameron’s stature as an unstoppable force in Hollywood. Cameron has built up enormous fame and power based on his reputation as a technical innovator – pushing the science and technology of modelmaking, digital animation and camera engineering. But Cameron is perhaps even more famous as the industry’s biggest risk-taker, which might have made him a lot of enemies if his risks hadn’t been so spectacularly rewarded in the past. In 1997, the film Titanic taught Hollywood a powerful lesson in Cameronomics: The director’s unquenchable thirst for authenticity and technological perfection required deep-sea exploratory filming, expensive scale models and pioneering computer graphics that ballooned the film’s budget to R1,5 million. This upped the ante for everyone involved and frightened the heck out of the studio bean counters, but the bet paid off – Titanic went on to make over R13 billion and win 11 Academy Awards.
A unique hybrid of scientist, explorer, inventor and artist, Cameron has made testing the limits of what is possible part of his standard operating procedure. He dreams almost impossibly big, and then invents ways to bring those dreams into reality. The technology of moviemaking is a personal mission to him, inextricably linked with the art. Each new film is an opportunity to advance the science of cinema. Judging from audiences’ reaction so far, Avatar is likely to change the way movies are captured, edited and even acted.
Filmmakers, especially those with a technical bent, admire Cameron for “his willingness to incorporate new technologies in his films without waiting for them to be perfected,” says Bruce Davis, the executive director of the Academy of Motion Picture Arts and Sciences. It adds to the risky nature of Cameron’s projects, but his storytelling has reaped enormous benefits. There’s a term in Hollywood for Cameron’s style of directing, Davis says: “They call this ‘building the parachute on the way down’.”
But repeatedly pulling off these feats of derring-do requires both the drive of an ambitious egomaniac and an engineer’s plodding patience. “You have to eat pressure for breakfast if you are going to do this job,” Cameron says. “On the one hand, pressure is a good thing. It makes you think about what you’re doing, your audience. You’re not making a personal statement, like a novel. But you can’t make a movie for everybody – that’s the kiss of death. You have to make it for yourself.”
Cameron’s dual-sided personality has roots in his upbringing – the brainy sci-fi geek from Chippewa, Ontario, near Niagara Falls in Canada, was raised by a painter mother and an engineer father. “It was always a parallel push between art and technology,” he says. “My approach to filmmaking was always very technical. I started off imagining not that I would be a director, but a special-effects practitioner.”
Unable to aff ord to go to film school in Los Angeles, Cameron supported himself as a truck driver and studied visual effects on weekends at the University of Southern California library, photocopying dissertations on optical printing and the sensitometry of film stocks. “ is is not bull,” he says. “I gave myself a great course on film FX for the cost of the copying.”
Cameron eventually landed a job on the effects crew of Roger Corman’s low-budget 1980 film Battle Beyond the Stars, but he didn’t tell anyone that he was an autodidact with no practical experience. When he was exposed to the reality of film production, it was very different from what he had imagined, he recalls: “It was totally gonzo problem solving. What do you do when Plans A, B and C have all crashed and burned by 9 am? at was my start. It wasn’t as a creative filmmaker – it was as a tech dude.”
Over the years, Cameron’s budgets have increased to become the biggest in the business, and digital technology has changed the realm of the possible in Hollywood, but Cameron is still very much the gonzo engineer. He helped found the special-effects company Digital Domain in the early 1990s, and he surrounds himself with Hollywood inventors such as Vince Pace, who developed special underwater lighting for Cameron’s 1989 undersea sci-fi thriller, The Abyss. Pace also worked with Cameron on Ghosts of the Abyss, a 2003 undersea 3D documentary that explored the wreck of the Titanic. For that movie, Pace and Cameron designed a unique hi-def 3D camera system that fused two Sony HDCF950 HD cameras 6 cm apart to mimic the stereoscopic separation of human eyes. The Fusion Camera System has since been used for 3D movies such as Journey to the Centre of the Earth and the upcoming Tron Legacy, and at sporting events such as the 2007 National Basketball Association finals.
The 3D experience is at the heart of Avatar. (In fact, some suspect that Cameron cannily delayed the movie’s release to wait for more cinemas to install 3D screens – there were more than 3 000 for the launch.) Stereoscopic moviemaking has historically been the novelty act of cinema. But Cameron sees 3D as a subtler experience. To film the live-action sequences of Avatar, he used a modified version of the Fusion camera. The new 3D camera creates an augmented-reality view for Cameron as he shoots, sensing its position on a motion-capture stage, then integrating the live actors into CG environments on the viewfinder. “It’s a unique way of shooting stereo movies,” says visual-effects supervisor Stephen Rosenbaum. “Cameron uses it to look into the environment; it’s not about beating people over the head with visual spectacle.” is immersive 3D brings a heightened believability to Avatar’s live-action sequences – gradually bringing viewers deeper into the exotic world of Pandora. In an early scene, Sully looks out the window as he flies over the giant trees and waterfalls of the jungle moon, and the depth afforded by the 3D perspective gives the planet mass and scale, making it as dizzyingly real for viewers as it is for him.
Shooting the virtual world
Yet live-action 3D was hardly the biggest technical challenge. Only about 25 per cent of the movie was created using traditional live performances on sets. The rest takes place in an entirely computergenerated world – combining performance capture with virtual environments that have never before been realised on film. Conjuring up this exotic world allowed Cameron to engage in “big-time design”, he says, with six-legged hammerhead thanators, armored direhorses, pterodactyllike banshees, hundreds of trees and plants, floating mountains and incredible landscapes, all created from scratch. He drew upon his experience with deep-sea biology and plant life for inspiration. Sigourney Weaver, who plays botanist Grace Augustine, calls it “the most ambitious movie I’ve ever been in. Every single plant and creature has come out of this crazy person’s head. This is what Cameron’s inner 14-year-old wanted to see.”
To bring his actors into this world, Cameron collaborated with Weta Digital, an effects house founded by The Lord of the Rings director Peter Jackson. Weta has created some of the most groundbreaking characters in recent years, using human performances to animate digital creatures such as Gollum in the Rings series and the great ape in Jackson’s 2005 version of King Kong. By now, the process of basic motion capture is well-established. Actors are dressed in “mocap” suits studded with reflective reference markers and stripes, then cameras capture the basic movements of a performance, which are later mapped to digital characters in a computer.
For actors, the process of performing within an imaginary world, squeezed into a leotard while pretending to inhabit an alien body, is a challenge. Motion-capture technology is capable of recording a 360-degree view of performances, so actors must play scenes with no idea where the “camera” will eventually be. Weaver found the experience liberating. “It’s simpler,” she says. “You just act. There’s no hair or makeup, nothing. It’s just you and the material. You forget everything but the story you’re telling.”
Directing within a virtual set is more difficult. Most directors choose their angles and shots on a computer screen in postproduction. But by then, most of the immediacy of the performance is lost. Cameron wanted to be able to see his actors moving within the virtual environments while still on the motion-capture stage (called the volume). So he challenged his virtual-production supervisor Glenn Derry to come up with a virtual camera that could show him a low-resolution view of Pandora as he shot the performances.
The resulting swing camera (so called because its screen could swing to any angle to give Cameron greater freedom of movement) is another of Avatar’s breakthrough technologies. The swing camera has no lens at all, only an LCD screen and markers that record its position and orientation within the volume relative to the actors. That position information is then run through an effects switcher, which feeds back low-resolution CG versions of both the actors and the environment of Pandora to the swing cam’s screen in real time.
This virtual camera allowed Cameron to shoot a scene simply by moving through the volume. Cameron could pick up the camera and shoot his actors photographically, as the performance occurred, or he could reshoot any scene by walking through the empty soundstage with the device after the actors were gone, capturing different camera angles as the scene replayed.
But all of this technology can lead right back into the uncanny valley, because capturing an actor’s movements is only a small step toward creating a believable digital character. Without the subtle expressions of the face, Cameron might as well be playing with marionettes. Getting this crucial element right required him to push Weta’s technology far beyond anything the company had done before.
In fact, Cameron doesn’t even like the term “motion capture” for the process used on Avatar. He prefers to call it “performance capture”. This may seem like semantics, but to Cameron, the subtle facial expressions that define an actor’s performance had been lost for many of the digital characters that have come before. In those films, the process of motion capture served only as a starting point for animators, who would finish the job with digital brush strokes. “Gollum’s face was entirely animated by hand,” says Weta Digital effects master Joe Letteri. “King Kong was a third or so straight performance capture. It was never automatic.” This time, Cameron wanted to keep the embellishment by animators to a minimum and let the actors drive their own performances.
To pull more data from the actors’ faces, Cameron reworked an old idea he had sketched on a napkin back in 1995: fasten a tiny camera to the front of a helmet to track every facial movement, from darting eyes and twitching noses to furrowing eyebrows and the tricky interaction of jaw, lips, teeth and tongue. “I knew I could not fail if I had a 100 per cent close-up of the actor 100 per cent of the time that travelled with them wherever they went,” he says. “That really makes a close-up come alive.”
The information from the cameras produced a digital framework, or rig, of an actor’s face. The rig was then given a set of rules that applied the muscle movements of each actor’s face to that of the avatar or the Na’vi that he or she was playing. To make a CG character express the same emotion as a human actor, the rig had to translate every arch of a human eyebrow directly to the digital character’s face.
But it turns out there is no magic formula that can supplant hard work and lots of trial and error. After Cameron complained about the uncanny-valley effect, Weta spent another year perfecting the rig on Worthington’s avatar by tweaking the algorithms that guided its movements and expressions until he came alive enough to meet Cameron’s sky-high standards. “It was torturous,” Letteri admits. But when Weta was finished, you could pour the motion-capture data into the rig and it would come out the other side right.
With so much attention focused on Avatar, anything short of perfection might not have been good enough. After all, Cameron was asking moviegoers to believe in a deep new universe of his own design, and to buy the concept that 3 m-tall blue aliens were able to communicate human emotions. If he was wrong, Avatar might be remembered as the moment when the battle for the uncanny valley was lost. If he was right, the technology would disappear behind the story line, and audiences would lose themselves in Avatar’s world. Judging from audience reactions so far, he’s aced it.
Cameron’s greatest FX hits
Battle Beyond the Stars
Effect: Spaceship backdrop
Producer Roger Corman wanted his actors in spaceships, but building sets was too expensive. So Cameron – who started as the film’s model builder and then became its art director – photocollaged matte shots and
projected them on a backdrop screen, then filmed the actors in front of it. The front-projection system is still in use today.
Effect: Robot self-surgery
After his directorial debut (1981’s Piranha Part Two), Cameron was under pressure to make his mark. He and FX whiz Stan Winston designed a prosthetic to make it appear as though the Terminator was taking apart his
own robotic arm – while allowing actor Arnold Schwarzenegger to wiggle his fingers. These days, the effect would be computergenerated.
Effect: Water tentacle
Innovations in The Abyss included illuminated water helmets and self-propelled camera platforms. But Cameron’s biggest advance was in computer graphics. A digitally rendered water tentacle was cinema’s first soft-surface character with a human face. The creature was based on scans of actors’ faces used to create a 3D polygonal mesh surface.
Terminator 2: Judgment Day
Effect: The T-1000 morphs into an LAPD patrolman
Cameron resurrected the “morph” technique used in The Abyss to turn actor Robert Patrick into the liquid-metal T-1000. This was the first time dialogue was synced with a computer-generated human face and full-body motion. It would take 18 years for visual-effects artists to create a photoreal CG human.
Effect: A virtual crowd on the ship’s deck
Cameron and effects house Digital Domain used motion capture to create the iconic scenes of many passengers walking on the Titanic’s decks. A few actors’ movements were captured, then Cameron replicated them. Director Peter Jackson and his effects house, Weta Digital, later refined the technology to populate huge battle scenes in The Lord of the Rings.
Ghosts of the Abyss
Effect: 3D dives to the Titanic wreckage
To shoot a series of scientific documentaries, Cameron and inventor Vince Pace developed a stereoscopic camera – really, two cameras aligned to mimic human depth perception. The rig used in Ghosts was a precursor to the Fusion 3D cameras Cameron relied on for Avatar’s live-action sequences.
Directing a virtual world
1. The volume
Most of James Cameron’s space epic, Avatar, was shot on a performance-capture stage, known as the volume, in Playa Vista, California. The volume was rimmed by 120 stationary video cameras, which could record the movements of all actors at once in 3D, with submillimetre precision. Data from the cameras was streamed into Autodesk software, which translates actors’ movements into digital characters in real time within a low-resolution computer-generated environment. So riding a fake banshee mockup onstage instantly translated to CG footage. Multiple cameramen were used on set for reference video, but because the volume essentially captures performances from every angle at once, Cameron could digitally render whatever angles and shots he wanted after the performance, adjusting the camera movements while viewing playback.
2. Digital close-up
Like many actors in Avatar, Zoë Saldana plays a fully computer-generated character, Na’vi princess Neytiri.
To map her movements to her digital doppelgänger, Saldana wore a motion-capture bodysuit with reference markers and stripes. She also wore a head rig designed by Cameron that aimed a small video camera at her face. That camera tracked green ink dots, painted on Saldana’s face, throughout the scene, giving Cameron closeup-level detail of changes in expression to map to Neytiri’s CG face.
3. Un-set playback
To shoot a scene within a totally CG world, Cameron had virtual production supervisor Glenn Derry rig
up augmented-reality cameras. Cameron could watch from the sidelines as his actors’ performances were
instantly mapped to their CG characters and displayed via an on-set screen. Or he could use a portable,
motion-tracking virtual camera to walk through the volume and view the CG environment of the movie on its LCD screen.
4. Final render
To transition from the CG produced on set to the photorealistic world of the finished movie, Cameron sent his
rough footage to Weta Digital in New Zealand. There, special-effects programmers used a facial solve program and facial action coding to translate the actors’ every minute muscle movement – blinks, twitches, frowns – to believable expressions on the faces of Pandora’s aliens.
3D comes home
Stereoscopic movies make more money per cinema than their 2D counterparts, but according to the 3D@Home Consortium, Hollywood studios generate as much as half their revenue from DVD sales. So the next big push for stereoscopic content is home theatre. But is it worth the upgrade? Here’s what you need to know. By Erin McCarthy
What equipment will I need to play 3D?
Some TVs, such as DLP rear-projection sets, can already play 3D content, but current LCDs and plasmas can’t, so most people will have to spring for a new set—Panasonic has announced stereoscopic plasmas for 2010. New 3D Blu-ray players are also coming this year, but some current machines, such as the PlayStation 3, will need only a simple firmware upgrade. Also required: shutter glasses, such as the Nvidia 3D Vision Kit.
What about content?
Right now, PC games are the only widely available stereoscopic content, but 3D@ Home’s Chris Chinnock expects approximately 35 movies to be released on the upcoming Blu-ray 3D format within the next year, and approximately 130 titles in the next two years – pricing for the new discs is still unknown. Stereoscopic
video games for PS3, Xbox and Wii are also coming this year, and 3D movies will eventually be
downloadable to PCs and TVs via services such as Roxio CinemaNow.