Screen grab

Further advancement in motion-capture technology is showing at a cinema near you, says Michael Spiteri.

Can an actor’s dramatic performance be captured in computer-generated animation?

Can photorealistic interpretations of human emotions such as love, joy, anger and fear be achieved in terms of megabytes? Just like their live action counterparts, computer-generated films are an existing fusion between art and technology at the service of a story, but in their making they present totally unique challenges.

There are two types of animation. Key-frame animators construct and manipulate 3D polygons on a frame-by-frame basis before adding various layers of textures and lighting to render the completed image. However, elements like water, due to translucency and reflective properties, are very hard to model in a virtual environment.

Therefore, by simple comparison, one can start to understand why, by far, the greatest challenge in computer-generated scenes is the depiction of people. The repertoire of motion that the human anatomy encompasses is vast, and familiar to all audiences.

In fact, when something is missing or exaggerated, we may not be able to pinpoint it, but we know something is not quite right. Jerkiness of movement and flatness of expression are the worst hurdles animators have to overcome when it comes to realism in animation.

That is why most computer-generated films opt to have anthropomorphic protagonists – animals or machines behaving like humans. Or else, as in Pixar’s The Incredibles and Up, humans are portrayed in stylised fashions.

Motion-capture, or mo-cap for short, is a technique that departs from frame-by-frame animation. It involves recording human bodily movements and mapping them onto computer-generated characters.

The idea behind mo-cap dates back to the 1800s, when Etienne Jules Marey came up with the revolutionary idea to record various stages of motion on a single photographic surface.

Traditional animators were aware of it as well. They attempted to replicate fluid motion in their films mainly by rotoscoping, tracing directly over pre-recorded live footage projected onto a frosted glass panel. A famous example of rotoscope use is Walt Disney’s first 1937 animated feature film Snow White and the Seven Dwarfs.

Computers were the next logical step and, as they became more advanced, motion capture evolved rapidly.

Primarily, there are four methods to capture motion from a targeted body. Prosthetic motion capture uses rotational and linear encoders rigged on a plastic endoskeleton that can be worn by the actor.

The second option is acoustic motion capture, which uses audio transmitters that click or bleep when moved. Electromagnetic motion capture consists of a central magnet with several field receivers attached to the actor’s limbs.

The most popular method is the optical fibre motion capture, which offers the most freedom to the performer because it requires almost no cabling.

Optical systems make use of directionally reflective balls, the markers, and special light detecting cameras. The cameras gather the information and feed it to a computer, where a 3D position of every single marker is calculated.

The major drawbacks of mo-cap are calibration, marker occlusion and processing time. Without precise calibration, the readings and, as a direct consequence, the movements of the virtual character are offset. It is almost inevitable that the same actor obstructs one camera vision or another and improper marker identification confuses the software.

Adding more markers and cameras is theoretically the best solution but it increases the volume of data to be processed and time is always a determining factor, especially when pressed to meet a release date.

Due to these restrictions, filmmakers often choose to merge motion capture with key-frame animation in an attempt to get the best of both. Perhaps the greatest achievement in this context was Gollum in Peter Jackson’s The Lord of the Rings trilogy.

The body language of Andy Serkis was directly transferred by motion capture, but subtler details like his facial features were left to key-frame animators. Serkis’s expressions were laboriously copied and replaced on every frame (24 frames for every second of film). Other notable achievements were the giant ape in King Kong, also by Peter Jackson, and the pirate Davy Jones in the Pirates of the Caribbean films.

No other film director has invested more energy and resources in mo-cap than Robert Zemeckis. The distinguished director dedicated his last three films to pushing the technique’s boundaries.

He pioneered performance capture to capture emotion as well as motion. The Polar Express, Beowulf and A Christmas Carol are animated films with photorealistic computer-generated human characters.

During the shoot of A Christmas Carol, the special light-detecting cameras gave way to 3D receptors that simultaneously recorded various actors interacting. Also, four HD cameras rigged on helmets monitored the shifting of painted dots, not markers, upon the facial area.

Practically every move of every crease and pore on the actor’s face was used to create a virtual skin. For the first time, watching the finished film it was difficult to dispute the fact that the characters on screen were conjuring true emotions.

In 2010, James Cameron employed similar equipment on The Volume, a cutting-edge huge performance capture stage he developed for his sci-fi epic Avatar. During the mammoth production, the crew went as far as making casts of the actor’s heads to construct apposite helmets to minimise reading errors.

The Adventures of Tintin: The Secret of the Unicorn will be released later this year. The film, shot entirely on a performance capture stage, will mark the first of a series of collaborations between Stephen Spielberg and Peter Jackson.

It seems performance capture is part of the future. Actors can now transform beyond recognition and natural human emotions can emanate from the most abstract forms imaginable.

Mr Spiteri is qualified in electrical and electronics engineering and has a passion for all things digital.