Pirates of the Caribbean: On Stranger Tides

First Cut
Starting their own plans with the original storyboards, Simon’s team took care to ask for the first edit early on because the post schedule on the sequence was very tight, among the first shot and delivered for the London location. The production also needed a rough cut back in the meantime to check how the edit was working. “That first edit was very long – over 10 minutes,” said VFX Supervisor Simon Stanley-Clamp. “Then it was cut back a few times before arriving at the 5 1/2 or so minutes you see in the final film.

“While awaiting the first edit, we went on set to gather what we needed to start designing CG buildings and environments to replace the very large blue screens in place at the three locations in and around London. We based the CG on existing architecture, often in side streets and mews. Well in advance of the plates arriving, we built 3D assets and tested our photogrammetry techniques in csPhotoMesh, the facility’s new software, to prove that we could capture stills of a building, model from the stills and create an effective environment.”
Once they were awarded the sequence, they had spent a week surveying the film’s main London locations at Greenwich, Hampton Court Palace and Middle Temple, plus the sets at Pinewood Studios in the docklands. The team captured numerous lighting and texture stills, including before and during set construction at Greenwich where Simon stayed on set each day of the three week shoot. They also managed to do a full set survey at Middle Temple, where the shoot had to be completed in a single day to accommodate the lawyers working there. The entire London section of the film was shot over three months.

Blue Screen Streets
Because the edit was uncertain for some time, the team wasn’t able to previs but used their stills and data capture to pre-build much of the architecture they would need for the different environments. For example, Greenwich had a distinctive look for which they built up a library of a dozen building variations, approved and ready to drop in, before they actually had any footage.

Street layout came from the stills, the storyboards and a few production designs, which Simon extrapolated. This helped them anticipate where they would shoot from. He would send back concept stills to production, showing the buildings he suggested to fill gaps and replace blue screens, sometimes with options. Few changes were made to these looks. The main criterion was that the buildings simply be completely unnoticeable and look ‘right’ for the situation.
The blue screens in some spots were enormous, in one case requiring that the team replace a full street of CG buildings. The justification for it was economic. The street was only needed for two or three shots, and an equivalent practical set would have been huge. Another massive blue screen was employed for the opening shot of London, passing through an arch with St Pauls cathedral in the distance, and others had to be replaced at either end of the Greenwich location to extend into the distance.

Pre-Composites
The first plates arrived in October and November. The team helped production trim the cut down and figure out was needed, shot by shot, by making rough pre-comps – taking still frames, putting them on cards and dropping them into position – so the editors could see the shots without blues screen and hone the looks as they liked. “It wasn’t true previs, but it wasn’t a finished comp either. The process also helped us decide what elements had to be shot,” said Simon.

“People walking through frame alone or in groups, shot on blue screen, were supplied abundantly for us to randomise in the plates, all shot in stereo. Later on, fire and smoke elements were shot in stereo as well. One critical smoke element was commissioned and shot only days before the sequence delivery. The story needed extra smoke as a neat escape route for Jack Sparrow, allowing him to swing undetected from a pub sign, but we didn’t have enough time for CG smoke development. We match moved the plate and layered up multiple bits of the practical smoke.”

“The first 10 or 12 minutes of the film’s opening London shots, coming across the water and leading into the sequence in King George’s palace before the carriage chase, were part of our award, the film essentially begins with our work,” said Simon. “We were even responsible for modelling and animating Jack’s CG cream puff for the palace dining room shots, which gets tossed around until it gets stuck on the chandelier. This chandelier was a massive, weighty practical prop controlled by giant pullies and motors that had to support Jack. But it didn’t swing naturally, so apart from Jack’s rig removal, we needed to carry out variable respeeds to give its motion the expected look.”

Variable Respeeds
Usually variable respeeds are frame or vector based for the smoothest interpolation. But in stereo, the associated clean up must extremely thorough and precise. They built a tool in Nuke that would do a ‘nearest’, vector or blended respeed, or a combination of any of these as required. They could feed in a QuickTime reference, get an exact match and choose whatever process worked best. Then they made sure it was performed exactly the same way in each eye.

Simon feels that, at this stage in stereo development, respeeds are best kept to a minimum, especially the variable type. “Sometimes they have to come back and be redone to make sure they look right and match the cut. They can even affect the audio because of the lipsync. The editors were usually quite helpful on this,” he said.

“This way, what we rendered is an exact match to each eye. Then we can use tools in Nuke, a nudge tool, for example, to adjust and fine-tune shots so that all elements sit at their true stereo depth. But generally, rendering and tracking tasks simply take twice as long, which made the whole project take substantially longer than it would have as a 2D project. However, this was something that the production and vendors were all aware of from the outset.” Tracking was usually done with 3D Equalizer or a Nuke track to finesse elements into place.
Consequently, from shoot to post, emphasis was on avoiding too much clean up work, making sure that what got into the plate was supposed to be there. Stereo cleanup means painting things out in precisely the same way in each eye to avoid artefacts and floating masks that attract the eye. Perhaps what gave us the most help was having their own stereographer on board, almost from the start of the shoot. As the plates turned over, he analysed them and picked up errors the team could correct themselves.

Stereographer
Simon said, “Convergence information wasn’t always immediately provided from production, causing delays, but the stereographer could advise us on the best convergence to set to allow us to go ahead. It was really helpful, especially combined with the in-house tools and using Nuke for all of the compositing. Being able to produce stereo QuickTimes for client review, for example, made a lot of difference to the workflow.”

Equipping the facility was a major but essential step. All compositors now have their own 3D monitors and Cinesite’s main theatre has been converted to Dolby D. A dedicated suite with stereo viewing system was built for ‘Pirates’. “You can’t guess about the images. They all have to be checked in the same way that they will be viewed in cinemas. The stereographer needed his own suite with monitors as well. Virtually every project now has a stereo agenda or deliverable, and now we are set up for it,” said Simon.

Changing Light
As the chase sequence was shot during the northern autumn, October and November, light was not consistent from set to set. Greenwich featured flat white-greyskies, which was a good match for the classic, gritty old London look specified in the early stages. Middle Temple was shot within the same period and didn’t cause many problems but over Pinewood’s exterior set, the skies were bright and sunny almost every day. This meant hanging huge diffusers over the sets to prevent sharp shadows and where the diffusers failed, the team had to remove shadows, re-grade some shots and replace a few backgrounds with murkier environments.

Slight rainfall at Greenwich wouldn’t have been an issue except for the stereo factor. The Pace rig the production were using has an exposed polarising mirror at the front. If rain falls on it, the drops appear in the footage as floating artefacts in the foreground. Cleaning these out in post is very time-consuming and expensive. At the shoot such problems were sometimes handled with eccentric measures like driving the rig backwards down the road, which gave them reversed shots but kept the rain off the mirror.

A very talented Jack Sparrow double performed many of the trickier stunts for actor Johnny Depp, and other stunt men stood in for him on specific manoeuvres. “But whenever he is recognisable as himself, it really is him – there were no face replacements for him. Instead stunt rigs were often used that required substantial digital removal from the stereo footage. In one case they had to replace a digital building to make sure the cable was totally cleaned out.

Frog Engineering
The pirates encounter poisonous frogs in the jungle, which Captain Barbossa captures in a jar. For the scene, Cinesite created and animated a full CG frog complete in every detail – a complete rig, wet eyes, in red, yellow, green and blue. For its size, it’s highly over-engineered,” admitted Simon. “You might even miss it. It was one of the first shots we completed, based on plates from the Hawaii location. I’m always more confident when I can survey the set myself for tracking and lighting data but since we didn’t have anyone working over there, we had to rely on the set information supplied to us.

Walk the Walk
Barbossa’s peg leg was an effect that appeared in most vendors’ shots. They shared out the work on it within their awarded shots. A physical peg leg was built as a looks reference and, again, stills of it were captured on location, plus texture reference shots, measurements and a cyber scan for an exact build and texture. Simon had shots taken of it just after each shoot – sometimes it appeared close-up – with the RED cameras to record it exactly as it had looked in the rest of the sequence under the same lighting with all the actors and gear still in place.
The rig was straightforward. They had built a model of Barbossa, and used it for shots when he was walking around, match moving it into the shot and animating only his lower torso performing the required walk. Not surprisingly, the way the actor walks and the way he is meant to appear on screen are different. The leg was meant to be rigid from the hip and unable to bend at the knee but as Geoffrey Rush walks around in his blue sock with the tracking markers, he does bend his knee.

“The cleanup was the tricky part, often needing a bit more than only the peg portion. The animation required a few trials to lock down while making sure that leg was perfectly straight! We weren’t match moving what the actor is doing but what it would look like if he weren’t bending his knee.

No Magic
“The hardest shot was Barbossa’s first, when the camera follows him into King George’s palace hall and he is seen virtually in silhouette. This made tracking and clean up harder, and we needed to make a replacement marble floor to reveal reflections and shadows from the leg. Invisibility and realism was crucial in our shots. Our team was not dealing with magic, lasers and collapsing buildings.

“We had all lighting scenarios to deal with as well. When Jack clashes with Angelica, posing as himself, in the sword fight in the Captain’s Daughter pub, the fire-lit interior is nearly dark. As the pair fight in the rafters overhead, they are wearing complicated rigs against the smoky ceiling behind them. To clean out the wires, we replaced the roof in CG and we also built the CG barrels you see in the background.”

Frozen in Time
The ILM Singapore team headed by VFX Supervisor Mohen Leo mainly took on a supporting role for ILM’s artists in San Francisco but two intriguing sequences were handled entirely in Singapore. The team needed to create miniature pirate ships, complete and realistic in every detail, inside a number of glass bottles. In the film’s story, the nefarious Blackbeard possesses magical powers that allow him to seize and shrink these ships to capture them in bottles. Furthermore, he traps each ship with precisely the same ocean conditions of its capture, freezing that moment in time and in miniature.

Although Mohen’s team had no one on set, he spent a month in San Francisco working with ILM’s supervisor there, Ben Snow, before the shoot. He took the opportunity to discuss technical approaches for the bottle sequences and what data they would need from set to complete the work in post.

Their main sequence involved Blackbeard’s full collection of bottles and takes place below decks on his ship. To get started on looks and aesthetics, the team focused on live action props in the plates – some model ships and a cabinet holding empty bottles, which the production crew had supplied as practical elements for the wider shots. In CG, the ILM team replaced the models, built ships and environments for the existing bottles, and built a large number of CG bottles with ships and environments inside.

Magic Moment
The team also needed to build a variety of ocean conditions, from calm seas to heavy storms. As the work got underway, the idea for the sequence evolved. The director Rob Marshall decided it would also be good to preserve the time of day and lighting conditions of their moment of capture, some in broad daylight, some at sunset, others at night.
This was particularly challenging because ultimately, they all had to go into the same overall environment of the background plate and tie into it. ILM’s concept artist John Bell prepared a few concepts and the Compositor Paolo Acri did some look development on one of the wider shots and started roughing in ideas for different conditions, light and other looks to show the director.

In the end, the project went one step further when one compositor, Ben Warner, tried putting canon fire and small explosions on one of the ships as if it were trapped in an endless battle. The director really liked this idea – it wasn’t just capturing weather and time but also a narrative moment in the bottles, so that each ship possessed its own story.
This resulted in one ship covered in ice caught in a snow flurry at night, for example, and Jack Sparrow’s own Black Pearl still had the parrot and monkey from the previous film, held in an eternal lightning storm with waves crashing against it. Within that scope, the team had tremendous creative freedom. The client gave them opportunities to try out different ideas and Rob chose the ones he liked, even a completely sunken ship with fish swimming by.

Lighting Interaction
For the vessels themselves they took inspiration from the model ships the art department supplied, plus ships they had in their library after working on all three previous ‘Pirates’ movies. They started with these resources, though Rob eventually diverged from the set design slightly and requested a Chinese junk in one bottle. They constructed close to a dozen different CG ships within CG bottles.

The representation of the bottles changed from shot to shot.  For close ups, practical bottles in the cabinet were removed to make way for the fully CG assets, and for the camera when taking reverse angle shots looking back at the actors as they discussed the ships. In these close shots, the lighting interaction between the ship and the bottle needed to be carefully designed and controlled, which was best achieved when both were built in CG.
In wide shots, the crews’ live action bottles were left on the cabinet and could be retained as secondary bottles, only briefly glimpsed, with the team’s CG ships, oceans and weather placed inside and motion applied. The team really enjoyed this chance to devote their efforts to something purely ‘pretty’, with almost more detail than could be appreciated in one viewing.

Ocean Looks
A second sequence occurs on the beach at the end of the movie, when Jack’s old friend Joshamee Gibbs returns the Black Pearl in its bottle to Jack, who must then figure out how to release his ship. The light in these shot presented a reversed lighting scenario, but the same challenge. Mohen said, “In order to let the ships keep their own time of day and still visually fit into the background plates we had to bridge the two lighting scenarios. In the dark sequence under deck we had light from the brighter bottles spill out onto the cabinet, while in the beach sequence we had the warm sunlight in the plate pick up some highlights on the Black Pearl in its otherwise gloomy overcast environment in the bottle.”

In all, the team completed over 40 of these ship and bottle shots. The shots were composed almost entirely of the plate material and their CG, with some rotoscope work from the plates and a few blue screen shots of Jack’s face shot later in the schedule. The CG pipeline at the facility comprises a combination of in-house and off-the shelf software. Modelling is done in Maya, creature simulations, effects and lighting are proprietary. Texturing, rotoscoping and paint are all done with a combination of tools. Compositing is in Nuke.
Effects lead John Kilshaw based the water and fluid simulations on ILM’s water software used on the third ‘Pirates’ film and more recently on ‘The Last Airbender’. They began with stormy, choppy and calm water looks. From there, the spray, mist, foam and underwater bubbles from churned up water were developed. All these simulations were run and placed into the individual bottle environments with the ships. By mixing and matching the ocean and lighting conditions, they generated numerous different looks for the different bottles.

Unforgiving
“The stereo 3D element of this project added a few special complications for our work,” said Mohen. “ILM had stereo experience from ‘Avatar’, of course, but this was the first time they were involved from the start and on such a large scale. Also, this was shot completely in stereo, with no conversion work. It was an opportunity to advance the stereo pipelines for compositing, for camera match move and layout, even for rotoscoping.

“One challenge was simply how unforgiving stereo really is for a VFX artist. The small cheats and inaccuracies are no longer viable. The viewer’s brain knows naturally how to perceive stereo vision, thus every detail and mistake is instantly recognized. Even a pixel or two out of place between left and right eyes will spoil the depth of a shot. It demands complete accuracy and the quality control is time-consuming.”
The fact that the bottles were made of thick hand-crafted glass that refracted within the bottle added a separate challenge. Had they rendered all their CG assets with those refraction effects built into their looks, they would have been inflexible in the composite. Nothing could have been added, such as practical elements, or tweaked.

Independent Render
Consequently, lighting Lead Simeon Bassett and Compositing Lead Jon Bowen modified their shaders and Nuke set ups in such a way that allowed them to render the bottle contents, especially the hero shots, without refractions from the glass and then apply them in the composite at the very end in a way that still worked in stereo. The result looked correct while allowing enough flexibility to adjust the looks exactly as required, independent of the refractions.

Compositing was all done in Nuke. Tracking was, again, done with proprietary tools but the stereo made it quite complex, demanding absolute precision. The layout team was finding the stereo plates substantially harder to manage but they have now advanced their pipeline and feel they can move onto other projects with more confidence.

“It’s not just the tools but the artists themselves who advance. We all learned how to judge stereo images with our own eyes – at first, we were all tempted to rely on 3D glasses for this, but it’s not always the best technique to identify errors. Once the eye and brain are trained, it can be much more effective to just flip back and forth between left and right images – without glasses. By detecting shifts between them, you can determine exactly where and why errors are occurring.”

“csPhotoMesh is photogrammetry and 3D scene reconstruction software, which was useful for building the carriage chase sequence environments,” said Michele. “It is a simple, flexible way to capture geometry. Given a set of digital images of a static scene, it produces a textured 3D mesh accurately representing the scene geometry and 3D cameras matching the original photos’ positions. The function is automatic – you just drop all the images in a directory and run the command. This kicks off a reconstruction process on our render farm resulting in a 3D mesh and camera positions ready for texturing.

“csStereoColourMatcher is also fully automated. Colour differences between stereoscopic image pairs can be caused, for example, by the beam splitter in the camera rig, or other factors that introduce significant colour shifts across different stereo views. Derived from vector-based analysis, we built it into the front end of all our compositing work for the film. It requires no user supervision and is completely integrated into the Nuke compositing system. We used it to colour balance more than 300 shots for the film.”