Keying

About keying: color keys, luminance keys, and difference keys

Keying is defining transparency by a particular color value or luminance value in an image. When you key out a value, all pixels that have colors or luminance values similar to that value become transparent.

Keying makes it easy to replace a background, which is especially useful when you work with objects too complex to mask easily. When you place a keyed layer over another layer, the result forms a composite, in which the background is visible wherever the keyed layer is transparent.

You often see composites made with keying techniques in movies, for example, when an actor appears to dangle from a helicopter or float in outer space. To create this effect, the actor is filmed in an appropriate position against a solid-color background screen. The background color is then keyed out and the scene with the actor is composited over a new background.

The technique of keying out a background of a consistent color is often called bluescreening or greenscreening, although you don’t have to use a blue or green screen; you can use any solid color for a background. Red screens are often used for shooting non-human objects, such as miniature models of cars and space ships. Magenta screens have been used for keying work in some feature films renowned for their visual effects. Other common terms for this kind of keying are color keying and chroma keying.

Difference keying works differently from color keying. Difference keying defines transparency with respect to a particular baseline background image. Instead of keying out a single-color screen, you can key out an arbitrary background. To use difference keying, you must have at least one frame that contains only the background; other frames are compared to this frame, and the background pixels are made transparent, leaving the foreground objects. Noise, grain, and other subtle variations can make difference keying very difficult to use in practice.

Keying effects, including Keylight

After Effects includes several built-in keying effects, as well as the Academy Award-winning Keylight effect, which excels at professional-quality color keying. (See Keying effects and Matte effects.)

For information on the Keylight effect, see its documentation in the folder in which the Keylight plug-in is installed, or on the Foundry website.

The Key Cleaner and Advanced Spill Suppressor effects are most effective when applied together, in that order, after a keying effect like Keylight. 

Use the Keylight + Key Cleaner + Advanced Spill Suppressor animation preset (located in the Image-Utilities presets folder) to apply all the three effects. The Advanced Spill Suppressor effect is turned off by default to allow you to sample the key color in the Keylight effect or if the footage does not have any color spill to be removed. For more information, see Key Cleaner and Advanced Spill Suppressor effect.

Mark Christiansen provides tips and techniques for using Keylight in an excerpt from his book After Effects Studio Techniques: Visual Effects and Compositing on the Peachpit Press website. In an excerpt from the “Color Keying in After Effects” chapter of After Effects Studio Techniques, Mark Christiansen provides detailed tips and techniques for color keying, including advice on which keying effects to avoid and how to overcome common keying challenges.

For a step-by-step tutorial demonstrating the use of the Color Difference Key effect, the Matte Choker effect, the Spill Suppressor effect, and garbage masks, see the “Keying in After Effects” chapter of the After Effects Classroom in a Book on the Peachpit Press website.

Rich Young collects more tips and resources for keying on his After Effects Portal website.

Tips on color keying and compositing from experienced compositor, Chris Zwar.

Shooting and acquiring footage for keying

• Light your color screen uniformly, and keep it free of wrinkles.

• Start with the highest-quality materials you can gather, such as film that you scan and digitize.

• Use uncompressed footage (or, at least, files with the least possible amount of compression). Many compression algorithms, especially the algorithms used in DV, HDV, and Motion JPEG, discard subtle variations in blue—which may be necessary to create a good key from a bluescreen. Use footage with the least color subsampling possible—for example, 4:2:2 rather than 4:2:0 or 4:1:1. 

Robbie Carman and Richard Harrington provide an excerpt on the Peachpit website from their book Video Made On A Mac that demonstrates how to plan, shoot, key, and composite a greenscreen shot.

Tips for keying with After Effects

• Use a garbage matte to roughly outline your subject so that you don’t have to waste time keying out parts of the background far from the foreground subject. (See Use a garbage matte.)
• Use a hold-out matte to roughly protect areas that are of a similar color to the background from being keyed out. (See Use a hold-out matte.)
• To help you view transparency, temporarily change the background color of the composition, or include a background layer behind the layer you are keying out. As you apply the keying effect to the layer in the foreground, the composition background (or a background layer) shows through, making it easy to view transparent areas. (See Composition settings.)
• For evenly lit footage, adjust keying controls on only one frame. Choose the most intricate frame of the scene, one involving fine detail such as hair and transparent or semitransparent objects, such as smoke or glass. If the lighting is constant, the same settings you apply to the first frame are applied to all subsequent frames. If lighting changes, you may need to adjust keying controls for other frames. Place keyframes for the first set of keying properties at the start of the scene. If you are setting keyframes for one property only, use Linear interpolation. For footage that requires keyframes for multiple interacting properties, use Hold interpolation. If you set keyframes for keying properties, you may want to check the results frame by frame. Intermediate keying values may appear, producing unexpected results.
• To key well-lit footage shot against a color screen, start with the Color Difference Key. Add the Advanced Spill Suppressor effect to remove traces of the key color, and then use one or more of the other Matte effects, if necessary. If you are not satisfied with the results, try starting again with the Linear Color Key.
• To key well-lit footage shot against multiple colors or unevenly lit footage shot against a bluescreen or greenscreen, start with the Color Range key. Add the Advanced Spill Suppressor
  and other effects to refine the matte. If you are not completely satisfied with the results, try starting with or adding the Linear Color Key.
• Using the Key Cleaner and the Advanced Spill Suppressor effect in sequence is the best way to go about applying Keying effects.  You can use the Keylight effect in combination with the Key Cleaner and Advanced Spill Suppressor effects in that order on a layer using the Keylight+Key Cleaner+Advanced Spill Suppressor animation preset in the Image-Utlities folder under Animation Presets. The Advanced Spill Suppressor effect is turned off by default to allow you to sample the key color in the Keylight effect or if the footage does not have any color spill to be removed. For more information, see Key Cleaner and Advanced Spill Suppressor effect.
• To key dark areas or shadows, use the Extract Key on the Luminance channel.
• To make a static background scene transparent, use the Difference Matte Key. Add the Simple Choker and other effects as needed to refine the matte.
• After you have used a key to create transparency, use Matte effects to remove traces of key color and create clean edges.
• Blurring the alpha channel after keying can soften the edges of the matte, which can improve compositing results.

Use a garbage matte

A garbage matte (or junk matte) removes unneeded portions of the scene, resulting in a rough area that contains only the subject that you want to keep. When you are working with a poorly lit or uneven color screen (for example, a bluescreen or greenscreen), sketching a garbage matte around the subject can greatly reduce the amount of work that you have to do in keying out the background. However, if you spend a lot of time making a perfect garbage matte that exactly outlines the subject—essentially rotoscoping—you lose the time-saving advantage of keying.

Use a hold-out matte

Use a hold-out matte (also known as a hold-back matte) to patch a scene to which a keying effect has been applied.

A hold-out matte is a masked-out portion of a duplicate of a layer that you have keyed. The duplicate is masked to include only the area of the image that contains the key color that you want to preserve as opaque. The hold-out matte is then placed directly on top of the keyed layer.