Managing and animating shape paths and masks

You animate mask paths and shape paths in much the same way that you animate other properties: set keyframes for the Mask Path or Path property, set paths at each keyframe, and After Effects will interpolate between these specified values.

Rotoscoping (commonly known as roto) is the drawing or painting on frames of a movie, using visual elements in the movie as a reference. A common kind of rotoscoping is tracing a path around an object in a movie and using that path as a mask to separate the object from its background. This allows you to work with the object and the background separately, so you can do things like apply different effects to the object than to its background or replace the background.

Rotoscoping in After Effects is mostly a matter of drawing masks, animating the mask path, and then using these masks to define a matte. Many additional tasks and techniques make this job easier, such as using motion tracking on the object before you begin drawing masks, and then using the motion tracking data to make a mask or matte automatically follow the object.

Rotoscoping tips

• Immediately after beginning to draw a mask, press Alt+Shift+M (Windows) or Option+Shift+M (Mac OS) to turn on keyframing for that mask and set a keyframe. This way, you won’t edit a mask frame-by-frame for several minutes (or longer) and then realize that you lost all of your work on previous frames because you forgot to click the stopwatch to make the mask path animated.
• Draw your masks on a white solid layer with its Video (eyeball) switch off, above the (locked) footage layer. This way, you run no risk of accidentally moving the footage layer when you manipulate the mask, and you can also much more easily apply tracking data to the mask. (You apply the tracking data to the invisible solid layer that holds the mask.) This also means that you don't lose your cached preview frames each time you manipulate the mask. (See Toggle visibility or influence of a layer or property group and Lock or unlock a layer.)
• Turn on the Preserve Constant Vertex Count preference. (See Designate the first vertex for a Bezier path.)
• When possible, transform (rotate, scale, move) the whole mask or a subset of the mask vertices instead of moving the vertices individually. This is both for efficiency and to avoid the chatter that comes from inconsistent movement across frames. (See Move vertices in free-transform mode.)
• Manual motion tracking is less time-consuming than manual rotoscoping. The more effort you spend getting good tracking data for various parts of your scene and object, the less time you'll spend drawing and fine-tuning masks. (See Tracking and stabilizing motion.)
• Use the variable-width mask feather feature for more control when feathering objects.

View mask paths and shapes

• To view mask paths for selected layers in the Timeline panel, press M.
• To view selected masks or shapes in the Timeline panel, press SS (press the S key twice).
• To view mask and shape paths in the Composition panel, click the Toggle Mask And Shape Path Visibility button at the bottom of the Composition panel.
• To view mask paths in the Layer panel, choose Masks from the Layer panel View menu.
• To hide a mask path while showing others, lock the mask by selecting its Lock switch in the Timeline panel, and then choose Layer > Mask > Hide Locked Masks.
• To isolate selected masks and hide others, choose Layer > Mask > Lock Other Masks, and then choose Layer > Mask Hide Locked Masks.

Select shape paths, shapes, and shape groups

You can select shape layers and their components at any of four levels of selection, referred to as selection modes:

The entire shape layer is selected. Transformations apply to the transform properties for the layer, in the Transform property group that is at the same level as the Contents property group.

An entire shape group is selected. Transformations apply to the transform properties for the group, in the Transform property group within the shape group in the Timeline panel.

Multiple vertices on one or more Bezier paths are selected. A free-transform bounding box is shown around the vertices in the Composition panel. By operating on this box, you can move multiple vertices with a single transformation. Transformations apply to the vertices themselves, which are contained within the Path property in the Timeline panel.

Only vertices are selected. In this mode, you can perform path-editing operations, such as adding vertices to a path and moving individual vertices.

For information on selecting masks, see Select masks, segments, and vertices.

Select a shape layer

• Click the layer name or layer duration bar in the Timeline panel.
• Using the Selection tool, click within the layer bounds in the Composition panel.
• To deselect all shapes on a layer but leave the shape layer selected, click within the layer bounds but outside all shape paths.

Select a shape group in group selection mode

• Using the Selection tool, double-click a member of the group in the Composition panel. Each time that you double-click, you descend another level in the group hierarchy.
• To activate the Direct Selection tool , hold Ctrl (Windows) or Command (Mac OS) with the Selection tool selected. Click a shape in the Composition panel with the Direct Selection tool to directly select that shape’s group, regardless of how deeply nested the shape is in the group hierarchy.
• To select a group that is contained within the same group as the group that is already selected, click the group to select.
• To add a group to a selection, Shift-click it. You can combine the Shift key with double-clicking and with the Direct Selection tool to add more deeply nested groups to the selection.

Select paths and vertices in path-editing mode

• To select a vertex, click the vertex with the Selection tool. To add vertices to the selection, Shift-click them.
• To select a path segment, click the segment with the Selection tool. To add segments to the selection, Shift-click them.
• To select an entire path, Alt-click (Windows) or Option-click (Mac OS) a segment or vertex of the path with the Selection tool, or select any portion of the path and press Ctrl+A (Windows) or Command+A (Mac OS).
• To select vertices by dragging, select a path or portion of a path to enter path-editing mode, and then drag with the marquee-selection tool to draw a marquee-selection box around the vertices to select. To add vertices to the selection, hold down the Shift key as you draw additional marquee-selection boxes.

Select all points on a path and enter free-transform mode

• Double-click a path segment while in path-editing mode or in group selection mode for a single shape.
• Select the Path property in the Timeline panel and press Ctrl+T (Windows) or Command+T (Mac OS).

Select masks, segments, and vertices

Unlike layers, masks can have more than one level of selection. You can select a mask as a whole path, which is appropriate when you want to move or resize a mask. However, if you want to change the path of a mask, select one or more points on it. Selected points appear solid, and unselected points appear hollow.

Select or deselect masks in the Layer or Composition panel

• To select a vertex on a mask, click the vertex with the Selection tool . To add vertices to the selection, Shift-click them.
• To select a mask segment, click the segment with the Selection tool. To add segments to the selection, Shift-click them.
• To select an entire mask, Alt-click (Windows) or Option-click (Mac OS) a segment, vertex, or handle of a mask with the Selection tool, or select any portion of the mask and choose Edit > Select All or press Ctrl+A (Windows) or Command+A (Mac OS). To add masks to the selection, Alt+Shift-click (Windows) or Option+Shift-click (Mac OS) them.
• To select masks by dragging, select a mask or portion of a mask to enter mask editing mode and then drag with the Selection tool to draw a marquee-selection box completely around the vertices or masks that you want to select. To add masks or vertices to the selection, hold down the Shift key as you draw additional marquee-selection boxes.
• To select all masks on a layer, select a mask on the layer, and choose Edit > Select All or press Ctrl+A (Windows) or Command+A (Mac OS).
• To deselect all masks, press Ctrl+Shift+A (Windows) or Command+Shift+A (Mac OS).
• To select an adjacent mask on a layer, press Alt+accent grave (`) (Windows) or Option+accent grave (`) (Mac OS) to select the next mask, or Shift+Alt+accent grave (`) (Windows) or Shift+Option+accent grave (`) (Mac OS) to select the previous mask.
• To deselect a mask, click anywhere other than on the mask.
• To remove a vertex or segment from a selection, Shift-click the vertex or segment.

Select masks in the Timeline panel

Lock or unlock masks

Locking a mask prevents you from selecting it in the Timeline, Composition, and Layer panels or setting it as a target in the Layer panel. Use this feature to avoid unwanted changes to the mask.

Move vertices in free-transform mode

You can scale and rotate an entire mask or shape path (or selected vertices in one or more paths) using the Free Transform Points command. When you use this command, a free-transform bounding box surrounds the selected vertices, and an anchor point appears in the center of the bounding box to mark the anchor point for the current transformation. You can scale and rotate the selected vertices by dragging the bounding box or its handles. You can also change the reference point from which the vertices are rotated or scaled by moving the bounding box anchor point. The free-transform bounding box handles and anchor point exist independently of the handles and anchor point for the layer.

Convert a path between manual Bezier and RotoBezier

You can convert any manual Bezier mask path or manual Bezier shape path to a RotoBezier path. If the manual Bezier path has direction handles that have been adjusted, this conversion changes the shape of the path, because After Effects calculates the curvature of RotoBezier segments automatically.

The conversion of a RotoBezier path to a manual Bezier path does not change the shape of the path.

Modify a Bezier mask path or shape path

You can change a Bezier mask path using the Selection tool and pen tools in the Layer or Composition panel. You can change a Bezier shape path using the Selection tool and pen tools in the Composition panel.

The pen tools—Add Vertex, Delete Vertex, and Convert Vertex tools—are grouped with the Pen tool in the Tools panel. To reveal these tools in the Tools panel, click and hold the Pen tool in the Tools panel.

When modifying a path, make sure that you click only existing vertices or segments; otherwise, you may create a new path instead.

Move, add, or delete a vertex

Adjust a path segment

Toggle a vertex between a smooth point and a corner point

Adjust the tension of a RotoBezier mask

Change a mask path numerically

Designate the first vertex for a Bezier path

To animate a path, After Effects designates the topmost vertex at the initial keyframe as the first vertex and numbers each successive vertex in ascending order from the first vertex. After Effects then assigns the same numbers to the corresponding vertices at all successive keyframes. After Effects interpolates the movement of each vertex from its initial position at one keyframe to the position of the correspondingly numbered vertex at the next keyframe. At any time during an animation, you can designate another vertex as the first vertex; this causes After Effects to renumber the vertices of the path. Renumbering vertices causes path animation to change, because After Effects then maps the new vertex numbers to the corresponding old vertex numbers still saved at successive keyframes.

When copying a closed path into a motion path, the vertex designated as the first vertex of the closed path is used as the beginning of the motion path. All motion paths are open paths.

Some shape path operations, such as Trim Paths, also use the first vertex as input to determine how to modify the path.

Animate a mask path with Smart Mask Interpolation

Smart Mask Interpolation provides a high level of control for creating mask path keyframes and smooth, realistic animation. After you select the mask path keyframes to interpolate, Smart Mask Interpolation creates intermediate keyframes based on settings you provide. The Info panel displays the progress of the interpolation and the number of keyframes created.

Move a mask or pan a layer behind a mask

You can adjust the area that is visible through a mask by either moving the mask in the Layer or Composition panel or panning (moving) the layer behind the mask in the Composition panel. When you move a mask, the Position values of the masked layer remain constant, and the mask moves in relation to other objects in the Composition panel.

When you use the Pan Behind (Anchor Point) tool to pan a layer behind a mask, the position of the mask remains constant in the Composition panel but changes in the Layer panel. The Position values of the masked layer change in relation to the composition. As you pan past the edges of the layer frame, the Mask Path values on the layer also change. Using the Pan Behind (Anchor Point) tool saves steps; without it, you would have to change the Position and Mask Path properties of the masked layer manually. You can animate a layer panning behind another layer by setting keyframes for the Position and Mask Path properties of the masked layer.

Move a mask

Pan a layer behind its mask

Face tracking and mask tracking

You can use the mask tracker to transform a mask so that it follows the motion of an object (or objects) in a movie. For more information, see Mask Tracking.

You can also use the mask tracker to accurately detect and track human faces. For more information, see Face Tracking.

Expression access to path points on masks, Bezier shapes, and brush strokes

You can use expressions to read and write the x and y coordinates of path points, or vertices for:

• Layer masks
• Bezier shapes   
• Brush strokes on the Paint and Roto Brush & Refine Edge effects.

The expression methods are similar to accessing path vertices via scripting. The expression method is named points() instead of vertices() for familiarity.

Access expression methods

You can access the expression methods using the following steps:

Listed below are example expression methods of the following path objects:

• Layer mask paths: layer("Dark Gray Solid 1").mask("Mask 1").path or .maskPath
• Bezier shape paths: layer("Shape Layer 1").content("Shape 1").content("Path 1").path
• Paint effect stroke paths: layer("Dark Gray Solid 1").effect("Paint").stroke("Brush 1").path
• Roto Brush & Refine Edge effect stroke paths: layer("Dark Gray Solid 1").effect("Roto Brush & Refine Edge").stroke("Foreground 1").path

Tips to work with these expression methods:

• Point and tangent values are returned as an array of [x,y] number pair arrays for the coordinates of the vertices. Coordinate values are rounded to four decimal places.
• Point and tangent arrays start with the first vertex on the path. You can change the first vertex of layer mask paths and Bezier shape paths using the Layer > Mask and Shape Path > Set First Vertex command.
• Coordinates for layer mask path points are relative to the layer's origin in its upper-left hand corner.
• Coordinates for Bezier shape path points are are relative to the anchor point of the path's shape group (ex., "Transform: Shape 1 > Anchor Point").
• Coordinates for brush stroke path points are relative to the start of the stroke; the first point is [0,0].
• The toComp() method, or other layer space transform methods, can be useful to convert coordinates from different layers into a common space or simply return the coordinates relative to the composition space.
• Because Bezier shape paths are relative to the anchor point of their local shape group and not the layer, their values may be offset from where you expect them to be. To work around this, set all Transform properties in the shape group to zero. Keep in mind that converting a parametric shape to a Bezier shape may affect the shape group's position.
• The createPath() method can be passed points, tangents, and isClosed methods of the same path or other paths. You can pass these methods unchanged to duplicate a path, or transform the point and tangent values before passing them to modify or animate the path.

Create Nulls from Paths panel

Create Nulls From Paths  scriptUI panel allows you to create intuitive animations driven by expressions, without manually writing expressions. You can directly link different paths to shapes, shapes to solids, etc. The Create Nulls From Paths panel uses expression access to path points to automatically link the nulls, so that you don't have to write the expression yourself.

The panel creates nulls for each path point on a mask path or Bezier shape path. You can use the following buttons in the panel to direct how the nulls perform:

• Points Follow Nulls: The button creates nulls that control the position of path points. The path points are attached to nulls and you can animate the nulls to drive the path points.
• Nulls Follow Points: The button creates nulls that are controlled by the position of path points. The nulls are attached to path points and you can animate the path points to move the nulls.
• Trace Path: The button creates a single null that follows the path. The position of the null is linked to the coordinates of the path. The rotation of null auto-orients to follow the path. Trace Path controls the movement of the null along the path and whether it loops. By default, keyframes are set to trace the path in 1 second, and loop is enabled.

Work with the panel