7. Working with Masks
Lesson overview
In this lesson, you’ll learn how to do the following:
Create a mask using the Pen tool.
Change a mask’s mode.
Edit a mask shape by controlling vertices and direction handles.
Feather a mask edge.
Replace the contents of a mask shape.
Adjust the position of a layer in 3D space to blend it with the rest of a shot.
Create a shadow effect.
Create a vignette.
This lesson will take about an hour to complete. If you haven’t already done so, download the project files for this lesson from adobepress.com/AfterEffectsCIB2022, following the instructions in the Getting Started section under “Accessing the lesson files and Web Edition.”
PROJECT: SEQUENCE FROM A COMMERCIAL
There will be times when you won’t need (or want) everything in a shot to be included in the final composite. Use masks to control what appears.
About masks
A mask in Adobe After Effects is a path, or outline, that is used to modify layer effects and properties. The most common use of masks is to modify a layer’s alpha channel. A mask consists of segments and vertices: Segments are the lines or curves that connect vertices. Vertices define where each segment of a path starts and ends.
A mask can be either an open or a closed path. An open path has a beginning point that is not the same as its end point; for example, a straight line is an open path. A closed path is continuous and has no beginning or end, such as a circle. Closed-path masks can create transparent areas for a layer. Open paths cannot create transparent areas for a layer, but are useful as parameters for an effect. For example, you can use an effect to generate a running light around a mask.
A mask belongs to a specific layer. Each layer can contain multiple masks.
You can draw masks in common geometric shapes—including polygons, ellipses, and stars—with the shape tools, or you can use the Pen tool to draw an arbitrary path. If you draw a shape when a layer is selected, the shape becomes a mask for that layer; the cursor displays a masked box (
) as you start to draw. If you draw a shape when no layer is selected, After Effects creates a shape layer; the cursor displays a star (
)as you start to draw.
Getting started
In this lesson, you will create a mask for the screen of a tablet and replace the screen’s original content with a movie. Then you’ll adjust the positioning of the new footage so that it fits the perspective of the shot. Finally, you’ll polish the scene by adding a shadow and creating a vignette effect.
Begin by previewing the movie and setting up the project.
Make sure the following files are in the Lessons/Lesson07 folder on your hard disk, or download them from adobepress.com now:
In the Assets folder: SeaTurtle.mov, TabletVideo.mov
In the Sample_Movie folder: Lesson07.mp4
Open and play the Lesson07.mp4 sample movie in Windows Movies & TV or QuickTime Player to see what you will create in this lesson. When you are done, close Windows Movies & TV or QuickTime Player. You may delete the sample movie from your hard disk if you have limited storage space.
When you begin the lesson, restore the default application settings for After Effects. See “Restoring default preferences” on page 3.
Start After Effects, and then immediately hold down Ctrl+Alt+Shift (Windows) or Command+Option+Shift (macOS) to restore default preferences settings. When prompted, click OK to delete your preferences. Click New Project in the Home window.
After Effects opens to display a new, untitled project.
Choose File > Save As > Save As, and navigate to the Lessons/Lesson07/Finished_Project folder.
Name the project Lesson07_Finished.aep, and then click Save.
Creating the composition
You’ll import two footage items for this exercise. Then, you’ll create the composition based on the aspect ratio and duration of one of the footage items.
Double-click an empty area of the Project panel to open the Import File dialog box.
Navigate to the Lessons/Lesson07/Assets folder, Shift-click to select the SeaTurtle.mov and TabletVideo.mov files, and then click Import or Open.
Deselect both files in the Project panel. Then select the TabletVideo.mov footage item, and drag it onto the Create A New Composition button (
) at the bottom of the Project panel.
After Effects creates a composition named TabletVideo and opens it in the Composition and Timeline panels.
Choose File > Save to save your work so far.
Creating a mask with the Pen tool
The TV screen is currently blank. To fill it with the video of a sea turtle, you need to mask the screen.
Tip
You can also create a mask using the Mocha effect. To learn more about using the Mocha effect, see After Effects Help.
Go to 2:00, where the video is in full color.
Make sure the TabletVideo.mov layer is selected in the Timeline panel, and then select the Pen tool (
) in the Tools panel.
The Pen tool creates straight lines or curved segments. The tablet screen itself is rectangular, so you’ll try using straight lines first.
Click the lower left corner of the tablet screen to place the first vertex.
Click the upper left corner of the tablet screen to place the second vertex. After Effects connects the two points with a segment.
Click to place a third vertex in the upper right corner of the tablet screen, and then click to place a fourth vertex where the fabric intersects with the right edge of the screen.
Move the Pen tool over the first vertex you created (in the lower left corner). When a circle appears next to the pointer (as in the image on the right below), click to close the mask path.
Editing a mask
Instead of masking the information inside the tablet, the mask has removed everything outside the tablet. You need to invert the mask. You’ll also use Bezier curves to create a more accurate mask.
Inverting the mask
For this project, you need everything inside the mask to be transparent and everything outside the mask to be opaque. You’ll invert the mask now.
Select the TabletVideo.mov layer in the Timeline panel, and press the M key to see the Mask Path property for the mask.
TipPressing the M key twice in quick succession displays all mask properties for the selected layer.
There are two ways to invert this mask: by choosing Subtract from the Mask Mode pop-up menu, or by selecting the Inverted option.
Select the Inverted option for Mask 1.
The mask inverts.
Press F2 or click an empty area of the Timeline panel to deselect the TabletVideo.mov layer.
Creating curved masks
Curved or freeform masks use Bezier curves to define the shape of the mask. Bezier curves give you the greatest control over the shape of the mask. With them, you can create straight lines with sharp angles, perfectly smooth curves, or a combination of the two.
You’ll use Bezier curves to adjust the mask’s edges around the fabric, red ball, and sea urchin shell that obscure part of the screen.
In the Timeline panel, select Mask 1, the mask for the TabletVideo.mov layer. Selecting Mask 1 makes the mask active and also selects all the vertices.
In the Tools panel, select the Add Vertex tool (
), which is hidden behind the Pen tool.
With the Add Vertex tool, click along the bottom edge of the mask to add a vertex.
Drag the new vertex down to the point where the red ball intersects with the screen.
Add another vertex next to the fabric, and drag it down to the point where the fabric and red ball intersect.
Add vertices at the two points where the sea urchin shell intersects the bottom edge of the screen.
In the Tools panel, select the Convert Vertex tool (
), which is hidden behind the Add Vertex tool.In the Composition panel, click the vertex where the fabric and red ball intersect. The Convert Vertex tool changes the corner vertex to smooth points.
Adjust the direction handles until the mask follows the contour of the fabric and the red ball.
Finally, adjust the direction handles for the vertices at the points where the sea urchin intersects with the bottom edge of the screen.
Deselect the TabletVideo.mov layer in the Timeline panel to check the edge of your mask. If you need to make adjustments, select Mask 1 in the TabletVideo.mov layer, and use the Convert Vertex tool to fine-tune the shape of the mask.
Choose File > Save to save your work.
Feathering the edges of a mask
The mask looks good, but the edge of the screen looks a little too sharp and unrealistic. To address this, you’ll feather, or soften, the edges.
Select the TabletVideo.mov layer in the Timeline panel, and press the F key to display the Mask Feather property for the mask.
Increase the Mask Feather amount to 3.0, 3.0 pixels.
Hide the properties for the TabletVideo.mov layer, and then choose File > Save to save your work.
Replacing the content of the mask
You are now ready to replace the background with the video of a sea turtle and blend it with the overall shot.
Tip
In effects that reference layers, you can use masks for greater control over the effect’s influence. You can also use expressions to animate mask points.
In the Project panel, select the SeaTurtle.mov file and drag it to the Timeline panel, placing it below the TabletVideo.mov layer.
Choose Fit Up To 100% from the Magnification Ratio pop-up menu at the bottom of the Composition panel so that you can see the whole composition.
NoteIf you’re using a Mac with Retina display, you’ll see Fit Up To 200%.
Select the Selection tool (
), and then drag the SeaTurtle.mov layer in the Composition panel slightly to the left, centering it in the tablet screen.
Go to 2:07. In the Timeline panel, drag the entire SeaTurtle.mov layer to the right so that it begins at 2:07 and ends at the same time as the TabletVideo.mov layer.
Repositioning and resizing the clip
The sea turtle clip is a little too big for the tablet screen and it’s not quite the right angle. You’ll convert it to a 3D layer, which will give you more control over its shape and size.
Still at 2:07, select the SeaTurtle.mov layer.
Press S to reveal the Scale property, and then change the Scale to 90%. Press Shift+R to reveal the Rotation property.
With the SeaTurtle.mov layer selected in the Timeline panel, click the 3D switch (
) for the layer.
NoteYou’ll learn more about 3D layers in Lessons 12, “Using 3D Features,” and 13, “Working with the 3D Camera Tracker.”
The 3D Transformation Gizmo appears in the Composition panel, and the Timeline panel includes the Orientation property and multiple Rotation properties. Most properties in the Timeline panel now have three values: From left to right, they represent the x, y, and z axes of the image. The z axis controls the depth of the layer. These axes are represented in the 3D Transformation Gizmo.
Make sure the Selection tool is selected, and then use the 3D Transformation Gizmo in the Composition panel to adjust the position of the video in the screen.
Drag the red arrow to move the layer horizontally, the green arrow to move it vertically, and the blue arrow to change the depth. Drag the ball along the curved red line to rotate the layer around the x axis; drag the green ball along the curved green line to rotate it around the y axis; and drag the blue ball along the curved blue line to rotate it around the z axis.
Make any additional adjustments, including changes to the Rotation values, so that your composition resembles the following image. Your values may not match ours, depending on how you shifted the SeaTurtle layer earlier.
TipTo specify whether the 3D Transformatiom Gizmo affects Rotation or Orientation values, select the Rotation tool in the Tools panel and choose an option from the Set menu. You can also type values directly in the Timeline panel instead of dragging in the Composition panel.
Hide the properties for the SeaTurtle.mov layer, and then choose File > Save to save your work.
Adjusting the opacity
The video appears abruptly. You’ll smooth its appearance by adjusting the opacity at the beginning.
Make sure you are still at 2:07 in the time ruler and the SeaTurtle.mov layer is selected. Then, press T to reveal the Opacity property.
Change the Opacity to 0%, and then click the stopwatch icon (
) to create an initial keyframe.Go to 3:15, and change the Opacity to 100%.
Hide the Opacity property, and deselect all layers.
Press the Home key or move the current-time indicator to the beginning of the time ruler, and then press the spacebar to preview your work. Press the spacebar again when you’re done.
Adding a shadow
The masked image looks convincing, but you can make it look even more realistic by adding a shadow to the screen.
Click an empty area in the Timeline panel to deselect all layers, and then choose Layer > New > Solid.
In the Solid Settings dialog box, name the layer Shadow, click the Make Comp Size button, choose a dark gray color ( we used R=34, G=34, B=34,), and then click OK.
Instead of trying to exactly re-create the shape of the TabletVideo.mov layer’s mask, it’s easier to copy it to the Shadow layer and adjust it.
Press the Home key or move the current-time indicator to the beginning of the time ruler.
Select the TabletVideo.mov layer in the Timeline panel, and press the M key to display the Mask Path property for the mask.
Select Mask 1, and then choose Edit > Copy or press Ctrl+C (Windows) or Command+C (macOS).
Select the Shadow layer in the Timeline panel, and then choose Edit > Paste or press Ctrl+V (Windows) or Command+V (macOS).
Hide the properties for the TabletVideo.mov layer.
This time, you want to keep the area inside the mask opaque and make the area outside the mask transparent.
Select the Shadow layer, and press the F key to reveal the Mask Feather property for the mask.
Change the Mask Feather value to 0 (zero).
Deselect the Inverted option. The Shadow layer now obscures the SeaTurtle.mov layer.
Adjusting the shape of the mask
The neighboring objects should cast a shadow, but it shouldn’t affect the entire screen. You’ll use the Pen tools to reduce the mask to cover only the lower right section of the screen.
Select the Delete Vertex tool (
), hidden beneath the Convert Vertex tool in the Tools panel.Click the upper left vertex to delete it.
Select the Selection tool (
) in the Tools panel, and then drag the lower left and upper right vertices closer to the lower right corner of the screen, as in the following image.
Select the Add Vertex tool (
), hidden beneath the Delete Vertex tool in the Tools panel.Add vertices to the upper path (currently a diagonal line), and then use the Convert Vertex (
) and Selection tools to position and shape the path into a curve similar to the following images.
Feathering the edges of the shadow
Shadows rarely have sharp edges. You’ll adjust the feather for the mask so that the shadow fades into the rest of the screen.
Go to 3:15.
Select the Mask Feather tool (
), hidden beneath the Convert Vertex tool () in the Tools panel.You used the Mask Feather property to adjusts the width of the feather the same amount all the way around the original mask. However, the Mask Feather tool gives you more flexibility, letting you vary the feathering width at points you define on the closed mask.
In the Timeline panel, select the Shadow layer. Then, in the Composition panel, click the lower left vertex to create a feather point.
Click the feather point again, and without releasing the mouse button, drag that feather point outward to increase the feathering margin for the entire mask.
Currently the feather, represented by dashed lines, extends evenly across the mask. You’ll add more feather points for greater flexibility.
Click the vertex on the right side of the sea urchin.
A feathering point is created, and the feather shrinks back to the mask edge on the lower and right sides of the screen, but the feathering remains on the upper side of the mask. The objects that are casting the shadow shouldn’t be shadowed.
Click the dashed line just under the vertex on the left side of the sea urchin, and drag the feathering boundary up to the screen.
(Optional) Adjust the feathering elsewhere by creating a feather point on the boundary and dragging it.
Now the shadow doesn’t appear on any of the objects.
The shape of the shadow is good, but it’s obscuring the image. You’ll change the opacity to reduce its effect.
Select the Shadow layer in the Timeline panel, and then press the T key to reveal the Opacity property. Change its value to 20%.
Press T to hide the Opacity property, and then press F2 or click an empty area in the Timeline panel to deselect all layers.
Applying a blending mode
To create unique interactions between layers, you may want to experiment with blending modes. Blending modes control how each layer blends with, or reacts to, layers beneath it. Blending modes for layers in After Effects are identical to blending modes in Adobe Photoshop.
From the Timeline panel menu, choose Columns > Modes to display the Mode pop-up menu.
Choose Darken from the Shadow layer’s Mode pop-up menu, and then deselect all layers so you can see the result.
The Darken blending mode enhances the screen image and boosts the colors underneath the Shadow layer.
Choose File > Save to save your work.
Creating a vignette
A popular effect in motion graphic design is to apply a vignette to the composition. This is often done to simulate light variations of a glass lens. It creates an interesting look that focuses the attention on the subject and sets the shot apart.
Click an empty area in the Timeline panel to deselect all layers, and then choose Layer > New > Solid.
In the Solid Settings dialog box, name this layer Warm Vignette, click the Make Comp Size button, use the eyedropper to select a dark brown tone from the metal rim of the vase to the left of the tablet, and then click OK.
In addition to the Pen tool, After Effects provides tools that let you easily create square and elliptical masks.
In the Tools panel, select the Ellipse tool (
), hidden behind the Rectangle tool (
).In the Composition panel, position the cross-hairs pointer in the upper left corner of the image. Drag to the opposite corner to create an elliptical shape that fills the image. Adjust the shape and position using the Selection tool, if necessary.
Expand the Mask 1 property in the Warm Vignette layer to display all of the mask properties for the layer.
Choose Subtract from the Mask 1 Mode pop-up menu.
Increase the Mask Feather amount to 200, 200 pixels.
Your composition should now resemble the following figure.
Even with this large feather amount, the vignette is intense and constricting. You can make the vignette warmer and less overwhelming by adjusting the Mask Expansion property. The Mask Expansion property represents, in pixels, how far from the original mask edge you are expanding or contracting the adjusted edge.
Increase Mask Expansion to 90 pixels.
Finally, you can use a blending mode to make the effect more subtle.
Choose Screen from the Mode pop-up menu for the Warm Vignette layer.
Hide the properties for the Warm Vignette layer, and then choose File > Save.
Deselect all layers. Then, press the spacebar to preview the movie. Press the spacebar again when you’re done.
Review answers
1 In After Effects, a mask is a path, or outline, that is used to modify layer effects and properties. The most common use of masks is to modify a layer’s alpha channel. A mask consists of segments and vertices.
2 A direction handle is used to control the shape and angle of a Bezier curve.
3 An open mask can be used to control effects or the placement of text; it does not define a region of transparency. A closed mask defines a region that will affect the alpha channel of a layer.
4 The Mask Feather tool lets you vary the width of feathering at different points on a mask. With the Mask Feather tool, click to add a Feather point, and then drag it.