About Masks

There are two types of masks that you can apply to a layer:

• Layer masks are composed of grayscale pixels that you edit with the painting or Selection tools (FIGURE 7.1).

  

• Vector masks are resolution independent and are created with the Pen or Shape tools. You cannot edit vector masks with the painting tools, but they are useful anytime you want sharp, defined edges (FIGURE 7.2).

  

For the majority of your masking needs, a layer mask is what’s called for, because it uses pixels. They are best for organic subjects such as hair, fur, or anything where the hardness of the edge varies around the subject border. Layer masks are also the more flexible of the mask types, because they can be edited with Brush tools.

When it comes to man-made objects or subjects with defined outlines that are mainly straight lines and graceful curves, it may be preferable to use the close cousin of the layer mask: a vector mask. The typical workflow for these is to create a pen path around the subject and then convert the path to a vector mask.

Both layer and vector masks appear to the right of the layer thumbnail in the Layers panel.

Adding Layer Masks

You can create a layer mask from a selection or by painting. Most often, you will make a mask from a selection, and then refine it with the painting tools. A “typical” mask looks like a stencil, with the white of the subject area surrounded by the black of the unselected areas.

To make a layer mask from a selection:

1. Make your initial selection (FIGURE 7.3).

   

2. On the Layers panel, click the image layer, layer group, or Smart Object to which you want to add the mask.

3. Do one of the following:

   • Click the Add Layer Mask button () at the bottom of the Layers panel.

   • Choose Layer > Layer Mask > Reveal Selection (FIGURE 7.4).

     

Depending on the image, you might want to hide rather than reveal the selected area.

To make a mask by hiding the selection:

1. Make your initial selection (FIGURE 7.5).

   

2. Alt/Option-click the Add Layer Mask button, or choose Layer > Layer Mask > Hide Selection (FIGURE 7.6). This is equivalent to choosing Select > Inverse before adding a layer mask.

   

Editing Layer Masks

You can continuously edit a layer mask to add to or subtract from the masked areas. Paint in black to hide; paint in white to reveal. The areas you paint in shades of gray appear in various levels of transparency—the lighter the gray the more of the layer is revealed and vice versa.

First, however, make sure you have the proper layer selected. When a layer mask thumbnail is selected, it has a small border that’s easy to miss (FIGURE 7.7).

If in doubt, you can confirm you have a mask selected by looking in the Properties panel and the document title tab (FIGURE 7.8).

To edit a layer mask:

1. On the Layers panel, click the Mask thumbnail of the layer you want to edit. A border appears around the layer mask thumbnail. In addition, the foreground and background colors become grayscale when the layer mask is active.

2. Select any of the editing or painting tools.

3. Paint on the mask in black to hide portions of the layer, paint in white to reveal portions of the layer (FIGURE 7.9). To partially reveal the layer, paint the mask with a shade of gray—a lighter gray reveals more, a darker gray hides more.

   

Painting a Layer Mask

Although a mask typically starts with a selection, it doesn’t have to. Alternatively, you can paint the mask from scratch.

To paint a layer mask:

1. Select the layer, layer group, or Smart Object you want to mask.

2. With no selection active, Click the Add Layer Mask button () (FIGURE 7.10).

   

3. With black as the foreground color, paint on the mask to conceal parts of the layer. Switch to white to paint back any areas that you want to restore (FIGURE 7.11).

   

4. Vary the brush size as necessary (FIGURE 7.12) with the [ and ] keys. Change the opacity of the brush with the number keys to paint in gray and partially reveal or conceal the layer.

   

To add to and subtract from a mask:

1. Reset the foreground and background colors to black and white by pressing D or clicking the small black/white swatches ().

2. Toggle between your foreground and background colors by pressing X or clicking the toggle switch () (FIGURE 7.13).

   

3. Paint on your mask layer with black to conceal an area, or touch up an area with white to reveal more.

4. If you get confused and paint in black when you should have painted in white or vice versa, just undo the step, press X to switch colors, and continue—layer masks are completely nondestructive.

Vector Masks

Vector masks are like layer masks, but have crisp, vector edges. If you have a manufactured subject requiring a crisp cut out, then a vector mask is the way to go. You can’t vary the softness/hardness of the mask edge so that certain parts are harder or softer than others, nor can you paint on a vector mask with a brush or gradient. You can soften the edge uniformly using the Feather slider on the Properties panel.

If using the Pen or shape tools to make a vector mask, make sure to choose Path as the tool mode on the Options bar, otherwise you’ll be making shape layers.

To add a vector mask:

1. Select the layer you want to mask.

2. Create a closed path with a Pen tool or Shape tool in Path mode (FIGURE 7.14).

   

3. Ctrl/Command-click the Add Mask icon at the bottom of the Layer panel. The first time you click creates a layer mask; the second time creates a vector mask. (A single click is enough if the layer already has a layer mask.) Alternatively, choose Layer > Vector Mask > Current Path.

To convert a custom shape to a vector mask:

1. With the Custom Shape tool, draw a shape layer (FIGURE 7.15). (If you are already in Path mode, skip to step 4.)

   

2. Select the shape layer with the Path Selection tool, and choose Edit > Cut (Ctrl/Command+X).

3. Select the layer that you want to mask, and choose Edit > Paste (Ctrl/Command+V). The shape will be pasted as a path to the layer (FIGURE 7.16).

   

4. Ctrl/Command-click the Add Mask icon at the bottom of the Layers panel (or click the icon twice) to convert the shape to a vector mask.

5. (Optional) Unlink the vector mask and the layer, so that you can move the layer and vector mask independently to experiment with how the shape crops the image.

6. Add a solid color fill layer below (FIGURE 7.17).

   

To edit a vector mask:

1. On the Layers panel, click the Mask thumbnail of the layer you want to edit. If the layer has both a layer mask and a vector mask, select the vector mask, which is the second mask. A border appears around the vector mask thumbnail. In addition, a blue path line appears along the mask edge.

2. Choose the Direct Selection tool (A), which shares the same toolspace as the Path Selection tool (FIGURE 7.18).

   

3. Click the path edge to activate the path anchor points. Click and drag either from the anchor points or the path segments connecting the anchor points to adjust the shape of the vector mask (FIGURE 7.19).

   

Creating a Simple Composition with Masks

Here’s a simple twist: Masks don’t have to be made from the layer to which they are attached.

To use selections from one layer as a layer mask for another:

1. Make a selection on one layer, even a rough one with the Quick Selection tool, as in FIGURE 7.25.

   

2. In the Layers panel, select a different layer to mask, and click the Add Layer Mask button (FIGURE 7.26).

   

3. Double-click the Layer Mask icon to bring up the Properties panel to fine-tune the Density and Feather settings as needed (FIGURE 7.27).

   

4. Use a vector mask to hide the white areas around and inside the frame. To create a vector mask, choose the Rectangle shape tool and change the tool mode to Path (FIGURE 7.28).

   

5. From the Path Operations menu on the Options bar, choose Exclude Overlapping Shapes (FIGURE 7.29).

   

6. Draw an exterior rectangle around the frame. Start with a rectangle that is close in size, then from its corner handles adjust the size and rotation of the rectangle to match the frame. Repeat this to draw an interior rectangle (FIGURE 7.30).

   

7. Hold Ctrl/Command and click Add Mask to convert the vector path into a vector mask (FIGURE 7.31).

   

Layer Masks and Adjustment Layers

Every time you add an adjustment layer, (see Chapter 9) you get a blank layer mask thrown in. If you have an active selection when you choose the adjustment layer, the non-selected areas will be black on the resulting layer mask. If you add the adjustment layer without an active selection, the layer mask attached to it will start out white, and so it has no effect—until you paint on it in black or gray to limit the adjustment to specific parts of the image.

To use a layer mask with an adjustment layer:

1. Make a selection of the part of the image you want to adjust, and choose Add Layer Mask (FIGURE 7.32). Depending on the result you’re after, you may want to inverse the selection (Ctrl+Shift+I/Command+Shift+I).

   

2. Choose an adjustment layer from the menu at the bottom of the Layers panel ().

3. If necessary, choose a painting tool and paint on the layer mask in black or white to refine the mask.

Gradient Layer Masks

You can use a gradient layer mask to achieve a subtle transition from one image to another.

To apply a gradient mask:

1. Create a document with two or more image layers (FIGURE 7.33).

   

2. Add a layer mask to the top layer by selecting the layer and clicking the Add Layer Mask icon ().

3. If necessary, restore the foreground colors to black and white by pressing X.

4. Select the Gradient tool. From the gradient picker, choose the Foreground To Transparent gradient and a linear gradient (FIGURE 7.34).

   

5. Making sure the layer mask is selected, make multiple swipes of the Gradient tool (from right to left in this case) to build up the gradient layer mask cumulatively (FIGURE 7.35).

   

You can also use a gradient layer mask in combination with adjustment layers to achieve a seamless transition from an adjusted to a non-adjusted portion of the image. This is particularly useful when you want to fix exposure in an image by darkening or lightening one side of the image, leaving the other side unchanged.

To apply a gradient mask to an adjustment layer:

1. Add a Curves adjustment layer (FIGURE 7.36).

   

2. Pull up the curve from the shadow area to increase the exposure (FIGURE 7.37).

   

3. To restore part of the image (in this case, the sky) to how it was before the adjustment, use a Foreground (black) To Transparent gradient on the layer mask that comes with the Curves adjustment: Select the layer mask, and drag from the top about halfway down the image. If necessary, you can make multiple swipes with the Gradient tool, to build the gradient layer mask cumulatively (FIGURE 7.38).

   

Managing Layer Masks

Here are some essential techniques for keeping on top of your layer masks.

While editing, it’s useful to swap back and forth between seeing a layer with and without its mask. Likewise, it’s sometimes easier to understand and edit a layer mask by viewing it in grayscale or as a color overlay on the layer. You can toggle these views as often as necessary.

To disable or enable a layer mask:

1. To see the layer without its mask, Shift-click the mask thumbnail in the Layers panel. A red X appears over the mask thumbnail, and all the layer’s content becomes visible (FIGURE 7.39).

   

2. Shift-click again to enable the mask.

To view a mask as grayscale:

1. Alt/Option-click the layer mask thumbnail to view just the grayscale mask (FIGURE 7.40).

   

2. Alt/Option-click the layer mask thumbnail again to view the layer. Alternatively, click the eye icon on the Properties panel.

To view the mask as a color overlay:

1. Alt/Option+Shift-click the layer mask thumbnail to view the mask as a color overlay. Or, open the Channels panel and click the eyeball next to the layer mask (FIGURE 7.41).

   

2. If you need to change the layer mask color—perhaps for more contrast against the colors in the image—double-click the layer mask channel in the Channels panel and choose a new mask color in the Layer Mask Display Options dialog.

To apply or delete a layer mask:

1. To permanently apply a layer mask, right-click the layer mask thumbnail, then choose Apply Layer Mask. But, be careful: any pixels made fully or partially transparent by the layer mask are permanently deleted, and can’t be recovered except through Undo.

2. To delete a layer mask without applying the changes, right-click the layer mask thumbnail and choose Delete Layer Mask (FIGURE 7.42).

   

Linked layers and masks

By default, a layer or group is linked to its layer mask or vector mask—you’ll see a link icon between the thumbnails in the Layers panel (FIGURE 7.44). When you move or transform (scale, rotate, etc.) either the layer or the mask, both layer and mask are moved or transformed together (FIGURE 7.45). Unlinking the layer from its mask lets you move the mask’s boundaries separately from the layer (FIGURE 7.46).

To unlink layers and masks:

• Click the link icon to link/unlink the layer and its mask.

Masks on Smart Objects

There are pros and cons to applying layer masks to the contents of a Smart Object, rather than to the Smart Object itself. A mask applied to a Smart Object isn’t protected from destructive editing (FIGURE 7.49) the way the contents of the Smart Object are. If you apply repeated transformations, the mask may be degraded. On the other hand, when you apply the mask to the Smart Object, rather than to its content, the mask is visible in the Layers panel, making this approach a more flexible way of working (FIGURE 7.50). This is especially true if the Smart Object contains multiple layers; a mask on the Smart Object masks everything in it. (For more on Smart Objects, see Chapter 14.)

If you want to make a Smart Object of a layer with a layer mask applied, but want to continue working on that layer mask, do the following.

To add a mask to a Smart Object:

1. Activate the selection from the layer mask by Ctrl/Command-clicking the layer mask thumbnail.

2. Right-click the layer mask thumbnail and choose Delete Layer Mask to delete the layer mask without applying it.

3. Right-click the layer thumbnail and choose Convert To Smart Object.

4. Convert the active selection into a layer mask for the Smart Object by clicking the Add Mask icon.

Luminosity Masks

A luminosity selection is proportional to the luminosity in the image. The lighter an area is, the more selected it will be. When you create a layer mask from such a selection, it will be a grayscale version of the image, “hiding” the darker parts and revealing the lighter parts. Amongst other things, this is useful for combining texture with an image.

To make a luminosity mask:

1. Create a document of two of more layers.

2. Select the top layer—in this case the texture—from which you will create the mask (FIGURE 7.51).

   

3. Go to the Channels panel, and Ctrl/Command-click the RGB composite channel to make the selection of the gray values of the image. You will see a marching ants selection on the image (FIGURE 7.52).

   

4. Return to the Layers panel, hide visibility of the texture layer, and select the layer beneath (FIGURE 7.53).

   

5. With the selection still active, click Add Layer Mask (FIGURE 7.54).

   

6. Create a solid color fill layer at the bottom of the layer stack (FIGURE 7.55).

   

7. (Optional) Select the Layer Mask and press Ctrl/Command+L to bring up the Levels dialog. Move the midpoint slider left to reduce the contrast (revealing more of the image) or right to increase the contrast (concealing more of the image) (FIGURE 7.56).

   

Combining Layer Masks and Vector Masks

If you have multiple layers with layer masks applied to them, you can mask the cumulative effect of those layers with a vector mask.

To mask more than one layer with a vector mask:

1. Create a document of two layers, and apply a gradient layer mask to the top layer, revealing part of the layer below (FIGURE 7.57).

   

2. Select the layers and press Ctrl/Command+G to put them in a group.

3. Click the newly created group to make it active, and then Ctrl/Command-click the Add Mask icon (FIGURE 7.58).

   

4. With the Ellipse tool drawing in Path mode, draw a circle on the vector mask (FIGURE 7.59).

   

It’s not often that you need to do so, but you can have both a layer mask and a vector mask attached to a single layer. (Just click the Layer Mask icon twice.)

Or, if you want to add different qualities of mask edge to different parts of the image, you can copy the layer, adding a layer mask to one copy and a vector mask to the other.

To apply a vector and a layer mask to copies of the same layer:

1. Add a vector mask (in FIGURE 7.60, a circle) to the layer.

   

2. Press Ctrl/Command+J to duplicate the layer (FIGURE 7.61).

   

3. Delete the vector mask from the top layer, and replace it with a layer mask (around the shape of the leaf) (FIGURE 7.62).

   

4. Adjust the feathering of the layer mask as necessary using the Properties panel.