Chapter 9

Stenciling

The stencil buffer is an off-screen buffer we can use to achieve special effects. The stencil buffer has the same resolution as the back buffer and depth buffer, such that the ijth pixel in the stencil buffer corresponds with the ijth pixel in the back buffer and depth buffer. Recall from the note in §4.1.5 that when a stencil buffer is specified, the stencil buffer always shares the depth buffer. As the name suggests, the stencil buffer works as a stencil and allows us to block the rendering of certain pixel fragments to the back buffer.

For instance, when implementing a mirror we need to reflect an object across the plane of the mirror; however, we only want to draw the reflection into the mirror. We can use the stencil buffer to block the rendering of the reflection unless it is being drawn into the mirror (see Figure 9.1).

Figure 9.1: (Left) Here we have a crate being reflected without using the stencil buffer. We see that the reflected crate is always reflected regardless of whether the reflection is in the mirror. (Right) By using the stencil buffer, we can block the reflected crate from being rendered unless it is being drawn in the mirror.

The stencil buffer (and also the depth buffer) is controlled via the ID3D10DepthStencilState interface. Like blending, the interface offers a flexible and powerful set of capabilities. Learning to use the stencil buffer effectively comes best by studying existing example applications. Once you understand a few applications of the stencil buffer, you will have a better idea of how it can be used for your own specific needs.

Objectives:

    To find out how to control the depth and stencil buffer settings with the ID3D10DepthStencilState interface.

    To learn how to implement mirrors by using the stencil buffer to prevent reflections from being drawn to non-mirror surfaces.

    To learn how to measure depth complexity using the stencil buffer.

9.1 Depth/Stencil Formats and Clearing

Recalling that the depth/stencil buffer is a texture, it must be created with certain data formats. The formats used for depth/stencil buffering are as follows:

   DXGI_FORMAT_D32_FLOAT_S8X24_UINT: Specifies a 32-bit floating-point depth buffer, with 8 bits (unsigned integer) reserved for the stencil buffer mapped to the [0, 255] range and 24 bits used for padding.

   DXGI_FORMAT_D24_UNORM_S8_UINT: Specifies an unsigned 24-bit depth buffer mapped to the [0, 1] range with 8 bits (unsigned integer) reserved for the stencil buffer mapped to the [0, 255] range.

In our D3DApp framework, when we create the depth buffer we specify:

Also, the stencil buffer should be reset to some value at the beginning of each frame. This is done with the following method (which also clears the depth buffer):

   pDepthStencilView: Pointer to the view of the depth/stencil buffer we want to clear.

   ClearFlags: Specify D3D10_CLEAR_DEPTH to clear the depth buffer only, D3D10_CLEAR_STENCIL to clear the stencil buffer only, or D3D10_CLEAR_DEPTH | D3D10_CLEAR_STENCIL to clear both.

   Depth: The float value to set each pixel in the depth buffer to; it must be a floating-point number x such that 0 ≤ x ≤ 1.

   Stencil: The integer value to set each pixel of the stencil buffer to; it must be an integer n such that 0 ≤ n ≤ 255.

In our D3DApp framework, we call this function in the D3DApp::drawScene method:

9.2 The Stencil Test

As previously stated, we can use the stencil buffer to block rendering to certain areas of the back buffer. The decision to block a particular pixel from being written is decided by the stencil test, which is given by the following:

The stencil test is performed as pixels get rasterized, assuming stenciling is enabled, and takes two operands:

   A left-hand-side (LHS) operand that is determined by ANDing an application-defined stencil reference value (StencilRef) with an application-defined masking value (StencilReadMask).

   A right-hand-side (RHS) operand that is determined by ANDing the entry already in the stencil buffer of the particular pixel we are testing (Value) with an application-defined masking value (StencilReadMask).

The stencil test then compares the LHS with the RHS as specified by an application-chosen comparison function , which returns a true or false value. We write the pixel to the back buffer if the test evaluates to true (assuming the depth test also passes). If the test evaluates to false, then we block the pixel from being written to the back buffer. And of course, if a pixel is rejected due to failing the stencil test, it is not written to the depth buffer either.

The operator is any one of the functions defined in the D3D10_COMPARISON_FUNC enumerated type:

   D3D10_COMPARISON_NEVER: The function always returns false.

   D3D10_COMPARISON_LESS: Replace with the < operator.

   D3D10_COMPARISON_EQUAL: Replace with the = = operator.

   D3D10_COMPARISON_LESS_EQUAL: Replace with the ≤ operator.

   D3D10_COMPARISON_GREATER: Replace with the > operator.

   D3D10_COMPARISON_NOT_EQUAL: Replace with the ! = operator.

   D3D10_COMPARISON_GREATER_EQUAL: Replace with the ≥ operator.

   D3D10_COMPARISON_ALWAYS: The function always returns true.

9.3 The Depth/Stencil State Block

The first step to creating an ID3D10DepthStencilState interface is to fill out a D3D10_DEPTH_STENCIL_DESC instance:

9.3.1 Depth Settings

   DepthEnable: Specify true to enable the depth test; specify false to disable it. When depth testing is disabled, the draw order matters, and a pixel fragment will be drawn even if it is behind an occluding object (review §4.1.5). If the depth test is disabled, elements in the depth buffer are not updated either, regardless of the DepthWriteMask setting.

   DepthWriteMask: This can be either D3D10_DEPTH_WRITE_MASK_ZERO or D3D10_DEPTH_WRITE_MASK_ALL, but not both. Assuming DepthEnable is set to true, D3D10_DEPTH_WRITE_MASK_ZERO disables writes to the depth buffer, but depth testing will still occur. D3D10_DEPTH_WRITE_MASK_ALL enables writes to the depth buffer; new depths will be written provided the depth and stencil test both pass.

   DepthFunc: Specify one of the members of the D3D10_COMPARISON_FUNC enumerated type to define the depth test comparison function. Usually this is always D3D10_COMPARISON_LESS so that the usual depth test is performed, as described in §4.1.5. That is, a pixel fragment is accepted provided its depth value is less than the depth of the previous pixel written to the back buffer. But as you can see, Direct3D allows you to customize the depth test if necessary.

9.3.2 Stencil Settings

   StencilEnable: Specify true to enable the stencil test; specify false to disable it.

   StencilReadMask: The StencilReadMask used in the stencil test:

The default does not mask any bits:

   StencilWriteMask: When the stencil buffer is being updated, we can mask off certain bits from being written to with the write mask. For example, if you wanted to prevent the top four bits from being written to, you could use the write mask of 0x0f. The default value does not mask any bits:

   FrontFace: A filled-out D3D10_DEPTH_STENCILOP_DESC structure indicating how the stencil buffer works for front-facing triangles.

   BackFace: A filled-out D3D10_DEPTH_STENCILOP_DESC structure indicating how the stencil buffer works for back-facing triangles.

   StencilFailOp: A member of the D3D10_STENCIL_OP enumerated type describing how the stencil buffer should be updated when the stencil test fails for a pixel fragment.

   StencilDepthFailOp: A member of the D3D10_STENCIL_OP enumerated type describing how the stencil buffer should be updated when the stencil test passes but the depth test fails for a pixel fragment.

   StencilPassOp: A member of the D3D10_STENCIL_OP enumerated type describing how the stencil buffer should be updated when the stencil test and depth test both pass for a pixel fragment.

   StencilFunc: A member of the D3D10_COMPARISON_FUNC enumerated type to define the stencil test comparison function.

   D3D10_STENCIL_OP_KEEP: Specifies to not change the stencil buffer; that is, keep the value currently there.

   D3D10_STENCIL_OP_ZERO: Specifies to set the stencil buffer entry to 0.

   D3D10_STENCIL_OP_REPLACE: Specifies to replace the stencil buffer entry with the stencil-reference value (StencilRef) used in the stencil test. Note that the StencilRef value is set when we bind the depth/stencil state block to the rendering pipeline (§9.3.3).

   D3D10_STENCIL_OP_INCR_SAT: Specifies to increment the stencil buffer entry. If the incremented value exceeds the maximum value (e.g., 255 for an 8-bit stencil buffer), then we clamp the entry to that maximum.

   D3D10_STENCIL_OP_DECR_SAT: Specifies to decrement the stencil buffer entry. If the decremented value is less than 0, then we clamp the entry to 0.

   D3D10_STENCIL_OP_INVERT: Specifies to invert the bits of the stencil buffer entry.

   D3D10_STENCIL_OP_INCR: Specifies to increment the stencil buffer entry. If the incremented value exceeds the maximum value (e.g., 255 for an 8-bit stencil buffer), then we wrap to 0.

   D3D10_STENCIL_OP_DECR: Specifies to decrement the stencil buffer entry. If the decremented value is less than 0, then we wrap to the maximum allowed value.

 

Note:    The BackFace settings are irrelevant in this case, since we do not render back-facing polygons due to backface culling. However, sometimes we do need to render back-facing polygons for certain graphics algorithms, or for transparent geometry (like the wire fence box, where we could see through the box to see the back sides). In these cases, the BackFace settings are relevant.

9.3.3 Creating and Binding a Depth/Stencil State

After we have filled out a D3D10_DEPTH_STENCIL_DESC structure, we can obtain a pointer to an ID3D10DepthStencilState interface with the following method:

   pDepthStencilDesc: Pointer to a filled-out D3D10_DEPTH_STENCIL_DESC structure describing the depth/stencil state block we want to create.

   ppDepthStencilState: Returns a pointer to the created ID3D10DepthStencilState interface.

Once an ID3D10DepthStencilState interface is created, we bind it to the output merger stage of the pipeline with the following method:

   pDepthStencilState: Pointer to the depth/stencil state block to set.

   StencilRef: The 32-bit stencil reference value to use in the stencil test.

As with the other state groups, a default depth/stencil state exists (basically the usual depth test with stenciling disabled). The default depth/stencil state can be restored by passing null for the first parameter of OMSetDepthStencilState:

9.3.4 Depth/Stencil States in Effect Files

A depth/stencil state can also be directly defined and set in an effect file:

The values you assign to the depth/stencil state object are like those you assign to the C++ structure, except without the prefix. For example, instead of specifying D3D10_STENCIL_OP_INCR we just specify INCR in the effect code. Incidentally, the state values we specify are not case sensitive; for example, INCR is equivalent to Incr.

9.4 Mirror Demo

Many surfaces in nature serve as mirrors and allow us to see the reflections of objects. This section describes how we can simulate mirrors for our 3D applications. Note that for simplicity, we reduce the task of implementing mirrors to planar surfaces only. For instance, a shiny car can display a reflection; however, a car’s body is smooth, round, and not planar. Instead, we render reflections such as those displayed in a shiny marble floor or in a mirror hanging on a wall — in other words, mirrors that lie on a plane.

Implementing mirrors programmatically requires us to solve two problems. First, we must learn how to reflect an object about an arbitrary plane so that we can draw the reflection correctly. Second, we must only display the reflection in a mirror; that is, we must somehow “mark” a surface as a mirror and then, as we are rendering, only draw the reflected object if it is in a mirror. Refer back to Figure 9.1, which first introduced this concept.

The first problem is easily solved with some analytic geometry, which is discussed in Appendix C. The second problem can be solved using the stencil buffer.

9.4.1 Overview

Figure 9.2: The eye sees the box reflection through the mirror. To simulate this, we reflect the box across the mirror plane and render the reflected box as usual.

Figure 9.2 shows that to draw a reflection of an object, we just need to reflect it over the mirror plane. However, this introduces problems, as seen in Figure 9.1. Namely, the reflection of the object (the crate in this case) is rendered on surfaces that are not mirrors (like the walls, for example). The reflection should only be seen through the mirror. We can solve this problem using the stencil buffer because the stencil buffer allows us to block rendering to certain areas on the back buffer. Thus we can use the stencil buffer to block the rendering of the reflected crate if it is not being rendered into the mirror. The following steps outline how this can be accomplished:

1.    Render the floor, walls, mirror, and crate to the back buffer as normal. Note that this step does not modify the stencil buffer.

2.    Clear the stencil buffer to 0. Figure 9.3 shows the back buffer and stencil buffer at this point.

Figure 9.3: The scene rendered to the back buffer and the stencil buffer cleared to 0 (denoted by light gray color). The black outlines drawn on the stencil buffer illustrate the relationship between the back buffer pixels and the stencil buffer pixels — they do not indicate any data drawn on the stencil buffer.

3.    Render the mirror to the stencil buffer. Set the stencil test to always succeed (D3D10_COMPARISON_ALWAYS) and specify that the stencil buffer entry should be replaced (D3D10_STENCIL_OP_REPLACE) with 1 if the test passes. If the depth test fails, we specify D3D10_STENCIL_OP_KEEP so that the stencil buffer is not changed if the depth test fails (this can happen if the crate obscures part of the mirror). Since we are only rendering the mirror to the stencil buffer, it follows that all the pixels in the stencil buffer will be 0 except for the pixels that correspond to the visible part of the mirror — they will have a 1. Figure 9.4 shows the updated stencil buffer. Essentially, we are marking the visible pixels of the mirror in the stencil buffer.

Figure 9.4: Rendering the mirror to the stencil buffer, we are essentially marking the pixels in the stencil buffer that correspond to the visible parts of the mirror. The solid black area on the stencil buffer denotes stencil entries set to 1. Note that the area on the stencil buffer occluded by the crate does not get set to 1 since it fails the depth test (the crate is in front of that part of the mirror).

4.    Now we render the reflected crate to the back buffer and stencil buffer. But recall that we only will render to the back buffer if the stencil test passes. This time, we set the stencil test to only succeed if the value in the stencil buffer is 1. In this way, the reflected crate will only be rendered to areas that have a 1 in their corresponding stencil buffer entry. Since the areas in the stencil buffer that correspond to the visible parts of the mirror are the only entries that have a 1, it follows that the reflected crate will only be rendered into the visible parts of the mirror.

9.4.2 Defining the Depth/Stencil States

To implement the previously described algorithm, we need two depth/stencil states. The first is used when drawing the mirror to mark the mirror pixels on the stencil buffer. The second is used to draw the reflected crate so that it is only drawn into the visible mirror.

Note that when drawing the reflected crate, we set the depth test to always pass. This is because the reflected crate is behind the wall, so the wall actually occludes it. In other words, if we used the normal depth test comparison function, the reflected crate would not be drawn since the depth test would fail. Moreover, we disable depth writes, as we should not update the depth buffer with the depths of the reflected crate that lies behind the wall/mirror.

9.4.3 Blending the Reflection

Our mirror, as shown in Figure 9.1, is not a perfect mirror and it has a texture of its own. Therefore, we blend the reflected crate with the mirror so that the final color is a weighted average of the reflected crate color and the mirror color. In other words, the crate is not reflected perfectly off the mirror — only a percentage of it is; the remaining percentage of color comes from the interaction between the light and mirror material.

Note that the crate texture does not have a source alpha — the WoodCrate01.dds image has no alpha channel. So how does this blending work? Well, we use D3D10_BLEND_BLEND_FACTOR as the source blend factor and D3D10_BLEND_INV_BLEND_FACTOR as the destination blend factor; this allows us to specify the blend factor color in the OMSetBlendState method:

These settings give the following blending equation:

C = 0.65 · C_src + 0.45 · C_dst

So we see 65% of the color comes from the crate and 45% of the color comes from the mirror. For reference, the following code shows how we create the blend state interface:

9.4.4 Drawing the Scene

The following code illustrates our draw method. We have omitted irrelevant details, such as setting constant buffer values, for brevity and clarity (see the Mirror demo example code for the full details).

 

Note:    When a triangle is reflected across a plane, its winding order does not reverse, and thus, its face normal does not reverse. Hence, outward-facing normals become inward-facing normals (see Figure 9.5) after reflection. To correct this, we tell Direct3D to interpret triangles with a counterclockwise winding order as front-facing and triangles with a clockwise winding order as back-facing (this is the opposite of our usual convention — §5.9.2). This effectively reflects the normal directions so that they are outward facing after reflection. We reverse the winding order convention by setting the following rasterizer state:

Figure 9.5: The polygon normals do not get reversed with reflection, which makes them inward facing after reflection.

 

Note:    When we draw the reflected crate, we also need to reflect the light source across the mirror plane. Otherwise, the lighting in the reflection would not be accurate.

9.5 Summary

   The stencil buffer is an off-screen buffer we can use to block the rendering of certain pixel fragments to the back buffer. The stencil buffer is shared with the depth buffer and thus has the same resolution as the depth buffer. Valid depth/stencil buffer formats are DXGI_FORMAT_D32_FLOAT_S8X24_UINT and DXGI_FORMAT_D24_UNORM_S8_UINT.

   The decision to block a particular pixel from being written is decided by the stencil test, which is given by the following:

where the operator is any one of the functions defined in the D3D10_COMPARISON_FUNC enumerated type. The StencilRef, StencilReadMask, StencilReadMask, and comparison operator are all application-defined quantities set with the Direct3D depth/stencil API. The Value quantity is the current value in the stencil buffer.

   The first step to creating an ID3D10DepthStencilState interface is to fill out a D3D10_DEPTH_STENCIL_DESC instance, which describes the depth/stencil state we want to create. After we have filled out a D3D10_DEPTH_STENCIL_DESC structure, we can obtain a pointer to an ID3D10Depth-StencilState interface with the ID3D10Device::CreateDepthStencilState method.    Finally, we bind a depth/stencil state block to the output merger stage of the pipeline with the ID3D10Device::OMSetDepthStencil-State method.

9.6 Exercises

1.    Modify the Mirror demo in the following way. First draw a wall with the following depth settings:

Next, draw a box behind the wall with these depth settings:

Does the wall occlude the box? Explain. What happens if you use the following to draw the wall instead?

Note that this exercise does not use the stencil buffer, so that should be disabled.

2.    Modify the Mirror demo by not reversing the triangle winding order convention. Does the reflected crate render correctly?

3.    Modify the Mirror demo by disabling the stencil test when rendering the reflected crate. Does the reflected crate render only into the mirror?

4.    Modify the Transparent Water demo from Chapter 8 to draw a cylinder (with no caps) at the center of the scene. Texture the cylinder with the 60-frame animated electric bolt animation found in this chapter’s directory using additive blending. Figure 9.6 shows an example of the output. (Hint: Refer back to §8.5.5 for the depth states to use when rendering additive blending geometry.)

Figure 9.6: Sample screenshot of the solution to Exercise 4.

5.    Depth complexity refers to the number of pixel fragments that compete, via the depth test, to be written to a particular entry in the back buffer. For example, a pixel we have drawn may be overwritten by a pixel that is closer to the camera (and this can happen several times before the closest pixel is actually figured out once the entire scene has been drawn). The pixel P in Figure 9.7 has a depth complexity of three since three pixel fragments compete for the pixel.

Figure 9.7: Multiple pixel fragments competing to be rendered to a single pixel on the projection window. In this scene, the pixel P has a depth complexity of three.

Potentially, the graphics card could fill a pixel several times each frame. This overdraw has performance implications, as the graphics card is wasting time processing pixels that eventually get overridden and are never seen. Consequently, it is useful to measure the depth complexity in a scene for performance analysis.

We can measure the depth complexity as follows: Render the scene and use the stencil buffer as a counter; that is, each pixel in the stencil buffer is originally cleared to zero, and every time a pixel fragment is processed, increment its count with D3D10_STENCIL_OP_INCR. The corresponding stencil buffer entry should always be incremented for every pixel fragment no matter what, so use the stencil comparison function D3D10_COMPARISON_ALWAYS. Then, for example, after the frame has been drawn, if the ijth pixel has a corresponding entry of five in the stencil buffer, then we know that five pixel fragments were processed for that pixel during that frame (i.e., the pixel has a depth complexity of five). Note that when counting the depth complexity, technically you only need to render the scene to the stencil buffer.

To visualize the depth complexity (stored in the stencil buffer), proceed as follows:

a.    Associate a color c_k for each level of depth complexity k. For example, blue for a depth complexity of one, green for a depth complexity of two, red for a depth complexity of three, and so on. (In very complex scenes where the depth complexity for a pixel could get very large, you probably do not want to associate a color for each level. Instead, you could associate a color for a range of disjoint levels. For example, pixels with depth complexity 1 to 5 are colored blue, pixels with depth complexity 6 to 10 are colored green, and so on.)

b.    Set the stencil buffer operation to D3D10_STENCIL_OP_KEEP so that we do not modify it anymore. (We modify the stencil buffer with D3DSTENCILOP_INCR when we are counting the depth complexity as the scene is rendered, but when writing the code to visualize the stencil buffer, we only need to read from the stencil buffer and we should not write to it.)

c.    For each level of depth complexity k:

1)    Set the stencil comparison function to D3D10_COMPARISON_EQUAL and set the stencil reference value to k.

2)    Draw a quad of color c_k that covers the entire projection window. Note that this will only color the pixels that have a depth complexity of k because of the preceding set stencil comparison function and reference value.

With this setup, we have colored each pixel based on its depth complexity uniquely, and so we can easily study the depth complexity of the scene. For this exercise, render the depth complexity of the scene used in the Clip Pixel demo from Chapter 8. Figure 9.8 shows a sample screenshot.

Figure 9.8: Sample screenshot of the solution to exercise 5.

 

Note:    The depth test occurs in the output merger stage of the pipeline, which occurs after the pixel shader stage. This means that a pixel fragment is processed through the pixel shader, even if it may ultimately be rejected by the depth test. However, modern hardware does an “early z-test” where the depth test is performed before the pixel shader. This way, a rejected pixel fragment will be discarded before being processed by a potentially expensive pixel shader. To take advantage of this optimization, you should try to render your non-blended game objects in front-to-back order with respect to the camera; in this way, the nearest objects will be drawn first, and objects behind them will fail the early z-test and not be processed further. This can be a significant performance benefit if your scene suffers from lots of overdraw due to a high depth complexity. We are not able to control the early z-test through the Direct3D API; the graphics driver is the one that decides if it is possible to perform the early z-test. For example, if a pixel shader modifies the pixel fragment’s depth value, then the early z-test is not possible, as the pixel shader must be executed before the depth test since the pixel shader modifies depth values. The opposite of this strategy is to render in back-to-front order, which would mean every pixel fragment would be processed only to be overwritten by a pixel in front of it.

 

Note:    We mentioned the ability to modify the depth of a pixel in the pixel shader. How does that work? A pixel shader can actually output a structure, not just a single color vector as we have been doing thus far:

The z-coordinate of the SV_Position element (pIn.posH.z) gives the unmodified pixel depth value. Using the special system value semantic SV_Depth, the pixel shader can output a modified depth value.