Creating and Using Alpha Maps/Channels

In this chapter, we are going to look at the most common methods of creating various forms of vegetation used in the games industry.

As creating foliage and leaves still relies heavily on alpha maps or a texture with an alpha channel, I’ll start off with a brief explanation.

An alpha map is a black-and-white or grayscale image, which controls the transparency of a surface when it’s applied to a mesh using a shader. In the example pictured in Fig. 4.1, the alpha map is on the left, the diffuse texture in the middle, and the final result applied to the mesh is on the right.

FIG 4.1

This works because the shader is interpreting values of 100% white as being fully opaque; and values of 100% black as being fully transparent. Any gray tones between 100% black and 100% white will be interpreted as being transparent.

Now that we know what an alpha map is, let’s look at how we go about creating them.

The first step is always the source image for the diffuse texture. Whether it is hand-painted or taken from a photo, you’ll need to have this first before you create the alpha map. I’ll use the previous example to demonstrate this.

This was initially taken from a photo and then modified using a mixture of Photoshop’s blending modes and hand–painting tools.

Initially, I ‘extracted’ the leaves I wanted from the original photo, and then used them to create the alpha channel.

FIG 4.2

There are a few different ways to do this.

I could simply create an alpha channel and start painting white around the leaves I want visible using the standard brush tool; but, this takes time and also means the background will remain. I wanted to remove this.

I started by creating a 512 × 512 image file. I then copied the source image into this file and scaled it to fit into the square texture page (CTRL + T). I then used the Lasso tool to create a selection around the area I wanted to retain.

I then inverted the selection by going to Select > Inverse on the menu bar. I then deleted the highlighted pixels. In a few key strokes, a large chunk of the unwanted image was removed.

For the remaining pixels around the leaves, we could either use the eraser, manually paint the pixels out, or we could use the magic wand tool to make pixel selections and delete them. The magic wand works best when the background is in a different color relative to the leaves, for example, silhouetted against a blue sky.

FIG 4.3

In the photo we are working with, there is a good level of contrast in the image; so we can use the magic wand tool. We need to keep the tolerance low, as the image is mainly green.

This time, I have decided to use the magic wand to quickly remove chunks of the image. I then finished off by removing the areas I didn’t want by using the eraser to manually erase the final few stray pixels.

I also adjusted the levels and hue and saturation on the final version before starting work on the diffuse texture.

FIG 4.4

Now that we have separated out the area of the image we want to keep, we can quickly use this to create the alpha channel – or at least a good starting point!

Select the whole of the layer that the leaves are on by holding CTRL + A, then select the small thumbnail image on the left of the layer’s name. This will select all pixels on the currently selected layer.

FIG 4.5

With this selection active, click on the Channels tab, go to the bottom right, and click on the new layer icon highlighted in Fig. 4.6.

FIG 4.6

This will create a new alpha channel in the texture file. Next, select the standard brush tool at 100% opacity and paint the selected area. The result is fairly good, but there are a few gray and white pixels around that will need painting over with 100% black (RGB 0,0,0).

When the alpha channel has been cleaned up, we can preview the results in Photoshop over the RGB image.

Do this by turning on the RGB channels by clicking on the box to the left of the RGB channel at the top of the channels list. This will show or hide these channels. The same applies to the alpha channel.

When the alpha channel is displayed over the RGB channels, it will only show the pixels under the white pixels of the alpha channel. In the next image, the entire purple area represents the black values of the alpha channel.

You can change the color and opacity of the alpha channels by double clicking on the alpha channel.

It’s worth noting that you can still paint and edit the alpha channel while viewing all the channels simultaneously.

FIG 4.7

The last step is to save the final texture (once the diffuse map is complete), along with the alpha channel, so it can be used in 3ds Max. There are two ways to display the alpha channel or map in real time in the viewport.

The diffuse texture can be saved with the alpha channel inside the same file using a format that stores alpha information. Some formats that do this are .BMP, .PNG, .TIFF, .TGA, and .DDS.

This texture containing the diffuse and alpha channels will need to be assigned to both the diffuse and opacity slots in a material. When it is assigned to the opacity slot, the bitmap parameters will need adjusting. By default, the Mono Channel Output is set to RGB intensity; it will need to be changed to Alpha in 3ds Max.

FIG 4.8

An alternative method of saving the Alpha texture is to save the alpha layer as a separate texture map.

This can be done by selecting the entire alpha layer by using CTRL + A while on the alpha layer to highlight it. Then, press CTRL + C to copy the selection and CTRL + V in the RGB Channels in the Layers tab to paste it as a new layer. Then, save the file as a separate texture file after flattening the image.

Then, we just need to assign this new texture map to the opacity slot in the 3ds Max material and assign the diffuse texture into the diffuse slot.

Both methods should display the diffuse map working with the alpha map in the viewport once Show Standard Map in Viewport is turned on in the material and the following display mode selected Main Menu > Views > Show Materials in Viewport as > Standard Display with Maps is also on.

This concludes our brief introduction to alpha maps. Next, we’ll take a look at diffuse maps.

Creating the Diffuse Textures for Vegetation

In this section, I’ll demonstrate how I created all the diffuse-texture maps in Fig. 4.9.

FIG 4.9

The first aspect we need to look at is the layout of the texture pages. The bushes in the first two images take up the entire texture sheet. This was intentional, as I intend the bushes to range from a medium to a large size, so giving them more texture space; this will result in a higher resolution texture map. All the textures in this case are 512 × 512 pixels.

The texture sheet in the top right consists of four different types of leaves that will be used to create a variety of different plant models. This includes a horizontally tiling stalk section at the bottom that will be shared between the various plants. Each leaf gets around 256 pixels squared of texture space. These leaves are intended for small- to medium-sized plants.

The benefit of sharing many elements on a single texture sheet is that there are less individual texture files to deal with. In terms of real-time rendering in a game engine, the engine will also only have to call upon one texture map and not four, making it more efficient.

I also follow this concept of texture-sharing through on the smaller models shown on the bottom right image. This consists of a variety of grasses and small plants, which will be used as a cheap solution that we will use to fill our environment full of vegetation.

The texture on the bottom left will be used as the foliage for a tree. The leaves run diagonally across the texture sheet so that the space is used more efficiently. If we were to have it vertical as we have with the bushes, this would result in a top heavy texture with a lot of wasted, empty space at the bottom.

Let’s move on to the actual texture creation.

The texture layout is the only real difference between all of these textures. The technique for creating the diffuse texture is largely the same for all of them.

I’m going to take the leaves used in the alpha map section to demonstrate this technique.

In the previous section, we extracted the leaves from the photo that are similar to Fig. 4.10.

FIG 4.10

The first step is creating a background. Use the color picker to select two different green colors, one for each swatch. Make sure there is enough contrast between the values so we get a good result.

Then, create a new layer (Shift + Ctrl + N). We’re going to use the clouds filter to create a random blotchy background. Go to the Main Menu > Filter > Render > Clouds.

FIG 4.11

You may need to try adjusting the two colors a few times to get a good result. You should now have an image similar to the one in Fig. 4.12.

The layer the leaves are on has the blend mode set to normal. You could easily play around with all the different modes to see the results. One technique may not work every time. It all really depends on the initial reference photo.

The issue that I have with the leaves as they are now is that I don’t like the varying colors and lighting from the photograph. In this instance, the color dodge blend mode worked really well to reduce these issues.

FIG 4.12

FIG 4.13

Obviously, now, the image is oversaturated; so we should use Image > Adjustments > Hue/Saturation to reduce this. Move the slider to the left to reduce the strength of the saturation. With the contrast reduced, I think we should reduce the brightness and contrast on this layer also a little bit. The results are shown in Fig. 4.14.

FIG 4.14

Now, copy the leaves layer (CTRL + J). Then, set the blend mode to normal. Then, use Shift + CTRL + U to desaturate the image. Then, type Shift + Ctrl + L to apply Auto Levels to the image. This will increase the contrast of the values in the layer, as in Fig. 4.15.

FIG 4.15

Set this layer blend mode to overlay. I also reduced the opacity to 46%. This will create contrast in the image. The previous steps have removed the contrast a little; so we are now putting it back in.

Just so that you can see the exact difference, have a look at both textures by comparing them side by side with each other, as in Fig. 4.16.

FIG 4.16

As you can see, my preferred texture on the right is softer, with lesser variations in color and contrast. As the grayscale layer set to overlay can also have its opacity adjusted, we can control the contrast easily.

Create two new layers and place them below the original layer of leaves and set to color dodge. We will use these to paint color variations back into the texture. I want to go through this process to add color details that were not in the original image.

For the first layer, I painted the stalks of the foliage brown and used a darker green with a little blue to paint some shadows. A lighter green was used to paint highlights. I used the standard Photoshop brush with a medium opacity. In Fig. 4.17, the new layer is on the left over the background and on the right, all the layers are displayed.

At this point, we can consider our texture as being complete.

Feel free to experiment with this process to create some really nice original textures. Figure 4.18 shows the second paint layer on the left over the background and the final result on the right.

I created the other three sets of leaves on the texture sheet in exactly the same way, using references from different photos. The horizontally tiling stalk element running along the bottom of the texture sheet was created by using the Rectangular Marquee tool (M) to create a selection.

FIG 4.17

FIG 4.18

I then created a new layer and used the Gradient tool set to reflect the gradient. This applied a cylindrical style shading to define the stalk.

FIG 4.19

I then set this gradient layer to overlay. As it then looked too smooth, I copied a section from another image and stretched it out horizontally over the stem and set it to soft light. This was just to add a bit of noise and variety to the stem. This layer was also tiled horizontally using the Offset tool.

FIG 4.20

The other textures were created in exactly the same way. For the dry bush, I have extracted the sections of tree branches from a photo. I then duplicated some elements and moved them around to fill out the texture sheet to a greater extent. After this, the background was created. The texture was then adjusted afterward to blend in nicely with the other textures of the scene.

FIG 4.21

The tree branch and foliage were created by using the leaves from the earlier section and taking a section of the bush from the previous texture. Figure 4.22 shows the steps in creating this texture.

FIG 4.22

First, the branch was rotated so that it starts in the bottom left corner. I then started to duplicate and move, rotate and scale the sets of leaves and place them all over the branch.

I tried to get as much variety as possible by tweaking the Lightness, Hue, and Saturation of some copies as I went along.

After I was happy with the result, I added a background. This time, the foliages blend mode was set to normal; so I did not use the cloud filter on the background.

The final steps were to duplicate the leaves and desaturate the image. I then set it to multiply to get some shading and contrast into the foliage. Another common trick that I do with foliage is to duplicate the final version and place this layer beneath the original. Then, I blur the copy. The reason for this is that if you use a file type or game engine that uses mip-maps, it will prevent the alpha and diffuse maps having issues like a ‘white halo’ effect on the textures.

Finally, there is a slightly different technique that I use for creating grass.

The two bottom sections of the grass texture were created by using photographs as a base, as in the previous examples. The other types of grass were created entirely in Photoshop.

First, create a new file with the dimensions of 512 × 512 pixels. Go to the Main Menu > Edit > Preferences > Guides, Grid, Slices & Count. Under the grid option, set the gridline to every 256 pixels. You can also change the color if you wish. I have set mine to red.

If the grid is not appearing, you can toggle it under View > Show > Grid or by pressing CTRL + H.

FIG 4.23

The grid should now be displayed over your texture, dividing it up into four sections of 256 pixels square. This tool is very useful when creating texture sheets as you can ensure that all the elements are of the correct resolution. I intend most of the grass to fit inside a 256-pixel square.

Then, using the Pen tool, lay out some curved paths that will be used to create the grass. I am trying to emulate the way grass grows with the flow of the paths. To use the Pen tool, simply left click on the image to start a path. Left click and hold elsewhere to create a new point. You can then adjust the curve of the line by moving the mouse. Then, repeat this process until you are happy with the number of paths. You can adjust the path’s anchor points after they are created using the Direct Selection tool. This is in the same area as the Path Selection tool. Hold the LMB to access it.

FIG 4.24

In Fig. 4.24, you will notice that I started the paths near the base of the bottom left quadrant. This was because when stroking the path, the Simulate Pen Pressure will be checked. This option will create a tapered line for the grass from bottom to top. The blades of grass should be thickest at the bottom and taper to a fine tip; so by starting the paths below the area we want to have the thickest will give the desired result.

To do this, select the paths with the Path Selection tool. Then, right click and choose the stroke path or go to Edit > Stroke. This tool takes the options from any tool in the drop-down list. For this, I used the Brush tool. Make sure that the Simulate Pen Pressure is ticked, then press OK. Figure 4.25 shows these options.

FIG 4.25

The end result should be similar to the grass on the left in Fig. 4.26. To finish off the grass, I used a soft airbrush with a low opacity to build up a gradient on the bottom of the grass. Then, with a standard brush on a medium opacity, I manually painted some highlights onto the blades of grass.

This completes our diffuse textures.

Always experiment using different methods, blend modes, and photographic reference.

Next, we’ll have a look at modeling vegetation.

FIG 4.26

Modeling Vegetation

Creating vegetation for games using alpha maps usually means that the geometry can remain quite simple. You only need a group of planes with some simple modeling, letting the textures do all the work.

A good approach to creating vegetation models is to look at the “flow” of the plant, or how a plant or tree grows. In addition, pay attention to the overall silhouette of the plant. This will determine the form of the mesh and how the components will be placed together to create the overall shape to achieve a realistic, natural look.

In Fig. 4.26b, I did a quick paint-over of a variety of trees to show how they differ in the way they grow. The blue lines represent the ‘flow’ of the foliage or how it grows on the branches. The red lines are the growth patterns of the trunks and branches. I use this approach to break down the shapes into simpler forms before I begin modeling the tree.

FIG 4.26b

This approach of breaking a complex object down into simpler parts also works well for plants. In Fig. 4.26c, I demonstrate this. On the left is the reference and in the middle are some quick sketch attempts at breaking down the shapes of the leaves and the growing patterns of the plants. Finally, silhouetted images of the final models are on the right to show how this technique has been applied.

FIG 4.26c

We’ll start with the grass models.

Create a square plane and apply the grass texture sheet to it and planar map it. Then, using the Slice Plane tool, divide up the different grass elements, as shown in Fig. 4.27. Then, duplicate the plane four more times. The idea is to delete all except one grass element on each plane. The end result will be that each mesh consists of only one type of grass. You should end up with five types.

FIG 4.27

In Fig. 4.28, on the top two grass meshes, I duplicated the polygon and rotated it 90° to create crossed planes. Then, the two planes were combined into one mesh (using Attach). This is the most common and basic grass model used in games. It is very cheap and can be duplicated a lot within a scene to quickly create dense areas of grass.

FIG 4.28

I then made a couple of more elaborate and interesting versions of this model, as shown in the bottom two examples in Fig. 4.28. Slice Plane was used to add in some additional edges to the plane; then, these new edges were used to add some bends to the polygons. The example on the right is a combination of this method and the crossed-plane method to create a really dense patch of grass.

You can make the grass models as complex or detailed as you like, by using a few different textures or adding extra edges to create smoother more detailed shapes. In current games, the methods shown here are usually enough.

The more details you add, the less efficient the models will be when they are placed down thousands of times within a scene.

In Fig. 4.30, I have cut around the texture to reduce the amount of unused transparency on the mesh.

This is a widely used optimization technique as unused transparency on the surface of the mesh will often still be calculated by the game engine when rendering, even if it is not visible.

In Fig. 4.29, the area highlighted in red is the area I have removed. This makes the mesh more efficient to render, especially when there are a lot of instances of the model in the scene.

I then used these extra vertices to add depth to the model by pulling them back. The meshes were then duplicated and positioned to act like crossed planes.

FIG 4.29

FIG 4.30

The next models are a selection of smaller plants. They are very simple models and designed to be cheap, so that they can be used many times in a scene. These models were created in the same way as the grass was, although three of them also have stems. These were created by moving, rotating, and scaling the vertices of a three-sided cylinder. You could start with a cylinder with a few height segments or use slice plane to add in some edge loops as you need them.

The leaves on the far right plant have more modeling detail on them. The cut tool was used to add extra geometry. The vertices were then adjusted to create a central crease in the leaf.

FIG 4.31

The next models created were the bushes. The starting point was a square flat plane. Another plane was added along with some extra edges, and the central vertices were moved out to create some depth.

I then built the bush out from a central point on the ground using duplicates of these two meshes. The model on the right (Fig. 4.32) is much cheaper and intended for use on the edges of the scene.

I could have cut around the bush as an optimization from the initial state before I duplicated them, as mentioned earlier, to reduce the surface area of unused transparency.

FIG 4.32

As an example to show how flexible vegetation meshes are, I have assigned the dry bush texture to this model. It looks a lot different from the green leaf version, even though it is the same mesh. This is easy to do as a texture swap as long as the texture layout is similar; this will mean you won’t need to remap anything.

FIG 4.33

The final model we will look at in this chapter is the tree. First, we need to start with a cylinder and delete the polygons on the top and bottom. Then using the slice plane tool, we add some edge loops.

Next, use these to shape the bottom of the trunk. Duplicate the mesh, scale it, and position it high up on the original mesh. This will form the basic shape of a branch. These steps are shown in Fig. 4.34.

FIG 4.34

Duplicate the trunk and branch meshes a few more times and use them to create the rest of the tree structure. Figure 4.35 shows the final version of our basic tree model.

After positioning all of the branches, I spent a few minutes modeling a few more branches to create some variety in the model.

The next step is to combine all the meshes together using Attach and then to stitch the branches into the main trunk mesh. Use the Cut tool to create the edges on the trunk to match the branches. Delete the faces on the trunk where the branch will connect. Then, use weld or target weld to stitch the vertices of the trunk and the branches together.

FIG 4.35

Attaching the branches to the trunk is a personal preference of mine as I prefer to have the branches and trunk as one solid mesh. It is ok to leave the branches intersecting the trunk, as shown in the version on the left of Fig. 4.35. I just prefer the look of the lighting and shading on the stitched version in the center.

When you are satisfied with the tree model, next we need to unwrap it.

Cylindrical mapping is the easiest technique to use, paying close attention to keeping the seams facing inward into the tree so that they’ll be hidden by foliage later.

Figure 4.36 shows the final unwrap of the trunk and branches. I tried to have as few UV shells as possible. There is one for the trunk and one for each branch.

Now that the tree itself is complete, it’s time to look at the foliage.

Just as we did before, I created a flat plane and one with some extra detail. I then used lots of duplicates of these to fill out the foliage on the tree, placing them around the branches. I move, rotate, and scale as I go. Figure 4.37 shows the two foliage meshes used on the left and the final tree on the right.

FIG 4.36

FIG 4.37

When the foliage was complete, it was attached to the trunk to create one mesh. The pivot point was then centered to the object and snapped to the bottom vertices at the base of the trunk. This will allow for easier placement of the tree in the environment later.

As it is still not possible to build current game environments full of thousands of trees, many cheaper solutions are used to make environments seem fuller than they actually are. One of these methods is the use of tree walls.

Tree walls are flat planes or simple meshes used on the edges of an environment. They usually consist of one texture that repeats (tiles) along one axis. Figure 4.38 shows a flat plane with the texture applied.

FIG 4.38

Figure 4.39 shows a similar mesh, but a section of a cylinder has been used to give the model more form.

FIG 4.39

As always, this method of construction can be taken further. I would personally use the previous tree-wall methods if they were to be placed a good distance away from the game play area. They are great background fillers, but they don’t hold up well to close inspection if a player is beside them.

FIG 4.40

One solution is to fill out the tree wall with other meshes and create a more expensive version that will sit well just off to the side of the game play area behind other assets. Figure 4.40 shows a good example of this method.

It was created from most of the meshes created in this tutorial.