Chapter 4

Efficient Graphics with OpenGL ES 2.0

We have seen what OpenGL 1.0 can offer, but there is so much more to this API. In this chapter, we take a look at the cutting edge in graphics development: OpenGL ES 2.0, a subset of OpenGL. We'll start with a brief description of the most important features OpenGL ES 2.0 can offer, including shaders, GLSL, and how they affect the Android platform. Then, will take a deeper look into OpenGL ES Shading Language (GLSL) by creating a neat Android project to render a geometric shape—icosahedrons—using OpenGL ES 2.0. Let's get started.

OpenGL ES 2.0 and Android

OpenGL ES 2.0 is a set of enhancements to OpenGL ES 1.0. It emphasizes a programmable 3D graphics pipeline with the ability to create shader and program objects and the ability to write vertex and fragment shaders in the GLSL.

OpenGL ES 2.0 provides the following desirable qualities for current graphics programming:

• A wider range of precision options for use in embedded devices using shading language similar to the desktop OpenGL 2.0
• Frame Buffer Objects to simplify surface management and offer a subset of functionality from the desktop FBO
• One hundred percent backward compatible with OpenGL ES 1.x and built with the latest standards and most advanced ideas available for graphics development

Android fully implements the OpenGL ES 2.0 specification. However, the following are some caveats you should consider before using this technology to build your games:

• OpenGL ES 2.0 is not supported in all versions of Android. Thus if you are targeting the largest breath of devices out there, you should stick with OpenGL ES 1.x.
• OpenGL ES 2.0 implements the slickest ideas and technology in graphics rendering; however, that doesn't necessarily mean that the code will be better or run faster.

It does provide a desirable characteristic, nevertheless. It is designed to reduce power consumption in embedded systems such as phones, thus it could reduce your game's overall power requirements and provide more efficient graphics rendering. All in all, having a solid knowledge of OpenGL ES 2.0 is a good thing for your résumé. Chances are that if you are a graphics developer looking for a job, the very first thing you'll be asked in a job interview is your knowledge about shaders and GLSL.

Shaders

Ashader is a simple program that describes the traits of either a vertex or a pixel. At the low level, a shader defines a set of software instructions used to calculate rendering effects with a high degree of flexibility. Shaders were created to replace the traditional desktop OpenGL fixed-function pipeline that allowed only common geometry transformation and pixel-shading functions. They provide the following advantages over the traditional desktop OpenGL pipeline:

• Customized effects can be applied for rendering
• A higher degree of flexibility
• Simplicity, and higher degree of reusability

There are three basic types of shaders implemented in OpenGL ES 2.0: vertex, fragment, and geometry.

Vertex Shaders

Vertexshaders are run once for each vertex given to the GPU and transform the 3D position in virtual space to the 2D coordinate for on-screen rendering. They can manipulate properties such as position, color, and texture coordinates, but cannot create new vertices.

Fragment Shaders

Fragmentshaders (also known as pixel shaders) calculate the color of individual pixels. They are typically used for scene lighting and related effects, such as bump mapping and color toning, and are often called many times per pixel for every object that is in the corresponding space.

Geometry Shaders

Geometryshaders can add and remove vertices from a mesh. They are used to generate geometry or to add volumetric detail to existing meshes that would be too costly to process on the CPU.

GLSL

GLSL is the OpenGL ES 2.0 Shading Language for programming vertex and fragment shaders that has been adapted for embedded platforms. It is meant to work together with OpenGL ES 1.1 to minimize the cost and power consumption of embedded devices like smartphones.

At the implementation level, GLSL is actually two closely-related languages: vertex shader and fragmentshader.

Vertex Shader Language (VSL)

At its simplest, VSL is a C style program to manipulate the attributes of a vertex. The following fragment defines a very simple vertex shader to set the rendering position to the position of the current vertex.

As we can see, the shader has a C-style syntax with main function where you simply declare GLSL instructions. In this case, we use two built-in variables:

• gl_Position: Sets the position of the vertex to be rendered
• gl_Vertex: Contains the position of the current vertex being processed

Fragment Shader Language (FSL)

The FSL is used to change the color attributes (RGBA) of the current pixel. For example, the following fragment sets the color of the current pixel to red RGBA (1, 0, 0, 0).

gl_FragColor is the built-in variable used to set the color of the current pixel. As with any programming language, GLSL provides all the things you would expect from a computer language, including:

• Variables and functions: All variables and functions must be declared before being used
• Basic types: This includes void, bool, int, float, vec2 (two-component float point vector),  boolean or integer 2, 3 or 4 component vectors,22, 33, or 44 float matrices
• Declaration scope: Determines where the declaration is visible; this includes  global and local variables, name spaces, plus re-declarations within the same scope
• Storage qualifiers: Qualifiers specified in front of the type—pure traditional C-style—including: local variables and constants
  

  New to GLSL, we also have:

  • attribute: defines the linkage between a vertex shader and OpenGL ES for per-vertex data
  • uniform: tells that the value does not change across the primitive being processed; forms the linkage between a shader, OpenGL ES, and the application
  • varying: defines that linkage between a vertex shader and a fragment shader for interpolated data
• Parameter qualifiers: These are the qualifiers passed to the arguments of a function, including:
  • in: a parameter is passed into a function
  • out: a parameter passed back out of a function, but not initialized
  • inout: a parameter is passed both into and out of a function
• Precision qualifiers: For floating point precision including: highp, mediump, and lowp for high, medium, and low precision, respectively
• Variance and the invariant qualifier: These are used to handle the possibility of getting different values from the same expression in different shaders. It is possible, in independent compilation, that two identical shaders are not exactly the same when they run, in which case there are two qualifiers—invariant and varying—to prevent or allow this.
• Operators and expressions: All the common operators and expression you would expect from your typical programming language (see the following sidebar for more details)
• Many other powerful features: These include built-in angle, trigonometry, exponential, matrix, and vector functions; built-in variables, and more

Need GLSL Help?

Now let's take a look at how we can implement a shader and use it in an Android program.

Anatomy of a Shader

The anatomy of a shader is defined by the following steps:

1. Create a program. The very first step is to create a shader program to be run within your main program—a program within a program, if you will.
2. Load the shader. Once you create the shader, you must load it from a string variable or file.
3. Attach. Next, you must attach the shader to the main program.
4. Link. This step compiles the shader code and checks for syntax errors.
5. Optional validation. It is always a good idea to validate the link status and handle errors appropriately.
6. Enable and use. Finally, you can enable it and use it against a set of vertices.

Creating the Shader Program

To create a shader object or program, we use the glCreateShader API call. It takes as a parameter a shader type: either GL_VERTEX_SHADER or GL_FRAGMENT_SHADER for vertex or fragment respectively. glCreateShader returns a non-zero value by which it can be referenced. The following fragment creates two shaders to load a vertex, and fragment shaders to draw an icosahedron (described later in this chapter).

We also make an API call to glCreateProgram, which creates an empty program object and returns a non-zero value by which it can be referenced. Shaders must be attached to a program. This provides a mechanism to specify the shader objects that will be linked to create a program. It also provides a means for checking the compatibility of the shaders that will be used to create a program. Next, we load it.

Loading the Shader

Ashader object is used to maintain the source code strings that define a shader. For this purpose, we can create a load function that invokes: glShaderSource and glCompileShader. glShaderSource takes as arguments the ID of the shader, the number of elements, a string containing the source code to be loaded, and an array of string lengths (NULL in this case). glCompileShader compiles the shader described by its reference ID. The following fragment describes the load function that will be used to draw the Icosahedron for an upcoming project.

As a bonus, you can also check the compilation status using the API call glGetShaderiv. It takes as arguments: a shader ID, a query constant (GL_COMPILE_STATUS, in this case, to check the compilation status), and the status of the query. If the status is not GL_TRUE, then the compilation errors can be extracted by calling glGetShaderInfoLog with the ID of the shader and a string buffer that described the nature of the error. The next step is attaching the shader to a program.

Attaching to the Shader

To attach our shader to the main program, use the API call glAttachShader. It takes as arguments the ID of the program object, to which a shader object will be attached, and the shader object that is to be attached, as shown in the following fragment.

We also use glBindAttribLocation to associate a user-defined attribute variable in the program object with a generic vertex attribute index. The name of the user-defined attribute variable is passed as a null-terminated string in the last argument. This allows the developer to declare variables in the master program and bind them to variables in the shader code.

Linking the Shader Program

To use the shaders, we must link the program that contains them by calling glLinkProgram with the reference ID of the program. Behind the scenes, OpenGL will create an executable that will run on the programmable fragment processor.

Getting the Link Status

The status of the link operation will be stored as part of the program object's state. It is always a good idea to check for errors by getting the status of the link using glGetProgramiv, very similar to the way we checked the compilation status but using the GL_LINK_STATUS constant in this particular case. The following fragment demonstrates how to do so.

Optional: Program Validation and Status

You should always validate program objects. It helps to see if you have syntax errors in your shader code. To validate a program, use the API call glValidateProgram with the reference ID of the program. Next, call glGetProgramiv with the program validation constant GL_VALIDATE_STATUS. The result of the validation will be returned in the last argument (ShaderStatus in this case). Then, simply check the status and handle the error accordingly, as shown in the following fragment.

Finally, enable and use the program.

Enabling and Using the Program

To starts things off, use glUseProgram with the program ID to install a program object as part of a current rendering state. A program object will contain an executable that will run on the vertex processor if it contains one or more shader objects of type GL_VERTEX_SHADER that have been successfully compiled and linked.

Remember the two local attributes (Position and Normal) we declared in the attach step? They must be enabled before they can take effect. By default, all client-side capabilities are disabled, including all generic vertex attribute arrays. If enabled, the values in the generic vertex attribute array will be accessed and used for rendering when calls are made to vertex array commands such as glDrawArrays, glDrawElements, glDrawRangeElements, glArrayElement, glMultiDrawElements, or glMultiDrawArrays.

Now let's put what we have learned so far into practice by building a neat Android project to draw an icosahedron usingshaders.

Invoking OpenGL ES 2.0 in Android

Before we jump into the Android project, there are three steps that we should keep in mind when using OpenGL ES 2.0 in Android as opposed to OpenGL ES 1.x.

1. The surface view Java class must use a custom context factory to enable 2.0 rendering.
2. The surface view class must use a custom EGLConfigChooser to be able to select an EGLConfig that supports 2.0. This is done by providing a config specification to eglChooseConfig() that has the attribute EGL10.ELG_RENDERABLE_TYPE containing the EGL_OPENGL_ES2_BIT flag set.
3. The surface view class must select the surface's format, then choose an EGLConfig that matches it exactly (with regards to red/green/blue/alpha channels bit depths). Failure to do so will result in an EGL_BAD_MATCH error.

Project Icosahedron

This is where we put our skills to the test with a cool Android project to draw an icosahedron using vertex and fragment shaders. The goals of the exercise are to:

• Demonstrate the use of vertex and fragment shaders using OpenGL ES 2.0.
• Demonstrate the use of hybrid activities (both Java and C++) to perform the geometry rendering. The project launcher, surface, and rendering thread will be created in Java. All OpenGL ES 2.0 rendering will be performed in C++, using JNI to glue both parts together.
• Demonstrate Android multi-touch functionality to manipulate the rotation speed (using swipes) and the zooming (using pinching) of the shape.

But before we start, let's take a look at what an icosahedron is.

Reviewing the Shape

An icosahedron is a regular polyhedron with 20 identical equilateral triangular faces, 30 edges, and 12 vertices (see Figure 4–1).

Figure 4–1. Regular icosahedron

The following Cartesian coordinates define the vertices of an icosahedron with edge-length 2, centered at the origin:

where φ = (1+√5)/2 is the golden ratio (also written asτ). Note that these vertices form five sets of three concentric, mutually orthogonal golden rectangles. In the OpenGL coordinate system, which ranges from [-1, 1] in all axes. The 12 vertices of the icosahedron are defined as the following:

The 20 triangular faces (that map to the vertices) are defined as follows:

This information will be used by the shaders in our C++ program to render the scene on screen, as you'll see later on. But first, let's take a look at the project in more detail.

Tackling the Project

We'll start by creating a new Android project to host the code. Start the Eclipse Android project wizard and create a new project, as shown in Figure 4–2.

Figure 4–2. Project properties for the icosahedron

Next, perform the following steps:

1. Give the application a name (anything you choose).
2. Enter a package name; com.opengl.shader in this case.
3. Select a main Activity name (ShadersActivity).
4. Select a minimum SDK API number (14 for Android 4) and click Finish.

Your project should look as shown in Figure 4–3.

Figure 4–3. Icosahedron project

Table 4–1 lists the files that compose the project.

Now let's look at the files in more detail.

Manifest

Listing 4–1 is the XML file created by the wizard. The most important thing to remember is to use the OpenGL ES 2.0 feature and set the required attribute to true.

The next step is the main activity.

Activity

Listing 4–2 shows the main program of an Android application. It is very simple: when the application starts, the onCreate method will be invoked. Within this method, a ShadersView object is created and set as the content view. A set of arguments may be passed to the C++ layer by invoking the setrenderer method with an array of strings.

Now, on to the surface view.

Surface View

The surface view is in charge of creating an OpenGL-capable, hardware-accelerated surface where objects can be drawn. The process is triggered by the setRenderer method. Because Android supports a plethora of graphics configuration, resolutions, and hardware specs, we don't know if the running device is set up to perform OpenGL ES 2.0 calls. Thus we must create a context factory class ContextFactory, which implements GLSurfaceView.EGLContextFactory. This class can then be used to tell Android that we wish to use an OpenGL ES 2.0-enabled context by giving the version as an attribute (see Listing 4–3).

ShadersView is also in charge of the following:

• Choosing  an EGLConfig that matches the format of the surface exactly. This is going to be done in the configuration chooser class later on. For example, the following fragment tells the surface to use an RGB565 configuration with a depth and stencil size.
  setEGLConfigChooser( new ConfigChooser(5, 6, 5, 0, depth, stencil) );
• Creating a setting for the surface renderer.
  mRenderer = new ViewRenderer();
setRenderer(mRenderer);

Surface Renderer

The surface renderer (ViewRenderer) contains the following methods, which trigger on different stages of the surface life cycle, as shown in Listing 4–4:

• onSurfaceCreated: fires only once when the surface is first created
• onSurfaceChanged: may fire multiple times whenever a surface change occurs; for example, when the device is rotated
• onDrawFrame: fires many times by the rendering thread when a frame is drawn

Next comes the critical GLES 2.0 configuration chooser.

OpenGL ES 2.0 Configuration Chooser

The configuration chooser is critical to select an EGLConfig that supports OpenGL ES 2.0. ConfigChooser implements the Android interface GLSurfaceView.EGLConfigChooser and must receive a configuration spec with the attribute EGL10.ELG_RENDERABLE_TYPE containing the EGL_OPENGL_ES2_BIT flag. With this information, it queries the display for all available configurations (see Listing 4–5).

With this information, it chooses the best configuration that matches the original configuration spec.

That will take care of the Java side of things; now let's shift gears to the C++ rendering. Table 4-1 described the native side of the project (contained in the ico.cppfile), which is the last piece of the puzzle. This file is in charge of the JNI function implementation; it also contains the source of the shaders, plus scene initialization and rendering. Let's take a look.

Native Icosahedron

The Java native functions declared in ViewRenderer.java are implemented in C++ using the syntax shown in Listing 4–6.

We have the following three C++ functions:

• Init: to initialize the scene
• Display: to draw a frame of the scene
• doSetRotationSpeed: to set the rotation speed

Before we look at the implementations, we must create the two shader programs, vertex and fragment, which will be used to draw the icosahedron.

Project Shaders

Listing 4–7 declares the two shader programs that will be used to compute the position and color of the vertices and the faces of the icosahedron.

Scene Initialization

The scene initialization in Listing 4–8 performs the following steps:

1. It creates two shader programs, vertex and fragment.
   Shader[0] = glCreateShader(GL_VERTEX_SHADER);
Shader[1] = glCreateShader(GL_FRAGMENT_SHADER);
2. It loads the vertex shader.
   LoadShader((char *)VertexShader, Shader[0]);
3. It loads the fragment shader. Note that VertexShader and FragmentShaderRed are two strings describing the shaders in Listing 3–28 from Chapter 3.
   LoadShader((char *)FragmentShaderRed, Shader[1]);
4. It creates the program and attaches the shaders and attributes.
   Program   = glCreateProgram();
glAttachShader(Program, Shader[0]);
glAttachShader(Program, Shader[1]);
5. It attaches the attributes or variables (Position and Normal) used by the master and shader programs to manipulate the vertex and face information of the icosahedron.
   glBindAttribLocation(Program, 0, "Position");
glBindAttribLocation(Program, 1, "Normal");
6. It links the program using its program ID: glLinkProgram(Program).
7. It validates the program status by querying the status using the GL_VALIDATE_STATUS constant.
   glValidateProgram(Program);
glGetProgramiv(Program, GL_VALIDATE_STATUS, &ShaderStatus);
if (ShaderStatus != GL_TRUE) {
  // handle error
}
8. It enables the program and attributes: Position (0) and Normal (1).
   glUseProgram                (Program);
glEnableVertexAttribArray   (0);
glEnableVertexAttribArray   (1);

Initialization is the first step and it is performed only once. Next, we tackle rendering.

Scene Rendering

Scene rendering is performed multiple times when a frame is to be drawn. Listing 4–9 defines the Display C++ function.

The Display C++ function performs the following steps:

1. It clears the screen with glClear (GL_COLOR_BUFFER_BIT);.
2. It creates a model matrix float Model[4][4] to setup a projection so that the object rotates around the Y axis. It also translates it appropriately to display.
3. It then constantly rotates the object as a function of time.
4. It enables the program with glUseProgram (Program);.
5. It binds the shader attributes Position (iProj) and Normal (iModel) to the projection (Proj) and Model matrices.
       glUniformMatrix4fv      (iProj, 1, false, (const float *)&Proj[0][0]);
    glUniformMatrix4fv      (iModel, 1, false, (const float *)&Model[0][0]);
6. It sets the icosahedron vertex information with
       glVertexAttribPointer   (0, 3, GL_FLOAT, 0, 0, &PtData[0][0]);
    glVertexAttribPointer   (1, 3, GL_FLOAT, GL_TRUE, 0, &PtData[0][0]);
7. Finally, it draws the icosahedron using the GL_TRIANGLES and the face information array described at the beginning of the project.
   glDrawElements (GL_TRIANGLES, sizeof(FaceData) / sizeof(unsigned short),
GL_UNSIGNED_SHORT, &FaceData[0][0]);

Setting the Rotation Speed

The rotation speed function in Listing 4–10 is a bonus C++ call that will be used in the next section to change the rotation speed whenever a finger is swiped in the display. To do so, it updates a global variable RotationSpeed, which is, in turn, used by the Display function to update the angle of the Model matrix used by the shaders. Because RotationSpeed is read by multiple threads, it is declared as volatile that tells the compiler to always re-read from memory when used.

This takes care of the C++ side of things. As a bonus, let's add swipe and pinch zooming functionality with Android's multi-touch APIs.

Adding Swipe and Multi-Touch Pinch for Zooming

As a bonus, this section describes how to use the Android multi-touch APIs to increase the rotation speed of the icosahedron by:

• Increasing the speed whenever finger-swiped to the right, or decrease it when swiping to the left. The rotation will switch from left to right whenever a threshold value is reached.
• Zooming the shape in or out whenever pinching inward or outward with two fingers.

Listing 4–11 describes the additions to the ShadersActivity class to perform such tasks.

To listen for touch events, an activity can overload the Android method:

The MotionEvent type contains all the information we need to access single or multi-touch attributes of the event. Among the most important are the following:

• getPointerCount: returns the number of pointers (or fingers) on screen
• getAction: returns the action constant being performed; for example,
  • ACTION_DOWN when the first pointer goes down
  • ACTION_UP when the first pointer goes up
  • ACTION_MOVE when the first pointer is dragged

Android supports up to four simultaneous pointers, thus when a second pointer goes down when the action returned will be ACTION_POINTER_2_DOWN, and so forth. When the user swipes on finger the pointer count will be 1. In that case, we simply save the XY coordinates of the finger and set the number of fingers to 1.

If pinching, the pointer count will be 2 and the action will be ACTION_POINTER_2_DOWN when the second finger goes down. In such case, we save the down coordinates of the second pointer by calling MotionEvent.getX and MotionEvent.getY with the index of the desired pointer. Also, set the number of fingers to 2.

Finally, when the swipe or pinch gesture completes, the MotionEvent.ACTION_UP or MotionEvent.ACTION_POINTER_2_UP actions will fire. Here, we calculate the delta coordinates for both fingers in the X and Y coordinates.

Next, we simply check the number of active fingers. A value of 1 indicates a swipe, 2 indicates a pinch. If swiping, we call the setRotationSpeed(DX1) function with the delta coordinates for the first finger in the X axis. If pinching, we call setPinch(DX1, DX2) with the deltas for both fingers in the X coordinate.

When swiping to the left, the delta X value will be negative; it will be positive when swiping to the right. In either case, we decrease or increase the rotation speed and call the view's setRotationSpeed function, which will invoke doSetRotation in C++.

For a pinch gesture, we must check if we are pinching inward or outward. When the deltas are DX1 > 0 && DX2 < 0, we have an inward pinch or zoom in; otherwise, it's a zoom out. In either case, we modify the width and height of the display by an arbitrary factor and invoke the view's setVideoSize(width, height) method. This method will invoke the C++ Init(w,h) subroutine.

Now let's compile and run the project in the device.

Compiling and Running

To compile the native library, start the Cygwin console in Windows, change to the project folder ch03.OpenGLES2Shaders, and use the Android compilation script:

The compilation script Android.mk is very simple, as shown in the following fragment. It defines a module called libicosahedron that is bound to the libraries: log (for text logging) and GLESv2 for OpenGL ES 2.0. When compilation completes, the shared library libicosahedron.so will be created in the libs/armeabi folder of your project.

The library can now be loaded within Java with the system call: System.loadLibrary(“icosahedron”). We're done! Connect the device to your computer, create a run configuration for the project, and launch it in your device. The result is shown in Figure 4–4. Try swiping a finger to the left or right to change the rotation speed or pinching to zoom in/out—and have some fun with OpenGL 2.0.

Figure 4–4. Icosahedron in action

Summary

We have seen the most important features of OpenGL ES 2.0 and how they can be used to create a complex shape using vertex and fragment shaders. Remember the following when looking at using OpenGL ES 2.0 in Android:

• OpenGL ES 2.0 is not supported in all versions of Android, thus if you want to reach the largest amount of users and devices, you should use OpenGL ES 1.x instead.
• OpenGL ES 2.0 provides more efficient power consumption than 1.x. Always keep that in mind when working on embedded devices.
• If you are interested in the latest and greatest techniques in graphics rendering, then OpenGL 2.0 is definitely the way to go.

In the following chapters, you will learn how easy it is to bring powerful PC hardware-accelerated game engines to the platform in record time and with minimal development costs. Carry on.