OpenGL ES

Three-dimensional rendering is becoming a more and more important feature of today's Android devices and applications. While becoming an expert in 3D rendering would take quite some time, the following section introduces some simple techniques that are easy to implement as well as some basic concepts you need to be aware of in order to start working with 3D rendering. If you want to learn more about OpenGL ES for Android, you can refer to “Pro OpenGL ES for Android” by Mike Smithwick and Mayank Verma.

Most recent Android devices support both OpenGL ES 1.1 and OpenGL ES 2.0 while older devices would support only OpenGL ES 1.1. As of December 2011, about 90% of the devices connecting to Android Market support OpenGL ES 2.0.

While OpenGL ES is a standard from the Khronos Group, several implementations exist. The most common GPUs supporting OpenGL ES in Android devices are:

• ARM Mali (example: Mali 400-MP4)
• Imagination Technologies PowerVR (example: PowerVR SGX543)
• Nvidia GeForce (Nvidia Tegra)
• Qualcomm Adreno (acquired from AMD, formerly ATI Imageon, and integrated into Qualcomm Snapdragon)

Extensions

The OpenGL standard supports extensions, allowing new features to be incorporated in certain GPUs. For example, the following extensions are supported by the Samsung Galaxy Tab 10.1 (based on Nvidia Tegra 2):

• GL_NV_platform_binary
• GL_OES_rgb8_rgba8
• GL_OES_EGL_sync
• GL_OES_fbo_render_mipmap
• GL_NV_depth_nonlinear
• GL_NV_draw_path
• GL_NV_texture_npot_2D_mipmap
• GL_OES_EGL_image
• GL_OES_EGL_image_external
• GL_OES_vertex_half_float
• GL_NV_framebuffer_vertex_attrib_array
• GL_NV_coverage_sample
• GL_OES_mapbuffer
• GL_ARB_draw_buffers
• GL_EXT_Cg_shaders
• GL_EXT_packed_float
• GL_OES_texture_half_float
• GL_OES_texture_float
• GL_EXT_texture_array
• GL_OES_compressed_ETC1_RGB8_texture
• GL_EXT_texture_compression_latc
• GL_EXT_texture_compression_dxt1
• GL_EXT_texture_compression_s3tc
• GL_EXT_texture_filter_anisotropic
• GL_NV_get_text_image
• GL_NV_read_buffer
• GL_NV_shader_framebuffer_fetch
• GL_NV_fbo_color_attachements
• GL_EXT_bgra
• GL_EXT_texture_format_BGRA8888
• GL_EXT_unpack_subimage
• GL_NV_texture_compression_st3c_update

Listing 8–15 shows how to retrieve the list of the extensions a device supports. Since OpenGL ES 2.0 is currently not supported by the Android emulator, make sure you run that code on an actual device.

You will need an OpenGL context to be able to successfully execute the code in Listing 8–15 and retrieve the list of extensions. For example, you can execute the code in your GLSurfaceView.Renderer's onSurfaceChanged() method. If you run that code and no OpenGL is available, then you will see a “call to OpenGL ES API with no current context” message in LogCat.

As you can see, the extensions above can be separated into multiple groups:

• GL_OES_*
• GL_ARB_*
• GL_EXT_*
• GL_NV_*

The GL_OES_* extensions are extensions that have been approved by the Khronos OpenGL ES Working Group. Even though these extensions are not required by OpenGL ES, they are widely available.

The GL_ARB_* extensions have been approved by the OpenGL Architecture Review Board. The GL_EXT_* extensions have been agreed upon by multiple vendors, whereas the GL_NV_* extensions are specific to Nvidia. By looking at what extensions are supported, you can usually tell which company provides a device's GPU:

• ARM extensions use the GL_ARM prefix.
• Imagination Technologies extensions use the GL_IMG prefix.
• Nvidia extensions use the GL_NV prefix.
• Qualcomm extensions use the GL_QUALCOMM, GL_AMD and GL_ATI prefixes.

Because not all devices support the same extensions (hello, fragmentation), you have to be extremely careful when optimizing for a specific device, as something that works on one device may not work on another. Typical examples include using Non-Power-Of-Two (NPOT) textures or a texture compression format that is supported on the device you test your application on but not on other devices. If you plan on supported non-Android devices, also check which extensions these devices support. For example, current Apple iPhone/iPod/iPad devices are all Imagination Technologies PowerVR-based (which may or may not share the same extensions) but future models may not be PowerVR-based.

Texture Compression

Textures will define what your OpenGL ES applications will look like. While it is easy to use uncompressed textures, such textures can quickly bloat your application. For example, an uncompressed 256×256 RGBA8888 texture could use 256 kilobytes of memory.

Uncompressed textures impact performance negatively as they are not as cacheable as compressed texture (because of their size), and they require more memory access (affecting also power consumption). Applications that use uncompressed textures are also bigger and therefore require more time to be downloaded and installed. Besides, the amount of memory on Android devices is typically limited, so your applications should use as little memory as possible.

In the list above, the following extensions show which texture compression formats are supported by the Samsung Galaxy Tab 10.1:

• GL_OES_compressed_ETC1_RGB8_texture
• GL_EXT_texture_compression_latc
• GL_EXT_texture_compression_dxt1
• GL_EXT_texture_compression_s3tc

The ETC1 compression format was created by Ericsson (ETC stands for Ericsson Texture Compression) and is supported by most Android devices (and by all Android devices that support OpenGL ES 2.0). It is therefore the safest choice to target as many devices as possible. This compression format uses 4 bits per pixel instead of 24 (ETC1 does not support alpha) and therefore achieves a 6× compression ratio: including the 16-byte header, a 256-by-256 ETC1 texture would use 32,784 bytes.

Multiple tools exist to create ETC1 textures. The Android SDK includes etc1tool, a command-line tool to encode PNG images to compressed ETC1 images. For example, the following line shows how to compress lassen.png (183 kilobytes) and generate the difference between the original image and the compressed one:

The output file, if not specified, is generated based on the name of the input file. In this particular case, the output file will be lassen.pkm (33 kilobytes). Figure 8–3 shows the difference between the compressed image and the original.

Figure 8–3. Original image (left) and difference with compressed image (right)

Unfortunately, the Android SDK's etc1tool does not provide many options. For those who want to be able to fine tune the compression and visual quality, another tool is recommended: etcpack. The etcpack tool can be downloaded from the Ericsson website (http://devtools.ericsson.com/etc) and allows for more options:

• Compression speed (slower speed = higher quality)
• Error metric (perceptual or non-perceptual)
• Orientation

You should always use the best quality possible before releasing your application when ETC1 textures are generated offline (that is, not on the Android device itself). Since the human eye is more sensitive to green than red or blue, you should also select “perceptual” as the error metric. (The algorithm will make green be closer to its original value at the expense of red and blue, decreasing the peak signal-to-noise ratio.)

While the difference may not be obvious, there is no reason to release an application that uses lower-quality textures as a higher-quality ETC1 texture will be the same size as a lower-quality one. Even though generating higher-quality textures takes more time and therefore many developers choose to work with lower-quality textures during development, you should always remember to switch to high-quality textures before releasing your application.

Alternatively, you can use the ARM Mali GPU Compression Tool, available for free at http://www.malideveloper.com/developer-resources/tools. This tool offers options similar to etcpack but with a graphical interface (a command-line version is also provided). Figure 8–4 shows the ARM Mali GPU Compression Tool in action with the same input file as above.

Figure 8–4. ARM Mali GPU compression tool

Figure 8–5 shows the difference in quality between a higher-quality (slow) compression and a lower-quality (fast) compression. Once again, the differences are not obvious.

Figure 8–5. Higher quality (left), medium (middle), and lower-quality (right)

As you can see on the ARM Mali developer website, many other tools are available. You can find many authoring and debugging tools on the ARM, Imagination Technologies, Nvidia, and Qualcomm websites:

• http://www.malideveloper.com/developer-resources/tools
• http://www.imgtec.com/powervr/insider/sdkdownloads
• http://developer.nvidia.com/tegra-android-development-pack
• http://developer.qualcomm.com/develop/mobile-technologies/graphics-optimization-adreno

For example, Imagination Technologies also provides a tool to create compressed textures, PVRTexTool, and the Nvidia website offers a complete SDK (including Android SDK, NDK, Eclipse, and sample code).

Starting in Android 2.2 (API level 8), the following classes are defined to help you work with ETC1 textures:

• android.opengl.ETC1
• android.opengl.ETC1Util
• android.opengl.ETC1Util.ETC1Texture

Your application can compress images into ETC1 textures dynamically with the ETC1.encodeImage() and ETC1Util.compressTexture() methods. This is useful when ETC1 textures cannot be generated offline, for example when the textures are based on pictures stored on the device (perhaps a picture of one of the user's contacts).

Other Texture Compression Formats

While ETC1 is the most common texture compression format, other formats exist and are supported by some devices. Among these are:

• PowerVR Texture Compression (PVRTC)
• ATI Texture Compression (ATC or ATITC)
• S3 Texture Compression (S3TC), with DXT1 to DXT5 variants

You should experiment with various compression formats depending on which devices you target. Because a particular GPU may be optimized for its own compression format (for example, PowerVR GPU optimized for PVRTC compression format), you may achieve better results with a proprietary compression format than with the more standard ETC1 compression format.

Manifest

Your OpenGL application's manifest should specify two things:

• Which OpenGL version it requires the device to support
• Which texture compression formats the application supports

Listing 8–16 shows how to specify the device should support OpenGL ES 2.0 and shows the application supports only two texture compression formats: ETC1 and PVRTC.

Android Market will use this information to filter applications when a device connects to Market. For example, a device that supports only OpenGL ES 1.1 will not see applications that use OpenGL ES 2.0. Similarly, a device that supports only ETC1 texture compression will not see applications that support only PVRTC and ATC textures.

If your application does not specify any texture compression format in its manifest, then Android Market will not apply any filtering based on compression format (and therefore will assume your application supports all compression formats). This could lead to users installing your application only to find out that the application does not work, potentially leading to bad reviews.

Mipmaps

Often, objects in a 3D scene appear in the background and do not use that much space on the screen. For example, using a 256×256 texture on an object that is only 10×10 pixels on the screen is a waste of resources (memory, bandwidth). Mipmaps solve this by providing multiple levels of detail for a texture, as shown in Figure 8–6.

Fige 8–6. Mipmaps from 256×256 to 1×1

While a mipmap set uses about 33% more memory than its lone original image, it may improve not only performance but also visual quality.

As you can see in Figure 8–6, each image is a version of the original image but at a different level of detail. Obviously, it is quite hard to see a flower in the 1×1 version of the texture.

The ARM Mali GPU Texture Compression Tool can generate mipmaps for you. Instead of generating only a single .pkm file, the tool will generate all levels of details all the way to 1×1 and will create a single .pkm file for each. Your application will then have to load these different levels one by one, for example by calling ETC1Util.loadTexture() or GLES20.glTexImage2D().

Because not all devices are equal, they may not all require the same levels of detail. For example, a Google TV device with a resolution of 1920×1080 (HD resolution) would typically require a higher level of detail than a Samsung Nexus S with a resolution of 800×480 (WVGA). Consequently, while a 256×256 texture may be needed for a Google TV device, the Nexus S may never have any use for such texture and would be ok using a 128×128 texture.

How the textures will be rendered also depends on what texture parameters are set with one of the glTexParameter() methods (for example, glTexParameteri()). For example, you can set the minifying function by calling glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MIN_FILTER, param) with param being one of the following values:

• GL_NEAREST
• GL_LINEAR
• GL_NEAREST_MIPMAP_NEAREST
• GL_NEAREST_MIPMAP_LINEAR
• GL_LINEAR_MIPMAP_NEAREST
• GL_LINEAR_MIPMAP_LINEAR

While GL_NEAREST is generally faster than GL_LINEAR, and GL_LINEAR is faster than the other four, better visual quality will be achieved with slower functions. What parameters your application will choose may depend on the user's preferences (if you provide a way for the user to configure the quality of rendering), however many users won't have the patience to try various settings to find a combination that works well on their devices. As a consequence, your application should do its best to determine the OpenGL configuration it should use.

Many things can be configured in OpenGL and some default values favor visual quality over performance, so you should become familiar with what your application can configure. For example, as far as textures are concerned, the documentation on http://www.khronos.org/opengles/sdk/docs/man/ will show you all the different parameters you can set.

Multiple APKs

Because you may want to support multiple texture compression formats (optimizing as much as possible for each device) and because mipmaps take more space, your application may go over the size limit defined by Android Market (currently 50 megabytes).

If that happens, you have basically three options:

• Reduce the size of your application, for example by supporting only ETC1 texture compression.
• Download textures from a remote server after the application is installed.
• Generate multiple APKs, each with its own set of textures.

Android Market lets you publish different APKs for your application, each targeting a different configuration. For example, you could have one APK using only ETC1 textures and a second one using only PVRTC textures—that is, an APK optimized for PowerVR-based Android devices. These APKs will share the same Android Market listing, and Android Market will take care of selecting the right APK for each device. Users won't have to worry about downloading and installing the right APK as the selection is automatic and transparent.

Of course, textures may not be the sole reason why you would need or want to release multiple APKs. For example, you may want to release a smaller APK for older devices and a larger APK with more features for newer devices. While shipping multiple APKs is possible, it makes your maintenance and release process more complicated and therefore it is recommended you try to ship a single APK whenever possible.

Shaders

OpenGL ES 2.0 supports the OpenGL ES Shading Language, replacing the fixed function transformation and fragment pipeline of OpenGL ES 1.×. Based on C, the language allows you to have more control over the OpenGL pipeline by writing your own vertex and fragment shaders.

Like any C program, shaders can go from very simple to extremely complex. While there is no single rule you have to follow, you should try to reduce complexity as much as possible in your shaders as it can have a significant impact on performance.

Scene Complexity

Obviously, a complex scene will take longer to render than a simple one. An easy way to increase the frame rate therefore is to simplify the scenes to render while maintaining an acceptable visual quality. For example, as you have already seen with textures, objects that are further away can be less detailed and can be made of fewer triangles. Simpler objects will use less memory and bandwidth.

Culling

Even though GPUs are good at geometry and can determine what has to be rendered, your application should do its best to eliminate objects that are outside of the viewing frustum so it does not send draw commands for objects that will simply be discarded because they are not visible.

There are numerous methods to cull objects or even triangles and while these methods are outside the scope of this book, a lower-than-expected frame rate may be caused by poor culling. For example, it is pretty easy to quickly eliminate objects that are behind the camera.

Render Mode

By default, the OpenGL renderer keeps rendering the scene, regardless of what might have changed. In some cases, the scene may not change between frames, and you may want to explicitly tell the renderer to only render the scene when you request it. You can achieve this by changing the GLSurfaceView's render mode.

You can set a GLSurfaceView's render mode by calling setRenderMode() with one of the following two values:

• RENDERMODE_CONTINUOUSLY
• RENDERMODE_WHEN_DIRTY

When the render mode is set to RENDERMODE_WHEN_DIRTY, the renderer will render the scene when the surface is created, or when GLSurfaceView.requestRender() is called.

Power Consumption

One of the great qualities of modern GPUs is that they can fall asleep or at least slow down very quickly during periods of inactivity. For example, the GPU can shut down (partially or fully) between two frames if it has nothing to do for a while, reducing power consumption and therefore increasing battery life.

Once you have achieved an acceptable frame rate for your application, you may still want to optimize further if only to reduce power consumption. The faster a frame is being rendered, the sooner the GPU can idle and the longer the battery life (and the longer a user can use your application). For some applications, one of the great ways to increase battery life with OpenGL is to render frames only when the scene changed (as described above with the RENDERMODE_WHEN_DIRTY render mode).