Pixel Rate Impacts Frame Rate

In modern mobile apps and web pages, the frame rate is the fundamental measure of graphics performance. Higher frame rates provide smoother animation. The GPU in computers and mobile phones is so powerful that it’s able to update every pixel of the display on every frame. For a variety of reasons, a microcontroller just can’t do the same; however, it’s possible to achieve animations at 30 or even 60 frames per second (fps).

Because a microcontroller cannot render a high frame rate when updating the entire display, the solution is to update only a subset of the display. You can design your user interface so that only relatively small parts of the display are updated at any one time. This significantly reduces the work required of the microcontroller, so the user sees a smooth, high frame rate animation just as on a mobile app or web page.

To achieve a high frame rate using a microcontroller, it’s helpful to think in terms of pixel rate—the number of pixels updated per second. The ESP32 and ESP8266 use a SPI bus to communicate with the display, and this connection runs at 40 MHz, providing a pixel rate of about 1,000,000 pixels per second, about 15 fps. Achieving the full theoretical pixel rate is generally not possible because of other factors; still, if your application updates only about 40% of the pixels in each frame—a pixel rate of about 30,000 pixels per frame—it can achieve a reliable frame rate of 30 fps. On the QVGA (320 x 240) displays used in this book, 30,000 pixels is about 40% of the total display area, which is more than enough motion to create a smooth, compelling user interface. Updating only 10,000 pixels per frame achieves 60 fps.

You might expect that the area of the screen updated on each frame must be a single rectangle. That would limit the design possibilities by limiting motion to one area of the display. Fortunately, though, this isn’t the case. As you’ll soon learn, you can have several different areas updating simultaneously, which can give the user the impression of motion on the full screen even though only a fraction of the actual pixels are changing.

Drawing Frames

There are other advantages of retained mode renderers, but these three should be enough to convince you that the memory and complexity costs justify using a retained mode renderer like Poco instead of the more common immediate mode renderer. The quality of the user interface rendering is so much higher that users have the impression they’re using a higher-quality product—one that belongs alongside their computer and phone rather than in a computer history museum.

Scanline Rendering

A QVGA display has 76,800 pixels, which means that a display with 16-bit pixels requires 153,600 bytes of memory to store one full frame. The ESP8266 has about 80 KB of total memory—only enough for a half frame, if your IoT product doesn’t use any other memory! The ESP32 has much more, but still, holding an entire frame in memory uses 50% or more of the total free memory at startup. The displays used in this book include memory for a single frame, so the microcontroller doesn’t have to store the entire frame, but it does need memory in which to render the frame. To minimize the memory required, the Poco renderer implements scanline rendering , a technique that divides the frame into horizontal strips as small as a single row of pixels; after each strip is rendered, it’s immediately transmitted to the display. This approach is more complex than rendering the entire frame at once, but it reduces the rendering memory requirements for a single 16-bit QVGA display from 153,600 bytes to 480 bytes—one 240-pixel scanline at two bytes per pixel—a memory savings of 99.68%!

There’s some performance overhead for each strip rendered, so there’s a benefit to reducing the number of strips by increasing their size—but of course this also increases the memory needed. The performance benefit decreases somewhat with each line added to a strip, so increasing beyond about eight scanlines isn’t usually worthwhile. If your project has some free memory or requires the highest performance rendering, you may want to have Poco render a few scanlines at a time; the upcoming chapters explain how to configure this.

Many modern microcontrollers, including the ESP32 and ESP8266, are able to use SPI asynchronously to transmit data to the display, which means that the microcontroller can do other work while that data is being transmitted. Poco uses asynchronous SPI transmission to render the next section of the display while the previous section is being transmitted to the display, and this simple parallel processing allows for a significant performance boost. To use this technique, Poco must have enough memory to hold at least two scanlines in memory: the previously rendered scanline that’s now being transmitted and the current scanline that’s now being rendered. Because this technique provides such a significant performance increase, Poco allocates two scanlines by default.

Restricting the Drawing Area

To understand the challenges of calculating the smallest possible update area, let’s look at an example of a bouncing ball. In each frame, the ball moves some number of pixels. In Figure 8-1, the ball moves a few pixels down and to the right. The smallest rectangle that encloses the previous and current position of the ball is a good first estimate of the smallest possible area of the screen to update.

Now consider the case in which the ball moves a much longer distance (Figure 8-2): the smallest rectangle that encloses the previous and current positions includes many pixels that didn’t actually change, but they’re redrawn because they’re included in the area to be updated.

As shown in Figure 8-3, in this case Piu recognizes that it’s more efficient to update two separate areas: the area enclosing the previous location of the ball, which fills to the background color, and the area enclosing the current location of the ball.

Piu actually goes one step further. In the first example, where the ball moves only a small distance—a distance that causes the current position to overlap with the previous position—Piu recognizes that a single enclosing rectangle isn’t the smallest possible update area; consequently (as shown in Figure 8-4), it updates three separate rectangles in this case, which avoids unnecessarily updating many background pixels that haven’t changed.

The calculations involved in optimizing the drawing area for a single bouncing ball are already surprisingly complex, and they would be even more complicated in an application with several bouncing balls that sometimes overlap. Piu automatically calculates the minimal set of rectangles for you; this does require time and memory, but the performance boost it gives makes it worthwhile. That’s because rendering performance is largely limited by the pixel rate of your application and Piu is automatically minimizing the pixel rate of your code.

Pixel Formats

Most displays support a single type of pixel. The QVGA color displays used in most of the examples in this book use a color 16-bit pixel that has 5 bits for red, 6 bits for green, and 5 bits for blue. Your mobile phone probably has 24-bit color pixels, with 8 bits each for red, green, and blue. While both kinds of pixels are adequate for displaying full-color user experiences, the 24-bit color pixels are able to show 256 times more colors (16,777,216 vs. 65,536). That difference means that images on an embedded device may have a less refined appearance, especially in areas filled with similar colors, such as a sunset. This can be a problem for photos, but it’s generally not an issue for user interfaces driven by microcontrollers if the design of the interface takes this limitation into consideration.

In addition to 16-bit color, a few displays support only 8-bit color. This is much more limited, allowing only 256 colors. Each pixel contains 3 bits for red, 3 bits for green, and 2 bits for blue. It’s possible to build a reasonable user interface with a display that uses this type of pixel, but it takes some work to carefully select colors that look good within the limits. In some cases, it can be beneficial to use 8-bit color pixels on a display that supports 16-bit pixels. This clearly doesn’t improve quality, but it does reduce the storage space required by assets and the time needed to render images. If you find your project is struggling to fit into available storage space or if rendering performance isn’t quite what you need, using 8-bit color pixels on a 16-bit display may be a viable solution.

There are also 4-bit color pixels, but it’s so difficult to achieve a professional result with these that they aren’t addressed here. However, 4-bit gray pixels—which can display 14 levels of gray plus black and white—are very useful. An ePaper display that’s unable to display color needs only gray pixels; since most ePaper displays are capable of showing only a few levels of gray, a 4-bit gray pixel is sufficient. Grayscale rendering is even faster than color rendering. You can use 4-bit gray pixels with a 16-bit color display to save even more storage space. There are also 8-bit gray pixels, which can display 254 levels of gray plus black and white; these provide excellent quality, but for many practical purposes 4-bit gray rendering is almost indistinguishable in quality from 8-bit gray pixels.

Some displays are just black and white. These displays tend to be small and low quality and to be used more for industrial IoT products than for consumer IoT products. A 1-bit pixel is sufficient for these displays; however, rendering well at 1 bit per pixel is very difficult. The Poco renderer does not support 1-bit pixel displays directly. Instead, the display driver receives 4-bit gray pixels and then reduces the image to 1-bit when transmitting it to the display.

Configuring a Host for a Pixel Format

In Chapter 1, you learned how to build a host using the mcconfig command line tool. On the command line, you use the -p option to pass the name of the hardware platform you’re targeting—for example, -p esp32 to build for an ESP32. For device targets that include a display, such as development boards from Moddable and M5Stack, the default pixel format is automatically configured for you. For example, when you build for Moddable One, Moddable Two, or M5Stack FIRE, the pixel format is set to rgb565le, for 16-bit color pixels; for Moddable Three, which has an ePaper display, it’s set to gray16, for 4-bit gray pixels.

The ILI9341 driver also supports 8-bit color and 8-bit gray pixels. You can use those with mcconfig by specifying rgb332 and gray256, respectively, in the -f command line option.

Freedom to Choose a Display

While the large variety of pixel formats available can seem confusing, it gives you options when creating a product. You can choose the display that best meets your requirements for quality, cost, and size. Poco is able to render pixels that work with your display, so you don’t have to choose a display based on your software stack’s limitations. In the next section, you’ll learn how to automatically transform the graphics assets in your project to match the display you’re using.

Masks

The most common type of asset used to build user interfaces with Poco and Piu is a mask. A mask is a grayscale image; you can think of it almost as a shape. Because the mask contains gray pixels, and not just black and white pixels, it can have smooth edges. Figure 8-5 shows two versions of a circle, the first as a grayscale mask and the second as a simple 1-bit mask, with their edges magnified to show the difference; note the gray edges in the magnification of the grayscale mask.

When Poco renders a grayscale mask, it doesn’t draw it as an image. If it did, the white pixels would hide the background, as illustrated in Figure 8-6.

Instead, Poco renders masks by treating black pixels as solid (fully opaque), white pixels as transparent (fully invisible), and the grayscale levels between as different levels of blending. The result corresponding to Figure 8-6 is shown in Figure 8-7, where the black circle is superimposed on the background (which is visible through the transparent white pixels) and the gray edges of the circle blend with the background, eliminating any jagged edges.

You probably want to include color in your user interface, in which case gray images don’t seem like an obvious solution. However, Poco lets you draw grayscale masks in any color. The black pixels are replaced by the color you choose, and the gray pixels blend that color with the background. Figure 8-8 shows the same circle mask drawn in blue (which appears gray in printed versions of this book).

The ability to draw a single grayscale mask in a variety of colors is very powerful, as it enables a single graphics asset to be displayed in different colors. This reduces the number of assets needed, saving storage space in your project.

Figure 8-9 shows some examples of grayscale masks used as user interface elements.

As you know from the “Pixel Formats” section, Poco defines two different kinds of grayscale pixels: 4-bit and 8-bit. All Poco masks are 4-bit grayscale, which allows for the smallest storage size and fastest rendering without sacrificing much quality.

Uncompressed Masks

When drawing the masks for the user interface, it’s often convenient to group several related elements together in a single graphics file. Many graphic designers prefer to work this way since it makes modifying the masks faster and easier. Because Poco supports clipped rendering, it’s able to use only part of a mask when drawing, so you have the option to organize your graphics files this way. The masks in Figure 8-10, showing several different states of a Wi-Fi connection, are combined in a single graphics file.

You can compress these combined mask images as described previously. However, there’s a performance penalty for using compression with a mask containing multiple images. That’s because to render a part of the compressed image, the decompressor must skip over the parts of the image above and to the left of the target area, which takes additional time. For some projects, the storage size reduction benefit of compression is more important than the performance reduction. You can keep the mask uncompressed by adding it to your manifest in a *-alpha section rather than the *-mask section (see Listing 8-2). Of course, your manifest may include both *-mask and *-alpha, to compress some masks while leaving others uncompressed.

"resources": {

    "*-alpha": [

        "./assets/wifi-states"

    ]

}

Listing 8-2.

Fonts

Fonts are a unique challenge in embedded development. Your computer and mobile phone have dozens, if not hundreds, of fonts built in. One or more of those fonts include nearly all the characters defined in the Unicode standard, meaning that there’s no text character your devices can’t display. On a microcontroller, there are no built-in fonts; the only fonts available to your project are the fonts you include in your project.

There are many fonts available for your computer, and it would be convenient to be able to use those same fonts in your IoT products. Most, if not all, of the fonts on your computer are stored in a format based on the TrueType scalable font technology created by Apple (the OpenType font format is a derivative of TrueType). Rendering these fonts on a microcontroller is possible but challenging, and the amount of code, memory, and CPU resources needed for rendering makes it impractical for many projects. The examples in this book use a simpler font format, a high-quality bitmap font. A TrueType-compatible renderer is available on the ESP32 and is introduced in this section.

Converting TrueType Fonts to Bitmap Fonts

Even though it’s probably impractical to use TrueType fonts in all of your projects, you can still use the fonts on your computer, by using your computer to convert the TrueType fonts to a format that can be easily handled by a microcontroller. The TrueType font is rendered in a specific point size to a bitmap, with all the characters being stored in a single bitmap. The bitmap uses 4-bit gray pixels, rather than black and white, to maintain the anti-aliasing of the original font. In addition, a .fnt file is generated that maps between Unicode character codes and rectangles in the font bitmap. This font format that combines a bitmap image with a map file is called BMFont, for “bitmap font.” There are several variations of BMFont; the Moddable SDK uses the binary BMFont format. Figure 8-11 shows an example of what the Open Sans font at 16-point size looks like in the BMFont format.

Notice that the characters are not arranged in the same order as in the Unicode or ASCII standard. For example, the letters A, B, and C do not appear in sequence. Instead, characters are arranged by height, to make the bitmap image as small as possible by minimizing the amount of unused white space.

The tools that may be used to create these bitmap files are not part of the Moddable SDK. Glyph Designer from 71 Squared works well. The Moddable SDK includes a suite of pre-built fonts in the BMFont format, so you can get started developing without any additional investment in tools.

The BMFont format has two files for each font: an image file, usually in the PNG format, and a font map file with a .fnt file extension. These two files should have the same name with different file name extensions, as in OpenSans-Regular-16.png and OpenSans-Regular-16.fnt. To add these to your project, include the name in your project manifest as shown in Listing 8-3.

"resources": {

    "*-mask": [

        "./assets/OpenSans-Regular-16"

    ]

}

Listing 8-3.

Notice that the *-mask section is the same one used for compressed grayscale masks. Fonts included in this way are compressed too; however, rather than the entire image being compressed, each character is compressed individually. This enables each character to be decompressed directly, avoiding the overhead that would otherwise be required to skip over the pixels above and to the left of each glyph.

The compressed glyphs are merged with the data from the .fnt file into a single resource. This results in compact font files that still retain excellent quality and can be rendered efficiently. The preceding Open Sans 16-point font example uses just 6,228 bytes of storage in total for both the compressed characters and the font metric information needed for layout and rendering. Additionally, because fonts are stored using the same compression format as grayscale masks, they may also be rendered in any color.

The BMFont format does not require fonts to be grayscale. This format is popular with game designers because it enables them to include creative, colorful fonts in their games. Full-color fonts are supported by Poco and Piu. They’re not commonly used on microcontrollers because they require significantly more storage. In case you want to try them out, the Moddable SDK contains examples to get you started.

Color Images

While grayscale masks are a powerful tool for building user interfaces, there are times when you need full-color images. Poco uses uncompressed bitmaps to support color images. These bitmaps provide excellent quality and performance; however, they can be quite large and so are typically used sparingly in interfaces for microcontrollers.

Full-color images are fully opaque; they have no blended or transparent areas. Figure 8-12 shows the quack JPEG image from the preceding manifest fragment rendered in a simple user interface.

Figure 8-13 shows the result. The duck image is the same one used in Figure 8-12; however, an alpha channel has been added to mask the background. Consequently, only the duck is drawn when the image is rendered.

When you include images with an alpha channel in your project, the build tools create two separate images: the full-color image as if you’d put the image into the *-color section and the uncompressed alpha channel as a 4-bit grayscale mask. The color resource is named quack-color.bmp and the compressed mask resource is named quack-alpha.bm4. Figure 8-14 shows the alpha channel used to mask the drawing of the duck image.

When Poco and Piu render the image, they use both the color image and the mask. You’ll learn how to do this in the next two chapters.

Rotating in Software

The most common technique for rotating the user interface is to draw the full interface into an offscreen memory buffer as if the display was at the rotated orientation. Then, when the pixels are transmitted to the display, they’re transformed to match the hardware orientation. This approach isn’t feasible on low-cost microcontrollers because there’s not enough memory available to store a full frame in an offscreen buffer.

Poco takes a very different approach: it rotates all image assets to the desired orientation during the build so that they don’t need to be rotated at runtime. This rotation is performed at the same time as any needed pixel format conversions. Then, when the application on the embedded device makes drawing calls, Poco only needs to rotate the drawing coordinates to the target orientation. With those two steps done, Poco renders as usual and the result appears rotated on the display. This approach has almost no measurable runtime overhead—no additional memory is used, and only a trivial amount of additional code is run to perform the coordinate transformation—so it’s almost a no-cost feature. Because software rotation is implemented entirely in the Poco renderer, it works with all displays.

There’s one notable limitation to software rotation: the rotation is fixed at build time and therefore can’t be changed at runtime. Consequently, this technique is useful for situations where the IoT product user interface needs to be at a different orientation than the native orientation of the display but not when it needs to respond to a user action, such as turning the screen. Hardware rotation, when available, overcomes this limitation.

Rotating in Hardware

Hardware rotation uses features of the display controller to rotate the image as the display receives the pixels from the microcontroller. Using hardware rotation requires both the display controller and the display driver to support the capability. Hardware rotation is fully supported by the ILI9341 driver for MIPI-compatible display controllers.

Hardware rotation is performed entirely at runtime, so there’s nothing to define in your build command line or project manifest. In fact, it’s important that you do not use both hardware and software rotation in your project; they’re not designed to work together, so the results can be unpredictable.

When the hardware rotation is changed, the display does not change. The full content of the display must be redrawn before the user sees the changed orientation. If you update only part of the screen after changing the rotation, the user will see part of the display drawn in the original orientation and other parts with the new orientation.