© Peter Hoddie and Lizzie Prader 2020
P. Hoddie, L. PraderIoT Development for ESP32 and ESP8266 with JavaScripthttps://doi.org/10.1007/978-1-4842-5070-9_5

5. Files and Data

Peter Hoddie1  and Lizzie Prader1
(1)
Menlo Park, CA, USA
 
Nearly every product has some data that it needs to ensure is available across restarts of the device, even if power is lost. On microcontrollers, flash memory is typically used for this non-volatile storage (NVS) memory. The same flash memory that holds the code of your application also stores the data your application uses and the data it creates. Here are some kinds of data your application might store:

  • Read-only data, such as the images that make up the user interface of your product or files containing static web pages served from an embedded web server

  • Small pieces of data that are both read and written—for example, user preferences and other long-term state

  • Large collections of data that are created as your product monitors operations—for example, while gathering data from its sensors

On computers and mobile devices, it’s common to use the file system for most, if not all, data storage needs. However, because of the constraints of embedded systems—code size limitations, highly constrained RAM, and severe performance constraints—the firmware often doesn’t even include a file system.

This chapter explains three different ways to work with stored data on embedded systems: files, preferences, and resources. The final section introduces direct access to flash memory, an advanced technique that provides the greatest flexibility.

When building your product, choose the data storage methods that best match your needs. Before assuming that files are the right choice, consider preferences and resources, which are lighter-weight ways to work with stored data.

Installing the Files and Data Host

You can run all of the examples referred to in this chapter by following the pattern described in Chapter 1: install the host on your device using mcconfig, then install example applications using mcrun.

The host is in the $EXAMPLES/ch5-files/host directory. Navigate to this directory from the command line and install it with mcconfig.

Files

The ESP32 and ESP8266 use SPIFFS (Serial Peripheral Interface Flash File System) for their file system in flash memory. SPIFFS is designed specifically for working with the NOR (NOT OR) flash memory used with many microcontrollers. While SPIFFS is nowhere near as fully featured as the file systems on a computer, it provides all the fundamental features you’ll need.

When using files on an embedded device, it’s important to keep in mind these limitations of the file system implementation:

  • SPIFFS is a flat file system, meaning that there are no real directories. All files are together in the SPIFFS root directory.

  • File names are limited to 32 bytes.

  • There are no file permissions or locks. All files may be read, written, and deleted.

  • The length of time for a write is unpredictable. It’s often fast, but when the file system needs to consolidate blocks it may block for some time.

This section focuses on accessing files using SPIFFS, which is available without adding any hardware and has a relatively small code size. On the ESP32, these same APIs may also be used to access an SD memory card formatted using the FAT32 file system.

File Classes

All access to the file system is done using classes in the file module:
import {File, Iterator, System} from "file";
The file module exports these three classes, as explained in detail in the following sections:

  • The File class performs operations on individual files, including read, write, delete, and rename.

  • The Iterator class returns the contents of a directory. On a flat file system like SPIFFS, Iterator is only available for the root directory.

  • The System class provides information about the file system storage, including total amount of storage and free space available.

File Paths

File paths are the strings you use to identify files and directories. The file module uses the slash character (/) to separate the parts of a path, as in /spiffs/data.txt.

Although SPIFFS is a flat file system with no subdirectories, it’s accessed with a root of /spiffs/ instead of / to support embedded devices that have more than one file system—for example, a built-in flash file system and an external SD card.

On the desktop simulator, the root varies based on the host platform. For example, on macOS, the default file system root is /Users/Shared/. When you write code intended to work in more than one environment, you can use the predefined value in the mc/config module to find the root for your host platform.
import config from "mc/config";
const root = config.file.root;

Because there may be multiple file systems, this root is just a convenient default place for files, not necessarily the sole file system available.

Each file system may have a different limit for the length of a file or directory name. Use the System.config static method to retrieve the maximum length for names in a specified root.
const spiffsConfig = System.config("/spiffs/");
let name = "this is a very long file name.txt";
if (name.length > spiffsConfig.maxPathLength)
    throw new Error("file name too long");

File Operations

This section describes methods that perform operations on a file, including deleting, creating, and opening. There are no methods to read or write the full content of a file, as that would often fail due to memory limitations; later sections introduce techniques for reading and writing.

Determining Whether a File Exists

Use the static exists method of the File class to determine whether a file exists:
if (File.exists(root + "test.txt"))
    trace("file exists ");
else
    trace("files does not exist ");

Deleting a File

To delete a file, use the static delete method of the File class:
File.delete(root + "goaway.txt");

The delete method returns true if successful and false otherwise. If the file doesn’t exist, delete returns true rather than throw an error, so there’s no need to surround its invocation with a try/catch block. The method does throw an error if the delete operation fails, but this happens only in rare circumstances, such as when the flash memory is worn out or the file system data structures are corrupt.

Renaming a File

To rename a file, use the static rename method of the File class. The first argument is the full path of the file to rename, whereas the second argument is only the new name.
File.rename(root + "oldname.txt", "newname.txt");
Note

The rename method is only for renaming a file. On file systems that support subdirectories, rename cannot be used to move a file from one directory to another.

Opening a File

To open a file, create an instance of the File class. The File constructor’s first parameter is the full path of the file to open. The optional second parameter is true to open in write mode (creating the file if it doesn’t exist) and either absent or false to open in read mode. Here’s an example of opening a file in read mode:
let file = new File(root + "test.txt");
The following example opens a file in write mode, creating the file if it doesn’t exist:
let file = new File(root + "test.txt", true);

The File constructor throws an error if there’s an error in opening a file, such as trying to open a nonexistent file in read mode.

When you’re done accessing a file, close the file instance to free the system resources it’s using:
file.close();

Writing to a File

This section introduces techniques for writing data to a file. You can use the File class to write both text and binary data. A file must be opened in write mode, or write operations will throw an error. To open in write mode, pass true as the second argument to the File constructor.

The file system automatically grows the file when you write data beyond the current size. There’s no support for truncating a file. To reduce a file’s size, create another file and copy the needed data into it from the original file.

Writing Text

The write method of the File class determines the kind of data you want to write from the type of the JavaScript object you pass to the call. To write text, pass a string. The following code from the $EXAMPLES/ch5-files/files example writes a single string to a file:
let file = new File(root + "test.txt", true);
file.write("this is a test");
file.close();

Strings are always written as UTF-8 data.

Writing Binary Data

To write binary data to a file, pass an ArrayBuffer to write. The following code from the $EXAMPLES/ch5-files/files example writes five 32-bit unsigned integers to a file. The values are in a Uint32Array, which uses an ArrayBuffer for its storage. The call to write gets the ArrayBuffer from the buffer property of the bytes array.
let bytes = Uint32Array.of(0, 1, 2, 3, 4);
let file = new File(root + "test.bin", true);
file.write(bytes.buffer);
file.close();
To write bytes (8-bit unsigned values), pass an integer value as the argument (see Listing 5-1).
let file = new File(root + "test.bin", true);
file.write(1);
file.write(2);
file.write(3);
file.close();

Listing 5-1.

Getting File Size

To determine the size of a file in bytes, you first open the file and then check its length property:
let file = new File(root + "test.txt");
let length = file.length;
trace(`test.txt is ${length} bytes `);
file.close();

The length property is read-only. It cannot be set to change the size of the file.

Writing Mixed Types

The write method lets you pass multiple arguments to write several pieces of data in a single call. This executes a little faster and keeps your code a little smaller. The following example writes an ArrayBuffer, four bytes, and one string in a single call to write:
let bytes = Uint32Array.of(0x01020304, 0x05060708);
let file = new File(root + "test.bin", true);
file.write(bytes.buffer, 9, 10, 11 12, "ONE TWO!");
file.close();
A hex dump of the file after the write looks like this:
04 03 02 01 08 07 06 05   .... ....
09 0A 0B 0C 79 78 69 32   .... ONE
84 87 79 33               TWO!

You might expect the first four bytes to be 01 02 03 04, but remember that instances of TypedArray, which includes Uint32Array, are stored in the host platform’s byte order, and the ESP32 and ESP8266 microcontrollers are both little-endian devices.

Reading from a File

This section introduces techniques for retrieving data from a file. You can use the File class to read both text and binary data. Most files are one or the other—all binary or all text data—though this is not required.

The File class supports reading a file in pieces, which enables you to control the maximum memory used when reading from the file.

Reading Text

Sometimes it’s useful to retrieve the entire contents of a file as a single text string. You do this by calling the read method with a single argument of String, which tells the file instance to read from the current position to the end of the file and put the result in a string. The following code from the $EXAMPLES/ch5-files/files example reads the contents from the test.txt file created earlier:
let file = new File(root + "test.txt");
let string = file.read(String);
trace(string + " ");
file.close();

The read method always starts reading from the current position. In this case, since the file has just been opened, the current position is 0, the start of the file.

Reading Text in Pieces

You may also use the read method to retrieve parts of a file, to minimize peak memory use. The optional second argument to read indicates the maximum number of bytes to read. This is the number of bytes that are read, with one exception: if reading the requested number of bytes would pass the end of the file, the text from the current position to the end of the file is read.

The example in Listing 5-2 reads a file in ten-byte pieces and traces them to the console. It compares the position property to the length property to determine when it has read all data from the file.
let file = new File(root + "test.txt");
while (file.position < file.length) {
    let string = file.read(String, 10);
    trace(string + " ");
}
file.close();

Listing 5-2.

On a computer, you might memory-map the file to simplify access to the data; however, that approach is not generally available on microcontrollers, as they typically lack an MMU (memory management unit) to perform the mapping. If you want to memory-map read-only data, resources are a good alternative, as explained later in this chapter.

Reading Binary Data

To read the entire file as binary data, call read with the single argument ArrayBuffer. The following code from the $EXAMPLES/ch5-files/files example reads the contents from the test.bin file created earlier:
let file = new File(root + "test.bin");
let buffer = file.read(ArrayBuffer);
file.close();
As when reading text, the binary read starts from the current position, which is 0 when the file opened, and continues to the end of the file. The data is returned in an ArrayBuffer. The following example wraps the returned buffer in a Uint8Array and displays the hexadecimal byte values on the console:
let bytes = new Uint8Array(buffer);
for (let i = 0; i < bytes.length; i++)
    trace(bytes[i].toString(16).padStart(2, "0"), " ");

Reading Binary Data in Pieces

The read method may also be used to retrieve binary data from arbitrary locations in a file. The example in Listing 5-3 reads the last four bytes of the file and displays the result as a 32-bit unsigned integer. The read location is specified by setting the position property to four bytes from the end of the file.
let file = new File(root + "test.bin");
file.position = file.length - 4;
let buffer = file.read(ArrayBuffer, 4);
file.close();
let value = (new Uint32(buffer))[0];

Listing 5-3.

Directories

The SPIFFS file system implements only a single directory, the root directory. Other file systems, such as FAT32, support an arbitrary number of subdirectories. In all cases, you use the Iterator class of the file module to list the files and subdirectories contained in a directory.

Iterating over Directories

Retrieving a list of the items in a directory is a two-step process. First you create an instance of the Iterator class for the directory over which to iterate; then you call the next method of the iterator to retrieve each item. When all items have been returned, the iterator returns undefined. Listing 5-4 from the $EXAMPLES/ch5-files/files example traces the files and directories contained in the root directory to the console.
let iterator = new Iterator(root);
let item;
while (item = iterator.next()) {
    if (undefined === item.length)
        trace(`${item.name.padEnd(32)} directory `);
    else
        trace(`${item.name.padEnd(32)} file ${item.length}` +
              "bytes ");
}

Listing 5-4.

The next method returns an object with properties that describe the item. The name property is always present. The length property is present only for files and indicates the number of bytes in the file. There’s no separate property to indicate whether the item is a file or directory, as the presence of the length property is sufficient for this purpose.

The iterator instance has a close method, which may be called to free the system resources used by the iterator. This is not usually necessary, however, because the iterator implementation automatically frees any system resources when it reaches the end of the items.

The Iterator class returns one item at a time, rather than a list of all items, to keep memory use to a minimum. The order in which items are returned depends on the underlying file system implementation. In the general case, you cannot assume, for example, that the items are returned in alphabetical order or that directories are returned before files.

Iterating with JavaScript Iterators

The JavaScript language provides an iterator feature that makes it easier to write code that uses iterators. For example, you can use the for-of loop syntax to iterate through the items. This language feature works with any instance that implements the iterator protocol , which the file module’s Iterator class does. This approach is a little more concise for you to code, at the expense of using a little more memory and CPU time. Listing 5-5 adapts Listing 5-4 to use the JavaScript iterator.
for (let item of new Iterator(root)) {
    if (undefined === item.length)
        trace(`${item.name.padEnd(32)} directory `);
    else
        trace(`${item.name.padEnd(32)} file ${item.length}` +
              "bytes ");
}

Listing 5-5.

Where iterators really shine is as inputs to functions that operate on iterators. For example, if you need an array containing all the items contained in a directory, you can simply pass the iterator instance to Array.from.
let items = Array.from(new Iterator(root));

Getting File System Information

The file module’s System object contains an info method to provide information about each file system root. You use this method to determine the total number of bytes of storage available and the number of bytes currently in use.
let info = System.info(root);
trace(`Used ${info.used} of ${info.total} `);

Preferences

Preferences are another tool for storing data on a microcontroller in your IoT product. They’re much more efficient than files but also much more limited. A file is well suited to storing a lot of information, whereas a preference stores only small pieces of information. Often in your product, you only need to keep track of a handful of user settings, and preferences are all you need for those situations; you may even exclude the file system entirely from your product.

Another advantage of using preferences is their reliability. The implementations of preferences for ESP32 and ESP8266 take steps to ensure that the preference data is not corrupted even if power is lost while the preferences are being updated. That level of reliability is more difficult to achieve in a file system, because the data structures are more complex.

The Preference Class

The preference module provides access to preferences. To use preferences in your code, import the Preference class from the preference module.
import Preference from "preference";

The JavaScript Preferences API introduced in this chapter is the same between microcontrollers; however, the underlying implementation is different. For example, on the ESP32, preferences are implemented using the NVS library in the ESP32 IDF SDK, whereas on the ESP8266, preferences are implemented by the Moddable SDK because there’s no system-provided equivalent. Since the implementations are different, there also are differences in behavior. The following sections note the differences you need to keep in mind.

Preference Names

Each preference is identified by two values, a domain and a name. These are similar to a simple file system path: the domain is like the directory name, and the name is like the file name. For example, consider a Wi-Fi light, where you want to save the user settings to restore when the power is turned on. You could use a light domain for all the light state preferences, with on, brightness, and color for names. The light might keep statistics data (such as the number of times the light has been turned on) in another domain, such as stats.

The domain and name values of a preference are always strings. Names are limited to 15 bytes on the ESP32 and 31 bytes on the ESP8266.

Preference Data

Preferences are not intended to replace a file system; it’s a common mistake to try to use them that way. Because the size of each individual preference is limited, as is the total storage available for all preferences, they’re far less general than a file system.

Each preference has a data type: a boolean, a 32-bit signed integer, a string, or an ArrayBuffer. Floating-point numeric values are not supported. The string type is often the most convenient to use but is also often the least efficient use of storage space. It you need to combine several values in a single preference, consider using an ArrayBuffer.

When you write a value, the type of the value is established based on the data provided. To change the type, write the value again. When you read a value, the value returned has the same type as the value that was written.

Note these differences between preference data on the ESP32 and on the ESP8266:

  • On the ESP32, the preference data space is configurable and is set to 16 KB in the hosts used in this book. On the ESP8266, the space for preference data is 4 KB.

  • On the ESP32, each preference may be up to 4,000 bytes of data; on the ESP8266, this value is limited to 64 bytes. If you’re writing code that you expect to run on several different microcontroller platforms, you need to design your preference values for the 64-byte data size.

Reading and Writing Preferences

Because preferences are just small pieces of data with a type, they’re much easier to read and write than a file. Listing 5-6 from the $EXAMPLES/ch5-files/preferences example writes four preferences to the example domain. The type of each value is used as the preference name. The set implementation determines the type of the preference based on the value passed in the third argument.
Preference.set("example", "boolean", true);
Preference.set("example", "integer", 1);
Preference.set("example", "string", "my value");
Preference.set("example", "arraybuffer",
               Uint8Array.of(1, 2, 3).buffer);

Listing 5-6.

Use the static get call to retrieve preference values, as shown in Listing 5-7. The type of the value returned matches the type of the value used in the set call.
let a = Preference.get("example", "boolean");     // true
let b = Preference.get("example", "integer");     // 1
let c = Preference.get("example", "string");      // "my value"
let d = Preference.get("example", "arraybuffer");
        // ArrayBuffer of [1, 2, 3]

Listing 5-7.

If no preference is found with the specified domain and name, the get call returns undefined:
let on = Preference.get("light", "on");
if (undefined === on)
    on = false;

Deleting Preferences

Use the delete method to remove a preference:
Preference.delete("example", "integer");

No error is thrown if a preference cannot be found with the specified domain and name. If there’s an error while updating flash memory to remove the preference, delete throws an error.

Don’t Use JSON

When building products in JavaScript for the web or computers, it’s common to store preferences using JSON—an approach that’s extremely easy to code and is flexible. It’s tempting to do the same when creating an embedded product using JavaScript; however, although it works in some products, it isn’t recommended, because it’s more likely to lead to failures later in the development process. Consider the following:

  • Storing preferences in a JSON file requires that your project include a file system—a large body of code that takes up some of the limited space in your flash memory.

  • A JSON object must be loaded into memory all at once, which means that accessing one preference value requires enough memory to hold all preference values.

  • Loading the JSON string data from a file and then parsing it to JavaScript objects takes considerably more time than just loading one value from a preference.

  • File systems are generally less error-resilient to power failures than preferences. Consequently, there’s a higher chance that user settings will be lost.

Using JSON may also seem like a good way to store several values in a single preference. This does work, but it has two limitations that make it an inadvisable choice in many cases. First, because preference data is limited to just 64 bytes on some devices, you cannot combine many values this way. Second, the overhead of the JSON format almost certainly means the preference data uses more storage than other methods. For example, the following code uses 24 bytes of storage to store three small integer values as JSON:
Preference.set("example", "json",
               JSON.stringify({a: 1, b: 2, c: 3}));
In contrast, this example requires just three bytes by using Uint8Array:
Preference.set("example", "bytes",
               Uint8Array.of(1, 2, 3).buffer);
Reading the values from the JSON version is easier:
let pref = JSON.parse(Preference.get("example", "json"));
Reading the values from the more storage-efficient version requires an additional line of code:
let pref = new Uint8Array(Preference.get("example", "bytes"));
pref = {a: pref[0], b: pref[1], c: pref[2]};

Security

The preference module provides no guarantees about the security of preference data. The domain, name, and value may be stored “in the clear” without any encryption or obfuscation. As with user data in files, you should take appropriate steps in your product to ensure that user data is adequately protected. Examples of sensitive user data that are commonly stored in IoT products are Wi-Fi passwords and cloud service account identifiers. At a minimum, you should consider applying some form of encryption to these values so that they cannot be read by an attacker scanning the flash memory of the device.

Some hosts do provide encrypted storage for preference data. With additional configuration, this is available on the ESP32, for example.

Resources

Resources are a tool for working with read-only data. They’re the most efficient way to embed large pieces of data in your project. Resources are usually accessed in place in flash memory, where they’re stored, and therefore use no RAM no matter how large the resource data is. The Moddable SDK uses resources for many different purposes, including TLS certificates, images, and audio, but there’s no restriction on the kind of data you can store in a resource.

The $EXAMPLES/ch5-files/resources example hosts a simple web page defined by mydata.dat, which is included as a resource. After you run the example, open a web browser and enter the IP address of your device, and you’ll see a web page that says “Hello, world”.

Adding Resources to a Project

Including a resource in your project requires two steps:
  1. 1.

    You add a file containing the resource data to your project. Often the resource files are placed in a subdirectory such as assets, data, or resources, but you can store them anywhere you like.

     
  2. 2.

    You add the file to the resources section of your manifest to tell the build tools to copy the file’s data to a resource.

     
Listing 5-8 is from the resources example’s manifest. It includes just one resource, mydata.dat, from the directory containing the manifest.
"resources": {
    "*": [
        "./mydata"
    ],
},

Listing 5-8.

The data file must have a .dat extension. However, the file name in the manifest must not include the extension; the build tools automatically locate your file with the .dat extension. It’s important that you do not include several files with the same name but different extensions (for example, mydata.dat and mydata.bin), as the tools may not find the one you expect first.

This chapter describes resource data that’s copied directly from your input file to the output binary without any changes. The build tools also have the ability to apply transformations to the data, such as converting images to a format optimized for your target microcontroller; Chapter 8 explains how to use resource transformations.

Accessing Resources

To access a resource, import the Resource class from the resource module:
import Resource from "resource";
You call the Resource class constructor with the path of the resource from the manifest. Note that the path always includes the file extension—.dat in this case.
let data = new Resource("mydata.dat");
The Resource constructor throws an error if it cannot find the requested resource. If you want to check whether a resource exists before calling the constructor, use the static exists method:
if (Resource.exists("mydata.dat")) {
    let data = new Resource("mydata.dat");
    ...
}

Using Resources

The Resource constructor returns the binary data as a HostBuffer. A HostBuffer is similar to an ArrayBuffer but, unlike an ArrayBuffer, the data of a HostBuffer may be read-only and consequently may be located in flash memory.

To get the number of bytes in a resource, use the byteLength property, just as with an ArrayBuffer:
let r1 = new Resource("mydata.dat");
let length = r1.byteLength;
Also as with an ArrayBuffer, you cannot access the data of a HostBuffer directly but must wrap it in a typed array or data view. The following example wraps a resource in a Uint8Array and traces the values to the console:
let r1 = new Resource("mydata.dat");
let bytes = new Uint8Array(r1);
for (let i = 0; i < bytes.length; i++)
    trace(bytes[i], " ");
This example wraps the resource in a DataView object to access its content as big-endian 32-bit unsigned integers:
let r1 = new Resource("mydata.dat");
let view = new DataView(r1);
for (let i = 0; i < view.byteLength; i += 4)
    trace(view.getUint32(i, false), " ");
Sometimes you want to modify the data in a resource. Because the data is read-only, you need to make a copy. The HostBuffer returned by the Resource constructor has a slice method that may be used to copy the resource data, in the same way as the slice method on an ArrayBuffer instance. For example, you could copy the entire resource to an ArrayBuffer in RAM as follows:
let r1 = new Resource("mydata.dat");
let clone = r1.slice(0);
The first argument to slice is the starting offset of the data to copy. The optional second argument is the ending offset to copy; if omitted, data is copied to the end of the resource. The following example extracts ten bytes of resource data starting at byte 20:
let r1 = new Resource("mydata.dat");
let fragment = r1.slice(20, 30);
The slice method supports an optional third argument, which is not provided by ArrayBuffer. This argument controls whether the data is copied into RAM. If it’s set to false, slice returns a HostBuffer referring to a fragment of the resource data, which is useful when you want to associate just part of a resource with an object without copying its data into RAM. For example, if there’s an array of five unsigned 16-bit data at offset 32 of the resource, you can create a Uint16Array that references it, as follows:
let r1 = new Resource("mydata.dat");
let values = new Uint16Array(r1.slice(32, 10, false));
You could achieve a similar result by using the optional byteOffset and length parameters of the Uint16Array constructor:
let r1 = new Resource("mydata.dat");
let values = new Uint16Array(r1, 32, 10);

The advantage of using slice is that it ensures that the full resource is unavailable to untrusted code with access to the values array. In the first of the preceding two examples, values.buffer has access to the entire resource, whereas in the second example it may be used only to access the five values in the Uint16Array.

Accessing Flash Memory Directly

All of the modules described in this chapter for storing and retrieving data—files, preferences, and resources—use the flash memory attached to the controller for data storage. Each approach for working with data in flash memory has its own benefits and limitations. In most cases, one of these approaches is a good fit for your product’s needs; in some situations, a more specialized approach may be more efficient. The flash module gives you direct access to flash storage. Using it well requires more work, but it’s worth the effort in some cases.

Warning

This is an advanced topic. Accessing flash memory directly is dangerous. You may crash your device or corrupt your data. You may even damage the flash memory, leaving your device unusable. Proceed with caution!

Flash Hardware Fundamentals

To be able to use the API provided by the flash module, it’s important to understand the fundamentals of the flash hardware.

The flash memory used with the ESP32 and ESP8266 microcontrollers is connected using an SPI (Serial Peripheral Interface) bus. Although reasonably fast to access, it’s still many times slower than accessing data in RAM.

Flash memory is organized into blocks (also called “sectors”). The size of a block varies depending on the flash memory component used. A common value is 4,096 bytes. When you’re reading and writing flash memory, you don’t usually need to be aware of the block size. However, the block size is important when you’re initializing flash memory.

The flash memory uses NOR technology to store data. This has the curious implication that an erased byte of flash memory has all bits set to 1, whereas it’s common to think of erased storage as being set to 0. You might think that you could simply set the freshly erased bytes to all zeros but, as you’ll see, that’s not a good idea with NOR flash memory.

When you write to NOR flash memory, you’re only writing 0 bits. Because the flash memory is erased to all 1 bits, this doesn’t matter on the first write. Consider two bytes (16 bits) of flash memory. They start out erased to all 1 bits.
11111111 11111111
Write two bytes to that, 1 and 2, and the result is straightforward:
00000001 00000010
The next step is where the result is unexpected. Here’s what happens when you then write the two bytes 2 and 1 to the same location:
00000000 00000000
The result is that both bytes are set to 0. Why? Remember that with NOR flash memory, a write sets only the 0 bits. Any bits in flash memory that are already set to 0 cannot be changed back to 1 with a write.

  • Flash 0. Write 0 => Flash 0.

  • Flash 0. Write 1 => Flash 0.

  • Flash 1. Write 0 => Flash 0.

  • Flash 1. Write 1 => Flash 1.

If a write can only change bits from 1 to 0, how are bits changed from 0 to 1? You use the flash erase method to do that. Unlike read and write, which may access any byte in the flash memory directly, erase is a bulk operation that sets all the bits in a block of flash memory to 1. You erase blocks aligned to the block size boundary, which means bytes 0 to 4,095 or bytes 4,096 to 8,191—not 1 to 4,096 because that’s not aligned to the start of a block, and not bytes 1 to 2 because that’s not a full block.

If you want to change one bit, you can read the entire block into RAM, erase the block, change the bit in RAM, and then write the block back. That works, but it’s slow, because erase is a relatively slow operation—many times slower than read and write. This approach also requires enough RAM to hold a full block, and there’s not always that much memory on a resource-constrained microcontroller. The biggest problem, however, is that flash memory wears out. Each block may be erased only a certain number of times, after which that block no longer stores data reliably; to preserve the device, you need to minimize the number of times you erase each block.

The good news is that the flash memory in your ESP32 or ESP8266 supports thousands, if not tens of thousands, of erase operations. The preference and file module implementations are aware of the limits and characteristics of NOR flash memory and take steps to minimize erases. If you’re accessing flash memory directly in a product that’s intended to be used for years, you need to do the same.

One commonly used strategy is incremental writes. In this approach, the current values are zeroed out and the new values written after the zeros in the block. This enables a single value to be updated many times without an erase. This approach is used by the preference module. The Frequently Updated Integer example later in this section explores the details of incremental writes.

Another common strategy is wear leveling. This approach attempts to erase each flash storage block the same number of times over the lifetime of the product, to ensure that no block (for example, the first block) wears out much sooner than the others due to more frequent access. The SPIFFS file system underlying the file module uses this technique.

Accessing Flash Partitions

The flash memory available to your microcontroller is accessed using the Flash class from the flash module:
import Flash from "flash";

Flash memory is divided into segments called partitions. For example, one partition contains your project’s code, another the preference data, and another the storage for the SPIFFS file system. Each partition is identified by a name.

To access the bytes in a partition, instantiate the Flash class with the name of the partition. When you install example applications using mcrun as introduced in Chapter 1, the byte code for the application is stored in the xs partition. The following line instantiates the Flash class to access it:
let xsPartition = new Flash("xs");

The partitions available to your code vary depending on the microcontroller and the host implementation. The xs partition that contains applications installed with mcrun is always available. The area used for the SPIFFS file system, named storage, is also generally always available; if you’re not using the SPIFFS file system in your project, you can use it for other purposes. Although these partitions are both present, their sizes vary by device.

On the ESP32, the ESP32 IDF from Espressif defines the partitions. The IDF provides a flexible partition mechanism that makes it possible for you to define your own partitions. On the ESP8266, the Moddable SDK defines the partitions, and they cannot easily be reconfigured.

On the ESP32, the Flash constructor searches the IDF partition map to match the partition name requested. Consequently, you can access the partition that contains the ESP32 preferences—which are implemented in the NVS library—with the name nvs, as declared in the partition map (the partitions.csv file in an IDF project).
let nvsPartition = new Flash("nvs");

Getting Partition Information

An instance of the Flash class has two read-only properties that provide important information about the partition: blockSize and byteLength .

The blockSize property indicates the number of bytes in a single block of the flash hardware. This value is often 4,096, but for robustness you should use the blockSize property rather than hardcode a constant value in your code. That way, your code can work unchanged on hardware that incorporates a different flash hardware component.
let storagePartition = new Flash("storage");
let blockSize = storagePartition.blockSize;

The blockSize property is important because it tells you both the alignment and the size of erase operations on the partition.

The byteLength property provides the total number of bytes available in the partition. The following example calculates the number of blocks in the partition:
let blocks = storagePartition.byteLength / blockSize;

The value of the byteLength property is always an integer multiple of the value of blockSize property, so the number of blocks is always an integer.

Reading from a Flash Partition

Use the read method to retrieve bytes from the flash storage partition. The read method takes two arguments: the offset into the partition and the number of bytes to read. The result of the read call is an ArrayBuffer. The following is an excerpt from the $EXAMPLES/ch5-files/flash-readwrite example:
let buffer = partition.read(0, 10);
let bytes = new Uint8Array(buffer);
for (let i = 0; i < bytes.byteLength; i++)
    trace(bytes[i] + " ");

This code retrieves the first ten bytes from the partition. It wraps the returned ArrayBuffer in a Uint8Array to trace the byte values to the console.

There are no restrictions on the offset and number of bytes to read, beyond the requirement that they’re within the partition. Specifically, a single call to read may cross a block boundary.

The read call copies the requested data from the partition into a new ArrayBuffer. Consequently, you should read flash memory in small fragments to use as little RAM as practical.

Erasing a Flash Partition

Use the erase method to reset all the bits in a flash partition to 1. The method takes a single argument, the number of the blocks to reset. This line erases the first block of the partition:
partition.erase(0);
The following code resets the entire partition. The erase operation is relatively slow; for a large partition—for example, the storage partition on the ESP8266—this operation takes several seconds.
let blocks = partition.byteLength / partition.blockSize;
for (let block = 0; block < blocks; block++)
    partition.erase(block);

Writing to a Flash Partition

Use the write method to change the values stored in the flash partition. This method takes three arguments: the offset at which to write the data into the partition, the number of bytes to write, and an ArrayBuffer containing the data. When the number of bytes to write is less than the size of the ArrayBuffer, only that number of bytes are written. The following example sets the first ten bytes of the partition to integers from 1 to 10:
let buffer = Uint8Array.of(1, 2, 3, 4, 5, 6, 7, 8, 9, 10).buffer;
partition.write(0, 10, buffer);

Keep in mind that writing sets only the 0 bits, as explained earlier in the “Flash Hardware Fundamentals” section. Therefore, it may be necessary to perform an erase before calling write.

Mapping a Flash Partition

On the ESP32, you have the option of memory-mapping the partition, which gives you read-only access to the contents of the partition using a typed array or data view constructor. To memory-map a partition, call the map method. The following code is taken from the $EXAMPLES/ch5-files/flash-map example:
let partition = new Flash("storage");
let buffer = partition.map();
let bytes = new Uint8Array(buffer);

The map property returns a HostBuffer that may be passed to a typed array or data view constructor to access the data. Memory-mapped partitions are a more convenient way to access data than the read call in some situations. Furthermore, because the data in the partition is not copied to RAM by the map method, RAM use is minimized.

The map method is unavailable on ES8266 due to hardware limitations that allow only memory mapping of the first megabyte of flash memory, the area reserved to store the firmware.

Example: Frequently Updated Integer

This section presents an example of directly accessing flash memory to maintain a 32-bit value more efficiently than is possible using a file or a preference. The example is for the situation where your product needs to update a value frequently in flash storage to ensure that it’s maintained reliably across reboots of the product.

The example uses a single block of flash memory. That’s typically 4,096 bytes, which is 1,024 times bigger than the 32-bit (four-byte) value being stored. The example takes advantage of the additional storage to reduce the number of erase operations, thereby prolonging the lifetime of the flash memory. For convenience, the block used is the first block of the storage partition, which prevents this example from being used with the SPIFFS file system.

The complete Frequently Updated Integer example is available at $EXAMPLES/ch5-files/flash-frequentupdate.

Initializing the Block

The first step is to open the storage partition:
let partition = new Flash("storage");
As shown in Listing 5-9, the next step is to check whether the block has been initialized. This is done by looking for a unique signature at the start of the block. If the signature is not found, the block is erased and the signature is written.
const SIGNATURE = 0xa82aa82a;
let signature = partition.read(0, 4);
signature = (new Uint32Array(signature))[0];
if (signature !== SIGNATURE)
    initialize(partition);
function initialize(partition) {
    let signature = Uint32Array.of(SIGNATURE);
    partition.erase(0);
    partition.write(0, 4, signature.buffer);
}

Listing 5-9.

Updating the Value

After the signature, the block has space to store 1023 copies of the counter. Listing 5-10 shows a write function that updates the value of the counter. It searches for the first unused 32-bit integer in the block and writes the value there. Recall that when a block is erased, all the bits are set to 1. That means any unused entries contain the value 0xFFFFFFFF (a 32-bit integer with all bits set to 1). If the block is full, it reinitializes the block and writes the value in the first free position.
function write(partition, newValue) {
    for (let i = 1; i < 1024; i++) {
        let currentValue = partition.read(i * 4, 4);
        currentValue = (new Uint32Array(currentValue))[0];
        if (0xFFFFFFFF === currentValue) {
            partition.write(i * 4, 4,                            Uint32Array.of(newValue).buffer);
            return;
        }
    }
    initialize(partition);
    partition.write(4, 4, Uint32Array.of(newValue).buffer);
}

Listing 5-10.

Reading the Value

The final part is the read function , shown in Listing 5-11. Like the write function, it searches for the first free entry. Once that’s found, read returns the value of the previous entry. If the search reaches the end of the block, the last value in the block is returned.
function read(partition) {
    let i;
    for (i = 1; i < 1024; i++) {
        let currentValue = partition.read(i * 4, 4);
        currentValue = (new Uint32Array(currentValue))[0];
        if (0xFFFFFFFF === currentValue)
            break;
    }
    let result = partition.read((i - 1) * 4, 4);
    return (new Uint32Array(result))[0];
}

Listing 5-11.

Benefits and Future Work

This example efficiently stores an integer value in flash memory. The value may be updated 1,023 times before the block needs to be erased. To understand the impact of this, consider a product that updates that value once a minute. That works out to 514 erase operations per year (60 * 24 * 365, which is 525,600 minutes per year, divided by 1,023 updates per erase rounds up to 514). Using a flash chip with support for 10,000 erase operations (a conservative estimate), the product has about a 19.5-year lifetime. If each write operation required an erase, the same product would wear out in only 7 days (60 * 24 * 7 is 10,080 writes per week).

The careful reader has noticed two limitations of this example: if power is lost in the write function after the erase and before the write, the current value will be lost; and the value cannot be set to 0xffffffff because that value is used to identify unused entries in the block. Solutions to these shortcomings are possible and are left as exercises for the reader.

Conclusion

In this chapter, you learned several different ways to store information in an embedded product. Files, preferences, and resources are the three primary ways to store data, and each is optimized for a different use of storage. You can use any combination of these approaches in your product. When designing your product, consider your storage needs to determine which approaches to use to make optimal use of available storage. Some situations are so specialized that none of these standard storage techniques are optimal; to address those cases, this chapter showed how flash memory works so that you can create your own storage methods.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.117.148.105