There are times when JavaScript isn’t the best language to use to implement parts of your IoT product. Fortunately, you don’t need to choose either JavaScript or C (or C++) to build your product: you can choose both. XS in C is a low-level C API provided by the XS JavaScript engine so that you can integrate C code into your JavaScript projects (or JavaScript code into your C projects!).
Performance – High-level languages, including JavaScript, can’t outperform optimized native code at high-performance tasks. You can add your own optimized native functions and invoke them from your JavaScript code.
Accessing hardware features – As a general-purpose programming language, JavaScript doesn’t have built-in support for the unique features of your host hardware. You can implement your own functions and classes to configure and use these.
Reusing existing native code – You may have a large body of existing native code that works well for your products, and you’d prefer not to have to rewrite it in JavaScript. You can use that code in your JavaScript projects by using XS in C to bridge between it and your JavaScript code.
XS in C lets you work with JavaScript features from C. As you know, JavaScript has capabilities that C doesn’t directly support, such as dynamic types and objects. Working with these features using XS in C can be awkward, but it becomes straightforward as you get some practice and learn some common patterns. This chapter introduces XS in C through a series of examples that demonstrate different techniques to build a bridge between JavaScript and C code.
Note that many engines that implement a high-level programming language provide an API to bridge between that language and native code. The Java language defines the Java Native Interface (JNI) for this purpose, and the V8 JavaScript engine provides a C++ API.
The information introduced in this chapter is an advanced topic. It assumes you’re comfortable programming in C and have a solid understanding of the basic JavaScript concepts discussed in this book.
Installing the Host
These command lines don’t specify a development board (for example, esp32/moddable_two) because the examples use only common features of the microcontroller and don’t depend on board-specific features.
When you build the examples with mcconfig, both the JavaScript and the C code are built. If an error occurs building either, it’s reported to the command line.
Generating Random Integers
The first example of native code integration generates random integers. You saw in Chapter 9 that the random-rectangles example uses random numbers generated by the JavaScript built-in function Math.random. That example is less efficient than it could be because Math.random returns a floating-point value, forcing Poco to convert several floating-point values to integers for each rectangle. Floating-point operations are generally slow on microcontrollers, and here they have no benefit. The C standard library’s rand function generates random integers, and the $EXAMPLES/ch11-native/random-integer example begins by using rand to generate random integers for JavaScript code.
Creating a Native Function
This syntax creates a JavaScript function named randomInt which, when called, invokes the native function xs_randomInt, essentially building a bridge from JavaScript to C. The use of @ here is not standard JavaScript syntax but a language extension provided by XS to simplify adding native code to your projects. Consequently, this code is unlikely to compile or work the same with other JavaScript engines.
Implementing a Native Function
Listing 11-1.
The include preprocessor command brings in the header file for XS in C. (The file name, xsmc, stands for “XS Microcontroller.”) There’s also an xs.h header file that’s used by some code. The two headers provide equivalent functionality, but the functions in the xsmc.h header file are more efficient and therefore preferred for use on microcontrollers.
The native function prototype of xs_randomInt is used for all functions that implement native methods using XS in C. The JavaScript arguments are not passed as arguments to the C function. You’ll see later in this chapter how to access the arguments.
This example needs to return a value—the result of calling rand. The result of rand is an integer, so this example uses xsmcSetInteger, a function that assigns a native 32-bit integer value to a JavaScript value. Here the JavaScript value is xsResult, which refers to the return value of the function on the JavaScript stack.
Using the Hardware Random Number Generator
You’ve seen how simple it is to declare, call, and implement a simple native function. When you run the random-integer example, you see 100 random numbers from 0 to 2,147,483,647 traced to the debug console. But when you restart the microcontroller and run the example a second time, you see the exact same list of numbers. That’s not very random. Why does it happen?
The rand function is a pseudo-random number generator. It’s an algorithm to generate numbers that appear random; however, when you restart the microcontroller you also restart the pseudo-random number generator algorithm, causing it to generate the same sequence of numbers. You can use the srand function to have the algorithm start a different sequence, but you must provide srand with a different starting point on each restart. The most common way to initialize the sequence is to use the current time. Unfortunately, many microcontrollers, including the ESP32 and ESP8266, don’t know the time at startup, so this technique can’t be applied.
Fortunately, many microcontrollers, including the ESP32 and ESP8266, have hardware to generate random numbers, and these values are more random than those generated by rand. The $EXAMPLES/ch11-native/random-integer-esp example shows how to use the hardware random number generator.
Not all random numbers are guaranteed to be sufficiently unpredictable to be safely used in security solutions, such as the TLS protocol that protects network connections. (Random numbers that have this guarantee are called cryptographically secure.) You should always verify that the source of random numbers you use meets the security requirements of your project. This isn’t easy to do, but it’s important, as a weak random number generator is a vulnerability in your project’s overall security.
Listing 11-2.
Both the ESP32 and ESP8266 hardware random number generators return 32-bit unsigned values. The xsmcSetInteger function requires a 32-bit signed value. Consequently, using the hardware random number technique changes the result of the JavaScript randomInt function to return a range of values from –2,147,483,648 to 2,147,483,647. Recall that when you use rand, all values are positive. You could use xsmcSetNumber instead to return the unsigned 32-bit value as a floating-point number; however, that runs counter to the goal of returning a random number as an integer value.
Usually you want a random number within a certain range, and generating a value within a range requires a division or modulo operation. The division operation typically requires a floating-point operation, since the result may have a fractional part. The modulo operation can use an integer divide if both operands are integers. However, instead of requiring the caller of randomInt to efficiently restrict the return value to the desired range, you can modify the native function to do that.
The next section addresses these issues.
Restricting Random Numbers to a Range
The native function must first retrieve the range passed as the first argument. The arguments are accessed by index using xsArg. Arguments are numbered starting at 0, so the first argument is accessed as xsArg(0). If the caller didn’t pass any arguments, xsArg(0) throws an exception; therefore, it’s not usually necessary for your native code to check the number of arguments passed. (If your function needs to know the number of arguments, use the xsmcArgc integer value.) The exceptions thrown by XS in C are ordinary JavaScript exceptions, which may be caught with a familiar try and catch blocks in the JavaScript code.
Listing 11-3.
It’s important to include error checking in the native code that bridges between your JavaScript and C code. JavaScript programmers expect the language to be safe—there should be no way to crash or corrupt the device—and the JavaScript engine and runtime do their best to achieve this goal. Your native code must do the same. For example, should the JavaScript code pass 0 for the range, the result is undefined by the C language. The modulo operation with a 0 on the right side on ESP32 generates an IntegerDivideByZero exception and on ESP8266 an Illegal Instruction exception, both of which reset the microcontroller.
Listing 11-4.
Comparing Random Number Approaches
The returned values are integers, not floating-point, allowing for more efficient execution on microcontrollers.
The returned values are efficiently limited to a requested range.
The numbers are more random because they use a hardware random number generator.
The native code isn’t portable. It builds successfully for only two microcontrollers.
You must build your native code as part of a host.
Native code is more complex to implement and debug and requires additional specialized knowledge.
When you have the option of adding native functionality to your project, you should base your decision on a balance of the advantages and the disadvantages.
The BitArray Class
JavaScript typed arrays, such as Uint8Array and Uint32Array, enable you to work with arrays of 8-, 16-, and 32-bit integer values using a minimum of memory. The BitArray class implements a 1-bit array—that is, an array that stores only the values 0 and 1. This is useful for efficiently storing a large number of samples received from a digital input.
This section introduces two variations of BitArray, each with the same JavaScript API. The first one uses a JavaScript ArrayBuffer to store the bits, while the second uses native memory allocated with the C calloc function.
Listing 11-5.
The first argument to both get and set is the index of the bit in the array to get or set. The index of the first array element is 0. The final line of the example toggles the value of the bit at index 3.
Using Memory Allocated by ArrayBuffer
Listing 11-6.
The constructor allocates an ArrayBuffer to hold the bit values. Because the memory of a new ArrayBuffer is always initialized to 0, no further initialization is needed. The number of bits to store is divided by 8 to determine the number of bytes needed and then rounded up using Math.ceil to ensure that there are enough bytes allocated when the number of bits isn’t evenly divisible by 8. The ArrayBuffer is assigned to the buffer property of the BitArray instance. The native implementations of get and set access the memory using the buffer property.
The get Function
Variables allocated using xsmcVars are accessed with xsVar, which is similar to xsArg but accesses local temporary variables instead of arguments to the function. The variables are automatically released when the native function that allocated them—in this case, xs_bitarray_get—returns. You should call xsmcVars only one time in a function, allocating all needed temporary variables at once.
The second argument, xsThis here, tells xsmcGet which object you want to retrieve the property from. The third argument, xsID_buffer here, specifies that the name of the property you want to retrieve is buffer.
The set Function
Listing 11-7.
Security Vulnerability
This implementation of BitArray, using memory allocated by ArrayBuffer, works well, but it has a critical flaw that makes it unsuitable for safe use in real products. The get and set functions don’t verify that the index argument is inside the bounds of the memory allocated. This enables code using this implementation of BitArray to read and write arbitrary memory on embedded devices, which can cause a crash or be used as the basis of a privacy attack. There are multiple ways to solve this problem; the next section discusses one of them.
Using Memory Allocated by calloc
The implementation of BitArray in the $EXAMPLES/ch11-native/bitarray-calloc example solves the security problem presented by the bitarray-arraybuffer example as just discussed. It stores the number of bits allocated by the constructor and then validates the index passed to the get and set calls against that stored value.
The BitArray implementation in the bitarray-calloc example uses calloc instead of ArrayBuffer to allocate memory. The memory allocated by these two approaches comes from two different pools of memory: memory allocated by calloc is taken from the native system memory heap, whereas memory allocated by ArrayBuffer is inside the memory heap managed by XS. Some hosts are configured with more free space in one of these pools than the other, which may influence your decision about where to allocate memory from. A little bit less code is required to work with the memory allocated by calloc, though that difference may not be significant.
The bitarray-calloc example illustrates some important techniques for integrating native code into your project. In addition to a native constructor, this BitArray class also has a native destructor to perform cleanup when an instance of the class is garbage-collected. In XS, an object with a native destructor is called a host object .
The Class Declaration
Listing 11-8.
The declarations of the get and set methods are the same as in the previous example, though the implementations are somewhat different.
The native constructor, destructor, and close functions are closely related. The next sections look at each in turn.
The Constructor
Listing 11-9.
Once the memory is allocated, the number of bits requested is stored at the start of the block. Because the memory is allocated using calloc, all bits are initialized to 0.
The call to xsmcSetHostData stores a reference to the memory allocated with this host object. This pointer is then available to all native methods of the object, through a call to xsmcGetHostData. You might be tempted to simply store the bytes pointer in a global variable; however, that approach fails when there’s more than one instance of the object, since the two objects can’t share a single C global variable. Using xsmcSetHostData to store the data pointer means that the implementation of BitArray supports an arbitrary number of simultaneous instances.
The Destructor
This is the first time in this book that you’ve seen a destructor. They’re common in C++ in working with objects, but they’re not a visible part of the JavaScript language. Instead, JavaScript automatically frees the memory used by objects when they’re garbage-collected. The JavaScript engine doesn’t know how to free the resources your host object allocated, such as the memory allocated with calloc. Therefore, you must implement a destructor.
Listing 11-10.
The function prototype of a destructor is different from regular native method calls. Instead of being passed a reference to the XS virtual machine as the, it has an argument that’s a data pointer, the same value you passed to xsmcSetHostData.
Because there’s no reference to the XS virtual machine (no the argument), you can’t make calls to XS in C. For example, you can’t call xsmcGetHostData, which is why the data pointer is always passed to the destructor function. That also means your destructor can’t create new objects, change the values of properties, or make function calls to the object. These limitations are necessary because the destructor is called from inside the garbage collector when such operations are unsafe.
The value of data may be NULL. This happens, for example, when the memory allocation in the constructor fails. As you’ll see in the next section, it also happens after the close method is called. Therefore, a good practice is to always check that the data argument isn’t NULL in your destructor before using it, as this example does.
The close Function
Chapters 3 and 5 contain examples of JavaScript objects that have a close method. This method releases any native resources—memory, file handles, network sockets, and so on—that the object owns. If the object isn’t explicitly closed, those resources are eventually released when the garbage collector determines that the object is no longer in use. However, there’s no way to know when the garbage collector will make that determination, which means it may be a very long time until the resources are freed. The close call solves this problem by giving code a way to explicitly free those resources.
Listing 11-11.
- 1.
The call to xsmcGetHostData retrieves the data pointer that was allocated in the constructor and associated with this object by the call to xsmcSetHostData.
- 2.
The data pointer is passed to the destructor, which does the work of releasing the resources.
- 3.
The call to xsmcSetHostData sets the saved data pointer to NULL. This ensures that, should close be called twice, the data pointer is freed only once.
The get and set Functions
Listing 11-12.
The implementation of set applies the same changes described for get in this section and so is not repeated here.
The length Property
The typed array classes include a length property in their instances which, as in instances of Array, indicates the number of elements in the array. This value is useful when you’re iterating over the array. Because this implementation of BitArray stores the number of bits allocated, it can also provide a length property.
Listing 11-13.
Advantages to This Approach
It validates the input values, eliminating the ability of sloppy code to cause a crash and of malicious code to breach privacy.
It provides a length property, making it more convenient to work with.
It uses system memory to store the bit data, reducing the memory used in the memory heap managed by the JavaScript engine.
It uses the host data feature of XS in C to keep track of the memory buffer, requiring less code and running faster than using a JavaScript property.
Wi-Fi Signal Notifications
You’ve learned how to implement a class to manage native resources as a host object. This next example shows how to make calls from C code back to JavaScript and how to configure a host object using a dictionary. Both these techniques are used by many of the host objects in the Moddable SDK.
The $EXAMPLES/ch11-native/wifi-rssi-notify example implements the WiFiRSSINotify class, which lets you register callbacks to invoke when the Wi-Fi signal strength crosses above and below a specified threshold. You might use this in your product to give the user an indication of when Wi-Fi is likely to perform well or to throttle the amount of network traffic you generate when the signal is weak. The class could be implemented entirely in JavaScript using Timer together with the net module introduced in the “Getting Network Information” section of Chapter 3. This implementation using native code is a bit more efficient and provides a convenient starting point to show how to configure your host object from a dictionary and how to invoke callback functions.
The Test Code
Listing 11-14.
Listing 11-15.
The WiFiRSSINotify Class
Default functions for the onWeakSignal and onStrongSignal callbacks are not part of the class. Before invoking a callback, WiFiRSSINotify confirms that the instance has a property with the callback’s name.
The Native RSSINotifyRecord Structure
Listing 11-16.
threshold – The RSSI threshold below which the signal is considered weak and at which the signal is considered strong.
state – The WiFiRSSINotify instance is always in one of three states: kRSSIUnknown when it’s created and then either kRSSIWeak or kRSSIStrong. This state is used to eliminate redundant callbacks when the state has not changed.
timer – A native timer used to implement polling.
the – A reference to the XS virtual machine that contains the WiFiRSSINotify instance. It’s used to invoke callbacks from the timer.
obj – A reference to the WiFiRSSINotify object that’s used to invoke callbacks from the timer. The type of this field, xsSlot, is used by XS to hold any JavaScript value. The xsArg, xsVar, and xsGet functions that you already know return values of type xsSlot.
Additional details about how these fields are used are provided in the following sections.
The Constructor
Listing 11-17.
Listing 11-18.
Listing 11-19.
Listing 11-20.
The call to modTimerAdd creates a timer that first fires after 1 millisecond and then fires at the interval specified by poll. When the timer fires, it calls the checkRSSI native function, passing it the value of rn. A later section shows how the native callback retrieves this value and invokes the JavaScript callbacks.
That’s the end of the xsTry block. Even in this relatively simple object, there are two exceptions that the constructor itself generates. In addition, the calls to xsmcToInteger throw exceptions when passed a value that can’t be converted to an integer. These many potentials for exceptions make it important for the constructor to ensure that no memory or other resources are orphaned if an exception is thrown. Using xsTry with xsCatch often helps with this.
You can only pass xsRemember a value in storage that your code allocated. If you call xsRemember with values such as xsThis, xsArg(0), xsVar(1), or other XS-provided values, it silently fails. As you might expect, there’s a corresponding xsForget call that needs to be called in close. The memory where the object is stored, rn->obj here, must persist until xsForget is called and therefore must not be a local variable in the constructor.
The Destructor
Listing 11-21.
The close Function
Listing 11-22.
The call to xsForget can’t be made in the destructor because the destructor can’t use XS in C, as explained previously.
The Callback
Listing 11-23.
Listing 11-24.
Listing 11-25.
Invoking a JavaScript function from native code requires a valid JavaScript stack frame. When a native method is called from JavaScript, XS has already created that stack frame. The checkRSSI function isn’t called by XS, but by modTimer, and therefore must set up the stack frame itself. It does this by calling xsBeginHost before the callback. It calls xsEndHost afterward to remove the stack frame that xsBeginHost creates. Both functions take the, a reference to the JavaScript virtual machine, as their sole argument. Between xsBeginHost and xsEndHost, you can make calls to XS in C as usual.
Listing 11-26.
You use xsCall1 to call functions with one argument (and xsCall0 to call functions with no arguments, xsCall2 for functions with two arguments, and so on, up to xsCall9).
Additional Techniques
You now know how to invoke native code from JavaScript code and JavaScript code from native code, giving you the power to integrate native code and scripts in whatever way makes the most sense for your project. This section briefly introduces several important topics that you may find useful when integrating native code into your own JavaScript-powered products. Along with discussing a variety of techniques to help you build the bridge between your native and JavaScript code, it includes warnings about some common mistakes.
Debugging Native Code
As you develop increasingly complex native code, you may need to debug that code. Although you may not have a native debugger available, your code can interact with xsbug.
Accessing Global Variables
Getting a Function’s Return Value
Getting Values
Listing 11-27.
All of these functions fail if the JavaScript value can’t be converted to the requested type. For example, xsmcToArrayBuffer fails if the value is a string.
Special care is required when working with the pointers to strings and with ArrayBuffer pointers. See the section “Ensuring Your Buffer Pointers Are Valid” for details.
Setting Values
Listing 11-28.
Listing 11-29.
Listing 11-30.
Determining a Value’s Type
The types returned by xsmcTypeOf are xsUndefinedType, xsNullType, xsBooleanType, xsIntegerType, xsNumberType, xsStringType, and xsReferenceType. Most of these correspond directly to JavaScript types you’re already familiar with. Notice, however, that there are types for both integers and numbers (floating-point values). While JavaScript itself uses the Number type for both, XS stores them as distinct types, as an optimization. If your native code checks whether a JavaScript value is of type Number, it needs to check for both xsIntegerType and xsNumberType.
Other useful prototypes defined by XS that may be used with xsmcIsInstanceOf include xsFunctionPrototype, xsDatePrototype, xsErrorPrototype, and xsTypedArrayPrototype. For a complete list, see the xs.h header file in the Moddable SDK.
Working with Strings
You’re guaranteed that strings you receive from XS are valid UTF-8. You must ensure that any strings you pass to XS are also valid UTF-8.
XS treats a null character (ASCII 0) as the end of the string, so don’t include any null characters in your strings. (Since the C language also uses the null character to terminate a string, this should be familiar.) Your code probably doesn’t intentionally create invalid UTF-8 strings or include null characters in a string, but they can sneak in when you import strings from a file or a network connection; it’s a good practice to validate these strings before passing them to XS.
In JavaScript, strings are read-only. No functions are provided to change the content of a string. You could choose to break this rule in your native code—but don’t! Doing so would break a fundamental assumption that JavaScript programmers rely on. Furthermore, it could cause a crash, as some strings are stored in read-only flash memory and attempting to write to them causes the microcontroller to reset.
The string pointer returned from xsmcToString can be invalidated when you make other calls using XS in C. The next section explains the details.
Ensuring Your Buffer Pointers Are Valid
When you call xsmcToString or xsmcToArrayBuffer, they don’t return a copy of the data; they return a pointer into an XS data structure. This behavior is important on microcontrollers, where the extra time and memory required to make a copy are unacceptable. The pointer may become invalid when you make a call to XS in C that causes the garbage collector to run. The garbage collector cannot free the ArrayBuffer or string, because they’re in use. However, the garbage collector may move the data structure when it compacts the memory heap to make more space by combining areas of free space.
Never use a pointer returned by XS in C after making another call to XS in C. This may seem challenging, but all the examples so far in this chapter have done exactly that.
Make a copy of the data. While this approach is not optimal, it’s occasionally necessary.
Integrating with C++
XS in C enables you to bridge not only between C and JavaScript code but also between C++ and JavaScript code. Although both JavaScript and C++ support objects, the details of how they implement objects and their features are quite different. Therefore, it’s usually unrealistic to try to create a direct mapping between your C++ classes and your JavaScript classes. Instead, design your JavaScript classes to make sense to JavaScript programmers and your C++ classes to make sense to C++ programmers. The bridge code you write using XS in C can translate between the two.
Using Threads
JavaScript is a single-threaded language; for this reason, the XS JavaScript engine is also single-threaded. This means that all calls to a single JavaScript virtual machine, as represented to native code by the, should be made from the same thread or task. You shouldn’t call XS in C from an interrupt or a thread other than the one that created the virtual machine.
Techniques that provide multitasking execution of JavaScript code, such as the Web Workers class, are built outside the JavaScript language. The Moddable SDK supports a subset of the Web Workers class on the ESP32, which enables several JavaScript virtual machines to coexist, each in their own thread. Each virtual machine is single-threaded, but several machines may run in parallel. The implementation of Web Workers for ESP32 respects the requirement that each individual JavaScript virtual machine is single-threaded.
Conclusion
The ability to bridge between JavaScript and native code using the XS in C API opens the door to many new possibilities for your projects. It enables you to optimize memory use, improve performance, reuse existing C and C++ code libraries, and access unique hardware capabilities. However, using XS in C is considerably more difficult than working in JavaScript, and consequently more error-prone. As a rule, using as little native code as practical tends to minimize the risks.
The XS in C documentation is a complete reference to the API. It’s part of the Moddable SDK.
All the classes in the Moddable SDK that access native capabilities are implemented using XS in C. If you’re curious about how they work, the source code is there for you to read and learn from.