IoT Development for ESP32 and ESP8266 with JavaScript
A Practical Guide to XS and the Moddable SDK
Heterosis, hybrid vigor, or outbreeding enhancement is the improved or increased function of any biological quality in a hybrid offspring. An offspring is heterotic if its traits are enhanced as a result of mixing the genetic contributions of its parents.
— Wikipedia, page 69105
The practical upshot of this is that if you take separate, inbred strains of, say, corn and breed them together, you get a great big vigorous plant, hence the name hybrid vigor.
The Moddable SDK, as described in this book, represents the hybrid vigor between embedded and JavaScript development. Using Moddable is a short path to very large amounts of corn.
If you’re an embedded developer, you delight in the ability to get close to the metal, to program on tiny, inexpensive devices without the affordances offered by development on large systems. Writing in C/C++ and/or assembly language gives you a great deal of control, but you often struggle with shoehorning functionality into these small devices, wrestle brittle development and debugging environments, and build ad hoc device-specific ways of updating code and managing resources. If the embedded device has a display or is capable of wireless communications, you need to track down the right tools to build, simulate, and test a wide variety of functionality in the absence of a rich underlying OS. A great deal of your energy goes into managing the constraints of these small systems rather than into the applications themselves.
If you’re a JavaScript developer, you delight in its productivity, versatility, and ubiquity. The language is forgiving to a novice yet extremely powerful in the hands of a master. It’s versatile enough to be used in front-end development as well as back-end infrastructure. Its huge community continually enriches the language and libraries and makes it arguably the most popular language ever. All this versatility and power comes with a price; until now a robust version of JavaScript has not scaled down to the point where it will fit on a small embedded device.
Moddable takes the best of these two strains and blends them together on an emerging class of powerful new microcontrollers. Embedded developers no longer have to regard JavaScript with envious eyes. They now have access to a versatile high-level language that still allows them to stay close to the metal and maintain tight control. JavaScript developers can reach ridiculously inexpensive devices with none of the friction of traditional embedded development, yet still work with affordances present on much larger systems.
It’s embarrassingly easy to be productive in the Moddable environment, with its rich SDK for managing secure web connectivity, Wi-Fi, Bluetooth, sound, graphics, user interfaces, and more; its elegant, full-featured debugging; and a wide variety of target hardware that ranges from inexpensive to nearly free.
My day job is investing in technology startups, and as such it pains me to see companies attempting embedded development with blunt tools. Moddable can save developers time and money and produce beautifully polished embedded applications. As an investor, I love tools that reduce risk and make companies more capital-efficient. As an avid maker, I love tools that make making even more fun.
If you develop embedded products, or if you’re a hobbyist or even just someone who loves writing code, Moddable’s hybrid vigor changes the game. Go grow some corn.
— Peter Barrett
This book is a hands-on guide to writing the software for IoT products. Each chapter is filled with compact, focused examples for you to learn from, study, run, and modify. When you finish this book, you’ll know the fundamentals of building sophisticated IoT products on low-cost hardware using modern JavaScript.
IoT products differ from traditional products in two ways: they have the ability to run software and they have the ability to communicate. Their communication is often over the internet, but it may be more local—for example, between products on your home Wi-Fi network or with your phone over a Bluetooth connection.
IoT products are often created by adding a microcontroller with Wi-Fi or Bluetooth capabilities to a traditional product. The cost of adding a microcontroller with communication capabilities is about one dollar today and continues to fall. At that price, nearly every product is going to be an IoT product—not just televisions and thermostats but also light bulbs, light switches, electric wall plugs, door locks, window shades, garage door openers, ceiling fans, rice cookers, refrigerators, and more.
The code in this book runs on the ESP32 and ESP8266 microcontrollers from Espressif, which offer remarkable power at an unprecedented cost. Unsurprisingly, they’re widely deployed in IoT products and extremely popular with makers and hobbyists. What you’ll learn in this book isn’t limited to these microcontrollers, however; it can be applied to a growing number of microcontrollers from manufacturers including Nordic, Qualcomm, and Silicon Labs.
Adding IoT hardware to a traditional product is the easy part; the hard part is the software. Software defines the product’s features and behavior. It determines whether the product is reliable and easy to use and whether it’s secure from external attacks and respects the privacy of users. Software decides what other products the product can communicate with, its energy use, the ease of adding new features over time, and much more.
Software is fundamental to IoT products, yet most of the industry continues to write the software for them using the same tools and techniques embedded software developers have used for decades. While the hardware has advanced by orders of magnitude, the software has not. That’s a problem, because much more is expected of the software in an IoT product today than the software in a digital thermostat from 1999.
JavaScript: A New Tool
This book introduces a new way of building the software for IoT products—a way that doesn’t try to reinvent the wheel by starting over. It adds one new tool to the many that embedded software developers have used for years: the JavaScript programming language. It may sound like a stretch to suggest that a programming language can transform the software of an IoT product, but it can. A modern high-level language is the perfect antidote to decades-old, low-level development methods.
JavaScript may seem an unlikely starting point for future generations of IoT products. After all, JavaScript began as a simple programming language to add a little interactivity to web pages at the dawn of the web. But as the web has evolved, JavaScript has evolved with it; it’s now a formally defined programming language standardized by an international committee with representatives from major companies, including Apple, Facebook, Google, Microsoft, Mozilla, and PayPal. The language has been made secure by two decades of attacks in the web browser. It’s been made powerful by the demands of increasingly sophisticated web pages. And it’s been made reliable and easy to use to meet the needs of millions of web developers around the world.
Developers working in JavaScript are incredibly productive and nimble. Within hours of a natural disaster, new websites appear with impressive features implemented in JavaScript. Major websites like Facebook and LinkedIn aren’t just built in JavaScript but deploy new features daily using JavaScript. Server-side JavaScript with Node.js now powers entire businesses, and many mobile apps are built in JavaScript.
JavaScript is ready for IoT developers to use. This book gets IoT developers ready to use JavaScript.
Recent Technical Advances
JavaScript wasn’t always a good fit for IoT. While the JavaScript engines built into web browsers are breathtakingly fast, that speed comes at a cost. Those engines are far too big to store in a low-cost microcontroller, and they require orders of magnitude more memory than is built into the hardware. The XS JavaScript engine is different: created by Kinoma and maintained by Moddable, it’s optimized for the constraints of the microcontrollers that power low-cost IoT products. XS is very small and still plenty fast.
To keep the engine small, you might expect XS to support only a subset of the JavaScript programming language, but that’s not the case. XS implements over 99% of the 2020 version of the JavaScript language specification; that’s more than any web browser. As an optimization, XS allows you to omit many features of the language to make the engine even smaller—but that’s your choice. All the features are there if you want to use them.
Having a small, efficient, and compatible JavaScript engine isn’t enough. The JavaScript language only performs computations; it needs a runtime to interact with the outside world. For a web page, the runtime is the web browser; for a web server, the runtime is Node.js. For IoT products, this book uses the Moddable SDK as the runtime.
The Moddable SDK includes a large collection of efficient modules for common tasks like communicating over the internet, controlling IoT hardware, managing security, and storing data. The Moddable SDK also has something unusual for an IoT runtime: deep support for high-quality, rich user interfaces on low-cost touch screens. While you may not yet be considering a screen for your IoT product, your future customers probably wish you would, because a screen makes your product easier to set up and use and enables it to provide more features.
For Embedded Programmers and Web Developers
There are no veteran IoT software developers, because the field is too new. We’re all still learning. Most IoT developers come from one of two backgrounds: there are developers building software for embedded systems, who are now being called on to create IoT products with connectivity, complex behaviors, and countless features; and there are developers working on the web, who are being called on to build IoT products on hardware that comes with resource and performance limits that are unimaginable on the web.
JavaScript is at the intersection of developers coming from the embedded and web software worlds. The JavaScript language itself embodies this intersection, arriving from the web with a syntax that looks a lot like the C and C++ languages long favored by embedded developers. JavaScript and C/C++ code can call each other, making it natural to combine both in a product. JavaScript has long been used for communication on the web, even giving rise to the JSON data interchange format commonly by IoT products to communicate.
This book is for developers with a background in either embedded software or web development. For experienced embedded developers unfamiliar with JavaScript, an early chapter introduces C and C++ programmers to JavaScript. For experienced web developers getting started in embedded systems, the book provides tips on memory and performance optimization unique to IoT product development.
Organization of This Book
Chapter 1 takes you through gathering all the hardware and software required for this book and running your first JavaScript application on a microcontroller. Along the way, it shows how to use the helpful features of xsbug, the JavaScript source-level debugger.
Chapter 2 is a fast-paced, practical introduction to JavaScript for developers who are already familiar with C or C++. The JavaScript language introduced here is the same language that’s used on the web, but since this book focuses on embedded systems, the chapter addresses some aspects of JavaScript that are seldom used by web developers.
Chapter 3 is all about your IoT device’s connection to the network, including different ways to connect, how to communicate using various network protocols, how to make secure connections, and the advanced topics of how to turn your device into a private Wi-Fi base station and how to use JavaScript promises with networking APIs.
Chapter 4 focuses on Bluetooth Low Energy (BLE), a wireless communication widely used between two devices in close proximity to each other. Products choose to use BLE instead of Wi-Fi if minimizing energy use is particularly important and when direct communication with another device, such as a mobile phone, is an acceptable alternative to internet access.
Chapter 5 explains how to work with stored data on embedded systems, given their code size limitations, constrained RAM, and performance constraints. It explains the three primary ways to store data—files, preferences, and resources—and introduces direct access to flash memory, an advanced technique that offers the greatest flexibility.
Chapter 6 gets you started on writing your own JavaScript code to interact with hardware. It includes examples that require just a few widely available, inexpensive sensors and actuators and shows you how to communicate with them directly using different hardware protocols.
Chapter 7 explains how to play sounds using an inexpensive speaker that’s easy to attach directly to an ESP32 or ESP8266. You’ll also learn how you can achieve higher-quality audio playback using an external I2S audio driver and how to choose the optimal audio format for your project.
Chapter 8 covers first the benefits and cost-effectiveness of adding a display to your IoT product and then the fundamentals of graphics on microcontrollers, including background on optimizations and constraints, information about how to add graphics assets to projects, and an introduction to various drawing methods. The next two chapters provide more details.
Chapter 9 discusses the Poco renderer through examples showing how to deliver high-quality, high-performance graphics on inexpensive microcontrollers. Poco is part of the Commodetto graphics library, which adds features including offscreen graphics buffers, bitmaps, and instantiation of graphics assets from resources, as demonstrated in some of the examples.
Chapter 10 elaborates on Piu, the object-oriented user interface framework that uses the Poco renderer to draw and simplifies the process of creating sophisticated user interfaces. The chapter provides an overview of how Piu works and introduces its key capabilities through a series of examples.
Chapter 11 presents the advanced topic of XS in C, the low-level C API provided by the XS JavaScript engine so that you can integrate C code into your JavaScript projects (or JavaScript code into your C projects). Using XS in C enables you to optimize memory use, improve performance, reuse existing C and C++ code libraries, and access unique hardware capabilities.
Finally, there’s a Glossary to remind you of the meanings of terms defined in this book.
What’s Next
If you’re a professional product developer, you can apply what you’ve learned to your IoT products.
If you operate a cloud service, you can build modules to help IoT product developers connect more easily with your service.
If you’re a sensor manufacturer, you can create modules to gather data from your sensors, streamlining the process of using those sensors in IoT products.
If you’re a maker or hobbyist, you can use your new knowledge and skills in your next project.
As you become proficient in using JavaScript to create IoT products, you may find that you spend less time and energy trying to make the product work at all and more time making it work better, do more, and be easier to use. That’s the real power of building IoT products in JavaScript.
One reason the JavaScript language has evolved so quickly for so long is that JavaScript developers have always shared their knowledge and experience in online communities. This sharing has resulted in a huge body of JavaScript source code being published under free and open source software licenses. This code is available to you to learn from, improve, and build on. As you create your own IoT projects using JavaScript, consider sharing them with others, so that other developers can learn from you and build on your work.
The GitHub repository for this book contains all the examples in the book, updates, and errata to correct any errors. You can open an issue to report a problem or ask a question.
The Moddable SDK includes reference documentation and examples for all the capabilities introduced in this book. It also includes the source code for the modules, so you can learn from how they work and enhance them to meet your needs. If you have questions or find a problem, you can open an issue.
The Moddable blog has in-depth articles about building IoT products using JavaScript. You can learn about the latest language capabilities supported by the XS JavaScript engine, security capabilities, and new features of the Moddable SDK.
Twitter is a great way to keep up with the latest developments. You can follow the authors of this book on Twitter at @lizzieprader and @phoddie and follow @moddabletech for the latest on the Moddable SDK, events, meetups, and new projects that developers have created.
Caroline Rose, editor – Caroline brought her legendary precision and clarity to this book. Everything about this book is better thanks to her patient work.
Mark Wharton, technical reviewer – Mark carefully reviewed the entire text, and then ran and reviewed every single example.
Masahiro Shioji, technical reviewer, Chapter 2 – Shioji-san brought the perspective of an embedded C developer to his thoughtful review.
Brian Friedkin, BLE advisor – Creator of the BLE implementation in the Moddable SDK, Brian made sure the explanations and examples in Chapter 4 are just right.
Patrick Soquet, advisor – As the inventor of the XS JavaScript engine, Patrick is the starting point for everything we know about embedded JavaScript.
Chris Krueger, illustrator – Chris turned our rough ideas and rougher sketches into the simple, clear images you see throughout the book.
Team Moddable – Our colleagues at Moddable have been incredibly supportive and patient over the many months it took to create this book.
is an engineer and entrepreneur focused on client software. He is recognized for crafting compact and efficient code that pushes the boundaries of user experience on consumer hardware. The software he and his teams have built has powered mass-market consumer products from companies including Apple, Palm, Sling, HP, Sony, and Whirlpool. Peter recognizes that the first users of any product are the developers creating it, and that those developers cannot build compelling consumer products on a foundation that’s unstable, complex, or confusing. He therefore champions investments in great tools and a simple runtime architecture.
Peter has founded several companies, including Kinoma, which merged into Marvell Semiconductor. He led QuickTime development at Apple during the 1990s as a Distinguished Engineer. He contributed to the development of the QuickTime file format and its adoption by ISO into the MPEG-4 standard. He is currently a member of the JavaScript language standards committee (ECMA TC39) and chair of ECMA TC53 for “Smart wearable systems and sensor-based devices.” Peter is particularly proud of his work putting both the KinomaJS framework and Darwin Streaming Server into open source. He continues to come to terms with the ten patents that bear his name.
is a software engineer at Moddable in the San Francisco Bay Area. She is an IoT skeptic working in the IoT space, hoping to make consumer IoT products and other embedded systems more open and customizable for the end user. She specializes in developing touch screen user interfaces for embedded systems and creating developer resources. Before Moddable, Lizzie worked as a developer relations engineer for the Kinoma team at Marvell Semiconductor, helping customers get the most out of Kinoma’s JavaScript-powered prototyping products. She earned her bachelor’s degree in Computer Science from UC Berkeley. When not sitting behind a computer, she enjoys being outdoors (in particular running, hiking, and swimming), reading, and playing the piano.
was born in England, was raised in Papua New Guinea and Australia, and worked as a software engineer for startups and large corporations in Australia, Japan, and the United States. He currently works at Amazon on the Alexa Automotive initiative. Since his early days programming the 6502 assembly language on Apple IIe and Commodore 64 computers, Mark excels at making the most of limited resources in constrained environments. Now in his spare time, Mark enjoys applying those skills to a new generation of low-cost, low-power devices using the Moddable SDK.
is a product creator who likes to bring technologies into easy-to-use products and release them to the world. He started in product development in the digital camera industry, at Sanyo Electric in Japan, and produced a QuickTime/MPEG-4 direct encoding video camera in collaboration with Peter Hoddie when Peter worked at Apple. Masahiro went on to spread the joy of creating something to university students, including operating a lab toward that end while continuing his own work in product development and new business development.
is a freelance technical writer and editor based in Palo Alto, California. With a career that has included 7 years working as a programmer, her focus has been on producing highly technical documentation, most notably as lead writer and editor of the original Inside Macintosh. Caroline was also Manager of Publications at NeXT and the editor of Apple’s quarterly technical journal, develop. Her subsequent freelance work began with coauthoring updated versions of Adobe’s PostScript and PDF reference manuals and went on to include clients such as AMD, Apple, Apress, Kinoma, Logitech, Nokia, and Sony. In her free time, Caroline enjoys playing guitar, singing, swimming, hiking, traveling, and working on her memoir.