Guts of the IoT: Building NodeMCU from Source for the ESP8266

Introduction

This is Part 1 of an advanced series on connecting to and working with the 'Internet of Things'. This project involves some pretty high-level concepts like editing source files and cross-compilation so be forewarned... this is not for the faint of heart. I recommend having some familiarity with compiling software from source in the Linux terminal before attempting this project.

What a time to be alive! Phones, refrigerators, socks, and even water bottles have microprocessors in them these days. As the Internet of Things materializes around us, it's getting cheaper and easier to connect arbitrary devices and sensors to the world wide web. One of the most exciting technologies for inexpensive IoT connectivity is the ESP8266 wireless microcontroller, from Chinese manufacturer Espressif. Originally marketed as a WiFi-enabled UART bridge for microcontrollers, the ESP8266 is actually a 32-bit Tensilica Xtensa LX106 processor running at 80 MHz with a full WiFi stack.
When the online community quickly realized this chip's potential, they started translating documentation to English and building tools to use it as a full-fledged embedded processor. The ESP8266 comes in several hardware variants but this article will focus on a breakout board for the ESP8266-12 called the NodeMCU devkit. I'm using  version 0.9 -- the updated version 1.0 is out in the wild but I haven't gotten my hands on it yet. They are mighty similar, with a few differences in the firmware and the USB-serial chip. If you have a Devkit V1.0 and this tutorial works for you, let me know in the comments. In any case, the Devkit is a breakout board for the ESP8266-12 and contains a CH341 USB-serial bridge (an alternative to the ubiquitous FTDI FTL232R), a micro-USB connector, and some buttons. Mine looks like this:

To get a hold of the NodeMCU, I ordered mine off of Ebay for about $10. Alibaba also has a wide selection to choose from. Still, if you prefer a reputable US distributor, Adafruit and Sparkfun each have their own breakout boards for the ESP8266-12. The firmware binary file you compile here will work on any ESP8266-12 board you choose, but the way to load it onto your respective board may vary. Due to the explosion of interest in this chip, many community-supported methods of programming exist to interact with it, for example there's an Arduino add-on, a port of MicroPython, and of course bare-metal C and C++. This article will focus on compiling and installing the NodeMCU firmware onto the device and setting it up to connect to WiFi, interact with the GPIO pins, and read a potentiometer input from the ADC.

 

Supplies Needed:

• 1x Computer running Linux (Ubuntu 15.04 Vivid Vervet)
• 1x NodeMCU Devkit VERSION 0.9 with ESP8266-12

 

Notes on Linux:

I'm running a 64-bit version of Ubuntu 15.04 Vivid Vervet with Linux kernel 3.19.0-22-generic. The commands used should be fairly system agnostic, but just keep in mind that you may have to adjust your commands when setting up dependencies -- different package managers like apt-get, pacman, and yum may have differing naming conventions for their packages.

 

Why Bother with NodeMCU and it's Source Code?

NodeMCU is based on the interpreted Lua scripting language and allows event-based programming of the ESP8266. Firmware binaries can already be customized and built online, so strictly speaking, you don't have to compile from source. However, since we're not scared of getting our hands dirty, we're going to learn how to strip away the cruft with more precision and also give ourselves more control and transparency over what's actually going in your hardware. So, with that said, the first and possibly least exciting part of dealing with a new processor is getting the toolchain and its dependencies set up so it can actually be programmed. This tutorial will focus on downloading the NodeMCU firmware source and building tools from github, selecting only the specific modules we want, compiling the module, and flashing it to the ESP8266 from a Linux environment. The commands apply specifically to Ubuntu 15.04 Vivid Vervet but could very easily be adapted to other versions and distributions -- just be aware that there could be subltle differences.

Getting Ready to Get Ready 

1) Support for the CH341 USB-serial bridge is built into the mainline Linux kernel but Windows and OSX users need to install drivers. Verify the device is functioning by plugging in the NodeMCU-Devkit and typing to following into the terminal:

dmesg | grep tty

You should see something like:

[50859.311796] usb 2-1.2: ch341-uart converter now attached to ttyUSB0

This is the serial port we will be communicating with when we send code to the ESP8266.

2) Install dependencies for compilation and communication to ESP8266:

sudo apt-get install make unrar autoconf automake libtool gcc g++ gperf flex bison \
texinfo gawk ncurses-dev libexpat-dev python python-serial sed git libtool-bin screen

The software development kit that we're going to use is the esp-open-sdk. It contains the tools to cross compile code for the Xtensa processor as well as the hardware abstraction library and esptool, a python script that can convert and upload code to the device.

3) Create a user-owned directory and clone the esp-open-sdk repository from Github:

mkdir ~/.opt && cd ~/.opt

git clone --recursive https://github.com/pfalcon/esp-open-sdk.git # --recursive since this repo contains more repos

4) Compile a standalone version of the open SDK with vendor SDK files built in. Then add the SDK's bin directory to your PATH environment variable:

cd esp-open-sdk

make STANDALONE=y

Sit back and relax.... it's gonna be a while.

The make command will take a while so go grab a cup of coffee or something while you wait. It's worth noting that while the standalone SDK is slightly easier to write code for locally, there are a few caveats. See the note from pfalcon here on portability and licenses. Then add the newly built tools' binary folder to the PATH environment variable. That way your system knows where to look when you try to compile Xtensa code.

echo 'PATH=$PATH:/opt/esp-open-sdk/xtensa-lx106-elf/bin' >> ~/.profile

PATH=$PATH:~/.opt/esp-open-sdk/xtensa-lx106-elf/bin

Now that we have the tools to make everything, now we can actually get the NodeMCU firmware and configure it to our specific needs.

5) Clone the Nodemcu firmware:

cd ~/.opt

git clone https://github.com/nodemcu/nodemcu-firmware.git

cd nodemcu-firmware

Nuts and bolts time! A common problem many users of NodeMCU are discussing is running out of room on the chip for program. Since the ESP8266 only has 64 KiB of instruction RAM and 96 KiB of data RAM, space is more or less at a premium. Why would you install I2C and u8g graphics libraries if you have no need for them for this project? By default, all the modules are built, so we're going to comment out the ones we don't want at this time:

6) Select modules to compile and install
Using your favorite editor (Sublime Text 3 with Vim key bindings thankyouverymuch), open ../nodemcu-firmware/app/include/user_modules.h
For this Hello World kind of a project, you only need the node, file, gpio, wifi, net, pwm, timer, adc, and uart modules. I edited my user_modules.h to look like this (I added some of my own comments for clarity):

                    #ifndef __USER_MODULES_H__
#define __USER_MODULES_H__

#define LUA_USE_BUILTIN_STRING  // for string.xxx()
#define LUA_USE_BUILTIN_TABLE  // for table.xxx()
#define LUA_USE_BUILTIN_COROUTINE // for coroutine.xxx()
#define LUA_USE_BUILTIN_MATH  // for math.xxx(), partially work
// #define LUA_USE_BUILTIN_IO   // for io.xxx(), partially work

// #define LUA_USE_BUILTIN_OS  // for os.xxx(), not work
// #define LUA_USE_BUILTIN_DEBUG // for debug.xxx(), not work

#define LUA_USE_MODULES

// Check out http://www.nodemcu.com/docs/ for more info on what you need and what you don't. 
#ifdef LUA_USE_MODULES
#define LUA_USE_MODULES_NODE  // NodeMCU system related functions (restart, info, etc)
#define LUA_USE_MODULES_FILE  // File system for managing lua files 
#define LUA_USE_MODULES_GPIO  // General purpose I/O. Good stuff!
#define LUA_USE_MODULES_WIFI  // Nowhere near as cool of a chip without this
#define LUA_USE_MODULES_NET  // Do things with TCP and UDP
#define LUA_USE_MODULES_PWM  // Pulse width modulation -- gonna dim some LEDs!
//#define LUA_USE_MODULES_I2C  // Peripheral communications library
//#define LUA_USE_MODULES_SPI  // Peripheral communications library
#define LUA_USE_MODULES_TMR  // Timer functions like delay() and all that 
#define LUA_USE_MODULES_ADC  // 10-bit analog-to-digital converter. Read pots & batts.
#define LUA_USE_MODULES_UART  // Need it to talk to the durn thing through CH341
//#define LUA_USE_MODULES_OW  // 1-Wire peripheral communications library
//#define LUA_USE_MODULES_BIT  // Bitwise manipulation (xor, not, and, or, etc)
//#define LUA_USE_MODULES_MQTT  // light weight messaging protocol on top of TCP/IP
//#define LUA_USE_MODULES_COAP  // IDK... No docs :'(
//#define LUA_USE_MODULES_U8G  // Neato burrito graphics lib: github.com/olikraus/u8glib
//#define LUA_USE_MODULES_WS2801 // IDK... No docs :'( More LEDs???
//#define LUA_USE_MODULES_WS2812 // Cool RGB LED strips
//#define LUA_USE_MODULES_CJSON  // Provides JSON support for Lua
//#define LUA_USE_MODULES_CRYPTO // Super secret spy stuff
//#define LUA_USE_MODULES_RC   // IDK... No docs :'(
//#define LUA_USE_MODULES_DHT  // IDK... No docs :'(

#endif /* LUA_USE_MODULES */

#endif /* __USER_MODULES_H__ */

                  

 

With the Devkit Version 0.9, we also have to modify the ../nodemcu-firmware/app/include/user_config.h to enable some v0.9-specific settings. Find line 4 and uncomment it so it looks like this:

#define DEVKIT_VERSION_0_9 1     // define this only if you use NodeMCU devkit v0.9

 

Let's Build It Already! 

7) Compile NodeMCU:

cd ~/.opt/nodemcu-firmware # make your way back to the firmware dir if you weren't there already

make # this might take a while

8) Add user to dialout group and flash the firmware to the device:
Before you can upload the compiled image or communicate with the device, you have to have permission to access to the serial port. /dev/ttyUSB0 is part of the dialout group, so to add yourself, type:

sudo adduser $USER dialout

Now you should be able to upload the NodeMCU image to your device:

make flash # this assumes the tty device is /dev/ttyUSB0

Phew! If everything worked properly and you didn't see any errors, you are now ready to start prototyping!
Note: in the guts of the make flash command, the makefile actually calls a Python script called esptool.py to write the compiled binaries to specific parts of memory in ESP8266. Esptool is its own separate project but NodeMCU cloned it and incorporated it into theirs. The literall command that is called by 'make flash' is:

../tools/esptool.py --port /dev/ttyUSB0 write_flash 0x00000 ../bin/0x00000.bin 0x10000 ../bin/0x10000.bin

So if your CH341 happens to be at a different location than /dev/ttyUSB0 (or you used Frightanic's custom build tool), just replace it with your specific device name and binaries.

Parting Notes

Now that the guts of our ESP8266 are now full of beautiful, Lua-interpreting goodness, next time we can focus on actually getting it to do something useful. We will discuss using Linux Screen to connect to NodeMCU's Lua interpreter and how to write scripts to interact with WiFi and the internet, as well as the onboard 10-bit ADC and GPIO. I'll also show you Luatool and other programs for interacting with the ESP8266 -- very exciting times ahead.