on the software front, you will need the arduino ide installed on your machine. see https://www.arduino.cc/en/software.

on the hardware front, you will need the following components −

esp32 dev board − necessary

micro usb cable − necessary for powering and programming the esp32

mpu6050 module − optional (you will need it only for the chapter specific to mpu6050)

light dependent resistor (ldr) with a normal resistor of comparable resistance or any other analog sensor − optional (you will need these only for the adc chapter)

oled display − optional (you will need it only for the chapter specific to the oled interface)

jumper wires − optional (you will need these wires for interfacing esp32 to mpu6050, ldr and/or oled display)

a note on github usage

as mentioned in the overview part, a github link is provided with each chapter that contains a code walkthrough. several of these codes have been taken from the examples that come along with the esp32 board in arduino. therefore, you need not make any extra effort to run them on your local machine. you can find them from file −> examples in the arduino ide once you install the esp32 board in arduino (we have a separate chapter for that). wherever an example code has been used, the exact path of the example code is mentioned.

all the codes which are not present in examples can be found in the following repository − https://github.com/yash-sanghvi/esp32. now, if you wish to download and run these codes on your local machine, you need to do the following −

click on the green button that reads 'code'.

if you are new to git, you can simply download the zip file and extract it into the folder of your choice. the subfolders contain the required arduino (.ino) files which you can then open in the arduino ide and compile and flash into the esp32.
if you are familiar with git and have git installed on your machine, you can copy the https address which is https://github.com/yash-sanghvi/esp32.git, navigate to the folder where you wish to clone this repository, open your git command−line and enter git clone https://github.com/yash-sanghvi/esp32.git

if you are new to git, you may be wondering why we should make the effort of cloning the repository, when downloading and extracting the zip will have the same effect. the answer is that downloading a zip is a one time process. in case, some changes happen to this repository in the future, the downloaded version in your local machine can't reflect those directly. you will need to download the zip again. if you clone the repository, you can fetch all the future changes by simply calling git pull. there's a lot more you can do with a clone. if there are multiple branches within the repository, you can switch branches just by git checkout branch−name. if you are downloading a zip, you will need to download a separate zip for each branch. the bottom line is that cloning is much more convenient in general. however, for this particular use case, since we don't anticipate major changes to this repository in the future and the master branch will only be of importance to you, you can go ahead and download zip, if you aren't very familiar with git.

have you noticed that a lot of everyday use things have become 'smart' lately? there are smart tvs, smart acs, smart refrigerators, and whatnot. what does the 'smartness' of these devices refer to? while the answer is somewhat different for each device, one common element of the smartness is 'connectedness'. your tv gets connected to your wifi so you can stream shows which you earlier watched only on your mobile phone. your ac is connected to the internet. you can send a command from another city from your mobile phone, and the ac at your home will turn on/off. your watch is connected to your mobile phone (via ble) and you can answer calls using your watch itself. all the things you commonly deal with are connected, as if in a net. it is an inter-net of things.

the above paragraph would have given you a feel of iot. the definition of iot, as per wikipedia, is as follows −

the internet of things (iot) describes the network of physical objects - "things" − that are embedded with sensors, software, and other technologies for the purpose of connecting and exchanging data with other devices and systems over the internet.

the above definition nails it. things embedded with sensors, and containing software, sharing data with other devices/ systems over the internet. this definition also clearly highlights two of the three major functional blocks of any iot device −

sense
process and store
transmit

sense

what do the iot devices sense? they can sense anything worth sensing. if your iot device is being installed in a garbage dump, it may check the garbage fill level. if your iot device is being installed in a factory, it may sense the electricity consumption. if your iot device is installed on a machine, it may sense the vibration signature of the machine to determine whether the machine is on or off or cutting. if your device is installed on a vehicle, it may sense the movement and location of your vehicle.

your iot device will sense anything which can help you save costs, increase profits, or warn you about any impending catastrophe. your traditional fire alarm is so close to being an iot device. it senses smoke, processes it to determine if the smoke concentration is above the safe level. it just doesn't transmit this information anywhere. however, if you have all the fire alarms in the building connected to the internet, and a dashboard in the security room, showing which room has caught fire, your fire alarm will very much be an iot device.

process and store

what processing/ storage happens onboard an iot device? this answer depends a lot on your use case. there are iot devices that do no processing onboard and simply transmit the raw sensor data to the servers. there are iot devices that do live video processing onboard to identify objects/ people. it depends on the volume of your data, the ram available, the final output desired, and the transmission bandwidth available. if your device gets a machine's vibration signature every millisecond, you will have 1000 readings in just one second. sending this volume of data to the server may not make sense in some cases (especially if you are on a low bandwidth network like nb−iot). in such cases, you may want to perform fft onboard the device and just send the frequencies and amplitudes of the vibration to the server. if your device senses the temperature and humidity in the atmosphere once every 5 mins, you may just need the formula to convert the raw readings to temperature and humidity and send it across. or you may just send the raw readings and let the server do the conversion. you may send every reading in this case.

almost all iot devices have some onboard memory to store the missed packets in case of a network error. some devices have config files which also require onboard storage. some devices keep the last x hours of data in their memory for future access. iot devices that perform heavy processing onboard definitely require storage to collect sufficient data before the processing starts. for instance, if your device performs fft on the vibration data after every 10,000 readings, it will need to store the incoming readings till the number reaches 10,000.

transmit

how do iot devices transmit data? well, there are several solutions available. some of them are −

cellular networks like gsm (2g), lte (4g), nb−iot
wi−fi
bluetooth and ble (bluetooth low energy)
lora

choosing the right transmission solution is a big decision in itself and depends a lot on the power available to you, your bandwidth requirements, communication distance, cost, and acceptable latency. your smartwatch may use ble to communicate with your mobile phone, your smart tv may use wifi, while a device installed in a vehicle may use a cellular network. an iot device made for agricultural applications, like soil moisture measurement, especially in remote areas, may use lora to communicate with another device, which in turn may have a wifi or ethernet connection. the ultimate goal is almost always to get the data on a server, and/or display it to the user on a dashboard/app.

wrapping up

if you are new to iot, this chapter would have given you a good overview of what the fuss is all about. if this has got you excited, then move on to the next chapter, where we talk about esp32, the system on chip (soc) microcontroller that this entire tutorial is all about. we discuss why esp32 is popular in the iot space and what functionalities it provides in the sensing, processing, storage, and transmission domains. see you there.

esp32 is the soc (system on chip) microcontroller which has gained massive popularity recently. whether the popularity of esp32 grew because of the growth of iot or whether iot grew because of the introduction of esp32 is debatable. if you know 10 people who have been part of the firmware development for any iot device, chances are that 7−8 of them would have worked on esp32 at some point. so what is the hype all about? why has esp32 become so popular so quickly? let's find out.

before we delve into the actual reasons for the popularity of esp32, let's take a look at some of its important specifications. the specs listed below belong to the esp32 wroom 32 variant.−

integrated crystal− 40 mhz
module interfaces− uart, spi, i2c, pwm, adc, dac, gpio, pulse counter, capacitive touch sensor
integrated spi flash− 4 mb
rom− 448 kb (for booting and core functions)
sram− 520 kb
integrated connectivity protocols− wifi, bluetooth, ble
on−chip sensor− hall sensor
operating temperature range− −40 − 85 degrees celsius
operating voltage− 3.3v
operating current− 80 ma (average)

with the above specifications in front of you, it is very easy to decipher the reasons for esp32's popularity. consider the requirements an iot device would have from its microcontroller (μc). if you've gone through the previous chapter, you'd have realized that the major operational blocks of any iot device are sensing, processing, storage, and transmitting. therefore, to begin with, the μc should be able to interface with a variety of sensors. it should support all the common communication protocols required for sensor interface: uart, i2c, spi. it should have adc and pulse counting capabilities. esp32 fulfills all of these requirements. on top of that, it also can interface with capacitive touch sensors. therefore, most common sensors can interface seamlessly with esp32.

secondly, the μc should be able to perform basic processing of the incoming sensor data, sometimes at high speeds, and have sufficient memory to store the data. esp32 has a max operating frequency of 40 mhz, which is sufficiently high. it has two cores, allowing parallel processing, which is a further add-on. finally, its 520 kb sram is sufficiently large for processing a large array of data onboard. many popular processes and transforms, like fft, peak detection, rms calculation, etc. can be performed onboard esp32. on the storage front, esp32 goes a step ahead of the conventional microcontrollers and provides a file system within the flash. out of the 4 mb of onboard flash, by default, 1.5 mb is reserved as spiffs (spi flash file system). think of it as a mini−sd card that lies within the chip itself. you can not only store data, but also text files, images, html and css files, and a lot more within spiffs. people have displayed beautiful webpages on wifi servers created using esp32, by storing html files within spiffs.

finally, for transmitting data, esp32 has integrated wifi and bluetooth stacks, which have proven to be a game-changer. no need to connect a separate module (like a gsm module or an lte module) for testing cloud communication. just have the esp32 board and a running wifi, and you can get started. esp32 allows you to use wifi in access point as well as station mode. while it supports tcp/ip, http, mqtt, and other traditional communication protocols, it also supports https. yep, you heard that right. it has a crypto−core or a crypto-accelerator, a dedicated piece of hardware whose job is to accelerate the encryption process. so you cannot only communicate with your web server, you can do so securely. ble support is also critical for several applications. of course, you can interface lte or gsm or lora modules with esp32. therefore, on the 'transmitting data' front as well, esp32 exceeds expectations.

with so many features, esp32 would be costing a fortune, right? that's the best part. esp32 dev modules cost in the ballpark of ₹ 500. not only that, the chip dimensions are quite small (25 mm x 18 mm, including the antenna area), allowing its use in devices requiring a very small form factor.

finally, esp32 can be programmed using the arduino ide, making the learning curve much less steep. isn't that great? are you excited to get your hands dirty with esp32? then let's start by installing the esp32 board in the arduino ide in the next chapter. see you there.

one very big advantage with esp32, which has aided its quick adoption and massive popularity, is the provision for programming the esp32 within the arduino ide.

now, i should point out here that arduino is not the only ide that helps you compile code for esp32 and flash it into the microcontroller. there is esp−idf which is the official development framework for esp32, which provides much more flexibility in terms of configuration options. however, it is hardly as intuitive and user−friendly as the arduino ide, and if you are starting out with esp32, arduino ide is ideal to get your hands dirty. also, with the number of supporting libraries built for esp32 in arduino, courtesy of the huge developer community, there's hardly any functionality of esp32 which can't be realized with the arduino ide. esp-idf is more suitable for the more advanced and experienced programmers, who need to stretch esp32 to its limits. if you are one of those, you are looking for the esp−idf getting started guide. others can follow along.

installation steps

now, to install the esp32 board in the arduino ide, you need to follow the below steps −

make sure you have arduino ide (preferably the latest version) installed on your machine
open arduino and go to file −> preferences
in the additional boards manager url, enter

https://dl.espressif.com/dl/package_esp32_index.json

in case you have an existing json file's url in the preferences (this is likely if you've installed esp8266, stm32duino, or any such additional board in the ide), you can just append the above path to the existing path, using a comma. an example is shown below, for esp8266 and esp32 boards −

http://arduino.esp8266.com/stable/package_esp8266com_index.json, https://dl.espressif.com/dl/package_esp32_index.json

go to tools −> board−> boards manager. a pop−up would open up. search for esp32 and install the esp32 by espressif systems board. the image below shows the board already installed because i had installed the board before preparing this tutorial.

verifying the installation

once your esp32 board has been installed, you can verify the installation by going to tools −> boards. you can see a whole bunch of boards under the esp32 arduino section. choose the board of your choice. if you are not sure which board best represents the one you have, you can choose esp32 dev module.

next, connect your board to your machine using the usb cable. you should see an additional com port under tools−> port. select that additional port. in case you see multiple ports, you can disconnect the usb and see which port disappeared. that port corresponds to esp32.

once the port is identified, pick any one example sketch from file −> examples. we will choose the startcounter example from file −> examples −> preferences −> startcounter.

open that sketch, compile it and flash it into the esp32 by clicking on the upload button (the right arrow button, besides the compile button).

then open the serial monitor using tools −> serial monitor, or simply by pressing ctrl + shift + m on your keyboard. you should see the counter value getting incremented after every esp32 restart.

congratulations!! you've set up the environment for working with esp32.

a key feature of esp32 that makes it so much more popular than its predecessor, esp8266, is the presence of two cores on the chip. this means that we can have two processes executing in parallel on two different cores. of course, you can argue that parallel operation can also be achieved on a single thread using freertos/ any other equivalent rtos. however, there is a difference between two processes running in parallel on a single core, and they running in parallel on different cores. on a single core, often, one thread has to wait for the other to pause before it can begin execution. on two cores, parallel execution is literally parallel, because they are literally occupying different processors.

sounds exciting? let's get started with a real example, that demonstrates how to create two tasks and assign them to specific cores within esp32.

code walkthrough

github link: https://github.com/

to use freertos within the arduino ide, no additional imports are required. it comes inbuilt. what we need to do is define two functions that we wish to run on the two cores. they are defined first. one function evaluates the first 25 terms of the fibonacci series and prints every 5th of them. it does so in a loop. the second function evaluates the sum of numbers from 1 to 100. it too does so in a loop. in other words, after calculating the sum from 1 to 100 once, it does so again, after printing the id of the core it is executing on. we are not printing all the numbers, but only every 5th number in both the sequences, because both the cores will try to access the same serial monitor. therefore, if we print every number, they will try to access the serial monitor at the same time frequently.

void print_fibonacci() {
   int n1 = 0;
   int n2 = 1;
   int term = 0;
   char print_buf[300];
   sprintf(print_buf, "term %d: %d\n", term, n1);
   serial.print(print_buf);
   term = term + 1;
   sprintf(print_buf, "term %d: %d\n", term, n1);
   serial.print(print_buf);
   for (;;) {
      term = term + 1;
      int n3 = n1 + n2;
      if(term%5 == 0){
      sprintf(print_buf, "term %d: %d\n", term, n3);
      serial.println(print_buf);
   }
   n1 = n2;
   n2 = n3;

   if (term >= 25) break;
   }
}
void sum_numbers() {
   int n1 = 1;
   int sum = 1;
   char print_buf[300];
   for (;;) {
      if(n1 %5 == 0){
         sprintf(print_buf, "                                                            term %d: %d\n", n1, sum);
         serial.println(print_buf);
      }
      n1 = n1 + 1;
      sum = sum+n1;
      if (n1 >= 100) break;
   }
}
void codefortask1( void * parameter ) {
   for (;;) {
      serial.print("code is running on core: ");serial.println( xportgetcoreid());
      print_fibonacci();
   }
}
void codefortask2( void * parameter ) {
   for (;;) {
      serial.print("                                                            code is running on core: ");serial.println( xportgetcoreid());
      sum_numbers();
   }
}

you can see above that we have shifted the print statement for task 2 to the right. this will help us differentiate between the prints happening from task 1 and task 2.

next we define task handles. task handles serve the purpose of referencing that particular task in other parts of the code. since we have two tasks, we will define two task handles.

taskhandle_t task1, task2;

now that the functions are ready, we can move to the setup part. within setup(), we simply pin the two tasks to the respective cores. first, let me show you the code snippet.

void setup() {
   serial.begin(115200);
   /*syntax for assigning task to a core:
   xtaskcreatepinnedtocore(
                    coretask,   // function to implement the task
                    "coretask", // name of the task 
                    10000,      // stack size in words 
                    null,       // task input parameter 
                    0,          // priority of the task 
                    null,       // task handle. 
                    taskcore);  // core where the task should run 
   */
   xtaskcreatepinnedtocore(    codefortask1,    "fibonaccitask",    5000,      null,    2,    &task1,    0);
   //delay(500);  // needed to start-up task1
   xtaskcreatepinnedtocore(    codefortask2,    "sumtask",    5000,    null,    2,    &task2,    1);
}

now let's dive deeper into the xtaskcreatepinnedtocore function. as you can see, it takes a total of 7 arguments. their description is as follows.

the first argument codefortask1 is the function that will be executed by the task
the second argument "fibonaccitask" is the label or name of that task
the third argument 1000 is the stack size in bytes that is allotted to this task
the fourth argument null is the task input parameter. basically, if you wish to input any parameter to the task, it goes here
the fifth argument 1 defines the priority of the task. the higher the value, the more is the priority of the task.
the sixth argument &task1 is the task handle
the final argument 0 is the code on which the task will run. if the value is 0, the task will run on core 0. if it is 1, the task will run on code 1.

finally, the loop can be left empty, since the two tasks running on the two cores are of more importance here.

void loop() {}

you can see the output on the serial monitor. note that there are no delays anywhere in the code. therefore, both the series getting incremented shows that the computations are happening in parallel. the core ids printed on the serial monitor also confirm that.

please note that arduino sketches, by default, run on core 1. this can be verified using serial.print( xportgetcoreid()); so if you add some code in loop(), it will run as another thread on core 1. in that case, core 0 will have a single task running, while core 1 will have two tasks running.

accelerometers and gyroscopes are widely used in industrial iot for measuring the health and operating parameters of various machines. mpu6050 is a popular six-axis accelerometer + gyroscope. it is a mems (micro-electro-mechanical systems) sensor, meaning it is very compact (as can be seen from the image below) and, for a wide range of frequencies, very accurate as well.

in this tutorial, we will see how to interface esp32 with the mpu6050. in the process, you will learn about the usage of the i2c (inter-integrated circuit) protocol, which will then enable you to interface the esp32 with several sensors and peripherals which communicate using the i2c protocol. you will need your esp32, an mpu6050, and a couple of jumper wires for this tutorial.

connecting mpu6050 with esp32

as shown in the image below, you need to connect the sda line of mpu6050 to pin 21 on esp32, scl line to pin 22, gnd to gnd, and vcc to 3v3 pin. the other pins of mpu6050 need not be connected.

code walkthrough

github link − https://github.com/

esp32, and arduino in general refer to the i2c protocol as 'wire'. therefore the required library import is wire.h

#include<wire.h>

next we define constants and global variables.

const int mpu_addr = 0x68; // i2c address of the mpu-6050
int16_t acx, acy, acz, tmp, gyx, gyy, gyz;

every i2c device has a fixed address using which other devices identify it and communicate with it. for mpu6050, that address is 0x68. we will use it later when initializing the i2c communication with the mpu6050. we next move to the setup code.

void setup() {
   serial.begin(115200);
   wire.begin(21, 22, 100000); // sda, scl, clock speed
   wire.begintransmission(mpu_addr);
   wire.write(0x6b);  // pwr_mgmt_1 register
   wire.write(0);     // set to zero (wakes up the mpu−6050)
   wire.endtransmission(true);
   serial.println("setup complete");
}

the first line is trivial. we are initiating communication with the serial monitor at 115200 baud rate. next, we begin the i2c communication. for that, we provide 3 arguments to the wire.begin() function.

these are the sda and scl pins and the clock speed. now, i2c communication requires two lines: the data line (sda) and the clock line (scl). on esp32, pins 21 and 22 are generally reserved for i2c, with 21 being sda and 22 being scl. for communicating with mpu6050, we have two speed options: 100kbit/s and 400kbit/s. we have chosen 100khz here. you can choose the higher speed option as well if your use-case requires it.

next, we indicate to the esp32 that we want to communicate with the chip which has the address equal to mpu_addr, using the wire.begintransmission() command. at this point, you would have guessed that one esp32 chip can communicate with multiple i2c peripherals. in fact, there are 128 unique addresses possible (address field is 7 bits long), and so the esp32 can communicate with 128 different peripherals using i2c, provided all of them have different addresses.

in the next couple of lines, we are setting the pwr_mgmt_1 register of mpu6050 to 0. this is used to wake up the mpu6050. the address 0x6b of the pwr_mgmt_1 register is the address within mpu6050's memory.

it has nothing to do with the i2c address of mpu6050. once the mpu is woken up, we end this particular transmission over i2c and our setup is complete, and we indicate that on the serial monitor using a print statement. now let's jump into the loop. you will notice that we pass a boolean true as an argument to wire.endtransmission. this tells the esp32 to send a stop command and release the i2c lines. if we replace true with false, the esp32 will send a restart instead of stop, keeping the connection active.

void loop() {
   wire.begintransmission(mpu_addr);
   wire.write(0x3b);  // starting with register 0x3b (accel_xout_h)
   wire.endtransmission(true);
   wire.begintransmission(mpu_addr);
   wire.requestfrom(mpu_addr, 14, true); // request a total of 14 registers
   acx = wire.read() −− 8 | wire.read(); // 0x3b (accel_xout_h) & 0x3c (accel_xout_l)
   acy = wire.read() −− 8 | wire.read(); // 0x3d (accel_yout_h) & 0x3e (accel_yout_l)
   acz = wire.read() −− 8 | wire.read(); // 0x3f (accel_zout_h) & 0x40 (accel_zout_l)
   tmp = wire.read() −− 8 | wire.read(); // 0x41 (temp_out_h) &  0x42 (temp_out_l)
   gyx = wire.read() −− 8 | wire.read(); // 0x3b (accel_xout_h) & 0x3c (accel_xout_l)
   gyy = wire.read() −− 8 | wire.read(); // 0x3d (accel_yout_h) & 0x3e (accel_yout_l)
   gyz = wire.read() −− 8 | wire.read(); // 0x3f (accel_zout_h) & 0x40 (accel_zout_l)


   serial.print(acx); serial.print(" , ");
   serial.print(acy); serial.print(" , ");
   serial.print(acz); serial.print(" , ");
   serial.print(gyx); serial.print(" , ");
   serial.print(gyy); serial.print(" , ");
   serial.print(gyz); serial.print("\n");
}

in the loop, if you scan through the above code snippet, you will see that we perform a total of two transmissions. in the first one, we indicate to the mpu6050 the address from which we would like to start reading the data, or rather set the mpu6050's internal pointer to this particular address. in the second transmission, we tell the mpu that we request 14 bytes starting from the address sent earlier. then we read the bytes one by one. you may notice that we don't have a wire.endtransmission(true) command at the end of read. this is because the third argument of wire.requestfrom(mpu,14,true) indicates to the esp32 to send a stop command after reading the required number of bytes. had we passed false instead of true, esp32 would have sent a restart command instead of stop command.

now, you might be wondering how was it determined that which register corresponds to which reading. the answer is the mpu6050 register map. it, as the name suggests, provides information on which value can be obtained from which register. based on this map, we realized that we understand that 0x3b and 0x3c correspond to the higher and lower bytes of the 16−bit x−direction acceleration value. the next two registers (0x3d and 0x3e) contain the higher and lower bytes of the 16−bit y−direction acceleration value, and so on. in between accelerometer and gyroscope readings, there are two bytes containing temperature readings, which we read and ignore, because we don't require them.

so with this, you can successfully acquire data from mpu6050 on an esp32. congratulations!! move on to the next tutorial for learning how to acquire data from an analog sensor on esp32.

references

another important category of sensors that you need to interface with esp32 is analog sensors. there are many types of analog sensors, ldrs (light dependent resistors), current and voltage sensors being popular examples. now, if you are familiar with how analogread works on any arduino board, like arduino uno, then this chapter will be a cakewalk for you because esp32 uses the same functions. there are only a few nuances you should be aware of, that will be covered in this chapter.

a brief about the analog to digital conversion (adc) process

every microcontroller which supports adc will have a defined resolution and a reference voltage. the reference voltage is generally the supply voltage. the analog voltage provided to the adc pin should be less than or equal to the reference voltage. the resolution indicates the number of bits that will be used to represent the digital value. thus, if the resolution is 8 bits, then the value will be represented by 8 bits, and the maximum value possible is 255. this maximum value corresponds to the value of the reference voltage. the values for other voltages are often derived by scaling.

thus, if the reference voltage is 5v and an 8−bit adc is used, then 5v corresponds to a reading of 255, 1v corresponds to a reading of (255/5*1) = 51, 2v corresponds to a reading (255/5*2) = 102 and so on. if we had a 12 bit adc, then 5v would correspond to a reading of 4095, 1v would correspond to a reading of (4095/5*1) = 819, and so on.

the reverse calculations can be performed similarly. if you get a value of 1000 on a 12 bit adc with a reference voltage of 3.3v, then it corresponds to a value of (1000/4095*3.3) = 0.8v approximately. if you get a reading of 825 on a 10 bit adc with a reference voltage of 5v, it corresponds to a value of (825/1023*5) = 4.03v approximately.

with the above explanation, it will be clear that both the reference voltage and the number of bits used for adc determine the minimum possible voltage change that can be detected. if the reference voltage is 5v and the resolution is 12-bit, you have 4095 values to represent a voltage range of 0−5v. thus, the minimum change that can be detected is 5v/4095 = 1.2mv. similarly, for a 5v and 8-bit reference voltage, you have only 255 values to represent a range of 0-5v. thus, the minimum change that can be detected is 5v/255 = 19.6mv, or about 16 times higher than the minimum change detected with a 12-bit resolution.

connecting the adc sensor with esp32

considering the popularity and availability of the sensor, we will use an ldr for the demonstration. we will essentially connect ldr in series with a regular resistor, and feed the voltage at the point connecting the two resistors to the adc pin of esp32. which pin? well, there are lots of them. esp32 boasts of 18 adc pins (8 in channel 1 and 10 in channel 2). however, channel 2 pins cannot be used along with wifi. and some pins of channel 1 are not exposed on some boards. therefore, i generally stick to the following 6 pins for adc− 32, 33, 34, 35, 36, 39. in the image shown below, an ldr with a resistance of 90k is connected to a resistor of resistance 150k. the free end of the ldr is connected to the 3.3v pin of esp32 and the free end of the resistor is connected to gnd. the common end of the ldr and the resistor is fed to the adc pin 36 (vn) of esp32.

code walkthrough

github link − https://github.com/

the code here is straightforward. no libraries need to be included. we just define the ldr pin as a constant, initialize serial in the setup(), and set the resolution of the adc. here we have set a resolution of 10-bits (meaning the maximum value is 1023). by default the resolution is 12-bits and for esp32, the minimum possible resolution is 9 bits.

const int ldr_pin = 36;

void setup() {
   // put your setup code here, to run once:
   serial.begin(115200);
   analogreadresolution(10); //default is 12. can be set between 9-12.
}

in the loop, we just read the value from the ldr pin and print it to the serial monitor. also, we convert it to voltage and print the corresponding voltage as well.

void loop() {
   // put your main code here, to run repeatedly:
   // ldr resistance: 90k ohms
   // resistance in series: 150k ohms
   // pinouts:
   // vcc −> 3.3 (connected to ldr free end)
   // gnd −> gnd (connected to resistor free end)
   // analog read −> vp (36) − intermediate between ldr and resistance. 
   int ldr_reading = analogread(ldr_pin);
   float ldr_voltage = ((float)ldr_reading*3.3/1023);
   serial.print("reading: ");serial.print(ldr_reading); serial.print("\t");serial.print("voltage: ");serial.println(ldr_voltage);
}

we have used 1023 as the divisor because we have set the adc resolution to 10 bits. in case you change the adc value to n, you need to change the divisor to (2^n −1). now place your hand on the ldr

we have used 1023 as the divisor because we have set the adc resolution to 10 bits. in case you change the adc value to n, you need to change the divisor to (2^n −1). now place your hand on the ldr, and see the effect on the voltage, and then flash a torch on the ldr and see the voltage swing to the opposite extreme on the serial monitor. that's it. you've successfully captured data from an analog sensor on esp32.

references

analog to digital converted on esp32

non−volatile storage is an important requirement for embedded systems. often, we want the chip to remember a couple of things, like setup variables, wifi credentials, etc. even between power cycles. it would be so inconvenient if we had to perform setup or config every time the device undergoes a power reset. esp32 has two popular non-volatile storage methods: preferences and spiffs. while preferences are generally used for storing key-value pairs, spiffs (spi flash file system), as the name suggests, is used for storing files and documents. in this chapter, let's focus on preferences.

preferences are stored in a section of the main flash memory with type as data and subtype as nvs. nvs stands for non−volatile storage. by default, 20 kb of space is reserved for preferences, so don't try to store a lot of bulky data in preferences. use spiffs for bulky data (spiffs has 1.5 mb of reserved space by default). what kind of key−value pairs can be stored within preferences? let's understand through the example code.

code walkthrough

we will use the example code provided. go to file −> examples −> preferences −> startcounter. it can also be found on github.

this code keeps a count of how many times the esp32 was reset. therefore, every time it wakes up, it fetches the existing count from preferences, increments it by one, and saves the updated count back to preferences. it then resets the esp32. you can see using the printed statements on the esp32 that the value of the count is not lost between resets, that it is indeed non−volatile.

this code is very heavily commented, and therefore, largely, self-explanatory. nevertheless, let's walk through the code.

we begin by including the preferences library.

#include <preferences.h>

next, we create an object of class preferences.

preferences preferences;

now let's look at the setup line by line. we begin by initializing serial.

void setup() {
   serial.begin(115200);
   serial.println();

next, we open preferences with a namespace. now, think of the preference storage like a bank locker−room. there are several lockers, and you open one at a time. the namespace is like the name of the locker. within each locker, there are key−value pairs that you can access. if the locker whose name you mentioned does not exist, then it will be created, and then you can add key−value pairs to that locker. why are there different lockers? to avoid clashes in the name. say you have a wifi library that uses preferences to store credentials and a bluetooth library that also uses preferences to store credentials. say both of these are being developed by different developers. what if both use the same key name credentials? this will obviously create a lot of confusion. however, if both of them have their keys in different lockers, there will be no confusion at all.

// open preferences with my-app namespace. each application module, library, etc
// has to use a namespace name to prevent key name collisions. we will open storage in
// rw-mode (second parameter has to be false).
// note: namespace name is limited to 15 chars.
preferences.begin("my−app", false);

the second argument false of preferences.begin() indicates that we want to both read from and write to this locker. if it was true, we could only read from the locker, not write to it. also, the namespace, as mentioned in the comments, shouldn't be more than 15 characters in length.

next, the code has a couple of commented statements, which you can make use of depending on the requirement. one enables you to clear the locker, and the other helps you delete a particular key−value pair from the locker (having the key as "counter")

// remove all preferences under the opened namespace
//preferences.clear();

// or remove the counter key only
//preferences.remove("counter");

as a next step, we get the value associated with the key "counter". now, for the first time when you run this program, there may be no such key existing. therefore, we also provide a default value of 0 as an argument to the preferences.getuint() function. what this tells esp32 is that if the key "counter" doesn't exist, create a new key-value pair, with key as "counter" and value as 0. also, note that we are using getuint because the value is of type unsigned int. other functions like getfloat, getstring, etc. need to be called depending on the type of the value. the full list of options can be found here.

unsigned int counter = preferences.getuint("counter", 0);

next, we increment this count by one and print it on the serial monitor.

// increase counter by 1
counter++;

// print the counter to serial monitor
serial.printf("current counter value: %u\n", counter);

we then store this updated value back to non-volatile storage. we are basically updating the value for the key "counter". next time the esp32 reads the value of the key "counter", it will get the incremented value.

// store the counter to the preferences
preferences.putuint("counter", counter);

finally, we close the preferences locker and restart the esp32 in 10 seconds.

// close the preferences
preferences.end();

// wait 10 seconds
serial.println("restarting in 10 seconds...");
delay(10000);
   
// restart esp
esp.restart();
}

because we restart esp32 before diving into the loop, the loop is never executed. therefore, it is kept blank

void loop() {}

this example demonstrates quite well how esp32 preferences storage is indeed non−volatile. when you check the printed statements on the serial monitor, you can see the count getting incremented between successive resets. this would not have happened with a local variable. it was only possible by using non−volatile storage through preferences.

references

in the previous chapter, we looked at preferences as one way of storing data in non−volatile storage and understood how they are used to store key-value pairs. in this one, we look at spiffs (spi flash file storage), which is used for storing larger data in the form of files. think of spiffs as a very small sd card onboard the esp32 chip itself. by default, about 1.5 mb of the onboard flash is allocated to spiffs. you can see that for yourself by going through tools −> partition scheme.

you can see that there are several other partition options available. however, let's not get there right now. changing the partition scheme will anyway not be required for most of your applications. all the chapters in this tutorial will work well with the default partition scheme.

now let's see the process of creating, modifying, reading, and deleting a file from spiffs, using an example.

code walkthrough

we will again use the example code provided. go to file −> examples −> spiffs −> spiffs_test. this code is ideal for understanding all the file operations possible with spiffs. it can also be found on github

we begin with the inclusion of two libraries: fs.h and spiffs.h. fs stands for file system.

#include "fs.h"
#include "spiffs.h"

next, you see a macro definition, format_spiffs_if_failed. there is an associated comment which suggests that you need to format the spiffs only the first time you run a test. what this means is that you can set the value of this macro to false after your first run. formatting the spiffs takes time, and need not be done every time you run your code. therefore, a practice that people follow is to have a separate code for formatting the spiffs, which they flash before flashing the main code. the main code doesn't include the format command. in this example though, for the sake of completeness, this macro has been kept as true.

/* you only need to format spiffs the first time you run a
   test or else use the spiffs plugin to create a partition
   https://github.com/me−no−dev/arduino−esp32fs−plugin */
#define format_spiffs_if_failed true

next, you can see that a number of functions have been defined for different file system operations. they are −

listdir − to list all directories
readfile − to read a specific file
writefile − to write to a file (this overwrites the content already present in the file)
appendfile − to append content to a file (use this when you want to add to the existing content, not overwrite it
renamefile − to change the name of a file
deletefile − to delete a file

void listdir(fs::fs &fs, const char * dirname, uint8_t levels){
   serial.printf("listing directory: %s\r\n", dirname);

   file root = fs.open(dirname);
   if(!root){
      serial.println("− failed to open directory");
      return;
   }
   if(!root.isdirectory()){
      serial.println(" − not a directory");
      return;
   }

   file file = root.opennextfile();
   while(file){
      if(file.isdirectory()){
         serial.print("  dir : ");
         serial.println(file.name());
         if(levels){
            listdir(fs, file.name(), levels -1);
         }
      } else {
         serial.print("  file: ");
         serial.print(file.name());
         serial.print("\tsize: ");
         serial.println(file.size());
      }
      file = root.opennextfile();
   }
}

void readfile(fs::fs &fs, const char * path){
   serial.printf("reading file: %s\r\n", path);

   file file = fs.open(path);
   if(!file || file.isdirectory()){
       serial.println("− failed to open file for reading");
       return;
   }

   serial.println("− read from file:");
   while(file.available()){
      serial.write(file.read());
   }
}

void writefile(fs::fs &fs, const char * path, const char * message){
   serial.printf("writing file: %s\r\n", path);

   file file = fs.open(path, file_write);
   if(!file){
      serial.println("− failed to open file for writing");
      return;
   }
   if(file.print(message)){
      serial.println("− file written");
   }else {
      serial.println("− frite failed");
   }
}

void appendfile(fs::fs &fs, const char * path, const char * message){
   serial.printf("appending to file: %s\r\n", path);

   file file = fs.open(path, file_append);
   if(!file){
      serial.println("− failed to open file for appending");
      return;
   }
   if(file.print(message)){
      serial.println("− message appended");
   } else {
      serial.println("− append failed");
   }
}

void renamefile(fs::fs &fs, const char * path1, const char * path2){
   serial.printf("renaming file %s to %s\r\n", path1, path2);
   if (fs.rename(path1, path2)) {
      serial.println("− file renamed");
   } else {
      serial.println("− rename failed");
   }
}

void deletefile(fs::fs &fs, const char * path){
   serial.printf("deleting file: %s\r\n", path);
   if(fs.remove(path)){
      serial.println("− file deleted");
   } else {
      serial.println("− delete failed");
   }
}

note that all of the above functions aren't asking for a file name. they are asking for the full file path. because this is a file system. you could have directories, subdirectories, and files within those subdirectories. therefore, esp32 needs to know the full path of the file you want to operate on.

next comes a function that isn't exactly a file operation function − testfileio. this is more of a time benchmarking function. it does the following −

writes about 1 mb (2048 * 512 bytes) of data to the file path that you provide and measures the write time
reads the same file and measures the read time

void testfileio(fs::fs &fs, const char * path){
   serial.printf("testing file i/o with %s\r\n", path);

   static uint8_t buf[512];
   size_t len = 0;
   file file = fs.open(path, file_write);
   if(!file){
      serial.println("− failed to open file for writing");
      return;
   }

   size_t i;
   serial.print("− writing" );
   uint32_t start = millis();
   for(i=0; i<2048; i++){
      if ((i & 0x001f) == 0x001f){
         serial.print(".");
      }
      file.write(buf, 512);
   }
   serial.println("");
   uint32_t end = millis() − start;
   serial.printf(" − %u bytes written in %u ms\r\n", 2048 * 512, end);
   file.close();

   file = fs.open(path);
   start = millis();
   end = start;
   i = 0;
   if(file && !file.isdirectory()){
      len = file.size();
         size_t flen = len;
         start = millis();
         serial.print("− reading" );
         while(len){
            size_t toread = len;
            if(toread > 512){
                toread = 512;
            }
            file.read(buf, toread);
            if ((i++ & 0x001f) == 0x001f){
              serial.print(".");
            }
            len −= toread;
         }
      serial.println("");
      end = millis() - start;
      serial.printf("- %u bytes read in %u ms\r\n", flen, end);
      file.close();
   } else {
      serial.println("- failed to open file for reading");
   }
}

note that the buf array is never initialized with any value. we may very well be writing garbage bytes to the file. that doesn't matter because the purpose of the function is to measure the write time and the read time.

once our functions have been defined, we move on to the setup, where the invocation of each of these functions is shown.

void setup(){
   serial.begin(115200);
   if(!spiffs.begin(format_spiffs_if_failed)){
      serial.println("spiffs mount failed");
      return;
   }
   listdir(spiffs, "/", 0);
   writefile(spiffs, "/hello.txt", "hello ");
   appendfile(spiffs, "/hello.txt", "world!\r\n");
   readfile(spiffs, "/hello.txt");
   renamefile(spiffs, "/hello.txt", "/foo.txt");
   readfile(spiffs, "/foo.txt");
   deletefile(spiffs, "/foo.txt");
   testfileio(spiffs, "/test.txt");
   deletefile(spiffs, "/test.txt");
   serial.println( "test complete" );
}

the setup does essentially the following −

it first initializes the spiffs using spiffs.begin(). the macro defined at the beginning is used here. when true, it formats the spiffs (time−consuming); when false, it initializes the spiffs without formatting.
it then lists all the directories at the root level. note that we have specified levels as 0. therefore, we are not listing the subdirectories within the directories. you can increase the nesting by incrementing the levels argument.
it then writes "hello" to a file hello.txt in the root. (the file will get created if it doesn't exist)
it then reads back hello.txt
it then renames hello.txt to foo.txt
it then reads foo.txt to see if the rename worked. you should see "hello" printed because that's what is stored in the file.
it then deletes foo.txt
it then performs the testfileio routine on a new file test.txt
once the routine is performed, it deletes test.txt

that's it. this example code very nicely lists down and tests all the functions you may want to use with spiffs. you can go ahead and modify this code, and play around with the different functions.

since we don't want to perform any recurring activity here, the loop is blank.

void loop(){
}

the output shown in the serial monitor will perhaps look like the image below −

note − if in case you get "spiffs mount failed" on running the sketch, set the value of format_spiffs_if_failed to false and try again.

references

esp32 arduino spiffs library

the combination of oled with esp32 is so popular that there are some boards of esp32 with the oled integrated. we'll, however, assume that you will be using a separate oled module with your esp32 board. if you have an oled module, it perhaps looks like the image below.

connecting the oled display module to esp32

like the mpu6050 module that we discussed in a previous chapter, the oled module also generally uses i2c for communication. therefore, the connection will be similar to the mpu6050 module. you need to connect the sda line to pin 21 on esp32, scl line to pin 22, gnd to gnd, and vcc to 3v3 pin

library for oled display

there are a number of libraries available for interfacing the oled display with esp32. you are free to use anyone you are comfortable with. for this example, we will use the 'esp8266 and esp32 oled driver for ssd1306 displays, by thingpulse, fabrice weinberg'. you can install this library from tools −> manage libraries. it can also be found on github

code walkthrough

the code becomes very simple thanks to the library we just installed. we will run a counter code, which will just count the seconds since the last reset and print them on the oled module. the code can be found on github

we begin with the inclusion of the ssd1306 library.

#include "ssd1306wire.h"

next, we define the oled pins and its i2c address. note that some oled modules contain an additional reset pin. a good example is the esp32 ttgo board, which comes with an inbuilt oled display. for that board, pin 16 is the reset pin. if you are connecting an external oled module to your esp32, you will most likely not use the reset pin. the i2c address of 0x3c is generally common for all oled modules.

//oled related variables
#define oled_addr   0x3c
#define oled_sda    21//4     //ttgo board without sd card has oled sda connected to pin 4 of esp32
#define oled_scl    22//15    //ttgo board without sd card has oled scl connected to pin 15 of esp32
#define oled_rst    16        //optional, ttgo board contains oled_rst connected to pin 16 of esp32

next, we create the oled display object and the counter variable.

ssd1306wire  display(oled_addr, oled_sda, oled_scl); 
int counter = 0;

after that, we define two functions. one for initializing the oled display (this function is redundant if your oled module doesn't contain a reset pin), and the other for printing text messages on the oled display. the showoledmessage() function breaks down the oled display area into 3 lines and asks for 3 strings, one for each line.

void initoled() {
   pinmode(oled_rst, output);
   //give a low to high pulse to the oled display to reset it
   //this is optional and not required for oled modules not containing a reset pin
   digitalwrite(oled_rst, low);
   delay(20);
   digitalwrite(oled_rst, high);
}
void showoledmessage(string line1, string line2, string line3) {
   display.init();                        // clears screen
   display.setfont(arialmt_plain_16);
   display.drawstring(0, 0, line1);       //  adds to buffer
   display.drawstring(0, 20, line2);
   display.drawstring(0, 40, line3);
   display.display();                     // displays content in buffer
}

finally, in the setup, we just initialize the oled display, and in the loop, we just utilize the first two lines of the display to show the counter.

void setup() {
   // put your setup code here, to run once:
   initoled();
}
void loop() {
   // put your main code here, to run repeatedly
   showoledmessage("num seconds is: ", string(counter), "");
   delay(1000);
   counter = counter+1;
}

that's it. congratulations on displaying your first text statements on the oled display.

the availability of a wifi stack is one of the main differentiators between esp32 and other microcontrollers. this chapter will give you a brief overview of the various wifi modes available on esp32. subsequent chapters cover the transmission of data of wifi using http, https, and mqtt. there are 3 primary modes in which the wifi can be configured on esp32:

station mode − this is like the wifi client mode. the esp32 connects to an available wifi field which in turn is connected to your internet. this is exactly similar to connecting your mobile phone to an available wifi network.
access point mode − this is equivalent to turning on the hotspot on your mobile phone so that other devices can connect to it. similarly, esp32 creates a wifi field around itself that other devices can connect to. esp32, however, does not have internet access by itself. therefore, with this mode, you can generally display only a couple of webpages hardcoded into esp32's memory. this mode is generally used to perform device setup during installation. say you are taking your esp32 to an unknown client site whose wifi credentials you don't know beforehand. you will program the esp32 to start operation in the access point mode. as soon as your mobile phone connects to the wifi field created by esp32, a page can open up (captive portal) and it will prompt you to enter wifi credentials. once you enter those credentials, the esp32, will switch to station mode and try to connect to the available wifi network using the credentials provided.
combined ap-sta mode − as you might have guessed, in this mode, esp32 is connected to an existing wifi network and at the same time it is creating its own field, which other devices can connect to.

most of the time, you will be using the esp32 in the station mode. in all the 3 subsequent chapters as well, we will be using the esp32 in the station mode. however, you should know about the ap mode as well and you are encouraged to explore examples of the ap mode yourself.

http (hypertext transfer protocol) is one of the most common forms of communications and with esp32 we can interact with any web server using http requests. let's understand how in this chapter.

a brief about http requests

the http request happens between a client and a server. a server, as the name suggests, 'serves' information to the client on request. a web server serves web pages generally. for instance, when you type https://www.linkedin.com/login in your internet browser, your pc or laptop acts as a client and requests for the page corresponding to the /login address, from the server hosting linkedin.com. you get an html page in return, which is then displayed by your browser.

http follows the request-response model, meaning that communication is always initiated by the client. the server cannot talk to any client out−of−the−blue, or can't start communication with any client. the communication always has to be initiated by the client in the form of a request and the server can only respond to that request. the response of the server contains the status code (remember 404? that's a status code) and, if applicable, the content requested. the list of all status codes can be found here.

now, how does a server identify an http request? through the structure of the request. an http request follows a fixed structure which consists of 3 parts:

the request line followed by carriage return line feed (crlf = \r\n)
zero or more header lines followed by crlf and an empty line, again followed by crlf
optional body

this is how a typical http request looks like:

post / http/1.1         //request line, containing request method (post in this case)
host: www.example.com   //headers
                        //empty line between headers
key1=value1&key2=value2   //body

this is how a server response looks like −

http/1.1 200 ok                     //response line; 200 is the status code
date: mon, 23 may 2005 22:38:34 gmt //headers
content-type: text/html; charset=utf-8
content-length: 155
last-modified: wed, 08 jan 2003 23:11:55 gmt
server: apache/1.3.3.7 (unix) (red-hat/linux)
etag: "3f80f−1b6−3e1cb03b"
accept-ranges: bytes
connection: close
                                    //empty line between headers and body
<html>						
  <head>
    <title>an example page</title>
  </head>
  <body>
    <p>hello world, this is a very simple html document.</p>
  </body>
</html>

in fact, there is a very good tutorial on http request structure on tutorialspoint itself. it also introduces you to the various request methods (get, post, put, etc.). for this chapter, we will be concerned with the get and post methods.

the get request contains all parameters in the form of a key value pair in the request url itself. for example, if instead of post, the same example request above was to be sent using get, it would look like:

get /test/demo_form.php?key1=value1&key2=value2 http/1.1   //request line
host: www.example.com                                     //headers	
                                                          //no need for a body

the post request, as you would have guessed by now, contains the parameters in the body instead of the url. there are several more differences between get and post, which you can read here. but the crux is that you will use post for sharing sensitive information, like passwords, with the server.

code walkthrough

for this chapter, we will write our http request from scratch. there are libraries like httpclient available specifically for handling the esp32 http requests which take care of constructing the http requests, but we will construct our request ourselves. that gives us much more flexibility. we will be restricting to the esp32 client mode for this tutorial. the http server mode is also possible with esp32, but that is for you to explore.

we will be using httpbin.org as our server. it is basically built for you to test your http requests. you can test get, post, and a variety of other methods using this server. see this.

the code can be found on github

we begin with the inclusion of the wifi library.

#include <wifi.h>

next, we will define some constants. for http, the port that is used is 80. that is the standard. similarly, we use 443 for https, 21 for ftp, 53 for dns, and so on. these are reserved port numbers.

const char* ssid = "your_ssid";
const char* password = "your_password";

const char* server = "httpbin.org";
const int port = 80;

finally, we create our wificlient object.

wificlient client

in the setup, we simply connect to the wifi in the station mode using the credentials provided.

void setup() {
   serial.begin(115200);
   wifi.mode(wifi_sta);          //the wifi is in station mode. the other is the softap mode
   wifi.begin(ssid, password);
   while (wifi.status() != wl_connected) {
      delay(500);
      serial.print(".");
   }
   serial.println("");  serial.print("wifi connected to: "); serial.println(ssid);  serial.println("ip address: ");  serial.println(wifi.localip());
   delay(2000);
}

the loop becomes important here. that's where the http request gets executed. we first begin by reading the chip id of our esp32. we will be sending that as a parameter to the server along with our name. we will construct the body of our http request using these parameters.

void loop() {
   int  conn;
   int chip_id = esp.getefusemac();;
   serial.printf("  flash chip id = %08x\t", chip_id);
   serial.println();
   serial.println();
   string body = "chipid=" + string(chip_id) + "&sentby=" + "your_name";
   int body_len = body.length();

notice the & before the sentby field. & is used as a separator between different key-value pairs in the http requests. next, we connect to the server.

serial.println(".....");
serial.println(); serial.print("for sending parameters, connecting to ");      serial.println(server);
conn = client.connect(server, port);

post request

if our connection is successful, client.connect() will return 1. we check that before making the request.

if (conn == 1)  {
   serial.println(); serial.print("sending parameters...");
   //request
   client.println("post /post http/1.1");
   //headers
   client.print("host: "); client.println(server);
   client.println("content-type: application/x−www−form−urlencoded");
   client.print("content-length: "); client.println(body_len);
   client.println("connection: close");
   client.println();
   //body
   client.println(body);
   client.println();

   //wait for server response
   while (client.available() == 0);

   //print server response
   while (client.available()) {
      char c = client.read();
      serial.write(c);
   }
} else {
   client.stop();
   serial.println("connection failed");
}

as you can see, we use the client.print() or client.println() for sending our request lines. the request, headers, and body are clearly indicated via comments. in the request line, post /post http/1.1 is equivalent to post http://httpbin.org/post http/1.1. since we have already mentioned the server in the client.connect(server,port), it is understood that /post refers to the server/post url.

for post requests especially, the content-length header is very important. without it, several servers assume that the content-length is 0, meaning there is no body. the content-type has been kept as application/x−www−form−urlencoded because our body represents a form data. in a typical form submission, you will have keys like name, address, etc., and corresponding values. you can have several other content types. for the full list, see this.

the connection: close header tells the server to close the connection after the request has been processed. you could have alternatively send connection: keep-alive if you wanted the connection to be kept alive after the request was processed.

these are just some of the headers that we could have included. the full list of http headers can be found here.

now, the httpbin.org/post url typically just echoes back our body. a sample response is the following −

http/1.1 200 ok
date: sat, 21 nov 2020 16:25:47 gmt
content−type: application/json
content−length: 402
connection: close
server: gunicorn/19.9.0
access−control−allow−origin: *
access−control−allow−credentials: true
{
   "args": {}, 
   "data": "", 
   "files": {}, 
   "form": {
      "chipid": "1780326616", 
      "sentby": "yash"
   }, 
   "headers": {
      "content−length": "34", 
      "content−type": "application/x−www−form−urlencoded", 
      "host": "httpbin.org", 
      "x-amzn−trace−id": "root=1−5fb93f8b−574bfb57002c108a1d7958bb"
   }, 
   "json": null, 
   "origin": "183.87.63.113", 
   "url": "http://httpbin.org/post"
}

as you can see, the content of the post body has been echoed back in the "form" field. you should see something similar to the above printed on your serial monitor. also note the url field. it clearly shows that the /post address in the request line was interpreted as http://httpbin.org/post.

finally, we will wait for 5 seconds, before ending the loop, and thus, making the request again.

  delay(5000);
}

get request

at this point, you would be wondering, what changes would you need to make to convert this post request to get request. it is quite simple actually. you would, first of all, invoke the /get address instead of /post. then you'll append the content of the body to the url after a ? sign. finally, you will replace the method to get. also, the content-length and content−type headers are no longer required, since your body is empty. thus, your request block would look like −

if (conn == 1) {
   string path = string("/get") + string("?") +body;
   serial.println(); serial.print("sending parameters...");
   //request
   client.println("get "+path+" http/1.1");
   //headers
   client.print("host: "); client.println(server);
   client.println("connection: close");
   client.println();
   //no body

   //wait for server response
   while (client.available() == 0);

   //print server response
   while (client.available()) {
      char c = client.read();
      serial.write(c);
   }
} else {
   client.stop();
   serial.println("connection failed");
}

the corresponding response would look like −

http/1.1 200 ok
date: tue, 17 nov 2020 18:05:34 gmt
content-type: application/json
content-length: 497
connection: close
server: gunicorn/19.9.0
access-control−allow−origin: *
access-control-allow-credentials: true

{
   "args": {
      "chipid": "3f:a0:a1:77:0d:84", 
      "sentby": "yash"
   }, 
   "headers": {
      "accept": "*/*", 
      "accept-encoding": "deflate, gzip", 
      "host": "httpbin.org", 
      "user-agent": "mozilla/5.0 (windows nt 10.0; win64; x64) applewebkit/537.36 (khtml, like gecko) chrome/86.0.4240.198 safari/537.36", 
      "x−amzn−trace−id": "root=1−5fb410ee−3630963b0b7980c959c34038"
   }, 
   "origin": "206.189.180.4", 
   "url": "https://httpbin.org/get?chipid=3f:a0:a1:77:0d:84&sentby=yash"
}

as you can see, the parameters send to the server are now returned in the args field, because they were sent as arguments in the url itself.

congratulations!! you've successfully sent your http requests using esp32.

references

we looked at transmitting data over http using esp32 in the previous chapter. in this one, we will transmit data over https. the s in https stands for 'secure'. basically, whatever data you transmit is encrypted using transport layer security (tls). this means that if someone is eavesdropping on your communication, they won't understand what you've transmitted. instead, what they'll get is some gibberish. covering how https works is beyond the scope of this chapter. but a simple google search will provide several useful resources for you to get started. in this chapter, we will see how to implement https on esp32.

converting any http request to https on esp32

in general, if you have a code written for sending an http request to the server, you can convert it to https following these simple steps −

change the library from wificlient to wificlientsecure (you need to include wificlientsecure.h)
change the port from 80 to 443

there is an optional fourth step: add ca certificate for the server. this step is optional because it doesn't affect the security of the communication. it just assures you that you are communicating with the correct server. if you don't provide the ca certificate, your communication will still be secure.

code walkthrough

the code you see below is very similar to the one used for the http communication. you are strongly advised to revisit that chapter. in this walkthrough, we will simply highlight the parts that are different from the http code.

the code can be found on github

we begin with the inclusion of the wifi library. we also need to include the wificlientsecure library here.

#include <wifi.h>
#include <wificlientsecure.h>

next, we will define the constants. note that the port is now 443 instead of 80.

const char* ssid = "your_ssid";
const char* password = "your_password";

const char* server = "httpbin.org";
const int port = 443;

next, instead of the wificlient object, we create the wificlientsecure object.

wificlientsecure client;

next, we define the ca certificate for our server (httpbin.org). now, you may be wondering how to get the ca certificate for our server. detailed steps are given here to get the ca certificate of any server using google chrome. in that same post, a note on the validity of ca certificates has been provided, and it is recommended to use the certificate of the certification authority, instead of the certificate of the server, especially for applications where you just program the device once and send it out in the field for years. the certification authority's certificate has a much longer validity (15+ years), compared to the server's certificate validity (1−2 years). therefore, we are using the certificate of the starfield class 2 certification authority (valid till 2034), instead of the certificate of httpbin.org (valid till feb 2021).

const char* ca_cert = \ 
"-----begin certificate-----\n" \
"miiedzccavegawibagibadanbgkqhkig9w0baqufadbomqswcqydvqqgewjvuzel\n"\
"mcmga1uechmcu3rhcmzpzwxkifrly2hub2xvz2llcywgsw5jljeymdaga1uecxmp\n"\
"u3rhcmzpzwxkiensyxnzidigq2vydglmawnhdglvbibbdxrob3jpdhkwhhcnmdqw\n"\
"nji5mtczote2whcnmzqwnji5mtczote2wjbomqswcqydvqqgewjvuzelmcmga1ue\n"\
"chmcu3rhcmzpzwxkifrly2hub2xvz2llcywgsw5jljeymdaga1uecxmpu3rhcmzp\n"\
"zwxkiensyxnzidigq2vydglmawnhdglvbibbdxrob3jpdhkwggegma0gcsqgsib3\n"\
"dqebaquaa4ibdqawggeiaoibaqc3msj+6xgmbiwtdbfk385n78gdgic/oav7pkaf\n"\
"8moh2ttybittkpskpd6e8j7ox+zlj0t1kky/e97gkvdir1mvnsofazmej2ycoadn\n"\
"+lq2cwqlzut3f+dzxkqzjrru6ybh838z1tbwj6+wrir/resp7defqgsho9t5iau0\n"\
"x9tdkyi22wy8sbi5gv2coj4qydvvbmvmepszgd3/cve8mc5fvj13c7jdbmzdi1aa\n"\
"k4umkhynarpkpw2vchmcudy96pztnbo8acr1zj3o/wsnf4azbl5kxznjhoe0nrra\n"\
"1w4tnsne35tfpe/w93bc6j67ea0cqmdrbnj41tpvi/jeoagragedo4hfmihcmb0g\n"\
"a1uddgqwbbs/x7frzt0fhvrbvazc1xdcdqmi5zcbkgydvr0jbigkmighgbs/x7fr\n"\
"zt0fhvrbvazc1xdcdqmi56fspgowadelmakga1uebhmcvvmxjtajbgnvbaothfn0\n"\
"yxjmawvszcbuzwnobm9sb2dpzxmsieluyy4xmjawbgnvbastkvn0yxjmawvszcbd\n"\
"bgfzcyayienlcnrpzmljyxrpb24gqxv0ag9yaxr5ggeamawga1udewqfmambaf8w\n"\
"dqyjkozihvcnaqefbqadggebaawdp4id0ckavagsafpzwdqbaycat1epoxkjktv3\n"\
"l7iezmdeatidh6gx70k1pncgqvhiv45yuapnp+yz3sfmh8lu+nlmpuxa2igvd56d\n"\
"eruix/u0f47zeud0/cwqtrv/p2jdlixtaasggh1o+re49l2l7shz3u0wixedyljl\n"\
"xy16paq8u4zt3vekyvggqqto8pt7dl5wxxp59fkdhemtlb71czbdzi0fmgakhynp\n"\
"vsjyacpq4xjdkvthcn2mqwplbqjliapbtjuhlbl90tsre9atvnziptnnvt51ckey\n"\
"wqpjirspnnvektelttqkbfi3qbfgmh95dmk/d5fs4c8ff5q=\n"\
"-----end certificate-----\n";

in the setup, we connect to the wifi in the station mode using the credentials provided, like before. here, we have the additional step of setting the ca certificate for our wifisecureclient. by doing this, we are telling the client that only communicate with the server if its ca certificate matches the one provided.

void setup() {
   serial.begin(115200);
   wifi.mode(wifi_sta);                    //the wifi is in station mode. the    other is the softap mode
   wifi.begin(ssid, password);
   while (wifi.status() != wl_connected) {
      delay(500);
      serial.print(".");
   }
   serial.println("");  serial.print("wifi connected to: "); serial.println(ssid);  serial.println("ip address: ");  serial.println(wifi.localip());
   client.setcacert(ca_cert);            //only communicate with the server if the ca certificates match
   delay(2000);
}

the loop is exactly the same as the one used in the http example.

void loop() {
   int  conn;
   int chip_id = esp.getefusemac();;
   serial.printf("  flash chip id = %08x\t", chip_id);
   serial.println();
   serial.println();
   string body = "chipid=" + string(chip_id) + "&sentby=" + "your_name";
   int body_len = body.length();

   serial.println(".....");
   serial.println(); serial.print("for sending parameters, connecting to "); serial.println(server);
   conn = client.connect(server, port);

   if (conn == 1) {
      serial.println(); serial.print("sending parameters...");
      //request
      client.println("post /post http/1.1");
      //headers
      client.print("host: "); client.println(server);
      client.println("content-type: application/x−www−form−urlencoded");
      client.print("content-length: "); client.println(body_len);
      client.println("connection: close");
      client.println();
      //body
      client.println(body);
      client.println();

      //wait for server response
      while (client.available() == 0);

      //print server response
      while (client.available()) {
         char c = client.read();
         serial.write(c);
      }
   } else {
      client.stop();
      serial.println("connection failed");
   }
   delay(5000);
}

the response to be expected from the server is also similar to the http example. the only difference is that the response received will also be secure. but we won't have to worry about decrypting the encrypted message. esp32 does that for us.

notice the url field in the server response. it contains https instead of http, confirming that our transmission was secure. in fact, if you edit the ca certificate slightly, say you just delete one character, and then try to run the sketch, you will see the connection getting failed.

however, if you remove the client.setcacert() line from the setup, the connection will get established again securely, even with the faulty ca certificate. this proves that setting the ca certificate doesn't affect the security of our communication. it just helps us verify that we are communicating with the right server. if we do set the certificate, then the esp32 won't communicate with the server unless the provided ca certificate matches the server's ca certificate. if we don't set the certificate, the esp32 will still communicate with the server securely.

congratulations!! you've successfully sent your https requests using esp32.

note − the hardware accelerator on esp32 that performs the encryption of messages for https, can support a maximum of 16384 bytes (16 kb) of data. therefore, if your message size exceeds 16 kb, you may need to break it down into chunks.

references

getting ca certificate for any server using google chrome

mqtt (message queuing telemetry transport) has gained a lot of prominence in the context of iot devices. it is a protocol that runs generally over tcp/ip. instead of the server−client model that we saw for http, mqtt uses the broker−client model. wikipedia defines mqtt brokers and clients as −

an mqtt broker is a server that receives all messages from the clients and then routes the messages to the appropriate destination clients. an mqtt client is any device (from a micro controller up to a full−fledged server) that runs an mqtt library and connects to an mqtt broker over a network.

think of the broker as a service like medium. the topics would be the medium publications, and the clients would be the medium users. a user (client) can post to a publication, and another user (client) who has subscribed to that publication (topic) would be told that a new post is available for reading. by now, you would have understood a major difference between http and mqtt. in http, your messages are directly sent to the intended server and you even get an acknowledgment in the form of status codes. in mqtt, you just send messages to the broker in the hope that your intended server(s) will take it from there. several features of mqtt turn out to be a boon if you are resource−constrained. they are listed below −

with mqtt, header overheads are very short and throughput is high. this helps save time and also battery.
mqtt sends information as a byte array instead of the text format. this makes the message lightweight.
because mqtt isn't dependent on the response from the server, the client is independent and can go to sleep (conserve battery) as soon as it has transmitted the message.

these are just some of the points which have resulted in the popularity of mqtt. you can get a more detailed comparison between mqtt and http here.

code walkthrough

in general, testing mqtt requires you to sign up for a free/ paid account with a broker. aws iot and azure iot are very popular platforms providing mqtt broker services, but they come with a lengthy signup and configuration process. luckily, there is a free broker service from hivemq which can be used for testing mqtt without any signup or configuration. it is ideal for those of you who are new to mqtt and just want to get your hands dirty, and also lets you focus more on the firmware of esp32. therefore, that is the broker we will be using for this chapter. of course, because it is a free service, there will be limitations. you can't share sensitive information, because all your messages are public, anyone can subscribe to your topics. for testing purposes, of course, these limitations won't matter.

the code can be found on github

we will be using the pubsubclient library. you can install it from tools −> manage libraries.

once the library is installed, we include wifi and pubsubclient libraries.

#include <wifi.h>
#include <pubsubclient.h>

next, we will define some constants. remember to replace the wifi credentials. the mqttserver and mqttport are the once mandated by http://www.mqtt−dashboard.com/.the mqtt_client_name, mqtt_pub_topic and mqtt_sub_topic can be any strings of your choice. just make sure that you do change their values. if multiple users copy the same code from this tutorial, you will receive a lot of messages from unknown clients when testing.

we also define the wificlient and mqttclient object. the mqttclient object requires the network client as an argument. if you are using ethernet, you would provide the ethernet client as an argument. since we are using wifi, we have provided the wifi client as an argument.

const char* ssid = "your_ssid";
const char* password = "your_password";

//the broker and port are provided by http://www.mqtt−dashboard.com/
char *mqttserver = "broker.hivemq.com";
int mqttport = 1883;

//replace these 3 with the strings of your choice
const char* mqtt_client_name = "espys2111";
const char* mqtt_pub_topic = "/ys/testpub"; //the topic to which our client will publish
const char* mqtt_sub_topic = "/ys/testsub"; //the topic to which our client will subscribe

wificlient client;
pubsubclient mqttclient(client);

next, we define the callback function. a callback function is an interrupt function. every time a new message is received from a subscribed topic, this function will be triggered. it has three arguments− the topic from which the message was received, the message as a byte array, and the length of the message. you can do whatever you want to do with that message (store it in spiffs, send it to another topic, and so on). here, we are just printing the topic and the message.

void callback(char* topic, byte* payload, unsigned int length) {
   serial.print("message received from: "); serial.println(topic);
   for (int i = 0; i < length; i++) {
      serial.print((char)payload[i]);
   }
   serial.println();
   serial.println();
}

in the setup, we connect to the wifi like in every other sketch. the last two lines concern mqtt. we set the server and port for mqtt and also the callback function.

void setup() {
   // put your setup code here, to run once:
   serial.begin(115200);
   wifi.mode(wifi_sta);                    //the wifi is in station mode
   wifi.begin(ssid, password);
   while (wifi.status() != wl_connected) {
      delay(500);
      serial.print(".");
   }
   serial.println("");  serial.print("wifi connected to: "); serial.println(ssid);  serial.println("ip address: ");     serial.println(wifi.localip());
   delay(2000);
   mqttclient.setserver(mqttserver, mqttport);
   mqttclient.setcallback(callback);
}

within the loop, we do the following:

if the client is not connected to the broker, we connect it using our client name.
once connected, we also subscribe our client to the mqtt_sub_topic.
we then publish a message to mqtt_pub_topic
we then run the mqttclient.loop(). this loop() function should be called regularly. it maintains the connection of the client with the broker and also helps the client process incoming messages. if you don't have this mqttclient.loop() line, you will be able to publish to mqtt_pub_topic, but won't get messages from mqtt_sub_topic, because the incoming messages are processed only when this line is called.
finally, we wait for 5 seconds, before starting this cycle again.

void loop() {
   // put your main code here, to run repeatedly:
   if (!mqttclient.connected()){
      while (!mqttclient.connected()){
         if(mqttclient.connect(mqtt_client_name)){
            serial.println("mqtt connected!");
            mqttclient.subscribe(mqtt_sub_topic);
         }
         else{
            serial.print(".");
         }
      }
   }
   mqttclient.publish(mqtt_pub_topic, "testmsg");
   serial.println("message published");
   mqttclient.loop();
   delay(5000);
}

testing the code

in order to test the above code, you need to go to www.hivemq.com

follow these steps once you are on that webpage −

click connect

click on add new topic subscription and enter the name of the topic to which your esp32 will publish (/ys/testpub in this case)

once you flash your esp32, you will start receiving messages on that topic every 5 seconds.

next, to test reception of message on esp32, enter the name of the topic your esp32 is subscribed to (ys/testsub in this case), then type a message in the message box and click publish. you should see the message on the serial monitor.

congratulations!! you've tested both publish and subscribe using mqtt on esp32.

references

this chapter gives you an introduction to transmitting data over bluetooth using esp32. arduino has a dedicated bluetoothserial library for esp32 that makes transmitting data over bluetooth as simple as transmitting data to the serial monitor. we will look at how to create a bluetooth field around the esp32, connect our smartphone to that field, and communicate with the esp32.

code walkthrough

we will be using an example code for this chapter. you can find it in file −> examples −> bluetoothserial −>serialtoserialbt. it can also be found on github.

we begin with the inclusion of the bluetoothserial library.

#include <bluetoothserial.h>

the next few lines of code are a bit irrelevant if you haven't messed around with your esp32. they check if bluetooth is configured enabled, and throw up a warning if it is not. bluetooth config is enabled by default on your esp32, so you may as well comment out these lines if you have used your esp32 strictly with the arduino ide. the make menuconfig which the error message refers to is actually accessed through esp−idf and not through arduino ide. so, the bottom line is, don't worry about these lines.

#if !defined(config_bt_enabled) || !defined(config_bluedroid_enabled)
#error bluetooth is not enabled! please run `make menuconfig` to and enable it
#endif

next, we define the bluetoothserial object.

bluetoothserial serialbt;

within setup, we will begin the bluetooth field around our esp32 using the serialbt.begin() function. this function takes an argument, the name of your bluetooth device (esp32 in this case). this is the name that will be shown when you scan for bluetooth networks on your cell phone.

void setup() {
   serial.begin(115200);
   serialbt.begin("esp32test"); //bluetooth device name
   serial.println("the device started, now you can pair it with bluetooth!");
}

now, the loop is quite straightforward here. if you have any text coming in on serial (i.e. text entered by you on the serial monitor), send it to over serialbt. if you have any text coming in from the serialbt, send it over serial, or, in other words, print it on the serial monitor.

void loop() {
   if (serial.available()) {
      serialbt.write(serial.read());
   }
   if (serialbt.available()) {
      serial.write(serialbt.read());
   }
   delay(20);
}

testing the code

to test this code, it is recommended that you download a serial bluetooth terminal app (either the one shown below or any equivalent app) on your smartphone. it will help you pair with your esp32, show you the messages received from esp32, and also help you send messages to your esp32.

to install it on an android device, click here. an equivalent app for ios could be bluterm.

you can find the screenshots of the tests performed with the serial bluetooth terminal app below. i had changed the bluetooth name of esp32 to 'esp32test345', because i had already paired my mobile phone with another esp32 whose bluetooth name was 'esp32test'. once the pairing is done, the device can be added in the serial bluetooth terminal app, and you can then communicate with your device as if communicating with another user on a messaging app.

pairing and communication

corresponding view of arduino serial terminal

congratulations. you've communicated with your esp32 using bluetooth. go ahead and explore the other examples that come along with the bluetoothserial library.

note − you may be tempted to use wifi and bluetooth simultaneously on esp32. this is not recommended. while esp32 has separate stacks for wifi and bluetooth, they share a common radio antenna. therefore, the behavior of the module when both the stacks are trying to access the antenna becomes unpredictable. it is recommended that only one stack accesses the antenna at a time.

in iot devices, the timestamp becomes an important attribute of the packet exchanged between the device and the server. therefore, it is necessary to have the correct time on your device at all times. one way is to use an rtc (real time clock) interfaced with your esp32. you can even use esp32's internal rtc. once given a reference time, it can correctly output future timestamps. but how will you get the reference time? one way is to hardcode the current time while programming the esp32. but that is not a neat method. secondly, the rtc is prone to drift and it is a good idea to keep providing it with reference timestamps regularly. in this chapter, we will see how to get the current time from ntp servers, feed it to esp32's internal rtc once, and print future timestamps.

a brief about ntp

ntp stands for network time protocol. it is a protocol for clock synchronization between computer systems. in layperson terms, there is a server sitting somewhere which maintains time accurately. whenever a client requests the current time from the ntp server, it sends back time accurate up to 100s of milliseconds. you can read more about ntp here. for esp32, there is an in−built time library that handles all the communication with the ntp servers. let's explore the use of that library in the code walkthrough below.

code walkthrough

we will use an in−built example for this walkthrough. it can be found in file −> examples −> esp32 −> time −> simpletime. it can also be found on github.

we begin with the inclusion of the wifi and the time libraries.

#include <wifi.h>
#include "time.h"

next, we define some global variables. replace the wifi ssid and password with the corresponding values for your wifi. next, we have defined the url for the ntp server. the gmtoffset_sec refers to the offset in seconds of your timezone from the gmt or the closely related utc. for instance, in india, where the timezone is 5 hours and 30 mins ahead of the utc, the gmtoffset_sec will be (5+0.5)*3600 = 19800.

the daylightoffset_sec is relevant for countries that have daylight savings. it can simply be set to 0 in other countries.

const char* ssid       = "your_ssid";
const char* password   = "your_pass";

const char* ntpserver = "pool.ntp.org";
const long  gmtoffset_sec = 3600;
const int   daylightoffset_sec = 3600;

next, you can see a function printlocaltime(). it simply fetches the local time from the internal rtc and prints it to serial.

void printlocaltime()
{
   struct tm timeinfo;
   if(!getlocaltime(&timeinfo)){
      serial.println("failed to obtain time");
      return;
   }
   serial.println(&timeinfo, "%a, %b %d %y %h:%m:%s");
}

you might be having three questions here −

where is the struct tm defined?
where is the getlocaltime() function defined?
what are the %a, %b, etc. formatters?

the struct tm is defined in the time.h file that we have included at the top. in fact, the time library is not an esp32 specific library. it is an avr library that is compatible to esp32. you can find the source code at here. if you look at the time.h file, you will see the struct tm.

struct tm {
   int8_t   tm_sec; /**< seconds after the minute - [ 0 to 59 ] */
   int8_t   tm_min; /**< minutes after the hour - [ 0 to 59 ] */
   int8_t   tm_hour; /**< hours since midnight - [ 0 to 23 ] */
   int8_t   tm_mday; /**< day of the month - [ 1 to 31 ] */
   int8_t   tm_wday; /**< days since sunday - [ 0 to 6 ] */
   int8_t   tm_mon; /**< months since january - [ 0 to 11 ] */
   int16_t  tm_year; /**< years since 1900 */
   int16_t  tm_yday; /**< days since january 1 - [ 0 to 365 ] */
   int16_t  tm_isdst; /**< daylight saving time flag */
};

now, the getlocaltime function is esp32 specific. it is defined in the esp32−hal−time.c file. it is a part of the arduino core for esp32 and doesn't need a separate include in arduino. you can see the source code here.

now, the meaning of the formatters is given below −

/*
   %a abbreviated weekday name
   %a full weekday name
   %b abbreviated month name
   %b full month name
   %c date and time representation for your locale
   %d day of month as a decimal number (01−31)
   %h hour in 24-hour format (00−23)
   %i hour in 12-hour format (01−12)
   %j day of year as decimal number (001−366)
   %m month as decimal number (01−12)
   %m minute as decimal number (00−59)
   %p current locale's a.m./p.m. indicator for 12−hour clock
   %s second as decimal number (00−59)
   %u week of year as decimal number,  sunday as first day of week (00−51)
   %w weekday as decimal number (0−6; sunday is 0)
   %w week of year as decimal number, monday as first day of week (00−51)
   %x date representation for current locale
   %x time representation for current locale
   %y year without century, as decimal number (00−99)
   %y year with century, as decimal number
   %z %z time-zone name or abbreviation, (no characters if time zone is unknown)
   %% percent sign
   you can include text literals (such as spaces and colons) to make a neater display or for padding between adjoining columns.
   you can suppress the display of leading zeroes  by using the "#" character  (%#d, %#h, %#i, %#j, %#m, %#m, %#s, %#u, %#w, %#w, %#y, %#y)
*/

thus, with our formatting scheme of %a, %b %d %y %h:%m:%s, we can expect the output to be similar to the following: sunday, november 15 2020 14:51:30.

now, coming to the setup and the loop. in the setup, we initialize serial, connect to the internet using our wifi, and configure the internal rtc of esp32 using the configtime() function. as you can see, that function takes in three arguments, the gmtoffset, the daylightoffset and the ntpserver. it will fetch the time from ntpserver in utc, apply the gmtoffset and the daylightoffset locally, and return the output time. this function, like getlocaltime, is defined in the esp32-hal-time.c file. as you can see from the file, tcp/ip protocol is used for fetching time from the ntp server.

once we've obtained the time from the ntp server and fed it to the internal rtc of the esp32, we no longer need wifi. thus, we disconnect the wifi and keep printing time in the loop every second. you can see on the serial monitor that the time gets incremented by one second in every print. this is because the internal rtc of esp32 maintains the time once it got the reference.

void setup()
{
   serial.begin(115200);
  
   //connect to wifi
   serial.printf("connecting to %s ", ssid);
   wifi.begin(ssid, password);
   while (wifi.status() != wl_connected) {
      delay(500);
      serial.print(".");
   }
   serial.println(" connected");
  
   //init and get the time
   configtime(gmtoffset_sec, daylightoffset_sec, ntpserver);
   printlocaltime();

   //disconnect wifi as it's no longer needed
   wifi.disconnect(true);
   wifi.mode(wifi_off);
}
void loop() {
  delay(1000);
  printlocaltime();
}

the serial monitor output will look like −

that's it. you've learned how to get the correct time from the ntp servers and configure your esp32's internal rtc. now, in whatever packets you send to the server, you can add the timestamp.

references

network time protocol - wikipedia

say you have a thousand iot devices out in the field. now, if one fine day, you find a bug in the production code, and wish to fix it, will you recall all the thousand devices and flash the new firmware in them? probably not! what you'll prefer to have is a way to update all the devices remotely, over-the-air. ota updates are very common these days. every now and then, you keep receiving software updates to your android or ios smartphones. just like software updates can happen remotely, so can firmware updates. in this chapter, we will look at how to update the firmware of esp32 remotely.

ota update process

the process is quite simple. the device first downloads the new firmware in chunks and stores it in a separate area of the memory. let's call this area the 'ota space'. let's call the area of the memory where the current code or the application code is stored as the 'application space'. once the entire firmware has been downloaded and verified, the device bootloader swings into action. consider the bootloader as a code written in a separate area of the memory (let's call it the 'bootloader space'), whose sole purpose is to load the correct code in the application space every time the device restarts.

thus, every time the device restarts, the code in the bootloader space gets executed first. most of the time, it simply passes control to the code in the application space. however, after downloading the newer firmware, when the device restarts, the bootloader will notice that a newer application code is available. so it will flash that newer code from the ota space into the application space and then give control to the code in the application space. the result will be that the device firmware will be upgraded.

now, digressing a bit, the bootloader can also flash the factory reset code from the 'factory reset space' to the application space, if the application code is corrupted, or a factory reset command is sent. also, often, the ota code and the factory reset codes are stored on external storage devices like an sd card or an external eeprom or flash chip, if the microcontroller doesn't have enough space. however, in the case of esp32, the ota code can be stored in the microcontroller's memory itself.

code walkthrough

we will be using an example code for this chapter. you can find it in file −> examples −> update −> aws_s3_ota_update. it can also be found on github.

this is one of the very detailed examples available for esp32 on arduino. the author of this sketch has even provided the expected serial monitor output of the sketch in comments. so while much of the code will be self−explanatory through the comments, we'll walk over the broad idea and also cover the important details. this code makes use of the update library which, like many other libraries, makes working with esp32 very easy, while keeping the rigorous work under−the−hood.

in this specific example, the author has kept the binary file of the new firmware in an aws s3 bucket. providing a detailed overview of aws s3 is beyond the scope of this chapter, but very broadly, s3 (simple storage service) is a cloud storage service provided by amazon web services (aws). think of it like google drive. you upload files to your drive and share a link with people to share it. similarly, you can upload a file to s3 and access it via a link. s3 is much more popular because a lot of other aws services can interface seamlessly with it. getting started with aws s3 will be easy. you can get help from several resources available through a quick google search. in the comments at the beginning of the sketch as well, a few steps to get started are mentioned.

an important recommendation to note is that you should use your own binary file for this code. the comments at the top of the sketch suggest that you can use the same binary file that the author has used. however, downloading a binary compiled on another machine/ another version of arduino ide has been known to cause errors sometimes in the ota process. also, using your own binary will make your learning more 'complete'. you can export the binary of any esp32 sketch by going to sketch −> export compiled binary. the binary (.bin) file gets saved in the same folder in which your arduino (.ino) file is saved.

once your binary is saved, you just need to upload it to s3 and add the link to the bucket and address of the binary file in your code. the binary you save should have some print statement to indicate that it is different from the code you flash in the esp32. a statement like "hello from s3" maybe. also, don't keep the s3 bucket link and bin address in the code as it is.

alright! enough talk! let's begin the walkthrough now. we will begin by including the wifi and update libraries.

#include <wifi.h>
#include <update.h>

next, we define a few variables, constants, and also the wificlient object. remember to add your own wifi credentials and s3 credentials.

wificlient client;

// variables to validate
// response from s3
long contentlength = 0;
bool isvalidcontenttype = false;

// your ssid and pswd that the chip needs
// to connect to
const char* ssid = "your−ssid";
const char* pswd = "your−ssid−pswd";

// s3 bucket config
string host = "bucket−name.s3.ap−south−1.amazonaws.com"; // host => bucket−name.s3.region.amazonaws.com
int port = 80; // non https. for https 443. as of today, https doesn't work.
string bin = "/sketch−name.ino.bin"; // bin file name with a slash in front.

next, a helper function getheadervalue() has been defined, which basically is used to check the value of a particular header. for example, if we get the header "content-length: 40" and it is stored in a string called headers, getheadervalue(headers,"content−length: ") will return 40.

// utility to extract header value from headers
string getheadervalue(string header, string headername) {
   return header.substring(strlen(headername.c_str()));
}

next, the main function execota(), which performs the ota. this function has the entire logic related to the ota. if you look at the setup, we simply connect to the wifi and call the execota() function.

void setup() {
   //begin serial
   serial.begin(115200);
   delay(10);

   serial.println("connecting to " + string(ssid));

   // connect to provided ssid and pswd
   wifi.begin(ssid, pswd);

   // wait for connection to establish
   while (wifi.status() != wl_connected) {
      serial.print("."); // keep the serial monitor lit!
      delay(500);
   }

   // connection succeed
   serial.println("");
   serial.println("connected to " + string(ssid));

   // execute ota update
   execota();
}

so you would have understood that understanding the execota function means understanding this entire code. therefore, let's begin the walkthrough of that function.

we begin by connecting to our host, which is the s3 bucket in this case. once connected, we fetch the bin file from the bucket, using a get request (refer to the http tutorial for more information on get requests)

void execota() {
   serial.println("connecting to: " + string(host));
   // connect to s3
   if (client.connect(host.c_str(), port)) {
   // connection succeed.
   // fecthing the bin
   serial.println("fetching bin: " + string(bin));

   // get the contents of the bin file
   client.print(string("get ") + bin + " http/1.1\r\n" +
      "host: " + host + "\r\n" +
      "cache-control: no-cache\r\n" +
      "connection: close\r\n\r\n");

next, we wait for the client to get connected. we give a maximum of 5 seconds for the connection to get established, otherwise we say that the connection has timed out and return.

unsigned long timeout = millis();
while (client.available() == 0) {
   if (millis() - timeout > 5000) {
      serial.println("client timeout !");
      client.stop();
      return;
   }
}

assuming that the code has not returned in the previous step, we have a successful connection established. the expected response from the server is provided in the comments. we begin by parsing that response. the response is read line by line, and each new line is stored in a variable called line. we specifically check for the following 3 things −

if the response status code is 200 (ok)
what is the content-length
whether the content type is application/octet-stream (this is the type expected for a binary file)

the first and third are required, and the second is just for information.

while (client.available()) {
   // read line till /n
   string line = client.readstringuntil('\n');
   // remove space, to check if the line is end of headers
   line.trim();

   // if the the line is empty,
   // this is end of headers
   // break the while and feed the
   // remaining `client` to the
   // update.writestream();
   if (!line.length()) {
   //headers ended
   break; // and get the ota started
   }

   // check if the http response is 200
   // else break and exit update
   if (line.startswith("http/1.1")) {
      if (line.indexof("200") < 0) {
         serial.println("got a non 200 status code from server. exiting ota update.");
         break;
      }
   }

   // extract headers here
   // start with content length
   if (line.startswith("content-length: ")) {
      contentlength = atol((getheadervalue(line, "content-length: ")).c_str());
      serial.println("got " + string(contentlength) + " bytes from server");
   }

   // next, the content type
   if (line.startswith("content-type: ")) {
      string contenttype = getheadervalue(line, "content-type: ");
      serial.println("got " + contenttype + " payload.");
      if (contenttype == "application/octet-stream") {
         isvalidcontenttype = true;
      }
   }
}

with this, the if block that checks if the connection with the server was successful ends. it is followed by the else block, which just prints that we were unable to establish connection to the server.

} else {
   // connect to s3 failed
   // may be try?
   // probably a choppy network?
   serial.println("connection to " + string(host) + " failed. please check your setup");
   // retry??
   // execota();
}

next, if we have hopefully received the correct response from the server, we will have a positive contentlength (remember, we had initialized it with 0 at the top and so it will still be 0 if we somehow did not reach that line where we parse the content−length header). also, we will have isvalidcontenttype as true (remember, we had initialized it with false). so we check if both of these conditions are true and if yes, proceed with the actual ota. note that so far, we have only made use of the wifi library, for interacting with the server. now if the server interaction turns out to be alright, we will begin use of the update library, otherwise, we simply print that there was no content in the server response and flush the client. if the response was indeed correct, we first check if there is enough space in the memory to store the ota file. by default, about 1.2 mb of space is reserved for the ota file. so if the contentlength exceeds that, update.begin() will return false. this 1.2mb number can change depending on the partitions of your esp32.

// check contentlength and content type
if (contentlength && isvalidcontenttype) {
   // check if there is enough to ota update
   bool canbegin = update.begin(contentlength);

now, if we indeed have space to store the ota file in memory, we begin writing the bytes to the memory area reserved for ota (the ota space), using the update.writestream() function. if we don't, we simply print that message and flush the client, and exit the ota process. the update.writestream() function returns the number of bytes that were written to the ota space. we then check if the number of bytes written is equal to the contentlength. if the update is completed, in which case the update.end() function will return true, we check if it has finished properly, i.e. all bytes are written, using the update.isfinished() function. if it returns true, meaning that all bytes have been written, we restart the esp32, so that the bootloader can flash the new code from the ota space to the application space, and our firmware gets upgraded. if it returns false, we print the error received.

   // if yes, begin
   if (canbegin) {
      serial.println("begin ota. this may take 2 − 5 mins to complete. things might be quite for a while.. patience!");
      // no activity would appear on the serial monitor
      // so be patient. this may take 2 - 5mins to complete
      size_t written = update.writestream(client);

      if (written == contentlength) {
         serial.println("written : " + string(written) + " successfully");
      } else {
         serial.println("written only : " + string(written) + "/" + string(contentlength) + ". retry?" );
         // retry??
         // execota();
      }

      if (update.end()) {
         serial.println("ota done!");
         if (update.isfinished()) {
            serial.println("update successfully completed. rebooting.");
            esp.restart();
         } else {
            serial.println("update not finished? something went wrong!");
         }
      } else {
         serial.println("error occurred. error #: " + string(update.geterror()));
      }
   } else {
      // not enough space to begin ota
      // understand the partitions and
      // space availability
      serial.println("not enough space to begin ota");
      client.flush();
   }
}

of course, you would have realized by now that we need not do anything in the loop here.

that's it. you've successfully upgraded the firmware of your esp32 chip remotely. if you are more curious about what each function of the update library does, you can refer to the comments in the update.h file.

references

ota on esp32

now that you are fairly familiar with esp32, let's look at its application. this is a chapter where i feel i need not tell you much. after going through the various chapters in this tutorial, you would have started forming ideas in your head. you would have already created a rough list of applications where you could use esp32. and the good news is that most of the applications you've listed are feasible.

however, esp32 is more feasible for some applications than others. in this chapter, my focus will be on making you understand the factors you should consider when deciding whether to use esp32 for an application or not. please note that this chapter is production focused, i.e., when we are talking of a scale of thousands or lakhs of devices. if you have a requirement of a handful of devices and esp32 can serve them, just simply go ahead and use esp32 without a second thought. also, for prototyping/ establishing proof of concept (poc), you can use esp32 without any hesitation.

one of the major advantages of esp32 is the presence of inbuilt wifi and bluetooth stacks and hardware. therefore, esp32 will be your choice of microcontroller in a static application where good wifi connectivity is guaranteed, say an environment monitoring application in, say, a laboratory. the presence of wifi stack on the module itself means you will have saved money on an additional networking module. however, if you use esp32 in an asset tracking application, where it keeps moving around, you will have to rely on a gsm or lte module for connectivity to the server (because you will not be guaranteed wifi availability). in such a scenario, esp32 loses the competitive advantage, and you may be better off using a cheaper microcontroller that can serve your purpose.

on similar lines, having a hardware accelerator for encrypting messages makes esp32 ideal for applications requiring secure communication (https). so if you are dealing with sensitive information that you don't want to fall in the wrong hands, using esp32 is advantageous over other microcontrollers that don't support encryption. an example application can be industrial iot in the defense sector.

the presence of two cores again makes esp32 your choice of microcontroller for processing-heavy applications like those receiving data at a very high baud rate and requiring the data processing and transmission to run on separate cores. several such applications can be found in industrial iot. but for a very light application, where you don't even require secure communication, a microcontroller having modest specs may prove to be more useful. after all, what is the use of having (and effectively paying for) two cores, when you can make do with one?

another factor to consider is the number of gpios and peripherals. esp32 has 3 uart channels. if you have an application where you need more than 3 uart channels, you may have to look for another microcontroller. similarly, esp32 has 34 programmable gpios which are more than sufficient for most applications. however, if your application does require more gpios, you may have to switch to another microcontroller

esp32's 1.5 mb of default spiffs provides you with more storage onboard the microcontroller than most other microcontrollers. if your storage requirements are within 1.5 mb, esp32 saves you the cost of an external sd card or flash chip. esp32 does wear-leveling within spiffs by itself, saving you a lot of development efforts as well. however, if your storage requirements are not met by esp32, again, the competitive advantage disappears.

esp32's 520 kb of ram is also more than sufficient for most applications. it is only for very heavy applications like image/video processing where this proves to be a bottleneck.

to summarize, esp32 has specs that are good enough to accommodate most of your applications. when scaling up production, you need to just make sure that the specs are not an overkill for you. in other words, if you can get the desired output with modest specs, you may be better off using a cheaper microcontroller and save money. these savings become significant when your production numbers increase by orders of magnitude. however, production aside, esp32 is definitely the ideal microcontroller for prototyping and establishing the poc.

if you have made it so far, congratulations! you are at a stage where you are not only familiar with esp32 but also have enough know-how to explore it further. and there is indeed a lot more to explore. several cool applications can be realized if you know some additional concepts. in this chapter, i'll just be giving you directions for this exploration. in fact, it will be much better if i just list them down. given below is the non−exhaustive list of topics/ areas which you can explore further, and concepts which you can learn about:

firmware

sleep modes − you will require to know these in applications which have a scarcity of power
timers − for scheduled tasks
interrupts − for tasks triggered by an asynchronous event
watchdog timeout − reset your microcontroller if it is stuck somewhere for too long
mutex and semaphore (rtos related) − when multiple threads want to access a shared resource
adjusting the partitions of esp32 for providing more space to spiffs

sensor interface

touch sensors with esp32
hall sensor with esp32
gnss receiver with esp32 (almost all iot devices have a gnss receiver for getting location information)

network related

ble (bluetooth low energy) with esp32
connecting esp32 to an external bluetooth field
wifi in access point (ap) with esp32
esp32 as a web server
captive portal using esp32 (in restaurants or airports, the screen that pops up as soon as you connect to the wifi, prompting you for your mobile number, that is called the captive portal)
data transfer using udp
data transfer using tcp
dns server
data transfer over lora
mqtt with brokers on aws/ azure

peripheral connectivity

can protocol (used for automotive applications)
i2c and spi with multiple slaves
interface of sd card with esp32
one wire protocol

data processing

fft on esp32
power calculations (rms value, peak to peak, power factor, etc.)

don't get overwhelmed by looking at the above list. you don't have to learn everything in a day. nor are you required to learn everything. as mentioned in the beginning, the above list just provides you with directions for further exploration. you can choose topics that suit your application. for all the topics where you have an example code available, there is no better place to get started. for others, you can read the documentation of the available libraries related to those topics, go through their .h files, and browse through the internet for examples. with such a massive online community, it is very difficult to not find help online. so, keep learning and keep exploring .

Save or share this article