A post originally published on Google’s TensorFlow Blog announced TensorFlow Lite Micro support on Espressif’s flagship SoC, ESP32. Here’s what it says:

ESP32 is already being used in a number of smart-home/connected-device projects with a variety of sensors and actuators connected to the microcontroller to sense the environment and act accordingly. With TensorFlow Lite for Microcontrollers executing on ESP32, this opens up scenarios for all kinds of use-cases that are triggered by local inference. ESP32 has 2 CPU cores and a bunch of optimizations, making it easier to run heavy TF Micro workloads. The Wi-Fi backhaul helps to raise remote events and trigger actions based on the inferences made.

Person Detection or a Door-Bell Camera?

As an example, we have modified the person_detection example that you all might be familiar with to make it a smart door-bell camera. We use the ESP-EYE developer kit for this demonstration. Note that this example uses person detection (it detects when a face is in front of the camera), and not person identification (identifying who the person is).

The ESP-EYE dev kit includes the ESP32 Wi-Fi/BT MCU coupled with a 2MP camera.

In Action

In our example, we will use this camera to observe and send out an email notification if we detect a person in the vicinity.

Check out a cool GIF showing this example in action, by clicking here!

Building It for Yourself

• Order the ESP-EYE: You can get the ESP-EYE Development Kit from your favourite distributor or from here. You will need a USB-to-microUSB cable for connecting this to your Windows/Linux/macOS host.
• Clone the repository: https://github.com/espressif/tensorflow/
• Setup your development host: Set up your development host with toolchains and utilities required to cross-build for ESP32. Follow the instructions of the ESP-IDF getting-started guide to set up the toolchain and the ESP-IDF itself.
• Generate the example: The example project can be generated with the following command:make -f tensorflow/lite/micro/tools/make/Makefile TARGET=esp generate_doorbell_camera_esp_project
• Build the example:
  • • Go to the example project directory: cd tensorflow/lite/micro/tools/make/gen/esp_xtensa-esp32/prj/doorbell_camera/esp-idf
    • Clone the esp32-camera component with following command:$ git clone https://github.com/espressif/esp32-camera components/esp32-camera
    • Configure the camera and the email address:idf.py menuconfig
    • Enter the Camera Pins configuration and SMTP Configuration menus to select the camera details, and also the email details.
    • Build the example:idf.py build
• Flash and Run the program: Use the following command to flash and run the program:idf.py --port /dev/ttyUSB0 flash monitor
• Now, whenever a person’s face is detected, the program will send out an email to the configured email address.

What Next?

Now that you have tried the door bell camera example, you may try the other applications that are part of the TF Micro repository: hello_world and micro_speech.

ESP32 is pretty powerful for a microcontroller. Clocked at 240MHz, with just a single core it can do the detection well under 1 second (roughly ~700ms; additional optimizations are on the way to reduce this even further). This leaves the second core free for other tasks from your application.

The TinyML book is an excellent resource for a thorough understanding of TensorFlow Lite for Microcontrollers.