On the Edge

Machine learning has changed the computing paradigm. Products today are built with artificial intelligence (AI) as a central attribute, and consumers are beginning to expect automation and human-like interactions with the devices they are using. However, much of the deep learning revolution has been limited to the cloud. Recently, and thanks to the performance increase of microcontroller and embedded devices, several machine learning libraries for mobile, embedded, and Internet of Things (IoT) devices have been announced that aim to offload computation to the edge. One such toolset is TensorFlow Lite, which I use in a real-world vertical farming operation.

Although building and training deep neural networks requires powerful servers provided with graphics processing units (GPUs) or tensor processing units (TPUs), machine learning inference requires fewer resources and can be executed on the edge. Developers can add intelligence to IoT devices that perform tasks such as anomaly detection, speech recognition, or regression (i.e., making a prediction), without depending on a cloud and thus wireless connectivity. Edge computing is so much more scalable, because the computation cost is spread over the end devices, instead of being centralized in a cloud. For some use cases, like speech recognition, AI on the edge guarantees better privacy, because personal data is kept on the device in your home or office.

In this article, I focus on the use of TensorFlow Lite [1] on IoT devices and, more precisely, in the application domain of vertical farming. Machine learning and artificial intelligence will help optimize vegetable yield or predict the harvesting date of plants.

The first section briefly introduces the concept of vertical farming before introducing TensorFlow Lite and looking at how it differs from TensorFlow. Then, I present the connected vertical farm and why and how it is used with TensorFlow Lite. The main body is a tutorial on the design and training of a salad greens growth model with TensorFlow, before optimizing and predicting on the edge (i.e., on device) the harvesting date of the salad greens with TensorFlow Lite. Finally, I conclude the article by summing up my thoughts on TensorFlow Lite and provide feedback and the insights gained from my experience.

Vertical Farming

Vertical farming is a new method of growing crops in vertically stacked layers in a fully controlled environment. Vertical farming aims to minimize water use and maximize productivity by growing crops stacked in a climate-controlled cabinet without soil in small amounts of nutrient-rich water.

Thus, vertical farming could help meet growing global food demands in a sustainable way by reducing distribution chains, producing fresh greens and vegetables with lower emissions close to populations, and providing higher nutrient produce with a better taste and look.

During the last five years, many companies have begun developing vertical farming systems, choosing structures like shipping containers, buildings, tunnels, or cabinet-like systems, which is the system I present in this article (Figure 1).

Figure 1: The vertical farm uses a glass cabinet as a greenhouse. Various sensors monitor the production of salad greens.

Two categories of soilless farming techniques are used in vertical farming:

In systems that use the hydroponic nutrient film technique (NFT), the roots of greens are submerged in a very shallow stream of water solution containing macronutrients (nitrogen, phosphorus, potassium, etc.). Optionally, in a hydroponic ebb and flow system, an inert medium (e.g., gravel or sand) is used as a soil substitute to provide support for the roots.

Unlike hydroponics, aeroponics does not require a liquid or solid medium in which to grow plants; instead, a liquid solution with nutrients is applied with a high-pressure jet or with misted fog, hereafter referred to as high-pressure aeroponics (HPA) and nebulization, respectively. Nebulization aeroponics is the most sustainable soilless growing technique, because it uses up to 90 percent less water than the most efficient conventional hydroponic systems and requires no replacement of growth medium.

Beyond technical and ecological considerations, the irrigation system strongly affects the growth of vegetables, which is why it is a key parameter to take into account in the artificial intelligence.

TensorFlow and TensorFlow Lite

TensorFlow is an end-to-end machine learning platform initially developed by the Google Brain team for internal Google use. The free and open source software library, released under Apache License 2.0 in 2015, allows a wide range of machine learning tasks, such as the design and training of neural networks, on many platforms, including CPUs, GPUs, and TPUs. November 2019 saw the release of TensorFlow 2.0, which was used for the project this article.

TensorFlow provides stable Python (for version 3.7) and C APIs. However, other programming languages supported without guaranteed API backward compatibility are C++, Go, Java, JavaScript, and Swift (early release). In this article, I use the Python API.

The TensorFlow platform is quite complex and comes with many components, like TensorBoard to debug and study the training steps of a model, TensorFlow.js to execute a model and run inference in a web browser, and TensorFlow Lite for devices with low memory and low computation capabilities.

TensorFlow Lite provides all the building blocks to convert and optimize a TensorFlow model on mobile, embedded, and IoT devices and to execute the model on these constrained edge devices. However, TensorFlow Lite, unlike TensorFlow, cannot be used to develop and train a machine learning model.

In short, TensorFlow Lite comes with two components: the TensorFlow Lite Converter and the TensorFlow Lite Runtime. The TensorFlow Lite Converter offers the necessary API and optimizers to reduce the memory footprint of a TensorFlow model, increase execution speed, and lower computation costs.

Usually, Python data scientists perform data engineering and preprocessing tasks on lab computers with pandas [2] or NumPy [3] libraries, before using TensorFlow to design and train a deep learning model. Note that the training phase can be offloaded to a GPU in the cloud. Then, on the lab computer, the researcher uses the TensorFlow Lite Converter, which comes with the TensorFlow distribution, to optimize and convert the trained model in a .tflite model optimized for edge devices. This .tflite model is then deployed to the edge device as a software update.

Finally, at run time, the edge device uses the TensorFlow Lite Runtime to execute the model and make predictions with live data from the vertical farm. A key point is that the TensorFlow distribution is not installed on the device, only the standalone TensorFlow Lite Runtime. Depending on the edge device platform, the TensorFlow Lite Runtime is either provided in an AAR package from JCenter for Android, a Python wheel package, or a C shared library for embedded devices.

Vertical Farming System

The cabinet for indoor vertical farming shown in Figure 1 has four shelves able to support 12 plants spread on a grid. The full vertical farming system is divided into four components as summarized in Figure 2:

Figure 2: The cupboard has four tiers of vegetables. The data ends up on a local Raspberry Pi.

• The cabinet with the sensors, actuators, and irrigation systems
• The motherboard, which for this project is a Raspberry Pi 3B+
• The cloud server
• The research and development (R&D) computer

The cabinet houses the following eight sensors and two cameras per layer: (1) carbon dioxide (CO2, ppm), (2) dissolved oxygen (ppm), (3) electrical conductivity (µS/cm), (4) oxidation-reduction (redox) potential (mV), (5) photosynthetic photon flux density (PPFD, µmol/sq m/sec), (6) water pH, (7) humidity (%), and (8) temperature (°C). The prototype implements the four different irrigation systems introduced earlier (one on each layer) to study their effect on plant growth and quality. Actuators help control the temperature, level of CO2, and lighting.

The Raspberry Pi is the brains of the system. It aggregates data from the sensors, controls the irrigation systems and actuators, and computes the plant leaf area with the help of cameras and several image processing algorithms developed with OpenCV [4] (Figure 3). At the same time, it runs the TensorFlow Lite Runtime to execute the plant growth prediction deep learning algorithm in real time. All the data is "cached" locally on the Raspberry Pi before being sent to the cloud API when Internet is available. The Raspberry Pi software is written in Python.

Figure 3: OpenCV keeps an eye on growth by measuring the size of lettuce heads. TensorFlow then predicts the harvest date.

The cloud has three main features: an API developed in Go to collect the data from the motherboard (i.e., the Raspberry Pi); a Postgres database to store the data transmitted by the cabinet motherboard, as well as information detailing the experiments and plant features (variety, date of planting, etc.), mainly for agronomic research purposes; and finally a web interface for data visualization developed with the Elasticsearch-Logstash-Kibana (ELK) stack [5].