Technical University of Denmark: World’s smallest fruit picker controlled by artificial intelligence

The goal of Kaare Hartvig Jensen, Associate Professor at DTU Physics, was to reduce the need for harvesting, transporting, and processing crops for the production of biofuels, pharmaceuticals, and other products. The new method of extracting the necessary substances, which are called plant metabolites, also eliminates the need for chemical and mechanical processes.

Plant metabolites consist of a wide range of extremely important chemicals. Many, such as the malaria drug artemisinin, have remarkable therapeutic properties, while others, like natural rubber or biofuel from tree sap, have mechanical properties.

Harvesting cell by cell
Because most plant metabolites are isolated in individual cells, the method of extracting the metabolites is also important, since the procedure affects both product purity and yield.

Usually the extraction involves grinding, centrifugation, and chemical treatment using solvents. This results in considerable pollution, which contributes to the high financial and environmental processing costs.

“All the substances are produced and stored inside individual cells in the plant. That’s where you have to go in if you want the pure material”
Kaare Hartvig Jensen, Associate Professor at DTU Physics.
“All the substances are produced and stored inside individual cells in the plant. That’s where you have to go in if you want the pure material. When you harvest the whole plant or separate the fruit from the branches, you also harvest a whole lot of tissue that doesn’t contain the substance you’re interested in,” explains Kaare Hartvig Jensen, Associate Professor at DTU Physics.

“So there are two perspectives to it. If you want to extract the pure substances, you need to do it cell by cell. And when you can do that, as we’ve shown, you don’t have to harvest the plant. Then you can put the little robot on and it can work without damaging the plant,” says Kaare.


The team is currently working with plants and leaves, but in the future this type of harvester may be used on a slightly larger scale.

The hope is that this unique approach can create a new source of biomass and spark research into a new area of sustainable energy production.

One thing the technology might be used for in the future is tapping energy from trees, which contain a lot of biofuel.

“In the forests of northern Canada and Russia, there are spruce forests with around 740 billion trees that are completely untouched. That’s about 25% of the total number of trees on the planet. By developing this technology, we can tap trees for sugar and make biofuel without chopping down or damaging the trees,” explains Kaare.

Artificial intelligence at a microscopic level
The cells in the fruit and leaves that the harvester looks for are 100 microns in diameter, and the tip of the needle is around 10 microns in diameter. The harvest is therefore happening on the scale of the width of a hair.

Magnus Valdemar Paludan, PhD student at DTU Physics who created the system of image analysis, image recognition, and robot control, explains.

“It’s all done with a microscope camera. To begin with, I manually marked pixels on the microscopy images showing the cells that the robot will harvest. That information can be used to train a computer to find similar cells in new images.”

Machine learning and a pre-existing neural network, GoogLeNet, are the building blocks of the technology. The network can already recognize macroscopic structures and can sift through an image and tell you if, for example, there’s an elephant or a red pepper hiding in the photo.

“We used a technique called transfer learning, where you use the existing neural network’s ability to recognize different objects in an image. By showing the computer a number of new images with the manually marked cells, we succeeded in adjusting the network’s parameters so it recognizes the microscopic metabolite-rich cells,” says Magnus.

“The harvester can then go in and take a picture of the leaf with the microscope camera, run it through the software, and recognize the cells it needs to harvest. Next, it can extract the chemicals automatically using a microrobot, while the rest of the plant remains undisturbed,” explains Magnus.

The results stem from Kaare Hartvig Jensen’s VILLUM Experiment project, for which he received a grant in 2017.