Robotics for Agrifood

Use Case Description

Enormous challenges, such as climate change, nitrogen leaching, and volatilization, EC Green Deal, workforce shortages, farmer aging, the energy crisis, etc., are putting pressure on our Flemish agriculture towards sustainability and require tremendous resilience from our farmers. At the same time, sustainability, self-sufficiency and food autonomy have never been more actual. 

The main objective of this use case is to enable precision robotics in agricultural (crop farming) environments. This will support the ongoing trend towards Precision Crop Farming, whereby the inputs (fertilizer, plant protection products, fuel, labor) are minimized to achieve an optimal net yield for sustainable agricultural production. An essential part is the increase in mapping and operation resolution. Precise robots and AI can process the exponentially growing amount of data and enable smart and precise actuation. The robot’s autonomy enables more precise field operations without compromising labor costs. Suddenly, this can make currently labor-intensive sustainable agricultural practices economically beneficial. In this way, we believe AI and robotics will play an important role in the twin transition towards a green, sustainable and more efficient digital agriculture. 

However, using a robot on a farm today comes with the risk of interchanging machine operation tasks with supervision and monitoring tasks. To increase the efficiency of using robots, it is important to increase the robots' adaptability to unforeseen events, assess the quality of the robot's operations, and continuously transfer information to optimize subsequent agronomic tasks. The automated robots operate the field, but in the meantime, the sensors collect burdens of data. The developments in sensor technology create possibilities for collecting high-dimensional data (e.g., LiDaR, high-resolution RGB, multispectral, hyperspectral). The data can be used to assess the quality of the operation, the field condition, and detect problems and prepare curative operations (e.g., growth delay, biotic stress, abiotic stress, etc.). However, the fast and autonomous processing of all these data collected on different robotic systems introduces some bottlenecks, such as merging various data streams (sensor fusion), data processing, and data transmission within a limited bandwidth. Dimensionality reduction and sensor fusion algorithms can extract meaningful information from the data, automating decision-making on the farm. The robotic control algorithms are often manually finetuned models that require expert knowledge during the implementation and sometimes the farmer’s supervision during the operation. Hierarchical control by reinforcement learning models in virtual and non-virtual environments has the potential to enable high-level decision-making and thus increase the adaptability of autonomous robots, reducing supervision time. Lastly, capturing the farmers' tacit knowledge and building trust among them is essential for the large-scale adoption of AI and robotics in agriculture. Explainable AI could, therefore, be a major enabler in the sector that gives the farmers complete insight and control of their farming practices.

In conclusion, the Robotics for Agrifood use case will contribute to high-dimensional data interpretation using sensor fusion and dimensionality reduction to make complex data usable for preparing successive field operations within a limited set-up time. Explainable AI will keep the farmer informed and in control. Meanwhile, the robot will be more adaptable, with a new reinforcement learning navigation- and task controller that better anticipates changing conditions and reduces the number and duration of (remote) interventions. 

Results, Highlights & Research Outputs 

Agricultural Robot Taskmap Operation Framework (ARTOF)

The Agricultural Robot Taskmap Operation Framework (ARTOF) was previously developed and is further improved in FAIR for better AI compatibility with, e.g., Reinforcement Learning (RL) and Vision-Language-Action VLA models. It provides the common functionality of task map execution for a wide range of robot configurations and farming applications based on Global Navigation Satellite System (GNSS) positioning. ARTOF is implemented on different in-house developed and modified agricultural robot platforms that support plant, soil, and technological research projects. The framework will facilitate data collection and enable quick, practical validation of the new developments within the FAIR project. 

Documentation available at https://artof-ilvo.github.io/
Software available at https://github.com/artof-ilvo

Sensor setup

Digital and physical test fields

Vineyard scan

Through indoor and outdoor experiments with the newly developed sensor setup, additional insights are obtained into the 3D modelling of vineyards under controlled conditions. The next step is to scale this technology to enable automated generation of a synthetic vineyard and to reduce the real-to-simulation (real-to-sim) gap, thereby providing a tool for creating a realistic environment for testing and validating AI-based research independent of the growing season.

Reinforcement Learning navigation controller

The first Reinforcement Learning (RL) navigation controller, developed in this use case, used GNSS sensor data and an occupancy map to navigate the Treebot through the vineyard. RL algorithms enable intelligent integration of the robot's navigation and task execution, shifting from "programming" the robot's task to higher-level instructions. This adaptability allows the robot to handle unexpected events effectively, ensuring complex task execution in dynamic environments. To extend these capabilities, the sensor setup combining LiDAR, camera and GNSS data will be used. Therefore, a unified input representation is being developed that is identical for training in simulation and real-world operation. To this end, an algorithm is designed to map the robot’s environment captured by the camera-LiDAR sensor setup into a comprehensive bird’s-eye-view representation with annotated regions for trees, humans, and other obstacles. This representation is consistent across simulation and reality and thereby suppresses the simulation-to-real (sim-to-real) gap when deploying the RL models in practice.

Hyperspectral dataset

Today, hyperspectral measurements are mostly done under controlled conditions in the lab. Applying this technology in the field is challenging. There are varying lighting conditions; dimensionality reduction is needed to produce manageable and transferable data.

To facilitate this research, a new dataset was created, whereby 60 Chardonnay grape bunches from 60 trees were scanned in the field and in laboratory conditions. Field scans were conducted using the robot arm on the Treebot platform and the Specim FX10 scanner. They were executed in different orientations to see if continuous driving while scanning would be possible.  The selected bunches were scanned again under laboratory conditions, and brix and acidity were determined on individual grapes.  Sixteen other trees were continuously scanned, and selected grape bunches and grapes were analyzed in the laboratory for reference to investigate the possibility of a continuous prediction to make the process quicker. 

This dataset allowed for the creation of a new model called “Light-Invariant Spectral Autoencoder (LISA)”. LISA is a domain invariant model for the prediction of the brix and acidity of grapes based on hyperspectral imaging (HSI). 

Relevant publication

Cornelissen, C., De Coninck, S., Willekens, A., Leroux, S., & Simoens, P. (2025). In-field mapping of grape yield and quality with illumination-invariant deep learning. IEEE INTERNET OF THINGS JOURNAL, 12(23), 49207–49220. https://doi.org/10.1109/JIOT.2025.3617805

Explanation for grape diseases

AI models are often described as "black boxes" because they lack transparency and interpretability. An explainable AI model that characterized vine leaf health was developed; the model’s decision was followed by providing the model's reasoning in natural language and suggesting actions for the farmer to undertake. As a proof of concept, the expert's answers were generated by GPT-4. In subsequent steps of the research, feedback from human experts will improve the model knowledge. The multi-modality of the model will be expanded, including other sensor data of the Treebot robot platform. Future iterations aim to explain the robot's actions based on a comprehensive assessment of the vineyard's condition.

Sustainable weed control

AI algorithms have an enormous potential for the optimisation of herbicide usage and sustainable weed control strategies. In the enrichment project POSING, an explainable vision AI tool for weed detection was improved to allow for the plant protection products (PPP) to be applied more effectively by optimising the means, the time for spraying and the spatial spot spraying application with enhanced explainability to the end-user. A demonstration of this concrete research output can be seen in the video below.

Outputs & Resources

Publications

Cornelissen, C., De Coninck, S., Willekens, A., Leroux, S., & Simoens, P. (2025). In-field mapping of grape yield and quality with illumination-invariant deep learning. IEEE Internet of Things Journal, 12(23), 49207–49220. https://doi.org/10.1109/JIOT.2025.3617805

Datasets

A hyperspectral vineyard dataset was created from 60 Chardonnay grape bunches from 60 trees, scanned both in the field and under laboratory conditions. Field scans were conducted using the Treebot platform and the Specim FX10 scanner, while laboratory measurements included brix and acidity measurements on individual grapes. Additional data is collected from production farms in collaboration with farmers and through partnerships with research and demonstration centres, to reflect the variability of agricultural contexts.

Software & Tools

ARTOF documentation: https://artof-ilvo.github.io/. ARTOF software: https://github.com/artof-ilvo. The data collection software for the new sensor setup will be further extended to function as a generic FAIR research tool.

Demonstrators

Demonstrators and proof-of-concept outputs include ARTOF, the sensor setup, ILVO’s robotic test field and synthetic vineyard environments, vineyard scanning experiments, the reinforcement learning navigation controller, the Light-Invariant Spectral Autoencoder (LISA), the explainable AI proof of concept for vine leaf health, and the POSING sustainable weed-control demonstrator.

Videos & Media

A demonstration video is referenced in the sustainable weed control section.

Team & Contact

Contact

Axel Willekens, ILVO; Jurgen Vangeyte, ILVO

Use Case Lead(s)

Axel Willekens, ILVO; Jurgen Vangeyte, ILVO

Principal Investigators / AI Expert Leads

Ann Nowé, Nikos Deligiannis, Kevin Mets, Hiep Luong, Pieter Simoens

Postdoctoral Researchers

Sam Leroux

PhD Researchers

Ciem Cornelissen; Laurens Diels; Tzoulio Chamiti

Research Groups

Agrifoodtechnology ILVO; VUB AI Lab; (Ann Nowé); VUB ETRO; (Nikos Deligiannis); UAntwerpen IDLab; (Kevin Mets); UGent IPI; (Hiep Luong); UGent IDLab (Pieter Simoens)