Embodied Agent (Industrial AGV) Performing Tasks Assisted by Operator

Use Case Description 

The use case demonstrates how Reinforcement Learning, AI vision and NLP (natural language processing) models can work together to improve the autonomy of autonomous systems in a dynamic environment as well as make interactions with them more intuitive to adjust to new tasks.

Results & Achievements

In 2021, we showed this approach applied to an autonomous tractor where an operator instructs the autonomous system by speech to move towards different objects in a parking lot. The results proved to be accurate and robust even in an outdoor condition. The objects mentioned by speech were correctly detected and matched 80% of the cases.

In 2022, the approach was extended to an autonomous forklift that operates in a warehouse with a more dynamic scene. RL models trained in simulations proved promising results for AGV navigation in unknown and dynamic environments. CNN models based on vision system were used to accurately make the warehouse inventory by reliably counting boxes with 97% precision and 90% recall by directing the navigation to get better viewpoints of uncertain detections.

In 2023, we continued improving the RL and AI Vision tracking by introducing more variability and unpredicted scenarios in the warehouse use case. We proved an integrated demo where RL based navigation can be done in a realistic time (few minutes to explore coupled of corridors). We also improve the tracking of the Vision AI NN and further train with other assets such as containers. Finally we associated the speech interaction with goal driven tasks.

Papers

 A multi-modal AI approach for intuitively instructable autonomous systems
 Gebellí Guinjoan, Ferran; Rademakers, Erwin; Chavali, Anil Kumar; Bey Temsamani, Abdellatif; Radevski, Gorjan; Tuytelaars, Tinne; Hutsebaut-Buysse, Matthias; Mets, Kevin; De Schepper, Tom; Latré, Steven; Mannens, Erik; Van hamme, Hugo
 International Journal on Advances in Systems and Measurements; 2023; pp. 

Directed Real-World Learned Exploration, M Hutsebaut-Buysse, FG Guinjoan, E Rademakers - 2023 IEEE/RSJ International Conference, 2023

Outreach

Different internal & external events

Uptake

In prospect

Basic Use Case Info

Period

2019-2023 

Contact

Abdel Bey

Research Groups

Flanders Make, KUL (Tinne Tuytelaars), KUL (Hugo Van Hamme), UAntwerp IDLab