Autonomous Systems

Use Case Description

Autonomous Systems (mobile robots, AGVs, Autonomous forklifts, drones, robotic arms, inspection systems etc.) are emerging in production floors to increase efficiency, but also for their flexibility. These systems are often manually programmed to deal with different production situations such as: handling parts, transportation of components from the warehouse to an assembly robot, inspecting for defects, etc.

Challenge 1: Intelligent Perception for dynamic and safe environments

Production floors are highly dynamic environments with mobile assets in random positions, walking operators, other moving vehicles, …. These systems should have situational awareness (localization of components, positioning of autonomous agents and operators, ...) using intelligent (AI) multi-modal (2D/3D Camera, Lidar, Radar, Localization systems, Sound) perception from the agent (ex: agv-mounted camera) and the environment (ex: ceiling-mounted cameras) to deal with unpredictable situations (to avoid catastrophic incidents).

Challenge 2: Precise Multi-agent Systems collaboration with zero-calibration

Multi-agent systems (MAS) often need to collaborate on tasks (transporting large objects with many AGVs, production with multiple agents each with their own skills and configurations, etc.) including low-level control of assets (lifting an object together) and supervisory control using inputs from intelligent perception.

Challenge 3: Communication and Real-time mapping/tracking for decision-support 

A digital twin of an autonomous agent in a simulation environment enables real-time monitoring and simulation of the arm's performance in a virtual environment for predictive maintenance, testing and optimizing tasks without interruption of actual production, improving efficiency, and reducing downtime. Having an actionable digital twin of a complete manufacturing environment requires real-time mapping and tracking of asset locations on the production floor using robust and fast wireless communication.

Goal of the use case

The use case Autonomous Systems in Manufacturing Floor addresses these challenges and allows the development and demonstration of AI techniques for intelligent perception, multi-agent collaboration, and real-time mapping & tracking of dynamic assets in a production floor.

Infrastructure and data

Two Flanders Make Infrastructures are being used to generate relevant data and to develop AI-enabled techniques to:

• Infraflex is a reconfigurable assembly system composed of hexagonal cells that can support an articulated robot, a carousel, and 6 add-on modules for additional equipment and providing a material flow via their automated docking ability in cooperation with mobile robots. A digital twin of the Infraflex setup, including the transport of components and add-ons,  is available in a simulation environment to test scheduling of orders using different setups, and to simulate their assembly.

• The AI forklift and AutoInfra are AGVs fitted with multimodal sensors (Radar, Lidar, 2D and stereo vision) to support the collection of data related to (Challenge 1) intelligent perception.

Results, Highlights & Research Outputs

Dataset 1: Towards An Autonomous Warehouse

Two scenarios are considered:

• Scenario 1: AI Forklift and AutoInfra doing a tour of the warehouse without people walking around. 

• Scenario 2: AI Forklift and AutoInfra doing a tour of the warehouse with people walking around. 

  • Detecting operators is important, especially while they are moving or assembling.

  • Combination with the reflection (zone 6, 8, 10 operator should be present)

  • Some people wear or carry safety-related items, such as a safety helmet, glasses, safety shoes, a fluo jacket, a safety button, a walkie-talkie, wearables, etc. Badge visible.

  • Equipment during moving/assembling

Multimodal AI for AGVs in warehouses

Visualization with Foxglove

Digital Twins

A digital twin is a virtual model of a physical object, system, or process that is continuously updated with real-time data. It serves as a digital replica that allows monitoring, analysis, simulation, and optimization of its physical counterpart.

Digital Twins for Synthetic Data generation

Dataset 2: Bin picking and kitting

• Bin Picking: Robots or humans grabbing specific parts from containers where items are stored randomly. Like finding a particular toy in a messy toy box.

• Kitting: Collecting all parts needed for a specific product and packaging them together before assembly.

Demonstrators

Demo 1: Edge AI-based Collision Avoidance AGV

Problem statement

Ensuring safe coexistence of automated guided vehicles (AGVs) with human workers is critical. Traditional safety measures, such as fixed speed limits, can be overly conservative, leading to reduced operational efficiency and productivity.

Goal of the demonstrator

• Demonstrate that specialized edge AI boards can be used for real-time decision-making while ensuring high uptime, low latency, and low power draw.

• Showcase a smart security braking system that uses obstacle detection (human).

Solution:

• People detection on the edge

  • Real-time Detection and Adjustment: Real-time (~12 FPS) detection of people and dynamic speed adjustment based on distance

  • People detection model: A Yolov10  model (quantized to 16 bits) provides a bounding box of the people in the field of view of the cameras. A lidar provides an estimate of the distance which can be used for the safety braking system.

• Braking Mechanism:

  • Sensor-based: A built-in safety system that relies on proximity sensors to come to a sudden stop when an obstacle is detected. However, if the speed is too high, the AGV will not be able to stop on time.

  • AI-assisted: The AGV keeps track of the distance to each person it has detected. Once an obstacle is detected at below a minimum distance, the AGV either comes to a sudden stop or gradually reduces its speed until it comes to a stop.

Hardware setup: An industrial Mir250 Robot is augmented with the following sensors:

• 5 cameras: 3 Front cameras with a a field of view between 120 and 160 degrees, 2 rear cameras

• 1 Lidar

• Edge processor: ORIN AGX 64 GB (4 TB SSD)

Demo 2: Multi-agent high-mix low-volume assembly

Problem statement

High-mix, low-volume manufacturing requires flexible orchestration of diverse autonomous agents while coping with dynamic production demands and changing environments. 

This setup enables testing, training and evaluation of intelligent perception, distributed decision-making, sim-to-real transfer, and resilient execution for autonomous systems in high-mix, low-volume manufacturing.

Goal of the demonstrator

• Develop and evaluate AI-driven methods for scheduling and coordination of multiple autonomous agents.

• Enable real-time synchronization between physical execution and digital twins for monitoring, validation, and adaptive rescheduling.

• Demonstrate cooperative multi-agent behaviors for complex assembly tasks under varying production orders.

Solution

The demonstrator integrates autonomous agents—including mobile robots, robotic arms, and AGVs—with a digital twin of the production floor. Production orders are translated into coordinated task plans, executed by the agents, and continuously mirrored in the simulation.

Physical assets:

• Autonomous forklift (pallet handling, aisle navigation) with onboard 3D LiDAR + RGB-D for perception.

• MiR-class AGVs for transport between storage, workcells, and shipping.

• UR-series robot arm on a reconfigurable workstation using InfraFlex quick-change tools (e.g., parallel gripper, vacuum, screwdriver, tensioning fixture) for variant assembly.

Digital twin & simulation

• Warehouse digital twin of existing physical AGVs, and a UR arm with the same tooling and payloads.

Automated Assembly of a flashlight: from bin-picking and kitting to assembly

Multiple autonomous systems—such as mobile robots, robotic arms, and automated guided vehicles (AGVs)— need to be orchestrated to collaborate to assemble different product variants. 

Outputs & Resources

Previous Realizations

Control of Cooperating Robots (2019-2023)

Datasets

• Dataset 1: Towards An Autonomous Warehouse.

• Dataset 2: Bin picking and kitting.

Digital Twins / Infrastructure

• Digital twins for synthetic data generation.

• Digital twin of the Flanders Make warehouses in Lommel and Kortrijk.

• Infraflex digital twin / co-creation flexible assembly lab facility.

Demonstrators / Applications

• Edge AI-based Collision Avoidance AGV.

• Multi-agent high-mix low-volume assembly.

• Automated assembly of a flashlight.

Outreach

• Flanders AI Forum (11/6/2024): Deep-dive

• Flanders AI Research Day (14/10/2024): Demo & Deep-dive

• On-site demonstrations @FM Kortrijk

Team & Contact

Contact

Bart Van Doninck (Use case owner & Operator 4.0/5.0 expert) - Flanders Make IM; Industry Use Cases follow up & Valorization: Wouter De Nijs / Abdel Bey-Temsamani - Flanders Make IM

Principal Investigators / AI Expert Leads

Maarten Van Der Burgt, imec - HSA; Tias Guns, KU Leuven - DTAI; Marian Verhelst, KU Leuven - MICAS; Jeroen Hoebeke, UGent - IDLab; Bart Dhoedt, UGent - IDLab; Pieter Simoens, UGent - IDLab; Sam Leroux, UGent - IDLab; Wilfried Philips, UGent - IPI; Bart Goossens, UGent - IPI; Ann Nowé, VUB AI Lab; Bram Van der Borgt, VUB AI Lab

Postdoctoral Researchers

David Van Hamme, UGent - IPI

PhD Researchers

Marco Foschini, KU Leuven - DTAI; Sander De Coninck, UGent - IDLab

Research Groups

• Development of chiplets for AI -research and sensor fusion.  (IMEC - HSA - PI: Maarten Van Der Burgt)

•  Combinatorial optimization and planning of MAS under uncertainty (KULeuven DTAI - PI: Tias Guns - PhD: Marco Foschini)

• Low latency edge AI for MAS (KU Leuven MICAS - PI: Marian Verhelst)

• Integration of chiplets, compute & communication into next generation Flanders AI SDK (WiFi TSN) (UGent IDLAB - PI: Jeroen Hoebeke)

•  Active perception & real-time interference: hierarchical world model for navigation (Ugent IDLAB - PI: Bart Dhoedt - PhD)

• Privacy preserving in MAS context (Ugent IDLAB - PI: Pieter Simoens, Sam Leroux - PhD: Sander De Coninck)

• Mapping sensor fusion algorithms onto edge devices; Cooperative scene analysis: accurate robust perception of AGV environment (UGent IPI - PI: Wilfried Philips, Bart Goossens and Postdoctoral researcher: David Van Hamme)

• Multi-agent perception and control (VUB AI Lab - PI:Ann Nowé, Bram Van der Borgt - PhD: TBD)