Industry

When the environment in which a robot operates is dynamic, it must be able to respond quickly to changes in the environment, which calls for a so-called closed-loop control algorithm, in which the robot does not blindly execute a plan, but constantly analyzes and processes novel sensor information, to adjust the plan online. This, of course, places strict demands on the processing time of the sensor information, as this determines how quickly the robot can respond to changes. The robot can also decide to search for more informative data, for example by taking a closer look when necessary (i.e. 'active sensing').

This type of robot control is also often used in the case of mobile robots such as autonomous vehicles or drones, which de facto have to deal with complex environments where fast reaction times are crucial.

The research within the Flanders AI Resarch Program

To process sensor data, deep learning techniques are becoming more and more popular, with increasingly larger deep learning models being used. The research program examines how these techniques can be executed as fast and (energy) efficient as possible, on the one hand by smartly adapting the model, and on the other by developing tailor-made hardware platforms. By co-designing both the machine learning models and the hardware platform, we can build high-performant systems for real-time control of robots. These AI techniques are used for subproblems (e.g. object classification and segmentation), as well as for end-to-end solutions using reinforcement learning. We also investigate how a robot can continuously learn from its experiences locally, by using either implicit (e.g. predicting future sensor data) or explicit (e.g. object grasped or not) reward signals.

Results and demonstrators

Demonstrator 1. Real-time object grasping by a robot. We equip the robot arm with an in-hand camera, such that the robot can determine the viewpoint. We demonstrate models that can be used for real-time object grasping, as well as models that provide the robot with an understanding of its workspace, in which the robot searches for informative data, for example by looking closer when necessary. Crucially, our models require a few orders of magnitude less parameters and data than what is accustomed in the state-of-the-art. For example, our robot can learn to grasp an object in minutes after just a single demonstration.

• E. De Coninck, T. Verbelen, P. Van Molle, P. Simoens, and B. Dhoedt, “Learning robots to grasp by demonstration,” ROBOTICS AND AUTONOMOUS SYSTEMS, vol. 127, 2020
• T. Van de Maele, T. Verbelen, O. Catal, C. De Boom, and B. Dhoedt, “Active vision for robot manipulators using the free energy principle,” FRONTIERS IN NEUROROBOTICS, vol. 15, 2021.

Demonstrator 2. A mobile, self-navigating robot finds its position and explores the environment. In the case of navigating robots, we build models that provide a low-dimensional representation of the sensor data, which can be used to efficiently find the robot's location in the environment. This is crucial in so-called simultaneous localization and mapping or SLAM algorithms. Using learned representations, our model can combine different data modalities in a compact map, while traditional SLAM methods use less scalable data structures such as point clouds or occupancy grids.

o   O. Catal, W. Jansen, T. Verbelen, B. Dhoedt, J. Steckel, “LatentSLAM: unsupervised multi-sensor representation learning for localization and mapping”, ICRA 2021.

Connect

• Contact person: imec/UGent: Tim Verbelen
• Other groups involved:  UGent (IDLab), KU Leuven (MICAS), imec

See also: https://vimeo.com/603989678

Connect

• Contactpersons:  Flanders Make: Maarten Witters, Robbert Hofman, Ali Bin Junaid, Abdel Bey-Temsamani
• Other groups involved: VUB (AI Lab, R&MM)

Connect

• Contact persons:  Flanders Make: Maarten Witters, Robbert Hofman, Ali Bin Junaid, Abdel Bey-Temsamani
•  Other groups involved: KU Leuven (PSI), UGent (WAVES), UAntwerpen (CLiPS, IDLab), VUB (AI Lab, DiMa, MOBI)

AGVs are frequently used in the manufacturing industry, logistics and agriculture and can already execute several tasks in an autonomous way.  The AGVs are also expected to execute a larger variety of tasks with a larger variety of objects.

The research within the Flanders AI Resarch Program

We develop new AI methods for intuitively controllable industrial autonomous vehicles. The vehicle must be able to perform tasks based on voice commands and must also avoid obstacles and be able to navigate in complex environments. A simple user interface is needed for easy vehicle control with voice commands. The interface being developed within this research program is multimodal (a combination of image and speech recognition). The user can give commands to the vehicle, ranging from simple commands to commands from the user that refer to objects detected using the cameras on the vehicle. This makes it possible to perform complex actions with the vehicle with a limited set of voice commands. 

The challenges are 1/ data-efficient learning because collecting data is very time-consuming, 2/ understanding spoken commands in an industrial environment. 3/ analyzing complex visual scenes, 4/ the multimodal aspect (combination of speech and vision), where the user can refer in spoken commands to objects that the cameras on the vehicle image and 5/ navigating in complex environments where the AGV has limited freedom to navigate. 

AI  techniques used:

1/ Pre-training and transfer learning to learn in a more efficient way. This means that we first work on similar data that is already available or easier to obtain before the AI is fine-tuned on the final data. 

2/ Spoken commands (in Dutch) are converted directly from speech to commands without the conversion from speech to text (“Spoken language understanding”)

3/ and 4/ Analysis of visual scenes, recognition of objects and relations between objects, recognition of objects from speech commands (representation learning, action recognition,...) 

5/ A vehicle is taught in a data-efficient way to navigate in complex environments and to avoid obstacles and with the help of “reinforcement learning”. Using a simulator allows learning without expensive experiments in the physical setup. 

Results and demonstrators

The AI methods are illustrated in two set-ups:

Set-up1 A tractor with a forklift implement to move objects equipped with multiple sensors including cameras and LIDARs. The tractor can be controlled in Dutch (where direction and speed can be indicated). Using the cameras, the tractor detects the objects that need to be moved. The user can also refer to these objects in the commands. The following tasks can already be performed: moving objects and interpreting and executing voice commands.

Set-up 2 An AGV in an office environment. This setup illustrates (in simulation) the use of reinforcement learning for navigation in a complex environment. 

 Connect

• Contact person Set-up 1: Flanders Make, Quiten Lauwers
• Contact person Set-up 2: UAntwerpen (IDLab), Tom De Schepper
• Other groups involved: KU Leuven (PSI)

Gradual automation of existing industrial sites is often challenging because of the stringent safety requirements in workspaces shared by human operators and mobile robots. Typically, sensors on the robot, e.g., single beam lidars, check the safety margin around the robot for intrusions, triggering safety stops when a person or other robot comes within this margin. This short-range safety field however limits the maximum safe velocity of the robot, and frequent safety stops severely reduce the efficiency. One solution is to adapt the site layout to reduce the number of person-robot interactions, but this requires a large investment and cannot easily be implemented in small phases to limit the interruption of operations.

The research within the Flanders AI Research Program

Within the Flanders AI Research Program, researchers are developing AI based methods to reliably keep track of the position and movement of human and robot operators on industrial sites at all times. To this end, data from different sensors is combined, typically camera with radar or lidar. Sensors fixed to the static infrastructure as well as robot-mounted sensors are used in this data fusion. The research focuses on two aspects: reliability in all conditions, and efficiency of the perception network.

To ensure people are well detected even in difficult light circumstances, novel neural networks are developed to process radar data, far exceeding the performance of traditional radar detection algorithms. The radar information is combined with camera analysis in a robust cooperative fusion framework, in which sensors share partly processed information at an early stage to selectively boost sensitivity. This approach increases the distinctive power of the sensors without introducing excess false detections.

To limit the computational cost of the neural networks used in analysis, distributed and selective processing paradigms are explored, taking into account blind spots of individual sensors as well as scene topology, exploiting spatial overlap and time redundancy to avoid repeatedly processing the entire field of vision of the sensors.

Results and demonstrators

Demonstrator 1. Car traffic in a crowded urban scene. By selectively activating neural networks based on scene topology, a six-fold reduction in processing time is realized compared to a state-of-the-art method from literature, without reduction in analysis quality, evaluated on a difficult public dataset for automotive applications.

Demonstrator 2 Radar-video person-detection.  A sensor node comprised of a 77GHz automotive radar and a basic RGB camera is coupled to a neural network to detect the position of people, even in low-light, blinding light or deep shadow conditions. The network can be generic (trained on a large dataset) or trained specifically for the space in which the sensor node is placed. This location-specific training can be performed fully unsupervised (without any manual labeling), by temporarily adding a lidar sensor to the node.

Connect

•  Contact person: imec/UGent: David Van Hamme
•  Other groups involved: UGent (IPI), UGent(IDLab), imec (AdvanceRF)

More information

-     Publications from the Flanders AI Research Program

o   Dimitrievski M, Van Hamme D, Veelaert P, Philips W. Cooperative Multi-Sensor Tracking of Vulnerable Road Users in the Presence of Missing Detections. Sensors. 2020; 20(17):4817. 

o   Dimitrievski, M., Shopovska, I., Van Hamme, D., Veelaert, P., & Philips, W. (2020). Weakly supervised deep learning method for vulnerable road user detection in FMCW radar. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Proceedings. Rhodes, Greece (Virtual Conference): IEEE. 

The additive manufacturing (AM) industry needs to be able to deliver products with a consistent high-quality for it to gain widespread usage in the industry. To deliver high quality, AM  needs to have means for a quality control on the printed part, a monitoring system that can predict the quality already during the printing and eventually a closed-loop control of the 3D printing process to improve quality and reduce resource usage.   

The research within the Flanders AI Research Program

The main goals are the research of both offline and online methods to detect defects and determine their probable cause. 

1. an AI offline smethod to automatically detect, localize and classify defects from computed tomography (CT) data. CT is a nondestructive method that allows an expert to assess the local quality of the part. 

2.  hardware and software for an on-line closed-loop control system to increase quality and productivity.

Results and demonstrators
The experimental setup is an open and modular platform for research into the 3D printing process. It is built on top of an industrial printer with building volume of 275 x 275 x 400 mm able to print in metal (316L stainless steel) using selective laser melting. It is equipped with an open control platform and has extensive monitoring capabilities. Sensors are used including the ones many 3D printers are already instrumented with such as high speed photodiodes (up to 100KHz) and cameras (up to 20KHz). 

Defect Detection in printed parts (off-line)

A 3D-CONV-GAN generative network has been trained to reconstruct a defect-free 3D structure of a printed object from its CT data.  In this way it is possible to automatically detect and localize defects by comparing the reconstructed structure with its original CT data. This method highlights the defects faster and with more precision than the classical methods.

Defect Detection during printing (on-line)

In order to enable the real-time detection, a technique for fast anomaly detection on time series using extreme learning machines have been implemented. It uses both a 3D-CNN and a LSTM network capable to analyze data coming from a high-speed sensors (e.g. a 20Khz video feed) monitoring the melting process in order to detect quality issues in real-time. It supports active learning to update the AI model on-the-fly. Hence there is no need for a separate calibration print and it handles incremental environmental changes. 

Connect

• Contact person: Flanders Make: Dries Verhees
• Other groups involved:   University of Antwerp (Vision Lab), University of Ghent (IPI, IDLab), KULeuven (STADIUS)

Connect

•     Contact persons: Flanders Make, Jeroen Jordens, Maarten Witters, Abdel Bey-Temsamani
•     Other groups involved: Universiteit Gent, IDLab Gent, KU Leuven (DTAI) en Uhasselt (Research Group Logistics), VUB (DIMA)

More Information

• Joining & Materials lab

In industry there are many fleets of similar machines, or machines operating in similar conditions. When commissioning or tuning controllers for such machines, time can be saved by reusing information from other machines in the fleet that are already operational. Also for monitoring of machines that are part of a  fleet information from other fleet members can be exploited to enhance performance. 

The research within the Flanders AI Resarch Program

A first application is the commissioning of a controller for a new machine relying on controllers obtained for similar machines. Especially for non-identical machines or varying conditions, for which tuning per machine or condition would be needed, a big gain can be achieved this way, when compared to model-based commissioning methods, or to the industrial practice of manual tuning.

A next application will look at making controllers learn during normal machine operation. The algorithms will adjust the policy, or the control actions chosen by the controller, taking into account the constantly changing state of the machine and the controller itself. This will be done relying on information observed during normal machine operation when the policy is active. Once these methods are successfully applied, they will enable adjusting to constant changes in the conditions or the machine behavior, at runtime, during normal machine operation.  Relying on multiple machines learning together will furthermore lead to an improved performance and a reduced learning period. 

Results and demonstrators

For this application a test setup is used representative for motions in looms, compressors, steering columns, … The setup has three separate crank rocker mechanisms, that each perform a series of similar yet non-identical stop start motions. 

For the improved commissioning a first approach relying on supervised learning has been applied. For the training a labeled dataset was used with examples of well-tuned controllers for various tasks and systems. With a random forest or a neural network this data is used to train a model that predicts for new tasks and/or systems good controller values, thereby reducing the needed time to find a good controller when compared to model-based approaches. For the neural-network-based method additionally simulation data can be added to expand the training dataset.

Connect

• Contact: Flanders Make, Bruno Depraetere, Abdel Bey-Temsamani
• Other groups  involved:  VUB 

More Information

• See also "Improved energ production of wind turbine parks" in the application domain Energy. 

Anomaly detection and condition-based maintenance of industrial machines are known applications of AI technologies. Despite the benefits, many industrial companies still lack confidence in the capabilities, performance, and reliability of anomaly detection and remaining useful life prediction methods.

There is a need for further research into:

1.     Real-time anomaly detection for complex industrial machines with high variability in failure, operational condition and other external disturbances where data is scarce, unlabeled or of poor quality.

2.     Energy-efficient AI systems that can simultaneously decentralized (edge) process real-time high-frequency signals and make decisions using energy-efficient AI-optimized processors.

The research within the Flanders AI Research Program

The research groups investigate:

1.     Data-efficient AI solutions for anomaly detection in industrial machines under high variabilities: how can knowledge-based models and data-driven methodologies be combined in a hybrid methodology? How can AI-based data generation and augmentation improve anomaly detection? (Research Challenge AI-Driven Data Science)

2.     Energy-efficient AI at the edge solutions for anomaly detection in industrial machines (Reserch Challenge AI in the Edge)

3.     Application of AI for health aware control (Research Challenge Multi-agent Collaborative AI )

Research is being conducted into the following AI techniques:

1.     For data-efficient AI solutions: Convolutional Neural Networks (CNN), Wavelet Scattering Networks (WSN), Transfer learning based on Domain Adversarial Neural Networks (DANN), Deep Support Vector Data Description (SVDD), Variational Autoencoders (VAE).

2.     For energy-efficient AI edge solutions: Spiking Neural Networks (SNN) with custom-developed edge hardware and supervised, unsupervised, hybrid edge-cloud Autoencoders (AE).

These methods are compared with a “state of the art” baseline consisting of methods using classical statistical features (e.g. rms, kurtosis, peak), methods based on physical models (e.g. fault frequencies based on envelope analysis), classic machine / deep learning methods (e.g. 1D CNN).

Results and demonstrators

Bearings are crucial mechanical components in many industrial machines but are also subject to wear which can lead to unplanned downtime. This use case employs high frequency (noisy) experimental machine data obtained from bearing lifetime tests performed at the Flanders Make Smart Maintenance living lab. Different AI methods are investigated and compared in terms of detection time, accuracy, data efficiency and computation time.

Connect

• Contact person: Flanders Make: Ted Ooijevaar
•  Other groups involved: KU Leuven (DTAI, LMSD), UGent (EElab, IDlab), imec, VUB

More Information

BOURGANA, Taoufik, BRIJDER, Robert, OOIJEVAAR, Ted, et al. Wavelet Scattering Network Based Bearing Fault Detection. In : PHM Society European Conference. 2021. p. 8-8.

Check out the use case “Prognostic Health Management of windturbines" in the application domain Energy.