Medical Imaging

Use Case Description

Medical decisions for diagnosis, therapy planning, and follow-up rely heavily on 2D and 3D medical imaging data of the patient, such as RX, CT, MRI, PET and US. Because of continuous technological advances in medical imaging equipment, the amount of imaging data to be analyzed in routine clinical practice is ever increasing, putting high pressure on the radiological workflow. Radiological reading of medical images is a complex process that requires extensive prior knowledge and expertise to detect and recognize subtle patterns in the images and discriminate abnormal findings from normal variability. More and more clinical applications also benefit from quantification of specific anatomical or functional parameters of the structures of interest, such as position, size, volume, shape, motion, deformation, density, perfusion, diffusion…. Such quantification often involves precise 3D localization and delineation of anatomical object boundaries in the images, of both normal anatomy as well as pathology, which is tedious and time-consuming and hence not feasible in routine clinical practice if to be performed manually by the expert. There is thus a strong need for reliable, automated computer-aided assessment and quantification of medical imaging data for clinical decision support in Radiology, Oncology, Cardiology, Neurology, Surgery and many other medical disciplines.

The research within the Flanders AI Research Program

In this project, we wish to develop machine learning strategies, implement them into AI-tools and validate their clinical application potential in a number of different, but methodologically related clinical decision support applications involving medical imaging data from different application domains, focusing in particular on applications in Radiology and Radiation Oncology. These applications have variable amounts of available training data, and priors over image structures, providing non-trivial and informative test cases for AI methods. Tasks can be broadly categorized into classification, regression, and image/volume segmentation problems.

Direct application of deep learning strategies that have been successful in related computer vision applications to medical imaging data is challenging because of the nature of the data itself (3D, multi-modality), the limited training data that is typically available (hundreds instead of millions of images), the need for expert input for clinically relevant assessment of performance (some errors are more significant than others), and the fact that ground truth may be unreliable due to observer variability induced by ambiguity in the images (disagreement between experts).   

Results

Automated delineation for radiotherapy treatment planning

Radiotherapy involves the delivery of a high radiation dose to the tumor while ensuring a low dose to the healthy surrounding organs that may be at risk for adverse effects of the radiation. Hence, a careful and detailed patient-specific 3D planning is required using a patient's CT scan as input, in which the tumor and the organs at risk are precisely delineated in order to define the high and low dose regions and construct an optimal treatment plan. These delineations are typically performed manually in clinical practice, which is a tedious and time consuming process due to the 3D nature of the images and due to the fact that organ contours may not be easily discriminatable on all slices due to image ambiguity. Moreover, manual delineation in medical images is subject to significant intra- and inter-observer variability, which may induce significant observer-related variability in the delivered dose distribution itself and hence in the clinical outcome of the therapy. To make the planning process more time efficient and less observer dependent, automated delineation tools are needed. In this context and in close collaboration with the clinical experts of UZ Leuven, an AI tool was developed and validated for automated CT delineation of 16 organs at risk that are of interest for head & neck radiotherapy treatment planning. Automated prediction of all contours at once is performed by a single neural network that processes the CT image at multiple resolutions using a patch-based approach, which was trained using a limited set of ~75 cases. While manual delination would take between 45' to 1h, automated delineation using this AI tool and subsequent expert verification, with manual corrections when needed, takes usually less than 10'. Moreover, observer variability is drastically reduced when the automated delineation is performed prospectively and subsequently presented to different observers for correction. The clinical benefits of AI-based automated delineation in routine clinical practice have been demonstrated within UZ Leuven. Similar approaches have been developed for organs at risk delineation in other anatomical sites (e.g. lung, abdomen) and also for automated delineation of the clinical target volume.

[1] Bollen, H., Willems, S., Wegge, M., Maes, F., Nuyts (2023). Benefits of automated gross tumor volume segmentation in head and neck cancer using multi-modality information. RADIOTHERAPY AND ONCOLOGY, 182, Art.No. ARTN 109574. doi: 10.1016/j.radonc.2023.109574 

[2] Buelens, P., Willems, S., Vandewinckele, L., Crijns, W., Maes, F., Weltens, C. with Buelens, P. (joint first author), Willems, S. (joint first author) (2022). Clinical evaluation of a deep learning model for segmentation of target volumes in breast cancer radiotherapy. Radiotherapy And Oncology, 171, 84-90. doi: 10.1016/j.radonc.2022.04.015

[3] van der Veen, J., Gulyban, A., Willems, S., Maes, F., Nuyts, S. (2021). Interobserver variability in organ at risk delineation in head and neck cancer. RADIATION ONCOLOGY, 16, Art.No. 120. doi: 10.1186/s13014-020-01677-2

[4] van der Veen, J., Willems, S., Bollen, H., Maes, F., Nuyts, S. (2020). Deep learning for elective neck delineation: More consistent and time efficient. RADIOTHERAPY AND ONCOLOGY, 153, 180-188. doi: 10.1016/j.radonc.2020.10.007   

[5] van der Veen, J., Willems, S., Deschuymer, S., Robben, D., Crijns, W., Maes, F., Nuyts, S. (2019). Benefits of deep learning for delineation of organs at risk in head and neck cancer. RADIOTHERAPY AND ONCOLOGY, 138, 68-74. doi: 10.1016/j.radonc.2019.05.010

[6] Willems, S., Crijns, W., La Greca Saint-Esteven, A., Van Der Veen, J., Robben, D., Depuydt, T., Nuyts, S., Haustermans, K., Maes, F. (2018). Clinical implementation of DeepVoxNet for auto-delineation of organs at risk in head and neck cancer patients in radiotherapy. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics): vol. 11041 LNCS, (223-232). Presented at the 7th international workshop on clinical image-based procedures: Translational research in medical imaging - CLIP 2018, held in conjunction with MICCAI 2018, Granada, Spain, 16 Sep 2018-16 Sep 2018. ISBN: 9783030012007. doi: 10.1007/978-3-030-01201-4_24 

[7] Willems, S., Maes, F. (sup.), Sterpin, E. (cosup.) (2023). Learning-based Computational Strategies for Treatment Adaptation in Image-guided Radiotherapy, PhD thesis, KU Leuven   

[8] Vandewinckele, L., Willems, S., Robben, D., Van Der Veen, J., Crijns, W., Nuyts, S., Maes, F. with Maes, F. (corresp. author) (2019). Segmentation of head-and-neck organs-at-risk in longitudinal CT scans combining deformable registrations and convolutional neural networks. Computer Methods in Biomechanics and Biomedical Engineering: Imaging and Visualization, 1-1. doi: 10.1080/21681163.2019.1673824 

[9] https://www.uzleuven.be/nl/nieuws/beter-bestralingsplan-voor-hoofd-en-halskanker-dankzij-artificiele-intelligentie

AI-based automated treatment planning in radiotherapy 

The aim of radiotherapy treatment planning is to optimize the parameters of the radiation device such that the prescribed dose is precisly delivered to the tumor with minimal dose to the surrounding healthy tissues. The device parameters differ for different treatment setups (e.g. IMRT, VMAT) and contain the directions from where to radiate (a number of discrete angles in case of IMRT, a continuous arc in case of VMAT), the intensity of the radiation (monitor units) and the shape of the radiation beam (varying aperture realized with a multi-leaf collimator). Optimisation of these parameters to ensure that all geometric and dosimetric constraints are met, is a complex process that is manually steered and which may require multiple iterations before a clinically acceptable plan is achieved. Machine learning techniques can help to automate this process by predicting a suitable plan based on similar previous cases, thereby decreasing planning time and reducing observer variability. In this project, the feasibility of using deep learning with convolutional neural networks (CNN) was investigated to directly predict the device parameters of a lung IMRT plan based on the CT scan of the patient and the desired 3D dose distribution, which itself was also estimated from the CT scan using a separate CNN. In an adaptive treatment setting, an initial plan is available that needs to be adapted to a change in anatomy of the patient. To this end, a CNN was trained that exploits longitudinal information to predict the device parameters of an adaptive lung IMRT plan directly from the patient anatomy, without the need for an intermediate 3D dose prediction. Another CNN was trained to predict the device parameters of a breast VMAT plan directly from patient anatomy. These predictions are subsequently given as initialisation to the optimiser for fine-tuning. We showed that direct device parameter prediction can significantly decrease treatment planning optimisation time, as equal dosimetric results were obtained in halve of the time with this workflow compared to an optimisation without this initialisation.

[1] Vandewinckele, L., Reynders, T., Weltens, C., Maes, F., Crijns, W. with Vandewinckele, L. (corresp. author) (2023). Deep learning based MLC aperture and monitor unit prediction as a warm start for breast VMAT optimisation. PHYSICS IN MEDICINE AND BIOLOGY, 68 (22), Art.No. ARTN 225013. doi: 10.1088/1361-6560/ad07f6

[2] Vandewinckele, L., Willems, S., Lambrecht, M., Berkovic, P., Maes, F., Crijns, W. with Vandewinckele, L. (corresp. author) (2022). Treatment plan prediction for lung IMRT using deep learning based fluence map generation. PHYSICA MEDICA-EUROPEAN JOURNAL OF MEDICAL PHYSICS, 99, 44-54. doi: 10.1016/j.ejmp.2022.05.008

[3] Vandewinckele, L., Crijns, W. (sup.), Maes, F. (cosup.), Lambrecht, M. (cosup.) (2024). Direct prediction of radiotherapy treatment plans using deep learning, PhD thesis, KU Leuven
 

Automated MRI quantification of mucle fat fraction in muscular dystrophy patients

Muscular dystrophies (MD) are a class of rare genetic diseases affecting specific muscle groups, resulting in progressive muscle weakness due to the gradual replacement of muscle tissue by fat. The degree of muscle fattening can be assessed with fat-sensitive magnetic resonance imaging (MRI) and objectively evaluated by quantifying the fat fraction percentage (FF%) per muscle. Conventionally, 2D manual assessment is performed on selected slices for the muscle tissue as a whole or specific muscle groups, as manual delineation of individual muscles in the MRI images is too time consuming and especially challenging in MD patients as the normal MRI contrast is affected by the pathology. To enable more precise 3D volumetric per muscle fat quantification, a deep learning based approach was developed for fully automated 3D segmentation and FF% quantification of the 18 individual muscles in the proximal leg from knee to hip based on an in-house created training set of expert delineated Dixon MRI images of both healthy subjects as well as patients with MD. Clinically relevant quantification accuracy was demonstrated for cases with low as well as medium to high fat infiltration for different types of MD. The tool was applied in several clinical studies to characterize disease progression using MRI in combination with functional measurements and patient-reported measures. 

[1] De Wel, B., Iterbeke, L., Huysmans, L., Peeters, R., Goosens, V., Dubuisson, N., van den Bergh, P., Van Parijs, V., Remiche, G., De Waele, L., Maes, F., Dupont, P., Claeys, K.G. (2024). Lessons for future clinical trials in adults with Becker muscular dystrophy: Disease progression detected by muscle magnetic resonance imaging, clinical and patient-reported outcome measures. EUROPEAN JOURNAL OF NEUROLOGY, 31 (7). doi: 10.1111/ene.16282 

[2] De Wel, B., Huysmans, L., Peeters, R., Ghysels, S., Byloos, K., Putzeys, G., Maes, F., Dupont, P., Claeys, K.G. (2024). Test-retest reliability and follow-up of muscle magnetic resonance elastography in adults with and without muscle diseases. JOURNAL OF CACHEXIA SARCOPENIA AND MUSCLE. doi: 10.1002/jcsm.13528

[3] De Wel, B., Huysmans, L., Depuydt, C.E., Goosens, V., Peeters, R., Santos, F.P., Thal, D.R., Dupont, P., Maes, F., Claeys, K.G. (2023). Histopathological correlations and fat replacement imaging patterns in recessive limb-girdle muscular dystrophy type 12. JOURNAL OF CACHEXIA SARCOPENIA AND MUSCLE, 14 (3), 1468-1481. doi: 10.1002/jcsm.13234 

[4] Huysmans, L., De Wel, B., Claeys, K.G., Maes, F. (2023). Automated MRI quantification of volumetric per-muscle fat fractions in the proximal leg of patients with muscular dystrophies. FRONTIERS IN NEUROLOGY, 14, Art.No. ARTN 1200727. doi: 10.3389/fneur.2023.1200727

[5] De Wel, B., Huysmans, L., Peeters, R., Goosens, V., Ghysels, S., Byloos, K., Putzeys, G., D'Hondt, A., De Bleecker, J.L., Dupont, P., Maes, F., Claeys, K.G. (2022). Prospective Natural History Study in 24 Adult Patients With LGMDR12 Over 2 Years of Follow-up Quantitative MRI and Clinical Outcome Measures. NEUROLOGY, 99 (6), E638-E649. doi: 10.1212/WNL.0000000000200708 

[6] De Wel, B., Claeys, K. (sup.), Maes, F. (cosup.), Dupont, P. (cosup.), Poesen, K. (cosup.) (2023). Towards trial readiness in hereditary neuromuscular diseases, PhD thesis, KU Leuven
 

iCovid

In 2020 this research has been applied to lung images during the Covid-19 pandemic. CT scans of the lungs deliver interesting information in understanding the disease and in the follow-up of the patients. Deep learning techniques are being used to accurately detect lung lesions (abnormal lung tissue). Researchers from the KU Leuven (PSI) and the VUB (ETRO) have developed and compared methods for fast and automatic analyzes of CT lung images. The results have been used in the iCovid project, a pro-bono project initiated by several hospitals and research organizations. The research results have been incorporated by icometrix NV in their cloud based image analyzes product, called ‘icolung’. Over 800 hospitals in Europe use this service on a daily basis. In a further stage, the iCovid project received funding from the EU H2020 program.

[1] Berenguer, A.D., Sahli, H., Joukovsky, B., Kvasnytsia, M., Dirks, I., Alioscha-Pérez, M., Deligiannis, N., Gonidakis, P., Sánchez, S.A., Brahimetaj, R., Papavasileiou, E., Chan, J.C-W., Li, F., Song, S., Yang, Y., Tilborghs, S., Willems, S., Eelbode, T., Bertels, J., Vandermeulen, D., Maes, F., Suetens, P., Fidon, L., Vercauteren, T., Robben, D., Brys, A., Smeets, D., Ilsen, B., Buls, N., Watté, N., Mey, J.D., Snoeckx, A., Parizel, P.M., Guiot, J., Deprez, L., Meunier, P., Gryspeerdt, S., Smet, K.D., Jansen, B., Vandemeulebroucke, J. (2020). Explainable-by-design Semi-Supervised Representation Learning for COVID-19 Diagnosis from CT Imaging.. arXiv. doi: 10.48550/arXiv.2011.11719  

[2] Tilborghs, S., Dirks, I., Fidon, L., Willems, S., Eelbode, T., Bertels, J., Ilsen, B., Brys, A., Dubbeldam, A., Buls, N., Gonidakis, P., Sánchez, S.A., Snoeckx, A., Parizel, P.M., Mey, J.D., Vandermeulen, D., Vercauteren, T., Robben, D., Smeets, D., Maes, F., Vandemeulebroucke, J., Suetens, P. (2020). Comparative study of deep learning methods for the automatic segmentation of lung, lesion and lesion type in CT scans of COVID-19 patients.. arXiv. doi: 10.48550/arXiv.2007.15546

[3] iCovid project: iCovid Lung Imaging AI

Fundamental issues in medical image analysis

In this use case, challenging image analysis problems are being tackled in the context of various medical imaging applications using AI-based computational strategies. This application-driven research helps to expose the fundamental issues of medical image analysis and the limitations of the current state of the art, providing inspiration for methodological research on these topics. Some fundamental issues that were explored in the context of this use case include: 

• incorporation of a prior global shape model in semantic segmentation to constrain local dense predictions to generate conforming shapes

• dealing with missing labels and data during training and prediction

• learning from user feedback in the form of corrections to an initial prediction

• characterization of the behavior of different loss functions for semantic segmentation in medical imaging applications

[1] Tilborghs, S., Bogaert, J., Maes, F. (2022). Shape constrained CNN for segmentation guided prediction of myocardial shape and pose parameters in cardiac MRI. Medical Image Analysis, 81, Art.No. 102533. doi: 10.1016/j.media.2022.102533

[2] Rahimpour, M., Bertels, J., Radwan, A., Vandermeulen, H., Sunaert, S., Vandermeulen, D., Maes, F., Goffin, K., Koole, M. (2021). Cross-modal distillation to improve MRI-based brain tumor segmentation with missing MRI sequences. Ieee Transactions On Biomedical Engineering, 1-12. doi: 10.1109/TBME.2021.3137561 

[3] Tilborghs, S., Bertels, J., Robben, D., Vandermeulen, D., Maes, F. (2022). The Dice loss in the context of missing or empty labels: introducing phi and epsilon. In: Lecture Notes in Computer Science. Presented at the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022), Singapore, 18 Sep 2022-22 Sep 2022. doi: 10.1007/978-3-031-16443-9_51   

[4] Willems, S., Bollen, H., Van Der Veen, J., Sterpin, E., Crijns, W., Nuyts, S., Maes, F. (2021). Learning from mistakes: An error-driven mechanism to improve segmentation performance based on expert feedback. In: CLIP/DCL/LL-COVID/PPML 2021, Lecture Notes in Computer Science: vol. 12969, (68-77). Presented at the 10th international workshop on clinical image-based procedures: Towards holistic patient models for personalised healthcare - CLIP 2021, held in conjunction with MICCAI 2021, Strasbourg, France, 27 Sep 2021-01 Oct 2021. ISBN: 978-3-030-90874-4. doi: 10.1007/978-3-030-90874-4_7 

[5] Eelbode, T., Bertels, J., Berman, M., Vandermeulen, D., Maes, F., Bisschops, R., Blaschko, M. with Eelbode, T. (joint first author), Bertels, J. (joint first author) (2020). Optimization for medical image segmentation: Theory and practice when evaluating with Dice Score or Jaccard Index. IEEE Transactions On Medical Imaging, 39 (11), 3679-3690. doi: 10.1109/TMI.2020.3002417 

Basic Use Case Information

Period

2019-2023

Contact

Matthew Blashko, Frederik Maes (KU Leuven-ESAT/PSI), Hichem Sahli (VUB-ETRO)

Research groups

KU Leuven (ESAT/PSI), VUB (ETRO), UHasselt (CENSTAT)

PhDs

Sofie Tilborghs (2021), Siri Willems (2023), Bram De Wel (2023), Liesbeth Vandewinckele (2024), Lotte Huysmans (in progress)