Retrieval-Augmented Visual and Language Modelling for Intelligent Document Processing (RAVLIDP)

This project aims to push the boundaries of intelligent document processing. By harnessing advanced visual and language modelling techniques, we aim at developing proof-of-concepts on intelligently understanding and retrieving information from documents. This involves extracting semantic-rich data from documents presented in various formats, including images, using advanced visual processing. We then parse this semantic information using large language models. Ultimately, we aim to intelligently retrieve information based on (combination of) these processes. Our goal is not solely to improve document searching; we aim at fundamentally enhancing document processing, making it smarter and more intuitive for everyone, particularly in industries where extensive document analysis still requires significant human effort.

The project has successfully developed and implemented advanced solutions for intelligent document processing. The delivered system combines sophisticated visual and language modeling techniques, demonstrated through a functioning prototype with proven capabilities in document understanding and information retrieval. The completed web-based prototype effectively processes documents through three key functionalities: accepting domain-specific user questions, automatically reviewing and identifying relevant documents, and generating appropriate answers based on the analyzed content. The system has been successfully deployed via a web interface. Additionally, the project has delivered an enhanced table parsing algorithm specifically designed for complex industrial documents, with documented improvements in performance on challenging cases. The system has demonstrated its ability to handle various document formats, including images, through advanced visual processing and large language model integration, making document processing more efficient and intuitive for users across different business sectors.

Downloads

Prototype for LLM based document reviewing and summarization (web interface QbD – link)

Contact: Leander Schietgat (VUB), David Verrier (VUB)
Researchers: Ann Dooms (PI), Tan Lu (co-PI)
Research groups: VUB DIMA