Internship Reinforcement learning of VLMs for table understanding-Saclay-H/F

Descripción del puesto:

Position description

Category

Miscellaneous

Contract

Internship

Job title

Internship Reinforcement learning of VLMs for table understanding-Saclay-H/F

Subject

Based in Saclay (Essonne), the LIST is one of the two institutes of CEA Tech, the Technological Research Division of the CEA. Dedicated to intelligent digital systems, its mission is to carry out technological developments of excellence on behalf of industrial partners, in order to create value.
Within the LIST, the Laboratory of Textual and Visual Semantic Analysis (LASTI) conducts its research in the field of natural language processing and computer vision to extract, classify and generate information. The laboratory's research themes include learning with few data, trustworthiness and multimodality.

Contract duration (months)

6 monhs

Job description

As an intern at the CEA, you will have the opportunity to work within a world-renowned research environment. Our teams consist of passionate and dedicated experts, providing an ideal setting for learning and collaboration. You will have access to state-of-the-art equipment and top-tier research resources to carry out your assignments. The work conducted may potentially lead to a scientific publication.

Context :
The team trains vision-language models (VLMs) that read an image of a table and output its structure. Manual table annotation is expensive, which limits classical supervised approaches.
This internship focuses on training VLMs with reinforcement learning (GRPO), using reward signals that do not depend on human annotations: for example, checking the geometric consistencies or generating synthetic tables whose structure is known by construction. The central question is which annotation-free signals correlate well with the real quality of a prediction, and how to combine them to improve the model on real documents.

What do we expect from you?
Get started with the team's training and evaluation environment
Analyse automatic evaluation methods
Propose and implement new approaches, and validate them through experiments
Work on the performance of the pipeline and the robustness of the models
Evaluate the models on public benchmarks
Document the results
The intern works on an existing pipeline, with access to a GPU cluster (Python, PyTorch, HuggingFace ecosystem, distributed training, experiment tracking). Supervision research engineers of the team, with a weekly meeting.

#Cea-List

Methods / Means

python linux

Applicant Profile

Students in their final year of studies (M2 or engineering school)
Python and PyTorch, Linux
Knowledge of deep learning and statistics
Good level of English
Appreciated: experience with HuggingFace Transformers, notions of reinforcement learning (GRPO, DPO, …)

Position location

Site

Saclay

Job location

France, Ile-de-France, Essonne (91)

Location

Saclay

Candidate criteria

Languages

English (Intermediate)

Prepared diploma

Bac+5 - Diplôme École d'ingénieurs