Track A: Large AI Models

TRACK A: Large AI Models

The rise of large multimodal models like ChatGPT has significantly influenced both research and public perception of AI in recent months. Theme track A, “Large AI Models,” dives into the foundational technology and the latest breakthroughs of such large models. The track will feature labs covering essential and fundamental topics such as language modeling, multimodal models, and training with massive datasets. Additionally, it will explore more advanced themes, including LLM alignment and efficiency, along with application-specific topics like using LLMs for code generation. Attendees will gain comprehensive insights from leading experts through a combination of theoretical and practical sessions.

Monday, September 9, 2024

Opening Speech

Keynote 1

Prof. Dr. rer. nat. Dr. h.c. mult. Wolfgang Wahlster (DFKI)
Professor Wolfgang Wahlster is a pioneer of AI in Germany and Europe as a founding director of the DFKI. He has served as an elected President of three international AI organizations: IJCAII, EurAI, and ACL. He is an elected Fellow of AAAI, EurAI, and GI. He laid some of the foundations for multimodal dialog systems, user modelling, and speech-to-speech translation cyber-physical production systems for the fourth industrial revolution (Industrie 4.0), a concept that he coined in 2010. Wahlster is a member of the Nobel Prize Academy in Stockholm, the German National Academy Leopoldina and three other prestigious academies. For his research, he has been awarded the German Future Prize, and the Grand Cross of Merit by the Federal President of Germany. (for more info see:

Industrial AI for Smart Manufacturing

In the next decade of Industry 4.0 a new generation of AI technologies will take smart factories to a new level. Large Language Models (LLMs) will be complemented by Large Process Models (LPMs) and Large Action Models (LAMs), so that generative AI models not only predict what to say or visualize next, but also what to do next with explanations of why these actions make sense.
Although deep learning is the most powerful machine learning method developed to date, it has already reached its inherent limits in many industrial application domains. It must be combined with various symbolic approaches in new system architectures. This leads to hybrid LxM (x=L,P, or M) technologies that use holonic multiagent architectures for combining neural approaches with symbolic reasoning technologies such as constraint solving, physics-based simulation and terminological reasoning in knowledge graphs.

Course 1

Christophe Cerisara (CNRS)
Christophe Cerisara is a French researcher at CNRS (National Centre for Scientific Research), specialized in machine learning models for natural language processing (NLP). He has created and is leading the SYNALP research team composed of about 20 NLP researchers since 2012. He is also the leader of the AI-NLP axis of the LORIA laboratory since 2019, and he has been referent for the French National Plan in AI in 2020. He has supervised more than 12 Ph.D. thesis, and has lead several projects about AI and training Large Language Models in the past few years.

Introduction to Large Language Models

Introduction to Large Language Models”] The first part of this course will give the basic principles of the transformer architecture and how the decoder can be trained to build a Large Language Model (LLM), including a short overview of its scaling laws. The second part will present how to use such a trained LLM, either directly through zero-shot and in-context learning, or through fine-tuning to adapt the LLM to a given task, but with a focus on the direct usage of the LLM on either low-end and high-end hardware and without going into the details of parameter-efficient fine-tuning and other advanced adaptation strategies. The third part (30′) will consist of a practical session about how to implement this with the huggingface transformers library. The prerequisites for this course are a good knowledge of python and of fundamentals of machine learning; some experience with pytorch is useful.

Tuesday, September 10, 2024

Keynote 2

Karën Fort (LORIA)
Karën Fort is a Professor at Université de Lorraine and does her research at the LORIA laboratory in Nancy, in the Inria team Semagramme. Her primary research interest is ethics in natural language processing (NLP), of which she is a pioneer: she organized the first colloquium on the subject in 2014, in France, followed by a national workshop (ETeRNAL 2015 and 2020) and a special issue of the TAL journal in 2016. She initiated the ethics and NLP French blog ( as well as the first survey on ethics in NLP (Fort & Couillault, 2016). She was co-chair of the first two ethics committees in the field (EMNLP 2020 and NAACL 2021) and is co-chair of the ethics committee of the association for computational linguistics (ACL). Beside her work on stereotypical biases (Névéol et al., 2022), she is interested in deontological ethics using NLP4NLP techniques (Abdalla et al, 2023).

Ethics in Natural Language Processing: don't look up!

With the success of neural methods, NLP has undergone a major revolution in the past decade: we now have (lots of) real users. Although the field already had an impact on society 10 years ago, it was easier to ignore. Ethics has now become a central issue. In this talk, I will show what can go wrong when we develop a system, if we are not careful enough. I’ll also provide tools and methodologies to better evaluate the impact of our work. More importantly, I’ll show that the issues we are facing are not limited to stereotypical biases and that we need to learn to question our work on a larger scale, as ethical thinking helps developing better systems.

Course 2

Malte Ostendorff (Deutsche Telekom)
Dr. Malte Ostendorff is a senior research engineer at Deutsche Telekom where he works on large language models (LLMs) and related topics. Previously, Malte was a senior researcher at the German Research Center for Artificial Intelligence (DFKI) and a Ph.D. student in the Scientific Information Analytics group at the University of Göttingen. Furthermore, Malte is a co-founder of Occiglot, a research collective for open-source language models for and by Europe, and a co-founder of Open Legal Data.

Training Data for Large Language Models

Large language models (LLMs) have emerged as a powerful technology underpinning state-of-the-art chatbots and various other natural language processing applications. Model sizes and computing resources that are used for building LLMs dominate the public discourse around these models, whereas one crucial aspect is often neglected – the LLM training data. LLMs are statistical models that learn from data and, therefore, the training data is crucial for LLMs and one of the main differentiators between different models. In this course, we will explore the datasets that were used by existing LLMs, automated tools for data curation and processing at the Web scale, and the most prominent sources where the data is coming from. We will discuss why commercial LLM providers are secretive about their data and what issues arise from training models on large-scale datasets. A basic understanding of what LLMs are and how they are trained is a prerequisite for this course.

Course 3


to be announced

Wednesday, September 11, 2024

Keynote 3


to be announced

Course 4


to be announced

Course 5

Jindong Gu (University of Oxford/Google DeepMind)
Dr. Jindong Gu is a senior research fellow at University of Oxford. He also partially works in Google DeepMind as a faculty researcher in Gemini Safety team. Prior to that, He received his Ph.D. Degree from University of Munich. His research focus is to build Responsible AI systems. Specifically, he is interested in the interpretability, robustness, privacy, and safety of visual perception, foundation models, robotic policy and planning, and their fusion towards general intelligent systems.

Responsible Generative AI

In recent years, generative AI (GenAI), like large language models and text-to-image models, has received significant attention across various domains. However, ensuring the responsible generation of content by these models is crucial for their real-world applicability. This raises an interesting question: What should responsible GenAI generate, and what should it not? This course will introduce the practical responsible requirements of both textual and visual generative models, outlining five key considerations: generating truthful content, avoiding toxic content, refusing harmful instruction, leaking no training data-related content, and ensuring generated content identifiable.

Thursday, September 12, 2024

Keynote 4


to be announced

Course 6



Course 7

Gerrit Großmann (DFKI)
Gerrit Großmann received his doctorate in Saarbrücken. His PhD topic was the behavior of stochastic processes on graphs and networks, including the spread of (online and offline) epidemics. He also worked within the interdisciplinary project NextAID, where he researched neuro-symbolic approaches for drug discovery, specifically by using diffusion models and graph neural networks.
Gerrit has been researching at DFKI in Saarbrücken and Kaiserslautern since 2023. His research interests there revolve around the question of how to integrate the distinct realms of discrete structures such as graphs and networks with the continuous nature of dynamic evolution, diffusion, and learning.

Language Models and Structured Knowledge in AI

Despite their groundbreaking impact, LLMs have their imperfections. This track examines the integration of LLMs with structured information like knowledge graphs. We investigate ways to improve the quality and reliability of LLMs and techniques for extracting structured data from them. By the end of the lab, you will have a first prototype of an implementation of an LLM combined with a knowledge graph. No specific experience working with LLMs is required, but some basic knowledge of deep learning is recommended.

Friday, September 13, 2024

Course 8

Alexandre Défossez (Kyutai)
Alexandre is part of the founding research team at Kyutai, a leading non profit research lab in Paris. Before he was a research scientist for 3 years at Meta AI Research, leading in particular the development of the AudioCraft framework (EnCodec, AudioGen, MusicGen). Alexandre completed his PhD at Facebook AI Research and INRIA Paris, working in particular on music source separation (Demucs).

Auto-regressive modeling of discrete audio tokens.

In this course, we will learn the theory around discrete audio modeling, covering the different components and techniques used (neural audio codec, multi stream transformers, etc), as well as the specificities of the audio domain. Then, we will apply theses techniques to fine tuning pre trained audio models to new datasets of audio. Attendees should have previous experience with Pytorch for the practical part of the class, along with a setup google account to use google Colab. Previous experience working with audio will help but is not required!

Comments are closed.