Professor Wolfgang Wahlster is a pioneer of AI in Germany and Europe as a founding director of the DFKI. He has served as an elected President of three international AI organizations: IJCAII, EurAI, and ACL. He is an elected Fellow of AAAI, EurAI, and GI. He laid some of the foundations for multimodal dialog systems, user modelling, and speech-to-speech translation cyber-physical production systems for the fourth industrial revolution (Industrie 4.0), a concept that he coined in 2010. Wahlster is a member of the Nobel Prize Academy in Stockholm, the German National Academy Leopoldina and three other prestigious academies. For his research, he has been awarded the German Future Prize, and the Grand Cross of Merit by the Federal President of Germany. (for more info see: https://www.wolfgang-wahlster.de/)

In the next decade of Industry 4.0 a new generation of AI technologies will take smart factories to a new level. Large Language Models (LLMs) will be complemented by Large Process Models (LPMs) and Large Action Models (LAMs), so that generative AI models not only predict what to say or visualize next, but also what to do next with explanations of why these actions make sense. Although deep learning is the most powerful machine learning method developed to date, it has already reached its inherent limits in many industrial application domains. It must be combined with various symbolic approaches in new system architectures. This leads to hybrid LxM (x=L,P, or M) technologies that use holonic multiagent architectures for combining neural approaches with symbolic reasoning technologies such as constraint solving, physics-based simulation and terminological reasoning in knowledge graphs.

Christophe Cerisara is a French researcher at CNRS (National Centre for Scientific Research), specialized in machine learning models for natural language processing (NLP). He has created and is leading the SYNALP research team composed of about 20 NLP researchers since 2012. He is also the leader of the AI-NLP axis of the LORIA laboratory since 2019, and he has been referent for the French National Plan in AI in 2020. He has supervised more than 12 Ph.D. thesis, and has lead several projects about AI and training Large Language Models in the past few years.

The first part of this course will give the basic principles of the transformer architecture and how the decoder can be trained to build a Large Language Model (LLM), including a short overview of its scaling laws. The second part will present how to use such a trained LLM, either directly through zero-shot and in-context learning, or through fine-tuning to adapt the LLM to a given task, but with a focus on the direct usage of the LLM on either low-end and high-end hardware and without going into the details of parameter-efficient fine-tuning and other advanced adaptation strategies. The third part (30′) will consist of a practical session about how to implement this with the huggingface transformers library. The prerequisites for this course are a good knowledge of python and of fundamentals of machine learning; some experience with pytorch is useful.

Kevin Baum, a computer scientist (M.Sc.) with a doctorate in philosophy, is the head of the Center for European Research in Trusted Artificial Intelligence (CERTAIN) and deputy head of the Neuro-Mechanistic Modeling (NMM) research department at the German Research Center for Artificial Intelligence (DFKI). In various interdisciplinary research projects, he has researched primarily on the sense and nonsense of transparency and explainability requirements for AI systems with regard to societal desiderata such as recognizing unfairness, the effectiveness of human oversight and enabling moral responsibility. He developed the award-winning lecture Ethics for Nerds, is and has been a member of various ethics committees, and is part of Algoright e.V., the interdisciplinary non-profit think tank for good digitalization and science communication.

“Progress in the field of AI is breathtaking. Large models, foundation models, multimodality: all this not only opens up a wide range of new possibilities, be it in code generation with LLMs or in robotics via Large Action Models, but also raises new societal and ethical challenges. In his provisional keynote, Kevin Baum provides an overview of current normative challenges, sorts out loose threads, and outlines some resulting research opportunities.”

Dr. Malte Ostendorff is a senior research engineer at Deutsche Telekom where he works on large language models (LLMs) and related topics. Previously, Malte was a senior researcher at the German Research Center for Artificial Intelligence (DFKI) and a Ph.D. student in the Scientific Information Analytics group at the University of Göttingen. Furthermore, Malte is a co-founder of Occiglot, a research collective for open-source language models for and by Europe, and a co-founder of Open Legal Data.

Large language models (LLMs) have emerged as a powerful technology underpinning state-of-the-art chatbots and various other natural language processing applications. Model sizes and computing resources that are used for building LLMs dominate the public discourse around these models, whereas one crucial aspect is often neglected – the LLM training data. LLMs are statistical models that learn from data and, therefore, the training data is crucial for LLMs and one of the main differentiators between different models. In this course, we will explore the datasets that were used by existing LLMs, automated tools for data curation and processing at the Web scale, and the most prominent sources where the data is coming from. We will discuss why commercial LLM providers are secretive about their data and what issues arise from training models on large-scale datasets. A basic understanding of what LLMs are and how they are trained is a prerequisite for this course.

Christophe Cerisara is a French researcher at CNRS (National Centre for Scientific Research), specialized in machine learning models for natural language processing (NLP). He has created and is leading the SYNALP research team composed of about 20 NLP researchers since 2012. He is also the leader of the AI-NLP axis of the LORIA laboratory since 2019, and he has been referent for the French National Plan in AI in 2020. He has supervised more than 12 Ph.D. thesis, and has lead several projects about AI and training Large Language Models in the past few years.

Beyond performance on tasks, another important characteristic of LLMs is their efficiency, i.e., their memory and computational costs during training, inference and deployment. In this course, I will first cover aspects related to efficient LLM training and then focus on low-data regimes.
The first part starts from discussing memory and computational costs of LLM, and covers concepts such as quantization, adapters, prefix tuning, qLoRA and model compression. The second part will briefly introduce notions related to transfer learning and continual learning: overfitting, learning to hallucinate, catastrophic forgetting, few-shot learning. The lab will focus on parameter-efficient methods.
The prerequisites for this course are the first introductory course about LLM and notions of machine learning.
I’d like to thank Simon Ostermann for providing the initial materials for this course.

Mariya Toneva leads the Bridging AI and Neuroscience group (BrAIN) at the Max Planck Institute for Software Systems. Her research is at the intersection of Machine Learning, Natural Language Processing, and Neuroscience, with a focus on building computational models of language processing in the brain that can also improve natural language processing systems. She obtained her PhD from Carnegie Mellon University in a joint program between Machine Learning and Neural Computation.

Current large language models (LLMs, e.g. ChatGPT, GPT-4, etc.) have impressive capabilities, but how closely do they actually align with the capabilities of the only system that truly understands complex language–the human brain? In this session, we will learn the core computational techniques for relating language in machines and language in the brain. We will also have a hands-on session with brain recordings of people processing complex language (e.g. reading a book).

The prerequisites are good familiarity with programming in python and basic machine learning concepts, such as regression and cross validation.

Dr. Jindong Gu is a senior research fellow at University of Oxford. He also partially works in Google DeepMind as a faculty researcher in Gemini Safety team. Prior to that, He received his Ph.D. Degree from University of Munich. His research focus is to build Responsible AI systems. Specifically, he is interested in the interpretability, robustness, privacy, and safety of visual perception, foundation models, robotic policy and planning, and their fusion towards general intelligent systems.

In recent years, generative AI (GenAI), like large language models and text-to-image models, has received significant attention across various domains. However, ensuring the responsible generation of content by these models is crucial for their real-world applicability. This raises an interesting question: What should responsible GenAI generate, and what should it not? This course will introduce the practical responsible requirements of both textual and visual generative models, outlining five key considerations: generating truthful content, avoiding toxic content, refusing harmful instruction, leaking no training data-related content, and ensuring generated content identifiable.

Xavier Hinaut is Research Scientist in Bio-inspired Machine Learning and Computational Neuroscience at Inria, Bordeaux, France since 2016. He received a MSc and Engineering degree form Compiègne Technology University (UTC), FR in 2008, a MSc in Cognitive Science & AI from EPHE, FR in 2019, then his PhD of Lyon University, FR in 2013. He is a member (Vice Chair) of IEEE CIS Task Force on Reservoir Computing. His work is at the frontier of neurosciences, machine learning, robotics and linguistics: from the modeling of human sentence processing to the analysis of birdsongs and their neural correlates. He both uses reservoirs for machine learning (e.g. birdsong classification) and models (e.g. sensorimotor models of how birds learn to sing). He manages the “DeepPool” ANR project on human sentence modeling with Deep Reservoirs architectures and the Inria Exploratory Action “BrainGPT” on Reservoir Transformers. He leads ReservoirPy development: the most up-to-date Python library for Reservoir Computing. https://github.com/reservoirpy/reservoirpy He is also involved in public outreach, notably by organising hackathons from which fun projects with reservoirs came out (ReMi project on reservoir generating MIDI and sounds).

Language involves several levels of abstraction, from small sound units like phonemes to contextual sentence-level understanding. Large Language Models (LLMs) have shown an impressive ability to predict human brain recordings. For instance, while a subject is listening to a book chapter from Harry Potter, LLMs can predict parts of brain imaging activity (recorded by functional Magnetic Resonance Imaging or Electroencephalography) at the phoneme or word level. These striking results are likely due to their hierarchical architectures and massive training data. Despite these feats, they differ significantly from how our brains work and provide little insight into the brain’s language processing. We will see how simple Recurrent Neural Networks like Reservoir Computing can model language acquisition from limited and ambiguous contextual data better than LSTMs. From these results, in the BrainGPT project, we explore various architectures inspired by both reservoirs and LLMs, combining random projections and attention mechanisms to build models that can be trained faster with less data and greater biological insight.

Dr. Lelarge is a researcher at INRIA and a lecturer in the computer science department of Ecole Normale Superieure. He graduated from Ecole Polytechnique, qualified as an engineer at Ecole Nationale Superieure des Telecommunications and received a PhD in Applied Mathematics from Ecole Polytechnique in 2005. His research interests include machine learning, deep learning, graphs and programming languages.

This course focuses on the use of large language models (LLMs) for generating and verifying code. Students will learn the foundational principles needed to build contemporary neural code generation systems, including techniques for formal-method aware fine-tuning models, performing inference, and evaluating results. This course explores new research in neural code generation, including improving interaction with human programmers, model reliability and adaptability, and applications to engineering, reasoning, and formal verification. The course encompasses algorithmic foundations, practical applications, and emerging research areas in neural code generation.

Gerrit Großmann received his doctorate in Saarbrücken. His PhD topic was the behavior of stochastic processes on graphs and networks, including the spread of (online and offline) epidemics. He also worked within the interdisciplinary project NextAID, where he researched neuro-symbolic approaches for drug discovery, specifically by using diffusion models and graph neural networks. Gerrit has been researching at DFKI in Saarbrücken and Kaiserslautern since 2023. His research interests there revolve around the question of how to integrate the distinct realms of discrete structures such as graphs and networks with the continuous nature of dynamic evolution, diffusion, and learning.

Despite their groundbreaking impact, LLMs have their imperfections. This track examines the integration of LLMs with structured information like knowledge graphs. We investigate ways to improve the quality and reliability of LLMs and techniques for extracting structured data from them. By the end of the lab, you will have a first prototype of an implementation of an LLM combined with a knowledge graph. No specific experience working with LLMs is required, but some basic knowledge of deep learning is recommended.

Islam Mesabah obtained his Master’s degree in Computer Science from the RPTU (Rhineland-Palatinate Technical University) in Kaiserslautern. His master’s thesis focused on the application of Large Language Models (LLMs) for effective code generation through the utilization of API documentation. Additionally, he researched text-style transfer evaluation using LLMs. Since 2023, Islam Mesabah has been serving as a researcher at the German Research Center for Artificial Intelligence (DFKI) in Kaiserslautern. His research at DFKI primarily explores the applications of Large Language Models and key information extraction and structuring from image documents. In addition to his research endeavors, Islam holds the position of teaching assistant for the “Engineering with Generative AI” course at RPTU Kaiserslautern.

Despite their groundbreaking impact, LLMs have their imperfections. This track examines the integration of LLMs with structured information like knowledge graphs. We investigate ways to improve the quality and reliability of LLMs and techniques for extracting structured data from them. By the end of the lab, you will have a first prototype of an implementation of an LLM combined with a knowledge graph. No specific experience working with LLMs is required, but some basic knowledge of deep learning is recommended.

Alexandre is part of the founding research team at Kyutai, a leading non profit research lab in Paris. Before he was a research scientist for 3 years at Meta AI Research, leading in particular the development of the AudioCraft framework (EnCodec, AudioGen, MusicGen). Alexandre completed his PhD at Facebook AI Research and INRIA Paris, working in particular on music source separation (Demucs).

In this course, we will learn the theory around discrete audio modeling, covering the different components and techniques used (neural audio codec, multi stream transformers, etc), as well as the specificities of the audio domain. Then, we will apply theses techniques to fine tuning pre trained audio models to new datasets of audio. Attendees should have previous experience with Pytorch for the practical part of the class, along with a setup google account to use google Colab. Previous experience working with audio will help but is not required!