Transformers at Work 2023

September 15, 2023 | Science Park Amsterdam

PAST EVENT. RECORDINGS AVAILABLE NOW!

TRANSFORMERS AT WORK

4th edition

After a very fast rise, Transformers have become the work horse of modern AI. From edge devices to the largest language models in the cloud, computer vision, robotics, reinforcement learning, they are everywhere. Even modern hardware has become optimized to the specifics of this Deep Learning architecture.

In the 4th edition of our workshop you will learn about the progress in Transformers from world-renowned researchers pushing the boundaries in their respective subfields such as Neural Information Retrieval, Conversation Agents, Large Language Models, and Multimodality. In contrast to earlier editions, the focus will be more on the shift towards real world applications of AI.

After the workshop food, drinks + live music (Roda da Holanda) create a unique opportunity to meet and have fun with fellow AI R&D folks from Industry and Academia at the beautiful Startup Village Amsterdam!

Meet our speakers:

Nils Reimers

(Cohere)

Connecting LLMs to your data

Suzan Verberne

(Leiden University)

Conversational Agents and Search

Douwe Kiela

(Contextual)

Multimodal LLMs

Madelon Hulsebos

(University of Amsterdam)

Table Representation Learning

Rodrigo Nogueira

(Zeta Alpha, UNICAMP)

Adapting LLMs for Languages and Domains

Raza Habib

(Humanloop)

LLMs for Production Applications

Konstantinos Bousmalis

(Google Deepmind)

A Self-improving Foundation Agent for Robotics

Corina Gurau

(Air Street, ex-Wayve)

World Models for Autonomous Driving

New edition, all the more value

LEARNING

We are proud to welcome researchers and leaders from industry and academia. Our speakers will show you the latest developments & most promising advances in the field. The starring role is of course reserved for Transformer models.

DISCOVERY

Advances in NLP and Search are changing the way AI systems can process unstructured data, which makes understanding of the state of the art and its implications for applications critical to getting ahead in virtually any industry.

NETWORKING

Make valuable connections and meet experts and peers not only in the AI industry, but also in all others touched by tech. "If you want to go somewhere, it is best to find someone who has already been there".

FUN

To celebrate the amazing progress in our field and kickstart the academic year with some fun, we can't conclude the evening without some entertainment. Drinks, food, & live music included, good company guaranteed!

What you can expect:

Time schedule:

12:00-13:00 Arrival and registration
13:00 Opening: Jakub Zavrel
13:30 - 14:45 Suzan Verberne, Nils Reimers, Rodrigo Nogueira
14:45 - 15:15 Coffee Break
15:15 - 16:30 Raza Habib, Madelon Hulsebos, Douwe Kiela
16:30 - 16:45 Break
16:45 - 17:30 Corina Gurau, Konstantinos Bousmalis
17:30 - 18:00 Panel "Transformers at Work"
18:00 Drinks, Food & Networking
19:30 Live music: Roda da Holanda

22:00 End

Find the full list of titles and abstracts below.

Speakers, Titles and Abstracts

Nils Reimers (Cohere)
Connecting LLMs to your data

Connecting LLMs to your data either with fine-tuning or retrieval augmented generation (RAG) became in the past 6 months extremely popular. In this talk, I will talk about different options with pro-/cons and challenges in this area. For RAG, semantic search plays a key role: I will cover the basics and the many shortcomings this approach has.

Rodrigo Nogueira (Zeta Alpha, UNICAMP)
Adapting LLMs for Languages and Domains

With the ongoing advancement and success of increasingly large and powerful language models, there's a growing perception that the future will move towards a singular monolithic model, equipped with vast knowledge and capable of tackling a wide range of tasks across various domains. Especially in the multilingual setting, this approach is appealing as it promises a single system that understands hundreds of languages and cultures. However, in this presentation, I will argue that such an approach is not the most computationally cost-effective. Models trained on specific languages and domains, derived from foundational "generalist" models, offer significant benefits at a comparatively lower cost. Specifically, I'll discuss the techniques and lessons learned from constructing language models tailored for the scientific, mathematical, financial domains, and for various specific languages. Finally, I'll delve into potential solutions to current issues of these models, such as mitigating the hallucination problem, and the challenges in building models that engage in continuous learning.

Madelon Hulsebos (University of Amsterdam)
Advances, challenges, and opportunities in Table Representation Learning

The impressive capabilities of transformers have been explored for applications over language, code, images, but the millions of tables have long been overlooked while tables dominate the organizational data landscape and give rise to peculiar challenges. Unlike natural language, tables come with structure, heterogeneous and messy data, relations across tables, contextual interactions, and metadata. Accurately and robustly embedding tables is, however, key to many real-world applications from data exploration and preparation to question answering and tabular ML. In this talk, I will discuss the general approaches taken towards adapting the transformer architecture towards tables and give an overview of the tasks already explored in this space. I will zoom in on some of the shortcomings of these approaches and close with the open challenges and opportunities.

Konstantinos Bousmalis (Google Deepmind)
A self-improving foundation agent for robotics

Learning from heterogeneous robotic experience has long been an impediment to quickly mastering new skills and embodiments in robotic learning. In this talk, I will discuss RoboCat, a self-improving foundation agent for robotic manipulation we created at Google DeepMind. RoboCat is a decision transformer able to ingest action-labelled visual experience from multiple robot embodiments. Its training data spans a large repertoire of motor control skills from simulated and real robotic arms with varying sets of observations and actions. Instructed with visual goals, RoboCat takes images as input to produce actions according to the embodiment it is using. I will talk about RoboCat's ability to generalise to new tasks and robots, and about how it can be used to generate data for subsequent training iterations, thus providing a basic building block for an autonomous improvement loop. I will finally discuss its growing cross-task transfer capabilities as we grow and diversify the training data.

Suzan Verberne (University of Leiden)
Transformers for information search: from document retrieval to source-grounded conversational agents

This talk will explore the inter-connections between information retrieval (IR) and generative large language models (LLMs), summarizing some of our research on domain-specific information retrieval, addressing long document retrieval with BERT-based approaches, then moving to conversational contexts and the use of generative LLMs for information dialogues, large-scale summarization, and data augmentation. The last part will give an outlook on retrieval-augmented conversational agents, introducing a number of challenges that we address in our current work: trustworthiness, efficiency, domain adaptation, and transparency.

Raza Habib (Humanloop)
Pitfalls and Best Practices — Lessons from LLMs in Production

Having seen hundreds of companies go on the journey from playground to production, in this talk, we’ll share case studies of what has and hasn’t worked. Raza shares what the common pitfalls are, emerging best practices, and suggestions for how to plan in such a quickly evolving space.

Douwe Kiela (Contextual AI)
Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs). Our system uses a language model to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever.

Corina Gurau (Air Street, ex-Wayve)
World Models for Autonomous Driving

Learning world models enables autonomous agents to better understand and reason about their environments. In this talk I will present a model that jointly learns a world model and driving policy from videos of expert demonstrations. The world model encompasses static scene elements, dynamic actors, and ego-vehicle behavior in an interpretable latent space. The model is capable of predicting diverse and plausible future states and actions, and can also execute complex driving manoeuvres from plans predicted entirely in imagination.

Registration