🔗

Yoshua Bengio

Original nameYoshua Bengio

Canadian Computer Scientist, Pioneer of the Attention Mechanism and Sequence Modeling

The Laws Behind the Technology · Theoretical Founders

Neural Machine TranslationThe Early Exploration of the Attention MechanismThe Montreal Institute for Learning Algorithms (Mila)

Who they are

Yoshua Bengio (1964– ) is a Canadian computer scientist who long worked at the University of Montreal on neural networks in natural language processing, making an important theoretical contribution to the "vanishing gradient" problem widely faced by recurrent neural networks handling long-sequence data, and in 2014 with collaborators proposing to bring the "attention mechanism" into neural machine translation models — letting the model, in generating each output word, dynamically give higher-weight attention to the most relevant part of the input sequence, rather than, as earlier models did, having to compress a whole input into a fixed-length vector. This early exploration of the attention mechanism provided an important intellectual precursor for the "Transformer" architecture a Google research team proposed three years later, wholly based on self-attention and wholly abandoning recurrent structure. Bengio founded the "Montreal Institute for Learning Algorithms" (Mila) in Montreal in 2016, which thereafter grew into one of the largest deep-learning research institutions in the world. For these foundational contributions he, with Hinton and LeCun, won the 2018 Turing Award. In recent years Bengio has continued in international public settings to warn of the safety and social risks advanced AI systems may bring, and is one of the core authors of several international AI-safety assessment reports.

Primary sourcesBahdanau, Cho, Bengio, "Neural Machine Translation by Jointly Learning to Align and Translate" (2014, an early attention-mechanism paper)

Key stories

Teaching the Model "Where to Look"

The attention mechanism Bengio and collaborators conceived let a neural machine translation model, in generating each word of the translation, dynamically review the part of the input original most relevant to the current word, rather than, as earlier models did, being forced to compress the whole input into a fixed-length vector and then decode it all at once — this design idea of "dynamic attention on demand" was thereafter proven one of the most important intellectual breakthroughs in deep learning’s handling of sequence data, and the Transformer architecture three years later precisely carried this idea to its extreme.

From Technical Founder to Risk-Warner

Bengio has in recent years invested considerable energy in continually warning, in international public and policy-making settings, of the safety hazards and social risks advanced AI systems may bring, taking part in writing several AI-safety assessment reports of wide international influence, and this shift of role from "laying the technical foundation" to "warning of the technical risk" appears to varying degrees in him and contemporary deep-learning founders such as Hinton, a rather distinctive mental journey for this generation of technical founders facing the wave of technology they themselves drove.

Relationships

Disciple

🎭Ian Goodfellow 🔮The "Transformer Eight"　The attention mechanism Bengio’s team proposed early provided an important intellectual precursor for the later Transformer architecture wholly based on self-attention

Ally

🧠Geoffrey Hinton 👁️Yann LeCun

Echoes today

Below are how modern works borrow or reinterpret this name or story — not the original material. The two differ, so keep them apart.

A core author of the international AI-safety reportsBengio is one of the core authors of several "International AI Safety Reports" supported by multiple governments, reports that have become an important reference for governments in setting AI regulatory policy.

Appears in

The Explosion of Deep Learning and Generative AIPerseverance in the "Neural Network Winter" and the Shock of AlexNet

← Back to Figures