The Thinking Machine - Part 2

Focus, Not Memory: The Rise of Attention

(If you missed Part 1 on RNNs and the memory problem, you can read it here.)

1. Introduction – A Change in Perspective

When machines first began tackling language, we taught them to remember.

Recurrent Neural Networks processed words one at a time, carrying memory forward like a fragile thread by hoping it wouldn’t break. LSTM and GRU added structure to that memory. They helped, but only up to a point.

The underlying mindset was always the same:

Preserve the past. Compress it. Hope it’s enough.

But at some point, researchers asked a different kind of question, a more human one:

What if the key to understanding isn’t memory alone, but focus?

Humans don’t replay everything we’ve ever heard in our heads. We don’t store entire conversations to make sense of the next word. Instead, we’re remarkably good at selective attention. We filter, prioritize, and zoom in on what matters, often without realizing it.

This was the insight that changed machine learning.

In this chapter, we’ll explore how the attention mechanism gave models a new kind of intelligence:

Not the ability to remember everything,
But the ability to look back selectively and learn what to focus on.

We’ll trace the origins of attention in neural networks, unpack its intuition, and set the stage for the architectural revolution that followed.

Let’s begin with something familiar:
How you understand language and why that might be the better blueprint.

2. The Human Analogy – Selective Focus

Imagine someone reading this sentence:

“Despite the stormy weather and delayed trains, the meeting was a success.”

To make sense of “was a success”, your brain naturally reaches back. Not to “weather” or “trains”, but to “meeting”. You didn’t re-read the whole sentence, and you certainly didn’t replay every previous word in order. You simply knew where to look.

This is the power of attention.

🧠 Humans Aren’t Sequence Machines

We don’t treat everything we hear or read equally.
Instead, we subconsciously assign importance to parts of what we perceive, based on relevance, expectation, or experience.
We’re constantly filtering:

What to notice
What to ignore
What to remember for now

This mental triage is fast, flexible, and rarely conscious. It’s why we can hold a conversation in a noisy room or follow the main thread of a complex book without remembering every word.

📚 From Human Focus to Machine Focus

Early neural networks lacked this ability.
They processed sequences blindly, treating each token with equal weight or relying on compressed memory to carry context forward.

But human cognition showed a different model:

Understanding doesn’t require memorizing everything.
It requires knowing what matters right now.

This inspired a shift from fixed memory structures to dynamic relevance estimation. A mechanism that could learn, during training, which parts of the input to “pay attention” to.

And so, in 2014, the first formal Attention mechanism appeared in the context of machine translation.

It wasn’t yet a revolution but it was a spark.

🧾 Recap: How Sequence Models Generate Language – One Word at a Time

Before we dive into attention mechanisms, let’s take a brief step back.

All the models we’ve discussed so far — RNNs, GRUs, LSTMs — are sequence-based. They process and generate tokens one at a time, in a chain-like fashion.

Let’s say we’re trying to generate a sentence like:

“The cat sat on the mat.”

A recurrent model doesn’t see the whole sentence at once. Instead, it builds it word by word, updating its internal context at each step:

Input → "The"      → Model state updated
Input → "cat"      → State updated again
Input → "sat"      → ...

At every step, the model uses the current context (its hidden state) to predict the next word.

🎯 What is “Context” in a Sequence Model?

You can think of the context as a running summary. A vector that accumulates knowledge about the sequence so far.

It contains:

Some memory of previous words,
Some notion of what might come next,
And (hopefully) enough information to make a good guess.

But here’s the catch:
That context is implicitly learned, and compressed into a single hidden state, and this especially in RNNs. The model can’t directly “look back” at specific words. It can only rely on what it has remembered in a vague, entangled form.

🧠 Where Attention Steps In

What if we could do better than just carrying a vague summary forward?

What if, instead of remembering everything equally, the model could look back at the entire sequence and decide what to focus on?

That’s what attention does:

It treats all previous tokens as available context, not just the latest state.
And it assigns weights to each part by telling the model what is more relevant right now.

Instead of dragging context along like a suitcase, the model can now selectively unpack what it needs.

This shift, from fixed, implicit memory to dynamic, weighted focus, is what turned attention into the most powerful idea in modern machine learning.

3. The First Attention Mechanisms in Machine Learning

The year was 2014, and researchers working on neural machine translation were facing a problem:

Even with LSTMs, it was difficult for the model to align source and target sequences and this especially in long sentences. The compressed memory of the encoder just wasn’t enough.

That’s when Bahdanau et al. introduced something radical:

A mechanism that allows the model to look back at the entire input sequence and learn where to focus when generating each output word.

🧭 The Key Insight: Dynamic Alignment

In traditional encoder-decoder setups:

The encoder compresses the entire input sequence into a single vector.
The decoder then tries to generate the output from that one vector.

But this creates a bottleneck, especially when translating longer sentences.

Bahdanau Attention changed that.
Instead of encoding everything into one fixed state, it allowed the decoder to:

Examine all encoder outputs at every step,
Assign a weight (attention score) to each one,
And compute a weighted sum of those encoder outputs forming so a custom context vector for each output word.

In simple terms:

Each time the model generates a word, it asks:
“Which parts of the input are relevant right now?”

🧠 Visual Intuition

You can think of the encoder as producing a list of representations: Encoder outputs: $\text{Encoder outputs: } [h_1, h_2, …, h_T]$

Then at each decoding step, the attention mechanism:

Scores how relevant each $h_t$ is to the current decoding state,
Applies softmax to turn scores into weights,
Computes a weighted average of the encoder outputs.

That weighted sum becomes the context vector used to generate the next word.

💡 Why It Worked So Well

Attention made the translation process adaptive:

It allowed the model to explicitly focus on relevant words, even if they were far away in the sequence.
It eliminated the need to cram all context into a single vector.
It improved both accuracy and interpretability. We could visualize which source words were being attended to.

Suddenly, machines weren’t just repeating learned patterns.
They were starting to look.

This mechanism wasn’t limited to translation for long.
Its flexibility made it appealing for many tasks:

Summarization
Image captioning
Question answering
Speech recognition

And as we’ll see next, the general structure of Attention, not just in application, but in form, became the blueprint for a new kind of modeling altogether.

4. Anatomy of Attention – Query, Key, Value

To understand modern attention mechanisms, we need to move from the intuitive idea of “focus” to a mathematical structure that can be learned and optimized.

That structure is based on three simple yet powerful components:

Query (Q), Key (K), and Value (V)

🎯 Intuition First

Let’s say you’re reading a sentence and trying to figure out the meaning of a word, for example, “it” in:

“The dog chased the ball because it was bouncing.”

Your brain tries to resolve what “it” refers to. To do this, it:

Forms a mental representation of “it” → the Query
Compares it against previous concepts (“dog”, “ball”) → the Keys
Retrieves the most relevant meaning → the Value

The attention mechanism does exactly this, but with vectors and dot products.

🔄 What Do Query, Key, and Value Really Mean?

Let’s slow down and look again at the three core components of attention:

Query, Key, and Value

At first glance, the names might seem cryptic or computer-sciencey like something out of a database or hash table. And in a way, that’s not far off.

But these terms each play a distinct and meaningful role in how attention works. Let’s unpack them more deeply.

🧠 Query: What I’m Looking For

A Query represents the current unit of meaning we’re trying to process. For example, a specific word in a sentence, or an image region, or a token in a code snippet.

It’s the model’s way of asking a question:

“What information do I need right now to understand or produce this next step?”

Each Query is unique to the current position in the sequence.
If we’re generating the word “is” in a sentence, the Query encodes the context around that position and asks: “What should I focus on to predict this word?”

🧷 Keys: What’s Available for Matching

A Key is a compact representation of each possible information source in the input.
If we’re working with a sentence, then each previous word (or token) has a Key which expresses what that token is about.

“Here’s what I represent. If you’re looking for something like me, I might be relevant.”

In other words:

The Query says: “I need X.”
The Key says: “I am Y — is that close to what you need?”

The dot product between a Query and a Key gives a similarity score. This is then a way of measuring how well a given token matches the current need.

🧳 Value: What You Actually Get If It’s Relevant

Finally, the Value is the actual content the model retrieves when a Key is relevant.

In many setups, the Key and Value are derived from the same source (e.g., the same token embedding). But their functions are different:

Keys are used for selection: they act like search tags or summaries.
Values are what the model collects and combines to form new understanding.

Once attention scores are computed, the model produces a weighted average over all the Values, where the weights reflect how well each Key matched the current Query.

🧭 Bringing It Together: A General Pattern

Here’s a more abstract way to view it — applicable across domains:

Role	Semantic Meaning	Example (language)	Example (images)
Query	What I need	A word I’m predicting	A region I’m analyzing
Key	What’s available	Context tokens	Surrounding patches
Value	What I’ll retrieve	Token embeddings	Visual features

This structure is domain-agnostic:
It works for words, pixels, audio frames, code tokens and anywhere where some parts of the input should influence others.

💬 Why This Structure Matters

At a high level:

The Query says: “Here’s what I need.”
The Keys say: “Here’s what I can offer.”
The dot product tells us: “How relevant are we to each other?”
The Values say: “Here’s the actual information you’ll use.”

This separation allows the model to:

Control influence between parts of the sequence
Learn soft, contextual relationships
Keep the process differentiable for training with backpropagation

And unlike RNNs, this relationship doesn’t rely on token position or memory — it’s content-based, dynamic, and parallel.

🧾 A Practical Example: Applying Attention to a Sentence

Let’s use this sentence:

“The cat sat on the mat.”

Let’s say the model is trying to process or generate the word “mat”.
That word will act as the Query so the model wants to know:

“What context should I focus on to understand or generate the word mat?”

🔍 Step 1: Assigning Roles

Query (Q): The current word we’re working on → “mat”
Keys (K): All previous words (available context):
- “The”, “cat”, “sat”, “on”, “the”
Values (V): The actual content or meaning of those previous words

In the real model, each of these words has been transformed into vector representations (via embeddings). But conceptually, you can think of:

The Query vector as representing: “I need information to understand ‘mat’”
Each Key vector as saying: “Here’s what this earlier word is about”
Each Value vector as: “Here’s what you’ll get if you attend to me”

🧠 Step 2: Similarity Scoring (Query vs. Keys)

The model compares the Query (mat) to each Key:

Key Word	Similarity to “mat”	Intuition
the	Low	Not very informative
cat	Medium	Related, but not directly
sat	Medium	Verb — provides context
on	High-ish	Important preposition
the	Low	Again, generic
→ Best match: on (and maybe sat)

These similarity scores are computed via dot products in vector space. And keep in mind: The similarity is based on word embeddings, so that the distance/direction has a meaning here. But conceptually, the model is asking:

“How much should I rely on each word when processing ‘mat’?”

🎯 Step 3: Softmax → Attention Weights

The raw similarity scores are turned into weights using softmax, making them sum to 1.

Let’s say the attention weights come out like this:

Word	Attention Weight
the	0.05
cat	0.10
sat	0.25
on	0.50
the	0.10

This means:

“To understand mat, I’ll rely mostly on on, a bit on sat, and very little on the rest.”

🧳 Step 4: Weighted Sum of Values

The model now blends the Values of each token according to the attention weights.
This produces a context vector: a learned representation of everything relevant for processing “mat” in this moment.

It’s like the model says:

“I’ll build my understanding of ‘mat’ from the parts of the sentence that told me what is on what.”

✨ Summary: What Just Happened?

Step	What It Means
Query	Represents what I want to understand (e.g., “mat”)
Keys	Represent what each other word offers
Similarity	Computes how relevant each word is
Softmax	Turns scores into attention weights
Values	Provide actual meaning content
Output	Blended context vector = “What I should know now”

And most importantly:

The model didn’t just look at everything equally — it learned to focus.

This is the breakthrough of attention.
Instead of blindly dragging along a memory state, we now let the model decide what matters and blend the past accordingly. And now to the math 😉

🧮 The Core Formula

Let’s define:

$Q \in \mathbb{R}^{n \times d_k}$: the set of Queries
$K \in \mathbb{R}^{n \times d_k}$: the set of Keys
$V \in \mathbb{R}^{n \times d_v}$: the set of Values

Then attention is computed as:

$$
\text{Attention}(Q, K, V) = \text{softmax} \left( \frac{Q K^\top}{\sqrt{d_k}} \right) V
$$

Let’s break this down:

1. Similarity Scores: $Q K^\top$

We take the dot product of each query vector with all key vectors. This gives us a score matrix that expresses how relevant each key is to a given query.

High dot product → high relevance
Low dot product → low relevance

This is how the model “decides” what to focus on.

2. Scaling: $\frac{1}{\sqrt{d_k}}$

Why scale the dot product?
Because as the dimensionality $d_k$ increases, the raw dot products can become large in magnitude, leading to sharp gradients after softmax. Scaling stabilizes this.

3. Softmax

We apply softmax to each row of the score matrix, turning the raw similarities into probabilities: a set of weights that sum to 1.

This is the attention distribution:
A learned set of weights over the available context.

4. Weighted Sum: $\text{softmax}(\cdot) V$

Finally, we use the attention weights to compute a weighted average of the value vectors.
The result is a context vector, one that reflects not just the current input, but everything else the model has chosen to focus on.

🔁 Each Position Gets Its Own Focus

Attention isn’t a one-time operation. It is applied for every position in the sequence.

Each token has:

Its own Query vector (what it’s trying to understand)
Shared Keys and Values from the entire sequence (the available context)

This means:

Every token attends to every other token.
Relevance is learned, so no handcrafted rules.

🔍 But What Is the Output of Attention, Really?

It’s important to pause and clarify a subtle but essential point:

The result of the attention mechanism (the context vector) is not a single number.
It’s a vector, just like the input Value vectors.

When we write:

$$
\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^\top}{\sqrt{d_k}}\right) V
$$

what actually happens is:

The softmax produces a set of attention weights (one per input token)
These weights are applied component-wise to each Value vector
The result is a new vector, where each dimension is a weighted sum of the corresponding dimension across all Values

In simple terms:

You’re not blending scalars.
You’re blending full semantic vectors like in mixing shades of meaning.

🧠 A Tiny Example (3D)

Imagine you’re attending to three previous words, and their Value vectors are:

cat → [0.2, 0.6, 0.4]
sat → [0.8, 0.1, 0.3]
on → [0.5, 0.9, 0.7]

With attention weights:

cat: 0.1
sat: 0.3
on: 0.6

You calculate:

$$
\text{Context Vector} = 0.1 \cdot [0.2, 0.6, 0.4] + 0.3 \cdot [0.8, 0.1, 0.3] + 0.6 \cdot [0.5, 0.9, 0.7] = [0.56, 0.63, 0.55]
$$

This resulting vector encodes the current focus of the model that is shaped by the most relevant tokens.

🎯 Why This Matters

The attention weights don’t just “make the vector bigger or smaller.”
They steer the vector in a different direction, directly towards the meanings that matter most for this specific context.

And that vector becomes:

Input to the next Transformer layer
Or the basis for the next word prediction
Or part of a classification decision

So yes:

The more weight a token gets, the more it pulls the resulting meaning in its direction.

That’s how deep learning models, quite literally, shift their understanding based on focus.

💡 Summary

Component	Role
Query (Q)	What I’m currently trying to understand
Key (K)	What’s available to match against
Value (V)	What I’ll retrieve if a match is found
Output	A weighted blend of values, based on attention scores

This mechanism is differentiable, interpretable, and learnable, which made it the perfect building block for large-scale neural architectures.

5. Why Attention Was a Breakthrough

The introduction of Attention wasn’t just a technical upgrade.
It was a shift in how we think about context, memory, and learning in neural networks.

Here’s why it changed everything:

⚡️ 1. It Made Context Explicit

Traditional RNNs and LSTMs carried context in the form of a hidden state. A compressed, fixed-size vector meant to summarize everything that came before.

But this state was opaque, lossy, and position-dependent.

With attention, the model doesn’t guess what to remember, it explicitly selects what to look at.

Every token in the sequence becomes directly and dynamically accessible, weighted according to relevance.
This turns context into something transparent and controllable.

🧮2. It Allowed Content-Based Access

RNNs access the past sequentially, step by step.
Attention removes this constraint entirely.

Instead of asking, “What happened right before?”, attention asks:

“Which parts of the entire input are relevant to me right now?”

This idea of content-based lookup is powerful.
It’s like building an internal search engine into the network.

The model doesn’t care if something happened 2 steps ago or 20.
If it’s important, it can find it instantly.

🚀 3. It Enabled Parallelization

Sequential models are slow to train. Each step depends on the previous one, making parallel computation difficult.

But attention mechanisms don’t have this dependency.
They let every position in the sequence compute its output at the same time, because:

The full sequence is available from the start
The computation is just a bunch of matrix multiplications

This made massively parallel hardware like GPUs shine.
Training became faster. Scaling became easier.
And suddenly, models could handle much more data.

🧠 4. It Was Learnable and Interpretable

Attention weights are differentiable and they can be optimized via backpropagation.

That means the model learns:

What to focus on for each type of input
How to shift attention based on task or structure
Even how to attend to nothing, if needed

And because attention weights are visible, we can inspect what the model is doing.
We can visualize them. We can debug them.
It’s one of the few places in deep learning where reasoning steps are observable.

🔑 5. It Scales With Meaning

The more complex the input, the more valuable attention becomes.

For short sequences, RNNs and LSTMs might do okay.
But for long sequences (documents, paragraphs, multi-sentence reasoning) attention shines.

It scales because:

Every position has full access to everything else
And the model can learn long-range dependencies, not just remember them

✅ Summary: What Attention Gave Us

Property	Traditional Models	Attention
Memory	Compressed into hidden state	Access to full sequence
Focus	Implicit	Explicit & dynamic
Scalability	Sequential, slow	Parallel, fast
Interpretability	Low	High
Learnability	Manual design	Fully trainable

Attention wasn’t just a clever trick — it was a doorway.

6. Outlook – A New Architecture Emerges

The attention mechanism solved a fundamental problem:

It let models choose what to focus on, rather than trying to remember everything.

But researchers quickly realized something even more powerful:

What if every word in a sentence could attend to every other word — including itself?

This idea gave birth to Self-Attention, a mechanism where each token dynamically relates to all others, regardless of position.

And when you stack layers of self-attention, interleaved with feedforward components and residual connections, you get something remarkable:

🎯 The Transformer: the architecture behind GPT, BERT, T5, and modern large language models.

In the next part of this series, we’ll explore:

What Self-Attention actually does (intuitively and mathematically)
Why it replaced RNNs completely in many NLP tasks
And how it forms the backbone of today’s most powerful models

The Thinking Machine – Part 2