Self-Attention – A Network That Talks to Itself
1. Introduction: From Attention to Self-Attention
Last time, we explored a simple but powerful idea: attention.
Instead of treating all words in a sentence equally, we let the model dynamically decide which words matter more when interpreting the current one. Focus, not memory.
This allowed us to move past the limits of pure recurrence. No more fragile hidden states dragging information through time. Just relevance: learned, weighted, and recalculated as needed.
But if you read closely, you may have noticed something.
In our previous example, attention was always computed from a single point of view. One token looks outward, scanning the others to build its own context. It’s a clever mechanism, but it’s also asymmetric. The others remain silent.
You might ask: what if every token had a say?
That’s where we’re headed.
In this part of the series, we meet self-attention as the architectural leap that turns attention into a full, parallel mechanism. Every token attends to every other token, all at once. The result is more than context: it’s a fully restructured sequence, encoded with deep, distributed meaning.
This is the mechanism that transformed attention from a supporting trick into the backbone of modern AI. But before we jump into math and layers, let’s understand the intuition and the problem it solves.
2. When One Perspective Isn’t Enough
Let’s return to the idea of attention. A single token, let’s say, the word “mat” at the end of a sentence, wants to understand its role. So it scans the previous words, weighs their importance, and builds a context.
It’s efficient. Focused. And slightly lonely.
Because in this setup, only one token is doing the thinking. The others are passive. Content that is to be scanned, judged, and used. They don’t get to update themselves. They don’t get to adapt to the sentence as a whole.
This may be enough for prediction.
When the goal is to guess the next word in a sentence, a one-way attention mechanism that looks at prior context can often do the job. It picks out what seems most relevant and makes a choice.
But it’s not enough for representation, not if we want each word in the sentence to carry a rich, updated meaning that reflects the full context.
Think of it this way:
Prediction is about answering a question.
Representation is about understanding a situation.
In a predictive setup, we might only care about what comes next. But in many tasks, such as translation, summarization, question answering, we want the model to truly understand the sentence. And for that, every word needs to know how it fits in.
We don’t just want to predict “mat” at the end of “the cat sat on the …”
We want to grasp that “cat” is the subject, “on” implies location, and “sat” ties them together.
Each of those words deserves to be revisited and reshaped by the context as it unfolds.
And that’s what self-attention makes possible:
A deep, distributed re-encoding of meaning. And not just a decision at the end of a chain.
If every word is embedded in a different local context, shouldn’t every word also be re-encoded based on that context? Not just used, but changed?
That’s the motivation behind self-attention.
We stop thinking of attention as a one-way spotlight and start seeing it as a conversation. Each token speaks, listens, and rewrites itself in relation to everything else.
The sentence is no longer a chain of dependencies.
It becomes a field of relationships.
And here’s where things shift. In self-attention, every token becomes both the center of attention and part of the background noise. The mechanism isn’t local, it’s global. And instead of focusing on one token’s question, we let every token ask its own question simultaneously.
Yes, it sounds computationally heavy. But it’s not. That’s the beauty.
Because when structured correctly, self-attention can be parallelized and calculated across the entire sequence at once. No recurrence, no delay, no whispering from one step to the next.
The sequence is seen all at once, and each element becomes context-aware in a way that RNNs and single-token attention could never manage.
Let’s pause here. And have a look back to the RNNs.
Recurrent networks process sequences step by step, updating a hidden state along the way. This state carries some information forward, but only in one direction, and always through a narrow bottleneck. It’s like trying to summarize a novel one sentence at a time, with no way to revise earlier paragraphs once new information appears.
Self-attention takes a very different route.
It lets each word consider the entire sequence in one go. Not just the past, not just a summary. And because this operation is fully parallel, it’s not just more powerful: it’s dramatically more efficient.
No more hidden states passed along like a fragile message.
No more vanishing gradients whispering from ten steps ago.
Just one matrix, one operation, and every token gets full access to the bigger picture.
That’s why self-attention didn’t merely complement RNNs. It started to replace them.
In many architectures today, the RNN is gone entirely.
In its place stands a mechanism that understands language not as a stream, but as a structure.
And that shift, from sequence to structure, is where modern AI truly begins.
3. Walking Through an Example (Sentence-Level)
Let’s make this concrete.
We’ll take a simple sentence:
“The cat sat on the table.”
Our goal is not to predict a word, but to contextualize each word.
That is the key difference. In self-attention, we’re not stepping through the sentence one token at a time. We are feeding the entire sentence into the mechanism at once, and asking:
For each word, which other words matter when forming its final meaning?
This is not a rhetorical question.
It’s a precise calculation and it happens for every word, in parallel.
Let’s pick one word to follow, say: “sat”.
The model learns to ask:
- What is the subject of “sat”? (→ “cat”)
- What is the location? (→ “on”, maybe even “table”)
- What could be ignored? (→ possibly the first “the”)
These weights are not hardcoded.
They are learned, based on data. And they are different for each layer and each head (we’ll come to that later).
Now, here’s the twist.
While “sat” is attending to the rest of the sentence, so is “cat”, and so is “table”.
Each word becomes both observer and observed.
The attention mechanism runs for each token, using that token as a reference point. And the result is a new vector: a re-encoded version of that word, now informed by everything around it.
This is self-attention in action.
One input sequence, many overlapping views, each tailored to a single word’s context.
Beyond Sentences
But let’s not stop here.
This idea is not exclusive to language.
Self-attention works on any kind of sequence, as long as it can be represented as a series of vectors:
- Sensor data over time
- Audio waveforms
- User click streams
- Video frame embeddings
- Even sequences of events or transactions
Because from the model’s point of view, a “token” is just a vector.
And the relationships between them (temporal, causal, structural) can all be learned through attention weights.
This is why self-attention is not just a language trick.
It is a general-purpose mechanism for extracting structured meaning from sequential data.
Text was just the first domain to reveal its power.
But it’s far from the last.
4. Self-Attention Mechanics: From Input to Context
Let’s now step fully into the machinery of self-attention. We won’t just describe what it does but we’ll also trace how it works. Intuitively first, then precisely.
The key idea is simple but powerful:
Every token in a sequence learns to look at all the others and decide which ones matter.
This is done not by rules, but by projection. Each word becomes three things:
- a query: what it’s interested in,
- a key: how it can be matched,
- and a value: what it can offer to the final output.
Let’s have a look at how this is done mathematically:
The Setup: From Tokens to Q, K, V
Assume we have a sentence with five words:
“The cat sat on table”
Each word is embedded into a vector of fixed size, say 128 dimensions. These are stacked into a matrix:
$$
X = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \\ x_4 \\ x_5 \\ \end{bmatrix} \quad \text{with shape } (5 \times 128)
$$
Now come the three projection matrices:
- \(W_Q \in \mathbb{R}^{128 \times 64}\) — for queries
- \(W_K \in \mathbb{R}^{128 \times 64}\) — for keys
- \(W_V \in \mathbb{R}^{128 \times 64}\) — for values
These matrices are learned during training. They determine how each token is projected into the attention space.
We apply them like this: \(Q = X W_Q,\quad K = X W_K,\quad V = X W_V\)
The result:
- \(Q, K, V \in \mathbb{R}^{5 \times 64}\)
- Each row now represents one word, seen through a different lens: as a query, a key, and a value.
The Core Operation: From Comparison to Context
Now the real magic begins. We want each token to decide which other tokens to attend to.
1. Compute Attention Scores
We take the dot product of \(Q\) and \(K^\top\):
$$
\text{Scores} = Q K^\top
$$
This yields a \(5 \times 5\) matrix: one row per word, one column per word it could attend to.
2. Scale and Normalize
To prevent exploding gradients, we scale the scores:
$$
\text{Scaled Scores} = \frac{Q K^\top}{\sqrt{d_k}} \quad \text{with } d_k = 64
$$
Then, we apply softmax to each row:
$$
\text{Attention Weights} = \text{softmax} \left( \frac{Q K^\top}{\sqrt{d_k}} \right)
$$
Each row now contains normalized weights which tell the model how much focus each word places on all the others.
3. Weighted Sum of Values
These attention weights are now used to blend the value vectors:
$$
\text{Output} = \text{Attention Weights} \cdot V
$$
The result is a new matrix of shape \((5 \times 64)\). Each row contains a contextualized representation of a token, no longer isolated, but infused with relevant information from the others.
What Learns, What Runs
It’s crucial to distinguish learning from execution:
- During training, the model learns the projection weights \(W_Q, W_K, W_V\) via gradient descent.
- During inference, those weights are fixed and the model simply applies the same matrix steps to new inputs.
So self-attention is not hardcoded logic.
It is a learned way of structuring input, based on how the model has found meaning across examples.
The Big Picture
Let’s step back.
- Each token becomes a query, a key, and a value.
- It uses the query to measure relevance (via keys),
- It gathers useful meaning (via values),
- And it creates a new, context-aware vector.
All tokens do this at once. In parallel. In one pass.
This is why self-attention is fast, expressive, and scalable.
And why it replaced recurrence not just functionally, but fundamentally.
5. Why Self-Attention Replaces RNNs
Let’s say it plainly:
A context vector created by self-attention is not just a memory. It’s a refined summary of meaning, constructed dynamically from the entire sequence, and weighted by relevance, not just order.
Let’s compare that briefly to older methods:
RNNs (including GRUs and LSTMs)
- They build context step by step:
At each time step, they update a hidden state \(h_t\), based only on the current input \(x_t\) and the previous hidden state \(h_{t-1}\). - This is sequential and compressive:
All past information gets “squeezed” into one vector. It’s like trying to summarize an entire paragraph by carrying one sentence forward each time. - As a result, the final context may forget early inputs, or blur together what matters.
Self-Attention
- It builds context in parallel:
Each word considers all other words at once. - And it’s selective:
Important words are weighted more heavily, irrelevant ones fade out. - The resulting vector for each token is not just its meaning, but its meaning in context.
This makes self-attention ideal for tasks where relationships matter more than position: like understanding syntax, resolving ambiguity, or modeling long-term dependencies.
So in the end:
Self-attention builds richer, sharper, more nuanced context representations and does so in fewer steps, with greater flexibility, and better scalability.
That’s the real superpower.
Self-Attention vs. RNN: Core Differences
| Aspect | RNN (e.g., LSTM, GRU) | Self-Attention |
|---|---|---|
| Processing style | Sequential (step by step) | Parallel (all at once) |
| Token interaction | One-way, time-ordered (usually past → present) | Full bidirectional context (or masked, if needed) |
| Memory | Hidden state passed through time | No memory needed — attends to full input each time |
| Speed | Slow (can’t parallelize over time) | Fast (matrix ops over all tokens) |
| Representation | Compresses context into a narrow state | Rich contextual embeddings per token |
| Scalability | Limited by depth and time | Scales well with layers and data |
Self-attention replaces the RNN unit. And this not just functionally, but conceptually.
It removes the dependency on sequence order for computation and builds representations based on relation, not recurrence.
And it does so:
- More flexibly (tokens can attend to any others, not just the past)
- More powerfully (context is not compressed into a single vector)
- More efficiently (fully parallel across the sequence)
That’s why the Transformer architecture (which is based on self-attention) has no recurrence at all.
Self-attention became the new unit: stacked, normalized, projected. And it proved not only sufficient, but superior.
6. Subtleties and Design Choices
If self-attention is so powerful, why does it need help?
Because there are things it doesn’t care about, unless we teach it to.
Let’s look at two of those things: order and variety.
1. Self-Attention Has No Sense of Order
Here’s a strange but important fact:
$$
\text{SelfAttention}(X) = \text{SelfAttention}(\text{Shuffled}(X))
$$
If you feed the same words in a different order, you get the same output, unless you do something about it.
Why?
Because the attention mechanism only compares the content of tokens.
It has no idea whether “cat” came before “sat” or after “table”.
For language, that’s a problem.
Order is not just helpful. It changes meaning.
2. Injecting Order: Positional Encodings
To fix this, we add extra information to each token embedding:
a positional encoding that represents its place in the sequence.
There are different ways to do this, but the original Transformer paper used sinusoidal functions. For a token at position \(pos\), and dimension \(i\), the encoding is:
$$
\text{PE}_{pos, 2i} = \sin\left(\frac{pos}{10000^{2i / d}}\right)
$$
$$
\text{PE}_{pos, 2i+1} = \cos\left(\frac{pos}{10000^{2i / d}}\right)
$$
This produces a unique vector for each position, with smooth transitions and predictable patterns. These vectors are then added to the input embeddings:
$$
X_{\text{final}} = X_{\text{embed}} + \text{PE}
$$
Now, self-attention has access to both the meaning and the position of each token and that without hardcoding the order into the architecture.
3. Outlook: More Heads, More Layers
What we’ve described so far is a single head of attention.
But modern architectures use multi-head attention.
Instead of one query-key-value projection, they learn multiple sets in parallel. Each head learns to focus on different types of relationships: syntax, semantics, alignment, or even word type.
After that, they stack the outputs into deeper layers.
Each layer builds on the one before: gradually creating more abstract, task-specific representations.
We’ll explore this in the next part of the insight series.
But for now, it’s enough to know:
Self-attention was just the beginning.
What followed was a full architectural revolution.
7. Outlook – The Transformer Emerges
Self-attention was never just a clever trick. It was a pivot.
Once you realize that every token can look at every other (directly, selectively, in parallel) you begin to rethink the very structure of sequence modeling.
No more waiting for the past to flow into the future.
No more compressing memory into one fragile state.
With self-attention, you can build entire layers of understanding and each one revisiting the input, each one refining the context.
This is the idea behind the Transformer.
It takes self-attention, stacks it, adds layer normalization, residuals, position encodings, and feedforward blocks and turns it into an engine.
Not just for language, but for anything that can be seen as a sequence.
In the next part, we will explore exactly how this engine works:
- How the Transformer replaces recurrence with structure
- Why stacking attention layers deepens meaning
- And how this laid the foundation for the entire foundation model era
So yes, self-attention changed everything.
But it was only Act One.
Leave a Reply
You must be logged in to post a comment.