The Thinking Machine - Part 1

The Memory Problem: Why RNNs Weren’t Enough

1. Introduction: What This Series Is About

When we talk about Artificial Intelligence, we often imagine machines that think.
But what does that really mean?

At the core of human thought lies context. This is our ability to remember, to focus, to relate what’s happening now to what came before. Language, reasoning, even emotions all rely on some kind of structured memory. And for decades, AI systems tried to mimic that: first crudely, then elegantly, then, one might say, desperately.

This series, The Thinking Machine, is about that journey.
A journey from simple sequence models to the architectures that power today’s large language models. It’s about the struggle to model thought, and the breakthroughs that led machines not to “think” in the human sense, but to approximate thought well enough to change the world.

But let’s be clear: this is not just another technical breakdown of the Transformer.
It’s a structured, human-readable narrative that explains:

Why the need for contextual understanding drove model evolution
How Attention shifted the paradigm
What makes Transformer architectures so powerful and where they might fall short

We’ll blend intuition, mathematics, and a bit of history, so that you can follow the evolution of ideas from early Recurrent Neural Networks to the architectural sophistication of GPT, BERT, and beyond.

📚 The Structure of the Series

Here’s what to expect in the coming chapters:

The Memory Problem: Why RNNs weren’t enough
Focus, Not Memory: The rise of Attention
Self-Attention: A network that talks to itself
Transformers: The architecture that changed everything
Variants: GPT, BERT, T5 and what makes them different
Position Matters: How Transformers learn order
What’s Next: Scaling, sparsity, memory and the future of thinking machines

By the end of this series, you’ll not only understand what a Transformer is. You’ll also understand why it had to be invented.

Let’s begin where the story starts:
With a machine that wanted to think but had trouble remembering.

2. The Early Days – Recurrent Neural Networks

Before Transformers and massive language models, the dominant approach to modeling sequences was built on a simple yet powerful idea:
The present depends on the past.

In natural language, every word we speak is shaped by the words that came before.
Say the sentence:

“I went to the bank to withdraw some…”
The next word probably isn’t “fish.”

We understand context. And in the early days of deep learning, Recurrent Neural Networks (RNNs) were designed to give machines a similar ability: to process information step by step, keeping track of what came before.

🧠 The Intuition Behind RNNs

At its heart, an RNN is a loop. Instead of processing all input at once, like a standard feedforward network, it moves through a sequence one element at a time. It maintains a hidden state, a kind of short-term memory, which is updated as new input arrives.

Mathematically, it’s often described as: \(h_t = \tanh(W_h h_{t-1} + W_x x_t + b)\)

Here:

\(x_t\) is the input at time step \(t\)
\(h_{t-1}\) is the previous hidden state
\(h_t\) is the new hidden state
\(W_h, W_x, b\) are trainable weights and bias

This recurrent structure allows information to “flow” from one time step to the next. In theory, the network can learn dependencies across time, like associating the word “bank” with “withdraw” several steps later.

📉 A Promise That Didn’t Hold Up

For a while, RNNs were promising. They powered early speech recognition systems, basic translation models, even music generation.

But then reality set in.

In practice, RNNs struggled with anything that required remembering something far in the past. The gradient signal during training would either vanish (fade to zero) or explode (grow uncontrollably) as it passed through many time steps. This made it nearly impossible to learn long-term dependencies reliably. A problem known as the vanishing/exploding gradient problem.

In other words:
RNNs were trying to think, but their memory kept failing them.

3. The Fundamental Weaknesses of RNNs

At first glance, Recurrent Neural Networks (RNNs) seem like a clever solution.
They allow a model to “remember” what it has seen before by carrying context forward one step at a time. But in practice, this approach runs into three major problems, each of which severely limits the ability of RNNs to model real-world language and sequences.

Let’s walk through them.

🚨 Problem 1: Vanishing and Exploding Gradients

When training deep neural networks, gradients, the signals used to update weights, must be propagated backward through time. But in RNNs, this backpropagation happens through many time steps, which makes things unstable.

If the gradients shrink slightly at each step, they vanish, and early layers learn nothing.
If the gradients grow slightly at each step, they explode, and the model becomes unstable.

This problem gets worse the longer the sequence is. Which is exactly the opposite of what we want for language, where context often spans many words or sentences.

🧠 Analogy: Imagine trying to recall the beginning of a conversation but your memory gets blurrier with every passing second, until you remember nothing.

🧱 Problem 2: Weak Long-Term Memory

Even if we solve the gradient issue (with tricks like gradient clipping), RNNs still struggle to maintain meaningful long-term dependencies. The hidden state \(h_t\) is constantly overwritten as new input arrives. It’s like having a single sticky note for memory, and rewriting it every time someone says something new.

This is especially bad in language tasks. Consider the sentence:

“The car that I bought yesterday , after much consideration, and despite all the warnings, is red.”

To understand “is red”, the model must remember “car”, skipping over an entire clause.
Standard RNNs usually fail here.

🔍 Problem 3: No Explicit Access to Past Inputs

Unlike architectures with memory buffers or external attention, RNNs can’t directly “look back” at previous tokens. All past information is compressed into the hidden state. If it wasn’t encoded well enough when it arrived, it’s lost forever.

This creates a bottleneck:
The hidden state must carry everything relevant from the past in a single vector (!).

In other words:

There’s no mechanism for selective recall.
No way to choose what to focus on.
Just a stream of updates, one after another.

🧩 Summary

Problem	Consequence
Vanishing gradients	Model stops learning over long sequences
Overwritten hidden states	Poor long-term memory
No selective attention	Lacks fine-grained context control

These issues aren’t just technical quirks. They define the limitations of RNNs as thinking systems. They can react to what just happened, but struggle to reflect on what happened before.

4. First Fixes – LSTM & GRU: Patching the Memory

Faced with the shortcomings of vanilla RNNs, researchers didn’t give up on the idea of sequential memory.

Instead, they asked a crucial question:

What if the network could decide what to remember and what to forget?

That’s the motivation behind two of the most influential architectures in deep learning history:

LSTM: Long Short-Term Memory (1997)
GRU: Gated Recurrent Unit (2014)

These models introduced the concept of gated memory by allowing the network to actively manage its internal state like a kind of rudimentary working memory.

🧠 The Core Idea: Gating Mechanisms

In a standard RNN, the hidden state \(h_t\) is updated blindly at every time step.
LSTM and GRU introduce gates: small neural networks that learn to control the flow of information.

They answer questions like:

Should we keep this information around?
Should we overwrite the current memory?
Should we use this memory to produce an output?

These gates are differentiable (i.e. trainable) and are learned alongside the rest of the model.

🧬 LSTM: Long Short-Term Memory

LSTM splits the memory into two parts:

A cell state \(c_t\): the long-term memory
A hidden state \(h_t\): the short-term memory used for output

It introduces three gates:

Forget Gate – decides what to discard from the cell state
Input Gate – decides what to write to the cell state
Output Gate – decides what to pass on to the next layer or time step

The resulting mechanism allows selective memory retention, which helps preserve information over long sequences.

🧠 Metaphor: It’s like a notepad with a checklist:

Keep this ✓
Erase that ✗
Write this new thing ✎

⚙️ GRU: Gated Recurrent Unit

The GRU simplifies the LSTM by merging the cell and hidden states, and using only two gates:

Update Gate: controls how much of the previous memory to keep
Reset Gate: controls how much of the past to forget when incorporating new input

Despite being simpler, GRUs perform comparably to LSTMs on many tasks and are faster to train.

🔬 Did They Solve the Problem?

Partially.
LSTM and GRU significantly improved the ability of RNNs to:

Handle longer dependencies
Learn more stable gradients
Perform well on language, speech, and time-series tasks

But two key limitations remained:

They were still sequential.
Each step depends on the one before it.
→ No parallelism = slow training, hard to scale
They still compressed context into a hidden state.
→ No explicit mechanism to look back at specific words

This meant RNNs, even with smarter memory, were fundamentally blind to structure.
They could process sequences, but not model relationships across them.

💡 The Bigger Realization

At some point, researchers began to wonder:

Maybe memory isn’t the answer.
Maybe instead of remembering everything…
…a model should learn to focus on the right things.

This is where the story turns and where everything changes.

5. The Three Problems That Led to Transformers

Smarter RNNs like LSTM and GRU gave us better memory.
But they still didn’t give us understanding.

The truth was becoming clear:
Even with gating mechanisms, these architectures inherently struggled with three fundamental challenges:

⚠️ Problem 1: Sequential Bottleneck

RNNs process one time step at a time.
This means that:

You can’t parallelize the sequence during training
You can’t jump around in the input
You have to wait for previous steps before continuing

This bottleneck isn’t just inconvenient. It’s a dealbreaker for modern scale. Training long documents or real-time conversations becomes painfully slow.

💬 Analogy: Imagine reading a book where you’re only allowed to see one word at a time.

🔒 Problem 2: Hidden-State Compression

Even LSTM and GRU compress all past information into a fixed-size hidden state.
This state acts as a bottleneck, which can’t represent all relevant details, especially in long sequences.

“The cat, which my friend adopted after finding it behind a gas station in a snowstorm three years ago, is black.”

To understand “is black”, you need to link it all the way back to “cat” , while ignoring irrelevant details like “snowstorm”.
That kind of selective linkage is nearly impossible with compressed memory.

🧠 RNNs have no way to explicitly retrieve the “cat” token. It’s either encoded or lost.

👁️‍🗨️ Problem 3: No Structured Access to Context

Humans don’t process language by compressing everything into a single number.
We focus. We attend to specific parts of what we’ve read or heard.

RNNs can’t do that. They treat all history equally unless they’ve learned to smuggle in a memory trick. There’s no built-in way to choose which past word is most relevant to the current one.

In other words:
RNNs try to remember everything, but fail to focus on what matters.

💣 The Consequences

These problems had been tolerated for years.
But as datasets grew, and tasks became more complex (translation, summarization, dialogue), the cracks widened.

Researchers needed a new paradigm.
One that allowed:

Parallel processing
Explicit access to all parts of the sequence
Fine-grained, trainable control over what to focus on

The answer came in 2017, with a now-legendary paper titled:

“Attention Is All You Need”

And it wasn’t just a catchy title.

It was a revolution.

6. Outlook: The Idea of Attention

So far, we’ve seen how RNNs, even in their improved forms, struggled with context, parallelism, and relevance.
They tried to remember, but failed to focus.

In the next part of this series, we’ll meet the concept that changed everything:
👉 Attention.

You’ll learn:

Why memory isn’t always the answer
How attention allows models to focus on what matters — dynamically
And how this seemingly simple idea laid the foundation for the Transformer revolution

Up to now, our models have processed language like someone squinting through a keyhole.

Next, we’ll give them a wide-angle lens.

The Thinking Machine – Part 1