RNNs, LSTMs & the Attention Mechanism

The sequence problem#

Text, speech, stock prices and sensor readings are sequences: order matters, and lengths vary. "The dog bit the man" and "The man bit the dog" use the same words but mean different things. A network for sequences needs to remember what came earlier.

This lesson follows the history that led to today's LLMs: RNN → LSTM → attention → transformer.

Recurrent Neural Networks (RNNs)#

An RNN reads a sequence one step at a time and carries a hidden state — a running memory — from step to step:

h_t = tanh(W_x · x_t + W_h · h_(t-1) + b)

At every step it combines the current input x_t with the previous memory h_(t-1). The same weights are reused at every step, so an RNN can process sequences of any length.

"I"  →[RNN]→ h1 →[RNN]→ h2 →[RNN]→ h3 → prediction
      ↑             ↑             ↑
     "I"          "love"       "Python"

Where RNNs struggle#

1. Vanishing gradients — during backpropagation through many time steps, gradients shrink until early words barely affect learning. In "The students who studied in the library all semester were tired", the model must remember "students" (plural) across many words.
2. No parallelism — step 100 must wait for step 99, so training on long sequences is slow, even on GPUs.

LSTMs: memory with gates#

Long Short-Term Memory (LSTM) networks add a separate cell state — a memory highway — controlled by three gates:

Because information can flow along the cell state with little change, gradients survive far longer. LSTMs (and the simpler GRU) powered machine translation, speech recognition and keyboard prediction through the 2010s.

The bottleneck in sequence-to-sequence models#

Early translation models used an encoder RNN to read the whole source sentence into one fixed-size vector, and a decoder RNN to generate the translation from it. Squeezing a 50-word sentence into one vector loses information — quality dropped sharply for long sentences.

Attention: look back at everything#

Attention removed the bottleneck. When generating each output word, the decoder looks at all encoder states and decides how much attention to pay to each one.

Translating "Main kitab parh raha hoon" to "I am reading a book": when producing "book", attention focuses mostly on "kitab".

The core computation:

1. Compare the current query (what I'm looking for) with every key (what each position offers) to get scores.
2. Turn scores into weights with softmax (they sum to 1).
3. Take the weighted sum of the values (the information at each position).

Attention(Q, K, V) = softmax(Q · Kᵀ / √d) · V

The √d scaling keeps scores in a stable range as vectors get larger.

Self-attention and the transformer#

The 2017 paper "Attention Is All You Need" asked: what if we drop recurrence entirely and use attention for everything?

• Self-attention lets every word attend to every other word in the same sentence, building context-aware representations in one step.
• Multi-head attention runs several attention operations in parallel, each learning different relationships (grammar, references, meaning).
• Positional encodings add word-order information, since attention alone ignores order.
• Everything is parallel — all positions are processed at once, making training on GPUs dramatically faster.

This transformer architecture is the foundation of GPT-style LLMs, BERT, vision transformers and most of modern AI.

RNN vs LSTM vs Transformer#

Where RNNs still appear#

Small on-device models, streaming signals processed one step at a time, and some time-series forecasting. Newer state-space models revisit recurrent ideas to handle very long sequences efficiently — so the concepts in this lesson remain relevant.

Common mistakes#

• Thinking LSTMs are obsolete knowledge. Interviewers still ask about them because they explain why attention was needed.
• Confusing attention with memory. Attention only sees what's inside the current context window.
• Forgetting positional information. Without positional encodings, a transformer treats a sentence like a bag of words.

Key takeaways#

• RNNs carry a hidden state through a sequence but suffer from vanishing gradients and slow, sequential training.
• LSTMs add gated memory to keep information over longer spans.
• Attention lets a model look at every position and weight what's relevant.
• Transformers use self-attention everywhere — parallel, scalable, and the basis of LLMs.