Deep Learning CourseLesson 10 of 16 27 min

Recurrent Neural Networks for Sequences

Deepak Mishra, Senior Technology Lead at Incedo

What you'll learn: how recurrent neural networks (RNNs) handle sequences, data where order matters and length varies, such as time series, text and audio. We start by training an LSTM forecaster in PyTorch, then open it up: the recurrence, backpropagation through time, and why plain RNNs forget. The questions along the way lead into eigenvalues and eigenvectors, which explain exactly when repeated matrix multiplication makes gradients vanish or explode, and into why the LSTM's additive memory fixes it.

Before you start: Lessons 1 to 9, especially matrices as functions (Lesson 5), the chain rule and vanishing gradients (Lesson 7), and skip connections (Lesson 9).

Forecast a time series

Given the last 30 readings of a noisy signal, predict the next one. We build training examples with a sliding window: each window of 30 values is an input, and the value right after it is the target. Then an LSTM reads each window one step at a time:

Python

import numpy as np
import torch
from torch import nn

rng = np.random.default_rng(0)
t = np.arange(0, 100, 0.1)
series = np.sin(t) + 0.1 * rng.normal(size=len(t))     # a noisy wave

window = 30
X = np.array([series[i:i + window] for i in range(len(series) - window)])
y = series[window:]
split = int(0.8 * len(X))                               # train on the past, test on the future
X_train = torch.tensor(X[:split, :, None], dtype=torch.float32)   # (samples, time steps, features)
y_train = torch.tensor(y[:split, None], dtype=torch.float32)
X_test = torch.tensor(X[split:, :, None], dtype=torch.float32)
y_test = torch.tensor(y[split:, None], dtype=torch.float32)

class Forecaster(nn.Module):
    def __init__(self, hidden=32):
        super().__init__()
        self.lstm = nn.LSTM(input_size=1, hidden_size=hidden, batch_first=True)
        self.head = nn.Linear(hidden, 1)

    def forward(self, x):
        outputs, _ = self.lstm(x)          # a hidden state for every time step
        return self.head(outputs[:, -1])   # predict from the last one

torch.manual_seed(0)
model = Forecaster()
optimizer = torch.optim.Adam(model.parameters(), lr=0.01)
for epoch in range(30):
    for i in torch.randperm(len(X_train)).split(32):
        loss = nn.functional.mse_loss(model(X_train[i]), y_train[i])
        optimizer.zero_grad(); loss.backward(); optimizer.step()

with torch.no_grad():
    lstm_mse = nn.functional.mse_loss(model(X_test), y_test).item()
naive_mse = nn.functional.mse_loss(X_test[:, -1], y_test).item()   # "tomorrow = today"
print(f"LSTM test MSE {lstm_mse:.4f}   naive baseline {naive_mse:.4f}   noise floor 0.0100")

Output

LSTM test MSE 0.0133   naive baseline 0.0192   noise floor 0.0100

The noise we added has variance 0.1² = 0.01, which no model can predict, so 0.01 is the best possible error. The naive forecast, which repeats the last value, scores about twice that. The LSTM closes most of the gap between the naive forecast and the floor: it has learned the shape of the wave from nothing but windows of numbers. Notice the split: for time series, never shuffle before splitting, or the model trains on the future.

What just happened

A dense network or CNN takes a fixed-size input all at once. An RNN reads a sequence one step at a time and carries a memory, the hidden state, from each step to the next. After reading 30 values, its hidden state is a summary of the whole window, and the final linear layer turns that summary into a prediction. The LSTM is a refined RNN cell; to see why it's needed, start with the plain one.

The recurrence

At each step t, a plain RNN combines the current input xₜ with its previous hidden state hₜ₋₁:

\begin{aligned} h_t &= \tanh(W_{xh}\, x_t + W_{hh}\, h_{t-1} + b) && h_0 = 0 \\[4pt] y_t &= W_{hy}\, h_t + b_y && \text{an output, if needed at this step} \end{aligned}

The same weights are used at every step, just as a CNN uses the same filter at every position. That's why one RNN handles sequences of any length with a fixed number of parameters, n_h · (n_x + n_h + 1) for the recurrent layer. Unrolled through time, an RNN is a deep network, one layer per time step, all sharing weights:

\begin{array}{ccccccccccccccc} h_0 & \longrightarrow & \boxed{\text{cell}} & \longrightarrow & h_1 & \longrightarrow & \boxed{\text{cell}} & \longrightarrow & h_2 & \longrightarrow & \boxed{\text{cell}} & \longrightarrow & h_3 & \longrightarrow & \cdots \\ & & \Big\uparrow & & & & \Big\uparrow & & & & \Big\uparrow & & & & \\ & & x_1 & & & & x_2 & & & & x_3 & & & & \end{array}

Python

def rnn_forward(xs, W_xh, W_hh, b):
    h = torch.zeros(W_hh.shape[0])
    for x in xs:                                  # one step per element, same weights each time
        h = torch.tanh(W_xh @ x + W_hh @ h + b)
    return h

torch.manual_seed(0)
rnn = nn.RNN(input_size=3, hidden_size=5)
xs = torch.randn(4, 3)                            # a sequence of 4 steps, 3 features each
ours = rnn_forward(xs, rnn.weight_ih_l0, rnn.weight_hh_l0, rnn.bias_ih_l0 + rnn.bias_hh_l0)
theirs = rnn(xs)[1].squeeze()                     # PyTorch's final hidden state
print(torch.allclose(ours, theirs, atol=1e-6))
print("parameters:", sum(p_.numel() for p_ in rnn.parameters()))   # 5*(3+5) + 2 biases of 5

Output

True
parameters: 50

(PyTorch keeps two bias vectors per layer where we use one; they simply add up.) RNNs fit several kinds of task: many-to-one (read a review, output a sentiment; read a window, output a forecast), one-to-many (generate a caption from one image), and many-to-many (translate a sentence, tag every word).

Training through time, and a memory test

To train an RNN, unroll it over the sequence and backpropagate through the unrolled graph, which is called backpropagation through time (BPTT). Because the weights are shared, each one's gradient is the sum of its contributions from every step. Here is a task that tests memory directly: each sequence starts with +1 or −1, followed by random noise, and the network must output the first value at the end. The longer the sequence, the further back it has to remember. We train a plain RNN on gaps of increasing length:

Python

def memory_task(length, n=2000, seed=0):
    g = torch.Generator().manual_seed(seed)
    first = torch.randint(0, 2, (n,), generator=g).float() * 2 - 1
    x = torch.randn(n, length, 1, generator=g)
    x[:, 0, 0] = first
    return x, (first > 0).float()

class Reader(nn.Module):
    def __init__(self, cell):
        super().__init__()
        self.rnn = cell(input_size=1, hidden_size=16, batch_first=True)
        self.head = nn.Linear(16, 1)
    def forward(self, x):
        return self.head(self.rnn(x)[0][:, -1]).squeeze(1)

def memory_accuracy(cell, length):
    torch.manual_seed(0)
    x, target = memory_task(length)
    model = Reader(cell)
    opt = torch.optim.Adam(model.parameters(), lr=0.005)
    for _ in range(400):
        loss = nn.functional.binary_cross_entropy_with_logits(model(x), target)
        opt.zero_grad(); loss.backward(); opt.step()
    x_test, t_test = memory_task(length, seed=1)
    with torch.no_grad():
        return ((model(x_test) > 0).float() == t_test).float().mean().item()

for length in [20, 100, 200]:
    print(f"plain RNN, remember across {length:3d} steps: {memory_accuracy(nn.RNN, length):.0%}")

Output

plain RNN, remember across  20 steps: 100%
plain RNN, remember across 100 steps: 100%
plain RNN, remember across 200 steps: 50%

On a task this simple, a plain RNN can remember surprisingly far: it is perfect across 20 and even 100 steps. But at 200 steps it collapses to 50%, a coin flip. It hasn't merely got worse; it can't learn the task at all, because the training signal from the last step no longer reaches the first. Why?

Why do gradients fade over time?

Question 1: backpropagating from the last step to the first, what happens to the gradient?

By the chain rule, the gradient flowing from step t back to step k is a product of one factor per step in between, and each factor contains W_hh:

\frac{\partial h_t}{\partial h_k} = \prod_{i = k+1}^{t} \operatorname{diag}\!\big(1 - h_i^2\big)\, W_{hh}

The tanh derivative, 1 − h², is at most 1, so roughly speaking the gradient gets multiplied by W_hh once per step. Over 50 steps that's W_hh⁵⁰. What does a matrix do when you multiply by it over and over? That question leads to eigenvalues.

What does repeated multiplication by a matrix do?

Question 2: what is Wⁿ x when n is large?

Lesson 5 described a matrix as a function that moves vectors, usually changing both their direction and length. But most square matrices have special directions that they don't turn at all, only stretch. A vector v in such a direction is an eigenvector, and the stretch factor λ is its eigenvalue:

W v = \lambda v

For eigenvectors, repeated multiplication is trivial: each application multiplies by λ again, so Wⁿ v = λⁿ v. If |λ| < 1, λⁿ shrinks to zero; if |λ| > 1, it blows up. And typically any vector can be written as a combination of the eigenvectors, x = c₁v₁ + c₂v₂ + …, so by linearity:

W^n x = c_1 \lambda_1^n v_1 + c_2 \lambda_2^n v_2 + \cdots

As n grows, the term with the largest |λ| dominates all the others. That largest |λ| is called the spectral radius. If it is below 1, Wⁿ x vanishes; above 1, it explodes; only at exactly 1 does it hold steady. Let's verify all three facts:

Python

W = torch.tensor([[0.9, 0.2], [0.1, 0.5]])
eigenvalues, eigenvectors = torch.linalg.eig(W)
lam, v = eigenvalues.real, eigenvectors.real
print("eigenvalues:", lam)
print("W v = lambda v:", torch.allclose(W @ v[:, 0], lam[0] * v[:, 0]))

x = torch.tensor([1.0, 1.0])
for n in [1, 10, 50]:
    Wn_x = torch.linalg.matrix_power(W, n) @ x
    print(f"n={n:2d}: |W^n x| = {Wn_x.norm():.6f}   largest |lambda|^n = {lam.abs().max() ** n:.6f}")

Output

eigenvalues: tensor([0.9449, 0.4551])
W v = lambda v: True
n= 1: |W^n x| = 1.252996   largest |lambda|^n = 0.944949
n=10: |W^n x| = 0.765896   largest |lambda|^n = 0.567653
n=50: |W^n x| = 0.079531   largest |lambda|^n = 0.058941

The length of Wⁿ x shrinks at the same rate as the largest eigenvalue raised to the n-th power: the two columns differ only by a constant factor, the size of x's component along the dominant eigenvector. For an RNN, this is the whole story of vanishing and exploding gradients: the gradient reaching a step 50 steps back is multiplied by something like W_hh⁵⁰, which vanishes when W_hh's spectral radius is below 1 (and the tanh derivatives only shrink it further) and explodes when it is above 1:

Python

def gradient_after(steps, radius, n=32):
    torch.manual_seed(0)
    W = torch.randn(n, n)
    W = W * radius / torch.linalg.eigvals(W).abs().max()     # set the spectral radius
    g = torch.ones(n)
    for _ in range(steps):
        g = W.T @ g                                           # one step of BPTT (ignoring tanh')
    return g.norm().item()

for radius in [0.8, 1.0, 1.2]:
    print(f"spectral radius {radius}: after 10 steps {gradient_after(10, radius):10.4f},"
          f"  after 50 steps {gradient_after(50, radius):14.6f}")

Output

spectral radius 0.8: after 10 steps     0.4817,  after 50 steps       0.000051
spectral radius 1.0: after 10 steps     4.4861,  after 50 steps       3.541229
spectral radius 1.2: after 10 steps    27.7765,  after 50 steps   32226.820312

Exploding gradients have a blunt but effective fix, gradient clipping: if the gradient's length exceeds a threshold c, rescale it to length c before the update (torch.nn.utils.clip_grad_norm_). Vanishing gradients need a different cell.

The LSTM

The long short-term memory cell (Hochreiter and Schmidhuber, 1997) adds a second state, the cell state cₜ, a conveyor belt of memory that runs through the sequence. Three gates, each a sigmoid producing numbers between 0 (closed) and 1 (open), control it:

\begin{aligned} \text{forget gate:}\quad & f_t = \sigma(W_f\,[h_{t-1}, x_t] + b_f) && \text{what to erase} \\ \text{input gate:}\quad & i_t = \sigma(W_i\,[h_{t-1}, x_t] + b_i) && \text{what to write} \\ \text{candidate:}\quad & \tilde c_t = \tanh(W_c\,[h_{t-1}, x_t] + b_c) && \text{what could be written} \\ \text{cell state:}\quad & c_t = f_t \odot c_{t-1} + i_t \odot \tilde c_t && \text{update the memory} \\ \text{output gate:}\quad & o_t = \sigma(W_o\,[h_{t-1}, x_t] + b_o) && \text{what to reveal} \\ \text{hidden state:}\quad & h_t = o_t \odot \tanh(c_t) && \end{aligned}

Reading a story: when a new character appears, the input gate writes "the subject is Priya" into the cell; the forget gate keeps it for paragraphs, then erases it when the subject changes; the output gate reveals it when the network has to choose between "she" and "he". An LSTM layer has four sets of weights, so 4 · n_h · (n_x + n_h + 1) parameters.

Why does the LSTM remember?

Question 3: what is different about the gradient through the cell state?

Differentiate the cell update cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ with respect to cₜ₋₁. Treating the gates as given, the derivative is just fₜ, element by element: no matrix multiplication, no squashing tanh. Over many steps the gradient becomes a product of forget gates, f₁·f₂·…·fₜ. If the network learns to keep its forget gate near 1 for information worth keeping, that product stays near 1 and the gradient survives hundreds of steps. It's the same idea as ResNet's skip connection, y = F(x) + x, where the identity carries the gradient: the cell state is an additive highway through time. We can measure it with autograd, varying the forget gate's bias:

Python

def cell_gradient(forget_bias, steps=50):
    torch.manual_seed(0)
    cell = nn.LSTMCell(input_size=1, hidden_size=8)
    with torch.no_grad():
        cell.bias_ih[8:16] = forget_bias              # PyTorch orders gates: input, forget, cell, output
        cell.bias_hh[8:16] = 0.0
    c0 = torch.zeros(1, 8, requires_grad=True)
    h, c = torch.zeros(1, 8), c0
    for x in torch.randn(steps, 1, 1):
        h, c = cell(x, (h, c))
    c.sum().backward()
    return c0.grad.abs().mean().item()

for bias in [-2.0, 0.0, 2.0, 5.0]:
    f = torch.sigmoid(torch.tensor(bias)).item()
    print(f"forget bias {bias:+.0f} (gate about {f:.2f}): gradient reaching step 0 = {cell_gradient(bias):.2e}")

Output

forget bias -2 (gate about 0.12): gradient reaching step 0 = 2.45e-34
forget bias +0 (gate about 0.50): gradient reaching step 0 = 2.70e-12
forget bias +2 (gate about 0.88): gradient reaching step 0 = 3.06e-02
forget bias +5 (gate about 0.99): gradient reaching step 0 = 1.83e+00

With the forget gate mostly closed, almost nothing reaches the start; with it open, the gradient survives 50 steps. Many implementations initialise the forget bias to 1 or more for exactly this reason.

GRU: a lighter alternative

The gated recurrent unit (Cho et al., 2014) merges cell and hidden state and uses two gates. It has about three quarters of the LSTM's parameters and often performs as well:

\begin{aligned} \text{update gate:}\quad & z_t = \sigma(W_z\,[h_{t-1}, x_t]) \\ \text{reset gate:}\quad & r_t = \sigma(W_r\,[h_{t-1}, x_t]) \\ \text{candidate:}\quad & \tilde h_t = \tanh(W_h\,[r_t \odot h_{t-1},\, x_t]) \\ \text{new state:}\quad & h_t = (1 - z_t) \odot h_{t-1} + z_t \odot \tilde h_t && \text{again additive: } z \text{ near 0 copies the past forward} \end{aligned}

In PyTorch it's a drop-in replacement: nn.GRU instead of nn.LSTM (the forecaster's forward method would take the same outputs).

Text and bidirectional RNNs

For text, each token id (Lesson 2) is turned into a vector by an embedding layer (Lesson 11), and the RNN reads the vectors in order. When the whole sequence is available up front, as with a review to classify, a bidirectional RNN runs one RNN forwards and another backwards and joins their hidden states, so every position sees context from both sides. For real-time forecasting it can't be used, since the future isn't known yet.

Python

class SentimentRNN(nn.Module):
    def __init__(self, vocab_size, embed_dim=64, hidden=64):
        super().__init__()
        self.embed = nn.Embedding(vocab_size, embed_dim)            # token id -> vector
        self.lstm = nn.LSTM(embed_dim, hidden, batch_first=True, bidirectional=True)
        self.head = nn.Linear(2 * hidden, 1)                        # forward + backward states

    def forward(self, token_ids):
        _, (h_n, _) = self.lstm(self.embed(token_ids))
        both_directions = torch.cat([h_n[0], h_n[1]], dim=1)
        return self.head(both_directions).squeeze(1)                # a positive/negative logit

model = SentimentRNN(vocab_size=10_000)
fake_batch = torch.randint(0, 10_000, (8, 200))                     # 8 reviews of 200 tokens
print(model(fake_batch).shape)                                      # torch.Size([8])

Output

torch.Size([8])

Limits of RNNs

Even LSTMs have two fundamental limits. They read one step at a time, so training can't be parallelised across a sequence, which is slow on long texts. And when one RNN encodes a sentence for another to translate, the whole sentence must squeeze through a single fixed-size hidden state. The attention mechanism was invented to remove that bottleneck, and the transformer, built from attention alone, removed both limits. RNNs remain useful for streaming, small and on-device problems, but transformers now dominate language (Lesson 12).

Summary

An RNN reads a sequence step by step, updating hₜ = tanh(W_xh xₜ + W_hh hₜ₋₁ + b) with the same weights at every step, and is trained by backpropagation through time. The gradient to an earlier step is a product of one factor per step, roughly W_hhⁿ. An eigenvector v of W satisfies Wv = λv, so Wⁿv = λⁿv, and for a general vector the largest |λ|, the spectral radius, dominates: below 1 the gradient vanishes, above 1 it explodes. Clipping tames explosions. The LSTM's cell state, cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ c̃ₜ, has derivative fₜ with respect to the previous cell, an additive highway like a skip connection, so it can remember across long gaps; the GRU does the same with fewer parameters.

Exercises

1. Find the eigenvalues and eigenvectors of [[2, 1], [1, 2]] by hand (solve det(W − λI) = 0), then check with torch.linalg.eig. What are W¹⁰ applied to each eigenvector?

2. A rotation matrix by 30° has no real eigenvectors. Why not, geometrically? What happens to the length of Rⁿx?

3. Run the memory task with nn.LSTM at 200 steps, first with default initialisation and then with the forget-gate bias set to 3 (as in the gradient experiment). Try several random seeds. How reliable is each? Then try nn.GRU.

4. Add gradient clipping (max norm 1.0) to the plain RNN on the memory task. Does it help with remembering across 50 steps? Explain why or why not.

5. In the forecaster, try windows of 5, 30 and 100 values. Which gives the lowest test error, and why might a longer window not always help?

6. Prove that if every forget gate value is at most 0.9, the gradient of c₅₀ with respect to c₀ through the cell state is at most 0.9⁵⁰. How small is that?