Deep Learning CourseLesson 2 of 16 23 min
Pretrained Text Models and the Geometry of Meaning
What you'll learn: how to use pretrained language models to judge the sentiment of a review, generate text, and search documents by meaning rather than by keywords, each in a few lines. Then we open the box: how text becomes numbers, and how meaning becomes geometry. That takes us into vectors, their length, the dot product and the angle between two vectors, built up one question at a time.
Before you start: Lesson 1, so you've seen a pretrained model, logits and softmax. A Colab notebook with a GPU is helpful but not required for this lesson.
Three things a pretrained model can do
The Hugging Face transformers library gives you thousands of pretrained models behind a single function, pipeline. Each line below downloads a model the first time it runs.
Sentiment analysis: is this text positive or negative?
from transformers import pipeline
sentiment = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")
print(sentiment([
"This course finally made backpropagation click for me.",
"The update deleted all my notes. Terrible.",
"The lecture was long, but honestly worth every minute.",
]))[{'label': 'NEGATIVE', 'score': 0.9831517934799194}, {'label': 'NEGATIVE', 'score': 0.9997929930686951}, {'label': 'POSITIVE', 'score': 0.9987037181854248}]Look closely at the first result. The model calls a clearly happy sentence NEGATIVE, with 98% confidence. It was fine-tuned on short movie-review snippets, and a sentence about a course that "finally" made something "click" is unlike anything it saw. As Lesson 1 warned, a model's confidence tells you nothing about whether the input resembles its training data. Always test a pretrained model on examples from your own domain before trusting it.
Text generation: continue a piece of text, one token at a time:
generator = pipeline("text-generation", model="gpt2")
result = generator("The best way to learn deep learning is", max_new_tokens=25,
do_sample=True, temperature=0.8)
print(result[0]["generated_text"])The best way to learn deep learning is to listen to it. Deep Learning The idea is that deep learning may be necessary for people who do not
Semantic search: find the documents that mean the same thing as a question, even when they share no words with it:
from sentence_transformers import SentenceTransformer
encoder = SentenceTransformer("all-MiniLM-L6-v2")
documents = [
"How do I reset my password?",
"Steps to recover access to your account",
"Our office is closed on public holidays.",
"The weather will be sunny tomorrow.",
"Pricing plans for teams and enterprises",
]
query = "I forgot my login credentials"
doc_vectors = encoder.encode(documents)
query_vector = encoder.encode(query)
scores = encoder.similarity(query_vector, doc_vectors)[0]
for score, doc in sorted(zip(scores.tolist(), documents), reverse=True):
print(f"{score:.2f} {doc}")0.68 How do I reset my password? 0.62 Steps to recover access to your account 0.07 Our office is closed on public holidays. 0.03 The weather will be sunny tomorrow. -0.03 Pricing plans for teams and enterprises
The two account-recovery documents come out on top, though "forgot", "login" and "credentials" appear in neither. A keyword search would have found nothing.
What just happened
Each of these models was pretrained on enormous amounts of text, learning the patterns of language: which words go together, how sentences are built, what tends to follow what. The sentiment model was then fine-tuned on movie reviews labelled positive or negative. The generator, GPT-2, was trained simply to predict the next word of web pages. The search model was trained so that sentences with similar meanings produce similar lists of numbers. As in Lesson 1, you're borrowing what someone else's model learned from data you could never collect yourself.
All three rest on the same two steps: turn text into numbers, then do arithmetic on the numbers. Let's look at both.
Text becomes tokens
A model can't read letters. A tokenizer splits text into pieces called tokens, from a fixed vocabulary, and replaces each with an id number. Common words are a single token; rare words are split into pieces, so nothing is ever completely unknown:
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
for text in ["The cat sat on the mat.", "Backpropagation is unbelievably elegant."]:
ids = tokenizer.encode(text)
print(ids)
print([tokenizer.decode([i]) for i in ids])
print("vocabulary size:", tokenizer.vocab_size)[464, 3797, 3332, 319, 262, 2603, 13] ['The', ' cat', ' sat', ' on', ' the', ' mat', '.'] [7282, 22930, 363, 341, 318, 48943, 19992, 13] ['Back', 'prop', 'ag', 'ation', ' is', ' unbelievably', ' elegant', '.'] vocabulary size: 50257
GPT-2 knows 50,257 tokens. When it generates, its output at each step is exactly what you met in Lesson 1: one logit per token in the vocabulary, turned by softmax into a probability distribution over 50,257 possible next tokens. Generating text is repeatedly sampling from that distribution:
import torch
from transformers import AutoModelForCausalLM
lm = AutoModelForCausalLM.from_pretrained("gpt2")
ids = tokenizer.encode("The capital of France is", return_tensors="pt")
with torch.no_grad():
logits = lm(ids).logits[0, -1] # scores for the token after the prompt
probs = torch.softmax(logits, dim=0)
top = probs.topk(5)
for prob, token_id in zip(top.values, top.indices):
print(f"{tokenizer.decode([token_id])!r:12} {prob.item():.1%}")' the' 8.5% ' now' 4.8% ' a' 4.6% ' France' 3.2% ' Paris' 3.2%
Paris is only fifth, behind " the", " now" and " a". GPT-2 is a small model from 2019, and it was trained to continue text plausibly, not to state facts: "The capital of France is the city of Paris" and "The capital of France is now…" are both plausible continuations. Larger, newer models put far more probability on " Paris", but they are still predicting text, which is why they can state falsehoods fluently (Lesson 13).
Meaning becomes a vector
The search model works differently. It turns a whole sentence into a fixed-length list of numbers, called an embedding:
v = encoder.encode("I forgot my login credentials")
print(v.shape) # (384,): every sentence becomes 384 numbers
print(v[:6])(384,) [-0.02265184 -0.01048861 0.00083868 0.00270868 -0.01129393 0.06997911]
Then it compared those lists, and somehow the comparison captured meaning. To see how, we need to understand what a list of numbers can be. That is where the questions start.
What is a vector?
Question 1: why call a list of 384 numbers a vector, and what can you do with one?
A vector is an ordered list of numbers, and the powerful idea is to treat it as a point, or an arrow from the origin to that point, in a space with one axis per number. The vector (3, 4) is the point 3 along the x-axis and 4 up the y-axis. A 384-number embedding is a point in a 384-dimensional space. Nobody can picture 384 dimensions, but every rule that works in 2 or 3 dimensions carries over unchanged, so we can reason in 2D and trust the maths in 384D.
Two operations make vectors useful. Adding vectors adds them component by component, which geometrically means placing the arrows head to tail. Multiplying a vector by a number (a scalar) scales every component, which stretches or shrinks the arrow, or flips it if the number is negative:
import numpy as np
a, b = np.array([3.0, 4.0]), np.array([1.0, -2.0])
print(a + b, 2 * a, -a)[4. 2.] [6. 8.] [-3. -4.]
How long is a vector?
Question 2: how do we measure the size of a vector, in any number of dimensions?
In 2D, the arrow (3, 4) is the long side of a right-angled triangle with sides 3 and 4, so by Pythagoras its length is √(3² + 4²) = 5. In 3D, apply Pythagoras twice: first across the floor, then up the wall. The floor diagonal of (x, y, z) has length √(x² + y²), and that diagonal and the height z form another right angle, so the full length is √(x² + y² + z²). Each extra dimension adds one more square under the root. The length, called the norm, is:
The distance between two points is the length of the arrow from one to the other, ‖a − b‖. A vector of length 1 is a unit vector, and dividing any vector by its length produces one: v / ‖v‖ points the same way but has length exactly 1.
print(np.linalg.norm(a)) # 5.0
print(np.sqrt(np.sum(a ** 2))) # the same, from the formula
u = a / np.linalg.norm(a)
print(u, np.linalg.norm(u)) # a unit vector in a's direction
print(np.linalg.norm(v)) # the embedding's length5.0 5.0 [0.6 0.8] 1.0 0.99999994
How do we compare two vectors?
Question 3: what single number measures how similar two vectors are?
The workhorse is the dot product: multiply matching components and add them up.
This looks like bookkeeping, but it has a remarkable geometric meaning. It equals the product of the two lengths times the cosine of the angle θ between the arrows:
Why on earth should multiplying components and adding them give you an angle? That deserves its own question.
Why does the dot product measure the angle?
Question 4: how can a sum of products be connected to an angle?
Draw a triangle with sides a, b and the third side a − b, joining their tips. The law of cosines, a generalisation of Pythagoras to triangles without a right angle, says:
(When θ = 90°, cos θ = 0 and this is exactly Pythagoras.) Now compute the same left-hand side with components instead, expanding each square (aᵢ − bᵢ)² = aᵢ² − 2aᵢbᵢ + bᵢ²:
Both lines describe the same length, so their right-hand sides must be equal. The ‖a‖² and ‖b‖² cancel, and what remains is a · b = ‖a‖ ‖b‖ cos θ. The algebra (multiply and add) and the geometry (lengths and angles) are two descriptions of one thing.
And the law of cosines itself? Drop a perpendicular from the tip of b onto a. It splits the triangle into two right-angled triangles; apply Pythagoras to each, and the cos θ appears as the length of b's shadow along a divided by ‖b‖. It is Pythagoras all the way down. We can check the result numerically by building vectors with known angles:
for degrees in [0, 45, 90, 135, 180]:
theta = np.radians(degrees)
p1 = np.array([2.0, 0.0]) # length 2, along x
p2 = 3 * np.array([np.cos(theta), np.sin(theta)]) # length 3, at the angle
print(f"{degrees:3d} deg: dot = {p1 @ p2:6.2f} |a||b|cos = {2 * 3 * np.cos(theta):6.2f}")0 deg: dot = 6.00 |a||b|cos = 6.00 45 deg: dot = 4.24 |a||b|cos = 4.24 90 deg: dot = 0.00 |a||b|cos = 0.00 135 deg: dot = -4.24 |a||b|cos = -4.24 180 deg: dot = -6.00 |a||b|cos = -6.00
Why compare by angle rather than distance?
Question 5: semantic search scored documents with cosine similarity. Why the angle, not the distance between points?
Rearranging the dot product formula isolates the angle:
A vector carries two pieces of information: its direction and its length. In embeddings the direction carries the meaning, while the length tends to reflect incidental things, such as how long or how common the text is. Cosine similarity ignores length entirely: scaling either vector doesn't change the angle. That's exactly what we want when we ask "do these mean the same thing?"
Can cosine similarity ever escape the range −1 to 1? No, and the reason is the Cauchy-Schwarz inequality, |a · b| ≤ ‖a‖ ‖b‖, which the gradient descent ladder in Lesson 7 uses as well. Many embedding models normalise their outputs to length 1, and then the dot product is the cosine similarity, with no division needed:
def cosine(x, y):
return (x @ y) / (np.linalg.norm(x) * np.linalg.norm(y))
q = encoder.encode(query)
d = encoder.encode(documents)
print(cosine(q, d[1]), cosine(10 * q, d[1])) # scaling doesn't change it
print([round(float(cosine(q, doc)), 2) for doc in d]) # the same scores as the search above
print(np.linalg.norm(d, axis=1).round(3)) # this model's vectors have length 10.6233954 0.6233954 [0.68, 0.62, 0.07, 0.03, -0.03] [1. 1. 1. 1. 1.]
What does a direction mean?
Question 6: if direction carries meaning, can a single direction stand for a concept?
Take a unit vector u. For any vector a, the dot product a · u = ‖a‖ cos θ is the length of a's shadow on u: its projection, how far a extends in u's direction. If u pointed in a "positive sentiment" direction, a · u would measure how positive a sentence is. Remarkably, trained embeddings do contain such directions, and we can find a rough one just by subtracting the embeddings of two contrasting sentences:
positive = encoder.encode("I absolutely love this, it is wonderful.")
negative = encoder.encode("I absolutely hate this, it is awful.")
direction = (positive - negative) / np.linalg.norm(positive - negative) # unit vector
for sentence in ["What a fantastic day!", "This is the worst service ever.",
"The meeting is at 3 pm.", "I'm quite pleased with the result."]:
print(f"{encoder.encode(sentence) @ direction:+.2f} {sentence}")+0.20 What a fantastic day! -0.33 This is the worst service ever. -0.01 The meeting is at 3 pm. +0.09 I'm quite pleased with the result.
Sentences project positively or negatively along this direction according to their sentiment, and a neutral scheduling sentence lands in between. Directions as concepts is one of the deepest ideas in modern AI, and it returns in Lesson 11 with word analogies such as king − man + woman ≈ queen.
Comparing one query with a million documents
Question 7: a search engine compares a query with millions of document vectors. Is that a million separate dot products?
Yes, but written as one operation. Stack the document vectors as the rows of a matrix D. Multiplying the matrix by the query vector computes the dot product of every row with the query in a single step, giving a vector of scores:
scores_by_hand = np.array([doc @ q for doc in d])
scores_matrix = d @ q # one matrix-vector product
print(np.allclose(scores_by_hand, scores_matrix))
print(documents[int(np.argmax(scores_matrix))])True How do I reset my password?
GPUs are built to do exactly this kind of operation at enormous speed. In fact, almost everything a neural network does is matrix multiplication, as Lesson 5 shows.
Summary
Pretrained models handle sentiment, generation and semantic search in a few lines. Text is split into tokens with ids; a generator outputs a softmax distribution over its whole vocabulary at each step, and an embedding model turns a sentence into a vector. Vectors are points or arrows in many dimensions. Their length comes from Pythagoras, ‖v‖ = √Σvᵢ², and the dot product Σaᵢbᵢ equals ‖a‖‖b‖cos θ, which follows from the law of cosines. Cosine similarity compares directions and ignores length, a dot product with a unit vector measures a projection, which can capture a concept, and a matrix-vector product computes many dot products at once.
Exercises
1. Run the sentiment model on ten sentences of your own, including sarcasm ("Oh great, another Monday"). Where does it fail, and why might it?
2. Show, using the dot product formula, that two non-zero vectors are perpendicular exactly when their dot product is 0. Then find a vector perpendicular to (3, 4) and check it.
3. Prove that the triangle inequality ‖a + b‖ ≤ ‖a‖ + ‖b‖ follows from the Cauchy-Schwarz inequality. (Hint: expand ‖a + b‖² as a dot product.)
4. Add twenty documents of your own to the semantic search, about a topic you know well, and write five queries that share no keywords with the right answers. How often does the top result match?
5. Generate 384-dimensional random vectors with np.random.randn and compute the cosine similarity of many random pairs. What values do you get, and why are random high-dimensional vectors almost always nearly perpendicular?
6. Find a "formal versus casual" direction the same way we found the sentiment direction, and rank a few sentences by their projection on it.
Comments