Embeddings: How AI Search Turns Your Page Into 384 Numbers

A text embedding is a list of numbers that encodes meaning. Two texts that mean similar things produce lists pointing in similar directions - which is how a machine compares meaning without matching a single word.

That one sentence sits behind every "AI search doesn't care about keywords" claim you have read this year. It is also almost never demonstrated. Every widget below runs in your browser: first on a hand-built eight-dimension model you can read with your eyes, then on a real 384-dimension sentence transformer, on your own text. The model is 22 MB and downloads only if you ask for it.

One of the sections below was written to say the opposite of what it says now. I ran the experiment before writing the prose, the numbers disagreed, and the section changed. That one is §5.

What is a text embedding?#

An embedding is a fixed-length list of numbers a model produces for a piece of text. The model is trained so that texts used in similar contexts get similar lists. Nothing about the words themselves survives - only their position in a shared space.

The useful mental image is a map of meaning. Words that get used the same way end up near each other, and "near" is something you can compute. Below is a deliberately tiny version: eight dimensions, each one named, so you can read the numbers instead of taking them on faith. The whole pipeline is one instrument - text in at panel 01, neighbours out at panel 05 - so you can watch a sentence become numbers without any step happening off screen. Type into it, click a token, and see what lives nearby.

toy_embed("cat")  // eight named dimensions

Two details in that toy are worth naming, because both are true of real models too. Function words like the and of carry no semantic weight, so every one of their numbers is zero - and a list of zeros points nowhere, which is why they pile up in the centre with no nearest anything. And a word the vocabulary has never seen is not rejected: catfish is rebuilt out of cat and fish. Hold on to that second one; §6 is about what a real tokenizer does with it.

What do the numbers in an embedding actually mean?#

In a real model, individual dimensions have no human-readable name. Meaning lives in the direction of the whole vector, not in any single number. The eight-dimension model above is a teaching lie - a useful one, because you can read it.

Nobody sat down and decided that dimension 4 of all-MiniLM-L6-v2 means "animal". The model learned 384 numbers per text by being shown a billion sentence pairs and nudged until similar sentences landed near each other. What each individual number does is distributed across all of them; researchers hunting for interpretable structure look for directions, not axes.

The map in panel 04 is a fixed projection of eight numbers down to two, so a point's position is computed from its vector and nothing else. Pick a token and drag one of its dimensions under panel 03: the point has to move, and the neighbours in panel 05 have to change with it. That is the only reason this demo means anything - a map with hand-placed coordinates could show you whatever its author believed.

Try it on milk. Push animal from 0.30 to 1.00 and it drifts out of the food cluster toward the animals. The word did not change; one number did.

What changes when you switch to a real model?#

Only the resolution. all-MiniLM-L6-v2 returns 384 numbers instead of 8, none of them nameable, and it embeds whole sentences rather than single words. The behaviour you just saw - similar meaning, similar direction - is identical.

Clicking below downloads the model into your browser once and caches it. Everything after that is local: no upload, no API key, no request leaving the tab. That is not a privacy flourish, it is just what these models now cost - 22 MB quantised, small enough that a web page can carry one.

embed("The cat drinks milk")

Those five numbers are not illustrative. They were produced by the same model file this page serves, and the widget reproduces them exactly - I have a test that fails if it ever stops doing so. Prefer a bigger canvas? The embedding playground does the same thing with more room.

How does a machine measure "similar meaning"?#

Cosine similarity measures the angle between two vectors, not the distance between them. It returns 1.0 for identical direction and 0 for unrelated, and it ignores length - which is why a nine-word sentence and a 177-word passage on the same subject can score in the same range.

The mistake worth unlearning: a score of 0.6 is not "60% similar". Real pairs of related sentences mostly live between 0.3 and 0.7, unrelated ones sit near 0.1, and the ceiling is reached only by near-identical text. Read the scale by comparison, never as a percentage.

similarity("How do I reset my password?", "I forgot my login")

One thing I did not expect to find: the score depends slightly on what else you embed at the same time. That same pair scores 0.3447 alone, 0.3638 when embedded together in one batch, and 0.3515 when a longer text joins the batch. Batching pads every sequence to the longest one present, and under 8-bit quantisation that shifts the result by about 0.02 - larger than differences this article treats as meaningful elsewhere. If you are benchmarking retrieval, embed one text per call, or at least keep your batching identical between runs.

Does repeating a keyword change the vector?#

Yes - upward. Repeating a keyword raises cosine similarity to a matching query, it does not lower it. A passage that repeats "vector database" eight times and answers nothing scores 0.420. A passage that answers the question without ever using the phrase scores 0.345.

This is the section I planned to write the other way round. The comfortable version of this claim - that stuffing pushes your vector away from the query - is repeated all over the AEO field, and it is wrong for this model. I tried three constructions to see whether the comfortable version could be rescued:

• Appending the phrase to a passage that already answers: 0.393 at one repetition, 0.472 at twelve. Up.
• Displacing the content, replacing real sentences with the phrase while holding length roughly constant: 0.465 for the clean answer, peaking at 0.629 when five of six sentences are gone. Only total degeneration, all six replaced, drops it to 0.410.
• Head to head, below. Up again.

query: "which database should I use for storing embeddings?"

So why does stuffing still not work? Because retrieval is the first step, not the last. The winning passage gets handed to a model that has to write an answer and cite something. A passage that won on repetition has nothing in it to quote. Stuffing buys you the shortlist and loses you the citation - which is a much more precise claim than "the vector moves away", and unlike that one, it survives contact with the numbers.

Two honest caveats. This is all-MiniLM-L6-v2, a small model, the one this page can afford to ship to your browser; larger retrievers and cross-encoder rerankers behave differently. And production retrieval is rarely cosine alone - hybrid scoring and rerankers exist precisely because this step is manipulable.

Related: the 3-word tag that hides your best pages from AI covers a failure that happens before any of this scoring runs.

What happens to words the model has never seen?#

Nothing breaks. The tokenizer splits an unknown word into sub-word pieces it does know, and the embedding is assembled from those pieces - which is why brand names, typos and new jargon still land somewhere sensible instead of failing.

This is the same trick the toy model played in §1, done properly. It also explains why consistency in naming pays: a product name split the same way every time accumulates a stable position, while three spellings of it scatter into three.

tokenize("alexturik")

The pieces are English pieces, and that has a price worth stating plainly: this model was trained on English. cat and кіт mean exactly the same thing and score 0.215, while cat and dog - different animals, same language - score 0.659. Meaning is not language-independent to a model that never saw the language. Multilingual checkpoints exist; they are an order of magnitude larger, which is why this page does not ship one.

Why does AI cite a section instead of the whole page?#

Because retrieval runs on chunks. Your page is split into passages, each passage gets its own vector, and only the best-matching passage is compared with the query. The page as a whole is never scored - so a strong section inside a weak page still wins.

The consequence for anything you write: every heading has to stand on its own, because it will be read on its own. "As we saw above" is invisible to a chunk that arrives without the above. This is also why documentation outranks blog posts in AI answers so often - it is already written in self-contained units.

query: "how is similarity measured"

Related: recency and authority, two schema fields that decide AI trust and how I added llms.txt to my Next.js blog both deal with what happens after your chunk is retrieved.

Running this yourself#

// npm i @huggingface/transformers
import { pipeline } from '@huggingface/transformers';
 
const extractor = await pipeline(
  'feature-extraction',
  'Xenova/all-MiniLM-L6-v2',
  { dtype: 'q8' }, // ~22 MB quantised
);
 
const out = await extractor('The cat drinks milk', {
  pooling: 'mean',
  normalize: true,
});
 
console.log(out.dims); // [1, 384]
console.log(out.tolist()[0]); // [0.0494, -0.0165, …]

With normalize: true the vectors come back unit length, which makes cosine similarity just a dot product - do not divide by the norms again:

const similarity = (a, b) => a.reduce((sum, x, i) => sum + x * b[i], 0);

Reference: Transformers.js docs, the all-MiniLM-L6-v2 model card, and MDN on Web Workers - run inference off the main thread or your page will freeze while it thinks.

Takeaways#

• Meaning is a direction, not a keyword. Similar texts point the same way; no word has to match.
• Dimensions have no names. Anything that tells you dimension 12 means "sentiment" is simplifying.
• Cosine ignores length. A nine-word sentence and a 177-word passage compete on equal terms.
• Repetition wins retrieval and loses the citation. Stuffing raises the similarity score. It just leaves nothing worth quoting.
• Retrieval is chunk-level. Every section must stand alone, because it will be read alone.
• Batching changes your numbers. Embed one text per call when the number matters.