Embeddings are lists of numbers (vectors) that a model produces to represent the meaning of a piece of content, such as a sentence, a document, an image or a product. Items with similar meaning get vectors that sit close together, so software can compare them with simple arithmetic. Embeddings are how a search system knows that “cancel my subscription” and “how do I stop paying monthly” are about the same thing, even though they share no words.
How Embeddings Work
An embedding model is a neural network trained so that related inputs end up near each other in a high-dimensional space. You send it text and get back a fixed-length vector, for example 1,024 or 3,072 numbers. The individual numbers have no human meaning; only the distances between vectors do.
To compare two items, you measure the distance or angle between their vectors. OpenAI’s documentation, for example, recommends cosine similarity and notes that its embeddings are normalised to length 1, so cosine similarity and Euclidean distance give the same ranking (OpenAI embeddings guide).
Some current models let you shorten the vector to save storage. The same OpenAI guide describes a dimensions parameter and reports that a shortened 256-dimension vector from its larger model scored better on the MTEB benchmark than a full vector from its older model. Shortening is a trade-off to measure on your own data, not a free win.
Types of embeddings in common use:
- Text embeddings for search, clustering and classification.
- Multimodal embeddings that place text and images in one space, so a text query can find a photo.
- Code embeddings for searching codebases by description.
- Sparse embeddings, where most values are zero and the non-zero ones correspond to terms. They behave more like keyword search and are often combined with dense embeddings.
What Embeddings Are Used For
- Semantic search and retrieval-augmented generation (RAG): find the passages most relevant to a question.
- Classification: route support tickets or tag documents, sometimes with no labelled training data at all.
- Clustering: group customer feedback or survey answers into themes, a common step with unstructured data.
- Deduplication: find near-identical records, documents or products.
- Recommendations: suggest items similar to what a user viewed.
- Anomaly detection: spot items far from everything else.
Embeddings are stored and searched in a vector database or a regular database with vector support.
Choosing an Embedding Model
Public leaderboards such as MTEB (Massive Text Embedding Benchmark) are a starting point. The benchmark’s own authors found that no single embedding method was best across all tasks (arXiv 2210.07316), and scores on public data do not predict results on your contracts, tickets or product catalogue. What decides the choice:
- Retrieval quality on your own test questions. Build 50 to 200 real questions with known correct passages and measure how often the right passage appears in the top results.
- Languages. A model that is strong in English may be weak in Ukrainian, German or Japanese.
- Domain vocabulary. Legal, medical and engineering text can need a domain model or a fine-tuned one.
- Where it runs. Hosted APIs are simple; open-weight embedding models can run inside your own network when data cannot leave it.
- Vector size and cost. Larger vectors cost more to store and search.
Common Failure Modes
- Mixing models. Vectors from two different models, or two versions of one, cannot be compared. A model change means re-embedding everything.
- Embedding the wrong unit. A whole 40-page document squeezed into one vector loses detail; tiny fragments lose context. Chunking strategy matters as much as the model.
- Trusting similarity as relevance. “Similar” is not the same as “correct”. A refund policy for one country looks very close to the policy for another.
- Ignoring exact terms. Product codes, error numbers and names are better matched with keyword search. Most production systems combine both.
- Treating vectors as anonymous. Embeddings derived from personal data should be handled with the same care as the source data under regulations such as GDPR, including deletion when the source record is deleted.
Related Terms
Embeddings are the input to a vector database and the basis of retrieval in RAG. They are also used in context engineering to decide which pieces of information to load into a model, and in LLM evaluation to measure how close an answer is to a reference.
Embeddings in Practice
Embedding models are handled as a versioned dependency. Each vector is stored with the name and version of the model that produced it and with its original text, so the index can be rebuilt when the model changes, and vectors from two models are never mixed in one index. The model itself is chosen by measuring retrieval on real questions from the client’s users, jargon and abbreviations included, rather than by leaderboard rank.
See also: RAG and CAG knowledge assistants.