Definition
Latent space is the high-dimensional mathematical space in which embeddings are organized. It is a learned, continuous representation space where the geometric positions of vectors encode meaning — similar concepts cluster together, and relationships between concepts are reflected by distances and directions.
The Word "Latent"
"Latent" means hidden or underlying. The latent space is not directly observed — it is the model's internal compressed representation of the world, learned entirely from data patterns.
Structure of Latent Space
- Every token, word, sentence, or concept maps to a point (vector) in this space
- The space has hundreds to thousands of dimensions (e.g., 4096 for LLaMA-3 7B)
- Directions in this space are meaningful:
- Early layers: syntactic structure (POS, morphology)
- Middle layers: semantic structure (entity types, coreference)
- Late layers: task-specific, output-oriented representations
- The final layer hidden states feed into the output (LM) head
- Encode query + documents into latent space
- Use cosine similarity to find nearest neighbors
- Powers RAG, recommendation engines, deduplication
- t-SNE and UMAP project high-dim latent space → 2D/3D
- Reveals clusters, outliers, and structure in data
- Train small classifiers on hidden states to discover what information each layer encodes
- Example: Does layer 12 know whether a word is a proper noun? Probe it.
- Interpolating between two points in latent space can generate smooth transitions between concepts (common in image generation, less so in text)
- Identify directions in latent space corresponding to behaviors (e.g., "sycophancy", "refusal")
- Activate or suppress behavior by adding/subtracting these vectors at inference time (activation steering / representation engineering)
- RAG quality: how well your retrieval works depends entirely on the quality of the embedding model's latent space
- Fine-tuning: fine-tuning shifts the latent space to accommodate new task structure
- Interpretability: understanding the latent space is central to mechanistic interpretability research
- Vector DBs: storing and querying vectors is storing and querying latent space points
- Embeddings, Attention, Hidden States, RAG, Semantic Search, Fine-Tuning, Activation Steering
- There may be a "gender" direction: king - man + woman ≈ queen
- A "capital city" direction: France - Paris + Berlin ≈ Germany
- A "sentiment" direction: positive sentiment tokens cluster in one region
How It's Built
1. During pre-training, the model adjusts its embedding matrix and attention weights to minimize next-token prediction loss
2. Tokens that appear in similar contexts get pulled toward each other in the space
3. Over billions of training steps, the space organizes itself to reflect the statistical structure of language
Geometric Intuitions
| Geometric Property | Semantic Meaning |
|-------------------|-----------------|
| Small distance (cosine similarity ≈ 1) | Semantically similar |
| Large distance (cosine similarity ≈ 0) | Semantically unrelated |
| Vector arithmetic | Analogical relationships |
| Clusters | Conceptual categories (animals, countries, verbs...) |
| Manifolds | Underlying structure of a concept domain |
Layers of Latent Space in a Transformer
A Transformer has multiple layers, and each produces its own latent representations (called hidden states):
Latent Space vs. Embedding Space
These terms are often used interchangeably but have a subtle distinction:
| Term | Meaning |
|------|---------|
| Embedding Space | The input embedding lookup table (static-ish) |
| Latent Space | The full internal representational space after all transformer layers |
| Hidden State | A vector in the latent space at a specific layer |