Ensiklopedia VibeKoding: Principles of Embeddings and Vector Retrieval.Ensiklopedia VibeKoding: Principles of Embeddings and Vector Retrieval.
How does a computer understand that "cats and dogs are similar, but neither is similar to a car"? To humans this is common sense, but to a computer, "cat", "dog", and "car" are just three unrelated strings. Embedding technology is the key to solving this problem — it turns text into numerical vectors, allowing computers to understand semantic "closeness and distance."How does a computer understand that "cats and dogs are similar, but neither is similar to a car"? To humans this is common sense, but to a computer, "cat", "dog", and "car" are just three unrelated strings. Embedding technology is the key to solving this problem — it turns text into numerical vectors, allowing computers to understand semantic "closeness and distance."
What will you learn in this article?What will you learn in this article?
After completing this chapter, you will gain:After completing this chapter, you will gain:
| Chapter | Content | Core Concepts |
|---|---|---|
| Chapter 1 | Embedding Concepts | Semantic space, vector representation |
| Chapter 2 | Similarity Calculation | Cosine similarity, Euclidean distance |
| Chapter 3 | Vector Indexing | Brute-force search vs. ANN |
| Chapter 4 | Vector Databases | Pinecone, Milvus, Chroma |
| Chapter 5 | End-to-End Pipeline | Text → Vector → Storage → Query |
------
In the world of natural language processing, there is a fundamental challenge: computers only understand numbers, not text.In the world of natural language processing, there is a fundamental challenge: computers only understand numbers, not text.
Early approaches assigned each word a numeric ID (One-Hot encoding), for example "cat"=001, "dog"=010, "car"=100. But this had a fatal flaw: all words are equally distant from each other. The distance from "cat" to "dog" is exactly the same as the distance from "cat" to "car" — which clearly defies our intuition.Early approaches assigned each word a numeric ID (One-Hot encoding), for example "cat"=001, "dog"=010, "car"=100. But this had a fatal flaw: all words are equally distant from each other. The distance from "cat" to "dog" is exactly the same as the distance from "cat" to "car" — which clearly defies our intuition.
The revolutionary insight of Embedding is this: it maps each word into a dense, low-dimensional vector space, where semantically similar words naturally cluster together. In this space, "cat" and "dog" are close neighbors, while "car" sits far away — the computer can finally "understand" semantics.The revolutionary insight of Embedding is this: it maps each word into a dense, low-dimensional vector space, where semantically similar words naturally cluster together. In this space, "cat" and "dog" are close neighbors, while "car" sits far away — the computer can finally "understand" semantics.
- One-Hot: Dimension = vocabulary size (potentially tens of thousands), each vector has only a single 1 and all other entries are 0 — sparse and devoid of semantics - Embedding: Dimensions typically 768–1536, every number carries meaning — dense and rich in semantic information - Key breakthrough: Word2Vec (2013) proved that "a word's meaning can be defined by its context," ushering in the Embedding era- One-Hot: Dimension = vocabulary size (potentially tens of thousands), each vector has only a single 1 and all other entries are 0 — sparse and devoid of semantics - Embedding: Dimensions typically 768–1536, every number carries meaning — dense and rich in semantic information - Key breakthrough: Word2Vec (2013) proved that "a word's meaning can be defined by its context," ushering in the Embedding era
------
The core idea of Embedding can be summarized in one sentence: use a set of numbers (a vector) to represent the meaning of a word or sentence.The core idea of Embedding can be summarized in one sentence: use a set of numbers (a vector) to represent the meaning of a word or sentence.
Imagine a two-dimensional coordinate system. Place "cat" at coordinates (0.2, 0.7), "dog" at (0.3, 0.6), and "car" at (0.9, 0.1). You'll notice "cat" and "dog" have similar coordinates, while "car" is far away from both. This is the intuition behind Embedding — semantic similarity becomes spatial distance.Imagine a two-dimensional coordinate system. Place "cat" at coordinates (0.2, 0.7), "dog" at (0.3, 0.6), and "car" at (0.9, 0.1). You'll notice "cat" and "dog" have similar coordinates, while "car" is far away from both. This is the intuition behind Embedding — semantic similarity becomes spatial distance.
1. Semantic clustering: Words with similar meanings automatically group together (animals in one cluster, food in another, technology in another) 2. Analogical relationships: Vector arithmetic can express semantic relationships — the classic example: king − man + woman ≈ queen 3. Dimensional meaning: Each dimension implicitly encodes some semantic feature (e.g., "is it an animal", "size", "sentiment", etc.)1. Semantic clustering: Words with similar meanings automatically group together (animals in one cluster, food in another, technology in another) 2. Analogical relationships: Vector arithmetic can express semantic relationships — the classic example: king − man + woman ≈ queen 3. Dimensional meaning: Each dimension implicitly encodes some semantic feature (e.g., "is it an animal", "size", "sentiment", etc.)
| Encoding Method | Dimensions | Semantic Information | Typical Application |
|---|---|---|---|
| One-Hot | Vocabulary size (~50,000) | None | Traditional NLP |
| Word2Vec | 100–300 | Word-level semantics | Word similarity, analogical reasoning |
| BERT Embedding | 768 | Contextual semantics | Sentence understanding, QA |
| OpenAI text-embedding-3 | 1536–3072 | Deep semantics | RAG, semantic search |
------
With vector representations in hand, the next question is naturally: how do we measure how similar two vectors are? It's like measuring how close two cities are on a map — you can measure the straight-line distance, or you can check whether they point in the same direction.With vector representations in hand, the next question is naturally: how do we measure how similar two vectors are? It's like measuring how close two cities are on a map — you can measure the straight-line distance, or you can check whether they point in the same direction.
- Cosine Similarity: Measures whether two vectors point in the same direction. Range: [-1, 1]. 1 means identical direction, 0 means orthogonal (unrelated), -1 means completely opposite. The go-to choice for text semantic comparison because it's unaffected by vector magnitude. - Euclidean Distance: Measures the straight-line distance between the endpoints of two vectors. Range: [0, ∞). 0 means they coincide exactly; the larger the value, the less similar. Suitable for scenarios where "absolute magnitude" matters.- Cosine Similarity: Measures whether two vectors point in the same direction. Range: [-1, 1]. 1 means identical direction, 0 means orthogonal (unrelated), -1 means completely opposite. The go-to choice for text semantic comparison because it's unaffected by vector magnitude. - Euclidean Distance: Measures the straight-line distance between the endpoints of two vectors. Range: [0, ∞). 0 means they coincide exactly; the larger the value, the less similar. Suitable for scenarios where "absolute magnitude" matters.
| Metric | Formula Intuition | Range | Use Cases |
|---|---|---|---|
| Cosine Similarity | Compares direction, ignores magnitude | [-1, 1] | Text semantic search, recommendation systems |
| Euclidean Distance | Straight-line distance between endpoints | [0, ∞) | Image features, cluster analysis |
| Dot Product | Direction × Magnitude | (-∞, +∞) | Fast computation with normalized vectors |
| Manhattan Distance | Distance traveled along coordinate axes | [0, ∞) | High-dimensional sparse vectors |
------
Suppose you have 1 million documents, each converted into a 1536-dimensional vector. A user asks a question, and you need to find the 10 most similar ones. The most direct approach is to compute similarity one by one — but that means performing 1 million 1536-dimensional vector operations, which is far too slow.Suppose you have 1 million documents, each converted into a 1536-dimensional vector. A user asks a question, and you need to find the 10 most similar ones. The most direct approach is to compute similarity one by one — but that means performing 1 million 1536-dimensional vector operations, which is far too slow.
This is the problem that vector indexing solves: trade space for time by building index structures through preprocessing, reducing retrieval speed from O(n) to approximately O(log n).This is the problem that vector indexing solves: trade space for time by building index structures through preprocessing, reducing retrieval speed from O(n) to approximately O(log n).
- Brute-force (Flat): Compare one by one. 100% accurate but slow. Suitable for small datasets (< 100k). - IVF (Inverted File Index): Partition the vector space into regions (clusters) first, then search only the nearest few regions during querying. Like organizing a library by subject — you only go to the relevant sections when looking for a book. - HNSW (Hierarchical Navigable Small World): Build a multi-layer graph structure, navigating from coarse to fine granularity layer by layer. Like first looking at a world map to locate a country, then a provincial map, and finally a street map. - PQ (Product Quantization): Compress high-dimensional vectors into short codes, sacrificing a small amount of precision for significant memory savings. Suitable for ultra-large-scale datasets.- Brute-force (Flat): Compare one by one. 100% accurate but slow. Suitable for small datasets (< 100k). - IVF (Inverted File Index): Partition the vector space into regions (clusters) first, then search only the nearest few regions during querying. Like organizing a library by subject — you only go to the relevant sections when looking for a book. - HNSW (Hierarchical Navigable Small World): Build a multi-layer graph structure, navigating from coarse to fine granularity layer by layer. Like first looking at a world map to locate a country, then a provincial map, and finally a street map. - PQ (Product Quantization): Compress high-dimensional vectors into short codes, sacrificing a small amount of precision for significant memory savings. Suitable for ultra-large-scale datasets.
| Index Type | Build Speed | Query Speed | Recall | Memory Usage | Suitable Scale |
|---|---|---|---|---|---|
| Flat (Brute-force) | No build needed | Slow | 100% | High | < 100k |
| IVF | Moderate | Fast | 95%+ | Medium | 100k–10M |
| HNSW | Slow | Very fast | 99%+ | High | 100k–10M |
| PQ | Moderate | Fast | 90%+ | Very low | > 10M |
| IVF-PQ | Moderate | Fast | 92%+ | Low | > 100M |
------
With vectors and indexing algorithms in hand, you need a place to store and manage them. Traditional databases (MySQL, PostgreSQL) excel at structured data but struggle with high-dimensional vector similarity search. Vector databases are designed specifically for this scenario.With vectors and indexing algorithms in hand, you need a place to store and manage them. Traditional databases (MySQL, PostgreSQL) excel at structured data but struggle with high-dimensional vector similarity search. Vector databases are designed specifically for this scenario.
1. Efficient storage: Storage formats optimized for high-dimensional floating-point vectors 2. ANN retrieval: Built-in support for multiple approximate nearest neighbor indexing algorithms (HNSW, IVF, etc.) 3. Metadata filtering: Support for filtering by tags, timestamps, and other criteria alongside vector search 4. Real-time updates: Support for dynamic insertion, deletion, and modification of vectors without rebuilding the entire index 5. Horizontal scaling: Distributed architecture supporting billion-scale vector collections1. Efficient storage: Storage formats optimized for high-dimensional floating-point vectors 2. ANN retrieval: Built-in support for multiple approximate nearest neighbor indexing algorithms (HNSW, IVF, etc.) 3. Metadata filtering: Support for filtering by tags, timestamps, and other criteria alongside vector search 4. Real-time updates: Support for dynamic insertion, deletion, and modification of vectors without rebuilding the entire index 5. Horizontal scaling: Distributed architecture supporting billion-scale vector collections
| Database | Type | Characteristics | Use Cases |
|---|---|---|---|
| Pinecone | Fully-managed cloud service | Zero ops, works out of the box | Rapid prototyping, small-to-medium production |
| Milvus | Open-source distributed | High performance, scalable | Large-scale production environments |
| Chroma | Open-source lightweight | Embeddable, clean API | Local development, small projects |
| Weaviate | Open-source cloud-native | Built-in vectorization, GraphQL | Scenarios requiring automatic vectorization |
| Qdrant | Open-source high-performance | Rust implementation, strong filtering | Scenarios requiring complex filtering |
| pgvector | PostgreSQL extension | Reuses existing PG infrastructure | Teams already using PostgreSQL |
------
Now that we understand each component, let's wire them together and see how a complete vector retrieval system works.Now that we understand each component, let's wire them together and see how a complete vector retrieval system works.
The entire flow splits into two tracks: offline ingestion (turning documents into vectors and storing them) and online querying (turning questions into vectors and searching).The entire flow splits into two tracks: offline ingestion (turning documents into vectors and storing them) and online querying (turning questions into vectors and searching).
1. Document loading: Read raw text from various sources (PDFs, web pages, databases) 2. Text preprocessing: Clean, denoise, and normalize (strip HTML tags, special characters, etc.) 3. Text chunking: Split long text into appropriately sized chunks according to a strategy (200–500 tokens) 4. Vectorization: Call an embedding model (e.g., OpenAI text-embedding-3-small) to convert each chunk into a vector 5. Store in vector database: Write the vectors along with the original text and metadata into the database1. Document loading: Read raw text from various sources (PDFs, web pages, databases) 2. Text preprocessing: Clean, denoise, and normalize (strip HTML tags, special characters, etc.) 3. Text chunking: Split long text into appropriately sized chunks according to a strategy (200–500 tokens) 4. Vectorization: Call an embedding model (e.g., OpenAI text-embedding-3-small) to convert each chunk into a vector 5. Store in vector database: Write the vectors along with the original text and metadata into the database
1. Receive query: User inputs a natural language question 2. Query vectorization: Convert the question into a vector using the same embedding model 3. Similarity retrieval: Search the vector database for the Top-K most similar document chunks 4. Post-processing: Rerank, deduplicate, filter by metadata 5. Return results: Return the most relevant document chunks to the caller (or pass them to an LLM to generate an answer)1. Receive query: User inputs a natural language question 2. Query vectorization: Convert the question into a vector using the same embedding model 3. Similarity retrieval: Search the vector database for the Top-K most similar document chunks 4. Post-processing: Rerank, deduplicate, filter by metadata 5. Return results: Return the most relevant document chunks to the caller (or pass them to an LLM to generate an answer)
| Stage | Key Decision | Recommended Approach |
|---|---|---|
| Embedding Model | Accuracy vs. cost vs. speed | OpenAI text-embedding-3-small (best value) |
| Chunking Strategy | Granularity vs. semantic integrity | Recursive chunking, 200–500 tokens |
| Vector Database | Scale vs. operational cost | Chroma for small projects, Pinecone/Milvus for production |
| Similarity Metric | Semantic vs. exact | Cosine similarity (go-to for text scenarios) |
| Top-K Value | Recall vs. noise | Retrieve 20 first, rerank and take Top 5 |
------
Embedding and vector retrieval form the bridge connecting "human language" and "machine understanding." They are also the foundational infrastructure for AI applications such as RAG, semantic search, and recommendation systems.Embedding and vector retrieval form the bridge connecting "human language" and "machine understanding." They are also the foundational infrastructure for AI applications such as RAG, semantic search, and recommendation systems.
Key takeaways from this chapter:Key takeaways from this chapter: