I delayed actually using embeddings because the word sounds like a research paper. It is not. An embedding is a list of numbers that stands in for a piece of text. Same length every time. Nearby lists mean nearby meaning, if the model is any good.
That is the whole trick. The rest is plumbing.
What I hold in my head
Take a sentence. Send it through a model. Get back 1536 floats (or 384, or 768, the number is a property of the model, not a law of nature). Store that array next to the original text.
At query time, embed the question the same way. Compare arrays. Return the texts whose arrays are closest.
No search index is required to understand this. A nested loop over ten chunks in a test file is enough:
function cosineSimilarity(a: number[], b: number[]): number {
if (a.length !== b.length) {
throw new Error('Vector length mismatch');
}
let dot = 0;
let magA = 0;
let magB = 0;
for (let i = 0; i < a.length; i += 1) {
dot += a[i] * b[i];
magA += a[i] * a[i];
magB += b[i] * b[i];
}
return dot / (Math.sqrt(magA) * Math.sqrt(magB));
}
When I could explain that function without looking it up, the libraries stopped feeling like magic.
Two arrays from two different models are not comparable
This one cost me an afternoon. I had a batch of chunks embedded with one model, then switched providers mid experiment and re-embedded only the new chunks with a different model. Same dimension count, coincidentally. The code ran. No errors. The similarity scores were just quietly wrong, every query returning a confident, meaningless top match.
Two embeddings are only comparable if the same model produced both. Not the same dimension count, the same model, ideally the same version of it. A 1536 length array from one model and a 1536 length array from another are just two arrays that happen to be the same size. They do not share a coordinate system. Cosine similarity between them is a number, but it is not a meaningful one.
The fix was boring: store the model name and version next to every embedding, and refuse to compare across a mismatch instead of letting it fail silently.
type EmbeddingRecord = {
vector: number[];
model: string;
createdAt: string;
};
If I change embedding models later, I re-embed everything. There is no partial migration where old and new vectors coexist in the same comparison. That single row of metadata is what makes that decision obvious the next time, instead of a debugging session.
Chunking is the product decision
The array is only as useful as the text you fed the model. I first embedded whole markdown files. Retrieval returned "the right document" and the wrong paragraph. Then I split on headings, then on overlapping windows of ~400 tokens.
There is no universal chunk size. There is only: what question will a user ask, and how much surrounding context does an answer need?
I am treating chunking as a product decision, not a preprocessing detail. The table schema should remember that:
type Chunk = {
id: string;
docId: string;
body: string;
headingPath: string[];
embedding: number[];
};
headingPath has already saved me once, when two chunks were numerically close and only the section title told me which one belonged in the answer.
Looking at what actually gets retrieved
Before I let anything call an LLM, I wrote a script that does nothing but print the top five matches for a hardcoded question, with their scores, and nothing else:
async function debugRetrieve(question: string, k = 5) {
const queryVector = await embed(question);
const results = chunks
.map((chunk) => ({
chunk,
score: cosineSimilarity(queryVector, chunk.embedding),
}))
.sort((a, b) => b.score - a.score)
.slice(0, k);
for (const { chunk, score } of results) {
console.log(score.toFixed(4), chunk.headingPath.join(' > '), '\n', chunk.body.slice(0, 120));
}
}
Running this against a dozen questions I already knew the answers to caught more bugs than any amount of staring at the ingestion code did. It's how I found a chunking script that was silently producing empty headingPath arrays for every chunk in one document, because the regex assumed headings started at the beginning of a line and one file used a different line ending. Every chunk still embedded fine. The score still returned a number. Nothing errored. It just made every match from that document indistinguishable from every other match from that document, and I would not have noticed without reading the raw output.
An LLM sitting on top of bad retrieval will still write a fluent, confident sentence. That is exactly why I don't trust it to tell me the retrieval is broken. I have to look at the arrays and the text myself, before the LLM ever gets involved.
Brute force works until it doesn't
The nested loop above is O(n), it checks every stored vector against the query vector, every time. For ten chunks that's instant. For a few hundred, still instant. I am not optimizing this yet, on purpose.
The reason people reach for pgvector's ivfflat or hnsw indexes, or a dedicated vector database, is that brute force stops being instant somewhere in the tens or hundreds of thousands of rows, depending on hardware. Those indexes trade a small amount of accuracy (they're approximate nearest neighbor, not exact) for a large amount of speed at that scale.
I don't have that problem yet. Adding an index before you need one just adds a second thing that can be subtly wrong, on top of the chunking and the model mismatch and everything else already in the pipeline. The honest brute force loop is also the easiest thing to verify by hand, which matters more right now than the query taking two milliseconds instead of two hundred.
The embedding call is a network call
Easy to forget, since it's wrapped in a function named something friendly like embed(). It's an HTTP request to someone else's server. It has latency, it has a rate limit, and it costs a fraction of a cent every time you make it.
Re-embedding the same chunk twice because a script re-ran is a real way to burn through a rate limit for no reason. The fix is a hash of the chunk's text, stored alongside the embedding:
type Chunk = {
id: string;
docId: string;
body: string;
headingPath: string[];
embedding: number[];
contentHash: string;
};
Before embedding a chunk, hash its body and compare against what's stored. Same hash, skip the call. Different hash, the text changed, re-embed it. It's a caching problem wearing an AI costume.
What I am not doing yet
I am not fine-tuning. I am not swapping models every afternoon. I am not building a vector database product. I am not indexing for scale I don't have. I am trying to get one retrieval path honest: same model in, same model out, scores I can explain, chunks a human could have picked, and a debug script I actually run before trusting the output.
The arrays are the easy part. Making the surrounding system tell the truth is the work.