Content revision 4e771eaf0390cc87b6c7c6fd8469f3508f6ceeec7119472e6c1ccf17e2e1e5d2

## Find a passage in its original context

AI × Self
Source ID aipersonalarchive:find-a-passage-in-its-original-context
Original https://www.henryw.me/#/aipersonalarchive#find-a-passage-in-its-original-context

Opening a long project page still leaves a visitor looking for the passage that answers their question. I made each Search result show the matching text and open its precise location. A visitor may remember a project’s name, or only the idea that made it interesting. Search accommodates both. A question about translating business requirements into software can lead back to my work at HSBC, even without the company name in the query. The result keeps enough of the passage visible to help the visitor decide where to go next. How Search finds related material Names such as HSBC are useful word matches, while a question about translating business requirements into software may describe the same work in different language. I combined lexical retrieval, which matches words, with semantic retrieval, which looks for related meaning.

When I build the site, a sentence-embedding model converts each passage into an embedding, a vector of numbers that represents its meaning. Related meanings tend to have similar vectors. At search time, the browser uses the same model to embed the query and compares it with those stored vectors. This lets each search run locally. Word matches still matter, especially for names and technical terms. I merge the two rankings with weighted reciprocal-rank fusion, giving precise word matches priority while retaining relevant semantic matches. The comparison shows what each method contributes. I use all-MiniLM-L6-v2, which produces 384-dimensional embeddings. I store their values as 8-bit integers with a per-vector scale to reduce the download. At query time, cosine similarity compares the vectors. Lexical ranking uses a TF-IDF-like score, giving informative words more weight.

Reciprocal-rank fusion combines list positions because the two scoring systems use different scales. A ranking always has a first result, even for an unrelated query. I tested realistic searches alongside deliberately unrelated ones. In that set, unrelated examples scored at or below about 0. 22, most useful matches were above 0. 45, and a few relevant ones fell near 0. I set the semantic floor at 0. 24 to retain those weaker matches while excluding clearly unrelated results. Adjust the threshold in the comparison to see how many passages remain. Its scores occupied a narrower range, making an absolute threshold less useful, and recall did not improve enough to justify switching. MiniLM gave me a clearer separation between relevant and unrelated material in these tests.