Retrieving knowledge that actually applies

Background

One big component of building an engineering learning system is deciding which past lessons apply to the task an agent is working on now. Similarity is a useful starting point, but two past lessons can sound relevant to the same task while only one applies to the system the agent is working on.

For example, an agent debugging a webhook integration might find two lessons about failed signature verification. One says to verify the request body exactly as received, before parsing the JSON. The other describes an integration that signs canonicalized JSON. Both lessons can be correct in their original setting, but the fix depends on how this integration’s signatures are generated.

An agent adding caching to a customer dashboard faces a similar choice. A lesson from a public product catalog recommends sharing cached responses across users. Another lesson, about private account data, requires separate cache entries for each customer. Both concern caching, but only the second applies to a dashboard showing customer-specific data.

These are the kinds of distinctions we wanted Groundtrack’s retrieval to handle. Returning superficially related lessons leaves the agent to work out which advice applies, alongside everything else it needs to do. It also leads to the following issues:

• Misleading guidance. Plausible but inapplicable advice can steer an agent toward the wrong implementation. A March 2026 study on distractor documents tested Qwen-based question-answering models with supporting passages alone and with irrelevant passages mixed in. Across three benchmarks, answer and citation scores were lower with distractors, even though all the supporting information was still present.

• Unnecessary work. Irrelevant or outdated context can contribute to repeated, unproductive actions. In one 300-task coding evaluation in AgentDiet (FSE 2026), cleaning up agent histories reduced average steps from 57.2 to 43.9 and runs hitting the 100-step limit from 66 to 26, with comparable task success.

• Repeated input costs. Results retained in the conversation remain part of subsequent model inputs. Their cost can persist across multiple steps of a task. Prompt caching can reduce the processing cost, but it does not remove irrelevant information from the context.

• Less room for useful context. Extra results compete for space with code, requirements, test output, and earlier decisions. As the history grows, the system may need to discard or summarize it sooner. Anthropic’s context-engineering guidance notes that aggressive compaction can lose subtle details needed later. These concerns begin before a context window is completely full.

When evaluating our own retrieval pipeline, we noticed this gap too. On the queries used in our final controlled comparison, vector search returning eight results covered 93.4% of the required knowledge groups, but only 21.1% of the returned records were labeled useful. A group counted as covered when at least one returned record supplied that requirement; coverage did not mean every result was applicable. These measurements describe retrieval quality, rather than the success rate of an agent completing the task.

Our goal was to preserve the knowledge an agent needs while returning only the guidance that applies (and returning nothing when no guidance applies). In other words, maximizing precision while maintaining near-100% required-knowledge coverage.

Useful results

21.1% → 85.1%

Required knowledge found

93.4% → 100%

Results per query

8 → 1.34

About 84% fewer tokens in the returned results.

Our previous vector search vs. our new approach, which checks whether the advice applies to the task. Both were scored on the same test questions.

How we tested retrieval

We constructed 1,332 experience records and 480 queries across 48 engineering task families. The records and queries were synthetic: we specified the operating conditions, the knowledge each query required, and which records supplied it.

The corpus included equivalent lessons, complementary requirements, advice for incompatible operating conditions, superseded or disputed guidance, and diagnostic notes that repeated a symptom without explaining what to do. Some queries required multiple distinct lessons. Others asked for information absent from the corpus, such as an undocumented configured value. The 160-query holdout included 32 of these no-answer queries, where the correct result was an empty list.

For the selector comparisons, we used three splits:

• Development: 160 queries for choosing prompts and models.
• Calibration: 160 queries for choosing selection thresholds.
• Holdout: 160 queries for measuring the frozen configuration. These produced the figures in the introduction.

Each task family stayed in one split, so differently worded versions of the same scenario could not appear in both development and holdout. Later experiments reused this holdout; they were frozen comparisons on an existing test set, rather than tests on newly unseen scenarios.

The model comparisons below use successful calls only. Within each chart or table, every method is scored on the same queries.

We scored two things separately:

• Precision: useful returned records divided by all returned records.
• Required-knowledge coverage: requirements covered by at least one returned record divided by all requirements. Equivalent records counted toward the same requirement.

For example, one benchmark query concerned a reporting service using pooled SQL connections. It required two rules: set tenant context within the report’s transaction, and wait for that setup to finish before issuing the query. Dense search returned eight records: two equivalent records covering the first rule and six irrelevant diagnostic notes. It missed the second rule. That gave 25% precision (2/8) and 50% coverage (1/2). Returning more copies of the first rule would not have fixed the missing requirement.

Returned records

25% precision 2 useful / 8 returned

1. 01Rule 1Useful
2. 02Rule 1Useful
3. 03—Irrelevant
4. 04—Irrelevant
5. 05—Irrelevant
6. 06—Irrelevant
7. 07—Irrelevant
8. 08—Irrelevant

Both useful records cover the same rule.

Required knowledge

50% coverage 1 covered / 2 required

• Rule 1Covered

  Set tenant context within the report's transaction.

  Supplied by records 01 and 02
• Rule 2Missing

  Wait for tenant setup to finish before issuing the query.

  Not supplied by any returned record

When required guidance was missing, we checked whether search had failed to find it or whether it had been found and then filtered out.

Why a similarity cutoff wasn’t enough

We started with vector search, which ranks experience records by how closely their embeddings match the query. Across the 160 holdout queries, the first 50 candidates contained records covering every required knowledge group. Search could find the knowledge. The problem was choosing what to return.

First, we varied the number of results. Of the 160 test queries, 60 were designed to make the choice harder: several records looked relevant, but some applied to different operating conditions, and some questions needed more than one lesson. On these 60 queries:

• 5 results: 42.6% coverage and 19.3% precision.
• 8 results: 81.5% coverage and 22.1% precision.
• 25 results: 100% coverage and 12.0% precision.

Returning fewer records dropped necessary guidance. Returning more recovered it, but added enough irrelevant records that almost nine out of ten results were not useful. A fixed count also forced an answer onto queries where nothing applied.

We then tried a minimum similarity score: return records above the cutoff and discard the rest. This allowed the search to return fewer results, including none, without changing the ranking.

The cutoff looked promising on the other 100 holdout queries. Precision rose from 19.9% to 32.8%, while required-knowledge coverage stayed at 100%. We chose the cutoff using the separate calibration split, not those holdout results.

But no tested cutoff passed our calibration checks for the harder scenarios: preserve at least 98% of required knowledge, miss none of the requirements marked critical, and return no records marked unsafe for the task. The setting that worked on the broader scenarios was not a general solution.

Similarity scores helped find related records, but did not reliably tell us whether their advice applied. Raising the cutoff could remove a needed rule along with the noise; lowering it could admit advice for the wrong conditions. We kept the broad search and moved on to testing rerankers that read the query and each candidate record together.

Trying a dedicated reranker

We next tested BGE, a model that reads the query and each record together, scores their relevance, and reorders the results. This gave it the actual text to compare, rather than relying only on embedding similarity.

We ran bge-reranker-base locally on candidates gathered through vector and keyword search. On the same 60 harder queries, we returned its eight highest-scoring records, without an additional score cutoff. Compared with the original vector search:

• Required-knowledge coverage fell from 81.5% to 77.8%.
• Precision fell from 22.1% to 20.6%.
• Critical requirements missed increased from 6 to 8.
• Records marked unsafe for the task fell from 23 to 1.

BGE returned less unsafe advice, but also left out more necessary guidance. That tradeoff did not meet our goal.

We checked the implementation before drawing conclusions. On 12 reference queries, the local model matched the hosted version’s first result and set of eight returned records. None of the inputs in the harder set had to be shortened to fit the model, so cutting off text did not explain the misses.

This result applies to the BGE model and setup we tested. It showed that a dedicated relevance score still wasn’t enough to choose the guidance we needed. Our next experiment asked a language model explicitly which records applied to the task.

Asking which advice actually applies

We gave a small, lightweight LLM (gpt-5.4-nano) the query and 32 candidate records drawn from vector and keyword search. We asked it to judge each record as directly useful, useful supporting guidance, or not useful. We kept records judged directly useful or useful as supporting guidance, returning at most eight. Records judged not useful were excluded.

The instructions made the distinction explicit: reject advice for the wrong version or operating conditions, and don’t treat generic guidance as an answer to a request for an exact value. For questions with multiple parts, a record could be useful even if it answered only one part.

On the same 60 harder queries, compared with the original vector search:

• Required-knowledge coverage rose from 81.5% to 100%.
• Precision rose from 22.1% to 47.9%.
• Critical requirements missed fell from 6 to 0.
• Records marked unsafe for the task fell from 23 to 1.

This recovered the missing knowledge while removing many irrelevant results. But more than half the returned records were still not useful, and one unsafe record remained.

On the other 100 queries, precision improved from 19.9% to 55.7%, but coverage fell from 100% to 91.1%. The model was more selective, but also discarded guidance those tasks needed.

Now speed was a problem, too. The search-and-selection step took 9.36 seconds at the 95th percentile on the harder queries, exceeding our two-second target. That did not include embedding the query or handling the MCP request.

Explicitly judging applicability was a promising direction, but this version was inconsistent and too slow. We next tested faster models and different ways of asking them to select the records.

Making selection fast enough

We first screened Llama, Qwen, and Gemma through Cloudflare Workers AI. Qwen3 30B-A3B (FP8) and Gemma 4 26B-A4B-IT went into the comparison of selection methods. We also included vector search and a lightweight MiniLM reranker as baselines.

This time, every method started with the same 50 records from vector search. We sent compact summaries and allowed the model to return any number of records, including zero, instead of limiting it to eight.

We compared four approaches:

• One record per call: Qwen judged each record independently, with 50 calls running in parallel.
• All records together: Qwen or Gemma saw all 50 records in one call and chose which to return.
• Smaller batches: Gemma judged five groups of ten records, then we combined the selections.
• Filter first: a similarity cutoff removed low-scoring records before Gemma judged the remaining list.

Each chart compares selection methods within one model. We tested different methods with Qwen and Gemma, so these are separate experiments.

We measured selection time after the records were ready, excluding the initial search. The main comparison allowed six seconds per query so we could examine quality beyond the two-second target.

Individual decisions or one shared list?

With Qwen, judging records independently improved precision, but lost coverage, including two critical requirements. It also took longer than judging the full list in one call.

All records together · 1 call One record per call · 50 parallel calls

Precision

Higher is better

1 call28.0%

50 calls46.1%

0100%

Knowledge coverage

Higher is better

1 call99.3%

50 calls91.3%

0100%

Selection time

95th percentile · Lower is better

1 call0.95 s

50 calls1.54 s

04 seconds

Times exclude the initial search and waits to stay within provider rate limits.

Can batching or filtering make selection faster?

Sending Gemma only the higher-scoring search results (about 35 records instead of 50) was faster, but slightly reduced precision and left out required knowledge. Splitting the list into batches returned more irrelevant records than sending all records together.

All together · 1 call 5 batches · 10 records each Filter first · About 35 records

Precision

Higher is better

All together78.1%

5 batches61.2%

Filter first77.0%

0100%

Knowledge coverage

Higher is better

All together96.1%

5 batches94.1%

Filter first93.1%

0100%

Selection time

95th percentile · Lower is better

All together2.69 s

5 batches3.32 s

Filter first1.74 s

04 seconds

Times exclude the initial search and waits to stay within provider rate limits.

The filter reduced Gemma’s input to about 35 records on average, but removed a record containing a required identifier. On the harder queries in this comparison, coverage was 84.4%.

None of these approaches met our quality requirements. Examining the misses revealed another problem: six test queries asked for exact identifiers that existed in the records’ detailed claims but were absent from the summaries we sent. Search had found the records, but the model could not see the requested information. We needed to fix that input before expecting better decisions.

Giving the model the missing information

We added a claim capsule to each record: a structured block containing its stored statements, the conditions where they apply, and whether they are active, disputed, or superseded. It also kept whether the evidence supported or contradicted each statement. Exact identifiers and conditions stayed attached to the advice. We copied existing fields rather than asking another model to rewrite them.

We reran Gemma with the same prompt, candidate records, and six-second deadline, adding only the capsules. Both versions were scored on the same queries.

Metric	Summary only	With capsule
Knowledge coverage	97.2%	100.0%
Precision	80.3%	79.1%

Four previously empty answers became correct one-record answers once the identifiers were visible. But the model still selected disputed advice and missed some useful supporting records. Capsules fixed the missing-information problem; choosing the right advice remained unfinished.

Comparing models with the fuller context

We kept the capsules and tested five Workers AI models. We also tightened the instruction: cover every distinct need, but choose just one record when several teach the same rule. Extra background advice wasn’t enough reason to include a record.

All five passed the initial output-format checks; we still had to measure whether they selected useful records. We evaluated them on the development set, allowing one call per query with a 1.95-second deadline.

Model	Precision	Knowledge coverage	Critical misses	Unsafe results
Gemma 426B-A4B-IT · Finalist	87.9%	98.9%	1	0
GLM 5.3 FlashFinalist	85.8%	98.9%	0	0
GPT-OSS20B	79.3%	98.9%	1	1
DeepSeek V4 Flash0731	80.6%	100.0%	0	0
Qwen330B-A3B (FP8)	49.8%	95.8%	1	2

All five models are compared on the same completed development queries. These are not the final-test results.

GLM and Gemma continued to calibration and the holdout evaluation. After calibration, we returned only records Gemma marked applicable; GLM also kept records it marked uncertain.

On a separate test set, we compared the questions where both models returned valid selections. Both covered every requirement. Gemma reached 85.0% precision, compared with 79.3% for GLM, so we continued with Gemma.

The resulting pipeline

The evaluated pipeline retrieved broadly, exposed the detailed claims, and asked Gemma to return the smallest sufficient set.

1. 01Vector searchFind 50 candidates
2. 02Claim capsulesAttach existing details
3. 03Gemma selectionOne call · 1.95-second limit
4. 040–N resultsReturn the selected records

For the final comparison against vector search, we included all test questions where Gemma returned a valid selection and rescored the original top-eight search results on those same questions. This broader set gives Gemma 85.1% precision, compared with 85.0% on the questions shared with GLM above. Both sides use the original labels.

Original vector search · Top eight Gemma + capsules · Selected set

Precision

Higher is better

Original21.1%

Gemma + capsules85.1%

0100%

Knowledge coverage

Higher is better

Original93.4%

Gemma + capsules100.0%

0100%

Mean results returned

Fewer results, with coverage retained

Original8.00

Gemma + capsules1.34

08 records

Approximately 84% fewer result tokens153,000 → 24,000 tokens across the matched queries

Estimated with the o200k_base tokenizer on both result sets in the same compact format. Counts cover returned results only, excluding the selector's input.

Gemma covered every required knowledge group with an average of 1.34 records instead of eight. It still returned unnecessary or disputed advice, leaving precision below our 90% target.

What remains unresolved

• Reliability and latency. The final scores cover valid responses; nine of 160 calls failed under the 1.95-second budget. Falling back to ordinary search keeps retrieval available but sacrifices selection quality. We need selection to finish reliably within a practical time budget.
• Precision. Selection precision reached 85.1%, a huge jump over our baseline 21.1%, but extra background advice, duplicates, and disputed claims remained problems despite the fuller input. Getting from 85% to 100% will be a more difficult research problem than getting from 20% to 85%.
• A smaller, trained selector. A model trained specifically to judge applicability may eventually replace the general-purpose LLM, as we try to further optimize latency, cost, precision, and knowledge coverage.

These experiments made it clear that vector and keyword search alone aren’t enough for the production retrieval system we’re building. Finding related records and deciding which advice applies are separate jobs. The system needs to do both well.

Retrieval is a small but crucial part of continual learning, the broader problem we’re working on at Groundtrack. Groundtrack is available to try today. It works with Codex, Claude Code, Cursor, OpenCode, and more.