Designing a Data Query Engine

It is better to know what you don't know, and design reliable systems.

The goal is infrastructure that lets a language model answer questions from a large corpus held in many formats: documents, spreadsheets, scanned pages, email, transcripts and database records. The model does the reading. The infrastructure decides what it reads, records where every fact came from, and makes each answer checkable against its sources.

A good answer names its sources, says how certain it is, and says plainly when the material doesn't settle the question.

Three constraints

Three constraints shape every decision in these notes.

• The context window is finite. A model reads a fixed number of tokens per request, and the corpus is many orders of magnitude larger. It is never given the corpus whole: each answer rests on the few passages selected for it, and anything not selected doesn't exist as far as the model is concerned. Retrieval, chunking and summarisation make that selection, and each one can lose information.
• Every answer shows its working. An answer states what it is, why it is correct, and how it was reached. The what is the claim. The why is the evidence, cited to its sources. The how is the method: a lookup, a comparison, a calculation or a chain of steps. An answer missing any of the three can read confidently and still be impossible to check.
• Some questions have more than one answer. Retrieved context can be incomplete, sources can disagree, and some questions need domain knowledge the corpus doesn't contain. That is a normal result, not a failure. The system reports competing answers side by side, states what is missing, and marks where a specialist has to decide.

The rest of this post takes them in turn: a problem, then the solution built for it. Each solution leaves everything below it unchanged, so the full system is the three read together.

---

Problem 1: The context window is finite

Every call to a model has a fixed budget for what it reads and what it writes. Whatever doesn't fit, the model never sees and never says.

One call reads a fixed slice of the input and writes a fixed slice of the output. Not to scale: the corpus is many times larger than the input window.

Character limits per call

Key insights

1. Business and domain knowledge is designed to fit. What goes into a call, and what comes out of it, stays within the limits by construction.
2. The system knows what it left out. Anything kept away because of a limit is recorded, so an answer can say what it didn't consider.
3. Limits are configuration, not code. When a new model ships, its limits are set in one place and the system adapts to them.

---

Solution 1: Handling the context limit

No document longer than the window can be read in one call, so documents are read as summaries first. Each document is split into segments by line number, and each segment is summarised on its own. Those summaries are summarised again, level by level, until one root summary covers the whole document.

A query starts at the root, opens only the branches that look relevant, and reads original lines only at the leaves it chooses. The model sees a few summaries and a few hundred lines of source, however long the document is. The tree is built once at ingest and reused by every query, so the cost is paid once per document, not once per question.

A summary tree over one document. Each leaf summarises one isolated line range, and each parent summarises its children. Every line belongs to exactly one leaf. Ranges never overlap and never leave a gap.

Design notes

• Segments are mutually isolated. Every line belongs to exactly one leaf, ranges never overlap, and no line is skipped. Line numbers are the key: they make each segment addressable, let a parent's range be derived from its children, and let an answer cite the exact lines it used.
• Windows follow the content, not a fixed size. A dynamic sliding window keeps everything about one context in a single window, so windows vary in length. Segmentation takes two runs:
  • First run: mark boundaries. The document is cut into provisional windows of equal size. Each call reads three whole windows: the current one, the one before it and the one after it. The model marks boundaries only inside the current window, and uses its neighbours to see where a context really starts and ends. The call then moves on by one window. Window size is set so three windows fit the input limit.
  • Second run: summarise. The document is cut at the marked boundaries into final windows, and each window is summarised on its own.

A dynamic sliding window. The first run reads each window with the window before and after it, and marks boundaries only in the current one. The second run summarises each resulting window.

A node in the tree:

type SummaryNode = {
  id: uuid;
  target_node_id?: uuid;  // the parent; absent on the root
  depth: number;          // 0 at the root
  is_leaf: boolean;
  line_start: number;     // inclusive
  line_end: number;       // inclusive
  summary: string;        // bullet points, one fact each
};

• A summary is bullet points. Each bullet carries one fact from the lines it covers. Bullets don't explain, connect or conclude: that is left to the query, which knows what it is looking for.
• Prompts protect objective content. Summarisation prompts are tuned until numbers, units, dates, names, identifiers and negations come through exactly as written. Nothing is inferred, rounded or reworded into a softer claim. Each summary is then checked mechanically: every number and name in it must appear in its own line range.

Lifecycle of a document

Possible queries

Each document carries a short, hand-written list of the questions it should be able to answer. The list tells the navigator what the document is for before it reads a single summary.

clinic-policy-2024.pdf
  - How much notice is needed to cancel an appointment?
  - Is there a fee for a missed appointment, and how much is it?
  - Who can approve an exception to the policy?

Where the list lives is a trade-off, and the choice is made by measurement.

Start with both. Run a fixed set of test queries against each placement, and keep the one that opens the right leaves most often for the fewest tokens.

Lifecycle of a query

Architecture

   AT INGEST                 STORES                    AT QUERY TIME

                                                       ┌──────────────┐
                                                       │    Query     │
                                                       └──────┬───────┘
   ┌──────────────┐         ┌──────────────┐           ┌──────▼───────┐
   │ Summariser   │────────►│ Summary tree │──────────►│ Navigator    │
   └──────▲───────┘         └──────────────┘           └──────┬───────┘
   ┌──────┴───────┐         ┌──────────────┐           ┌──────▼───────┐
   │ Segmenter    │────────►│ Line index   │──────────►│ Reader       │
   └──────▲───────┘         └──────────────┘           └──────┬───────┘
   ┌──────┴───────┐                                     ┌──────▼───────┐
   │ Documents    │                                     │ Answer writer│
   └──────────────┘                                     └──────────────┘

Ingest builds the summary tree once; every query reads down it. The summariser, segmenter, navigator and answer writer are the steps that call a model.

---

Problem 2: Every answer shows its working

An answer on its own is only a claim. Every answer the system returns comes in three parts, so a reader can check it without asking the model again. The why is the part most often missing, and the one that makes an answer checkable.

• What is the answer? The claim, stated plainly.
• Why is it correct? The evidence, cited to its sources.
• How did we reach it? The method, step by step.

One answer, in three parts

Which clinics missed the most appointments last quarter?

Key insights

1. An answer without its why and how isn't returned. The three parts are produced together, not added afterwards.
2. The why cites the passages and records the answer rests on, so a reader can check it without running the query again.
3. The how records every step, including what was left out, so the answer can be repeated or corrected.

---

Solution 2: Searching the summary tree

Solution 2 extends Solution 1. The summary tree was built to fit a document into the window. Here it becomes a search index. A query walks down the tree from the root, and at each node the model makes one small choice: answer here, open some of the children, or move on. The walk ends in an answer that carries its what, why and how, which is what Problem 2 asks for.

Routes run in parallel

A node can send the search down more than one child, and that is expected. A question about cancellation fees may need the policy section and the billing section. Each child opened starts its own route. Routes run in parallel, because each one reads only its own node, and they are merged when the answer is written.

A stops at depth 1: its summary already states the what. B runs to a leaf and reads lines 859-960 for the how. Only the nodes on the two routes are read; everything else is never loaded.

Cost grows with depth, not length

A route reads one node per level, so its cost follows the depth of the tree, not the length of the document. With b children per node, a document of n leaves is about log₍b₎ n levels deep. A document ten thousand times longer adds a handful of reads, not ten thousand. Parallel routes multiply this by the number of routes, and every read is a small call that fits the window with room to spare.

Where a route stops

A route stops at the highest node that can answer the part of the query it is after. Going deeper costs reads. Stopping too early costs evidence.

A route after the what can stop as soon as a summary states the claim. A route after the how always ends at a leaf, because only source lines can show how an answer was reached. The why sits between them: the prompt defines what counts as a valid reason, and the model applies that to what it has read.

Generating an answer from a node

Every node is read with the same prompt. The format is fixed, so the output can be parsed and checked mechanically, and the same prompt works at the root, halfway down and at a leaf. At a leaf, the bullets are replaced by numbered source lines.

SYSTEM
  You read one node of a summary tree and make one choice.
  Reasoning rules: {what counts as a valid reason in this corpus}
  Possible queries: {the list for this document}
  Examples: {shots: a node, and the choice it should get}

NODE n-0412 · lines 721–960 · depth 2
  - {bullet}
  - {bullet}

CHILDREN
  [A] n-0419 · lines 721–858 · {its first bullet}
  [B] n-0420 · lines 859–960 · {its first bullet}

QUERY
  {the query, split into what, why and how}

CHOOSE ONE
  ANSWER     The answer is in this node. Cite its bullets or lines.
  DESCEND    Open these children: A, B, or both.
  NOT_HERE   Nothing here or below bears on the query.

RESPOND IN JSON
  { "choice": "ANSWER" | "DESCEND" | "NOT_HERE",
    "open": ["A"],
    "what": "…",
    "why": "…",
    "how": [{ "line": 902, "quote": "…" }] }

Choose, don't invent

Hallucination starts when a model is asked an open question and fills the gap with something plausible. The node prompt never asks one.

• Every decision is a pick from a closed list. One of three choices, children by letter, evidence by line number. The model never writes a fact that isn't already in front of it.
• Evidence is quoted, then checked. Each quote must match its bullet or line exactly before the answer is accepted.
• There is always a way out. NOT_HERE is on every list, so the model is never pushed to produce something when nothing fits.
• Anything off the list is rejected. A response with an unknown choice, child or line is discarded and the node is read again.

Multi-shot refining

The prompt improves by adding worked examples, and only from failures.

1. Run a fixed set of test queries, each with the node it should stop at and the answer it should give.
2. Collect every wrong choice: a route that stopped too early, went too deep or opened the wrong child.
3. Turn each one into a shot: the node as the model saw it, the right choice, and the reason for it.
4. Run the test set again. Keep a shot only if it fixes something without breaking something else.

Shots cost input tokens on every call, so the set is pruned as well as grown. A shot that no longer changes any result is removed.

Lifecycle of a query

Ingest is unchanged from Solution 1. What this solution adds is a planner and a parallel search in place of the single navigator, and a citation check on every answer.

---

Problem 3: Some questions have more than one answer

One query can find two passages that answer it differently. Neither has to be wrong: the documents disagree, or each covers part of the question.

                 ┌────────────────────────────────────────┐
                 │  How much notice does a cancellation   │
                 │  need?                                 │
                 └───────────┬─────────────────┬──────────┘
                             │                 │
                ┌────────────▼───────┐ ┌───────▼────────────┐
                │ 24 hours' notice   │ │ 48 hours' notice   │
                │ clinic-policy-     │ │ front-desk-        │
                │ 2024.pdf           │ │ handbook.docx      │
                └────────────────────┘ └────────────────────┘

One query, two passages, two answers. Both are returned with their sources instead of one being picked silently.

Why it happens, and what the system does

Key insights

1. More than one answer is a result, not an error. It is reported as it is, not smoothed into one.
2. Every answer carries its source, so a disagreement can be traced and settled by someone who knows.
3. Doubt is stated with its reason: missing context, conflicting sources, or a question outside the corpus.

---

Solution 3: Shaped by the use case

Solutions 1 and 2 build the core: an index that fits any document into the window, and a search that returns answers with their what, why and how. Problem 3 showed that a search can return more than one answer. The core shouldn't decide which ones matter, because that depends on who is asking and what they need to do next.

Solution 3 puts that decision in a layer of use-case strategies on top of the core. The core stays the same for every use case. The strategies change.

Top N is enough

More answers aren't more help. Past the first few, each extra result costs the reader time and the next call tokens, and adds less than the one before. So the candidates are ranked and the top N are shown, where N is set by the use case, not by how many answers exist.

The rest are held back, not thrown away. The reader sees how many there are and can open them in one step. Nothing is hidden, only ordered.

Seven candidates, ranked. The line under the third is N for this use case. The number held back is always shown.

What decides the order

• Evidence. An answer cited to source lines outranks one that rests on a summary.
• Agreement. An answer found on more than one route outranks one found once.
• Source weight. Set by the business: a signed policy outranks a handbook, and a current version outranks an old one.
• Fit. How directly the answer addresses the what, why or how that was asked.

Disagreement is never ranked away. When N is 1 and a lower-ranked answer contradicts the top one, the answer carries a flag that says so. That keeps Problem 3 honest at any N.

N follows the use case

Follow-up questions

An answer usually prompts the next question. Rather than start again from scratch, the reader picks the answers worth keeping and follows up from them.

        ┌──────────────── searched again, from the picked nodes rather than the root ───────────────┐
        │                                                                                           │
   ┌────▼───┐     ┌──────────┐     ┌──────────┐     ┌─────────┐     ┌───────────┐                  │
   │  Ask   │────►│  Search  │────►│  Top N   │────►│  Pick   │────►│ Follow up │──────────────────┘
   └────────┘     └──────────┘     └──────────┘     └─────────┘     └───────────┘
    a query       routes in         ranked           reader keeps    suggested
                  parallel          answers          some            or typed

The follow-up loop. Picking is the reader's one decision; everything else is the system's.

• Picking is quick. An answer is kept with one tap. The picked set is a short list of node ids and line ranges, not a copy of their text.
• A follow-up starts from the picks. Search is scoped to the picked nodes and their neighbours, so it is faster, cheaper and stays on topic. If nothing there answers, it widens to the root and says so.
• Suggested follow-ups are choices, not inventions. They come from three places: the parts of the query not yet answered, often the why or the how; the picked node's unopened children; and the document's possible queries. The reader taps one or types their own.
• The thread stays short. Only the picks carry forward, so a long conversation doesn't fill the window.

Strategies on top of the core

Everything the business decides sits in one layer. A strategy is configuration: it changes what the core is asked and how its results are shown, never how the core indexes or searches. A new use case is a new strategy, not a new pipeline.

The core is built once. Strategies sit on top and change per use case; adding one changes nothing below it.

What a strategy sets:

Lifecycle of a query

---

The full solution

The three solutions are layers of one system. Solution 1 makes any document readable within the context limit. Solution 2 searches what Solution 1 built and returns answers that show their working. Solution 3 decides, per use case, how many of those answers to show and what the reader can do next. Each solution left everything below it unchanged, so the full system is the three read together.

In brief

What is fixed, and what is configured

• Fixed: the index and the search. Conversion, windows, summaries, the tree, the routes and citation checks work the same for every use case.
• Configured per model. Window size, summary length and how many windows a call reads follow the model's limits, set in one place.
• Configured per use case. Scope, top N, ranking, depth, follow-ups and format.

Lifecycle of a document

A document is processed once, when it arrives, and nothing about that depends on who will ask about it. It becomes numbered lines, is cut into windows that each hold one context whole, is summarised window by window and checked against its own lines, and rolls up into a tree.

1. Convert. PDF, spreadsheet, email, scan or transcript becomes numbered lines of text.
2. Mark boundaries. First run: each window is read with its neighbours.
3. Summarise. Second run: one leaf of bullet points per window.
4. Check. Every number and name is found in its own lines.
5. Build tree. Leaves roll up, level by level, to one root.
6. Store. The summary tree and the line index, built once and searched by every query.

Lifecycle of a query

A query is processed every time it is asked. The use case's strategy sets its scope and shape. The planner splits it into what, why and how, and routes search the tree in parallel, each stopping where its part is answered. Only routes after the how read source lines. Answers are chosen rather than invented and checked, then ranked, and the top N are shown. A pick starts the next question where the last one ended.

1. Load strategy. Scope, top N, ranking, depth and format.
2. Plan. The query is split into what, why and how.
3. Search. Routes descend in parallel and stop where their part is answered.
4. Read lines. Only the how reads source lines.
5. Answer and check. Chosen, not invented; every quote checked.
6. Rank. Evidence, agreement, source weight and fit.
7. Show top N. The rest held back, disagreement flagged.
8. Pick. The reader keeps some answers.
9. Follow up. Searched again, from the picks.

It is better to know what you don't know, and design reliable systems.