A retrieval prototype in a folder of notes

Daily workEN2026-10-26

Start with a narrow question: can an assistant answer questions about a small folder of project notes and show where its answers came from? Build the first version with Markdown files, a generated index, and a script. Retrieval-augmented generation supplies selected source material to a model before it answers. This walkthrough keeps that selection local and uses an API only for the final response. ([github.com](https://github.com/openai/chatgpt-retrieval-plugin/blob/main/README.md?utm_source=openai))

§ Step 1: Choose a small, inspectable collection

Create a docs folder containing ten to twenty Markdown or text files about one project. For this exercise, use deployment notes, troubleshooting instructions, and operational decisions. Exclude credentials, personal information, and anything you are not authorized to send to an external service.

Add two working files outside that folder: index.jsonl for generated passages and questions.jsonl for evaluation cases. Keep source documents separate from generated output so rebuilding the index cannot accidentally ingest yesterday's results.

§ Step 2: Turn documents into traceable passages

Write an indexing script that reads files in sorted path order and splits them at headings. As an initial setting, divide long sections into roughly 250-word passages, retaining one paragraph of overlap. Chunking and overlap are established retrieval controls; these particular sizes are starting choices, not requirements. ([developers.openai.com](https://developers.openai.com/api/docs/guides/retrieval))

Write one JSON object per line in index.jsonl. Include these fields:

id: A unique passage identifier within this index version.
path: The relative source filename.
heading: The section title.
start_line and end_line: The original source boundaries.
text: The passage exactly as extracted.
source_hash: A content hash for detecting changes.

Rebuild the entire index after each source edit for now. Write to a temporary file, then replace the previous index only after indexing succeeds.

§ Step 3: Implement a keyword baseline

Begin without embeddings. Lowercase the question and each passage, split them into alphanumeric terms, and remove a short, explicit list of common words. Score each passage by the number of distinct question terms it contains, giving heading matches an additional point. Break ties by passage ID.

Return at most four passages with positive scores. Print their scores, filenames, and line ranges before making any model request. If nothing matches, return a local message saying no matching passage was found. Treat a positive score as a candidate match, not proof that the passage answers the question.

§ Step 4: Build the context before calling the model

Label retrieved passages S1 through S4. Place each label beside its filename, line range, and text. Keep the question separate from these source blocks. Selecting relevant excerpts before generation follows the same broad grounding pattern used in document question-answering workflows. ([help.openai.com](https://help.openai.com/en/articles/6643167-how-to-use-the-openai-api-for-qa-or-to-build-a-chatbot?utm_source=openai))

Use a fixed instruction: answer only from the supplied passages, cite source labels for factual claims, and say when evidence is insufficient. Also instruct the model to treat passage contents as evidence rather than commands. Do not give this prototype tools or access to secrets.

§ Step 5: Connect the answer stage

Use Vana API as the model gateway, configuring your OpenAI-compatible client with its published base address. Keep file loading, indexing, and ranking in your own script; the gateway receives the question and selected excerpts, not an instruction to search your disk. ([vanaapi.com](https://vanaapi.com/))

Create a Vana API account when you are ready to test this stage. Store the API key in an environment variable and keep the model identifier in configuration. Gemini 2.5 Flash is one option for the initial answer stage. Validate every returned source label against the labels you actually supplied.

§ Step 6: Record failures before expanding

Add twelve test questions: direct lookups, questions requiring two passages, paraphrases, and questions the documents cannot answer. Record retrieved IDs, answer text, source-label validity, latency, and reported token usage. Read the cited passages yourself; a valid label does not establish that the answer is supported.

If paraphrases repeatedly miss the right passage, consider semantic retrieval next. Embeddings represent text numerically and support ranking by relatedness rather than literal word overlap. You can retain the plain-file records while changing that ranking stage. ([developers.openai.com](https://developers.openai.com/api/docs/guides/embeddings?utm_source=openai))

Stop once each failed answer can be traced to selection, missing evidence, or generation. That is the useful first milestone: a small retrieval system whose behavior you can inspect.