Flagship ML case study
Local RAG Toolkit
A configurable local document pipeline for asking grounded questions, inspecting sources, and comparing retrieval tradeoffs.
Grounded answers without sending private documents to a hosted service.
The toolkit is designed for experimenting with retrieval-augmented generation on a laptop while keeping source documents, embeddings, and generation local. The central product requirement is traceability: an answer should make it possible to inspect the supporting source.
Configurable components make tradeoffs inspectable.
- PDF, Markdown, and text ingestion feed a configurable chunking pipeline.
- ChromaDB stores vectorized chunks for local retrieval.
- Ollama-backed embedding and generation models can be changed without rewriting the application flow.
- A command-line workflow supports repeatable ingestion and querying.
- A FastAPI interface provides a lightweight browser-facing experience while keeping the core pipeline modular.
The next iteration turns quality into a reproducible experiment.
A focused evaluation lab is planned around a versioned document corpus and golden questions. It will compare a small number of chunking and retrieval configurations and report retrieval relevance, citation correctness, answer support, and latency.
- Recall@k and mean reciprocal rank for expected source retrieval.
- Citation/source attribution checks independent of prose quality.
- Latency by ingestion, retrieval, and generation stage.
- Error categories for missing evidence, wrong evidence, and unsupported synthesis.
Local does not automatically mean accurate.
Model choice, chunk boundaries, context limits, and document quality can all change the result. The case study will keep those constraints visible and will not use an LLM judge as the only source of evaluation truth.
Project evidence
Measured results will replace assumptions.
Evaluation results are intentionally omitted until they can be reproduced and documented with the relevant dataset, configuration, and hardware context.