ContextIQ
Eval-first RAG for production knowledge systems: hybrid retrieval, citations, refusal, and a CI quality gate.
- Role
- Full Stack · RAG Engineering
- Type
- Eval-first RAG platform
- Client
- Personal · Open source
- Status
- Live
Outcome
Measured RAG end to end: dense-only recall@5 was 3.9% on the golden set; hybrid + rerank reached 89.3% context recall with 100% faithfulness and 100% refusal accuracy on unanswerable queries.
89.3%
Context recall
100%
Faithfulness
100%
Refusal accuracy
0%
Hallucination
Tutorial RAG stops at embed-and-cosine: exact terms (API names, quotas, error codes) get missed, models invent APIs when context is thin, and changing chunking silently tanks quality with no regression test.
Full pipeline: ingest structured docs, chunk with measured strategies, embed (BGE-small), retrieve dense + sparse → Reciprocal Rank Fusion → rerank, generate with citations or refuse, evaluate 75 golden queries in CI against floors and a baseline.
- Frontend
- Next.js App Router on Vercel — landing, streaming chat (/chat?demo=1), eval and trace dashboards.
- Backend
- Python Starlette API on Railway — ingest, retrieve, generate, trace stages.
- APIs
- SSE streaming for chat; JSON for eval metrics and job status.
- Databases
- Optional pgvector for embeddings; corpus and indexes versioned with the repo shape.
- Infrastructure
- Docker Compose for local full stack; split deploy Vercel + Railway in production.
- Auth
- Optional API key; rate limits and input caps on public demo routes.
- Deployment
- Vercel (frontend) + Railway (API). Local path is $0; Bedrock Haiku cost surfaced in UI when used.
- Hybrid retrieval (dense + BM25) with RRF fusion and reranking
- Grounded answers with source citations; refusal on weak context or injection-shaped queries
- Golden eval suite (75 queries) with context recall, faithfulness, refusal, and hallucination metrics
- Product surfaces: /chat demo, /eval dashboard, /failures, /traces, /architecture
- Streaming SSE chat; optional OpenAI-compatible and local (Ollama) generation paths
- API auth, rate limits, input caps; per-stage latency and cost in traces
- Docker Compose locally; Vercel (UI) + Railway (Python API) in production
Challenge
Dense-only vector search missed exact strings in technical docs.
Decision
Measure hybrid retrieval instead of guessing.
Implementation
BM25 + dense embeddings, RRF merge, then rerank. Tracked recall@5 on the same answerable golden set.
Result
Recall@5 rose from 3.9% (dense-only) to 89.3% context recall with hybrid + rerank on that suite.
Challenge
Models answered when they should refuse.
Decision
Refusal is a first-class outcome, tested in eval.
Implementation
Nine unanswerable golden queries; generation path refuses when context is insufficient.
Result
100% refusal accuracy on those nine; 0% hallucination rate on the published suite.
Challenge
Retrieval regressions do not show up in unit tests.
Decision
CI eval gate on committed metrics, separate from TypeScript/Python unit tests.
Implementation
GitHub Actions runs eval/golden.jsonl; PR fails if context recall drops below 0.85 vs baseline.
Result
Chunking and retrieval changes cannot merge without measured proof.
Eval before “make it smarter”
golden.jsonl and baseline metrics exist before tuning prompts or chunk sizes.
Alternative. Eyeball a few chat answers after each change.
Tradeoff. Upfront eval work; regressions become visible instead of silent.
Hybrid + rerank over dense-only
This corpus is full of exact tokens embeddings blur.
Alternative. Single embedding index.
Tradeoff. More moving parts; measured 3.9% → 89.3% recall story on the same set.
Extractive / grounded generation for faithfulness metrics
100% faithfulness on the suite is on answers tied to retrieved context—not a frontier model free-writing.
Alternative. Chatty LLM with post-hoc citation.
Tradeoff. Faithfulness number is honest but not a claim about GPT-4-level robustness.
- Problem
- Support and docs Q&A fails when retrieval misses exact terms and models invent APIs.
- Model / provider
- BGE-small embeddings; OpenAI-compatible / Ollama / optional Bedrock Haiku for generation
- Why this model
- Small embedding model for cost; generation path chosen per environment (local $0 vs hosted).
- Architecture
- Ingest → chunk → embed → hybrid retrieve → RRF → rerank → cite or refuse → trace.
- Retrieval
- Dense + BM25 sparse, reciprocal rank fusion, rerank; structural chunking default in production.
- Validation
- Golden eval in CI; refusal and faithfulness scored separately from unit tests.
- Failure handling
- Refuse when ungrounded; /failures page documents remaining miss cases.
- Rate limiting
- Public demo rate limits and optional API key.
- Cost controls
- Local stack $0; Bedrock Haiku pricing shown in UI when that path is enabled.
- Evaluation
- 75-query golden set; GitHub Actions fails PRs below recall floor 0.85 vs baseline.
- Human in the loop
- Eval dashboard and failure browser for operator review—not fully autonomous support.
- BuiltEnd-to-end RAG pipeline, hybrid retrieval, and eval suite
- BuiltNext.js product surfaces (chat, eval, failures, traces, architecture)
- BuiltPython SSE API, Docker Compose, CI eval gate, Vercel + Railway deploy
- Context precision 41.6% on the suite — next lever is tighter chunks / rerank / citation packing.
- Demo auth is API key or open rate limit — not multi-tenant SaaS.
Published suite (n=75): 89.3% context recall, 100% faithfulness (extractive path), 100% refusal accuracy (9/9 unanswerable), 0% hallucination. Context precision 41.6% — documented as the open quality lever.