RAG Hit Rate Lab

Free Review to certify

Build an eval set, measure Hit@k, then improve retrieval with chunking / query rewrite / hybrid — numbers before vibes.

Skills you'll learn

Hit@k evaluation Corpus chunking Query rewrite Error analysis RAG ablations

~140 min · +70 pts

SCENARIO: CareerAck wants a help/search assistant over paths, virtual projects, AI Breakthrough FAQ, and job-skill notes.
Stakeholders ask "does RAG work?" — you answer with Hit@k, not demos alone.

Starter: /assets/templates/rag-hit-lab/
  - corpus/ (chunked docs with ids)
  - eval/questions.jsonl (gold chunk ids)
  - scripts/eval_hit.py (offline keyword baseline)
  - REPORT.md + RUBRIC.md

Must deliver:
1) Baseline Hit@1 / Hit@3 / Hit@5 on the provided eval set
2) At least TWO interventions (e.g. chunking change, query rewrite, hybrid/keyword boost, metadata filter)
3) After metrics table + delta vs baseline
4) Error analysis: ≥5 misses with why (wrong chunk / query / corpus gap)
5) One-page recommendation: what to ship next for production RAG

Offline-first: keyword/BM25-style scoring is enough. Optional embeddings/DeepSeek are bonus only.
Submit: report (link or paste) + eval outputs + reflection (120+ chars).

Environment setup (start here)

Offline RAG eval. Copy /assets/templates/rag-hit-lab/, run eval_hit.py (stdlib only), improve Hit@k with ≥2 interventions, fill REPORT.md.

Python required

Pick your OS and follow from step 1:

Windows macOS

1) Install Python 3.10+ from https://www.python.org/downloads/ (tick Add to PATH).
2) Copy folder public/assets/templates/rag-hit-lab to e.g. C:\rag-hit-lab
3) Open PowerShell: cd C:\rag-hit-lab
4) python scripts\eval_hit.py --mode baseline
5) Also try --mode rewrite, title_boost, hybrid
6) Fill REPORT.md with metrics, error analysis, recommendation; submit on CareerAck.

1) python3 --version (install via brew/python.org if missing).
2) Copy rag-hit-lab starter to e.g. ~/Documents/rag-hit-lab
3) cd ~/Documents/rag-hit-lab
4) python3 scripts/eval_hit.py --mode baseline
5) Run rewrite / title_boost / hybrid; compare Hit@k
6) Complete REPORT.md + RUBRIC self-score; submit.

How do I know it worked?

outputs/baseline.json exists with hit@1/3/5 summary; REPORT has before/after table.

Common issues

python not found → use python3 or reinstall with PATH. No corpus → keep corpus/*.md next to scripts/. Hit@k unchanged → edit REWRITE_RULES or re-chunk; do not only change LLM.