Eval Lab

Problem

Teams change a prompt or a model and learn from their users that quality dropped.

Try it live

Pick a system's prompt, such as the LB-01 reply drafter. Edit it and run it on ten cases of its golden set, on Groq, Workers AI or both. Compare score, latency, tokens and calls with production, then open every case that changed.

Proves

LLM evaluation methodology, multi-provider routing decisions, and CI discipline.

Signal chain

1. golden set
2. run variants
3. rule-based graders
4. LLM judge
5. confidence intervals
6. CI gate

Stack

Flask async · asyncio fan-out · httpx · Postgres · GitHub Actions

Engineering highlights

• Flask with an asyncio pipeline of its own: a run is up to 40 model calls that mostly wait on the network, so one event loop sends them four at a time, and production's cached results cost none.
• Visitor runs use rule-based graders. The LLM judge runs nightly, calibrated against human labels before its scores count.
• In CI, a gate fails a change whose score falls below its committed baseline, and each night an advisor says which fallback models pass on each route.