Measure before you build: a sandbox for de-risking agentic AI

The agentic-AI ecosystem moves faster than anyone can validate it. New retrieval tricks, orchestration frameworks, and memory patterns appear weekly, each with a compelling demo and no evidence. ZenLab (Zen Agentic Lab) is my answer: a sandbox-only monorepo with one discipline — measure before you build. A technique graduates only when an experiment says it should, a promotion gate enforced by the repo’s structure rather than by good intentions.

Two results that show the method

Twenty-one topics across ten tracks sit over a shared core: hybrid retrieval, multi-agent orchestration, POMDP decision-making, off-policy learning, MLOps governance, and more. Two results illustrate how it works.

The hybrid retrieval track implements BM25 + dense retrieval fused with Reciprocal Rank Fusion (k=60) and a BGE cross-encoder reranker, with a labelled-query eval runner for chunking and rerank ablations — so retrieval choices are measured, not guessed. And a POMDP belief-agent demo quantifies the value of information on the classic Tiger problem, moving expected reward from −45 to +4 simply by reasoning about when to look before acting.

The safety floor — and the bug that proved it

Every agentic demo in the lab obeys one deterministic rule: LLM agents propose, plain code disposes. The flagship is a concurrent trading committee (asyncio.gather) whose verdicts pass through a deterministic veto arbiter. That design paid for itself: it reproduced a real fail-open audit bug — a parse failure silently approving a trade — and fixed it into a fail-closed ABSTAIN.

The failure mode that motivates the rule showed up on schedule. The rule caught it.

Always green, by design

The default path runs with zero API keys and no Docker. Heavy dependencies are lazy-imported, so the 43-test suite stays green in CI on every pull request — make check runs the tests, ruff lint, mypy --strict, a flagship compile, and five end-to-end demos. Reproducibility isn’t a footnote; it’s what makes the evidence trustworthy.

Governance as a track, not an afterthought

A governed lakehouse/MLOps track covers Delta Lake, Unity Catalog, MLflow model registry, the Feast feature store, drift monitoring, and model serving — taught OSS-offline with a mapping to a Databricks production setup. Alongside it runs an applied profit-gate ML pipeline: bronze/silver/gold data layers, leakage-safe feature contracts, calibrated classifiers, and shadow-only scoring with explicit champion approval. Models are never auto-promoted.

What the lab is for

Most teams adopt agentic techniques the way ZenLab refuses to: demo first, evidence never. Running the lab has produced a personal playbook of what actually survives measurement — deterministic safety floors, shadow deployment, champion/challenger gates — and, just as valuably, a list of things that looked exciting and didn’t.

The structure itself is the reusable part. Because every topic is a runnable, tested, offline lesson, the lab doubles as a curriculum: clone it, run an experiment, and let the numbers tell you whether the shiny new technique deserves a place in your architecture.