Six bodies of work. Each card names the claim, the evidence, and the kind of role it maps to.
Evals engineeringBenchmarking
I build instruments that catch what harnesses get silently wrong
I read the open-source code I depend on closely enough to find real defects, then fix them upstream. In codebase-memory-mcp, fan-in queries ignored references written in markdown and shell; the fix is accepted in principle by the maintainer, open, not merged. In claude-mem, observation search results carried no date; the fix is routed into the maintainer’s search read-path plan, open, not merged. Status as of 15 September 2026; details in the note.
- Judge panels for spec verification carry Bayesian reliability priors per claim type, held-out seals so calibration can’t be peeked at pre-evaluation, and Kaplan-Meier retry budgets.
- Measured my own agent-router’s inter-rater ceiling (two LLM raters, Cohen κ 0.554; agreement on routing shape only 39%) and report every routing score against that ceiling, never against 100%.
Agent memorySystems design
Cortex — an agent memory system with write governance and an audit trail
271,360 lines of Python and 6,788 passing tests at v4.21.0, 43 tagged releases from 11 June to 10 September 2026, benchmark suites for LoCoMo, LongMemEval, MemoryAgentBench and bespoke probes. The parts other memory systems don’t have: an origin-allowlisted write gate so fetched web text can’t promote itself into long-term memory; supersession enforced at read time so a corrected fact can never be re-injected; and injection receipts: every auto-injected memory carries a marker the why tool resolves to what was actually in context, without claiming causation.
- Dual backend (SQLite / PostgreSQL+pgvector), lazy SQL-side decay, consolidation modeled on complementary learning systems.
- The A/B benchmark exposed a freshness-policy gap in my own serving path; root-caused to three unwired seams, fix plan public.
Code intelligenceRust
A code-graph MCP server that tells you when it might be wrong
83,724 lines of Rust, 1,200+ tests and 26 MCP tools at v0.11.1: tree-sitter indexing across 11 languages, cross-file resolution ladder (static → macro tables → stdlib → LSP), community detection, blast-radius queries. Its distinctive feature is epistemic honesty as API: impact results carry exact vs lower-bound markers with the named carriers of uncertainty, every heuristic edge has a confidence and resolution method, and a coverage sidecar records what the index did not parse.
- Pre-registered falsifiable benchmark, content-hashed corpus, committed before execution. It falsified my own recall hypothesis (0.83 vs substring 1.00), and that loss is published in the README.
- Supply chain: CodeQL, nightly fuzzing, cargo-audit/deny with empty ignore list, Sigstore attestation + CycloneDX SBOM in release CI.
Agent infraHarness engineering
Deterministic guardrails around a non-deterministic model
A plugin family for Claude Code: context management that checkpoints while headroom remains (the warn level blocks the stop once, persists a mechanical checkpoint plus a token-budgeted memory-writer pass, then resumes, so the hard cap becomes a formality); true session spend accounting that parses each subagent’s own transcript; burn-rate pacing in the statusline. Plus a 120-agent orchestration layer with worktree-per-agent isolation and identity injected at the spawn site so a subagent cannot forge its own memory scope.
- Every documented count is derived by a CI-checked command (
doc-count-check fails the build when docs and tree disagree).
- Gates degrade open but never silently: one-line stderr trace on every non-fatal path, a policy adopted after an audit found a silently disabled hook.
ProvenanceTechnical watch
An evidence-first technical watch for the agent ecosystem
A deterministic daily scout over a fleet of upstream repos (GitHub metadata only, no code execution, fails closed on missing evidence) feeding a five-state proof ladder: CLAIMED → IMPLEMENTED → VERIFIED → COMPARED → ACTIONABLE. A README can only ever establish CLAIMED; parity claims require a matched benchmark citation, enforced in CI.
- Dossiers keep retracted root causes in the tree, because the chronology of being wrong is itself evidence.
Shipped productCase study
Japonais 2027, the harness dogfooded
An offline, single-file PWA I use daily to work toward JLPT N3, built with my agent team under the same zetetic standards as everything above. I led it from an iPad by usage reports: I described the symptom, the agent reproduced it against real data before believing it, fixed the minimum, then locked the fix with a test replaying my exact report. Executable invariants gate every release; red output reads “do not ship” and nothing ships.
No content was invented: every kanji meaning, etymology and reading comes from a cited source (KANJIDIC2, CJKVI IDS, makemeahanzi, Janome), and a hole in a source stays a visible hole. Supabase Postgres behind row-level security on every row.
- When I disagreed with the app’s numbers, we didn’t debate. The agent built an audit screen crossing two independent measurements of the same work, and the screenshot settled it.
- The code stays private because the Irodori curriculum (Japan Foundation) isn’t redistributable. The method that built it is public: read the full case study.