Hunter Brennick AI Systems & Advisory ↗
AI Systems Orchestration
Part III · Applying And Deciding/Chapter 10
10

Gaps And Project Ideas

From learning map to possible builds: run ledgers, failure corpora, tool benches, model routers, promotion ladders, and orchestration labs.

8 min read1 figures
The gist
  • The fastest way to get better at AI orchestration is to collect failures.
  • Backbone builds: a shared run ledger, a failure corpus, a tool fitness bench.
  • Workflows graduate up the runtime ladder on pain, not fashion.
FIVE GAPS · EIGHT BUILDS THE GAPS shared run ledger one schema for every run in every system failure corpus every bad run becomes an asset tool fitness tests judge whether tools are agent friendly routing scorecards which model earns each task promotion ladder when a workflow graduates to a runtime THE BUILDS orchestration lab one task through every pattern black box recorder a standard trace bundle for any run workflow builder rough idea to artifact contract review swarm artifact mediated code review tool fitness bench score tool ergonomics before agents suffer router playground same task through many model routes improvement loop human gated system upgrades ops room operations for agent teammates the fastest way to get better is to collect failures · start with the build that creates reusable evidence
FIG 10.1The opportunity map: five gaps on the left, eight possible builds on the right.

Things You Might Be Missing

These are not accusations. They are opportunities worth watching.

1. A Shared Agent Run Ledger

It's common to end up with run metadata in one system. Command telemetry lives in another. Request and step metrics live in a third. There is no shared view across them.

Potential idea:

agent-run-ledger
  run_id
  system
  workflow_or_service
  model
  tools_called
  artifacts_written
  trace_path
  eval_result
  cost
  latency
  human_interventions
  failure_mode

This could become the backbone for learning across all three repos.

2. A Failure Corpus

The fastest way to get better at AI orchestration is to collect failures.

Potential idea:

failures/
  task.md
  expected.md
  actual.md
  trace.json
  tool_outputs/
  diagnosis.md
  fixed_by.md

Every bad run becomes training material. It becomes eval material. It becomes design material.

3. Tool Fitness Tests

You have MCP and tool surfaces. The next step is judging whether tools are agent-friendly.

Potential idea:

tool-fitness/<tool-name>/
  schema-review.md
  eval-cases.json
  latency.md
  failure-modes.md
  prompt-injection-notes.md
Start with
  • Zendesk MCP
  • GitHub
  • Linear
  • local file and codebase tools

4. Model Routing Scorecards

Not every task needs the strongest model.

Potential idea:

model-router-scorecard.md
  task type
  cheap model pass rate
  strong model pass rate
  cost delta
  latency delta
  recommended default

This is where AI becomes economically scalable.

5. A Workflow-To-Runtime Promotion Ladder

You need a clean rule for when something graduates from prompt/workflow to runner/runtime.

Potential ladder:

Level 1: prose workflow
Level 2: artifact-mediated workflow
Level 3: scripted deterministic runner
Level 4: graph runner with checkpoints
Level 5: scheduled/long-running service
Level 6: self-improving loop with evals and memory

The promotion rule should be pain-based. Only move up when the lower level cannot handle:

  • retries
  • parallelism
  • state
  • evals
  • latency

Potential Projects

These are ideas worth keeping in a scratchpad.

1. AI Orchestration Lab

A tiny local lab that can run the same task through multiple orchestration patterns:

one-shot
single-agent-with-tools
planner-executor
parallel specialists
generator-verifier
graph runtime
human-in-loop

Output:

runs/<task>/<pattern>/
  input.md
  trace.json
  artifacts/
  eval.md
  cost-latency.json
  notes.md

Why it is useful: it stops you from debating patterns abstractly. You see which one wins on real tasks.

2. Agent Black Box Recorder

A standard trace/artifact bundle for any agent run.

Capture:

prompt or instructions version
model
tool calls
files touched
artifacts written
approvals requested
eval score
final decision

Why it is useful: debugging gets easier. Evals get easier. Trust gets easier.

3. Workflow Builder

A workflow that turns a rough idea into:

workflow.md
manifest.yaml
artifact contract
run metadata contract
eval plan
sync plan

Why it is useful: it makes new workflows less ad hoc. It teaches the system to build itself carefully.

4. PR Review Swarm

Artifact-mediated review:

recon.md
risk-review.md
verification.md
final-report.md
run-meta.json

Why it is useful: it is a clean test for multi-agent value. It does not require a runtime engine first.

5. Tool Fitness Bench

A benchmark for MCP/tool ergonomics.

Test whether the model can:

  • discover the right tool
  • call it correctly
  • use small results
  • recover from failures
  • avoid unsafe calls

Why it is useful: tool design is a multiplier. Bad tools quietly destroy agent performance.

6. Model Router Playground

Send the same tasks through different models and routes.

Measure
  • pass rate
  • cost
  • latency
  • tool call quality
  • hallucination rate
  • the need for human correction

Why it is useful: it builds instinct for when to spend reasoning.

7. Self-Improving Workflow Loop

Not magical self-improvement. A grounded loop:

run workflow
  -> collect trace
  -> evaluate failure
  -> propose prompt/tool/workflow patch
  -> human approves
  -> rerun eval bank
  -> promote if better

Why it is useful: it turns failures into system upgrades. It keeps humans in the promotion path.

8. AI Staff Ops Room

A dashboard-ish concept for agent work would show:

  • what is running
  • what is blocked
  • what needs approval
  • what failed
  • what got expensive
  • what learned something
  • what should be retried

Why it is useful: AI systems that become teammates need operations. Chat threads alone are not enough.

Next chapterChapter 11 · Fast Path And Reading List The compressed path: build enough taste quickly that you can make good system choices this week.