Compound AI Systems
The shift from single model calls to systems: retrieval, code execution, verifiers, sampling, model programs, and feedback loops.
- AI progress often comes from systems around the model, not the model alone.
- The best system is rarely the most agentic. It is the one with the strongest feedback loop.
- "Ask the model harder" is one lever. Search, verification, tools, and state are the others.
Module 3: Compound AI Systems
This is the important shift: AI progress often comes from systems around the model.
Several real systems show what this shift looks like in practice.
Retrieval plus model is Google Search. It combines several pieces:
- retrieval
- ranking models
- a knowledge graph
- a generated summary layered on top
No single model call replicates that.
Model plus code execution shows up in a coding assistant that writes a function and then runs the test suite against it instead of asserting the code works.
Sampling plus scoring is how AlphaCode 2 operates. It generates on the order of a million candidate solutions per problem. It then uses a separate model to filter them down to a handful worth submitting.
Generator plus verifier means one model produces an answer while a second model checks it. Even basic retrieval augmented generation improves once a second pass verifies the output is actually supported by the retrieved source instead of trusting the first draft.
Planner plus executor splits the work. One model determines the steps. Another executes them.
Symbolic tools plus LLMs means delegating to a calculator or a database query or a unit test instead of asking the model to compute the answer itself. Code already does that reliably.
Model programs optimized against metrics replace hand tuning a prompt by feel. Evaluate it against a scored test set and let the results decide what works.
Build by solving one task three ways. Take a simple classification: is this support ticket a billing issue or not.
One shot classifies directly from the ticket text. It is the cheapest and fastest approach and the least reliable on ambiguous tickets.
Prompt plus tool call gives the model an account lookup tool before it classifies. That costs one extra call but produces a meaningful accuracy gain whenever account status actually matters to the decision.
Sample and vote generates five classifications and takes the majority. Five calls buy one or two extra correct answers. That is worth it only when an incorrect classification is costly.
Compare these across all three before deciding which one fits:
- quality
- cost
- latency
- failure modes
The best system is often not the most agentic one. It is the one with the strongest feedback loop. Asking the model harder is only one lever. Other levers worth pulling first include:
- search
- verification
- tools
- state
One task run three ways teaches more than a month of framework tourism.