Classic Orchestration
The non AI foundation: DAGs, queues, retries, state machines, durable execution, controllers, and the old systems lessons that still matter.
- Most AI orchestration problems are old systems problems wearing a new hat.
- The vocabulary that transfers: DAGs, queues, retries, idempotency, state machines, durable execution.
- If a normal workflow can solve it, do not make it an autonomous agent yet.
- If the run outlasts a chat message, persist state.
Module 1: Ordinary Orchestration Before AI
Start here because most AI orchestration problems are old systems problems wearing a new hat.
Learn the pieces before the acronyms pile up. DAGs and task dependencies. Queues and workers. Retries and backoff. Idempotency. State machines. Durable execution. Human approval gates. Controller loops. Observability. None of this is AI specific. It is the vocabulary of any system that has to keep working after something fails partway through.
Build a small example to make the ideas concrete. Draw a tiny workflow for a real task: research then draft then audit then publish. Add retries. Add a checkpoint. Add one human approval gate. Then ask the question that matters most. What happens if step 3 fails after step 2 already succeeded?
What Actually Happens When Step 3 Fails
Picture a workflow. Charge the customer. Email the receipt. Update the spreadsheet. Step 2 succeeds. Step 3 crashes. The system retries.
A retry with no protection charges the customer twice. That is the failure mode idempotency exists to prevent.
The standard fix is used by Stripe and most payment systems. Every operation carries an idempotency key. This key is a unique identifier for that one attempt. The server checks whether it has already processed that key. If so it returns the stored result instead of running the operation again. If not it runs the operation and stores the result under that key.
Durable execution engines like Temporal take this further. Every step gets written to a permanent event log as it completes. If the worker crashes partway through a step a new worker replays the log and resumes exactly where execution stopped. Even a pending retry survives the crash since the timer lives in the log and not in worker memory.
An AI agent calling a payment API or a ticketing system runs into this same failure mode. The agent is not special here. It needs the same idempotency keys and the same durable state any other caller would need.
A few rules of thumb are worth building into instinct. A normal workflow that can solve the problem does not need to be an autonomous agent yet. A step with side effects needs an idempotency key or must be reversible. A run that can last longer than a single chat message needs to persist state somewhere that survives a crash.
The tools will keep changing. The failure modes will not.