Hunter Brennick AI Systems & Advisory ↗
AI Systems Orchestration
Part IV · Operating Agentic Systems/Chapter 13
13

Learning Loops And Continuous Improvement

How an AI system gets better after each run without pretending the agent should blindly rewrite itself.

8 min or 35 min detailed10 figures
The gist
  • The loop that compounds: observe → diagnose → propose → sandbox → gate → apply → verify → remember.
  • MAPE-K is the older name. The new part is agents helping with Analyze and Plan.
  • For real engineering work, Level 4 human gated apply is the sweet spot.
  • Improvement without evidence is opinion. Autonomy without gates is risk.

The Missing Operating Loop

Earlier chapters covered
  • models
  • tools
  • workflows
  • graphs
  • protocols
  • evals
  • stack choices

This chapter is about the loop that makes all of that compound:

observe what happened
  -> diagnose what went wrong or slowed down
  -> propose a focused improvement
  -> test it somewhere safe
  -> ask for approval when consequences matter
  -> apply the change
  -> verify it worked
  -> remember the lesson

That is the operating layer. It is the difference between a clever AI demo and a system that actually improves.

Terminology First

Use continuous improvement for the broad practice. That term is not new. It comes from quality engineering and Lean and Kaizen and process-improvement traditions.

Use AI-assisted continuous improvement when agents help observe or diagnose or propose or test or document improvements.

Use agentic continuous improvement when the system itself performs meaningful parts of the loop with tools and memory and artifacts and approval gates.

Do not present labels like CI^2 as established industry terminology. They can be private shorthand. The defensible language is:

  • continuous improvement
  • PDSA/PDCA
  • Kaizen
  • DevOps
  • SRE
  • platform engineering
  • AIOps
  • MLOps
  • self-adaptive systems
  • MAPE-K

The Simple Loop

THE OPERATING LOOP observetraces · evals · cost diagnoseevidence · not vibes proposethe smallest fix sandboxtest somewhere safe gateask when it matters applyto the canonical source verifystrongest local checks rememberstart smarter next run the important part is not that AI does every step · it is that every step leaves evidence
FIG 13.1The operating loop: observe through remember with evidence at every step.

For AI systems each step has a concrete shape:

Step What it means in an AI system
Observe Collect run artifacts, traces, eval results, failures, cost, latency, and user corrections.
Diagnose Explain the likely cause using evidence, not vibes.
Propose Draft a small improvement: prompt, workflow, tool schema, eval, guardrail, or documentation change.
Sandbox Test the change against fake data, fake home directories, or isolated runtime state.
Gate Ask a human before risky writes, sends, deletes, installs, or policy changes.
Apply Make the approved change in the canonical source of truth.
Verify Run the strongest relevant local checks.
Remember Save the result so the next run starts smarter.

The important part is not that AI does every step. The important part is that every step leaves evidence.

MAPE-K Is The Older Name

A useful older model is MAPE-K. It stands for Monitor and Analyze and Plan and Execute and Knowledge.

It comes from autonomic computing and self-adaptive systems. It maps surprisingly well to agentic systems.

MAPE-K · THE OLDER NAME monitor traces · validation · failures · cost analyze root cause · patterns · risks plan ranked fixes · experiments · tasks execute approved changes only knowledge sources · policies · modelsmemory · artifacts what happened what to do do it · with gates evidence in the knowledge in the middle is shared state · not the model's token window and not memory alone
FIG 13.2MAPE-K: the older name for the same loop, with shared knowledge in the middle.

Knowledge in MAPE-K means the shared information the loop uses:

  • current sources
  • policies
  • system models
  • evaluated artifacts
  • retained lessons

It is not the model's token window. It is not memory alone.

The new part is that agents can now help with Analyze and Plan in a way that used to require a person manually reading every artifact.

The dangerous part is Execute. That is where human gates matter.

The Safety Ladder

Not every improvement loop deserves the same autonomy.

Level 1: Observe only
Level 2: Recommend
Level 3: Sandbox
Level 4: Human-gated apply
Level 5: Autonomous apply for narrow, reversible, proven changes

For real engineering work Level 4 is the sweet spot.

THE SAFETY LADDER L1 · observe only reports · dashboards · health checks L2 · recommend ranked options · task drafts L3 · sandbox candidate patches in isolated roots L4 · human gated apply approved patches · PRs · syncs the sweet spot L5 · autonomous apply only narrow · reversible · proven changes autonomy is earned level by level not every improvement loop deserves the same autonomy · most real engineering work should live at L4
FIG 13.3The safety ladder: not every improvement loop deserves the same autonomy.
The system can
  • notice patterns
  • write a proposal
  • build a candidate patch
  • run checks

A human still approves meaningful changes before they touch canonical source or external systems or user-visible outputs.

How This Fits A Real Platform

A platform built around authored workflows is often already close to this loop. Workflows define repeatable procedures. Generated client surfaces make those workflows available. Run artifacts preserve evidence. Validation and preflight checks catch drift. Research and ideation workflows turn unclear questions into grounded options. Memory keeps prior lessons from disappearing.

The next move is not a giant autonomous runtime. The next move is a better improvement loop:

run
  -> artifact
  -> eval or review
  -> correction tag
  -> improvement proposal
  -> sandbox
  -> approval
  -> workflow-sync or patch

That keeps the system ambitious without making it sloppy.

What To Practice

Take one repeated failure and turn it into an improvement loop.

Good candidates include
  • the agent using the wrong term
  • the agent reading too broadly
  • a workflow missing a source coverage section
  • a tool call failing for the same reason twice
  • a generated install drifting from canonical source
  • a report that is useful but too expensive in tokens

For each one write:

failure:
evidence:
likely cause:
smallest fix:
eval/check:
risk:
approval needed:

That is the smallest useful continuous-improvement artifact.

What To Read With This Chapter

Chapter 05 · Context, Tools, And Protocolsthis book↗Chapter 07 · Evals, Guardrails, And Safetythis book↗Chapter 10 · Gaps And Project Ideasthis book↗Chapter 13b · Detailed Versionthis book↗Chapter 14 · Measuring AI-Assisted Software Deliverythis book↗

The short version:

Improvement without evidence becomes opinion. Autonomy without gates becomes risk. The useful middle is evidence-driven and human-gated improvement.

There is more

This chapter also exists as a long form deep dive: the timeline, the vocabulary, seven large scale organization examples, and a concrete first version to build.

This is the detailed companion to Chapter 13. The shorter chapter gives the clean operating model. This version keeps the longer article with examples, terminology, field context, and real world case studies.

Continuous Improvement Is The Next Software Feedback Loop

A practical guide to the AI-assisted wave after CI/CD.

The Simple Idea

Continuous integration asks: did this change break the build?

Continuous deployment asks: can this change safely reach users?

Continuous improvement asks: what should we make better next, based on evidence?

That last question is not new. Factories, quality teams, and operations teams have been asking versions of it for almost a century. What is new is that software systems can now inspect themselves, read their own artifacts, compare their own workflows, propose changes, run tests, draft pull requests, and explain why a change is worth making.

That is the shift.

Terminology correction: "continuous improvement" is not new, and it is not a new category invented by AI. The best-practice framing is:

  • Continuous improvement for the broad management and quality practice.
  • PDSA/PDCA, Kaizen, Lean, and quality engineering for the older process-improvement lineage.
  • DevOps, SRE, platform engineering, and value stream management for the software delivery lineage.
  • AIOps, MLOps, self-adaptive systems, and autonomic computing for the automated operations and adaptive-systems lineage.
  • AI-assisted continuous improvement or agentic continuous improvement for the newer AI-assisted version, with the caveat that this is emerging terminology, not yet a single standardized discipline.

Not "the AI takes over." More like:

the system gets a nervous system, a memory, and a careful assistant.

For a platform like this, the natural version is not a fully autonomous patch machine. It is an evidence loop. The loop watches workflows, MCP configs, generated installs, reports, validations, tool versions, and repeated friction points. It turns that into ranked improvement proposals.

A Quick Timeline

Continuous improvement is old. AI-assisted continuous improvement is new.

Era What emerged Simple meaning
1920s-1930s Statistical quality control, Shewhart cycles Measure the process, improve the process.
1950s Toyota Production System, Kaizen Everyone improves the system continuously.
1980s Kaizen popularized globally Small improvements become management practice.
1990s-2000s Agile, XP, continuous integration Integrate code often so problems surface earlier.
2010 Continuous delivery becomes mainstream Keep software always releasable.
2010s DevOps, SRE, observability, error budgets Use production signals to balance speed and reliability.
Late 2010s-2020s MLOps, CI/CD/CT Models need continuous training, monitoring, and retraining.
2023-now Agentic software engineering AI agents can read, test, patch, review, and synthesize across code and tools.

So the new thing is not "continuous improvement" as a phrase. The new thing is continuous improvement with software agents that can actually do a meaningful part of the loop.

Terminology Check

Some of the casual phrasing around this topic is useful for brainstorming. It should not ship unchanged in a technical document.

Casual phrase Better term Why
"tier-1 software companies" large-scale software organizations, major technology companies, or hyperscale technology companies "Tier-1" is understandable but not a standard software engineering category. It can mean different things in venture, vendor management, recruiting, or personal taste.
"the next wave after CI/CD" AI-assisted continuous improvement layered on CI/CD, SRE, platform engineering, and MLOps Continuous improvement predates CI/CD. AI is changing who can perform parts of the loop, not inventing the loop.
"self-improving agent platform" human-gated self-adaptive platform engineering More precise. It says the platform can observe, analyze, and propose changes, but humans still gate risky execution.
"agent auto-fixes itself" closed-loop remediation with human-in-the-loop approval "Auto-fix" hides the important safety model: evidence, sandboxing, validation, and approval.
"CI^2" internal shorthand only CI^2 is not established industry terminology. It is fine as a private label if you define it, but do not present it as a known field.
"AI-native continuous improvement" AI-assisted continuous improvement or agentic continuous improvement "AI-native" is common product language, but "AI-assisted" is more precise when humans still own decisions.

The clean version for a technical document reads like this:

We are exploring AI-assisted continuous improvement for a local internal developer platform, using a human-gated MAPE-K-style feedback loop.

That sounds less flashy, but it is much more defensible.

Best-Practice Vocabulary

These are the terms worth using around this work.

Term Plain meaning What it looks like in practice
Feedback loop A system observes an outcome and uses that information to adjust future behavior. Reports, validations, audits, and memory should change what the next run recommends.
PDSA / PDCA Plan-Do-Study-Act or Plan-Do-Check-Act; a classic improvement cycle. Propose a workflow change, test it in a sandbox, inspect validation, then decide whether to apply.
Kaizen Continuous small improvements, usually from people closest to the work. Fix repeated agent mistakes, prompt drift, stale aliases, and validation gaps before they become large rewrites.
Value stream management Looking at the full path from idea to delivered value and finding delay, waste, and rework. Track where agent work stalls: source discovery, MCP auth, generated installs, validation, memory, or review.
DORA metrics Delivery-performance metrics. The current DORA guide lists change lead time, deployment frequency, failed deployment recovery time, change fail rate, and deployment rework rate. Useful once a platform starts tracking how long changes take and how often they cause failed validation or rollback.
Progressive delivery Releasing changes gradually with controls such as feature flags, canaries, and health checks. Apply generated install or MCP changes to one client or fake-home lab before touching every live client.
SLO / error budget A reliability target and the amount of acceptable unreliability left. Define acceptable failure rates for workflow validation, smoke checks, or generated install drift.
Toil Repetitive manual work that could be automated or designed away. Repeated correction of the same agent mistake is toil. Repeated MCP diagnosis is toil.
Internal developer platform A platform that gives developers paved, supported ways to build and ship. A control plane repo like this acts like a local internal developer platform for agents and workflows.
Golden path / paved road The recommended, supported path through a platform. Canonical workflows, generated skills, validators, and run metadata are the paved roads.
Guardrail A safety constraint that prevents unsafe behavior without blocking all progress. Schema checks, smoke tests, approval gates, sandbox roots, and explicit write scopes.
Policy-as-code Rules encoded in executable checks rather than tribal knowledge. "Do not edit generated skills directly" should be enforced by validation where possible, not only remembered.
DevSecOps Integrating secure-development practices into the software delivery lifecycle. Security checks, provenance, dependency review, and write-scope rules should be part of the improvement loop, not a late manual audit.
AIOps Applying AI and automation to operations signals such as logs, metrics, events, and incidents. Useful if the platform starts analyzing failures, tool health, and runtime drift across many runs.
MLOps / continuous training Operational practices for ML systems, including model monitoring, retraining, and drift detection. Relevant by analogy: prompts, workflow instructions, and evals also drift and need monitoring.
Self-adaptive systems Systems that change behavior in response to monitored conditions. The safe version is recommend-or-sandbox first, not immediate self-modification.
MAPE-K Monitor, Analyze, Plan, Execute, Knowledge. The clean architecture model for a self-improvement radar.
Human-in-the-loop A person participates before the system proceeds. Meaningful changes get approved before they touch canonical files.
Human-on-the-loop A person supervises automation and can intervene. Useful later for low-risk recurring checks that run automatically but report exceptions.
Evals Repeatable tests for AI behavior, quality, and regressions. Agent workflows need evals for over-scoping, missing citations, wrong tool routing, or unsafe edits.
Drift detection Detecting when actual behavior diverges from intended behavior. Compare generated installs, MCP availability, run metadata, and workflow outputs against the canonical contract.

The Conveyor Belt Analogy

Think of a factory conveyor belt.

CI checks whether each part fits.

CD moves the part to the customer safely.

Continuous improvement watches the whole factory and asks:

  • Why do parts jam at station three?
  • Why does this inspection fail every Friday?
  • Why do operators keep writing the same workaround on sticky notes?
  • Why does one tool create cleanup work for another team?
  • Which tiny process change would remove the most repeated pain?

Now replace the factory with a local agent system.

  • Workflows are the stations.
  • MCP configs are the wiring.
  • Generated client installs are the packaging.
  • Run artifacts are the inspection records.
  • Memory is the old maintenance notebook.
  • Agents are the mechanics who can inspect, propose, and sometimes repair.

That is the shape.

The Core Loop

The simplest model is:

THE OPERATING LOOP observetraces · evals · cost diagnoseevidence · not vibes proposethe smallest fix sandboxtest somewhere safe gateask when it matters applyto the canonical source verifystrongest local checks rememberstart smarter next run the important part is not that AI does every step · it is that every step leaves evidence
FIG 13.4The core loop: observe, diagnose, propose, sandbox, gate, apply, verify, remember.

For software, the loop should be deliberately boring:

  • Observe real signals.
  • Diagnose with evidence.
  • Propose ranked options.
  • Sandbox risky changes.
  • Ask for human approval.
  • Apply only approved changes.
  • Verify with the strongest local checks.
  • Save the result so the next agent does not start from scratch.

The key is that "continuous" does not mean "reckless." It means the system keeps looking.

The Academic Name: MAPE-K

There is an older field called autonomic computing. IBM and others worked on self-managing systems in the early 2000s. A common model is MAPE-K:

  • Monitor
  • Analyze
  • Plan
  • Execute
  • Knowledge

That maps cleanly onto agentic systems:

MAPE-K · THE OLDER NAME monitor traces · validation · failures · cost analyze root cause · patterns · risks plan ranked fixes · experiments · tasks execute approved changes only knowledge sources · policies · modelsmemory · artifacts what happened what to do do it · with gates evidence in the knowledge in the middle is shared state · not the model's token window and not memory alone
FIG 13.5MAPE-K with a real platform as the executed surface.

Here, Knowledge is the shared base used by the control loop. It can include current sources, policies, system models, evaluated artifacts, and retained lessons. It is broader than memory and separate from the temporary context assembled for one model call.

The modern twist is that the Analyze and Plan steps can now be done by agents that read code, docs, PRs, release notes, logs, reports, and local state.

The dangerous twist is that Execute is tempting.

That is where gates matter most.

The Safety Ladder

Not all automation has the same risk.

THE SAFETY LADDER L1 · observe only reports · dashboards · health checks L2 · recommend ranked options · task drafts L3 · sandbox candidate patches in isolated roots L4 · human gated apply approved patches · PRs · syncs the sweet spot L5 · autonomous apply only narrow · reversible · proven changes autonomy is earned level by level not every improvement loop deserves the same autonomy · most real engineering work should live at L4
FIG 13.6Five levels of loop autonomy from observe only to narrow autonomous apply.

In most setups, the sweet spot is Levels 1 through 4.

Level 5 should be rare. It should be limited to tiny, reversible, machine-verifiable changes. Think "update a generated report index" or "refresh a cache," not "rewrite the MCP profile" or "edit all workflow instructions."

What This Looks Like In Practice

A platform like this often already has the bones of this in place:

  • a canonical workflow source
  • generated client surfaces
  • run artifacts and state
  • validation commands
  • smoke checks
  • golden checks
  • release radar
  • memory from prior runs
  • a stable artifact convention
  • multiple agent clients

That is more than a pile of scripts. It is a control plane.

The next step is a recurring self-improvement radar:

RADAR SHAPED LOOP schedule run checks scan artifacts check releases summarize drift rankideas write report ask a humanbefore any change is applied approved lessons feed the next cycle
FIG 13.7A release radar shaped loop: schedule, scan, summarize, rank, report, then ask.
The radar should answer
  • What is failing?
  • What is stale?
  • What has drifted?
  • What got slower or noisier?
  • What new upstream capability matters?
  • What workflows keep causing repeated corrections?
  • What should be improved first?

It should not silently edit canonical files.

The Three Kinds Of Improvement

Continuous improvement in software is not one thing. It splits into three practical buckets.

1. Product Improvement

The thing users see gets better.

Examples
cleaner onboardingfaster searchbetter error statesfewer broken flowsaccessibility fixesmore useful defaults

AI helps by reading feedback, logs, analytics, session traces, tickets, and support threads.

2. Engineering System Improvement

The way engineers ship work gets better.

Examples
faster testsless flaky CIsmaller PRsbetter code review guidancestale flag cleanupsafer deploy gatesautomatic dependency triage

AI helps by finding patterns across diffs, build failures, incidents, and old reports.

3. Agent/Workflow Improvement

The agents themselves get better.

Examples
better workflow instructionsbetter run metadatabetter validation scriptsbetter tool routingfewer repeated mistakesbetter memory retrievalless token waste

This is the bucket a platform like this lives in.

The New Wave

The new wave is not just "AI writes code."

That is the flashy part, but not the whole system.

The bigger pattern is:

THE NEW WAVE FLYWHEEL AI reads more contexttraces · artifacts · sources Proposes better changessmaller · targeted · explained Runs more verificationtests · evals · sandboxes Leaves better evidencereports · diffs · provenance Humans approve higherlevel decisions with confidence each turn buys the next
FIG 13.8The new wave flywheel: better evidence buys higher level human approval.

The best teams will not simply ask, "How much code can AI generate?"

They will ask
  • Can AI reduce the time from signal to diagnosis?
  • Can AI turn repeated pain into good backlog items?
  • Can AI remove stale code safely?
  • Can AI write better tests around risky changes?
  • Can AI compare current behavior to intended contracts?
  • Can AI help us learn from every incident, PR, and support case?

That is a richer idea than code generation.

It is operational learning.

In best-practice terms, this is a combination of:

  • continuous improvement from quality engineering
  • continuous delivery and progressive delivery from DevOps
  • toil reduction and error-budget thinking from SRE
  • internal developer platforms and golden paths from platform engineering
  • closed-loop remediation from operations automation
  • AIOps and MLOps where AI systems and operational signals are involved
  • self-adaptive systems / MAPE-K from autonomic computing
  • agentic software engineering from the current AI coding-agent wave

That is why there is not one perfect field label. The precise version is cross-disciplinary.

Seven Large-Scale Software Organization Examples

These are examples from large-scale software organizations. This guide avoids the term "tier-1" because it is imprecise.

These examples are not all the same. Some are established automation, some are AI agents, some are safe deployment, and some are MLOps. Together they show where the industry is heading.

Important boundary: these public examples prove that the patterns exist. They do not prove that each company runs the same integrated self-improvement loop internally. Any connection back to a platform of this kind is an analogy and a design inference. It is not a claim about a company's private architecture.

1. Google: SRE, Error Budgets, Toil Reduction, and MLOps

Google's SRE practice is one of the cleanest examples of continuous improvement in software operations.

The error budget idea is simple: do not argue endlessly about speed versus reliability. Define an SLO, measure real reliability, and let the remaining error budget guide how much risk the team can take.

Google also treats toil as a problem to engineer away. If a repetitive operations task keeps coming back, it should become automation, a better system, or a design change.

On the ML side, Google Cloud's MLOps guidance extends CI/CD into CI/CD/CT, where CT means continuous training. That matters because models are not done when deployed. Data changes, behavior drifts, and the system needs monitoring and retraining loops.

Why it matters:

A useful equivalent of an error budget is "how much agent slop is tolerable before freezing new workflow additions and improving validation." A useful toil signal is "how many times the same class of mistake needed correcting."

2. Microsoft and GitHub: Coding Agents Plus Safe Deployment

GitHub Copilot coding agent can work in the background, take on GitHub issues, create branches, raise pull requests, and request review when done. That is a direct example of AI entering the software change loop.

Microsoft also publishes safe deployment guidance around progressive exposure, health checks, feature flags, circuit breakers, and stopping rollout when health changes. That is the other half of the story: agents can move faster, so deployment systems need stronger brakes.

Why it matters:

The lesson is not "let Copilot merge everything." The lesson is "agents can prepare work. The platform still needs review, health checks, and rollout policy."

3. AWS: Amazon Q Developer Agents

Amazon Q Developer can help explain code, generate code, write tests, scan for vulnerabilities, upgrade code, and perform agent-style feature development. In GitHub preview, Amazon Q can respond to an issue or slash command, implement a feature or bug fix, open a pull request, and iterate based on PR feedback.

This is AI-assisted continuous improvement moving into the issue and review loop.

Why it matters:

A platform like this can borrow the shape without copying the product. An issue comes in. An agent researches it. An agent proposes a fix. An agent patches it in a sandbox. A human reviews it. Validation decides whether it can proceed.

4. OpenAI: Codex as Background Software Engineering Agent

OpenAI Codex can run coding tasks in isolated cloud environments, work in parallel, answer codebase questions, fix bugs, run tests, draft PRs, and perform code review. The important design point is isolation: each task gets its own sandbox.

That is exactly the right mental model for platform experiments.

Why it matters:

A local test version should use a fake HOME and fake XDG_* roots before touching real generated installs or client configs. Let the agent try things. Just make the first blast radius artificial.

5. Meta: Sapienz, SapFix, and Infer

Meta has been doing automated software improvement for years.

Sapienz automatically designs and runs tests at scale. SapFix can generate candidate patches for bugs found by tools like Sapienz and Infer, validate those patches, and send them to human reviewers. Infer is Meta's static analysis tool for finding deeper bugs before shipping.

The key detail: SapFix was not designed to deploy its own fixes without people. It generated and validated candidate patches, then routed them to humans.

Why it matters:

This is the pattern worth copying. Autonomous evidence and candidate fixes. Human approval before anything gets applied.

6. Netflix: Automated Canary Analysis

Netflix's Kayenta work is a classic continuous improvement and safe deployment example. Instead of asking humans to stare at graphs and decide whether a canary looks healthy, Kayenta compares canary and baseline metrics and produces a judgment that can inform rollout or rollback.

Netflix has also extended the canary idea into data validation, using automated data canaries to catch bad metadata transformations before they affect members broadly.

Why it matters:

The equivalent here is not user traffic. It is validation signals. A new workflow or MCP change should canary through generated output checks, smoke checks, and connector checks, landing on one client before all clients.

7. Uber: Piranha for Automated Stale Code Cleanup

Uber built Piranha to remove stale feature flag code. Feature flags are useful for rollout, but once a flag is fully rolled out or abandoned, the leftover code becomes technical debt. Piranha scans source code, generates diffs to remove stale flag paths, and routes those diffs for review.

Uber describes an ongoing pipeline that periodically finds stale flags, generates diffs, creates tasks, and reminds owners.

Why it matters:

This is very close to what a platform like this needs. Feature flags may not exist here, but stale workflow names do, along with deprecated metadata fields, old generated surfaces, retired aliases, and repeated compatibility cruft. A Piranha-like cleanup radar could find those and draft removals safely.

The Pattern Across All Seven

The best examples have the same bones:

THE PATTERN ACROSS ALL SEVEN Signalfailure · drift · stalecode · release · ticket Analysisclassify and explain Candidatepatch · PR · taskrollout decision Validationtests · metrics · canarystatic analysis Humanreview · approvereject Memoryartifactlesson every lesson becomes the next signal's context
FIG 13.9The pattern across all seven organizations: signal to memory through a human gate.

The companies differ in tools, but the management philosophy is similar:

  • make hidden problems visible
  • reduce manual judgment where machines are better
  • keep humans in charge where context matters
  • turn recurring pain into system changes
  • save evidence for the next loop

What To Call This

Here is the clean terminology split.

Use case Best term
Broad management practice continuous improvement
Software delivery practice DevOps continuous improvement or continuous delivery feedback loop
Reliability practice SRE feedback loop, error-budget-driven reliability work, or toil reduction
Platform practice platform engineering continuous improvement or internal developer platform improvement loop
Automated operations practice closed-loop remediation, AIOps, or autonomic computing
Academic systems framing self-adaptive systems or MAPE-K feedback loop
AI coding-agent framing agentic software engineering
Likely article or product framing AI-assisted continuous improvement for platform engineering

For work like this, the most accurate name is:

AI-assisted continuous improvement for platform engineering

A shorter internal label works too:

Agentic continuous improvement

But it should be defined every time:

Agentic continuous improvement means a human-gated software feedback loop where agents observe evidence, analyze drift or waste, propose improvements, test candidates in a sandbox, and preserve the lesson in artifacts.

CI^2 should not be presented as an industry term. It is catchy, but it is private shorthand unless a document explicitly defines it.

What Field Is This In?

It crosses fields.

Field What it contributes
Quality engineering continuous improvement, PDSA, defect reduction
Industrial engineering process design, Kaizen, waste reduction
Software engineering tests, refactoring, code review, static analysis
DevOps CI/CD, release automation, feedback loops
SRE error budgets, toil reduction, production learning
DevSecOps secure development practices, provenance, vulnerability response
Platform engineering internal developer platforms, golden paths, paved roads, developer experience
MLOps continuous training, model monitoring, drift detection
Autonomic computing self-managing systems, MAPE-K
Agentic software engineering AI agents that inspect, plan, patch, and verify

Work like this sits mostly in platform engineering, DevOps, SRE, autonomic computing, and agentic software engineering.

It also touches AIOps once the platform begins analyzing operational signals. It touches MLOps once prompts, evals, and model behavior get treated as monitored production assets.

Assumption Boundaries

This article is intentionally not assuming a few things.

This does not assume AI should directly modify a platform without review. The safer model is human-gated remediation. Observe. Recommend. Sandbox. Validate. Then ask before applying.

This does not assume the public examples below expose each company's full internal process. The examples show public patterns: SRE, MLOps, coding agents, safe deployment, canary analysis, automated testing, static analysis, and stale-code cleanup.

This does not assume a platform like this is already a production-grade internal developer platform. It behaves like a local control plane for agent workflows, generated client surfaces, MCP configuration, validation, and run artifacts. That makes platform-engineering terminology appropriate, but it should stay grounded in the actual implementation.

This does not assume "continuous improvement" means continuous code changes. A healthy improvement loop may often decide "watch," "document," "add an eval," "tighten a validator," or "do nothing."

This does not assume metrics from CI/CD transfer directly. DORA-style metrics are useful vocabulary, but a platform like this may need domain-specific measures. Validation failure rate. Generated-install drift. Repeated correction count. Stale artifact age. Workflow over-scope rate. Time from failed run to diagnosed cause.

This does not assume memory is truth. Memory is a useful input, but current code, current generated files, current validation output, and current external docs should win when they disagree.

The Practical Version

Turning this into a real capability should not start with "agent auto-edits the repo every night."

A better starting point looks like this:

The technical pattern is:

A CONTROL PLANE FOR IMPROVEMENT signalsvalidation · drift · corrections classifierfailure · toil · debt · risk backlogranked by value and risk sandbox labisolated roots gatesschema · smoke · evals · diff decisionno action · watch · patch knowledge baserun meta · durable lessons every lesson becomes the next cycle's signal · nothing applies without passing the gates
FIG 13.10A control plane for improvement: signals to knowledge base through gates.

The operating principle:

The platform should be allowed to notice almost everything, propose many things, sandbox some things, and apply only approved things.

Phase 1: Continuous Observation

Run a scheduled report that checks:

  • schema validation
  • sync validation
  • golden fixtures
  • install smoke checks
  • release radar
  • stale generated surfaces
  • stale aliases or deprecated fields
  • repeated workflow correction patterns
  • broken MCP availability or auth hints
  • token usage spikes
  • old report artifacts with unresolved blockers
Output
  • one Markdown report
  • one JSON run metadata file
  • no repo edits
  • no generated install edits

Phase 2: Continuous Recommendation

Turn the report into ranked proposals:

  • "Fix this validator gap."
  • "Refresh this MCP pin."
  • "Remove this retired alias."
  • "Add a run-meta validator for this workflow."
  • "Split this workflow because it is doing two jobs."
  • "Improve this prompt because agents repeatedly over-scope."
Output
  • top 3 changes
  • expected value
  • risk
  • validation path
  • whether a fake-home lab is needed

Phase 3: Sandbox Execution

For approved candidates
  • create fake HOME
  • redirect XDG_CONFIG_HOME
  • redirect XDG_DATA_HOME
  • redirect XDG_STATE_HOME
  • redirect XDG_CACHE_HOME
  • run generate/install/validate there
  • compare outputs
  • write a patch proposal
Output
  • patch or diff
  • validation results
  • files touched
  • rollback notes

Phase 4: Human-Gated Apply

Only after approval
  • edit canonical source
  • regenerate outputs
  • run validation
  • update goldens if appropriate
  • run smoke checks
  • write final report
Output
  • real platform change
  • validation proof
  • durable memory

What Not To Do

Do not make the system fully autonomous too early.

Avoid
!nightly unreviewed edits to workflow instructions
!silent MCP manifest changes
!automatic generated install rewrites
!automatic client config edits
!broad dependency upgrades without rollback
!AI-authored tasks with no source evidence
!memory updates that turn guesses into "facts"

The failure mode is not just a bad patch. The failure mode is trust decay.

Once you stop trusting the automation, the loop dies.

A Good First Version

A good first version would be boring and useful.

Name:

self-improvement-radar

Runs
  • weekly
  • on demand before big platform changes
  • after failed validation
Inputs
  • local validation commands
  • release radar
  • latest workflow artifacts
  • recent migration sidecars
  • local memory summaries
  • optional network release lookup
Outputs
  • report.md
  • run-meta.json
  • findings.json
  • optional task-drafts.md
Decision
  • no action
  • watch
  • manual review
  • sandbox candidate
  • ready for approved implementation

The best first promise:

It will not change the platform on its own. It will say what is worth changing, and why.

That is enough to be valuable.

Example Findings A Radar Could Produce

This section makes the idea less abstract.

Example 1: Generated Install Drift

Signal
  • canonical workflow changed
  • generated Codex skill differs
  • smoke check still passes
Diagnosis
  • the installed surface is usable but stale
  • the risk is confusion, not immediate failure
Recommendation
  • regenerate client surfaces
  • run golden and smoke checks
  • record the migration sidecar
Decision
  • manual review
Good fit
✓generated files touch user-facing agent behavior, so the change should be visible.

Example 2: Repeated Over-Scope Correction

Signal
  • multiple runs show the user correcting agents for broadening a task
  • final reports mention scope drift
Diagnosis
  • the workflow instruction is under-specified or not enforced by evals
Recommendation
  • add a workflow-specific scope rule
  • add an eval that fails when the agent edits outside the requested surface
Decision
  • sandbox candidate
Good fit
✓prompt changes are hard to verify by static checks alone, so evals should be added with the instruction change.

Example 3: MCP Availability Regression

Signal
  • configured MCP appears in the manifest
  • installed client config exists
  • runtime tool does not appear in the active client
Diagnosis
  • likely auth, launch environment, plugin availability, or generated-install mismatch
Recommendation
  • run a connector availability probe
  • compare canonical manifest, generated config, installed config, and active runtime
  • write a finding before changing config
Decision
  • manual review
Good fit
✓auth and runtime state are easy to misdiagnose if the report only reads files.

Example 4: Stale Compatibility Code

Signal
  • old metadata field is still supported
  • no recent run emits it
  • validators now expect the replacement field
Diagnosis
  • possible cleanup candidate
Recommendation
  • search current workflows and artifacts
  • add a deprecation check
  • propose removal only if no current producer remains
Decision
  • watch or sandbox candidate
Good fit
✓stale-code cleanup is valuable, but deleting compatibility paths without evidence can break old artifacts.

Example 5: Token Usage Spike

Signal
  • local token report shows a spike in cached or fresh input tokens
  • large context files were loaded repeatedly
Diagnosis
  • likely workflow context hygiene problem
Recommendation
  • tighten skill-loading instructions
  • add artifact summaries
  • prefer targeted search over bulk reads
Decision
  • task
Good fit
✓this is process improvement rather than a bug fix.

Final Take

Continuous improvement is not new.

What is new is that the improvement loop can now be partially staffed by agents.

The old world
  • people inspect
  • people diagnose
  • people propose
  • people patch
  • people verify
  • people remember, inconsistently
The new wave
  • systems inspect themselves
  • agents summarize the evidence
  • agents propose options
  • agents test candidates in sandboxes
  • humans approve meaningful changes
  • artifacts preserve the lesson

For a platform like this, that is the right ambition.

Not a self-driving codebase.

A platform that keeps a calm eye on itself, learns from every run, and brings the next best improvement with receipts.

Sources

＋Further reading for this chapter The annotated must reads list this loop is built on: Accelerate, the Google SRE books, MAPE-K, hidden technical debt, MLOps, and the lean roots.
Next chapterChapter 14 · Measuring AI-Assisted Software Delivery A measurement model for deciding whether AI is actually helping software work, not just making more output.