Agentic Data Development Follows a Lifecycle.
AI data lifecycle uses humans, agents, and controls to guide execution while preserving decisions and reducing verification risk.
Key Takeaways
- The Agentic Data Development Lifecycle (ADDLC) is a vendor-neutral framework for AI-driven data engineering in which agents execute work, controls enforce rules, and humans retain decision rights over intent, meaning, and acceptance.
- As coding becomes less scarce, the main bottlenecks shift to human judgment, verification capacity, and operational attention, requiring process safeguards against sprawl, untrusted quality, shadow autonomy, and artifact chaos.
- ADDLC organizes work through 3 actors, 3 loops, 7 phases, and 4 gates, emphasizing specification as the system of record and treating generated pipelines, tests, and code as derived artifacts.
- Autonomy is granted per change class based on reversibility and blast radius, not per agent, while verification, Build/Run separation, evidence lineage, and stewarded master-data changes protect correctness and trust.
The Agentic Data Development Lifecycle (ADDLC), published as a specification on GitHub, is a vendor-neutral way to run data engineering when AI agents perform most execution and humans retain decision rights over intent, meaning and acceptance. It uses 3 actors, 3 loops, 7 phases and 4 gates. Its rule is short: agents propose, controls enforce, humans decide. Autonomy is granted per class of change, by reversibility and blast radius, not per agent and not by enthusiasm; it is earned with evidence and lost with escaped defects. Early results from one large platform’s reference and streaming market data show lower effort and faster delivery, but the evidence is observational.
Why code stopped being the bottleneck
In the last year, coding agents with access to a warehouse, a dbt project and a catalog made ingestion jobs, staging models, tests and documentation almost free. The bottleneck moved from writing code to specifying meaning and verifying outcomes. A mis-specified join, a metric that silently changes meaning, or a wrong identifier mapping in reference data still costs as much as before, often more, because agents produce change faster than people can review it.
Three resources are scarce: judgment about meaning, such as what an active instrument is, which source wins when vendors disagree on a listing date, or whether a negative trade size is an error or a correction; verification capacity, because unverified change is not velocity but deferred incidents; and operational attention, because every new asset is something somebody has to run and generation without a path to operation moves cost from Build to Run. Teams that adopt agents without process change see sprawl, self-certified quality, untrusted documentation, shadow autonomy when agents inherit an engineer’s production credentials, and artifact chaos when agents copy files instead of versioning them. ADDLC is organized to protect these three resources.
What the model says
The 3 actors are humans, agents and controls. Humans decide intent, meaning, acceptance and exceptions; typical roles are data product owner, data steward, data engineer, DataOps operator and consumer. Agents have execution rights only: draft, build, test, document, triage and propose fixes, with separate Build, verify and Run identities. Controls enforce continuous integration pipelines, contract validators, policy as code, access control, quality gates and promotion rules; they are deterministic and cannot be persuaded.
The model applies DAMA-DMBOK’s separation of governance and management, like the separation of audit and accounting in finance: oversight is exercised by humans and enforced by deterministic controls, while management is increasingly executed by agents. The computational half of Zhamak Dehghani’s federated computational governance is what “controls enforce” requires: policies as code, applied automatically, everywhere.
Its 7 phases are framed, specified, built, verified, released, operated and retired. The three loops include an intent loop for business and governance—Frame and Specify—where humans lead and agents prepare. The model includes gate criteria, an autonomy scale from Manual to Self-organizing, metrics, a maturity model and self-assessment. Build and Run are segregated by design, for humans and agents. Master and reference data get a stewarded change channel of their own. It builds on DAMA-DMBOK, Data Mesh and Data Management at Scale rather than replacing them. The normative specification, templates and self-assessment are on GitHub: https://github.com/savvakrasavasavva/addlc
Ten principles guide decisions. Martin Kleppmann distinguishes systems of record from derived data that can be recomputed; the specification—contract, semantics, quality expectations and acceptance examples—is the system of record, while pipelines, directed acyclic graphs and tests are derived artifacts an agent can regenerate. Generated code is not a deterministic function of its specification, which is why verification exists. When code and specification disagree, that is a defect: fix the code or formally revise the specification; neither wins silently.
Meaning is human-owned: grain, keys, survivorship rules, metric definitions, glossary terms and quality thresholds need accountable owners. Agents draft them, owners approve them, and nobody infers them at runtime. No agent certifies its own work; verification uses independent tests derived from the specification, reconciliation against an independent source, or a separate verifier agent that sees specification and output but not builder reasoning. Autonomy is granted per change class: reversible documentation and irreversible golden-record matching rules deserve different answers, even from the same agent.
Context is a product: glossary, domain model, contracts, standards, agent instruction files and skills need owners, versions, reviews and tests, making DMBOK metadata management a product with a demanding consumer. Every action leaves evidence that is attributable to an agent identity and human sponsor, explainable by specification and decision, and reconstructable from inputs, tool calls and outputs, extending lineage from where data came from to why a change happened.
Build and Run are segregated: Build agents have no write access to production data, Run agents cannot change code or models, and they use different identities, credentials and queues. The split separates identities and permissions, not accountability; a team that builds and runs its own product still uses separate Build and Run identities, because non-human workers otherwise do everything. Master-data changes are stewarded data operations, not deployments: correcting a record, remapping an identifier or patching history goes through an isolated, auditable and reversible channel, never through a code path or an ad hoc SQL session. Reuse is required before generation, otherwise near-free generation becomes permanent sprawl. Measurement focuses on lead time, human effort, failure rates, detection and verification yield per change class, not generated code or pipeline counts.
How earlier work compares
The gap is a vendor-neutral model that assigns decision rights among humans, agents and controls; defines gates with evidence criteria; segregates Build from Run; governs autonomy explicitly; defines metrics and maturity; and treats master and reference data as the special case. Coalesce’s “The Agentic Data Development Lifecycle in Practice” by Mikkel Dengsøe, October 2026, is the closest prior art: a four-stage product workflow—Define, Build, Govern and Observe—with a clear split between what the agent does and what people decide, and spec-driven development in Git. It is anchored in one product and does not define gate criteria, accountable deciders, autonomy policy, metrics or maturity.
Thoughtworks’ “Making Your Data Ready for Agentic AI” by Pramod Sadalage and Prem Chandrasekaran, martinfowler.com, August 2026, gives five attributes of AI-ready data, a context layer of domain, semantic and capability models, and a staged autonomy ladder. It treats agents mainly as consumers and curators of data rather than builders of pipelines. “The Self-Organizing Data Stack” by Ananth Packkildurai, Data Engineering Weekly, October 2026, describes an end state in which an engine maintains the dependency graph through five loops, with assets as contracts and a firm rule: never let an agent grade its own work. It gives a destination, not the path.
Holistics’ “Agentic Data Engineering” by Huy Nguyen, September 2026, covers what agents can do across the stack and why the semantic layer is the contract between engineering and analytics, but has no lifecycle. “From Data Quality to Quality of Agentic Data Use” by Ania Cravero, Jorge Díaz-Villegas and Zihao Xiao, Applied Sciences, September 2026, sets out six dimensions of agentic data use and a staged control lifecycle for an agent acting under delegated authority. It is rigorous, but focused on executing a data task rather than a team’s operating model. The ADLC family, including IBM, EPAM, AWS AI-DLC and others, covers lifecycles for building agents, or for software built by agents. A comparative review in Towards AI found six definitions and zero consensus, and none targets data engineering.
What early results show
Numbers from applying the model to the reference and streaming market data of one large platform, before and after, are observational, not a controlled experiment. Part of the gain comes from organizational changes made at the same time. Human effort per integration is about 4-5x lower, with a target of roughly 15x. Lead time from request to data in production is about 5x shorter. Integrations delivered per quarter by the same team are about 8x more. The share of incidents first reported by consumers rather than by monitoring is about 17x lower. Specs and packages accepted at the first review are nearly 2x higher, approaching all of them. Incident triage done by agents moved from none to a target of about three quarters.
For teams, the useful question is not how autonomous an agent is, but which classes of change may be executed without a human, and why. The model tries to make that answer evidence-based: reversible, low blast-radius changes can earn more autonomy, while irreversible changes to meaning, golden records or reference data stay under human decision rights, enforced by controls and verified independently.
Meta description: A vendor-neutral Agentic Data Development Lifecycle sets 3 actors, 4 gates and an autonomy scale, while early results show large efficiency gains.
Tags: Agentic Data Development Lifecycle, data engineering, AI agents, data governance, autonomy
Featured image: Abstract friendly illustration of humans, agents and controls around a seven-phase data pipeline, no real people, no logos, no text.

Comments (0)