- Agents now write to enterprise data at scale. On Databricks’ Lakebase service they create 80% of new databases and 97% of test and development environments.
- Every damaging action in the past year’s best-known agent incidents was permitted. How bad each one got depended on whether an earlier copy of the data survived.
- Governance decides who may change data and can run outside storage. Provenance records what changed and keeps the earlier state, so storage has to write it. Most enterprises file provenance under governance and bolt it on.
- We set out the requirement as the Agent Write Contract: five clauses for every agent change, in line with where bank supervisors such as Singapore’s MAS are taking AI agents.
- S3 versioning, soft deletes and database restores keep old copies but do not link versions or record why. Storage built for agent updates and regulator questions needs the record inside every commit.
Provenance used to be a regulator’s question to CFOs: where did this number come from, and what happened to the data between the source system and the report? Agents now need the same answer. They update records, merge duplicates, close tickets and delete what looks like clutter, and each of those writes alters what the next person, report or model will treat as fact. Agents and regulators, it turns out, want the same thing: governed data.
A bad change made by a person usually leaves a trail in a ticket, an email or someone’s memory. An agent makes it in seconds, under a valid credential, often unwatched, so the only dependable record is the one the data platform keeps. That record is becoming the control that decides whether enterprises let agents write at all.
From reading data to writing it
Most AI governance so far has dealt with inputs: training data, retrieval, who sees the answer. That suited AI that mostly read.
Databricks’ 2026 State of AI Agents report, built from usage across more than 20,000 organizations, found that AI agents now create 80% of the databases on its Lakebase service, built on Neon, and 97% of database testing and development environments there.
In January Gartner said organizations "can no longer implicitly trust data or assume it was human generated." It expects half to adopt zero-trust data governance by 2028, with AI-produced data identified and tagged.
Writes carry more risk than reads: a bad answer can be ignored, but a bad write feeds everything that reads the data afterwards.
What the past year’s agent incidents have in common
Public reports of agent mishaps piled up between late 2025 and mid-2026. Two involved production systems at companies, and both turned on what the platform underneath the agent allowed.
| Incident | What happened | What decided the outcome |
|---|---|---|
| AWS, December 2025 (reported February 2026) | The Financial Times reported that engineers let Amazon’s Kiro coding agent make changes, and it chose to delete and recreate an environment, causing a 13-hour interruption to AWS Cost Explorer in one region of mainland China. Amazon disputes this and attributes the event to misconfigured access controls. | On either account, the permissions allowed a destructive action, and the dispute shows how hard attribution gets with agents. |
| PocketOS, April 2026 | A Cursor agent running Anthropic’s Claude Opus 4.6, working on a staging task, found an API token and deleted a production database volume on Railway in nine seconds. Volume-level backups went with it; the newest backup the company could reach was three months old. | Railway’s API ran the delete immediately, while its dashboard had a 48-hour delay. Railway recovered the data from separate disaster backups and has since made every API delete a 48-hour soft delete. |
In both, the outcome depended on the system underneath the agent rather than on the model: whether a destructive change could happen instantly, whether it was recorded with enough context to explain later, and whether an earlier state survived somewhere the agent could not reach.
Governance and provenance are two different jobs
The usual response to stories like these is to tighten permissions, and that is the right first step. It also exposes a habit worth questioning: most enterprises file provenance under governance. Governance programmes list lineage among their duties, and catalogs such as Microsoft Purview and Databricks Unity Catalog present lineage as a governance feature. Catalog lineage maps which tables and columns feed which. It does not keep the earlier state of the data or say why a value changed, and it cannot restore anything.
Filed that way, provenance becomes one more tool attached beside the data, and agents that write make the cost visible. The two controls act at different moments, fail in different ways and have to live in different places.
Governance acts before a change. It decides which agents exist, whose authority they borrow, what their credentials can reach, and which actions a person must approve first.
Provenance acts during and after a change. It records what happened and keeps what was there before: which version changed, into what, from which inputs, by whom and why. A delete leaves the earlier state in place and adds an entry to a history that cannot be rewritten.
| Governance | Provenance | |
|---|---|---|
| Question it answers | Who is allowed to do this? | What was done, and what was there before? |
| When it acts | Before the write | During and after the write |
| Typical controls | Agent identities, least-privilege scopes, policy checks, approval gates | Versioned history, recorded author and reason, lineage between versions, retained deletes |
| Where it can live | Outside storage: identity systems, gateways, catalogs | Inside storage, written in the same commit as the change |
| What a failure looks like | An agent reaches data or tools it should never have had | A permitted change turns out to be wrong, and nobody can show or restore the earlier state |
The large platforms moved quickly on governance this year. Microsoft’s Purview audit log records agent tool calls as their own events, Snowflake lets admins require two approvals before destructive changes, and Amazon’s Bedrock AgentCore checks every agent tool call against policy. All of it runs before the action, where governance belongs. A gateway log shows that an agent called delete; the deleted data, and any way back, has to come from storage.
Apply that to the incidents. The PocketOS agent used a real, fully permissioned token, and Amazon’s own explanation of the Kiro event is a misconfigured access control. By each system’s own rules, the destructive step was allowed. Governance sets what an agent may attempt; it has no way to judge whether a particular permitted change was correct.
That judgement comes later, when someone notices a missing record and asks what changed, who changed it, why, and whether it can be put back. Only the data platform can answer. In our earlier piece on storage failures we described the same split as the write path and the read path; once agents write data, the two have to be designed together.
The Agent Write Contract: data provenance for AI agents in five clauses
Bank supervisors are moving the same way. On 7 October 2026 the Monetary Authority of Singapore published final AI risk management guidelines for financial institutions. They bring AI agents into scope and expect controls covering data management, human oversight, monitoring and change management, phased in from October 2027. Like most supervisory guidance, they describe outcomes and leave the data platform to each institution, so governance and provenance have to turn them into a test. Every agent change should meet five clauses. The first draws on governance and provenance; the rest are provenance, with governance returning for irreversible actions.
- Clause 1 · Governance + provenanceAttributedThe change records which agent made it, on whose behalf, and under which permission. Without this you see a service account, or a person who followed an agent’s advice.
- Clause 2 · ProvenanceExplainedThe change records why it was made and from which inputs. Otherwise you can see that a value changed, with no idea what the agent was trying to do.
- Clause 3 · ProvenanceVisibleThe people who own the data can see a before-and-after before anyone else relies on it. If they can’t, they find out from a downstream report weeks later.
- Clause 4 · ProvenancePreservedThe earlier state stays intact, and a delete is recorded as an event. Lacking that, the only copy of the old data sits in a backup, if anywhere.
- Clause 5 · Provenance, gated by governanceReversibleOne change can be undone on its own without a restore, and anything that cannot be undone needs a person to approve it. Without this, recovery means rolling everything back to an old point and losing good work along with the bad.
The Explained clause needs care: an agent’s stated reason is its own account. After deleting the PocketOS database, the agent fluently listed the rules it had broken, including that it had guessed instead of checking. Such explanations need not reflect what drove the action. So the record should hold what the agent read and did: its inputs, their data versions and its tool calls. The stated reason is one field in that record, never the evidence.
Visibility builds the most trust and breaks first at scale. An owner can review one batch of 312 removals in minutes, but nobody can review 40,000 changes a day, and asking them to produces rubber-stamping. Singapore’s IMDA agentic AI framework warns about this automation bias. In practice that means tiers: changes to working data nothing certified depends on go through, medium-impact changes are sampled in batches, and anything that alters approved figures, removes records or cannot be undone waits for a named person. Every tier writes the same record, so unreviewed changes can still be found and reversed. (We made the same case for answers in the trust dividend.)
Undo, versions and immutability are not the same thing
Amazon S3, which sits under most enterprise data lakes, makes a useful test case. With S3 Versioning turned on, a delete that does not name a version erases nothing. S3 adds a delete marker, the object appears deleted, and every earlier version can still be restored. S3 Object Lock goes further and makes individual versions write-once. In compliance mode, no user, including the account’s root user, can overwrite or delete a locked version before its retention date.
That protection is real, and it still falls short of provenance in four ways:
- Each object is versioned on its own. S3 knows that
customers.parquethas five versions, but it has no record that version 4 was built from versions 2 and 3 of two other files, or that six objects changed together as one correction. - A version carries no reason. CloudTrail can record who made a change, and only if object-level data events have been turned on, which they are not by default. Nothing records why.
- Versions can still be removed. Anyone with the right permission can permanently delete a version, lifecycle rules expire old versions on a schedule, and Object Lock’s governance mode can be bypassed with a specific permission. Only compliance mode holds against everyone.
- S3 has no concept of a table or of approval. A table spans many files, so rebuilding it as it stood last month means working out which version of each file was current then, and nothing marks which state was signed off.
Open table formats close part of this gap. Apache Iceberg groups files into table snapshots, and its branches let a team write changes, check them and then publish, a pattern called write-audit-publish. Author and reason stay optional, though, and maintenance expires old snapshots unless a branch or tag keeps them.
Other common mechanisms have their own limits:
| Mechanism | Example | What it gives you | Where it stops |
|---|---|---|---|
| Undo window | Railway’s 48-hour soft delete | A deleted object can be restored for a fixed period. | After the window, the data is gone. It records that something was deleted, but not why. |
| Point-in-time recovery | Database restore to a timestamp | The whole database as it stood at a moment, restored to a new instance. | Brings back everything from that moment, including what was wrong, and discards good changes made since. |
| Immutable ledger | An append-only log over immutable files | Every change is a new entry with author, reason and the versions it came from, and removals are entries too. | Storage grows, so retention has to be set deliberately. |
When a delete has to be real
Immutable history runs into one hard limit: some deletes are legally required to be real. The EU’s GDPR and India’s Digital Personal Data Protection Act both give individuals a right to erasure. Both exempt data the law requires a company to keep, so a bank’s regulatory records may stay while a marketing profile goes.
A ledger can honour both. One common approach encrypts personal fields with a key per person and destroys the key on erasure, sometimes called crypto-shredding: the records stay in place but can no longer be read. Another rewrites the affected files without the person’s data and records who erased it, when and under which request. Either way, the history shows that an erasure happened.
Design provenance into the storage
Agents write in two places. In systems of record such as CRM, ERP and ticketing, the vendor’s audit log holds the record, and change data capture brings it into the data platform. Inside the data platform, agents merge master data, fix records that fail quality checks and adjust curated silver and gold tables. Those writes feed every report and model downstream, so storage has to carry their provenance.
Governance can sit outside storage. Provenance cannot: a log kept beside the data records only the writes that pass through it. Railway’s dashboard delayed deletes by 48 hours; its API did not, and the PocketOS agent used the API. If data can change without its record changing in the same commit, the record will eventually miss something.
Five properties matter most where regulators ask how a figure was produced, or agents hand work on:
- One commit, one record. The change and its record are written together. Storage refuses any write that lacks an author, the party it acts for, and its input versions.
- Files never change; removals are entries. Data is written once, never edited in place. Retention follows the longer of the regulatory and business horizons, and physical deletion happens only through a recorded erasure path.
- Handovers carry versions. A handover to another agent or a person names the data version and change set, so the record of who did what continues.
- Approval is a state in storage. Proposed changes stay apart from the published version until a person or rule publishes them, and publishing is itself an entry.
- Traceability is a query. A regulator’s question about a figure is answered by walking the ledger back to its sources, from the same records the platform runs on.
A bank’s agent cleaning its customer master:
| When | What happened | What storage keeps |
|---|---|---|
| Day 1 | For the data owner, the agent proposes removing 312 duplicates, matches attached. | A ledger entry for a new version excluding the 312. Their file is unchanged. |
| Day 1 | The owner reviews the change and publishes it. | An approval entry. Reports and agents now read it. |
| Day 22 | Finance finds four were real customers sharing an address. | The four are still in the original file; no backup needed. |
| Day 22 | An analyst restores the four; the owner approves. | A new file holding only the four records, entered in the ledger as a third version. The other 308 stay excluded. |
No file was rewritten. The ledger decides which files and records make up each version, and every read assembles the latest one from it. Every step names who acted and why. We built our storage architecture this way, with agent identity and access controls in front, as described on Security Assurance. The ledger works over any object store if every write goes through it. It has two costs: versions built from many small files need regular compaction, and the on-whose-behalf identity must survive every agent handover, which many stacks cannot do.
Earlier this month we argued that your next AI failure will be a storage failure, because most enterprises cannot show what data an AI read. Agents that write add a second test: proving what they changed and changing it back.
Frequently asked questions
What is data provenance for AI agents?
It is the record of where a piece of data came from and every change made to it since: which agent made each change, on whose behalf, why, from which inputs, and what the data looked like before. It lets an organization explain an agent’s changes and reverse the wrong ones.
How is data provenance different from data governance?
Governance works before a change: it decides who may act on which data and what needs approval, and it can run outside storage, in gateways and catalogs. Provenance works during and after a change: it keeps the earlier state, even after a delete, and records what changed, by whom, why and from which inputs. It has to be written by storage in the same commit as the change, or it will miss writes.
Is S3 versioning or a soft delete enough to protect data from AI agents?
Both help you recover data. S3 Versioning keeps every version of each object, but versions are not linked, carry no reason, and can be expired or deleted unless locked in compliance mode. A soft delete lasts only for its window. Agent writes also need an append-only ledger that ties versions together, records author and reason, and treats removals as entries.
How does an immutable audit trail handle the right to erasure?
Some deletes are legally required to be real, under laws such as the GDPR and India’s DPDP Act. A ledger can honour them by destroying a per-person encryption key, or by rewriting the affected files without the person’s data. Either way it records that the erasure happened, who did it and under which request, without keeping the erased content.
References
- Databricks, State of AI Agents 2026, January 2026. Aggregated, anonymized usage data from more than 20,000 organizations, November 2024 to October 2025.
- Gartner, Gartner Predicts by 2028, 50% of Organizations Will Adopt Zero-Trust Data Governance as Unverified AI-Generated Data Grows, 21 January 2026.
- Engadget, 13-hour AWS outage reportedly caused by Amazon’s own AI tools, February 2026, reporting the Financial Times and Amazon’s statement; and GeekWire, Amazon pushes back on Financial Times report blaming AI coding tools for AWS outages, February 2026.
- Tom’s Hardware, Claude-powered AI coding agent deletes entire company database in 9 seconds, 27 April 2026; and Decrypt, AI Agent Deletes Startup’s Database in 9 Seconds, Founder Says, April 2026.
- Railway, Your AI wants to nuke your database. Guardrails fix that., April 2026.
- Amazon Web Services, Deleting object versions from a versioning-enabled bucket, Locking objects with Object Lock and Enabling CloudTrail event logging for S3 buckets and objects, Amazon S3 user guide.
- Apache Iceberg, Branching and Tagging, project documentation; and Dremio, Exploring Branches and Tags in Apache Iceberg using Spark.
- Monetary Authority of Singapore, Guidelines on Artificial Intelligence Risk Management for Financial Institutions, 7 October 2026, as summarized by Allen & Gledhill (21 October 2026) and PwC Singapore.
- General Data Protection Regulation, Article 17: Right to erasure, including the exception for legal obligations in Article 17(3)(b); and India’s Digital Personal Data Protection Act, 2023, Section 12: Right to correction and erasure of personal data.
- Amazon Web Services, RestoreDBInstanceToPointInTime, Amazon RDS API reference.
- OWASP GenAI Security Project, OWASP Top 10 for Agentic Applications for 2026, 9 December 2025.
- Microsoft Learn, Audit log activities, section on Agent 365 activities.
- Microsoft Learn, Data lineage in Microsoft Purview; Databricks, Capture and view data lineage with Unity Catalog.
- Snowflake, Cortex AI Gateway and advanced AI security announced at Black Hat 2026, 5 August 2026.
- Amazon Web Services, Policy in Amazon Bedrock AgentCore: Control agent-to-tool interactions, developer guide.
- Databricks, Databricks and Neon will deliver serverless Postgres for developers and AI agents, May 2025.
- Infocomm Media Development Authority (IMDA), Model AI Governance Framework for Agentic AI, 22 January 2026, as summarized by Human Resources Online and Covington’s Inside Privacy.
- Jones Walker, NIST’s AI Agent Standards Initiative: Why Autonomous AI Just Became Washington’s Problem, 26 February 2026.
- EU AI Act Explorer, Digital Omnibus on AI, in force 27 July 2026; and Gibson Dunn, EU AI Act Omnibus Agreement: Postponed High-Risk Deadlines and Other Key Changes, May 2026.


