Cost scales with the table
Changing a hundred rows rewrites the million around them. Every edit gets slower as the dataset grows.
Wekalp: only the affected file is rewritten, so cost tracks the change, not the table.
/ Technology · Storage Architecture
Auditable data storage that never overwrites.
Every data load, correction and approval writes a new immutable file and one entry in an append-only transaction log — nothing is modified in place. Each table version is computed from the log, any earlier version can be rebuilt, and every row traces back to its source. Lakehouse reliability, with a complete audit trail.
ACID transactions · Time travel · Per-file checksums · Column-level encryption · Single-tenant, cloud or on-premises
ACID
Transactions on every write
Each change commits completely or not at all. Readers never see partial data.
Immutable
Data files
Updates write new files. Existing files are never overwritten or edited.
Time travel
To any earlier version
Rebuild a table exactly as it stood at any version, within the retention period.
Mandatory
Audit metadata
Author, reason, lineage and checksum recorded on every change — rejected writes included.
Why storage matters
Why typical data storage fails an audit.
Enterprise data arrives in batches from many systems and teams, is checked and corrected over days, then approved and often sent to an outside party — a regulator, a clearing house, a counterparty. Most storage keeps one copy per table and overwrites it on every change. That fails in four ways.
A crash corrupts the only copy
A rewrite that fails halfway leaves a half-written file where the good one used to be.
Wekalp: atomic commits. A failed write never becomes visible, and the previous version stays intact.
Concurrent users collide
Two people saving the same table means the last writer silently wins and the other's work disappears.
Wekalp: concurrent loads run in parallel; edits to the same file are serialised, never lost.
Nothing records where a row came from
No trace of who changed what, why, from which file — or what the outside party actually accepted.
Wekalp: every change records author, reason and lineage; finalized data is kept as a golden record at every layer.
Guiding principles
The core fundamentals of an AI‑ready storage layer.
The same principles as modern lakehouse formats, with the audit trail made mandatory rather than optional.
A
Atomic
One log entry is the commit. New data is invisible until it commits, then fully present.
C
Consistent
Schema and file validity are enforced before commit; each data file is added once and removed at most once.
I
Isolated
Every reader sees a consistent snapshot. Concurrent edits to the same file are serialised by a lock.
D
Durable
Transactional commits on durable object storage, with a per-file checksum to verify the bytes later.
Immutable storage
Files are written once. A change creates a new file; nothing is overwritten.
Append-only transaction log
Every change is a new log entry, in strict order. History is never edited.
Schema enforcement
Bad data is rejected before it is written, not cleaned up afterwards.
Fail-safe validation
Anything unexpected — an unknown record, a checksum that no longer matches — stops the process. Nothing is guessed.
Snapshot isolation
Readers always see a complete, consistent version — never a half-written one.
Concurrency control
Edits to the same data are queued behind a lock. No lost updates, no silent merges.
Copy-on-write updates
Only the affected file is rewritten as a new copy. The original stays intact.
Time travel
Rebuild the data exactly as it stood at any earlier version.
Data lineage and audit trail
Every file records who, when, why and what it came from — refused attempts too.
Integrity checksums
Each file’s checksum is recorded at commit, so any change to the bytes is detectable.
Column-level encryption
Sensitive columns are encrypted before storage and decrypted only for roles allowed to see them.
Golden record
Each layer’s finalized data is kept as a golden record, separate from work in progress — the trusted input to the next step.
Key management, access control and the audit log of user activity are covered on Security Assurance.
Architecture
The architectural advantage.
Two layers, as in Delta Lake and Apache Iceberg: data files in object storage, and a transaction log that defines every version of a table.
01 · SourcesData sources
System extractsAPIsDatabase connectorsFile dropsJobs & AI agentsPartner responses
02 · IngestIngestion and validation
Schema & quality checks→Write data file→Compute checksum→Commit to transaction log
Validation runs before anything is written. The commit is always the final step.
03 · StoreStorage layer
Metadata layer
Transaction log
Append-only record of every file added or removed, in commit order, with full audit metadata.
Object storage
Immutable data files
Open columnar format, sensitive columns encrypted, checksum recorded at commit.
04 · QueryQuery layer
Read current snapshot from log→Scan only those files→Return results
Every query sees one consistent table version — no partial writes, no merging at read time.
Transaction log
What each log entry records.
Every lakehouse log records which files make up a table. Here, the audit metadata is mandatory on every entry — this log is what the rest of the page calls the ledger.
Integrity rule: each data file is added once and removed at most once.
Core · defines the table
Commit#9
OperationAdd file f7
Table versionv1
Audit metadata · mandatory
ReasonCorrection
Lineagefrom f1 + f6
AuthorData steward
Checksum9f2c…e1
Rows100,000
Timestamp30 Sep · 14:02
Record lifecycle
One lifecycle, at every layer.
Bronze, Silver and Gold work as in any medallion architecture: raw data, transformed tables, business-ready outputs. Wekalp adds one lifecycle to every layer, on the same transaction log: data is worked on, approved internally and finalized into a golden record before it feeds the next layer. That lets business users correct and own data at every layer, with a complete audit trail.
BronzeRaw layerData as ingested from source systems
- WorkingLoads, reloads, multi-part loads, corrections and removals
- Internally approvedQuality checks passed, signed off, frozen
- Golden recordFeeds the Silver layer
SilverTransformed layerFlattened, analytical and code-mapped tables
- WorkingTransformations, then corrections by business users — e.g. a fixed code mapping
- Internally approvedQuality checks passed, signed off, frozen
- Golden recordFeeds the Gold layer
GoldBusiness-ready layerAggregations and business-ready outputs
- WorkingAggregations and outputs, adjusted with a full audit trail
- Internally approvedQuality checks passed, signed off, frozen
- Golden recordReleased to every consumer
ConsumersGolden records, aggregated or granular, feed
AI and LLMsAgents and models grounded in certified data
Regulatory submissionsFilings to supervisors and other external parties
Business ownersDashboards, analysis and decisions
Applications and APIsDownstream systems and partners
Rejected or changed later? A new working version opens at the layer that needs the fix, carried with its error codes. A golden record is never edited in place.
The ledger replay below walks through the Bronze-layer operations step by step; Silver and Gold tables use exactly the same ones.
What Wekalp adds to a standard medallion architecture
Corrections inside each layer
Reloads, corrections and removals in Silver and Gold too, each a new version, with no rebuild from raw data.
Approval before data moves on
Internal sign-off and a golden record per layer. The standard pattern has no approval step.
A mandatory audit trail
Reason, author and lineage on every change, and external rejections reopen the right layer.
Owned by business users
The people who know the data correct and approve it, not only engineers through pipelines.
Standard medallion as described by Databricks, where a wrong value in Silver or Gold is fixed by rebuilding from raw data and rerunning the pipeline.
Retention and cleanup
Version history
Superseded files are kept for time travel and audit. Reads only touch current files, so performance does not degrade.
Retention period and vacuum
A scheduled job removes files older than your retention period. Every deletion is itself logged.
Orphan cleanup
Files left by a failed write are never visible as data and are removed automatically.
This is the storage beneath Wekalp Regulatory Reporting, where the external party is a supervisor accepting or rejecting records one by one, as in COCREE 2.0 reporting.
See it work
See the ledger in action.
One quarter of loan data. Three systems send batches, a duplicate is removed, quality checks flag records, and the fixes are applied. Step through ten moments and watch what the ledger records and what the data looks like after each one. These are Bronze-layer operations; Silver and Gold tables use the same ones.
Both fixes go in, one after the other
Analyst 1’s fixes make f12; Analyst 2 then retries on f12 and makes f13. Where both changed a record, the later value stands and the earlier one stays readable.
Recorded in the ledger
- Removef7Superseded by Analyst 1’s fixesAnalyst 1
- Addf12f7 with 400 records replaced from f10Analyst 1
- Removef12Superseded by Analyst 2’s fixesAnalyst 2
- Addf13f12 with 250 records replaced from f11Analyst 2
Current data
f2100,000
f5120,000
f8100,000
f13100,000
420,000records (unchanged)
Kept for audit, not counted
f1f3f4f7f9f12
f6f10f11
Changed in this stepHolds flagged records
Show all 19 ledger entries as a table
| # | Event | File | Why | Derived from | By | Rows |
|---|---|---|---|---|---|---|
| 1 | Add | f1 | Ingest | — | System 1 · Core banking | 100,000 |
| 2 | Add | f2 | Ingest | — | System 1 · Core banking | 100,000 |
| 3 | Add | f3 | Ingest | — | System 1 · Core banking | 100,000 |
| 4 | Add | f4 | Ingest | — | System 3 · Branch files | 80,000 |
| 5 | Add | f5 | Ingest | — | System 2 · Loan origination | 120,000 |
| 6 | Remove | f4 | Dropped | — | Data steward | 80,000 |
| 7 | Evidence | f6 | Correction | — | Data steward | 1,500 |
| 8 | Remove | f1 | Correction | — | Data steward | 100,000 |
| 9 | Add | f7 | Correction | f1 + f6 | Data steward | 100,000 |
| 10 | Remove | f3 | Correction | — | Data steward | 100,000 |
| 11 | Add | f8 | Correction | f3 + f6 | Data steward | 100,000 |
| 12 | Evidence | f9 | Correction | — | Analyst 1 | 2,000 |
| 13 | Remove | f9 | Correction rejected | — | Analyst 1 | 2,000 |
| 14 | Evidence | f10 | Correction | — | Analyst 1 | 400 |
| 15 | Evidence | f11 | Correction | — | Analyst 2 | 250 |
| 16 | Remove | f7 | Correction | — | Analyst 1 | 100,000 |
| 17 | Add | f12 | Correction | f7 + f10 | Analyst 1 | 100,000 |
| 18 | Remove | f12 | Correction | — | Analyst 2 | 100,000 |
| 19 | Add | f13 | Correction | f12 + f11 | Analyst 2 | 100,000 |
Built for enterprise AI
Why it matters for enterprise AI.
AI is only as reliable as the data it reads. The same controls that satisfy an auditor give models and agents versioned, traceable and governed data.
Reproducible datasets
Train and evaluate models on an exact data version, rebuilt at any time — not today’s data.
Data provenance
Every AI answer can cite the exact file versions and checksums it was built from.
Governed agent writes
AI agents write through the same validated, logged path as any other source — nothing in place.
Certified data for AI
Point agents at the golden record for facts; working data stays separate until it is approved.
Quality-gated training data
Models only use data whose quality checks are current; any change marks old results stale.
Unified ingestion
Extracts, APIs, database connectors, files and agent output all land through one validated path.
Why this matters beyond the platform: why your next AI failure will be a storage failure, and the four questions every AI answer has to survive.
Peer comparison
How it compares with lakehouse table formats.
Delta Lake, Apache Iceberg and Apache Hudi are excellent general-purpose table formats, and Wekalp shares their core guarantees. The difference is audit depth — and, by design, a narrower focus.
| Capability | Wekalp | Delta Lake |
Apache Iceberg |
Apache Hudi |
|---|---|---|---|---|
| Core table guarantees | ||||
| ACID transactions | ✓ | ✓ | ✓ | ✓ |
| Immutable data files | ✓ | ✓ | ✓ | ✓ |
| Schema enforcement on write | ✓ | ✓ | ✓ | ✓ |
| Time travel | ✓ | ✓ | ✓ | ✓ |
| Row-level updates and deletes | ✓ | ✓ | ✓ | ✓ |
| Merge-on-read updates | ✕1 | ✓ | ✓ | ✓ |
| Concurrent writers | ✓ | ✓ | ✓ | ✓ |
| Audit and lineage | ||||
| Commit history (operation, time) | ✓ | ✓ | ✓ | ✓ |
| Author recorded on every change | ✓ | 2 | 3 | 3 |
| Reason recorded on every change | ✓ | 3 | 3 | 3 |
| File-level lineage | ✓ | 4 | 4 | 5 |
| Rejected writes recorded | ✓ | ✕ | ✕ | ✕ |
| Input files retained as evidence | ✓ | ✕ | ✕ | ✕ |
| Per-file content checksum | ✓ | ✕6 | ✕ | ✕ |
| Data lifecycle | ||||
| Approve and freeze a version | ✓ | ✕ | 7 | 8 |
| Golden record per layer (finalized, versioned) | ✓ | ✕ | ✕ | ✕ |
| Audited corrections on transformed tables | ✓ | 3 | 3 | 3 |
| Retention and file cleanup | ✓ | ✓ | ✓ | ✓ |
✓ Built in Partial or optional✕ Not supported
- Wekalp: by design, an update rewrites the affected file, so reads never merge changes.
- Delta Lake: user details are captured by some engines (for example Databricks), not guaranteed by the protocol.
- Optional custom commit metadata (Delta commit info, Iceberg snapshot properties, Hudi commit metadata) — not mandatory per change.
- Delta Lake and Iceberg: files removed and added in one commit are linked only at commit level.
- Hudi: file groups track successive versions of the same file.
- Delta Lake: an optional per-version checksum file covers table-level metrics, not file contents.
- Iceberg: branches and tags support a write-audit-publish workflow.
- Hudi: savepoints preserve a commit for restore.
Full technical specification
For architects and auditors. The model rests on transactional commits, immutable object storage, open columnar files and a business key per table. If your review needs more, send it over.
- Model
- Immutable data files plus an append-only transaction log. A table version is every file the log has added and not yet removed.
- Transactions
- ACID commits. One log entry is the commit; each data file is added once and removed at most once.
- Concurrency
- New data loads in parallel; changes to an existing file are serialised by a fail-fast lock and re-checked after locking. Nothing is merged silently.
- Versioning
- Working versions per table and period; approval freezes a version for good, and any later change opens a new one.
- Lineage
- Every log entry records its author, time, reason and source files — rejected writes included.
- Evidence
- Incoming correction files are retained as evidence, whether applied or rejected.
- Golden records
- Each layer’s finalized data is kept as an immutable golden record per table and period, rolled forward on each finalization. Bronze and Silver golden records feed the next layer; Gold golden records feed AI, reporting, regulatory submissions and applications.
- Integrity
- Per-file checksums recorded at commit and verified before anything is sent out, on demand and on a schedule; row-level hashes detect real change.
- Encryption
- Sensitive columns encrypted before a file is stored and decrypted only inside the engine, for permitted roles. Details on Security Assurance.
- History
- Time travel to any earlier version or golden record until the retention period expires; every vacuum is itself logged.
- Openness
- Open columnar files in standard object storage. No proprietary format, no separate catalog service, no cluster runtime.
- Deployment
- Single-tenant, in your own cloud or data centre or on Wekalp's hosted cloud.
- Configuration
- Sensitive columns, checks and severities, output formats and retention thresholds. The model itself does not change between deployments.
Based on the open-source specifications and documentation of each format as of October 2026; managed platforms built on them may add features.
Common questions
Frequently asked questions.
Is this Delta Lake?
It is built on the same core idea — immutable data files plus a log of commits — but implemented natively on the application's own transactional database and object storage, rather than by running the open-source Delta Lake package. The log is an append-only table, not a directory of JSON files, and it records more than a lakehouse log does: why each event happened, which files it derived from, the input files themselves, and a golden record of finalized data at every layer.
Why keep the ledger in a database instead of in object storage?
Because the database already provides transactions and row locks. Creating a table version and committing its first file is one transaction, and the lock that serialises two edits to the same file is an ordinary row lock. Nothing extra has to run to coordinate writers: no catalog service, no lock provider and no Spark runtime.
Can anyone edit or delete data after it is written?
No. Data files are never modified or deleted in place. Every ingestion, removal and correction writes a new file and appends a new log entry; the previous file stays in storage. Files are only deleted by a scheduled vacuum job after your retention period, and each deletion is itself logged.
What happens if a write crashes halfway?
The commit to the transaction log is always the last step, after the file is fully written and its checksum recorded. A crash before that step leaves an orphan file that no query will ever see, never partial data. Writes in progress are tracked, so orphan files are found by lookup and cleaned up automatically.
How do two people correct the same data at the same time?
Each data file has a lock. The first edit takes it; the second is told the file is in use and retries. After the first commits, the second re-checks that its file is still current and applies its change to the new version. On a shared record the later correction wins, and the earlier value stays readable in history. Nothing is merged silently.
Does data have to arrive as files?
No. Data can be ingested from a scheduled system extract, an API feed, a database connector, a file drop, or the output of an automated job or AI agent. Each batch passes the same checks, is written once, and gets the same log entry, attribution and lineage.
How does this help enterprise AI initiatives?
AI needs data it can be held to. Every answer can be traced to the exact data files and checksums it used; any training or evaluation dataset can be rebuilt as it stood at a given version; AI agents write through the same validated, logged path as any other source; and finalized data sits in a golden record, separate from work in progress.
Does storage grow without limit?
Superseded files are kept until your retention period expires, then removed by a scheduled vacuum job. Queries only read current files, so history costs storage, not speed. Within the retention period, any earlier version can be rebuilt.
Talk to us about audit-ready data.
Bring the hardest question your auditors, regulators or risk team ask about your data. We will show you how the platform answers it.
- An architecture walkthrough, principle by principle
- A replay of your own data's lifecycle on the platform
- How it runs in your cloud or data centre