File pruning
The ledger identifies the files a query, process or run needs. No other file is opened.
/ Technology · Compute Architecture · Shared engine
The compute engine: DuckDB, Arrow and Polars on governed Parquet.
The engine every query and every orchestrated run shares. It reads only what a job needs, merges files only when it must, runs spreadsheet logic as versioned rules and keeps every job inside a memory limit.
Four-step pruning · File sweep · Excel formulas as rules · Four kinds of tables · Memory capped per job · Production sizing
Engines and execution
Engines and query execution.
Three engines, one path
DuckDB, Arrow and Polars or pandas.
No single engine is best at everything. DuckDB does the relational work, Polars or pandas run the Python logic on a small, pruned frame, and Apache Arrow hands data between them without copying it.
Open-source editions throughout: DuckDB and Polars (MIT), pandas (BSD-3), Apache Arrow (Apache 2.0).
One execution path, three engines
Read less
Reading only the data a job needs.
Every read is cut down in four steps before any processing starts: files, then versions and dates, then columns, then rows. Everything else stays in storage. In our own platform, scanning in DuckDB before pandas cut memory on large operations by 80–90% (The Kitchen Revolution).
Partition pruning
Within the files the ledger lists, only the versions and business dates the run needs are read.
Column pruning (projection pushdown)
Only the columns the query or process uses are read and handed on.
Row pruning (predicate pushdown)
Filters are applied as the data is read, and row groups that cannot match are skipped.
File sweep
The file sweep: merge only when it is required.
The records a run needs often sit across several current files. Building one merged file from them is expensive, so in most runs no merged file is created: DuckDB queries the current parts directly and unions the results. A merged file is built only when an execution or assertion has to see every record in one place.
- Ledger-driven: as in Delta Lake, only the files the log lists as current are read.
- Light by default: one query across the file list, or one per part, with the results unioned. Nothing is copied.
- Merged when mandatory: for example duplicate checks across all loads, or totals reconciled across the whole set.
Two ways to read the current files
Business rules
Excel formulas become versioned rules.
Spreadsheet logic is translated into versioned rules, then run as SQL in DuckDB or as Python in Polars. Each rule is tested in a sandbox before it touches live data, and can be read back as spreadsheet logic.
VLOOKUP · exact match → LEFT JOIN
Excel formula
=VLOOKUP(C2, Branch_Master!A:D, 4, FALSE)Versioned rule
R-114 v3
Loans.region ← BranchMaster.region
key: branch_code
match: exact
no match: flag, do not guessSQL · DuckDB
SELECT l.*, b.region,
b.branch_code IS NULL AS r114_no_match
FROM loans l
LEFT JOIN branch_master b
ON l.branch_code = b.branch_codePython · Polars
loans.join(
branch_master.select("branch_code", "region"),
on="branch_code", how="left",
).with_columns(
pl.col("region").is_null().alias("r114_no_match")
)A missing branch becomes a logged validation result, not a silent #N/A, and a duplicate key in the master is flagged instead of Excel quietly taking the first row.
VLOOKUP · approximate match → ASOF JOIN
Excel formula
=VLOOKUP(G2, Rate_Bands!A:B, 2, TRUE)Versioned rule
R-207 v1
Loans.rate_band ← RateBands.band
range: balance ≥ lower_bound
match: nearest lower bandSQL · DuckDB
SELECT l.*, r.band
FROM loans l
ASOF LEFT JOIN rate_bands r
ON l.balance >= r.lower_boundPython · Polars
loans.sort("balance").join_asof(
rate_bands.sort("lower_bound"),
left_on="balance", right_on="lower_bound",
)A lookup that silently depends on the sheet being sorted becomes an explicit range rule.
SUMIFS → GROUP BY
Excel formula
=SUMIFS(Loans!K:K, Loans!C:C, A2, Loans!H:H, "Active")Versioned rule
R-031 v5
Branch.active_balance = SUM(Loans.outstanding_balance)
group: branch_code
filter: status = 'Active'SQL · DuckDB
SELECT branch_code,
SUM(outstanding_balance) AS active_balance
FROM loans
WHERE status = 'Active'
GROUP BY branch_codePython · Polars
loans.filter(pl.col("status") == "Active")
.group_by("branch_code")
.agg(pl.col("outstanding_balance").sum()
.alias("active_balance"))One rule replaces a formula copied down thousands of rows, and runs on every branch at once.
Table types
Table types the engine reads and writes.
Data moves from source tables to worktables to business tables, with SQL views for anything a data team prefers to write as a query. All of it is Parquet in the same versioned storage, so there is no separate serving database to license, size or secure.
Source tables
Source tables, by load type.
Nothing is changed on the way in. The type of source decides how each new load is treated.
Transactional
Each load is added to the ones before it: payments, journals, trades.
Positional
Each load replaces that date's picture, and can carry forward: balances, inventory.
Master
Each load is the current list, and the latest version is used: customers, products.
Master with history
Rows carry start and end dates, so you can ask what was true on any date.
Built on top of sources
Worktables, business tables and SQL views.
Every kind is versioned by business date, quality-checked and read under the same row- and column-level access. Orchestrated runs can also produce intermediate tables, kept as stored tables when needed for audit, and reports.
Worktables
The working tabs teams built in Excel, learned once and rebuilt every run, with audited adjustments.
Business tables
Sources and worktables flattened into one wide, query-ready table, with a glossary for AI.
SQL views
Tables defined by a query, for exploring, tidying a source or building a business table.
Stored or live
Built once per date and kept, or run as a query when read. One setting, nothing else changes.
How versions, golden records and the medallion lifecycle work: Storage Architecture.
Memory
Memory management for Python and DuckDB jobs.
Data jobs usually run out of memory for one of two reasons: pandas makes copies as it works (its creator, Wes McKinney, recommends 10 times as much RAM as data), or a long-running Python process holds on to memory from earlier jobs. Wekalp handles memory the way databases do: keep the working set small, cap it, spill what does not fit to disk, and release everything when the job ends.
Small frames for Python
DuckDB filters rows and columns first, so pandas or Polars only receives the small result.
An explicit limit on every job
DuckDB, Arrow buffers and Python run inside a set memory limit, sized per job type.
Overflow goes to a scratch disk
Above the limit, DuckDB writes the overflow to a scratch disk, so a large file slows down instead of failing.
Freed when the job ends
Memory and scratch files are released after every job, and each run's peak memory is logged.
A private, size-limited scratch disk
A standard Kubernetes emptyDir of 10 GiB, private to the job's container, which runs non-root with a read-only file system.
Encrypted, then deleted
Scratch files sit on encrypted disks and are deleted after each job. Where disk is not permitted, jobs run fully in memory, with more memory reserved; a file larger than the limit then fails.
Memory over time, job after job · schematic, not measured
Exports follow the same rule: large exports run at constant memory, whatever the file size.
Performance
Compute performance in production.
Measured on a regulatory credit-data submission at a live customer. At each size, processing, validation and write-back of the results complete within 300 seconds. Memory grows by 2 GiB for every 75 million extra data points.
Regulatory validation and write-back · resource limits per run · completed within 300 s
How it scales. A larger run gets more CPU and memory, as the chart shows, and spills to the scratch disk rather than failing if a file outgrows its limit. Heavily read tables are stored once per business date instead of recomputed, and compute on Kubernetes autoscales with the number of jobs and users.
This workload runs inside Wekalp Regulatory Reporting.
Common questions
Frequently asked questions.
How do you stop pandas and Python jobs from leaking memory?
We don't rely on leak-free code. Heavy scans and joins run in DuckDB, so pandas only sees a small, pruned frame. Each job runs under a memory limit, and its memory is released when the job ends, so nothing builds up from one job to the next.
What happens when a job needs more memory than it has?
DuckDB spills to a size-limited 10 GiB scratch disk beyond its limit, so the job slows down rather than fails. If a job is ever stopped, nothing it wrote is visible, and the scheduler retries it from the last checkpoint.
Does writing to disk during processing create a security risk?
The scratch disk is private to the job's container, limited to 10 GiB, and deleted when the job completes and when the container stops. The container runs as non-root with a read-only file system, so the scratch disk is the only writable path, and the disks are encrypted at rest. If disk scratch is not permitted, processing can run fully in memory instead, at the cost of more reserved memory.
How fast is it on a real workload?
In production, a regulatory credit-data validation of 300 million data points, about 1.9 million loan records of 158 fields each, completes within 300 seconds on 7 vCPU and 18 GiB, including writing the results back. 150 million data points complete in the same time on 4 vCPU and 14 GiB.
What kinds of tables does the platform build?
Four kinds: source tables, which hold data exactly as it arrives (transactional, positional, master or master with history); worktables, the working data teams used to build in spreadsheets; business tables, consolidated and query-ready for analytics, APIs and AI; and SQL views, tables defined with a query. Any of them is stored or live, all on one storage tier, so there is no separate serving database.
What does the file sweep do?
It decides how a run reads records spread across several current files. In most runs no merged file is created: DuckDB queries the current parts directly, across the file list or one part at a time, and unions the results, which keeps large reads light. A merged file is built only when an execution or assertion has to see every record in one place, such as a duplicate check across all loads. Removed and superseded files are never read.
How are Excel formulas turned into rules?
Spreadsheet formulas are parsed into versioned rules and run as SQL in DuckDB or Python in Polars. An exact-match VLOOKUP becomes a left join with an explicit no-match flag, an approximate VLOOKUP becomes an as-of range join, and SUMIFS becomes a filtered GROUP BY. Each rule is tested in a sandbox before it touches live data and can be read back as spreadsheet logic.
What if a calculation can't be written in SQL?
It runs as Python on Polars or pandas, inside the same governed execution path, with data handed over from DuckDB through Apache Arrow without copying.
Talk to us about your compute workloads.
Send us one report, reconciliation or pipeline that takes too long or keeps failing today. We'll run it on the platform and show you the execution path, the memory it needed and the time it took.
- Benchmarked on your data shape, not a demo set
- A walkthrough of the engine and scheduler for your architects
- A comparison with your current infrastructure spend