Home How We Work Security Assurance Resources
/ Technology · Compute Architecture

One compute engine for queries and orchestrated runs.

Every query, from an analyst, an AI conversation or an API, and every orchestrated run uses the same in-process engines — DuckDB, Apache Arrow and Polars or pandas — reading Parquet where it sits. Every job runs under a memory limit and gives it all back when it finishes. Results are written back as new, versioned tables. No idle capacity to pay for, and no separate database to keep in sync.

See how it works Bring us your slowest report
In-process engines · Four kinds of tables, stored or live · SQL directly on files · Built-in scheduler · Memory capped per job · Single-tenant, cloud or on-premises
Illustrative query plan. Figures are an example, not a benchmark.
Zero
Idle compute

Nothing to keep warm or pay for while no job is running.

300M
Data points in 5 minutes

Validated and written back, on 7 vCPU and 18 GiB, in production.

Capped
Memory for every job

Released when the job ends, so leaked memory never builds up.

No database
To load or keep in sync

Full SQL runs directly on the governed Parquet files.

Why compute matters

Why typical data compute is costly, fragile or ungoverned.

Most data compute is either a warehouse billed for its peak or Python scripts that leak memory and leave no record, with SQL, Python, schedules and spreadsheet formulas spread across both. It fails in four ways.

01 · Cost

Compute billed for peak capacity

Warehouses and always-on compute are sized for month-end, then billed by the hour or the credit, busy or idle.

Wekalp: compute is used only while a job runs. No idle compute, no credits.
02 · Memory

Unbounded Python memory growth

A long-running pandas process copies frames and rarely hands memory back. The job that ran fine on Monday is killed on Friday.

Wekalp: every job runs under a memory cap, overflow goes to a scratch disk so large files slow down instead of failing, and memory is released after every job.
03 · AI access

AI tools given direct access to data

Chat and copilot tools are often connected straight to tables, so what the model can see depends on how it was set up.

Wekalp: the model sees only metadata and writes the query, which runs under the user's row-level rules.
04 · Failed runs

A failed run needs an engineer

Someone has to dig through logs to find the cause, rebuild the inputs and re-run by hand, and the old output is often overwritten.

Wekalp: the Command Center shows every run with its inputs and rule versions, and business owners re-run with the input versions they choose.
Overview

Two workloads, one engine.

Compute does two jobs. Querying answers a request for data, from a technical user, an analyst, a business user in conversation or another system. Orchestration runs automated batches that build downstream tables, visuals and reports. Both use the same engine. Orchestration adds a layer of control and governance on top, and the Command Center puts that control in people's hands.

Part 2Querying
  • Build tablesTechnical users
  • Explore dataAnalysts
  • Ask in conversationBusiness users
  • Published APIsApps and third parties
Part 3Orchestration
  • Automated batch runsDownstream data, visuals and reports
Control layer DecidePlanRunRecord Operated from the Command Center
Part 1Shared engine
CompileExecuteWrite backStore Memory capped, spilled and released on every job
Part 1Shared engine
1.1 · How a job runs

How a compute job runs.

A dashboard click, an API call, an AI agent's question and a nightly batch all take the same route. Compute sits between a governed compiler and versioned storage, and holds nothing between jobs.

  1. 01AskOne door in
  2. 02CompileOne SQL plan
  3. 03OrchestrateDependency order
  4. 04ExecuteIn-process engines
  5. 05Write backACID, versioned
  6. 06StoreOne atomic commit
01 · Ask

One entry point

People, apps, AI agents and the scheduler all use the same entry point.

02 · Compile

Compiled to a governed SQL plan

The versioned definition, its parameters and the asker's row filter become one SQL plan.

03 · Orchestrate

Queries run at once, batches are planned

A query goes straight to the engine; an orchestrated run is decided, planned and ordered first (Part 3).

04 · Execute

In-process execution, memory-capped

The file sweep feeds DuckDB, which hands data to Polars or pandas through Arrow.

05 · Write back

Written back to the data lake, ACID intact

Results land as new versions with the same Delta Lake-style ACID guarantees as the storage layer. Earlier versions stay readable.

06 · Store

Atomic commit to the ledger

New Parquet files appear in your object store in one ledger commit. Storage Architecture →

1.2 · Engines and execution

Engines and query execution.

Three engines, one path

DuckDB, Arrow and Polars or pandas.

No single engine is best at everything. DuckDB does the relational work, Polars or pandas run the Python logic on a small, pruned frame, and Apache Arrow hands data between them without copying it.

Open-source editions throughout: DuckDB and Polars (MIT), pandas (BSD-3), Apache Arrow (Apache 2.0).

One execution path, three engines
Third-party, TPC-H-derived results published by Polars (May 2025). Not a Wekalp benchmark.
Read less

Reading only the data a job needs.

Every read is cut down in four steps before any processing starts: files, then versions and dates, then columns, then rows. Everything else stays in storage. In our own platform, scanning in DuckDB before pandas cut memory on large operations by 80–90% (The Kitchen Revolution).

Step 1 · Files

File pruning

The ledger identifies the files a query, process or run needs. No other file is opened.

Step 2 · Versions and dates

Partition pruning

Within those files, only the specific versions and business dates the run needs are read.

Step 3 · Columns

Column pruning (projection pushdown)

Only the columns the query or process uses are read and handed on.

Step 4 · Rows

Row pruning (predicate pushdown)

Filters are applied as the data is read, and row groups that cannot match are skipped.

File sweep

The file sweep: merge only when it is required.

The records a run needs often sit across several current files. Building one merged file from them is expensive, so in most runs no merged file is created: DuckDB queries the current parts directly and unions the results. A merged file is built only when an execution or assertion has to see every record in one place.

  • Ledger-driven: as in Delta Lake, only the files the log lists as current are read.
  • Light by default: one query across the file list, or one per part, with the results unioned. Nothing is copied.
  • Merged when mandatory: for example duplicate checks across all loads, or totals reconciled across the whole set.
Two ways to read the current files
Dashed files are removed or superseded. They stay in storage for audit and are never read.
Business rules

Excel formulas become versioned rules.

Spreadsheet logic is translated into versioned rules, then run as SQL in DuckDB or as Python in Polars. Each rule is tested in a sandbox before it touches live data, and can be read back as spreadsheet logic.

VLOOKUP · exact match → LEFT JOIN
Excel formula
=VLOOKUP(C2, Branch_Master!A:D, 4, FALSE)
Versioned rule
R-114 v3
Loans.region ← BranchMaster.region
key:       branch_code
match:     exact
no match:  flag, do not guess
SQL · DuckDB
SELECT l.*, b.region,
       b.branch_code IS NULL AS r114_no_match
FROM loans l
LEFT JOIN branch_master b
  ON l.branch_code = b.branch_code
Python · Polars
loans.join(
    branch_master.select("branch_code", "region"),
    on="branch_code", how="left",
).with_columns(
    pl.col("region").is_null().alias("r114_no_match")
)

A missing branch becomes a logged validation result, not a silent #N/A, and a duplicate key in the master is flagged instead of Excel quietly taking the first row.

VLOOKUP · approximate match → ASOF JOIN
Excel formula
=VLOOKUP(G2, Rate_Bands!A:B, 2, TRUE)
Versioned rule
R-207 v1
Loans.rate_band ← RateBands.band
range:     balance ≥ lower_bound
match:     nearest lower band
SQL · DuckDB
SELECT l.*, r.band
FROM loans l
ASOF LEFT JOIN rate_bands r
  ON l.balance >= r.lower_bound
Python · Polars
loans.sort("balance").join_asof(
    rate_bands.sort("lower_bound"),
    left_on="balance", right_on="lower_bound",
)

A lookup that silently depends on the sheet being sorted becomes an explicit range rule.

SUMIFS → GROUP BY
Excel formula
=SUMIFS(Loans!K:K, Loans!C:C, A2, Loans!H:H, "Active")
Versioned rule
R-031 v5
Branch.active_balance = SUM(Loans.outstanding_balance)
group:     branch_code
filter:    status = 'Active'
SQL · DuckDB
SELECT branch_code,
       SUM(outstanding_balance) AS active_balance
FROM loans
WHERE status = 'Active'
GROUP BY branch_code
Python · Polars
loans.filter(pl.col("status") == "Active")
     .group_by("branch_code")
     .agg(pl.col("outstanding_balance").sum()
          .alias("active_balance"))

One rule replaces a formula copied down thousands of rows, and runs on every branch at once.

1.3 · Data artifacts

Table types the engine reads and writes.

Data moves from source tables to worktables to business tables, with SQL views for anything a data team prefers to write as a query. All of it is Parquet in the same versioned storage, so there is no separate serving database to license, size or secure.

Source tables

Source tables, by load type.

Nothing is changed on the way in. The type of source decides how each new load is treated.

Transactional

Each load is added to the ones before it: payments, journals, trades.

Positional

Each load replaces that date's picture, and can carry forward: balances, inventory.

Master

Each load is the current list, and the latest version is used: customers, products.

Master with history

Rows carry start and end dates, so you can ask what was true on any date.

Built on top of sources

Worktables, business tables and SQL views.

Every kind is versioned by business date, quality-checked and read through the same access rules.

Worktables

The working tabs teams built in Excel, learned once and rebuilt every run, with audited adjustments.

Business tables

Sources and worktables flattened into one wide, query-ready table, with a glossary for AI.

SQL views

Tables defined by a query, for exploring, tidying a source or building a business table.

Stored or live

Built once per date and kept, or run as a query when read. One setting, nothing else changes.

How versions, golden records and the medallion lifecycle work: Storage Architecture.

1.4 · Memory

Memory management for Python and DuckDB jobs.

Data jobs usually run out of memory for one of two reasons: pandas makes copies as it works (its creator, Wes McKinney, recommends 10 times as much RAM as data), or a long-running Python process holds on to memory from earlier jobs. Wekalp handles memory the way databases do: keep the working set small, cap it, spill what does not fit to disk, and release everything when the job ends.

Step 1 · Read less

Small frames for Python

DuckDB filters rows and columns first, so pandas or Polars only receives the small result.

Step 2 · Cap

An explicit limit on every job

DuckDB, Arrow buffers and Python run inside a set memory limit, sized per job type.

Step 3 · Spill

Overflow goes to a scratch disk

Above the limit, DuckDB writes the overflow to a scratch disk, so a large file slows down instead of failing.

Step 4 · Release

Freed when the job ends

Memory and scratch files are released after every job, and each run's peak memory is logged.

Step 5 · Isolate

A private, size-limited scratch folder

A standard Kubernetes emptyDir of 10 GiB, private to the job's container, which runs non-root with a read-only file system.

Step 6 · Clean up

Encrypted, then deleted

Scratch files sit on encrypted disks and are deleted after each job. Where disk is not permitted, jobs run fully in memory instead.

Memory over time, job after job · schematic, not measured
1.5 · Performance

Compute performance in production.

Measured on a regulatory credit-data submission at a live customer. At each size, processing, validation and write-back of the results complete within 300 seconds. Memory grows by 2 GiB for every 75 million extra data points.

Regulatory validation and write-back · resource limits per run · completed within 300 s
Three levels of workload tested for performance: regulatory credit-data validation. Each record is a 158-field loan contract (terms, payments, arrears, PFRS 9 impairment, up to six guarantees with collateral and six linked parties), so 300M data points is about 1.9 million contracts. Every field gets format, numeric and master-data checks, and the results are written back. A data point is one field value in one record. Limits shown; requests are 3, 4 and 6 vCPU and 12, 14 and 16 GiB.

How it scales. A larger run gets more CPU and memory, as the chart shows, and spills to the scratch disk rather than failing if a file outgrows its limit. Heavily read tables are stored once per business date instead of recomputed, and compute on Kubernetes autoscales with the number of jobs and users.

This workload runs inside Wekalp Regulatory Reporting.

Part 2Querying
2.1 · Ways to query

Four ways to query the same data.

Whoever asks, the request takes the shared path: compiled from governed definitions with the asker's row-level rules, run on the files in place, and logged.

Technical users

Build and update tables

SQL on the files to build or update stored tables and materialized views.

Analysts

Explore data

Query any table, at any version or business date, without loading it anywhere.

Business users

Ask in conversation

Questions in plain language, answered as tables or charts. The model never sees the data.

Apps and third parties

Published APIs

Every table is available through a published API, on an API key basis, under the same access rules.

2.2 · SQL on files

SQL directly on Parquet files.

The SQL that runs the platform's own pipelines runs directly on the Parquet in storage. Nothing is imported into a database or warehouse first, and every query still goes through the same compiler, row-level rules and audit log.

  • One SQL, every use: analysis, scheduled jobs, APIs and AI agents.
  • Any table, any date: as it stands now or on an earlier business date.
  • Nothing to sync: no import job, no second copy, no serving database.
Getting from files to an answer
2.3 · Conversational analytics

How AI questions become DuckDB queries.

Conversational analytics and visualization use the same query path as everything else. The model reasons over governed metadata to write the query; it is never given the data. The query then runs through the same compiler and access rules as a dashboard or an API call.

Question in, with the asker's role

A router sends the question to the right agent, carrying who is asking.

The model sees metadata, not data

Metric and entity definitions, relationships and the glossary. Never rows.

Compiled under the user's access rules

The compiler adds the row filter, so a prompt cannot widen what a role can see.

Run on the files, returned as a chart

DuckDB runs the query in place; the answer comes back as a table or chart.

Part 3Orchestration
3.1 · What orchestration adds

Orchestration: the same engine, with control.

Automated batches build downstream data, visuals and reports on the shared engine. What orchestration adds is control and governance: the system decides when to run and what to rebuild, plans the order, logs every step and writes versioned results, and people can step in at any point. There is no separate workflow engine to license.

3.2 · Execution engine

How a run is decided, planned, run and recorded.

Before anything is calculated, the engine decides whether the run can start, works out what it needs, what can be reused and in what order to build the rest. That keeps large batch runs fast, within memory and fully traceable.

One run, stage by stage · reused or recalculated
Dashed: unchanged, reused at its current version. Coral: recalculated as a new version. Illustrative.
01
DecideShould the run start?
  1. 1TriggersOn a schedule, on an event such as a file landing or an upstream run finishing, or by hand.
  2. 2ReadinessInputs and dependencies are checked; a missing input holds the run.
  3. 3Sign-off gatesSteps that need approval wait for it before continuing.
02
PlanWork out what the run needs
  1. 4Building blocksFind every source, table and report the run touches.
  2. 5VersionsPick the right version of each one.
  3. 6OrderSequence the artifacts, then the columns and cells inside each.
  4. 7Reuse or recalculateRebuild only what changed; reuse the rest.
03
RunCalculate in stages
  1. 8Stage by stageWorktables, then intermediate tables, then reports, logged as it runs.
  2. 9SQL and Excel formulasBoth run in one path; formulas are parsed into scripts by document intelligence.
  3. 10RetriesFailed steps retry as configured, then alert the owner and pause.
04
RecordVersion and audit everything
  1. 11New versionsLog the version of every artifact used and created.
  2. 12Audit trailWhich artifacts, which versions, and how each value was calculated.

Intermediate tables can also be kept as stored tables when users need them for audit.

3.3 · Command Center

Monitoring and re-running jobs.

The Command Center is the interface for that control. Business owners run the whole show from one screen. Watch six moments, from spotting a failed run to re-running it with the inputs they choose.

  1. Step 01

    Watch every run

    Scheduled report groups sit beside the batch history: what ran, what triggered it and how long it took.

    • LoggedBatch 45 · scheduler · 3m 25s
    • FailedBatch 42 · costing report
    • ActionPause, resume or execute now
    Command Center with scheduled report groups and a batch execution history showing completed and failed batches with trigger and duration
  2. Step 02

    See what a run depends on

    Open a group to see its reports and every source, master file and parameter, with the version each will use.

    • ReadyPrimary sales data · v1
    • ReadyProduct master · v3
    • MissingSKU master · the run waits
    A report group expanded to show its five reports and dependencies, with source and master file versions and a missing SKU master flagged
  3. Step 03

    Schedule or start a run

    Pick report groups, then set the frequency, time and start date, or run a group straight away.

    • GroupPrimary sales dashboard demo
    • ScheduleDaily at 06:00 from 01 Dec 2026
    Schedule new execution dialog with one report group selected and a daily 06:00 schedule configured
  4. Step 04

    Spot the failed run

    The execution history marks the failed run. Open it to see what triggered it and which group it belongs to.

    • FailedRun 141 · 30 Jun · 08:16 PM
    • ActionAnalyse execution, then re-run
    Manual run screen with the execution history showing run 141 as failed and its details opened
  5. Step 05

    Review before re-running

    The review lists dependent reports and sources, and lets you pick the version of each input the re-run reads.

    • Scope5 reports · no dependent reports
    • VersionASM target data · v5 selected
    • ActionAdd to queue
    Re-execution queue review listing the dependent sources for a report group, each with its selected version
  6. Step 06

    Confirm and execute

    Confirm the queued run and choose whether to rebuild tables and keep adjustments already posted.

    • OnRecreate tables from inputs
    • OffRetain report adjustments
    • RunPrimary sales dashboard executes
    Execute confirmation listing the reports to run, with options to recreate tables and retain report adjustments
Comparison

How it compares with warehouses, lakehouses and scripts.

Lakehouse platforms, warehouses and Python scripts each solve part of the problem. The table shows what each typically includes out of the box. These are general patterns, not a review of any one product; many platforms offer options that close some gaps.

Wekalp compared with lakehouse platforms, cloud warehouses and Python scripts
Capability Wekalp Lakehouse
platform		 Cloud
warehouse		 Python
scripts
Cost
Nothing billed while no job runs✓11✕
No per-credit, per-scan or per-query metering✓✕✕✓
Execution
SQL directly on open Parquet files✓✓23
Python logic in the same governed path as SQL✓✓4✕
Memory capped and released for every job✓✓✓✕
Spreadsheet formulas translated into versioned rules✓✕✕✕
Orchestration
Built-in scheduler with event triggers✓✓5✕
Sign-off steps before a run continues✓✕✕✕
Business users re-run with chosen input versions✓6✕✕
Governance and AI
Every run recorded with its inputs and rule versions✓77✕
AI writes queries from metadata, never sees the data✓88✕
AI queries run under the same row-level rules✓88✕
✓ Built in Partial or optional✕ Not supported
  1. Auto-suspend, auto-stop or serverless options reduce idle cost; billing is still per compute-hour or credit.
  2. Through external tables or a lakehouse format, usually alongside native storage.
  3. With a library such as DuckDB or pyarrow, governed by whoever wrote the script.
  4. Through user-defined functions or add-on runtimes.
  5. Native tasks or jobs exist; event-driven runs often need a separate workflow tool.
  6. Time travel makes earlier versions available; choosing them for a re-run is usually an engineering task.
  7. Job logs, query history and table history are kept, but rule versions and approvals are not tied to each run.
  8. AI assistants exist; access rules and data exposure depend on how they are configured.
Open by design

Your data stays readable, inside or outside Wekalp.

Every table is available through published APIs, on an API key basis, so BI tools, warehouses and downstream applications can pull data on their own schedule. Underneath, the data is standard Parquet and the ledger is JSON, both in your own object storage. Leaving never means a migration project.

  • Published APIs per table. Pull any table into another system with an API key.
  • Same access rules on the way out. Row-level rules apply to API requests as they do inside the platform.
  • Parquet in your storage. Readable by any Parquet-compatible engine.
  • Ledger as JSON. The version history sits alongside the data, in your account.
Common questions

Frequently asked questions.

How do you stop pandas and Python jobs from leaking memory?

We don't rely on leak-free code. Heavy scans and joins run in DuckDB, so pandas only sees a small, pruned frame. Each job runs under a memory ceiling, and its memory is released when the job ends, so nothing builds up from one job to the next.

What happens when a job needs more memory than it has?

DuckDB spills to a size-limited 10 GiB scratch area on disk beyond its limit, so the job slows down rather than fails. If a job is ever stopped, nothing it wrote is visible, and the scheduler retries it from the last checkpoint.

Does writing to disk during processing create a security risk?

The scratch area is private to the job's container, limited to 10 GiB, and deleted when the job completes and when the container stops. The container runs as non-root with a read-only file system, so the scratch folder is the only writable path, and the disks are encrypted at rest. If disk scratch is not permitted, processing can run fully in memory instead, at the cost of more reserved memory.

How fast is it on a real workload?

In production, a regulatory credit-data validation of 300 million data points, about 1.9 million loan records of 158 fields each, completes within 300 seconds on 7 vCPU and 18 GiB, including writing the results back. 150 million data points complete in the same time on 4 vCPU and 14 GiB.

What kinds of tables does the platform build?

Four kinds: source tables, which hold data exactly as it arrives (transactional, positional, master or master with history); worktables, the working data teams used to build in spreadsheets; business tables, consolidated and query-ready for analytics, APIs and AI; and SQL views, tables defined with a query. Any of them is stored or live, all on one storage tier, so there is no separate serving database.

Can we query data without loading it into a database?

Yes. The same SQL the platform's pipelines use runs directly on the Parquet files in storage, on any table type and any version. Nothing is imported into a database or warehouse, and every query still applies your row-level rules and is logged.

Does the AI model see our data when we ask questions in chat?

No. For conversational analytics and visualization, the model only sees governed metadata: metric and entity definitions, relationships and the business glossary. It uses that context to write the query, which is compiled with the user's row-level rules and run by DuckDB on the files, the same way a dashboard query runs. The model has no access path of its own, so a prompt cannot widen what a user's role can see.

What does the file sweep do?

It decides how a run reads records spread across several current files. In most runs no merged file is created: DuckDB queries the current parts directly, across the file list or one part at a time, and unions the results, which keeps large reads light. A merged file is built only when an execution or assertion has to see every record in one place, such as a duplicate check across all loads. Removed and superseded files are never read.

How does a run decide what to recalculate?

The execution engine first identifies the building blocks a run needs (sources, worktables, business tables, intermediate tables and reports) and the right version of each. Artifacts whose inputs have not changed are reused; the rest are recalculated in order, worktables first, then intermediate tables, then reports. Every artifact used or created is logged with its version, with an audit trail of how each value was calculated.

How are Excel formulas turned into rules?

Spreadsheet formulas are parsed into versioned rules and run as SQL in DuckDB or Python in Polars. An exact-match VLOOKUP becomes a left join with an explicit no-match flag, an approximate VLOOKUP becomes an as-of range join, and SUMIFS becomes a filtered GROUP BY. Each rule is tested in a sandbox before it touches live data and can be read back as spreadsheet logic.

How do we get data out of Wekalp, or leave?

Every table is available through published APIs on an API key basis, under the same row-level rules, so other systems can pull data on their own schedule. Underneath, the data is standard Parquet and the ledger is JSON, both in your own object storage, readable without Wekalp.

Do we still need Airflow or another workflow tool?

No. The scheduler and Command Center are part of the platform. Runs start on a schedule, by hand, or on events such as a file landing or a process completing. Steps run in dependency order, failures retry as configured, owners are alerted, and runs can be paused, restarted from a checkpoint or held for sign-off, with every run logged.

Do we need a separate warehouse or lakehouse platform?

No. Wekalp includes its own engines, versioned storage and scheduler, so there is no separate warehouse, lakehouse or serving database to license. The engines are open source and chosen for each kind of work.

What if a calculation can't be written in SQL?

It runs as Python on Polars or pandas, inside the same governed execution path, with data handed over from DuckDB through Apache Arrow without copying.

Can it run entirely on-premises?

Yes. Every component runs in containers on Kubernetes in your own data centre, with standard on-premises object storage. No cloud-provider service is required.

Do we need a data engineer to tune it?

No. The platform configures the engines. The main performance decision is whether a table is stored or live, which is one setting on the table and changes nothing that reads from it.

Talk to us about your compute workloads.

Send us one report, reconciliation or pipeline that takes too long or keeps failing today. We'll run it on the platform and show you the execution path, the memory it needed and the time it took.

  • Benchmarked on your data shape, not a demo set
  • A walkthrough of the engine tier and scheduler for your architects
  • A comparison with your current infrastructure spend
Talk to our architects Storage Architecture