DataFrame

Functions

.fn.select

Run a functional select with explicit expressions:

.fn.select[table; partitions; where; group; operations; limit]

Python bindings accept polars.Expr in these arguments, including expressions inside lists and dictionaries. Expressions returned by the engine are also converted to polars.Expr. Expression exchange uses Polars JSON serialization; expression structures unsupported by either side can raise conversion errors.

import polars as pl
from chili import ChiliEngine

engine = ChiliEngine(pepper=True)
engine.set_var("trades", pl.DataFrame({
    "sym": ["a", "b", "a"],
    "qty": [1, 2, 3],
}))
result = engine.fn_call(
    ".fn.select",
    ["trades", [], [], [pl.col("sym")], [pl.col("qty").sum()], 0],
)
# result contains sym: ["a", "b"], qty: [4, 2]
engine.shutdown()

Plain strings and symbols remain literal values. For example, grouping by ["sym"] groups by the constant text "sym"; [pl.col("sym")] groups by the column. In Pepper, use col each ks to convert a list of names to column expressions.

collect

Collect a lazy DataFrame into an eager DataFrame. Used with lazy evaluation mode (-L or --lazy).

cols

Get the columns of a dataframe.

describe

Get the description of a dataframe.

explode

Explode a dataframe by a list of rows.

extend

Extend a dataframe with another dataframe without rechunking.

flag

Check if the sort flag with series.

flip

Flip a dictionary into a dataframe.

hstack

Horizontally stack two dataframes.

pivot

Pivot a dataframe.

schema

Get the schema of a dataframe.

transpose

Transpose a dataframe.

unique

Stable row deduplication keeping the first occurrence of each duplicate row.

unpivot

Unpivot a dataframe.

vstack

Vertically stack two dataframes.

xasc

Sort a dataframe in ascending order.

xdesc

Sort a dataframe in descending order.

xrename

Rename columns of a dataframe.

xreorder

Reorder columns of a dataframe.

upsert

Append rows to a dataframe with upsert[table; data] (pepper) or upsert(table, data) (chili).

A table symbol updates the dataframe in global state atomically and returns the number of incoming rows. If the table does not exist, data must be a dataframe and is used to create it. A dataframe value returns a new dataframe containing the appended rows; the original dataframe is unchanged.

upsertn

Append rows and retain at most the last n rows with upsertn[table; data; n] (pepper) or upsertn(table, data, n) (chili). Rows are retained in append order; no sorting or deduplication is performed.

• A table symbol updates global state. Appending and trimming are one atomic operation. The return value is the number of incoming rows, including rows discarded by the limit.
• A dataframe value returns a new dataframe and leaves the original unchanged.
• For table symbols, trimming happens before appending when the incoming batch has at least n rows or the stored table is already larger than n. Only rows that can survive are appended. Ordinary partial updates append in place and then trim, avoiding an extra copy of the stored table. Dataframe-value calls always trim a separate result before appending.
• Schema validation still applies, including when n is zero; a failed update leaves the stored table unchanged. Polars slices may share backing buffers, so the row limit is not a strict limit on resident memory.
• If the table does not exist, data must be a dataframe; the new table contains at most its last n rows.
• n equal to zero retains an empty dataframe with the same schema. Negative limits are rejected.
• When fewer than n rows are available, all available rows are retained. Unlike -n#, upsertn does not repeat rows to reach the limit.

Examples:

t: ([]x: 1 2 3, price: 10.0 20.0 30.0);
added: upsertn(`t, ([]x: 4 5, price: 40.0 50.0), 3);
// added is 2; t contains x: 3 4 5

bounded: upsertn(t, [6, 60.0], 2);
// bounded contains x: 5 6; t still contains x: 3 4 5

t: ([]x: 1 2 3; price: 10.0 20.0 30.0);
added: upsertn[`t; ([]x: 4 5; price: 40.0 50.0); 3];
// added is 2; t contains x: 3 4 5

bounded: upsertn[t; (6; 60.0); 2];
// bounded contains x: 5 6; t still contains x: 3 4 5

insert

Insert rows into a dataframe in global state. It will set the dataframe to the id if it doesn't exist. Also, it will keep the last row by key columns. This is a thread safe operation.