API reference — Data¶
Typed ingest, the synthetic generator, dataset helpers and column screening. Guide with real examples: Data.
generate¶
pythongenerate((n_rows=10000, n_numeric=5, cardinalities=None, hierarchy_levels=3, branching=None, group_effect_strength=0.5, hierarchy_effect_strength=0.3, interaction_strength=0.2, noise=0.1, task='regression', n_classes=2, tail_behavior='gaussian', missingness=0.0, seed=42))
Generate an adjustable benchmark dataset with known ground truth.
The knobs control how much signal lives in numeric effects, categorical
levels, hierarchical groups and their interaction — so learners can be
scored on structure recovery, not just accuracy. Deterministic for a
given seed.
Args: n_rows: number of rows to generate. n_numeric: numeric feature columns; the first two carry linear signal. cardinalities: level count per categorical column (each >= 2). hierarchy_levels: depth of the categorical hierarchy (0 = none). branching: child count per hierarchy level (last value repeats). group_effect_strength: scale of categorical level effects. hierarchy_effect_strength: scale of hierarchy node effects. interaction_strength: scale of the numeric x categorical interaction. noise: scale of the residual noise. task: "regression" or "classification". n_classes: class count for classification (equal-frequency cuts). tail_behavior: "gaussian", "heavy" (Student-t residuals) or "bathtub" (noise grows with the signal — the DAR motivation). missingness: fraction of feature cells set to missing, in [0, 1). seed: RNG seed.
Returns: dict with columns (name -> values, None = missing), target,
labels (classification only) and truth — the true effects plus
truth["variance"], the realized variance share of each component.
Raises: ValueError: invalid parameter combination.
load_dataset¶
pythonload_dataset((name: 'str', auto_download: 'bool' = True) -> 'dict')
Load one dataset as name/columns/dtypes/n_rows/n_cols.
columns maps column name to a list with None for missing values;
dtypes is "numeric" or "categorical" (strings/dictionaries).
list_datasets¶
pythonlist_datasets(() -> 'list[str]')
Names of currently available local datasets (parquet stems in repo or cache).
summarize¶
pythonsummarize((values))
Summary of observed values, with missing counts shown by default.
NAs and NaNs are excluded from the statistics but always reported in
n_missing — nothing is silently hidden.
Args: values: numeric values (None/NaN = missing).
Returns: dict with count (observed), n_missing, sum, mean, std,
min, q1, median, q3, max (NaN for the statistics of an
all-missing input).
winnow¶
pythonwinnow((x, y, w=None, min_relevance=0.05))
Winnowing: screen columns by univariate relevance against the target and
keep the relevant ones (the cheap pre-split screening LightGBM's docs call
feature pre-filtering / winnowing).
Args:
x: rows of predictors; None entries are missing.
y: targets (rows with None targets or zero weight are dropped).
w: optional row weights.
min_relevance: keep columns with relevance >= this (default 0.05).
Returns: {"kept": [column indices], "relevance": [score per column]}.
DatasetBuilder¶
pythonclass DatasetBuilder((n_category=None, na_policy='allow', digits=None, add_intercept=False))
Ingest any data into the typed dataset.
Owns the ingestion semantics: n_category category inference, explicit NA
policy, digit rounding and an optional intercept column. Missing is None
or NaN — both are counted in na_counts, and filling is opt-in only.
Args (constructor):
n_category: integral numeric columns with at most this many distinct
levels become categorical; values < 1 mean a fraction of rows
(None = no inference).
na_policy: "allow" (keep missing), "drop" (drop rows containing any
missing) or "impute" (numeric mean, categorical mode).
digits: round floats to this many decimal places (None = no rounding).
add_intercept: append a constant 1.0 column named intercept.
DatasetBuilder.add_categorical¶
pythonadd_categorical((self, /, name, values))
Add a categorical column from strings; the vocabulary is first-seen
order. None = missing.
Args: name: column name. values: one level per row (None = missing).
DatasetBuilder.add_numeric¶
pythonadd_numeric((self, /, name, values))
Add a numeric column. NaN is treated as missing.
Args: name: column name. values: one value per row (None/NaN = missing).
DatasetBuilder.build¶
pythonbuild((self, /))
Build the typed dataset with the configured policies applied.
Returns: dict with columns (name -> values), dtypes ("numeric" or
"categorical"), na_counts (missing per column), n_rows,
n_cols.
Raises: ValueError: inconsistent column lengths, duplicate names, invalid policy usage (e.g. "drop" removing every row).