From notebook to a training pipeline

The starting point is forecast.ipynb: data loading, cleaning, feature engineering, training and evaluation, all in one notebook with cells run in a particular order. It works on the author's laptop. Our first job is to make it reproducible and testable.

Restructure into a package

crumb/
  src/crumb/
    data.py        load and validate raw data
    features.py    feature engineering
    train.py       training entry point
    evaluate.py    metrics and reports
  tests/
  configs/train.yaml
  pyproject.toml

Move each logical step from the notebook into a function with clear inputs and outputs. Keep the notebook for exploration, but make it import from the package instead of defining its own copies.

One entry point

def main(config_path: str) -> None:
    cfg = load_config(config_path)
    raw = load_sales(cfg.data.path, start=cfg.data.start, end=cfg.data.end)
    validate_sales(raw)
    X, y = build_features(raw, cfg.features)
    X_train, X_val, y_train, y_val = time_split(X, y, cfg.split.val_weeks)
    model = train_model(X_train, y_train, cfg.model)
    metrics = evaluate(model, X_val, y_val)
    save_artifacts(model, metrics, cfg.output_dir)

Running uv run python -m crumb.train configs/train.yaml now produces the same result every time, given the same data and config.

Configuration, not code edits

Parameters such as the date range, feature options and model hyperparameters live in a YAML config file. Changing an experiment means changing the config, which is easy to review and to record alongside results.

Time aware splits

Sales forecasting is a time series problem. A random split would leak future information into training, giving optimistic scores. We use a time based split: train on earlier weeks, validate on the most recent weeks. The notebook used a random split, and fixing this alone changed the validation error noticeably, a common discovery when productionising.

Pin the environment

Lock dependency versions with a lock file so the training environment is identical across machines and over time. Record the Python version too. A model trained with one version of a library may load incorrectly or behave slightly differently with another.

Seeds and determinism

Set random seeds for every library that uses randomness. Some algorithms remain slightly non deterministic with multiple threads; know where that happens, and judge improvements by more than tiny differences.

Tests for the pipeline

Write tests for the feature functions with small, hand made inputs where you know the correct output, such as the day of week and holiday flags. Feature code is where training serving skew begins, so it deserves the most tests.