perdix 0.3.0__tar.gz

perdix-0.3.0/CHANGELOG.md

@@ -0,0 +1,222 @@
+# Changelog
+
+## 0.3.0 (2026-11-02)
+
+The first release on PyPI (`pip install perdix`), and the first under the name
+Perdix: the name FeatureLab was taken there. It adds a split by group,
+formulas bred by genetic programming, Ask Claude in rounds, the app in a
+browser, the turbofan engines example and a start for newcomers.
+
+### Added
+
+- A license: Perdix is free software under the GNU LGPL, version 3 or
+  later. What it generates (exported scripts, SQL, recipe files, pipelines,
+  transformed data) may be used under any terms; see LICENSE.
+- Ask Claude in rounds (*Rounds* in the dialog, `perdix propose
+  --rounds`). Each round after the first tells Claude what its earlier
+  proposals gained on the training rows, or why they were dropped; held-out
+  scores are never sent. The prompt is cached across rounds.
+- Find features can breed formulas by genetic programming (*Candidates from:
+  Genetic programming*, `perdix search --generator gp`): + − × ÷, log,
+  sqrt and abs over the numeric columns, bred on part of the training rows and
+  screened on the rest like any other candidate. Each formula found is a
+  *Formula* step. At the 20,000-row cap the breeding takes about 5 s, and a
+  search about as long as one over the fixed list.
+- Perdix in a browser: `perdix webapp` writes one HTML file (about
+  0.3 MB) that runs the core with Pyodide. It loads data and recipes, edits
+  steps, evaluates, finds features, checks for leakage and downloads the
+  recipe, the script or the output. Pyodide and its packages come from the
+  jsDelivr CDN on first open; the data stays in the browser.
+- A split by group (*Split: Group*, `split = "group"`): whole groups of the group
+  column go to the training or the held-out rows, in a seeded order, so no
+  engine, machine or customer has rows on both sides. *Find features* and Ask
+  Claude's rounds keep groups whole in their own check on the training rows
+  too. A step that makes new rows keeps the training groups. FeatureLab 0.2.0
+  cannot read a recipe with this split.
+- A seventh example and tutorial, *Turbofan engines*, on NASA's C-MAPSS data
+  (subset FD001, 100 engines run to failure): the remaining useful life of
+  each engine, the split by engine, smoothing, each engine's own history and
+  drift features. A random split scores the same recipe R²
+  0.989 instead of 0.933. On NASA's own test engines, scored as published
+  results are, the recipe and the plain model *Evaluate* uses reach RMSE 12.1
+  (score 238), within the range of published deep networks. NASA states no
+  license for the data, so Perdix does not ship them: *Settings* (new,
+  Ctrl+,) has a *Datasets* section that downloads NASA's archive from NASA's
+  page, checks it and installs it in `~/.perdix`, or installs a copy we
+  downloaded ourselves; `perdix examples download turbofan-engines` does
+  the same. The browser app leaves this example out.
+- *Formula* has a new option, *Infinite results become missing*. A parameter
+  added after 0.2.0 is saved only when it differs from its default, so a
+  recipe that leaves the option off still opens in FeatureLab 0.2.0.
+
+- A start for newcomers. A table opened for a recipe without steps starts at
+  *Fit on* 80 % (the earliest rows with a time column, else a random sample),
+  and *Time* is set when the table has exactly one date/time column. Before,
+  it started at 100 %, where Evaluate cannot score and the leakage report
+  warns. A note in the Dataset card asks for the target while none is set.
+  Undo takes it all back. The web app starts a CSV file the same way.
+- The empty Recipe card lists the first three suggested steps, each with a
+  reason and a *+*. Before, the suggestions were only under Evaluate.
+- The leakage checks run in the background after every change, on the
+  recipe's output (at most 50,000 rows). The Analysis tab shows how many
+  findings there are, for example *Analysis (2)*, and a new high-severity
+  finding comes as a notification. Before, they ran only on *Run analysis*.
+- *Find features* has a button on the app bar. *Ask Claude* says in its
+  dialog when the `anthropic` package is missing, before anything is asked.
+- *Clip outliers* has a new option, *Outliers*: `clip` (as before), `missing`
+  (they become missing values, for an imputation step to fill) or `drop_rows`
+  (rows with an outlier are removed; not in a scikit-learn pipeline). The SQL
+  export writes all three; the plain scikit-learn export only clipping.
+- *Best k* can score by chi-square (*Score: chi2*), for counts and 0/1 columns
+  against a class target.
+- A new step, *Optimize memory* (`clean.compact`): floats as float32, integers
+  in the smallest type that holds them, and text with few distinct values as
+  category. Its fitted values show the memory before and after.
+- The suggestions propose *Mostly missing* for columns that miss more than
+  half of their values, and leave those columns out of every other rule.
+- `perdix suggest -o RECIPE` writes the suggestions as a recipe to start
+  from, with the roles and the 80 % split. `perdix -h` lists the commands
+  in groups, in the order of the work, and every command's `-h` has a
+  description.
+
+### Fixed
+
+- The Data page cut off the first letters of a new or changed column's name:
+  its width left out the *+* or the dot before the name.
+- Opening the demo after working on another table kept that table's recipe,
+  cleared its target and set none, while the notification named
+  `failure_24h`. The demo now starts a new recipe with its roles, one undo
+  away from the old one; a recipe made for the demo's data is kept.
+- Opening an example over another table warned that the old recipe's roles
+  were "not in the data" before the example's recipe replaced them, and the
+  Dataset card named the file `data.csv` instead of the example.
+- A step that needs a target or a time column said so even when it was set.
+  The note now shows only while the role is unset, as a warning.
+- The *Split* buttons in the Dataset card were cut off at the default window
+  size ("arlies", "andor"), and with DejaVu Sans, the font Linux falls back on
+  without Inter, Segoe UI or Roboto. The left column now widens to what its
+  content needs.
+- *Run analysis* could end at "Not run yet" with nothing to show. The
+  recompute that follows opening a table or editing the recipe refreshed the
+  page while the analysis ran, and the refresh cancelled it even when the data
+  and the recipe were unchanged. A run now finishes unless they change.
+- pandas 3 works. It stores text as a string dtype, not `object`, so date
+  columns were not recognised; text with several UTC offsets now raises
+  instead of giving objects; and short codes like `M01` or `Mar` parse as
+  dates in year 1. All three are handled, and results under pandas 2 are
+  unchanged, bit for bit. Found by the new CI job on Python 3.12.
+- A rare crash: Python's cycle collector could run in a background thread and
+  free Qt objects there. It now runs from a timer in the GUI thread.
+- Ties between equally frequent values were broken by whatever order
+  `value_counts` gave, which changes with the numpy version. *Impute missing
+  values* with *most frequent* now takes the smallest of the tied numbers (as
+  scikit-learn does) or the first tied text, and the analysis's top
+  categories list ties by text. On columns where most values occur once this
+  changes the filled value; everything else is unchanged.
+- Ask Claude summarised the wrong rows after a step that makes new rows
+  (resampling): it took the raw table's training split, so held-out rows
+  entered the statistics sent to Claude and the rows proposals were checked
+  on. It now uses the training split of the recipe's own output.
+
+### Changed
+
+- FeatureLab is now Perdix: another project already has the name
+  `featurelab` on PyPI. The package, the import and the commands are
+  `perdix` and `perdix-gui`, the launcher is `perdix.sh`, and the environment
+  variables are `PERDIX_PLUGINS`, `PERDIX_CACHE_MB` and `PERDIX_VENV`.
+  Plugins go in `~/.perdix/plugins`; move them there from
+  `~/.featurelab/plugins`. A recipe keeps its format version under `perdix`.
+  One saved before, under `featurelab`, opens as before, and FeatureLab 0.2.0
+  opens the new ones. On its first start the GUI takes over FeatureLab's
+  settings and autosaved session. A pipeline pickled by FeatureLab
+  (`featurelab pipeline`) refers to `featurelab` classes and does not load
+  under Perdix: export it again.
+- *Try the demo dataset* opens the Machine failure example's data (the demo
+  and the example were the same data under two names) with an empty recipe,
+  and Getting started's walkthrough beside it. The demo's data now have a
+  file, so the session restores them on the next start.
+- Suggestions leave out a column that nearly equals the target (a target
+  proxy in the leakage report) and suggest dropping it with the identifiers.
+  On the churn example they no longer one-hot encode `cancellation_reason`.
+- Getting started puts the first recipe before the tour of the window and
+  has a section on opening your own data. The README installs `.[io]` for
+  users; `dev` is for working on Perdix.
+- With no step selected, the catalog's description box shrinks to a line, and
+  the Tables hint in the Dataset card to one line, so more steps fit.
+- Find features scores candidates by what they add to a model that already
+  has every column (*Model gain*, after OpenFE's feature boosting) instead of
+  by mutual information, which is still offered (*Score by*, `perdix
+  search --method mi`). A copy of an existing column, or a transform of one,
+  no longer ranks high.
+- Evaluate, Find features and Ask Claude start from the output the GUI has
+  already computed instead of fitting the recipe again. Scoring a candidate
+  or a proposal fits only the step it adds. The scores are unchanged, bit for
+  bit. A feature search on an 86,400 × 291 frame took 54 s and now takes 36 s.
+
+## 0.2.0 (2026-10-20)
+
+This release adds most of the roadmap. A recipe can now be scored, checked
+for drift, extended with suggested or searched features, and exported as SQL.
+Every result of 0.1.0 is unchanged, bit for bit, unless a fix below says
+otherwise.
+
+### Added
+
+- Evaluation. `featurelab evaluate` and the *Evaluate* page score a recipe on
+  held-out rows with a gradient-boosting model, step by step, so we see
+  which steps earn their place.
+- Stability. Population stability index per feature and adversarial
+  validation between training and held-out rows (`featurelab stability`, the
+  *Stability* tab under Analysis).
+- Suggestions and search. `featurelab suggest` proposes steps from the data,
+  shown on the *Evaluate* page; `featurelab search` tries candidate features
+  and keeps those that score (*Features → Find features…*).
+- Ask Claude (`featurelab propose`, *Features → Ask Claude for features…*).
+  Claude proposes features from a summary of the columns, never the rows.
+  Each proposal is checked and scored before it can be added. It needs `ANTHROPIC_API_KEY`
+  and the `ai` extra.
+- Data contracts. `featurelab validate` checks new data against the types,
+  ranges and categories the recipe was fitted on.
+- Second tables, listed under *Tables* in the dataset panel, and a
+  point-in-time aggregate step (`join.aggregate`) that joins them without
+  looking into the future.
+- Geo steps: `geo.distance`, `geo.bearing` and `geo.geohash`.
+- Encoders: weight of evidence, James–Stein, ordered target, rare-label
+  grouping and similarity encoding.
+- Steps: `clean.fuzzy_group`, `ts.catch22`, `ts.decompose`, a forecast
+  `horizon` on the time-series steps, `datetime.holidays`, `text.embed`, and
+  the selectors `select.boruta`, `select.mrmr`, `select.permutation` and
+  `select.stability`. There are now 68 steps in 13 categories.
+- Plugins. Steps in a plugin folder are loaded at start (`featurelab plugins`).
+- Exports. `featurelab export-sql` writes a recipe as a SQL query, and
+  `featurelab export-sklearn` writes a plain scikit-learn pipeline. Both
+  cover a subset of the steps and say which ones they cannot export.
+- Large data. `featurelab apply --chunksize` transforms a file in chunks. In
+  the GUI we can work on a sample: the first rows, a random sample, or the
+  earliest or latest rows by time.
+- Documentation for all of the above, and a changelog.
+
+### Changed
+
+- Run analysis no longer refits the recipe to find the training rows: the
+  session keeps them with each step's output. Mutual information runs on
+  threads. On an 86,400 × 291 frame the analysis took 26 s and now takes 15 s.
+- k-means is bit-reproducible: it runs single-threaded under a lock.
+- Steps and the GUI are faster (shallow copies, partial sorts for KNN,
+  segment indexers for rolling and expanding windows, deduplicated text).
+- The GUI has a new design: an app bar, cards, a welcome screen, toasts and
+  a logo.
+- scikit-learn 1.5 or newer is required.
+
+### Fixed
+
+- About 70 bugs from five reviews. Among them: the training split with
+  repeated index labels, missing times and resampled bins; parsing of times
+  around daylight-saving changes; several encoders; focus in the step editor.
+
+## 0.1.0 (2026-10-20)
+
+First release. A recipe engine with typed step parameters, a step registry,
+training splits and exact pandas code generation; exports to a Python script
+and a scikit-learn pipeline; profiling, correlation, VIF, target relevance and
+a leakage report; a command line and a PySide6 GUI.