@ketecode/loom 0.0.0-stage → 0.2.0

package/skills/dart-testing/SKILL.md

@@ -0,0 +1,164 @@
+---
+name: dart-testing
+description: Writes deterministic Dart and Flutter tests with package:test and flutter_test, fakes collaborators with mocktail, asserts on Futures and Streams with expectLater and stream matchers, controls time with fake_async, and sets up basic golden image tests. Use when adding or fixing unit, widget or golden tests in a Dart or Flutter project, mocking a repository or HTTP client with mocktail, or chasing a flaky async test that hangs, times out or passes only sometimes.
+license: MIT
+compatibility: Any Agent Skills host. Needs the Dart SDK (dart test) or Flutter (flutter test). mocktail and fake_async are pub packages added as dev dependencies.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: testing
+  kete-loom/risk: runs-commands
+  kete-loom/requires: "dart"
+  kete-loom/owner: "Kete-org"
+---
+
+# Dart testing
+
+Tests that run fast, never touch the network, and fail for one reason. Use `dart test` for pure Dart packages and `flutter test` for anything importing Flutter. Integration tests on a device (`integration_test`) and browser E2E are out of scope here.
+
+## 0. Before you write
+
+1. Check `dev_dependencies` in `pubspec.yaml`: `test` (pure Dart) or `flutter_test` (Flutter, from the SDK), plus `mocktail` if present. If the project uses `mockito` instead, follow it; don't add a second mocking library.
+2. Run the existing suite once (`dart test` or `flutter test`) and note failures that were already there.
+3. Mirror the source layout: `lib/src/cart.dart` is tested in `test/src/cart_test.dart`. Files must end in `_test.dart` to be picked up.
+
+## 1. Test shape
+
+```dart
+import 'package:test/test.dart'; // Flutter: package:flutter_test/flutter_test.dart
+
+void main() {
+  group('Cart.total', () {
+    late Cart cart;
+    setUp(() => cart = Cart());
+
+    test('is zero when empty', () {
+      expect(cart.total, Money.zero);
+    });
+
+    test('throws on negative quantity', () {
+      expect(() => cart.add(item, quantity: -1), throwsA(isA<ArgumentError>()));
+    });
+  });
+}
+```
+
+- One behaviour per `test`; name it as a sentence a reader understands in the failure log.
+- Build fresh state in `setUp`; register cleanup with `addTearDown(...)` right where the resource is created.
+- Prefer specific matchers (`equals`, `isA<T>().having((e) => e.code, 'code', 404)`, `containsAll`, `closeTo`) over `isTrue` on a hand-built boolean, so failures print useful values.
+- Run one test while iterating: `dart test test/src/cart_test.dart --name "is zero"`.
+
+## 2. Mocking with mocktail
+
+```dart
+class MockUserApi extends Mock implements UserApi {}
+class FakeUri extends Fake implements Uri {}
+
+setUpAll(() => registerFallbackValue(FakeUri()));
+
+test('maps 404 to UserNotFound', () async {
+  final api = MockUserApi();
+  when(() => api.fetch(any())).thenThrow(const HttpFailure(404));
+  final repo = UserRepository(api);
+
+  await expectLater(repo.load('42'), throwsA(isA<UserNotFound>()));
+  verify(() => api.fetch(any())).called(1);
+});
+```
+
+- Stub Future- and Stream-returning methods with `thenAnswer((_) async => value)` / `thenAnswer((_) => Stream.value(x))`, not `thenReturn`.
+- An unstubbed method that returns a non-nullable type throws at call time; stub every call the code under test makes.
+- `any()` with a custom type needs `registerFallbackValue` once, usually in `setUpAll`. Named arguments use `any(named: 'id')`.
+- `verifyNever(...)` for calls that must not happen; `captureAny()` with `verify(...).captured` to inspect arguments.
+- Mock only boundaries (HTTP, database, platform channels, clock). Never mock the class under test or simple value types; use real instances or a hand-written `Fake`.
+
+## 3. Async code
+
+- Make the test body `async` and `await` the call. A test that returns before its future completes passes without checking anything.
+- Assert on a future with `await expectLater(future, completion(equals(x)))` or `throwsA(...)`. Use `expectLater`, not `expect`, when the matcher is asynchronous.
+- Never use real `Future.delayed` or sleeps to wait for something. Inject the delay or use `fake_async` (section 5).
+- The default per-test timeout is 30 seconds; a hang usually means an un-completed `Completer`, an un-closed stream, or a missing stub. Fix the cause instead of raising `timeout:`.
+
+## 4. Streams
+
+```dart
+test('emits loading then data then closes', () async {
+  final states = bloc.states; // the Stream under test
+  final check = expectLater(states, emitsInOrder([isA<Loading>(), isA<Loaded>(), emitsDone]));
+  bloc.load();
+  await check;
+});
+```
+
+- Start `expectLater` before triggering the events, then await it, so no event is missed on a broadcast stream.
+- Matchers: `emits(x)`, `emitsInOrder([...])`, `emitsThrough(x)`, `emitsError(isA<E>())`, `mayEmit(x)`, `neverEmits(x)`, `emitsDone`.
+- Collect a finite stream with `await stream.toList()` when order and count are all that matter.
+- Close controllers in `addTearDown(controller.close)` so a failing test does not leak listeners into the next one.
+
+## 5. Time with fake_async
+
+Add `fake_async` as a dev dependency. It controls `dart:async` time: `Timer`, `Future.delayed`, `Stream.periodic` and microtasks. It cannot control `DateTime.now()` or `Stopwatch`, because they are not part of `dart:async`.
+
+```dart
+test('retries after 2 seconds', () {
+  fakeAsync((async) {
+    final client = RetryingClient(api);
+    client.send(request);
+    async.elapse(const Duration(seconds: 2));
+    verify(() => api.post(any())).called(2);
+  });
+});
+```
+
+- The callback is synchronous; don't `await` inside it. Use `async.flushMicrotasks()` to run pending futures and `async.elapse(d)` to advance timers.
+- When logic reads the current time, have it use `clock.now()` / `clock.stopwatch()` from `package:clock`; `fakeAsync` overrides those automatically, so they follow `elapse`.
+- In widget tests, time is already fake: use `tester.pump(duration)` instead of `fakeAsync`.
+
+## 6. Widget tests (flutter_test)
+
+```dart
+testWidgets('shows error text when login fails', (tester) async {
+  when(() => auth.login(any(), any())).thenThrow(const AuthFailure());
+  await tester.pumpWidget(MaterialApp(home: LoginPage(auth: auth)));
+
+  await tester.enterText(find.byKey(const Key('email')), 'a@b.co');
+  await tester.tap(find.text('Sign in'));
+  await tester.pumpAndSettle();
+
+  expect(find.text('Invalid credentials'), findsOneWidget);
+});
+```
+
+- Wrap the widget in the minimum ancestors it needs (`MaterialApp`, providers, `MediaQuery`) and inject fakes through constructors or the project's DI.
+- `pump()` renders one frame; `pumpAndSettle()` runs until no frames are scheduled and times out on endless animations (spinners). Use `pump(duration)` for those.
+- Real I/O (files, sockets) does not complete inside the fake zone; wrap it in `await tester.runAsync(() async {...})` or, better, fake it.
+- Find by `Key`, text or semantics label; avoid finding by widget index.
+
+## 7. Golden tests (basics)
+
+```dart
+testWidgets('price tag golden', (tester) async {
+  await tester.pumpWidget(const MaterialApp(home: PriceTag(amount: 1250)));
+  await expectLater(find.byType(PriceTag), matchesGoldenFile('goldens/price_tag.png'));
+});
+```
+
+- Create or update reference images with `flutter test --update-goldens`, look at the PNGs, and commit them. Never update goldens just to make a red test green without checking the diff images written to a `failures/` folder next to the test file (the failure message prints the path).
+- Goldens differ across operating systems and font setups. Generate and compare them on one platform (usually CI's), and tag them (`@Tags(['golden'])`) so they can run separately.
+- Set a fixed surface size (`tester.view.physicalSize` and `devicePixelRatio`, reset in `addTearDown`) so output does not depend on the host.
+- If the project uses a golden helper package, follow it; don't add one unasked.
+
+## Stop and ask the user
+
+- The code under test reads globals, singletons or `DateTime.now()` directly and can't be tested without refactoring production code.
+- Fixing a failing test would mean changing the expected behaviour rather than the code.
+- Goldens would need regenerating across many files.
+
+## Never do
+
+- Hit the real network, real databases or real platform channels in unit or widget tests.
+- Skip (`skip:`) or delete a failing test to get green, or raise timeouts to hide a hang.
+- Use `sleep` or real delays to wait for async work.
+
+## Report back
+
+Tests added or changed (file and test names), the command run (`dart test ...` / `flutter test ...`) with pass/fail counts, and any behaviour left untested and why.

package/skills/dart-testing/evals/triggers.json

@@ -0,0 +1,18 @@
+{
+  "should_trigger": [
+    "Write mocktail tests for my Dart UserRepository that wraps an http client",
+    "How do I test that this Dart stream emits loading, then data, then closes?",
+    "My flutter widget test times out on pumpAndSettle, how do I fix it?",
+    "Add a golden test for the PriceTag widget in our Flutter app",
+    "Test this retry logic with a 2 second delay in Dart without actually waiting",
+    "mocktail says I need registerFallbackValue for my custom type, what does that mean"
+  ],
+  "should_not_trigger": [
+    "Convert this Dart enum and switch into a sealed class hierarchy",
+    "Review my Flutter screen for stream subscription leaks and unawaited futures",
+    "dart run build_runner build fails with conflicting outputs, fix it",
+    "Write pytest fixtures for my FastAPI service",
+    "Set up Playwright end-to-end tests for our web checkout",
+    "Write Swift Testing tests with #expect for my iOS view model"
+  ]
+}

package/skills/data-analysis-notebooks/SKILL.md

@@ -0,0 +1,113 @@
+---
+name: data-analysis-notebooks
+description: Keeps Jupyter and other notebook analyses reproducible from a clean kernel, with a set project layout, a pinned environment, raw data treated as read-only, pandas and polars code free of chained assignment and silent dtype or join errors, honest plots, and statistics that avoid p-hacking, multiple-comparison and leakage traps, then moves code that has settled into tested modules. Use when starting or cleaning up an exploratory analysis notebook, when a notebook gives different results on rerun, when reviewing the statistics or charts in an analysis, or when turning notebook code into a script or package others can run.
+license: MIT
+compatibility: Any Agent Skills host. Examples use Python with Jupyter, pandas or polars, and matplotlib; the rules carry over to R Markdown, Quarto and other notebook tools.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: runs-commands
+  kete-loom/owner: "Kete-org"
+---
+
+# Data analysis notebooks
+
+A notebook is finished when someone else can run it top to bottom on a fresh kernel and get the same numbers and figures.
+
+## 0. Assess first
+
+1. Restart the kernel and run all cells (`jupyter nbconvert --to notebook --execute analysis.ipynb --output /tmp/check.ipynb`). Record which cell fails or which numbers change. This is the baseline.
+2. Find the environment file (`pyproject.toml` with a lock file, `requirements.txt`, `environment.yml`). If none exists, create one before changing code.
+3. Find where the data comes from. If it is a file only on someone's laptop or a query with no saved text, stop and ask the user how to get it reproducibly.
+
+## 1. Project layout
+
+```
+project/
+├── data/raw/          # read-only originals, never edited, usually git-ignored
+├── data/processed/    # produced by code, safe to delete and regenerate
+├── notebooks/         # numbered: 01-explore.ipynb, 02-model.ipynb
+├── src/<package>/     # functions that notebooks import
+├── tests/
+├── reports/figures/
+└── pyproject.toml + lock file (or environment.yml)
+```
+
+- Paths are relative to the project root (`pathlib.Path`), never `/Users/me/Downloads/...`.
+- Secrets and connection strings come from environment variables, never cells.
+- Commit notebooks with outputs stripped (`nbstripout --install`) or pair each with a text copy (`jupytext --set-formats ipynb,py:percent`) so diffs are reviewable.
+
+## 2. Pin the environment
+
+- Use one tool consistently: `uv` (`uv add pandas`, commit `uv.lock`), `pip-tools`, `poetry`, or `conda` with an exported `environment.yml`.
+- Pin exact versions of the analysis libraries. pandas 3.0 changed defaults (Copy-on-Write always on, a new `str` dtype for text columns, microsecond datetime resolution), so results can differ between pandas 2.x and 3.x.
+- Register the kernel from that environment (`python -m ipykernel install --user --name <project>`), and record the Python version.
+
+## 3. Notebook hygiene
+
+1. First cell: imports, settings, seeds (`rng = np.random.default_rng(42)`), and paths. No imports further down.
+2. Run cells in order only. No cell may depend on a variable created by a cell below it or by a deleted cell.
+3. Parameters (dates, thresholds, file names) in one cell near the top, tagged `parameters` if the notebook will be run with papermill.
+4. One question per notebook. Over roughly 50 cells or several unrelated questions, split it.
+5. Every result cell that a conclusion relies on prints the number it relies on, not just a chart.
+6. Never mutate raw data in place; write derived data to `data/processed/` with code.
+
+## 4. pandas and polars correctness
+
+Full checklist in `references/dataframe-pitfalls.md`. The core rules:
+
+- **Assignment:** use `df.loc[mask, "col"] = value`. Chained assignment (`df[mask]["col"] = value`) never updates `df` under pandas 3.0 Copy-on-Write; it only raises a `ChainedAssignmentError` warning, which is easy to miss.
+- **Joins:** always pass `validate="one_to_one"` / `"many_to_one"` in pandas `merge` (polars: `validate="1:1"` / `"m:1"`), and check row counts before and after. A silent many-to-many join is the commonest wrong-answer bug.
+- **Types:** declare dtypes when reading (`dtype=`, `parse_dates=`, polars `schema_overrides=`). Check `df.dtypes` and `df.isna().sum()` after loading. IDs with leading zeros are strings.
+- **Missing values:** decide per column (drop, impute, keep as category) and write the decision down; `mean()` skips NaN silently.
+- **Polars:** prefer lazy queries (`pl.scan_parquet(...)...collect()`) on large files; expressions inside `with_columns`, not Python loops or `map_elements` when a built-in expression exists.
+- **Assert as you go:** `assert df["id"].is_unique`, `assert len(out) == len(left)`, value ranges. Cheap asserts catch the errors plots hide.
+
+## 5. Plotting
+
+- Label axes with units, title the question the chart answers, and state n.
+- Start bar charts at zero. Use the same axis scales when panels are compared.
+- Show uncertainty (confidence intervals, error bars, or raw points) whenever you show an estimate.
+- Prefer colour-blind-safe palettes (matplotlib `viridis`, `cividis`) and never rely on colour alone.
+- Save figures from code (`fig.savefig("reports/figures/fig1.png", dpi=200, bbox_inches="tight")`), not screenshots.
+
+## 6. Statistics pitfalls (stop and flag)
+
+Detail and fixes in `references/statistics-pitfalls.md`. Flag to the user, do not silently "fix":
+
+- Many tests, one reported: correct for multiple comparisons (Holm, Benjamini-Hochberg) or report all tests run.
+- Deciding the analysis after seeing the data (choosing subgroups, outlier rules or covariates by which gives p < 0.05).
+- Train/test leakage: scaling, imputation or feature selection fitted on all data before splitting; time-series split at random.
+- Non-independent observations (repeated measures, clustered data) analysed as independent.
+- Correlation reported as causation.
+- p-values without effect sizes and confidence intervals; "not significant" read as "no effect".
+- Simpson's paradox: a trend that reverses within groups. Check the main result by key subgroups.
+
+If the analysis supports a regulated, clinical or financial decision, say a qualified statistician should review it.
+
+## 7. Move code out of the notebook
+
+When a cell is reused, longer than about 20 lines, or the analysis is going into production:
+
+1. Move the function into `src/<package>/`, with a docstring and type hints, and import it back (`%load_ext autoreload` and `%autoreload 2` while developing).
+2. Add a test in `tests/` with a small fixed DataFrame that covers the edge cases you found (empty, missing values, duplicates).
+3. Leave narrative, charts and parameter choices in the notebook. Leave logic in modules.
+4. For scheduled or repeated runs, convert to a script (`jupyter nbconvert --to script`) and clean it up, or run the notebook with `papermill` and parameters. Production pipelines belong in a pipeline tool, not a notebook.
+5. Re-run the notebook from a clean kernel and confirm the numbers match the baseline from step 0.
+
+## Never
+
+- Delete or edit files in `data/raw/`.
+- Report a number that only appears after running cells out of order.
+- Drop rows or outliers without printing how many and why.
+- Commit large data files, credentials or notebooks full of personal data outputs.
+
+## Report back
+
+```
+Reruns clean: <yes/no, which cell failed before>
+Environment: <tool, lock file, Python and key library versions>
+Changes: <layout, hygiene, dataframe fixes, code moved to src/ with tests>
+Statistics flags: <each issue, where, suggested fix>
+Numbers changed vs baseline: <none, or which and why>
+```

package/skills/data-analysis-notebooks/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "My Jupyter notebook gives a different churn rate every time I restart the kernel and run it, help me find out why",
+    "Clean up this exploratory pandas notebook: pin the environment, fix the merge that duplicates rows and move the helpers into a module",
+    "Check the statistics in my A/B test analysis notebook, I ran a t-test on twenty metrics and reported the two that were significant",
+    "Turn the cleaning cells in 02-explore.ipynb into a tested Python package the rest of the team can import",
+    "Set up a project structure for a new research analysis in notebooks so the results are reproducible"
+  ],
+  "should_not_trigger": [
+    "Build an Airflow pipeline that refreshes the warehouse tables every hour",
+    "Create a Grafana dashboard for our API latency metrics",
+    "Fix the shape mismatch error in my PyTorch training loop",
+    "Search PubMed for studies on statin use and dementia risk and summarise them",
+    "Write a Snakemake rule that calls variants with bcftools"
+  ]
+}

package/skills/data-analysis-notebooks/references/dataframe-pitfalls.md

@@ -0,0 +1,49 @@
+# DataFrame pitfalls (pandas and polars)
+
+## Loading
+
+- pandas: `pd.read_csv(path, dtype={"zip": "string", "id": "string"}, parse_dates=["created_at"])`. Prefer Parquet over CSV for intermediates; it keeps types.
+- polars: `pl.read_csv(path, schema_overrides={"zip": pl.String})`, `pl.scan_csv` / `pl.scan_parquet` for lazy reading.
+- Check straight after loading: shape, `dtypes`, null counts, duplicate keys, min/max of numeric and date columns.
+
+## Assignment and copies (pandas)
+
+- pandas 3.0 makes Copy-on-Write the only mode: any indexing result behaves as a copy. `df[df.a > 0]["b"] = 1` never updates `df` (only a `ChainedAssignmentError` warning). Use `df.loc[df.a > 0, "b"] = 1`.
+- `SettingWithCopyWarning` is gone in 3.0; do not rely on warnings to catch mistakes, assert on the result.
+- Avoid `inplace=True`; reassign instead (`df = df.dropna(subset=["x"])`).
+
+## Strings and dates (pandas 3.0)
+
+- Text columns infer to the `str` dtype (backed by PyArrow when installed), not `object`. Code that checks `dtype == object` for strings will break.
+- Default datetime resolution is microseconds when inferred from strings. Compare timestamps with matching time zones; localise with `tz_localize`, convert with `tz_convert`.
+
+## Joins
+
+```python
+before = len(left)
+out = left.merge(right, on="customer_id", how="left", validate="many_to_one", indicator=True)
+assert len(out) == before
+print(out["_merge"].value_counts())
+```
+
+polars: `left.join(right, on="customer_id", how="left", validate="m:1")`.
+
+- Check key dtypes and formats match on both sides. pandas raises on int vs string keys, but `"007"` vs `"7"` (an ID read as a number on one side) silently matches nothing.
+- Strip whitespace and normalise case on string keys before joining.
+
+## Grouping and aggregation
+
+- `groupby(..., dropna=False)` if missing keys are a real group; pandas drops them by default.
+- Use named aggregation (`df.groupby("g").agg(n=("id", "size"), mean_x=("x", "mean"))`) so columns say what they are.
+- `size` counts rows including nulls; `count` excludes nulls.
+- polars: `df.group_by("g").agg(pl.len().alias("n"), pl.col("x").mean())`; order is not guaranteed unless `maintain_order=True`.
+
+## Floats and money
+
+- Compare floats with tolerance (`np.isclose`). Money: integer minor units or `decimal`, not float sums.
+
+## Performance without changing results
+
+- Vectorise; avoid `iterrows` and `apply(axis=1)` for arithmetic.
+- `category` dtype for low-cardinality strings in pandas.
+- polars lazy queries: build the whole query, then `.collect()` once; `explain()` shows the plan.

package/skills/data-analysis-notebooks/references/statistics-pitfalls.md

@@ -0,0 +1,24 @@
+# Statistics pitfalls in exploratory analysis
+
+For each, the sign to look for and what to do. Report the issue to the user; changing a conclusion is their call.
+
+| Pitfall | Sign in the notebook | What to do |
+|---|---|---|
+| Multiple comparisons | Loops over many columns, groups or metrics with a p-value each; only the significant ones reported | Report every test run; adjust with Holm (`statsmodels.stats.multitest.multipletests(p, method="holm")`) or Benjamini-Hochberg (`method="fdr_bh"`) |
+| Garden of forking paths / p-hacking | Outlier rules, subgroups or covariates changed between runs; cells commented out after a non-significant result | Write the analysis plan first; label later analyses "exploratory" |
+| Optional stopping | Collecting more data until p < 0.05 | Fix sample size in advance, or use a sequential design built for it |
+| Leakage | `fit` on the full dataset before `train_test_split`; target-derived features; random split of time series | Fit preprocessing inside a pipeline (`sklearn.pipeline.Pipeline`) on train only; split by time or group (`TimeSeriesSplit`, `GroupKFold`) |
+| Non-independence | Several rows per person, site or day treated as independent samples | Aggregate to the unit of analysis, or use mixed models / cluster-robust errors |
+| Wrong test assumptions | t-test on heavily skewed small samples; chi-square with expected counts below 5 | Check assumptions; use a rank-based test, exact test, or bootstrap |
+| Significance without size | Only p-values reported | Report effect size with a confidence interval |
+| Absence of evidence | "p = 0.2 so no effect" | Say the data cannot distinguish; show the interval |
+| Simpson's paradox | Overall trend differs from within-group trends | Stratify by the likely confounder and show both |
+| Survivorship and selection | Analysing only customers who stayed, only completed records | State who is missing and why; compare with the full population if possible |
+| Regression to the mean | Extreme groups "improve" on remeasurement | Compare against a control group |
+| Base rates | Accuracy reported on imbalanced classes | Report precision, recall, and the base rate; use a confusion matrix |
+| Correlation as causation | Causal wording ("drives", "causes") from observational data | Use associational wording unless the design supports causal claims |
+
+## Randomness
+
+- Seed every random step (`np.random.default_rng(seed)`, `random_state=` in scikit-learn). Report results across several seeds when they vary.
+- Bootstrap intervals: state the number of resamples and the seed.

package/skills/data-pipelines/SKILL.md

@@ -0,0 +1,90 @@
+---
+name: data-pipelines
+description: Batch data pipeline engineering for extract-load-transform jobs run by an orchestrator such as Airflow, Dagster or Prefect, built so every task is idempotent and safe to rerun for any past interval, partitioned by logical date, guarded by row-count, freshness and schema checks before data is published, tolerant of late-arriving records, and traceable through lineage. Use when building or fixing a scheduled ingestion or ETL/ELT job, writing an Airflow DAG, Dagster asset or Prefect flow, planning a backfill, or chasing duplicate or missing rows after a rerun.
+license: MIT
+compatibility: Any Agent Skills host. Needs the project's orchestrator CLI (airflow, dagster or prefect) and warehouse client to run anything; the design rules apply to any batch pipeline.
+metadata:
+  kete-loom/version: "0.1.0"
+  kete-loom/category: data
+  kete-loom/risk: runs-commands
+  kete-loom/owner: "Kete-org"
+---
+
+# Batch data pipelines
+
+Build pipelines where running any task twice for the same interval gives the same result, any past day can be rebuilt on demand, and bad data stops before anyone reads it.
+
+## Ground rules
+
+1. **One run = one interval.** Every task processes a bounded window given by the orchestrator (`data_interval_start`/`data_interval_end` in Airflow, a partition key in Dagster, a parameter in Prefect). Never use `now()` or "since last run" inside task logic; it makes reruns and backfills give different answers.
+2. **Idempotent writes.** A task replaces its output for its window: `DELETE ... WHERE partition = :d` then `INSERT` in one transaction, `MERGE` on a natural key, `INSERT OVERWRITE` of a partition, or write to a staging table and swap. Plain `INSERT`/append is only safe with a dedup step downstream on a unique key.
+3. **Land raw first.** Extract to an immutable raw zone (object storage or a raw schema) keyed by source and interval, exactly as received. Transform from raw, so a logic bug is fixed by rerunning transforms, not by re-hitting the source.
+4. **Check, then publish.** Data quality checks run between build and publish. A failed blocking check stops downstream tasks; consumers never see half-built or failed output.
+5. **Small, single-purpose tasks.** Extract, load, transform, check, publish are separate tasks so one can be retried alone. Pass references (table names, paths) between tasks, not data.
+6. **Secrets from the orchestrator's connection or secret store**, never in DAG code or the repo.
+
+## Workflow
+
+1. **Inventory.** Search the codebase for the orchestrator and its version (`airflow version`, `dagster --version`, `prefect version`), existing pipelines, the warehouse, and how tests run. Follow the existing patterns.
+2. **Define the contract** for each output: grain (one row per what), primary key, partition column, freshness target ("yesterday's data by 06:00 UTC"), and who consumes it. If the grain or key is unclear, **stop and ask the user**.
+3. **Pick the interval and schedule** from the freshness target and how the source exposes changes (full snapshot, `updated_at` watermark, change data capture, append-only events).
+4. **Write extract** for one interval: paginate, retry transient errors with backoff, set timeouts, and save to the raw zone at a deterministic path such as `raw/<source>/<table>/dt=YYYY-MM-DD/`.
+5. **Write load and transform** with one of the idempotent patterns from rule 2. If transforms live in dbt, the orchestrator calls dbt; model design is the `dbt-patterns` skill.
+6. **Add checks** (next section) as their own tasks or asset checks.
+7. **Handle late data** (below).
+8. **Wire retries and alerts**: retries with delay on extract and load, no blind retries on checks, an alert on failure and on a missed freshness target.
+9. **Test**: unit-test transform logic on small fixtures, then run one interval end to end, then run the same interval again and confirm row counts and checksums do not change.
+
+Orchestrator-specific code (Airflow TaskFlow and backfills, Dagster partitioned assets and asset checks, Prefect flows and retries) is in `references/orchestrators.md`. Read it before writing or changing pipeline code.
+
+## Data quality checks
+
+Run at least these on every published table:
+
+| Check | Example rule | Blocking? |
+|---|---|---|
+| Row count | not zero; within ±X% of the trailing 7-interval median | yes for zero, warn for drift |
+| Primary key | unique and not null | yes |
+| Freshness | `max(loaded_at)` within target | yes |
+| Schema | expected columns and types present | yes |
+| Referential | foreign keys exist in the parent table | warn, unless the user says block |
+| Accepted values | status in the known set | warn |
+| Reconciliation | totals match the source (count or sum per interval) | yes for money and counts reported outward |
+
+Decide thresholds with the user; do not invent business tolerances.
+
+## Late-arriving data
+
+- Ask the user how late data can arrive, then pick one:
+  - **Lookback window**: each run also rebuilds the previous N intervals. Simple; costs N times the compute.
+  - **Partition by event time, reprocess on arrival**: detect which past partitions received new raw rows and rerun those.
+  - **Partition by load time**: never late by definition; consumers filter by event time and accept that history changes.
+- Keep both `event_time` and `loaded_at` on every row.
+- Document the cut-off after which late rows are dropped or routed to a quarantine table, and count them.
+
+## Backfills
+
+1. Confirm the task is idempotent (ground rule 2) and that the source still holds the history.
+2. **Stop and ask the user** before backfilling more than a few intervals against a production source or warehouse: confirm the date range, cost, source rate limits, and whether consumers will see changing numbers.
+3. Run oldest first with capped concurrency, then re-run checks for the whole range.
+
+## Lineage
+
+- Name inputs and outputs explicitly (Airflow assets, Dagster asset dependencies, dbt `ref`/`source`) so the orchestrator draws the graph.
+- If the project uses OpenLineage, emit through the orchestrator's OpenLineage integration instead of hand-written events.
+- Record in each output table or its docs: source system, pipeline name, and the code version that built it.
+
+## Never
+
+- Never delete or overwrite the raw zone to "fix" a run.
+- Never mark a failed check as passed, or downgrade it to a warning, to get a run green without the user agreeing.
+- Never run a backfill or `--full-refresh` against production without confirmation.
+- Never use the wall clock to decide which data a task reads.
+
+## Report back
+
+- Pipelines and tasks added or changed, with the output table, grain, key and partition.
+- The idempotency pattern used for each write.
+- Checks added and which ones block.
+- Late-data policy and backfill instructions.
+- Commands run and their results, including the rerun-same-interval test.

package/skills/data-pipelines/evals/triggers.json

@@ -0,0 +1,16 @@
+{
+  "should_trigger": [
+    "Write an Airflow DAG that pulls yesterday's orders from the Shopify API into S3 and loads them into Snowflake",
+    "Our nightly ETL job inserts duplicate rows every time we rerun a failed day, make it idempotent",
+    "Convert this cron-driven Python ingestion script into Dagster partitioned assets with checks",
+    "We need to backfill six months of event data after fixing a bug in the transform step, how do we do it safely?",
+    "Events keep arriving up to three days late and our daily aggregates are wrong, how should the pipeline handle that?"
+  ],
+  "should_not_trigger": [
+    "Restructure our dbt project into staging, intermediate and marts folders",
+    "Build a scraper that collects product prices from a retailer's listing pages",
+    "Decide which KPIs and charts should go on the weekly sales dashboard",
+    "Design ClickHouse tables for ingesting clickstream events with a materialised view rollup",
+    "Explore this CSV in a Jupyter notebook and plot the monthly churn rate"
+  ]
+}

package/skills/data-pipelines/references/orchestrators.md

@@ -0,0 +1,64 @@
+# Orchestrator notes
+
+Check the installed version first; APIs differ between major versions. The official docs for the installed version are the final word.
+
+## Airflow (3.x)
+
+- Write DAGs with the TaskFlow API (`@dag`, `@task`) and pass `schedule=` (the old `schedule_interval` is gone in Airflow 3).
+- `catchup` defaults to `False` in Airflow 3 (`[scheduler] catchup_by_default`). Set it explicitly on every DAG so behaviour does not depend on config.
+- Read the window from the task context: `data_interval_start`, `data_interval_end`, `logical_date`. With the default `CronTriggerTimetable` in Airflow 3 the data interval can be zero-length; if the task needs a real window, use `CronDataIntervalTimetable` or compute the window from `logical_date` explicitly, and test it.
+- Datasets were renamed **assets** in Airflow 3 (`from airflow.sdk import Asset`). Declare `outlets=[Asset(...)]` on producers and `schedule=[Asset(...)]` on consumers for data-aware scheduling and lineage.
+- Retries: `default_args={"retries": 3, "retry_delay": timedelta(minutes=5), "retry_exponential_backoff": 2.0}`. Current Airflow 3 docs type `retry_exponential_backoff` as a float multiplier (0 = constant delay, 2.0 = doubling); Airflow 2 took a bool, so check the installed version. Set `execution_timeout` on every task.
+- Limit parallel runs with `max_active_runs` on the DAG and pools for shared sources.
+- Keep top-level DAG code cheap: no network or database calls at import time; the parser runs it constantly.
+- Backfill (Airflow 3): `airflow backfill create --dag-id <id> --from-date 2025-01-01 --to-date 2025-01-31 --reprocess-behavior failed --max-active-runs 2`. `--reprocess-behavior` is `none`, `failed` or `completed`.
+- Test one run locally with `airflow dags test <dag_id> <date>`.
+- Secrets come from Connections and Variables backed by a secrets backend, read inside the task, not at parse time.
+
+## Dagster
+
+- Model outputs as **software-defined assets** (`@dg.asset`); dependencies come from function parameters or `deps=`.
+- Partition by time with `DailyPartitionsDefinition(start_date="2025-01-01")`; read `context.partition_key` (or `context.partition_time_window`) inside the asset and write only that partition.
+- Quality checks: `@dg.asset_check(asset=my_asset)` returning `AssetCheckResult(passed=..., severity=...)`, or `blocking=True` so downstream assets don't materialize on failure.
+- Freshness and schedules: build a job from an asset selection and attach a schedule, or use declarative automation (`automation_condition=`).
+- Backfill partitions from the UI or `dagster asset materialize --select <asset> --partition-range <start>...<end>` (the `--partition-range` flag appears in recent `dagster asset materialize --help` output but not in the CLI reference page; Dagster now recommends the `dg` CLI, so check `--help` for the installed version).
+- Resources (`ConfigurableResource`) hold connections; secrets come from environment variables via `dg.EnvVar("NAME")`.
+
+## Prefect (3.x)
+
+- `@flow` for the pipeline, `@task` for steps. Pass the interval as flow parameters (`start: date, end: date`) so a backfill is just a run with old parameters.
+- Retries: `@task(retries=3, retry_delay_seconds=[10, 60, 300])`; set `timeout_seconds`.
+- Task caching: the default `cache_policy` keys on inputs, task source and the flow run ID, so a new run recomputes. A custom policy such as `INPUTS` can skip work across runs; make sure the key includes the interval, or use `NO_CACHE` for writes, otherwise a rerun silently does nothing.
+- Deploy with `prefect deploy` / `flow.serve()` and a cron schedule; secrets come from Prefect `Secret` blocks or environment variables.
+- Concurrency limits for shared sources: global concurrency limits or tags.
+
+## Idempotent write snippets
+
+Delete-insert one partition (Postgres, Snowflake, Redshift style):
+
+```sql
+BEGIN;
+DELETE FROM analytics.orders_daily WHERE order_date = :ds;
+INSERT INTO analytics.orders_daily SELECT ... FROM raw.orders WHERE order_date = :ds;
+COMMIT;
+```
+
+Merge on a natural key:
+
+```sql
+MERGE INTO analytics.customers t
+USING staging.customers_batch s ON t.customer_id = s.customer_id
+WHEN MATCHED THEN UPDATE SET ...
+WHEN NOT MATCHED THEN INSERT (...) VALUES (...);
+```
+
+BigQuery or Spark/Hive partition overwrite: write the interval's rows with `INSERT OVERWRITE` (Spark, with dynamic partition overwrite mode) or a `WRITE_TRUNCATE` load job targeting the partition decorator (`table$20250131`) in BigQuery (with `bq load`, pass `--replace=true` to overwrite the partition instead of appending).
+
+## Sources
+
+- Airflow 3 upgrade notes (catchup default, `CronTriggerTimetable`, `airflow.sdk.Asset`): https://airflow.apache.org/docs/apache-airflow/stable/installation/upgrading_to_airflow3.html
+- Airflow CLI (`backfill create`, `dags test`): https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html
+- Airflow Task SDK `BaseOperator` (`retry_exponential_backoff`): https://airflow.apache.org/docs/task-sdk/stable/api.html
+- Dagster asset checks: https://dagster.io/docs/guides/test/asset-checks and https://dagster.io/docs/api/dagster/asset-checks
+- Prefect caching and retries: https://docs.prefect.io/v3/concepts/caching and https://docs.prefect.io/v3/how-to-guides/workflows/retries
+- BigQuery partition loads: https://docs.cloud.google.com/bigquery/docs/load-data-partitioned-tables