@zigrivers/scaffold 3.7.0 → 3.9.0

package/content/knowledge/library/library-versioning.md

@@ -0,0 +1,300 @@
+---
+name: library-versioning
+description: Semver discipline, breaking change detection, release automation, and changelog management for published libraries
+topics: [library, versioning, semver, breaking-changes, release-automation, changelog, changesets]
+---
+
+Library versioning is a communication protocol with consumers. Semver (Semantic Versioning) is not merely a numbering scheme — it is a contract about backward compatibility. Breaking that contract without a major version bump is one of the most damaging things a library can do. Consumers set version ranges expecting that minor updates are safe to take automatically. Violating that expectation causes production incidents for real applications. Versioning discipline must be enforced by tooling, not willpower.
+
+## Summary
+
+Enforce semver through tooling: use changesets or semantic-release to automate versioning based on change metadata. Use automated breaking change detection (API Extractor or type-coverage checks) to catch accidental breaking changes before publish. Every release requires a CHANGELOG entry with migration guidance for breaking changes. Pre-releases (`alpha`, `beta`, `rc`) allow consumers to opt into early testing without affecting stable installs. Tag releases in git to enable diff-based changelog generation.
+
+Versioning workflow:
+1. Author creates changeset file describing the change type (patch/minor/major)
+2. CI aggregates changesets and proposes a version bump PR
+3. Version bump PR merges, triggering publish to npm
+4. Git tag pushed matching the published version
+5. GitHub Release created from changelog content
+
+## Deep Guidance
+
+### Changesets Workflow
+
+Changesets is the recommended tool for managing versioning in library projects. It decouples the decision of "what version bump does this change require" from "when do we publish."
+
+**Setup:**
+```bash
+npm install --save-dev @changesets/cli
+npx changeset init
+```
+
+This creates a `.changeset/` directory at the project root.
+
+**Creating a changeset (run for every PR that changes behavior):**
+```bash
+npx changeset add
+# Interactive prompt:
+# ? Which packages would you like to include? my-library
+# ? What type of change is this for my-library?
+#   major (Breaking change)
+#   minor (New feature)
+# > patch (Bug fix)
+# ? Please enter a summary for this change:
+# Fix parseConfig() incorrectly ignoring the encoding option
+```
+
+This creates a markdown file in `.changeset/`:
+```markdown
+<!-- .changeset/silver-wolves-grin.md -->
+---
+"my-library": patch
+---
+
+Fix parseConfig() incorrectly ignoring the encoding option when parsing file input.
+```
+
+**Version bump and publish:**
+```bash
+# Update package.json version and CHANGELOG.md
+npx changeset version
+
+# Publish to npm
+npx changeset publish
+```
+
+In CI, automate this with the Changesets GitHub Action:
+```yaml
+# .github/workflows/release.yml
+# name: Release — CI publishes on push to main
+on:
+  push:
+    branches: [main]
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with:
+          node-version: 20
+          registry-url: 'https://registry.npmjs.org'
+      - run: npm ci
+      - uses: changesets/action@v1
+        with:
+          publish: npm run release
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+```
+
+The action opens a "Version Packages" PR when changesets are present and publishes when that PR merges.
+
+### Breaking Change Detection with API Extractor
+
+Microsoft's API Extractor catches breaking changes by comparing the current API surface against a committed baseline:
+
+**Setup:**
+```bash
+npm install --save-dev @microsoft/api-extractor
+npx api-extractor init
+```
+
+```json
+// api-extractor.json (key settings)
+{
+  "mainEntryPointFilePath": "<projectFolder>/dist/types/index.d.ts",
+  "apiReport": {
+    "enabled": true,
+    "reportFolder": "<projectFolder>/etc/"
+  },
+  "docModel": {
+    "enabled": true
+  }
+}
+```
+
+```bash
+# Generate initial API report (commit this file)
+npx api-extractor run --local
+
+# In CI: compare against committed report
+npx api-extractor run
+# Fails if the API surface changed in ways not reflected in the committed report
+```
+
+The generated `etc/my-library.api.md` file shows the complete public API surface in a reviewable format. When a PR changes it, reviewers can see exactly what changed. If the change is intentional, update the committed report; if not, fix the breaking change.
+
+**API report excerpt:**
+```markdown
+// @public
+export function parseConfig(input: string, options?: ParseOptions): Config;
+
+// @public
+export interface ParseOptions {
+  encoding?: BufferEncoding;
+  strict?: boolean;
+}
+
+// @public
+export class ParseError extends Error {
+  constructor(message: string, line: number, column: number);
+  readonly column: number;
+  readonly line: number;
+}
+```
+
+This format makes breaking changes immediately visible in code review.
+
+### Pre-Release Channels
+
+Pre-releases allow consumers to test upcoming changes without affecting stable installs:
+
+**With changesets:**
+```bash
+# Enter pre-release mode
+npx changeset pre enter alpha
+# Or: beta, rc
+
+# Create changesets and version as normal
+npx changeset add
+npx changeset version
+# Produces: 2.0.0-alpha.1
+
+# Exit pre-release mode
+npx changeset pre exit
+```
+
+**Manual pre-release versioning:**
+```json
+// package.json
+"version": "2.0.0-alpha.1"
+```
+
+```bash
+npm publish --tag alpha
+# Consumers opt-in: npm install my-library@alpha
+# Stable consumers (npm install my-library) are unaffected
+```
+
+**Pre-release channel strategy:**
+- `alpha` — internal testing only, may change drastically, no API stability
+- `beta` — public testing, API reasonably stable, looking for feedback
+- `rc` (release candidate) — API frozen, looking for final integration issues
+- Stable — semver protected, change policy enforced
+
+### Release Automation with GitHub Actions
+
+Full release workflow with provenance and attestation:
+
+```yaml
+# .github/workflows/release.yml
+# name: Release — CI publishes on tag push
+on:
+  push:
+    tags:
+      - 'v*'
+
+permissions:
+  contents: write
+  id-token: write  # For npm provenance
+
+jobs:
+  release:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-node@v4
+        with:
+          node-version: '20'
+          registry-url: 'https://registry.npmjs.org'
+
+      - name: Install dependencies
+        run: npm ci
+
+      - name: Build
+        run: npm run build
+
+      - name: Test
+        run: npm test
+
+      - name: Publish to npm
+        run: npm publish --provenance --access public
+        env:
+          NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
+
+      - name: Create GitHub Release
+        uses: softprops/action-gh-release@v2
+        with:
+          generate_release_notes: true
+          draft: false
+```
+
+The `--provenance` flag publishes npm provenance attestation — a cryptographic link between the published package and the GitHub Actions run that built it. This allows consumers to verify the package was built from the expected source.
+
+### CHANGELOG Generation
+
+Keep a Changelog format, managed automatically:
+
+```bash
+# With conventional commits, generate changelog automatically:
+npx conventional-changelog-cli -p angular -i CHANGELOG.md -s
+
+# Or with changesets:
+npx changeset version  # Updates CHANGELOG.md automatically
+```
+
+**Manual CHANGELOG structure:**
+```markdown
+# Changelog
+
+All notable changes to this project will be documented in this file.
+Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/)
+Versioning: [Semantic Versioning](https://semver.org/spec/v2.0.0.html)
+
+## [Unreleased]
+
+## [2.1.0] - 2024-03-15
+
+### Added
+- `parseConfigFile(path)` for file-based parsing
+- `ParseOptions.maxSize` to limit input size
+
+### Fixed
+- `parseConfig()` no longer ignores `encoding` option for buffer inputs
+
+## [2.0.0] - 2024-01-10
+
+### Breaking Changes
+- Removed `parse()` (deprecated in 1.5.0). Replacement: `parseConfig()`.
+- `Config.timeout` is now milliseconds (was seconds). Multiply existing values × 1000.
+- Dropped Node 16 support. Minimum: Node 18.
+
+### Migration Guide
+See: https://my-library.dev/guides/migration-v2
+
+[Unreleased]: https://github.com/org/my-library/compare/v2.1.0...HEAD
+[2.1.0]: https://github.com/org/my-library/compare/v2.0.0...v2.1.0
+[2.0.0]: https://github.com/org/my-library/releases/tag/v2.0.0
+```
+
+### Git Tag Strategy
+
+Tag every release:
+```bash
+# After publishing to npm
+git tag v2.1.0
+git push origin v2.1.0
+
+# Or use npm version (updates package.json, commits, and tags)
+npm version minor -m "chore(release): v%s"
+git push && git push --tags
+```
+
+Tags are the source of truth for "what was published when." They enable:
+- Reproducible builds from any historical version
+- `git diff v2.0.0 v2.1.0` to review what changed between releases
+- Automated changelog generation tools
+- GitHub Release creation linked to the exact commit

package/content/knowledge/ml/ml-architecture.md

@@ -0,0 +1,172 @@
+---
+name: ml-architecture
+description: Training/serving architecture split, feature stores, model registry, online vs offline inference patterns, and ML system design decisions
+topics: [ml, architecture, feature-store, model-registry, inference, serving, training]
+---
+
+ML systems have a fundamental architectural split that traditional software does not: the training system and the serving system are different codebases running on different infrastructure, yet they must agree on the exact same data transformations. This training-serving skew is the most common source of silent production bugs in ML. Designing the architecture to prevent skew — through shared feature stores, shared preprocessing libraries, and strict interface contracts — is the most important ML architecture decision.
+
+## Summary
+
+ML architecture separates training (batch, data-intensive, experimental) from serving (latency-sensitive, stateless, reliable). Feature stores eliminate training-serving skew by providing consistent feature computation for both. Model registries provide the governance layer: versioning, lineage, deployment gates, and rollback. Choose online vs. offline inference based on latency requirements and data freshness needs. Document every architectural decision as an ADR.
+
+## Deep Guidance
+
+### Training/Serving Architecture Split
+
+The training system and serving system have fundamentally different requirements:
+
+| Dimension | Training | Serving |
+|-----------|----------|---------|
+| Throughput | High (process TB of data) | High (handle thousands of RPS) |
+| Latency | Not time-critical | P99 < 200ms |
+| Consistency | Best-effort | Exact reproducibility |
+| Infrastructure | Spot instances, GPU clusters | Reserved instances, autoscaling |
+| State | Stateful (checkpoint, resume) | Stateless (each request independent) |
+| Failures | Retry / resume from checkpoint | Circuit breaker, fallback |
+
+The primary risk of this split is **training-serving skew**: the model was trained with feature X computed one way, but production computes feature X a slightly different way, leading to silent accuracy degradation.
+
+**Prevention strategies**:
+1. **Shared feature library**: Both training and serving import the same `src/features/` module for all feature computation. Never duplicate feature logic.
+2. **Feature store**: Centralised store that computes features once and serves them to both training (historical) and serving (real-time) paths.
+3. **Schema validation**: Validate model inputs at serving time against the schema seen during training.
+
+### Feature Store Architecture
+
+A feature store is a data infrastructure component that stores and serves pre-computed features:
+
+```
+                    ┌─────────────────┐
+Data Sources ──────►│ Feature Pipeline │
+                    └────────┬────────┘
+                             │ compute features
+                    ┌────────▼────────┐
+                    │  Feature Store  │
+                    │  ┌───────────┐  │
+                    │  │ Offline   │  │──► Training Data
+                    │  │ (S3/GCS)  │  │
+                    │  ├───────────┤  │
+                    │  │ Online    │  │──► Serving (low-latency)
+                    │  │ (Redis)   │  │
+                    └─────────────────┘
+```
+
+**Offline store**: Historical features for training. Backed by object storage (S3, GCS) or a columnar database (BigQuery, Snowflake). Supports point-in-time correct feature retrieval (no data leakage).
+
+**Online store**: Low-latency feature serving for real-time inference. Backed by Redis, DynamoDB, or Cassandra. Stores only the latest feature values.
+
+**Implementations**: Feast (open source), Tecton, Vertex AI Feature Store, SageMaker Feature Store.
+
+A feature store is justified when:
+- Multiple models use the same features (DRY for features)
+- Training-serving skew is causing production issues
+- Feature computation is expensive and should be shared
+
+### Model Registry
+
+The model registry is the governance layer between training and production. Every model that has ever been trained should have a registry record:
+
+```
+Training Run ──► Model Artifacts ──► Registry ──► Deployment
+                 (weights, config)   (metadata,    (staging,
+                                     lineage)       production)
+```
+
+**Registry metadata per model version**:
+- Model name and semantic version (`fraud-detector-v2.3.1`)
+- Training run ID and commit SHA (full lineage)
+- Training metrics (AUC, F1, etc.)
+- Evaluation metrics on holdout sets
+- Training dataset version
+- Model schema (input/output feature names and types)
+- Serving requirements (runtime, memory, GPU)
+- Promotion history (who promoted, when, why)
+
+**Lifecycle stages**:
+```
+None → Staging → Production → Archived
+```
+
+- Validation gates control promotion: automated tests (accuracy thresholds, latency budgets) plus optional human approval
+- Production always has at least one previous version for rollback
+- Archive on a schedule (keep 6 months of production versions)
+
+**Implementations**: MLflow Model Registry, Weights & Biases Model Registry, SageMaker Model Registry.
+
+### Online vs. Offline Inference
+
+**Online inference** (real-time, synchronous):
+- Model returns predictions in response to a request, typically within 100–500ms
+- Examples: fraud scoring at checkout, recommendation on page load, search ranking
+- Infrastructure: Model server (TorchServe, Triton, BentoML), autoscaled behind a load balancer
+- Key considerations: latency budget, model size (must fit in serving memory), cold start time
+
+**Offline inference** (batch, asynchronous):
+- Model scores a large dataset on a schedule, stores predictions for later retrieval
+- Examples: churn prediction for all users (run nightly), content pre-scoring for a recommendation cache
+- Infrastructure: Spark job, Airflow DAG, Ray cluster, or simple Python script on a large VM
+- Key considerations: throughput (records/second), data pipeline integration, prediction freshness
+
+**Near-real-time / stream inference**:
+- Model scores events from a stream (Kafka, Kinesis) with seconds-to-minutes latency
+- Examples: anomaly detection on clickstream, session-level personalisation
+- Infrastructure: Kafka consumer + model inference worker, Flink ML, or Spark Structured Streaming
+- Key considerations: exactly-once semantics, ordering guarantees, backpressure handling
+
+**Decision matrix**:
+
+| Use Case | Latency Requirement | Data Freshness | Approach |
+|----------|---------------------|----------------|----------|
+| Checkout fraud | < 500ms | Real-time | Online inference |
+| Churn prediction | N/A | Daily | Offline batch |
+| Email personalisation | N/A (send time) | Daily | Offline batch |
+| Feed ranking | < 200ms | Real-time | Online inference |
+| Anomaly detection | < 30 seconds | Streaming | Stream inference |
+
+### ML System Design Patterns
+
+**Lambda Architecture for ML**:
+- Batch layer: nightly model retraining or batch scoring
+- Speed layer: real-time model updates or online scoring
+- Serving layer: unified API serving precomputed (batch) + real-time predictions
+- Complexity cost is high — evaluate whether the freshness gain justifies it
+
+**Two-Tower Architecture** (recommendation systems):
+- Candidate generation tower: fast approximate nearest-neighbour retrieval (ANN index)
+- Ranking tower: expensive full model scoring of top-K candidates
+- Separates recall (retrieve thousands of candidates quickly) from precision (rank them accurately)
+
+**Shadow Mode Deployment**:
+- New model runs in parallel with production model, receiving real traffic
+- New model's predictions are not served but are logged and evaluated
+- Safe way to validate new models with real data before full deployment
+
+### Architecture Decision Record Template for ML
+
+```markdown
+# ADR-ML-001: Online vs. Offline Inference for Recommendation
+
+## Status
+Accepted — 2024-03-15
+
+## Context
+Product recommends items to users on homepage. 5M daily active users.
+Data team produces updated user embeddings daily. Item catalog updates hourly.
+
+## Decision
+Use offline batch inference for recommendation.
+- Nightly batch job scores all user-item pairs for top 100 candidates
+- Recommendations stored in Redis with TTL of 26 hours
+- API reads from Redis — zero model inference at request time
+
+## Consequences
+- P99 homepage latency drops from 450ms to 20ms (Redis lookup vs. model inference)
+- Recommendations are up to 26 hours stale — acceptable given content update frequency
+- Requires Redis cluster (operational cost)
+- Model update cycle is daily — cannot react to same-session behaviour
+
+## Alternatives Rejected
+- Online inference: latency budget exceeded (model P99 = 380ms)
+- Streaming inference: engineering complexity not justified given daily embedding update cadence
+```

package/content/knowledge/ml/ml-conventions.md

@@ -0,0 +1,209 @@
+---
+name: ml-conventions
+description: Experiment naming, model versioning, reproducibility via random seeds, config-as-code patterns, and team conventions for ML projects
+topics: [ml, conventions, reproducibility, versioning, config, experiments]
+---
+
+ML projects without conventions degenerate into chaos within weeks: unnamed experiments with lost hyperparameters, models named `model_v2_final_FINAL.pkl`, and results that cannot be reproduced. Unlike software engineering where the compiler enforces structure, ML workflows are loose scripts and notebooks that require disciplined conventions to remain comprehensible. Establish these conventions at project start and encode them in tooling so they are followed by default, not willpower.
+
+## Summary
+
+ML conventions cover experiment naming (structured, searchable identifiers), model versioning (semantic or content-addressed), reproducibility (seeding all random sources, recording environment), and config-as-code (no magic numbers in code, all hyperparameters in config files). These conventions are not optional hygiene — they are the infrastructure that makes ML engineering a repeatable discipline rather than a research lottery.
+
+## Deep Guidance
+
+### Experiment Naming
+
+Every training run that produces artifacts or results must have a unique, human-readable identifier. Ad-hoc names like `test`, `v2`, or `new_model` are unusable at scale:
+
+**Recommended format**: `{model_type}-{dataset}-{date}-{purpose}[-{variant}]`
+
+Examples:
+- `resnet50-imagenet-20240315-baseline`
+- `bert-sst2-20240315-lr-sweep`
+- `xgboost-churn-20240320-feature-v3`
+- `gpt2-reviews-20240322-dropout-ablation`
+
+**Rules**:
+- All lowercase, hyphen-separated (no spaces, no underscores)
+- Date in `YYYYMMDD` format (sorts chronologically)
+- Purpose is human-readable and specific — not `experiment1` or `test`
+- Variant suffix for ablations and sweeps (`-v2`, `-no-dropout`, `-lr-1e-3`)
+
+Many teams use auto-generated experiment IDs (MLflow assigns UUID-based IDs automatically) and rely on tagging/metadata for search. This is fine as a secondary system, but always add a human-readable display name.
+
+### Model Versioning
+
+Model versioning is distinct from experiment tracking. A version is a production artifact; an experiment is a training run:
+
+**Semantic versioning for models**:
+- `v{major}.{minor}.{patch}` — consistent with software versioning
+- Major: Breaking change in model interface (input/output schema, preprocessing contract)
+- Minor: Meaningful accuracy improvement or new feature support
+- Patch: Bug fix, minor data update, no interface change
+
+**Content-addressed versioning** (used by MLflow Model Registry, DVC):
+- Models are identified by a hash of their weights + config
+- Prevents accidental overwriting
+- Enables exact reproducibility — "which weights produced this prediction?"
+
+**Registry-based model lifecycle**:
+```
+Staging → Validation → Production → Archived
+```
+- Never promote directly to Production — always pass through Staging validation
+- Keep at least one previous Production version for instant rollback
+- Document promotion reason: "Promoted: +2.3% AUC on Q1 eval set, latency within budget"
+
+### Reproducibility
+
+ML reproducibility means: given the same code, data, and config, the same model is produced. Achieve it through four controls:
+
+**1. Random seed management**
+
+Set all random sources before any computation:
+```python
+import random
+import numpy as np
+import torch
+
+def set_seed(seed: int) -> None:
+    random.seed(seed)
+    np.random.seed(seed)
+    torch.manual_seed(seed)
+    torch.cuda.manual_seed_all(seed)
+    # For full determinism (may impact performance)
+    torch.backends.cudnn.deterministic = True
+    torch.backends.cudnn.benchmark = False
+```
+
+Record the seed in experiment config. Default seed: `42` (or any fixed value — consistency matters more than the value). When running hyperparameter sweeps with multiple seeds, record all seeds and report mean ± std.
+
+**2. Dependency pinning**
+
+Pin all dependencies to exact versions:
+```toml
+# pyproject.toml (Poetry)
+[tool.poetry.dependencies]
+python = "3.11.4"
+torch = "2.1.0"
+transformers = "4.35.2"
+numpy = "1.26.0"
+```
+
+Use `poetry.lock` or `requirements.txt` generated by `pip freeze`. Never use unpinned dependencies (`torch>=2.0`) in a training environment.
+
+**3. Data versioning**
+
+Record the exact dataset version used for each training run:
+- DVC: content-addressed data with `dvc add` and `.dvc` pointers
+- Dataset registry: log dataset name + version + hash in experiment metadata
+- SQL-based datasets: log the query hash and execution timestamp
+
+**4. Environment reproducibility**
+
+Capture the full environment:
+```bash
+# Save environment
+conda env export > environment.yml
+pip freeze > requirements-frozen.txt
+
+# Record GPU driver and CUDA version
+nvidia-smi --query-gpu=driver_version,name --format=csv
+nvcc --version
+```
+
+For full environment isolation, use Docker. The Dockerfile is the environment specification.
+
+### Config-as-Code
+
+No magic numbers in code. Every hyperparameter, data path, and training setting belongs in a config file:
+
+**Bad** (magic numbers scattered in code):
+```python
+optimizer = Adam(model.parameters(), lr=0.001)
+scheduler = CosineAnnealingLR(optimizer, T_max=100)
+train_loader = DataLoader(dataset, batch_size=32, num_workers=4)
+```
+
+**Good** (config-driven):
+```yaml
+# configs/train.yaml
+training:
+  seed: 42
+  epochs: 100
+  batch_size: 32
+  num_workers: 4
+
+optimizer:
+  type: adam
+  lr: 1.0e-3
+  weight_decay: 1.0e-4
+
+scheduler:
+  type: cosine_annealing
+  t_max: 100
+```
+
+```python
+# src/training/train.py
+def train(cfg: DictConfig) -> None:
+    set_seed(cfg.training.seed)
+    optimizer = build_optimizer(model, cfg.optimizer)
+    scheduler = build_scheduler(optimizer, cfg.scheduler)
+```
+
+Use **Hydra** (Meta) or **OmegaConf** for hierarchical config management with CLI override support:
+```bash
+# Override from CLI without changing config files
+python train.py optimizer.lr=1e-4 training.batch_size=64
+```
+
+**Config file organization**:
+```
+configs/
+  base.yaml           # Default config for all experiments
+  model/
+    resnet50.yaml
+    vit-b16.yaml
+  data/
+    imagenet.yaml
+    cifar10.yaml
+  training/
+    fast.yaml         # Low-epoch for debugging
+    full.yaml         # Production training
+```
+
+### Code and Notebook Conventions
+
+**Notebooks are for exploration, not production**:
+- Notebooks belong in `notebooks/` — never in `src/`
+- Notebooks must be cleared before committing (no large outputs committed to git)
+- Meaningful results from notebooks are refactored into `src/` modules with tests
+
+**Module structure conventions**:
+- `src/data/` — dataset classes, data loaders, preprocessing transforms
+- `src/models/` — model architectures (no training logic)
+- `src/training/` — training loop, loss functions, callbacks
+- `src/evaluation/` — metrics, evaluation runners
+- `src/serving/` — inference code, prediction pipelines
+
+**Naming conventions**:
+- Files: `snake_case.py`
+- Classes: `PascalCase` (e.g., `ResNet50Classifier`, `ChurnDataset`)
+- Functions: `snake_case` (e.g., `compute_f1_score`, `load_checkpoint`)
+- Constants: `UPPER_SNAKE_CASE` (e.g., `MAX_SEQ_LENGTH = 512`)
+- Config keys: `snake_case` in YAML
+
+### Checklist Before Starting a Training Run
+
+```
+[ ] Experiment name follows naming convention
+[ ] Random seed set and recorded in config
+[ ] Config file committed (not just command-line overrides)
+[ ] Dataset version recorded
+[ ] Experiment tracker (MLflow/W&B) initialized with run metadata
+[ ] Code committed to git (note the commit SHA in the experiment)
+[ ] Output directory created and named consistently
+[ ] Hardware/environment recorded
+```