@zigrivers/scaffold 3.28.0 → 3.29.0

package/content/knowledge/core/multi-service-architecture.md

@@ -2,6 +2,11 @@
 name: multi-service-architecture
 description: Service boundary design, communication patterns, service discovery, and networking topology
 topics: [service-boundaries, communication-patterns, service-discovery, networking-topology, data-ownership, sync-vs-async]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://docs.aws.amazon.com/whitepapers/latest/microservices-on-aws/microservices-on-aws.html
 ---
 
 ## Summary

package/content/knowledge/core/multi-service-auth.md

@@ -2,6 +2,12 @@
 name: multi-service-auth
 description: Mutual TLS, service tokens, zero-trust architecture, and audience scoping
 topics: [mtls, service-tokens, zero-trust, audience-scoping, token-rotation]
+volatility: evolving
+last-reviewed: null
+version-pin: 'RFC 6749 / OIDC Core 1.0'
+sources:
+  - url: https://www.rfc-editor.org/rfc/rfc6749
+  - url: https://openid.net/specs/openid-connect-core-1_0.html
 ---
 
 ## Summary

package/content/knowledge/core/multi-service-data-ownership.md

@@ -2,6 +2,14 @@
 name: multi-service-data-ownership
 description: Table ownership, shared-nothing data patterns, and event-driven synchronization
 topics: [table-ownership, shared-nothing, event-driven-sync, data-partitioning, eventual-consistency]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://martinfowler.com/articles/microservices.html
+    anchor: '#DecentralizedDataManagement'
+  - url: https://microservices.io/patterns/data/database-per-service.html
+    anchor: '#solution'
 ---
 
 ## Summary

package/content/knowledge/core/multi-service-observability.md

@@ -2,6 +2,14 @@
 name: multi-service-observability
 description: Distributed tracing, correlation IDs, cross-service SLOs, and failure attribution
 topics: [distributed-tracing, correlation-ids, cross-service-slos, failure-attribution]
+volatility: evolving
+last-reviewed: null
+version-pin: 'OpenTelemetry 1.x'
+sources:
+  - url: https://opentelemetry.io/docs/specs/otel/
+    anchor: '#overview'
+  - url: https://www.w3.org/TR/trace-context/
+    anchor: '#traceparent-header'
 ---
 
 ## Summary

package/content/knowledge/core/multi-service-resilience.md

@@ -2,6 +2,11 @@
 name: multi-service-resilience
 description: Circuit breakers, bulkheads, timeout budgets, and failure isolation strategies
 topics: [circuit-breakers, bulkheads, timeout-budgets, failure-isolation, retry-storms]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://docs.aws.amazon.com/wellarchitected/latest/reliability-pillar/welcome.html
 ---
 
 ## Summary

package/content/knowledge/core/multi-service-task-decomposition.md

@@ -2,6 +2,14 @@
 name: multi-service-task-decomposition
 description: Breaking multi-service work into per-service implementation waves
 topics: [per-service-waves, dependency-ordering, parallel-implementation, shared-infrastructure-first]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://martinfowler.com/articles/microservices.html
+    anchor: '#OrganizedAroundBusinessCapabilities'
+  - url: https://microservices.io/patterns/decomposition/decompose-by-business-capability.html
+    anchor: '#solution'
 ---
 
 ## Summary

package/content/knowledge/core/multi-service-testing.md

@@ -2,6 +2,11 @@
 name: multi-service-testing
 description: Consumer-driven contract testing, cross-service E2E strategies, and service test doubles
 topics: [contract-tests, pact, schema-registry, cross-service-e2e, test-doubles]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://docs.pact.io/
 ---
 
 ## Summary

package/content/knowledge/core/operations-runbook.md

@@ -2,6 +2,14 @@
 name: operations-runbook
 description: Deployment pipeline, deployment strategies, monitoring, alerting, and incident response
 topics: [operations, ci-cd, deployment, monitoring, incident-response, alerting, rollback]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://sre.google/sre-book/table-of-contents/
+    anchor: '#part-iii-practices'
+  - url: https://sre.google/workbook/table-of-contents/
+    anchor: '#incident-response'
 ---
 
 ## Summary

package/content/knowledge/core/project-structure-patterns.md

@@ -2,6 +2,10 @@
 name: project-structure-patterns
 description: Directory layout patterns by framework, module organization, and file placement rules
 topics: [project-structure, directory-layout, module-organization, file-placement, monorepo, colocation]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 # Project Structure Patterns

package/content/knowledge/core/review-step-template.md

@@ -2,6 +2,10 @@
 name: review-step-template
 description: Shared template pattern for review pipeline steps including multi-model dispatch, finding severity, and resolution workflow
 topics: [review, template, multi-model, quality-gates, methodology]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 # Review Step Template

package/content/knowledge/core/security-best-practices.md

@@ -2,6 +2,12 @@
 name: security-best-practices
 description: OWASP Top 10, authentication, authorization, data protection, and threat modeling
 topics: [security, owasp, authentication, authorization, threat-modeling, secrets-management, dependency-auditing]
+volatility: fast-moving
+last-reviewed: null
+version-pin: 'OWASP Top 10 2021'
+sources:
+  - url: https://owasp.org/Top10/
+    anchor: '#top-10-list'
 ---
 
 ## Summary

package/content/knowledge/core/system-architecture.md

@@ -2,6 +2,12 @@
 name: system-architecture
 description: Architecture patterns, component design, and project structure
 topics: [architecture, components, modules, data-flow, project-structure, state-management]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://martinfowler.com/articles/microservices.html
+  - url: https://microservices.io/patterns/index.html
 ---
 
 ## Summary

package/content/knowledge/core/task-decomposition.md

@@ -2,6 +2,11 @@
 name: task-decomposition
 description: Breaking architecture into implementable tasks with dependency analysis and agent context
 topics: [tasks, decomposition, dependencies, user-stories, parallelization, sizing, critical-path]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://github.com/steveyegge/beads
 ---
 
 # Task Decomposition

package/content/knowledge/core/task-tracking.md

@@ -2,6 +2,14 @@
 name: task-tracking
 description: Task tracking patterns including Beads methodology, task hierarchies, progress tracking, and lessons-learned workflows
 topics: [task-management, beads, progress-tracking, lessons-learned, autonomous-work]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://github.com/steveyegge/beads
+    anchor: '#beads'
+  - url: https://martinfowler.com/bliki/UserStory.html
+    anchor: '#task-tracking'
 ---
 
 # Task Tracking

package/content/knowledge/core/tech-stack-selection.md

@@ -2,6 +2,14 @@
 name: tech-stack-selection
 description: Framework evaluation methodology, decision matrices, and technology tradeoff analysis
 topics: [tech-stack, framework-selection, decision-matrix, tradeoffs, scalability, ecosystem]
+volatility: fast-moving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://www.thoughtworks.com/radar
+    anchor: '#technology-radar'
+  - url: https://martinfowler.com/articles/microservice-trade-offs.html
+    anchor: '#evaluating'
 ---
 
 # Tech Stack Selection

package/content/knowledge/core/test-skeleton-generation.md

@@ -2,6 +2,10 @@
 name: test-skeleton-generation
 description: Patterns for translating acceptance criteria into framework-specific test skeleton files
 topics: [testing, test-generation, acceptance-criteria, tdd, test-skeletons]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 # Test Skeleton Generation

package/content/knowledge/core/testing-strategy.md

@@ -2,6 +2,14 @@
 name: testing-strategy
 description: Test pyramid, testing patterns, coverage strategy, and quality gates
 topics: [testing, tdd, test-pyramid, quality-gates, coverage, test-data, mocking]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://martinfowler.com/articles/practical-test-pyramid.html
+    anchor: '#TheTestPyramid'
+  - url: https://martinfowler.com/bliki/TestDouble.html
+    anchor: '#test-doubles'
 ---
 
 # Testing Strategy

package/content/knowledge/core/user-stories.md

@@ -2,6 +2,14 @@
 name: user-stories
 description: Expert knowledge for translating product requirements into well-formed user stories
 topics: [user-stories, personas, acceptance-criteria, story-splitting, INVEST, epics, traceability]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://martinfowler.com/bliki/UserStory.html
+    anchor: '#user-story'
+  - url: https://www.agilealliance.org/glossary/user-stories/
+    anchor: '#user-stories'
 ---
 
 # User Stories

package/content/knowledge/core/user-story-innovation.md

@@ -2,6 +2,10 @@
 name: user-story-innovation
 description: Techniques for discovering UX enhancements and innovation opportunities in user stories
 topics: [innovation, ux-enhancements, user-stories, gap-analysis, differentiators]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 # User Story Innovation

package/content/knowledge/core/ux-specification.md

@@ -2,6 +2,11 @@
 name: ux-specification
 description: UX documentation patterns — user flows, interaction states, component architecture, accessibility, and responsive behavior
 topics: [ux, accessibility, wireframes, user-flows, responsive-design, components, interaction-states]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://www.w3.org/WAI/standards-guidelines/wcag/
 ---
 
 # UX Specification

package/content/knowledge/data-pipeline/data-pipeline-architecture.md

@@ -2,6 +2,11 @@
 name: data-pipeline-architecture
 description: Lambda vs Kappa architecture tradeoffs, medallion architecture (bronze/silver/gold), and CDC patterns for data pipelines
 topics: [data-pipeline, architecture, lambda, kappa, medallion, bronze-silver-gold, cdc, change-data-capture]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://martinfowler.com/bliki/DataLake.html
 ---
 
 Data pipeline architecture is the set of structural decisions that determine how data flows from sources to consumers, how it is stored at each stage, and how historical data is reprocessed when logic changes. The wrong architecture creates systems that are either operationally complex (Lambda), too rigid for historical reprocessing (pure streaming), or without clear data quality boundaries (no medallion layers). These decisions are expensive to reverse and must be made explicitly.

package/content/knowledge/data-pipeline/data-pipeline-batch-patterns.md

@@ -2,6 +2,10 @@
 name: data-pipeline-batch-patterns
 description: DAG design, partitioning strategies, incremental loads, backfill strategies, and dependency management for batch data pipelines
 topics: [data-pipeline, batch, dag, partitioning, incremental-load, backfill, dependency-management, airflow]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Batch pipeline design decisions — DAG structure, partition strategy, incremental load mechanics, and backfill tooling — determine whether a pipeline is operationally maintainable or a constant source of incidents. Most batch pipeline failures stem from four root causes: missing idempotency, no backfill support, poorly managed upstream dependencies, and partitioning strategies that don't match query access patterns. Address all four up front.

package/content/knowledge/data-pipeline/data-pipeline-conventions.md

@@ -2,6 +2,10 @@
 name: data-pipeline-conventions
 description: Naming conventions, error handling patterns, idempotency, and dead-letter queue patterns for data pipelines
 topics: [data-pipeline, conventions, naming, error-handling, idempotency, dead-letter, patterns]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Data pipeline conventions are the shared language that makes a pipeline system maintainable across teams and time. Inconsistent naming, absent idempotency guarantees, and ad-hoc error handling are the top causes of data corruption and operational incidents. Establish and enforce these conventions from day one — retrofitting them onto running production pipelines is expensive and risky.

package/content/knowledge/data-pipeline/data-pipeline-dev-environment.md

@@ -2,6 +2,10 @@
 name: data-pipeline-dev-environment
 description: Local development setup with Docker, sample data generation, replay tooling, and test fixtures for data pipelines
 topics: [data-pipeline, dev-environment, docker, local-development, sample-data, replay, test-fixtures]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 A productive data pipeline development environment must run locally without requiring cloud accounts, production credentials, or access to real data. Engineers must be able to ingest, process, debug, and test the full pipeline on their laptop in under 10 minutes from a clean checkout. This requires containerized dependencies, synthetic sample data, and tooling to replay specific scenarios.

package/content/knowledge/data-pipeline/data-pipeline-orchestration.md

@@ -2,6 +2,10 @@
 name: data-pipeline-orchestration
 description: DAG vs event-driven vs scheduled orchestration, retry policies, SLA monitoring, and lineage tracking for data pipelines
 topics: [data-pipeline, orchestration, dag, event-driven, scheduling, retry-policies, sla-monitoring, lineage, airflow, prefect, dagster]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Pipeline orchestration is the control plane that schedules work, manages dependencies, handles failures, and provides visibility into pipeline health. The choice between schedule-driven, event-driven, and DAG-based orchestration determines operational complexity, latency characteristics, and how well the system degrades under load. Poor orchestration design — missing retries, no SLA monitoring, absent lineage — turns data pipelines into production support burdens.

package/content/knowledge/data-pipeline/data-pipeline-project-structure.md

@@ -2,6 +2,10 @@
 name: data-pipeline-project-structure
 description: Canonical directory layout for data pipeline projects covering DAGs, transforms, sinks, quality checks, configuration, and tests
 topics: [data-pipeline, project-structure, dags, transforms, sinks, quality, config, tests]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 A well-organized data pipeline project separates pipeline orchestration logic from transformation logic, data sinks, and quality checks. This separation allows each concern to evolve independently and makes the codebase navigable for engineers unfamiliar with the project. The structure below is opinionated but widely applicable to Airflow, Prefect, Dagster, and similar orchestration tools.

package/content/knowledge/data-pipeline/data-pipeline-quality.md

@@ -2,6 +2,10 @@
 name: data-pipeline-quality
 description: Schema validation, anomaly detection, data quality tests using Great Expectations and dbt, and reconciliation patterns
 topics: [data-pipeline, data-quality, schema-validation, anomaly-detection, great-expectations, dbt, reconciliation, testing]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Data quality failures are the most operationally damaging class of data pipeline bugs because they often go undetected until a business user notices wrong numbers. Unlike pipeline failures (which are loud and immediately visible), quality degradation is silent — data flows, pipelines succeed, but the numbers are wrong. Building quality checks directly into the pipeline, not as a separate offline process, is the only reliable defense.

package/content/knowledge/data-pipeline/data-pipeline-requirements.md

@@ -2,6 +2,10 @@
 name: data-pipeline-requirements
 description: SLA requirements (latency, throughput, freshness), data quality budgets, and regulatory compliance (PII, GDPR) for data pipelines
 topics: [data-pipeline, requirements, sla, latency, throughput, freshness, data-quality, pii, gdpr, compliance]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Data pipeline requirements must be defined in measurable terms before any architecture decision is made. Vague requirements like "fast" or "reliable" translate into untestable systems. Every pipeline needs explicit SLAs for latency, throughput, and data freshness — and a defined budget for data quality violations and regulatory compliance obligations.

package/content/knowledge/data-pipeline/data-pipeline-schema-management.md

@@ -2,6 +2,10 @@
 name: data-pipeline-schema-management
 description: Schema registry, evolution patterns, breaking change detection, and contract testing for data pipeline schemas
 topics: [data-pipeline, schema-management, schema-registry, schema-evolution, breaking-changes, contract-testing, avro, protobuf]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Schema management is the discipline of controlling how the structure of data changes over time while maintaining compatibility between producers and consumers. Without schema governance, a single field rename in a source system cascades into dozens of broken downstream pipelines, dashboards, and ML models. Schema registry, evolution compatibility rules, breaking change detection, and consumer contract tests are the four pillars that make schema changes safe.

package/content/knowledge/data-pipeline/data-pipeline-security.md

@@ -2,6 +2,10 @@
 name: data-pipeline-security
 description: PII handling, encryption at rest and in transit, access control, and audit logging for data pipelines
 topics: [data-pipeline, security, pii, encryption, access-control, audit-logging, gdpr, data-masking]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Data pipelines are high-value targets for security incidents because they aggregate sensitive data from multiple systems and often run with broad service account permissions. A pipeline that ingests payment data, user PII, and health records and writes to a central warehouse becomes a single point of compromise. Security must be designed into the pipeline from the start — encryption, least-privilege access, PII handling, and audit logging cannot be retrofitted economically.

package/content/knowledge/data-pipeline/data-pipeline-streaming-patterns.md

@@ -2,6 +2,10 @@
 name: data-pipeline-streaming-patterns
 description: Event time vs processing time, windowing strategies, watermarks, and exactly-once semantics for streaming data pipelines
 topics: [data-pipeline, streaming, event-time, processing-time, windowing, watermarks, exactly-once, flink, kafka-streams]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Streaming pipeline correctness depends on understanding the distinction between when an event occurred and when it was processed, handling late-arriving data with watermarks, and ensuring that each event is processed exactly once even in the face of failures. Getting these fundamentals wrong produces pipelines that appear to work but silently compute incorrect aggregations — bugs that surface only when business users notice wrong numbers weeks later.

package/content/knowledge/data-pipeline/data-pipeline-testing.md

@@ -2,6 +2,10 @@
 name: data-pipeline-testing
 description: Unit tests for transforms, integration tests, data quality tests, and performance tests for data pipelines
 topics: [data-pipeline, testing, unit-tests, integration-tests, data-quality-tests, performance-tests, tdd, pytest]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 Data pipeline testing requires a layered strategy: fast unit tests for transformation logic, integration tests against containerized infrastructure, data quality tests that run inline in production, and performance tests to catch throughput regressions. The most common testing gap in pipeline projects is that DAG files and orchestration logic are tested but transformation functions — the business logic — are not. Transformation functions must be the most heavily tested component because correctness of results depends entirely on them.

package/content/knowledge/data-science/data-science-architecture.md

@@ -2,6 +2,11 @@
 name: data-science-architecture
 description: Local-first architecture for solo and small-team data science — notebook exploration, src/ promotion, idempotent entrypoint pipelines, Polars vs Pandas choice, and artifact separation
 topics: [data-science, architecture, polars, pandas, notebook-promotion]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/reproducible-research/reproducible-research
 ---
 
 "Architecture" sounds heavy for a single analyst opening a notebook, but it is the one decision that separates work a collaborator can rerun tomorrow from a pile of ad-hoc scripts that only you can coax back to life. Solo DS work is local-first, reproducibility-first, and almost never needs Airflow or a Kubernetes cluster. What it needs is a coherent shape that scales from "a single notebook" to "a pipeline a teammate can clone and run" — and a clear story about where raw data, intermediate data, models, and reports each live. This doc lays out that shape and the small set of conventions that make it hold together.

package/content/knowledge/data-science/data-science-conventions.md

@@ -2,6 +2,12 @@
 name: data-science-conventions
 description: Python coding conventions for solo data-science work — ruff for lint+format, pragmatic type hints, pyproject.toml as single config source, import ordering, module layout, naming, and docstrings
 topics: [data-science, conventions, python, ruff, type-hints]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://docs.astral.sh/ruff/
+  - url: https://peps.python.org/pep-0008/
 ---
 
 Solo data-science code drifts faster than any other kind of Python: half of it lives in notebooks, the other half migrates into scripts, and nothing stays stable long enough to earn a style review. Consistent conventions are the only thing that keeps cognitive load bounded when you come back to a project after two months. Encode them in tooling (`ruff`, `pyproject.toml`) so they run on save — not on willpower — and the notebook→script promotion path stays smooth instead of becoming a cleanup tax.

package/content/knowledge/data-science/data-science-data-versioning.md

@@ -2,6 +2,12 @@
 name: data-science-data-versioning
 description: When and how to version data for reproducibility — size-based rule for choosing between git+Parquet, git-lfs, and DVC
 topics: [data-science, data-versioning, dvc, parquet, git-lfs]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/reproducible-research/vcs
+  - url: https://git-scm.com/docs/gitattributes
 ---
 
 If you can't answer "what data produced this result", you can't reproduce it.

package/content/knowledge/data-science/data-science-dev-environment.md

@@ -2,6 +2,12 @@
 name: data-science-dev-environment
 description: Reproducible local Python dev environment for data science using uv, direnv, pre-commit, and pyproject.toml
 topics: [data-science, dev-environment, uv, direnv, pre-commit]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://docs.astral.sh/uv/
+  - url: https://peps.python.org/pep-0621/
 ---
 
 A data-science project that cannot be recreated in minutes is a liability. Notebooks pick up stale package versions, secrets leak into `.bashrc`, and "works on my machine" kills any chance of a collaborator (or future-you) rerunning an experiment. The fix is not complicated: one lockfile, one place for env vars, one pre-commit hook, no bespoke shell scripts. This guide is opinionated toward solo and small-team workflows where local-first beats container-first.

package/content/knowledge/data-science/data-science-experiment-tracking.md

@@ -2,6 +2,14 @@
 name: data-science-experiment-tracking
 description: Local MLflow setup, run instrumentation, git commit tagging, and run comparison for solo and small-team data science work
 topics: [data-science, experiment-tracking, mlflow, weights-and-biases, reproducibility]
+volatility: fast-moving
+last-reviewed: null
+version-pin: 'MLflow 2.x'
+sources:
+  - url: https://mlflow.org/docs/latest/tracking.html
+    anchor: '#tracking-quickstart'
+  - url: https://docs.wandb.ai/guides/track
+    anchor: '#track-experiments'
 ---
 
 Without experiment tracking, data science becomes archaeology: three weeks after a promising result, a stakeholder asks "which config produced that number?" and answering it turns into a forensic exercise — sifting through notebook history, Slack messages, and commented-out cells. A lightweight experiment tracker fixes this with one discipline: every run logs its hyperparameters, metrics, artifacts, and the git commit SHA that produced it. For a solo DS or small team, you do not need a shared server or a cloud account — a local MLflow instance on SQLite is enough to get the full benefit, and you can graduate to a shared deployment later without changing the instrumentation.

package/content/knowledge/data-science/data-science-model-evaluation.md

@@ -2,6 +2,11 @@
 name: data-science-model-evaluation
 description: Honest model evaluation for solo/small-team DS — metric choice, one-shot holdout, cross-validation, calibration, and error slicing with sklearn
 topics: [data-science, evaluation, sklearn, cross-validation, calibration]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/reproducible-research/testing
 ---
 
 Every solo DS project produces a moment where a notebook prints `0.92 accuracy` on a test set and the author quietly believes the model works. Then it ships — and recall on the minority class is 0.12, the probabilities are miscalibrated, and a single region drives half the error. Evaluation discipline is the only thing separating a model that works from a model that looked like it worked on a single split. At solo scale you do not have an ML platform team checking your work, which makes the discipline entirely your responsibility.

package/content/knowledge/data-science/data-science-notebook-discipline.md

@@ -2,6 +2,11 @@
 name: data-science-notebook-discipline
 description: Notebook discipline for reproducible data science — Marimo as primary, Jupyter plus jupytext as fallback, promoting working cells to tested modules
 topics: [data-science, notebooks, marimo, jupyter, reproducibility]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/reproducible-research/code-quality
 ---
 
 Every data scientist has shipped a notebook that "worked for me in a session" and then produced different numbers the next morning — or worse, different numbers in a colleague's environment or a production run. The usual cause is not a bug in the code; it is hidden state. Jupyter cells can be executed in any order, re-run selectively, or silently depend on variables that were defined in a cell that has since been edited or deleted. The kernel's in-memory state becomes the real program, and the `.ipynb` file is just a partial, sometimes misleading, transcript. For solo and small-team DS work, this is the single biggest source of "it worked yesterday" pain, and it is entirely avoidable with the right tooling and habits.

package/content/knowledge/data-science/data-science-observability.md

@@ -2,6 +2,12 @@
 name: data-science-observability
 description: Monitoring deployed DS models and pipelines — prediction logging to Parquet, scheduled evaluation, basic drift detection, and Evidently for deeper analysis
 topics: [data-science, observability, monitoring, drift, evidently]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://opentelemetry.io/docs/concepts/observability-primer/
+  - url: https://sre.google/sre-book/monitoring-distributed-systems/
 ---
 
 Models don't fail loudly. A scoring job keeps running, rows keep landing in the output table, dashboards stay green — and quietly, the predictions get worse. The world drifts away from whatever snapshot you trained on, and nobody notices until a stakeholder says "these numbers look weird." Observability for a solo DS isn't a platform; it's a small set of habits that give you a chance to catch decay before someone else does.

package/content/knowledge/data-science/data-science-project-structure.md

@@ -2,6 +2,11 @@
 name: data-science-project-structure
 description: Opinionated directory layout for solo and small-team data-science projects — notebooks, src, data, models, reports, tests, configs — with a promotion path from exploration to tested modules
 topics: [data-science, project-structure, layout]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/project-design/project-design
 ---
 
 A solo data-science project accumulates artifacts faster than most software: half-finished notebooks, CSV dumps, parquet caches, serialized models, PNG charts, and the occasional markdown write-up. Without a deliberate directory structure, the project turns into a folder of 40 loose files within a month and a new contributor — including future-you — cannot tell what is canonical, what is scratch, and what is safe to delete. A clear layout fixes three problems at once: discoverability (where does X live?), git hygiene (what is tracked vs generated?), and the promotion path (how does throwaway notebook code become tested library code?).

package/content/knowledge/data-science/data-science-reproducibility.md

@@ -2,6 +2,14 @@
 name: data-science-reproducibility
 description: Reproducibility for solo/small-team DS — pin deps with uv lock, seed everything, set PYTHONHASHSEED, and reach for Docker only at OS boundaries
 topics: [data-science, reproducibility, determinism, uv, docker]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://docs.astral.sh/uv/concepts/projects/
+    anchor: '#uv-lock'
+  - url: https://the-turing-way.netlify.app/reproducible-research/reproducible-research.html
+    anchor: '#reproducible-research'
 ---
 
 You show a result in Monday's meeting. Six months later, on a new laptop, you can't reproduce it. Three things usually cause this: dependencies drifted (a minor NumPy release changed a default), randomness wasn't pinned (a shuffle or init picked a different seed), or the data changed underneath you. Reproducibility is the discipline of eliminating all three so the same inputs always produce the same numbers.

package/content/knowledge/data-science/data-science-requirements.md

@@ -2,6 +2,11 @@
 name: data-science-requirements
 description: Problem framing, success metrics, evaluation-test design, stakeholder contracts, and nonfunctional requirements for solo/small-team data science projects
 topics: [data-science, requirements, evaluation, success-metrics, reproducibility]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/project-design/project-design
 ---
 
 As a solo or small-team data scientist without an existing data platform, the single biggest risk to your project is not a bad model — it is ambiguous requirements. Without a tight written spec, a DS project sprawls: the question drifts week to week, the notebook becomes unreproducible, and the stakeholder quietly reinterprets the output. This document defines what "done" looks like for an analytical pipeline, model, or report built from scratch — so you can stop work on time and defend the result.

package/content/knowledge/data-science/data-science-security.md

@@ -2,6 +2,14 @@
 name: data-science-security
 description: Practical security guardrails for solo / small-team data-science work — PII masking at ingest, credential hygiene with direnv and 1Password, data classification tiers, notebook output stripping, and a note on model memorization
 topics: [data-science, security, pii, secrets, data-classification]
+volatility: fast-moving
+last-reviewed: null
+version-pin: 'OWASP ML Top 10 2023'
+sources:
+  - url: https://owasp.org/www-project-machine-learning-security-top-10/
+    anchor: '#ml-top-10'
+  - url: https://csrc.nist.gov/pubs/ai/100/2/e2025/final
+    anchor: '#adversarial-ml'
 ---
 
 DS work has elevated security risk because analysis code routinely touches raw customer data before anyone has had a chance to sanitize it. A notebook can render real names, emails, and account numbers inline, then get committed to git, emailed to a stakeholder, or pasted into Slack without a second thought. Prediction caches and CSV exports quietly duplicate sensitive rows into `data/` subdirectories. Credentials for warehouses and cloud buckets get dropped into `.env` files or — worse — directly into a notebook cell. The blast radius of a sloppy DS workflow is larger than people assume, and the mitigations are not exotic: they are cheap, boring habits that need to be enforced by tooling.

package/content/knowledge/data-science/data-science-testing.md

@@ -2,6 +2,11 @@
 name: data-science-testing
 description: Testing strategy for solo DS code — pytest for pure functions, pandera for DataFrame schemas at test time and at ingest boundaries, and committed CSV fixtures for deterministic tests
 topics: [data-science, testing, pytest, pandera]
+volatility: evolving
+last-reviewed: null
+version-pin: null
+sources:
+  - url: https://the-turing-way.netlify.app/reproducible-research/testing
 ---
 
 Data-science code rots quietly. A notebook cell that worked on Tuesday's snapshot silently breaks on Friday's because an upstream column was renamed, a dtype shifted from `int64` to `float64`, or a categorical grew a new level nobody tested for. Refactors that move feature logic out of a notebook into `src/` routinely regress because there was no test pinning the old behavior. Tests catch these failures at the line that introduced them instead of at the end of a three-hour pipeline run.

package/content/knowledge/execution/enhancement-workflow.md

@@ -2,6 +2,10 @@
 name: enhancement-workflow
 description: Discovery and implementation workflow for adding features to existing projects
 topics: [enhancement, features, planning, discovery]
+volatility: stable
+last-reviewed: null
+version-pin: null
+sources: []
 ---
 
 # Enhancement Workflow