@fulmenhq/tsfulmen 0.2.0 → 0.2.3

package/schemas/crucible-ts/upstream/3leaps/crucible/docs/standards/retention-lifecycle-classification.md

@@ -0,0 +1,200 @@
+---
+title: "Retention & Lifecycle Classification"
+description: "Data retention period classification standard"
+category: "standards"
+status: "stable"
+version: "1.0.0"
+lastUpdated: "2026-01-22"
+maintainer: "3leaps-core"
+reviewers: ["compliance", "legal"]
+approvers: ["3leapsdave"]
+tags: ["classification", "retention", "lifecycle", "compliance", "data-governance"]
+content_license: "CC0"
+relatedDocs:
+  - "docs/standards/data-sensitivity-classification.md"
+  - "config/classifiers/dimensions/retention-lifecycle.dimension.json"
+audience: "all"
+---
+
+# Retention & Lifecycle Classification
+
+This standard defines retention period classifications for data across all 3leaps ecosystems. It provides a consistent framework for:
+
+- **Deletion SLAs** - When data must be deleted
+- **Archival Policies** - When and how to archive
+- **Compliance Requirements** - Regulatory retention obligations
+- **Storage Optimization** - Tiered storage decisions
+- **Legal Hold Procedures** - Preservation requirements
+
+---
+
+## Retention Levels
+
+### Unknown
+
+**Retention not yet classified; must be classified before storage provisioning.**
+
+| Aspect        | Requirement                                     |
+| ------------- | ----------------------------------------------- |
+| **Retention** | Undefined; treat as indefinite until classified |
+| **Backup**    | Basic backup until policy determined            |
+| **Storage**   | Staging/quarantine                              |
+| **Deletion**  | Prohibited until classified                     |
+| **Use Cases** | New data sources, imported datasets             |
+
+**Operational Notes**: Gate storage tier decisions and deletion policies on explicit classification. Do not auto-delete `unknown` retention data.
+
+---
+
+### Transient
+
+**Short-lived data (≤7 days); auto-deleted, no backup required.**
+
+| Aspect        | Requirement                              |
+| ------------- | ---------------------------------------- |
+| **Retention** | ≤7 days                                  |
+| **Backup**    | Not required                             |
+| **Storage**   | Hot/ephemeral                            |
+| **Deletion**  | Automatic expiry                         |
+| **Use Cases** | Session cache, temp files, debug buffers |
+
+**Operational Notes**:
+
+- Use TTL-based storage (Redis, S3 lifecycle)
+- No archive or backup pipelines needed
+- Safe to delete without approval
+
+---
+
+### Short
+
+**Short retention (≤90 days); routine cleanup, basic backup.**
+
+| Aspect        | Requirement                             |
+| ------------- | --------------------------------------- |
+| **Retention** | ≤90 days                                |
+| **Backup**    | Basic (daily snapshots)                 |
+| **Storage**   | Standard                                |
+| **Deletion**  | Scheduled cleanup jobs                  |
+| **Use Cases** | Debug logs, dev environments, test data |
+
+**Operational Notes**:
+
+- Configure automated cleanup jobs
+- Basic backup for recovery during retention window
+- Review before deletion if business value uncertain
+
+---
+
+### Standard
+
+**Standard retention (≤2 years); regular backup, tiered storage.**
+
+| Aspect        | Requirement                                             |
+| ------------- | ------------------------------------------------------- |
+| **Retention** | ≤2 years                                                |
+| **Backup**    | Regular (with point-in-time recovery)                   |
+| **Storage**   | Tiered (hot → warm → cold)                              |
+| **Deletion**  | Policy-driven with approval                             |
+| **Use Cases** | Transaction history, operational data, business records |
+
+**Operational Notes**:
+
+- Default retention tier for most business data
+- Implement storage tiering for cost optimization
+- Document deletion policy and approval process
+
+---
+
+### Long
+
+**Long retention (>2 years); archive storage, compliance backup.**
+
+| Aspect        | Requirement                                        |
+| ------------- | -------------------------------------------------- |
+| **Retention** | >2 years (often 7+ years)                          |
+| **Backup**    | Compliance-grade (immutable, verified)             |
+| **Storage**   | Archive/cold (Glacier, etc.)                       |
+| **Deletion**  | Regulatory approval required                       |
+| **Use Cases** | Audit records, financial data, regulatory archives |
+
+**Operational Notes**:
+
+- Use archive storage classes for cost efficiency
+- Ensure compliance with industry regulations (SOX, HIPAA, etc.)
+- Implement integrity verification (checksums, attestation)
+
+---
+
+### Legal Hold
+
+**Indefinite retention; immutable, protected from deletion.**
+
+| Aspect        | Requirement                                                     |
+| ------------- | --------------------------------------------------------------- |
+| **Retention** | Indefinite (until released)                                     |
+| **Backup**    | Immutable, geographically redundant                             |
+| **Storage**   | WORM (Write-Once-Read-Many)                                     |
+| **Deletion**  | Prohibited without legal release                                |
+| **Use Cases** | Litigation evidence, regulatory investigation, breach forensics |
+
+**Operational Notes**:
+
+- Implement legal hold flag in data management systems
+- Chain of custody documentation required
+- Legal team controls release authorization
+- Prevent any modification or deletion
+
+---
+
+## Decision Guide
+
+```
+How long must we keep this data?
+
+├── Only while actively needed (hours/days) → transient
+├── Weeks to months (debugging, dev cycles) → short
+├── 1-2 years (operational history) → standard
+├── Years (regulatory, audit requirements) → long
+└── Until legal/regulatory release → legal-hold
+```
+
+---
+
+## Compliance Mapping
+
+| Regulation | Typical Retention | Recommended Tier |
+| ---------- | ----------------- | ---------------- |
+| SOX        | 7 years           | long             |
+| HIPAA      | 6 years           | long             |
+| GDPR       | Varies (minimize) | standard or less |
+| PCI DSS    | 1 year minimum    | standard         |
+| SEC 17a-4  | 6 years           | long             |
+| Litigation | Until resolved    | legal-hold       |
+
+---
+
+## Handling Matrix
+
+| Retention Level | Storage Tier  | Backup Strategy  | Deletion Process    | Cost Profile |
+| --------------- | ------------- | ---------------- | ------------------- | ------------ |
+| **transient**   | Hot/ephemeral | None             | Auto-expiry         | Lowest       |
+| **short**       | Standard      | Daily snapshots  | Scheduled jobs      | Low          |
+| **standard**    | Tiered        | Regular + PITR   | Policy + approval   | Medium       |
+| **long**        | Archive/cold  | Compliance-grade | Regulatory approval | Higher       |
+| **legal-hold**  | WORM          | Immutable        | Legal release only  | Highest      |
+
+---
+
+## Machine-Readable Definition
+
+- **Dimension Config**: `config/classifiers/dimensions/retention-lifecycle.dimension.json`
+- **Schema**: `schemas/classifiers/v0/dimension-definition.schema.json`
+
+---
+
+## Attribution
+
+This standard is the canonical reference for retention classification across 3leaps ecosystems. Downstream consumers should reference or vendor this standard rather than maintaining independent copies.
+
+**Review Cycle**: Semiannual with compliance and legal teams.

package/schemas/crucible-ts/upstream/3leaps/crucible/docs/standards/schema-stability-classification.md

@@ -0,0 +1,205 @@
+---
+title: "Schema Stability Classification"
+description: "Schema evolution and stability classification standard"
+category: "standards"
+status: "stable"
+version: "1.0.0"
+lastUpdated: "2026-01-22"
+maintainer: "3leaps-core"
+reviewers: ["platform", "api-standards"]
+approvers: ["3leapsdave"]
+tags: ["classification", "schema", "versioning", "api", "stability"]
+content_license: "CC0"
+relatedDocs:
+  - "docs/decisions/ADR-0001-schema-config-versioning.md"
+  - "config/classifiers/dimensions/schema-stability.dimension.json"
+audience: "all"
+---
+
+# Schema Stability Classification
+
+This standard defines stability levels for schemas, APIs, and configuration formats across all 3leaps ecosystems. It provides a consistent framework for:
+
+- **Compatibility Expectations** - What consumers can rely on
+- **Versioning Cadence** - How often changes occur
+- **Deprecation Windows** - How much notice before breaking changes
+- **Consumer Guidance** - Whether to adopt and how to track changes
+
+---
+
+## Stability Levels
+
+### Unknown
+
+**Stability not yet classified; must be classified before consumers adopt.**
+
+| Aspect               | Requirement                           |
+| -------------------- | ------------------------------------- |
+| **Compatibility**    | Unknown; assume none                  |
+| **Breaking Changes** | Possible at any time                  |
+| **Versioning**       | Not yet versioned                     |
+| **Deprecation**      | N/A                                   |
+| **Consumer Advice**  | Do not adopt; wait for classification |
+
+**Use Cases**: New schemas under initial development, imported definitions pending review.
+
+**Operational Notes**: Block production adoption of `unknown` stability schemas. Require explicit classification before publishing to consumers.
+
+---
+
+### Experimental
+
+**No stability guarantees; may change without notice.**
+
+| Aspect               | Requirement                                  |
+| -------------------- | -------------------------------------------- |
+| **Compatibility**    | None guaranteed                              |
+| **Breaking Changes** | Any time, no notice required                 |
+| **Versioning**       | v0.x or -alpha/-preview suffix               |
+| **Deprecation**      | None required                                |
+| **Consumer Advice**  | For evaluation only; expect breaking changes |
+
+**Use Cases**: Early prototypes, alpha features, proof-of-concept APIs, `v0/` schema directories.
+
+**Naming Conventions**:
+
+- Version: `v0.x.x`, `0.x`, or `vX-alpha`
+- Path: `/v0/`, `/preview/`, `/alpha/`
+
+---
+
+### Evolving
+
+**Active development; additive changes expected, breaking changes with notice.**
+
+| Aspect               | Requirement                                 |
+| -------------------- | ------------------------------------------- |
+| **Compatibility**    | Additive changes backward-compatible        |
+| **Breaking Changes** | With notice (changelog, deprecation period) |
+| **Versioning**       | v0.x with clear changelog                   |
+| **Deprecation**      | Minimum 1 minor version warning             |
+| **Consumer Advice**  | Safe for development; track changelog       |
+
+**Use Cases**: Beta features, active development APIs, schemas under iteration.
+
+**Naming Conventions**:
+
+- Version: `v0.x.x` or `vX-beta`
+- Path: `/v0/`, `/beta/`
+
+---
+
+### Stable
+
+**Production-ready; breaking changes require major version bump and deprecation period.**
+
+| Aspect               | Requirement                                      |
+| -------------------- | ------------------------------------------------ |
+| **Compatibility**    | Full backward compatibility within major version |
+| **Breaking Changes** | Major version bump required                      |
+| **Versioning**       | Semantic versioning (v1.x.x, v2.x.x)             |
+| **Deprecation**      | Minimum 6 months or 2 minor versions             |
+| **Consumer Advice**  | Production safe; follow semver guidelines        |
+
+**Use Cases**: Production APIs, released schemas, public interfaces.
+
+**Naming Conventions**:
+
+- Version: `v1.x.x`, `v2.x.x` (semver)
+- Path: `/v1/`, `/v2/`
+
+---
+
+### Frozen
+
+**No changes expected; security patches only.**
+
+| Aspect               | Requirement                                 |
+| -------------------- | ------------------------------------------- |
+| **Compatibility**    | No changes (security-only exceptions)       |
+| **Breaking Changes** | Only for critical security fixes            |
+| **Versioning**       | Patch versions only (vX.Y.Z → vX.Y.Z+1)     |
+| **Deprecation**      | N/A (may transition to deprecated)          |
+| **Consumer Advice**  | Maximum stability; plan for eventual sunset |
+
+**Use Cases**: Legacy APIs maintained for compatibility, long-term support versions.
+
+---
+
+### Deprecated
+
+**Scheduled for removal; migration path documented.**
+
+| Aspect               | Requirement                                      |
+| -------------------- | ------------------------------------------------ |
+| **Compatibility**    | Maintained until removal date                    |
+| **Breaking Changes** | None (frozen until removal)                      |
+| **Versioning**       | No new versions                                  |
+| **Deprecation**      | Removal date published, migration guide provided |
+| **Consumer Advice**  | Migrate immediately; removal imminent            |
+
+**Use Cases**: Sunset APIs, replaced schemas, legacy versions.
+
+**Required Documentation**:
+
+- Removal date
+- Replacement/migration path
+- Migration guide
+
+---
+
+## Lifecycle Transitions
+
+```
+experimental → evolving → stable → frozen → deprecated → removed
+                  ↑           |
+                  └───────────┘ (reactivation rare)
+```
+
+### Typical Progression
+
+1. **experimental**: Initial development, rapid iteration
+2. **evolving**: Beta users onboarded, stabilizing
+3. **stable**: GA release, production use
+4. **frozen**: Maintenance mode, new version available
+5. **deprecated**: Sunset announced, migration period
+6. **removed**: No longer available
+
+---
+
+## Versioning Guidelines
+
+| Stability Level | Version Pattern | Example      |
+| --------------- | --------------- | ------------ |
+| experimental    | v0.x.x          | v0.3.0-alpha |
+| evolving        | v0.x.x          | v0.8.0-beta  |
+| stable          | vX.Y.Z          | v1.2.3       |
+| frozen          | vX.Y.Z (patch)  | v1.2.4       |
+| deprecated      | vX.Y.Z (frozen) | v1.2.4       |
+
+---
+
+## Consumer Guidance Matrix
+
+| If you need...              | Use stability level |
+| --------------------------- | ------------------- |
+| Bleeding edge features      | experimental        |
+| New features with some risk | evolving            |
+| Production reliability      | stable              |
+| Maximum stability           | frozen              |
+| (Don't use)                 | deprecated          |
+
+---
+
+## Machine-Readable Definition
+
+- **Dimension Config**: `config/classifiers/dimensions/schema-stability.dimension.json`
+- **Schema**: `schemas/classifiers/v0/dimension-definition.schema.json`
+
+---
+
+## Attribution
+
+This standard is the canonical reference for schema stability classification across 3leaps ecosystems. Downstream consumers should reference or vendor this standard rather than maintaining independent copies.
+
+**Review Cycle**: Quarterly with platform and API standards teams.

package/schemas/crucible-ts/upstream/3leaps/crucible/docs/standards/velocity-mode-classification.md

@@ -0,0 +1,222 @@
+---
+title: "Velocity Mode Classification"
+description: "Data processing velocity pattern classification standard"
+category: "standards"
+status: "stable"
+version: "1.0.0"
+lastUpdated: "2026-01-22"
+maintainer: "3leaps-core"
+reviewers: ["platform", "data-engineering"]
+approvers: ["3leapsdave"]
+tags: ["classification", "velocity", "streaming", "batch", "data-engineering"]
+content_license: "CC0"
+relatedDocs:
+  - "docs/standards/volatility-classification.md"
+  - "docs/standards/volume-tier-classification.md"
+  - "config/classifiers/dimensions/velocity-mode.dimension.json"
+audience: "all"
+---
+
+# Velocity Mode Classification
+
+This standard defines velocity mode classifications for data processing across all 3leaps ecosystems. It provides a consistent framework for:
+
+- **Pipeline Topology** - Choosing batch vs. streaming architecture
+- **Infrastructure Requirements** - Selecting appropriate platforms
+- **Processing Semantics** - At-least-once, exactly-once, etc.
+- **Latency Expectations** - Setting SLAs for data freshness
+- **Cost Planning** - Understanding operational cost profiles
+
+---
+
+## Relationship to Volatility
+
+Velocity mode and volatility are complementary:
+
+- **Volatility** describes _how often data changes_ (update cadence)
+- **Velocity mode** describes _how data is processed_ (processing pattern)
+
+Common pairings:
+
+| Volatility | Typical Velocity Mode    |
+| ---------- | ------------------------ |
+| static     | batch                    |
+| monthly    | batch                    |
+| weekly     | batch                    |
+| daily      | batch                    |
+| hourly     | micro-batch or batch     |
+| streaming  | streaming or micro-batch |
+
+---
+
+## Velocity Modes
+
+### Unknown
+
+**Velocity mode not yet classified; must be classified before pipeline design.**
+
+| Aspect           | Characteristic         |
+| ---------------- | ---------------------- |
+| **Latency**      | Unknown                |
+| **Processing**   | Cannot design pipeline |
+| **Data State**   | Unknown                |
+| **Semantics**    | Cannot determine       |
+| **Cost Profile** | Cannot estimate        |
+
+**Use Cases**: New data sources, requirements gathering phase.
+
+**Operational Notes**: Gate pipeline design and infrastructure provisioning on explicit classification. Analyze source characteristics and latency requirements to determine appropriate mode.
+
+---
+
+### Batch
+
+**Scheduled batch processing; data at rest, periodic execution.**
+
+| Aspect           | Characteristic                       |
+| ---------------- | ------------------------------------ |
+| **Latency**      | Hours to days                        |
+| **Processing**   | Scheduled jobs (cron, Airflow, etc.) |
+| **Data State**   | At rest (complete dataset available) |
+| **Semantics**    | Full reprocessing possible           |
+| **Cost Profile** | Lowest (compute only when running)   |
+
+**Use Cases**: ETL pipelines, nightly aggregations, reporting, data warehouse loads.
+
+**Infrastructure**: Airflow, dbt, Spark batch, serverless functions.
+
+---
+
+### Micro-batch
+
+**Frequent small batches (minutes); near-real-time with batch semantics.**
+
+| Aspect           | Characteristic                         |
+| ---------------- | -------------------------------------- |
+| **Latency**      | Minutes (1-15 min typical)             |
+| **Processing**   | Triggered batches, windowed processing |
+| **Data State**   | Recent windows (tumbling/sliding)      |
+| **Semantics**    | Windowed exactly-once achievable       |
+| **Cost Profile** | Moderate (frequent compute cycles)     |
+
+**Use Cases**: Dashboard metrics, near-real-time analytics, alerting with delay tolerance.
+
+**Infrastructure**: Spark Structured Streaming, Flink (batch mode), scheduled lambdas.
+
+---
+
+### Streaming
+
+**Event-driven continuous processing; data in motion, sub-second latency.**
+
+| Aspect           | Characteristic                       |
+| ---------------- | ------------------------------------ |
+| **Latency**      | Sub-second to seconds                |
+| **Processing**   | Continuous (always-on)               |
+| **Data State**   | In motion (event at a time)          |
+| **Semantics**    | Exactly-once requires careful design |
+| **Cost Profile** | Highest (always-on compute)          |
+
+**Use Cases**: Real-time fraud detection, live dashboards, event sourcing, IoT processing.
+
+**Infrastructure**: Kafka Streams, Flink, Kinesis, Pulsar.
+
+**Considerations**:
+
+- Requires streaming infrastructure (message brokers)
+- Complex exactly-once semantics
+- Backpressure and scaling considerations
+- Higher operational complexity
+
+---
+
+### Hybrid
+
+**Combined batch and streaming; lambda/kappa architecture patterns.**
+
+| Aspect           | Characteristic                     |
+| ---------------- | ---------------------------------- |
+| **Latency**      | Mixed (real-time + historical)     |
+| **Processing**   | Parallel batch and streaming paths |
+| **Data State**   | Both at rest and in motion         |
+| **Semantics**    | Varies by path                     |
+| **Cost Profile** | High (maintaining both systems)    |
+
+**Use Cases**: Analytics platforms needing both historical and real-time, ML feature stores.
+
+**Architecture Patterns**:
+
+**Lambda Architecture**:
+
+```
+Events → [Streaming Layer] → Real-time Views
+      ↘ [Batch Layer] → Batch Views
+                     → [Serving Layer] → Query
+```
+
+**Kappa Architecture**:
+
+```
+Events → [Streaming Layer] → Views
+      → [Reprocessing] (replay from log)
+```
+
+---
+
+## Decision Guide
+
+```
+How quickly must data be processed after arrival?
+
+├── Hours/days acceptable → batch
+├── Minutes acceptable → micro-batch
+├── Seconds/sub-second required → streaming
+└── Need both real-time and historical → hybrid
+```
+
+### Cost-Latency Tradeoff
+
+| Velocity Mode | Latency    | Cost     | Complexity |
+| ------------- | ---------- | -------- | ---------- |
+| batch         | Hours-days | Lowest   | Simplest   |
+| micro-batch   | Minutes    | Moderate | Low        |
+| streaming     | Sub-second | High     | High       |
+| hybrid        | Mixed      | Highest  | Highest    |
+
+**Guidance**: Start with batch unless latency requirements demand otherwise. Micro-batch is often a good compromise.
+
+---
+
+## Processing Semantics
+
+| Velocity Mode   | At-least-once | Exactly-once | At-most-once |
+| --------------- | ------------- | ------------ | ------------ |
+| **batch**       | Easy          | Easy         | Easy         |
+| **micro-batch** | Easy          | Achievable   | Easy         |
+| **streaming**   | Default       | Complex      | Possible     |
+| **hybrid**      | By path       | By path      | By path      |
+
+---
+
+## Combining with Other Dimensions
+
+| Combination                                      | Implication                               |
+| ------------------------------------------------ | ----------------------------------------- |
+| `velocity: streaming` + `volume: large`          | Requires streaming at scale (Kafka+Flink) |
+| `velocity: batch` + `volatility: streaming`      | Mismatch—reconsider architecture          |
+| `velocity: micro-batch` + `retention: transient` | Good for dashboards with short history    |
+
+---
+
+## Machine-Readable Definition
+
+- **Dimension Config**: `config/classifiers/dimensions/velocity-mode.dimension.json`
+- **Schema**: `schemas/classifiers/v0/dimension-definition.schema.json`
+
+---
+
+## Attribution
+
+This standard is the canonical reference for velocity mode classification across 3leaps ecosystems. Downstream consumers should reference or vendor this standard rather than maintaining independent copies.
+
+**Review Cycle**: Semiannual with platform and data engineering teams.