@fulmenhq/tsfulmen 0.2.0 → 0.2.2

package/schemas/crucible-ts/upstream/3leaps/crucible/docs/standards/volatility-classification.md

@@ -0,0 +1,209 @@
+---
+title: "Volatility & Update Cadence Classification"
+description: "Standard for classifying data and configuration update frequency"
+category: "standards"
+status: "stable"
+version: "1.0.0"
+lastUpdated: "2026-01-22"
+maintainer: "3leaps-core"
+reviewers: ["platform", "data-engineering"]
+approvers: ["3leapsdave"]
+tags: ["classification", "volatility", "cadence", "scheduling", "data-lifecycle"]
+content_license: "CC0"
+relatedDocs:
+  - "schemas/classifiers/v0/dimension-definition.schema.json"
+  - "config/classifiers/dimensions/volatility.dimension.json"
+  - "docs/standards/data-sensitivity-classification.md"
+audience: "all"
+---
+
+# Volatility & Update Cadence Classification
+
+This standard defines update cadence levels for data and configuration across all 3leaps ecosystems. It provides a consistent framework for:
+
+- **Freshness SLAs** - Setting expectations for data currency
+- **Scheduling** - Determining batch job and pipeline frequencies
+- **Partitioning** - Informing time-based partitioning strategies
+- **Caching** - Setting appropriate TTLs and invalidation policies
+- **Resource Planning** - Estimating compute and storage requirements
+
+## Volatility Levels
+
+Volatility is an **ordinal** dimension—higher values indicate more frequent updates.
+
+| Level | Key         | Description             | Typical Use Cases                                       |
+| ----- | ----------- | ----------------------- | ------------------------------------------------------- |
+| 0     | `unknown`   | Not yet classified      | New data sources pending classification                 |
+| 1     | `static`    | No scheduled updates    | Reference data, schemas, standards, one-time snapshots  |
+| 2     | `monthly`   | Roughly monthly batches | Financial reports, compliance audits, capacity planning |
+| 3     | `weekly`    | Roughly weekly batches  | Product catalogs, pricing updates, aggregated metrics   |
+| 4     | `daily`     | Daily batches           | Transaction summaries, daily snapshots, ETL pipelines   |
+| 5     | `hourly`    | Sub-daily batches       | Operational metrics, near-real-time dashboards, alerts  |
+| 6     | `streaming` | Event-driven continuous | Real-time telemetry, live feeds, event sourcing         |
+
+---
+
+## Level Details
+
+### Unknown (0)
+
+**Volatility not yet classified; must be classified before operational use.**
+
+| Aspect           | Requirement                                                  |
+| ---------------- | ------------------------------------------------------------ |
+| **Scheduling**   | Prohibited until classified                                  |
+| **Caching**      | Conservative defaults only (no long TTL assumptions)         |
+| **Partitioning** | Avoid production partition strategy decisions                |
+| **Use Cases**    | New data feeds, imported datasets, unprofiled or new sources |
+
+**Operational Notes**: Gate operational use on explicit classification. Do not treat missing or unknown volatility as “static” or “daily” by default.
+
+---
+
+### Static (1)
+
+**No scheduled updates—one-time or ad-hoc changes only.**
+
+- Changes require explicit versioning and release process
+- Safe to cache indefinitely (until version changes)
+- Examples: JSON schemas, role definitions, reference taxonomies
+
+**Operational Implications**:
+
+- No scheduled refresh jobs needed
+- Version-based cache invalidation
+- Changes go through PR/review process
+
+---
+
+### Monthly (2)
+
+**Batch updates roughly monthly.**
+
+- Typically aligned with business cycles (month-end close, reporting periods)
+- Allow 24-48 hour processing windows
+- Examples: Financial statements, compliance reports, capacity forecasts
+
+**Operational Implications**:
+
+- Schedule during low-traffic windows
+- Plan for larger batch sizes
+- Coordinate with downstream consumers on refresh dates
+
+---
+
+### Weekly (3)
+
+**Batch updates roughly weekly.**
+
+- Common for curated datasets that balance freshness and processing cost
+- Examples: Product catalogs, aggregated analytics, weekly digests
+
+**Operational Implications**:
+
+- Typical refresh: weekends or early morning
+- Moderate batch sizes
+- Weekly SLA monitoring
+
+---
+
+### Daily (4)
+
+**Daily batches—the most common cadence for operational data.**
+
+- Standard for transactional summaries and operational reporting
+- Examples: Daily sales, order summaries, log aggregations
+
+**Operational Implications**:
+
+- Nightly batch windows (typically 00:00-06:00)
+- Date-partitioned storage recommended
+- T+1 data availability expectations
+
+---
+
+### Hourly (5)
+
+**Sub-daily batches—hourly or more frequent.**
+
+- Bridges gap between batch and streaming
+- Examples: Operational dashboards, alerting thresholds, rate limit counters
+
+**Operational Implications**:
+
+- Micro-batch processing
+- Hour-partitioned or rolling windows
+- Higher compute costs than daily
+- Consider streaming if approaching minute-level freshness needs
+
+---
+
+### Streaming (6)
+
+**Event-driven continuous updates—sub-minute latency.**
+
+- True real-time processing
+- Examples: Live telemetry, event sourcing, real-time fraud detection
+
+**Operational Implications**:
+
+- Requires streaming infrastructure (Kafka, Kinesis, Pulsar)
+- Continuous compute costs
+- Complex exactly-once semantics
+- Backpressure and scaling considerations
+
+---
+
+## Decision Guide
+
+```
+How quickly must consumers see new data?
+
+├── "Whenever we release a new version" → static
+├── "By the end of the month" → monthly
+├── "Within a week" → weekly
+├── "Next business day" → daily
+├── "Within hours" → hourly
+└── "Immediately / real-time" → streaming
+```
+
+### Cost-Freshness Tradeoff
+
+| Volatility | Relative Cost | Freshness           | Complexity |
+| ---------- | ------------- | ------------------- | ---------- |
+| static     | Lowest        | Stale until release | Simplest   |
+| monthly    | Low           | Up to 30 days       | Simple     |
+| weekly     | Low-Medium    | Up to 7 days        | Simple     |
+| daily      | Medium        | Up to 24 hours      | Moderate   |
+| hourly     | Medium-High   | Up to 1 hour        | Moderate   |
+| streaming  | Highest       | Sub-minute          | Complex    |
+
+**Guidance**: Start with the lowest volatility that meets business requirements. Upgrading to higher frequency is easier than optimizing an over-engineered streaming system.
+
+---
+
+## Combining with Other Dimensions
+
+Volatility works alongside other classifiers:
+
+| Combination                                         | Implication                                           |
+| --------------------------------------------------- | ----------------------------------------------------- |
+| `sensitivity: 4-personal` + `volatility: streaming` | Real-time PII requires streaming encryption and audit |
+| `sensitivity: 0-public` + `volatility: static`      | Cacheable forever, CDN-friendly                       |
+| `volatility: daily` + partitioning                  | Use date-based partitions                             |
+| `volatility: streaming` + storage                   | Consider append-only / event log storage              |
+
+---
+
+## Machine-Readable Definition
+
+- **Dimension Config**: `config/classifiers/dimensions/volatility.dimension.json`
+- **Schema**: `schemas/classifiers/v0/dimension-definition.schema.json`
+
+---
+
+## Attribution
+
+This standard is the canonical reference for volatility classification across 3leaps ecosystems. Downstream consumers should reference or vendor this standard rather than maintaining independent copies.
+
+**Review Cycle**: Semiannual with platform and data engineering teams.

package/schemas/crucible-ts/upstream/3leaps/crucible/docs/standards/volume-tier-classification.md

@@ -0,0 +1,200 @@
+---
+title: "Volume Tier Classification"
+description: "Data volume classification standard for scale planning"
+category: "standards"
+status: "stable"
+version: "1.0.0"
+lastUpdated: "2026-01-22"
+maintainer: "3leaps-core"
+reviewers: ["platform", "data-engineering"]
+approvers: ["3leapsdave"]
+tags: ["classification", "volume", "scale", "data-engineering", "partitioning"]
+content_license: "CC0"
+relatedDocs:
+  - "docs/standards/velocity-mode-classification.md"
+  - "config/classifiers/dimensions/volume-tier.dimension.json"
+audience: "all"
+---
+
+# Volume Tier Classification
+
+This standard defines volume tier levels for data across all 3leaps ecosystems. It provides a consistent framework for:
+
+- **Scale Planning** - Estimating infrastructure requirements
+- **Partitioning Strategies** - Optimal data organization
+- **File Sizing** - Appropriate chunk sizes for processing
+- **Storage Selection** - Choosing storage systems and formats
+- **Benchmark Profiles** - Performance testing at appropriate scale
+
+Volume tier is an **ordinal** dimension—higher values indicate larger scale.
+
+---
+
+## Volume Tiers
+
+| Tier        | Row Count | Typical Size | Processing Model           |
+| ----------- | --------- | ------------ | -------------------------- |
+| **unknown** | Unknown   | Unknown      | Cannot provision           |
+| **tiny**    | ≤100K     | <100 MB      | In-memory, single file     |
+| **small**   | ≤10M      | <10 GB       | Single-node                |
+| **medium**  | ≤1B       | <1 TB        | Distributed beneficial     |
+| **large**   | ≤100B     | <100 TB      | Distributed required       |
+| **massive** | >100B     | >100 TB      | Specialized infrastructure |
+
+---
+
+## Tier Details
+
+### Unknown
+
+**Volume not yet classified; must be classified before infrastructure provisioning.**
+
+| Aspect             | Guidance                                  |
+| ------------------ | ----------------------------------------- |
+| **Processing**     | Unknown; cannot provision infrastructure  |
+| **Storage Format** | Staging only                              |
+| **Partitioning**   | Cannot determine                          |
+| **Infrastructure** | Quarantine/staging environment            |
+| **Use Cases**      | New data feeds, imports pending profiling |
+
+**Operational Notes**: Gate infrastructure provisioning decisions on explicit classification. Profile data to determine appropriate tier before production deployment.
+
+---
+
+### Tiny (≤100K rows)
+
+**Very small datasets; single-file, in-memory processing.**
+
+| Aspect             | Guidance                                  |
+| ------------------ | ----------------------------------------- |
+| **Processing**     | In-memory (pandas, DuckDB, etc.)          |
+| **Storage Format** | CSV, JSON, single Parquet file            |
+| **Partitioning**   | None needed                               |
+| **Infrastructure** | Local machine, small container            |
+| **Use Cases**      | Test fixtures, config data, lookup tables |
+
+---
+
+### Small (≤10M rows)
+
+**Small datasets; single-node processing, moderate file sizes.**
+
+| Aspect             | Guidance                                      |
+| ------------------ | --------------------------------------------- |
+| **Processing**     | Single-node (laptop, small VM)                |
+| **Storage Format** | Parquet, CSV with compression                 |
+| **Partitioning**   | Optional (by date if time-series)             |
+| **Infrastructure** | Standard compute, local SSD                   |
+| **Use Cases**      | Product catalogs, user tables, reference data |
+
+---
+
+### Medium (≤1B rows)
+
+**Medium datasets; partitioned storage, distributed processing beneficial.**
+
+| Aspect             | Guidance                                   |
+| ------------------ | ------------------------------------------ |
+| **Processing**     | Distributed beneficial (Spark, Dask)       |
+| **Storage Format** | Columnar (Parquet, ORC) required           |
+| **Partitioning**   | Required (date, key columns)               |
+| **Infrastructure** | Cloud data warehouse, distributed compute  |
+| **Use Cases**      | Transaction history, event logs, analytics |
+
+**Optimization Tips**:
+
+- Partition by date for time-series data
+- Use predicate pushdown for queries
+- Consider data lake with metadata layer
+
+---
+
+### Large (≤100B rows)
+
+**Large datasets; distributed processing required, columnar formats.**
+
+| Aspect             | Guidance                                 |
+| ------------------ | ---------------------------------------- |
+| **Processing**     | Distributed required (Spark, Presto)     |
+| **Storage Format** | Columnar with compression (Parquet+Zstd) |
+| **Partitioning**   | Multi-level (date + key)                 |
+| **Infrastructure** | Data lake, distributed compute clusters  |
+| **Use Cases**      | Telemetry, clickstream, IoT sensors      |
+
+**Optimization Tips**:
+
+- Aggressive partitioning and clustering
+- Z-ordering or data skipping indexes
+- Consider separate hot/warm/cold storage
+
+---
+
+### Massive (>100B rows)
+
+**Massive datasets; specialized infrastructure, aggressive partitioning.**
+
+| Aspect             | Guidance                                         |
+| ------------------ | ------------------------------------------------ |
+| **Processing**     | Specialized systems (BigQuery, Redshift, custom) |
+| **Storage Format** | Native formats, custom codecs                    |
+| **Partitioning**   | Heavy (multi-dimension, sharding)                |
+| **Infrastructure** | Enterprise data platforms, dedicated clusters    |
+| **Use Cases**      | Global clickstream, genomics, simulation         |
+
+**Optimization Tips**:
+
+- Work with platform specialists
+- Consider materialized views/aggregates
+- Pre-compute common queries
+- Evaluate specialized databases
+
+---
+
+## Decision Guide
+
+```
+How many rows in your dataset?
+
+├── Thousands (≤100K) → tiny
+├── Millions (≤10M) → small
+├── Hundreds of millions (≤1B) → medium
+├── Tens of billions (≤100B) → large
+└── Hundreds of billions+ → massive
+```
+
+---
+
+## Infrastructure Recommendations
+
+| Volume Tier | Storage System             | Compute                | Format             |
+| ----------- | -------------------------- | ---------------------- | ------------------ |
+| **tiny**    | Local FS, S3 single file   | Local, small container | CSV, JSON          |
+| **small**   | S3/GCS, local SSD          | Single VM, serverless  | Parquet            |
+| **medium**  | Data lake (Delta, Iceberg) | Spark, serverless SQL  | Parquet + metadata |
+| **large**   | Data lake, warehouse       | Spark cluster, Presto  | Parquet + Zstd     |
+| **massive** | Enterprise DW, BigQuery    | Dedicated clusters     | Native formats     |
+
+---
+
+## Combining with Other Dimensions
+
+| Combination                             | Implication                                |
+| --------------------------------------- | ------------------------------------------ |
+| `volume: large` + `velocity: streaming` | Requires streaming infrastructure at scale |
+| `volume: tiny` + `sensitivity: 4`       | Small but needs secure handling            |
+| `volume: massive` + `retention: long`   | Archive storage strategy critical          |
+
+---
+
+## Machine-Readable Definition
+
+- **Dimension Config**: `config/classifiers/dimensions/volume-tier.dimension.json`
+- **Schema**: `schemas/classifiers/v0/dimension-definition.schema.json`
+
+---
+
+## Attribution
+
+This standard is the canonical reference for volume tier classification across 3leaps ecosystems. Downstream consumers should reference or vendor this standard rather than maintaining independent copies.
+
+**Review Cycle**: Semiannual with platform and data engineering teams.

package/schemas/crucible-ts/upstream/3leaps/crucible/schemas/ailink/v0/README.md

@@ -0,0 +1,48 @@
+# AILink Schemas v0
+
+Schemas for AI-powered backend integration.
+
+**Status**: Unstable (v0) - breaking changes may occur without notice.
+
+## Schemas
+
+| Schema                        | Purpose                                     |
+| ----------------------------- | ------------------------------------------- |
+| `prompt.schema.json`          | AILink prompt configuration                 |
+| `search-response.schema.json` | Base response structure for search/analysis |
+
+## Usage
+
+### Prompt Configuration
+
+Prompts use YAML frontmatter validated against `prompt.schema.json`:
+
+```yaml
+slug: my-prompt
+name: My Prompt
+description: What this prompt does
+version: 1.0.0
+input:
+  required_variables:
+    - query
+tools:
+  - type: web_search
+```
+
+The prompt body (after `---`) contains the system template in markdown.
+
+### Response Validation
+
+Responses are validated against `search-response.schema.json` or a domain-specific schema that extends it.
+
+## Schema URLs
+
+```
+https://schemas.3leaps.dev/ailink/v0/prompt.schema.json
+https://schemas.3leaps.dev/ailink/v0/search-response.schema.json
+```
+
+## Related
+
+- [namelens/namelens](https://github.com/namelens/namelens) - Reference implementation
+- [FulmenHQ Crucible](https://github.com/fulmenhq/crucible) - Enterprise extensions

package/schemas/crucible-ts/upstream/3leaps/{ailink → crucible/schemas/ailink}/v0/prompt.schema.json

@@ -4,12 +4,7 @@
   "title": "AILink Prompt Configuration",
   "description": "Schema for AILink prompt configuration files. Prompts define how AI backends process requests, including input variables, tools, and response validation.",
   "type": "object",
-  "required": [
-    "slug",
-    "name",
-    "description",
-    "version"
-  ],
+  "required": ["slug", "name", "description", "version"],
   "properties": {
     "slug": {
       "type": "string",
@@ -74,12 +69,7 @@
           "type": "array",
           "items": {
             "type": "string",
-            "enum": [
-              "image/png",
-              "image/jpeg",
-              "image/webp",
-              "image/gif"
-            ]
+            "enum": ["image/png", "image/jpeg", "image/webp", "image/gif"]
           },
           "description": "Accepted image MIME types (if accepts_images is true)"
         },
@@ -125,9 +115,7 @@
         },
         {
           "type": "object",
-          "required": [
-            "$ref"
-          ],
+          "required": ["$ref"],
           "properties": {
             "$ref": {
               "type": "string",
@@ -185,9 +173,7 @@
   "$defs": {
     "tool": {
       "type": "object",
-      "required": [
-        "type"
-      ],
+      "required": ["type"],
       "properties": {
         "type": {
           "type": "string",

package/schemas/crucible-ts/upstream/3leaps/{ailink → crucible/schemas/ailink}/v0/search-response.schema.json

@@ -4,9 +4,7 @@
   "title": "AILink Search Response",
   "description": "Base schema for AILink search/analysis response validation. Extensible for domain-specific responses.",
   "type": "object",
-  "required": [
-    "summary"
-  ],
+  "required": ["summary"],
   "properties": {
     "summary": {
       "type": "string",
@@ -15,13 +13,7 @@
     },
     "risk_level": {
       "type": "string",
-      "enum": [
-        "low",
-        "medium",
-        "high",
-        "critical",
-        "unknown"
-      ],
+      "enum": ["low", "medium", "high", "critical", "unknown"],
       "description": "Overall risk assessment"
     },
     "confidence": {
@@ -63,22 +55,11 @@
   "$defs": {
     "mention": {
       "type": "object",
-      "required": [
-        "source",
-        "description"
-      ],
+      "required": ["source", "description"],
       "properties": {
         "source": {
           "type": "string",
-          "enum": [
-            "web",
-            "news",
-            "social",
-            "github",
-            "registry",
-            "trademark",
-            "other"
-          ],
+          "enum": ["web", "news", "social", "github", "registry", "trademark", "other"],
           "description": "Source type of the mention"
         },
         "description": {
@@ -92,21 +73,12 @@
         },
         "relevance": {
           "type": "string",
-          "enum": [
-            "high",
-            "medium",
-            "low"
-          ],
+          "enum": ["high", "medium", "low"],
           "description": "Relevance to the query"
         },
         "sentiment": {
           "type": "string",
-          "enum": [
-            "positive",
-            "neutral",
-            "negative",
-            "mixed"
-          ],
+          "enum": ["positive", "neutral", "negative", "mixed"],
           "description": "Sentiment of the mention"
         },
         "date": {
@@ -119,9 +91,7 @@
     },
     "attachment": {
       "type": "object",
-      "required": [
-        "type"
-      ],
+      "required": ["type"],
       "properties": {
         "type": {
           "type": "string",