@documentation-robotics/cli 0.2.0 → 0.4.0

package/dist/integrations/github_copilot/skills/dr_09_dataset_layer/SKILL.md

@@ -15,13 +15,13 @@ triggers:
     "index",
     "field"
   ]
-version: 0.8.1
+version: 0.8.3
 ---
 
 # Data Store Layer Skill
 
 **Layer Number:** 08
-**Specification:** Metadata Model Spec v0.8.1
+**Specification:** Metadata Model Spec v0.8.3
 **Purpose:** Defines paradigm-neutral physical storage modeling, capturing databases, collections/tables, fields/columns, indexes, views, stored logic, validation rules, access patterns, event handlers, and retention policies across relational, document, key-value, time-series, and graph stores.
 
 ---
@@ -46,6 +46,11 @@ This layer supports **multiple storage paradigms**: relational (PostgreSQL, MySQ
 
 **Central Entity:** The **Collection** (table, document collection, stream) is the core modeling unit.
 
+> **`storedlogic` vs `collection` — common mistake:**
+> `storedlogic` is for stored **procedures, triggers, and user-defined functions** — executable logic that lives inside the database.
+> For relational **tables** (or document collections, streams, or buckets), use `collection` with `collectionType: TABLE` (or `DOCUMENT`, `STREAM`, etc.).
+> Using `storedlogic` for a table will fail schema validation.
+>
 > **CLI Introspection:** Run `dr schema types data-store` for the authoritative, always-current list of node types.
 > Run `dr schema node <type-id>` for full attribute details on any type (e.g., `dr schema node data-store.collection`).
 
@@ -71,6 +76,53 @@ This layer supports **multiple storage paradigms**: relational (PostgreSQL, MySQ
 
 ---
 
+## Type Decision Tree
+
+Use this decision tree **before assigning a type** to any storage element.
+
+Evaluate questions top-to-bottom. Stop at the first YES match.
+If none match, reconsider whether the concept belongs in a different layer.
+
+> **storedlogic vs. eventhandler:** If the element describes _what fires and when_ (the reactive trigger mechanism), use `eventhandler`. If it describes _the computation or logic that runs_ (a function, procedure, or script), use `storedlogic`.
+
+```
+Is this a database server/instance/cluster?
+  YES → data-store.database  (e.g., PostgreSQL instance, MongoDB Atlas cluster)
+
+Is this a logical grouping of collections (schema, keyspace, database prefix)?
+  YES → data-store.namespace  (e.g., PostgreSQL schema, Cassandra keyspace, MongoDB database)
+
+Is this a primary storage unit (table, document collection, stream, bucket, topic)?
+  YES → data-store.collection  (e.g., users table, orders collection, events stream)
+
+Is this a field or column definition inside a collection?
+  YES → data-store.field  (e.g., email VARCHAR, user_id UUID, created_at TIMESTAMP)
+
+Is this a query optimization index (B-tree, hash, compound, text, vector)?
+  YES → data-store.index  (e.g., idx_users_email, full-text search index)
+
+Is this a derived or materialized view over one or more collections?
+  YES → data-store.view  (e.g., active_users_view, monthly_revenue_mv)
+
+Is this a stored procedure, function, or user-defined aggregate in the database?
+  YES → data-store.storedlogic  (e.g., calculate_discount(), get_user_stats(), normalize_email())
+
+Is this a database-level validation constraint or schema enforcement rule?
+  YES → data-store.validationrule  (e.g., check constraint, JSON schema validator,
+        foreign key [database-enforced referential integrity — not a cross-layer relationship])
+
+Is this a named query access pattern describing how the application reads or writes data?
+  YES → data-store.accesspattern  (e.g., get-user-by-email, list-orders-by-date, time-range-query)
+
+Is this a CDC handler, database trigger, or event-driven data workflow?
+  YES → data-store.eventhandler  (e.g., on-insert audit log, DynamoDB Streams handler)
+
+Is this a TTL, archival, or data lifecycle rule?
+  YES → data-store.retentionpolicy  (e.g., 90-day audit log TTL, GDPR deletion policy)
+```
+
+---
+
 ## When to Use This Skill
 
 Activate when the user:
@@ -85,19 +137,81 @@ Activate when the user:
 
 ---
 
-## Cross-Layer Relationships
-
-**Outgoing (Data Store → Other Layers):**
-
-- `x-json-schema` → Data Model Layer (what logical schema does this implement?)
-- `x-governed-by-*` → Security Layer (data access policies)
-- `x-apm-performance-metrics` → APM Layer (query performance monitoring)
+## Intra-Layer Relationships
+
+Containment is one level at a time (database → namespace → collection → fields, indexes, validation
+rules, event handlers). Stored data is passive: collections, views, fields and databases are accessed,
+referenced and mapped, and never serve or trigger. Behaviour in the store is stored logic, event
+handlers and access patterns.
+
+| Source Type     | Predicate    | Target Type    | Example                                                                |
+| --------------- | ------------ | -------------- | ---------------------------------------------------------------------- |
+| Database        | composes     | Namespace      | "orders-db" composes the `public` schema                               |
+| Database        | composes     | Collection     | "sessions-redis" composes the `session:*` key collection               |
+| Database        | replicates   | Database       | "orders-db-primary" replicates to "orders-db-replica"                  |
+| Database        | federates    | Database       | "analytics-trino" federates "orders-db"                                |
+| Namespace       | composes     | Collection     | `public` schema composes the `orders` table                            |
+| Namespace       | composes     | View           | `reporting` schema composes `monthly_revenue_mv`                       |
+| Namespace       | composes     | StoredLogic    | `public` schema composes `calculate_discount()`                        |
+| Collection      | composes     | Field          | `orders` composes `orders.customer_id`                                 |
+| Collection      | composes     | Index          | `orders` composes `idx_orders_customer_id`                             |
+| Collection      | composes     | ValidationRule | `orders` composes the `chk_orders_total_positive` check constraint     |
+| Collection      | composes     | EventHandler   | `orders` composes the `orders_audit_trigger`                           |
+| Collection      | references   | Collection     | `orders` references `customers` (foreign key)                          |
+| Field           | composes     | Field          | `address` sub-document composes `address.city`                         |
+| Index           | aggregates   | Field          | `idx_orders_customer_date` aggregates `customer_id` and `created_at`   |
+| AccessPattern   | accesses     | Collection     | "list-orders-by-customer" accesses `orders`                            |
+| AccessPattern   | uses         | Index          | "list-orders-by-customer" uses `idx_orders_customer_date`              |
+| AccessPattern   | references   | Field          | "list-orders-by-customer" references `orders.customer_id`              |
+| View            | derives-from | Collection     | `active_users_view` derives from `users`                               |
+| View            | derives-from | View           | `quarterly_revenue_mv` derives from `monthly_revenue_mv`               |
+| StoredLogic     | accesses     | Collection     | `get_user_stats()` accesses `orders`                                   |
+| StoredLogic     | migrates     | Collection     | `migrate_2024_10_split_address()` migrates `customers`                 |
+| EventHandler    | references   | Field          | `orders_status_trigger` references `orders.status` (AFTER UPDATE OF)   |
+| EventHandler    | triggers     | StoredLogic    | `orders_audit_trigger` triggers `write_audit_row()`                    |
+| ValidationRule  | references   | Field          | `uq_users_email` references `users.email`                              |
+| ValidationRule  | cascades-to  | Collection     | `fk_order_items_order` (`ON DELETE CASCADE`) cascades to `order_items` |
+| RetentionPolicy | governs      | Collection     | "Audit Log Retention" governs `audit_log`                              |
+| RetentionPolicy | archives-to  | Collection     | "Audit Log Retention" archives to `audit_log_archive`                  |
 
-**Incoming (Other Layers → Data Store):**
+---
 
-- Data Model Layer → Data Store (physical storage mapping)
-- Application Layer → Data Store (database connections)
-- Technology Layer → Data Store (hosting infrastructure)
+## Cross-Layer References
+
+Cross-layer links are created with `dr relationship add`, not inline YAML attributes.
+
+### Outgoing References (Data Store → Other Layers)
+
+| Source Type     | Predicate  | Target Type                       | Example                                                           |
+| --------------- | ---------- | --------------------------------- | ----------------------------------------------------------------- |
+| Collection      | realizes   | `business.businessobject`         | `orders` table realizes the "Customer Order" business object      |
+| Collection      | realizes   | `security.secureresource`         | `patients` collection realizes the "Patient Records" resource     |
+| Collection      | references | `security.dataclassification`     | `payments` collection references the "PCI" classification         |
+| Field           | references | `security.dataclassification`     | `users.email` references the "PII" classification                 |
+| Field           | requires   | `security.fieldaccesscontrol`     | `customers.ssn` requires the "SSN Masking" field control          |
+| Field           | maps-to    | `api.parameter`                   | `orders.id` maps to the `{orderId}` path parameter                |
+| Database        | depends-on | `technology.systemsoftware`       | "orders-db" depends on "PostgreSQL 16"                            |
+| Database        | depends-on | `technology.node`                 | "orders-db" depends on the "db-primary-01" node                   |
+| Database        | satisfies  | `security.securitypolicy`         | "orders-db" satisfies the "Encryption at Rest" policy             |
+| StoredLogic     | realizes   | `application.applicationfunction` | `calculate_discount()` realizes "Apply Promotions"                |
+| EventHandler    | triggers   | `application.applicationevent`    | "orders CDC stream" triggers "Order Placed"                       |
+| AccessPattern   | serves     | `application.applicationfunction` | "get-user-by-email" serves "Authenticate User"                    |
+| View            | maps-to    | `api.response`                    | `order_summary_view` maps to the `GET /orders/summary` response   |
+| RetentionPolicy | satisfies  | `security.retentionpolicy`        | "Audit Log Retention" satisfies the "SOC2 Audit Retention" policy |
+
+### Incoming References (Other Layers → Data Store)
+
+| Source Type                        | Predicate | Target Type    | Example                                                          |
+| ---------------------------------- | --------- | -------------- | ---------------------------------------------------------------- |
+| `application.applicationcomponent` | accesses  | Collection     | "OrderRepository" accesses `orders`                              |
+| `application.applicationfunction`  | accesses  | Collection     | "findOrdersByCustomer" accesses `orders`                         |
+| `application.applicationservice`   | accesses  | View           | "Reporting Service" accesses `monthly_revenue_mv`                |
+| `data-model.objectschema`          | maps-to   | Collection     | "User" schema maps to the `users` table                          |
+| `data-model.schemadefinition`      | maps-to   | Collection     | "AuditEntry" definition maps to `audit_log`                      |
+| `testing.testcoveragetarget`       | covers    | StoredLogic    | "Discount rules" coverage target covers `calculate_discount()`   |
+| `testing.testcasesketch`           | tests     | ValidationRule | "Reject negative total" sketch tests `chk_orders_total_positive` |
+| `apm.metricinstrument`             | monitors  | Database       | "db.client.connections.usage" monitors "orders-db"               |
+| `apm.span`                         | monitors  | Collection     | "SELECT orders" span monitors `orders`                           |
 
 ---
 
@@ -106,7 +220,7 @@ Activate when the user:
 1. **Paradigm-neutral modeling** — Use `collection`/`field` regardless of whether the underlying store is relational or document
 2. **Access patterns first** — For NoSQL (DynamoDB, Cassandra), define `AccessPattern` entities before collections
 3. **Indexes** — Add indexes for frequent query paths; use `AccessPattern` to document which index serves which pattern
-4. **PII marking** — Use `x-pii` on `field` entities to mark sensitive data
+4. **PII marking** — Link sensitive `field` entities to a security `dataclassification` node via `field.satisfies.security.dataclassification`; note PII status in the field `description`
 5. **Retention policies** — Always add a `RetentionPolicy` for collections with regulatory or storage requirements
 6. **Stored logic** — Capture stored procedures, triggers, and UDFs as `StoredLogic` entities
 7. **Event handlers** — Document CDC (change-data-capture) and event-driven triggers as `EventHandler` entities
@@ -118,28 +232,28 @@ Activate when the user:
 
 ```bash
 # Add a database instance
-dr add data-store database --name "users-db"
+dr add data-store database "users-db"
 
 # Add a namespace (schema or keyspace)
-dr add data-store namespace --name "public" --property parentDatabase=data-store.database.users-db
+dr add data-store namespace "public" --description "Default database namespace"
 
 # Add a collection (table or document collection)
-dr add data-store collection --name "users" --property parentNamespace=data-store.namespace.public
+dr add data-store collection "users" --description "User records collection"
 
 # Add a field to a collection
-dr add data-store field --name "email" --property dataType=string --property nullable=false
+dr add data-store field "email" --description "User email address"
 
 # Add an index
-dr add data-store index --name "idx-users-email" --property fields='["email"]' --property unique=true
+dr add data-store index "idx-users-email" --description "Index on email field"
 
 # Add an access pattern (for NoSQL capacity planning)
-dr add data-store accesspattern --name "get-user-by-email" --property queryType=point-lookup
+dr add data-store accesspattern "get-user-by-email" --description "Point lookup by email"
 
 # List collections
-dr list data-store collection
+dr list data-store --type collection
 
 # Validate data-store layer
-dr validate --layer data-store
+dr validate --layers data-store
 
 # Introspect available types
 dr schema types data-store
@@ -150,58 +264,71 @@ dr schema types data-store
 ## Example: Users Collection (Paradigm-Neutral)
 
 ```yaml
+# Collection — use collectionType to specify the paradigm-specific storage unit
 id: data-store.collection.users
 name: "Users Collection"
 type: collection
+description: "User account records — relational table (PostgreSQL)"
+properties:
+  collectionType: TABLE
+  partitionKey: "id"
+  validationSchema: data-model.object-schema.user
+```
+
+```yaml
+# Fields are separate data-store.field elements — not nested inside the collection
+id: data-store.field.users-id
+name: "Users ID"
+type: field
+description: "Primary key"
+properties:
+  dataType: uuid
+  nullable: false
+  fieldRole: PARTITION_KEY
+
+id: data-store.field.users-email
+name: "Users Email"
+type: field
+description: "User email address — PII"
 properties:
-  parentNamespace: data-store.namespace.public
-  paradigm: relational # or: document, key-value, time-series, graph
-  fields:
-    - id:
-        dataType: uuid
-        nullable: false
-        primaryKey: true
-    - email:
-        dataType: string
-        nullable: false
-        x-pii: true
-        x-encrypted: true
-    - username:
-        dataType: string
-        nullable: false
-    - created_at:
-        dataType: timestamp
-        nullable: false
-  x-json-schema: data-model.object-schema.user
-  x-apm-performance-metrics:
-    - apm.metric.users-query-latency
+  dataType: string
+  nullable: false
+
+id: data-store.field.users-created-at
+name: "Users Created At"
+type: field
+properties:
+  dataType: timestamp
+  nullable: false
 ```
 
-### Access Pattern (for DynamoDB/NoSQL)
+### Access Pattern
 
 ```yaml
 id: data-store.accesspattern.get-user-by-email
 name: "Get User by Email"
 type: accesspattern
+description: "Point lookup by email — used for login and profile fetch"
 properties:
-  collection: data-store.collection.users
-  queryType: point-lookup
-  keyAttributes: ["email"]
-  consistencyLevel: strong
-  estimatedRps: 500
+  patternType: POINT_READ
+  targetCollection: data-store.collection.users
+  keyCondition: "email"
+  consistencyRequirement: STRONG
+  expectedFrequency: HIGH_THROUGHPUT
 ```
 
 ### Retention Policy
 
 ```yaml
-id: data-store.retentionpolicy.users-audit-log
-name: "Users Audit Log Retention"
+id: data-store.retentionpolicy.audit-log-retention
+name: "Audit Log Retention"
 type: retentionpolicy
+description: "365-day retention for regulatory compliance (SOC2, GDPR Article 30)"
 properties:
-  collection: data-store.collection.users-audit-log
-  ttlDays: 365
-  archiveAfterDays: 90
-  regulatoryBasis: "SOC2, GDPR Article 30"
+  targetCollection: data-store.collection.users-audit-log
+  retentionDuration: "P365D"
+  action: ARCHIVE
+  enabled: true
 ```
 
 ---
@@ -210,7 +337,32 @@ properties:
 
 - ❌ Using SQL-only concepts (Table, Column, Constraint) — use paradigm-neutral `collection`, `field`, `validationrule`
 - ❌ Skipping `AccessPattern` for NoSQL stores (DynamoDB, Cassandra) — define access patterns first
-- ❌ Not marking PII fields with `x-pii`
-- ❌ Missing cross-layer links to data model layer (`x-json-schema`)
+- ❌ Nesting fields inline inside a collection element — `field` entities are always separate elements linked via `collection.composes.field`
+- ❌ Using invented `x-pii`, `x-json-schema`, or `x-apm-performance-metrics` attributes — these are not in the spec; use relationships (`field satisfies security.dataclassification`, `data-model.objectschema maps-to collection`) instead
+- ❌ Using `ttlDays` / `archiveAfterDays` in retentionpolicy — use `retentionDuration` (ISO 8601, e.g. `"P365D"`) and `action` (enum: DELETE | ARCHIVE | ...)
 - ❌ Forgetting `RetentionPolicy` for regulated data
 - ❌ Not documenting `EventHandler` for CDC or change-triggered workflows
+
+---
+
+## Coverage Completeness Checklist
+
+Before declaring data-store layer extraction complete, verify each type was considered:
+
+- [ ] **database** — Database instance (any paradigm: relational, document, key-value, time-series, graph)
+- [ ] **namespace** — Logical grouping of collections (PostgreSQL schema, Cassandra keyspace, MongoDB database)
+- [ ] **collection** — Primary storage unit (table, document collection, stream, bucket)
+- [ ] **field** — Field or column definition with data type and constraints
+- [ ] **index** — Query optimization index (B-tree, hash, compound, text, geospatial, vector)
+- [ ] **view** — Derived or materialized view over one or more collections
+- [ ] **storedlogic** — Stored procedures, functions, and user-defined aggregates
+- [ ] **validationrule** — Database-level validation constraint or schema enforcement rule
+- [ ] **accesspattern** — Named query access pattern (especially required for NoSQL: DynamoDB, Cassandra, MongoDB)
+- [ ] **eventhandler** — Event-driven trigger or change-data-capture handler
+- [ ] **retentionpolicy** — Data lifecycle, TTL, and retention rule definition
+
+If any type has ZERO elements, explicitly decide:
+"This type doesn't apply to this codebase" with reasoning.
+
+> **Note:** `accesspattern` is strongly recommended for any NoSQL store (DynamoDB, Cassandra, Firestore) — NoSQL schema design is driven by access patterns.
+> `retentionpolicy` is strongly recommended for any collection subject to regulatory requirements (GDPR, SOC2, HIPAA).