opencode-skills-collection 2.0.0 → 2.0.2

package/bundled-skills/monte-carlo-push-ingestion/references/direct-http-api.md

@@ -0,0 +1,207 @@
+# Direct HTTP API (without pycarlo)
+
+The `pycarlo` SDK is optional. You can call the push APIs directly over HTTPS from any
+language or tool (curl, Postman, etc.) as long as you:
+- authenticate with an integration key whose scope is `Ingestion`
+- send a JSON body that matches the ingest schema
+- send to the correct integration gateway endpoint
+
+## Endpoint
+
+The host is environment-specific:
+- **Production**: `https://integrations.getmontecarlo.com`
+
+## Authentication headers
+
+All requests use the same headers:
+```
+x-mcd-id:       <integration-key-id>
+x-mcd-token:    <integration-key-secret>
+Content-Type:   application/json
+```
+
+## Response
+
+On success, all endpoints return:
+```json
+{"invocation_id": "<uuid>"}
+```
+
+Save the `invocation_id` — it is the primary trace ID for debugging across downstream systems.
+
+---
+
+## Metadata example
+
+`POST /ingest/v1/metadata`
+
+```bash
+curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/metadata" \
+  -H "Content-Type: application/json" \
+  -H "x-mcd-id: <integration-key-id>" \
+  -H "x-mcd-token: <integration-key-secret>" \
+  -d '{
+    "event_type": "RELATIONAL_ASSET",
+    "resource": {
+      "uuid": "<warehouse-uuid>",
+      "resource_type": "snowflake"
+    },
+    "events": [
+      {
+        "type": "TABLE",
+        "metadata": {
+          "name": "orders",
+          "database": "analytics",
+          "schema": "public",
+          "description": "Orders table"
+        },
+        "fields": [
+          {"name": "id", "type": "INTEGER"},
+          {"name": "amount", "type": "DECIMAL(10,2)"}
+        ],
+        "volume": {
+          "row_count": 1000000,
+          "byte_count": 111111111
+        },
+        "freshness": {
+          "last_update_time": "2026-03-12T14:30:00Z"
+        }
+      }
+    ]
+  }'
+```
+
+`volume` and `freshness` are optional — you can push schema-only metadata.
+
+---
+
+## Table lineage example
+
+`POST /ingest/v1/lineage` with `event_type: "LINEAGE"`
+
+```bash
+curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/lineage" \
+  -H "Content-Type: application/json" \
+  -H "x-mcd-id: <integration-key-id>" \
+  -H "x-mcd-token: <integration-key-secret>" \
+  -d '{
+    "event_type": "LINEAGE",
+    "resource": {
+      "uuid": "<warehouse-uuid>",
+      "resource_type": "snowflake"
+    },
+    "events": [
+      {
+        "source": {
+          "name": "orders_raw",
+          "database": "analytics",
+          "schema": "public"
+        },
+        "destination": {
+          "name": "orders_curated",
+          "database": "analytics",
+          "schema": "public"
+        }
+      }
+    ]
+  }'
+```
+
+---
+
+## Column lineage example
+
+`POST /ingest/v1/lineage` with `event_type: "COLUMN_LINEAGE"`
+
+Same endpoint as table lineage. Column lineage automatically creates the parent table-level
+edge too.
+
+```bash
+curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/lineage" \
+  -H "Content-Type: application/json" \
+  -H "x-mcd-id: <integration-key-id>" \
+  -H "x-mcd-token: <integration-key-secret>" \
+  -d '{
+    "event_type": "COLUMN_LINEAGE",
+    "resource": {
+      "uuid": "<warehouse-uuid>",
+      "resource_type": "snowflake"
+    },
+    "events": [
+      {
+        "source": {
+          "name": "customers",
+          "database": "analytics",
+          "schema": "public"
+        },
+        "destination": {
+          "name": "customer_orders",
+          "database": "analytics",
+          "schema": "public"
+        },
+        "col_mappings": [
+          {
+            "destination_col": "customer_id",
+            "source_cols": ["customer_id"]
+          },
+          {
+            "destination_col": "full_name",
+            "source_cols": ["first_name", "last_name"]
+          }
+        ]
+      }
+    ]
+  }'
+```
+
+---
+
+## Query log example
+
+`POST /ingest/v1/querylogs`
+
+**Important**: this endpoint uses `log_type` instead of `resource_type` in the resource object.
+This is the only endpoint where the field name differs.
+
+```bash
+curl -X POST "https://integrations.getmontecarlo.com/ingest/v1/querylogs" \
+  -H "Content-Type: application/json" \
+  -H "x-mcd-id: <integration-key-id>" \
+  -H "x-mcd-token: <integration-key-secret>" \
+  -d '{
+    "event_type": "QUERY_LOG",
+    "resource": {
+      "uuid": "<warehouse-uuid>",
+      "log_type": "snowflake"
+    },
+    "events": [
+      {
+        "start_time": "2026-03-02T12:00:00Z",
+        "end_time": "2026-03-02T12:00:05Z",
+        "query_text": "SELECT * FROM analytics.public.orders",
+        "query_id": "query-123",
+        "user": "analyst@company.com",
+        "returned_rows": 10
+      }
+    ]
+  }'
+```
+
+Supported `log_type` values: `snowflake`, `bigquery`, `databricks`, `redshift`, `hive-s3`,
+`athena`, `teradata`, `clickhouse`, `databricks-metastore-sql-warehouse`, `s3`, `presto-s3`.
+
+---
+
+## Batching
+
+The compressed request body must not exceed **1MB** (Kinesis limit). For large payloads, split
+events into multiple requests. Each request returns its own `invocation_id`.
+
+## Expiration summary
+
+| Flow | Expiration |
+|---|---|
+| Table metadata | Never expires |
+| Table lineage | Never expires |
+| Column lineage | Expires after 10 days |
+| Query logs | Same as pulled query logs |

package/bundled-skills/monte-carlo-push-ingestion/references/prerequisites.md

@@ -0,0 +1,150 @@
+# Prerequisites
+
+## Two keys, two purposes
+
+Push ingestion requires **two separate Monte Carlo API keys** — one for pushing data, one
+for reading/verifying it. They use identical header names but different endpoints.
+
+| Key | Purpose | Endpoint |
+|---|---|---|
+| **Ingestion key** (scope=`Ingestion`) | Push metadata, lineage, query logs | `https://integrations.getmontecarlo.com` |
+| **GraphQL API key** | Verify pushed data, run management mutations | `https://api.getmontecarlo.com/graphql` |
+
+Both authenticate with:
+```
+x-mcd-id:    <key-id>
+x-mcd-token: <key-secret>
+```
+
+The secret for both is shown **only once** at creation time — store it securely immediately.
+
+---
+
+## Create the Ingestion key (for pushing)
+
+Use the Monte Carlo CLI:
+
+```bash
+montecarlo integrations create-key \
+  --scope Ingestion \
+  --description "Push ingestion key"
+```
+
+Output:
+```
+Key id:     <id>
+Key secret: <secret>    ← only shown once
+```
+
+Install the CLI if needed:
+```bash
+pip install montecarlodata
+montecarlo configure   # enter your API key when prompted
+```
+
+**Optional — restrict to a specific warehouse:**
+If you want the key to only work for one warehouse UUID, use the GraphQL mutation instead:
+
+```graphql
+mutation {
+  createIntegrationKey(
+    description: "Push key for warehouse XYZ"
+    scope: Ingestion
+    warehouseIds: ["<warehouse-uuid>"]
+  ) {
+    key { id secret }
+  }
+}
+```
+
+---
+
+## Create the GraphQL API key (for verification)
+
+1. Go to **https://getmontecarlo.com/settings/api**
+2. Click **Add**
+3. Choose key type (personal or account-level — account-level requires Account Owner role)
+4. Copy the **Key ID** and **Secret** immediately
+
+The GraphQL endpoint is: `https://api.getmontecarlo.com/graphql`
+
+Test it:
+```bash
+curl -s -X POST https://api.getmontecarlo.com/graphql \
+  -H "x-mcd-id: <id>" \
+  -H "x-mcd-token: <secret>" \
+  -H "Content-Type: application/json" \
+  -d '{"query": "{ getUser { email } }"}' | python3 -m json.tool
+```
+
+---
+
+## Find your warehouse (resource) UUID
+
+The Ingestion key needs to reference the correct MC resource UUID. To find it:
+
+```graphql
+query {
+  getUser {
+    account {
+      warehouses {
+        uuid
+        name
+        connectionType
+      }
+    }
+  }
+}
+```
+
+Or in the MC UI: **Settings → Integrations** → click the warehouse → copy the UUID from the URL.
+
+---
+
+## Install pycarlo (optional)
+
+The pycarlo SDK simplifies push calls, but is not required. You can also call the push APIs
+directly via HTTP/curl — see `references/direct-http-api.md`.
+
+```bash
+pip install pycarlo
+```
+
+Initialize the ingestion client in your script:
+
+```python
+from pycarlo.core import Client, Session
+from pycarlo.features.ingestion import IngestionService
+
+client = Client(session=Session(
+    mcd_id="<ingestion-key-id>",
+    mcd_token="<ingestion-key-secret>",
+    scope="Ingestion",
+))
+service = IngestionService(mc_client=client)
+```
+
+Load credentials from environment variables (recommended):
+
+```python
+import os
+service = IngestionService(mc_client=Client(session=Session(
+    mcd_id=os.environ["MCD_INGEST_ID"],
+    mcd_token=os.environ["MCD_INGEST_TOKEN"],
+    scope="Ingestion",
+)))
+```
+
+---
+
+## Environment variable conventions
+
+The script templates use these env var names by default:
+
+| Variable | Key type | Used by |
+|---|---|---|
+| `MCD_INGEST_ID` | Ingestion key ID | push and collect_and_push scripts |
+| `MCD_INGEST_TOKEN` | Ingestion key secret | push and collect_and_push scripts |
+| `MCD_ID` | GraphQL API key ID | verification scripts, slash commands |
+| `MCD_TOKEN` | GraphQL API key secret | verification scripts, slash commands |
+| `MCD_RESOURCE_UUID` | Warehouse UUID | all scripts |

package/bundled-skills/monte-carlo-push-ingestion/references/push-lineage.md

@@ -0,0 +1,160 @@
+# Pushing Table and Column Lineage
+
+## Overview
+
+Both table-level and column-level lineage use the same endpoint: `POST /ingest/v1/lineage`.
+The `event_type` field distinguishes them:
+- `LINEAGE` — table-level: source table → destination table
+- `COLUMN_LINEAGE` — column-level: source table.column → destination table.column
+  (also automatically creates the parent table-level edge)
+
+Push lineage is **typically visible in the MC lineage graph within seconds to a few minutes**
+via the fast direct path (PushLineageProcessor → S3 CSVs → neo4jLineageLoaderPrivate → Neo4j).
+
+**Expiration**:
+- Pushed **table lineage does not expire** (`expire_at = 9999-12-31`).
+- Pushed **column lineage expires after 10 days** (same as pulled column lineage).
+
+**Batching**: For large numbers of lineage events, split into batches. The compressed request
+body must not exceed **1MB** (Kinesis limit).
+
+## pycarlo models
+
+```python
+from pycarlo.features.ingestion import (
+    IngestionService,
+    LineageEvent,
+    LineageAssetRef,
+    ColumnLineageField,
+    ColumnLineageSourceField,
+)
+```
+
+## Table lineage example
+
+```python
+event = LineageEvent(
+    destination=LineageAssetRef(
+        database="analytics",
+        schema="public",
+        table="customer_orders",
+    ),
+    sources=[
+        LineageAssetRef(database="analytics", schema="public", table="customers"),
+        LineageAssetRef(database="analytics", schema="public", table="orders"),
+    ],
+)
+
+result = service.send_lineage(
+    resource_uuid="<your-resource-uuid>",
+    resource_type="data-lake",
+    events=[event],
+)
+invocation_id = service.extract_invocation_id(result)
+print("invocation_id:", invocation_id)
+```
+
+## Column lineage example
+
+```python
+event = LineageEvent(
+    destination=LineageAssetRef(
+        database="analytics",
+        schema="public",
+        table="customer_orders",
+    ),
+    sources=[
+        LineageAssetRef(database="analytics", schema="public", table="customers"),
+        LineageAssetRef(database="analytics", schema="public", table="orders"),
+    ],
+    # column mappings: dest_col ← src_table.src_col
+    fields=[
+        ColumnLineageField(
+            destination_field="customer_id",
+            source_fields=[
+                ColumnLineageSourceField(
+                    database="analytics", schema="public",
+                    table="customers", field="customer_id",
+                )
+            ],
+        ),
+        ColumnLineageField(
+            destination_field="order_amount",
+            source_fields=[
+                ColumnLineageSourceField(
+                    database="analytics", schema="public",
+                    table="orders", field="amount",
+                )
+            ],
+        ),
+    ],
+)
+
+result = service.send_lineage(
+    resource_uuid=resource_uuid,
+    resource_type="data-lake",
+    events=[event],
+)
+```
+
+Column lineage push automatically creates a table-level edge too, so you don't need to
+send separate table and column lineage events for the same relationship.
+
+## Extracting lineage from SQL logs
+
+For warehouses that don't expose a native lineage table, extract lineage by parsing query
+history SQL for `CREATE TABLE AS SELECT`, `INSERT INTO ... SELECT`, and `MERGE INTO` patterns.
+
+Simplified example regex:
+```python
+import re
+
+CTAS_PATTERN = re.compile(
+    r"CREATE\s+(?:OR\s+REPLACE\s+)?TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?(\S+)\s+AS\s+SELECT",
+    re.IGNORECASE,
+)
+INSERT_PATTERN = re.compile(
+    r"INSERT\s+(?:OVERWRITE\s+)?(?:INTO\s+)?(\S+).*?FROM\s+(\S+)",
+    re.IGNORECASE | re.DOTALL,
+)
+```
+
+For Snowflake, BigQuery, and Redshift the query history tables provide this SQL.
+For Databricks, use `system.access.table_lineage` directly (no parsing needed).
+For Hive, parse the HiveServer2 log file.
+
+## Output manifest (include invocation_id)
+
+```python
+manifest = {
+    "resource_uuid": resource_uuid,
+    "invocation_id": service.extract_invocation_id(result),   # ← save this
+    "collected_at": datetime.now(tz=timezone.utc).isoformat(),
+    "edges": [
+        {
+            "destination": {"database": e.destination.database, "table": e.destination.table},
+            "sources": [{"database": s.database, "table": s.table} for s in e.sources],
+        }
+        for e in events
+    ],
+}
+with open("lineage_output.json", "w") as f:
+    json.dump(manifest, f, indent=2)
+```
+
+## How push lineage is distinguished from query-derived lineage
+
+Push-ingested lineage nodes and edges carry `origin = push_ingest` in Neo4j and
+`origin_type = DIRECT_LINEAGE` in the normalized lineage model. This prevents the lineage
+DAG from overwriting them with query-log-derived edges and gives MC a clear audit trail.
+
+## Neo4j node expiry
+
+Push-ingested **table lineage** nodes and edges are written with `expire_at = 9999-12-31`
+(never expire). This is handled internally by PushLineageProcessor — you do not need to set
+this manually when using `send_lineage()`.
+
+Push-ingested **column lineage** expires after **10 days**, same as pulled column lineage.
+
+For custom nodes created via GraphQL mutations, you **do** need to set
+`expireAt: "9999-12-31"` explicitly — see `references/custom-lineage.md`.

package/bundled-skills/monte-carlo-push-ingestion/references/push-metadata.md

@@ -0,0 +1,158 @@
+# Pushing Table Metadata
+
+## Overview
+
+Metadata push sends three types of signals per table:
+- **Schema** — column names and types
+- **Volume** — row count and byte count
+- **Freshness** — last update timestamp
+
+All three travel together in a single `RelationalAsset` object via `POST /ingest/v1/metadata`.
+
+**Expiration**: Pushed table metadata **does not expire**. Once pushed, it remains in Monte
+Carlo until explicitly deleted via `deletePushIngestedTables`.
+
+**Batching**: For large numbers of tables, split assets into batches. The compressed request
+body must not exceed **1MB** (Kinesis limit).
+
+## pycarlo models
+
+```python
+from pycarlo.features.ingestion import (
+    IngestionService,
+    RelationalAsset,
+    AssetMetadata,
+    AssetField,
+    AssetVolume,
+    AssetFreshness,
+)
+```
+
+## Minimal example
+
+```python
+asset = RelationalAsset(
+    type="TABLE",  # ONLY "TABLE" or "VIEW" — normalize warehouse-native values
+    metadata=AssetMetadata(
+        name="orders",
+        database="analytics",
+        schema="public",
+        description="Order transactions",
+    ),
+    fields=[
+        AssetField(name="order_id", type="INTEGER"),
+        AssetField(name="amount",   type="DECIMAL"),
+        AssetField(name="created_at", type="TIMESTAMP"),
+    ],
+    volume=AssetVolume(
+        row_count=1_500_000,
+        byte_count=250_000_000,
+    ),
+    freshness=AssetFreshness(
+        last_update_time="2024-03-01T12:00:00Z",  # ISO 8601 string, NOT a datetime object
+    ),
+)
+
+result = service.send_metadata(
+    resource_uuid="<your-resource-uuid>",
+    resource_type="data-lake",   # see note below on resource_type
+    events=[asset],
+)
+invocation_id = service.extract_invocation_id(result)
+print("invocation_id:", invocation_id)   # save this!
+```
+
+## resource_type
+
+The `resource_type` value must match the type of the MC resource (warehouse connection) you
+are pushing to. Use the same string that appears in the MC UI or the `connectionType` field
+from `getUser { account { warehouses { connectionType } } }`.
+
+Common values:
+- `"data-lake"` — Hive, EMR, Glue, generic data lake connections
+- `"snowflake"` — Snowflake
+- `"bigquery"` — BigQuery
+- `"databricks"` — Databricks Unity Catalog
+- `"redshift"` — Redshift
+
+## Asset type
+
+The `type` parameter on `RelationalAsset` must be one of two values (uppercase):
+- `"TABLE"` — tables, external tables, dynamic tables, materialized views, etc.
+- `"VIEW"` — views, secure views
+
+**Important**: Warehouse-native type values like `"BASE TABLE"` (Snowflake), `"MANAGED"` /
+`"EXTERNAL"` (Databricks), or `"MATERIALIZED_VIEW"` (BigQuery) are **NOT accepted** by the
+MC API and will cause a 400 error. Always normalize to `"TABLE"` or `"VIEW"` before pushing.
+
+## Field types
+
+Normalize to SQL-standard uppercase strings. Monte Carlo accepts any string but canonical
+values like `INTEGER`, `BIGINT`, `VARCHAR`, `FLOAT`, `BOOLEAN`, `TIMESTAMP`, `DATE`,
+`DECIMAL`, `ARRAY`, `STRUCT` work best with downstream features.
+
+## Volume and freshness are optional
+
+If your warehouse doesn't expose row counts or last-modified timestamps, omit `volume`
+and/or `freshness` — schema-only metadata is valid.
+
+If you send `freshness`, each push must carry a **changed** `last_update_time` to count as
+a new data point for the anomaly detector (repeated identical timestamps don't advance the
+training clock).
+
+## Freshness + volume only mode (skip schema)
+
+For periodic pushes (e.g. hourly cron), you often don't need to re-collect the full schema
+on every run — field definitions rarely change. Collection scripts can support a
+`--only-freshness-and-volume` flag that skips the `COLUMNS` / `INFORMATION_SCHEMA` query
+and omits `fields` from the manifest. This is significantly faster on warehouses with many
+tables. Use the full collection (with fields) on the first push and on a daily schedule,
+and the freshness+volume only mode for hourly pushes in between. See the
+[BigQuery Iceberg example](https://github.com/monte-carlo-data/mcd-public-resources/tree/main/examples/push-ingestion/bigquery/push-iceberg-tables)
+for a working implementation of this pattern.
+
+## Batch multiple tables
+
+`events` accepts a list. Push all tables in a single call or in batches:
+
+```python
+result = service.send_metadata(
+    resource_uuid=resource_uuid,
+    resource_type="data-lake",
+    events=[asset1, asset2, asset3, ...],
+)
+```
+
+## Output manifest (include invocation_id)
+
+Always write a local manifest so you can trace issues later:
+
+```python
+import json
+from datetime import datetime, timezone
+
+manifest = {
+    "resource_uuid": resource_uuid,
+    "invocation_id": service.extract_invocation_id(result),   # ← critical for debugging
+    "collected_at": datetime.now(tz=timezone.utc).isoformat(),
+    "assets": [
+        {
+            "database": a.metadata.database,
+            "schema": a.metadata.schema,
+            "table": a.metadata.name,
+            "row_count": a.volume.row_count if a.volume else None,
+            "fields": [{"name": f.name, "type": f.type} for f in a.fields],
+        }
+        for a in assets
+    ],
+}
+with open("metadata_output.json", "w") as f:
+    json.dump(manifest, f, indent=2)
+```
+
+## Push frequency for anomaly detection
+
+To keep volume and freshness anomaly detectors active:
+- Push **at most once per hour** (pushing more frequently produces unpredictable behavior)
+- Push **consistently** — gaps longer than a few days will deactivate detectors
+- See `references/anomaly-detection.md` for minimum sample requirements