@revos/cli 0.1.4 → 0.2.1

package/dist/templates/AGENTS.md

@@ -11,12 +11,22 @@ This is a RevOS data engineering project for **<%=orgName%>** organization.
 
 ## Key Commands
 
-| Command               | Description                      |
-| --------------------- | -------------------------------- |
-| `revos auth login`    | Authenticate with RevOS          |
-| `revos overlays push` | Push Cube overlays to RevOS      |
-| `revos overlays pull` | Pull Cube overlays from RevOS    |
-| `revos overlays diff` | Compare local vs remote overlays |
-| `dbt run`             | Run dbt models                   |
-| `dbt test`            | Test dbt models                  |
-| `bq ls`               | List BigQuery datasets/tables    |
+| Command                          | Description                           |
+| -------------------------------- | ------------------------------------- |
+| `revos auth login`               | Authenticate with RevOS               |
+| `revos overlays push`            | Push Cube overlays to RevOS           |
+| `revos overlays pull`            | Pull Cube overlays from RevOS         |
+| `revos overlays diff`            | Compare local vs remote overlays      |
+| `revos integrations list`        | Open integrations page in RevOS UI    |
+| `revos integrations create`      | Open RevOS UI to add a data source    |
+| `revos integrations update <id>` | Open RevOS UI to edit an integration  |
+| `dbt run`                        | Run dbt models                        |
+| `dbt test`                       | Test dbt models                       |
+| `bq ls`                          | List BigQuery datasets/tables         |
+
+## Data Sources
+
+When the user wants to add or manage data sources (integrations):
+- To add a new data source: suggest `revos integrations create`
+- To view existing integrations: suggest `revos integrations list`
+- If a BigQuery dataset is empty or has no tables, mention that integrations can be configured with `revos integrations create`, or sample data can be loaded if available

package/dist/templates/skills/create-dbt-transformations/SKILL.md

@@ -5,7 +5,7 @@ description: Create new dbt transformations (bronze/silver/gold models) in the R
 
 # Create dbt Transformations
 
-Use this skill to generate SQL models, declare sources, update `schema.yml`, and validate models with `revos dbt run` / `revos dbt test`.
+Use this skill to generate SQL models, declare sources, update `schema.yml`, and validate models with `dbt run` / `dbt test`.
 
 For BigQuery exploration (listing datasets, inspecting raw tables, previewing rows, null rates), load the `explore-lakehouse` skill. If that skill is not installed, fall back to:
 
@@ -72,15 +72,13 @@ Always declare raw tables as sources before referencing them. Do not use bare fu
 
 ## Standard dbt Commands
 
-Always use the `revos` wrapper:
-
 ```bash
-revos dbt parse                               # validate syntax (no warehouse)
-revos dbt compile --select <model>            # resolve refs, produce compiled SQL
-revos dbt run --select <model>                # execute against warehouse
-revos dbt test --select <model>               # run tests
-revos dbt build --select <model>              # run + test
-revos dbt build --select path:models/<layer>  # entire layer
+dbt parse                               # validate syntax (no warehouse)
+dbt compile --select <model>            # resolve refs, produce compiled SQL
+dbt run --select <model>                # execute against warehouse
+dbt test --select <model>               # run tests
+dbt build --select <model>              # run + test
+dbt build --select path:models/<layer>  # entire layer
 ```
 
 ---
@@ -99,8 +97,8 @@ For each transformation (one at a time — do not batch):
 6. Generate `dbt/models/<layer>/<model_name>.sql`.
 7. Detect the primary key (Checkpoint 3 if ambiguous).
 8. Add model entry to `dbt/models/<layer>/schema.yml` with PK and FK tests. See [schema-conventions.md](references/schema-conventions.md).
-9. Run `revos dbt run --select <model_name>` and report result.
-10. Run `revos dbt test --select <model_name>` and report result.
+9. Run `dbt run --select <model_name>` and report result.
+10. Run `dbt test --select <model_name>` and report result.
 11. Summarize (see Final Response Format).
 
 For multiple transformations in one request: repeat steps 1–11 per model in order.
@@ -169,6 +167,16 @@ If none produce a clear answer → Checkpoint 3.
 
 A column is a FK candidate if it matches `<entity>_id` where `<entity>` ≠ model's own entity, is not part of the PK, and is not nullable by design. Add `not_null` test only (no `relationships` tests by default).
 
+## Timestamp Column Propagation (Gold Models)
+
+Every gold model **must** propagate at least one timestamp column so downstream Cube overlays can use SQL-based `refresh_key` (see `create-semantic-model` skill). Priority:
+
+1. `_airbyte_extracted_at` — present on all Airbyte sources; always propagate if available in upstream.
+2. `updated_at` / `modified_at` — CDC-friendly streams.
+3. `created_at` — insert-only fact tables.
+
+If the upstream source has none of these, document it in a SQL comment: `-- no timestamp column available from source`.
+
 ## SQL File Generation
 
 See [sql-templates.md](references/sql-templates.md) for:
@@ -206,8 +214,8 @@ Tests:
 - not_null on <fk>: added
 
 Validation:
-- revos dbt run:  passed | failed
-- revos dbt test: passed | failed
+- dbt run:  passed | failed
+- dbt test: passed | failed
 
 Physical table after run:
 `<resolved_dataset>.<model_name>`

package/dist/templates/skills/create-dbt-transformations/references/edge-cases.md

@@ -34,7 +34,7 @@ Declare it as a source in `dbt/models/bronze/schema.yml` first (see [schema-conv
 
 1. Show the error verbatim — do not paraphrase warehouse errors.
 2. Offer to fix the SQL based on the error message.
-3. Do not proceed to `revos dbt test` until run succeeds.
+3. Do not proceed to `dbt test` until run succeeds.
 
 ## test fails
 

package/dist/templates/skills/create-semantic-model/SKILL.md

@@ -15,8 +15,7 @@ project layout, finding gold models, resolving dbt model names to BigQuery table
 creating bridge models from JSON arrays, and dbt validation commands.
 
 If `create-dbt-transformations` is not installed: discover gold models directly via
-`find dbt/models/gold -name "*.sql"`, run dbt commands without the `revos` wrapper
-(`dbt parse`, `dbt run`, `dbt test`), and skip bridge model delegation.
+`find dbt/models/gold -name "*.sql"` and skip bridge model delegation.
 Warn the user: "The `create-dbt-transformations` skill is not installed — bridge model creation and dbt validation are limited."
 
 If BigQuery exploration is needed (listing tables, inspecting schemas, previewing rows),
@@ -95,7 +94,7 @@ Follow these phases in order. Do not skip ahead.
 
 1. Discover gold models via `find dbt/models/gold -name "*.sql"`.
 2. If none exist, stop and tell the user to create gold models first via `create-dbt-transformations`.
-3. Inspect 1-2 existing overlays in `semantic/` to detect conventions (map-style vs list-style, `extends:`, `public:`, `refresh_key` style). Apply detected conventions to new overlays.
+3. Inspect 1-2 existing overlays in `semantic/` to detect conventions (`extends:`, `public:`, `refresh_key` style). Apply detected conventions to new overlays. Always use flat single-cube YAML (never `cubes:` or `views:` root).
 4. If the user named a specific model, find it. If not found, stop.
 5. Otherwise list all discovered gold models and ask which should participate (Checkpoint 1).
 6. Keep the full discovered list available for connector search in Phase 3.
@@ -238,13 +237,14 @@ Create Cube.dev YAML files in `semantic/`. Follow the existing style detected in
 
 Key rules:
 
-1. File name = cube `name` (no `gold_` prefix) + `.yml`.
-2. `sql_table` uses fully qualified BigQuery reference with `gold_` prefix.
-3. Every confirmed relationship gets joins in both directions.
-4. Bridge/junction cubes use `public: false`.
-5. Every overlay includes `refresh_key`. Prefer `_airbyte_extracted_at`, fall back to other timestamps, then `every: 1 hour`.
-6. `refresh_key.sql` references the same table as `sql_table`.
-7. Tag unvalidated joins with `# UNVALIDATED: <reason>`.
+1. **One cube per file, flat YAML.** Each overlay file contains a single cube starting with `name:` at the root level. Never wrap with `cubes:` or `views:` at the root.
+2. File name = cube `name` (no `gold_` prefix) + `.yml`.
+3. `sql_table` uses fully qualified BigQuery reference with `gold_` prefix.
+4. Every confirmed relationship gets joins in both directions.
+5. Bridge/junction cubes use `public: false`.
+6. Every overlay **must** include a SQL-based `refresh_key`. Use `SELECT MAX(<timestamp_col>)` with columns in this priority: `_airbyte_extracted_at` (present on all Airbyte sources), `updated_at`/`modified_at` (CDC streams), `created_at` (insert-only facts). Only use `every: <interval>` as absolute last resort when **no timestamp column exists in the table** — add a YAML comment explaining why (e.g. `# no timestamp column available`).
+7. `refresh_key.sql` references the same table as `sql_table`.
+8. Tag unvalidated joins with `# UNVALIDATED: <reason>`.
 
 See [references/cube-examples.md](references/cube-examples.md) for canonical standard cube, bridge cube, join direction examples, and refresh key variants.
 
@@ -252,7 +252,7 @@ See [references/cube-examples.md](references/cube-examples.md) for canonical sta
 
 ## Phase 9: Validate Generated Files
 
-1. If `create-dbt-transformations` was invoked (bridge model), it already validated dbt models. Otherwise run `revos dbt parse`.
+1. If `create-dbt-transformations` was invoked (bridge model), it already validated dbt models. Otherwise run `dbt parse`.
 2. Verify physical tables exist in BigQuery: `bq show <dataset>.<table_name>`. If missing, document as pending.
 3. Verify generated overlays match conventions: flat YAML, correct naming, correct `sql_table`, all dimensions present, `refresh_key` included, joins in both directions.
 

package/dist/templates/skills/create-semantic-model/references/cube-examples.md

@@ -9,6 +9,7 @@
 - [Refresh Key Variants](#refresh-key-variants)
 - [Type Mapping](#type-mapping)
 - [Measure Suggestions](#measure-suggestions)
+- [Common Mistakes](#common-mistakes)
 
 ---
 
@@ -215,24 +216,68 @@ joins:
 
 ## Refresh Key Variants
 
-Priority order:
+SQL-based refresh keys are **required**. They ensure caches invalidate only when data actually changes, instead of on a fixed timer.
+
+Priority — use the first available timestamp column:
 
 ```yaml
-# 1. Airbyte timestamp (preferred)
+# 1. Airbyte timestamp (preferred — present on all Airbyte sources)
 refresh_key:
   sql: "SELECT MAX(_airbyte_extracted_at) FROM `<dataset>.<gold_model>`"
 
-# 2. Other reliable timestamp
+# 2. CDC / update timestamp
 refresh_key:
   sql: "SELECT MAX(updated_at) FROM `<dataset>.<gold_model>`"
 
-# 3. Default fallback
+# 3. Insert-only fact table
 refresh_key:
-  every: 1 hour
+  sql: "SELECT MAX(created_at) FROM `<dataset>.<gold_model>`"
+
+# 4. Last resort — ONLY when the table has no timestamp column at all
+# Add a comment explaining why:
+refresh_key:
+  every: 1 hour  # no timestamp column available in this table
 ```
 
 `refresh_key.sql` must reference the same fully qualified table as the cube's `sql_table`.
 
+### Common Mistakes
+
+```yaml
+# BAD — fixed cadence, ignores actual data changes.
+# This forces cache rebuild every hour even when data hasn't changed.
+refresh_key:
+  every: 1 hour
+
+# GOOD — only invalidates when new rows arrive
+refresh_key:
+  sql: "SELECT MAX(_airbyte_extracted_at) FROM `<dataset>.<gold_model>`"
+```
+
+```yaml
+# BAD — omitting refresh_key entirely (cube uses default, which may be too aggressive)
+name: my_cube
+sql_table: ...
+
+# GOOD — always include an explicit refresh_key
+name: my_cube
+sql_table: ...
+refresh_key:
+  sql: "SELECT MAX(_airbyte_extracted_at) FROM `<dataset>.<gold_model>`"
+```
+
+```yaml
+# BAD — refresh_key.sql references a different table than sql_table
+sql_table: "`project.dataset.gold_orders`"
+refresh_key:
+  sql: "SELECT MAX(_airbyte_extracted_at) FROM `project.dataset.gold_customers`"
+
+# GOOD — same table in both
+sql_table: "`project.dataset.gold_orders`"
+refresh_key:
+  sql: "SELECT MAX(_airbyte_extracted_at) FROM `project.dataset.gold_orders`"
+```
+
 ---
 
 ## Type Mapping
@@ -265,3 +310,36 @@ updated_at  -> last_updated_at (max)
 ```
 
 `count_distinct` on FK columns: define inside the cube that owns the FK, not the parent cube. Joins can produce row fan-out that distorts distinct counts.
+
+---
+
+## Common Mistakes
+
+### Wrapping with `cubes:` or `views:` at the root
+
+RevOS expects one cube per file with flat YAML — `name:` at the root level. The Cube.dev docs show a multi-cube `cubes:` list format, but RevOS does not use it.
+
+```yaml
+# BAD — multi-cube list format
+cubes:
+  - name: hubspot_companies
+    sql_table: "`dataset.gold_hubspot_companies`"
+    dimensions:
+      id:
+        sql: "${CUBE}.id"
+        type: string
+        primary_key: true
+```
+
+```yaml
+# GOOD — flat single-cube format
+name: hubspot_companies
+sql_table: "`dataset.gold_hubspot_companies`"
+dimensions:
+  id:
+    sql: "${CUBE}.id"
+    type: string
+    primary_key: true
+```
+
+Same applies to `views:` — never use it as a root key.

package/dist/templates/skills/explore-lakehouse/SKILL.md

@@ -85,9 +85,13 @@ FROM \`$GOOGLE_CLOUD_PROJECT.$REVOS_BQ_DATASET.<table>\`
 ### "What's in my database?" / general overview
 
 1. List tables in the org's dataset: `bq ls $REVOS_BQ_DATASET`
-2. Infer the data source and domain from table name prefixes (e.g. `salesforce_*`, `stripe_*`, `hubspot_*`)
-3. Group tables by source/domain
-4. Return: sources found, table count per source, table types (TABLE/VIEW), one-line description per group
+2. If the dataset is empty (no tables), tell the user:
+   - They can add data sources by running `revos integrations create` to open the RevOS UI
+   - They can view existing integrations with `revos integrations list`
+   - Stop here — no further exploration is possible without data
+3. Infer the data source and domain from table name prefixes (e.g. `salesforce_*`, `stripe_*`, `hubspot_*`)
+4. Group tables by source/domain
+5. Return: sources found, table count per source, table types (TABLE/VIEW), one-line description per group
 
 ### "What layer is this data?" / bronze–silver–gold assessment
 

package/dist/templates/skills/load-sample-data/SKILL.md

@@ -0,0 +1,119 @@
+---
+name: load-sample-data
+description: >
+  Populate a BigQuery dataset with sample data from public datasets using bq cp.
+  Use when asked to: load sample data, populate the lakehouse, add demo data, seed the dataset,
+  get started with sample tables, or when the user needs data to explore.
+---
+
+# Load Sample Data
+
+Copy sample tables from `bigquery-public-data` into the user's dataset using `bq cp`. All copied tables are prefixed with `sample_` so they can be easily identified and cleaned up later.
+
+## Environment
+
+Verify env vars before running any command. If either is empty, ask the user to set it.
+
+```bash
+echo "Project: $GOOGLE_CLOUD_PROJECT"
+echo "Dataset: $REVOS_BQ_DATASET"
+```
+
+- `$GOOGLE_CLOUD_PROJECT` — BQ project ID
+- `$REVOS_BQ_DATASET` — target dataset
+
+## Step 1: Check Existing Tables
+
+```bash
+bq ls $REVOS_BQ_DATASET
+```
+
+If tables exist, list them. They will be relevant in Step 3 for collision handling.
+
+## Step 2: Present Sample Dataset Catalog
+
+```text
+Available sample datasets:
+
+1. thelook_ecommerce (default) — B2C e-commerce data
+   Tables: sample_users, sample_orders, sample_order_items, sample_products, sample_events, sample_inventory_items, sample_distribution_centers
+   Rows: ~100K users, ~300K orders
+   Good for: customer analytics, purchase funnels, product performance
+
+2. google_analytics_sample — Web analytics session data
+   Tables: sample_ga_sessions (single table, one day snapshot)
+   Rows: ~900K sessions
+   Good for: web traffic analysis, user behavior, channel attribution
+
+3. austin_bikeshare — Bikeshare trip and station data
+   Tables: sample_bikeshare_trips, sample_bikeshare_stations
+   Rows: ~1.3M trips
+   Good for: geospatial analysis, demand forecasting, utilization metrics
+```
+
+## Step 3: Copy Tables
+
+If any table names from the chosen sample dataset collide with existing tables (from Step 1), ask the user whether to **overwrite** or **skip** each collision before copying. Skip collisions the user chose not to overwrite.
+
+### thelook_ecommerce
+
+```bash
+for table in users orders order_items products events inventory_items distribution_centers; do
+  echo "Copying sample_$table..."
+  bq cp -f bigquery-public-data:thelook_ecommerce.$table \
+    $GOOGLE_CLOUD_PROJECT:$REVOS_BQ_DATASET.sample_$table
+done
+```
+
+### google_analytics_sample
+
+The `ga_sessions` table is date-sharded. Copy a representative day:
+
+```bash
+echo "Copying sample_ga_sessions..."
+bq cp -f bigquery-public-data:google_analytics_sample.ga_sessions_20170801 \
+  $GOOGLE_CLOUD_PROJECT:$REVOS_BQ_DATASET.sample_ga_sessions
+```
+
+### austin_bikeshare
+
+```bash
+for table in bikeshare_trips bikeshare_stations; do
+  echo "Copying sample_$table..."
+  bq cp -f bigquery-public-data:austin_bikeshare.$table \
+    $GOOGLE_CLOUD_PROJECT:$REVOS_BQ_DATASET.sample_$table
+done
+```
+
+## Step 4: Verify
+
+```bash
+for table in <copied_tables>; do
+  echo -n "$table: "
+  bq query --nouse_legacy_sql --format=csv \
+    "SELECT COUNT(*) FROM \`$GOOGLE_CLOUD_PROJECT.$REVOS_BQ_DATASET.$table\`" 2>/dev/null | tail -1
+done
+```
+
+## Final Response
+
+```text
+Sample data loaded into $REVOS_BQ_DATASET.
+
+Source: bigquery-public-data:<dataset_name>
+Tables copied:
+- <table_1>: <row_count> rows
+- <table_2>: <row_count> rows
+
+Next steps:
+- Run "explore lakehouse" to inspect the data
+- Run "create dbt transformations" to build bronze/silver/gold models
+- Run "create semantic model" to generate Cube overlays
+```
+
+## Rules
+
+- Use `bq cp -f` (not `CREATE TABLE AS SELECT`) — faster, no query costs, preserves schema. The `-f` flag is required to avoid cross-region confirmation prompts that block non-interactive shells.
+- Show progress for each table being copied.
+- Report any failures clearly with the `bq` error message.
+- Always prefix destination tables with `sample_` — this allows easy identification and cleanup of sample data.

package/package.json

@@ -1,7 +1,8 @@
 {
   "name": "@revos/cli",
-  "version": "0.1.4",
+  "version": "0.2.1",
   "description": "RevOS CLI for managing RevOS platform resources",
+  "license": "MIT",
   "type": "module",
   "main": "dist/index.js",
   "types": "dist/index.d.ts",
@@ -31,12 +32,16 @@
       },
       "overlays": {
         "description": "Manage Cube overlays"
+      },
+      "integrations": {
+        "description": "Manage integrations (opens RevOS UI)"
       }
     }
   },
   "files": [
     "dist",
-    "bin"
+    "bin",
+    "LICENSE"
   ],
   "dependencies": {
     "@inquirer/search": "^4.1.7",
@@ -57,7 +62,7 @@
     "jest": "^29.7.0",
     "tsdown": "^0.21.9",
     "typescript": "^5.7.2",
-    "@revosai/config-eslint": "0.0.0"
+    "@revos/config-eslint": "0.0.0"
   },
   "publishConfig": {
     "registry": "https://registry.npmjs.org",