datamasque-cli 1.2.0__tar.gz → 1.3.0__tar.gz

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## v1.3.0
+
+### Added
+- `dm system ai-engine show` and `dm system ai-engine set <URL>` — view and
+  configure the AI Engine URL.
+
 ## v1.2.0
 
 ### Added

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datamasque-cli
-Version: 1.2.0
+Version: 1.3.0
 Summary: Official command-line interface for the DataMasque data-masking platform.
 Project-URL: Homepage, https://datamasque.com/
 Project-URL: Repository, https://github.com/datamasque/datamasque-cli
@@ -200,7 +200,7 @@ dm libraries usage <name>                         # Show rulesets using it
 ### In-flight masking
 
 The IFM service runs alongside the admin server,
-reached at `<DataMasque URL>/ifm` via the standard nginx topology.
+reached at `<DataMasque URL>/ifm`.
 
 ```console
 dm ifm list                                            # List ruleset plans
@@ -278,6 +278,8 @@ dm system upload-licence ./licence.lic          # Upload a licence file
 dm system logs -o logs.tar.gz                   # Download application logs
 dm system admin-install --email admin@co.com    # Initial admin setup
 dm system set-locality AU                       # Set system locality
+dm system ai-engine show                        # Show the configured AI Engine URL
+dm system ai-engine set <URL>                   # Point DataMasque at an AI Engine
 ```
 
 ## JSON output

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/README.md

@@ -170,7 +170,7 @@ dm libraries usage <name>                         # Show rulesets using it
 ### In-flight masking
 
 The IFM service runs alongside the admin server,
-reached at `<DataMasque URL>/ifm` via the standard nginx topology.
+reached at `<DataMasque URL>/ifm`.
 
 ```console
 dm ifm list                                            # List ruleset plans
@@ -248,6 +248,8 @@ dm system upload-licence ./licence.lic          # Upload a licence file
 dm system logs -o logs.tar.gz                   # Download application logs
 dm system admin-install --email admin@co.com    # Initial admin setup
 dm system set-locality AU                       # Set system locality
+dm system ai-engine show                        # Show the configured AI Engine URL
+dm system ai-engine set <URL>                   # Point DataMasque at an AI Engine
 ```
 
 ## JSON output

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/claude-skills/datamasque-cli/skills/datamasque-cli/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: datamasque-cli
-description: Use when the user wants to interact with a DataMasque instance — start masking runs, check run status, list connections or rulesets, manage seeds, manage ruleset libraries, check system health, or any task involving the DataMasque API. Triggers on "mask the data", "start a run", "check the run", "list connections", "list rulesets", "upload a seed", "check DataMasque health", "dm status", "ruleset library", or any request to operate DataMasque programmatically.
+description: Use when the user wants to interact with a DataMasque instance — start masking runs, check run status, list connections or rulesets, manage seeds, manage ruleset libraries, check system health, configure the AI Engine, or any task involving the DataMasque API. Triggers on "mask the data", "start a run", "check the run", "list connections", "list rulesets", "upload a seed", "check DataMasque health", "dm status", "ruleset library", "configure the AI Engine", "set the AI Engine URL", or any request to operate DataMasque programmatically.
 argument-hint: e.g. "start a run with docx_masking on var_input_docx"
 user-invocable: true
 ---

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/claude-skills/ruleset-builder/.claude-plugin/plugin.json

@@ -1,6 +1,6 @@
 {
   "name": "ruleset-builder",
-  "version": "1.0.0",
+  "version": "1.1.0",
   "description": "Convert auto-generated DataMasque rulesets into production-ready form. Validate and iterate.",
   "author": { "name": "DataMasque Ltd" },
   "repository": "https://github.com/datamasque/datamasque-cli",

datamasque_cli-1.3.0/claude-skills/ruleset-builder/skills/ruleset-builder/SKILL.md

@@ -0,0 +1,176 @@
+---
+name: ruleset-builder
+description: Use when the user wants to turn auto-generated DataMasque rulesets into production-ready ones — extract a `ruleset_library`, add `hash_columns`, refine a ruleset, or clean up generated YAML. Triggers on "ruleset builder", "build ruleset", "refine ruleset", "add hash columns", "add ruleset library", "production ruleset", "clean up ruleset".
+argument-hint: e.g. "build a ruleset from these generated files"
+user-invocable: true
+---
+
+# Ruleset Builder
+
+Transform auto-generated DataMasque rulesets into production-ready rulesets with three improvements:
+1. **`ruleset_library` references** — `$ref` links replacing every repeated inline mask
+2. **`hash_columns`** — on every applicable `mask_table` task for deterministic consistency
+3. **Clean structure** — `skip_defaults`, no doc blocks, validated
+
+FK cascade is automatic: mask the parent PK with `imitate_unique` (or `imitate_uuid` / `imitate_nz_ird`) and the engine replicates the rule onto every FK column referencing it. **Do NOT add explicit rules for FK columns.** Avoid `from_unique_imitate` and `mask_unique_key` (both deprecated). Never skip IDs.
+
+5-step process (1–5). Use `TaskCreate` to track all 5; report after each step before proceeding. The prompt must include business domain and application type — ask if missing.
+
+---
+
+## Step 1: Report versions
+
+Report the Ruleset Builder version (from `plugin.json`) and `dm version` so the operator can correlate output with releases.
+
+---
+
+## Step 2: Read reference docs
+
+Canonical mask reference:
+<https://portal.datamasque.com/portal/documentation/latest/masking-functions-overview.html>
+
+Read all of these before any other work:
+```
+${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/fk-cascade.md
+${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/mask-definitions-guide.md
+${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/hash-columns-guide.md
+${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/ruleset-yaml-reference.md
+```
+
+---
+
+## Step 3: Extract ruleset_library
+
+Write a Python script using `ruamel.yaml` (`uv pip install ruamel.yaml`).
+
+Process the input YAML. For each `mask_table` task, replace every inline mask with a `$ref` to a rule in `ruleset_library.yaml`. Build the library progressively — read its current state at the start of each iteration, create it if absent.
+
+The library `masks` section structure:
+```yaml
+version: "1.0"
+masks:
+  rule_name:
+    type: rule_type
+    ...params
+```
+
+### Classification rules (apply in order)
+
+**1. ID columns** — any column ending in `_ID`, `_NO`, `_NR`, `_NBR` is an entity identifier.
+- **FK side: drop the rule entirely.** If an ID column is a foreign key (the table's `Foreign Keys` metadata in the discovery CSV has an entry for it), do NOT emit a rule for it. The engine cascades automatically from the parent PK rule. See `fk-cascade.md`.
+- **PK side: use `imitate_unique` with `seed:`.** Strip adjective/verb prefixes before the noun: `PREVIOUS_`, `OLD_`, `TRANSFERRED_`, `PRIOR_`, `CURR_`, `NEW_`, `NEXT_`, `ALT_`, `PARENT_`, `CHILD_`, `SOURCE_`, `TARGET_`, `ORIG_`, `PENDING_`, `ARCHIVED_`, `DELETED_`. Extract the core entity (`PREVIOUS_INVOICE_ID` → `invoice`).
+- Library entry name: `{entity}_id`. Reference it as `$ref: "Global/RuleLib#masks/{entity}_id"`.
+- Library entry body: `type: imitate_unique`, `seed: "{entity}"`. The `seed` is optional but recommended: it namespaces by entity so unrelated IDs don't collide (e.g. `customer.id=42` doesn't mask to the same value as `product.id=42`). Doesn't affect FK cascade.
+- This overrides whatever mask was originally generated (even `from_random_number`).
+
+**2. Named patterns** — detect by mask structure:
+
+| Pattern         | Detection                                                                                                                                           | Library rule           |
+|-----------------|-----------------------------------------------------------------------------------------------------------------------------------------------------|------------------------|
+| Email           | `chain(concat(concat(firstName+lastName, glue='.')+email_suffix)+transform_case(lower))`                                                            | `email_address`        |
+| Full name       | `chain(concat(firstName+lastName, glue=' ')+take_substring)` OR plain `concat(firstName+lastName, glue=' ')` — column not containing USERNAME/LOGIN | `full_name`            |
+| Username        | Same mask as full_name but column name contains USERNAME, USER_NAME, LOGIN, LOGON                                                                   | `username`             |
+| First name only | `from_file` with firstNames seed                                                                                                                    | `name_first`           |
+| Last name only  | `from_file` with lastNames seed                                                                                                                     | `name_last`            |
+| DOB             | Column name contains DOB/BIRTH/DATE_OF_BIRTH — use `retain_age` regardless of original type                                                         | `dob`                  |
+| Company         | `chain(from_file(companies)+take_substring)`                                                                                                        | `company_name`         |
+| Country name    | `from_file(country_codes, seed_column=name)`                                                                                                        | `country_name`         |
+| Country alpha-2 | `from_file(country_codes, seed_column=alpha_2)`                                                                                                     | `country_code_2`       |
+| Country alpha-3 | `from_file(country_codes, seed_column=alpha_3)`                                                                                                     | `country_code_3`       |
+| Phone/fax       | `imitate` on column name containing PHONE, TEL, FAX, MOBILE, CELL                                                                                   | `phone`                |
+| Address line 1  | `from_file(addresses, seed_column=street_address)` on LINE_1/ADDRESS_LINE_1 columns                                                                 | `address_line1`        |
+| Address line N  | Same for LINE_2, LINE_3 etc.                                                                                                                        | `address_lineN`        |
+| Address full    | `from_file(addresses, seed_column=street_address)` on non-line-numbered columns                                                                     | `address_full`         |
+| Address expr    | `concat(address+city+state+postcode, glue=', ')`                                                                                                    | `network_address_expr` |
+| City            | `from_file(addresses, seed_column=city)`                                                                                                            | `city`                 |
+| Postcode        | `from_file(addresses, seed_column=postcode)`                                                                                                        | `post_code`            |
+| Suburb          | `from_file(addresses, seed_column=suburb)`                                                                                                          | `suburb`               |
+| Occupation      | `from_file(occupations)`                                                                                                                            | `occupation`           |
+
+**3. Remaining** — group by column name concept. Where column names share a root (e.g., `RESULT3_VALUE`, `RESULT5_VALUE` → `result_value`; `GENERAL_2`, `GENERAL_6` → `general`), use one shared rule. Strip adjective prefixes. Use first occurrence's parameters.
+
+- `imitate_unique` (non-ID cols) → `{col_group}: type: imitate_unique, seed: "{col_group}"` (seed recommended for namespacing; see ID columns section).
+- `from_random_date` → `{col_group}: type: from_random_date, min/max from first occurrence`
+- `from_random_number` → `{col_group}: type: from_random_number, min/max from first occurrence`
+- String catch-all → `{col_group}: type: imitate_unique, seed: "{col_group}"` (use `imitate` only for types `imitate_unique` can't handle, e.g. datetime, bool).
+- Complex chains → keep structure, group by column name
+
+### Output format
+
+`Global/RuleLib` below is a placeholder for `<namespace>/<library_name>` — substitute the operator's real values, and create the library with `dm libraries create` before running the ruleset.
+
+```yaml
+version: '1.0'
+skip_defaults:
+  - ''
+  - null
+imports:
+  - Global/RuleLib
+
+tasks:
+  - type: mask_table
+    table: '"SCHEMA"."TABLE"'
+    key: '"ROWID"'
+    rules:
+      - column: '"FIRST_NAME"'
+        masks:
+          - $ref: "Global/RuleLib#masks/name_first"
+```
+
+Do NOT write a custom YAML serializer. Use `ruamel.yaml` round-trip dumper. Use `DoubleQuotedScalarString` for `$ref` values.
+
+**Report:** "Step 3 done — extracted N rule library definitions: [list each name and usage count]."
+
+---
+
+## Step 4: Add hash_columns
+
+Write a Python script that:
+
+**Parse the discovery CSV** (comma-separated):
+`Selected`, `Table schema`, `Table name`, `Column name`, `Data Type`, `Constraint`, `Foreign Keys`, `Max Length`, `Numeric Precision`, `Numeric Scale`, `Reason for flag`, `Flagged by`, `Data classifications`
+
+Build a lookup of `(schema, table)` → columns with constraint and FK metadata:
+- `Constraint` patterns: `Primary(COL)`, `Unique(COL)`, `Foreign(COL)`
+- `Foreign Keys` JSON: `["FK_NAME", "SCHEMA.TABLE.COLUMN"]` — index 1 gives the referenced table
+
+**For each `mask_table` task:**
+
+1. **Pick hash column** using this priority:
+   - **Parent-entity FK first**: find FK columns where the referenced table is the parent of the current table — i.e., the current table name *starts with* the referenced table name (e.g., `ACCOUNT_HISTORY` starts with `ACCOUNT` → use `ACCOUNT_ID`). This avoids choosing lookup-table FKs (e.g., don't choose `ACCOUNT_TYPE_ID` in `ACCOUNT` just because it has a FK).
+   - **PK fallback**: if no parent-entity FK found, use the Primary Key column (never `ROWID`)
+   - **Archive table fallback**: if no PK in the CSV (archive tables `_A`, `_A_R`, `_R` often lack explicit keys), strip the suffix and look up the base table recursively
+   - **Composite PKs**: prefer `*_ID` or `*_NO` columns; deduplicate derivatives (`ACCOUNT_ID` + `PREVIOUS_ACCOUNT_ID` → keep `ACCOUNT_ID`)
+   - **Skip** if no suitable column found
+
+2. Insert `hash_columns: ["COLUMN_NAME"]` after the `key:` field
+
+3. Verify all rules in output are `$ref` — fix any remaining inline masks
+
+4. Write to output file
+
+**Report:** "Step 4 done — added hash_columns to N tables, skipped M (all-unique), skipped K (no suitable key). Top hash columns: [column → count]."
+
+---
+
+## Step 5: Validate and clean up
+
+Remove any comment lines containing `ROWID`.
+
+Run `dm rulesets validate --file <output_file> --type database`
+(use `file` for file-masking rulesets).
+
+Fix any errors and re-validate until passing.
+
+---
+
+## Summary
+
+| Metric                     | Value          |
+|----------------------------|----------------|
+| Total tables               | N              |
+| Mask definitions extracted | N (list names) |
+| Tables with hash_columns   | N              |
+| Tables skipped (no key)    | N              |
+| Validation                 | passed/failed  |
+| Output file                | path           |

datamasque_cli-1.3.0/claude-skills/ruleset-builder/skills/ruleset-builder/references/fk-cascade.md

@@ -0,0 +1,109 @@
+# FK Cascade Invariant
+
+The most important rule when refining a DataMasque ruleset that spans
+related tables. Get this wrong and you either leak identity (by skipping
+IDs entirely) or break the engine (by adding rules for FK columns).
+
+## The rule
+
+**Mask only the parent PK column. The engine cascades the same masked value
+to every FK column referencing it.**
+
+Three masks support this cascade:
+
+- `imitate_unique` — recommended for new work.
+- `imitate_uuid` — for UUID-shaped IDs.
+- `imitate_nz_ird` — for NZ IRD numbers.
+
+(`from_unique_imitate` and `mask_unique_key` are deprecated; do not emit.)
+
+When `mask_table` runs and a rule on a referenced column uses one of these
+masks, the engine:
+
+1. Discovers child tables with FKs referencing this column.
+2. Auto-replicates the parent's rule onto every FK column.
+3. Same mask config → same masked output → joins survive.
+
+This is documented at
+<https://portal.datamasque.com/portal/documentation/latest/unique-masks.html>:
+
+> "You can apply an `imitate_unique` mask to a primary key column or a
+> column that is used as a foreign key in another table. References will be
+> updated automatically. Composite primary keys are supported."
+
+## Worked example
+
+Schema:
+- `customers.id` (PK), `customers.email`
+- `orders.id` (PK), `orders.customer_id` (FK → `customers.id`), `orders.tracking_number`
+
+Correct ruleset:
+
+```yaml
+- type: mask_table
+  table: customers
+  key: id
+  rules:
+    - column: id
+      masks:
+        - type: imitate_unique
+          seed: customer
+    - column: email
+      masks:
+        - type: from_file
+          seed_file: DataMasque_emails.csv
+          seed_column: email
+
+- type: mask_table
+  table: orders
+  key: id
+  rules:
+    # customer_id is intentionally absent — the engine replicates the
+    # `customers.id` rule onto it automatically. Adding it here would
+    # be rejected by the runtime FK check.
+    - column: tracking_number
+      masks:
+        - type: imitate_unique
+          seed: tracking
+```
+
+After the run, `orders.customer_id` holds the same masked values as
+`customers.id`, joins remain intact, and `tracking_number` is independently
+masked with its own seed.
+
+## Anti-patterns to refuse
+
+- **Adding explicit FK rules** ("I'll mask both PK and FK with shared
+  `$ref` so the cascade works"). The runtime rejects this by default with
+  the error:
+  *"To preserve referential integrity, the following foreign key columns
+  cannot be directly masked by this task."*
+  The engine will replicate the rule for you; adding your own conflicts.
+- **Skipping IDs to "preserve FK joins"**. Leaves identifiers in plain
+  sight. Mask the parent PK with `imitate_unique` — joins survive via
+  the auto-cascade.
+- **Inventing linking parameters** (`source_table`, `source_column`,
+  `parent_column`, `link_to`). None of these exist on any DataMasque mask.
+- **Inventing a hashing mask** (`hash_text`, `hash`, `link`, `match_id`).
+  None of these exist. `imitate_unique` is the deterministic mask.
+- **Using `from_unique_imitate` or `mask_unique_key`**. Both deprecated.
+  `imitate_unique` replaces both.
+
+## Cross-run consistency requires `run_secret`
+
+Within a single run, `imitate_unique` is deterministic via a per-run
+`insecure_seed`. Across runs, the cascade only holds if the run is
+invoked with a `run_secret`. Without it, the same input maps to a
+different masked value next run. If cross-run consistency matters, flag
+this in the final summary.
+
+## Self-check before finishing
+
+For each FK relationship in the schema:
+
+1. Is the parent PK masked with `imitate_unique`, `imitate_uuid`, or
+   `imitate_nz_ird`?
+2. Is the FK column **absent** from your output (no explicit rule)?
+3. Are `from_unique_imitate` and `mask_unique_key` absent from your output?
+
+If any answer is "no", fix it before validation.

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/claude-skills/ruleset-builder/skills/ruleset-builder/references/hash-columns-guide.md

@@ -71,14 +71,14 @@ hash_columns:
 
 Every table belongs to a domain entity. Find the column that identifies that entity:
 
-| Domain | Typical hash column | Examples |
-|--------|-------------------|----------|
-| Customer | `cust_id`, `customer_id`, `client_id` | CUST_MASTER, CUST_ADDRESS |
-| Account | `acc_id`, `account_id`, `account_no` | DEP_ACCOUNT, DEP_EMAIL_ALERT |
-| Card | `card_id`, `card_no` | CARD_MASTER, CARD_INSURANCE |
-| Loan | `loan_id`, `loan_no` | LOAN_COLLATERAL, LOAN_GUARANTOR |
-| Employee | `emp_id`, `emp_no`, `employee_id` | COM_EMPLOYEE, COM_EMP_ROLE |
-| Transaction | `tx_id`, `trf_id`, `fx_tx_id` | TRF_MASTER, FX_RECEIPT |
+| Domain      | Typical hash column                   | Examples                        |
+|-------------|---------------------------------------|---------------------------------|
+| Customer    | `cust_id`, `customer_id`, `client_id` | CUST_MASTER, CUST_ADDRESS       |
+| Account     | `acc_id`, `account_id`, `account_no`  | DEP_ACCOUNT, DEP_EMAIL_ALERT    |
+| Card        | `card_id`, `card_no`                  | CARD_MASTER, CARD_INSURANCE     |
+| Loan        | `loan_id`, `loan_no`                  | LOAN_COLLATERAL, LOAN_GUARANTOR |
+| Employee    | `emp_id`, `emp_no`, `employee_id`     | COM_EMPLOYEE, COM_EMP_ROLE      |
+| Transaction | `tx_id`, `trf_id`, `fx_tx_id`         | TRF_MASTER, FX_RECEIPT          |
 
 ### Step 2: Check foreign keys in the DDL
 

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/claude-skills/ruleset-builder/skills/ruleset-builder/references/mask-definitions-guide.md

@@ -175,14 +175,14 @@ tasks:
 
 Common seed files for `from_file` masks:
 
-| Category | Files |
-|----------|-------|
-| Names | `DataMasque_firstNames_mixed.csv`, `DataMasque_lastNames_v2.csv` |
+| Category  | Files                                                                                                 |
+|-----------|-------------------------------------------------------------------------------------------------------|
+| Names     | `DataMasque_firstNames_mixed.csv`, `DataMasque_lastNames_v2.csv`                                      |
 | Addresses | `DataMasque_US_addresses.csv`, `DataMasque_AU_addresses_real.csv`, `DataMasque_NZ_addresses_real.csv` |
-| Companies | `DataMasque_companies.csv`, `DataMasque_NZ_companies.csv`, `DataMasque_AU_companies.csv` |
-| Email | `DataMasque_fake_email_suffixes.csv`, `DataMasque_email_suffixes.csv` |
-| Reference | `DataMasque_country_codes.csv`, `DataMasque_occupations.csv` |
-| Cards | `DataMasque_credit_card_numbers.csv`, `DataMasque_credit_card_prefixes.csv` |
+| Companies | `DataMasque_companies.csv`, `DataMasque_NZ_companies.csv`, `DataMasque_AU_companies.csv`              |
+| Email     | `DataMasque_fake_email_suffixes.csv`, `DataMasque_email_suffixes.csv`                                 |
+| Reference | `DataMasque_country_codes.csv`, `DataMasque_occupations.csv`                                          |
+| Cards     | `DataMasque_credit_card_numbers.csv`, `DataMasque_credit_card_prefixes.csv`                           |
 
 Regional variants exist for BR, IN, AU, NZ, US.
 Use `from_file` when there are more than ~50 distinct values;

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/claude-skills/ruleset-builder/skills/ruleset-builder/references/ruleset-libraries-guide.md

@@ -143,13 +143,13 @@ tasks: [...]
 
 ## Libraries vs YAML Anchors
 
-| Feature | YAML Anchors (`&`/`*`) | Libraries (`$ref`) |
-|---------|----------------------|-------------------|
-| Scope | Within one ruleset | Across multiple rulesets |
-| Management | Inline in YAML | Managed via API/CLI, versioned |
-| Syntax | `<<: *anchor_name` | `$ref: "lib#path"` |
-| Override | `<<:` merge key | Not supported (use as-is) |
-| Best for | Single-ruleset reuse | Organisation-wide standards |
+| Feature    | YAML Anchors (`&`/`*`) | Libraries (`$ref`)             |
+|------------|------------------------|--------------------------------|
+| Scope      | Within one ruleset     | Across multiple rulesets       |
+| Management | Inline in YAML         | Managed via API/CLI, versioned |
+| Syntax     | `<<: *anchor_name`     | `$ref: "lib#path"`             |
+| Override   | `<<:` merge key        | Not supported (use as-is)      |
+| Best for   | Single-ruleset reuse   | Organisation-wide standards    |
 
 **Recommendation:**
 - Start with YAML anchors (`mask_definitions`) for within-ruleset deduplication

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/claude-skills/ruleset-builder/skills/ruleset-builder/references/ruleset-yaml-reference.md

@@ -72,6 +72,16 @@ For PostgreSQL/MySQL, plain names work: `table: users`, `key: id`.
 
 ## Mask Types Quick Reference
 
+This is the **closed list** of every `type:` value DataMasque accepts. Do
+not invent mask types or parameters (no `source_table`, no `link_to`, no
+`parent_column` — none exist). For per-mask parameter details, see the
+canonical source:
+<https://portal.datamasque.com/portal/documentation/latest/masking-functions-overview.html>.
+
+For a deterministic hash, use `imitate_unique` (or `imitate_uuid` for UUIDs)
+optionally with `seed:` to namespace. The cascade is automatic; no
+cross-table reference parameter exists. See `fk-cascade.md`.
+
 ### Generic
 - `from_fixed` — fixed replacement value
 - `from_column` — copy from another column
@@ -124,6 +134,20 @@ For PostgreSQL/MySQL, plain names work: `table: users`, `key: id`.
 ### Document
 - `json` — mask JSON fields within a column
 - `xml` — mask XML elements within a column
+- `unstructured_text` — mask entities inside free text
+
+### Commonly-hallucinated names that do NOT exist
+
+These plausible-sounding names are not in DataMasque. Refuse to emit them:
+
+| Hallucinated name                                           | What was wanted                   | Use instead                                                   |
+|-------------------------------------------------------------|-----------------------------------|---------------------------------------------------------------|
+| `hash_text`, `hash`                                         | deterministic hash of a value     | `imitate_unique` (or `imitate_uuid` for UUIDs)                |
+| `link`, `match_id`, `link_to`                               | join two columns after masking    | shared `imitate_unique` config on both sides                  |
+| `from_random_words`                                         | random words / short text         | `from_random_text` (random chars) or `from_file`              |
+| `from_random_string`                                        | random string                     | `from_random_text`                                            |
+| `redact`, `mask_value`                                      | constant placeholder              | `from_fixed` with `value:`                                    |
+| `source_table`, `source_column`, `parent_column`, `link_to` | param to point a FK at its parent | does not exist — cascade is automatic with shared mask config |
 
 ## skip_defaults
 

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "datamasque-cli"
-version = "1.2.0"
+version = "1.3.0"
 description = "Official command-line interface for the DataMasque data-masking platform."
 authors = [
     { name = "DataMasque Ltd" },

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/src/datamasque_cli/commands/system.py

@@ -1,4 +1,4 @@
-"""System-level commands: health, licence, logs, admin-install."""
+"""System administration commands."""
 
 from __future__ import annotations
 
@@ -116,3 +116,34 @@ def set_locality(
     client = get_client(profile)
     client.set_locality(locality)
     print_success(f"Locality set to '{locality}'.")
+
+
+ai_engine_app = typer.Typer(help="Configure the AI Engine.", no_args_is_help=True)
+app.add_typer(ai_engine_app, name="ai-engine")
+
+
+@ai_engine_app.command("show")
+def ai_engine_show(
+    profile: str | None = typer.Option(None, "--profile", "-p", help="Profile to use"),
+    is_json: bool = typer.Option(False, "--json", help="Output as JSON"),
+) -> None:
+    """Show the configured AI Engine URL."""
+    client = get_client(profile)
+    response = client.make_request("GET", "/api/settings/")
+    url = response.json().get("dm_ai_engine_url") or None
+    if should_emit_json(is_json):
+        print_json({"dm_ai_engine_url": url})
+        return
+    # An empty table cell would look like a rendering bug.
+    render_output({"dm_ai_engine_url": url or "<not configured>"}, is_json=False, title="AI Engine")
+
+
+@ai_engine_app.command("set")
+def ai_engine_set(
+    url: str = typer.Argument(help="AI Engine base URL"),
+    profile: str | None = typer.Option(None, "--profile", "-p", help="Profile to use"),
+) -> None:
+    """Point DataMasque at an AI Engine."""
+    client = get_client(profile)
+    client.make_request("PATCH", "/api/settings/", data={"dm_ai_engine_url": url})
+    print_success(f"AI Engine URL set to '{url}'.")

datamasque_cli-1.3.0/tests/commands/test_system.py

@@ -0,0 +1,82 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from unittest.mock import MagicMock, patch
+
+import pytest
+from datamasque.client.models.license import LicenseInfo, SwitchableLicenseMetadata
+from typer.testing import CliRunner
+
+from datamasque_cli.main import app
+
+MODULE = "datamasque_cli.commands.system"
+
+
+@patch(f"{MODULE}.get_client")
+def test_licence_projects_to_user_facing_fields(mock_get_client: MagicMock, runner: CliRunner) -> None:
+    client = MagicMock()
+    mock_get_client.return_value = client
+    client.get_current_license_info.return_value = LicenseInfo(
+        uuid="lic-123",
+        name="Test Licence",
+        type="standard",
+        is_expired=False,
+        uploadable=True,
+        expiry_date=datetime(2027, 6, 1, tzinfo=UTC),
+        days_until_expiry=400,
+        platform_name="DataMasque",
+        # Noisy nested field that should NOT appear in the projected output.
+        switchable_license_metadata=SwitchableLicenseMetadata(license_source="aws"),
+    )
+
+    result = runner.invoke(app, ["system", "licence", "--json"])
+
+    assert result.exit_code == 0
+    assert '"uuid": "lic-123"' in result.stdout
+    assert '"days_until_expiry": 400' in result.stdout
+    assert '"platform_name": "DataMasque"' in result.stdout
+    assert "switchable_license_metadata" not in result.stdout
+    assert "license_source" not in result.stdout
+
+
+@pytest.mark.parametrize(
+    ("extra_args", "settings_url", "expected_output"),
+    [
+        (["--json"], "http://engine.example.com:9021", '"dm_ai_engine_url": "http://engine.example.com:9021"'),
+        ([], "http://engine.example.com:9021", "http://engine.example.com:9021"),
+        ([], None, "<not configured>"),
+        ([], "", "<not configured>"),
+    ],
+)
+@patch(f"{MODULE}.get_client")
+def test_ai_engine_show(
+    mock_get_client: MagicMock,
+    runner: CliRunner,
+    extra_args: list[str],
+    settings_url: str | None,
+    expected_output: str,
+) -> None:
+    client = MagicMock()
+    mock_get_client.return_value = client
+    response = MagicMock()
+    response.json.return_value = {"dm_ai_engine_url": settings_url}
+    client.make_request.return_value = response
+
+    result = runner.invoke(app, ["system", "ai-engine", "show", *extra_args])
+
+    assert result.exit_code == 0
+    client.make_request.assert_called_once_with("GET", "/api/settings/")
+    assert expected_output in result.stdout
+
+
+@patch(f"{MODULE}.get_client")
+def test_ai_engine_set_patches_settings_with_url(mock_get_client: MagicMock, runner: CliRunner) -> None:
+    client = MagicMock()
+    mock_get_client.return_value = client
+
+    result = runner.invoke(app, ["system", "ai-engine", "set", "http://engine.example.com:9021"])
+
+    assert result.exit_code == 0
+    client.make_request.assert_called_once_with(
+        "PATCH", "/api/settings/", data={"dm_ai_engine_url": "http://engine.example.com:9021"}
+    )

{datamasque_cli-1.2.0 → datamasque_cli-1.3.0}/uv.lock

@@ -141,7 +141,7 @@ wheels = [
 
 [[package]]
 name = "datamasque-cli"
-version = "1.2.0"
+version = "1.3.0"
 source = { editable = "." }
 dependencies = [
     { name = "datamasque-python" },

datamasque_cli-1.2.0/claude-skills/ruleset-builder/skills/ruleset-builder/SKILL.md

@@ -1,175 +0,0 @@
----
-name: ruleset-builder
-description: Use when the user wants to turn auto-generated DataMasque rulesets into production-ready ones — extract a `ruleset_library`, add `hash_columns`, refine a ruleset, or clean up generated YAML. Triggers on "ruleset builder", "build ruleset", "refine ruleset", "add hash columns", "add ruleset library", "production ruleset", "clean up ruleset".
-argument-hint: e.g. "build a ruleset from these generated files"
-user-invocable: true
----
-
-# Ruleset Builder
-
-Transform auto-generated DataMasque rulesets into production-ready rulesets with three improvements:
-1. **`ruleset_library` references** — `$ref` links replacing every repeated inline mask
-2. **`hash_columns`** — on every applicable `mask_table` task for deterministic consistency
-3. **Clean structure** — `skip_defaults`, no doc blocks, validated
-
-**4-step process. Complete all 4 steps. Report after each step before proceeding.**
-
-Use `TaskCreate` for all 4 steps before starting. The prompt must include business domain and application type — ask if missing.
-
----
-
-## Step 0: Report version
-Report: **Version 1.5**
-
----
-
-## Step 1: Read reference docs
-
-Read all three before any other work:
-```
-${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/mask-definitions-guide.md
-${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/hash-columns-guide.md
-${CLAUDE_PLUGIN_ROOT}/skills/ruleset-builder/references/ruleset-yaml-reference.md
-```
-
----
-
-## Step 2: Extract ruleset_library
-
-Write a Python script using `ruamel.yaml` (`uv pip install ruamel.yaml`).
-
-Process the input YAML. For each `mask_table` task, replace every inline mask with a `$ref` to a rule in `ruleset_library.yaml`. Build the library progressively — read its current state at the start of each iteration, create it if absent.
-
-The library `masks` section structure:
-```yaml
-version: "1.0"
-masks:
-  rule_name:
-    type: rule_type
-    ...params
-```
-
-### Classification rules (apply in order)
-
-**1. ID columns** — any column ending in `_ID`, `_NO`, `_NR`, `_NBR` is an entity identifier.
-- Strip adjective/verb prefixes before the noun: `PREVIOUS_`, `OLD_`, `TRANSFERRED_`, `PRIOR_`, `CURR_`, `NEW_`, `NEXT_`, `ALT_`, `PARENT_`, `CHILD_`, `SOURCE_`, `TARGET_`, `ORIG_`, `PENDING_`, `ARCHIVED_`, `DELETED_`
-- Extract the core entity: `PREVIOUS_INVOICE_ID` → `invoice`, `TRANSFERRED_ACCOUNT_ID` → `account`, `INVOICE_ACCOUNT_ID` → `invoice_account` (compound kept — no prefix stripped)
-- Group all derivatives to one rule: `$ref: "Global/RuleLib#masks/{entity}_id"`
-- Library entry: `type: imitate_unique`, `seed: "{entity}"` — **seed is required**
-- This overrides whatever mask was originally generated (even `imitate_unique`, `from_random_number`, etc.)
-
-**2. Named patterns** — detect by mask structure:
-
-| Pattern | Detection | Library rule |
-|---------|-----------|--------------|
-| Email | `chain(concat(concat(firstName+lastName, glue='.')+email_suffix)+transform_case(lower))` | `email_address` |
-| Full name | `chain(concat(firstName+lastName, glue=' ')+take_substring)` OR plain `concat(firstName+lastName, glue=' ')` — column not containing USERNAME/LOGIN | `full_name` |
-| Username | Same mask as full_name but column name contains USERNAME, USER_NAME, LOGIN, LOGON | `username` |
-| First name only | `from_file` with firstNames seed | `name_first` |
-| Last name only | `from_file` with lastNames seed | `name_last` |
-| DOB | Column name contains DOB/BIRTH/DATE_OF_BIRTH — use `retain_age` regardless of original type | `dob` |
-| Company | `chain(from_file(companies)+take_substring)` | `company_name` |
-| Country name | `from_file(country_codes, seed_column=name)` | `country_name` |
-| Country alpha-2 | `from_file(country_codes, seed_column=alpha_2)` | `country_code_2` |
-| Country alpha-3 | `from_file(country_codes, seed_column=alpha_3)` | `country_code_3` |
-| Phone/fax | `imitate` on column name containing PHONE, TEL, FAX, MOBILE, CELL | `phone` |
-| Address line 1 | `from_file(addresses, seed_column=street_address)` on LINE_1/ADDRESS_LINE_1 columns | `address_line1` |
-| Address line N | Same for LINE_2, LINE_3 etc. | `address_lineN` |
-| Address full | `from_file(addresses, seed_column=street_address)` on non-line-numbered columns | `address_full` |
-| Address expr | `concat(address+city+state+postcode, glue=', ')` | `network_address_expr` |
-| City | `from_file(addresses, seed_column=city)` | `city` |
-| Postcode | `from_file(addresses, seed_column=postcode)` | `post_code` |
-| Suburb | `from_file(addresses, seed_column=suburb)` | `suburb` |
-| Occupation | `from_file(occupations)` | `occupation` |
-
-**3. Remaining** — group by column name concept. Where column names share a root (e.g., `RESULT3_VALUE`, `RESULT5_VALUE` → `result_value`; `GENERAL_2`, `GENERAL_6` → `general`), use one shared rule. Strip adjective prefixes. Use first occurrence's parameters.
-
-- `imitate_unique` (non-ID cols) → `{col_group}: type: imitate_unique, seed: "{col_group}"` — **seed is required**
-- `from_random_date` → `{col_group}: type: from_random_date, min/max from first occurrence`
-- `from_random_number` → `{col_group}: type: from_random_number, min/max from first occurrence`
-- `imitate` (non-phone) → `{col_group}: type: imitate`
-- Complex chains → keep structure, group by column name
-
-> **Critical rule:** Every `imitate_unique` entry in `ruleset_library.yaml` MUST have a `seed` value.
-> - Entity ID rules: `seed: "{entity_name}"` (e.g., `account_id` → `seed: "account"`)
-> - All other `imitate_unique` rules: `seed: "{rule_name}"` (e.g., `field_name` → `seed: "field_name"`)
-
-### Output format
-
-```yaml
-version: '1.0'
-skip_defaults:
-  - ''
-  - null
-imports:
-  - Global/RuleLib
-
-tasks:
-  - type: mask_table
-    table: '"SCHEMA"."TABLE"'
-    key: '"ROWID"'
-    rules:
-      - column: '"FIRST_NAME"'
-        masks:
-          - $ref: "Global/RuleLib#masks/name_first"
-```
-
-Do NOT write a custom YAML serializer. Use `ruamel.yaml` round-trip dumper. Use `DoubleQuotedScalarString` for `$ref` values.
-
-**Report:** "Step 2 done — extracted N rule library definitions: [list each name and usage count]."
-
----
-
-## Step 3: Add hash_columns
-
-Write a Python script that:
-
-**Parse the discovery CSV** (comma-separated):
-`Selected`, `Table schema`, `Table name`, `Column name`, `Data Type`, `Constraint`, `Foreign Keys`, `Max Length`, `Numeric Precision`, `Numeric Scale`, `Reason for flag`, `Flagged by`, `Data classifications`
-
-Build a lookup of `(schema, table)` → columns with constraint and FK metadata:
-- `Constraint` patterns: `Primary(COL)`, `Unique(COL)`, `Foreign(COL)`
-- `Foreign Keys` JSON: `["FK_NAME", "SCHEMA.TABLE.COLUMN"]` — index 1 gives the referenced table
-
-**For each `mask_table` task:**
-
-1. **Pick hash column** using this priority:
-   - **Parent-entity FK first**: find FK columns where the referenced table is the parent of the current table — i.e., the current table name *starts with* the referenced table name (e.g., `ACCOUNT_HISTORY` starts with `ACCOUNT` → use `ACCOUNT_ID`). This avoids choosing lookup-table FKs (e.g., don't choose `ACCOUNT_TYPE_ID` in `ACCOUNT` just because it has a FK).
-   - **PK fallback**: if no parent-entity FK found, use the Primary Key column (never `ROWID`)
-   - **Archive table fallback**: if no PK in the CSV (archive tables `_A`, `_A_R`, `_R` often lack explicit keys), strip the suffix and look up the base table recursively
-   - **Composite PKs**: prefer `*_ID` or `*_NO` columns; deduplicate derivatives (`ACCOUNT_ID` + `PREVIOUS_ACCOUNT_ID` → keep `ACCOUNT_ID`)
-   - **Skip** if no suitable column found
-
-2. Insert `hash_columns: ["COLUMN_NAME"]` after the `key:` field
-
-3. Verify all rules in output are `$ref` — fix any remaining inline masks
-
-4. Write to output file
-
-**Report:** "Step 3 done — added hash_columns to N tables, skipped M (all-unique), skipped K (no suitable key). Top hash columns: [column → count]."
-
----
-
-## Step 4: Validate and clean up
-
-Remove any comment lines containing `ROWID`.
-
-Run:
-```bash
-dm rulesets validate --file <output_file>
-```
-
-Fix any errors and re-validate until passing.
-
----
-
-## Summary
-
-| Metric | Value |
-|--------|-------|
-| Total tables | N |
-| Mask definitions extracted | N (list names) |
-| Tables with hash_columns | N |
-| Tables skipped (no key) | N |
-| Validation | passed/failed |
-| Output file | path |

datamasque_cli-1.2.0/tests/commands/test_system.py

@@ -1,38 +0,0 @@
-from __future__ import annotations
-
-from datetime import UTC, datetime
-from unittest.mock import MagicMock, patch
-
-from datamasque.client.models.license import LicenseInfo, SwitchableLicenseMetadata
-from typer.testing import CliRunner
-
-from datamasque_cli.main import app
-
-MODULE = "datamasque_cli.commands.system"
-
-
-@patch(f"{MODULE}.get_client")
-def test_licence_projects_to_user_facing_fields(mock_get_client: MagicMock, runner: CliRunner) -> None:
-    client = MagicMock()
-    mock_get_client.return_value = client
-    client.get_current_license_info.return_value = LicenseInfo(
-        uuid="lic-123",
-        name="Test Licence",
-        type="standard",
-        is_expired=False,
-        uploadable=True,
-        expiry_date=datetime(2027, 6, 1, tzinfo=UTC),
-        days_until_expiry=400,
-        platform_name="DataMasque",
-        # Noisy nested field that should NOT appear in the projected output.
-        switchable_license_metadata=SwitchableLicenseMetadata(license_source="aws"),
-    )
-
-    result = runner.invoke(app, ["system", "licence", "--json"])
-
-    assert result.exit_code == 0
-    assert '"uuid": "lic-123"' in result.stdout
-    assert '"days_until_expiry": 400' in result.stdout
-    assert '"platform_name": "DataMasque"' in result.stdout
-    assert "switchable_license_metadata" not in result.stdout
-    assert "license_source" not in result.stdout