@dforge-core/dforge-mcp 0.1.0-rc.11 → 0.1.0-rc.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (43) hide show
  1. package/CHANGELOG.md +132 -0
  2. package/README.md +75 -21
  3. package/dist/server.js +2405 -272
  4. package/docs/creating-modules.md +7 -3
  5. package/package.json +11 -6
  6. package/resources/docs/conventions.md +20 -14
  7. package/resources/schemas/entity.schema.json +8 -1
  8. package/resources/schemas/reports.schema.json +4 -0
  9. package/skills/dforge-mcp-author/SKILL.md +227 -95
  10. package/skills/dforge-mcp-author/examples/simple-todo/README.md +38 -0
  11. package/skills/dforge-mcp-author/examples/simple-todo/entities/todo_item.json +83 -0
  12. package/skills/dforge-mcp-author/examples/simple-todo/entities/todo_list.json +43 -0
  13. package/skills/dforge-mcp-author/examples/simple-todo/logic/actions/mark_done.dsl +6 -0
  14. package/skills/dforge-mcp-author/examples/simple-todo/manifest.json +32 -0
  15. package/skills/dforge-mcp-author/examples/simple-todo/security/roles.json +10 -0
  16. package/skills/dforge-mcp-author/examples/simple-todo/seed-data/01-lists.json +17 -0
  17. package/skills/dforge-mcp-author/examples/simple-todo/ui/actions.json +11 -0
  18. package/skills/dforge-mcp-author/examples/simple-todo/ui/data_views.json +35 -0
  19. package/skills/dforge-mcp-author/examples/simple-todo/ui/menus.json +28 -0
  20. package/skills/dforge-mcp-author/references/action-dsl.md +397 -0
  21. package/skills/dforge-mcp-author/references/column-types.md +168 -0
  22. package/skills/dforge-mcp-author/references/conventions.md +177 -0
  23. package/skills/dforge-mcp-author/references/data-migration.md +270 -0
  24. package/skills/dforge-mcp-author/references/data-views.md +192 -0
  25. package/skills/dforge-mcp-author/references/excel-import.md +61 -0
  26. package/skills/dforge-mcp-author/references/field-types.md +144 -0
  27. package/skills/dforge-mcp-author/references/filters.md +326 -0
  28. package/skills/dforge-mcp-author/references/flags.md +73 -0
  29. package/skills/dforge-mcp-author/references/formulas.md +206 -0
  30. package/skills/dforge-mcp-author/references/jobs.md +149 -0
  31. package/skills/dforge-mcp-author/references/manifest.md +123 -0
  32. package/skills/dforge-mcp-author/references/menus.md +164 -0
  33. package/skills/dforge-mcp-author/references/number-sequences.md +117 -0
  34. package/skills/dforge-mcp-author/references/print-templates.md +159 -0
  35. package/skills/dforge-mcp-author/references/queries.md +312 -0
  36. package/skills/dforge-mcp-author/references/reports.md +398 -0
  37. package/skills/dforge-mcp-author/references/schema-import.md +331 -0
  38. package/skills/dforge-mcp-author/references/security.md +244 -0
  39. package/skills/dforge-mcp-author/references/settings.md +120 -0
  40. package/skills/dforge-mcp-author/references/traits.md +153 -0
  41. package/skills/dforge-mcp-author/references/translations.md +158 -0
  42. package/skills/dforge-mcp-author/references/validation-checklist.md +182 -0
  43. package/skills/dforge-mcp-author/scripts/xlsx_to_model.py +198 -0
@@ -0,0 +1,168 @@
1
+ # Column Types Reference
2
+
3
+ Every column has a `columnType` (optional for the default physical type). Seven values exist:
4
+
5
+ | `columnType` | Physical DB column? | Description |
6
+ |---|---|---|
7
+ | (omitted) or `"D"` | Yes | **Data column** — default. Maps to a real SQL column. |
8
+ | `"R"` | No | **Reference column** — virtual N:1 lookup. Paired with a hidden FK column (see FK+Reference pattern). |
9
+ | `"S"` | No | **Set column** — virtual 1:N backwards reference. Used for detail grids. |
10
+ | `"F"` | No | **Formula column** — virtual computed value. Evaluated by the formula engine. |
11
+ | `"A"` | Yes | **Accumulation register column** — stores posted state for accumulation registers. Advanced (accounting modules). |
12
+ | `"L"` | Yes | **Ledger register column** — stores posted state for double-entry bookkeeping. Advanced (accounting modules). |
13
+ | `"G"` | Yes | **Generated column** — DB-level computed aggregate (e.g. SUM over child set via trigger, or PostgreSQL `GENERATED ALWAYS AS`). The aggregated child column **must be physical** (a `D` column) — never a virtual `F`/`R`/`S` column. For a simple roll-up total, prefer `F` (see below). |
14
+
15
+ **For most module development, you'll only use D, R, S, and F.** Types A, L, and G are for advanced accounting/registry modules that use the `postable`, `accumulation`, or `ledger` traits.
16
+
17
+ ## Data columns (`columnType` omitted or `"D"`)
18
+
19
+ Most columns. They map 1:1 to a physical SQL column.
20
+
21
+ ```json
22
+ "first_name": {
23
+ "dbDatatype": "varchar",
24
+ "fieldTypeCd": "text",
25
+ "flags": "VEM",
26
+ "maxLen": 100,
27
+ "orderNum": 20,
28
+ "description": "First Name"
29
+ }
30
+ ```
31
+
32
+ ## Reference columns (`columnType: "R"`)
33
+
34
+ Virtual N:1 lookup. No physical column. Always **paired with a hidden FK column** that does hold the physical data. Together they form the **FK+Reference pattern**.
35
+
36
+ ```json
37
+ // Hidden FK column — physical, hidden
38
+ "account_id": {
39
+ "dbDatatype": "cuid",
40
+ "flags": "EM",
41
+ "orderNum": 90,
42
+ "description": "Account ID"
43
+ },
44
+ // Visible Reference column — virtual, shown as lookup picker
45
+ "account": {
46
+ "columnType": "R",
47
+ "fieldTypeCd": "lookup",
48
+ "flags": "VEM",
49
+ "orderNum": 95,
50
+ "description": "Account",
51
+ "link": {
52
+ "entity": "account",
53
+ "thisKey": "account_id",
54
+ "otherKey": "account_id"
55
+ }
56
+ }
57
+ ```
58
+
59
+ Plus declare the FK constraint in the entity's `references` block:
60
+
61
+ ```json
62
+ "references": {
63
+ "FK_Contact_Account": {
64
+ "from": { "field": "account_id" },
65
+ "to": { "entity": "account", "field": "account_id" }
66
+ }
67
+ }
68
+ ```
69
+
70
+ > **FK column `dbDatatype` must exactly match the referenced entity's PK `dbDatatype`.** Never guess.
71
+ > - Entities using the `identity` trait → PK is `dbDatatype: "cuid"` → FK column must also be `"cuid"`
72
+ > - Cross-module or legacy entities → call `dforge_module_inspect` on the referenced module and read the PK column's `dbDatatype` before declaring the FK
73
+ > - **`bigint`, `integer`, `int8` are wrong values for FK columns** — even though `cuid` is physically stored as int8, the platform type name is `cuid`, not `bigint`
74
+
75
+ **Why two columns?** The FK stores the actual value. The Reference column configures how the value is rendered in the UI (as a typeahead picker showing the target's `toString`). They must be kept in sync — column order in JSON doesn't matter but `orderNum` determines UI ordering.
76
+
77
+ ## Set columns (`columnType: "S"`)
78
+
79
+ Virtual 1:N backwards reference. Shows a grid of related records on the parent's detail view.
80
+
81
+ ```json
82
+ "activities": {
83
+ "columnType": "S",
84
+ "fieldTypeCd": "grid",
85
+ "flags": "VEM",
86
+ "orderNum": 150,
87
+ "description": "Activities",
88
+ "link": {
89
+ "entity": "activity",
90
+ "thisKey": "contact_id", // column on THIS entity (parent PK)
91
+ "otherKey": "contact_id" // column on the OTHER entity (FK back to parent)
92
+ }
93
+ }
94
+ ```
95
+
96
+ `link.entity` is the child entity. `link.thisKey` is the column on the current entity (typically the PK). `link.otherKey` is the FK column on the child entity pointing back to the parent.
97
+
98
+ Set columns are **not** declared in the `references` block — the FK already lives on the child entity.
99
+
100
+ ## Formula columns (`columnType: "F"`)
101
+
102
+ Virtual computed column. No physical storage. Value is calculated at runtime by the formula engine.
103
+
104
+ ```json
105
+ "full_name": {
106
+ "columnType": "F",
107
+ "fieldTypeCd": "text",
108
+ "baseDatatypeCd": "string",
109
+ "flags": "V",
110
+ "orderNum": 25,
111
+ "formula": "[first_name] + ' ' + [last_name]",
112
+ "description": "Full Name"
113
+ }
114
+ ```
115
+
116
+ **Required fields for formula columns**:
117
+
118
+ - `columnType: "F"`
119
+ - `fieldTypeCd` — how to render it
120
+ - `baseDatatypeCd` — **required** on formula columns even though there's no physical column; used for type inference
121
+ - `flags: "V"` (usually — formula columns are almost always read-only)
122
+ - `formula` — the formula expression
123
+ - `orderNum`
124
+ - `description`
125
+
126
+ Do **not** include `dbDatatype` on formula columns (there's no physical storage).
127
+
128
+ See `formulas.md` for syntax.
129
+
130
+ ## Roll-up totals over child rows — use `F`, not `G`
131
+
132
+ To total a child set on a header (line items → order total, movements → quantity on hand),
133
+ **use a Formula column (`"F"`) with `SUM([set].[field])`**. It is computed at query time, so it
134
+ can reference **any** child column — including the child's own formula columns.
135
+
136
+ ```json
137
+ // purchase_order.total_amount — query-time roll-up (installs cleanly)
138
+ "total_amount": {
139
+ "columnType": "F",
140
+ "fieldTypeCd": "currency",
141
+ "baseDatatypeCd": "number",
142
+ "flags": "V",
143
+ "orderNum": 90,
144
+ "formula": "SUM([lines].[line_total])", // line_total may itself be a formula column
145
+ "description": "Total Amount"
146
+ }
147
+ ```
148
+
149
+ > ⛔ **Never make a `G` (Generated) column aggregate a virtual `F` column.** A `G` aggregate is
150
+ > maintained by a database trigger that reads the child's `OLD`/`NEW` *physical* values. A formula
151
+ > (`F`) column has no physical storage, so the trigger references a column that doesn't exist and
152
+ > **install fails** with `db_error: column old.<field> does not exist`. If you need a stored
153
+ > (`G`) aggregate over a *derived* quantity (e.g. a signed movement quantity), make that child
154
+ > column **physical** first — either a plain `D` column written by the action logic, or a
155
+ > same-row `G`/`GENERATED ALWAYS AS` column with a `dbDatatype` — then aggregate that. For most
156
+ > modules a query-time `F` roll-up is the right choice; reserve `G` for high-volume stored aggregates.
157
+
158
+ ## Quick reference: which columnType do I use?
159
+
160
+ - **Plain text/number/date/etc. field that stores a value** → omit `columnType` (default `"D"`)
161
+ - **"This contact belongs to one account"** → `"R"` (plus hidden FK)
162
+ - **"This account has many contacts"** → `"S"` on the account side
163
+ - **"Compute full_name from first_name + last_name"** → `"F"`
164
+ - **"User picker" (points to admin.user)** → omit `columnType`, use `fieldTypeCd: "user"` (writes user ID directly, no paired FK)
165
+ - **"Polymorphic link to any entity"** → omit `columnType`, use `fieldTypeCd: "entitylink"` (stored as JSON)
166
+ - **"Roll-up total over child rows"** (line items → header total, movements → quantity on hand) → `"F"` with `SUM([set].[field])` (query-time; see "Roll-up totals" above). Reserve `"G"` for high-volume *stored* aggregates, and only over a **physical** child column — never over an `F` column.
167
+ - **"Accumulation register state"** → `"A"` (advanced accounting — requires `postable` + `accumulation` traits)
168
+ - **"Double-entry ledger state"** → `"L"` (advanced accounting — requires `postable` + `ledger` traits)
@@ -0,0 +1,177 @@
1
+ # Module Conventions Reference
2
+
3
+ Rules that apply across the entire module package. Violating these produces technically-valid but non-idiomatic modules, and in several cases breaks the installer.
4
+
5
+ ## Naming
6
+
7
+ | Thing | Convention | Example |
8
+ |---|---|---|
9
+ | Module `code` | lowercase, letters/digits/underscore | `crm`, `my_module`, `wms_fin` |
10
+ | Entity keys / `dbObject` | `snake_case`, singular | `contact`, `opportunity_line` |
11
+ | Column codes | `snake_case` | `first_name`, `account_id` |
12
+ | Data view codes | `snake_case` | `contact_list`, `deal_kanban` |
13
+ | Report codes | `snake_case` | `sales_pipeline` |
14
+ | Action codes | `snake_case` | `change_stage`, `send_welcome` |
15
+ | Menu codes | `snake_case` | `sales`, `pipeline` |
16
+ | Role codes | `snake_case`, domain-specific | `sales_rep`, `sales_admin` |
17
+ | Setting codes | `snake_case` | `vat_rate`, `invoice_prefix` |
18
+ | Trait names | lowercase single word | `identity`, `audit` |
19
+ | Seed data files | numbered prefix | `01-countries.json`, `02-currencies.json` |
20
+ | Action DSL files | `<action_code>.dsl` | `change_stage.dsl` |
21
+ | Reference keys | `FK_<FromEntity>_<Purpose>` | `FK_Contact_Account` |
22
+
23
+ Everything is **case-sensitive**. Don't mix snake_case and camelCase.
24
+
25
+ ## File layout
26
+
27
+ Standard module directory structure:
28
+
29
+ ```
30
+ my_module/
31
+ ├── manifest.json
32
+ ├── README.md # module overview
33
+ ├── CHANGELOG.md # version history
34
+ ├── MODULE-INFO.md # user-facing intro (shown in module picker)
35
+ ├── entities/
36
+ │ └── <entity>.json
37
+ ├── logic/
38
+ │ └── actions/
39
+ │ └── <action>.dsl
40
+ ├── ui/
41
+ │ ├── data_views.json
42
+ │ ├── menus.json
43
+ │ ├── actions.json
44
+ │ ├── folders.json
45
+ │ └── reports/
46
+ │ └── <report>.json
47
+ ├── security/
48
+ │ └── roles.json
49
+ ├── settings.json
50
+ ├── seed-data/
51
+ │ ├── 01-<first>.json
52
+ │ └── 02-<second>.json
53
+ ├── translations/
54
+ │ └── de-DE.json # IETF tags, one file per non-English locale
55
+ ├── print_templates/
56
+ │ └── <template>.scriban
57
+ ├── files/ # static assets
58
+ └── webhooks.json # optional
59
+ ```
60
+
61
+ Only include what your module actually uses — not every module needs `print_templates/` or `webhooks.json`.
62
+
63
+ ## The FK+Reference pattern (again, because it's critical)
64
+
65
+ **Every reference to another entity is two columns**:
66
+
67
+ 1. Hidden FK column with `flags: "EM"` and `dbDatatype` matching the target PK type
68
+ 2. Visible Reference column with `columnType: "R"`, `fieldTypeCd: "lookup"`, `flags: "VEM"`, and a `link` object pointing at the target
69
+
70
+ Plus the actual FK constraint declared in the entity's `references` block.
71
+
72
+ See `column-types.md` for the full example.
73
+
74
+ ## Bridge modules
75
+
76
+ When two core modules need to integrate (e.g. CRM and Finance), create a **bridge module**:
77
+
78
+ - Name it with a hyphen: `crm-fin`, `wms-fin`, `crm-pricing`
79
+ - Depends on **both** core modules
80
+ - Owns the integration: extension entities, cross-module actions, cross-module data views
81
+ - Core modules stay independent and don't know about each other
82
+
83
+ Extension entities in a bridge module have `"extends": "fin.invoice"` at the top. Physical columns in the extension go into a 1:1 ext table.
84
+
85
+ ## Seed data ordering
86
+
87
+ Seed files run in numeric prefix order. Put entities with FK dependencies **after** their targets:
88
+
89
+ ```
90
+ seed-data/
91
+ ├── 01-currencies.json # no FKs, insert first
92
+ ├── 02-countries.json # references currency
93
+ ├── 03-regions.json # references country
94
+ └── 04-offices.json # references region
95
+ ```
96
+
97
+ Each seed file has two top-level keys: `"entityCode"` (NOT `"entity"`) and `"records"` (array). **Using `"entity"` instead of `"entityCode"` is a silent failure** — the installer reads an empty entity code and the INSERT fails.
98
+
99
+ Use explicit **numeric** PKs in seed data so cross-file references work. The `identity` trait creates `cuid` columns which are `int8` (bigint) — **NOT UUIDs**. Use simple integers like `1001`, `1002`:
100
+
101
+ ```json
102
+ {
103
+ "entityCode": "country",
104
+ "records": [
105
+ { "country_id": 1001, "code": "US", "name": "United States" },
106
+ { "country_id": 1002, "code": "DE", "name": "Germany" }
107
+ ]
108
+ }
109
+ ```
110
+
111
+ **Common mistake**: using UUID strings like `"10000000-0000-0000-0000-000000000001"` — **wrong**. The `cuid` datatype is a numeric `int8`, not a UUID string. Seed data PKs must be numbers. Use a numbering scheme that avoids collisions (e.g. 1001-1099 for countries, 2001-2099 for regions, etc.).
112
+
113
+ ## Translations
114
+
115
+ Translation files live in `translations/` and are named after the **IETF locale tag** (`de-DE.json`, `uk-UA.json`, etc.). Filenames are auto-discovered; declare which ones you ship in `manifest.supportedLocales`. English is the default — labels and descriptions live in entity/view JSON, so **don't** create `en.json` or `en-US.json` (they're skipped by the installer).
116
+
117
+ ```json
118
+ // translations/de-DE.json
119
+ {
120
+ "contact": "Kontakt",
121
+ "account": "Kunde",
122
+ "contact_list.label": "Alle Kontakte"
123
+ }
124
+ ```
125
+
126
+ Missing keys fall back to the English value declared in the source JSON.
127
+
128
+ ## Versioning — always bump before packaging
129
+
130
+ The manifest has **two version numbers** that must be managed:
131
+
132
+ | Field | When to bump | Example |
133
+ |---|---|---|
134
+ | `version` | **Every release** — any change at all (bug fix, feature, schema change) | `0.1.0` → `0.2.0` |
135
+ | `dbSchemaVersion` | **Only when the DB schema changes** — new entity, new column, removed column, changed type, new constraint, new index | `0.0.1` → `0.0.2` |
136
+
137
+ Both use **semver** (`MAJOR.MINOR.PATCH`).
138
+
139
+ **Rules**:
140
+
141
+ - If you added an action but no new columns → bump `version` only, keep `dbSchemaVersion`.
142
+ - If you added a new entity or column → bump both `version` and `dbSchemaVersion`.
143
+ - If you changed a view or menu → bump `version` only.
144
+ - If you changed translations only → bump `version` only.
145
+ - The installer uses `dbSchemaVersion` to decide whether to run schema migrations. Wrong version = skipped migrations.
146
+ - **Never ship a package without bumping at least `version`** from the previous release. The installer may reject or silently skip a re-install with the same version.
147
+ - For brand-new modules, start at `version: "0.1.0"` and `dbSchemaVersion: "0.0.1"`.
148
+
149
+ ## `orderNum` — always set it
150
+
151
+ On every column, data view, menu item, setting, and action that appears in a list. Without `orderNum`, UI ordering is undefined. Use widely-spaced values (10, 20, 30, 40) so you can insert between them later.
152
+
153
+ ## Consistency across the module
154
+
155
+ - All timestamps use the same trait (`audit`)
156
+ - All primary keys use the same trait (`identity`) with consistent PK type (`cuid`)
157
+ - All entities have `toString`
158
+ - All views use `dataSources` array
159
+ - All roles use `rights`
160
+ - All menus use nested dicts + `dataViewCode`
161
+ - All actions have `canExecute:` (even if it's just `true`)
162
+
163
+ ## Do not invent
164
+
165
+ - Do not invent field types (`rating`, `geolocation`, `signature`) — not in the catalog
166
+ - Do not invent column types beyond `D` / `R` / `S` / `F`
167
+ - Do not invent role rights letters beyond `SIUDC` + `E`
168
+ - Do not invent view types beyond the registered ones
169
+ - Do not invent menu item types beyond `V` / `R` / `D` / `A`
170
+ - Do not invent DSL functions
171
+ - Do not invent formula functions
172
+
173
+ If you genuinely need something new, **say so** and ask the user how to work around it with existing primitives.
174
+
175
+ ## Reference
176
+
177
+ This file summarises the key conventions. If you need to verify a pattern not covered here, ask the user or check the reference modules in the `examples/` directory.
@@ -0,0 +1,270 @@
1
+ # Data Migration Reference
2
+
3
+ When the user has an existing database (typically after running the schema importer to produce a rough-cut module) and wants to **move the actual rows** into dForge, write them a migration script they can run locally.
4
+
5
+ This assumes:
6
+ - dForge is running locally (e.g. `docker compose up`).
7
+ - The legacy database is reachable from the developer's machine.
8
+ - The target is a **dev tenant**, never production.
9
+
10
+ ## Script template — Python / psycopg
11
+
12
+ This is the most portable choice. Works against any source database with a Python driver, writes to dForge Postgres.
13
+
14
+ ```python
15
+ #!/usr/bin/env python3
16
+ """
17
+ Migration script: old_crm → dForge my_crm module
18
+ Generated on 2026-04-08
19
+ """
20
+
21
+ import argparse
22
+ import sys
23
+ import uuid
24
+ import psycopg
25
+ from datetime import datetime
26
+ from typing import Dict
27
+
28
+ MIGRATION_USER_ID = "00000000-0000-0000-0000-000000000001" # override with --migration-user
29
+
30
+ def cuid() -> str:
31
+ """Generate a new cuid-like identifier for target PKs."""
32
+ return str(uuid.uuid4())
33
+
34
+ def main():
35
+ parser = argparse.ArgumentParser()
36
+ parser.add_argument("--source", required=True, help="Source DB connection string")
37
+ parser.add_argument("--target", required=True, help="Target dForge tenant DB connection string")
38
+ parser.add_argument("--migration-user", default=MIGRATION_USER_ID)
39
+ parser.add_argument("--dry-run", action="store_true", help="Simulate without committing")
40
+ args = parser.parse_args()
41
+
42
+ src = psycopg.connect(args.source)
43
+ tgt = psycopg.connect(args.target)
44
+
45
+ try:
46
+ with tgt.transaction():
47
+ pk_map: Dict[str, Dict[int, str]] = {} # source_table → {source_id → target_id}
48
+
49
+ # Phase 1: parents (no FK deps)
50
+ migrate_accounts(src, tgt, pk_map, args.migration_user)
51
+ migrate_products(src, tgt, pk_map, args.migration_user)
52
+
53
+ # Phase 2: children (depend on parents)
54
+ migrate_contacts(src, tgt, pk_map, args.migration_user)
55
+ migrate_opportunities(src, tgt, pk_map, args.migration_user)
56
+
57
+ # Phase 3: grandchildren
58
+ migrate_opportunity_lines(src, tgt, pk_map, args.migration_user)
59
+ migrate_activities(src, tgt, pk_map, args.migration_user)
60
+
61
+ if args.dry_run:
62
+ print("[DRY RUN] Rolling back.")
63
+ raise SystemExit(0)
64
+
65
+ print("Migration committed successfully.")
66
+
67
+ except Exception as e:
68
+ print(f"Migration failed: {e}", file=sys.stderr)
69
+ sys.exit(1)
70
+ finally:
71
+ src.close()
72
+ tgt.close()
73
+
74
+
75
+ def migrate_accounts(src, tgt, pk_map, migration_user):
76
+ print("Migrating accounts…")
77
+ pk_map["accounts"] = {}
78
+ with src.cursor() as sc, tgt.cursor() as tc:
79
+ sc.execute("SELECT id, name, phone, website, created_at FROM accounts")
80
+ count = 0
81
+ for row in sc:
82
+ old_id, name, phone, website, created_at = row
83
+ new_id = cuid()
84
+ pk_map["accounts"][old_id] = new_id
85
+ tc.execute(
86
+ """
87
+ INSERT INTO crm.account
88
+ (account_id, name, phone, website, created_date, created_by, last_updated, last_updated_by)
89
+ VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
90
+ """,
91
+ (new_id, name, phone, website, created_at or datetime.utcnow(),
92
+ migration_user, created_at or datetime.utcnow(), migration_user),
93
+ )
94
+ count += 1
95
+ print(f" {count} accounts migrated")
96
+
97
+
98
+ def migrate_contacts(src, tgt, pk_map, migration_user):
99
+ print("Migrating contacts…")
100
+ pk_map["contacts"] = {}
101
+ skipped = 0
102
+ with src.cursor() as sc, tgt.cursor() as tc:
103
+ sc.execute("SELECT id, first_name, last_name, email, phone, account_id, created_at FROM contacts")
104
+ count = 0
105
+ for row in sc:
106
+ old_id, first_name, last_name, email, phone, old_account_id, created_at = row
107
+
108
+ # Resolve FK via pk_map
109
+ new_account_id = pk_map["accounts"].get(old_account_id)
110
+ if old_account_id and not new_account_id:
111
+ print(f" Skipping contact {old_id}: account {old_account_id} not found in mapping")
112
+ skipped += 1
113
+ continue
114
+
115
+ new_id = cuid()
116
+ pk_map["contacts"][old_id] = new_id
117
+ tc.execute(
118
+ """
119
+ INSERT INTO crm.contact
120
+ (contact_id, first_name, last_name, email, phone, account_id,
121
+ created_date, created_by, last_updated, last_updated_by)
122
+ VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
123
+ """,
124
+ (new_id, first_name, last_name, email, phone, new_account_id,
125
+ created_at or datetime.utcnow(), migration_user,
126
+ created_at or datetime.utcnow(), migration_user),
127
+ )
128
+ count += 1
129
+ print(f" {count} contacts migrated, {skipped} skipped")
130
+
131
+
132
+ # … migrate_products, migrate_opportunities, migrate_opportunity_lines, migrate_activities follow the same pattern
133
+
134
+
135
+ if __name__ == "__main__":
136
+ main()
137
+ ```
138
+
139
+ ## Script template — Pure SQL (Postgres → Postgres via dblink)
140
+
141
+ When both databases are PostgreSQL and you want the fastest possible migration, use `dblink` or `postgres_fdw`. This requires the `dblink` extension to be installed on the target.
142
+
143
+ ```sql
144
+ -- migrate.sql — run against the dForge tenant DB
145
+ -- Connect to the legacy source via dblink
146
+
147
+ BEGIN;
148
+
149
+ -- Ensure dblink is available
150
+ CREATE EXTENSION IF NOT EXISTS dblink;
151
+
152
+ -- Create a PK mapping temp table
153
+ CREATE TEMP TABLE pk_map (
154
+ source_table text NOT NULL,
155
+ source_id bigint NOT NULL,
156
+ target_id text NOT NULL,
157
+ PRIMARY KEY (source_table, source_id)
158
+ );
159
+
160
+ -- Phase 1: Accounts
161
+ INSERT INTO crm.account
162
+ (account_id, name, phone, website, created_date, created_by, last_updated, last_updated_by)
163
+ SELECT
164
+ gen_random_uuid()::text,
165
+ src.name,
166
+ src.phone,
167
+ src.website,
168
+ COALESCE(src.created_at, NOW()),
169
+ '00000000-0000-0000-0000-000000000001'::text,
170
+ COALESCE(src.created_at, NOW()),
171
+ '00000000-0000-0000-0000-000000000001'::text
172
+ FROM dblink(
173
+ 'host=legacy.local dbname=old_crm user=readonly',
174
+ 'SELECT id, name, phone, website, created_at FROM accounts'
175
+ ) AS src(id bigint, name text, phone text, website text, created_at timestamp)
176
+ RETURNING account_id;
177
+
178
+ -- Record mappings for FK resolution
179
+ -- (In practice, do this in the same INSERT using a CTE with RETURNING)
180
+
181
+ -- Phase 2: Contacts (needs account FK resolution)
182
+ -- …
183
+
184
+ COMMIT;
185
+ ```
186
+
187
+ Pure SQL is faster but harder to read and debug. **Prefer the Python script unless the dataset is very large.**
188
+
189
+ ## Key rules
190
+
191
+ ### PK generation
192
+
193
+ Source uses integer PKs, dForge uses cuid (string). Generate a **new cuid for every row** and record it in the mapping table. Never preserve source integer PKs.
194
+
195
+ ### FK resolution order
196
+
197
+ Topologically sort the entities by FK dependencies. Parents first, then children, then grandchildren. If the source schema has cycles, break them: insert parents with NULL FKs first, then UPDATE the FKs in a second pass.
198
+
199
+ ### Audit fields
200
+
201
+ Set `created_by` / `last_updated_by` to a designated migration user. The zero user (`00000000-0000-0000-0000-000000000001`) is a common convention for "system-generated" rows. Ask the user if you're unsure which user to attribute to. (Note: `created_by`/`last_updated_by` only exist if the entity uses `audit-full` trait; with just `audit`, only timestamps exist.)
202
+
203
+ Set `created_date` / `last_updated` from source timestamps if available, otherwise `NOW()`. Note: dForge audit column names are `created_date` and `last_updated` — NOT `created_at`/`updated_at`.
204
+
205
+ ### Enum conversion
206
+
207
+ If the schema importer converted a lookup table (like `statuses`) to a `dropdown` column, the migration script must map source integer FK values to the new string option values. Read `params.options` on the target column and the importer's `MIGRATION_NOTES.md` to find the mapping.
208
+
209
+ ### NULL handling
210
+
211
+ Source column is nullable, but the target column has `NOT NULL` (because the importer detected `NOT NULL` or the user made it mandatory). The script should either:
212
+
213
+ - Provide a sensible default, or
214
+ - Skip the row and log it
215
+
216
+ Never silently insert NULL into a NOT NULL column — it will fail.
217
+
218
+ ### Type conversions
219
+
220
+ Common cases:
221
+
222
+ - `varchar(50)` → `varchar(100)` — safe
223
+ - `varchar(200)` → `varchar(100)` — **detect and truncate + warn, or skip + log**
224
+ - `int` enum → `dropdown` string — use the importer's option map
225
+ - Latin-1 → UTF-8 — re-encode bytes
226
+ - Naive timestamp → timestamptz — assume a tz (ask user) or keep naive
227
+
228
+ ### Idempotency
229
+
230
+ Wrap the migration in a transaction. A failed run rolls back cleanly. For re-runs, either:
231
+
232
+ 1. Truncate target tables before running (destructive — explicit user approval)
233
+ 2. Use `ON CONFLICT DO NOTHING` with deterministic PKs derived from source (e.g. `md5(source_id)`)
234
+ 3. Simply drop and reinstall the module, then re-run the migration
235
+
236
+ Default: **transaction + fail-fast**. Let the user re-run after fixing whatever failed.
237
+
238
+ ## Dry-run mode
239
+
240
+ **Always** include `--dry-run`. In dry-run mode:
241
+
242
+ 1. Connect to both DBs
243
+ 2. Run all SELECTs
244
+ 3. Do all transforms
245
+ 4. Print a summary: `"Would insert 1,243 accounts, 87 contacts, 3,412 activities. 12 rows would be skipped."`
246
+ 5. Rollback before committing
247
+
248
+ The user reviews, then re-runs without `--dry-run` to apply.
249
+
250
+ ## What NOT to do
251
+
252
+ - **Do not go through the dForge API.** The RPC stack is 100x slower than direct SQL and adds no safety for a one-shot migration. Direct INSERTs into the tenant DB are the right tool.
253
+ - **Do not skip dry-run mode.** It's the only safe way to preview.
254
+ - **Do not migrate to production without multiple explicit confirmations.** Dev/local only by default. If the user insists on production, make the script check `--i-know-this-is-production` and print a scary warning.
255
+ - **Do not hardcode credentials.** Pass via arguments or env vars.
256
+ - **Do not silently drop rows.** Every skipped row gets logged to stderr with a reason.
257
+ - **Do not assume target tables exist.** Check first; bail if the module isn't installed.
258
+ - **Do not migrate views, stored procs, or triggers.** Those are listed in `MIGRATION_NOTES.md` and converted manually to dForge constructs (formulas, actions, reports).
259
+
260
+ ## Produce a MIGRATION.md alongside the script
261
+
262
+ When you generate the migration script, also generate (or update) a `MIGRATION.md` in the module folder with:
263
+
264
+ - What the script does in plain English
265
+ - How to run it (example command lines for dry-run and actual run)
266
+ - What gets migrated and what doesn't
267
+ - Known caveats specific to this dataset
268
+ - Rollback instructions
269
+
270
+ The user reads `MIGRATION.md` first, then runs the script.