@dforge-core/dforge-mcp 0.1.0-rc.11 → 0.1.0-rc.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +132 -0
- package/README.md +75 -21
- package/dist/server.js +2405 -272
- package/docs/creating-modules.md +7 -3
- package/package.json +11 -6
- package/resources/docs/conventions.md +20 -14
- package/resources/schemas/entity.schema.json +8 -1
- package/resources/schemas/reports.schema.json +4 -0
- package/skills/dforge-mcp-author/SKILL.md +227 -95
- package/skills/dforge-mcp-author/examples/simple-todo/README.md +38 -0
- package/skills/dforge-mcp-author/examples/simple-todo/entities/todo_item.json +83 -0
- package/skills/dforge-mcp-author/examples/simple-todo/entities/todo_list.json +43 -0
- package/skills/dforge-mcp-author/examples/simple-todo/logic/actions/mark_done.dsl +6 -0
- package/skills/dforge-mcp-author/examples/simple-todo/manifest.json +32 -0
- package/skills/dforge-mcp-author/examples/simple-todo/security/roles.json +10 -0
- package/skills/dforge-mcp-author/examples/simple-todo/seed-data/01-lists.json +17 -0
- package/skills/dforge-mcp-author/examples/simple-todo/ui/actions.json +11 -0
- package/skills/dforge-mcp-author/examples/simple-todo/ui/data_views.json +35 -0
- package/skills/dforge-mcp-author/examples/simple-todo/ui/menus.json +28 -0
- package/skills/dforge-mcp-author/references/action-dsl.md +397 -0
- package/skills/dforge-mcp-author/references/column-types.md +168 -0
- package/skills/dforge-mcp-author/references/conventions.md +177 -0
- package/skills/dforge-mcp-author/references/data-migration.md +270 -0
- package/skills/dforge-mcp-author/references/data-views.md +192 -0
- package/skills/dforge-mcp-author/references/excel-import.md +61 -0
- package/skills/dforge-mcp-author/references/field-types.md +144 -0
- package/skills/dforge-mcp-author/references/filters.md +326 -0
- package/skills/dforge-mcp-author/references/flags.md +73 -0
- package/skills/dforge-mcp-author/references/formulas.md +206 -0
- package/skills/dforge-mcp-author/references/jobs.md +149 -0
- package/skills/dforge-mcp-author/references/manifest.md +123 -0
- package/skills/dforge-mcp-author/references/menus.md +164 -0
- package/skills/dforge-mcp-author/references/number-sequences.md +117 -0
- package/skills/dforge-mcp-author/references/print-templates.md +159 -0
- package/skills/dforge-mcp-author/references/queries.md +312 -0
- package/skills/dforge-mcp-author/references/reports.md +398 -0
- package/skills/dforge-mcp-author/references/schema-import.md +331 -0
- package/skills/dforge-mcp-author/references/security.md +244 -0
- package/skills/dforge-mcp-author/references/settings.md +120 -0
- package/skills/dforge-mcp-author/references/traits.md +153 -0
- package/skills/dforge-mcp-author/references/translations.md +158 -0
- package/skills/dforge-mcp-author/references/validation-checklist.md +182 -0
- package/skills/dforge-mcp-author/scripts/xlsx_to_model.py +198 -0
|
@@ -0,0 +1,168 @@
|
|
|
1
|
+
# Column Types Reference
|
|
2
|
+
|
|
3
|
+
Every column has a `columnType` (optional for the default physical type). Seven values exist:
|
|
4
|
+
|
|
5
|
+
| `columnType` | Physical DB column? | Description |
|
|
6
|
+
|---|---|---|
|
|
7
|
+
| (omitted) or `"D"` | Yes | **Data column** — default. Maps to a real SQL column. |
|
|
8
|
+
| `"R"` | No | **Reference column** — virtual N:1 lookup. Paired with a hidden FK column (see FK+Reference pattern). |
|
|
9
|
+
| `"S"` | No | **Set column** — virtual 1:N backwards reference. Used for detail grids. |
|
|
10
|
+
| `"F"` | No | **Formula column** — virtual computed value. Evaluated by the formula engine. |
|
|
11
|
+
| `"A"` | Yes | **Accumulation register column** — stores posted state for accumulation registers. Advanced (accounting modules). |
|
|
12
|
+
| `"L"` | Yes | **Ledger register column** — stores posted state for double-entry bookkeeping. Advanced (accounting modules). |
|
|
13
|
+
| `"G"` | Yes | **Generated column** — DB-level computed aggregate (e.g. SUM over child set via trigger, or PostgreSQL `GENERATED ALWAYS AS`). The aggregated child column **must be physical** (a `D` column) — never a virtual `F`/`R`/`S` column. For a simple roll-up total, prefer `F` (see below). |
|
|
14
|
+
|
|
15
|
+
**For most module development, you'll only use D, R, S, and F.** Types A, L, and G are for advanced accounting/registry modules that use the `postable`, `accumulation`, or `ledger` traits.
|
|
16
|
+
|
|
17
|
+
## Data columns (`columnType` omitted or `"D"`)
|
|
18
|
+
|
|
19
|
+
Most columns. They map 1:1 to a physical SQL column.
|
|
20
|
+
|
|
21
|
+
```json
|
|
22
|
+
"first_name": {
|
|
23
|
+
"dbDatatype": "varchar",
|
|
24
|
+
"fieldTypeCd": "text",
|
|
25
|
+
"flags": "VEM",
|
|
26
|
+
"maxLen": 100,
|
|
27
|
+
"orderNum": 20,
|
|
28
|
+
"description": "First Name"
|
|
29
|
+
}
|
|
30
|
+
```
|
|
31
|
+
|
|
32
|
+
## Reference columns (`columnType: "R"`)
|
|
33
|
+
|
|
34
|
+
Virtual N:1 lookup. No physical column. Always **paired with a hidden FK column** that does hold the physical data. Together they form the **FK+Reference pattern**.
|
|
35
|
+
|
|
36
|
+
```json
|
|
37
|
+
// Hidden FK column — physical, hidden
|
|
38
|
+
"account_id": {
|
|
39
|
+
"dbDatatype": "cuid",
|
|
40
|
+
"flags": "EM",
|
|
41
|
+
"orderNum": 90,
|
|
42
|
+
"description": "Account ID"
|
|
43
|
+
},
|
|
44
|
+
// Visible Reference column — virtual, shown as lookup picker
|
|
45
|
+
"account": {
|
|
46
|
+
"columnType": "R",
|
|
47
|
+
"fieldTypeCd": "lookup",
|
|
48
|
+
"flags": "VEM",
|
|
49
|
+
"orderNum": 95,
|
|
50
|
+
"description": "Account",
|
|
51
|
+
"link": {
|
|
52
|
+
"entity": "account",
|
|
53
|
+
"thisKey": "account_id",
|
|
54
|
+
"otherKey": "account_id"
|
|
55
|
+
}
|
|
56
|
+
}
|
|
57
|
+
```
|
|
58
|
+
|
|
59
|
+
Plus declare the FK constraint in the entity's `references` block:
|
|
60
|
+
|
|
61
|
+
```json
|
|
62
|
+
"references": {
|
|
63
|
+
"FK_Contact_Account": {
|
|
64
|
+
"from": { "field": "account_id" },
|
|
65
|
+
"to": { "entity": "account", "field": "account_id" }
|
|
66
|
+
}
|
|
67
|
+
}
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
> **FK column `dbDatatype` must exactly match the referenced entity's PK `dbDatatype`.** Never guess.
|
|
71
|
+
> - Entities using the `identity` trait → PK is `dbDatatype: "cuid"` → FK column must also be `"cuid"`
|
|
72
|
+
> - Cross-module or legacy entities → call `dforge_module_inspect` on the referenced module and read the PK column's `dbDatatype` before declaring the FK
|
|
73
|
+
> - **`bigint`, `integer`, `int8` are wrong values for FK columns** — even though `cuid` is physically stored as int8, the platform type name is `cuid`, not `bigint`
|
|
74
|
+
|
|
75
|
+
**Why two columns?** The FK stores the actual value. The Reference column configures how the value is rendered in the UI (as a typeahead picker showing the target's `toString`). They must be kept in sync — column order in JSON doesn't matter but `orderNum` determines UI ordering.
|
|
76
|
+
|
|
77
|
+
## Set columns (`columnType: "S"`)
|
|
78
|
+
|
|
79
|
+
Virtual 1:N backwards reference. Shows a grid of related records on the parent's detail view.
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
"activities": {
|
|
83
|
+
"columnType": "S",
|
|
84
|
+
"fieldTypeCd": "grid",
|
|
85
|
+
"flags": "VEM",
|
|
86
|
+
"orderNum": 150,
|
|
87
|
+
"description": "Activities",
|
|
88
|
+
"link": {
|
|
89
|
+
"entity": "activity",
|
|
90
|
+
"thisKey": "contact_id", // column on THIS entity (parent PK)
|
|
91
|
+
"otherKey": "contact_id" // column on the OTHER entity (FK back to parent)
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
`link.entity` is the child entity. `link.thisKey` is the column on the current entity (typically the PK). `link.otherKey` is the FK column on the child entity pointing back to the parent.
|
|
97
|
+
|
|
98
|
+
Set columns are **not** declared in the `references` block — the FK already lives on the child entity.
|
|
99
|
+
|
|
100
|
+
## Formula columns (`columnType: "F"`)
|
|
101
|
+
|
|
102
|
+
Virtual computed column. No physical storage. Value is calculated at runtime by the formula engine.
|
|
103
|
+
|
|
104
|
+
```json
|
|
105
|
+
"full_name": {
|
|
106
|
+
"columnType": "F",
|
|
107
|
+
"fieldTypeCd": "text",
|
|
108
|
+
"baseDatatypeCd": "string",
|
|
109
|
+
"flags": "V",
|
|
110
|
+
"orderNum": 25,
|
|
111
|
+
"formula": "[first_name] + ' ' + [last_name]",
|
|
112
|
+
"description": "Full Name"
|
|
113
|
+
}
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
**Required fields for formula columns**:
|
|
117
|
+
|
|
118
|
+
- `columnType: "F"`
|
|
119
|
+
- `fieldTypeCd` — how to render it
|
|
120
|
+
- `baseDatatypeCd` — **required** on formula columns even though there's no physical column; used for type inference
|
|
121
|
+
- `flags: "V"` (usually — formula columns are almost always read-only)
|
|
122
|
+
- `formula` — the formula expression
|
|
123
|
+
- `orderNum`
|
|
124
|
+
- `description`
|
|
125
|
+
|
|
126
|
+
Do **not** include `dbDatatype` on formula columns (there's no physical storage).
|
|
127
|
+
|
|
128
|
+
See `formulas.md` for syntax.
|
|
129
|
+
|
|
130
|
+
## Roll-up totals over child rows — use `F`, not `G`
|
|
131
|
+
|
|
132
|
+
To total a child set on a header (line items → order total, movements → quantity on hand),
|
|
133
|
+
**use a Formula column (`"F"`) with `SUM([set].[field])`**. It is computed at query time, so it
|
|
134
|
+
can reference **any** child column — including the child's own formula columns.
|
|
135
|
+
|
|
136
|
+
```json
|
|
137
|
+
// purchase_order.total_amount — query-time roll-up (installs cleanly)
|
|
138
|
+
"total_amount": {
|
|
139
|
+
"columnType": "F",
|
|
140
|
+
"fieldTypeCd": "currency",
|
|
141
|
+
"baseDatatypeCd": "number",
|
|
142
|
+
"flags": "V",
|
|
143
|
+
"orderNum": 90,
|
|
144
|
+
"formula": "SUM([lines].[line_total])", // line_total may itself be a formula column
|
|
145
|
+
"description": "Total Amount"
|
|
146
|
+
}
|
|
147
|
+
```
|
|
148
|
+
|
|
149
|
+
> ⛔ **Never make a `G` (Generated) column aggregate a virtual `F` column.** A `G` aggregate is
|
|
150
|
+
> maintained by a database trigger that reads the child's `OLD`/`NEW` *physical* values. A formula
|
|
151
|
+
> (`F`) column has no physical storage, so the trigger references a column that doesn't exist and
|
|
152
|
+
> **install fails** with `db_error: column old.<field> does not exist`. If you need a stored
|
|
153
|
+
> (`G`) aggregate over a *derived* quantity (e.g. a signed movement quantity), make that child
|
|
154
|
+
> column **physical** first — either a plain `D` column written by the action logic, or a
|
|
155
|
+
> same-row `G`/`GENERATED ALWAYS AS` column with a `dbDatatype` — then aggregate that. For most
|
|
156
|
+
> modules a query-time `F` roll-up is the right choice; reserve `G` for high-volume stored aggregates.
|
|
157
|
+
|
|
158
|
+
## Quick reference: which columnType do I use?
|
|
159
|
+
|
|
160
|
+
- **Plain text/number/date/etc. field that stores a value** → omit `columnType` (default `"D"`)
|
|
161
|
+
- **"This contact belongs to one account"** → `"R"` (plus hidden FK)
|
|
162
|
+
- **"This account has many contacts"** → `"S"` on the account side
|
|
163
|
+
- **"Compute full_name from first_name + last_name"** → `"F"`
|
|
164
|
+
- **"User picker" (points to admin.user)** → omit `columnType`, use `fieldTypeCd: "user"` (writes user ID directly, no paired FK)
|
|
165
|
+
- **"Polymorphic link to any entity"** → omit `columnType`, use `fieldTypeCd: "entitylink"` (stored as JSON)
|
|
166
|
+
- **"Roll-up total over child rows"** (line items → header total, movements → quantity on hand) → `"F"` with `SUM([set].[field])` (query-time; see "Roll-up totals" above). Reserve `"G"` for high-volume *stored* aggregates, and only over a **physical** child column — never over an `F` column.
|
|
167
|
+
- **"Accumulation register state"** → `"A"` (advanced accounting — requires `postable` + `accumulation` traits)
|
|
168
|
+
- **"Double-entry ledger state"** → `"L"` (advanced accounting — requires `postable` + `ledger` traits)
|
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# Module Conventions Reference
|
|
2
|
+
|
|
3
|
+
Rules that apply across the entire module package. Violating these produces technically-valid but non-idiomatic modules, and in several cases breaks the installer.
|
|
4
|
+
|
|
5
|
+
## Naming
|
|
6
|
+
|
|
7
|
+
| Thing | Convention | Example |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| Module `code` | lowercase, letters/digits/underscore | `crm`, `my_module`, `wms_fin` |
|
|
10
|
+
| Entity keys / `dbObject` | `snake_case`, singular | `contact`, `opportunity_line` |
|
|
11
|
+
| Column codes | `snake_case` | `first_name`, `account_id` |
|
|
12
|
+
| Data view codes | `snake_case` | `contact_list`, `deal_kanban` |
|
|
13
|
+
| Report codes | `snake_case` | `sales_pipeline` |
|
|
14
|
+
| Action codes | `snake_case` | `change_stage`, `send_welcome` |
|
|
15
|
+
| Menu codes | `snake_case` | `sales`, `pipeline` |
|
|
16
|
+
| Role codes | `snake_case`, domain-specific | `sales_rep`, `sales_admin` |
|
|
17
|
+
| Setting codes | `snake_case` | `vat_rate`, `invoice_prefix` |
|
|
18
|
+
| Trait names | lowercase single word | `identity`, `audit` |
|
|
19
|
+
| Seed data files | numbered prefix | `01-countries.json`, `02-currencies.json` |
|
|
20
|
+
| Action DSL files | `<action_code>.dsl` | `change_stage.dsl` |
|
|
21
|
+
| Reference keys | `FK_<FromEntity>_<Purpose>` | `FK_Contact_Account` |
|
|
22
|
+
|
|
23
|
+
Everything is **case-sensitive**. Don't mix snake_case and camelCase.
|
|
24
|
+
|
|
25
|
+
## File layout
|
|
26
|
+
|
|
27
|
+
Standard module directory structure:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
my_module/
|
|
31
|
+
├── manifest.json
|
|
32
|
+
├── README.md # module overview
|
|
33
|
+
├── CHANGELOG.md # version history
|
|
34
|
+
├── MODULE-INFO.md # user-facing intro (shown in module picker)
|
|
35
|
+
├── entities/
|
|
36
|
+
│ └── <entity>.json
|
|
37
|
+
├── logic/
|
|
38
|
+
│ └── actions/
|
|
39
|
+
│ └── <action>.dsl
|
|
40
|
+
├── ui/
|
|
41
|
+
│ ├── data_views.json
|
|
42
|
+
│ ├── menus.json
|
|
43
|
+
│ ├── actions.json
|
|
44
|
+
│ ├── folders.json
|
|
45
|
+
│ └── reports/
|
|
46
|
+
│ └── <report>.json
|
|
47
|
+
├── security/
|
|
48
|
+
│ └── roles.json
|
|
49
|
+
├── settings.json
|
|
50
|
+
├── seed-data/
|
|
51
|
+
│ ├── 01-<first>.json
|
|
52
|
+
│ └── 02-<second>.json
|
|
53
|
+
├── translations/
|
|
54
|
+
│ └── de-DE.json # IETF tags, one file per non-English locale
|
|
55
|
+
├── print_templates/
|
|
56
|
+
│ └── <template>.scriban
|
|
57
|
+
├── files/ # static assets
|
|
58
|
+
└── webhooks.json # optional
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Only include what your module actually uses — not every module needs `print_templates/` or `webhooks.json`.
|
|
62
|
+
|
|
63
|
+
## The FK+Reference pattern (again, because it's critical)
|
|
64
|
+
|
|
65
|
+
**Every reference to another entity is two columns**:
|
|
66
|
+
|
|
67
|
+
1. Hidden FK column with `flags: "EM"` and `dbDatatype` matching the target PK type
|
|
68
|
+
2. Visible Reference column with `columnType: "R"`, `fieldTypeCd: "lookup"`, `flags: "VEM"`, and a `link` object pointing at the target
|
|
69
|
+
|
|
70
|
+
Plus the actual FK constraint declared in the entity's `references` block.
|
|
71
|
+
|
|
72
|
+
See `column-types.md` for the full example.
|
|
73
|
+
|
|
74
|
+
## Bridge modules
|
|
75
|
+
|
|
76
|
+
When two core modules need to integrate (e.g. CRM and Finance), create a **bridge module**:
|
|
77
|
+
|
|
78
|
+
- Name it with a hyphen: `crm-fin`, `wms-fin`, `crm-pricing`
|
|
79
|
+
- Depends on **both** core modules
|
|
80
|
+
- Owns the integration: extension entities, cross-module actions, cross-module data views
|
|
81
|
+
- Core modules stay independent and don't know about each other
|
|
82
|
+
|
|
83
|
+
Extension entities in a bridge module have `"extends": "fin.invoice"` at the top. Physical columns in the extension go into a 1:1 ext table.
|
|
84
|
+
|
|
85
|
+
## Seed data ordering
|
|
86
|
+
|
|
87
|
+
Seed files run in numeric prefix order. Put entities with FK dependencies **after** their targets:
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
seed-data/
|
|
91
|
+
├── 01-currencies.json # no FKs, insert first
|
|
92
|
+
├── 02-countries.json # references currency
|
|
93
|
+
├── 03-regions.json # references country
|
|
94
|
+
└── 04-offices.json # references region
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Each seed file has two top-level keys: `"entityCode"` (NOT `"entity"`) and `"records"` (array). **Using `"entity"` instead of `"entityCode"` is a silent failure** — the installer reads an empty entity code and the INSERT fails.
|
|
98
|
+
|
|
99
|
+
Use explicit **numeric** PKs in seed data so cross-file references work. The `identity` trait creates `cuid` columns which are `int8` (bigint) — **NOT UUIDs**. Use simple integers like `1001`, `1002`:
|
|
100
|
+
|
|
101
|
+
```json
|
|
102
|
+
{
|
|
103
|
+
"entityCode": "country",
|
|
104
|
+
"records": [
|
|
105
|
+
{ "country_id": 1001, "code": "US", "name": "United States" },
|
|
106
|
+
{ "country_id": 1002, "code": "DE", "name": "Germany" }
|
|
107
|
+
]
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
**Common mistake**: using UUID strings like `"10000000-0000-0000-0000-000000000001"` — **wrong**. The `cuid` datatype is a numeric `int8`, not a UUID string. Seed data PKs must be numbers. Use a numbering scheme that avoids collisions (e.g. 1001-1099 for countries, 2001-2099 for regions, etc.).
|
|
112
|
+
|
|
113
|
+
## Translations
|
|
114
|
+
|
|
115
|
+
Translation files live in `translations/` and are named after the **IETF locale tag** (`de-DE.json`, `uk-UA.json`, etc.). Filenames are auto-discovered; declare which ones you ship in `manifest.supportedLocales`. English is the default — labels and descriptions live in entity/view JSON, so **don't** create `en.json` or `en-US.json` (they're skipped by the installer).
|
|
116
|
+
|
|
117
|
+
```json
|
|
118
|
+
// translations/de-DE.json
|
|
119
|
+
{
|
|
120
|
+
"contact": "Kontakt",
|
|
121
|
+
"account": "Kunde",
|
|
122
|
+
"contact_list.label": "Alle Kontakte"
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Missing keys fall back to the English value declared in the source JSON.
|
|
127
|
+
|
|
128
|
+
## Versioning — always bump before packaging
|
|
129
|
+
|
|
130
|
+
The manifest has **two version numbers** that must be managed:
|
|
131
|
+
|
|
132
|
+
| Field | When to bump | Example |
|
|
133
|
+
|---|---|---|
|
|
134
|
+
| `version` | **Every release** — any change at all (bug fix, feature, schema change) | `0.1.0` → `0.2.0` |
|
|
135
|
+
| `dbSchemaVersion` | **Only when the DB schema changes** — new entity, new column, removed column, changed type, new constraint, new index | `0.0.1` → `0.0.2` |
|
|
136
|
+
|
|
137
|
+
Both use **semver** (`MAJOR.MINOR.PATCH`).
|
|
138
|
+
|
|
139
|
+
**Rules**:
|
|
140
|
+
|
|
141
|
+
- If you added an action but no new columns → bump `version` only, keep `dbSchemaVersion`.
|
|
142
|
+
- If you added a new entity or column → bump both `version` and `dbSchemaVersion`.
|
|
143
|
+
- If you changed a view or menu → bump `version` only.
|
|
144
|
+
- If you changed translations only → bump `version` only.
|
|
145
|
+
- The installer uses `dbSchemaVersion` to decide whether to run schema migrations. Wrong version = skipped migrations.
|
|
146
|
+
- **Never ship a package without bumping at least `version`** from the previous release. The installer may reject or silently skip a re-install with the same version.
|
|
147
|
+
- For brand-new modules, start at `version: "0.1.0"` and `dbSchemaVersion: "0.0.1"`.
|
|
148
|
+
|
|
149
|
+
## `orderNum` — always set it
|
|
150
|
+
|
|
151
|
+
On every column, data view, menu item, setting, and action that appears in a list. Without `orderNum`, UI ordering is undefined. Use widely-spaced values (10, 20, 30, 40) so you can insert between them later.
|
|
152
|
+
|
|
153
|
+
## Consistency across the module
|
|
154
|
+
|
|
155
|
+
- All timestamps use the same trait (`audit`)
|
|
156
|
+
- All primary keys use the same trait (`identity`) with consistent PK type (`cuid`)
|
|
157
|
+
- All entities have `toString`
|
|
158
|
+
- All views use `dataSources` array
|
|
159
|
+
- All roles use `rights`
|
|
160
|
+
- All menus use nested dicts + `dataViewCode`
|
|
161
|
+
- All actions have `canExecute:` (even if it's just `true`)
|
|
162
|
+
|
|
163
|
+
## Do not invent
|
|
164
|
+
|
|
165
|
+
- Do not invent field types (`rating`, `geolocation`, `signature`) — not in the catalog
|
|
166
|
+
- Do not invent column types beyond `D` / `R` / `S` / `F`
|
|
167
|
+
- Do not invent role rights letters beyond `SIUDC` + `E`
|
|
168
|
+
- Do not invent view types beyond the registered ones
|
|
169
|
+
- Do not invent menu item types beyond `V` / `R` / `D` / `A`
|
|
170
|
+
- Do not invent DSL functions
|
|
171
|
+
- Do not invent formula functions
|
|
172
|
+
|
|
173
|
+
If you genuinely need something new, **say so** and ask the user how to work around it with existing primitives.
|
|
174
|
+
|
|
175
|
+
## Reference
|
|
176
|
+
|
|
177
|
+
This file summarises the key conventions. If you need to verify a pattern not covered here, ask the user or check the reference modules in the `examples/` directory.
|
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# Data Migration Reference
|
|
2
|
+
|
|
3
|
+
When the user has an existing database (typically after running the schema importer to produce a rough-cut module) and wants to **move the actual rows** into dForge, write them a migration script they can run locally.
|
|
4
|
+
|
|
5
|
+
This assumes:
|
|
6
|
+
- dForge is running locally (e.g. `docker compose up`).
|
|
7
|
+
- The legacy database is reachable from the developer's machine.
|
|
8
|
+
- The target is a **dev tenant**, never production.
|
|
9
|
+
|
|
10
|
+
## Script template — Python / psycopg
|
|
11
|
+
|
|
12
|
+
This is the most portable choice. Works against any source database with a Python driver, writes to dForge Postgres.
|
|
13
|
+
|
|
14
|
+
```python
|
|
15
|
+
#!/usr/bin/env python3
|
|
16
|
+
"""
|
|
17
|
+
Migration script: old_crm → dForge my_crm module
|
|
18
|
+
Generated on 2026-04-08
|
|
19
|
+
"""
|
|
20
|
+
|
|
21
|
+
import argparse
|
|
22
|
+
import sys
|
|
23
|
+
import uuid
|
|
24
|
+
import psycopg
|
|
25
|
+
from datetime import datetime
|
|
26
|
+
from typing import Dict
|
|
27
|
+
|
|
28
|
+
MIGRATION_USER_ID = "00000000-0000-0000-0000-000000000001" # override with --migration-user
|
|
29
|
+
|
|
30
|
+
def cuid() -> str:
|
|
31
|
+
"""Generate a new cuid-like identifier for target PKs."""
|
|
32
|
+
return str(uuid.uuid4())
|
|
33
|
+
|
|
34
|
+
def main():
|
|
35
|
+
parser = argparse.ArgumentParser()
|
|
36
|
+
parser.add_argument("--source", required=True, help="Source DB connection string")
|
|
37
|
+
parser.add_argument("--target", required=True, help="Target dForge tenant DB connection string")
|
|
38
|
+
parser.add_argument("--migration-user", default=MIGRATION_USER_ID)
|
|
39
|
+
parser.add_argument("--dry-run", action="store_true", help="Simulate without committing")
|
|
40
|
+
args = parser.parse_args()
|
|
41
|
+
|
|
42
|
+
src = psycopg.connect(args.source)
|
|
43
|
+
tgt = psycopg.connect(args.target)
|
|
44
|
+
|
|
45
|
+
try:
|
|
46
|
+
with tgt.transaction():
|
|
47
|
+
pk_map: Dict[str, Dict[int, str]] = {} # source_table → {source_id → target_id}
|
|
48
|
+
|
|
49
|
+
# Phase 1: parents (no FK deps)
|
|
50
|
+
migrate_accounts(src, tgt, pk_map, args.migration_user)
|
|
51
|
+
migrate_products(src, tgt, pk_map, args.migration_user)
|
|
52
|
+
|
|
53
|
+
# Phase 2: children (depend on parents)
|
|
54
|
+
migrate_contacts(src, tgt, pk_map, args.migration_user)
|
|
55
|
+
migrate_opportunities(src, tgt, pk_map, args.migration_user)
|
|
56
|
+
|
|
57
|
+
# Phase 3: grandchildren
|
|
58
|
+
migrate_opportunity_lines(src, tgt, pk_map, args.migration_user)
|
|
59
|
+
migrate_activities(src, tgt, pk_map, args.migration_user)
|
|
60
|
+
|
|
61
|
+
if args.dry_run:
|
|
62
|
+
print("[DRY RUN] Rolling back.")
|
|
63
|
+
raise SystemExit(0)
|
|
64
|
+
|
|
65
|
+
print("Migration committed successfully.")
|
|
66
|
+
|
|
67
|
+
except Exception as e:
|
|
68
|
+
print(f"Migration failed: {e}", file=sys.stderr)
|
|
69
|
+
sys.exit(1)
|
|
70
|
+
finally:
|
|
71
|
+
src.close()
|
|
72
|
+
tgt.close()
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
def migrate_accounts(src, tgt, pk_map, migration_user):
|
|
76
|
+
print("Migrating accounts…")
|
|
77
|
+
pk_map["accounts"] = {}
|
|
78
|
+
with src.cursor() as sc, tgt.cursor() as tc:
|
|
79
|
+
sc.execute("SELECT id, name, phone, website, created_at FROM accounts")
|
|
80
|
+
count = 0
|
|
81
|
+
for row in sc:
|
|
82
|
+
old_id, name, phone, website, created_at = row
|
|
83
|
+
new_id = cuid()
|
|
84
|
+
pk_map["accounts"][old_id] = new_id
|
|
85
|
+
tc.execute(
|
|
86
|
+
"""
|
|
87
|
+
INSERT INTO crm.account
|
|
88
|
+
(account_id, name, phone, website, created_date, created_by, last_updated, last_updated_by)
|
|
89
|
+
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
|
90
|
+
""",
|
|
91
|
+
(new_id, name, phone, website, created_at or datetime.utcnow(),
|
|
92
|
+
migration_user, created_at or datetime.utcnow(), migration_user),
|
|
93
|
+
)
|
|
94
|
+
count += 1
|
|
95
|
+
print(f" {count} accounts migrated")
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def migrate_contacts(src, tgt, pk_map, migration_user):
|
|
99
|
+
print("Migrating contacts…")
|
|
100
|
+
pk_map["contacts"] = {}
|
|
101
|
+
skipped = 0
|
|
102
|
+
with src.cursor() as sc, tgt.cursor() as tc:
|
|
103
|
+
sc.execute("SELECT id, first_name, last_name, email, phone, account_id, created_at FROM contacts")
|
|
104
|
+
count = 0
|
|
105
|
+
for row in sc:
|
|
106
|
+
old_id, first_name, last_name, email, phone, old_account_id, created_at = row
|
|
107
|
+
|
|
108
|
+
# Resolve FK via pk_map
|
|
109
|
+
new_account_id = pk_map["accounts"].get(old_account_id)
|
|
110
|
+
if old_account_id and not new_account_id:
|
|
111
|
+
print(f" Skipping contact {old_id}: account {old_account_id} not found in mapping")
|
|
112
|
+
skipped += 1
|
|
113
|
+
continue
|
|
114
|
+
|
|
115
|
+
new_id = cuid()
|
|
116
|
+
pk_map["contacts"][old_id] = new_id
|
|
117
|
+
tc.execute(
|
|
118
|
+
"""
|
|
119
|
+
INSERT INTO crm.contact
|
|
120
|
+
(contact_id, first_name, last_name, email, phone, account_id,
|
|
121
|
+
created_date, created_by, last_updated, last_updated_by)
|
|
122
|
+
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
|
|
123
|
+
""",
|
|
124
|
+
(new_id, first_name, last_name, email, phone, new_account_id,
|
|
125
|
+
created_at or datetime.utcnow(), migration_user,
|
|
126
|
+
created_at or datetime.utcnow(), migration_user),
|
|
127
|
+
)
|
|
128
|
+
count += 1
|
|
129
|
+
print(f" {count} contacts migrated, {skipped} skipped")
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
# … migrate_products, migrate_opportunities, migrate_opportunity_lines, migrate_activities follow the same pattern
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
if __name__ == "__main__":
|
|
136
|
+
main()
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
## Script template — Pure SQL (Postgres → Postgres via dblink)
|
|
140
|
+
|
|
141
|
+
When both databases are PostgreSQL and you want the fastest possible migration, use `dblink` or `postgres_fdw`. This requires the `dblink` extension to be installed on the target.
|
|
142
|
+
|
|
143
|
+
```sql
|
|
144
|
+
-- migrate.sql — run against the dForge tenant DB
|
|
145
|
+
-- Connect to the legacy source via dblink
|
|
146
|
+
|
|
147
|
+
BEGIN;
|
|
148
|
+
|
|
149
|
+
-- Ensure dblink is available
|
|
150
|
+
CREATE EXTENSION IF NOT EXISTS dblink;
|
|
151
|
+
|
|
152
|
+
-- Create a PK mapping temp table
|
|
153
|
+
CREATE TEMP TABLE pk_map (
|
|
154
|
+
source_table text NOT NULL,
|
|
155
|
+
source_id bigint NOT NULL,
|
|
156
|
+
target_id text NOT NULL,
|
|
157
|
+
PRIMARY KEY (source_table, source_id)
|
|
158
|
+
);
|
|
159
|
+
|
|
160
|
+
-- Phase 1: Accounts
|
|
161
|
+
INSERT INTO crm.account
|
|
162
|
+
(account_id, name, phone, website, created_date, created_by, last_updated, last_updated_by)
|
|
163
|
+
SELECT
|
|
164
|
+
gen_random_uuid()::text,
|
|
165
|
+
src.name,
|
|
166
|
+
src.phone,
|
|
167
|
+
src.website,
|
|
168
|
+
COALESCE(src.created_at, NOW()),
|
|
169
|
+
'00000000-0000-0000-0000-000000000001'::text,
|
|
170
|
+
COALESCE(src.created_at, NOW()),
|
|
171
|
+
'00000000-0000-0000-0000-000000000001'::text
|
|
172
|
+
FROM dblink(
|
|
173
|
+
'host=legacy.local dbname=old_crm user=readonly',
|
|
174
|
+
'SELECT id, name, phone, website, created_at FROM accounts'
|
|
175
|
+
) AS src(id bigint, name text, phone text, website text, created_at timestamp)
|
|
176
|
+
RETURNING account_id;
|
|
177
|
+
|
|
178
|
+
-- Record mappings for FK resolution
|
|
179
|
+
-- (In practice, do this in the same INSERT using a CTE with RETURNING)
|
|
180
|
+
|
|
181
|
+
-- Phase 2: Contacts (needs account FK resolution)
|
|
182
|
+
-- …
|
|
183
|
+
|
|
184
|
+
COMMIT;
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Pure SQL is faster but harder to read and debug. **Prefer the Python script unless the dataset is very large.**
|
|
188
|
+
|
|
189
|
+
## Key rules
|
|
190
|
+
|
|
191
|
+
### PK generation
|
|
192
|
+
|
|
193
|
+
Source uses integer PKs, dForge uses cuid (string). Generate a **new cuid for every row** and record it in the mapping table. Never preserve source integer PKs.
|
|
194
|
+
|
|
195
|
+
### FK resolution order
|
|
196
|
+
|
|
197
|
+
Topologically sort the entities by FK dependencies. Parents first, then children, then grandchildren. If the source schema has cycles, break them: insert parents with NULL FKs first, then UPDATE the FKs in a second pass.
|
|
198
|
+
|
|
199
|
+
### Audit fields
|
|
200
|
+
|
|
201
|
+
Set `created_by` / `last_updated_by` to a designated migration user. The zero user (`00000000-0000-0000-0000-000000000001`) is a common convention for "system-generated" rows. Ask the user if you're unsure which user to attribute to. (Note: `created_by`/`last_updated_by` only exist if the entity uses `audit-full` trait; with just `audit`, only timestamps exist.)
|
|
202
|
+
|
|
203
|
+
Set `created_date` / `last_updated` from source timestamps if available, otherwise `NOW()`. Note: dForge audit column names are `created_date` and `last_updated` — NOT `created_at`/`updated_at`.
|
|
204
|
+
|
|
205
|
+
### Enum conversion
|
|
206
|
+
|
|
207
|
+
If the schema importer converted a lookup table (like `statuses`) to a `dropdown` column, the migration script must map source integer FK values to the new string option values. Read `params.options` on the target column and the importer's `MIGRATION_NOTES.md` to find the mapping.
|
|
208
|
+
|
|
209
|
+
### NULL handling
|
|
210
|
+
|
|
211
|
+
Source column is nullable, but the target column has `NOT NULL` (because the importer detected `NOT NULL` or the user made it mandatory). The script should either:
|
|
212
|
+
|
|
213
|
+
- Provide a sensible default, or
|
|
214
|
+
- Skip the row and log it
|
|
215
|
+
|
|
216
|
+
Never silently insert NULL into a NOT NULL column — it will fail.
|
|
217
|
+
|
|
218
|
+
### Type conversions
|
|
219
|
+
|
|
220
|
+
Common cases:
|
|
221
|
+
|
|
222
|
+
- `varchar(50)` → `varchar(100)` — safe
|
|
223
|
+
- `varchar(200)` → `varchar(100)` — **detect and truncate + warn, or skip + log**
|
|
224
|
+
- `int` enum → `dropdown` string — use the importer's option map
|
|
225
|
+
- Latin-1 → UTF-8 — re-encode bytes
|
|
226
|
+
- Naive timestamp → timestamptz — assume a tz (ask user) or keep naive
|
|
227
|
+
|
|
228
|
+
### Idempotency
|
|
229
|
+
|
|
230
|
+
Wrap the migration in a transaction. A failed run rolls back cleanly. For re-runs, either:
|
|
231
|
+
|
|
232
|
+
1. Truncate target tables before running (destructive — explicit user approval)
|
|
233
|
+
2. Use `ON CONFLICT DO NOTHING` with deterministic PKs derived from source (e.g. `md5(source_id)`)
|
|
234
|
+
3. Simply drop and reinstall the module, then re-run the migration
|
|
235
|
+
|
|
236
|
+
Default: **transaction + fail-fast**. Let the user re-run after fixing whatever failed.
|
|
237
|
+
|
|
238
|
+
## Dry-run mode
|
|
239
|
+
|
|
240
|
+
**Always** include `--dry-run`. In dry-run mode:
|
|
241
|
+
|
|
242
|
+
1. Connect to both DBs
|
|
243
|
+
2. Run all SELECTs
|
|
244
|
+
3. Do all transforms
|
|
245
|
+
4. Print a summary: `"Would insert 1,243 accounts, 87 contacts, 3,412 activities. 12 rows would be skipped."`
|
|
246
|
+
5. Rollback before committing
|
|
247
|
+
|
|
248
|
+
The user reviews, then re-runs without `--dry-run` to apply.
|
|
249
|
+
|
|
250
|
+
## What NOT to do
|
|
251
|
+
|
|
252
|
+
- **Do not go through the dForge API.** The RPC stack is 100x slower than direct SQL and adds no safety for a one-shot migration. Direct INSERTs into the tenant DB are the right tool.
|
|
253
|
+
- **Do not skip dry-run mode.** It's the only safe way to preview.
|
|
254
|
+
- **Do not migrate to production without multiple explicit confirmations.** Dev/local only by default. If the user insists on production, make the script check `--i-know-this-is-production` and print a scary warning.
|
|
255
|
+
- **Do not hardcode credentials.** Pass via arguments or env vars.
|
|
256
|
+
- **Do not silently drop rows.** Every skipped row gets logged to stderr with a reason.
|
|
257
|
+
- **Do not assume target tables exist.** Check first; bail if the module isn't installed.
|
|
258
|
+
- **Do not migrate views, stored procs, or triggers.** Those are listed in `MIGRATION_NOTES.md` and converted manually to dForge constructs (formulas, actions, reports).
|
|
259
|
+
|
|
260
|
+
## Produce a MIGRATION.md alongside the script
|
|
261
|
+
|
|
262
|
+
When you generate the migration script, also generate (or update) a `MIGRATION.md` in the module folder with:
|
|
263
|
+
|
|
264
|
+
- What the script does in plain English
|
|
265
|
+
- How to run it (example command lines for dry-run and actual run)
|
|
266
|
+
- What gets migrated and what doesn't
|
|
267
|
+
- Known caveats specific to this dataset
|
|
268
|
+
- Rollback instructions
|
|
269
|
+
|
|
270
|
+
The user reads `MIGRATION.md` first, then runs the script.
|