@dforge-core/dforge-mcp 0.1.0-rc.9 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +132 -0
- package/README.md +84 -27
- package/dist/server.js +2957 -616
- package/docs/creating-modules.md +11 -4
- package/package.json +11 -6
- package/resources/docs/conventions.md +22 -16
- package/resources/docs/dsl-reference.md +767 -0
- package/resources/schemas/entity.schema.json +8 -1
- package/resources/schemas/print-templates.schema.json +82 -0
- package/resources/schemas/reports.schema.json +4 -0
- package/resources/schemas/triggers.schema.json +59 -0
- package/skills/dforge-mcp-author/SKILL.md +269 -69
- package/skills/dforge-mcp-author/examples/matrix-budget/README.md +43 -0
- package/skills/dforge-mcp-author/examples/matrix-budget/entities/budget_category.json +24 -0
- package/skills/dforge-mcp-author/examples/matrix-budget/entities/budget_line.json +56 -0
- package/skills/dforge-mcp-author/examples/matrix-budget/manifest.json +30 -0
- package/skills/dforge-mcp-author/examples/matrix-budget/security/roles.json +9 -0
- package/skills/dforge-mcp-author/examples/matrix-budget/seed-data/01-categories.json +8 -0
- package/skills/dforge-mcp-author/examples/matrix-budget/ui/data_views.json +42 -0
- package/skills/dforge-mcp-author/examples/simple-todo/README.md +38 -0
- package/skills/dforge-mcp-author/examples/simple-todo/entities/todo_item.json +83 -0
- package/skills/dforge-mcp-author/examples/simple-todo/entities/todo_list.json +43 -0
- package/skills/dforge-mcp-author/examples/simple-todo/logic/actions/mark_done.dsl +6 -0
- package/skills/dforge-mcp-author/examples/simple-todo/manifest.json +32 -0
- package/skills/dforge-mcp-author/examples/simple-todo/security/roles.json +10 -0
- package/skills/dforge-mcp-author/examples/simple-todo/seed-data/01-lists.json +17 -0
- package/skills/dforge-mcp-author/examples/simple-todo/ui/actions.json +11 -0
- package/skills/dforge-mcp-author/examples/simple-todo/ui/data_views.json +35 -0
- package/skills/dforge-mcp-author/examples/simple-todo/ui/menus.json +28 -0
- package/skills/dforge-mcp-author/references/action-dsl.md +397 -0
- package/skills/dforge-mcp-author/references/column-types.md +168 -0
- package/skills/dforge-mcp-author/references/conventions.md +177 -0
- package/skills/dforge-mcp-author/references/data-migration.md +270 -0
- package/skills/dforge-mcp-author/references/data-views.md +243 -0
- package/skills/dforge-mcp-author/references/excel-import.md +61 -0
- package/skills/dforge-mcp-author/references/field-types.md +144 -0
- package/skills/dforge-mcp-author/references/filters.md +326 -0
- package/skills/dforge-mcp-author/references/flags.md +73 -0
- package/skills/dforge-mcp-author/references/formulas.md +206 -0
- package/skills/dforge-mcp-author/references/jobs.md +149 -0
- package/skills/dforge-mcp-author/references/manifest.md +123 -0
- package/skills/dforge-mcp-author/references/menus.md +164 -0
- package/skills/dforge-mcp-author/references/number-sequences.md +117 -0
- package/skills/dforge-mcp-author/references/print-templates.md +159 -0
- package/skills/dforge-mcp-author/references/queries.md +312 -0
- package/skills/dforge-mcp-author/references/reports.md +398 -0
- package/skills/dforge-mcp-author/references/schema-import.md +331 -0
- package/skills/dforge-mcp-author/references/security.md +244 -0
- package/skills/dforge-mcp-author/references/settings.md +120 -0
- package/skills/dforge-mcp-author/references/traits.md +153 -0
- package/skills/dforge-mcp-author/references/translations.md +158 -0
- package/skills/dforge-mcp-author/references/validation-checklist.md +183 -0
- package/skills/dforge-mcp-author/scripts/xlsx_to_model.py +198 -0
|
@@ -0,0 +1,177 @@
|
|
|
1
|
+
# Module Conventions Reference
|
|
2
|
+
|
|
3
|
+
Rules that apply across the entire module package. Violating these produces technically-valid but non-idiomatic modules, and in several cases breaks the installer.
|
|
4
|
+
|
|
5
|
+
## Naming
|
|
6
|
+
|
|
7
|
+
| Thing | Convention | Example |
|
|
8
|
+
|---|---|---|
|
|
9
|
+
| Module `code` | lowercase, letters/digits/underscore | `crm`, `my_module`, `wms_fin` |
|
|
10
|
+
| Entity keys / `dbObject` | `snake_case`, singular | `contact`, `opportunity_line` |
|
|
11
|
+
| Column codes | `snake_case` | `first_name`, `account_id` |
|
|
12
|
+
| Data view codes | `snake_case` | `contact_list`, `deal_kanban` |
|
|
13
|
+
| Report codes | `snake_case` | `sales_pipeline` |
|
|
14
|
+
| Action codes | `snake_case` | `change_stage`, `send_welcome` |
|
|
15
|
+
| Menu codes | `snake_case` | `sales`, `pipeline` |
|
|
16
|
+
| Role codes | `snake_case`, domain-specific | `sales_rep`, `sales_admin` |
|
|
17
|
+
| Setting codes | `snake_case` | `vat_rate`, `invoice_prefix` |
|
|
18
|
+
| Trait names | lowercase single word | `identity`, `audit` |
|
|
19
|
+
| Seed data files | numbered prefix | `01-countries.json`, `02-currencies.json` |
|
|
20
|
+
| Action DSL files | `<action_code>.dsl` | `change_stage.dsl` |
|
|
21
|
+
| Reference keys | `FK_<FromEntity>_<Purpose>` | `FK_Contact_Account` |
|
|
22
|
+
|
|
23
|
+
Everything is **case-sensitive**. Don't mix snake_case and camelCase.
|
|
24
|
+
|
|
25
|
+
## File layout
|
|
26
|
+
|
|
27
|
+
Standard module directory structure:
|
|
28
|
+
|
|
29
|
+
```
|
|
30
|
+
my_module/
|
|
31
|
+
├── manifest.json
|
|
32
|
+
├── README.md # module overview
|
|
33
|
+
├── CHANGELOG.md # version history
|
|
34
|
+
├── MODULE-INFO.md # user-facing intro (shown in module picker)
|
|
35
|
+
├── entities/
|
|
36
|
+
│ └── <entity>.json
|
|
37
|
+
├── logic/
|
|
38
|
+
│ └── actions/
|
|
39
|
+
│ └── <action>.dsl
|
|
40
|
+
├── ui/
|
|
41
|
+
│ ├── data_views.json
|
|
42
|
+
│ ├── menus.json
|
|
43
|
+
│ ├── actions.json
|
|
44
|
+
│ ├── folders.json
|
|
45
|
+
│ └── reports/
|
|
46
|
+
│ └── <report>.json
|
|
47
|
+
├── security/
|
|
48
|
+
│ └── roles.json
|
|
49
|
+
├── settings.json
|
|
50
|
+
├── seed-data/
|
|
51
|
+
│ ├── 01-<first>.json
|
|
52
|
+
│ └── 02-<second>.json
|
|
53
|
+
├── translations/
|
|
54
|
+
│ └── de-DE.json # IETF tags, one file per non-English locale
|
|
55
|
+
├── print_templates/
|
|
56
|
+
│ └── <template>.scriban
|
|
57
|
+
├── files/ # static assets
|
|
58
|
+
└── webhooks.json # optional
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Only include what your module actually uses — not every module needs `print_templates/` or `webhooks.json`.
|
|
62
|
+
|
|
63
|
+
## The FK+Reference pattern (again, because it's critical)
|
|
64
|
+
|
|
65
|
+
**Every reference to another entity is two columns**:
|
|
66
|
+
|
|
67
|
+
1. Hidden FK column with `flags: "EM"` and `dbDatatype` matching the target PK type
|
|
68
|
+
2. Visible Reference column with `columnType: "R"`, `fieldTypeCd: "lookup"`, `flags: "VEM"`, and a `link` object pointing at the target
|
|
69
|
+
|
|
70
|
+
Plus the actual FK constraint declared in the entity's `references` block.
|
|
71
|
+
|
|
72
|
+
See `column-types.md` for the full example.
|
|
73
|
+
|
|
74
|
+
## Bridge modules
|
|
75
|
+
|
|
76
|
+
When two core modules need to integrate (e.g. CRM and Finance), create a **bridge module**:
|
|
77
|
+
|
|
78
|
+
- Name it with a hyphen: `crm-fin`, `wms-fin`, `crm-pricing`
|
|
79
|
+
- Depends on **both** core modules
|
|
80
|
+
- Owns the integration: extension entities, cross-module actions, cross-module data views
|
|
81
|
+
- Core modules stay independent and don't know about each other
|
|
82
|
+
|
|
83
|
+
Extension entities in a bridge module have `"extends": "fin.invoice"` at the top. Physical columns in the extension go into a 1:1 ext table.
|
|
84
|
+
|
|
85
|
+
## Seed data ordering
|
|
86
|
+
|
|
87
|
+
Seed files run in numeric prefix order. Put entities with FK dependencies **after** their targets:
|
|
88
|
+
|
|
89
|
+
```
|
|
90
|
+
seed-data/
|
|
91
|
+
├── 01-currencies.json # no FKs, insert first
|
|
92
|
+
├── 02-countries.json # references currency
|
|
93
|
+
├── 03-regions.json # references country
|
|
94
|
+
└── 04-offices.json # references region
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Each seed file has two top-level keys: `"entityCode"` (NOT `"entity"`) and `"records"` (array). **Using `"entity"` instead of `"entityCode"` is a silent failure** — the installer reads an empty entity code and the INSERT fails.
|
|
98
|
+
|
|
99
|
+
Use explicit **numeric** PKs in seed data so cross-file references work. The `identity` trait creates `cuid` columns which are `int8` (bigint) — **NOT UUIDs**. Use simple integers like `1001`, `1002`:
|
|
100
|
+
|
|
101
|
+
```json
|
|
102
|
+
{
|
|
103
|
+
"entityCode": "country",
|
|
104
|
+
"records": [
|
|
105
|
+
{ "country_id": 1001, "code": "US", "name": "United States" },
|
|
106
|
+
{ "country_id": 1002, "code": "DE", "name": "Germany" }
|
|
107
|
+
]
|
|
108
|
+
}
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
**Common mistake**: using UUID strings like `"10000000-0000-0000-0000-000000000001"` — **wrong**. The `cuid` datatype is a numeric `int8`, not a UUID string. Seed data PKs must be numbers. Use a numbering scheme that avoids collisions (e.g. 1001-1099 for countries, 2001-2099 for regions, etc.).
|
|
112
|
+
|
|
113
|
+
## Translations
|
|
114
|
+
|
|
115
|
+
Translation files live in `translations/` and are named after the **IETF locale tag** (`de-DE.json`, `uk-UA.json`, etc.). Filenames are auto-discovered; declare which ones you ship in `manifest.supportedLocales`. English is the default — labels and descriptions live in entity/view JSON, so **don't** create `en.json` or `en-US.json` (they're skipped by the installer).
|
|
116
|
+
|
|
117
|
+
```json
|
|
118
|
+
// translations/de-DE.json
|
|
119
|
+
{
|
|
120
|
+
"contact": "Kontakt",
|
|
121
|
+
"account": "Kunde",
|
|
122
|
+
"contact_list.label": "Alle Kontakte"
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
Missing keys fall back to the English value declared in the source JSON.
|
|
127
|
+
|
|
128
|
+
## Versioning — always bump before packaging
|
|
129
|
+
|
|
130
|
+
The manifest has **two version numbers** that must be managed:
|
|
131
|
+
|
|
132
|
+
| Field | When to bump | Example |
|
|
133
|
+
|---|---|---|
|
|
134
|
+
| `version` | **Every release** — any change at all (bug fix, feature, schema change) | `0.1.0` → `0.2.0` |
|
|
135
|
+
| `dbSchemaVersion` | **Only when the DB schema changes** — new entity, new column, removed column, changed type, new constraint, new index | `0.0.1` → `0.0.2` |
|
|
136
|
+
|
|
137
|
+
Both use **semver** (`MAJOR.MINOR.PATCH`).
|
|
138
|
+
|
|
139
|
+
**Rules**:
|
|
140
|
+
|
|
141
|
+
- If you added an action but no new columns → bump `version` only, keep `dbSchemaVersion`.
|
|
142
|
+
- If you added a new entity or column → bump both `version` and `dbSchemaVersion`.
|
|
143
|
+
- If you changed a view or menu → bump `version` only.
|
|
144
|
+
- If you changed translations only → bump `version` only.
|
|
145
|
+
- The installer uses `dbSchemaVersion` to decide whether to run schema migrations. Wrong version = skipped migrations.
|
|
146
|
+
- **Never ship a package without bumping at least `version`** from the previous release. The installer may reject or silently skip a re-install with the same version.
|
|
147
|
+
- For brand-new modules, start at `version: "0.1.0"` and `dbSchemaVersion: "0.0.1"`.
|
|
148
|
+
|
|
149
|
+
## `orderNum` — always set it
|
|
150
|
+
|
|
151
|
+
On every column, data view, menu item, setting, and action that appears in a list. Without `orderNum`, UI ordering is undefined. Use widely-spaced values (10, 20, 30, 40) so you can insert between them later.
|
|
152
|
+
|
|
153
|
+
## Consistency across the module
|
|
154
|
+
|
|
155
|
+
- All timestamps use the same trait (`audit`)
|
|
156
|
+
- All primary keys use the same trait (`identity`) with consistent PK type (`cuid`)
|
|
157
|
+
- All entities have `toString`
|
|
158
|
+
- All views use `dataSources` array
|
|
159
|
+
- All roles use `rights`
|
|
160
|
+
- All menus use nested dicts + `dataViewCode`
|
|
161
|
+
- All actions have `canExecute:` (even if it's just `true`)
|
|
162
|
+
|
|
163
|
+
## Do not invent
|
|
164
|
+
|
|
165
|
+
- Do not invent field types (`rating`, `geolocation`, `signature`) — not in the catalog
|
|
166
|
+
- Do not invent column types beyond `D` / `R` / `S` / `F`
|
|
167
|
+
- Do not invent role rights letters beyond `SIUDC` + `E`
|
|
168
|
+
- Do not invent view types beyond the registered ones
|
|
169
|
+
- Do not invent menu item types beyond `V` / `R` / `D` / `A`
|
|
170
|
+
- Do not invent DSL functions
|
|
171
|
+
- Do not invent formula functions
|
|
172
|
+
|
|
173
|
+
If you genuinely need something new, **say so** and ask the user how to work around it with existing primitives.
|
|
174
|
+
|
|
175
|
+
## Reference
|
|
176
|
+
|
|
177
|
+
This file summarises the key conventions. If you need to verify a pattern not covered here, ask the user or check the reference modules in the `examples/` directory.
|
|
@@ -0,0 +1,270 @@
|
|
|
1
|
+
# Data Migration Reference
|
|
2
|
+
|
|
3
|
+
When the user has an existing database (typically after running the schema importer to produce a rough-cut module) and wants to **move the actual rows** into dForge, write them a migration script they can run locally.
|
|
4
|
+
|
|
5
|
+
This assumes:
|
|
6
|
+
- dForge is running locally (e.g. `docker compose up`).
|
|
7
|
+
- The legacy database is reachable from the developer's machine.
|
|
8
|
+
- The target is a **dev tenant**, never production.
|
|
9
|
+
|
|
10
|
+
## Script template — Python / psycopg
|
|
11
|
+
|
|
12
|
+
This is the most portable choice. Works against any source database with a Python driver, writes to dForge Postgres.
|
|
13
|
+
|
|
14
|
+
```python
|
|
15
|
+
#!/usr/bin/env python3
|
|
16
|
+
"""
|
|
17
|
+
Migration script: old_crm → dForge my_crm module
|
|
18
|
+
Generated on 2026-04-08
|
|
19
|
+
"""
|
|
20
|
+
|
|
21
|
+
import argparse
|
|
22
|
+
import sys
|
|
23
|
+
import uuid
|
|
24
|
+
import psycopg
|
|
25
|
+
from datetime import datetime
|
|
26
|
+
from typing import Dict
|
|
27
|
+
|
|
28
|
+
MIGRATION_USER_ID = "00000000-0000-0000-0000-000000000001" # override with --migration-user
|
|
29
|
+
|
|
30
|
+
def cuid() -> str:
|
|
31
|
+
"""Generate a new cuid-like identifier for target PKs."""
|
|
32
|
+
return str(uuid.uuid4())
|
|
33
|
+
|
|
34
|
+
def main():
|
|
35
|
+
parser = argparse.ArgumentParser()
|
|
36
|
+
parser.add_argument("--source", required=True, help="Source DB connection string")
|
|
37
|
+
parser.add_argument("--target", required=True, help="Target dForge tenant DB connection string")
|
|
38
|
+
parser.add_argument("--migration-user", default=MIGRATION_USER_ID)
|
|
39
|
+
parser.add_argument("--dry-run", action="store_true", help="Simulate without committing")
|
|
40
|
+
args = parser.parse_args()
|
|
41
|
+
|
|
42
|
+
src = psycopg.connect(args.source)
|
|
43
|
+
tgt = psycopg.connect(args.target)
|
|
44
|
+
|
|
45
|
+
try:
|
|
46
|
+
with tgt.transaction():
|
|
47
|
+
pk_map: Dict[str, Dict[int, str]] = {} # source_table → {source_id → target_id}
|
|
48
|
+
|
|
49
|
+
# Phase 1: parents (no FK deps)
|
|
50
|
+
migrate_accounts(src, tgt, pk_map, args.migration_user)
|
|
51
|
+
migrate_products(src, tgt, pk_map, args.migration_user)
|
|
52
|
+
|
|
53
|
+
# Phase 2: children (depend on parents)
|
|
54
|
+
migrate_contacts(src, tgt, pk_map, args.migration_user)
|
|
55
|
+
migrate_opportunities(src, tgt, pk_map, args.migration_user)
|
|
56
|
+
|
|
57
|
+
# Phase 3: grandchildren
|
|
58
|
+
migrate_opportunity_lines(src, tgt, pk_map, args.migration_user)
|
|
59
|
+
migrate_activities(src, tgt, pk_map, args.migration_user)
|
|
60
|
+
|
|
61
|
+
if args.dry_run:
|
|
62
|
+
print("[DRY RUN] Rolling back.")
|
|
63
|
+
raise SystemExit(0)
|
|
64
|
+
|
|
65
|
+
print("Migration committed successfully.")
|
|
66
|
+
|
|
67
|
+
except Exception as e:
|
|
68
|
+
print(f"Migration failed: {e}", file=sys.stderr)
|
|
69
|
+
sys.exit(1)
|
|
70
|
+
finally:
|
|
71
|
+
src.close()
|
|
72
|
+
tgt.close()
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
def migrate_accounts(src, tgt, pk_map, migration_user):
|
|
76
|
+
print("Migrating accounts…")
|
|
77
|
+
pk_map["accounts"] = {}
|
|
78
|
+
with src.cursor() as sc, tgt.cursor() as tc:
|
|
79
|
+
sc.execute("SELECT id, name, phone, website, created_at FROM accounts")
|
|
80
|
+
count = 0
|
|
81
|
+
for row in sc:
|
|
82
|
+
old_id, name, phone, website, created_at = row
|
|
83
|
+
new_id = cuid()
|
|
84
|
+
pk_map["accounts"][old_id] = new_id
|
|
85
|
+
tc.execute(
|
|
86
|
+
"""
|
|
87
|
+
INSERT INTO crm.account
|
|
88
|
+
(account_id, name, phone, website, created_date, created_by, last_updated, last_updated_by)
|
|
89
|
+
VALUES (%s, %s, %s, %s, %s, %s, %s, %s)
|
|
90
|
+
""",
|
|
91
|
+
(new_id, name, phone, website, created_at or datetime.utcnow(),
|
|
92
|
+
migration_user, created_at or datetime.utcnow(), migration_user),
|
|
93
|
+
)
|
|
94
|
+
count += 1
|
|
95
|
+
print(f" {count} accounts migrated")
|
|
96
|
+
|
|
97
|
+
|
|
98
|
+
def migrate_contacts(src, tgt, pk_map, migration_user):
|
|
99
|
+
print("Migrating contacts…")
|
|
100
|
+
pk_map["contacts"] = {}
|
|
101
|
+
skipped = 0
|
|
102
|
+
with src.cursor() as sc, tgt.cursor() as tc:
|
|
103
|
+
sc.execute("SELECT id, first_name, last_name, email, phone, account_id, created_at FROM contacts")
|
|
104
|
+
count = 0
|
|
105
|
+
for row in sc:
|
|
106
|
+
old_id, first_name, last_name, email, phone, old_account_id, created_at = row
|
|
107
|
+
|
|
108
|
+
# Resolve FK via pk_map
|
|
109
|
+
new_account_id = pk_map["accounts"].get(old_account_id)
|
|
110
|
+
if old_account_id and not new_account_id:
|
|
111
|
+
print(f" Skipping contact {old_id}: account {old_account_id} not found in mapping")
|
|
112
|
+
skipped += 1
|
|
113
|
+
continue
|
|
114
|
+
|
|
115
|
+
new_id = cuid()
|
|
116
|
+
pk_map["contacts"][old_id] = new_id
|
|
117
|
+
tc.execute(
|
|
118
|
+
"""
|
|
119
|
+
INSERT INTO crm.contact
|
|
120
|
+
(contact_id, first_name, last_name, email, phone, account_id,
|
|
121
|
+
created_date, created_by, last_updated, last_updated_by)
|
|
122
|
+
VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
|
|
123
|
+
""",
|
|
124
|
+
(new_id, first_name, last_name, email, phone, new_account_id,
|
|
125
|
+
created_at or datetime.utcnow(), migration_user,
|
|
126
|
+
created_at or datetime.utcnow(), migration_user),
|
|
127
|
+
)
|
|
128
|
+
count += 1
|
|
129
|
+
print(f" {count} contacts migrated, {skipped} skipped")
|
|
130
|
+
|
|
131
|
+
|
|
132
|
+
# … migrate_products, migrate_opportunities, migrate_opportunity_lines, migrate_activities follow the same pattern
|
|
133
|
+
|
|
134
|
+
|
|
135
|
+
if __name__ == "__main__":
|
|
136
|
+
main()
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
## Script template — Pure SQL (Postgres → Postgres via dblink)
|
|
140
|
+
|
|
141
|
+
When both databases are PostgreSQL and you want the fastest possible migration, use `dblink` or `postgres_fdw`. This requires the `dblink` extension to be installed on the target.
|
|
142
|
+
|
|
143
|
+
```sql
|
|
144
|
+
-- migrate.sql — run against the dForge tenant DB
|
|
145
|
+
-- Connect to the legacy source via dblink
|
|
146
|
+
|
|
147
|
+
BEGIN;
|
|
148
|
+
|
|
149
|
+
-- Ensure dblink is available
|
|
150
|
+
CREATE EXTENSION IF NOT EXISTS dblink;
|
|
151
|
+
|
|
152
|
+
-- Create a PK mapping temp table
|
|
153
|
+
CREATE TEMP TABLE pk_map (
|
|
154
|
+
source_table text NOT NULL,
|
|
155
|
+
source_id bigint NOT NULL,
|
|
156
|
+
target_id text NOT NULL,
|
|
157
|
+
PRIMARY KEY (source_table, source_id)
|
|
158
|
+
);
|
|
159
|
+
|
|
160
|
+
-- Phase 1: Accounts
|
|
161
|
+
INSERT INTO crm.account
|
|
162
|
+
(account_id, name, phone, website, created_date, created_by, last_updated, last_updated_by)
|
|
163
|
+
SELECT
|
|
164
|
+
gen_random_uuid()::text,
|
|
165
|
+
src.name,
|
|
166
|
+
src.phone,
|
|
167
|
+
src.website,
|
|
168
|
+
COALESCE(src.created_at, NOW()),
|
|
169
|
+
'00000000-0000-0000-0000-000000000001'::text,
|
|
170
|
+
COALESCE(src.created_at, NOW()),
|
|
171
|
+
'00000000-0000-0000-0000-000000000001'::text
|
|
172
|
+
FROM dblink(
|
|
173
|
+
'host=legacy.local dbname=old_crm user=readonly',
|
|
174
|
+
'SELECT id, name, phone, website, created_at FROM accounts'
|
|
175
|
+
) AS src(id bigint, name text, phone text, website text, created_at timestamp)
|
|
176
|
+
RETURNING account_id;
|
|
177
|
+
|
|
178
|
+
-- Record mappings for FK resolution
|
|
179
|
+
-- (In practice, do this in the same INSERT using a CTE with RETURNING)
|
|
180
|
+
|
|
181
|
+
-- Phase 2: Contacts (needs account FK resolution)
|
|
182
|
+
-- …
|
|
183
|
+
|
|
184
|
+
COMMIT;
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Pure SQL is faster but harder to read and debug. **Prefer the Python script unless the dataset is very large.**
|
|
188
|
+
|
|
189
|
+
## Key rules
|
|
190
|
+
|
|
191
|
+
### PK generation
|
|
192
|
+
|
|
193
|
+
Source uses integer PKs, dForge uses cuid (string). Generate a **new cuid for every row** and record it in the mapping table. Never preserve source integer PKs.
|
|
194
|
+
|
|
195
|
+
### FK resolution order
|
|
196
|
+
|
|
197
|
+
Topologically sort the entities by FK dependencies. Parents first, then children, then grandchildren. If the source schema has cycles, break them: insert parents with NULL FKs first, then UPDATE the FKs in a second pass.
|
|
198
|
+
|
|
199
|
+
### Audit fields
|
|
200
|
+
|
|
201
|
+
Set `created_by` / `last_updated_by` to a designated migration user. The zero user (`00000000-0000-0000-0000-000000000001`) is a common convention for "system-generated" rows. Ask the user if you're unsure which user to attribute to. (Note: `created_by`/`last_updated_by` only exist if the entity uses `audit-full` trait; with just `audit`, only timestamps exist.)
|
|
202
|
+
|
|
203
|
+
Set `created_date` / `last_updated` from source timestamps if available, otherwise `NOW()`. Note: dForge audit column names are `created_date` and `last_updated` — NOT `created_at`/`updated_at`.
|
|
204
|
+
|
|
205
|
+
### Enum conversion
|
|
206
|
+
|
|
207
|
+
If the schema importer converted a lookup table (like `statuses`) to a `dropdown` column, the migration script must map source integer FK values to the new string option values. Read `params.options` on the target column and the importer's `MIGRATION_NOTES.md` to find the mapping.
|
|
208
|
+
|
|
209
|
+
### NULL handling
|
|
210
|
+
|
|
211
|
+
Source column is nullable, but the target column has `NOT NULL` (because the importer detected `NOT NULL` or the user made it mandatory). The script should either:
|
|
212
|
+
|
|
213
|
+
- Provide a sensible default, or
|
|
214
|
+
- Skip the row and log it
|
|
215
|
+
|
|
216
|
+
Never silently insert NULL into a NOT NULL column — it will fail.
|
|
217
|
+
|
|
218
|
+
### Type conversions
|
|
219
|
+
|
|
220
|
+
Common cases:
|
|
221
|
+
|
|
222
|
+
- `varchar(50)` → `varchar(100)` — safe
|
|
223
|
+
- `varchar(200)` → `varchar(100)` — **detect and truncate + warn, or skip + log**
|
|
224
|
+
- `int` enum → `dropdown` string — use the importer's option map
|
|
225
|
+
- Latin-1 → UTF-8 — re-encode bytes
|
|
226
|
+
- Naive timestamp → timestamptz — assume a tz (ask user) or keep naive
|
|
227
|
+
|
|
228
|
+
### Idempotency
|
|
229
|
+
|
|
230
|
+
Wrap the migration in a transaction. A failed run rolls back cleanly. For re-runs, either:
|
|
231
|
+
|
|
232
|
+
1. Truncate target tables before running (destructive — explicit user approval)
|
|
233
|
+
2. Use `ON CONFLICT DO NOTHING` with deterministic PKs derived from source (e.g. `md5(source_id)`)
|
|
234
|
+
3. Simply drop and reinstall the module, then re-run the migration
|
|
235
|
+
|
|
236
|
+
Default: **transaction + fail-fast**. Let the user re-run after fixing whatever failed.
|
|
237
|
+
|
|
238
|
+
## Dry-run mode
|
|
239
|
+
|
|
240
|
+
**Always** include `--dry-run`. In dry-run mode:
|
|
241
|
+
|
|
242
|
+
1. Connect to both DBs
|
|
243
|
+
2. Run all SELECTs
|
|
244
|
+
3. Do all transforms
|
|
245
|
+
4. Print a summary: `"Would insert 1,243 accounts, 87 contacts, 3,412 activities. 12 rows would be skipped."`
|
|
246
|
+
5. Rollback before committing
|
|
247
|
+
|
|
248
|
+
The user reviews, then re-runs without `--dry-run` to apply.
|
|
249
|
+
|
|
250
|
+
## What NOT to do
|
|
251
|
+
|
|
252
|
+
- **Do not go through the dForge API.** The RPC stack is 100x slower than direct SQL and adds no safety for a one-shot migration. Direct INSERTs into the tenant DB are the right tool.
|
|
253
|
+
- **Do not skip dry-run mode.** It's the only safe way to preview.
|
|
254
|
+
- **Do not migrate to production without multiple explicit confirmations.** Dev/local only by default. If the user insists on production, make the script check `--i-know-this-is-production` and print a scary warning.
|
|
255
|
+
- **Do not hardcode credentials.** Pass via arguments or env vars.
|
|
256
|
+
- **Do not silently drop rows.** Every skipped row gets logged to stderr with a reason.
|
|
257
|
+
- **Do not assume target tables exist.** Check first; bail if the module isn't installed.
|
|
258
|
+
- **Do not migrate views, stored procs, or triggers.** Those are listed in `MIGRATION_NOTES.md` and converted manually to dForge constructs (formulas, actions, reports).
|
|
259
|
+
|
|
260
|
+
## Produce a MIGRATION.md alongside the script
|
|
261
|
+
|
|
262
|
+
When you generate the migration script, also generate (or update) a `MIGRATION.md` in the module folder with:
|
|
263
|
+
|
|
264
|
+
- What the script does in plain English
|
|
265
|
+
- How to run it (example command lines for dry-run and actual run)
|
|
266
|
+
- What gets migrated and what doesn't
|
|
267
|
+
- Known caveats specific to this dataset
|
|
268
|
+
- Rollback instructions
|
|
269
|
+
|
|
270
|
+
The user reads `MIGRATION.md` first, then runs the script.
|
|
@@ -0,0 +1,243 @@
|
|
|
1
|
+
# Data Views Reference
|
|
2
|
+
|
|
3
|
+
Data views define how users see and interact with entity data. One entity can have many data views (grid, kanban, gallery, tree-grid, list, calendar, matrix, card).
|
|
4
|
+
|
|
5
|
+
Lives in: `ui/data_views.json`
|
|
6
|
+
|
|
7
|
+
## Structure
|
|
8
|
+
|
|
9
|
+
```json
|
|
10
|
+
{
|
|
11
|
+
"contact_list": {
|
|
12
|
+
"label": "All Contacts",
|
|
13
|
+
"description": "Primary list of contacts",
|
|
14
|
+
"viewType": "grid",
|
|
15
|
+
"icon": "bi-people",
|
|
16
|
+
"dataSources": [{
|
|
17
|
+
"entityCode": "contact",
|
|
18
|
+
"label": "Contacts",
|
|
19
|
+
"columns": [
|
|
20
|
+
{ "column_cd": "first_name", "width": 150 },
|
|
21
|
+
{ "column_cd": "last_name", "width": 150 },
|
|
22
|
+
{ "column_cd": "email", "width": 220 },
|
|
23
|
+
{ "column_cd": "phone", "width": 130 },
|
|
24
|
+
{ "column_cd": "account", "width": 200 }
|
|
25
|
+
],
|
|
26
|
+
"filter": null
|
|
27
|
+
}],
|
|
28
|
+
"order": ["last_name"]
|
|
29
|
+
}
|
|
30
|
+
}
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
## Critical rule — `dataSources` array
|
|
34
|
+
|
|
35
|
+
**Always** put `entityCode`, `columns`, and per-source `filter` inside a `dataSources` array. Never at the root. The view-level `order` and `filter` sit at the view-def root, alongside `viewType`.
|
|
36
|
+
|
|
37
|
+
```json
|
|
38
|
+
// WRONG — will fail validation
|
|
39
|
+
{
|
|
40
|
+
"contact_list": {
|
|
41
|
+
"label": "Contacts",
|
|
42
|
+
"viewType": "grid",
|
|
43
|
+
"entityCode": "contact", // WRONG — should be inside dataSources
|
|
44
|
+
"columns": [ /* ... */ ] // WRONG — should be inside dataSources
|
|
45
|
+
}
|
|
46
|
+
}
|
|
47
|
+
|
|
48
|
+
// RIGHT
|
|
49
|
+
{
|
|
50
|
+
"contact_list": {
|
|
51
|
+
"label": "Contacts",
|
|
52
|
+
"viewType": "grid",
|
|
53
|
+
"dataSources": [{
|
|
54
|
+
"entityCode": "contact",
|
|
55
|
+
"columns": [ /* ... */ ]
|
|
56
|
+
}]
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
## View types
|
|
62
|
+
|
|
63
|
+
| `viewType` | Description | `viewConfig` | When to use |
|
|
64
|
+
|---|---|---|---|
|
|
65
|
+
| `grid` | Spreadsheet-style table with sortable columns | — | Default for most entities; bulk data |
|
|
66
|
+
| `list` | Compact list view | — | Mobile-friendly, card-sized items |
|
|
67
|
+
| `kanban` | Columns grouped by a status field | `groupByField` | Workflow entities with stages (deals, tickets) |
|
|
68
|
+
| `gallery` | Image-forward card grid | — | Image-heavy entities (products, real estate) |
|
|
69
|
+
| `tree-grid` | Hierarchical grid with expand/collapse | — | Tree structures (folders, categories, org charts) |
|
|
70
|
+
| `calendar` | Calendar display of date-based records | `dateField`, `titleField` | Schedules, due dates, events |
|
|
71
|
+
| `matrix` | Pivot grid: `rowAxis` × `colAxis`, one value-source record per cell | `rowAxis`, `colAxis`, `cell` | Cross-tab data: P&L (accounts × periods), timesheets (tasks × days), budgets (categories × months) |
|
|
72
|
+
| `card` | Single-record detail view | — | Record detail pages (implicit — every entity has one) |
|
|
73
|
+
|
|
74
|
+
## View config (`viewConfig`)
|
|
75
|
+
|
|
76
|
+
Some view types need additional configuration via the `viewConfig` property:
|
|
77
|
+
|
|
78
|
+
### Kanban
|
|
79
|
+
|
|
80
|
+
```json
|
|
81
|
+
{
|
|
82
|
+
"opportunities_kanban": {
|
|
83
|
+
"viewType": "kanban",
|
|
84
|
+
"label": "Sales Pipeline",
|
|
85
|
+
"dataSources": [{
|
|
86
|
+
"entityCode": "opportunity",
|
|
87
|
+
"columns": ["opportunity_name", "account", "total_amount", "owner_id"]
|
|
88
|
+
}],
|
|
89
|
+
"viewConfig": {
|
|
90
|
+
"groupByField": "stage"
|
|
91
|
+
}
|
|
92
|
+
}
|
|
93
|
+
}
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
`groupByField` — the dropdown/status column to group cards by. Each distinct value becomes a kanban column.
|
|
97
|
+
|
|
98
|
+
### Calendar
|
|
99
|
+
|
|
100
|
+
```json
|
|
101
|
+
{
|
|
102
|
+
"invoices_calendar": {
|
|
103
|
+
"viewType": "calendar",
|
|
104
|
+
"label": "Invoice Calendar",
|
|
105
|
+
"dataSources": [{
|
|
106
|
+
"entityCode": "invoice",
|
|
107
|
+
"columns": ["invoice_number", "customer", "total_amount", "due_date"]
|
|
108
|
+
}],
|
|
109
|
+
"viewConfig": {
|
|
110
|
+
"dateField": "due_date",
|
|
111
|
+
"titleField": "invoice_number"
|
|
112
|
+
}
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
```
|
|
116
|
+
|
|
117
|
+
- `dateField` — the date column that determines where records appear on the calendar
|
|
118
|
+
- `titleField` — the column whose value is shown as the calendar event title
|
|
119
|
+
|
|
120
|
+
### Matrix
|
|
121
|
+
|
|
122
|
+
A pivot grid. The view's primary `dataSources` entity is the **cell** entity (one record per row×column intersection). `viewConfig` declares the two axes and how the cell record maps onto them.
|
|
123
|
+
|
|
124
|
+
```json
|
|
125
|
+
{
|
|
126
|
+
"income_statement_matrix": {
|
|
127
|
+
"viewType": "matrix",
|
|
128
|
+
"label": "Income Statement (Matrix)",
|
|
129
|
+
"dataSources": [{
|
|
130
|
+
"entityCode": "balance_register",
|
|
131
|
+
"columns": ["account_id", "period_key", "pl_amount"]
|
|
132
|
+
}],
|
|
133
|
+
"viewConfig": {
|
|
134
|
+
"rowAxis": {
|
|
135
|
+
"kind": "dataset",
|
|
136
|
+
"entity": "account",
|
|
137
|
+
"labelField": "account_name",
|
|
138
|
+
"filter": { "c": "statement", "o": "eq", "v": "IncomeStatement" },
|
|
139
|
+
"sort": [{ "col": "account_code", "dir": "asc" }]
|
|
140
|
+
},
|
|
141
|
+
"colAxis": {
|
|
142
|
+
"kind": "dataset",
|
|
143
|
+
"entity": "accounting_period",
|
|
144
|
+
"labelField": "description",
|
|
145
|
+
"lockedField": "closed",
|
|
146
|
+
"hideEmpty": true,
|
|
147
|
+
"sort": [{ "col": "period_key", "dir": "asc" }]
|
|
148
|
+
},
|
|
149
|
+
"cell": {
|
|
150
|
+
"entity": "balance_register",
|
|
151
|
+
"rowKey": "account_id",
|
|
152
|
+
"colKey": "period_key",
|
|
153
|
+
"fields": ["pl_amount"],
|
|
154
|
+
"editable": false,
|
|
155
|
+
"drill": true
|
|
156
|
+
}
|
|
157
|
+
}
|
|
158
|
+
}
|
|
159
|
+
}
|
|
160
|
+
```
|
|
161
|
+
|
|
162
|
+
**`rowAxis` / `colAxis`** — what the rows and columns are. Three `kind`s:
|
|
163
|
+
|
|
164
|
+
- `"dataset"` — axis values are an entity's records. `entity` + `labelField` required; optional `keyField` (defaults to the axis entity PK — what the cell's `rowKey`/`colKey` matches), `lockedField` (boolean column → that axis value's cells are read-only), `filter` (same `{c,o,v}` grammar as everywhere — scopes which axis records load), `hideEmpty` (**column axis only** — drop columns with no cell record in any row, e.g. periods with no postings; ignored if it would blank the whole grid), `sort` (`[{ col, dir }]`).
|
|
165
|
+
- `"dropdown"` — axis values are a dropdown/flags column's options. Just `kind` + `column` (an `"entity.column"` reference). Codes line up by construction.
|
|
166
|
+
- `"date"` — generated date window, **column axis only** (not valid on `rowAxis`). `kind` + `grain: "day"` + `window: "week"` (v1).
|
|
167
|
+
|
|
168
|
+
**`cell`** — the value-source record at each intersection. `entity` (must match the primary dataSource), `rowKey` / `colKey` (cell columns matching the row/column axis keys), `fields` (cell columns rendered in each cell). Optional: `cardinality: "one"` (v1 default), `editable`, `drill` (read-only cells become clickable → open the cell record and its child sets), `seedFromRow` (`{ cellField: rowAxisField }` copied into new cell records on insert), `seedCurrentUser` (cell fields set to the current user id on insert).
|
|
169
|
+
|
|
170
|
+
## Columns
|
|
171
|
+
|
|
172
|
+
Each column is either a simple string or an object:
|
|
173
|
+
|
|
174
|
+
```json
|
|
175
|
+
"columns": [
|
|
176
|
+
"first_name", // just column code, default width
|
|
177
|
+
{ "column_cd": "email", "width": 220 }, // with width override
|
|
178
|
+
{ "column_cd": "account", "width": 200, "visible": true }
|
|
179
|
+
]
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
## Filters
|
|
183
|
+
|
|
184
|
+
Filters use dForge's canonical JSON filter format — see [filters.md](filters.md) for the full grammar. Keys are `c` (column), `o` (operator), `v` (value) for conditions, and `g` (group operator), `i` (items) for groups.
|
|
185
|
+
|
|
186
|
+
Single condition:
|
|
187
|
+
|
|
188
|
+
```json
|
|
189
|
+
"filter": { "c": "status", "o": "=", "v": "active" }
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
Group:
|
|
193
|
+
|
|
194
|
+
```json
|
|
195
|
+
"filter": {
|
|
196
|
+
"g": "and",
|
|
197
|
+
"i": [
|
|
198
|
+
{ "c": "status", "o": "=", "v": "active" },
|
|
199
|
+
{ "c": "created_date", "o": ">", "v": "2026-01-01" }
|
|
200
|
+
]
|
|
201
|
+
}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
Do **not** use `op`/`args`/`column`/`value` — that format is rejected.
|
|
205
|
+
|
|
206
|
+
## Sort (`order`)
|
|
207
|
+
|
|
208
|
+
Sort lives at the **view-def root** (not inside `dataSources[]`) under the key **`order`** — a `string[]` of column codes. Direction is encoded in the string itself:
|
|
209
|
+
|
|
210
|
+
- Bare column code → **ascending** (the default; no prefix needed)
|
|
211
|
+
- Leading `-` → **descending**
|
|
212
|
+
|
|
213
|
+
```json
|
|
214
|
+
"order": ["last_name", "first_name"] // both ascending
|
|
215
|
+
"order": ["-created_date"] // descending (audit trait provides created_date)
|
|
216
|
+
"order": ["-created_date", "last_name"] // primary desc, then asc tiebreak
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
The array order is the sort precedence: the first entry is the primary sort key, subsequent entries are tiebreakers.
|
|
220
|
+
|
|
221
|
+
This matches the shape the runtime persists and `data.get` accepts. **Do not** use `"sort"`, `[{ column_cd, direction }]`, or `[{ c, d }]` here — those shapes belong to `query.run` and `report.run` (see [queries.md](queries.md)), not `data_views.json`.
|
|
222
|
+
|
|
223
|
+
## Multi-source views
|
|
224
|
+
|
|
225
|
+
A view can show data from multiple entities (e.g. a dashboard-style view mixing contacts and activities). Each entry in `dataSources` is independent with its own columns/filter.
|
|
226
|
+
|
|
227
|
+
```json
|
|
228
|
+
"dataSources": [
|
|
229
|
+
{ "entityCode": "contact", "columns": [ /* ... */ ] },
|
|
230
|
+
{ "entityCode": "activity", "columns": [ /* ... */ ] }
|
|
231
|
+
]
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
Most views only have one source. Order is view-level and applies to the primary (level 0) source.
|
|
235
|
+
|
|
236
|
+
## Common mistakes
|
|
237
|
+
|
|
238
|
+
- Putting `entityCode` or `columns` at the root — **wrong**. They go inside `dataSources[]`.
|
|
239
|
+
- Using `viewCode` or `view_cd` — **wrong**. The code is the dictionary key.
|
|
240
|
+
- Using `"sort"` for data view ordering — **wrong**. The field is `"order"` and it lives at view-def root (not inside `dataSources[]`).
|
|
241
|
+
- Object-array order like `[{ column_cd: "...", direction: "asc" }]` — **wrong shape for data views**. Use `string[]` with leading `-` for descending.
|
|
242
|
+
- `viewType` is **optional** — omitted or unknown values fall back to `grid` at runtime. Set it explicitly for any non-grid view (`kanban`, `calendar`, `matrix`, …); a `matrix` view also **requires** a `viewConfig` with `rowAxis`/`colAxis`/`cell`.
|
|
243
|
+
- Inventing view types like `"spreadsheet"` or `"table"` — **wrong**. Use `grid`.
|