@clivly/core 0.8.0 → 0.9.1

package/README.md

@@ -1,34 +1,114 @@
 # @clivly/core
 
-ORM- and auth-agnostic contracts and shared domain types for [Clivly](https://clivly.com) — the CRM that lives inside your app.
+ORM- and auth-agnostic contracts and shared domain types for
+[Clivly](https://clivly.com) — the CRM that lives inside your app.
 
-This package defines the interfaces that ORM and auth adapters implement. It has no runtime dependencies.
+This package defines the interfaces that ORM and auth adapters implement, the
+entity-config schema shared by the SDK and the dashboard's mapping table, and
+the view compiler that turns a mapping into SQL. Its only runtime dependency is
+`zod`.
+
+> **Most integrators never install this directly.** `defineClivlyConfig` is
+> re-exported from `clivly/sdk`, which is where the
+> [integration guide](../../docs/guides/connecting-to-clivly-cloud.md) tells you
+> to import it from. Reach for `@clivly/core` when you are building an adapter
+> or working with the entity config outside an SDK instance.
+
+## Prerequisites
+
+| Requirement | Version / note |
+| --- | --- |
+| Node.js | **≥ 22** |
+| `drizzle-orm` | **Optional peer** — only needed for the `/drizzle` subpath |
+
+## Installation
+
+```bash
+npm install @clivly/core
+```
 
 ## Exports
 
+Every subpath below is importable independently, so you pull in only what you
+use. The root (`.`) re-exports the contracts, entity config, mapping form, sync
+engine and view compiler — but **not** `/drizzle` (which needs the optional
+peer), `/entity-heuristics`, `/mapping-score`, `/integration-state` or
+`/env-file`.
+
 | Entry | Contents |
-|---|---|
-| `.` | Everything below, re-exported |
+| --- | --- |
+| `@clivly/core` | The re-exported surface: contracts, entity config, mapping form, sync engine, view compiler, projections, shared types |
 | `@clivly/core/adapter` | `CRMAdapter` — the interface every ORM implementation fulfils (org-scoped; contacts, companies, deals, tasks, activities) |
-| `@clivly/core/auth-adapter` | `ClivlyAuthAdapter`, `authCustom()`, `MembershipResolver` — pluggable host-app identity + the `crm_members` authorization layer |
+| `@clivly/core/auth-adapter` | `ClivlyAuthAdapter`, `ClivlyUser`, `authCustom()`, `MembershipResolver`, `CrmMembership` — pluggable host-app identity + the `crm_members` authorization layer |
+| `@clivly/core/entity-config` | `defineClivlyConfig`, `validateEntitiesConfig`, `entitiesConfigSchema`, `CANONICAL_FIELDS`, `CRM_CONCEPTS`, `ClivlyConfigError`, and the config types |
 | `@clivly/core/types` | Shared domain types (`Contact`, `Company`, `Deal`, `Task`, `Note`, `Activity`, inputs, filters, `CrmRole`) |
-| `@clivly/core/drizzle` | `discoverFromDrizzle(schema)` — derive the discovery shape from a Drizzle schema object. Requires the optional `drizzle-orm` peer dependency |
+| `@clivly/core/drizzle` | `discoverFromDrizzle(schema)` — derive the discovery shape (including keys, types, nullability and FK edges) from a Drizzle schema object; plus `buildUniqueConstraints` / `buildColumnsMeta` for reflecting a single table. Requires the optional `drizzle-orm` peer |
+| `@clivly/core/view-compiler` | `compileEntityView`, `compileEntityViews`, `explainEntitiesConfig`, `DEFAULT_VIEW_PREFIX` |
+| `@clivly/core/mappings-config` | `mappingsToEntitiesConfig`, `EntityMappingRow` — turn `crm_entity_mappings` rows into an entities config |
+| `@clivly/core/mapping-form` | `buildFilterFromDiscriminator`, `buildRelationships`, `relationshipViaSignature` — mapping-UI helpers |
+| `@clivly/core/mapping-score` | Scoring helpers for ranking candidate mappings |
+| `@clivly/core/entity-heuristics` | Table-name heuristics that guess which table holds contacts/companies |
+| `@clivly/core/sync-engine` | `reconcile`, `runSync`, `SyncStore` and friends — the mirror reconciliation engine |
+| `@clivly/core/integration-state` | `INSTALL_COMMAND` and the shared onboarding state machine |
+| `@clivly/core/env-file` | `envFileFor(framework)` — the conventional env filename per framework |
+
+## Five-minute quick start
 
-## Schema discovery (Drizzle)
+### Schema discovery (Drizzle)
 
-`discoverFromDrizzle` turns your Drizzle schema into Clivly's discovery shape, so you don't hand-author table/column arrays:
+`discoverFromDrizzle` turns your Drizzle schema into Clivly's discovery shape,
+so you don't hand-author table/column arrays:
 
 ```ts
 import { discoverFromDrizzle } from "@clivly/core/drizzle";
 import * as schema from "./db/schema";
 
 const tables = discoverFromDrizzle(schema);
-// → [{ name: "participants", columns: ["id", "full_name", "email", …] }, …]
+// → [
+//     {
+//       name: "participants",
+//       columns: ["id", "full_name", "email", …],
+//       columnsMeta: [
+//         { name: "id", type: "uuid", nullable: false, isPrimaryKey: true, isForeignKey: false },
+//         { name: "org_id", type: "uuid", nullable: false, isPrimaryKey: false,
+//           isForeignKey: true, references: { table: "organizations", column: "id" } },
+//         …
+//       ],
+//       uniqueConstraints: [["id"], ["email"]],
+//     },
+//     …
+//   ]
+```
+
+Column names are the **real DB names** (`full_name`), not the Drizzle property
+keys (`fullName`).
+
+`columnsMeta` and `uniqueConstraints` are what [derived
+objects](#derived-objects-projections) validate against — without them a
+projection cannot be authored, so keep this wired up rather than hand-writing
+`{ name, columns }` pairs.
+
+### Defining an entity config
+
+```ts
+import { defineClivlyConfig } from "@clivly/core/entity-config";
+
+const entities = defineClivlyConfig({
+  entities: {
+    participants: {
+      concept: "contact",
+      source: "participants",                        // the host table name
+      fields: { name: "full_name", email: "email" }, // canonical key → source column
+    },
+  },
+});
 ```
 
-Column names are the real DB names (`full_name`), not the Drizzle keys (`fullName`). `drizzle-orm` is an **optional peer dependency** — only needed when you import this subpath.
+`defineClivlyConfig` validates at load and warns on any non-canonical field key
+that would mirror as empty. `CANONICAL_FIELDS` lists the accepted keys per
+concept; `CRM_CONCEPTS` lists the concepts.
 
-## Usage
+### Implementing an adapter
 
 ```ts
 import type { CRMAdapter } from "@clivly/core";
@@ -39,9 +119,78 @@ function useAdapter(crm: CRMAdapter) {
 }
 ```
 
-## The `$raw` escape hatch
+## Derived objects (projections)
 
-Entity `filter`s and relationship `via`s accept a `$raw` form — an opaque SQL fragment for predicates the operators can't express (e.g. a cross-table condition):
+An ordinary entity maps one table. A **projection** derives a custom object from
+a base table plus explicit joins, so an object that spans tables — an
+`Enrollment` built from `enrollments`, `students` and `courses` — can be
+expressed at all.
+
+```ts
+import { validateProjection, type ProjectionMapping } from "@clivly/core";
+
+const enrollment: ProjectionMapping = {
+  version: 1,
+  kind: "projection",
+  entityKey: "enrollment",
+  baseTable: "enrollments",
+  joins: [
+    {
+      key: "stu",
+      table: "students",
+      type: "left",
+      on: [{ left: { table: "stu", column: "id" }, op: "eq",
+             right: { table: "base", column: "student_id" } }],
+    },
+  ],
+  fieldBindings: {
+    student_name: { from: { table: "stu", column: "full_name" } },
+    term: { from: { table: "base", column: "term" } },
+  },
+  identity: { kind: "base_pk" },
+};
+
+const errors = validateProjection(enrollment, tables, "projection");
+```
+
+Refer to the base table as `base` and each join by its `key`. Validation is
+**fail-closed**: a table whose discovery payload lacks `columnsMeta` /
+`uniqueConstraints` is rejected rather than assumed safe, because identity
+uniqueness cannot be proven without them.
+
+### Fan-out
+
+A join that does not cover a unique key of the joined table can match many rows
+per base row. That is allowed — but the base table's primary key is then no
+longer unique per output row, so validation enforces: at most one fan-out join,
+no `base_pk` identity on a fanning join, and an identity that includes a unique
+key of the fanning table. `fanOutJoins` reports which joins fan out.
+
+Composite identities concatenate their parts with `::`, so every part must be
+`NOT NULL` (`CONCAT_WS` skips nulls) and non-text (a value containing `::` could
+collide).
+
+### Building one interactively
+
+`ProjectionDraft` and its helpers exist so a builder UI stays presentational —
+every rule is a pure function of draft + schema:
+
+| Helper | Purpose |
+| --- | --- |
+| `emptyProjectionDraft(baseTable)` | A blank draft |
+| `suggestJoinConditions(base, joinTable, schema, key)` | Pre-fill a join's `ON` from foreign-key metadata |
+| `draftToProjection(draft)` | `ProjectionMapping`, or `null` while incomplete |
+| `draftErrors(draft, schema)` | Validation errors for the current draft |
+| `describeCardinality(draft, schema)` | Per-join `{ joinKey, fansOut }` for inline hints |
+
+Full guide, including how to choose an identity and the current limits:
+[`docs/guides/derived-objects.md`](https://github.com/amani-joseph/clivly.com/blob/master/docs/guides/derived-objects.md).
+
+## Advanced: the `$raw` escape hatch
+
+Entity `filter`s and relationship `via`s accept a `$raw` form — an opaque SQL
+fragment for predicates the operators can't express (e.g. a cross-table
+condition):
 
 ```ts
 defineClivlyConfig({
@@ -56,14 +205,21 @@ defineClivlyConfig({
 });
 ```
 
-**`$raw` is unchecked.** `validateEntitiesConfig` deliberately **skips** `$raw` filters and joins — it can't know your columns are real or your SQL is valid. Two consequences:
+**`$raw` is unchecked.** `validateEntitiesConfig` deliberately **skips** `$raw`
+filters and joins — it can't know your columns are real or your SQL is valid.
+Two consequences:
 
-- **Injection safety is on you.** Never interpolate untrusted input into a `$raw` string; treat it like handwritten SQL. Everything else in the config is escaped for you; `$raw` is not.
+- **Injection safety is on you.** Never interpolate untrusted input into a
+  `$raw` string; treat it like handwritten SQL. Everything else in the config is
+  escaped for you; `$raw` is not.
 - **Mistakes surface at sync time, not config time** — unless you dry-run it.
 
 ### Dry-run with `explainEntitiesConfig`
 
-`explainEntitiesConfig(config, run, options?)` compiles each entity's view and executes it via a runner you provide, so a broken `$raw` fragment fails **before** the first sync. It never throws — it returns one `ExplainResult` per entity:
+`explainEntitiesConfig(config, run, options?)` compiles each entity's view and
+executes it via a runner you provide, so a broken `$raw` fragment fails
+**before** the first sync. It never throws — it returns one `ExplainResult` per
+entity:
 
 ```ts
 import { explainEntitiesConfig } from "@clivly/core";
@@ -81,6 +237,16 @@ for (const r of results) {
 }
 ```
 
+`clivly status` runs the same mapping dry-run as one of its setup checks, so you
+get this during normal verification without wiring it up yourself.
+
+## Related documentation
+
+- **[Connecting to Clivly Cloud](../../docs/guides/connecting-to-clivly-cloud.md)** — the canonical end-to-end guide
+- [`@clivly/sdk` README](../sdk/README.md) — the runtime that consumes this config
+- [`clivly` CLI README](../cli/README.md) — every command and flag
+- [`STABILITY.md`](../../STABILITY.md) — import tiers before 1.0
+
 ## License
 
 MIT

package/dist/drizzle.cjs

@@ -1,7 +1,83 @@
 Object.defineProperty(exports, Symbol.toStringTag, { value: "Module" });
 let drizzle_orm = require("drizzle-orm");
+let drizzle_orm_pg_core = require("drizzle-orm/pg-core");
 //#region src/drizzle.ts
 /**
+* Every unique key on a table, as DB column-name tuples. Primary keys count —
+* fan-out detection asks "do these join columns cover a unique key?", and a PK
+* answers that as well as any explicit unique constraint.
+*
+* Returns `undefined` if reflection fails (non-pg table, or a drizzle version
+* without config access), meaning "unknown, cannot determine". Returns `[]`
+* when reflection succeeds and the table genuinely has none. That distinction
+* is load-bearing: `hasRichMeta` uses it to tell "metadata unavailable" from
+* "confirmed no constraints", and projection validation fails closed on the
+* former.
+*/
+function buildUniqueConstraints(table) {
+	const keys = [];
+	try {
+		const config = (0, drizzle_orm_pg_core.getTableConfig)(table);
+		for (const column of config.columns ?? []) if (column.isUnique || column.primary) keys.push([column.name]);
+		for (const pk of config.primaryKeys ?? []) keys.push(pk.columns.map((c) => c.name));
+		for (const constraint of config.uniqueConstraints ?? []) keys.push(constraint.columns.map((c) => c.name));
+		for (const index of config.indexes ?? []) {
+			if (!index.config?.unique) continue;
+			const columns = (index.config.columns ?? []).map((c) => c?.name).filter((name) => Boolean(name));
+			if (columns.length > 0) keys.push(columns);
+		}
+	} catch {
+		return;
+	}
+	const seen = /* @__PURE__ */ new Set();
+	return keys.filter((key) => {
+		const id = [...key].sort().join("\0");
+		if (seen.has(id) || key.length === 0) return false;
+		seen.add(id);
+		return true;
+	});
+}
+/**
+* Per-column metadata (type, PK, nullability, FK edges) for a Drizzle table.
+* Structural only — no DB round-trip, so it is safe to recompute on every
+* heartbeat. FK reflection is best-effort; a non-Postgres table still gets
+* name/type/PK/nullable.
+*/
+function buildColumnsMeta(table) {
+	const columns = (0, drizzle_orm.getTableColumns)(table);
+	const fkByColumn = /* @__PURE__ */ new Map();
+	try {
+		for (const fk of (0, drizzle_orm_pg_core.getTableConfig)(table).foreignKeys) {
+			const ref = fk.reference();
+			const foreignTable = (0, drizzle_orm.getTableName)(ref.foreignTable);
+			ref.columns.forEach((col, index) => {
+				const foreignCol = ref.foreignColumns[index];
+				if (foreignCol) fkByColumn.set(col.name, {
+					table: foreignTable,
+					column: foreignCol.name
+				});
+			});
+		}
+	} catch {}
+	return Object.values(columns).map((col) => {
+		const references = fkByColumn.get(col.name);
+		let type;
+		try {
+			type = col.getSQLType();
+		} catch {
+			type = void 0;
+		}
+		return {
+			name: col.name,
+			type,
+			nullable: !col.notNull,
+			isPrimaryKey: Boolean(col.primary),
+			isForeignKey: references !== void 0,
+			...references ? { references } : {}
+		};
+	});
+}
+/**
 * Derive Clivly's discovery shape (table + column names) from a Drizzle schema
 * object — the `* as schema` you pass to `drizzle(client, { schema })`. This
 * delivers the "reads your schema" promise for Drizzle hosts: no hand-authored
@@ -21,12 +97,17 @@ function discoverFromDrizzle(schema) {
 	for (const value of Object.values(schema)) {
 		if (!(0, drizzle_orm.is)(value, drizzle_orm.Table)) continue;
 		const columns = (0, drizzle_orm.getTableColumns)(value);
+		const uniqueConstraints = buildUniqueConstraints(value);
 		tables.push({
 			name: (0, drizzle_orm.getTableName)(value),
-			columns: Object.values(columns).map((column) => column.name)
+			columns: Object.values(columns).map((column) => column.name),
+			columnsMeta: buildColumnsMeta(value),
+			...uniqueConstraints === void 0 ? {} : { uniqueConstraints }
 		});
 	}
 	return tables;
 }
 //#endregion
+exports.buildColumnsMeta = buildColumnsMeta;
+exports.buildUniqueConstraints = buildUniqueConstraints;
 exports.discoverFromDrizzle = discoverFromDrizzle;

package/dist/drizzle.d.cts

@@ -1,5 +1,26 @@
-import { DiscoveredSchemaTable } from "./entity-config.cjs";
+import { f as DiscoveredColumnMeta, p as DiscoveredSchemaTable } from "./entity-config-CLxI8dVb.cjs";
+import { Table } from "drizzle-orm";
 //#region src/drizzle.d.ts
+/**
+ * Every unique key on a table, as DB column-name tuples. Primary keys count —
+ * fan-out detection asks "do these join columns cover a unique key?", and a PK
+ * answers that as well as any explicit unique constraint.
+ *
+ * Returns `undefined` if reflection fails (non-pg table, or a drizzle version
+ * without config access), meaning "unknown, cannot determine". Returns `[]`
+ * when reflection succeeds and the table genuinely has none. That distinction
+ * is load-bearing: `hasRichMeta` uses it to tell "metadata unavailable" from
+ * "confirmed no constraints", and projection validation fails closed on the
+ * former.
+ */
+declare function buildUniqueConstraints(table: unknown): string[][] | undefined;
+/**
+ * Per-column metadata (type, PK, nullability, FK edges) for a Drizzle table.
+ * Structural only — no DB round-trip, so it is safe to recompute on every
+ * heartbeat. FK reflection is best-effort; a non-Postgres table still gets
+ * name/type/PK/nullable.
+ */
+declare function buildColumnsMeta(table: Table): DiscoveredColumnMeta[];
 /**
  * Derive Clivly's discovery shape (table + column names) from a Drizzle schema
  * object — the `* as schema` you pass to `drizzle(client, { schema })`. This
@@ -17,4 +38,4 @@ import { DiscoveredSchemaTable } from "./entity-config.cjs";
  */
 declare function discoverFromDrizzle(schema: Record<string, unknown>): DiscoveredSchemaTable[];
 //#endregion
-export { discoverFromDrizzle };
+export { buildColumnsMeta, buildUniqueConstraints, discoverFromDrizzle };

package/dist/drizzle.d.mts

@@ -1,5 +1,26 @@
-import { DiscoveredSchemaTable } from "./entity-config.mjs";
+import { f as DiscoveredColumnMeta, p as DiscoveredSchemaTable } from "./entity-config-CLxI8dVb.mjs";
+import { Table } from "drizzle-orm";
 //#region src/drizzle.d.ts
+/**
+ * Every unique key on a table, as DB column-name tuples. Primary keys count —
+ * fan-out detection asks "do these join columns cover a unique key?", and a PK
+ * answers that as well as any explicit unique constraint.
+ *
+ * Returns `undefined` if reflection fails (non-pg table, or a drizzle version
+ * without config access), meaning "unknown, cannot determine". Returns `[]`
+ * when reflection succeeds and the table genuinely has none. That distinction
+ * is load-bearing: `hasRichMeta` uses it to tell "metadata unavailable" from
+ * "confirmed no constraints", and projection validation fails closed on the
+ * former.
+ */
+declare function buildUniqueConstraints(table: unknown): string[][] | undefined;
+/**
+ * Per-column metadata (type, PK, nullability, FK edges) for a Drizzle table.
+ * Structural only — no DB round-trip, so it is safe to recompute on every
+ * heartbeat. FK reflection is best-effort; a non-Postgres table still gets
+ * name/type/PK/nullable.
+ */
+declare function buildColumnsMeta(table: Table): DiscoveredColumnMeta[];
 /**
  * Derive Clivly's discovery shape (table + column names) from a Drizzle schema
  * object — the `* as schema` you pass to `drizzle(client, { schema })`. This
@@ -17,4 +38,4 @@ import { DiscoveredSchemaTable } from "./entity-config.mjs";
  */
 declare function discoverFromDrizzle(schema: Record<string, unknown>): DiscoveredSchemaTable[];
 //#endregion
-export { discoverFromDrizzle };
+export { buildColumnsMeta, buildUniqueConstraints, discoverFromDrizzle };

package/dist/drizzle.mjs

@@ -1,6 +1,82 @@
 import { Table, getTableColumns, getTableName, is } from "drizzle-orm";
+import { getTableConfig } from "drizzle-orm/pg-core";
 //#region src/drizzle.ts
 /**
+* Every unique key on a table, as DB column-name tuples. Primary keys count —
+* fan-out detection asks "do these join columns cover a unique key?", and a PK
+* answers that as well as any explicit unique constraint.
+*
+* Returns `undefined` if reflection fails (non-pg table, or a drizzle version
+* without config access), meaning "unknown, cannot determine". Returns `[]`
+* when reflection succeeds and the table genuinely has none. That distinction
+* is load-bearing: `hasRichMeta` uses it to tell "metadata unavailable" from
+* "confirmed no constraints", and projection validation fails closed on the
+* former.
+*/
+function buildUniqueConstraints(table) {
+	const keys = [];
+	try {
+		const config = getTableConfig(table);
+		for (const column of config.columns ?? []) if (column.isUnique || column.primary) keys.push([column.name]);
+		for (const pk of config.primaryKeys ?? []) keys.push(pk.columns.map((c) => c.name));
+		for (const constraint of config.uniqueConstraints ?? []) keys.push(constraint.columns.map((c) => c.name));
+		for (const index of config.indexes ?? []) {
+			if (!index.config?.unique) continue;
+			const columns = (index.config.columns ?? []).map((c) => c?.name).filter((name) => Boolean(name));
+			if (columns.length > 0) keys.push(columns);
+		}
+	} catch {
+		return;
+	}
+	const seen = /* @__PURE__ */ new Set();
+	return keys.filter((key) => {
+		const id = [...key].sort().join("\0");
+		if (seen.has(id) || key.length === 0) return false;
+		seen.add(id);
+		return true;
+	});
+}
+/**
+* Per-column metadata (type, PK, nullability, FK edges) for a Drizzle table.
+* Structural only — no DB round-trip, so it is safe to recompute on every
+* heartbeat. FK reflection is best-effort; a non-Postgres table still gets
+* name/type/PK/nullable.
+*/
+function buildColumnsMeta(table) {
+	const columns = getTableColumns(table);
+	const fkByColumn = /* @__PURE__ */ new Map();
+	try {
+		for (const fk of getTableConfig(table).foreignKeys) {
+			const ref = fk.reference();
+			const foreignTable = getTableName(ref.foreignTable);
+			ref.columns.forEach((col, index) => {
+				const foreignCol = ref.foreignColumns[index];
+				if (foreignCol) fkByColumn.set(col.name, {
+					table: foreignTable,
+					column: foreignCol.name
+				});
+			});
+		}
+	} catch {}
+	return Object.values(columns).map((col) => {
+		const references = fkByColumn.get(col.name);
+		let type;
+		try {
+			type = col.getSQLType();
+		} catch {
+			type = void 0;
+		}
+		return {
+			name: col.name,
+			type,
+			nullable: !col.notNull,
+			isPrimaryKey: Boolean(col.primary),
+			isForeignKey: references !== void 0,
+			...references ? { references } : {}
+		};
+	});
+}
+/**
 * Derive Clivly's discovery shape (table + column names) from a Drizzle schema
 * object — the `* as schema` you pass to `drizzle(client, { schema })`. This
 * delivers the "reads your schema" promise for Drizzle hosts: no hand-authored
@@ -20,12 +96,15 @@ function discoverFromDrizzle(schema) {
 	for (const value of Object.values(schema)) {
 		if (!is(value, Table)) continue;
 		const columns = getTableColumns(value);
+		const uniqueConstraints = buildUniqueConstraints(value);
 		tables.push({
 			name: getTableName(value),
-			columns: Object.values(columns).map((column) => column.name)
+			columns: Object.values(columns).map((column) => column.name),
+			columnsMeta: buildColumnsMeta(value),
+			...uniqueConstraints === void 0 ? {} : { uniqueConstraints }
 		});
 	}
 	return tables;
 }
 //#endregion
-export { discoverFromDrizzle };
+export { buildColumnsMeta, buildUniqueConstraints, discoverFromDrizzle };