@collective-intelligence/extensions-sdk-backend 0.2.7 → 0.2.9
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +4 -0
- package/README.md +7 -0
- package/dist/effect/datasets/provider-implementations.d.ts +14 -2
- package/dist/effect.js +1 -1
- package/docs/effect.md +3 -0
- package/docs/guide/ai/index.md +349 -0
- package/docs/guide/data/connectors.md +204 -0
- package/docs/guide/data/datasets.md +156 -0
- package/docs/guide/data/knowledge.md +211 -0
- package/docs/guide/develop/extensions.md +208 -0
- package/docs/guide/develop/external-apps.md +245 -0
- package/docs/guide/develop/sdks.md +233 -0
- package/docs/guide/index.md +78 -0
- package/docs/guide/platform/access.md +244 -0
- package/docs/guide/platform/architecture.md +142 -0
- package/docs/guide/platform/entrypoints.md +236 -0
- package/docs/guide/scenarios/index.md +107 -0
- package/docs/guide/template/index.md +276 -0
- package/docs/guide/template/state-machines.md +294 -0
- package/package.json +4 -4
|
@@ -0,0 +1,204 @@
|
|
|
1
|
+
# Connectors
|
|
2
|
+
|
|
3
|
+
Connect outside systems, map their fields to Knowledge schemas, and run and watch imports.
|
|
4
|
+
|
|
5
|
+
A connector imports records from an outside system, such as Notion, HubSpot or a SQL database, into Knowledge records. A Documents connector does the same for files, which it keeps in a Documents collection. An import passes through several stages: CI reads the source page by page, stages the records, commits the accepted ones into Knowledge, and then indexes them for search. A demo that shows imported data has to wait for the stage it needs.
|
|
6
|
+
|
|
7
|
+
## Connections, templates and instances
|
|
8
|
+
|
|
9
|
+
Three objects work together, and a demo usually declares two of them.
|
|
10
|
+
|
|
11
|
+
A connection holds credentials. A connection template, such as `builtin.integrations.notion.connections.internal-integration-token`, defines the credential form and a test, and an Organization connection holds the actual credentials. Connectors, tools and Dataset providers each point at a connection through a slot, so one connection can serve all of them.
|
|
12
|
+
|
|
13
|
+
A connector template is the import capability that a package ships, for example the "Notion API" connector `builtin.integrations.notion.connectors.api`. A connector instance is one configured use of that template in an Organization, with its settings, connection slots, entity bindings, pipelines, schedules and run state. Notion, for example, allows several instances in one Organization.
|
|
14
|
+
|
|
15
|
+
```mermaid
|
|
16
|
+
flowchart LR
|
|
17
|
+
CT[Connection template] --> OC[Organization connection with credentials]
|
|
18
|
+
KT[Connector template] --> CI[Connector instance]
|
|
19
|
+
CI -- slot --> OC
|
|
20
|
+
CI -- entity bindings --> KS[Knowledge schemas]
|
|
21
|
+
CI -- pipelines --> SC[Schedules: full or delta]
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Credentials
|
|
25
|
+
|
|
26
|
+
A package declares a connection with `declareConnectionInstance`. The secret comes from a package variable, so the release itself never contains it:
|
|
27
|
+
|
|
28
|
+
```ts
|
|
29
|
+
const variables = { notionToken: { type: "secret" } } as const
|
|
30
|
+
const vars = packageVariables(variables)
|
|
31
|
+
|
|
32
|
+
const notionConnection = declareConnectionInstance({
|
|
33
|
+
manifest: { fqn: "acme.connections.notion", version: "1.0.0" },
|
|
34
|
+
presentation: { displayName: "Company Notion" },
|
|
35
|
+
definition: {
|
|
36
|
+
templateFqn:
|
|
37
|
+
"builtin.integrations.notion.connections.internal-integration-token",
|
|
38
|
+
lifecycle: { deployMode: "upsert", readonly: false },
|
|
39
|
+
credentials: { integrationToken: vars.notionToken }
|
|
40
|
+
}
|
|
41
|
+
})
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
At deploy time, the Inputs File links the variable to an Organization secret:
|
|
45
|
+
|
|
46
|
+
```json
|
|
47
|
+
{
|
|
48
|
+
"variables": {
|
|
49
|
+
"notionToken": { "link": "NOTION_INTEGRATION_TOKEN" }
|
|
50
|
+
}
|
|
51
|
+
}
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
`ci deploy --inputs-template` prints every connection slot and variable that the Inputs File must fill. CI stores the link itself as the credential, so a later rotation of the secret reaches the connection. When the linked secret does not exist yet, CI creates it as pending, and the connection makes no provider calls until someone fills in the value. A field that the template declares as secret stays secret even when the setup marks it otherwise.
|
|
55
|
+
|
|
56
|
+
## Declare a connector instance
|
|
57
|
+
|
|
58
|
+
`declareKnowledgeConnectorInstance` names the template, fills its connection slots, binds remote collections to Knowledge schemas and sets the import pipelines. Each connection slot takes exactly one of `{ setupKey }`, which an administrator fills at install, or `{ connectionInstanceFqn }`, which points at a connection that the package declares. The package must also list the integration, such as `{ fqn: "builtin.integrations.notion" }`, in its `dependencies`.
|
|
59
|
+
|
|
60
|
+
```ts
|
|
61
|
+
const customers = declareKnowledgeConnectorInstance({
|
|
62
|
+
manifest: { fqn: "com.acme.sales.instances.customers", version: "1.0.0" },
|
|
63
|
+
presentation: { displayName: "CRM customers" },
|
|
64
|
+
definition: {
|
|
65
|
+
templateFqn: "com.acme.integrations.crm",
|
|
66
|
+
lifecycle: { deployMode: "upsert", readonly: false },
|
|
67
|
+
enabled: true,
|
|
68
|
+
connections: { crm: { setupKey: "crm" } },
|
|
69
|
+
entityBindings: [
|
|
70
|
+
{
|
|
71
|
+
key: "customers",
|
|
72
|
+
schemaFqn: "com.acme.sales.customer",
|
|
73
|
+
bindingMode: "managed",
|
|
74
|
+
remoteCollectionId: "customers",
|
|
75
|
+
fieldMap: [
|
|
76
|
+
{
|
|
77
|
+
op: "path",
|
|
78
|
+
remote: "name",
|
|
79
|
+
local: "name",
|
|
80
|
+
direction: "inbound",
|
|
81
|
+
skipIfNull: false
|
|
82
|
+
}
|
|
83
|
+
]
|
|
84
|
+
}
|
|
85
|
+
],
|
|
86
|
+
pipelines: [
|
|
87
|
+
{
|
|
88
|
+
key: "nightly",
|
|
89
|
+
definition: {
|
|
90
|
+
steps: [{ tasks: [{ kind: "entity", bindingKey: "customers" }] }]
|
|
91
|
+
},
|
|
92
|
+
schedules: [
|
|
93
|
+
{
|
|
94
|
+
key: "nightly_delta",
|
|
95
|
+
cronExpression: "0 2 * * *",
|
|
96
|
+
timezone: "Europe/Rome",
|
|
97
|
+
importMode: "delta",
|
|
98
|
+
enabled: true
|
|
99
|
+
}
|
|
100
|
+
]
|
|
101
|
+
}
|
|
102
|
+
]
|
|
103
|
+
}
|
|
104
|
+
})
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
> **Warning:** Deploying a package that declares a connector instance creates no records and
|
|
108
|
+
> starts no import. Data arrives only when a schedule fires or someone starts
|
|
109
|
+
> the pipeline by hand, so plan for that step before you open the demo.
|
|
110
|
+
|
|
111
|
+
A redeploy follows the `lifecycle` of the instance. `init_only` keeps the existing configuration. `upsert` merges the new declaration, the previous declaration and the live configuration, so a manual edit to an unrelated field survives, and CI refuses changes that compete for the same field. Imported records, run history and cursors are runtime state, and a redeploy never resets them. When you remove a binding from the declaration, CI archives it; when you remove a pipeline, CI disables it. Imported records stay in both cases.
|
|
112
|
+
|
|
113
|
+
## Discovery and field mapping
|
|
114
|
+
|
|
115
|
+
During setup, a connector reports the remote collections and fields it finds, each with a type hint such as `{ kind: "datetime" }`. Knowledge setup uses the hint to choose a local field type. Notion dates, for example, arrive as `datetime` because a Notion date can carry a time.
|
|
116
|
+
|
|
117
|
+
Running setup again does not repair a field that already exists. Setup skips remote models that are already bound, and an existing field keeps its type. To change a `date` field to `datetime` later, update both the schema property and the field definition.
|
|
118
|
+
|
|
119
|
+
A binding's `fieldMap` lists the rules that copy remote values into local fields.
|
|
120
|
+
|
|
121
|
+
| Rule | What it does |
|
|
122
|
+
| ----------- | -------------------------------------------------------------------------------------------------------------- |
|
|
123
|
+
| `path` | Copies one remote value into a local field |
|
|
124
|
+
| `reference` | Stores remote IDs as relation field references to records of another binding of the same connector instance |
|
|
125
|
+
| `embedded` | Maps a nested remote object into a record of another binding, creates that record when needed, and links to it |
|
|
126
|
+
|
|
127
|
+
The `remote` value is a dot path with at most one `[]` segment, which takes every item of an array. This `reference` rule fills a `companies` relation field from HubSpot-style associations:
|
|
128
|
+
|
|
129
|
+
```ts
|
|
130
|
+
{
|
|
131
|
+
op: "reference",
|
|
132
|
+
local: "companies",
|
|
133
|
+
remote: "associations.companies.results[].id",
|
|
134
|
+
direction: "inbound",
|
|
135
|
+
skipIfNull: true,
|
|
136
|
+
}
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Each entity binding has a binding mode, `managed` or `external`, and the template declares which modes it supports. Notion supports both. An `external` binding refuses the managed-only `updateMode` and `outboundStrategy` options.
|
|
140
|
+
|
|
141
|
+
## Imports
|
|
142
|
+
|
|
143
|
+
```mermaid
|
|
144
|
+
flowchart LR
|
|
145
|
+
T[Schedule or manual start] --> P[Pull source pages]
|
|
146
|
+
P --> S[Staged records]
|
|
147
|
+
S --> C[Commit into Knowledge]
|
|
148
|
+
C --> I[Embedding and indexing]
|
|
149
|
+
```
|
|
150
|
+
|
|
151
|
+
A FULL import lists everything in the source. A DELTA import reads only what changed since the Delta Since time, which is the start of the binding's last successful phase minus the connector's overlap. A failed phase does not move Delta Since. The overlap (`deltaOverlapSeconds`, 300 seconds by default and 120 for Notion) reads the edge of the window again to catch late changes. The Notion connector recommends a delta schedule every five minutes. A SQL connector can run a delta import only when you choose a watermark column for the binding, the column that records when a row changed; without one, it runs full imports only.
|
|
152
|
+
|
|
153
|
+
The import writes staged records first and then commits the accepted records into Knowledge. Treat an import as finished only when CI has committed the staged records and the indexing work that follows. Before each remote page, the import saves its cursor and counters. When a worker stops in the middle of a call, the next attempt repeats at most that one page, and repeating a page is safe because page commits are idempotent. Full imports resume from their cursor as well, so a restart does not scan the whole source again.
|
|
154
|
+
|
|
155
|
+
When a source value does not fit the schema field type, for example a Notion date with a time going into a `date` field, the batch fails with a non-retryable `entity_data` error that names the record. The run stays failed until you fix the data or the field type.
|
|
156
|
+
|
|
157
|
+
## Deletions
|
|
158
|
+
|
|
159
|
+
Every Knowledge connector supports deletion on full runs. A full run deletes the records that are missing from a complete listing of the source, and a page that cannot prove the listing complete deletes nothing.
|
|
160
|
+
|
|
161
|
+
> **Note:** Deletion on delta runs is planned. Until then, count on full runs to remove
|
|
162
|
+
> records that were deleted at the source.
|
|
163
|
+
|
|
164
|
+
For a Notion demo, this means that a page deleted in Notion leaves CI on the next full run and stays through the five-minute delta runs.
|
|
165
|
+
|
|
166
|
+
## Health
|
|
167
|
+
|
|
168
|
+
Health belongs to the Organization connection, and CI copies it onto each connector that uses that connection. A failed connection test marks the connection `unhealthy` and records the error. In Knowledge setup, **Refresh data sources** runs the connection refresh again before it updates the connector.
|
|
169
|
+
|
|
170
|
+
An operator follows an import in several places: the connection test result, the readiness of each binding, the status of each pipeline run, the counts of staged records, and the embedding work that follows the commit.
|
|
171
|
+
|
|
172
|
+
## Built-in integrations
|
|
173
|
+
|
|
174
|
+
The platform ships these integrations. Some import data, and others add only connections, tools and workflow nodes.
|
|
175
|
+
|
|
176
|
+
| Role | Integrations |
|
|
177
|
+
| ----------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- |
|
|
178
|
+
| Knowledge connectors | Notion, HubSpot, Monday, SQL (PostgreSQL and MySQL), Structured File (with an FTP source), the external application Knowledge gateway |
|
|
179
|
+
| Documents connectors | Microsoft 365 Drive, Google Drive |
|
|
180
|
+
| Dataset providers | Microsoft Fabric, Databricks |
|
|
181
|
+
| Connections, tools and actions only | Apollo, Lusha, Slack, Telegram, Twilio, WhatsApp |
|
|
182
|
+
|
|
183
|
+
Lusha shows the difference. It has an API key connection and prospecting tools and workflow actions, and it imports nothing. Use it to enrich a Lead from a button, and use a connector such as HubSpot to sync a CRM. The [Datasets](datasets.md) page covers Fabric and Databricks.
|
|
184
|
+
|
|
185
|
+
## Knowledge sync and Documents sync
|
|
186
|
+
|
|
187
|
+
Knowledge connectors and Documents connectors have different owners and different rules. A Documents connector never creates Knowledge records, and it does not use the Knowledge import stages or the Knowledge deletion rules.
|
|
188
|
+
|
|
189
|
+
| | Knowledge connector | Documents connector |
|
|
190
|
+
| ------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
191
|
+
| Writes | Knowledge records in bound schemas | Files and versions in one Connected collection |
|
|
192
|
+
| Configured by | A connector instance with entity bindings and pipelines | One Documents connector instance per Connected collection |
|
|
193
|
+
| Schedule | Cron schedules per pipeline, FULL or DELTA | CI checks every minute and starts a sync five minutes after the last attempt finished |
|
|
194
|
+
| Package declaration | `declareKnowledgeConnectorInstance` | `declareDocumentCollection` with a connection slot and remote folders |
|
|
195
|
+
| Folder exclusions | Not applied | Excluded Folder Names skip whole folders |
|
|
196
|
+
|
|
197
|
+
A Documents sync moves its provider checkpoint forward only after it applies the changes. Two Connected collections on the same provider account cannot overlap their root folders. A finished Documents sync still has to index the files before search finds them, as the [Knowledge](knowledge.md) page explains.
|
|
198
|
+
|
|
199
|
+
## Related pages
|
|
200
|
+
|
|
201
|
+
- [Knowledge](knowledge.md)
|
|
202
|
+
- [Datasets](datasets.md)
|
|
203
|
+
- [Extensions](../develop/extensions.md)
|
|
204
|
+
- [Access](../platform/access.md)
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
# Datasets
|
|
2
|
+
|
|
3
|
+
Query typed tables of measurements, events and warehouse data, and control who can read them.
|
|
4
|
+
|
|
5
|
+
A Dataset is a typed collection of rows that belongs to an Organization. Each Dataset has a stable identity, a versioned definition, a policy that says who may query it, and provenance on every result. Datasets hold data that has many rows and few edits, such as meter readings, monthly sales figures or a table in a customer's data warehouse. Agents, workflows and Studio query that data in bounded, typed requests.
|
|
6
|
+
|
|
7
|
+
## Dataset or Knowledge schema
|
|
8
|
+
|
|
9
|
+
Knowledge models things that people talk about, edit, link and search, with one record per thing. A Dataset models observations about those things, with one row per measurement, event or transaction. The platform does not create a Knowledge record for each row.
|
|
10
|
+
|
|
11
|
+
| | Knowledge schema | Dataset |
|
|
12
|
+
| -------------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
|
|
13
|
+
| One row or record is | A real thing: a customer, a Lead, a goal | An observation: a reading, a transaction, an event |
|
|
14
|
+
| People edit rows | Yes, in forms, Studio and the API | No; rows come from file imports or from the source system |
|
|
15
|
+
| Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema |
|
|
16
|
+
| Queries | Field filters, text and semantic search, relation paths | `records`, `aggregate` and `timeSeries` |
|
|
17
|
+
| Access | Knowledge data policies across schemas and domains | One policy per Dataset with an explicit list of principals |
|
|
18
|
+
|
|
19
|
+
A strategy Metric shows the split. The Metric itself is a Knowledge record with an owner, a target, a unit, a status and links to goals, and its history of measured values is a Dataset. A Dataset field such as `customerId` can bind to the Customer schema in Knowledge. That binding links the two without copying rows into Knowledge, and deleting one side does not delete the other.
|
|
20
|
+
|
|
21
|
+
```mermaid
|
|
22
|
+
flowchart LR
|
|
23
|
+
subgraph K[Knowledge]
|
|
24
|
+
KS[Customer schema]
|
|
25
|
+
end
|
|
26
|
+
subgraph D[Datasets]
|
|
27
|
+
DS[Orders Dataset]
|
|
28
|
+
end
|
|
29
|
+
K --- KC[Records, relation fields, semantic search]
|
|
30
|
+
D --- DC[Typed rows, measures, aggregates, time buckets]
|
|
31
|
+
DS -. optional binding on customerId .-> KS
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Managed and External Datasets
|
|
35
|
+
|
|
36
|
+
A Dataset has one of two backings. CI stores the rows of a Managed Dataset in PostgreSQL, in the Organization Dataset Store. The rows of an External Dataset stay in the customer's system, and CI queries them live. Both backings use the same query contract, so tools, workflows and Studio do not need to know which one they read. The backing shows only in the provenance, the capabilities and the freshness of a result.
|
|
37
|
+
|
|
38
|
+
CI keeps no hidden copy of External data. Every query reaches the provider and CI caches no results, so when the provider is down the query fails with no fallback.
|
|
39
|
+
|
|
40
|
+
| Provider | Backing | Source |
|
|
41
|
+
| ----------------------------------------------------------------------- | -------- | ---------------------------------------------------------- |
|
|
42
|
+
| `builtin.core.datasets.providers.postgres` | Managed | Rows stored in CI |
|
|
43
|
+
| Microsoft Fabric (package `builtin.integrations.microsoft-fabric`) | External | Fabric Warehouses and Lakehouse SQL analytics endpoints |
|
|
44
|
+
| Databricks (package `builtin.integrations.databricks`) | External | Unity Catalog tables and views through one SQL Warehouse |
|
|
45
|
+
| Remote Gateway (`builtin.core.external-apps.providers.remote-external`) | External | A customer server that exposes rows through the server SDK |
|
|
46
|
+
|
|
47
|
+
The Fabric and Databricks packages do not install automatically, so install the package before you declare a Dataset that uses it. A Fabric connection holds one service principal from the customer's Microsoft Entra tenant, with a tenant ID, a client ID and a client secret or certificate. A Databricks connection uses service principal OAuth and rejects personal access tokens. A Fabric source location has exactly three parts: workspace, item and object. To combine several sources, combine them in the customer's warehouse and point the Dataset at the result. New Lakehouse tables can take a few minutes to appear on the Fabric SQL analytics endpoint, so refresh the metadata in Fabric if a new table is missing.
|
|
48
|
+
|
|
49
|
+
A Remote Gateway Dataset answers `records` queries only, with at most 1,000 rows, 1 MiB and 60 seconds per request. CI checks the Dataset policy first, and the customer server can narrow the result further but cannot widen it.
|
|
50
|
+
|
|
51
|
+
## Load a Managed Dataset
|
|
52
|
+
|
|
53
|
+
In the Dashboard you create a Dataset in one of two ways: import a structured file as a Managed Dataset, or connect a compatible External source. A Managed import accepts CSV, Excel `.xlsx`, JSON arrays of records, JSON Lines and Parquet. CI inspects the file and proposes field types, identity fields and candidate time fields, and a person reviews and approves the proposal before CI loads any rows. There is no spreadsheet-style row editor.
|
|
54
|
+
|
|
55
|
+
Ingestion runs in the background. The API answers `202 Accepted` with a run ID, and you follow the run until it finishes. Each run uses one mode: `append` adds rows, `upsert` inserts or updates rows by their identity fields, and `replace` swaps all visible rows in one step. If an `upsert` run contains the same identity twice, CI rejects the run and does not pick one row for you.
|
|
56
|
+
|
|
57
|
+
The table template of the Dataset decides which modes it allows.
|
|
58
|
+
|
|
59
|
+
| Template | Purpose | Modes |
|
|
60
|
+
| ----------------- | -------------------------------- | ----------------------------- |
|
|
61
|
+
| `generic-table` | General typed records | `append`, `upsert`, `replace` |
|
|
62
|
+
| `entity-snapshot` | Current state keyed by entity | `upsert`, `replace` |
|
|
63
|
+
| `time-series` | Time-indexed measures | `append` |
|
|
64
|
+
| `event-log` | Append-only events | `append` |
|
|
65
|
+
| `fact-table` | Dimensions, measures and periods | `append`, `upsert` |
|
|
66
|
+
| `status-update` | Append-only status history | `append` |
|
|
67
|
+
|
|
68
|
+
## Query a Dataset
|
|
69
|
+
|
|
70
|
+
A Dataset query has one of three kinds. `records` returns selected rows with filters, sort and pagination. `aggregate` groups measures by dimensions with `count`, `countDistinct`, `sum`, `average`, `minimum` and `maximum`. `timeSeries` buckets measures over a time field, with a grain from `minute` to `year`.
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
type RecordsQuery = Readonly<{
|
|
74
|
+
kind: "records"
|
|
75
|
+
datasetId: string
|
|
76
|
+
definitionRevision?: string
|
|
77
|
+
select: readonly string[]
|
|
78
|
+
filters?: DatasetFilter
|
|
79
|
+
sort?: readonly DatasetSort[]
|
|
80
|
+
page: DatasetPageRequest
|
|
81
|
+
}>
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
A Managed `records` query returns at most 50 fields, 100 rows and 1 MiB per response. Filters are combined with AND and use `eq`, `ne`, `lt`, `lte`, `gt`, `gte`, `contains` and `isNull`. CI returns 64-bit integers and decimals as exact JSON strings, so parse them before you do arithmetic.
|
|
85
|
+
|
|
86
|
+
Datasets accept no SQL. Calculated measures use a constrained expression model, and an operation that the provider cannot run fails with a typed capability error. CI never loads a whole table to compute a result in memory.
|
|
87
|
+
|
|
88
|
+
> **Note:** The External providers that ship today answer `records` and `aggregate`, and
|
|
89
|
+
> the Remote Gateway answers `records` only. Use a Managed Dataset when the demo
|
|
90
|
+
> needs `timeSeries`.
|
|
91
|
+
|
|
92
|
+
Every result carries its provenance: the Dataset ID, the definition revision, the query kind, the backing, the freshness when the provider knows it, a correlation ID, the timing, and warnings for partial or stale results. Provenance never contains credentials or physical table names.
|
|
93
|
+
|
|
94
|
+
## Access
|
|
95
|
+
|
|
96
|
+
Each Dataset has its own policy, and the policy lists its principals by ID. It holds `principals` (each a `user` or `service` with an `id`), `allowedOperations` (`describe`, `records`, `aggregate`, `time_series`), `allowedFields`, `sensitiveFields`, an optional `rowFilter`, and the limits `maxRows`, `maxBytes` and `maxTimeoutMs`. A user also needs the Organization permission to query Datasets, and a Dataset grant does not give Organization membership or capabilities.
|
|
97
|
+
|
|
98
|
+
Knowledge access works differently: Knowledge data policies apply across schemas and domains. The [Access](../platform/access.md) page explains both models.
|
|
99
|
+
|
|
100
|
+
> **Warning:** A package-declared Dataset without a `dataPolicy` grants access to nobody,
|
|
101
|
+
> including the person who deployed it. Always declare the principals that the
|
|
102
|
+
> demo needs.
|
|
103
|
+
|
|
104
|
+
## Declare a Dataset in a package
|
|
105
|
+
|
|
106
|
+
`declareDatasetConnectorInstance` declares a Dataset in the demo package. An External declaration names the provider, a connection slot, the source location and the policy:
|
|
107
|
+
|
|
108
|
+
```ts
|
|
109
|
+
const orders = declareDatasetConnectorInstance({
|
|
110
|
+
manifest: { fqn: "com.acme.sales.instances.orders", version: "1.0.0" },
|
|
111
|
+
presentation: { displayName: "Warehouse orders" },
|
|
112
|
+
definition: {
|
|
113
|
+
lifecycle: { deployMode: "init_only", readonly: false },
|
|
114
|
+
bindingMode: "external",
|
|
115
|
+
providerFqn: "com.acme.integrations.warehouse",
|
|
116
|
+
connection: { setupKey: "warehouse" },
|
|
117
|
+
sourceLocation: {
|
|
118
|
+
segments: [
|
|
119
|
+
{ kind: "container", opaqueId: "sales", label: "Sales" },
|
|
120
|
+
{ kind: "dataset", opaqueId: "table:orders", label: "Orders" }
|
|
121
|
+
]
|
|
122
|
+
},
|
|
123
|
+
dataPolicy: {
|
|
124
|
+
principals: [{ kind: "user", id: "7f824c46-..." }],
|
|
125
|
+
allowedOperations: ["describe", "records"],
|
|
126
|
+
allowedFields: ["id", "total"],
|
|
127
|
+
maxRows: 1000,
|
|
128
|
+
maxBytes: 1048576
|
|
129
|
+
}
|
|
130
|
+
}
|
|
131
|
+
})
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
A deploy configures Datasets and loads no data. An External Dataset is ready to query live once its connection works, and its optional schedules run health checks or propose definition changes for review. A Managed declaration creates an empty Dataset, and loading rows stays a manual import. CI refuses a Managed declaration when the Organization Dataset Store is not enabled. Removing the package never deletes Managed rows.
|
|
135
|
+
|
|
136
|
+
## Datasets for agents and workflows
|
|
137
|
+
|
|
138
|
+
Agents use one tool, Query Dataset (`builtin.core.datasets.tools.query`). It lists, describes and queries the Datasets that an agent may use, with bounded `records`, `aggregate` and `timeSeries` requests. You choose those Datasets by ID in the Skill binding, and an empty selection gives the agent no Datasets. The agent never passes SQL, provider names, connections or table locations. The [AI](../ai/index.md) page explains Skills and tool bindings.
|
|
139
|
+
|
|
140
|
+
Workflows have one node for each operation: describe, query records, query aggregate and query time series. Studio shows Datasets in their own area, apart from the Knowledge views.
|
|
141
|
+
|
|
142
|
+
## Pick the right tool for a demo
|
|
143
|
+
|
|
144
|
+
- The customer has a warehouse in Fabric or Databricks: use an External Dataset and query it live.
|
|
145
|
+
- The customer has a CSV or Excel export and wants to see trends: import it into a Managed Dataset.
|
|
146
|
+
- Users must edit records and link them to each other: use Knowledge.
|
|
147
|
+
|
|
148
|
+
Datasets are sized for about 100,000 to 1,000,000 data points per Organization and do not fit a high-volume telemetry stream. Joins across Datasets, writes back to External Datasets, streaming ingestion and copying External data into a Managed Dataset are not available. A route for external applications to query Datasets directly is planned; today agents, workflows and Studio run the queries.
|
|
149
|
+
|
|
150
|
+
## Related pages
|
|
151
|
+
|
|
152
|
+
- [Knowledge](knowledge.md)
|
|
153
|
+
- [Connectors](connectors.md)
|
|
154
|
+
- [Access](../platform/access.md)
|
|
155
|
+
- [AI](../ai/index.md)
|
|
156
|
+
- [Extensions](../develop/extensions.md)
|
|
@@ -0,0 +1,211 @@
|
|
|
1
|
+
# Knowledge
|
|
2
|
+
|
|
3
|
+
Declare schemas, store typed records, query them, and read and watch them from a demo app.
|
|
4
|
+
|
|
5
|
+
Knowledge stores the business records of a demo, such as Leads and Opportunities. A schema defines the fields that a record type can have, and a record is one instance that matches that schema. Schemas live in a Knowledge domain, a named container that the Organization owns. In the template, the extension package `acme.app.knowledge` declares one domain with a Lead schema and an Opportunity schema, and the demo app reads and writes Lead and Opportunity records through the SDKs.
|
|
6
|
+
|
|
7
|
+
```mermaid
|
|
8
|
+
flowchart LR
|
|
9
|
+
P[Extension package] --> D[Knowledge domain]
|
|
10
|
+
D --> L[Lead schema]
|
|
11
|
+
D --> O[Opportunity schema]
|
|
12
|
+
L --> LRec[Lead records]
|
|
13
|
+
O --> ORec[Opportunity records]
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Schemas that a package or a CLI deploy creates have the definition type `custom`, and a custom schema can point a relation field at one of the platform's `builtin` schemas.
|
|
17
|
+
|
|
18
|
+
## Field kinds
|
|
19
|
+
|
|
20
|
+
A schema combines ordinary value fields with three special kinds. Each special kind has its own declaration block, and the platform generates its value schema.
|
|
21
|
+
|
|
22
|
+
| Kind | Declared in | Stored value | Use it for |
|
|
23
|
+
| -------------- | ----------------------------------- | --------------------------------------------------------------------------------------------------------------- | -------------------------------------------------- |
|
|
24
|
+
| Value field | `schema` properties | `string`, `number` (with `numberKind: "integer"` for whole numbers), `datetime`, `string_select` with `options` | Names, amounts, dates, status values |
|
|
25
|
+
| Relation field | `fieldSemantics["/<key>"].relation` | One record ID or an array of record IDs | Links between records, such as Opportunity to Lead |
|
|
26
|
+
| User field | `userFields` | One user ID or an ordered list of user IDs | Owners, reviewers, the users who act for a record |
|
|
27
|
+
| File field | `fileFields` | A File value with `id`, `url`, `filename`, `mimeType`, `size` and `source` | Attachments such as a signed PDF |
|
|
28
|
+
|
|
29
|
+
## Declare a schema in a package
|
|
30
|
+
|
|
31
|
+
A demo declares its schemas in code, inside its extension package. This is the template's Lead schema, shortened:
|
|
32
|
+
|
|
33
|
+
```ts
|
|
34
|
+
declareKnowledgeEntitySchema({
|
|
35
|
+
definition: {
|
|
36
|
+
domainFqn: domain.manifest.fqn,
|
|
37
|
+
fileFields: {
|
|
38
|
+
attachment: {
|
|
39
|
+
accepts: ["application/pdf", "image/jpeg", "image/png"],
|
|
40
|
+
maxBytes: 5 * 1024 * 1024,
|
|
41
|
+
multiple: false,
|
|
42
|
+
visibility: "private"
|
|
43
|
+
}
|
|
44
|
+
},
|
|
45
|
+
lifecycle: { deployMode: "upsert", readonly: true },
|
|
46
|
+
schema: knowledgeRecordSchema({
|
|
47
|
+
additionalProperties: false,
|
|
48
|
+
properties: {
|
|
49
|
+
company: { required: true, type: "string" },
|
|
50
|
+
email: { required: true, type: "string" },
|
|
51
|
+
name: { minLength: 1, required: true, type: "string" },
|
|
52
|
+
score: { numberKind: "integer", type: "number" },
|
|
53
|
+
status: {
|
|
54
|
+
options: ["converted", "open", "qualified"],
|
|
55
|
+
type: "string_select"
|
|
56
|
+
},
|
|
57
|
+
createdAt: { type: "datetime" }
|
|
58
|
+
}
|
|
59
|
+
})
|
|
60
|
+
},
|
|
61
|
+
manifest: { fqn: "acme.app.schemas.lead", version },
|
|
62
|
+
presentation: {
|
|
63
|
+
displayName: "Lead",
|
|
64
|
+
description: "A prospective Acme customer."
|
|
65
|
+
}
|
|
66
|
+
})
|
|
67
|
+
```
|
|
68
|
+
|
|
69
|
+
Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`, Dashboard administrators cannot edit the definition, so the code stays the only source of truth. Choose `readonly: true` when the demo code owns the schema and you redeploy to change it, and `readonly: false` when a customer administrator should keep editing the schema in the Dashboard after the deploy.
|
|
70
|
+
|
|
71
|
+
The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the demo app (its capabilities, the fields you can filter, sort and search on, and the create and update shapes) and never reaches CI. CI receives only the schema above, so when you add a field, change both files.
|
|
72
|
+
|
|
73
|
+
The CLI puts the schema into an Organization in five steps: `ci package build`, `validate`, `publish`, `promote` and `install --organization <id>`. The [Extensions](../develop/extensions.md) page shows the full commands.
|
|
74
|
+
|
|
75
|
+
## Relation fields
|
|
76
|
+
|
|
77
|
+
A relation field holds references to records of exactly one target schema. Declare it under `fieldSemantics` with a target schema and a cardinality, and leave the property out of the value schema because the platform generates its value schema:
|
|
78
|
+
|
|
79
|
+
```ts
|
|
80
|
+
fieldSemantics: {
|
|
81
|
+
"/companies": {
|
|
82
|
+
relation: {
|
|
83
|
+
targetSchemaFqn: "core.schemas.crm.company",
|
|
84
|
+
cardinality: "many_to_many",
|
|
85
|
+
onDelete: "restrict",
|
|
86
|
+
},
|
|
87
|
+
},
|
|
88
|
+
}
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Cardinality reads from the source record to the target record. "One" means at most one, so a link is never required.
|
|
92
|
+
|
|
93
|
+
| Cardinality | Targets per source | Sources per target |
|
|
94
|
+
| -------------- | ------------------ | ------------------ |
|
|
95
|
+
| `one_to_one` | At most one | At most one |
|
|
96
|
+
| `one_to_many` | Many | At most one |
|
|
97
|
+
| `many_to_one` | At most one | Many |
|
|
98
|
+
| `many_to_many` | Many | Many |
|
|
99
|
+
|
|
100
|
+
To show the same links from the other side, declare an inverse field on the target schema with `inverseOf: { schemaFqn, field }`. An inverse field is read-only, and a write to it fails, even a write of `null` or an empty array. `onDelete` controls what happens when someone physically deletes a target record: `clear` (the default) removes the reference, and `restrict` refuses the delete. A logical delete keeps the references.
|
|
101
|
+
|
|
102
|
+
> **Warning:** If you declare `Company.contacts` and `Contact.company` as two forward fields,
|
|
103
|
+
> you get two separate relationships that do not stay in sync. Declare one
|
|
104
|
+
> forward field and make the other side its inverse.
|
|
105
|
+
|
|
106
|
+
In the template, Opportunity stores `leadId` as a plain required string. That is enough to display the ID. If your demo needs to navigate from an Opportunity to its Lead, expand the Lead in one query, or filter on `lead.company`, replace `leadId` with a `many_to_one` relation field that targets the Lead schema.
|
|
107
|
+
|
|
108
|
+
## User fields and file fields
|
|
109
|
+
|
|
110
|
+
A User field stores the ID of one Organization member or External App User, or an ordered list of them. The platform resolves names, emails and avatars when it displays the field and never stores them. Selecting a user in the field does not assign work, send a notification or grant access.
|
|
111
|
+
|
|
112
|
+
```ts
|
|
113
|
+
userFields: {
|
|
114
|
+
owner: { multiple: false },
|
|
115
|
+
reviewers: { multiple: true },
|
|
116
|
+
}
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
One User field per schema can set `accounts: true`. The users in that field act for the record, and access rules can test it with `isMe("accounts")` or `includesMe("accounts")`, for example to let a user write only the records that list them. The platform owns this field: package policies, app users and agent tools cannot change it, but a workflow `add-account` step can. The [Access](../platform/access.md) page explains the rules that read it.
|
|
120
|
+
|
|
121
|
+
A file field stores files. Its read value is `{ id, url, filename, mimeType, size, source }`, where `source` is `managed` when CI stores the bytes and `external_url` when the file stays at a remote URL. To upload, the app asks CI for an upload slot. The slot is single use and belongs to one file field. CI refuses an upload larger than `maxBytes`, and the template also checks the size in the browser so that it sends nothing too large. To update the Lead attachment, the template sends `{ upload }` for a new file, `{ id }` to keep the current file, or `null` to remove it.
|
|
122
|
+
|
|
123
|
+
## Query records
|
|
124
|
+
|
|
125
|
+
Field filters narrow records with `{ field, operator, value }` and operators such as `eq`, `in`, `exists` and `is_empty`, and text search matches plain text. Semantic search ranks records by embeddings, so it works only on a schema with embeddings enabled, and each result carries its `match.semantic_score`.
|
|
126
|
+
|
|
127
|
+
The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an admin edit, CI reprocesses the existing records automatically. You do not need a new import or an Organization reset.
|
|
128
|
+
|
|
129
|
+
> **Warning:** `embedEntities` alone gives no semantic search over records. Select at least
|
|
130
|
+
> one field with `includeInItemEmbedding`, or the schema has no record vector
|
|
131
|
+
> and shows a configuration status instead.
|
|
132
|
+
|
|
133
|
+
A filter path such as `company.industry` follows declared relation fields for at most three hops, and `include` returns the related records in an `included` tree beside the references. Includes go at most three levels deep and return at most 100 readable records per field, with a `truncated` marker when they stop. Each included record passes the same read check as a direct read, so `include` never shows a record that the caller cannot read. Relevance-ranked search rejects filters on relation fields.
|
|
134
|
+
|
|
135
|
+
> **Note:** Knowledge has no graph database and no graph traversal. Older material that
|
|
136
|
+
> describes graph edges or relation bindings refers to a retired model. Relation
|
|
137
|
+
> paths and `include` follow the relation fields that the schema declares.
|
|
138
|
+
|
|
139
|
+
## Retrieval for agents
|
|
140
|
+
|
|
141
|
+
Agents and workflows read Knowledge through built-in tools. Knowledge Query (`builtin.core.knowledge.tools.query`) reads Knowledge only, and Documents Query (`builtin.core.documents.tools.query`) searches Documents only. Advanced Information Retrieval (`builtin.core.information.tools.advanced-retrieval`) is the only tool that searches both, and it does so by default.
|
|
142
|
+
|
|
143
|
+
```mermaid
|
|
144
|
+
flowchart LR
|
|
145
|
+
K[Knowledge Query] --> KP[Knowledge policy and record filters]
|
|
146
|
+
D[Documents Query] --> DP[Documents policy and collection paths]
|
|
147
|
+
A[Advanced Information Retrieval] --> KP
|
|
148
|
+
A --> DP
|
|
149
|
+
KP --> R[PostgreSQL ranking]
|
|
150
|
+
DP --> R
|
|
151
|
+
R --> C[Live authorization recheck]
|
|
152
|
+
C --> E[Results]
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
PostgreSQL holds the source of truth and runs semantic search with pgvector. The tool input has no field for a provider, a vector or a ranking setting, so an agent cannot change how CI routes the query. Before CI returns a result, it checks the current policies again, which means that a revoked grant takes effect on the next query. The [AI](../ai/index.md) page covers tools and Skills.
|
|
156
|
+
|
|
157
|
+
## Documents and Knowledge
|
|
158
|
+
|
|
159
|
+
Knowledge holds typed records with fields. Documents holds files in collections and folders, with versions and permissions per path. A Document Collection is either Local, where people upload files, or Connected, where a Documents connector keeps it in sync with folders in Microsoft 365 or Google Drive.
|
|
160
|
+
|
|
161
|
+
The two areas share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both areas also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization.
|
|
162
|
+
|
|
163
|
+
A finished sync does not mean that the new files are searchable yet, because indexing runs after the sync. A workflow Wait at level `sync` waits for the import only, while level `indexing` also waits for indexing. Use `indexing` before a demo step that searches the new content. The [Connectors](connectors.md) page compares Knowledge sync with Documents sync.
|
|
164
|
+
|
|
165
|
+
## Read and write from an app
|
|
166
|
+
|
|
167
|
+
A demo app reaches records in two ways, and the template uses both.
|
|
168
|
+
|
|
169
|
+
| | Browser | Server |
|
|
170
|
+
| ----------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
|
171
|
+
| API | Entity hooks from `defineEntity({ contract, config })` in `@collective-intelligence/external-apps-client-sdk-react/resources` | `ServerClient.records` from `@collective-intelligence/external-apps-server-sdk/effect` |
|
|
172
|
+
| Acts as | The signed-in user | The app credential (`CI_CLIENT_ID`, `CI_CLIENT_SECRET`) |
|
|
173
|
+
| Operations | Whatever the entity contract allows: create, get, search, update, delete, upload | `create`, `get`, `search`, `update`, `delete` |
|
|
174
|
+
| Typical use | Forms, lists and tables | Server routes and the template's Read Port |
|
|
175
|
+
|
|
176
|
+
The server client takes the Organization and the schema on each call, and the schema FQN works as `schemaId`. A missing record fails with the `CiError` code `not_found`. This is the template's server read, without the wrapper that turns `not_found` into an empty result:
|
|
177
|
+
|
|
178
|
+
```ts
|
|
179
|
+
const record =
|
|
180
|
+
yield *
|
|
181
|
+
client.records.get({
|
|
182
|
+
id,
|
|
183
|
+
organizationId: options.organizationId,
|
|
184
|
+
schemaId: leadSchemaFqn
|
|
185
|
+
})
|
|
186
|
+
const fields = yield * decodeLead(record.value)
|
|
187
|
+
return { ...fields, id: record.id }
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
`records.search` accepts a filter, text search, sort, pagination and `include`. App filtering goes through `POST /application/v1/resources/:schemaId/search`, and relation field pickers use `POST /application/v1/knowledge/autocomplete`. The SDKs do not wrap semantic search; agents use the retrieval tools for that.
|
|
191
|
+
|
|
192
|
+
> **Warning:** The server reads with the app credential, so that credential itself needs read
|
|
193
|
+
> access to Leads. CI accepts a user's token only when the user holds the
|
|
194
|
+
> application gate. See [Access](../platform/access.md).
|
|
195
|
+
|
|
196
|
+
## Change events
|
|
197
|
+
|
|
198
|
+
Every write to a record produces a change event, whether the write comes from the demo app, Studio, an agent or a workflow. Database triggers write a change log entry for each changed record, with one entry per record per transaction. A sync that changes nothing writes no entry.
|
|
199
|
+
|
|
200
|
+
An app subscribes to a topic per schema, for example `{ schema: "acme.app.schemas.lead" }`. `useRealtimeTopic(topic, changes => ...)` receives batches of changes that carry record IDs only, and `useEntityChanges(contract)` gives a function that refreshes the entity reads for those IDs. CI checks read access for each change before it sends it. When CI cannot decide whether a reader may see a delete, it sends that reader a `reset` instead of the ID, and the client must then refetch.
|
|
201
|
+
|
|
202
|
+
To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or the convert-lead workflow.
|
|
203
|
+
|
|
204
|
+
## Related pages
|
|
205
|
+
|
|
206
|
+
- [Datasets](datasets.md)
|
|
207
|
+
- [Connectors](connectors.md)
|
|
208
|
+
- [Access](../platform/access.md)
|
|
209
|
+
- [SDKs](../develop/sdks.md)
|
|
210
|
+
- [Extensions](../develop/extensions.md)
|
|
211
|
+
- [AI](../ai/index.md)
|