@collective-intelligence/external-apps-server-sdk 0.3.2 → 0.3.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +10 -2
- package/dist/admin.d.ts +3 -3
- package/dist/chunks/{index-vf83jqc8.js → index-hzx6hj46.js} +2 -2
- package/dist/effect.d.ts +270 -2
- package/dist/effect.js +1 -1
- package/dist/index.d.ts +270 -2
- package/dist/index.js +1 -1
- package/docs/guide/ai/index.md +36 -17
- package/docs/guide/ai/studio.md +245 -0
- package/docs/guide/data/connectors.md +81 -18
- package/docs/guide/data/datasets.md +6 -17
- package/docs/guide/data/documents.md +235 -0
- package/docs/guide/data/index.md +62 -0
- package/docs/guide/data/knowledge.md +21 -18
- package/docs/guide/develop/extensions.md +15 -14
- package/docs/guide/develop/external-apps.md +48 -14
- package/docs/guide/develop/infrastructure.md +135 -0
- package/docs/guide/develop/sdks.md +2 -2
- package/docs/guide/index.md +11 -7
- package/docs/guide/platform/access.md +82 -17
- package/docs/guide/platform/architecture.md +27 -18
- package/docs/guide/platform/entrypoints.md +125 -48
- package/docs/guide/scenarios/index.md +69 -46
- package/docs/guide/template/index.md +22 -9
- package/docs/guide/template/state-machines.md +1 -1
- package/package.json +4 -4
|
@@ -0,0 +1,245 @@
|
|
|
1
|
+
# Studio
|
|
2
|
+
|
|
3
|
+
How Organization members work in the Studio desktop app, how a package ships Studio apps, forms and widgets, and how Studio shares one solution with your external application.
|
|
4
|
+
|
|
5
|
+
Studio is the CI app where the people of an Organization do their work. They chat with agents, use Skills, open Studio apps, fill forms, decide approvals and find Documents. Admins set up the Organization in the Dashboard, and members use what it offers them in Studio. A package can ship a complete Studio app, and the `next-effect-app` template ships one: the Sales desk, which lists the same Leads that the template's web app reads and writes. This page uses the Sales desk as the running example.
|
|
6
|
+
|
|
7
|
+
## Install and connect Studio
|
|
8
|
+
|
|
9
|
+
Studio is a desktop app. Each Organization has a Studio install page in the Dashboard at `/get-studio/<organization-id>`, and nobody types a backend URL:
|
|
10
|
+
|
|
11
|
+
1. Open the install page and pick a platform: macOS (Apple Silicon or Intel), Windows or Linux. The page detects one for you.
|
|
12
|
+
2. Download and install Studio, then keep it open.
|
|
13
|
+
3. Click Connect Studio on the install page. The browser asks to open Collective Intelligence; choose Open.
|
|
14
|
+
4. If the button does not work, click Copy connect link and paste the link in Studio under "Connect button didn't work?".
|
|
15
|
+
|
|
16
|
+
The connect link is signed, and the backend verifies it before Studio stores the connection. When the page says "Studio setup isn't finished for this organization", an Organization admin must complete the Studio setup first.
|
|
17
|
+
|
|
18
|
+
After it connects, Studio shows the Organization's name, cover and logo on its loading and login screens, also offline. Before it opens the workspace, Studio checks the Organization's Stable and Minimum desktop versions, and it saves the open tabs before it installs an update.
|
|
19
|
+
|
|
20
|
+
## Who can open Studio
|
|
21
|
+
|
|
22
|
+
A member uses Studio only through the `studio` entry capability. The built-in Studio user profile holds it, together with the platform capabilities for conversations, agents, Knowledge, workflows, datasets and notifications. A guest acts only through the entry of an external application, so a guest cannot use Studio.
|
|
23
|
+
|
|
24
|
+
> **Warning:** The template's Sales rep and Sales manager profiles do not allow
|
|
25
|
+
> `core.studio`. Before a Sales member can open the Sales desk, check that the
|
|
26
|
+
> member also holds the Studio user profile or another profile that allows
|
|
27
|
+
> `core.studio`. [Access](../platform/access.md) explains profiles and grants.
|
|
28
|
+
|
|
29
|
+
What a person sees inside Studio still follows the grants. A Studio user reads no Knowledge record until a Knowledge Data Policy or a profile opens it, and a Studio app or page with an empty audience is offered to nobody.
|
|
30
|
+
|
|
31
|
+
## What a Studio app holds
|
|
32
|
+
|
|
33
|
+
A Studio app is a set of sections and pages in a left sidebar. Each page holds views. Admins build apps in the Dashboard builder, and a package can declare one. These built-in pages are available to every app:
|
|
34
|
+
|
|
35
|
+
| Page | What it does |
|
|
36
|
+
| --------------- | --------------------------------------------------------------- |
|
|
37
|
+
| Chat | The conversational workspace with agents and Skills |
|
|
38
|
+
| Agents | Lists the agents offered to the person and starts conversations |
|
|
39
|
+
| Skills | Lists the Skills the person can use in chat |
|
|
40
|
+
| Approvals | Lists pending approval decisions |
|
|
41
|
+
| Notifications | Shows Studio notifications, including approval requests |
|
|
42
|
+
| Activity Center | The person's work inbox of activities in progress and done |
|
|
43
|
+
| Documents | Browses, searches and downloads Documents |
|
|
44
|
+
|
|
45
|
+
The built-in views are tabs, a responsive grid, a report, a schema list and a schema entity detail for Knowledge records, a Skill form, a trigger form, an agent form, workflow runs, agent runs and a Dataset area. Their references have the form `builtin:view.<name>`, for example `builtin:view.schema-list`, and pages use `builtin:page.<name>`.
|
|
46
|
+
|
|
47
|
+
## Chat with agents
|
|
48
|
+
|
|
49
|
+
The Agents page lists every agent whose audience includes the person, and each one opens a conversation. When an agent is missing, check its audience first. The template's Lead qualifier has `audience: { applications: [application] }`, so the web app can call it and no Sales member sees it in Studio. To offer it to Sales members too, pass the Sales audience into the agent's declaration and keep the application:
|
|
50
|
+
|
|
51
|
+
```ts
|
|
52
|
+
declareAgent({
|
|
53
|
+
definition: {
|
|
54
|
+
audience: { ...audience, applications: [application] }
|
|
55
|
+
// ...
|
|
56
|
+
}
|
|
57
|
+
// ...
|
|
58
|
+
})
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
The template already passes `audience` (the Sales Group) into `leadQualificationComponents` for the tool and the Skill, so `leadQualifierAgent` needs that parameter as well. A tool binding with an approval mode pauses a chat turn until a person decides. [AI primitives](index.md) explains agents, Skills, tools and approvals in detail.
|
|
62
|
+
|
|
63
|
+
## Ship a Studio app from a package
|
|
64
|
+
|
|
65
|
+
`declareStudioApp` declares a Declared Studio App, which CI creates and publishes when it installs the package. `declareStudioAppTemplate` instead declares a starting point that the builder copies into a new app, and the copy keeps no link to the package. These rules apply to a Declared Studio App:
|
|
66
|
+
|
|
67
|
+
- Its `lifecycle` is always `{ deployMode: "upsert", readonly: true }`, and the SDK refuses other values.
|
|
68
|
+
- The app and each page take an `audience`. Without one, CI offers them to nobody. Sections show while one of their pages is offered, and views follow their page.
|
|
69
|
+
- Studio never shows a page that the person cannot access, and it shows no empty placeholder for it.
|
|
70
|
+
- A later release finds the app by its FQN, so a new slug moves the same app.
|
|
71
|
+
- A missing workflow, trigger, agent, schema, component or audience reference makes the install write nothing.
|
|
72
|
+
- The builder shows the app as package-owned and read-only. Change it through a new release.
|
|
73
|
+
- An uninstall disables the app, and a new install enables it again.
|
|
74
|
+
|
|
75
|
+
The Sales desk has one section and one Leads page, offered to the Sales Group. The page places a Knowledge schema list and the Lead score card widget, both bound to the Lead schema:
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
export const salesDeskApp = declareStudioApp({
|
|
79
|
+
definition: {
|
|
80
|
+
audience: salesMembers,
|
|
81
|
+
definition: {
|
|
82
|
+
name: "Sales desk",
|
|
83
|
+
slug: "acme-sales-desk",
|
|
84
|
+
sections: [
|
|
85
|
+
{
|
|
86
|
+
fqn: `${salesDeskAppFqn}.sections.sales`,
|
|
87
|
+
label: "Sales",
|
|
88
|
+
slug: "sales",
|
|
89
|
+
orderIndex: 0,
|
|
90
|
+
pages: [
|
|
91
|
+
{
|
|
92
|
+
audience: salesMembers,
|
|
93
|
+
fqn: `${salesDeskAppFqn}.pages.leads`,
|
|
94
|
+
label: "Leads",
|
|
95
|
+
slug: "leads",
|
|
96
|
+
kind: "generic",
|
|
97
|
+
routeSegment: "leads",
|
|
98
|
+
orderIndex: 0,
|
|
99
|
+
// No `mutations`: the list is read-only; the web app writes Leads.
|
|
100
|
+
views: [
|
|
101
|
+
{
|
|
102
|
+
componentRef: "builtin:view.schema-list",
|
|
103
|
+
dataBinding: { kind: "schema", schema: leadSchemaFqn },
|
|
104
|
+
fqn: `${salesDeskAppFqn}.views.leads-list`,
|
|
105
|
+
orderIndex: 0,
|
|
106
|
+
slot: "main"
|
|
107
|
+
},
|
|
108
|
+
{
|
|
109
|
+
componentRef: leadScoreCardComponentFqn,
|
|
110
|
+
dataBinding: { kind: "schema", schema: leadSchemaFqn },
|
|
111
|
+
fqn: `${salesDeskAppFqn}.views.lead-score`,
|
|
112
|
+
orderIndex: 1,
|
|
113
|
+
slot: "main"
|
|
114
|
+
}
|
|
115
|
+
]
|
|
116
|
+
}
|
|
117
|
+
]
|
|
118
|
+
}
|
|
119
|
+
]
|
|
120
|
+
},
|
|
121
|
+
lifecycle: { deployMode: "upsert", readonly: true }
|
|
122
|
+
},
|
|
123
|
+
manifest: { fqn: salesDeskAppFqn, version: VERSION },
|
|
124
|
+
presentation: {
|
|
125
|
+
displayName: "Sales desk",
|
|
126
|
+
description: "Leads of the Acme CRM with the Lead score card."
|
|
127
|
+
}
|
|
128
|
+
})
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
To let Sales members edit Leads in Studio, set `props.mutations.create`, `update` or `delete` on the schema list to `true`. A schema entity detail view becomes editable when `update` is on. Each write still passes the Lead write policy, which in the template requires CRM `manage`.
|
|
132
|
+
|
|
133
|
+
## Forms
|
|
134
|
+
|
|
135
|
+
Three built-in views collect input and start a run. Each one runs as the Studio user who submits it:
|
|
136
|
+
|
|
137
|
+
| Form | Declared with | What it runs |
|
|
138
|
+
| ----------------------------------------- | --------------------------------------------------------------------------------------- | ----------------------------------------------------- |
|
|
139
|
+
| Trigger form, `builtin:view.trigger-form` | `withStudioTriggerForm(trigger, { organizationGroup, fields })` on an On-demand trigger | The workflow, once every field holds a valid value |
|
|
140
|
+
| Agent form, `builtin:view.agent-form` | `withStudioAgentForm(agent, { fields })` | The declared agent, with the field values as its task |
|
|
141
|
+
| Skill form, `builtin:view.skill-form` | The Dashboard builder | One Skill with structured input |
|
|
142
|
+
|
|
143
|
+
### Trigger form fields
|
|
144
|
+
|
|
145
|
+
A trigger form shows and runs only for members of its `organizationGroup`, including subgroups. Its fields can be `text`, `textarea`, `number`, `boolean`, `enum`, `json` with a JSON Schema, `rich_text`, `file` (PDF, JPEG or PNG up to 10 MiB by default), `entities` (Knowledge records of one schema that the person can read) and `user_select` (Organization members). Workflow nodes read a value as `{{ trigger.form.<key> }}`. A link can open the form prefilled through the query parameter `prefill.<triggerRef>`, so put no secrets in such a link.
|
|
146
|
+
|
|
147
|
+
### Agent forms
|
|
148
|
+
|
|
149
|
+
An agent form names its agent by FQN in `props.agentRef` and runs it inline within the agent's audience. The optional `props.resultWidget` names a widget that renders the agent's output. Agent form fields of type `file` or `entities` have no input yet: the view shows "This field cannot be filled in an agent form yet." and keeps Run disabled.
|
|
150
|
+
|
|
151
|
+
### Form runs and approvals
|
|
152
|
+
|
|
153
|
+
A run from an agent form or a Skill form keeps its history, result, follow-ups, retries, cancellation and approvals together. A Skill form set to `ask_always` asks a person to approve every installed tool call, even on bindings set to `never`, and `automatic` skips approval. The other authorization checks still apply.
|
|
154
|
+
|
|
155
|
+
## Run lists, Documents and datasets
|
|
156
|
+
|
|
157
|
+
The workflow runs view (`props.workflows`) and the agent runs view (`props.agents`) list the runs of the workflows or agents they name, for the people who can see the view. A retry starts a new run as the person who retries it, with the original input. A workflow retry needs an On-demand trigger on that version. The lists show status, times and error codes, and never inputs, outputs or messages.
|
|
158
|
+
|
|
159
|
+
The Documents page is a finder. A person browses the Document Collections and folders they can read, searches filenames and document content in one field, downloads the stored file of a Document Version, or opens a cloud file in its Microsoft or Google app. Studio has no built-in document viewer. [Documents](../data/documents.md) explains collections and versions.
|
|
160
|
+
|
|
161
|
+
The Dataset area lists the datasets a person can see, with each dataset's definition, policy, health, freshness and lineage. Its query workspace offers only the fields and operations that the dataset policy allows, and the create panel appears only for a person who can administer datasets. [Datasets](../data/datasets.md) explains the policies.
|
|
162
|
+
|
|
163
|
+
## Approvals
|
|
164
|
+
|
|
165
|
+
The Approvals page lists the decisions that wait for the person. A notification for an approval shows the facts of the pending action, such as what it sends, to whom, and which Skill or workflow asks. CI freezes those facts when the request opens and shows secrets as omitted. The reviewer decides and never becomes the identity that runs the call.
|
|
166
|
+
|
|
167
|
+
## Widgets from packages
|
|
168
|
+
|
|
169
|
+
A widget is a React component that a package ships and a Studio page places. The template's Lead score card needs four pieces:
|
|
170
|
+
|
|
171
|
+
1. A backend declaration in the package's `components`: `declareStudioComponent` with `category: "widget"`, a `propsSchema` and a `sourceRef` that names the package's Studio contribution.
|
|
172
|
+
2. A frontend declaration: `declareFrontendComponent` with a `load` callback and a `studio_component` target, returned by the package's `loadFrontendCatalog`.
|
|
173
|
+
3. A hosted contribution for the Studio host in `declarePackage({ hostedContributions })`.
|
|
174
|
+
4. A view of a Studio app that places the widget by its FQN in `componentRef`, as the Sales desk does.
|
|
175
|
+
|
|
176
|
+
```ts
|
|
177
|
+
declareFrontendCatalogContribution({
|
|
178
|
+
compatibility: { react: "^19.0.0" },
|
|
179
|
+
fqn: leadScoreCardContributionFqn,
|
|
180
|
+
host: "studio",
|
|
181
|
+
load: async () => await import("./studio")
|
|
182
|
+
})
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
Every placed widget gets a `context` prop with the app, page, view and published revision ids. The prop is `null` in the builder preview and before the app is published, so a widget renders an inert state then. The Lead score card shows a Preview badge in that case.
|
|
186
|
+
|
|
187
|
+
A widget holds no CI credentials. It calls CI through the Studio API client from `@collective-intelligence/extensions-sdk-frontend-react/studio-api`, which Studio binds to its own signed-in connection, so each request runs with the signed-in person's permissions. The client covers the view-data search, read and update calls and the reads of the two run-list views. The Lead score card reads the Leads of its own view:
|
|
188
|
+
|
|
189
|
+
```tsx
|
|
190
|
+
const leads = useQuery({
|
|
191
|
+
...studioViewDataSearchOptions({
|
|
192
|
+
...studioViewScope(context),
|
|
193
|
+
body: { limit: SEARCH_LIMIT }
|
|
194
|
+
}),
|
|
195
|
+
select: leadsOf
|
|
196
|
+
})
|
|
197
|
+
```
|
|
198
|
+
|
|
199
|
+
A widget writes through a `fromEffect` XState actor from `@collective-intelligence/extensions-sdk-frontend-react/effect` and mounts `EffectRuntimeProvider` at its root. Package widgets render in the Studio desktop app, which downloads and verifies the package release.
|
|
200
|
+
|
|
201
|
+
> **Note:** The Lead score card shows the stored `score` field of each Lead, or "Not
|
|
202
|
+
> scored yet" when the field is empty. People set that field in the web app's
|
|
203
|
+
> Lead form. The card runs no agent, and nothing in the template writes the Lead
|
|
204
|
+
> qualifier's score back to the Lead.
|
|
205
|
+
|
|
206
|
+
> **Warning:** An install of a package with a Studio contribution waits until a Studio
|
|
207
|
+
> desktop app of that Organization reports the release ready. A pipeline has no
|
|
208
|
+
> Studio, so pass `--skip-studio-readiness` to `ci deploy`, as the template
|
|
209
|
+
> does. A Studio that connects later loads the release then.
|
|
210
|
+
|
|
211
|
+
## Studio and your external application
|
|
212
|
+
|
|
213
|
+
The template's package installs everything that Studio and the web app share. Studio people and web app users are the same CI users, and both sides read the same Lead records:
|
|
214
|
+
|
|
215
|
+
```mermaid
|
|
216
|
+
flowchart LR
|
|
217
|
+
pkg["Extension package"] -->|installs| org["Organization"]
|
|
218
|
+
org --> desk["Sales desk in Studio"]
|
|
219
|
+
org --> app["External application and BFF"]
|
|
220
|
+
desk -->|reads| leads["Lead records"]
|
|
221
|
+
app -->|reads and writes| leads
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
One `ci deploy` installs the Lead schema, the Sales Group and its grants, the external application, the convert-lead workflow, the Lead qualifier with its Skill and tool, and the Sales desk with its widget. The Sales Group's grants decide what a Sales member does in the web app, and the same Group is the audience of the Sales desk. A Lead that a Sales manager edits in the web app is the record that the Sales desk lists. The web app cuts each user's access to the application's Max Privileges, and Studio sessions have no such cut, so the same person can see more in Studio than through the app.
|
|
225
|
+
|
|
226
|
+
Workflows and agents can start from either side:
|
|
227
|
+
|
|
228
|
+
| Primitive | From the web app | From Studio |
|
|
229
|
+
| --------- | ----------------------------------------------------------------------------- | ------------------------------------------- |
|
|
230
|
+
| Workflow | An External application trigger, started by the server's Fixed workflow route | An On-demand trigger with a trigger form |
|
|
231
|
+
| Agent | `agents.invoke` or `conversations.continue` from the app server | The Agents and Chat pages, or an agent form |
|
|
232
|
+
| Audience | `applications: [application]` | `groups`, `profiles` or `organization` |
|
|
233
|
+
|
|
234
|
+
A trigger form needs a trigger that people start, so an External application trigger cannot back it. One workflow can declare both triggers when the same work must start from both sides. The template's convert-lead workflow has only the External application trigger.
|
|
235
|
+
|
|
236
|
+
A run acts as the identity that started it, and every tool call in the run uses that identity's current grants. A Sales manager who starts a workflow from Studio runs it as themselves when the workflow's `runIdentity` is `triggering_user`, and as the workflow's run-as service principal when it is `service_principal`. A run that no person starts acts as that service principal. [AI primitives](index.md) lists every case.
|
|
237
|
+
|
|
238
|
+
## Related pages
|
|
239
|
+
|
|
240
|
+
- [AI primitives](index.md)
|
|
241
|
+
- [Access](../platform/access.md)
|
|
242
|
+
- [Extensions](../develop/extensions.md)
|
|
243
|
+
- [External apps](../develop/external-apps.md)
|
|
244
|
+
- [Documents](../data/documents.md)
|
|
245
|
+
- [Datasets](../data/datasets.md)
|
|
@@ -2,11 +2,13 @@
|
|
|
2
2
|
|
|
3
3
|
Connect outside systems, map their fields to Knowledge schemas, and run and watch imports.
|
|
4
4
|
|
|
5
|
-
A connector
|
|
5
|
+
A connector keeps a Data resource in sync with an outside system. CI has two kinds. A Knowledge connector imports records from a source such as Notion, HubSpot or a SQL database into Knowledge records. A Documents connector imports files from Microsoft 365 or Google Drive into one Connected Collection of Documents. The [Data resources](index.md) overview compares Knowledge, Datasets and Documents, and the [Documents](documents.md) page covers Document Collections.
|
|
6
|
+
|
|
7
|
+
A Knowledge import passes through several stages: CI reads the source page by page, stages the records, commits the accepted ones into Knowledge, and then indexes them for search. An app or agent that reads imported data has to wait for the stage it needs. Most of this page covers Knowledge connectors. The last section compares them with Documents connectors.
|
|
6
8
|
|
|
7
9
|
## Connections, templates and instances
|
|
8
10
|
|
|
9
|
-
Three objects work together, and a
|
|
11
|
+
Three objects work together, and a package usually declares two of them.
|
|
10
12
|
|
|
11
13
|
A connection holds credentials. A connection template, such as `builtin.integrations.notion.connections.internal-integration-token`, defines the credential form and a test, and an Organization connection holds the actual credentials. Connectors, tools and Dataset providers each point at a connection through a slot, so one connection can serve all of them.
|
|
12
14
|
|
|
@@ -55,7 +57,7 @@ At deploy time, the Inputs File links the variable to an Organization secret:
|
|
|
55
57
|
|
|
56
58
|
## Declare a connector instance
|
|
57
59
|
|
|
58
|
-
`declareKnowledgeConnectorInstance` names the template, fills its connection slots, binds remote collections to Knowledge schemas and sets the import pipelines. Each connection slot takes exactly one of `{ setupKey }
|
|
60
|
+
`declareKnowledgeConnectorInstance` names the template, fills its connection slots, binds remote collections to Knowledge schemas and sets the import pipelines. Each connection slot takes exactly one of `{ setupKey }` or `{ connectionInstanceFqn }`. The Inputs File fills a `setupKey` slot at install with an existing connection or a new one. A `connectionInstanceFqn` points at a connection that the package declares. The package must also list the integration, such as `{ fqn: "builtin.integrations.notion" }`, in its `dependencies`.
|
|
59
61
|
|
|
60
62
|
```ts
|
|
61
63
|
const customers = declareKnowledgeConnectorInstance({
|
|
@@ -136,7 +138,7 @@ The `remote` value is a dot path with at most one `[]` segment, which takes ever
|
|
|
136
138
|
}
|
|
137
139
|
```
|
|
138
140
|
|
|
139
|
-
|
|
141
|
+
The records that a Knowledge connector writes and keeps in sync are connected records. Each import brings the source changes into them, and the [Deletions](#deletions) section explains when a connected record leaves CI because its source item is gone.
|
|
140
142
|
|
|
141
143
|
## Imports
|
|
142
144
|
|
|
@@ -148,7 +150,7 @@ flowchart LR
|
|
|
148
150
|
C --> I[Embedding and indexing]
|
|
149
151
|
```
|
|
150
152
|
|
|
151
|
-
A FULL import lists everything in the source. A DELTA import reads only what changed since the Delta Since time, which is the start of the binding's last successful phase minus the connector's overlap. A failed phase does not move Delta Since. The overlap (`deltaOverlapSeconds`, 300 seconds by default and 120 for Notion) reads the edge of the window again to catch late changes. The Notion connector recommends a delta schedule every five minutes. A SQL connector
|
|
153
|
+
A FULL import lists everything in the source. A DELTA import reads only what changed since the Delta Since time, which is the start of the binding's last successful phase minus the connector's overlap. A failed phase does not move Delta Since. The overlap (`deltaOverlapSeconds`, 300 seconds by default and 120 for Notion) reads the edge of the window again to catch late changes. The Notion connector recommends a delta schedule every five minutes. A SQL connector reads only changed rows when you choose a watermark column for the binding: a date, timestamp or integer column that records when a row changed. Without one, a delta import still runs, but it reads every row, adds a warning to the run, and removes nothing.
|
|
152
154
|
|
|
153
155
|
The import writes staged records first and then commits the accepted records into Knowledge. Treat an import as finished only when CI has committed the staged records and the indexing work that follows. Before each remote page, the import saves its cursor and counters. When a worker stops in the middle of a call, the next attempt repeats at most that one page, and repeating a page is safe because page commits are idempotent. Full imports resume from their cursor as well, so a restart does not scan the whole source again.
|
|
154
156
|
|
|
@@ -156,12 +158,70 @@ When a source value does not fit the schema field type, for example a Notion dat
|
|
|
156
158
|
|
|
157
159
|
## Deletions
|
|
158
160
|
|
|
159
|
-
|
|
161
|
+
When an item is deleted at the source, the connector decides when its connected record leaves CI. Each Knowledge connector template declares one of two deletion capabilities:
|
|
162
|
+
|
|
163
|
+
- `full`: a full run removes the records whose items are missing from a complete listing of the source, and a delta run removes nothing. A template that declares no capability has this one.
|
|
164
|
+
- `full_and_delta`: a full run removes records in the same way, and a delta run also removes the records that the source reports as deleted in its change feed.
|
|
165
|
+
|
|
166
|
+
A listing page that cannot prove the listing complete removes nothing. When a connector can read a single item, CI also asks the source to confirm that the item is gone before it removes the record.
|
|
167
|
+
|
|
168
|
+
| Knowledge connector | Sync modes | What a delta run reads | Removes on full runs | Removes on delta runs |
|
|
169
|
+
| -------------------------------------- | -------------------------- | ------------------------------------------------------------------ | -------------------- | --------------------- |
|
|
170
|
+
| Notion | Full, delta | Pages edited since the last successful run | Yes | No |
|
|
171
|
+
| HubSpot | Full, delta | Records modified since the last successful run | Yes | No |
|
|
172
|
+
| Monday | Full; delta for items only | Items updated since the last successful run | Yes | No |
|
|
173
|
+
| SQL (PostgreSQL, MySQL) | Full, delta | Rows past the watermark column, or every row without one | Yes | No |
|
|
174
|
+
| Structured File | Full, delta | Nothing when the file is unchanged, the whole file when it changed | Yes | No |
|
|
175
|
+
| External application Knowledge gateway | Full, delta | The change feed that the external application exposes for the data | Yes | Yes |
|
|
176
|
+
|
|
177
|
+
The external application Knowledge gateway is the connector for a customer's own system. Its full run lists every item that the external application exposes, and its delta run reads the application's change feed, where a deleted item removes the record during that run. The application implements both for each Knowledge resource that it shares through its Gateway. The [External apps](../develop/external-apps.md) page explains Gateways.
|
|
178
|
+
|
|
179
|
+
Each entity binding can also limit deletions. `remoteDeletionPropagation` takes `never`, `full` or `full_and_delta`. When you leave it out, the binding follows the connector capability, and CI refuses a value above the capability. `maxRemoteDeletionsPerRun` is the deletion brake for full runs: when a full run finds more records to remove than the limit, it removes none of them and reports that the brake tripped. The brake does not apply to delta runs. A delta run on the external application Knowledge gateway removes every deletion that its change feed reports, however many there are.
|
|
180
|
+
|
|
181
|
+
```ts
|
|
182
|
+
{
|
|
183
|
+
key: "customers",
|
|
184
|
+
schemaFqn: "com.acme.sales.customer",
|
|
185
|
+
bindingMode: "managed",
|
|
186
|
+
remoteCollectionId: "customers",
|
|
187
|
+
remoteDeletionPropagation: "full",
|
|
188
|
+
maxRemoteDeletionsPerRun: 50,
|
|
189
|
+
fieldMap: [/* ... */],
|
|
190
|
+
}
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
In the Dashboard, the binding shows the same setting under **Fields map**, **Settings**, **Remote deletions**, and it offers only the values that the connector allows. The brake has no Dashboard field. The run detail shows, for each binding, how many records the run removed or why it removed none.
|
|
194
|
+
|
|
195
|
+
> **Warning:** Set `maxRemoteDeletionsPerRun` before the first full run on customer data. If
|
|
196
|
+
> a listing then misses many items by mistake, the run removes nothing and the
|
|
197
|
+
> run detail shows that the brake tripped.
|
|
198
|
+
|
|
199
|
+
For a Notion demo, a page deleted in Notion stays in CI through the delta runs and leaves on the next full run. The next section explains how to schedule that full run.
|
|
200
|
+
|
|
201
|
+
## Choose a sync schedule
|
|
202
|
+
|
|
203
|
+
Start from how old the data in the demo may be. For a Knowledge connector, a change at the source reaches CI within one delta interval plus the run time. A deletion at the source reaches CI within one full-run interval plus the run time, unless the connector removes records on delta runs. Choose the delta schedule from the first figure and the full schedule from the second.
|
|
204
|
+
|
|
205
|
+
Some connectors recommend schedules. The Dashboard Knowledge setup suggests them when it creates bindings. A package does not receive them, so it declares its own `schedules` on each pipeline.
|
|
206
|
+
|
|
207
|
+
| Connector | Recommended schedules |
|
|
208
|
+
| --------------------------------------- | ----------------------------------------------------------------------- |
|
|
209
|
+
| Notion | Delta every 5 minutes (`*/5 * * * *`) |
|
|
210
|
+
| HubSpot | Delta every 15 minutes (`*/15 * * * *`), full every night (`0 3 * * *`) |
|
|
211
|
+
| Monday | Delta every 15 minutes (`*/15 * * * *`), full every night (`0 3 * * *`) |
|
|
212
|
+
| SQL, Structured File, Knowledge gateway | None |
|
|
213
|
+
|
|
214
|
+
HubSpot and Monday pair frequent delta runs with a nightly full run because their delta runs cannot see deletions. Notion recommends only the delta schedule, and a Notion delta run removes nothing either. Add a periodic full schedule to every Notion pipeline so that pages deleted in Notion leave CI, and set its frequency from how long a deleted page may stay visible in the demo.
|
|
215
|
+
|
|
216
|
+
For SQL, choose a watermark column before you schedule delta runs. Without one, a delta run reads as much as a full run and removes nothing, so schedule full runs instead. A Structured File delta run costs little when the file has not changed, because CI compares the file version or its content first, but deletions still need a full run. The Knowledge gateway removes records on delta runs, so frequent delta runs keep both changes and deletions current. Keep an occasional full run for the gateway as well: when the application rejects the stored feed position, CI reads the feed again from the start, and that first delta run removes nothing.
|
|
217
|
+
|
|
218
|
+
Every run calls the source, and a full run reads every record again. HubSpot and Monday stop a run when the account reaches its daily API limit, so frequent full runs can use up the quota that the delta runs need. CI retries the shorter rate limits of all three sources.
|
|
219
|
+
|
|
220
|
+
By default, CI checks for due schedules once a minute, so a schedule shorter than one minute gains nothing. A connector instance has one scheduled run at a time. When a schedule comes due during another run of the same instance, it starts after that run ends, and missed times do not pile up. A long full run therefore delays the delta runs of every pipeline in the instance, so schedule full runs at a quiet hour. Each delta run also reads again the overlap window before its start time, 120 seconds for Notion and Monday and 300 seconds for HubSpot and by default.
|
|
160
221
|
|
|
161
|
-
|
|
162
|
-
> records that were deleted at the source.
|
|
222
|
+
Indexing and embeddings run after the commit, so search finds new records a little after the run ends. When a run removes a record, the record leaves CI at the commit, and the search index follows.
|
|
163
223
|
|
|
164
|
-
|
|
224
|
+
A Documents connector has no schedule to choose. CI checks every minute and starts the sync of a Connected Collection five minutes after its last attempt finished, failed attempts included. You can turn scheduled sync off for a collection, or start a sync by hand, which skips the wait. Each sync applies the deletions that the provider reports.
|
|
165
225
|
|
|
166
226
|
## Health
|
|
167
227
|
|
|
@@ -184,21 +244,24 @@ Lusha shows the difference. It has an API key connection and prospecting tools a
|
|
|
184
244
|
|
|
185
245
|
## Knowledge sync and Documents sync
|
|
186
246
|
|
|
187
|
-
Knowledge connectors and Documents connectors have different owners and different rules. A Documents connector never creates Knowledge records, and it does not use the Knowledge import stages or the Knowledge deletion
|
|
247
|
+
Knowledge connectors and Documents connectors have different owners and different rules. A Documents connector never creates Knowledge records, and it does not use the Knowledge import stages or the Knowledge deletion capabilities.
|
|
188
248
|
|
|
189
|
-
| | Knowledge connector
|
|
190
|
-
| ------------------- |
|
|
191
|
-
| Writes |
|
|
192
|
-
| Configured by | A connector instance with entity bindings and pipelines
|
|
193
|
-
| Schedule | Cron schedules per pipeline, FULL or DELTA
|
|
194
|
-
|
|
|
195
|
-
|
|
|
249
|
+
| | Knowledge connector | Documents connector |
|
|
250
|
+
| ------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
|
251
|
+
| Writes | Connected records in bound Knowledge schemas | Files and versions in one Connected Collection |
|
|
252
|
+
| Configured by | A connector instance with entity bindings and pipelines | One Documents connector instance per Connected Collection |
|
|
253
|
+
| Schedule | Cron schedules per pipeline, FULL or DELTA | CI checks every minute and starts a sync five minutes after the last attempt finished |
|
|
254
|
+
| Deletions | Depend on the connector capability, as in the table above | Every sync applies the deletions that the provider reports |
|
|
255
|
+
| Package declaration | `declareKnowledgeConnectorInstance` | `declareDocumentCollection` with a connection slot and remote folders |
|
|
256
|
+
| Folder exclusions | Not applied | Excluded Folder Names skip whole folders |
|
|
196
257
|
|
|
197
|
-
A Documents sync moves its provider checkpoint forward only after it applies the changes. Two Connected
|
|
258
|
+
A Documents sync moves its provider checkpoint forward only after it applies the changes. Two Connected Collections on the same provider account cannot overlap their root folders. A finished Documents sync still has to index the files before search finds them, as the [Documents](documents.md) and [Knowledge](knowledge.md) pages explain.
|
|
198
259
|
|
|
199
260
|
## Related pages
|
|
200
261
|
|
|
262
|
+
- [Data resources](index.md)
|
|
201
263
|
- [Knowledge](knowledge.md)
|
|
264
|
+
- [Documents](documents.md)
|
|
202
265
|
- [Datasets](datasets.md)
|
|
203
266
|
- [Extensions](../develop/extensions.md)
|
|
204
267
|
- [Access](../platform/access.md)
|
|
@@ -4,17 +4,9 @@ Query typed tables of measurements, events and warehouse data, and control who c
|
|
|
4
4
|
|
|
5
5
|
A Dataset is a typed collection of rows that belongs to an Organization. Each Dataset has a stable identity, a versioned definition, a policy that says who may query it, and provenance on every result. Datasets hold data that has many rows and few edits, such as meter readings, monthly sales figures or a table in a customer's data warehouse. Agents, workflows and Studio query that data in bounded, typed requests.
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## Datasets and Knowledge
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
| | Knowledge schema | Dataset |
|
|
12
|
-
| -------------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
|
|
13
|
-
| One row or record is | A real thing: a customer, a Lead, a goal | An observation: a reading, a transaction, an event |
|
|
14
|
-
| People edit rows | Yes, in forms, Studio and the API | No; rows come from file imports or from the source system |
|
|
15
|
-
| Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema |
|
|
16
|
-
| Queries | Field filters, text and semantic search, relation paths | `records`, `aggregate` and `timeSeries` |
|
|
17
|
-
| Access | Knowledge data policies across schemas and domains | One policy per Dataset with an explicit list of principals |
|
|
9
|
+
A Dataset holds observations about the things that Knowledge records describe, with one row per measurement, event or transaction. The [Data resources](index.md) page compares Datasets with Knowledge and Documents and lists which kind fits which demo situation.
|
|
18
10
|
|
|
19
11
|
A strategy Metric shows the split. The Metric itself is a Knowledge record with an owner, a target, a unit, a status and links to goals, and its history of measured values is a Dataset. A Dataset field such as `customerId` can bind to the Customer schema in Knowledge. That binding links the two without copying rows into Knowledge, and deleting one side does not delete the other.
|
|
20
12
|
|
|
@@ -95,7 +87,7 @@ Every result carries its provenance: the Dataset ID, the definition revision, th
|
|
|
95
87
|
|
|
96
88
|
Each Dataset has its own policy, and the policy lists its principals by ID. It holds `principals` (each a `user` or `service` with an `id`), `allowedOperations` (`describe`, `records`, `aggregate`, `time_series`), `allowedFields`, `sensitiveFields`, an optional `rowFilter`, and the limits `maxRows`, `maxBytes` and `maxTimeoutMs`. A user also needs the Organization permission to query Datasets, and a Dataset grant does not give Organization membership or capabilities.
|
|
97
89
|
|
|
98
|
-
Knowledge access works differently: Knowledge
|
|
90
|
+
Knowledge access works differently: each Knowledge Data Policy targets one schema, and no policy targets a whole domain. The [Access](../platform/access.md) page explains both models.
|
|
99
91
|
|
|
100
92
|
> **Warning:** A package-declared Dataset without a `dataPolicy` grants access to nobody,
|
|
101
93
|
> including the person who deployed it. Always declare the principals that the
|
|
@@ -103,7 +95,7 @@ Knowledge access works differently: Knowledge data policies apply across schemas
|
|
|
103
95
|
|
|
104
96
|
## Declare a Dataset in a package
|
|
105
97
|
|
|
106
|
-
`declareDatasetConnectorInstance` declares a Dataset in
|
|
98
|
+
`declareDatasetConnectorInstance` declares a Dataset in a package. An External declaration names the provider, a connection slot, the source location and the policy:
|
|
107
99
|
|
|
108
100
|
```ts
|
|
109
101
|
const orders = declareDatasetConnectorInstance({
|
|
@@ -139,16 +131,13 @@ Agents use one tool, Query Dataset (`builtin.core.datasets.tools.query`). It lis
|
|
|
139
131
|
|
|
140
132
|
Workflows have one node for each operation: describe, query records, query aggregate and query time series. Studio shows Datasets in their own area, apart from the Knowledge views.
|
|
141
133
|
|
|
142
|
-
##
|
|
143
|
-
|
|
144
|
-
- The customer has a warehouse in Fabric or Databricks: use an External Dataset and query it live.
|
|
145
|
-
- The customer has a CSV or Excel export and wants to see trends: import it into a Managed Dataset.
|
|
146
|
-
- Users must edit records and link them to each other: use Knowledge.
|
|
134
|
+
## Limits
|
|
147
135
|
|
|
148
136
|
Datasets are sized for about 100,000 to 1,000,000 data points per Organization and do not fit a high-volume telemetry stream. Joins across Datasets, writes back to External Datasets, streaming ingestion and copying External data into a Managed Dataset are not available. A route for external applications to query Datasets directly is planned; today agents, workflows and Studio run the queries.
|
|
149
137
|
|
|
150
138
|
## Related pages
|
|
151
139
|
|
|
140
|
+
- [Data resources](index.md)
|
|
152
141
|
- [Knowledge](knowledge.md)
|
|
153
142
|
- [Connectors](connectors.md)
|
|
154
143
|
- [Access](../platform/access.md)
|