@collective-intelligence/extensions-sdk 0.3.2 → 0.3.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -2,11 +2,13 @@
2
2
 
3
3
  Connect outside systems, map their fields to Knowledge schemas, and run and watch imports.
4
4
 
5
- A connector imports records from an outside system, such as Notion, HubSpot or a SQL database, into Knowledge records. A Documents connector does the same for files, which it keeps in a Documents collection. An import passes through several stages: CI reads the source page by page, stages the records, commits the accepted ones into Knowledge, and then indexes them for search. A demo that shows imported data has to wait for the stage it needs.
5
+ A connector keeps a Data resource in sync with an outside system. CI has two kinds. A Knowledge connector imports records from a source such as Notion, HubSpot or a SQL database into Knowledge records. A Documents connector imports files from Microsoft 365 or Google Drive into one Connected Collection of Documents. The [Data resources](index.md) overview compares Knowledge, Datasets and Documents, and the [Documents](documents.md) page covers Document Collections.
6
+
7
+ A Knowledge import passes through several stages: CI reads the source page by page, stages the records, commits the accepted ones into Knowledge, and then indexes them for search. An app or agent that reads imported data has to wait for the stage it needs. Most of this page covers Knowledge connectors. The last section compares them with Documents connectors.
6
8
 
7
9
  ## Connections, templates and instances
8
10
 
9
- Three objects work together, and a demo usually declares two of them.
11
+ Three objects work together, and a package usually declares two of them.
10
12
 
11
13
  A connection holds credentials. A connection template, such as `builtin.integrations.notion.connections.internal-integration-token`, defines the credential form and a test, and an Organization connection holds the actual credentials. Connectors, tools and Dataset providers each point at a connection through a slot, so one connection can serve all of them.
12
14
 
@@ -55,7 +57,7 @@ At deploy time, the Inputs File links the variable to an Organization secret:
55
57
 
56
58
  ## Declare a connector instance
57
59
 
58
- `declareKnowledgeConnectorInstance` names the template, fills its connection slots, binds remote collections to Knowledge schemas and sets the import pipelines. Each connection slot takes exactly one of `{ setupKey }`, which an administrator fills at install, or `{ connectionInstanceFqn }`, which points at a connection that the package declares. The package must also list the integration, such as `{ fqn: "builtin.integrations.notion" }`, in its `dependencies`.
60
+ `declareKnowledgeConnectorInstance` names the template, fills its connection slots, binds remote collections to Knowledge schemas and sets the import pipelines. Each connection slot takes exactly one of `{ setupKey }` or `{ connectionInstanceFqn }`. The Inputs File fills a `setupKey` slot at install with an existing connection or a new one. A `connectionInstanceFqn` points at a connection that the package declares. The package must also list the integration, such as `{ fqn: "builtin.integrations.notion" }`, in its `dependencies`.
59
61
 
60
62
  ```ts
61
63
  const customers = declareKnowledgeConnectorInstance({
@@ -136,7 +138,7 @@ The `remote` value is a dot path with at most one `[]` segment, which takes ever
136
138
  }
137
139
  ```
138
140
 
139
- Each entity binding has a binding mode, `managed` or `external`, and the template declares which modes it supports. Notion supports both. An `external` binding refuses the managed-only `updateMode` and `outboundStrategy` options.
141
+ The records that a Knowledge connector writes and keeps in sync are connected records. Each import brings the source changes into them, and the [Deletions](#deletions) section explains when a connected record leaves CI because its source item is gone.
140
142
 
141
143
  ## Imports
142
144
 
@@ -148,7 +150,7 @@ flowchart LR
148
150
  C --> I[Embedding and indexing]
149
151
  ```
150
152
 
151
- A FULL import lists everything in the source. A DELTA import reads only what changed since the Delta Since time, which is the start of the binding's last successful phase minus the connector's overlap. A failed phase does not move Delta Since. The overlap (`deltaOverlapSeconds`, 300 seconds by default and 120 for Notion) reads the edge of the window again to catch late changes. The Notion connector recommends a delta schedule every five minutes. A SQL connector can run a delta import only when you choose a watermark column for the binding, the column that records when a row changed; without one, it runs full imports only.
153
+ A FULL import lists everything in the source. A DELTA import reads only what changed since the Delta Since time, which is the start of the binding's last successful phase minus the connector's overlap. A failed phase does not move Delta Since. The overlap (`deltaOverlapSeconds`, 300 seconds by default and 120 for Notion) reads the edge of the window again to catch late changes. The Notion connector recommends a delta schedule every five minutes. A SQL connector reads only changed rows when you choose a watermark column for the binding: a date, timestamp or integer column that records when a row changed. Without one, a delta import still runs, but it reads every row, adds a warning to the run, and removes nothing.
152
154
 
153
155
  The import writes staged records first and then commits the accepted records into Knowledge. Treat an import as finished only when CI has committed the staged records and the indexing work that follows. Before each remote page, the import saves its cursor and counters. When a worker stops in the middle of a call, the next attempt repeats at most that one page, and repeating a page is safe because page commits are idempotent. Full imports resume from their cursor as well, so a restart does not scan the whole source again.
154
156
 
@@ -156,12 +158,70 @@ When a source value does not fit the schema field type, for example a Notion dat
156
158
 
157
159
  ## Deletions
158
160
 
159
- Every Knowledge connector supports deletion on full runs. A full run deletes the records that are missing from a complete listing of the source, and a page that cannot prove the listing complete deletes nothing.
161
+ When an item is deleted at the source, the connector decides when its connected record leaves CI. Each Knowledge connector template declares one of two deletion capabilities:
162
+
163
+ - `full`: a full run removes the records whose items are missing from a complete listing of the source, and a delta run removes nothing. A template that declares no capability has this one.
164
+ - `full_and_delta`: a full run removes records in the same way, and a delta run also removes the records that the source reports as deleted in its change feed.
165
+
166
+ A listing page that cannot prove the listing complete removes nothing. When a connector can read a single item, CI also asks the source to confirm that the item is gone before it removes the record.
167
+
168
+ | Knowledge connector | Sync modes | What a delta run reads | Removes on full runs | Removes on delta runs |
169
+ | -------------------------------------- | -------------------------- | ------------------------------------------------------------------ | -------------------- | --------------------- |
170
+ | Notion | Full, delta | Pages edited since the last successful run | Yes | No |
171
+ | HubSpot | Full, delta | Records modified since the last successful run | Yes | No |
172
+ | Monday | Full; delta for items only | Items updated since the last successful run | Yes | No |
173
+ | SQL (PostgreSQL, MySQL) | Full, delta | Rows past the watermark column, or every row without one | Yes | No |
174
+ | Structured File | Full, delta | Nothing when the file is unchanged, the whole file when it changed | Yes | No |
175
+ | External application Knowledge gateway | Full, delta | The change feed that the external application exposes for the data | Yes | Yes |
176
+
177
+ The external application Knowledge gateway is the connector for a customer's own system. Its full run lists every item that the external application exposes, and its delta run reads the application's change feed, where a deleted item removes the record during that run. The application implements both for each Knowledge resource that it shares through its Gateway. The [External apps](../develop/external-apps.md) page explains Gateways.
178
+
179
+ Each entity binding can also limit deletions. `remoteDeletionPropagation` takes `never`, `full` or `full_and_delta`. When you leave it out, the binding follows the connector capability, and CI refuses a value above the capability. `maxRemoteDeletionsPerRun` is the deletion brake for full runs: when a full run finds more records to remove than the limit, it removes none of them and reports that the brake tripped. The brake does not apply to delta runs. A delta run on the external application Knowledge gateway removes every deletion that its change feed reports, however many there are.
180
+
181
+ ```ts
182
+ {
183
+ key: "customers",
184
+ schemaFqn: "com.acme.sales.customer",
185
+ bindingMode: "managed",
186
+ remoteCollectionId: "customers",
187
+ remoteDeletionPropagation: "full",
188
+ maxRemoteDeletionsPerRun: 50,
189
+ fieldMap: [/* ... */],
190
+ }
191
+ ```
192
+
193
+ In the Dashboard, the binding shows the same setting under **Fields map**, **Settings**, **Remote deletions**, and it offers only the values that the connector allows. The brake has no Dashboard field. The run detail shows, for each binding, how many records the run removed or why it removed none.
194
+
195
+ > **Warning:** Set `maxRemoteDeletionsPerRun` before the first full run on customer data. If
196
+ > a listing then misses many items by mistake, the run removes nothing and the
197
+ > run detail shows that the brake tripped.
198
+
199
+ For a Notion demo, a page deleted in Notion stays in CI through the delta runs and leaves on the next full run. The next section explains how to schedule that full run.
200
+
201
+ ## Choose a sync schedule
202
+
203
+ Start from how old the data in the demo may be. For a Knowledge connector, a change at the source reaches CI within one delta interval plus the run time. A deletion at the source reaches CI within one full-run interval plus the run time, unless the connector removes records on delta runs. Choose the delta schedule from the first figure and the full schedule from the second.
204
+
205
+ Some connectors recommend schedules. The Dashboard Knowledge setup suggests them when it creates bindings. A package does not receive them, so it declares its own `schedules` on each pipeline.
206
+
207
+ | Connector | Recommended schedules |
208
+ | --------------------------------------- | ----------------------------------------------------------------------- |
209
+ | Notion | Delta every 5 minutes (`*/5 * * * *`) |
210
+ | HubSpot | Delta every 15 minutes (`*/15 * * * *`), full every night (`0 3 * * *`) |
211
+ | Monday | Delta every 15 minutes (`*/15 * * * *`), full every night (`0 3 * * *`) |
212
+ | SQL, Structured File, Knowledge gateway | None |
213
+
214
+ HubSpot and Monday pair frequent delta runs with a nightly full run because their delta runs cannot see deletions. Notion recommends only the delta schedule, and a Notion delta run removes nothing either. Add a periodic full schedule to every Notion pipeline so that pages deleted in Notion leave CI, and set its frequency from how long a deleted page may stay visible in the demo.
215
+
216
+ For SQL, choose a watermark column before you schedule delta runs. Without one, a delta run reads as much as a full run and removes nothing, so schedule full runs instead. A Structured File delta run costs little when the file has not changed, because CI compares the file version or its content first, but deletions still need a full run. The Knowledge gateway removes records on delta runs, so frequent delta runs keep both changes and deletions current. Keep an occasional full run for the gateway as well: when the application rejects the stored feed position, CI reads the feed again from the start, and that first delta run removes nothing.
217
+
218
+ Every run calls the source, and a full run reads every record again. HubSpot and Monday stop a run when the account reaches its daily API limit, so frequent full runs can use up the quota that the delta runs need. CI retries the shorter rate limits of all three sources.
219
+
220
+ By default, CI checks for due schedules once a minute, so a schedule shorter than one minute gains nothing. A connector instance has one scheduled run at a time. When a schedule comes due during another run of the same instance, it starts after that run ends, and missed times do not pile up. A long full run therefore delays the delta runs of every pipeline in the instance, so schedule full runs at a quiet hour. Each delta run also reads again the overlap window before its start time, 120 seconds for Notion and Monday and 300 seconds for HubSpot and by default.
160
221
 
161
- > **Note:** Deletion on delta runs is planned. Until then, count on full runs to remove
162
- > records that were deleted at the source.
222
+ Indexing and embeddings run after the commit, so search finds new records a little after the run ends. When a run removes a record, the record leaves CI at the commit, and the search index follows.
163
223
 
164
- For a Notion demo, this means that a page deleted in Notion leaves CI on the next full run and stays through the five-minute delta runs.
224
+ A Documents connector has no schedule to choose. CI checks every minute and starts the sync of a Connected Collection five minutes after its last attempt finished, failed attempts included. You can turn scheduled sync off for a collection, or start a sync by hand, which skips the wait. Each sync applies the deletions that the provider reports.
165
225
 
166
226
  ## Health
167
227
 
@@ -184,21 +244,24 @@ Lusha shows the difference. It has an API key connection and prospecting tools a
184
244
 
185
245
  ## Knowledge sync and Documents sync
186
246
 
187
- Knowledge connectors and Documents connectors have different owners and different rules. A Documents connector never creates Knowledge records, and it does not use the Knowledge import stages or the Knowledge deletion rules.
247
+ Knowledge connectors and Documents connectors have different owners and different rules. A Documents connector never creates Knowledge records, and it does not use the Knowledge import stages or the Knowledge deletion capabilities.
188
248
 
189
- | | Knowledge connector | Documents connector |
190
- | ------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------- |
191
- | Writes | Knowledge records in bound schemas | Files and versions in one Connected collection |
192
- | Configured by | A connector instance with entity bindings and pipelines | One Documents connector instance per Connected collection |
193
- | Schedule | Cron schedules per pipeline, FULL or DELTA | CI checks every minute and starts a sync five minutes after the last attempt finished |
194
- | Package declaration | `declareKnowledgeConnectorInstance` | `declareDocumentCollection` with a connection slot and remote folders |
195
- | Folder exclusions | Not applied | Excluded Folder Names skip whole folders |
249
+ | | Knowledge connector | Documents connector |
250
+ | ------------------- | --------------------------------------------------------- | ------------------------------------------------------------------------------------- |
251
+ | Writes | Connected records in bound Knowledge schemas | Files and versions in one Connected Collection |
252
+ | Configured by | A connector instance with entity bindings and pipelines | One Documents connector instance per Connected Collection |
253
+ | Schedule | Cron schedules per pipeline, FULL or DELTA | CI checks every minute and starts a sync five minutes after the last attempt finished |
254
+ | Deletions | Depend on the connector capability, as in the table above | Every sync applies the deletions that the provider reports |
255
+ | Package declaration | `declareKnowledgeConnectorInstance` | `declareDocumentCollection` with a connection slot and remote folders |
256
+ | Folder exclusions | Not applied | Excluded Folder Names skip whole folders |
196
257
 
197
- A Documents sync moves its provider checkpoint forward only after it applies the changes. Two Connected collections on the same provider account cannot overlap their root folders. A finished Documents sync still has to index the files before search finds them, as the [Knowledge](knowledge.md) page explains.
258
+ A Documents sync moves its provider checkpoint forward only after it applies the changes. Two Connected Collections on the same provider account cannot overlap their root folders. A finished Documents sync still has to index the files before search finds them, as the [Documents](documents.md) and [Knowledge](knowledge.md) pages explain.
198
259
 
199
260
  ## Related pages
200
261
 
262
+ - [Data resources](index.md)
201
263
  - [Knowledge](knowledge.md)
264
+ - [Documents](documents.md)
202
265
  - [Datasets](datasets.md)
203
266
  - [Extensions](../develop/extensions.md)
204
267
  - [Access](../platform/access.md)
@@ -4,17 +4,9 @@ Query typed tables of measurements, events and warehouse data, and control who c
4
4
 
5
5
  A Dataset is a typed collection of rows that belongs to an Organization. Each Dataset has a stable identity, a versioned definition, a policy that says who may query it, and provenance on every result. Datasets hold data that has many rows and few edits, such as meter readings, monthly sales figures or a table in a customer's data warehouse. Agents, workflows and Studio query that data in bounded, typed requests.
6
6
 
7
- ## Dataset or Knowledge schema
7
+ ## Datasets and Knowledge
8
8
 
9
- Knowledge models things that people talk about, edit, link and search, with one record per thing. A Dataset models observations about those things, with one row per measurement, event or transaction. The platform does not create a Knowledge record for each row.
10
-
11
- | | Knowledge schema | Dataset |
12
- | -------------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
13
- | One row or record is | A real thing: a customer, a Lead, a goal | An observation: a reading, a transaction, an event |
14
- | People edit rows | Yes, in forms, Studio and the API | No; rows come from file imports or from the source system |
15
- | Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema |
16
- | Queries | Field filters, text and semantic search, relation paths | `records`, `aggregate` and `timeSeries` |
17
- | Access | Knowledge data policies across schemas and domains | One policy per Dataset with an explicit list of principals |
9
+ A Dataset holds observations about the things that Knowledge records describe, with one row per measurement, event or transaction. The [Data resources](index.md) page compares Datasets with Knowledge and Documents and lists which kind fits which demo situation.
18
10
 
19
11
  A strategy Metric shows the split. The Metric itself is a Knowledge record with an owner, a target, a unit, a status and links to goals, and its history of measured values is a Dataset. A Dataset field such as `customerId` can bind to the Customer schema in Knowledge. That binding links the two without copying rows into Knowledge, and deleting one side does not delete the other.
20
12
 
@@ -95,7 +87,7 @@ Every result carries its provenance: the Dataset ID, the definition revision, th
95
87
 
96
88
  Each Dataset has its own policy, and the policy lists its principals by ID. It holds `principals` (each a `user` or `service` with an `id`), `allowedOperations` (`describe`, `records`, `aggregate`, `time_series`), `allowedFields`, `sensitiveFields`, an optional `rowFilter`, and the limits `maxRows`, `maxBytes` and `maxTimeoutMs`. A user also needs the Organization permission to query Datasets, and a Dataset grant does not give Organization membership or capabilities.
97
89
 
98
- Knowledge access works differently: Knowledge data policies apply across schemas and domains. The [Access](../platform/access.md) page explains both models.
90
+ Knowledge access works differently: each Knowledge Data Policy targets one schema, and no policy targets a whole domain. The [Access](../platform/access.md) page explains both models.
99
91
 
100
92
  > **Warning:** A package-declared Dataset without a `dataPolicy` grants access to nobody,
101
93
  > including the person who deployed it. Always declare the principals that the
@@ -103,7 +95,7 @@ Knowledge access works differently: Knowledge data policies apply across schemas
103
95
 
104
96
  ## Declare a Dataset in a package
105
97
 
106
- `declareDatasetConnectorInstance` declares a Dataset in the demo package. An External declaration names the provider, a connection slot, the source location and the policy:
98
+ `declareDatasetConnectorInstance` declares a Dataset in a package. An External declaration names the provider, a connection slot, the source location and the policy:
107
99
 
108
100
  ```ts
109
101
  const orders = declareDatasetConnectorInstance({
@@ -139,16 +131,13 @@ Agents use one tool, Query Dataset (`builtin.core.datasets.tools.query`). It lis
139
131
 
140
132
  Workflows have one node for each operation: describe, query records, query aggregate and query time series. Studio shows Datasets in their own area, apart from the Knowledge views.
141
133
 
142
- ## Pick the right tool for a demo
143
-
144
- - The customer has a warehouse in Fabric or Databricks: use an External Dataset and query it live.
145
- - The customer has a CSV or Excel export and wants to see trends: import it into a Managed Dataset.
146
- - Users must edit records and link them to each other: use Knowledge.
134
+ ## Limits
147
135
 
148
136
  Datasets are sized for about 100,000 to 1,000,000 data points per Organization and do not fit a high-volume telemetry stream. Joins across Datasets, writes back to External Datasets, streaming ingestion and copying External data into a Managed Dataset are not available. A route for external applications to query Datasets directly is planned; today agents, workflows and Studio run the queries.
149
137
 
150
138
  ## Related pages
151
139
 
140
+ - [Data resources](index.md)
152
141
  - [Knowledge](knowledge.md)
153
142
  - [Connectors](connectors.md)
154
143
  - [Access](../platform/access.md)
@@ -0,0 +1,235 @@
1
+ # Documents
2
+
3
+ Keep files in Document Collections, sync them from Microsoft 365 or Google Drive, control who reads them, and know when their content becomes searchable.
4
+
5
+ Documents is the Data resource for files. It keeps files in Document Collections, with folders, Documents and Versions, and with permissions on each collection and folder. Knowledge holds typed records with fields, and Documents holds files that people browse in Studio and that agents and workflows search. A Knowledge file field also stores its file through Documents, in an internal collection that Documents browse and search never show; a reader reaches that file through the Knowledge record. See [Knowledge](knowledge.md) for records and file fields.
6
+
7
+ ## Collections, Documents and Versions
8
+
9
+ | Term | Meaning |
10
+ | -------------------- | ------------------------------------------------------------------------------------------------------ |
11
+ | Document Collection | The container and permission boundary. Each collection has one source: Local or Connected. |
12
+ | Local Collection | A collection with no outside system. CI holds the authoritative copy of its files. |
13
+ | Connected Collection | A collection that one Documents connector keeps in sync with folders in Microsoft 365 or Google Drive. |
14
+ | Document | The identity of one file. It sits in exactly one folder. |
15
+ | Document Version | An immutable revision of a Document. Changed bytes create a new Version; a rename or a move does not. |
16
+
17
+ Files enter a Local Collection through the API and through workflows. A Connected Collection follows its provider: the outside drive stays authoritative, and each sync imports its changes as new Documents and Versions. Two actions in Studio show the difference. Download returns the bytes of the stored Version that the reader selected, and open original opens the live file at the provider, which can be newer than the last sync.
18
+
19
+ ## Where people work with Documents
20
+
21
+ Organization admins and Tenant admins manage collections in the Dashboard under **Documents**, then **Collections**. **Add new** opens Setup. A Local Collection takes two steps, Source and Details. A Connected Collection adds a Connection step, where you pick a Documents connector and a connection, and a Sync step with **Automatic sync**, which is on by default. Details holds the name, the remote folders and the two indexing options, **Index content** and **Index overview**.
22
+
23
+ The detail page of a collection has a **Files** tab with a folder explorer and file details, including Versions, and a **Permission tester** tab. It also shows the indexing and sync settings, the sync status and a **Sync now** action. An Organization admin can inspect folders, files and Version identifiers there without permission to read their content.
24
+
25
+ End users read Documents in Studio. The Documents page lists the collections that the user can read, shows their folders and files, searches by file name and content, downloads the selected Version and opens the original at the provider. It has no upload or edit controls. Plain text files up to 1 MiB show a text preview; other files show a fallback and stay downloadable.
26
+
27
+ > **Warning:** Both indexing options are off by default, in Dashboard Setup and in a package
28
+ > declaration. With both off, CI uses no embeddings and no AI processing for the
29
+ > collection's files, so semantic search and the retrieval tools find nothing in
30
+ > their content. Turn on **Index content** (or `indexContent: true`) for any
31
+ > demo step that searches inside files.
32
+
33
+ ## Permissions
34
+
35
+ Documents has three operations. `read` is called View in the Dashboard and opens the current Version of each file. `history` is called View history and opens older Versions and the Version list. `write` creates, updates and deletes. A Document Data Policy opens one of these operations on one collection or one folder for the holders of a capability. Unlike a Knowledge Data Policy, it has no `where` clause and no Grant Axes.
36
+
37
+ The core default policies open View to holders of the core Documents data capability at `read` and `write` at `manage`. No default policy opens View history, so members see older Versions only where a package Document Data Policy opens `history`. Holders of the Documents admin capability at `manage` get all three operations.
38
+
39
+ A package declares its own capability and the policies that use it:
40
+
41
+ ```ts
42
+ import {
43
+ access,
44
+ declareAccessCapabilities,
45
+ declareAccessDocumentDataPolicy
46
+ } from "@collective-intelligence/extensions-sdk"
47
+
48
+ export const hr = declareAccessCapabilities({
49
+ manifest: { fqn: "acme.hr", version: "1.0.0" },
50
+ definition: {
51
+ nodes: { policies: { kind: "readManage" } },
52
+ lifecycle: { deployMode: "upsert", readonly: true }
53
+ }
54
+ })
55
+
56
+ export const hrPoliciesRead = declareAccessDocumentDataPolicy({
57
+ manifest: { fqn: "acme.hr.policies.hr-policies-read", version: "1.0.0" },
58
+ definition: {
59
+ collection: hrPolicies, // the declaration in the next section
60
+ operation: "read",
61
+ capability: access.requires(hr.policies, "read")
62
+ }
63
+ })
64
+ ```
65
+
66
+ A policy on a folder uses `folder: { collection, target: { drive, folders } }` in place of `collection`. A second policy with `operation: "history"` lets the same people open older Versions. Who holds the capability comes from Access Profiles and Access Grants, which [Grants and permissions](../platform/access.md) explains.
67
+
68
+ To check a result, open the collection's **Permission tester**, choose **View collection as** and pick a person. The tester shows View, View history and Update for each file. It tests the Documents that already exist, so a Connected Collection that has not imported any file yet shows an empty result. Check the sync status before you read an empty result as a denial.
69
+
70
+ ## Declare a Connected Collection in a package
71
+
72
+ `declareDocumentCollection` declares one Connected Collection whose identity is the declaration FQN. No option declares a Local Collection.
73
+
74
+ ```ts
75
+ import {
76
+ declareDocumentCollection,
77
+ packageVariables
78
+ } from "@collective-intelligence/extensions-sdk"
79
+
80
+ const vars = packageVariables({ hrDrive: { type: "string" } })
81
+
82
+ export const hrPolicies = declareDocumentCollection({
83
+ manifest: { fqn: "acme.documents.hr-policies", version: "1.0.0" },
84
+ presentation: {
85
+ displayName: "HR policies",
86
+ description: "Company HR policies"
87
+ },
88
+ definition: {
89
+ connectorFqn:
90
+ "builtin.integrations.microsoft-365.connectors.documents-drive",
91
+ connection: { setupKey: "microsoft365" },
92
+ title: "HR Policies",
93
+ indexContent: true,
94
+ remoteRoots: [{ drive: vars.hrDrive, folders: ["Policies"] }]
95
+ }
96
+ })
97
+ ```
98
+
99
+ `connection` names a connection that the package declares (`{ connectionInstanceFqn }`) or an existing one (`{ setupKey }`, filled with a `{ connectionId }` in the Inputs File). `indexContent` and `indexOverview` default to `false`, and `syncEnabled` defaults to `true`. A remote root is a path `{ drive, folders }` or a `{ remoteId }`. On Microsoft 365, `drive` is the drive title `<siteName> / <libraryName>` that the Dashboard source browser shows, and an empty `folders` names the whole drive. Each value can be a Package Variable, so one release serves environments with different folders.
100
+
101
+ You can leave out `connectorFqn`. The deploy then takes the one Documents connector among the package dependencies whose connection template matches, and it refuses the deploy when none or several match. One `ci deploy` also installs a missing built-in dependency such as `builtin.integrations.microsoft-365`.
102
+
103
+ Package declaration is verified for Microsoft 365 only. Create a Google Drive collection through Dashboard Setup, which offers every installed Documents connector.
104
+
105
+ The first deploy resolves each folder at the provider before it changes anything, so a missing folder refuses the deploy. A later deploy of the same declaration writes nothing and never starts a sync. Removing the declaration, or uninstalling the package, keeps the collection and its files.
106
+
107
+ > **Note:** A Microsoft 365 app that has only the `Files.SelectedOperations.Selected`
108
+ > permission cannot list drives, so a path of names cannot resolve. Declare each
109
+ > root of such a tenant as `{remoteId}`.
110
+
111
+ ## Sync
112
+
113
+ | Connector | FQN | Reads |
114
+ | ----------------------- | ------------------------------------------------------------------ | ------------------------------------------------------ |
115
+ | Microsoft 365 Documents | `builtin.integrations.microsoft-365.connectors.documents-drive` | OneDrive and SharePoint drives through Microsoft Graph |
116
+ | Google Drive Documents | `builtin.integrations.google-workspace.connectors.documents-drive` | My Drive and shared drives |
117
+
118
+ Each Connected Collection has its own Documents connector instance, and two Connected Collections on the same provider account cannot overlap their root folders. Google Drive exports native files as Office files: Docs as DOCX, Sheets as XLSX, Slides as PPTX and Drawings as PNG. It skips native types that have no export, and the Microsoft 365 connector skips OneNote notebooks.
119
+
120
+ With **Automatic sync** on, CI starts a sync five minutes after the last attempt ended, also after a failed one. **Sync now** starts one at once. With Automatic sync off, the collection syncs only on Sync now. A sync saves its provider position only after it applies the changes, so an interrupted sync continues without losing any. A Local Collection has no sync status.
121
+
122
+ | Status | Meaning |
123
+ | ----------- | --------------------------------------------------------------------------------------------------------------- |
124
+ | Not synced | No sync has completed yet. |
125
+ | Syncing | A sync is requested or running. |
126
+ | Up to date | A sync completed in the last ten minutes, or at any time when Automatic sync is off. |
127
+ | Sync issues | The last sync failed, a Quarantined File is open, or Automatic sync is on and no sync completed in ten minutes. |
128
+
129
+ Excluded Folder Names keep archive folders out. The package declaration sets them with `excludedFolderNames`, for example `["archives"]`, and no Dashboard field changes them. A file below a folder with one of these names, at any depth under a root, is never downloaded, read or embedded. The match ignores case, accents, outer spaces and one pair of `[]` or `()` around the name, and the whole name must match. A root is never excluded by its own name. A Document that was imported before its folder took an excluded name leaves search, browse and the AI tools, and it comes back with the same identity when the folder leaves the excluded name. A changed list makes the next sync walk every folder again.
130
+
131
+ A file whose download fails three times on the same provider revision becomes a Quarantined File. The sync stops fetching it and shows it as an error in **Sync issues**, which sets the status to Sync issues. **Sync now** tries every Quarantined File once more, and a new revision of the file at the provider starts it again. Other failed downloads retry on their own and show as warnings that do not change the status.
132
+
133
+ ## Supported file formats
134
+
135
+ | Group | Formats | How CI reads them | Needs |
136
+ | ------------- | ---------------------------------------- | ---------------------------------------------------------- | ---------------------- |
137
+ | Text | TXT, MD, CSV, JSON | Text reader | Nothing |
138
+ | PDF | PDF | Text and structure, page OCR for pages with almost no text | A vision model for OCR |
139
+ | Office | DOCX, XLSX, PPTX | Office readers | Nothing |
140
+ | Legacy Office | DOC, XLS, PPT, RTF | Conversion to DOCX, XLSX or PPTX, then the Office readers | The Compute service |
141
+ | Images | JPEG, PNG, WebP | Visible text and a description from a vision model | A vision model |
142
+ | Audio | AAC, AIFF, FLAC, M4A, MP3, OGG, WAV, WMA | Speech transcription | The Compute service |
143
+ | Video | AVI, MKV, MOV, MP4, WebM, WMV | Speech transcription of the audio track | The Compute service |
144
+
145
+ CI reads no other format. HTML, TIF, TIFF, BMP and GIF files, for example, sync and download like any file, but CI does not index their content. The Compute service is a separate CI service that runs transcription and Office conversion.
146
+
147
+ Each entry path has its own list of accepted formats. Documents accepts all formats in the table. Knowledge file fields accept DOC, DOCX, JPEG, MD, PDF, PNG, PPT, PPTX, RTF, TXT, WebP, XLS and XLSX. A CI deployment can narrow either list. A file outside the list still syncs, lists and downloads, and its indexing status is `unsupported`. Without the Compute service, a legacy Office file is `unsupported` with the reason that conversion is not available.
148
+
149
+ ## What CI reads from each format
150
+
151
+ For a PDF, CI reads the native text and rebuilds its structure. In a PDF of three or more pages it removes headers and footers that repeat across pages. It finds headings up to six levels from their numbering and font size, and the document title on page one. It rebuilds tables from ruled grids or from aligned columns, also when a table continues on the next page, and it reads multi-column pages column by column. A page with fewer than 50 letters and digits goes to page OCR, and a PDF with almost no text is read by OCR as a whole.
152
+
153
+ For Office files, CI keeps what the author sees:
154
+
155
+ - DOCX headings come from Word heading styles in any language, and from font size only when the file uses no heading style. Text boxes, footnotes and endnotes stay next to the text that holds them. Deleted tracked changes and comments stay out, and headers and footers are kept as metadata.
156
+ - PPTX gives the shown slides with their groups, tables and speaker notes. Hidden slides give nothing, and the title of the first slide is the document title.
157
+ - XLSX gives one table per visible sheet. Hidden sheets, rows and columns give nothing, dates use ISO 8601, and a text cell such as `00123` keeps its leading zeros.
158
+
159
+ A merged cell repeats its value down the rows it covers, so each table row has the full column count.
160
+
161
+ DOC, XLS, PPT and RTF files go through one conversion with LibreOffice in the Compute service, and the Office readers then read the converted copy. The copy stays internal: search results, citations and downloads always point to the original file. The conversion never runs macros and never fetches external content. A corrupt or password-protected file, a conversion over 60 seconds or an empty result fails that file.
162
+
163
+ Images in DOCX, PPTX and PDF files go to the vision model too, as do stand-alone JPEG, PNG and WebP files. Each kept image gives its visible text and a short description. CI drops images whose short side is under 128 pixels, keeps at most 50 images per file and reads each distinct image once. Each described image uses one page of the Organization's daily OCR budget. A file with an image that failed or found no budget is `partial`.
164
+
165
+ Audio and video go to the Compute service for transcription, which detects the language. A video gives the speech of its audio track; CI does not index video frames. Opening, downloading or searching a file never starts a transcription.
166
+
167
+ The file indexing strategy decides how much of this work runs. CI takes the Organization's setting, else the tenant default, else `text_with_image_descriptions`.
168
+
169
+ - `text_only` reads text only and drops images.
170
+ - `text_with_ocr` adds page OCR for sparse PDF pages and reads stand-alone image files.
171
+ - `text_with_image_descriptions`, the default, also describes every kept image.
172
+ - `hybrid_multimodal` also stores an image vector for each kept image, and it needs an embedding model that accepts images.
173
+
174
+ Every strategy except `text_only` needs an enabled vision model.
175
+
176
+ | Limit | Default |
177
+ | ---------------------------------- | ---------------------------------------------------- |
178
+ | PDF page OCR and image reading | PDFs up to 32 MiB; text extraction has no size limit |
179
+ | One image sent to the vision model | 8 MiB |
180
+ | Audio and video transcription | 100 MiB and 30 minutes per file |
181
+ | Legacy Office conversion | 60 seconds per file |
182
+ | Studio text preview | 1 MiB |
183
+
184
+ Above 32 MiB, a PDF keeps the native text of its sparse pages and becomes `partial`, and a PDF with almost no text fails.
185
+
186
+ ## How indexing works
187
+
188
+ ```mermaid
189
+ flowchart LR
190
+ S[Sync or API write] --> V[New Document Version]
191
+ V --> E[Extraction: text, structure, OCR, images, speech]
192
+ E --> F[Fragments]
193
+ F --> M[Embeddings]
194
+ M --> Q[Searchable]
195
+ E -. optional .-> O[Overview]
196
+ ```
197
+
198
+ Knowledge file fields and Documents share one processing pipeline. Extraction turns a file into text and structure. Segmentation cuts that text into Fragments, the units that search and the retrieval tools return. CI then embeds each Fragment with the document title in front, so a passage from page 40 still says which file it comes from. With **Index overview** on, CI also writes a short summary of the file.
199
+
200
+ The pipeline keeps the structure that extraction found. CI splits a paragraph only when it is too long for one Fragment, and then between sentences. Fragments do not overlap. A table becomes groups of whole rows, and every group repeats the table header, so each Fragment is a readable table. A table never shares a Fragment with prose, and a heading joins the content that follows it.
201
+
202
+ CI stores one result for the same bytes in the same Organization with the same processing settings, and every Document and Knowledge file with those bytes uses that result. A duplicate file costs no second extraction or embedding. When the processing settings change, for example the file indexing strategy, CI processes the files again in the background, and the old result serves search until the new one is ready.
203
+
204
+ A file is searchable when its Fragments and their vectors are saved. CI does not wait for the overview. Only the current Version of each Document is in semantic search. While a new Version is processed, search keeps the previous Version, and it switches when the new one is ready.
205
+
206
+ ## Wait for indexing in a workflow
207
+
208
+ > **Warning:** A finished sync means that the files are imported. Indexing runs after the
209
+ > sync, so the new content is not searchable yet.
210
+
211
+ The Documents workflow nodes come in two branches, Local and Connected. Both have Create, List, Get, Update, Delete and Restore Collection, and Search Documents, Search Within Document, Expand Passage and Read Document Range. The Connected branch adds Start Collection Sync, Get Collection Sync Status, Set Collection Sync Settings and Wait for Collection Sync.
212
+
213
+ Wait for Collection Sync takes the import reference that Create Collection or Start Collection Sync returns, and a `wait` level. Level `sync`, the default, waits for that import. Level `indexing` also waits for the enabled indexing of the files that the import brought in. Use `indexing` before a demo step that searches the new content. Level `indexing` needs at least one indexing option on the collection, and the wait fails when both are off. An optional `expiresAt` ends the wait through an `expired` outcome. The wait never starts indexing and never changes the collection's settings.
214
+
215
+ A search node returns what is searchable when it runs. In a test, an empty result from a random query does not prove that a file is missing, because semantic search ranks by meaning. Test an empty result with a scope that you know is empty or with a filter that excludes everything.
216
+
217
+ ## Search and retrieval tools
218
+
219
+ Studio search matches file names at once and adds content matches from the semantic index. A file-name match works before indexing, and a content match needs an indexed Version. One search returns at most 100 results.
220
+
221
+ Agents and Skills search Documents through two built-in tools. Bind them as [AI](../ai/index.md) describes.
222
+
223
+ - Documents Query (`builtin.core.documents.tools.query`) searches Documents only. Each result has a title, a score, an overview, a snippet and up to three ranked passages with their location. An empty `documentCollections` list searches every readable collection, and `documentId` searches inside one file. Its `coverage` output lists readable files that are still indexing, partial, failed or over budget.
224
+ - Advanced Information Retrieval (`builtin.core.information.tools.advanced-retrieval`) searches Knowledge and Documents together, and `sources` defaults to both. It comes with Expand Passage and Read Source Range to read the text around a passage. It also returns a file from a Knowledge file field when the reader can read that record.
225
+
226
+ Every search checks the reader's current permissions. A file that the reader may not read looks the same as a file that does not exist.
227
+
228
+ ## Related pages
229
+
230
+ - [Data resources](index.md)
231
+ - [Knowledge](knowledge.md)
232
+ - [Connectors](connectors.md)
233
+ - [Grants and permissions](../platform/access.md)
234
+ - [AI](../ai/index.md)
235
+ - [Extensions](../develop/extensions.md)
@@ -0,0 +1,62 @@
1
+ # Data resources
2
+
3
+ Choose between Knowledge, Datasets and Documents, and see where each kind keeps its data.
4
+
5
+ Collective Intelligence (CI) has three kinds of Data resource: Knowledge, Datasets and Documents. Each kind has its own unit. A Knowledge schema holds records, a Dataset holds typed rows, and a Document Collection holds files. Choose the kind for each part of your demo first, then open its page. Every Data resource belongs to one Organization; [Platform architecture](../platform/architecture.md) explains how CI keeps the data of Organizations apart.
6
+
7
+ ## Which kind to use
8
+
9
+ - Users create, edit and link items such as a customer, a Lead or a goal: use Knowledge.
10
+ - The customer keeps a warehouse in Microsoft Fabric or Databricks: use an External Dataset, which CI queries live.
11
+ - The customer has a CSV or Excel export and wants to see trends or totals: import it into a Managed Dataset.
12
+ - People read, search and cite files such as contracts, manuals or reports: use Documents. Pick a Connected Collection when the files live in Microsoft 365 or Google Drive, and a Local Collection when your demo adds the files itself.
13
+ - One record needs an attached file, such as a signed PDF on a Lead: use a file field on the Knowledge schema.
14
+
15
+ Knowledge and Datasets often appear in the same demo. Knowledge models the things that people talk about, edit and link, with one record per thing. A Dataset models observations about those things, with one row per measurement, event or transaction, and CI does not create a Knowledge record for each row. The [Datasets](datasets.md) page shows how a Dataset field can bind to a Knowledge schema without copying rows.
16
+
17
+ ## Compare the three kinds
18
+
19
+ | | Knowledge | Datasets | Documents |
20
+ | --------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
21
+ | One item is | A record of a schema | A row of a Dataset | A Document with immutable Versions, in a Document Collection |
22
+ | Who changes it | People in forms, Studio and apps, plus agents and workflows | Ingestion runs for a Managed Dataset, the source system for an External one | The API and workflows for a Local Collection, sync for a Connected one |
23
+ | Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema | Folders inside the collection |
24
+ | Queries | Field filters, text search, semantic search and relation paths | `records`, `aggregate` and `timeSeries` | Filename search and semantic search over content |
25
+ | Access | Knowledge Data Policies, each on one schema | One policy per Dataset with an explicit list of principals | Document Data Policies and rules per collection, folder and file |
26
+ | Tool for agents | Knowledge Query | Query Dataset | Documents Query |
27
+
28
+ Advanced Information Retrieval is the one tool that searches more than one kind in a single call. It covers Knowledge and Documents. Agents query Datasets with Query Dataset only. The [AI](../ai/index.md) page explains tools and Skills.
29
+
30
+ ## Where the data lives
31
+
32
+ Two questions describe each mode. The first asks who holds the authoritative copy: a managed mode keeps it in CI, and an external mode leaves it in the outside system. The second asks whether a connector keeps the data in sync from an outside system; data that a connector syncs is connected.
33
+
34
+ ```mermaid
35
+ flowchart LR
36
+ W[People, apps, agents, workflows] --> K[Knowledge records]
37
+ S[Outside system] -->|connector sync| KC[Connected Knowledge records]
38
+ F[File import] --> M[Managed Dataset]
39
+ S -.->|live query, no copy| E[External Dataset]
40
+ A[API and workflows] --> L[Local Collection]
41
+ S -->|connector sync| C[Connected Collection]
42
+ ```
43
+
44
+ | Kind and mode | Stored in CI | Authoritative copy | Changes come from |
45
+ | --------------------------- | -------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------- |
46
+ | Knowledge records | Yes | CI | People, apps, agents and workflows |
47
+ | Connected Knowledge records | Yes, as a synced copy | The source system; with an outbound binding, CI edits also reach it | Connector pipeline runs, and CI edits when the schema allows them |
48
+ | Managed Dataset | Yes, in PostgreSQL | CI | Ingestion runs that append, upsert or replace rows |
49
+ | External Dataset | No | The source system | The source system only |
50
+ | Local Collection | Yes | CI | The API and workflows |
51
+ | Connected Collection | Yes, as immutable Versions | The source system | Sync runs from Microsoft 365 or Google Drive |
52
+
53
+ An External Dataset has no copy in CI. Every query reaches the provider, CI caches no results, and a query fails when the provider is down. CI writes nothing back to an External Dataset. Documents have no live mode: both collection types store their files in CI, and Studio can open the current file in the provider for a Connected Collection. For connected Knowledge records, the schema's `localMutationPolicy` decides whether people can edit them in CI. `always` allows edits, `connector_backed` allows them only while a connector binding sends edits back to the source, and `never` refuses them. The [Connectors](connectors.md) page explains bindings and sync.
54
+
55
+ ## Related pages
56
+
57
+ - [Knowledge](knowledge.md)
58
+ - [Datasets](datasets.md)
59
+ - [Documents](documents.md)
60
+ - [Connectors](connectors.md)
61
+ - [Platform architecture](../platform/architecture.md)
62
+ - [AI](../ai/index.md)