@collective-intelligence/extensions-sdk-frontend 0.3.1 → 0.3.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -4,17 +4,9 @@ Query typed tables of measurements, events and warehouse data, and control who c
4
4
 
5
5
  A Dataset is a typed collection of rows that belongs to an Organization. Each Dataset has a stable identity, a versioned definition, a policy that says who may query it, and provenance on every result. Datasets hold data that has many rows and few edits, such as meter readings, monthly sales figures or a table in a customer's data warehouse. Agents, workflows and Studio query that data in bounded, typed requests.
6
6
 
7
- ## Dataset or Knowledge schema
7
+ ## Datasets and Knowledge
8
8
 
9
- Knowledge models things that people talk about, edit, link and search, with one record per thing. A Dataset models observations about those things, with one row per measurement, event or transaction. The platform does not create a Knowledge record for each row.
10
-
11
- | | Knowledge schema | Dataset |
12
- | -------------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
13
- | One row or record is | A real thing: a customer, a Lead, a goal | An observation: a reading, a transaction, an event |
14
- | People edit rows | Yes, in forms, Studio and the API | No; rows come from file imports or from the source system |
15
- | Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema |
16
- | Queries | Field filters, text and semantic search, relation paths | `records`, `aggregate` and `timeSeries` |
17
- | Access | Knowledge data policies across schemas and domains | One policy per Dataset with an explicit list of principals |
9
+ A Dataset holds observations about the things that Knowledge records describe, with one row per measurement, event or transaction. The [Data resources](index.md) page compares Datasets with Knowledge and Documents and lists which kind fits which demo situation.
18
10
 
19
11
  A strategy Metric shows the split. The Metric itself is a Knowledge record with an owner, a target, a unit, a status and links to goals, and its history of measured values is a Dataset. A Dataset field such as `customerId` can bind to the Customer schema in Knowledge. That binding links the two without copying rows into Knowledge, and deleting one side does not delete the other.
20
12
 
@@ -95,7 +87,7 @@ Every result carries its provenance: the Dataset ID, the definition revision, th
95
87
 
96
88
  Each Dataset has its own policy, and the policy lists its principals by ID. It holds `principals` (each a `user` or `service` with an `id`), `allowedOperations` (`describe`, `records`, `aggregate`, `time_series`), `allowedFields`, `sensitiveFields`, an optional `rowFilter`, and the limits `maxRows`, `maxBytes` and `maxTimeoutMs`. A user also needs the Organization permission to query Datasets, and a Dataset grant does not give Organization membership or capabilities.
97
89
 
98
- Knowledge access works differently: Knowledge data policies apply across schemas and domains. The [Access](../platform/access.md) page explains both models.
90
+ Knowledge access works differently: each Knowledge Data Policy targets one schema, and no policy targets a whole domain. The [Access](../platform/access.md) page explains both models.
99
91
 
100
92
  > **Warning:** A package-declared Dataset without a `dataPolicy` grants access to nobody,
101
93
  > including the person who deployed it. Always declare the principals that the
@@ -103,7 +95,7 @@ Knowledge access works differently: Knowledge data policies apply across schemas
103
95
 
104
96
  ## Declare a Dataset in a package
105
97
 
106
- `declareDatasetConnectorInstance` declares a Dataset in the demo package. An External declaration names the provider, a connection slot, the source location and the policy:
98
+ `declareDatasetConnectorInstance` declares a Dataset in a package. An External declaration names the provider, a connection slot, the source location and the policy:
107
99
 
108
100
  ```ts
109
101
  const orders = declareDatasetConnectorInstance({
@@ -139,16 +131,13 @@ Agents use one tool, Query Dataset (`builtin.core.datasets.tools.query`). It lis
139
131
 
140
132
  Workflows have one node for each operation: describe, query records, query aggregate and query time series. Studio shows Datasets in their own area, apart from the Knowledge views.
141
133
 
142
- ## Pick the right tool for a demo
143
-
144
- - The customer has a warehouse in Fabric or Databricks: use an External Dataset and query it live.
145
- - The customer has a CSV or Excel export and wants to see trends: import it into a Managed Dataset.
146
- - Users must edit records and link them to each other: use Knowledge.
134
+ ## Limits
147
135
 
148
136
  Datasets are sized for about 100,000 to 1,000,000 data points per Organization and do not fit a high-volume telemetry stream. Joins across Datasets, writes back to External Datasets, streaming ingestion and copying External data into a Managed Dataset are not available. A route for external applications to query Datasets directly is planned; today agents, workflows and Studio run the queries.
149
137
 
150
138
  ## Related pages
151
139
 
140
+ - [Data resources](index.md)
152
141
  - [Knowledge](knowledge.md)
153
142
  - [Connectors](connectors.md)
154
143
  - [Access](../platform/access.md)
@@ -0,0 +1,235 @@
1
+ # Documents
2
+
3
+ Keep files in Document Collections, sync them from Microsoft 365 or Google Drive, control who reads them, and know when their content becomes searchable.
4
+
5
+ Documents is the Data resource for files. It keeps files in Document Collections, with folders, Documents and Versions, and with permissions on each collection and folder. Knowledge holds typed records with fields, and Documents holds files that people browse in Studio and that agents and workflows search. A Knowledge file field also stores its file through Documents, in an internal collection that Documents browse and search never show; a reader reaches that file through the Knowledge record. See [Knowledge](knowledge.md) for records and file fields.
6
+
7
+ ## Collections, Documents and Versions
8
+
9
+ | Term | Meaning |
10
+ | -------------------- | ------------------------------------------------------------------------------------------------------ |
11
+ | Document Collection | The container and permission boundary. Each collection has one source: Local or Connected. |
12
+ | Local Collection | A collection with no outside system. CI holds the authoritative copy of its files. |
13
+ | Connected Collection | A collection that one Documents connector keeps in sync with folders in Microsoft 365 or Google Drive. |
14
+ | Document | The identity of one file. It sits in exactly one folder. |
15
+ | Document Version | An immutable revision of a Document. Changed bytes create a new Version; a rename or a move does not. |
16
+
17
+ Files enter a Local Collection through the API and through workflows. A Connected Collection follows its provider: the outside drive stays authoritative, and each sync imports its changes as new Documents and Versions. Two actions in Studio show the difference. Download returns the bytes of the stored Version that the reader selected, and open original opens the live file at the provider, which can be newer than the last sync.
18
+
19
+ ## Where people work with Documents
20
+
21
+ Organization admins and Tenant admins manage collections in the Dashboard under **Documents**, then **Collections**. **Add new** opens Setup. A Local Collection takes two steps, Source and Details. A Connected Collection adds a Connection step, where you pick a Documents connector and a connection, and a Sync step with **Automatic sync**, which is on by default. Details holds the name, the remote folders and the two indexing options, **Index content** and **Index overview**.
22
+
23
+ The detail page of a collection has a **Files** tab with a folder explorer and file details, including Versions, and a **Permission tester** tab. It also shows the indexing and sync settings, the sync status and a **Sync now** action. An Organization admin can inspect folders, files and Version identifiers there without permission to read their content.
24
+
25
+ End users read Documents in Studio. The Documents page lists the collections that the user can read, shows their folders and files, searches by file name and content, downloads the selected Version and opens the original at the provider. It has no upload or edit controls. Plain text files up to 1 MiB show a text preview; other files show a fallback and stay downloadable.
26
+
27
+ > **Warning:** Both indexing options are off by default, in Dashboard Setup and in a package
28
+ > declaration. With both off, CI uses no embeddings and no AI processing for the
29
+ > collection's files, so semantic search and the retrieval tools find nothing in
30
+ > their content. Turn on **Index content** (or `indexContent: true`) for any
31
+ > demo step that searches inside files.
32
+
33
+ ## Permissions
34
+
35
+ Documents has three operations. `read` is called View in the Dashboard and opens the current Version of each file. `history` is called View history and opens older Versions and the Version list. `write` creates, updates and deletes. A Document Data Policy opens one of these operations on one collection or one folder for the holders of a capability. Unlike a Knowledge Data Policy, it has no `where` clause and no Grant Axes.
36
+
37
+ The core default policies open View to holders of the core Documents data capability at `read` and `write` at `manage`. No default policy opens View history, so members see older Versions only where a package Document Data Policy opens `history`. Holders of the Documents admin capability at `manage` get all three operations.
38
+
39
+ A package declares its own capability and the policies that use it:
40
+
41
+ ```ts
42
+ import {
43
+ access,
44
+ declareAccessCapabilities,
45
+ declareAccessDocumentDataPolicy
46
+ } from "@collective-intelligence/extensions-sdk"
47
+
48
+ export const hr = declareAccessCapabilities({
49
+ manifest: { fqn: "acme.hr", version: "1.0.0" },
50
+ definition: {
51
+ nodes: { policies: { kind: "readManage" } },
52
+ lifecycle: { deployMode: "upsert", readonly: true }
53
+ }
54
+ })
55
+
56
+ export const hrPoliciesRead = declareAccessDocumentDataPolicy({
57
+ manifest: { fqn: "acme.hr.policies.hr-policies-read", version: "1.0.0" },
58
+ definition: {
59
+ collection: hrPolicies, // the declaration in the next section
60
+ operation: "read",
61
+ capability: access.requires(hr.policies, "read")
62
+ }
63
+ })
64
+ ```
65
+
66
+ A policy on a folder uses `folder: { collection, target: { drive, folders } }` in place of `collection`. A second policy with `operation: "history"` lets the same people open older Versions. Who holds the capability comes from Access Profiles and Access Grants, which [Grants and permissions](../platform/access.md) explains.
67
+
68
+ To check a result, open the collection's **Permission tester**, choose **View collection as** and pick a person. The tester shows View, View history and Update for each file. It tests the Documents that already exist, so a Connected Collection that has not imported any file yet shows an empty result. Check the sync status before you read an empty result as a denial.
69
+
70
+ ## Declare a Connected Collection in a package
71
+
72
+ `declareDocumentCollection` declares one Connected Collection whose identity is the declaration FQN. No option declares a Local Collection.
73
+
74
+ ```ts
75
+ import {
76
+ declareDocumentCollection,
77
+ packageVariables
78
+ } from "@collective-intelligence/extensions-sdk"
79
+
80
+ const vars = packageVariables({ hrDrive: { type: "string" } })
81
+
82
+ export const hrPolicies = declareDocumentCollection({
83
+ manifest: { fqn: "acme.documents.hr-policies", version: "1.0.0" },
84
+ presentation: {
85
+ displayName: "HR policies",
86
+ description: "Company HR policies"
87
+ },
88
+ definition: {
89
+ connectorFqn:
90
+ "builtin.integrations.microsoft-365.connectors.documents-drive",
91
+ connection: { setupKey: "microsoft365" },
92
+ title: "HR Policies",
93
+ indexContent: true,
94
+ remoteRoots: [{ drive: vars.hrDrive, folders: ["Policies"] }]
95
+ }
96
+ })
97
+ ```
98
+
99
+ `connection` names a connection that the package declares (`{ connectionInstanceFqn }`) or an existing one (`{ setupKey }`, filled with a `{ connectionId }` in the Inputs File). `indexContent` and `indexOverview` default to `false`, and `syncEnabled` defaults to `true`. A remote root is a path `{ drive, folders }` or a `{ remoteId }`. On Microsoft 365, `drive` is the drive title `<siteName> / <libraryName>` that the Dashboard source browser shows, and an empty `folders` names the whole drive. Each value can be a Package Variable, so one release serves environments with different folders.
100
+
101
+ You can leave out `connectorFqn`. The deploy then takes the one Documents connector among the package dependencies whose connection template matches, and it refuses the deploy when none or several match. One `ci deploy` also installs a missing built-in dependency such as `builtin.integrations.microsoft-365`.
102
+
103
+ Package declaration is verified for Microsoft 365 only. Create a Google Drive collection through Dashboard Setup, which offers every installed Documents connector.
104
+
105
+ The first deploy resolves each folder at the provider before it changes anything, so a missing folder refuses the deploy. A later deploy of the same declaration writes nothing and never starts a sync. Removing the declaration, or uninstalling the package, keeps the collection and its files.
106
+
107
+ > **Note:** A Microsoft 365 app that has only the `Files.SelectedOperations.Selected`
108
+ > permission cannot list drives, so a path of names cannot resolve. Declare each
109
+ > root of such a tenant as `{remoteId}`.
110
+
111
+ ## Sync
112
+
113
+ | Connector | FQN | Reads |
114
+ | ----------------------- | ------------------------------------------------------------------ | ------------------------------------------------------ |
115
+ | Microsoft 365 Documents | `builtin.integrations.microsoft-365.connectors.documents-drive` | OneDrive and SharePoint drives through Microsoft Graph |
116
+ | Google Drive Documents | `builtin.integrations.google-workspace.connectors.documents-drive` | My Drive and shared drives |
117
+
118
+ Each Connected Collection has its own Documents connector instance, and two Connected Collections on the same provider account cannot overlap their root folders. Google Drive exports native files as Office files: Docs as DOCX, Sheets as XLSX, Slides as PPTX and Drawings as PNG. It skips native types that have no export, and the Microsoft 365 connector skips OneNote notebooks.
119
+
120
+ With **Automatic sync** on, CI starts a sync five minutes after the last attempt ended, also after a failed one. **Sync now** starts one at once. With Automatic sync off, the collection syncs only on Sync now. A sync saves its provider position only after it applies the changes, so an interrupted sync continues without losing any. A Local Collection has no sync status.
121
+
122
+ | Status | Meaning |
123
+ | ----------- | --------------------------------------------------------------------------------------------------------------- |
124
+ | Not synced | No sync has completed yet. |
125
+ | Syncing | A sync is requested or running. |
126
+ | Up to date | A sync completed in the last ten minutes, or at any time when Automatic sync is off. |
127
+ | Sync issues | The last sync failed, a Quarantined File is open, or Automatic sync is on and no sync completed in ten minutes. |
128
+
129
+ Excluded Folder Names keep archive folders out. The package declaration sets them with `excludedFolderNames`, for example `["archives"]`, and no Dashboard field changes them. A file below a folder with one of these names, at any depth under a root, is never downloaded, read or embedded. The match ignores case, accents, outer spaces and one pair of `[]` or `()` around the name, and the whole name must match. A root is never excluded by its own name. A Document that was imported before its folder took an excluded name leaves search, browse and the AI tools, and it comes back with the same identity when the folder leaves the excluded name. A changed list makes the next sync walk every folder again.
130
+
131
+ A file whose download fails three times on the same provider revision becomes a Quarantined File. The sync stops fetching it and shows it as an error in **Sync issues**, which sets the status to Sync issues. **Sync now** tries every Quarantined File once more, and a new revision of the file at the provider starts it again. Other failed downloads retry on their own and show as warnings that do not change the status.
132
+
133
+ ## Supported file formats
134
+
135
+ | Group | Formats | How CI reads them | Needs |
136
+ | ------------- | ---------------------------------------- | ---------------------------------------------------------- | ---------------------- |
137
+ | Text | TXT, MD, CSV, JSON | Text reader | Nothing |
138
+ | PDF | PDF | Text and structure, page OCR for pages with almost no text | A vision model for OCR |
139
+ | Office | DOCX, XLSX, PPTX | Office readers | Nothing |
140
+ | Legacy Office | DOC, XLS, PPT, RTF | Conversion to DOCX, XLSX or PPTX, then the Office readers | The Compute service |
141
+ | Images | JPEG, PNG, WebP | Visible text and a description from a vision model | A vision model |
142
+ | Audio | AAC, AIFF, FLAC, M4A, MP3, OGG, WAV, WMA | Speech transcription | The Compute service |
143
+ | Video | AVI, MKV, MOV, MP4, WebM, WMV | Speech transcription of the audio track | The Compute service |
144
+
145
+ CI reads no other format. HTML, TIF, TIFF, BMP and GIF files, for example, sync and download like any file, but CI does not index their content. The Compute service is a separate CI service that runs transcription and Office conversion.
146
+
147
+ Each entry path has its own list of accepted formats. Documents accepts all formats in the table. Knowledge file fields accept DOC, DOCX, JPEG, MD, PDF, PNG, PPT, PPTX, RTF, TXT, WebP, XLS and XLSX. A CI deployment can narrow either list. A file outside the list still syncs, lists and downloads, and its indexing status is `unsupported`. Without the Compute service, a legacy Office file is `unsupported` with the reason that conversion is not available.
148
+
149
+ ## What CI reads from each format
150
+
151
+ For a PDF, CI reads the native text and rebuilds its structure. In a PDF of three or more pages it removes headers and footers that repeat across pages. It finds headings up to six levels from their numbering and font size, and the document title on page one. It rebuilds tables from ruled grids or from aligned columns, also when a table continues on the next page, and it reads multi-column pages column by column. A page with fewer than 50 letters and digits goes to page OCR, and a PDF with almost no text is read by OCR as a whole.
152
+
153
+ For Office files, CI keeps what the author sees:
154
+
155
+ - DOCX headings come from Word heading styles in any language, and from font size only when the file uses no heading style. Text boxes, footnotes and endnotes stay next to the text that holds them. Deleted tracked changes and comments stay out, and headers and footers are kept as metadata.
156
+ - PPTX gives the shown slides with their groups, tables and speaker notes. Hidden slides give nothing, and the title of the first slide is the document title.
157
+ - XLSX gives one table per visible sheet. Hidden sheets, rows and columns give nothing, dates use ISO 8601, and a text cell such as `00123` keeps its leading zeros.
158
+
159
+ A merged cell repeats its value down the rows it covers, so each table row has the full column count.
160
+
161
+ DOC, XLS, PPT and RTF files go through one conversion with LibreOffice in the Compute service, and the Office readers then read the converted copy. The copy stays internal: search results, citations and downloads always point to the original file. The conversion never runs macros and never fetches external content. A corrupt or password-protected file, a conversion over 60 seconds or an empty result fails that file.
162
+
163
+ Images in DOCX, PPTX and PDF files go to the vision model too, as do stand-alone JPEG, PNG and WebP files. Each kept image gives its visible text and a short description. CI drops images whose short side is under 128 pixels, keeps at most 50 images per file and reads each distinct image once. Each described image uses one page of the Organization's daily OCR budget. A file with an image that failed or found no budget is `partial`.
164
+
165
+ Audio and video go to the Compute service for transcription, which detects the language. A video gives the speech of its audio track; CI does not index video frames. Opening, downloading or searching a file never starts a transcription.
166
+
167
+ The file indexing strategy decides how much of this work runs. CI takes the Organization's setting, else the tenant default, else `text_with_image_descriptions`.
168
+
169
+ - `text_only` reads text only and drops images.
170
+ - `text_with_ocr` adds page OCR for sparse PDF pages and reads stand-alone image files.
171
+ - `text_with_image_descriptions`, the default, also describes every kept image.
172
+ - `hybrid_multimodal` also stores an image vector for each kept image, and it needs an embedding model that accepts images.
173
+
174
+ Every strategy except `text_only` needs an enabled vision model.
175
+
176
+ | Limit | Default |
177
+ | ---------------------------------- | ---------------------------------------------------- |
178
+ | PDF page OCR and image reading | PDFs up to 32 MiB; text extraction has no size limit |
179
+ | One image sent to the vision model | 8 MiB |
180
+ | Audio and video transcription | 100 MiB and 30 minutes per file |
181
+ | Legacy Office conversion | 60 seconds per file |
182
+ | Studio text preview | 1 MiB |
183
+
184
+ Above 32 MiB, a PDF keeps the native text of its sparse pages and becomes `partial`, and a PDF with almost no text fails.
185
+
186
+ ## How indexing works
187
+
188
+ ```mermaid
189
+ flowchart LR
190
+ S[Sync or API write] --> V[New Document Version]
191
+ V --> E[Extraction: text, structure, OCR, images, speech]
192
+ E --> F[Fragments]
193
+ F --> M[Embeddings]
194
+ M --> Q[Searchable]
195
+ E -. optional .-> O[Overview]
196
+ ```
197
+
198
+ Knowledge file fields and Documents share one processing pipeline. Extraction turns a file into text and structure. Segmentation cuts that text into Fragments, the units that search and the retrieval tools return. CI then embeds each Fragment with the document title in front, so a passage from page 40 still says which file it comes from. With **Index overview** on, CI also writes a short summary of the file.
199
+
200
+ The pipeline keeps the structure that extraction found. CI splits a paragraph only when it is too long for one Fragment, and then between sentences. Fragments do not overlap. A table becomes groups of whole rows, and every group repeats the table header, so each Fragment is a readable table. A table never shares a Fragment with prose, and a heading joins the content that follows it.
201
+
202
+ CI stores one result for the same bytes in the same Organization with the same processing settings, and every Document and Knowledge file with those bytes uses that result. A duplicate file costs no second extraction or embedding. When the processing settings change, for example the file indexing strategy, CI processes the files again in the background, and the old result serves search until the new one is ready.
203
+
204
+ A file is searchable when its Fragments and their vectors are saved. CI does not wait for the overview. Only the current Version of each Document is in semantic search. While a new Version is processed, search keeps the previous Version, and it switches when the new one is ready.
205
+
206
+ ## Wait for indexing in a workflow
207
+
208
+ > **Warning:** A finished sync means that the files are imported. Indexing runs after the
209
+ > sync, so the new content is not searchable yet.
210
+
211
+ The Documents workflow nodes come in two branches, Local and Connected. Both have Create, List, Get, Update, Delete and Restore Collection, and Search Documents, Search Within Document, Expand Passage and Read Document Range. The Connected branch adds Start Collection Sync, Get Collection Sync Status, Set Collection Sync Settings and Wait for Collection Sync.
212
+
213
+ Wait for Collection Sync takes the import reference that Create Collection or Start Collection Sync returns, and a `wait` level. Level `sync`, the default, waits for that import. Level `indexing` also waits for the enabled indexing of the files that the import brought in. Use `indexing` before a demo step that searches the new content. Level `indexing` needs at least one indexing option on the collection, and the wait fails when both are off. An optional `expiresAt` ends the wait through an `expired` outcome. The wait never starts indexing and never changes the collection's settings.
214
+
215
+ A search node returns what is searchable when it runs. In a test, an empty result from a random query does not prove that a file is missing, because semantic search ranks by meaning. Test an empty result with a scope that you know is empty or with a filter that excludes everything.
216
+
217
+ ## Search and retrieval tools
218
+
219
+ Studio search matches file names at once and adds content matches from the semantic index. A file-name match works before indexing, and a content match needs an indexed Version. One search returns at most 100 results.
220
+
221
+ Agents and Skills search Documents through two built-in tools. Bind them as [AI](../ai/index.md) describes.
222
+
223
+ - Documents Query (`builtin.core.documents.tools.query`) searches Documents only. Each result has a title, a score, an overview, a snippet and up to three ranked passages with their location. An empty `documentCollections` list searches every readable collection, and `documentId` searches inside one file. Its `coverage` output lists readable files that are still indexing, partial, failed or over budget.
224
+ - Advanced Information Retrieval (`builtin.core.information.tools.advanced-retrieval`) searches Knowledge and Documents together, and `sources` defaults to both. It comes with Expand Passage and Read Source Range to read the text around a passage. It also returns a file from a Knowledge file field when the reader can read that record.
225
+
226
+ Every search checks the reader's current permissions. A file that the reader may not read looks the same as a file that does not exist.
227
+
228
+ ## Related pages
229
+
230
+ - [Data resources](index.md)
231
+ - [Knowledge](knowledge.md)
232
+ - [Connectors](connectors.md)
233
+ - [Grants and permissions](../platform/access.md)
234
+ - [AI](../ai/index.md)
235
+ - [Extensions](../develop/extensions.md)
@@ -0,0 +1,62 @@
1
+ # Data resources
2
+
3
+ Choose between Knowledge, Datasets and Documents, and see where each kind keeps its data.
4
+
5
+ Collective Intelligence (CI) has three kinds of Data resource: Knowledge, Datasets and Documents. Each kind has its own unit. A Knowledge schema holds records, a Dataset holds typed rows, and a Document Collection holds files. Choose the kind for each part of your demo first, then open its page. Every Data resource belongs to one Organization; [Platform architecture](../platform/architecture.md) explains how CI keeps the data of Organizations apart.
6
+
7
+ ## Which kind to use
8
+
9
+ - Users create, edit and link items such as a customer, a Lead or a goal: use Knowledge.
10
+ - The customer keeps a warehouse in Microsoft Fabric or Databricks: use an External Dataset, which CI queries live.
11
+ - The customer has a CSV or Excel export and wants to see trends or totals: import it into a Managed Dataset.
12
+ - People read, search and cite files such as contracts, manuals or reports: use Documents. Pick a Connected Collection when the files live in Microsoft 365 or Google Drive, and a Local Collection when your demo adds the files itself.
13
+ - One record needs an attached file, such as a signed PDF on a Lead: use a file field on the Knowledge schema.
14
+
15
+ Knowledge and Datasets often appear in the same demo. Knowledge models the things that people talk about, edit and link, with one record per thing. A Dataset models observations about those things, with one row per measurement, event or transaction, and CI does not create a Knowledge record for each row. The [Datasets](datasets.md) page shows how a Dataset field can bind to a Knowledge schema without copying rows.
16
+
17
+ ## Compare the three kinds
18
+
19
+ | | Knowledge | Datasets | Documents |
20
+ | --------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
21
+ | One item is | A record of a schema | A row of a Dataset | A Document with immutable Versions, in a Document Collection |
22
+ | Who changes it | People in forms, Studio and apps, plus agents and workflows | Ingestion runs for a Managed Dataset, the source system for an External one | The API and workflows for a Local Collection, sync for a Connected one |
23
+ | Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema | Folders inside the collection |
24
+ | Queries | Field filters, text search, semantic search and relation paths | `records`, `aggregate` and `timeSeries` | Filename search and semantic search over content |
25
+ | Access | Knowledge Data Policies, each on one schema | One policy per Dataset with an explicit list of principals | Document Data Policies and rules per collection, folder and file |
26
+ | Tool for agents | Knowledge Query | Query Dataset | Documents Query |
27
+
28
+ Advanced Information Retrieval is the one tool that searches more than one kind in a single call. It covers Knowledge and Documents. Agents query Datasets with Query Dataset only. The [AI](../ai/index.md) page explains tools and Skills.
29
+
30
+ ## Where the data lives
31
+
32
+ Two questions describe each mode. The first asks who holds the authoritative copy: a managed mode keeps it in CI, and an external mode leaves it in the outside system. The second asks whether a connector keeps the data in sync from an outside system; data that a connector syncs is connected.
33
+
34
+ ```mermaid
35
+ flowchart LR
36
+ W[People, apps, agents, workflows] --> K[Knowledge records]
37
+ S[Outside system] -->|connector sync| KC[Connected Knowledge records]
38
+ F[File import] --> M[Managed Dataset]
39
+ S -.->|live query, no copy| E[External Dataset]
40
+ A[API and workflows] --> L[Local Collection]
41
+ S -->|connector sync| C[Connected Collection]
42
+ ```
43
+
44
+ | Kind and mode | Stored in CI | Authoritative copy | Changes come from |
45
+ | --------------------------- | -------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------- |
46
+ | Knowledge records | Yes | CI | People, apps, agents and workflows |
47
+ | Connected Knowledge records | Yes, as a synced copy | The source system; with an outbound binding, CI edits also reach it | Connector pipeline runs, and CI edits when the schema allows them |
48
+ | Managed Dataset | Yes, in PostgreSQL | CI | Ingestion runs that append, upsert or replace rows |
49
+ | External Dataset | No | The source system | The source system only |
50
+ | Local Collection | Yes | CI | The API and workflows |
51
+ | Connected Collection | Yes, as immutable Versions | The source system | Sync runs from Microsoft 365 or Google Drive |
52
+
53
+ An External Dataset has no copy in CI. Every query reaches the provider, CI caches no results, and a query fails when the provider is down. CI writes nothing back to an External Dataset. Documents have no live mode: both collection types store their files in CI, and Studio can open the current file in the provider for a Connected Collection. For connected Knowledge records, the schema's `localMutationPolicy` decides whether people can edit them in CI. `always` allows edits, `connector_backed` allows them only while a connector binding sends edits back to the source, and `never` refuses them. The [Connectors](connectors.md) page explains bindings and sync.
54
+
55
+ ## Related pages
56
+
57
+ - [Knowledge](knowledge.md)
58
+ - [Datasets](datasets.md)
59
+ - [Documents](documents.md)
60
+ - [Connectors](connectors.md)
61
+ - [Platform architecture](../platform/architecture.md)
62
+ - [AI](../ai/index.md)
@@ -1,8 +1,10 @@
1
1
  # Knowledge
2
2
 
3
- Declare schemas, store typed records, query them, and read and watch them from a demo app.
3
+ Declare schemas, store typed records, query them, and read and watch them from an external app.
4
4
 
5
- Knowledge stores the business records of a demo, such as Leads and Opportunities. A schema defines the fields that a record type can have, and a record is one instance that matches that schema. Schemas live in a Knowledge domain, a named container that the Organization owns. In the template, the extension package `acme.app.knowledge` declares one domain with a Lead schema and an Opportunity schema, and the demo app reads and writes Lead and Opportunity records through the SDKs.
5
+ Knowledge holds structured business records, such as Leads and Opportunities, and relation fields link records to each other. A schema defines the fields that a record type can have, and a record is one instance that matches that schema. Schemas live in a Knowledge domain, a named group of related schemas that the Organization owns. In the template, the extension package `acme.app.knowledge` declares the Acme CRM domain with a Lead schema and an Opportunity schema, and the template's web app reads and writes Lead and Opportunity records through the SDKs.
6
+
7
+ A domain holds schemas and never records. It can set a default for record versioning that its schemas use when they set none. Access policies target schemas, so a domain does not grant or limit access. The [Data resources](index.md) page compares Knowledge with Datasets and Documents.
6
8
 
7
9
  ```mermaid
8
10
  flowchart LR
@@ -28,7 +30,7 @@ A schema combines ordinary value fields with three special kinds. Each special k
28
30
 
29
31
  ## Declare a schema in a package
30
32
 
31
- A demo declares its schemas in code, inside its extension package. This is the template's Lead schema, shortened:
33
+ A customer solution declares its schemas in code, inside its extension package. This is the template's Lead schema, shortened:
32
34
 
33
35
  ```ts
34
36
  declareKnowledgeEntitySchema({
@@ -66,9 +68,9 @@ declareKnowledgeEntitySchema({
66
68
  })
67
69
  ```
68
70
 
69
- Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`, Dashboard administrators cannot edit the definition, so the code stays the only source of truth. Choose `readonly: true` when the demo code owns the schema and you redeploy to change it, and `readonly: false` when a customer administrator should keep editing the schema in the Dashboard after the deploy.
71
+ Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`, an Organization admin cannot edit the definition in the Dashboard, so the code stays the only source of truth. Choose `readonly: true` when the package code owns the schema and you redeploy to change it, and `readonly: false` when the customer's Organization admin should keep editing the schema in the Dashboard after the deploy.
70
72
 
71
- The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the demo app (its capabilities, the fields you can filter, sort and search on, and the create and update shapes) and never reaches CI. CI receives only the schema above, so when you add a field, change both files.
73
+ The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the web app (its capabilities, the fields you can filter, sort and search on, and the create and update shapes) and never reaches CI. CI receives only the schema above, so when you add a field, change both files.
72
74
 
73
75
  The CLI puts the schema into an Organization in five steps: `ci package build`, `validate`, `publish`, `promote` and `install --organization <id>`. The [Extensions](../develop/extensions.md) page shows the full commands.
74
76
 
@@ -124,7 +126,7 @@ A file field stores files. Its read value is `{ id, url, filename, mimeType, siz
124
126
 
125
127
  Field filters narrow records with `{ field, operator, value }` and operators such as `eq`, `in`, `exists` and `is_empty`, and text search matches plain text. Semantic search ranks records by embeddings, so it works only on a schema with embeddings enabled, and each result carries its `match.semantic_score`.
126
128
 
127
- The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an admin edit, CI reprocesses the existing records automatically. You do not need a new import or an Organization reset.
129
+ The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an edit in the Dashboard, CI reprocesses the existing records automatically. You do not need a new import or an Organization reset.
128
130
 
129
131
  > **Warning:** `embedEntities` alone gives no semantic search over records. Select at least
130
132
  > one field with `includeInItemEmbedding`, or the schema has no record vector
@@ -156,20 +158,16 @@ PostgreSQL holds the source of truth and runs semantic search with pgvector. The
156
158
 
157
159
  ## Documents and Knowledge
158
160
 
159
- Knowledge holds typed records with fields. Documents holds files in collections and folders, with versions and permissions per path. A Document Collection is either Local, where people upload files, or Connected, where a Documents connector keeps it in sync with folders in Microsoft 365 or Google Drive.
160
-
161
- The two areas share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both areas also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization.
162
-
163
- A finished sync does not mean that the new files are searchable yet, because indexing runs after the sync. A workflow Wait at level `sync` waits for the import only, while level `indexing` also waits for indexing. Use `indexing` before a demo step that searches the new content. The [Connectors](connectors.md) page compares Knowledge sync with Documents sync.
161
+ Knowledge and Documents share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both kinds also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization. The [Documents](documents.md) page covers Document Collections, sync and indexing.
164
162
 
165
163
  ## Read and write from an app
166
164
 
167
- A demo app reaches records in two ways, and the template uses both.
165
+ An external app reaches records in two ways, and the template uses both.
168
166
 
169
167
  | | Browser | Server |
170
168
  | ----------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
171
169
  | API | Entity hooks from `defineEntity({ contract, config })` in `@collective-intelligence/external-apps-client-sdk-react/resources` | `ServerClient.records` from `@collective-intelligence/external-apps-server-sdk/effect` |
172
- | Acts as | The signed-in user | The app credential (`CI_CLIENT_ID`, `CI_CLIENT_SECRET`) |
170
+ | Acts as | The signed-in user | A Server Credential, with the Access Grants of the app's server |
173
171
  | Operations | Whatever the entity contract allows: create, get, search, update, delete, upload | `create`, `get`, `search`, `update`, `delete` |
174
172
  | Typical use | Forms, lists and tables | Server routes and the template's Read Port |
175
173
 
@@ -189,21 +187,26 @@ return { ...fields, id: record.id }
189
187
 
190
188
  `records.search` accepts a filter, text search, sort, pagination and `include`. App filtering goes through `POST /application/v1/resources/:schemaId/search`, and relation field pickers use `POST /application/v1/knowledge/autocomplete`. The SDKs do not wrap semantic search; agents use the retrieval tools for that.
191
189
 
192
- > **Warning:** The server reads with the app credential, so that credential itself needs read
193
- > access to Leads. CI accepts a user's token only when the user holds the
194
- > application gate. See [Access](../platform/access.md).
190
+ > **Warning:** Server calls need a Server Credential (`dsc_`), and CI refuses the enrollment
191
+ > client secret on server calls because that secret only signs users in. Create
192
+ > one with `ci apps credentials upsert <app> --name <name> --env-file <file>`.
193
+ > The Server Credential acts with the Access Grants of the app's server, so
194
+ > those grants need read access to Leads. CI accepts a user's token only when
195
+ > the user holds the application gate. See [Access](../platform/access.md).
195
196
 
196
197
  ## Change events
197
198
 
198
- Every write to a record produces a change event, whether the write comes from the demo app, Studio, an agent or a workflow. Database triggers write a change log entry for each changed record, with one entry per record per transaction. A sync that changes nothing writes no entry.
199
+ Every write to a record produces a change event, whether the write comes from an external app, Studio, an agent or a workflow. Database triggers write a change log entry for each changed record, with one entry per record per transaction. A sync that changes nothing writes no entry.
199
200
 
200
201
  An app subscribes to a topic per schema, for example `{ schema: "acme.app.schemas.lead" }`. `useRealtimeTopic(topic, changes => ...)` receives batches of changes that carry record IDs only, and `useEntityChanges(contract)` gives a function that refreshes the entity reads for those IDs. CI checks read access for each change before it sends it. When CI cannot decide whether a reader may see a delete, it sends that reader a `reset` instead of the ID, and the client must then refetch.
201
202
 
202
- To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or the convert-lead workflow.
203
+ To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or a workflow that writes the Lead.
203
204
 
204
205
  ## Related pages
205
206
 
207
+ - [Data resources](index.md)
206
208
  - [Datasets](datasets.md)
209
+ - [Documents](documents.md)
207
210
  - [Connectors](connectors.md)
208
211
  - [Access](../platform/access.md)
209
212
  - [SDKs](../develop/sdks.md)
@@ -2,7 +2,7 @@
2
2
 
3
3
  What a package declares, how the template wires it, how to deploy it with ci deploy, and the rules its code runs under.
4
4
 
5
- A package is a TypeScript module whose default export is `declarePackage({ manifest, components, hostedContributions })`. The `ci` CLI builds that module into a release, which is one immutable published version of the package. An administrator then installs the release into one Organization, and the installation records which release that Organization runs.
5
+ A package is a TypeScript module whose default export is `declarePackage({ manifest, components, hostedContributions })`. The `ci` CLI builds that module into a release, which is one immutable published version of the package. An Organization admin then installs the release into one Organization, and the installation records which release that Organization runs.
6
6
 
7
7
  The template's package declares the Lead and Opportunity schemas, the Sales group tree, the external application of the web app, Sales Access Profiles and grants, the Max Privileges of the application's tokens, the convert-lead workflow, the lead qualifier Agent with its Skill and tool, and the Sales desk Studio app. [SDKs](sdks.md) explains which `@collective-intelligence/*` package each part imports.
8
8
 
@@ -41,7 +41,7 @@ Executable kinds also take an `implementation` class from `@collective-intellige
41
41
 
42
42
  A component refers to another component of the same package through the handle that its `declare*` call returned. For a built-in component or one from another package, pass the FQN string instead, for example `"builtin.core.knowledge.tools.query"`. TypeScript does not check these strings. Install resolves them and rolls the package back when one does not exist, and the SDK publishes no list of built-in FQNs, so the template pins its FQNs in a `bun test` that catches a typo before deploy.
43
43
 
44
- A package that declares Agents or Skills needs an administrator to install it. A package can depend on another package by FQN only, for example `dependencies: [{ fqn: "com.acme.base" }]`, and each install picks the latest eligible release of the dependency by promotion order.
44
+ A package that declares Agents or Skills needs a person to install it; an install with no signed-in user fails. A package can depend on another package by FQN only, for example `dependencies: [{ fqn: "com.acme.base" }]`, and each install picks the latest eligible release of the dependency by promotion order.
45
45
 
46
46
  ## How the template wires the package
47
47
 
@@ -102,13 +102,13 @@ To add a component to the template:
102
102
 
103
103
  Every content component sets `lifecycle: { deployMode, readonly }`.
104
104
 
105
- | Option | Effect |
106
- | ------------------------- | -------------------------------------------------------------------------------------------- |
107
- | `deployMode: "upsert"` | Each deploy brings the definition back in line with the package. |
108
- | `deployMode: "init_only"` | The first deploy creates the definition. Later deploys leave an administrator's edits alone. |
109
- | `readonly: true` | Nobody can edit the definition by hand. Records of a schema stay editable. |
105
+ | Option | Effect |
106
+ | ------------------------- | ------------------------------------------------------------------------------------------------- |
107
+ | `deployMode: "upsert"` | Each deploy brings the definition back in line with the package. |
108
+ | `deployMode: "init_only"` | The first deploy creates the definition. Later deploys leave an Organization admin's edits alone. |
109
+ | `readonly: true` | Nobody can edit the definition by hand. Records of a schema stay editable. |
110
110
 
111
- The template uses `upsert` with `readonly: true` for almost everything. The convert-lead workflow and the Sales reps grant use `init_only` with `readonly: false`, so an administrator can change them after install and an update keeps that change.
111
+ The template uses `upsert` with `readonly: true` for almost everything. The convert-lead workflow and the Sales reps grant use `init_only` with `readonly: false`, so an Organization admin can change them after install and an update keeps that change.
112
112
 
113
113
  ## Package variables
114
114
 
@@ -131,7 +131,7 @@ const vars = packageVariables(variables)
131
131
  export default declarePackage({ components, manifest, variables })
132
132
  ```
133
133
 
134
- A variable has a type of `string`, `number`, `boolean`, `string[]`, `json` or `secret`. A variable without a default is required at deploy, and a `secret` cannot have a default. When a deploy provisions a secret without a value, the install still succeeds, but runtime use fails until an administrator sets it.
134
+ A variable has a type of `string`, `number`, `boolean`, `string[]`, `json` or `secret`. A variable without a default is required at deploy, and a `secret` cannot have a default. When a deploy provisions a secret without a value, the install still succeeds, but runtime use fails until a Tenant admin sets its value in the Organization variables. An Organization admin can read the variables but cannot change them.
135
135
 
136
136
  A handle cannot go in `fqn`, `version`, `kind`, `lifecycle`, a schema projection, or a reference to another component by FQN such as `schemaFqn` or `templateFqn`. The package FQN cannot be a variable either.
137
137
 
@@ -189,12 +189,13 @@ Every call gets an `AbortSignal` in `context.signal` and can have a deadline in
189
189
 
190
190
  ## After install
191
191
 
192
- Some parts of the demo need a person after the first deploy:
192
+ Some parts of a package need a person after the first deploy:
193
193
 
194
- 1. Rotate the client secret. A confidential application has no secret after install. An administrator with `external_apps:manage` runs `ci apps rotate-secret <application-id>` or clicks Rotate secret on the Dashboard External Apps page. CI shows the secret once, and the web app stores it as `CI_CLIENT_SECRET`.
195
- 2. Add users to the groups. The package creates groups empty and grants profiles to groups. In the template, an administrator adds users to the Sales group or its `managers` subgroup.
196
- 3. Set every secret variable that the deploy left without a value.
197
- 4. Enroll the web app with `ci apps enroll --env-file apps/web/.env.local`, which writes the `CI_*` values the server SDK reads.
194
+ 1. Write the app's sign-in values. A package-declared application refuses `ci apps enroll`, so run `ci apps env <application-id-or-fqn> --env-file apps/web/.env.local`, which writes the `CI_*` values the server SDK reads.
195
+ 2. Rotate the sign-in secret. A confidential application has no secret after install. An Organization admin runs `ci apps rotate-secret <application-id>` or clicks Rotate secret on the Dashboard External Apps page. CI shows the secret once, and the web app stores it as `CI_CLIENT_SECRET`. This secret signs users in only.
196
+ 3. Create a Server Credential for the app's own server calls with `ci apps credentials upsert <application-id-or-fqn> --name <name> --env-file <file>`. CI refuses the sign-in secret on server calls. [External apps](external-apps.md) explains both secrets.
197
+ 4. Add users to the groups. The package creates groups empty and grants profiles to groups. In the template, an Organization admin adds users to the Sales group or its `managers` subgroup.
198
+ 5. Set every secret variable that the deploy left without a value.
198
199
 
199
200
  The Dashboard Extensions page lists packages on Installed and Available tabs. Install and uninstall show a spinner until the real outcome, and leaving the page does not cancel the operation. A configured Knowledge connector instance, even a disabled one, or a saved connection blocks uninstall, and the Dashboard shows the error next to the package.
200
201