@collective-intelligence/extensions-sdk-frontend 0.3.2 → 0.3.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/docs/guide/ai/index.md +36 -17
- package/docs/guide/ai/studio.md +245 -0
- package/docs/guide/data/connectors.md +81 -18
- package/docs/guide/data/datasets.md +6 -17
- package/docs/guide/data/documents.md +235 -0
- package/docs/guide/data/index.md +62 -0
- package/docs/guide/data/knowledge.md +21 -18
- package/docs/guide/develop/extensions.md +68 -15
- package/docs/guide/develop/external-apps.md +48 -14
- package/docs/guide/develop/infrastructure.md +135 -0
- package/docs/guide/develop/sdks.md +2 -2
- package/docs/guide/index.md +12 -8
- package/docs/guide/platform/access.md +82 -17
- package/docs/guide/platform/architecture.md +27 -18
- package/docs/guide/platform/entrypoints.md +128 -47
- package/docs/guide/scenarios/index.md +69 -46
- package/docs/guide/template/index.md +22 -9
- package/docs/guide/template/state-machines.md +1 -1
- package/package.json +3 -3
|
@@ -4,17 +4,9 @@ Query typed tables of measurements, events and warehouse data, and control who c
|
|
|
4
4
|
|
|
5
5
|
A Dataset is a typed collection of rows that belongs to an Organization. Each Dataset has a stable identity, a versioned definition, a policy that says who may query it, and provenance on every result. Datasets hold data that has many rows and few edits, such as meter readings, monthly sales figures or a table in a customer's data warehouse. Agents, workflows and Studio query that data in bounded, typed requests.
|
|
6
6
|
|
|
7
|
-
##
|
|
7
|
+
## Datasets and Knowledge
|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
| | Knowledge schema | Dataset |
|
|
12
|
-
| -------------------- | ------------------------------------------------------- | -------------------------------------------------------------- |
|
|
13
|
-
| One row or record is | A real thing: a customer, a Lead, a goal | An observation: a reading, a transaction, an event |
|
|
14
|
-
| People edit rows | Yes, in forms, Studio and the API | No; rows come from file imports or from the source system |
|
|
15
|
-
| Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema |
|
|
16
|
-
| Queries | Field filters, text and semantic search, relation paths | `records`, `aggregate` and `timeSeries` |
|
|
17
|
-
| Access | Knowledge data policies across schemas and domains | One policy per Dataset with an explicit list of principals |
|
|
9
|
+
A Dataset holds observations about the things that Knowledge records describe, with one row per measurement, event or transaction. The [Data resources](index.md) page compares Datasets with Knowledge and Documents and lists which kind fits which demo situation.
|
|
18
10
|
|
|
19
11
|
A strategy Metric shows the split. The Metric itself is a Knowledge record with an owner, a target, a unit, a status and links to goals, and its history of measured values is a Dataset. A Dataset field such as `customerId` can bind to the Customer schema in Knowledge. That binding links the two without copying rows into Knowledge, and deleting one side does not delete the other.
|
|
20
12
|
|
|
@@ -95,7 +87,7 @@ Every result carries its provenance: the Dataset ID, the definition revision, th
|
|
|
95
87
|
|
|
96
88
|
Each Dataset has its own policy, and the policy lists its principals by ID. It holds `principals` (each a `user` or `service` with an `id`), `allowedOperations` (`describe`, `records`, `aggregate`, `time_series`), `allowedFields`, `sensitiveFields`, an optional `rowFilter`, and the limits `maxRows`, `maxBytes` and `maxTimeoutMs`. A user also needs the Organization permission to query Datasets, and a Dataset grant does not give Organization membership or capabilities.
|
|
97
89
|
|
|
98
|
-
Knowledge access works differently: Knowledge
|
|
90
|
+
Knowledge access works differently: each Knowledge Data Policy targets one schema, and no policy targets a whole domain. The [Access](../platform/access.md) page explains both models.
|
|
99
91
|
|
|
100
92
|
> **Warning:** A package-declared Dataset without a `dataPolicy` grants access to nobody,
|
|
101
93
|
> including the person who deployed it. Always declare the principals that the
|
|
@@ -103,7 +95,7 @@ Knowledge access works differently: Knowledge data policies apply across schemas
|
|
|
103
95
|
|
|
104
96
|
## Declare a Dataset in a package
|
|
105
97
|
|
|
106
|
-
`declareDatasetConnectorInstance` declares a Dataset in
|
|
98
|
+
`declareDatasetConnectorInstance` declares a Dataset in a package. An External declaration names the provider, a connection slot, the source location and the policy:
|
|
107
99
|
|
|
108
100
|
```ts
|
|
109
101
|
const orders = declareDatasetConnectorInstance({
|
|
@@ -139,16 +131,13 @@ Agents use one tool, Query Dataset (`builtin.core.datasets.tools.query`). It lis
|
|
|
139
131
|
|
|
140
132
|
Workflows have one node for each operation: describe, query records, query aggregate and query time series. Studio shows Datasets in their own area, apart from the Knowledge views.
|
|
141
133
|
|
|
142
|
-
##
|
|
143
|
-
|
|
144
|
-
- The customer has a warehouse in Fabric or Databricks: use an External Dataset and query it live.
|
|
145
|
-
- The customer has a CSV or Excel export and wants to see trends: import it into a Managed Dataset.
|
|
146
|
-
- Users must edit records and link them to each other: use Knowledge.
|
|
134
|
+
## Limits
|
|
147
135
|
|
|
148
136
|
Datasets are sized for about 100,000 to 1,000,000 data points per Organization and do not fit a high-volume telemetry stream. Joins across Datasets, writes back to External Datasets, streaming ingestion and copying External data into a Managed Dataset are not available. A route for external applications to query Datasets directly is planned; today agents, workflows and Studio run the queries.
|
|
149
137
|
|
|
150
138
|
## Related pages
|
|
151
139
|
|
|
140
|
+
- [Data resources](index.md)
|
|
152
141
|
- [Knowledge](knowledge.md)
|
|
153
142
|
- [Connectors](connectors.md)
|
|
154
143
|
- [Access](../platform/access.md)
|
|
@@ -0,0 +1,235 @@
|
|
|
1
|
+
# Documents
|
|
2
|
+
|
|
3
|
+
Keep files in Document Collections, sync them from Microsoft 365 or Google Drive, control who reads them, and know when their content becomes searchable.
|
|
4
|
+
|
|
5
|
+
Documents is the Data resource for files. It keeps files in Document Collections, with folders, Documents and Versions, and with permissions on each collection and folder. Knowledge holds typed records with fields, and Documents holds files that people browse in Studio and that agents and workflows search. A Knowledge file field also stores its file through Documents, in an internal collection that Documents browse and search never show; a reader reaches that file through the Knowledge record. See [Knowledge](knowledge.md) for records and file fields.
|
|
6
|
+
|
|
7
|
+
## Collections, Documents and Versions
|
|
8
|
+
|
|
9
|
+
| Term | Meaning |
|
|
10
|
+
| -------------------- | ------------------------------------------------------------------------------------------------------ |
|
|
11
|
+
| Document Collection | The container and permission boundary. Each collection has one source: Local or Connected. |
|
|
12
|
+
| Local Collection | A collection with no outside system. CI holds the authoritative copy of its files. |
|
|
13
|
+
| Connected Collection | A collection that one Documents connector keeps in sync with folders in Microsoft 365 or Google Drive. |
|
|
14
|
+
| Document | The identity of one file. It sits in exactly one folder. |
|
|
15
|
+
| Document Version | An immutable revision of a Document. Changed bytes create a new Version; a rename or a move does not. |
|
|
16
|
+
|
|
17
|
+
Files enter a Local Collection through the API and through workflows. A Connected Collection follows its provider: the outside drive stays authoritative, and each sync imports its changes as new Documents and Versions. Two actions in Studio show the difference. Download returns the bytes of the stored Version that the reader selected, and open original opens the live file at the provider, which can be newer than the last sync.
|
|
18
|
+
|
|
19
|
+
## Where people work with Documents
|
|
20
|
+
|
|
21
|
+
Organization admins and Tenant admins manage collections in the Dashboard under **Documents**, then **Collections**. **Add new** opens Setup. A Local Collection takes two steps, Source and Details. A Connected Collection adds a Connection step, where you pick a Documents connector and a connection, and a Sync step with **Automatic sync**, which is on by default. Details holds the name, the remote folders and the two indexing options, **Index content** and **Index overview**.
|
|
22
|
+
|
|
23
|
+
The detail page of a collection has a **Files** tab with a folder explorer and file details, including Versions, and a **Permission tester** tab. It also shows the indexing and sync settings, the sync status and a **Sync now** action. An Organization admin can inspect folders, files and Version identifiers there without permission to read their content.
|
|
24
|
+
|
|
25
|
+
End users read Documents in Studio. The Documents page lists the collections that the user can read, shows their folders and files, searches by file name and content, downloads the selected Version and opens the original at the provider. It has no upload or edit controls. Plain text files up to 1 MiB show a text preview; other files show a fallback and stay downloadable.
|
|
26
|
+
|
|
27
|
+
> **Warning:** Both indexing options are off by default, in Dashboard Setup and in a package
|
|
28
|
+
> declaration. With both off, CI uses no embeddings and no AI processing for the
|
|
29
|
+
> collection's files, so semantic search and the retrieval tools find nothing in
|
|
30
|
+
> their content. Turn on **Index content** (or `indexContent: true`) for any
|
|
31
|
+
> demo step that searches inside files.
|
|
32
|
+
|
|
33
|
+
## Permissions
|
|
34
|
+
|
|
35
|
+
Documents has three operations. `read` is called View in the Dashboard and opens the current Version of each file. `history` is called View history and opens older Versions and the Version list. `write` creates, updates and deletes. A Document Data Policy opens one of these operations on one collection or one folder for the holders of a capability. Unlike a Knowledge Data Policy, it has no `where` clause and no Grant Axes.
|
|
36
|
+
|
|
37
|
+
The core default policies open View to holders of the core Documents data capability at `read` and `write` at `manage`. No default policy opens View history, so members see older Versions only where a package Document Data Policy opens `history`. Holders of the Documents admin capability at `manage` get all three operations.
|
|
38
|
+
|
|
39
|
+
A package declares its own capability and the policies that use it:
|
|
40
|
+
|
|
41
|
+
```ts
|
|
42
|
+
import {
|
|
43
|
+
access,
|
|
44
|
+
declareAccessCapabilities,
|
|
45
|
+
declareAccessDocumentDataPolicy
|
|
46
|
+
} from "@collective-intelligence/extensions-sdk"
|
|
47
|
+
|
|
48
|
+
export const hr = declareAccessCapabilities({
|
|
49
|
+
manifest: { fqn: "acme.hr", version: "1.0.0" },
|
|
50
|
+
definition: {
|
|
51
|
+
nodes: { policies: { kind: "readManage" } },
|
|
52
|
+
lifecycle: { deployMode: "upsert", readonly: true }
|
|
53
|
+
}
|
|
54
|
+
})
|
|
55
|
+
|
|
56
|
+
export const hrPoliciesRead = declareAccessDocumentDataPolicy({
|
|
57
|
+
manifest: { fqn: "acme.hr.policies.hr-policies-read", version: "1.0.0" },
|
|
58
|
+
definition: {
|
|
59
|
+
collection: hrPolicies, // the declaration in the next section
|
|
60
|
+
operation: "read",
|
|
61
|
+
capability: access.requires(hr.policies, "read")
|
|
62
|
+
}
|
|
63
|
+
})
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
A policy on a folder uses `folder: { collection, target: { drive, folders } }` in place of `collection`. A second policy with `operation: "history"` lets the same people open older Versions. Who holds the capability comes from Access Profiles and Access Grants, which [Grants and permissions](../platform/access.md) explains.
|
|
67
|
+
|
|
68
|
+
To check a result, open the collection's **Permission tester**, choose **View collection as** and pick a person. The tester shows View, View history and Update for each file. It tests the Documents that already exist, so a Connected Collection that has not imported any file yet shows an empty result. Check the sync status before you read an empty result as a denial.
|
|
69
|
+
|
|
70
|
+
## Declare a Connected Collection in a package
|
|
71
|
+
|
|
72
|
+
`declareDocumentCollection` declares one Connected Collection whose identity is the declaration FQN. No option declares a Local Collection.
|
|
73
|
+
|
|
74
|
+
```ts
|
|
75
|
+
import {
|
|
76
|
+
declareDocumentCollection,
|
|
77
|
+
packageVariables
|
|
78
|
+
} from "@collective-intelligence/extensions-sdk"
|
|
79
|
+
|
|
80
|
+
const vars = packageVariables({ hrDrive: { type: "string" } })
|
|
81
|
+
|
|
82
|
+
export const hrPolicies = declareDocumentCollection({
|
|
83
|
+
manifest: { fqn: "acme.documents.hr-policies", version: "1.0.0" },
|
|
84
|
+
presentation: {
|
|
85
|
+
displayName: "HR policies",
|
|
86
|
+
description: "Company HR policies"
|
|
87
|
+
},
|
|
88
|
+
definition: {
|
|
89
|
+
connectorFqn:
|
|
90
|
+
"builtin.integrations.microsoft-365.connectors.documents-drive",
|
|
91
|
+
connection: { setupKey: "microsoft365" },
|
|
92
|
+
title: "HR Policies",
|
|
93
|
+
indexContent: true,
|
|
94
|
+
remoteRoots: [{ drive: vars.hrDrive, folders: ["Policies"] }]
|
|
95
|
+
}
|
|
96
|
+
})
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
`connection` names a connection that the package declares (`{ connectionInstanceFqn }`) or an existing one (`{ setupKey }`, filled with a `{ connectionId }` in the Inputs File). `indexContent` and `indexOverview` default to `false`, and `syncEnabled` defaults to `true`. A remote root is a path `{ drive, folders }` or a `{ remoteId }`. On Microsoft 365, `drive` is the drive title `<siteName> / <libraryName>` that the Dashboard source browser shows, and an empty `folders` names the whole drive. Each value can be a Package Variable, so one release serves environments with different folders.
|
|
100
|
+
|
|
101
|
+
You can leave out `connectorFqn`. The deploy then takes the one Documents connector among the package dependencies whose connection template matches, and it refuses the deploy when none or several match. One `ci deploy` also installs a missing built-in dependency such as `builtin.integrations.microsoft-365`.
|
|
102
|
+
|
|
103
|
+
Package declaration is verified for Microsoft 365 only. Create a Google Drive collection through Dashboard Setup, which offers every installed Documents connector.
|
|
104
|
+
|
|
105
|
+
The first deploy resolves each folder at the provider before it changes anything, so a missing folder refuses the deploy. A later deploy of the same declaration writes nothing and never starts a sync. Removing the declaration, or uninstalling the package, keeps the collection and its files.
|
|
106
|
+
|
|
107
|
+
> **Note:** A Microsoft 365 app that has only the `Files.SelectedOperations.Selected`
|
|
108
|
+
> permission cannot list drives, so a path of names cannot resolve. Declare each
|
|
109
|
+
> root of such a tenant as `{remoteId}`.
|
|
110
|
+
|
|
111
|
+
## Sync
|
|
112
|
+
|
|
113
|
+
| Connector | FQN | Reads |
|
|
114
|
+
| ----------------------- | ------------------------------------------------------------------ | ------------------------------------------------------ |
|
|
115
|
+
| Microsoft 365 Documents | `builtin.integrations.microsoft-365.connectors.documents-drive` | OneDrive and SharePoint drives through Microsoft Graph |
|
|
116
|
+
| Google Drive Documents | `builtin.integrations.google-workspace.connectors.documents-drive` | My Drive and shared drives |
|
|
117
|
+
|
|
118
|
+
Each Connected Collection has its own Documents connector instance, and two Connected Collections on the same provider account cannot overlap their root folders. Google Drive exports native files as Office files: Docs as DOCX, Sheets as XLSX, Slides as PPTX and Drawings as PNG. It skips native types that have no export, and the Microsoft 365 connector skips OneNote notebooks.
|
|
119
|
+
|
|
120
|
+
With **Automatic sync** on, CI starts a sync five minutes after the last attempt ended, also after a failed one. **Sync now** starts one at once. With Automatic sync off, the collection syncs only on Sync now. A sync saves its provider position only after it applies the changes, so an interrupted sync continues without losing any. A Local Collection has no sync status.
|
|
121
|
+
|
|
122
|
+
| Status | Meaning |
|
|
123
|
+
| ----------- | --------------------------------------------------------------------------------------------------------------- |
|
|
124
|
+
| Not synced | No sync has completed yet. |
|
|
125
|
+
| Syncing | A sync is requested or running. |
|
|
126
|
+
| Up to date | A sync completed in the last ten minutes, or at any time when Automatic sync is off. |
|
|
127
|
+
| Sync issues | The last sync failed, a Quarantined File is open, or Automatic sync is on and no sync completed in ten minutes. |
|
|
128
|
+
|
|
129
|
+
Excluded Folder Names keep archive folders out. The package declaration sets them with `excludedFolderNames`, for example `["archives"]`, and no Dashboard field changes them. A file below a folder with one of these names, at any depth under a root, is never downloaded, read or embedded. The match ignores case, accents, outer spaces and one pair of `[]` or `()` around the name, and the whole name must match. A root is never excluded by its own name. A Document that was imported before its folder took an excluded name leaves search, browse and the AI tools, and it comes back with the same identity when the folder leaves the excluded name. A changed list makes the next sync walk every folder again.
|
|
130
|
+
|
|
131
|
+
A file whose download fails three times on the same provider revision becomes a Quarantined File. The sync stops fetching it and shows it as an error in **Sync issues**, which sets the status to Sync issues. **Sync now** tries every Quarantined File once more, and a new revision of the file at the provider starts it again. Other failed downloads retry on their own and show as warnings that do not change the status.
|
|
132
|
+
|
|
133
|
+
## Supported file formats
|
|
134
|
+
|
|
135
|
+
| Group | Formats | How CI reads them | Needs |
|
|
136
|
+
| ------------- | ---------------------------------------- | ---------------------------------------------------------- | ---------------------- |
|
|
137
|
+
| Text | TXT, MD, CSV, JSON | Text reader | Nothing |
|
|
138
|
+
| PDF | PDF | Text and structure, page OCR for pages with almost no text | A vision model for OCR |
|
|
139
|
+
| Office | DOCX, XLSX, PPTX | Office readers | Nothing |
|
|
140
|
+
| Legacy Office | DOC, XLS, PPT, RTF | Conversion to DOCX, XLSX or PPTX, then the Office readers | The Compute service |
|
|
141
|
+
| Images | JPEG, PNG, WebP | Visible text and a description from a vision model | A vision model |
|
|
142
|
+
| Audio | AAC, AIFF, FLAC, M4A, MP3, OGG, WAV, WMA | Speech transcription | The Compute service |
|
|
143
|
+
| Video | AVI, MKV, MOV, MP4, WebM, WMV | Speech transcription of the audio track | The Compute service |
|
|
144
|
+
|
|
145
|
+
CI reads no other format. HTML, TIF, TIFF, BMP and GIF files, for example, sync and download like any file, but CI does not index their content. The Compute service is a separate CI service that runs transcription and Office conversion.
|
|
146
|
+
|
|
147
|
+
Each entry path has its own list of accepted formats. Documents accepts all formats in the table. Knowledge file fields accept DOC, DOCX, JPEG, MD, PDF, PNG, PPT, PPTX, RTF, TXT, WebP, XLS and XLSX. A CI deployment can narrow either list. A file outside the list still syncs, lists and downloads, and its indexing status is `unsupported`. Without the Compute service, a legacy Office file is `unsupported` with the reason that conversion is not available.
|
|
148
|
+
|
|
149
|
+
## What CI reads from each format
|
|
150
|
+
|
|
151
|
+
For a PDF, CI reads the native text and rebuilds its structure. In a PDF of three or more pages it removes headers and footers that repeat across pages. It finds headings up to six levels from their numbering and font size, and the document title on page one. It rebuilds tables from ruled grids or from aligned columns, also when a table continues on the next page, and it reads multi-column pages column by column. A page with fewer than 50 letters and digits goes to page OCR, and a PDF with almost no text is read by OCR as a whole.
|
|
152
|
+
|
|
153
|
+
For Office files, CI keeps what the author sees:
|
|
154
|
+
|
|
155
|
+
- DOCX headings come from Word heading styles in any language, and from font size only when the file uses no heading style. Text boxes, footnotes and endnotes stay next to the text that holds them. Deleted tracked changes and comments stay out, and headers and footers are kept as metadata.
|
|
156
|
+
- PPTX gives the shown slides with their groups, tables and speaker notes. Hidden slides give nothing, and the title of the first slide is the document title.
|
|
157
|
+
- XLSX gives one table per visible sheet. Hidden sheets, rows and columns give nothing, dates use ISO 8601, and a text cell such as `00123` keeps its leading zeros.
|
|
158
|
+
|
|
159
|
+
A merged cell repeats its value down the rows it covers, so each table row has the full column count.
|
|
160
|
+
|
|
161
|
+
DOC, XLS, PPT and RTF files go through one conversion with LibreOffice in the Compute service, and the Office readers then read the converted copy. The copy stays internal: search results, citations and downloads always point to the original file. The conversion never runs macros and never fetches external content. A corrupt or password-protected file, a conversion over 60 seconds or an empty result fails that file.
|
|
162
|
+
|
|
163
|
+
Images in DOCX, PPTX and PDF files go to the vision model too, as do stand-alone JPEG, PNG and WebP files. Each kept image gives its visible text and a short description. CI drops images whose short side is under 128 pixels, keeps at most 50 images per file and reads each distinct image once. Each described image uses one page of the Organization's daily OCR budget. A file with an image that failed or found no budget is `partial`.
|
|
164
|
+
|
|
165
|
+
Audio and video go to the Compute service for transcription, which detects the language. A video gives the speech of its audio track; CI does not index video frames. Opening, downloading or searching a file never starts a transcription.
|
|
166
|
+
|
|
167
|
+
The file indexing strategy decides how much of this work runs. CI takes the Organization's setting, else the tenant default, else `text_with_image_descriptions`.
|
|
168
|
+
|
|
169
|
+
- `text_only` reads text only and drops images.
|
|
170
|
+
- `text_with_ocr` adds page OCR for sparse PDF pages and reads stand-alone image files.
|
|
171
|
+
- `text_with_image_descriptions`, the default, also describes every kept image.
|
|
172
|
+
- `hybrid_multimodal` also stores an image vector for each kept image, and it needs an embedding model that accepts images.
|
|
173
|
+
|
|
174
|
+
Every strategy except `text_only` needs an enabled vision model.
|
|
175
|
+
|
|
176
|
+
| Limit | Default |
|
|
177
|
+
| ---------------------------------- | ---------------------------------------------------- |
|
|
178
|
+
| PDF page OCR and image reading | PDFs up to 32 MiB; text extraction has no size limit |
|
|
179
|
+
| One image sent to the vision model | 8 MiB |
|
|
180
|
+
| Audio and video transcription | 100 MiB and 30 minutes per file |
|
|
181
|
+
| Legacy Office conversion | 60 seconds per file |
|
|
182
|
+
| Studio text preview | 1 MiB |
|
|
183
|
+
|
|
184
|
+
Above 32 MiB, a PDF keeps the native text of its sparse pages and becomes `partial`, and a PDF with almost no text fails.
|
|
185
|
+
|
|
186
|
+
## How indexing works
|
|
187
|
+
|
|
188
|
+
```mermaid
|
|
189
|
+
flowchart LR
|
|
190
|
+
S[Sync or API write] --> V[New Document Version]
|
|
191
|
+
V --> E[Extraction: text, structure, OCR, images, speech]
|
|
192
|
+
E --> F[Fragments]
|
|
193
|
+
F --> M[Embeddings]
|
|
194
|
+
M --> Q[Searchable]
|
|
195
|
+
E -. optional .-> O[Overview]
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
Knowledge file fields and Documents share one processing pipeline. Extraction turns a file into text and structure. Segmentation cuts that text into Fragments, the units that search and the retrieval tools return. CI then embeds each Fragment with the document title in front, so a passage from page 40 still says which file it comes from. With **Index overview** on, CI also writes a short summary of the file.
|
|
199
|
+
|
|
200
|
+
The pipeline keeps the structure that extraction found. CI splits a paragraph only when it is too long for one Fragment, and then between sentences. Fragments do not overlap. A table becomes groups of whole rows, and every group repeats the table header, so each Fragment is a readable table. A table never shares a Fragment with prose, and a heading joins the content that follows it.
|
|
201
|
+
|
|
202
|
+
CI stores one result for the same bytes in the same Organization with the same processing settings, and every Document and Knowledge file with those bytes uses that result. A duplicate file costs no second extraction or embedding. When the processing settings change, for example the file indexing strategy, CI processes the files again in the background, and the old result serves search until the new one is ready.
|
|
203
|
+
|
|
204
|
+
A file is searchable when its Fragments and their vectors are saved. CI does not wait for the overview. Only the current Version of each Document is in semantic search. While a new Version is processed, search keeps the previous Version, and it switches when the new one is ready.
|
|
205
|
+
|
|
206
|
+
## Wait for indexing in a workflow
|
|
207
|
+
|
|
208
|
+
> **Warning:** A finished sync means that the files are imported. Indexing runs after the
|
|
209
|
+
> sync, so the new content is not searchable yet.
|
|
210
|
+
|
|
211
|
+
The Documents workflow nodes come in two branches, Local and Connected. Both have Create, List, Get, Update, Delete and Restore Collection, and Search Documents, Search Within Document, Expand Passage and Read Document Range. The Connected branch adds Start Collection Sync, Get Collection Sync Status, Set Collection Sync Settings and Wait for Collection Sync.
|
|
212
|
+
|
|
213
|
+
Wait for Collection Sync takes the import reference that Create Collection or Start Collection Sync returns, and a `wait` level. Level `sync`, the default, waits for that import. Level `indexing` also waits for the enabled indexing of the files that the import brought in. Use `indexing` before a demo step that searches the new content. Level `indexing` needs at least one indexing option on the collection, and the wait fails when both are off. An optional `expiresAt` ends the wait through an `expired` outcome. The wait never starts indexing and never changes the collection's settings.
|
|
214
|
+
|
|
215
|
+
A search node returns what is searchable when it runs. In a test, an empty result from a random query does not prove that a file is missing, because semantic search ranks by meaning. Test an empty result with a scope that you know is empty or with a filter that excludes everything.
|
|
216
|
+
|
|
217
|
+
## Search and retrieval tools
|
|
218
|
+
|
|
219
|
+
Studio search matches file names at once and adds content matches from the semantic index. A file-name match works before indexing, and a content match needs an indexed Version. One search returns at most 100 results.
|
|
220
|
+
|
|
221
|
+
Agents and Skills search Documents through two built-in tools. Bind them as [AI](../ai/index.md) describes.
|
|
222
|
+
|
|
223
|
+
- Documents Query (`builtin.core.documents.tools.query`) searches Documents only. Each result has a title, a score, an overview, a snippet and up to three ranked passages with their location. An empty `documentCollections` list searches every readable collection, and `documentId` searches inside one file. Its `coverage` output lists readable files that are still indexing, partial, failed or over budget.
|
|
224
|
+
- Advanced Information Retrieval (`builtin.core.information.tools.advanced-retrieval`) searches Knowledge and Documents together, and `sources` defaults to both. It comes with Expand Passage and Read Source Range to read the text around a passage. It also returns a file from a Knowledge file field when the reader can read that record.
|
|
225
|
+
|
|
226
|
+
Every search checks the reader's current permissions. A file that the reader may not read looks the same as a file that does not exist.
|
|
227
|
+
|
|
228
|
+
## Related pages
|
|
229
|
+
|
|
230
|
+
- [Data resources](index.md)
|
|
231
|
+
- [Knowledge](knowledge.md)
|
|
232
|
+
- [Connectors](connectors.md)
|
|
233
|
+
- [Grants and permissions](../platform/access.md)
|
|
234
|
+
- [AI](../ai/index.md)
|
|
235
|
+
- [Extensions](../develop/extensions.md)
|
|
@@ -0,0 +1,62 @@
|
|
|
1
|
+
# Data resources
|
|
2
|
+
|
|
3
|
+
Choose between Knowledge, Datasets and Documents, and see where each kind keeps its data.
|
|
4
|
+
|
|
5
|
+
Collective Intelligence (CI) has three kinds of Data resource: Knowledge, Datasets and Documents. Each kind has its own unit. A Knowledge schema holds records, a Dataset holds typed rows, and a Document Collection holds files. Choose the kind for each part of your demo first, then open its page. Every Data resource belongs to one Organization; [Platform architecture](../platform/architecture.md) explains how CI keeps the data of Organizations apart.
|
|
6
|
+
|
|
7
|
+
## Which kind to use
|
|
8
|
+
|
|
9
|
+
- Users create, edit and link items such as a customer, a Lead or a goal: use Knowledge.
|
|
10
|
+
- The customer keeps a warehouse in Microsoft Fabric or Databricks: use an External Dataset, which CI queries live.
|
|
11
|
+
- The customer has a CSV or Excel export and wants to see trends or totals: import it into a Managed Dataset.
|
|
12
|
+
- People read, search and cite files such as contracts, manuals or reports: use Documents. Pick a Connected Collection when the files live in Microsoft 365 or Google Drive, and a Local Collection when your demo adds the files itself.
|
|
13
|
+
- One record needs an attached file, such as a signed PDF on a Lead: use a file field on the Knowledge schema.
|
|
14
|
+
|
|
15
|
+
Knowledge and Datasets often appear in the same demo. Knowledge models the things that people talk about, edit and link, with one record per thing. A Dataset models observations about those things, with one row per measurement, event or transaction, and CI does not create a Knowledge record for each row. The [Datasets](datasets.md) page shows how a Dataset field can bind to a Knowledge schema without copying rows.
|
|
16
|
+
|
|
17
|
+
## Compare the three kinds
|
|
18
|
+
|
|
19
|
+
| | Knowledge | Datasets | Documents |
|
|
20
|
+
| --------------- | -------------------------------------------------------------- | --------------------------------------------------------------------------- | ---------------------------------------------------------------------- |
|
|
21
|
+
| One item is | A record of a schema | A row of a Dataset | A Document with immutable Versions, in a Document Collection |
|
|
22
|
+
| Who changes it | People in forms, Studio and apps, plus agents and workflows | Ingestion runs for a Managed Dataset, the source system for an External one | The API and workflows for a Local Collection, sync for a Connected one |
|
|
23
|
+
| Links | Relation fields between records | An optional binding from a Dataset field to a Knowledge schema | Folders inside the collection |
|
|
24
|
+
| Queries | Field filters, text search, semantic search and relation paths | `records`, `aggregate` and `timeSeries` | Filename search and semantic search over content |
|
|
25
|
+
| Access | Knowledge Data Policies, each on one schema | One policy per Dataset with an explicit list of principals | Document Data Policies and rules per collection, folder and file |
|
|
26
|
+
| Tool for agents | Knowledge Query | Query Dataset | Documents Query |
|
|
27
|
+
|
|
28
|
+
Advanced Information Retrieval is the one tool that searches more than one kind in a single call. It covers Knowledge and Documents. Agents query Datasets with Query Dataset only. The [AI](../ai/index.md) page explains tools and Skills.
|
|
29
|
+
|
|
30
|
+
## Where the data lives
|
|
31
|
+
|
|
32
|
+
Two questions describe each mode. The first asks who holds the authoritative copy: a managed mode keeps it in CI, and an external mode leaves it in the outside system. The second asks whether a connector keeps the data in sync from an outside system; data that a connector syncs is connected.
|
|
33
|
+
|
|
34
|
+
```mermaid
|
|
35
|
+
flowchart LR
|
|
36
|
+
W[People, apps, agents, workflows] --> K[Knowledge records]
|
|
37
|
+
S[Outside system] -->|connector sync| KC[Connected Knowledge records]
|
|
38
|
+
F[File import] --> M[Managed Dataset]
|
|
39
|
+
S -.->|live query, no copy| E[External Dataset]
|
|
40
|
+
A[API and workflows] --> L[Local Collection]
|
|
41
|
+
S -->|connector sync| C[Connected Collection]
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
| Kind and mode | Stored in CI | Authoritative copy | Changes come from |
|
|
45
|
+
| --------------------------- | -------------------------- | ------------------------------------------------------------------- | ----------------------------------------------------------------- |
|
|
46
|
+
| Knowledge records | Yes | CI | People, apps, agents and workflows |
|
|
47
|
+
| Connected Knowledge records | Yes, as a synced copy | The source system; with an outbound binding, CI edits also reach it | Connector pipeline runs, and CI edits when the schema allows them |
|
|
48
|
+
| Managed Dataset | Yes, in PostgreSQL | CI | Ingestion runs that append, upsert or replace rows |
|
|
49
|
+
| External Dataset | No | The source system | The source system only |
|
|
50
|
+
| Local Collection | Yes | CI | The API and workflows |
|
|
51
|
+
| Connected Collection | Yes, as immutable Versions | The source system | Sync runs from Microsoft 365 or Google Drive |
|
|
52
|
+
|
|
53
|
+
An External Dataset has no copy in CI. Every query reaches the provider, CI caches no results, and a query fails when the provider is down. CI writes nothing back to an External Dataset. Documents have no live mode: both collection types store their files in CI, and Studio can open the current file in the provider for a Connected Collection. For connected Knowledge records, the schema's `localMutationPolicy` decides whether people can edit them in CI. `always` allows edits, `connector_backed` allows them only while a connector binding sends edits back to the source, and `never` refuses them. The [Connectors](connectors.md) page explains bindings and sync.
|
|
54
|
+
|
|
55
|
+
## Related pages
|
|
56
|
+
|
|
57
|
+
- [Knowledge](knowledge.md)
|
|
58
|
+
- [Datasets](datasets.md)
|
|
59
|
+
- [Documents](documents.md)
|
|
60
|
+
- [Connectors](connectors.md)
|
|
61
|
+
- [Platform architecture](../platform/architecture.md)
|
|
62
|
+
- [AI](../ai/index.md)
|
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
# Knowledge
|
|
2
2
|
|
|
3
|
-
Declare schemas, store typed records, query them, and read and watch them from
|
|
3
|
+
Declare schemas, store typed records, query them, and read and watch them from an external app.
|
|
4
4
|
|
|
5
|
-
Knowledge
|
|
5
|
+
Knowledge holds structured business records, such as Leads and Opportunities, and relation fields link records to each other. A schema defines the fields that a record type can have, and a record is one instance that matches that schema. Schemas live in a Knowledge domain, a named group of related schemas that the Organization owns. In the template, the extension package `acme.app.knowledge` declares the Acme CRM domain with a Lead schema and an Opportunity schema, and the template's web app reads and writes Lead and Opportunity records through the SDKs.
|
|
6
|
+
|
|
7
|
+
A domain holds schemas and never records. It can set a default for record versioning that its schemas use when they set none. Access policies target schemas, so a domain does not grant or limit access. The [Data resources](index.md) page compares Knowledge with Datasets and Documents.
|
|
6
8
|
|
|
7
9
|
```mermaid
|
|
8
10
|
flowchart LR
|
|
@@ -28,7 +30,7 @@ A schema combines ordinary value fields with three special kinds. Each special k
|
|
|
28
30
|
|
|
29
31
|
## Declare a schema in a package
|
|
30
32
|
|
|
31
|
-
A
|
|
33
|
+
A customer solution declares its schemas in code, inside its extension package. This is the template's Lead schema, shortened:
|
|
32
34
|
|
|
33
35
|
```ts
|
|
34
36
|
declareKnowledgeEntitySchema({
|
|
@@ -66,9 +68,9 @@ declareKnowledgeEntitySchema({
|
|
|
66
68
|
})
|
|
67
69
|
```
|
|
68
70
|
|
|
69
|
-
Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`,
|
|
71
|
+
Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`, an Organization admin cannot edit the definition in the Dashboard, so the code stays the only source of truth. Choose `readonly: true` when the package code owns the schema and you redeploy to change it, and `readonly: false` when the customer's Organization admin should keep editing the schema in the Dashboard after the deploy.
|
|
70
72
|
|
|
71
|
-
The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the
|
|
73
|
+
The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the web app (its capabilities, the fields you can filter, sort and search on, and the create and update shapes) and never reaches CI. CI receives only the schema above, so when you add a field, change both files.
|
|
72
74
|
|
|
73
75
|
The CLI puts the schema into an Organization in five steps: `ci package build`, `validate`, `publish`, `promote` and `install --organization <id>`. The [Extensions](../develop/extensions.md) page shows the full commands.
|
|
74
76
|
|
|
@@ -124,7 +126,7 @@ A file field stores files. Its read value is `{ id, url, filename, mimeType, siz
|
|
|
124
126
|
|
|
125
127
|
Field filters narrow records with `{ field, operator, value }` and operators such as `eq`, `in`, `exists` and `is_empty`, and text search matches plain text. Semantic search ranks records by embeddings, so it works only on a schema with embeddings enabled, and each result carries its `match.semantic_score`.
|
|
126
128
|
|
|
127
|
-
The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an
|
|
129
|
+
The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an edit in the Dashboard, CI reprocesses the existing records automatically. You do not need a new import or an Organization reset.
|
|
128
130
|
|
|
129
131
|
> **Warning:** `embedEntities` alone gives no semantic search over records. Select at least
|
|
130
132
|
> one field with `includeInItemEmbedding`, or the schema has no record vector
|
|
@@ -156,20 +158,16 @@ PostgreSQL holds the source of truth and runs semantic search with pgvector. The
|
|
|
156
158
|
|
|
157
159
|
## Documents and Knowledge
|
|
158
160
|
|
|
159
|
-
Knowledge
|
|
160
|
-
|
|
161
|
-
The two areas share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both areas also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization.
|
|
162
|
-
|
|
163
|
-
A finished sync does not mean that the new files are searchable yet, because indexing runs after the sync. A workflow Wait at level `sync` waits for the import only, while level `indexing` also waits for indexing. Use `indexing` before a demo step that searches the new content. The [Connectors](connectors.md) page compares Knowledge sync with Documents sync.
|
|
161
|
+
Knowledge and Documents share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both kinds also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization. The [Documents](documents.md) page covers Document Collections, sync and indexing.
|
|
164
162
|
|
|
165
163
|
## Read and write from an app
|
|
166
164
|
|
|
167
|
-
|
|
165
|
+
An external app reaches records in two ways, and the template uses both.
|
|
168
166
|
|
|
169
167
|
| | Browser | Server |
|
|
170
168
|
| ----------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
|
171
169
|
| API | Entity hooks from `defineEntity({ contract, config })` in `@collective-intelligence/external-apps-client-sdk-react/resources` | `ServerClient.records` from `@collective-intelligence/external-apps-server-sdk/effect` |
|
|
172
|
-
| Acts as | The signed-in user |
|
|
170
|
+
| Acts as | The signed-in user | A Server Credential, with the Access Grants of the app's server |
|
|
173
171
|
| Operations | Whatever the entity contract allows: create, get, search, update, delete, upload | `create`, `get`, `search`, `update`, `delete` |
|
|
174
172
|
| Typical use | Forms, lists and tables | Server routes and the template's Read Port |
|
|
175
173
|
|
|
@@ -189,21 +187,26 @@ return { ...fields, id: record.id }
|
|
|
189
187
|
|
|
190
188
|
`records.search` accepts a filter, text search, sort, pagination and `include`. App filtering goes through `POST /application/v1/resources/:schemaId/search`, and relation field pickers use `POST /application/v1/knowledge/autocomplete`. The SDKs do not wrap semantic search; agents use the retrieval tools for that.
|
|
191
189
|
|
|
192
|
-
> **Warning:**
|
|
193
|
-
>
|
|
194
|
-
>
|
|
190
|
+
> **Warning:** Server calls need a Server Credential (`dsc_`), and CI refuses the enrollment
|
|
191
|
+
> client secret on server calls because that secret only signs users in. Create
|
|
192
|
+
> one with `ci apps credentials upsert <app> --name <name> --env-file <file>`.
|
|
193
|
+
> The Server Credential acts with the Access Grants of the app's server, so
|
|
194
|
+
> those grants need read access to Leads. CI accepts a user's token only when
|
|
195
|
+
> the user holds the application gate. See [Access](../platform/access.md).
|
|
195
196
|
|
|
196
197
|
## Change events
|
|
197
198
|
|
|
198
|
-
Every write to a record produces a change event, whether the write comes from
|
|
199
|
+
Every write to a record produces a change event, whether the write comes from an external app, Studio, an agent or a workflow. Database triggers write a change log entry for each changed record, with one entry per record per transaction. A sync that changes nothing writes no entry.
|
|
199
200
|
|
|
200
201
|
An app subscribes to a topic per schema, for example `{ schema: "acme.app.schemas.lead" }`. `useRealtimeTopic(topic, changes => ...)` receives batches of changes that carry record IDs only, and `useEntityChanges(contract)` gives a function that refreshes the entity reads for those IDs. CI checks read access for each change before it sends it. When CI cannot decide whether a reader may see a delete, it sends that reader a `reset` instead of the ID, and the client must then refetch.
|
|
201
202
|
|
|
202
|
-
To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or the
|
|
203
|
+
To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or a workflow that writes the Lead.
|
|
203
204
|
|
|
204
205
|
## Related pages
|
|
205
206
|
|
|
207
|
+
- [Data resources](index.md)
|
|
206
208
|
- [Datasets](datasets.md)
|
|
209
|
+
- [Documents](documents.md)
|
|
207
210
|
- [Connectors](connectors.md)
|
|
208
211
|
- [Access](../platform/access.md)
|
|
209
212
|
- [SDKs](../develop/sdks.md)
|