@collective-intelligence/extensions-sdk 0.1.15 → 0.1.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1286,6 +1286,23 @@ manifest and retain sources in `schemaSidecars`. `canonicalPackageManifest`
1286
1286
  therefore contains no native objects or functions. Projection changes are
1287
1287
  intentional manifest and declaration-hash changes.
1288
1288
 
1289
+ ### Schema slots
1290
+
1291
+ `packageSchemaSlots(definition)` lists every schema position that a package
1292
+ declares. It accepts a `PackageDefinition` or any `{ manifest }` holder and
1293
+ returns `PackageSchemaSlot[]` in primitive order. Each slot has
1294
+ `{ location, direction, profile, projection }`. The declaration position sets
1295
+ `profile` and `direction`; an adapter label never does. A host checks each
1296
+ `projection` against its `profile` with `assertConsumerProfile`, and tools that
1297
+ read schemas, such as code generators, use the same list.
1298
+
1299
+ Slots exist for tool, agent, connection, connector, notification channel,
1300
+ workflow node, field and template, Studio component, and knowledge schema
1301
+ declarations. `datasetTableTemplate` adds `configuration` and `requirements`
1302
+ (`runtime_json`, input). `externalAppGateway` adds the `input` and `output` of
1303
+ each expected operation (`runtime_json`). A declaration kind without a handler
1304
+ in the reader fails type-checking.
1305
+
1289
1306
  ### JSON value boundaries
1290
1307
 
1291
1308
  `declareWorkflowNode` always derives the node editor from the projected
@@ -1679,7 +1696,8 @@ that collection's resource ids as
1679
1696
 
1680
1697
  `PackageKnowledgeConnector` implementations receive one decoded envelope containing
1681
1698
  `settings`, exact named `connections`, and the operation `request`. Optional
1682
- `browse`, `discover`, `pull`, `push`, and `read` methods remain package-owned.
1699
+ `browse`, `discover`, `pull`, `push`, `read`, and `readRecord` methods remain
1700
+ package-owned.
1683
1701
  Open `contributions` identify package presentation/control integrations by FQN,
1684
1702
  connection slot, connection template, capability, and portable JSON metadata;
1685
1703
  hosts route those identities without provider-name unions or registries.
@@ -2433,6 +2451,78 @@ Add these settings to the existing schema declaration's `definition` object:
2433
2451
  }
2434
2452
  ```
2435
2453
 
2454
+ ### Remote deletion propagation
2455
+
2456
+ Author rule: a Knowledge connector must offer full deletion. A full import run
2457
+ deletes each record whose item is absent from a proven-complete listing (the
2458
+ **Census**). Every connector therefore gives every item a stable item ID and
2459
+ every listing page census evidence. A page without valid evidence deletes
2460
+ nothing.
2461
+
2462
+ A connector declares its **Remote deletion capability** with `remoteDeletion`:
2463
+
2464
+ ```ts
2465
+ // Full deletion only. Omitting `remoteDeletion` means the same.
2466
+ remoteDeletion: { capability: "full" }
2467
+
2468
+ // Full deletion, and a delta run that deletes from a Tombstone feed.
2469
+ remoteDeletion: { capability: "full_and_delta", feed: "server_filtered" }
2470
+ ```
2471
+
2472
+ An omitted `remoteDeletion` means `{ capability: "full" }`. There is no `none`.
2473
+ The type `KnowledgeConnectorRemoteDeletion` allows a `feed` (`"server_filtered"`
2474
+ or `"client_filtered"`) only with `full_and_delta`. Declare `full_and_delta` only
2475
+ when the source offers a **Tombstone feed**. The backend stores the declaration
2476
+ on the connector template, and consumers read it from the template DTO.
2477
+
2478
+ What a connector gives, by capability:
2479
+
2480
+ - **Stable item ID.** `remoteId(remote)` of the Knowledge adapter returns the
2481
+ same ID for the same item on every run (see "Knowledge adapter identity").
2482
+ `KnowledgeConnectorPullPage.itemMetadata` aligns provider identity and
2483
+ revision with `items` by index.
2484
+ - **Census evidence.** Each `KnowledgeConnectorPullPage` carries `census`
2485
+ (`KnowledgeConnectorCensusPageEvidence`): `sourceId`, `requestCursor` (`null`
2486
+ on the first page), `terminal` (true exactly when `nextCursor` is `null`), and
2487
+ `listing`. `listing` is `"complete"` when the page lists every item of the
2488
+ source, `"filtered"` when a query or remote filter hides items, and `"capped"`
2489
+ when a provider or result limit stops the listing. Only pages that are all
2490
+ `"complete"` form a Census.
2491
+ - **Item proof (optional).** A connector that can read one item implements the
2492
+ `readRecord` method of `PackageKnowledgeConnector` (wire operation
2493
+ `read_record`) and returns a `KnowledgeConnectorRecordProof`: `trashed`,
2494
+ `not_found`, `moved` (with the `sourceId` it moved to), `live`, or
2495
+ `not_verified` (with a `reason`). The host deletes a record only when this proof
2496
+ agrees with the absence. The method `read` is not this proof: a connector can
2497
+ use `read` for another purpose, such as a file preview, and the host ignores
2498
+ it for deletion. Without `readRecord`, the proven-complete listing is the
2499
+ proof (the listing-only path).
2500
+ - **Tombstone feed (`full_and_delta` only).** A delta page lists tombstones in
2501
+ `itemMetadata` with `deletedAt` and `deletionKind`. The last page of a feed run
2502
+ gives the **Feed cursor** as `deltaCursor`; without it the next run has no start
2503
+ point. When the source rejects the cursor, return a
2504
+ `KnowledgeConnectorPullFault` with the code `cursor_expired`. The host then
2505
+ clears the cursor, and the next run of the binding reads in full.
2506
+
2507
+ An entity binding can declare `remoteDeletionPropagation: "never" | "full" |
2508
+ "full_and_delta"`. The values are ordered `never` < `full` < `full_and_delta`. An
2509
+ omitted value creates a binding with no stored setting, which follows the
2510
+ capability, and keeps the stored setting on redeploy. A declared value replaces
2511
+ the stored setting. A value above the capability is rejected with the error
2512
+ `remote_deletion_above_capability` on every write path. The admin API also
2513
+ accepts `null` to reset a binding to follow the capability. The binding DTO
2514
+ returns the computed `effectiveRemoteDeletionPropagation`; read it instead of
2515
+ recomputing it.
2516
+
2517
+ An entity binding can also declare `maxRemoteDeletionsPerRun`, the **Deletion
2518
+ brake**: a positive integer, or `null` for no limit. A Census with more
2519
+ candidates than this number deletes nothing. A run reports its **Deletion
2520
+ outcome** in the `deletion` field of the pipeline run: `applied`, `skipped`, or
2521
+ `incomplete`, each with its counts or reason.
2522
+
2523
+ The setting and the brake are operational configuration: binding definition
2524
+ snapshots exclude them, and definition rollback preserves their current values.
2525
+
2436
2526
  ### Per-source run data access
2437
2527
 
2438
2528
  `declareAgent`, `declareSkill`, and `declareWorkflow` accept optional
@@ -36,6 +36,7 @@ export type KnowledgeConnectorInstanceBindingOptions = Readonly<{
36
36
  updateMode?: "full" | "partial";
37
37
  outboundStrategy?: "local_first" | "remote_first";
38
38
  }>;
39
+ export type KnowledgeConnectorRemoteDeletionPropagation = "never" | "full" | "full_and_delta";
39
40
  export type KnowledgeConnectorPushEntityMatch = "remote_ref" | "unique_fields";
40
41
  export type KnowledgeConnectorInstanceEntityBinding = KnowledgeConnectorInstanceBindingOptions & Readonly<{
41
42
  schemaFqn: string;
@@ -50,6 +51,17 @@ export type KnowledgeConnectorInstanceEntityBinding = KnowledgeConnectorInstance
50
51
  * the unique values of every entity the push writes.
51
52
  */
52
53
  pushEntityMatch?: KnowledgeConnectorPushEntityMatch;
54
+ /**
55
+ * Most records one run may delete; a census with more candidates
56
+ * deletes nothing. `null` sets no limit.
57
+ */
58
+ maxRemoteDeletionsPerRun?: number | null;
59
+ /**
60
+ * Omission creates a binding with no stored setting, which follows the
61
+ * connector capability, and keeps the stored setting on redeploy. A
62
+ * setting above the capability is rejected.
63
+ */
64
+ remoteDeletionPropagation?: KnowledgeConnectorRemoteDeletionPropagation;
53
65
  suppressEntityEvents?: boolean;
54
66
  }>;
55
67
  export type KnowledgeConnectorInstanceRelationBinding = KnowledgeConnectorInstanceBindingOptions & Readonly<{
@@ -144,6 +144,21 @@ export type DocumentsDeltaCursorExpired = Readonly<{
144
144
  code: "cursor_expired";
145
145
  message: string;
146
146
  }>;
147
+ /**
148
+ * What a connector can do about **Remote deletion propagation**. A `feed`
149
+ * exists only with `full_and_delta`, so the union allows no other pairing.
150
+ * - `full`: a full import run deletes each record whose item is absent from a
151
+ * proven-complete listing (the **Census**).
152
+ * - `full_and_delta`: a delta run also deletes from a **Tombstone feed**.
153
+ * `server_filtered` means the source filters the feed; `client_filtered`
154
+ * means the connector filters by the time of its last sync.
155
+ */
156
+ export type KnowledgeConnectorRemoteDeletion = Readonly<{
157
+ capability: "full";
158
+ }> | Readonly<{
159
+ capability: "full_and_delta";
160
+ feed: "server_filtered" | "client_filtered";
161
+ }>;
147
162
  export type KnowledgeConnectorDeclaration = PackagePrimitiveBase<"knowledgeConnector"> & Readonly<{
148
163
  autoActivate?: boolean;
149
164
  /** Creates the singleton instance when the package is installed. */
@@ -160,6 +175,15 @@ export type KnowledgeConnectorDeclaration = PackagePrimitiveBase<"knowledgeConne
160
175
  isSingleton?: boolean;
161
176
  /** Enables per-entity-binding outbound sync when the runtime implements it. */
162
177
  outboundCapability?: "declared_and_enablable" | "none";
178
+ /**
179
+ * Author rule: a connector must offer full deletion. Omission means
180
+ * `{ capability: "full" }`; there is no `none`. A connector therefore
181
+ * gives every item a stable item ID and every listing page census
182
+ * evidence (`KnowledgeConnectorCensusPageEvidence`); a page without
183
+ * valid evidence deletes nothing. Declare `full_and_delta` only when
184
+ * the source offers a tombstone feed.
185
+ */
186
+ remoteDeletion?: KnowledgeConnectorRemoteDeletion;
163
187
  /** Suggested schedules for root bindings created from discovered remotes. */
164
188
  recommendedImportSchedules?: readonly KnowledgePredefinedSchedule[];
165
189
  relationSchemaCapabilities?: readonly KnowledgeRelationSchemaCapability[];
@@ -0,0 +1,84 @@
1
+ import type { JsonObject } from "../foundation/json.js";
2
+ /**
3
+ * Evidence that one listing page belongs to a **Census**. A census is proven
4
+ * only when every page of the run states `listing: "complete"`; a `filtered`
5
+ * or `capped` page, or a page without valid evidence, deletes nothing.
6
+ */
7
+ export type KnowledgeConnectorCensusPageEvidence = Readonly<{
8
+ /** Stable identity of the listed source. */
9
+ sourceId: string;
10
+ /** The cursor that requested this page; `null` for the first page. */
11
+ requestCursor: string | null;
12
+ /** True exactly when the page has no next cursor. */
13
+ terminal: boolean;
14
+ /**
15
+ * `complete` lists every item of the source. `filtered` hides items by a
16
+ * query or a remote filter. `capped` stops at a provider or result limit.
17
+ */
18
+ listing: "complete" | "filtered" | "capped";
19
+ }>;
20
+ /**
21
+ * What a connector learns when it reads one item by its ID with the operation
22
+ * `readRecord` (wire name `read_record`). Hosts delete a record only after a
23
+ * proven-complete census, and use this proof where the connector implements
24
+ * `readRecord`. The operation `read` is not this proof. A connector without
25
+ * `readRecord` gives none, and the host uses the listing-only path.
26
+ */
27
+ export type KnowledgeConnectorRecordProof = Readonly<{
28
+ remoteId: string;
29
+ }> & (Readonly<{
30
+ status: "trashed" | "not_found";
31
+ }> | Readonly<{
32
+ status: "moved";
33
+ sourceId: string | null;
34
+ }> | Readonly<{
35
+ status: "live";
36
+ sourceId: string;
37
+ item: JsonObject;
38
+ }> | Readonly<{
39
+ status: "not_verified";
40
+ reason: string;
41
+ }>);
42
+ /** Provider identity and revision of one item, aligned by index with `items`. */
43
+ export type KnowledgeConnectorRemoteItemMetadata = Readonly<{
44
+ externalId: string;
45
+ revision: string;
46
+ deletedAt?: string;
47
+ deletionKind?: "hard" | "soft";
48
+ updatedAt?: string;
49
+ }>;
50
+ /**
51
+ * A remote record the connector skipped while reading a page, such as a
52
+ * spreadsheet row without an identity. It names where, never a value.
53
+ */
54
+ export type KnowledgeConnectorRejectedRecord = Readonly<{
55
+ code: string;
56
+ location: string;
57
+ field?: string;
58
+ }>;
59
+ /** One page of items that a connector pulls from its source. */
60
+ export type KnowledgeConnectorPullPage = Readonly<{
61
+ items: readonly JsonObject[];
62
+ /** `null` on the last page. */
63
+ nextCursor: string | null;
64
+ /** Without evidence, the page proves no **Census** and deletes nothing. */
65
+ census?: KnowledgeConnectorCensusPageEvidence;
66
+ /**
67
+ * The **Feed cursor**, given on the last page of a feed run. A run without
68
+ * one leaves the next run without a start point.
69
+ */
70
+ deltaCursor?: string;
71
+ itemMetadata?: readonly KnowledgeConnectorRemoteItemMetadata[];
72
+ rejected?: readonly KnowledgeConnectorRejectedRecord[];
73
+ }>;
74
+ /**
75
+ * A pull that fails instead of giving a page. The code `cursor_expired` means
76
+ * the source rejected the **Feed cursor**; the host then clears the cursor, and
77
+ * the next run of the binding reads in full.
78
+ */
79
+ export type KnowledgeConnectorPullFault = Readonly<{
80
+ fault: Readonly<{
81
+ code: string;
82
+ message: string;
83
+ }>;
84
+ }>;
package/dist/index.d.ts CHANGED
@@ -27,6 +27,7 @@ export { isVariableReference, packageVariables, packageVariableTypes, packageVar
27
27
  export type { PackageVariable, PackageVariableDeclaration, PackageVariables, PackageVariablesDeclaration, PackageVariableType, PackageVariableValue, VariableReference, Varying, VaryingDefinition, VaryingExcludedKey, VaryingStructure } from "./package/variables.js";
28
28
  export { normalizeNotificationChannelCapabilities } from "./components/notification-channel-capabilities.js";
29
29
  export { renderKnowledgeRemoteReferenceUrl } from "./components/knowledge-remote-reference.js";
30
+ export type { KnowledgeConnectorCensusPageEvidence, KnowledgeConnectorPullFault, KnowledgeConnectorPullPage, KnowledgeConnectorRecordProof, KnowledgeConnectorRejectedRecord, KnowledgeConnectorRemoteItemMetadata } from "./components/knowledge-remote-deletion.js";
30
31
  export { BACKEND_EXECUTION_LANES, declareHostedContribution, FRONTEND_CATALOG_CONTRACT, FRONTEND_CATALOG_EXPORT_NAME, HOSTS } from "./hosted/contributions.js";
31
32
  export type { BackendExecutionLane, BackendHostedContributionManifest, ExtensionHost, FrontendCatalogContract, FrontendCatalogContributionManifest, FrontendCatalogExportName, FrontendExtensionHost, HostedContributionDeclaration, HostedContributionInput, HostedContributionManifest, HostedContributionSourceIdentity } from "./hosted/contributions.js";
32
33
  export type { ComponentManifest, ComponentPresentation, DeclarationHandle, DeclarationReference, DeclaredComponent, DocumentsConnectorImplementation, PackageSchemaRuntimeSidecar, MetadataComponentManifest, ToolClassDeclaration, WorkflowFieldClassDeclaration, KnowledgeEntityAdapterClassDeclaration, ConnectionClassDeclaration, KnowledgeConnectorClassDeclaration, KnowledgeConnectorOAuthClassDeclaration, EventListenerClassDeclaration, FieldTransformerClassDeclaration, NotificationChannelClassDeclaration, KnowledgeRelationAdapterClassDeclaration, ToolSettingsOptionsClassDeclaration, WorkflowNodeClassDeclaration, WorkflowNodePortInput } from "./components/authoring-core.js";
@@ -59,7 +60,7 @@ export { organizationSettingsViolation } from "./components/organization-setting
59
60
  export type { OrganizationSettingsApplicationConsent, OrganizationSettingsAuthentication, OrganizationSettingsDeclaration, OrganizationSettingsLocalMethod, OrganizationSettingsMedia, OrganizationSettingsSso, OrganizationSettingsSsoRole, VaryingOrganizationSettings } from "./components/organization-settings.js";
60
61
  export { declareAgent, declareSkill, duplicateToolBinding, toolBindingDeclaration } from "./components/agents.js";
61
62
  export type { AgentActivityPolicy, AgentAudience, AgentAudienceDeclaration, AgentDeclaration, AgentDefinitionInput, AgentMemberAudience, AgentModelTier, DeclaredToolBindings, KnowledgeScopeSelection, SkillDeclaration, SkillDefinitionInput, ToolBinding, ToolBindingApproval, ToolBindingDeclaration, ToolBindingKnowledgeScope } from "./components/agents.js";
62
- export type { DocumentsDeltaCursorExpired, DocumentsDeltaItem, DocumentsDeltaPage, DocumentsDeltaRequest, DocumentsDrive, DocumentsDrivesPage, DocumentsFolder, DocumentsFoldersPage, DocumentsFetchedContent, DocumentsRootAncestry, DocumentsSource, KnowledgeEntityAdapterDeclaration, ConnectionDeclaration, DatasetParserDeclaration, ExternalDatasetProviderDeclaration, KnowledgeConnectorContribution, KnowledgeConnectorBindingExtraConfig, KnowledgeConnectorDeclaration, KnowledgeConnectorFieldMapRule, KnowledgeEntitySchemaCapability, KnowledgeConnectorOAuthDeclaration, KnowledgeConnectorOAuthTokens, KnowledgeConnectorRemoteReference, EventListenerDeclaration, EventListenerInternalOccurrence, EventListenerPollRequest, EventListenerPollResult, EventListenerWebhookRequest, KnowledgePredefinedSchedule, ManagedDatasetProviderDeclaration, KnowledgeRelationAdapterDeclaration, KnowledgeRelationSchemaCapability } from "./components/connectors.js";
63
+ export type { DocumentsDeltaCursorExpired, DocumentsDeltaItem, DocumentsDeltaPage, DocumentsDeltaRequest, DocumentsDrive, DocumentsDrivesPage, DocumentsFolder, DocumentsFoldersPage, DocumentsFetchedContent, DocumentsRootAncestry, DocumentsSource, KnowledgeEntityAdapterDeclaration, ConnectionDeclaration, DatasetParserDeclaration, ExternalDatasetProviderDeclaration, KnowledgeConnectorContribution, KnowledgeConnectorBindingExtraConfig, KnowledgeConnectorDeclaration, KnowledgeConnectorRemoteDeletion, KnowledgeConnectorFieldMapRule, KnowledgeEntitySchemaCapability, KnowledgeConnectorOAuthDeclaration, KnowledgeConnectorOAuthTokens, KnowledgeConnectorRemoteReference, EventListenerDeclaration, EventListenerInternalOccurrence, EventListenerPollRequest, EventListenerPollResult, EventListenerWebhookRequest, KnowledgePredefinedSchedule, ManagedDatasetProviderDeclaration, KnowledgeRelationAdapterDeclaration, KnowledgeRelationSchemaCapability } from "./components/connectors.js";
63
64
  export type { PackagePrimitiveDeclaration } from "./components/declaration.js";
64
65
  export type { DatasetBlueprintClassDeclaration, DatasetBlueprintDeclaration, DatasetTableTemplateClassDeclaration, DatasetTableTemplateDeclaration, DatasetTableTemplateHint, DatasetTableTemplatePhysicalCode, DatasetTableTemplateSemanticRole, DatasetTableTemplateUniqueness } from "./components/templates.js";
65
66
  export type { KnowledgeAutocompleteConfiguration, KnowledgeAutocompleteFieldReference, KnowledgeEntityRelationField, KnowledgeEntitySchemaDeclaration, KnowledgeDisplayDefaults, KnowledgeFieldLocalization, KnowledgeFieldPresentation, KnowledgeFieldSemantics, KnowledgeFileField, KnowledgeFileFieldDeclaration, KnowledgeUserField, KnowledgeUserFieldDeclaration, KnowledgeManagedProjection, LocalizedFieldText, KnowledgePolicyDeclaration, KnowledgeRelationSchemaDeclaration } from "./components/knowledge.js";
@@ -73,6 +74,8 @@ export type { DatasetParserClassDeclaration, ExternalDatasetProviderClassDeclara
73
74
  export type { EmailProviderDeclaration, InboundMessagingProviderDefinition, OrganizationSsoDeclaration, OutboundMessagingProviderDeclaration } from "./components/connectors.js";
74
75
  export type { ConversationReference, InboundAttachment, InboundMessage, MessageReceipt, OutboundMessage } from "./components/messaging.js";
75
76
  export { parsePackageManifest } from "./package/parse-manifest.js";
77
+ export { packageSchemaSlots } from "./package/schema-slots.js";
78
+ export type { PackageSchemaSlot } from "./package/schema-slots.js";
76
79
  export { WorkflowEditorWidgets, WorkflowTemplateKinds, WorkflowTriggerLifecycleAdapterIds } from "./components/workflows.js";
77
80
  export { parseWorkflowTriggerActivation } from "./package/workflow-readers.js";
78
81
  export { parseWorkflowNodeContract, parseWorkflowReusableDependency } from "./package/workflow-readers.js";
@@ -80,7 +83,7 @@ export { parseJsonSchemaProjection } from "./package/readers.js";
80
83
  export { deriveWorkflowEditor, WorkflowJsonSchemaFormat } from "./components/workflow-editor.js";
81
84
  export type { WorkflowEditorDerivationInput } from "./components/workflow-editor.js";
82
85
  export { declareKnowledgeConnectorInstance, declareDatasetConnectorInstance } from "./components/metadata-authoring.js";
83
- export type { ConnectorInstanceConnection, KnowledgeConnectorInstanceDeclaration, DatasetConnectorInstanceDeclaration, KnowledgeConnectorInstanceSettings, KnowledgeConnectorInstanceRemoteFilters, KnowledgeConnectorInstanceBindingOptions, KnowledgeConnectorInstanceEntityBinding, KnowledgeConnectorPushEntityMatch, KnowledgeConnectorInstanceRelationBinding, KnowledgeConnectorInstancePipeline, KnowledgeConnectorInstanceSchedule, KnowledgeConnectorInstanceDataPolicy, DatasetConnectorInstanceDataPolicy, DatasetConnectorInstanceSchedule } from "./components/connector-instances.js";
86
+ export type { ConnectorInstanceConnection, KnowledgeConnectorInstanceDeclaration, DatasetConnectorInstanceDeclaration, KnowledgeConnectorInstanceSettings, KnowledgeConnectorInstanceRemoteFilters, KnowledgeConnectorInstanceBindingOptions, KnowledgeConnectorInstanceEntityBinding, KnowledgeConnectorPushEntityMatch, KnowledgeConnectorRemoteDeletionPropagation, KnowledgeConnectorInstanceRelationBinding, KnowledgeConnectorInstancePipeline, KnowledgeConnectorInstanceSchedule, KnowledgeConnectorInstanceDataPolicy, DatasetConnectorInstanceDataPolicy, DatasetConnectorInstanceSchedule } from "./components/connector-instances.js";
84
87
  export type { KnowledgePolicyBody, KnowledgePolicyPredicate, KnowledgePolicyRef } from "./components/knowledge.js";
85
88
  export type { ServicePrincipalDeclaration, ServicePrincipalOrganizationRole } from "./components/service-principal.js";
86
89
  export { declareConnectionInstance } from "./components/metadata-authoring.js";