@cratis/pi 7.114.0 → 7.115.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (26) hide show
  1. package/package/corpus/harnesses/copilot/instructions/reactors.instructions.md +1 -1
  2. package/package/corpus/product-sync.json +4 -4
  3. package/package/corpus/rules/reactors.md +1 -1
  4. package/package/corpus/skills/cratis-chronicle-client-dotnet/SKILL.md +4 -0
  5. package/package/corpus/skills/cratis-chronicle-client-dotnet/references/append-outcomes.md +78 -0
  6. package/package/corpus/skills/cratis-chronicle-client-dotnet/references/connection-options.md +13 -0
  7. package/package/corpus/skills/cratis-chronicle-client-dotnet/references/hosting.md +23 -0
  8. package/package/corpus/skills/cratis-chronicle-compliance/SKILL.md +6 -0
  9. package/package/corpus/skills/cratis-chronicle-decision-reads/references/choosing-how-a-rule-holds.md +8 -4
  10. package/package/corpus/skills/cratis-chronicle-event-constraints/SKILL.md +16 -5
  11. package/package/corpus/skills/cratis-chronicle-hosting/SKILL.md +25 -9
  12. package/package/corpus/skills/cratis-chronicle-hosting/references/clustering-and-schema.md +84 -0
  13. package/package/corpus/skills/cratis-chronicle-hosting/references/operations.md +30 -2
  14. package/package/corpus/skills/cratis-chronicle-hosting/references/storage-resolution.md +30 -0
  15. package/package/corpus/skills/cratis-chronicle-hosting/references/webhooks.md +110 -0
  16. package/package/corpus/skills/cratis-chronicle-projection/references/definition-changes.md +33 -0
  17. package/package/corpus/skills/cratis-chronicle-projection/references/fluent-builder.md +12 -0
  18. package/package/corpus/skills/cratis-chronicle-reactor/SKILL.md +12 -0
  19. package/package/corpus/skills/cratis-chronicle-reactor/references/return-event-metadata.md +13 -0
  20. package/package/corpus/skills/cratis-chronicle-read-model/SKILL.md +3 -1
  21. package/package/corpus/skills/cratis-chronicle-read-model/references/indexing.md +44 -0
  22. package/package/corpus/skills/cratis-chronicle-read-model-specifications/SKILL.md +4 -0
  23. package/package/corpus/skills/cratis-chronicle-read-model-specifications/references/seeded-state.md +35 -0
  24. package/package/corpus/skills/cratis-chronicle-reducer/SKILL.md +14 -0
  25. package/package/corpus/skills/cratis-chronicle-reducer/references/replay-hooks-and-definition-changes.md +7 -0
  26. package/package.json +1 -1
@@ -225,7 +225,7 @@ rejects Warning-level reactor/event-type drift (not Information-level read-model
225
225
  findings). Each reactor class should have a focused purpose. Multiple handler methods in one reactor are fine if they serve the same automation concern.
226
226
  5. **Failure behavior** — If a reactor throws, *or* a returned side-effect event fails to append (constraint violation, concurrency violation, or error), the failing event-source partition pauses until the issue is resolved. Repeated failures can **quarantine** the observer: once `QuarantineOnFailedPartitionCount`/`QuarantineOnFailedPartitionPercentage` (under `Observers`) is crossed, the observer enters the `Quarantined` state — reminders cancelled, retries stopped, automatic recovery suppressed. **A quarantine ends when an operator clears it (`ClearObserverQuarantine()`), or when the observer is subscribed again** — for an application observer that is when the client connects again, for example after a redeploy; Kernel-owned observers are subscribed again when the Kernel starts. A periodic watchdog (default 60s, `WatchdogInterval`) re-routes stuck/dead observers but does not rescue quarantined ones. Design for resilience.
227
227
  6. **Don't throw to validate malformed inbound events** — reactors are not data-quality validators; invalid payloads must be rejected at the command/append site. When a malformed cross-service fact reaches a consumer, throwing just to reject it pauses the partition and can quarantine the whole observer. Instead append a clear failure/dead-letter event (e.g. `ProvisioningFailed`) or surface it via the operational failure path, and skip partial side effects.
228
- 7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class.
228
+ 7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class. From Chronicle 19.90.0, the .NET client runs different partitions concurrently (one partition's batches stay ordered); shared dependencies must be thread-safe, or set `ChronicleOptions.Reactors.MaxConcurrentPartitions = 1`. See **cratis-chronicle-reactor** for verified options and limits.
229
229
  8. **A new side-effect reactor must not act on history.** A reactor that is new in a release (or has a new id) catches up from the start of the event log when it first registers, and `[OnceOnly]` does not cover that first catch-up because it is not a replay. If its handlers cause side effects (jobs, AI calls, external calls, notifications), start the observer at the tail on first registration or make the handler ignore facts that predate its registration, and specify that history causes nothing.
230
230
  9. **An unattended, irreversible pass reads the log, not the sink — and carries a fuse.** A reactor that deletes, revokes, bills or notifies *because a read model says something is absent* is trusting a materialized sink that is only as current as its observer: a paused partition, a replay in progress or a lagging projection makes "absent" true of the sink and false of the world. Decide from a **`[Passive]`** read model (computed on demand from the events at the moment the reactor runs) or from the events themselves. And the code shape cannot tell an **empty subject set** ("nothing to consider" — fail-safe) from an **empty qualifying set** ("nothing survived the filter" — fail-destructive: every subject is about to be acted on), so an unattended pass states its expected population, refuses when the count is implausible, and caps how many subjects one run may touch — the same discipline as [guards-and-fuses.md](./guards-and-fuses.instructions.md), applied to a reactor. A sweep that decides from a read model also refuses while that model's observer is replaying or behind, because a rebuilding model reads as "nothing was ever processed"; specify the replaying case.
231
231
  10. **Provider clients stay behind a translation or adapter reactor.** A `[Command]` or `IReactor` in a domain slice that injects or imports a provider client (GitHub, payment, mail SDK) is a finding: put the client behind one adapter slice and let domain slices work with our events and commands. Judgment-shaped today; a candidate for a namespace-boundary analyzer.
@@ -284,9 +284,9 @@
284
284
  }
285
285
  ],
286
286
  "synced": {
287
- "version": "19.81.1",
288
- "date": "2026-10-17",
289
- "by": "Cratis/AI#773"
287
+ "version": "19.110.1",
288
+ "date": "2026-10-25",
289
+ "by": "Cratis/AI#820"
290
290
  },
291
291
  "covers": {
292
292
  "skills": [
@@ -323,7 +323,7 @@
323
323
  ],
324
324
  "rules": []
325
325
  },
326
- "notes": "Reviewed releases 19.65.1-19.81.1 at tags: subscription generation pins, StartAtTail, namespace removal/reconnect, storage connection resolution (deploy 19.74.2+), definition refusal recovery, conflict event fields, .NET occurrences/lifecycle/receipts and testing identities, full-history per-key reads, child all-event mapping repair, projection SoftwareVersion, bounded job acceptance/recovery, immediate-append refusal ownership and namespace subscription reconciliation. Existing protected-read guidance retained. Chronicle#5028 definition round-trip/restart-replay fix is unreleased and not taught as fixed. No new kernel/transport probe."
326
+ "notes": "Reviewed all 60 stable releases 19.82.0-19.110.1 against tagged v19.110.1 source: append uncertainty and constraint-index repair, .NET keyed appends and multi-scope guards, request completion ownership, concurrent reactor/reducer partitions and returned-event receipts, projection replay/upgrade repairs, seeded scenarios, SQL/composite indexes, SQL membership/schema ownership, Localhost single-writer fencing, health probes, Azure identity/Key Vault purge, webhook target validation/retries/managed identity/real-delivery probes, and disabled-by-default projection-generation groundwork. 19.110.0 .NET packages were not published; use 19.110.1. Other covered skills were reviewed for release relevance without mechanical changes. Existing protected-read guidance retained. No new kernel/transport probe or capability."
327
327
  },
328
328
  {
329
329
  "id": "arc-go",
@@ -224,7 +224,7 @@ rejects Warning-level reactor/event-type drift (not Information-level read-model
224
224
  findings). Each reactor class should have a focused purpose. Multiple handler methods in one reactor are fine if they serve the same automation concern.
225
225
  5. **Failure behavior** — If a reactor throws, *or* a returned side-effect event fails to append (constraint violation, concurrency violation, or error), the failing event-source partition pauses until the issue is resolved. Repeated failures can **quarantine** the observer: once `QuarantineOnFailedPartitionCount`/`QuarantineOnFailedPartitionPercentage` (under `Observers`) is crossed, the observer enters the `Quarantined` state — reminders cancelled, retries stopped, automatic recovery suppressed. **A quarantine ends when an operator clears it (`ClearObserverQuarantine()`), or when the observer is subscribed again** — for an application observer that is when the client connects again, for example after a redeploy; Kernel-owned observers are subscribed again when the Kernel starts. A periodic watchdog (default 60s, `WatchdogInterval`) re-routes stuck/dead observers but does not rescue quarantined ones. Design for resilience.
226
226
  6. **Don't throw to validate malformed inbound events** — reactors are not data-quality validators; invalid payloads must be rejected at the command/append site. When a malformed cross-service fact reaches a consumer, throwing just to reject it pauses the partition and can quarantine the whole observer. Instead append a clear failure/dead-letter event (e.g. `ProvisioningFailed`) or surface it via the operational failure path, and skip partial side effects.
227
- 7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class.
227
+ 7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class. From Chronicle 19.90.0, the .NET client runs different partitions concurrently (one partition's batches stay ordered); shared dependencies must be thread-safe, or set `ChronicleOptions.Reactors.MaxConcurrentPartitions = 1`. See **cratis-chronicle-reactor** for verified options and limits.
228
228
  8. **A new side-effect reactor must not act on history.** A reactor that is new in a release (or has a new id) catches up from the start of the event log when it first registers, and `[OnceOnly]` does not cover that first catch-up because it is not a replay. If its handlers cause side effects (jobs, AI calls, external calls, notifications), start the observer at the tail on first registration or make the handler ignore facts that predate its registration, and specify that history causes nothing.
229
229
  9. **An unattended, irreversible pass reads the log, not the sink — and carries a fuse.** A reactor that deletes, revokes, bills or notifies *because a read model says something is absent* is trusting a materialized sink that is only as current as its observer: a paused partition, a replay in progress or a lagging projection makes "absent" true of the sink and false of the world. Decide from a **`[Passive]`** read model (computed on demand from the events at the moment the reactor runs) or from the events themselves. And the code shape cannot tell an **empty subject set** ("nothing to consider" — fail-safe) from an **empty qualifying set** ("nothing survived the filter" — fail-destructive: every subject is about to be acted on), so an unattended pass states its expected population, refuses when the count is implausible, and caps how many subjects one run may touch — the same discipline as [guards-and-fuses.md](./guards-and-fuses.md), applied to a reactor. A sweep that decides from a read model also refuses while that model's observer is replaying or behind, because a rebuilding model reads as "nothing was ever processed"; specify the replaying case.
230
230
  10. **Provider clients stay behind a translation or adapter reactor.** A `[Command]` or `IReactor` in a domain slice that injects or imports a provider client (GitHub, payment, mail SDK) is a finding: put the client behind one adapter slice and let domain slices work with our events and commands. Judgment-shaped today; a candidate for a namespace-boundary analyzer.
@@ -225,6 +225,10 @@ if (!result.IsSuccess)
225
225
  several event sources in one batch.
226
226
  - **There is no `AppendAnonymous`.** It does not exist anywhere in the product.
227
227
 
228
+ For unknown outcomes, `AppendOnce` / `AppendManyOnce`, keyed retry conflicts and
229
+ several concurrency guards for one source, read
230
+ [append outcomes and retry safety](references/append-outcomes.md).
231
+
228
232
  Reading back is on `IEventSequence`: `GetForEventSourceIdAndEventTypes`,
229
233
  `HasEventsFor`, `GetFromSequenceNumber`, `GetNextSequenceNumber`, and
230
234
  `GetTailSequenceNumber`. Redaction is `Redact`; subject erasure is a separate
@@ -0,0 +1,78 @@
1
+ <!-- Copyright (c) Cratis. All rights reserved. -->
2
+ <!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
3
+
4
+ # Append outcomes and retry safety
5
+
6
+ Verified against the tagged sources in `Source/Clients/DotNET/EventSequences/`
7
+ (`IEventSequence.cs`, `EventSequence.cs`, `IdempotencyKey.cs`, `AppendResult.cs`,
8
+ `AppendManyResult.cs`, `AppendError.cs`) and
9
+ [appending events](https://github.com/Cratis/Chronicle/blob/v19.110.1/Documentation/events/appending.mdx).
10
+ These are .NET client APIs; do not assume other SDKs expose the same methods.
11
+
12
+ ## Unknown is not failure-before-write
13
+
14
+ Requires Chronicle 19.82.2 for the distinct unknown-outcome and index-unavailable errors.
15
+ Requires Chronicle 19.108.0 for node-loss/timeout unknown-outcome handling.
16
+
17
+ `AppendError.AppendOutcomeUnknown` means the kernel cannot confirm whether the
18
+ events were stored, including an in-flight append interrupted by node loss or
19
+ a grain-call timeout. Do not retry an
20
+ unkeyed append blindly: read the source's events for the attempt's correlation
21
+ id, and append again only when none belongs to that attempt.
22
+
23
+ An `Errors` entry with value `ConstraintIndexUnavailable` means a governing unique/closing index
24
+ still needs repair and **nothing was stored**. Retry unchanged after a short
25
+ delay. The kernel repairs constraint indexes from the event log; restarting
26
+ clients or removing constraints is not a repair strategy.
27
+
28
+ ## Append once
29
+
30
+ Requires Chronicle .NET client 19.92.0 and kernel 19.91.0 for keyed appends.
31
+
32
+ ```csharp
33
+ using Cratis.Chronicle.EventSequences;
34
+
35
+ // Keep this key and payload unchanged across retries of this logical operation.
36
+ var key = new IdempotencyKey("order-confirmation-42");
37
+ var result = await eventLog.AppendOnce(orderId, new OrderConfirmed(), key);
38
+ ```
39
+
40
+ - `IdempotencyKey` accepts 1–512 characters. Never put personal data in it.
41
+ - Check `IEventSequence.SupportsIdempotentAppends`. A kernel without support
42
+ causes `IdempotentAppendsNotSupported` before sending; do not fall back to
43
+ unkeyed retries when duplicate protection is required.
44
+ - `AppendOnce`, `AppendManyOnce`, `AppendOnceWithNamedTags` and
45
+ `AppendManyOnceWithNamedTags` are on `IEventSequence` (and thus `IEventLog`).
46
+ A batch key derives each event key with `IdempotencyKey.ForIndex(index)`;
47
+ preserve the batch's order and content across retries.
48
+ - `EventForEventSourceId.IdempotencyKey` keys individual events in a batch,
49
+ including a mixed keyed/unkeyed batch spanning sources. Unkeyed entries do
50
+ not acquire duplicate protection just because other entries have keys.
51
+ - `AppendResult.Outcome` / `AppendManyResult.Outcomes` distinguish `Appended`,
52
+ `AlreadyAppended`, and `AlreadyAppendedAfterErasure`. An already-appended
53
+ outcome is success, not permission to re-create an erased event.
54
+ - Inspect `IdempotencyConflicts`: a key reused for different content or generation,
55
+ or a match that cannot still be verified, is not a successful retry. A keyed
56
+ retry after an unknown outcome must use the **same** key and content.
57
+ - Keys are refused on events staged in a unit of work and on appends carrying
58
+ effect receipts: these throw `IdempotentAppendsNotSupported`, not an unkeyed
59
+ fallback. Keyed appends are not a replacement for DCB or unit-of-work ownership.
60
+
61
+ Requires Chronicle Testing 19.99.0 for dropped keyed-append replies.
62
+
63
+ The Testing client supports keyed appends. `DropNext(DroppedReply.Append)` can lose a keyed append reply after persistence;
64
+ retry with the same key and assert exactly one stored event and an
65
+ already-appended outcome. This proves retry behavior, not an exactly-once
66
+ external side effect.
67
+
68
+ ## Several concurrency guards for one source
69
+
70
+ Requires Chronicle 19.84.0 for multiple concurrency guards for one source.
71
+
72
+ `AppendMany` overloads accept
73
+ `IEnumerable<KeyValuePair<EventSourceId, ConcurrencyScope>>`. Repeat a source id
74
+ for distinct stream guards; a dictionary cannot represent those entries. The
75
+ kernel checks the conjunction before any write, preserving batch atomicity.
76
+ A violation identifies `ScopeIndex` and `Scope`. Older kernels still refuse
77
+ unsupported combinations with `IncompatibleConcurrencyScopesForEventSource`;
78
+ never discard a guard to make an append work.
@@ -59,3 +59,16 @@ Read [host wiring](hosting.md) for the deployment distinction.
59
59
  > TLS enabled (the default) and no configured certificate. **A production connection string must set
60
60
  > `skipTlsValidation=false`** (or `Tls.SkipCertificateValidation = false`)
61
61
  > against a server whose certificate is actually verifiable.
62
+
63
+ ## Partition concurrency
64
+
65
+ Requires Chronicle 19.90.0 for concurrent .NET reactor/reducer partitions.
66
+
67
+ `ChronicleOptions.Reactors.MaxConcurrentPartitions` and
68
+ `ChronicleOptions.Reducers.MaxConcurrentPartitions` independently limit parallel
69
+ partitions per observer. Both default to processor count capped at 8 and require
70
+ at least 1. Set 1 for one-at-a-time handling if shared state is not thread-safe;
71
+ one partition's batches remain ordered. These are .NET client settings, not
72
+ kernel `observers` settings. Verified at **v19.110.1**:
73
+ `Source/Clients/DotNET/Reactors/ReactorOptions.cs`, `Reducers/ReducerOptions.cs`
74
+ and `Observation/PartitionedDeliveryScheduler.cs`.
@@ -57,3 +57,26 @@ configured, from bootstrap configuration (configured applications are recreated
57
57
  on startup). Changing the image or supplying new client credentials alone does
58
58
  not remove that old application. See **cratis-chronicle-hosting** for kernel
59
59
  certificates, private h2c and production setup.
60
+
61
+ ## Request completion ownership (19.110.1+ .NET packages)
62
+
63
+ Verified at **v19.110.1**: `Source/Clients/DotNET/Transactions/`
64
+ (`IUnitOfWorkCompletionOwnershipFeature.cs`, `IUnitOfWorkCompletionClaim.cs`,
65
+ `UnitOfWorkCompletionResult.cs`), `Source/Clients/AspNetCore/Transactions/UnitOfWorkMiddleware.cs`
66
+ and `Source/Clients/AspNetCore/ChronicleUnitOfWorkOptions.cs`.
67
+
68
+ Nested hosts can claim a request's completion with
69
+ `IUnitOfWorkCompletionOwnershipFeature.TryClaim`. Complete through the claim
70
+ and inspect its result before writing the response. An uncompleted claim is
71
+ rolled back by the outer middleware, never silently committed. This does not
72
+ grant arbitrary nested code the decision-read owner capability described in
73
+ **cratis-chronicle-decision-reads**. `UnitOfWorkCompletionOutcome.Unknown` means a
74
+ technical commit failure, not proof of rollback. Decision conflicts also occur
75
+ in `ConcurrencyViolations`; avoid reporting the same source's conflict twice.
76
+
77
+ Compatibility middleware logs a failed post-handler commit without changing a
78
+ response already sent. Opt in to problem responses **before headers start** with
79
+ `ChronicleAspNetCoreOptions.UnitOfWork.ReportFailuresBeforeResponseStarts`:
80
+ concurrency/decision conflict is 409, constraint violation is 400, other append
81
+ failure is 500. A started response cannot be repaired; complete through the
82
+ claim first when a success response must mean the write committed.
@@ -242,6 +242,12 @@ requirement is only to annotate the types.
242
242
  Key storage is per namespace and is a hosting decision, not an application one. Chronicle ships
243
243
  MongoDB (the default), SQL, in-memory, HashiCorp Vault, and Azure Key Vault key
244
244
  storage, configured under `Cratis:Chronicle:Compliance:Encryption`.
245
+ From 19.99.1, Azure Key Vault erasure **purges every key revision**, not just
246
+ soft-deletes it. The vault must grant purge permission and have purge protection
247
+ disabled; a refused purge fails erasure rather than reporting success. Do not
248
+ silently weaken the erasure contract to fit a protected vault. Verified at
249
+ **v19.110.1**, `Source/Kernel/Storage.AzureKeyVault/AzureKeyVaultEncryptionKeyStorage.cs`
250
+ and `Documentation/hosting/azure.md`.
245
251
 
246
252
  > **Operational trap.** Switching to a dedicated compliance key store without
247
253
  > setting `MigrateFromDefaultStorage` leaves every existing key unreachable.
@@ -158,10 +158,14 @@ A constraint binds every writer, whichever client or service appends, per namesp
158
158
  and per event sequence. It cannot count, sum or test a predicate; it expresses
159
159
  uniqueness (with release on a removal event), create-once and closed streams.
160
160
 
161
- - The claim is settled after the durable append (not yet implemented in every
162
- store). A failed index update is logged and a later duplicate may pass until a
163
- reindex. SQL and InMemory stores do not yet settle claims. A constraint is the
164
- right mechanism, not an unconditional proof.
161
+ - From Chronicle 19.82.2, a failed index update after a durable append marks a
162
+ gap. A later governed append repairs that gap from the event log or refuses
163
+ with a `ConstraintIndexUnavailable` error value before writing. Do not treat
164
+ that retryable refusal as a duplicate or remove the constraint. Verified at
165
+ **v19.110.1**, `Source/Kernel/Core/EventSequences/EventSequence.AppendSettlement.cs`
166
+ and `EventSequence.Validation.cs`; see
167
+ [append outcomes](../../cratis-chronicle-client-dotnet/references/append-outcomes.md)
168
+ for the distinct unknown-outcome case.
165
169
  - Tenants in different namespaces can claim the same value.
166
170
  - `[RemoveConstraint]` releases attribute constraints (`[Unique]` on a property or
167
171
  on the event type); on a fluent `IConstraint` it compiles and does nothing, so
@@ -17,11 +17,22 @@ A constraint binds **every** writer, whichever client or service appends, so it
17
17
  the mechanism for uniqueness and create-once. It cannot count, sum or test a
18
18
  predicate: capacity and stock rules need a guarded read (read protection `On`), a stream decision or an
19
19
  aggregate. Read [choosing how a rule holds](../cratis-chronicle-decision-reads/references/choosing-how-a-rule-holds.md)
20
- when it is unclear which one a rule needs. Limits: the claim is settled after the
21
- durable append, so a failed index update can let a later duplicate
22
- through until a reindex; constraints apply per event store namespace and event
23
- sequence, so tenants in different namespaces can claim the same value; and a
24
- constraint is not re-run when the kernel retries a duplicate sequence number.
20
+ when it is unclear which one a rule needs.
21
+
22
+ Requires Chronicle 19.82.2 for constraint-index gap repair.
23
+
24
+ A failed
25
+ post-append index update marks a gap: subsequent governed appends repair the
26
+ index from the event log or return `ConstraintIndexUnavailable` in the result's
27
+ error collection without writing. Do not remove the constraint or blindly retry an unknown
28
+ append outcome; see [append outcomes](../cratis-chronicle-client-dotnet/references/append-outcomes.md).
29
+ Constraints apply per event store namespace and event sequence, so tenants in
30
+ different namespaces can claim the same value. A constraint is not re-run when
31
+ the kernel retries a duplicate sequence number.
32
+
33
+ Verified in the tagged kernel sources listed by [append outcomes](../cratis-chronicle-client-dotnet/references/append-outcomes.md):
34
+ `Source/Kernel/Core/EventSequences/EventSequence.AppendSettlement.cs` and
35
+ `EventSequence.Validation.cs`.
25
36
 
26
37
  ## Verified product sources
27
38
 
@@ -48,8 +48,9 @@ operator/kernel surface, not an application-profile layout rule.
48
48
  development database ports on loopback only; a host application needing MongoDB
49
49
  must be able to reach port 27017. Do not expose unauthenticated MongoDB.
50
50
 
51
- Read references/storage-resolution.md when choosing per-namespace destinations,
52
- separate schema/runtime identities, SQL schemas, or disabling automatic creation.
51
+ Read [storage resolution](references/storage-resolution.md) when choosing
52
+ per-namespace destinations, schema/runtime identities, Azure authentication,
53
+ SQL schemas, or disabling automatic creation.
53
54
 
54
55
  ## Supply certificates before starting production
55
56
 
@@ -111,18 +112,29 @@ than increasing limits without accounting for memory.
111
112
 
112
113
  Set `health.port` for a dedicated HTTP/1.1 probe listener. `health.tls=false`
113
114
  removes TLS only there; `health.exclusive=true` restricts it to the configured
114
- health path (`/health` by default). Exclusive is **false by default**, so an
115
+ health path (`/health` by default) and its `/live` and `/ready` subpaths. Exclusive is **false by default**, so an
115
116
  ordinary plaintext probe port also exposes other HTTP/1.1 endpoints. Keep it
116
117
  internal. `health.tls=true` needs the top-level TLS certificate even when main
117
118
  TLS is disabled. The production chiseled container has no shell/curl: use an
118
119
  orchestrator/external probe, not an in-container shell healthcheck.
119
120
 
121
+ Requires Chronicle 19.109.0 for separate readiness/liveness routes.
122
+
123
+ `/health` and `/health/ready` remain readiness (including startup
124
+ and storage); use `/health/live` for liveness/startup so a replay backlog does
125
+ not restart a healthy-but-unready silo. A custom base path moves all three
126
+ routes. Verified in the tagged sources linked from [clustering and schema ownership](references/clustering-and-schema.md),
127
+ `Documentation/hosting/configuration/health-endpoint.md`
128
+ and `Source/Kernel/Server/KernelReadinessHealthCheck.cs`.
129
+
120
130
  ## Form one cluster, not several over shared data
121
131
 
122
- Every node must explicitly set `clustering.type=MongoDB` and share the cluster
123
- and service ids and MongoDB storage. Default `Localhost` forms isolated
124
- single-node clusters: sharing a database does not join them. Only MongoDB
125
- supports multi-node storage; PostgreSQL, SQL Server and SQLite are single-node.
132
+ Every node must select shared membership and share cluster/service ids and the
133
+ cluster database: `clustering.type=MongoDB` for MongoDB, or `Sql` for PostgreSQL
134
+ and SQL Server. SQLite remains single-node. Default `Localhost` does not join
135
+ nodes; persistent stores guard it with a single-writer lease. Read [clustering and schema ownership](references/clustering-and-schema.md)
136
+ for lease failure behavior, SQL migration/Verify modes and the disabled-by-default
137
+ projection-generation groundwork.
126
138
  `Clustering:ResponseTimeout` bounds Orleans transport waiting; increasing observer
127
139
  subscriber timeout cannot exceed that transport deadline.
128
140
 
@@ -132,12 +144,16 @@ and upgrading a cluster. See **cratis-chronicle-web-workbench** for browser
132
144
  exposure and **cratis-chronicle-cli-operations** for authorized live operations.
133
145
  Several services share one kernel with one event store each: see **cratis-multi-service-architecture**, and **cratis-multi-service-local-orchestration** for an AppHost or compose file.
134
146
 
147
+ Read [webhook delivery policy](references/webhooks.md) for target validation,
148
+ independent retries, managed identity and **real-delivery** quarantine probes.
149
+ Ordinary observer quarantine guidance does not imply webhooks stay silent.
150
+
135
151
  ## Verify
136
152
 
137
153
  - The build/image and intended environment agree; no default development
138
154
  application remains in reused production storage.
139
- - Storage is reachable and MongoDB is a replica set; multiple nodes share MongoDB
140
- membership, cluster id, service id and certificate ring.
155
+ - Storage is reachable and MongoDB is a replica set; multiple nodes share valid
156
+ MongoDB/Sql membership, cluster id, service id and certificate ring.
141
157
  - Certificates load; clients validate production TLS (`skipTlsValidation=false`).
142
158
  - A probe observes health without publishing a plaintext Workbench/API port.
143
159
  - Proxy upstream protocol and trust are tested from the actual immediate peer.
@@ -0,0 +1,84 @@
1
+ <!-- Copyright (c) Cratis. All rights reserved. -->
2
+ <!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
3
+
4
+ # Cluster membership, fencing and SQL schema ownership
5
+
6
+ Verified at Chronicle **v19.110.1**: `Documentation/hosting/configuration/`
7
+ (`clustering.md`, `storage.mdx`, `health-endpoint.md`),
8
+ `Source/Kernel/Core/Clustering/SingleWriterLeaseKeeper.cs`,
9
+ `Source/Kernel/Core/Configuration/{SchemaMode,SchemaAction,Features}.cs` and
10
+ `Source/Kernel/Server/KernelReadinessHealthCheck.cs`.
11
+
12
+ ## Choose membership for the storage
13
+
14
+ Requires Chronicle 19.105.0 for SQL multi-node membership and provider validation.
15
+
16
+ PostgreSQL and SQL Server support multi-node membership with
17
+ `clustering.type = Sql`; MongoDB uses `MongoDB`. Every node must share the
18
+ cluster database, `clusterId`, `serviceId` and certificates. SQL clustering with SQLite or a non-SQL provider fails at startup; it is not a
19
+ way to cluster SQLite. Sharing a database while using `Localhost` does not form
20
+ a cluster. Silo and gateway ports must remain reachable between nodes.
21
+
22
+ ## Localhost single-writer lease
23
+
24
+ Requires Chronicle 19.110.0 on every contender for the Localhost lease. With MongoDB
25
+ storage the lease requires MongoDB 5.0 or later, and it is on by default.
26
+
27
+ `Localhost` on persistent storage (MongoDB, PostgreSQL, SQL Server,
28
+ SQLite) takes the fixed `kernel` lease in the cluster database **before** Orleans
29
+ runtime startup. `InMemory` is process-local and takes no lease; MongoDB/Sql
30
+ membership do not use this guard. Different cluster/service ids are diagnostic,
31
+ not a way to bypass the fixed lease.
32
+
33
+ - `clustering.singleWriterLease.enabled` is automatic by default; disable it
34
+ only with another proven single-writer mechanism.
35
+ - `ttl` defaults to 30 seconds (minimum 5); `acquireTimeout` defaults to 2 minutes.
36
+ Zero timeout makes one attempt. A second node waits or refuses startup and
37
+ stays unready; it does not become a safe second writer.
38
+ - Failed renewal marks health unhealthy and starts shutdown. Failure to confirm
39
+ renewal within half the TTL hard-fences the process. Shutdown retains the
40
+ lease while silo work drains; a crash successor waits for expiry. Do not
41
+ delete a live lease to get another node started.
42
+ - Every contender must support the lease; an older binary does not take it.
43
+
44
+ ## Least-privilege SQL deployments
45
+
46
+ Requires Chronicle 19.109.0 for schema actions and Verify runtime mode.
47
+
48
+ `--migrate-only` (or
49
+ `Cratis__Chronicle__Storage__SchemaAction=MigrateAndExit`) applies SQL migrations
50
+ for the cluster, known stores/namespaces and jobs, then exits without serving.
51
+ Run it with the authorized schema identity before starting runtime-only nodes.
52
+ `--verify-schema` / `SchemaAction=VerifyAndExit` checks without changing schema;
53
+ repeatable `--event-store` and `--namespace` with migration pre-provision new
54
+ destinations. Exit codes: 0 success, 2 schema not current, 3 lock timeout, 4
55
+ invalid options/forbidden database creation, 1 other failure. MongoDB and
56
+ in-memory schema actions succeed without doing work; do not call that a SQL
57
+ migration check.
58
+
59
+ `Cratis__Chronicle__Storage__SchemaMode=Verify` performs no database/schema
60
+ creation or migrations. Missing schema at startup or a newly used namespace
61
+ fails with instructions to migrate. `Migrate` remains the default. Provision
62
+ new destinations before routing tenants to them; `createIfMissing=false` is
63
+ not equivalent to Verify and does not replace the migration step.
64
+
65
+ SQL schema-lock wait and recovered insert-collision metrics aid diagnosis;
66
+ readiness stays unhealthy until startup finishes. Do not restart a migrating
67
+ node using a readiness endpoint as liveness.
68
+
69
+ ## Projection generations are not enabled by these releases
70
+
71
+ The tagged sources provide guarded groundwork, not an instruction to enable it.
72
+ `Features.ProjectionGenerations` remains off by default. MongoDB generation
73
+ sink writes need majority transactions/fences, so startup refuses a standalone
74
+ MongoDB with the feature on. Materialization binding catalogs exist for
75
+ MongoDB and in-memory; SQL explicitly refuses that catalog. Generation-owned
76
+ collections refuse drop, rename or clear.
77
+
78
+ The per-store generation initializer is explicit and idempotent, records durable
79
+ receipts/blockers, and refuses adoption if it cannot prove exclusion; it is not
80
+ run automatically and does nothing with the feature disabled. Initialized stores
81
+ have a minimum kernel generation floor; older kernels and state reset are
82
+ refused. Do not recommend enabling the feature, deleting fences, or resetting
83
+ state to resolve an ordinary replay problem. Consult the tagged product sources
84
+ and accepted design before planning adoption.
@@ -26,18 +26,46 @@ clearing one does not clear the other. Inspect the cause before authorizing clea
26
26
  or retry. Subscriber timeout abandons waiting, not subscriber work: redelivery
27
27
  can overlap work that still runs, so side effects must remain idempotent.
28
28
 
29
+ Requires Chronicle 19.109.0 for abandoned-delivery holds.
30
+ Requires Chronicle 19.110.0 for webhook quarantine probes.
31
+
32
+ `observers.abandonedDeliveryHold` defaults to 300 seconds **after the delivery
33
+ deadline**. A positive value below `subscriberTimeout` uses that timeout instead;
34
+ zero or less disables the hold. Retry/catch-up/replay wait for a barrier on the
35
+ **original subscriber activation**, not a replacement client. An unavailable
36
+ original silo is not proof that its work drained. Late results are discarded;
37
+ a late reducer result cannot write to its sink.
38
+
39
+ The hold is bounded, not cancellation: user code might still run after expiry,
40
+ so external side effects must remain idempotent. Clearing persisted failure
41
+ history followed by restart loses that history. Inspect abandoned-delivery logs
42
+ and metrics before reducing the hold. Verified in
43
+ `Documentation/hosting/configuration/observers.md` and
44
+ `Source/Kernel/Core/Observation/Observer.DeliveryFences.cs`.
45
+
29
46
  Quarantine persists through grain deactivation. Watchdog ticks and catch-up/replay
30
47
  completion do not end it. An operator clear or a fresh subscription does;
31
48
  a kernel restart only helps observers that actually receive a fresh subscription.
32
49
  Event-store subscriptions remain quarantined until operator clear. Recovery can
33
- quarantine again if the cause remains.
50
+ quarantine again if the cause remains. Webhooks are an exception: eligible
51
+ quarantines can probe with a **real pending event** and
52
+ resume after success. Read [webhook policy](webhooks.md) before promising an
53
+ operator that quarantine prevents all traffic; webhook timeout cancellation
54
+ also drains the HTTP operation before starting another delivery.
34
55
 
35
56
  Projection replay and catch-up each use a **single globally ordered job step**;
36
57
  a new catch-up failure stops that step rather than letting other sources proceed.
37
58
  Full replay is inactive until completion, then routes to catch-up/live unless
38
59
  quarantined. Partition quarantine stops that partition's automatic retries;
39
60
  observer quarantine suppresses automatic recovery for the whole observer.
40
- Sources: `Documentation/concepts/observers.mdx`,
61
+ Projection replay skips a failed partition while continuing
62
+ healthy partitions, then schedules failed partitions for recovery after replay
63
+ completes. Catch-up still stops its ordered step on a new failure. Replaying
64
+ **global** projections is refused before any write (Requires Chronicle 19.101.3). For MongoDB
65
+ catch-up/bulk-write upgrade repairs, read
66
+ [projection definition changes](../../cratis-chronicle-projection/references/definition-changes.md).
67
+
68
+ Sources (verified through **v19.110.1**): `Documentation/concepts/observers.mdx`,
41
69
  `Documentation/projections/registration-lifecycle.md`, and
42
70
  `Source/Kernel/Core/Observation/Jobs/{CatchUpObserver,ReplayObserver}.cs`.
43
71
 
@@ -46,6 +46,36 @@ access tokens refreshed through resolution. Password-based MongoDB clients do
46
46
  not acquire a new password merely because configuration changes; restart them
47
47
  when rotating that credential.
48
48
 
49
+ ## Azure identity
50
+
51
+ Verified through **v19.110.1**: `Documentation/hosting/azure.md`,
52
+ `Documentation/hosting/configuration/storage-connection-resolution.md`,
53
+ `Source/Kernel/Core/Configuration/PostgreSqlEntra.cs` and
54
+ `Source/Kernel/Storage.Sql/Connections/PostgreSqlEntraAuthentication.cs`.
55
+
56
+ Requires Chronicle 19.96.0 for AzureCredential HTTP resolver authentication.
57
+
58
+ The HTTP resolver's `Authentication.Type=AzureCredential` supports
59
+ `Credential=ManagedIdentity` (default), `WorkloadIdentity` or `Default`, plus
60
+ `Scope` and optional `ClientId`. The resolver identity is not the database
61
+ identity. Prefer the specific production credential rather than accidentally
62
+ using a developer credential through the default chain.
63
+
64
+ Requires Chronicle 19.92.2 for native Azure SQL identity authentication.
65
+ Requires Chronicle 19.109.0 for native PostgreSQL Entra authentication.
66
+
67
+ Azure SQL uses Microsoft.Data.SqlClient's native authentication connection-string
68
+ settings (managed identity, workload identity, or default chain), not a
69
+ Chronicle-specific password flow. Native Entra authentication for
70
+ Azure Database for PostgreSQL uses `storage.postgreSqlEntra.enabled=true`, optional
71
+ `managedIdentityClientId`, and a PostgreSQL `Username` with **no password**.
72
+ Tokens refresh on pooled connections. Do not combine it with a resolver that
73
+ returns credentials; placement and schema resolution remain supported.
74
+
75
+ For separate least-privilege schema/runtime deployments, read
76
+ [clustering and schema ownership](clustering-and-schema.md): `--migrate-only`
77
+ and `SchemaMode=Verify` are distinct from `createIfMissing=false`.
78
+
49
79
  ## Verify
50
80
 
51
81
  Use the real target provider in integration specs: resolve two namespace
@@ -0,0 +1,110 @@
1
+ <!-- Copyright (c) Cratis. All rights reserved. -->
2
+ <!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
3
+
4
+ # Webhook delivery policy
5
+
6
+ ## Contents
7
+
8
+ - Validate the destination, not just the registration
9
+ - Independent delivery retries
10
+ - OAuth scope and token refresh
11
+ - Managed identity
12
+ - Quarantine probes are real deliveries
13
+
14
+ Verified at Chronicle **v19.110.1**: `Documentation/webhooks/index.mdx`,
15
+ `Source/Kernel/Core/Configuration/{Webhooks,WebhookManagedIdentity,WebhookProbe}.cs`,
16
+ `Source/Kernel/Core/Observation/Webhooks/` (including `Validation/`) and
17
+ `Source/Kernel/Server/Observation/Webhooks/ManagedIdentityCredentialFactory.cs`.
18
+ Configuration below sits under `Cratis:Chronicle:Webhooks`.
19
+
20
+ ## Validate the destination, not just the registration
21
+
22
+ Requires Chronicle 19.100.0 for webhook target validation.
23
+
24
+ `Validation` is `Warn` (default) or `Enforce`. Warn logs violations without
25
+ blocking, except unapproved cloud metadata addresses. Do not mistake the default
26
+ for full SSRF protection or approve metadata hosts in normal deployments.
27
+ Enforce requires HTTPS and approved hosts, checking private/special resolved
28
+ addresses at connect time too. Redirects are not followed. OAuth authorities use the same policy.
29
+ `AllowInsecureHttp` applies only in Development.
30
+
31
+ `AllowedHosts` accepts exact hosts and wildcard subdomains; an event store's
32
+ `EventStoreAllowedHosts` entry **replaces** the global list. An exact host entry
33
+ can intentionally admit that host's private addresses. Bound exceptions to
34
+ trusted receivers; wildcards approve subdomains, not their parent domain.
35
+ Never widen the list just to silence a refusal.
36
+
37
+ Requires Chronicle 19.106.0 for validation of stored active definitions.
38
+
39
+ Existing active definitions are checked on activation/restart.
40
+ Enforce quarantines a disallowed target without deleting/deactivating the
41
+ stored definition; Warn logs once per activation and continues. Fix the target
42
+ or approved policy, then clear quarantine: clearing revalidates, so it does not
43
+ bypass a bad target. A connect-time policy failure is a configuration problem,
44
+ not a transient failure retried for 24 hours.
45
+
46
+ ## Independent delivery retries
47
+
48
+ Requires Chronicle 19.104.0 for webhook-specific retry policy.
49
+
50
+ Each delivery makes one HTTP attempt. `Retry` is independent of global observer
51
+ settings: delays are 1 s, 5 s, 30 s, 2 min, 10 min, 30 min, then hourly, with
52
+ ±20% jitter. The budget is 24 hours, then the partition is parked. Five parked
53
+ partitions or at least half the failing partitions parked quarantines the
54
+ observer. Override the schedule, maximum delay, attempts, budget and quarantine
55
+ thresholds only with a bounded recovery policy. `RetryDelaySeconds` is retained
56
+ but has no effect.
57
+
58
+ Retry deadlines/counts/positions survive restart; manual partition retry overrides
59
+ a pending deadline. A per-target (scheme/host/port) circuit defers deliveries
60
+ without consuming attempts. `429`/`503` with valid `Retry-After` defer without
61
+ consuming attempts, capped by `MaximumRetryAfterSeconds` (default 24 hours).
62
+ `410` parks immediately. Other `4xx` except `408` and `429` quarantine as
63
+ configuration failures. Catch-up/replay respect the pending deadline.
64
+
65
+ ## OAuth scope and token refresh
66
+
67
+ Requires Chronicle 19.98.0 for OAuth scope, token reuse and one-shot 401 refresh.
68
+
69
+ The .NET webhook builder accepts
70
+ `WithOAuth(authority, clientId, clientSecret, scope)`; scope is optional, and
71
+ re-registering without it clears a previously configured scope. Entra client
72
+ credentials need the receiver's `api://<application-id>/.default` scope. Tokens
73
+ are shared by token endpoint/client/scope/credentials and refreshed before
74
+ expiry; tokens lacking expiry are not cached. A 401 refreshes and retries the
75
+ same request once; other responses do not trigger an OAuth retry. This is
76
+ separate from the observer retry policy. Never put credentials in examples or
77
+ logs. Verified in `Source/Clients/DotNET/Webhooks/IWebhookDefinitionBuilder.cs` and
78
+ the tagged webhook documentation.
79
+
80
+ ## Managed identity
81
+
82
+ Requires Chronicle kernel 19.110.0 and .NET packages 19.110.1 for managed identity webhooks.
83
+
84
+ System-assigned/user-assigned Azure managed identity and AKS workload identity
85
+ can authorize a webhook without storing a client secret. The operator must
86
+ allow exact scopes in `ManagedIdentity.AllowedScopes`; the empty default
87
+ disables this. Non-empty client ids must be in `AllowedClientIds`; an empty id
88
+ needs `AllowSystemAssigned=true`. The receiver must explicitly match the host
89
+ allow-list **even in Warn mode**; delivery rechecks approval before acquiring a
90
+ token. This lends the **kernel's** identity, not the registering client's. Its
91
+ principal still needs the receiver's application role. Register with the .NET
92
+ builder's `WithManagedIdentity(scope, clientId)` (client id optional). Tokens are
93
+ memory-only and refreshed before expiry. A 401 discards the cached token and
94
+ its credential instance, then retries once; Azure's own cache means a different
95
+ token is not guaranteed. Never log tokens. Verified in
96
+ `Documentation/hosting/azure.md` and `Source/Clients/DotNET/Webhooks/IWebhookDefinitionBuilder.cs`.
97
+
98
+ ## Quarantine probes are real deliveries
99
+
100
+ Requires Chronicle 19.110.0 for automatic webhook quarantine probes.
101
+
102
+ `Probe.Enabled` defaults to true. Retry-budget or delivery/configuration
103
+ quarantines can probe with exponential jittered backoff (initial 60 s, maximum
104
+ 1 hour, timeout 30 s); manual quarantines and parked partitions are not
105
+ probed. A probe sends the **first pending event** without clearing quarantine
106
+ first. Success records it as handled, clears quarantine and resumes in order.
107
+ Failure leaves quarantine intact. Probes revalidate targets and consume neither
108
+ normal retry budget nor circuit samples. Set `Probe.Enabled=false` when
109
+ quarantined receivers must get no automatic traffic. Receivers must remain
110
+ idempotent; a probe is not a read-only health check.
@@ -13,6 +13,18 @@ when independence cannot be proved. Counters and joins are repaired by replay,
13
13
  not by deploying corrected mappings: earlier double-counts remain until rebuilt.
14
14
  A `FromAll` subscription change redefines existing projections.
15
15
 
16
+ Adding mappings for event types that already have history requires
17
+ **full** replay: `Automatic` rebuilds; `PartialOnly` and `Manual` recommend it.
18
+ The kernel no longer replays those mappings onto live state, where watermarks
19
+ could discard historical writes or counters could count twice. No history for
20
+ the added types means no action. Explicit partition replay is unchanged.
21
+ Verified in `Source/Kernel/Core/Projections/Projection.cs`
22
+ (`EvolveForAllNamespaces`).
23
+
24
+ Requires Chronicle 19.101.3 for explicit refusal of global projection replay.
25
+
26
+ Global projection replay (full or partition) is refused before writing. Do not promise to repair globally scoped state by replay.
27
+
16
28
  Projection replay and catch-up each walk the sequence in **one globally ordered
17
29
  job step**, not parallel per-source steps. A new failure during catch-up stops
18
30
  that ordered step even if other sources are healthy. Full replay keeps the
@@ -49,5 +61,26 @@ revives a retired observer, or restarts a stopped/deleted capture. A kernel
49
61
  restart or a newly added namespace still ends projection quarantine through a
50
62
  fresh subscription; do not interpret ordinary watchdog repair as operator clear.
51
63
 
64
+ ## Upgrade repair checks
65
+
66
+ Verified through **v19.110.1**:
67
+ `Source/Kernel/Storage.MongoDB/Sinks/Sink.cs`,
68
+ `Source/Kernel/Core/Projections/Engine/KeyResolvers.ParentHistory.cs` and the
69
+ tagged release notes:
70
+
71
+ Requires Chronicle 19.110.0 for MongoDB resumed bulk-window preservation, replaced-window fencing and historical parent-key resolution.
72
+
73
+ - MongoDB catch-ups resumed while clients restarted/re-registered could lose
74
+ unflushed writes on older kernels. Upgrade, then replay affected
75
+ projections, including catch-ups with `BulkWindowLost` or completed failures;
76
+ silent loss may leave no log entry. The fixed sink preserves the window,
77
+ refuses a window replaced before the first checkpoint and
78
+ prevents queued writes crossing replay/catch-up collections.
79
+ - With content-keyed parents, children rebuilt before this fix may be under the
80
+ wrong parent. Replay after upgrading; key lookup now uses the child's event
81
+ position rather than the parent's latest key.
82
+ - These are kernel-backed storage/replay behaviors. In-process projection
83
+ scenarios do not prove bulk durability or historical parent-key resolution.
84
+
52
85
  Closing a stream does not erase its events or prevent a projection replay.
53
86
  Keep the closing fact as history; see **cratis-chronicle-event-constraints**.
@@ -55,6 +55,12 @@ model-bound `[FromEvery]` has no opt-out. `FromAll` can count into dictionary
55
55
  properties with a per-event context key (`Projections/FromEveryBuilder.cs`,
56
56
  `IFromEveryBuilder.cs`, `Documentation/projections/declarative/`).
57
57
 
58
+ Requires Chronicle 19.88.0 for all-event constant mappings.
59
+
60
+ Fluent `FromEvery` and `FromAll` property mappings support `ToValue(...)` for
61
+ concepts, enums and `null`, like per-event `Set(...).ToValue(...)`. Verified at
62
+ **v19.110.1**, `Projections/AllSetBuilder.cs` and `IAllSetBuilder.cs`.
63
+
58
64
  ## Inherited from `IProjectionBuilder<TReadModel, TBuilder>`
59
65
 
60
66
  ```csharp
@@ -83,6 +89,12 @@ Chaining `.NotRewindable()`, `.Passive()`, `.ContainerName()`, or
83
89
  `RemovedWith` and `RemovedWithJoin` hand you the **concrete**
84
90
  `RemovedWithBuilder<,>` and `RemovedWithJoinBuilder<,>` classes, not interfaces.
85
91
 
92
+ Requires Chronicle 19.93.0 for the root-removal-through-join startup warning.
93
+
94
+ `RemovedWithJoin` removes **children**, not a root read model. The .NET client warns at startup about root removal through a join (fluent,
95
+ model-bound or explicit); it still removes nothing. Use `RemovedWith` for the
96
+ root. Verified in `Projections/RootRemovalViaJoin.cs` at the tag above.
97
+
86
98
  ## Keys live on the per-event builder
87
99
 
88
100
  `From<TEvent>` hands the callback an `IFromBuilder<TReadModel, TEvent>`, which
@@ -239,6 +239,18 @@ Read [references/domain-failures.md](references/domain-failures.md) when handlin
239
239
  Inject through the primary constructor; hold no mutable state (CHR0031 warning).
240
240
  Keep one automation concern per reactor; several related handlers are fine.
241
241
 
242
+ Requires Chronicle 19.90.0 for concurrent .NET reactor partitions.
243
+
244
+ Different partitions of one .NET reactor run concurrently;
245
+ batches for one partition retain arrival order. Shared collaborators must be
246
+ thread-safe. Set `ChronicleOptions.Reactors.MaxConcurrentPartitions = 1` when
247
+ one-at-a-time handling is required; the default is processor count capped at 8,
248
+ and values below 1 are refused. This is not ordering across different observers.
249
+ Verified in the tagged sources listed in
250
+ [connection options](../cratis-chronicle-client-dotnet/references/connection-options.md):
251
+ `Source/Clients/DotNET/Reactors/ReactorOptions.cs`,
252
+ `Reactors.cs` and `Observation/PartitionedDeliveryScheduler.cs`.
253
+
242
254
  ## Specifications
243
255
 
244
256
  Start with `Specification` plus a direct call for pure reactor decisions; add `ReactorScenario<TReactor>` only where invocation, dependency wiring or side effects contribute proof. Specify the contract, not only the happy path: field lineage, contract-authorized filtering, repeated delivery, and unrelated events.
@@ -14,6 +14,19 @@ analyzers flag synchronous custom returns even when the runtime handler accepts
14
14
  Return events rather than inject `IEventLog`: **a convention, not an analyzer
15
15
  contract**. Returned appends stay inside Chronicle's transaction.
16
16
 
17
+ Requires Chronicle 19.94.0 for receipt-keyed returned events.
18
+
19
+ Returned events on receipt-capable kernels have stable effect keys: retries, crash redelivery and same-generation replay do not append them
20
+ again. This covers bare, cross-source and mixed returned-event results, not
21
+ arbitrary HTTP calls or independently injected `IEventLog` appends. Older kernels
22
+ retain the previous at-least-once behavior. Preserve deterministic return
23
+ ordering/content; a changed result is a conflict, not a reason to discard the
24
+ receipt. Keep `ReactorDelivery`-based idempotency for external side effects and
25
+ `[OnceOnly]` for deliberate replay exclusion. Verified at **v19.110.1**:
26
+ `Source/Clients/DotNET/Reactors/SideEffects/ReactorEffectKeys.cs`,
27
+ `EventResultHandler.cs`, `MixedSideEffectsResultHandler.cs` and
28
+ `Documentation/reactors/side-effects.mdx`.
29
+
17
30
  ```csharp
18
31
  // One event, on the triggering event's own event source.
19
32
  public Task<<ResultEvent>> <MethodName>(<EventName> @event, EventContext context) =>
@@ -75,7 +75,9 @@ public record <ReadModelName>(
75
75
  reserves those for compliance-subject tracking, and analyzer `CHR0035`
76
76
  rejects them.
77
77
  - `Cratis.Chronicle.ReadModels.IndexAttribute` (`[Index]`) marks a property for
78
- indexing in the sink.
78
+ indexing in the sink, including SQL. `[CompositeIndex]` on the class declares ordered multi-property indexes and
79
+ `[Index(IsDescending = true)]` declares descending ones. Read
80
+ [indexing](references/indexing.md) for valid paths, flags and store-specific limits.
79
81
 
80
82
  ### Shape the model for its consumer
81
83
 
@@ -0,0 +1,44 @@
1
+ <!-- Copyright (c) Cratis. All rights reserved. -->
2
+ <!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
3
+
4
+ # Read-model indexes
5
+
6
+ Verified against the tagged `Source/Clients/DotNET/ReadModels/IndexAttribute.cs`,
7
+ `CompositeIndexAttribute.cs` and
8
+ [indexing documentation](https://github.com/Cratis/Chronicle/blob/v19.110.1/Documentation/read-models/indexing.mdx).
9
+
10
+ Requires Chronicle 19.103.0 for SQL indexes.
11
+ Requires Chronicle 19.105.0 for SQL Server string index length bounds.
12
+ Requires Chronicle kernel 19.110.0 for SQL nested-path indexes and PostgreSQL GIN
13
+ collection indexes.
14
+ Requires Chronicle .NET packages 19.110.1 for composite/descending indexes and an
15
+ index for every use of a reused nested type.
16
+
17
+ Import `Cratis.Chronicle.ReadModels`.
18
+
19
+ ```csharp
20
+ [ReadModel]
21
+ [CompositeIndex(nameof(CustomerId), nameof(Created), IsDescending = new[] { false, true })]
22
+ public record OrdersByCustomer(
23
+ Guid Id,
24
+ [property: Index] Guid CustomerId,
25
+ [property: Index(IsDescending = true)] DateTimeOffset Created);
26
+ ```
27
+
28
+ - `[Index]` targets a property or record parameter; `[CompositeIndex]` targets
29
+ a class and is repeatable. Composite property order matters.
30
+ - Dotted paths name nested properties. `IsDescending` is one flag on `[Index]`,
31
+ an array on `[CompositeIndex]`; omit the array for all ascending or provide
32
+ one flag per property. Empty/duplicate/unknown paths and mismatched flag
33
+ counts are refused at registration. Indexes are not uniqueness constraints.
34
+ Use an event constraint for a value that must be unique at append time;
35
+ a read-side index is not a write-side invariant.
36
+ - MongoDB and SQL create indexes at read-model registration. PostgreSQL uses
37
+ GIN for collection paths, which cannot be combined in a composite index with
38
+ ordinary fields. SQL Server and SQLite skip collection indexes with a warning.
39
+ - SQL Server indexed string columns honor `MaxLength`, `StringLength` or
40
+ `Length`; otherwise the bound is 450 characters. Overlong writes fail the
41
+ projection partition. Replay existing read models to apply bounded columns.
42
+ - Indexes on reused nested types are expanded for every path, not
43
+ just the first occurrence. Test the actual store's query behavior and inspect
44
+ its schema; an in-process scenario does not prove physical indexes exist.
@@ -153,6 +153,10 @@ identifier properties populate; when one does not, investigate the projection
153
153
  rather than the harness. Skip only on a reproduced harness gap, and put the
154
154
  specific reason in the skip message.
155
155
 
156
+ For pre-seeding instances, `ProcessAppendedEvents(...)` and schema-generator
157
+ sharing, read [seeded state and appended-event processing](references/seeded-state.md).
158
+ These are setup/processing surfaces, not proof of kernel persistence or replay.
159
+
156
160
  ## Step 4 — Know what the scenario picked
157
161
 
158
162
  `WithProjection` overrides discovery, including a reducer. Otherwise:
@@ -0,0 +1,35 @@
1
+ <!-- Copyright (c) Cratis. All rights reserved. -->
2
+ <!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
3
+
4
+ # Seeded state and appended-event processing
5
+
6
+ Requires Chronicle 19.83.0 for seeded state; verified against v19.110.1 sources.
7
+
8
+ Verified in `Source/Clients/Testing/ReadModels/`
9
+ (`ReadModelSourceGivenBuilder.cs`, `ReadModelScenario.cs`) and
10
+ `Source/Clients/Testing/Events/EventStoreForTesting.cs`.
11
+
12
+ `await scenario.Given.ForEventSourceId(id).ReadModel(instance)`
13
+ sets initial state for both `InstanceForEventSourceId(id)` and
14
+ `scenario.ReadModels.GetInstanceById<T>(id)`; it does not create event history.
15
+ Use it for handler setup, not as proof that a projection built the instance.
16
+
17
+ Requires Chronicle 19.82.0 for appended-event processing with real contexts.
18
+
19
+ `ProcessAppendedEvents(...)` folds `AppendedEvent` values rather
20
+ than appending them a second time, preserving their real `EventContext`
21
+ (sequence number and occurred time) and sorting across sources by sequence
22
+ number. Use distinct sequence numbers. Bare events supplied through `Given`
23
+ or `ProcessEventsFor` still use synthetic contexts. With seeded instances,
24
+ the fold's results replace the scenario's `ReadModels` instances; this does
25
+ not prove kernel storage or replay durability. Seeding the same key twice
26
+ throws `ReadModelAlreadySeeded`; models with no projection/reducer keep
27
+ lookup-only behavior.
28
+
29
+ Requires Chronicle 19.106.0 for shared default test-store schema generation.
30
+
31
+ Default test stores share a JSON schema generator per process,
32
+ while each store gets a separate schema copy. Set
33
+ `EventStoreForTesting.ShareSchemaGenerator = false` for per-store metadata
34
+ providers; custom service providers already bypass sharing. Configure the
35
+ static switch at suite setup, not concurrently between specifications.
@@ -245,6 +245,20 @@ same store catalog: `AmbiguousReadModelBuilder` rejects it. `[Passive]`, sequenc
245
245
  selection, filters and `[Reducer]` alone do not establish projection ownership.
246
246
  Remove the old builder when switching; registration is not a data migration.
247
247
 
248
+ ## Partition delivery
249
+
250
+ Requires Chronicle 19.90.0 for concurrent .NET reducer partitions.
251
+
252
+ Different partitions of one .NET reducer can run at the
253
+ same time; batches for the same partition remain in arrival order. Keep shared
254
+ collaborators thread-safe, or set
255
+ `ChronicleOptions.Reducers.MaxConcurrentPartitions = 1`. The default is processor
256
+ count capped at 8; values below 1 are refused. A global ordering assumption is
257
+ not justified by per-partition ordering. Verified in the tagged sources listed
258
+ by [connection options](../cratis-chronicle-client-dotnet/references/connection-options.md):
259
+ `Source/Clients/DotNET/Reducers/ReducerOptions.cs`, `Reducers.cs` and
260
+ `Observation/PartitionedDeliveryScheduler.cs`.
261
+
248
262
  ## Replay
249
263
 
250
264
  Read [references/replay-hooks-and-definition-changes.md](references/replay-hooks-and-definition-changes.md) when a self model needs `BeginReplay`/`EndReplay` hooks (they return exactly `Task`; do not assume delivery is exactly once) or when a code, key or filter change must replay (`observers.definitionEvolution`). Requires Chronicle 19.29.4 for method-fingerprint change detection; assembly versions are ignored.
@@ -33,3 +33,10 @@ partial replay only when history proves they follow all previously consumed
33
33
  events per source. Interleaved history requires full replay. Manual policies
34
34
  leave recommendations; they do not silently rebuild. Source:
35
35
  `Documentation/hosting/configuration/observers.md`.
36
+
37
+ At **v19.110.1**, replay writes reduced state to a separate sink and switches to
38
+ it only when replay completes. Abandoning a failed reducer replay does not
39
+ publish partially rebuilt state. This is a kernel sink boundary, not proof that
40
+ external side effects are reversed. Verified in
41
+ `Source/Kernel/Core/Observation/Jobs/ReplayObserver.cs` and the tagged observer
42
+ configuration documentation.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@cratis/pi",
3
- "version": "7.114.0",
3
+ "version": "7.115.0",
4
4
  "description": "Configuration-aware Cratis AI integration for Pi",
5
5
  "type": "module",
6
6
  "files": [