@cratis/pi 7.114.0 → 7.115.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/package/corpus/harnesses/copilot/instructions/reactors.instructions.md +1 -1
- package/package/corpus/product-sync.json +4 -4
- package/package/corpus/rules/reactors.md +1 -1
- package/package/corpus/skills/cratis-chronicle-client-dotnet/SKILL.md +4 -0
- package/package/corpus/skills/cratis-chronicle-client-dotnet/references/append-outcomes.md +78 -0
- package/package/corpus/skills/cratis-chronicle-client-dotnet/references/connection-options.md +13 -0
- package/package/corpus/skills/cratis-chronicle-client-dotnet/references/hosting.md +23 -0
- package/package/corpus/skills/cratis-chronicle-compliance/SKILL.md +6 -0
- package/package/corpus/skills/cratis-chronicle-decision-reads/references/choosing-how-a-rule-holds.md +8 -4
- package/package/corpus/skills/cratis-chronicle-event-constraints/SKILL.md +16 -5
- package/package/corpus/skills/cratis-chronicle-hosting/SKILL.md +25 -9
- package/package/corpus/skills/cratis-chronicle-hosting/references/clustering-and-schema.md +84 -0
- package/package/corpus/skills/cratis-chronicle-hosting/references/operations.md +30 -2
- package/package/corpus/skills/cratis-chronicle-hosting/references/storage-resolution.md +30 -0
- package/package/corpus/skills/cratis-chronicle-hosting/references/webhooks.md +110 -0
- package/package/corpus/skills/cratis-chronicle-projection/references/definition-changes.md +33 -0
- package/package/corpus/skills/cratis-chronicle-projection/references/fluent-builder.md +12 -0
- package/package/corpus/skills/cratis-chronicle-reactor/SKILL.md +12 -0
- package/package/corpus/skills/cratis-chronicle-reactor/references/return-event-metadata.md +13 -0
- package/package/corpus/skills/cratis-chronicle-read-model/SKILL.md +3 -1
- package/package/corpus/skills/cratis-chronicle-read-model/references/indexing.md +44 -0
- package/package/corpus/skills/cratis-chronicle-read-model-specifications/SKILL.md +4 -0
- package/package/corpus/skills/cratis-chronicle-read-model-specifications/references/seeded-state.md +35 -0
- package/package/corpus/skills/cratis-chronicle-reducer/SKILL.md +14 -0
- package/package/corpus/skills/cratis-chronicle-reducer/references/replay-hooks-and-definition-changes.md +7 -0
- package/package.json +1 -1
|
@@ -225,7 +225,7 @@ rejects Warning-level reactor/event-type drift (not Information-level read-model
|
|
|
225
225
|
findings). Each reactor class should have a focused purpose. Multiple handler methods in one reactor are fine if they serve the same automation concern.
|
|
226
226
|
5. **Failure behavior** — If a reactor throws, *or* a returned side-effect event fails to append (constraint violation, concurrency violation, or error), the failing event-source partition pauses until the issue is resolved. Repeated failures can **quarantine** the observer: once `QuarantineOnFailedPartitionCount`/`QuarantineOnFailedPartitionPercentage` (under `Observers`) is crossed, the observer enters the `Quarantined` state — reminders cancelled, retries stopped, automatic recovery suppressed. **A quarantine ends when an operator clears it (`ClearObserverQuarantine()`), or when the observer is subscribed again** — for an application observer that is when the client connects again, for example after a redeploy; Kernel-owned observers are subscribed again when the Kernel starts. A periodic watchdog (default 60s, `WatchdogInterval`) re-routes stuck/dead observers but does not rescue quarantined ones. Design for resilience.
|
|
227
227
|
6. **Don't throw to validate malformed inbound events** — reactors are not data-quality validators; invalid payloads must be rejected at the command/append site. When a malformed cross-service fact reaches a consumer, throwing just to reject it pauses the partition and can quarantine the whole observer. Instead append a clear failure/dead-letter event (e.g. `ProvisioningFailed`) or surface it via the operational failure path, and skip partial side effects.
|
|
228
|
-
7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class.
|
|
228
|
+
7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class. From Chronicle 19.90.0, the .NET client runs different partitions concurrently (one partition's batches stay ordered); shared dependencies must be thread-safe, or set `ChronicleOptions.Reactors.MaxConcurrentPartitions = 1`. See **cratis-chronicle-reactor** for verified options and limits.
|
|
229
229
|
8. **A new side-effect reactor must not act on history.** A reactor that is new in a release (or has a new id) catches up from the start of the event log when it first registers, and `[OnceOnly]` does not cover that first catch-up because it is not a replay. If its handlers cause side effects (jobs, AI calls, external calls, notifications), start the observer at the tail on first registration or make the handler ignore facts that predate its registration, and specify that history causes nothing.
|
|
230
230
|
9. **An unattended, irreversible pass reads the log, not the sink — and carries a fuse.** A reactor that deletes, revokes, bills or notifies *because a read model says something is absent* is trusting a materialized sink that is only as current as its observer: a paused partition, a replay in progress or a lagging projection makes "absent" true of the sink and false of the world. Decide from a **`[Passive]`** read model (computed on demand from the events at the moment the reactor runs) or from the events themselves. And the code shape cannot tell an **empty subject set** ("nothing to consider" — fail-safe) from an **empty qualifying set** ("nothing survived the filter" — fail-destructive: every subject is about to be acted on), so an unattended pass states its expected population, refuses when the count is implausible, and caps how many subjects one run may touch — the same discipline as [guards-and-fuses.md](./guards-and-fuses.instructions.md), applied to a reactor. A sweep that decides from a read model also refuses while that model's observer is replaying or behind, because a rebuilding model reads as "nothing was ever processed"; specify the replaying case.
|
|
231
231
|
10. **Provider clients stay behind a translation or adapter reactor.** A `[Command]` or `IReactor` in a domain slice that injects or imports a provider client (GitHub, payment, mail SDK) is a finding: put the client behind one adapter slice and let domain slices work with our events and commands. Judgment-shaped today; a candidate for a namespace-boundary analyzer.
|
|
@@ -284,9 +284,9 @@
|
|
|
284
284
|
}
|
|
285
285
|
],
|
|
286
286
|
"synced": {
|
|
287
|
-
"version": "19.
|
|
288
|
-
"date": "2026-10-
|
|
289
|
-
"by": "Cratis/AI#
|
|
287
|
+
"version": "19.110.1",
|
|
288
|
+
"date": "2026-10-25",
|
|
289
|
+
"by": "Cratis/AI#820"
|
|
290
290
|
},
|
|
291
291
|
"covers": {
|
|
292
292
|
"skills": [
|
|
@@ -323,7 +323,7 @@
|
|
|
323
323
|
],
|
|
324
324
|
"rules": []
|
|
325
325
|
},
|
|
326
|
-
"notes": "Reviewed releases 19.
|
|
326
|
+
"notes": "Reviewed all 60 stable releases 19.82.0-19.110.1 against tagged v19.110.1 source: append uncertainty and constraint-index repair, .NET keyed appends and multi-scope guards, request completion ownership, concurrent reactor/reducer partitions and returned-event receipts, projection replay/upgrade repairs, seeded scenarios, SQL/composite indexes, SQL membership/schema ownership, Localhost single-writer fencing, health probes, Azure identity/Key Vault purge, webhook target validation/retries/managed identity/real-delivery probes, and disabled-by-default projection-generation groundwork. 19.110.0 .NET packages were not published; use 19.110.1. Other covered skills were reviewed for release relevance without mechanical changes. Existing protected-read guidance retained. No new kernel/transport probe or capability."
|
|
327
327
|
},
|
|
328
328
|
{
|
|
329
329
|
"id": "arc-go",
|
|
@@ -224,7 +224,7 @@ rejects Warning-level reactor/event-type drift (not Information-level read-model
|
|
|
224
224
|
findings). Each reactor class should have a focused purpose. Multiple handler methods in one reactor are fine if they serve the same automation concern.
|
|
225
225
|
5. **Failure behavior** — If a reactor throws, *or* a returned side-effect event fails to append (constraint violation, concurrency violation, or error), the failing event-source partition pauses until the issue is resolved. Repeated failures can **quarantine** the observer: once `QuarantineOnFailedPartitionCount`/`QuarantineOnFailedPartitionPercentage` (under `Observers`) is crossed, the observer enters the `Quarantined` state — reminders cancelled, retries stopped, automatic recovery suppressed. **A quarantine ends when an operator clears it (`ClearObserverQuarantine()`), or when the observer is subscribed again** — for an application observer that is when the client connects again, for example after a redeploy; Kernel-owned observers are subscribed again when the Kernel starts. A periodic watchdog (default 60s, `WatchdogInterval`) re-routes stuck/dead observers but does not rescue quarantined ones. Design for resilience.
|
|
226
226
|
6. **Don't throw to validate malformed inbound events** — reactors are not data-quality validators; invalid payloads must be rejected at the command/append site. When a malformed cross-service fact reaches a consumer, throwing just to reject it pauses the partition and can quarantine the whole observer. Instead append a clear failure/dead-letter event (e.g. `ProvisioningFailed`) or surface it via the operational failure path, and skip partial side effects.
|
|
227
|
-
7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class.
|
|
227
|
+
7. **No state** — Reactors should be stateless. Inject dependencies via primary constructor, but do not store mutable state on the class. From Chronicle 19.90.0, the .NET client runs different partitions concurrently (one partition's batches stay ordered); shared dependencies must be thread-safe, or set `ChronicleOptions.Reactors.MaxConcurrentPartitions = 1`. See **cratis-chronicle-reactor** for verified options and limits.
|
|
228
228
|
8. **A new side-effect reactor must not act on history.** A reactor that is new in a release (or has a new id) catches up from the start of the event log when it first registers, and `[OnceOnly]` does not cover that first catch-up because it is not a replay. If its handlers cause side effects (jobs, AI calls, external calls, notifications), start the observer at the tail on first registration or make the handler ignore facts that predate its registration, and specify that history causes nothing.
|
|
229
229
|
9. **An unattended, irreversible pass reads the log, not the sink — and carries a fuse.** A reactor that deletes, revokes, bills or notifies *because a read model says something is absent* is trusting a materialized sink that is only as current as its observer: a paused partition, a replay in progress or a lagging projection makes "absent" true of the sink and false of the world. Decide from a **`[Passive]`** read model (computed on demand from the events at the moment the reactor runs) or from the events themselves. And the code shape cannot tell an **empty subject set** ("nothing to consider" — fail-safe) from an **empty qualifying set** ("nothing survived the filter" — fail-destructive: every subject is about to be acted on), so an unattended pass states its expected population, refuses when the count is implausible, and caps how many subjects one run may touch — the same discipline as [guards-and-fuses.md](./guards-and-fuses.md), applied to a reactor. A sweep that decides from a read model also refuses while that model's observer is replaying or behind, because a rebuilding model reads as "nothing was ever processed"; specify the replaying case.
|
|
230
230
|
10. **Provider clients stay behind a translation or adapter reactor.** A `[Command]` or `IReactor` in a domain slice that injects or imports a provider client (GitHub, payment, mail SDK) is a finding: put the client behind one adapter slice and let domain slices work with our events and commands. Judgment-shaped today; a candidate for a namespace-boundary analyzer.
|
|
@@ -225,6 +225,10 @@ if (!result.IsSuccess)
|
|
|
225
225
|
several event sources in one batch.
|
|
226
226
|
- **There is no `AppendAnonymous`.** It does not exist anywhere in the product.
|
|
227
227
|
|
|
228
|
+
For unknown outcomes, `AppendOnce` / `AppendManyOnce`, keyed retry conflicts and
|
|
229
|
+
several concurrency guards for one source, read
|
|
230
|
+
[append outcomes and retry safety](references/append-outcomes.md).
|
|
231
|
+
|
|
228
232
|
Reading back is on `IEventSequence`: `GetForEventSourceIdAndEventTypes`,
|
|
229
233
|
`HasEventsFor`, `GetFromSequenceNumber`, `GetNextSequenceNumber`, and
|
|
230
234
|
`GetTailSequenceNumber`. Redaction is `Redact`; subject erasure is a separate
|
|
@@ -0,0 +1,78 @@
|
|
|
1
|
+
<!-- Copyright (c) Cratis. All rights reserved. -->
|
|
2
|
+
<!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
|
|
3
|
+
|
|
4
|
+
# Append outcomes and retry safety
|
|
5
|
+
|
|
6
|
+
Verified against the tagged sources in `Source/Clients/DotNET/EventSequences/`
|
|
7
|
+
(`IEventSequence.cs`, `EventSequence.cs`, `IdempotencyKey.cs`, `AppendResult.cs`,
|
|
8
|
+
`AppendManyResult.cs`, `AppendError.cs`) and
|
|
9
|
+
[appending events](https://github.com/Cratis/Chronicle/blob/v19.110.1/Documentation/events/appending.mdx).
|
|
10
|
+
These are .NET client APIs; do not assume other SDKs expose the same methods.
|
|
11
|
+
|
|
12
|
+
## Unknown is not failure-before-write
|
|
13
|
+
|
|
14
|
+
Requires Chronicle 19.82.2 for the distinct unknown-outcome and index-unavailable errors.
|
|
15
|
+
Requires Chronicle 19.108.0 for node-loss/timeout unknown-outcome handling.
|
|
16
|
+
|
|
17
|
+
`AppendError.AppendOutcomeUnknown` means the kernel cannot confirm whether the
|
|
18
|
+
events were stored, including an in-flight append interrupted by node loss or
|
|
19
|
+
a grain-call timeout. Do not retry an
|
|
20
|
+
unkeyed append blindly: read the source's events for the attempt's correlation
|
|
21
|
+
id, and append again only when none belongs to that attempt.
|
|
22
|
+
|
|
23
|
+
An `Errors` entry with value `ConstraintIndexUnavailable` means a governing unique/closing index
|
|
24
|
+
still needs repair and **nothing was stored**. Retry unchanged after a short
|
|
25
|
+
delay. The kernel repairs constraint indexes from the event log; restarting
|
|
26
|
+
clients or removing constraints is not a repair strategy.
|
|
27
|
+
|
|
28
|
+
## Append once
|
|
29
|
+
|
|
30
|
+
Requires Chronicle .NET client 19.92.0 and kernel 19.91.0 for keyed appends.
|
|
31
|
+
|
|
32
|
+
```csharp
|
|
33
|
+
using Cratis.Chronicle.EventSequences;
|
|
34
|
+
|
|
35
|
+
// Keep this key and payload unchanged across retries of this logical operation.
|
|
36
|
+
var key = new IdempotencyKey("order-confirmation-42");
|
|
37
|
+
var result = await eventLog.AppendOnce(orderId, new OrderConfirmed(), key);
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
- `IdempotencyKey` accepts 1–512 characters. Never put personal data in it.
|
|
41
|
+
- Check `IEventSequence.SupportsIdempotentAppends`. A kernel without support
|
|
42
|
+
causes `IdempotentAppendsNotSupported` before sending; do not fall back to
|
|
43
|
+
unkeyed retries when duplicate protection is required.
|
|
44
|
+
- `AppendOnce`, `AppendManyOnce`, `AppendOnceWithNamedTags` and
|
|
45
|
+
`AppendManyOnceWithNamedTags` are on `IEventSequence` (and thus `IEventLog`).
|
|
46
|
+
A batch key derives each event key with `IdempotencyKey.ForIndex(index)`;
|
|
47
|
+
preserve the batch's order and content across retries.
|
|
48
|
+
- `EventForEventSourceId.IdempotencyKey` keys individual events in a batch,
|
|
49
|
+
including a mixed keyed/unkeyed batch spanning sources. Unkeyed entries do
|
|
50
|
+
not acquire duplicate protection just because other entries have keys.
|
|
51
|
+
- `AppendResult.Outcome` / `AppendManyResult.Outcomes` distinguish `Appended`,
|
|
52
|
+
`AlreadyAppended`, and `AlreadyAppendedAfterErasure`. An already-appended
|
|
53
|
+
outcome is success, not permission to re-create an erased event.
|
|
54
|
+
- Inspect `IdempotencyConflicts`: a key reused for different content or generation,
|
|
55
|
+
or a match that cannot still be verified, is not a successful retry. A keyed
|
|
56
|
+
retry after an unknown outcome must use the **same** key and content.
|
|
57
|
+
- Keys are refused on events staged in a unit of work and on appends carrying
|
|
58
|
+
effect receipts: these throw `IdempotentAppendsNotSupported`, not an unkeyed
|
|
59
|
+
fallback. Keyed appends are not a replacement for DCB or unit-of-work ownership.
|
|
60
|
+
|
|
61
|
+
Requires Chronicle Testing 19.99.0 for dropped keyed-append replies.
|
|
62
|
+
|
|
63
|
+
The Testing client supports keyed appends. `DropNext(DroppedReply.Append)` can lose a keyed append reply after persistence;
|
|
64
|
+
retry with the same key and assert exactly one stored event and an
|
|
65
|
+
already-appended outcome. This proves retry behavior, not an exactly-once
|
|
66
|
+
external side effect.
|
|
67
|
+
|
|
68
|
+
## Several concurrency guards for one source
|
|
69
|
+
|
|
70
|
+
Requires Chronicle 19.84.0 for multiple concurrency guards for one source.
|
|
71
|
+
|
|
72
|
+
`AppendMany` overloads accept
|
|
73
|
+
`IEnumerable<KeyValuePair<EventSourceId, ConcurrencyScope>>`. Repeat a source id
|
|
74
|
+
for distinct stream guards; a dictionary cannot represent those entries. The
|
|
75
|
+
kernel checks the conjunction before any write, preserving batch atomicity.
|
|
76
|
+
A violation identifies `ScopeIndex` and `Scope`. Older kernels still refuse
|
|
77
|
+
unsupported combinations with `IncompatibleConcurrencyScopesForEventSource`;
|
|
78
|
+
never discard a guard to make an append work.
|
package/package/corpus/skills/cratis-chronicle-client-dotnet/references/connection-options.md
CHANGED
|
@@ -59,3 +59,16 @@ Read [host wiring](hosting.md) for the deployment distinction.
|
|
|
59
59
|
> TLS enabled (the default) and no configured certificate. **A production connection string must set
|
|
60
60
|
> `skipTlsValidation=false`** (or `Tls.SkipCertificateValidation = false`)
|
|
61
61
|
> against a server whose certificate is actually verifiable.
|
|
62
|
+
|
|
63
|
+
## Partition concurrency
|
|
64
|
+
|
|
65
|
+
Requires Chronicle 19.90.0 for concurrent .NET reactor/reducer partitions.
|
|
66
|
+
|
|
67
|
+
`ChronicleOptions.Reactors.MaxConcurrentPartitions` and
|
|
68
|
+
`ChronicleOptions.Reducers.MaxConcurrentPartitions` independently limit parallel
|
|
69
|
+
partitions per observer. Both default to processor count capped at 8 and require
|
|
70
|
+
at least 1. Set 1 for one-at-a-time handling if shared state is not thread-safe;
|
|
71
|
+
one partition's batches remain ordered. These are .NET client settings, not
|
|
72
|
+
kernel `observers` settings. Verified at **v19.110.1**:
|
|
73
|
+
`Source/Clients/DotNET/Reactors/ReactorOptions.cs`, `Reducers/ReducerOptions.cs`
|
|
74
|
+
and `Observation/PartitionedDeliveryScheduler.cs`.
|
|
@@ -57,3 +57,26 @@ configured, from bootstrap configuration (configured applications are recreated
|
|
|
57
57
|
on startup). Changing the image or supplying new client credentials alone does
|
|
58
58
|
not remove that old application. See **cratis-chronicle-hosting** for kernel
|
|
59
59
|
certificates, private h2c and production setup.
|
|
60
|
+
|
|
61
|
+
## Request completion ownership (19.110.1+ .NET packages)
|
|
62
|
+
|
|
63
|
+
Verified at **v19.110.1**: `Source/Clients/DotNET/Transactions/`
|
|
64
|
+
(`IUnitOfWorkCompletionOwnershipFeature.cs`, `IUnitOfWorkCompletionClaim.cs`,
|
|
65
|
+
`UnitOfWorkCompletionResult.cs`), `Source/Clients/AspNetCore/Transactions/UnitOfWorkMiddleware.cs`
|
|
66
|
+
and `Source/Clients/AspNetCore/ChronicleUnitOfWorkOptions.cs`.
|
|
67
|
+
|
|
68
|
+
Nested hosts can claim a request's completion with
|
|
69
|
+
`IUnitOfWorkCompletionOwnershipFeature.TryClaim`. Complete through the claim
|
|
70
|
+
and inspect its result before writing the response. An uncompleted claim is
|
|
71
|
+
rolled back by the outer middleware, never silently committed. This does not
|
|
72
|
+
grant arbitrary nested code the decision-read owner capability described in
|
|
73
|
+
**cratis-chronicle-decision-reads**. `UnitOfWorkCompletionOutcome.Unknown` means a
|
|
74
|
+
technical commit failure, not proof of rollback. Decision conflicts also occur
|
|
75
|
+
in `ConcurrencyViolations`; avoid reporting the same source's conflict twice.
|
|
76
|
+
|
|
77
|
+
Compatibility middleware logs a failed post-handler commit without changing a
|
|
78
|
+
response already sent. Opt in to problem responses **before headers start** with
|
|
79
|
+
`ChronicleAspNetCoreOptions.UnitOfWork.ReportFailuresBeforeResponseStarts`:
|
|
80
|
+
concurrency/decision conflict is 409, constraint violation is 400, other append
|
|
81
|
+
failure is 500. A started response cannot be repaired; complete through the
|
|
82
|
+
claim first when a success response must mean the write committed.
|
|
@@ -242,6 +242,12 @@ requirement is only to annotate the types.
|
|
|
242
242
|
Key storage is per namespace and is a hosting decision, not an application one. Chronicle ships
|
|
243
243
|
MongoDB (the default), SQL, in-memory, HashiCorp Vault, and Azure Key Vault key
|
|
244
244
|
storage, configured under `Cratis:Chronicle:Compliance:Encryption`.
|
|
245
|
+
From 19.99.1, Azure Key Vault erasure **purges every key revision**, not just
|
|
246
|
+
soft-deletes it. The vault must grant purge permission and have purge protection
|
|
247
|
+
disabled; a refused purge fails erasure rather than reporting success. Do not
|
|
248
|
+
silently weaken the erasure contract to fit a protected vault. Verified at
|
|
249
|
+
**v19.110.1**, `Source/Kernel/Storage.AzureKeyVault/AzureKeyVaultEncryptionKeyStorage.cs`
|
|
250
|
+
and `Documentation/hosting/azure.md`.
|
|
245
251
|
|
|
246
252
|
> **Operational trap.** Switching to a dedicated compliance key store without
|
|
247
253
|
> setting `MigrateFromDefaultStorage` leaves every existing key unreachable.
|
|
@@ -158,10 +158,14 @@ A constraint binds every writer, whichever client or service appends, per namesp
|
|
|
158
158
|
and per event sequence. It cannot count, sum or test a predicate; it expresses
|
|
159
159
|
uniqueness (with release on a removal event), create-once and closed streams.
|
|
160
160
|
|
|
161
|
-
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
161
|
+
- From Chronicle 19.82.2, a failed index update after a durable append marks a
|
|
162
|
+
gap. A later governed append repairs that gap from the event log or refuses
|
|
163
|
+
with a `ConstraintIndexUnavailable` error value before writing. Do not treat
|
|
164
|
+
that retryable refusal as a duplicate or remove the constraint. Verified at
|
|
165
|
+
**v19.110.1**, `Source/Kernel/Core/EventSequences/EventSequence.AppendSettlement.cs`
|
|
166
|
+
and `EventSequence.Validation.cs`; see
|
|
167
|
+
[append outcomes](../../cratis-chronicle-client-dotnet/references/append-outcomes.md)
|
|
168
|
+
for the distinct unknown-outcome case.
|
|
165
169
|
- Tenants in different namespaces can claim the same value.
|
|
166
170
|
- `[RemoveConstraint]` releases attribute constraints (`[Unique]` on a property or
|
|
167
171
|
on the event type); on a fluent `IConstraint` it compiles and does nothing, so
|
|
@@ -17,11 +17,22 @@ A constraint binds **every** writer, whichever client or service appends, so it
|
|
|
17
17
|
the mechanism for uniqueness and create-once. It cannot count, sum or test a
|
|
18
18
|
predicate: capacity and stock rules need a guarded read (read protection `On`), a stream decision or an
|
|
19
19
|
aggregate. Read [choosing how a rule holds](../cratis-chronicle-decision-reads/references/choosing-how-a-rule-holds.md)
|
|
20
|
-
when it is unclear which one a rule needs.
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
20
|
+
when it is unclear which one a rule needs.
|
|
21
|
+
|
|
22
|
+
Requires Chronicle 19.82.2 for constraint-index gap repair.
|
|
23
|
+
|
|
24
|
+
A failed
|
|
25
|
+
post-append index update marks a gap: subsequent governed appends repair the
|
|
26
|
+
index from the event log or return `ConstraintIndexUnavailable` in the result's
|
|
27
|
+
error collection without writing. Do not remove the constraint or blindly retry an unknown
|
|
28
|
+
append outcome; see [append outcomes](../cratis-chronicle-client-dotnet/references/append-outcomes.md).
|
|
29
|
+
Constraints apply per event store namespace and event sequence, so tenants in
|
|
30
|
+
different namespaces can claim the same value. A constraint is not re-run when
|
|
31
|
+
the kernel retries a duplicate sequence number.
|
|
32
|
+
|
|
33
|
+
Verified in the tagged kernel sources listed by [append outcomes](../cratis-chronicle-client-dotnet/references/append-outcomes.md):
|
|
34
|
+
`Source/Kernel/Core/EventSequences/EventSequence.AppendSettlement.cs` and
|
|
35
|
+
`EventSequence.Validation.cs`.
|
|
25
36
|
|
|
26
37
|
## Verified product sources
|
|
27
38
|
|
|
@@ -48,8 +48,9 @@ operator/kernel surface, not an application-profile layout rule.
|
|
|
48
48
|
development database ports on loopback only; a host application needing MongoDB
|
|
49
49
|
must be able to reach port 27017. Do not expose unauthenticated MongoDB.
|
|
50
50
|
|
|
51
|
-
Read references/storage-resolution.md when choosing
|
|
52
|
-
|
|
51
|
+
Read [storage resolution](references/storage-resolution.md) when choosing
|
|
52
|
+
per-namespace destinations, schema/runtime identities, Azure authentication,
|
|
53
|
+
SQL schemas, or disabling automatic creation.
|
|
53
54
|
|
|
54
55
|
## Supply certificates before starting production
|
|
55
56
|
|
|
@@ -111,18 +112,29 @@ than increasing limits without accounting for memory.
|
|
|
111
112
|
|
|
112
113
|
Set `health.port` for a dedicated HTTP/1.1 probe listener. `health.tls=false`
|
|
113
114
|
removes TLS only there; `health.exclusive=true` restricts it to the configured
|
|
114
|
-
health path (`/health` by default). Exclusive is **false by default**, so an
|
|
115
|
+
health path (`/health` by default) and its `/live` and `/ready` subpaths. Exclusive is **false by default**, so an
|
|
115
116
|
ordinary plaintext probe port also exposes other HTTP/1.1 endpoints. Keep it
|
|
116
117
|
internal. `health.tls=true` needs the top-level TLS certificate even when main
|
|
117
118
|
TLS is disabled. The production chiseled container has no shell/curl: use an
|
|
118
119
|
orchestrator/external probe, not an in-container shell healthcheck.
|
|
119
120
|
|
|
121
|
+
Requires Chronicle 19.109.0 for separate readiness/liveness routes.
|
|
122
|
+
|
|
123
|
+
`/health` and `/health/ready` remain readiness (including startup
|
|
124
|
+
and storage); use `/health/live` for liveness/startup so a replay backlog does
|
|
125
|
+
not restart a healthy-but-unready silo. A custom base path moves all three
|
|
126
|
+
routes. Verified in the tagged sources linked from [clustering and schema ownership](references/clustering-and-schema.md),
|
|
127
|
+
`Documentation/hosting/configuration/health-endpoint.md`
|
|
128
|
+
and `Source/Kernel/Server/KernelReadinessHealthCheck.cs`.
|
|
129
|
+
|
|
120
130
|
## Form one cluster, not several over shared data
|
|
121
131
|
|
|
122
|
-
Every node must
|
|
123
|
-
|
|
124
|
-
single-node
|
|
125
|
-
|
|
132
|
+
Every node must select shared membership and share cluster/service ids and the
|
|
133
|
+
cluster database: `clustering.type=MongoDB` for MongoDB, or `Sql` for PostgreSQL
|
|
134
|
+
and SQL Server. SQLite remains single-node. Default `Localhost` does not join
|
|
135
|
+
nodes; persistent stores guard it with a single-writer lease. Read [clustering and schema ownership](references/clustering-and-schema.md)
|
|
136
|
+
for lease failure behavior, SQL migration/Verify modes and the disabled-by-default
|
|
137
|
+
projection-generation groundwork.
|
|
126
138
|
`Clustering:ResponseTimeout` bounds Orleans transport waiting; increasing observer
|
|
127
139
|
subscriber timeout cannot exceed that transport deadline.
|
|
128
140
|
|
|
@@ -132,12 +144,16 @@ and upgrading a cluster. See **cratis-chronicle-web-workbench** for browser
|
|
|
132
144
|
exposure and **cratis-chronicle-cli-operations** for authorized live operations.
|
|
133
145
|
Several services share one kernel with one event store each: see **cratis-multi-service-architecture**, and **cratis-multi-service-local-orchestration** for an AppHost or compose file.
|
|
134
146
|
|
|
147
|
+
Read [webhook delivery policy](references/webhooks.md) for target validation,
|
|
148
|
+
independent retries, managed identity and **real-delivery** quarantine probes.
|
|
149
|
+
Ordinary observer quarantine guidance does not imply webhooks stay silent.
|
|
150
|
+
|
|
135
151
|
## Verify
|
|
136
152
|
|
|
137
153
|
- The build/image and intended environment agree; no default development
|
|
138
154
|
application remains in reused production storage.
|
|
139
|
-
- Storage is reachable and MongoDB is a replica set; multiple nodes share
|
|
140
|
-
membership, cluster id, service id and certificate ring.
|
|
155
|
+
- Storage is reachable and MongoDB is a replica set; multiple nodes share valid
|
|
156
|
+
MongoDB/Sql membership, cluster id, service id and certificate ring.
|
|
141
157
|
- Certificates load; clients validate production TLS (`skipTlsValidation=false`).
|
|
142
158
|
- A probe observes health without publishing a plaintext Workbench/API port.
|
|
143
159
|
- Proxy upstream protocol and trust are tested from the actual immediate peer.
|
|
@@ -0,0 +1,84 @@
|
|
|
1
|
+
<!-- Copyright (c) Cratis. All rights reserved. -->
|
|
2
|
+
<!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
|
|
3
|
+
|
|
4
|
+
# Cluster membership, fencing and SQL schema ownership
|
|
5
|
+
|
|
6
|
+
Verified at Chronicle **v19.110.1**: `Documentation/hosting/configuration/`
|
|
7
|
+
(`clustering.md`, `storage.mdx`, `health-endpoint.md`),
|
|
8
|
+
`Source/Kernel/Core/Clustering/SingleWriterLeaseKeeper.cs`,
|
|
9
|
+
`Source/Kernel/Core/Configuration/{SchemaMode,SchemaAction,Features}.cs` and
|
|
10
|
+
`Source/Kernel/Server/KernelReadinessHealthCheck.cs`.
|
|
11
|
+
|
|
12
|
+
## Choose membership for the storage
|
|
13
|
+
|
|
14
|
+
Requires Chronicle 19.105.0 for SQL multi-node membership and provider validation.
|
|
15
|
+
|
|
16
|
+
PostgreSQL and SQL Server support multi-node membership with
|
|
17
|
+
`clustering.type = Sql`; MongoDB uses `MongoDB`. Every node must share the
|
|
18
|
+
cluster database, `clusterId`, `serviceId` and certificates. SQL clustering with SQLite or a non-SQL provider fails at startup; it is not a
|
|
19
|
+
way to cluster SQLite. Sharing a database while using `Localhost` does not form
|
|
20
|
+
a cluster. Silo and gateway ports must remain reachable between nodes.
|
|
21
|
+
|
|
22
|
+
## Localhost single-writer lease
|
|
23
|
+
|
|
24
|
+
Requires Chronicle 19.110.0 on every contender for the Localhost lease. With MongoDB
|
|
25
|
+
storage the lease requires MongoDB 5.0 or later, and it is on by default.
|
|
26
|
+
|
|
27
|
+
`Localhost` on persistent storage (MongoDB, PostgreSQL, SQL Server,
|
|
28
|
+
SQLite) takes the fixed `kernel` lease in the cluster database **before** Orleans
|
|
29
|
+
runtime startup. `InMemory` is process-local and takes no lease; MongoDB/Sql
|
|
30
|
+
membership do not use this guard. Different cluster/service ids are diagnostic,
|
|
31
|
+
not a way to bypass the fixed lease.
|
|
32
|
+
|
|
33
|
+
- `clustering.singleWriterLease.enabled` is automatic by default; disable it
|
|
34
|
+
only with another proven single-writer mechanism.
|
|
35
|
+
- `ttl` defaults to 30 seconds (minimum 5); `acquireTimeout` defaults to 2 minutes.
|
|
36
|
+
Zero timeout makes one attempt. A second node waits or refuses startup and
|
|
37
|
+
stays unready; it does not become a safe second writer.
|
|
38
|
+
- Failed renewal marks health unhealthy and starts shutdown. Failure to confirm
|
|
39
|
+
renewal within half the TTL hard-fences the process. Shutdown retains the
|
|
40
|
+
lease while silo work drains; a crash successor waits for expiry. Do not
|
|
41
|
+
delete a live lease to get another node started.
|
|
42
|
+
- Every contender must support the lease; an older binary does not take it.
|
|
43
|
+
|
|
44
|
+
## Least-privilege SQL deployments
|
|
45
|
+
|
|
46
|
+
Requires Chronicle 19.109.0 for schema actions and Verify runtime mode.
|
|
47
|
+
|
|
48
|
+
`--migrate-only` (or
|
|
49
|
+
`Cratis__Chronicle__Storage__SchemaAction=MigrateAndExit`) applies SQL migrations
|
|
50
|
+
for the cluster, known stores/namespaces and jobs, then exits without serving.
|
|
51
|
+
Run it with the authorized schema identity before starting runtime-only nodes.
|
|
52
|
+
`--verify-schema` / `SchemaAction=VerifyAndExit` checks without changing schema;
|
|
53
|
+
repeatable `--event-store` and `--namespace` with migration pre-provision new
|
|
54
|
+
destinations. Exit codes: 0 success, 2 schema not current, 3 lock timeout, 4
|
|
55
|
+
invalid options/forbidden database creation, 1 other failure. MongoDB and
|
|
56
|
+
in-memory schema actions succeed without doing work; do not call that a SQL
|
|
57
|
+
migration check.
|
|
58
|
+
|
|
59
|
+
`Cratis__Chronicle__Storage__SchemaMode=Verify` performs no database/schema
|
|
60
|
+
creation or migrations. Missing schema at startup or a newly used namespace
|
|
61
|
+
fails with instructions to migrate. `Migrate` remains the default. Provision
|
|
62
|
+
new destinations before routing tenants to them; `createIfMissing=false` is
|
|
63
|
+
not equivalent to Verify and does not replace the migration step.
|
|
64
|
+
|
|
65
|
+
SQL schema-lock wait and recovered insert-collision metrics aid diagnosis;
|
|
66
|
+
readiness stays unhealthy until startup finishes. Do not restart a migrating
|
|
67
|
+
node using a readiness endpoint as liveness.
|
|
68
|
+
|
|
69
|
+
## Projection generations are not enabled by these releases
|
|
70
|
+
|
|
71
|
+
The tagged sources provide guarded groundwork, not an instruction to enable it.
|
|
72
|
+
`Features.ProjectionGenerations` remains off by default. MongoDB generation
|
|
73
|
+
sink writes need majority transactions/fences, so startup refuses a standalone
|
|
74
|
+
MongoDB with the feature on. Materialization binding catalogs exist for
|
|
75
|
+
MongoDB and in-memory; SQL explicitly refuses that catalog. Generation-owned
|
|
76
|
+
collections refuse drop, rename or clear.
|
|
77
|
+
|
|
78
|
+
The per-store generation initializer is explicit and idempotent, records durable
|
|
79
|
+
receipts/blockers, and refuses adoption if it cannot prove exclusion; it is not
|
|
80
|
+
run automatically and does nothing with the feature disabled. Initialized stores
|
|
81
|
+
have a minimum kernel generation floor; older kernels and state reset are
|
|
82
|
+
refused. Do not recommend enabling the feature, deleting fences, or resetting
|
|
83
|
+
state to resolve an ordinary replay problem. Consult the tagged product sources
|
|
84
|
+
and accepted design before planning adoption.
|
|
@@ -26,18 +26,46 @@ clearing one does not clear the other. Inspect the cause before authorizing clea
|
|
|
26
26
|
or retry. Subscriber timeout abandons waiting, not subscriber work: redelivery
|
|
27
27
|
can overlap work that still runs, so side effects must remain idempotent.
|
|
28
28
|
|
|
29
|
+
Requires Chronicle 19.109.0 for abandoned-delivery holds.
|
|
30
|
+
Requires Chronicle 19.110.0 for webhook quarantine probes.
|
|
31
|
+
|
|
32
|
+
`observers.abandonedDeliveryHold` defaults to 300 seconds **after the delivery
|
|
33
|
+
deadline**. A positive value below `subscriberTimeout` uses that timeout instead;
|
|
34
|
+
zero or less disables the hold. Retry/catch-up/replay wait for a barrier on the
|
|
35
|
+
**original subscriber activation**, not a replacement client. An unavailable
|
|
36
|
+
original silo is not proof that its work drained. Late results are discarded;
|
|
37
|
+
a late reducer result cannot write to its sink.
|
|
38
|
+
|
|
39
|
+
The hold is bounded, not cancellation: user code might still run after expiry,
|
|
40
|
+
so external side effects must remain idempotent. Clearing persisted failure
|
|
41
|
+
history followed by restart loses that history. Inspect abandoned-delivery logs
|
|
42
|
+
and metrics before reducing the hold. Verified in
|
|
43
|
+
`Documentation/hosting/configuration/observers.md` and
|
|
44
|
+
`Source/Kernel/Core/Observation/Observer.DeliveryFences.cs`.
|
|
45
|
+
|
|
29
46
|
Quarantine persists through grain deactivation. Watchdog ticks and catch-up/replay
|
|
30
47
|
completion do not end it. An operator clear or a fresh subscription does;
|
|
31
48
|
a kernel restart only helps observers that actually receive a fresh subscription.
|
|
32
49
|
Event-store subscriptions remain quarantined until operator clear. Recovery can
|
|
33
|
-
quarantine again if the cause remains.
|
|
50
|
+
quarantine again if the cause remains. Webhooks are an exception: eligible
|
|
51
|
+
quarantines can probe with a **real pending event** and
|
|
52
|
+
resume after success. Read [webhook policy](webhooks.md) before promising an
|
|
53
|
+
operator that quarantine prevents all traffic; webhook timeout cancellation
|
|
54
|
+
also drains the HTTP operation before starting another delivery.
|
|
34
55
|
|
|
35
56
|
Projection replay and catch-up each use a **single globally ordered job step**;
|
|
36
57
|
a new catch-up failure stops that step rather than letting other sources proceed.
|
|
37
58
|
Full replay is inactive until completion, then routes to catch-up/live unless
|
|
38
59
|
quarantined. Partition quarantine stops that partition's automatic retries;
|
|
39
60
|
observer quarantine suppresses automatic recovery for the whole observer.
|
|
40
|
-
|
|
61
|
+
Projection replay skips a failed partition while continuing
|
|
62
|
+
healthy partitions, then schedules failed partitions for recovery after replay
|
|
63
|
+
completes. Catch-up still stops its ordered step on a new failure. Replaying
|
|
64
|
+
**global** projections is refused before any write (Requires Chronicle 19.101.3). For MongoDB
|
|
65
|
+
catch-up/bulk-write upgrade repairs, read
|
|
66
|
+
[projection definition changes](../../cratis-chronicle-projection/references/definition-changes.md).
|
|
67
|
+
|
|
68
|
+
Sources (verified through **v19.110.1**): `Documentation/concepts/observers.mdx`,
|
|
41
69
|
`Documentation/projections/registration-lifecycle.md`, and
|
|
42
70
|
`Source/Kernel/Core/Observation/Jobs/{CatchUpObserver,ReplayObserver}.cs`.
|
|
43
71
|
|
|
@@ -46,6 +46,36 @@ access tokens refreshed through resolution. Password-based MongoDB clients do
|
|
|
46
46
|
not acquire a new password merely because configuration changes; restart them
|
|
47
47
|
when rotating that credential.
|
|
48
48
|
|
|
49
|
+
## Azure identity
|
|
50
|
+
|
|
51
|
+
Verified through **v19.110.1**: `Documentation/hosting/azure.md`,
|
|
52
|
+
`Documentation/hosting/configuration/storage-connection-resolution.md`,
|
|
53
|
+
`Source/Kernel/Core/Configuration/PostgreSqlEntra.cs` and
|
|
54
|
+
`Source/Kernel/Storage.Sql/Connections/PostgreSqlEntraAuthentication.cs`.
|
|
55
|
+
|
|
56
|
+
Requires Chronicle 19.96.0 for AzureCredential HTTP resolver authentication.
|
|
57
|
+
|
|
58
|
+
The HTTP resolver's `Authentication.Type=AzureCredential` supports
|
|
59
|
+
`Credential=ManagedIdentity` (default), `WorkloadIdentity` or `Default`, plus
|
|
60
|
+
`Scope` and optional `ClientId`. The resolver identity is not the database
|
|
61
|
+
identity. Prefer the specific production credential rather than accidentally
|
|
62
|
+
using a developer credential through the default chain.
|
|
63
|
+
|
|
64
|
+
Requires Chronicle 19.92.2 for native Azure SQL identity authentication.
|
|
65
|
+
Requires Chronicle 19.109.0 for native PostgreSQL Entra authentication.
|
|
66
|
+
|
|
67
|
+
Azure SQL uses Microsoft.Data.SqlClient's native authentication connection-string
|
|
68
|
+
settings (managed identity, workload identity, or default chain), not a
|
|
69
|
+
Chronicle-specific password flow. Native Entra authentication for
|
|
70
|
+
Azure Database for PostgreSQL uses `storage.postgreSqlEntra.enabled=true`, optional
|
|
71
|
+
`managedIdentityClientId`, and a PostgreSQL `Username` with **no password**.
|
|
72
|
+
Tokens refresh on pooled connections. Do not combine it with a resolver that
|
|
73
|
+
returns credentials; placement and schema resolution remain supported.
|
|
74
|
+
|
|
75
|
+
For separate least-privilege schema/runtime deployments, read
|
|
76
|
+
[clustering and schema ownership](clustering-and-schema.md): `--migrate-only`
|
|
77
|
+
and `SchemaMode=Verify` are distinct from `createIfMissing=false`.
|
|
78
|
+
|
|
49
79
|
## Verify
|
|
50
80
|
|
|
51
81
|
Use the real target provider in integration specs: resolve two namespace
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
<!-- Copyright (c) Cratis. All rights reserved. -->
|
|
2
|
+
<!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
|
|
3
|
+
|
|
4
|
+
# Webhook delivery policy
|
|
5
|
+
|
|
6
|
+
## Contents
|
|
7
|
+
|
|
8
|
+
- Validate the destination, not just the registration
|
|
9
|
+
- Independent delivery retries
|
|
10
|
+
- OAuth scope and token refresh
|
|
11
|
+
- Managed identity
|
|
12
|
+
- Quarantine probes are real deliveries
|
|
13
|
+
|
|
14
|
+
Verified at Chronicle **v19.110.1**: `Documentation/webhooks/index.mdx`,
|
|
15
|
+
`Source/Kernel/Core/Configuration/{Webhooks,WebhookManagedIdentity,WebhookProbe}.cs`,
|
|
16
|
+
`Source/Kernel/Core/Observation/Webhooks/` (including `Validation/`) and
|
|
17
|
+
`Source/Kernel/Server/Observation/Webhooks/ManagedIdentityCredentialFactory.cs`.
|
|
18
|
+
Configuration below sits under `Cratis:Chronicle:Webhooks`.
|
|
19
|
+
|
|
20
|
+
## Validate the destination, not just the registration
|
|
21
|
+
|
|
22
|
+
Requires Chronicle 19.100.0 for webhook target validation.
|
|
23
|
+
|
|
24
|
+
`Validation` is `Warn` (default) or `Enforce`. Warn logs violations without
|
|
25
|
+
blocking, except unapproved cloud metadata addresses. Do not mistake the default
|
|
26
|
+
for full SSRF protection or approve metadata hosts in normal deployments.
|
|
27
|
+
Enforce requires HTTPS and approved hosts, checking private/special resolved
|
|
28
|
+
addresses at connect time too. Redirects are not followed. OAuth authorities use the same policy.
|
|
29
|
+
`AllowInsecureHttp` applies only in Development.
|
|
30
|
+
|
|
31
|
+
`AllowedHosts` accepts exact hosts and wildcard subdomains; an event store's
|
|
32
|
+
`EventStoreAllowedHosts` entry **replaces** the global list. An exact host entry
|
|
33
|
+
can intentionally admit that host's private addresses. Bound exceptions to
|
|
34
|
+
trusted receivers; wildcards approve subdomains, not their parent domain.
|
|
35
|
+
Never widen the list just to silence a refusal.
|
|
36
|
+
|
|
37
|
+
Requires Chronicle 19.106.0 for validation of stored active definitions.
|
|
38
|
+
|
|
39
|
+
Existing active definitions are checked on activation/restart.
|
|
40
|
+
Enforce quarantines a disallowed target without deleting/deactivating the
|
|
41
|
+
stored definition; Warn logs once per activation and continues. Fix the target
|
|
42
|
+
or approved policy, then clear quarantine: clearing revalidates, so it does not
|
|
43
|
+
bypass a bad target. A connect-time policy failure is a configuration problem,
|
|
44
|
+
not a transient failure retried for 24 hours.
|
|
45
|
+
|
|
46
|
+
## Independent delivery retries
|
|
47
|
+
|
|
48
|
+
Requires Chronicle 19.104.0 for webhook-specific retry policy.
|
|
49
|
+
|
|
50
|
+
Each delivery makes one HTTP attempt. `Retry` is independent of global observer
|
|
51
|
+
settings: delays are 1 s, 5 s, 30 s, 2 min, 10 min, 30 min, then hourly, with
|
|
52
|
+
±20% jitter. The budget is 24 hours, then the partition is parked. Five parked
|
|
53
|
+
partitions or at least half the failing partitions parked quarantines the
|
|
54
|
+
observer. Override the schedule, maximum delay, attempts, budget and quarantine
|
|
55
|
+
thresholds only with a bounded recovery policy. `RetryDelaySeconds` is retained
|
|
56
|
+
but has no effect.
|
|
57
|
+
|
|
58
|
+
Retry deadlines/counts/positions survive restart; manual partition retry overrides
|
|
59
|
+
a pending deadline. A per-target (scheme/host/port) circuit defers deliveries
|
|
60
|
+
without consuming attempts. `429`/`503` with valid `Retry-After` defer without
|
|
61
|
+
consuming attempts, capped by `MaximumRetryAfterSeconds` (default 24 hours).
|
|
62
|
+
`410` parks immediately. Other `4xx` except `408` and `429` quarantine as
|
|
63
|
+
configuration failures. Catch-up/replay respect the pending deadline.
|
|
64
|
+
|
|
65
|
+
## OAuth scope and token refresh
|
|
66
|
+
|
|
67
|
+
Requires Chronicle 19.98.0 for OAuth scope, token reuse and one-shot 401 refresh.
|
|
68
|
+
|
|
69
|
+
The .NET webhook builder accepts
|
|
70
|
+
`WithOAuth(authority, clientId, clientSecret, scope)`; scope is optional, and
|
|
71
|
+
re-registering without it clears a previously configured scope. Entra client
|
|
72
|
+
credentials need the receiver's `api://<application-id>/.default` scope. Tokens
|
|
73
|
+
are shared by token endpoint/client/scope/credentials and refreshed before
|
|
74
|
+
expiry; tokens lacking expiry are not cached. A 401 refreshes and retries the
|
|
75
|
+
same request once; other responses do not trigger an OAuth retry. This is
|
|
76
|
+
separate from the observer retry policy. Never put credentials in examples or
|
|
77
|
+
logs. Verified in `Source/Clients/DotNET/Webhooks/IWebhookDefinitionBuilder.cs` and
|
|
78
|
+
the tagged webhook documentation.
|
|
79
|
+
|
|
80
|
+
## Managed identity
|
|
81
|
+
|
|
82
|
+
Requires Chronicle kernel 19.110.0 and .NET packages 19.110.1 for managed identity webhooks.
|
|
83
|
+
|
|
84
|
+
System-assigned/user-assigned Azure managed identity and AKS workload identity
|
|
85
|
+
can authorize a webhook without storing a client secret. The operator must
|
|
86
|
+
allow exact scopes in `ManagedIdentity.AllowedScopes`; the empty default
|
|
87
|
+
disables this. Non-empty client ids must be in `AllowedClientIds`; an empty id
|
|
88
|
+
needs `AllowSystemAssigned=true`. The receiver must explicitly match the host
|
|
89
|
+
allow-list **even in Warn mode**; delivery rechecks approval before acquiring a
|
|
90
|
+
token. This lends the **kernel's** identity, not the registering client's. Its
|
|
91
|
+
principal still needs the receiver's application role. Register with the .NET
|
|
92
|
+
builder's `WithManagedIdentity(scope, clientId)` (client id optional). Tokens are
|
|
93
|
+
memory-only and refreshed before expiry. A 401 discards the cached token and
|
|
94
|
+
its credential instance, then retries once; Azure's own cache means a different
|
|
95
|
+
token is not guaranteed. Never log tokens. Verified in
|
|
96
|
+
`Documentation/hosting/azure.md` and `Source/Clients/DotNET/Webhooks/IWebhookDefinitionBuilder.cs`.
|
|
97
|
+
|
|
98
|
+
## Quarantine probes are real deliveries
|
|
99
|
+
|
|
100
|
+
Requires Chronicle 19.110.0 for automatic webhook quarantine probes.
|
|
101
|
+
|
|
102
|
+
`Probe.Enabled` defaults to true. Retry-budget or delivery/configuration
|
|
103
|
+
quarantines can probe with exponential jittered backoff (initial 60 s, maximum
|
|
104
|
+
1 hour, timeout 30 s); manual quarantines and parked partitions are not
|
|
105
|
+
probed. A probe sends the **first pending event** without clearing quarantine
|
|
106
|
+
first. Success records it as handled, clears quarantine and resumes in order.
|
|
107
|
+
Failure leaves quarantine intact. Probes revalidate targets and consume neither
|
|
108
|
+
normal retry budget nor circuit samples. Set `Probe.Enabled=false` when
|
|
109
|
+
quarantined receivers must get no automatic traffic. Receivers must remain
|
|
110
|
+
idempotent; a probe is not a read-only health check.
|
|
@@ -13,6 +13,18 @@ when independence cannot be proved. Counters and joins are repaired by replay,
|
|
|
13
13
|
not by deploying corrected mappings: earlier double-counts remain until rebuilt.
|
|
14
14
|
A `FromAll` subscription change redefines existing projections.
|
|
15
15
|
|
|
16
|
+
Adding mappings for event types that already have history requires
|
|
17
|
+
**full** replay: `Automatic` rebuilds; `PartialOnly` and `Manual` recommend it.
|
|
18
|
+
The kernel no longer replays those mappings onto live state, where watermarks
|
|
19
|
+
could discard historical writes or counters could count twice. No history for
|
|
20
|
+
the added types means no action. Explicit partition replay is unchanged.
|
|
21
|
+
Verified in `Source/Kernel/Core/Projections/Projection.cs`
|
|
22
|
+
(`EvolveForAllNamespaces`).
|
|
23
|
+
|
|
24
|
+
Requires Chronicle 19.101.3 for explicit refusal of global projection replay.
|
|
25
|
+
|
|
26
|
+
Global projection replay (full or partition) is refused before writing. Do not promise to repair globally scoped state by replay.
|
|
27
|
+
|
|
16
28
|
Projection replay and catch-up each walk the sequence in **one globally ordered
|
|
17
29
|
job step**, not parallel per-source steps. A new failure during catch-up stops
|
|
18
30
|
that ordered step even if other sources are healthy. Full replay keeps the
|
|
@@ -49,5 +61,26 @@ revives a retired observer, or restarts a stopped/deleted capture. A kernel
|
|
|
49
61
|
restart or a newly added namespace still ends projection quarantine through a
|
|
50
62
|
fresh subscription; do not interpret ordinary watchdog repair as operator clear.
|
|
51
63
|
|
|
64
|
+
## Upgrade repair checks
|
|
65
|
+
|
|
66
|
+
Verified through **v19.110.1**:
|
|
67
|
+
`Source/Kernel/Storage.MongoDB/Sinks/Sink.cs`,
|
|
68
|
+
`Source/Kernel/Core/Projections/Engine/KeyResolvers.ParentHistory.cs` and the
|
|
69
|
+
tagged release notes:
|
|
70
|
+
|
|
71
|
+
Requires Chronicle 19.110.0 for MongoDB resumed bulk-window preservation, replaced-window fencing and historical parent-key resolution.
|
|
72
|
+
|
|
73
|
+
- MongoDB catch-ups resumed while clients restarted/re-registered could lose
|
|
74
|
+
unflushed writes on older kernels. Upgrade, then replay affected
|
|
75
|
+
projections, including catch-ups with `BulkWindowLost` or completed failures;
|
|
76
|
+
silent loss may leave no log entry. The fixed sink preserves the window,
|
|
77
|
+
refuses a window replaced before the first checkpoint and
|
|
78
|
+
prevents queued writes crossing replay/catch-up collections.
|
|
79
|
+
- With content-keyed parents, children rebuilt before this fix may be under the
|
|
80
|
+
wrong parent. Replay after upgrading; key lookup now uses the child's event
|
|
81
|
+
position rather than the parent's latest key.
|
|
82
|
+
- These are kernel-backed storage/replay behaviors. In-process projection
|
|
83
|
+
scenarios do not prove bulk durability or historical parent-key resolution.
|
|
84
|
+
|
|
52
85
|
Closing a stream does not erase its events or prevent a projection replay.
|
|
53
86
|
Keep the closing fact as history; see **cratis-chronicle-event-constraints**.
|
|
@@ -55,6 +55,12 @@ model-bound `[FromEvery]` has no opt-out. `FromAll` can count into dictionary
|
|
|
55
55
|
properties with a per-event context key (`Projections/FromEveryBuilder.cs`,
|
|
56
56
|
`IFromEveryBuilder.cs`, `Documentation/projections/declarative/`).
|
|
57
57
|
|
|
58
|
+
Requires Chronicle 19.88.0 for all-event constant mappings.
|
|
59
|
+
|
|
60
|
+
Fluent `FromEvery` and `FromAll` property mappings support `ToValue(...)` for
|
|
61
|
+
concepts, enums and `null`, like per-event `Set(...).ToValue(...)`. Verified at
|
|
62
|
+
**v19.110.1**, `Projections/AllSetBuilder.cs` and `IAllSetBuilder.cs`.
|
|
63
|
+
|
|
58
64
|
## Inherited from `IProjectionBuilder<TReadModel, TBuilder>`
|
|
59
65
|
|
|
60
66
|
```csharp
|
|
@@ -83,6 +89,12 @@ Chaining `.NotRewindable()`, `.Passive()`, `.ContainerName()`, or
|
|
|
83
89
|
`RemovedWith` and `RemovedWithJoin` hand you the **concrete**
|
|
84
90
|
`RemovedWithBuilder<,>` and `RemovedWithJoinBuilder<,>` classes, not interfaces.
|
|
85
91
|
|
|
92
|
+
Requires Chronicle 19.93.0 for the root-removal-through-join startup warning.
|
|
93
|
+
|
|
94
|
+
`RemovedWithJoin` removes **children**, not a root read model. The .NET client warns at startup about root removal through a join (fluent,
|
|
95
|
+
model-bound or explicit); it still removes nothing. Use `RemovedWith` for the
|
|
96
|
+
root. Verified in `Projections/RootRemovalViaJoin.cs` at the tag above.
|
|
97
|
+
|
|
86
98
|
## Keys live on the per-event builder
|
|
87
99
|
|
|
88
100
|
`From<TEvent>` hands the callback an `IFromBuilder<TReadModel, TEvent>`, which
|
|
@@ -239,6 +239,18 @@ Read [references/domain-failures.md](references/domain-failures.md) when handlin
|
|
|
239
239
|
Inject through the primary constructor; hold no mutable state (CHR0031 warning).
|
|
240
240
|
Keep one automation concern per reactor; several related handlers are fine.
|
|
241
241
|
|
|
242
|
+
Requires Chronicle 19.90.0 for concurrent .NET reactor partitions.
|
|
243
|
+
|
|
244
|
+
Different partitions of one .NET reactor run concurrently;
|
|
245
|
+
batches for one partition retain arrival order. Shared collaborators must be
|
|
246
|
+
thread-safe. Set `ChronicleOptions.Reactors.MaxConcurrentPartitions = 1` when
|
|
247
|
+
one-at-a-time handling is required; the default is processor count capped at 8,
|
|
248
|
+
and values below 1 are refused. This is not ordering across different observers.
|
|
249
|
+
Verified in the tagged sources listed in
|
|
250
|
+
[connection options](../cratis-chronicle-client-dotnet/references/connection-options.md):
|
|
251
|
+
`Source/Clients/DotNET/Reactors/ReactorOptions.cs`,
|
|
252
|
+
`Reactors.cs` and `Observation/PartitionedDeliveryScheduler.cs`.
|
|
253
|
+
|
|
242
254
|
## Specifications
|
|
243
255
|
|
|
244
256
|
Start with `Specification` plus a direct call for pure reactor decisions; add `ReactorScenario<TReactor>` only where invocation, dependency wiring or side effects contribute proof. Specify the contract, not only the happy path: field lineage, contract-authorized filtering, repeated delivery, and unrelated events.
|
|
@@ -14,6 +14,19 @@ analyzers flag synchronous custom returns even when the runtime handler accepts
|
|
|
14
14
|
Return events rather than inject `IEventLog`: **a convention, not an analyzer
|
|
15
15
|
contract**. Returned appends stay inside Chronicle's transaction.
|
|
16
16
|
|
|
17
|
+
Requires Chronicle 19.94.0 for receipt-keyed returned events.
|
|
18
|
+
|
|
19
|
+
Returned events on receipt-capable kernels have stable effect keys: retries, crash redelivery and same-generation replay do not append them
|
|
20
|
+
again. This covers bare, cross-source and mixed returned-event results, not
|
|
21
|
+
arbitrary HTTP calls or independently injected `IEventLog` appends. Older kernels
|
|
22
|
+
retain the previous at-least-once behavior. Preserve deterministic return
|
|
23
|
+
ordering/content; a changed result is a conflict, not a reason to discard the
|
|
24
|
+
receipt. Keep `ReactorDelivery`-based idempotency for external side effects and
|
|
25
|
+
`[OnceOnly]` for deliberate replay exclusion. Verified at **v19.110.1**:
|
|
26
|
+
`Source/Clients/DotNET/Reactors/SideEffects/ReactorEffectKeys.cs`,
|
|
27
|
+
`EventResultHandler.cs`, `MixedSideEffectsResultHandler.cs` and
|
|
28
|
+
`Documentation/reactors/side-effects.mdx`.
|
|
29
|
+
|
|
17
30
|
```csharp
|
|
18
31
|
// One event, on the triggering event's own event source.
|
|
19
32
|
public Task<<ResultEvent>> <MethodName>(<EventName> @event, EventContext context) =>
|
|
@@ -75,7 +75,9 @@ public record <ReadModelName>(
|
|
|
75
75
|
reserves those for compliance-subject tracking, and analyzer `CHR0035`
|
|
76
76
|
rejects them.
|
|
77
77
|
- `Cratis.Chronicle.ReadModels.IndexAttribute` (`[Index]`) marks a property for
|
|
78
|
-
indexing in the sink.
|
|
78
|
+
indexing in the sink, including SQL. `[CompositeIndex]` on the class declares ordered multi-property indexes and
|
|
79
|
+
`[Index(IsDescending = true)]` declares descending ones. Read
|
|
80
|
+
[indexing](references/indexing.md) for valid paths, flags and store-specific limits.
|
|
79
81
|
|
|
80
82
|
### Shape the model for its consumer
|
|
81
83
|
|
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
<!-- Copyright (c) Cratis. All rights reserved. -->
|
|
2
|
+
<!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
|
|
3
|
+
|
|
4
|
+
# Read-model indexes
|
|
5
|
+
|
|
6
|
+
Verified against the tagged `Source/Clients/DotNET/ReadModels/IndexAttribute.cs`,
|
|
7
|
+
`CompositeIndexAttribute.cs` and
|
|
8
|
+
[indexing documentation](https://github.com/Cratis/Chronicle/blob/v19.110.1/Documentation/read-models/indexing.mdx).
|
|
9
|
+
|
|
10
|
+
Requires Chronicle 19.103.0 for SQL indexes.
|
|
11
|
+
Requires Chronicle 19.105.0 for SQL Server string index length bounds.
|
|
12
|
+
Requires Chronicle kernel 19.110.0 for SQL nested-path indexes and PostgreSQL GIN
|
|
13
|
+
collection indexes.
|
|
14
|
+
Requires Chronicle .NET packages 19.110.1 for composite/descending indexes and an
|
|
15
|
+
index for every use of a reused nested type.
|
|
16
|
+
|
|
17
|
+
Import `Cratis.Chronicle.ReadModels`.
|
|
18
|
+
|
|
19
|
+
```csharp
|
|
20
|
+
[ReadModel]
|
|
21
|
+
[CompositeIndex(nameof(CustomerId), nameof(Created), IsDescending = new[] { false, true })]
|
|
22
|
+
public record OrdersByCustomer(
|
|
23
|
+
Guid Id,
|
|
24
|
+
[property: Index] Guid CustomerId,
|
|
25
|
+
[property: Index(IsDescending = true)] DateTimeOffset Created);
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
- `[Index]` targets a property or record parameter; `[CompositeIndex]` targets
|
|
29
|
+
a class and is repeatable. Composite property order matters.
|
|
30
|
+
- Dotted paths name nested properties. `IsDescending` is one flag on `[Index]`,
|
|
31
|
+
an array on `[CompositeIndex]`; omit the array for all ascending or provide
|
|
32
|
+
one flag per property. Empty/duplicate/unknown paths and mismatched flag
|
|
33
|
+
counts are refused at registration. Indexes are not uniqueness constraints.
|
|
34
|
+
Use an event constraint for a value that must be unique at append time;
|
|
35
|
+
a read-side index is not a write-side invariant.
|
|
36
|
+
- MongoDB and SQL create indexes at read-model registration. PostgreSQL uses
|
|
37
|
+
GIN for collection paths, which cannot be combined in a composite index with
|
|
38
|
+
ordinary fields. SQL Server and SQLite skip collection indexes with a warning.
|
|
39
|
+
- SQL Server indexed string columns honor `MaxLength`, `StringLength` or
|
|
40
|
+
`Length`; otherwise the bound is 450 characters. Overlong writes fail the
|
|
41
|
+
projection partition. Replay existing read models to apply bounded columns.
|
|
42
|
+
- Indexes on reused nested types are expanded for every path, not
|
|
43
|
+
just the first occurrence. Test the actual store's query behavior and inspect
|
|
44
|
+
its schema; an in-process scenario does not prove physical indexes exist.
|
|
@@ -153,6 +153,10 @@ identifier properties populate; when one does not, investigate the projection
|
|
|
153
153
|
rather than the harness. Skip only on a reproduced harness gap, and put the
|
|
154
154
|
specific reason in the skip message.
|
|
155
155
|
|
|
156
|
+
For pre-seeding instances, `ProcessAppendedEvents(...)` and schema-generator
|
|
157
|
+
sharing, read [seeded state and appended-event processing](references/seeded-state.md).
|
|
158
|
+
These are setup/processing surfaces, not proof of kernel persistence or replay.
|
|
159
|
+
|
|
156
160
|
## Step 4 — Know what the scenario picked
|
|
157
161
|
|
|
158
162
|
`WithProjection` overrides discovery, including a reducer. Otherwise:
|
package/package/corpus/skills/cratis-chronicle-read-model-specifications/references/seeded-state.md
ADDED
|
@@ -0,0 +1,35 @@
|
|
|
1
|
+
<!-- Copyright (c) Cratis. All rights reserved. -->
|
|
2
|
+
<!-- Licensed under the MIT license. See LICENSE file in the project root for full license information. -->
|
|
3
|
+
|
|
4
|
+
# Seeded state and appended-event processing
|
|
5
|
+
|
|
6
|
+
Requires Chronicle 19.83.0 for seeded state; verified against v19.110.1 sources.
|
|
7
|
+
|
|
8
|
+
Verified in `Source/Clients/Testing/ReadModels/`
|
|
9
|
+
(`ReadModelSourceGivenBuilder.cs`, `ReadModelScenario.cs`) and
|
|
10
|
+
`Source/Clients/Testing/Events/EventStoreForTesting.cs`.
|
|
11
|
+
|
|
12
|
+
`await scenario.Given.ForEventSourceId(id).ReadModel(instance)`
|
|
13
|
+
sets initial state for both `InstanceForEventSourceId(id)` and
|
|
14
|
+
`scenario.ReadModels.GetInstanceById<T>(id)`; it does not create event history.
|
|
15
|
+
Use it for handler setup, not as proof that a projection built the instance.
|
|
16
|
+
|
|
17
|
+
Requires Chronicle 19.82.0 for appended-event processing with real contexts.
|
|
18
|
+
|
|
19
|
+
`ProcessAppendedEvents(...)` folds `AppendedEvent` values rather
|
|
20
|
+
than appending them a second time, preserving their real `EventContext`
|
|
21
|
+
(sequence number and occurred time) and sorting across sources by sequence
|
|
22
|
+
number. Use distinct sequence numbers. Bare events supplied through `Given`
|
|
23
|
+
or `ProcessEventsFor` still use synthetic contexts. With seeded instances,
|
|
24
|
+
the fold's results replace the scenario's `ReadModels` instances; this does
|
|
25
|
+
not prove kernel storage or replay durability. Seeding the same key twice
|
|
26
|
+
throws `ReadModelAlreadySeeded`; models with no projection/reducer keep
|
|
27
|
+
lookup-only behavior.
|
|
28
|
+
|
|
29
|
+
Requires Chronicle 19.106.0 for shared default test-store schema generation.
|
|
30
|
+
|
|
31
|
+
Default test stores share a JSON schema generator per process,
|
|
32
|
+
while each store gets a separate schema copy. Set
|
|
33
|
+
`EventStoreForTesting.ShareSchemaGenerator = false` for per-store metadata
|
|
34
|
+
providers; custom service providers already bypass sharing. Configure the
|
|
35
|
+
static switch at suite setup, not concurrently between specifications.
|
|
@@ -245,6 +245,20 @@ same store catalog: `AmbiguousReadModelBuilder` rejects it. `[Passive]`, sequenc
|
|
|
245
245
|
selection, filters and `[Reducer]` alone do not establish projection ownership.
|
|
246
246
|
Remove the old builder when switching; registration is not a data migration.
|
|
247
247
|
|
|
248
|
+
## Partition delivery
|
|
249
|
+
|
|
250
|
+
Requires Chronicle 19.90.0 for concurrent .NET reducer partitions.
|
|
251
|
+
|
|
252
|
+
Different partitions of one .NET reducer can run at the
|
|
253
|
+
same time; batches for the same partition remain in arrival order. Keep shared
|
|
254
|
+
collaborators thread-safe, or set
|
|
255
|
+
`ChronicleOptions.Reducers.MaxConcurrentPartitions = 1`. The default is processor
|
|
256
|
+
count capped at 8; values below 1 are refused. A global ordering assumption is
|
|
257
|
+
not justified by per-partition ordering. Verified in the tagged sources listed
|
|
258
|
+
by [connection options](../cratis-chronicle-client-dotnet/references/connection-options.md):
|
|
259
|
+
`Source/Clients/DotNET/Reducers/ReducerOptions.cs`, `Reducers.cs` and
|
|
260
|
+
`Observation/PartitionedDeliveryScheduler.cs`.
|
|
261
|
+
|
|
248
262
|
## Replay
|
|
249
263
|
|
|
250
264
|
Read [references/replay-hooks-and-definition-changes.md](references/replay-hooks-and-definition-changes.md) when a self model needs `BeginReplay`/`EndReplay` hooks (they return exactly `Task`; do not assume delivery is exactly once) or when a code, key or filter change must replay (`observers.definitionEvolution`). Requires Chronicle 19.29.4 for method-fingerprint change detection; assembly versions are ignored.
|
|
@@ -33,3 +33,10 @@ partial replay only when history proves they follow all previously consumed
|
|
|
33
33
|
events per source. Interleaved history requires full replay. Manual policies
|
|
34
34
|
leave recommendations; they do not silently rebuild. Source:
|
|
35
35
|
`Documentation/hosting/configuration/observers.md`.
|
|
36
|
+
|
|
37
|
+
At **v19.110.1**, replay writes reduced state to a separate sink and switches to
|
|
38
|
+
it only when replay completes. Abandoning a failed reducer replay does not
|
|
39
|
+
publish partially rebuilt state. This is a kernel sink boundary, not proof that
|
|
40
|
+
external side effects are reversed. Verified in
|
|
41
|
+
`Source/Kernel/Core/Observation/Jobs/ReplayObserver.cs` and the tagged observer
|
|
42
|
+
configuration documentation.
|