@zio.dev/zio-blocks 0.0.51 → 0.0.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/adr/2026-07-18-data-migration.md +123 -0
- package/guides/async-getting-started.md +687 -0
- package/guides/compile-time-resource-safety-with-scope.md +6 -0
- package/guides/getting-started-with-mux.md +0 -112
- package/guides/query-dsl-extending.md +1 -1
- package/guides/query-dsl-fluent-builder.md +1 -1
- package/guides/query-dsl-reified-optics.md +1 -1
- package/guides/query-dsl-sql.md +395 -1
- package/guides/sql-checked-interpolation.md +173 -0
- package/guides/sql-transactions.md +286 -0
- package/guides/telemetry-guide.md +131 -70
- package/guides/zio-schema-migration.md +6 -6
- package/index.md +200 -559
- package/package.json +1 -1
- package/reference/async.md +1379 -531
- package/reference/chunk.md +3 -3
- package/reference/codegen/index.md +1 -1
- package/reference/combinators.md +4 -4
- package/reference/config/config-decoder.md +460 -0
- package/reference/config/config-source.md +489 -0
- package/reference/config/errors.md +278 -0
- package/reference/config/flags.md +369 -0
- package/reference/config/formats.md +314 -0
- package/reference/config/index.md +304 -0
- package/reference/config/rollout.md +336 -0
- package/reference/context.md +6 -49
- package/reference/data-migration.md +269 -0
- package/reference/datastar/attributes.md +302 -0
- package/reference/datastar/events.md +234 -0
- package/reference/datastar/index.md +256 -0
- package/reference/datastar/signals.md +230 -0
- package/reference/datastar/sse.md +295 -0
- package/reference/datastar.md +2 -2
- package/reference/docs.md +2 -2
- package/reference/endpoint/bulk-creation.md +96 -0
- package/reference/endpoint/endpoint.md +1 -0
- package/reference/endpoint/index.md +9 -89
- package/reference/endpoint/path-codec.md +12 -24
- package/reference/endpoint/route-pattern.md +4 -6
- package/reference/endpoint/segment-codec.md +19 -32
- package/reference/html.md +313 -9
- package/reference/htmx/index.md +4 -52
- package/reference/htmx/response-headers.md +240 -0
- package/reference/http-model/headers.md +735 -0
- package/reference/http-model/index.md +3 -1
- package/reference/http-model/model.md +107 -71
- package/reference/http-model/schema-codecs.md +522 -0
- package/reference/http-model/schema.md +6 -3
- package/reference/http-model/server-sent-event.md +341 -0
- package/reference/jwt.md +195 -0
- package/reference/maybe.md +128 -11
- package/reference/media-type.md +2 -2
- package/reference/mux.mdx +7 -2
- package/reference/openapi.md +3 -3
- package/reference/projection.md +654 -0
- package/reference/resource-management/index.md +1 -1
- package/reference/resource-management/resource.md +2 -98
- package/reference/resource-management/scope.md +1 -209
- package/reference/resource-management/wire.md +4 -50
- package/reference/ringbuffer/advanced.mdx +1 -1
- package/reference/ringbuffer/index.mdx +3 -3
- package/reference/ringbuffer/mpmc.mdx +38 -4
- package/reference/ringbuffer/mpsc.mdx +36 -4
- package/reference/ringbuffer/spmc.mdx +1 -1
- package/reference/ringbuffer/spsc.mdx +87 -15
- package/reference/schema/allows.md +0 -96
- package/reference/schema/binding.md +2 -2
- package/reference/schema/built-in-codecs/avro.md +2 -2
- package/reference/schema/built-in-codecs/bson.md +50 -20
- package/reference/schema/built-in-codecs/csv.md +2 -2
- package/reference/schema/built-in-codecs/index.md +3 -3
- package/reference/schema/built-in-codecs/json/index.md +2 -2
- package/reference/schema/built-in-codecs/json/json.md +1 -0
- package/reference/schema/built-in-codecs/messagepack.md +3 -3
- package/reference/schema/built-in-codecs/thrift.md +2 -2
- package/reference/schema/built-in-codecs/toon.md +3 -3
- package/reference/schema/built-in-codecs/yaml.md +2 -2
- package/reference/schema/codec.md +11 -11
- package/reference/schema/dynamic-optic.md +48 -3
- package/reference/schema/dynamic-schema.md +3 -3
- package/reference/schema/index.md +2 -0
- package/reference/schema/path-interpolator.md +2 -0
- package/reference/schema/reflect-transformer.md +140 -0
- package/reference/schema/schema-evolution/as.md +4 -4
- package/reference/schema/schema-evolution/into.md +2 -2
- package/reference/schema/schema-expr.md +2 -2
- package/reference/schema/schema-search.md +263 -0
- package/reference/schema/schema.md +10 -2
- package/reference/schema/type-class-derivation.md +1 -1
- package/reference/smithy.md +502 -3
- package/reference/sql/db-codec-deriver.md +3 -3
- package/reference/sql/db-codec.md +22 -22
- package/reference/sql/db-con.md +4 -4
- package/reference/sql/db-connection.md +1 -1
- package/reference/sql/db-param.md +1 -1
- package/reference/sql/db-result-reader.md +4 -2
- package/reference/sql/db-tx.md +46 -14
- package/reference/sql/ddl.md +1 -1
- package/reference/sql/frag.md +44 -10
- package/reference/sql/index.md +7 -7
- package/reference/sql/repo.md +15 -15
- package/reference/sql/sql-dialect.md +1 -1
- package/reference/sql/sql-logger.md +1 -1
- package/reference/sql/sql-name-mapper.md +3 -3
- package/reference/sql/table-metadata.md +3 -3
- package/reference/sql/table.md +10 -10
- package/reference/sql/transactor-zio.md +1 -1
- package/reference/sql/transactor.md +21 -11
- package/reference/sql-zio.md +2 -2
- package/reference/streams/core/index.md +32 -0
- package/reference/streams/{pipeline.md → core/pipeline.md} +210 -74
- package/reference/streams/{sink.md → core/sink.md} +331 -353
- package/reference/streams/{stream.md → core/stream.md} +919 -209
- package/reference/streams/execution-and-compatibility/async-execution.md +822 -0
- package/reference/streams/execution-and-compatibility/index.md +35 -0
- package/reference/streams/execution-and-compatibility/platform-differences.md +297 -0
- package/reference/streams/execution-and-compatibility/scala-2-compatibility.md +88 -0
- package/reference/streams/execution-and-compatibility/zero-boxing.md +393 -0
- package/reference/streams/index.md +140 -67
- package/reference/streams/primitives/index.md +30 -0
- package/reference/streams/primitives/reader.md +1992 -0
- package/reference/streams/{writer.md → primitives/writer.md} +254 -98
- package/reference/telemetry/common/any-value.md +90 -0
- package/reference/telemetry/common/attribute-key.md +87 -0
- package/reference/telemetry/common/attributes.md +118 -0
- package/reference/telemetry/common/index.md +39 -0
- package/reference/telemetry/common/instrumentation-scope.md +24 -0
- package/reference/telemetry/common/resource.md +34 -0
- package/reference/telemetry/index.md +311 -0
- package/reference/telemetry/logging/index.md +197 -0
- package/reference/telemetry/logging/log-enrichment.md +72 -0
- package/reference/telemetry/logging/log-formatter.md +100 -0
- package/reference/telemetry/logging/log-record-processor.md +56 -0
- package/reference/telemetry/logging/log-record.md +44 -0
- package/reference/telemetry/logging/log-writer.md +64 -0
- package/reference/telemetry/logging/logger-provider.md +142 -0
- package/reference/telemetry/logging/logger.md +83 -0
- package/reference/telemetry/logging/severity.md +62 -0
- package/reference/telemetry/metrics/index.md +150 -0
- package/reference/telemetry/metrics/instruments.md +183 -0
- package/reference/telemetry/metrics/labeled-instruments.md +74 -0
- package/reference/telemetry/metrics/meter-provider.md +76 -0
- package/reference/telemetry/metrics/meter.md +98 -0
- package/reference/telemetry/metrics/metric-data.md +57 -0
- package/reference/telemetry/otel/custom-exporter.md +216 -0
- package/reference/telemetry/otel/index.md +212 -0
- package/reference/telemetry/tracing/index.md +155 -0
- package/reference/telemetry/tracing/sampler.md +89 -0
- package/reference/telemetry/tracing/span-builder.md +57 -0
- package/reference/telemetry/tracing/span-context.md +39 -0
- package/reference/telemetry/tracing/span-data.md +32 -0
- package/reference/telemetry/tracing/span-kind.md +55 -0
- package/reference/telemetry/tracing/span-processor.md +53 -0
- package/reference/telemetry/tracing/span-status.md +47 -0
- package/reference/telemetry/tracing/span.md +117 -0
- package/reference/telemetry/tracing/tracer-provider.md +91 -0
- package/reference/telemetry/tracing/tracer.md +52 -0
- package/reference/typeid.md +0 -64
- package/sidebars.js +365 -185
- package/undocumented-report.md +528 -270
- package/reference/config.md +0 -158
- package/reference/streams/concurrent-operators.md +0 -106
- package/reference/streams/reader.md +0 -1284
- package/reference/streams/scala-2-compatibility.md +0 -55
- package/reference/streams/zero-boxing.md +0 -275
- package/reference/telemetry.md +0 -693
|
@@ -0,0 +1,654 @@
|
|
|
1
|
+
---
|
|
2
|
+
id: projection
|
|
3
|
+
title: "Projection"
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
`zio-blocks-projection` provides event-sourced projections backed by per-entity
|
|
7
|
+
SQLite files. Each projection type gets its own isolated `.db` file, giving you
|
|
8
|
+
WAL-mode writes, offline queryability, and zero contention between unrelated
|
|
9
|
+
entities.
|
|
10
|
+
|
|
11
|
+
## Installation
|
|
12
|
+
|
|
13
|
+
```scala
|
|
14
|
+
libraryDependencies += "dev.zio" %% "zio-blocks-projection" % "0.0.56"
|
|
15
|
+
|
|
16
|
+
// JVM only — SQLite-backed stores
|
|
17
|
+
libraryDependencies += "dev.zio" %% "zio-blocks-projection" % "0.0.56"
|
|
18
|
+
|
|
19
|
+
// ZIO integration (transactor cache, engine runtime)
|
|
20
|
+
libraryDependencies += "dev.zio" %% "zio-blocks-sql-zio" % "0.0.56"
|
|
21
|
+
|
|
22
|
+
// for SQLite persistence add: libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0" // or InMemory fallback if not present
|
|
23
|
+
libraryDependencies += "org.xerial" % "sqlite-jdbc" % "3.53.4.0"
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
`sqlite-jdbc` is `% Test` by default in `zio-blocks-projection`; consumers that
|
|
27
|
+
want SQLite persistence at runtime must add the dependency above explicitly.
|
|
28
|
+
If it is not present the engine falls back to `InMemoryProjectionStore`.
|
|
29
|
+
|
|
30
|
+
## Overview
|
|
31
|
+
|
|
32
|
+
Projections turn an event stream into queryable read models. Instead of
|
|
33
|
+
querying the event store directly, you define how events map to entities, and
|
|
34
|
+
the engine materializes those entities into SQLite files you can read
|
|
35
|
+
anytime.
|
|
36
|
+
|
|
37
|
+
**Why per-entity SQLite?**
|
|
38
|
+
|
|
39
|
+
A single shared database means every projection contends on the same locks,
|
|
40
|
+
the same WAL, and the same file. With per-entity files, each projection type
|
|
41
|
+
gets isolated storage:
|
|
42
|
+
|
|
43
|
+
| Approach | Contention | WAL contention | Offline access | Backup granularity |
|
|
44
|
+
|----------|-----------|---------------|----------------|-------------------|
|
|
45
|
+
| Single DB | High | High | Yes | Full DB |
|
|
46
|
+
| Per-entity SQLite | Zero | Zero | Yes | Per projection |
|
|
47
|
+
|
|
48
|
+
Each projection store lives at `<basePath>/<specName>.db`. Global
|
|
49
|
+
aggregates go to `global/<specName>.db`. The `TransactorCache` reuses
|
|
50
|
+
connections via an LRU cache so you don't open hundreds of file handles.
|
|
51
|
+
|
|
52
|
+
**Three projection scopes:**
|
|
53
|
+
|
|
54
|
+
- **PerEntity** (default): one row per entity ID. Events routed by
|
|
55
|
+
`ctx.entityId`. Good for user profiles, order state, anything keyed by
|
|
56
|
+
a single ID.
|
|
57
|
+
- **CrossEntity**: events routed by an arbitrary key extracted from the
|
|
58
|
+
event. Multiple event types converge on the same projection. Good for
|
|
59
|
+
"all repos owned by user X" views.
|
|
60
|
+
- **Global**: a single shared row (or a few rows keyed by a grouping
|
|
61
|
+
string). Events from all sources funnel into atomic counter updates.
|
|
62
|
+
Good for dashboards, daily aggregates, live counters.
|
|
63
|
+
|
|
64
|
+
## Quick Start
|
|
65
|
+
|
|
66
|
+
Define your events, your projection entity, wire up a spec, and start the
|
|
67
|
+
engine:
|
|
68
|
+
|
|
69
|
+
```scala
|
|
70
|
+
import zio.*
|
|
71
|
+
import zio.blocks.projection.*
|
|
72
|
+
import zio.blocks.projection.testing.TestEngine
|
|
73
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
74
|
+
|
|
75
|
+
// 1. Define events
|
|
76
|
+
case class UserCreated(name: String, email: String)
|
|
77
|
+
object UserCreated {
|
|
78
|
+
implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
|
|
79
|
+
}
|
|
80
|
+
|
|
81
|
+
// 2. Define the projection entity
|
|
82
|
+
case class UserProfile(@Modifier.id id: String, name: String, email: String)
|
|
83
|
+
object UserProfile {
|
|
84
|
+
implicit val schema: Schema[UserProfile] = Schema.derived[UserProfile]
|
|
85
|
+
implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
|
|
86
|
+
}
|
|
87
|
+
|
|
88
|
+
// 3. Define the projection
|
|
89
|
+
val projection = Projection[UserProfile]("userProfiles")
|
|
90
|
+
.from("users")
|
|
91
|
+
.routeToSelf
|
|
92
|
+
.on[UserCreated]
|
|
93
|
+
.insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
|
|
94
|
+
|
|
95
|
+
// 4. Create a test engine (auto-creates stores and hubs)
|
|
96
|
+
val program: ZIO[Scope, Throwable, Unit] = for {
|
|
97
|
+
engine <- TestEngine.make(projection)
|
|
98
|
+
// Append events
|
|
99
|
+
_ <- engine.append("user-1", UserCreated("Alice", "alice@example.com"))
|
|
100
|
+
_ <- engine.append("user-2", UserCreated("Bob", "bob@example.com"))
|
|
101
|
+
// Query
|
|
102
|
+
u1 <- engine.query(projection, "user-1")
|
|
103
|
+
u2 <- engine.query(projection, "user-2")
|
|
104
|
+
} yield ()
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
The engine starts a catch-up fiber that reads all historical events, then
|
|
108
|
+
switches to live mode via the Hub subscription. Query results appear once
|
|
109
|
+
the engine processes the relevant events.
|
|
110
|
+
|
|
111
|
+
## Entity Path Conventions
|
|
112
|
+
|
|
113
|
+
`EntityPath[A]` tells the engine two things: which folder stores the SQLite
|
|
114
|
+
file, and which field holds the entity ID. The ID field is identified via
|
|
115
|
+
`@Modifier.id` (from `zio.blocks.schema.Modifier`) or, for ergonomics, a
|
|
116
|
+
field named `id`.
|
|
117
|
+
|
|
118
|
+
```scala
|
|
119
|
+
import zio.blocks.projection.*
|
|
120
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
121
|
+
|
|
122
|
+
case class Order(@Modifier.id id: Long, total: BigDecimal)
|
|
123
|
+
object Order {
|
|
124
|
+
implicit val schema: Schema[Order] = Schema.derived[Order]
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
val orderId = "42" // String id
|
|
128
|
+
val raw = orderId // String = "42"
|
|
129
|
+
```
|
|
130
|
+
|
|
131
|
+
```scala
|
|
132
|
+
import zio.blocks.projection.*
|
|
133
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
134
|
+
|
|
135
|
+
case class UserProfile(@Modifier.id userId: String, name: String, email: String)
|
|
136
|
+
object UserProfile {
|
|
137
|
+
implicit val schema: Schema[UserProfile] = Schema.derived[UserProfile]
|
|
138
|
+
implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
|
|
139
|
+
}
|
|
140
|
+
// basePath = "users", entityIdField = "userId"
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
**Derivation rules** (`EntityPath.derived[A]`):
|
|
144
|
+
|
|
145
|
+
1. Look for a field annotated with `@Modifier.id`. If found, use it as
|
|
146
|
+
the entity ID field.
|
|
147
|
+
2. Otherwise, find the field named `id`.
|
|
148
|
+
3. If neither is found, fail with `Entity must have @Modifier.id field`.
|
|
149
|
+
4. Derive the folder name from the ID field name:
|
|
150
|
+
- Strip trailing `Id` suffix: `userId` → `user`
|
|
151
|
+
- Convert to snake_case: `userId` → `user_id`
|
|
152
|
+
- Pluralize: `user` → `users`
|
|
153
|
+
- Result: `basePath = "users"`, `entityIdField = "userId"`
|
|
154
|
+
|
|
155
|
+
If the field is exactly `id`, `basePath` is `ids` — prefer `userId` or use `@path` to override.
|
|
156
|
+
|
|
157
|
+
**Override with `@path`:**
|
|
158
|
+
|
|
159
|
+
```scala
|
|
160
|
+
import zio.blocks.projection.*
|
|
161
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
162
|
+
|
|
163
|
+
@path("custom_users")
|
|
164
|
+
case class UserProfile(@Modifier.id id: String, name: String)
|
|
165
|
+
object UserProfile {
|
|
166
|
+
implicit val schema: Schema[UserProfile] = Schema.derived[UserProfile]
|
|
167
|
+
implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
|
|
168
|
+
}
|
|
169
|
+
// basePath = "custom_users", entityIdField = "id"
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
**Manual construction:**
|
|
173
|
+
|
|
174
|
+
```scala
|
|
175
|
+
import zio.blocks.projection.*
|
|
176
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
177
|
+
|
|
178
|
+
case class MyEntity(@Modifier.id id: String, value: Int)
|
|
179
|
+
object MyEntity {
|
|
180
|
+
implicit val schema: Schema[MyEntity] = Schema.derived[MyEntity]
|
|
181
|
+
implicit val entityPath: EntityPath[MyEntity] = EntityPath[MyEntity]("my_entities", "id")
|
|
182
|
+
}
|
|
183
|
+
```
|
|
184
|
+
|
|
185
|
+
## Multi-Source Projections
|
|
186
|
+
|
|
187
|
+
Cross-entity projections receive events from multiple sources and route them
|
|
188
|
+
by a key extracted from the event:
|
|
189
|
+
|
|
190
|
+
```scala
|
|
191
|
+
import zio.blocks.projection.*
|
|
192
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
193
|
+
|
|
194
|
+
case class RepoCreated(ownerId: String, repoName: String)
|
|
195
|
+
object RepoCreated {
|
|
196
|
+
implicit val schema: Schema[RepoCreated] = Schema.derived[RepoCreated]
|
|
197
|
+
}
|
|
198
|
+
|
|
199
|
+
case class RepoListEntry(@Modifier.id id: String, ownerId: String, repoName: String)
|
|
200
|
+
object RepoListEntry {
|
|
201
|
+
implicit val schema: Schema[RepoListEntry] = Schema.derived[RepoListEntry]
|
|
202
|
+
implicit val entityPath: EntityPath[RepoListEntry] = EntityPath.derived[RepoListEntry]
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
val spec = Projection[RepoListEntry]("repoListEntries")
|
|
206
|
+
.from("repos")
|
|
207
|
+
.routedBy[RepoCreated](_.ownerId)
|
|
208
|
+
.on[RepoCreated]
|
|
209
|
+
.custom((e, _) => ProjectionAction.Upsert(RepoListEntry(e.ownerId, e.ownerId, e.repoName)))
|
|
210
|
+
|
|
211
|
+
// spec.scope == ProjectionScope.CrossEntity(extractor)
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
The `routedBy` call tells the engine to extract a routing key from each
|
|
215
|
+
event. Events with the same key go to the same shard store, so querying
|
|
216
|
+
`engine.query(spec, "alice")` searches Alice's shard first.
|
|
217
|
+
|
|
218
|
+
**Routing modes:**
|
|
219
|
+
|
|
220
|
+
| Mode | Behavior | Scope derived |
|
|
221
|
+
|------|----------|---------------|
|
|
222
|
+
| `.routeToSelf` | Uses `ctx.entityId` as key | `PerEntity` |
|
|
223
|
+
| `.routedBy[E](_.field)` | Extracts key from event | `CrossEntity` |
|
|
224
|
+
| `.routeToAll` | All events go to same store | `Global` (if `isGlobal`) |
|
|
225
|
+
|
|
226
|
+
## Aggregate Projections
|
|
227
|
+
|
|
228
|
+
Global projections aggregate events from multiple sources into a single
|
|
229
|
+
row (or a few rows keyed by a grouping string). The engine applies atomic
|
|
230
|
+
counter updates:
|
|
231
|
+
|
|
232
|
+
```scala
|
|
233
|
+
import zio.blocks.projection.*
|
|
234
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
235
|
+
|
|
236
|
+
case class UserCreated(name: String, email: String)
|
|
237
|
+
object UserCreated {
|
|
238
|
+
implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
|
|
239
|
+
}
|
|
240
|
+
case class RepoCreated(ownerId: String, repoName: String)
|
|
241
|
+
object RepoCreated {
|
|
242
|
+
implicit val schema: Schema[RepoCreated] = Schema.derived[RepoCreated]
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
case class DailyStats(@Modifier.id date: String, userCount: Int, repoCount: Int)
|
|
246
|
+
object DailyStats {
|
|
247
|
+
implicit val schema: Schema[DailyStats] = Schema.derived[DailyStats]
|
|
248
|
+
implicit val entityPath: EntityPath[DailyStats] = EntityPath.derived[DailyStats]
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
val spec = Projection
|
|
252
|
+
.global[DailyStats]("dailyStats")
|
|
253
|
+
.from("users")
|
|
254
|
+
.routeToAll
|
|
255
|
+
.on[UserCreated]
|
|
256
|
+
.aggregate(FieldUpdate.Increment("user_count", 1L))
|
|
257
|
+
.from("repos")
|
|
258
|
+
.routeToAll
|
|
259
|
+
.on[RepoCreated]
|
|
260
|
+
.aggregate(FieldUpdate.Increment("repo_count", 1L))
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
**FieldUpdate operations:**
|
|
264
|
+
|
|
265
|
+
| Operation | SQL translation | Description |
|
|
266
|
+
|-----------|----------------|-------------|
|
|
267
|
+
| `Set(field, value)` | `SET col = ?` | Replace field value |
|
|
268
|
+
| `Increment(field, by)` | `SET col = COALESCE(col,0) + ?` | Atomic increment |
|
|
269
|
+
| `Decrement(field, by)` | `SET col = COALESCE(col,0) - ?` | Atomic decrement |
|
|
270
|
+
| `Max(field, value)` | `SET col = MAX(COALESCE(col, ?), ?)` | Keep highest value |
|
|
271
|
+
| `Min(field, value)` | `SET col = MIN(COALESCE(col, ?), ?)` | Keep lowest value |
|
|
272
|
+
|
|
273
|
+
The `COALESCE` handles missing rows. The engine runs `INSERT OR IGNORE`
|
|
274
|
+
before any `UPDATE`, so incrementing a counter that doesn't exist yet
|
|
275
|
+
creates the row with value `1` (not `0 + 1`).
|
|
276
|
+
|
|
277
|
+
**Concurrent safety:** Counter operations are atomic at the SQL level. Ten
|
|
278
|
+
fibers each incrementing the same counter produce the correct total without
|
|
279
|
+
lost updates.
|
|
280
|
+
|
|
281
|
+
## Schema Evolution
|
|
282
|
+
|
|
283
|
+
When you change a projection entity's schema (add a field, rename a column),
|
|
284
|
+
the engine detects the mismatch and rebuilds the projection from events.
|
|
285
|
+
|
|
286
|
+
### How it works
|
|
287
|
+
|
|
288
|
+
1. On startup, `ProjectionEngine` computes `SchemaHash.compute[A]` for
|
|
289
|
+
each spec's entity type. This is a SHA-256 hash of the schema structure
|
|
290
|
+
(field names, types, order).
|
|
291
|
+
2. The hash is stored in `_projection_meta.schema_hash`.
|
|
292
|
+
3. If the stored hash doesn't match the current hash, the engine rebuilds:
|
|
293
|
+
- Truncate the projection store
|
|
294
|
+
- Replay all events from the EventStore through the spec's handlers
|
|
295
|
+
- Store the new hash
|
|
296
|
+
|
|
297
|
+
```scala
|
|
298
|
+
import zio.blocks.projection.*
|
|
299
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
300
|
+
|
|
301
|
+
case class UserProfileV1(@Modifier.id id: String, name: String)
|
|
302
|
+
object UserProfileV1 {
|
|
303
|
+
implicit val schema: Schema[UserProfileV1] = Schema.derived[UserProfileV1]
|
|
304
|
+
}
|
|
305
|
+
|
|
306
|
+
case class UserProfileV2(@Modifier.id id: String, name: String, email: String)
|
|
307
|
+
object UserProfileV2 {
|
|
308
|
+
implicit val schema: Schema[UserProfileV2] = Schema.derived[UserProfileV2]
|
|
309
|
+
}
|
|
310
|
+
|
|
311
|
+
val hashV1 = SchemaHash.compute[UserProfileV1]
|
|
312
|
+
val hashV2 = SchemaHash.compute[UserProfileV2]
|
|
313
|
+
// hashV1 != hashV2 because V2 has an extra "email" field
|
|
314
|
+
```
|
|
315
|
+
|
|
316
|
+
### Lazy rebuild
|
|
317
|
+
|
|
318
|
+
Set `lazyRebuild = true` in `ProjectionEngineConfig` to defer rebuilds.
|
|
319
|
+
The engine marks specs needing rebuild but doesn't block startup. The first
|
|
320
|
+
query to a spec triggers its rebuild:
|
|
321
|
+
|
|
322
|
+
```scala
|
|
323
|
+
import zio.blocks.projection.ProjectionEngineConfig
|
|
324
|
+
|
|
325
|
+
val config = ProjectionEngineConfig(
|
|
326
|
+
batchSize = 100,
|
|
327
|
+
batchTimeout = zio.Duration.fromMillis(50),
|
|
328
|
+
ringCapacity = 4096,
|
|
329
|
+
rebuildParallelism = 4,
|
|
330
|
+
lazyRebuild = true
|
|
331
|
+
)
|
|
332
|
+
```
|
|
333
|
+
|
|
334
|
+
### Migration shortcut
|
|
335
|
+
|
|
336
|
+
For simple `AddField` migrations (adding columns with defaults), the engine
|
|
337
|
+
uses `ALTER TABLE ADD COLUMN` instead of a full rebuild. This is much faster
|
|
338
|
+
for large projections:
|
|
339
|
+
|
|
340
|
+
```scala
|
|
341
|
+
import zio.blocks.schema.migration.Migration
|
|
342
|
+
|
|
343
|
+
// Assumes UserProfileV1 and UserProfileV2 are defined as above
|
|
344
|
+
val migration = Migration
|
|
345
|
+
.newBuilder[UserProfileV1, UserProfileV2]
|
|
346
|
+
.addField(_.email, "")
|
|
347
|
+
.build
|
|
348
|
+
|
|
349
|
+
// Engine detects this is a simple AddField migration
|
|
350
|
+
// and uses ALTER TABLE instead of full rebuild
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
## Event Tagging and Migration
|
|
354
|
+
|
|
355
|
+
Event tags identify variant cases for serialization. By default, the tag is
|
|
356
|
+
the case class name (e.g., `"UserCreated"`). You can override with numeric
|
|
357
|
+
tags or rename cases across versions.
|
|
358
|
+
|
|
359
|
+
### String tags (default)
|
|
360
|
+
|
|
361
|
+
For a sealed trait `UserEvent` with cases `UserCreated` and `UserDeleted`,
|
|
362
|
+
the tags are `"UserCreated"` and `"UserDeleted"`.
|
|
363
|
+
|
|
364
|
+
### Numeric tags
|
|
365
|
+
|
|
366
|
+
Use the `@eventTag` annotation for stable numeric identifiers:
|
|
367
|
+
|
|
368
|
+
```scala
|
|
369
|
+
import zio.blocks.projection.eventTag
|
|
370
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
371
|
+
|
|
372
|
+
sealed trait UserEvent
|
|
373
|
+
object UserEvent {
|
|
374
|
+
@eventTag(1)
|
|
375
|
+
case class Created(name: String) extends UserEvent
|
|
376
|
+
object Created {
|
|
377
|
+
implicit val schema: Schema[Created] = Schema.derived[Created]
|
|
378
|
+
}
|
|
379
|
+
|
|
380
|
+
@eventTag(2)
|
|
381
|
+
case class Deleted(userId: String) extends UserEvent
|
|
382
|
+
object Deleted {
|
|
383
|
+
implicit val schema: Schema[Deleted] = Schema.derived[Deleted]
|
|
384
|
+
}
|
|
385
|
+
|
|
386
|
+
implicit val schema: Schema[UserEvent] = Schema.derived[UserEvent]
|
|
387
|
+
}
|
|
388
|
+
```
|
|
389
|
+
|
|
390
|
+
### Tag aliases via Migration
|
|
391
|
+
|
|
392
|
+
When you rename a case, old events in the database still have the old tag.
|
|
393
|
+
Use `Migration.renameCase` to create an alias:
|
|
394
|
+
|
|
395
|
+
```scala
|
|
396
|
+
import zio.blocks.schema.migration.Migration
|
|
397
|
+
import zio.blocks.projection.TagResolver
|
|
398
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
399
|
+
|
|
400
|
+
sealed trait LoginEvent
|
|
401
|
+
object LoginEvent {
|
|
402
|
+
case class UserLoggedIn(userId: String) extends LoginEvent
|
|
403
|
+
object UserLoggedIn {
|
|
404
|
+
implicit val loggedInSchema: Schema[UserLoggedIn] = Schema.derived
|
|
405
|
+
}
|
|
406
|
+
case class UserAuthenticated(userId: String) extends LoginEvent
|
|
407
|
+
object UserAuthenticated {
|
|
408
|
+
implicit val authSchema: Schema[UserAuthenticated] = Schema.derived
|
|
409
|
+
}
|
|
410
|
+
implicit val loginEventSchema: Schema[LoginEvent] = Schema.derived
|
|
411
|
+
}
|
|
412
|
+
|
|
413
|
+
val migration = Migration
|
|
414
|
+
.newBuilder[LoginEvent, LoginEvent]
|
|
415
|
+
.renameCase("UserLoggedIn", "UserAuthenticated")
|
|
416
|
+
.build
|
|
417
|
+
|
|
418
|
+
val tagInfo = TagResolver.resolve[LoginEvent](migration)
|
|
419
|
+
// tagInfo.aliases == Map("UserLoggedIn" -> "UserAuthenticated")
|
|
420
|
+
// tagInfo.allTags == Set("UserLoggedIn", "UserAuthenticated")
|
|
421
|
+
```
|
|
422
|
+
|
|
423
|
+
The `TagInfo` object provides:
|
|
424
|
+
|
|
425
|
+
- `expandRequested(tags)` — returns the union of requested tags plus all
|
|
426
|
+
their old aliases, so selective reads fetch both old and new events
|
|
427
|
+
- `isOldTag(tag)` — true if the tag was renamed
|
|
428
|
+
- `currentTagFor(oldTag)` — returns the current tag for an old tag
|
|
429
|
+
- `normalize(tag)` — maps old tags to current tags
|
|
430
|
+
|
|
431
|
+
### Transitive chains
|
|
432
|
+
|
|
433
|
+
If you rename `A → B` then `B → C`, the resolver collapses this to `A → C`.
|
|
434
|
+
All three tags are queryable.
|
|
435
|
+
|
|
436
|
+
### Startup warning
|
|
437
|
+
|
|
438
|
+
On startup, the `SQLiteEventStore` checks all distinct tags in the database
|
|
439
|
+
against the known tag set. Unknown tags produce a warning log, not an error.
|
|
440
|
+
This catches typos or events from a different schema version without crashing
|
|
441
|
+
the application.
|
|
442
|
+
|
|
443
|
+
## Testing
|
|
444
|
+
|
|
445
|
+
The projection module ships with in-memory implementations for unit tests.
|
|
446
|
+
No SQLite required.
|
|
447
|
+
|
|
448
|
+
### InMemoryProjectionStore
|
|
449
|
+
|
|
450
|
+
A `ProjectionStore[A]` backed by `Ref[Map[String, A]]`. Supports all
|
|
451
|
+
operations: insert, upsert, updateFields, delete, truncate, schema hash
|
|
452
|
+
tracking.
|
|
453
|
+
|
|
454
|
+
```scala
|
|
455
|
+
import zio.*
|
|
456
|
+
import zio.blocks.projection.*
|
|
457
|
+
import zio.blocks.projection.testing.InMemoryProjectionStore
|
|
458
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
459
|
+
|
|
460
|
+
case class UserProfile(@Modifier.id id: String, name: String, email: String)
|
|
461
|
+
object UserProfile {
|
|
462
|
+
implicit val schema: Schema[UserProfile] = Schema.derived[UserProfile]
|
|
463
|
+
implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
|
|
464
|
+
}
|
|
465
|
+
|
|
466
|
+
val test: ZIO[Any, Throwable, Unit] = for {
|
|
467
|
+
store <- InMemoryProjectionStore.make[UserProfile]
|
|
468
|
+
_ <- store.insert(UserProfile("u1", "Alice", "alice@example.com"))
|
|
469
|
+
alice <- store.findById("u1")
|
|
470
|
+
// alice == Some(UserProfile("u1", "Alice", "alice@example.com"))
|
|
471
|
+
} yield ()
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
### TestEngine
|
|
475
|
+
|
|
476
|
+
A simple test helper that auto-creates `InMemoryProjectionStore` per
|
|
477
|
+
projection and `Hub` per source. No manual `makeWithStores` wiring:
|
|
478
|
+
|
|
479
|
+
```scala
|
|
480
|
+
import zio.*
|
|
481
|
+
import zio.blocks.projection.*
|
|
482
|
+
import zio.blocks.projection.testing.TestEngine
|
|
483
|
+
import zio.blocks.schema.{Modifier, Schema}
|
|
484
|
+
|
|
485
|
+
case class UserProfile(@Modifier.id id: String, name: String, email: String)
|
|
486
|
+
object UserProfile {
|
|
487
|
+
implicit val schema: Schema[UserProfile] = Schema.derived[UserProfile]
|
|
488
|
+
implicit val entityPath: EntityPath[UserProfile] = EntityPath.derived[UserProfile]
|
|
489
|
+
}
|
|
490
|
+
case class UserCreated(name: String, email: String)
|
|
491
|
+
object UserCreated {
|
|
492
|
+
implicit val schema: Schema[UserCreated] = Schema.derived[UserCreated]
|
|
493
|
+
}
|
|
494
|
+
|
|
495
|
+
val projection = Projection[UserProfile]("userProfiles")
|
|
496
|
+
.from("users").routeToSelf
|
|
497
|
+
.on[UserCreated]
|
|
498
|
+
.insert((e, ctx) => UserProfile(ctx.entityId, e.name, e.email))
|
|
499
|
+
|
|
500
|
+
val test: ZIO[Scope, Throwable, Unit] = for {
|
|
501
|
+
engine <- TestEngine.make(projection)
|
|
502
|
+
_ <- engine.append("user-1", UserCreated("Alice", "alice@example.com"))
|
|
503
|
+
result <- engine.query(projection, "user-1")
|
|
504
|
+
// result == Some(UserProfile("user-1", "Alice", "alice@example.com"))
|
|
505
|
+
} yield ()
|
|
506
|
+
```
|
|
507
|
+
|
|
508
|
+
### TestContext
|
|
509
|
+
|
|
510
|
+
Factory for `ProjectionContext` values in tests:
|
|
511
|
+
|
|
512
|
+
```scala
|
|
513
|
+
import zio.blocks.projection.testing.TestContext
|
|
514
|
+
import java.time.Instant
|
|
515
|
+
|
|
516
|
+
val ctx1 = TestContext.make(entityId = "user-1")
|
|
517
|
+
// ProjectionContext("user-1", now, 0, None)
|
|
518
|
+
|
|
519
|
+
val ctx2 = TestContext.makeWithSource(entityId = "user-1", sourceEntityId = "src-1")
|
|
520
|
+
// ProjectionContext("user-1", now, 0, Some("src-1"))
|
|
521
|
+
|
|
522
|
+
val ctx3 = TestContext.withSeq(entityId = "user-1", seq = 42L)
|
|
523
|
+
// ProjectionContext("user-1", now, 42, None)
|
|
524
|
+
```
|
|
525
|
+
|
|
526
|
+
## Configuration
|
|
527
|
+
|
|
528
|
+
### ProjectionEngineConfig
|
|
529
|
+
|
|
530
|
+
Controls batching, ring buffer size, rebuild behavior, and schema
|
|
531
|
+
evolution:
|
|
532
|
+
|
|
533
|
+
```scala
|
|
534
|
+
import zio.*
|
|
535
|
+
import zio.blocks.projection.ProjectionEngineConfig
|
|
536
|
+
|
|
537
|
+
val config = ProjectionEngineConfig(
|
|
538
|
+
batchSize = 100, // events per batch in catch-up
|
|
539
|
+
batchTimeout = 50.millis, // max wait before flushing partial batch
|
|
540
|
+
ringCapacity = 4096, // Hub capacity for live events
|
|
541
|
+
rebuildParallelism = 4, // concurrent rebuild fibers
|
|
542
|
+
lazyRebuild = false // true = defer rebuilds to first query
|
|
543
|
+
)
|
|
544
|
+
```
|
|
545
|
+
|
|
546
|
+
| Parameter | Default | Description |
|
|
547
|
+
|-----------|---------|-------------|
|
|
548
|
+
| `batchSize` | 100 | Number of events processed per batch during catch-up |
|
|
549
|
+
| `batchTimeout` | 50ms | Maximum wait before flushing a partial batch |
|
|
550
|
+
| `ringCapacity` | 4096 | Capacity of the Hub ring buffer for live events |
|
|
551
|
+
| `rebuildParallelism` | 4 | Number of concurrent rebuild fibers |
|
|
552
|
+
| `lazyRebuild` | false | Defer schema rebuilds to first query |
|
|
553
|
+
| `evolution` | default | `SchemaEvolutionConfig` sub-config |
|
|
554
|
+
|
|
555
|
+
### TransactorCacheConfig
|
|
556
|
+
|
|
557
|
+
Controls the SQLite transactor cache:
|
|
558
|
+
|
|
559
|
+
```scala
|
|
560
|
+
import zio.blocks.projection.TransactorCacheConfig
|
|
561
|
+
|
|
562
|
+
val cacheConfig = TransactorCacheConfig(maxSize = 256)
|
|
563
|
+
```
|
|
564
|
+
|
|
565
|
+
| Parameter | Default | Description |
|
|
566
|
+
|-----------|---------|-------------|
|
|
567
|
+
| `maxSize` | 256 | Maximum number of cached SQLite connections |
|
|
568
|
+
|
|
569
|
+
The cache uses LRU eviction. Each cached transactor sets `PRAGMA journal_mode=WAL`
|
|
570
|
+
and `PRAGMA synchronous=NORMAL` on first open. The cache is `Scope`-managed:
|
|
571
|
+
all transactors close when the scope exits.
|
|
572
|
+
|
|
573
|
+
### Scope lifecycle
|
|
574
|
+
|
|
575
|
+
The `ProjectionEngine.make` and `TransactorCache.make` methods require
|
|
576
|
+
`ZIO[Scope, ...]`. This ensures resources are cleaned up when your
|
|
577
|
+
application shuts down:
|
|
578
|
+
|
|
579
|
+
```scala
|
|
580
|
+
import zio.*
|
|
581
|
+
import zio.blocks.projection.*
|
|
582
|
+
|
|
583
|
+
val app: ZIO[Scope, Throwable, Unit] = for {
|
|
584
|
+
cache <- TransactorCache.make()
|
|
585
|
+
engine <- ProjectionEngine.makeWithConfig(
|
|
586
|
+
ProjectionEngineConfig.default,
|
|
587
|
+
/* specs... */
|
|
588
|
+
)
|
|
589
|
+
_ <- engine.start
|
|
590
|
+
} yield ()
|
|
591
|
+
// cache and engine cleaned up when scope closes
|
|
592
|
+
```
|
|
593
|
+
|
|
594
|
+
## API Reference
|
|
595
|
+
|
|
596
|
+
| Type | Package | Description |
|
|
597
|
+
|------|---------|-------------|
|
|
598
|
+
| `@Modifier.id` | `zio.blocks.schema.Modifier` | Annotation marking the entity ID field |
|
|
599
|
+
| `EntityPath[A]` | `zio.blocks.projection` | Folder name + ID field for a projection entity |
|
|
600
|
+
| `@path(name)` | `zio.blocks.projection` | Annotation to override the derived folder name |
|
|
601
|
+
| `@eventTag(n)` | `zio.blocks.projection` | Annotation for numeric event tags |
|
|
602
|
+
| `Projection[A]` | `zio.blocks.projection` | Defines a projection: name, schema, handlers, routing |
|
|
603
|
+
| `ProjectionAction[+A]` | `zio.blocks.projection` | Enum: Insert, Upsert, Update, Delete, Truncate, Noop |
|
|
604
|
+
| `FieldUpdate` | `zio.blocks.projection` | Enum: Set, Increment, Decrement, Max, Min |
|
|
605
|
+
| `ProjectionContext` | `zio.blocks.projection` | Event metadata: entityId, timestamp, seq, sourceEntityId |
|
|
606
|
+
| `EventEnvelope[+E]` | `zio.blocks.projection` | Event wrapper: seq, tag, event, timestamp, entityId |
|
|
607
|
+
| `EventStore[E]` | `zio.blocks.projection` | Trait: append, readFrom, readAll, subscribe |
|
|
608
|
+
| `SQLiteEventStore[E]` | `zio.blocks.projection` | SQLite-backed EventStore with tag migration |
|
|
609
|
+
| `ProjectionStore[A]` | `zio.blocks.projection` | Trait: insert, upsert, updateFields, delete, truncate, findById |
|
|
610
|
+
| `InMemoryProjectionStore[A]` | `zio.blocks.projection.testing` | In-memory ProjectionStore for tests |
|
|
611
|
+
| `TransactorCache` | `zio.blocks.projection` | LRU cache of SQLite Transactors |
|
|
612
|
+
| `TransactorCacheConfig` | `zio.blocks.projection` | Configuration for TransactorCache |
|
|
613
|
+
| `ProjectionEngine` | `zio.blocks.projection` | Orchestrates catch-up + live processing |
|
|
614
|
+
| `ProjectionEngineConfig` | `zio.blocks.projection` | Engine configuration (batching, rebuild, etc.) |
|
|
615
|
+
| `SchemaHash` | `zio.blocks.projection` | SHA-256 hash of Schema structure for evolution detection |
|
|
616
|
+
| `SchemaEvolution` | `zio.blocks.projection` | Hash check, rebuild, migration shortcut |
|
|
617
|
+
| `SchemaEvolutionConfig` | `zio.blocks.projection` | Evolution config (parallelism, lazy, migration shortcut) |
|
|
618
|
+
| `TagResolver` | `zio.blocks.projection` | Builds TagInfo from Schema + optional Migration |
|
|
619
|
+
| `TagInfo` | `zio.blocks.projection` | Alias map, all tags, old tag detection, value migration |
|
|
620
|
+
| `AggregateProjection` | `zio.blocks.projection` | Helpers for global aggregate specs with counters |
|
|
621
|
+
| `TestEngine` | `zio.blocks.projection.testing` | Simple test engine with auto-created stores and hubs |
|
|
622
|
+
| `TestProjectionEngine` | `zio.blocks.projection.testing` | Synchronous test engine (deprecated, use TestEngine) |
|
|
623
|
+
| `TestContext` | `zio.blocks.projection.testing` | Factory for ProjectionContext in tests |
|
|
624
|
+
|
|
625
|
+
### Projection methods
|
|
626
|
+
|
|
627
|
+
| Method | Description |
|
|
628
|
+
|--------|-------------|
|
|
629
|
+
| `Projection[A](name)` | Create a per-entity spec (requires `Schema[A]` + `EntityPath[A]`) |
|
|
630
|
+
| `Projection.global[A](name)` | Create a global spec (requires `Schema[A]`) |
|
|
631
|
+
| `.from(sourceName)` | Bind to a named event source |
|
|
632
|
+
| `.on[E]` | Register a handler for event type `E` |
|
|
633
|
+
| `.insert((E, ProjectionContext) => A)` | Handler: insert a new entity |
|
|
634
|
+
| `.update((E, ProjectionContext) => A)` | Handler: replace the entity |
|
|
635
|
+
| `.updateWithField(fieldName, (E, ProjectionContext) => Any)` | Handler: update a single field |
|
|
636
|
+
| `.delete` | Handler: delete the entity |
|
|
637
|
+
| `.custom((E, ProjectionContext) => ProjectionAction[A])` | Handler: arbitrary action |
|
|
638
|
+
| `.aggregate(FieldUpdate)` | Handler: atomic counter update |
|
|
639
|
+
| `.routedBy[E](E => String)` | Route events by extracted key |
|
|
640
|
+
| `.routeToSelf` | Route events by `ctx.entityId` |
|
|
641
|
+
| `.routeToAll` | Send all events to same store |
|
|
642
|
+
| `spec.scope` | Derived scope: `PerEntity`, `CrossEntity`, or `Global` |
|
|
643
|
+
|
|
644
|
+
### ProjectionEngine methods
|
|
645
|
+
|
|
646
|
+
| Method | Description |
|
|
647
|
+
|--------|-------------|
|
|
648
|
+
| `ProjectionEngine.make(specs*)` | Create engine with default config (requires `Scope`) |
|
|
649
|
+
| `ProjectionEngine.makeWithConfig(config, specs*)` | Create engine with custom config |
|
|
650
|
+
| `engine.start` | Start catch-up + live processing (requires `Scope`) |
|
|
651
|
+
| `engine.query(spec, entityId)` | Query an entity by ID |
|
|
652
|
+
| `engine.queryByName(specName, entityId)` | Query by spec name string |
|
|
653
|
+
| `engine.registerMigration(spec, migration)` | Register a migration for schema evolution |
|
|
654
|
+
| `engine.transactorCache` | Access the underlying TransactorCache |
|