@zio.dev/zio-blocks 0.0.33 → 0.0.51

package/reference/schema/migration.md

@@ -0,0 +1,297 @@
+---
+id: migration
+title: "Migration"
+---
+
+`Migration[A, B]` is ZIO Blocks Schema's typed API for evolving data from one schema version to another. It wraps a fully serializable [`DynamicMigration`](#dynamicmigration) core with typed source and target schemas, giving you a builder-based workflow for adding fields, dropping fields, renaming fields, changing types, migrating nested values, transforming collections and maps, and composing migrations.
+
+## Overview
+
+The migration system is split into three layers:
+
+- **`Migration[A, B]`** — typed migration between source type `A` and target type `B`
+- **`DynamicMigration`** — pure runtime representation built from serializable actions
+- **`MigrationBuilder[A, B, Changeset]`** — macro-validated builder that accumulates actions and checks that the target shape is fully handled before `build`
+
+```
+Migration[A, B]
+  ├── sourceSchema: Schema[A]
+  ├── targetSchema: Schema[B]
+  └── dynamicMigration: DynamicMigration
+         └── actions: Chunk[MigrationAction]
+                ├── AddField / DropField / RenameField
+                ├── TransformField / ChangeFieldType
+                ├── MandateField / OptionalizeField
+                ├── RenameCase / TransformCase
+                ├── MigrateField
+                ├── TransformElements / TransformKeys / TransformValues
+                └── Irreversible
+```
+
+`Migration` is the user-facing entry point. `DynamicMigration` is the transport-friendly representation you can inspect, serialize, or apply dynamically.
+
+## Creating a Migration
+
+The normal entry point is `Migration.newBuilder[A, B]`:
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+
+case class PersonV1(name: String)
+case class PersonV2(name: String, age: Int)
+
+object PersonV1 {
+  implicit val schema: Schema[PersonV1] = Schema.derived
+}
+
+object PersonV2 {
+  implicit val schema: Schema[PersonV2] = Schema.derived
+}
+
+val migration: Migration[PersonV1, PersonV2] =
+  Migration
+    .newBuilder[PersonV1, PersonV2]
+    .addField(_.age, SchemaExpr.literal(0))
+    .build
+```
+
+Applying the migration converts the source value to [`DynamicValue`](./dynamic-value.md), applies the dynamic actions, then converts the result back into the target type:
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+
+case class PersonV1(name: String)
+case class PersonV2(name: String, age: Int)
+
+object PersonV1 {
+  implicit val schema: Schema[PersonV1] = Schema.derived
+}
+
+object PersonV2 {
+  implicit val schema: Schema[PersonV2] = Schema.derived
+}
+
+val migration = Migration
+  .newBuilder[PersonV1, PersonV2]
+  .addField(_.age, SchemaExpr.literal(0))
+  .build
+
+val result = migration(PersonV1("Alice"))
+// Right(PersonV2("Alice", 0))
+```
+
+## Core Operations
+
+### `Migration#apply`
+
+Transforms a typed value:
+
+```scala
+def apply(value: A): Either[SchemaError, B]
+```
+
+### `Migration#reverse`
+
+Returns the **structural reverse** of the migration:
+
+```scala
+def reverse: Migration[B, A]
+```
+
+This is best-effort at runtime. Actions that lose information (for example `TransformField`, `ChangeFieldType`, `OptionalizeField`, or collection-wide transforms) reverse to `Irreversible`, which causes reverse execution to fail with a descriptive error instead of silently fabricating data.
+
+### Composition
+
+Migrations compose with `++` or `andThen`:
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+
+case class PersonV1(name: String)
+case class PersonV2(name: String, age: Int)
+case class PersonV3(name: String, age: Int, city: String)
+
+object PersonV1 { implicit val schema: Schema[PersonV1] = Schema.derived }
+object PersonV2 { implicit val schema: Schema[PersonV2] = Schema.derived }
+object PersonV3 { implicit val schema: Schema[PersonV3] = Schema.derived }
+
+val migration = Migration
+  .newBuilder[PersonV1, PersonV2]
+  .addField(_.age, SchemaExpr.literal(0))
+  .build
+
+val addCity = Migration
+  .newBuilder[PersonV2, PersonV3]
+  .addField(_.city, SchemaExpr.literal(""))
+  .build
+
+val combined: Migration[PersonV1, PersonV3] = migration ++ addCity
+```
+
+## Builder Operations
+
+`MigrationBuilder` uses selector expressions such as `_.field`, `_.nested.field`, `_.items.each`, and `_.data.eachValue`. The builder's third type parameter, `Changeset`, tracks which operations have already been applied so `build` can validate completeness.
+
+### Record operations
+
+- `addField(_.target, default)`
+- `dropField(_.source, defaultForReverse)`
+- `renameField(_.from, _.to)`
+- `transformField(_.from, _.to, expr)`
+- `mandateField(_.optionalSource, _.target, default)`
+- `optionalizeField(_.source, _.optionalTarget)`
+- `changeFieldType(_.source, expr)`
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+
+case class UserV1(firstName: String, nickname: Option[String], age: Int)
+case class UserV2(fullName: String, nickname: String, age: String)
+
+object UserV1 {
+  implicit val schema: Schema[UserV1] = Schema.derived
+}
+
+object UserV2 {
+  implicit val schema: Schema[UserV2] = Schema.derived
+}
+
+val userMigration = Migration
+  .newBuilder[UserV1, UserV2]
+  .renameField(_.firstName, _.fullName)
+  .mandateField(_.nickname, _.nickname, SchemaExpr.literal("anonymous"))
+  .changeFieldType(_.age, SchemaExpr.literal("30"))
+  .build
+```
+
+`transformField` and `changeFieldType` evaluate their `SchemaExpr` against the currently focused field value, not the entire root record. In practice, that means the expression should be written in terms of the value being replaced.
+
+### Nested migration with `migrateField`
+
+When a nested value has its own migration, use `migrateField`:
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.migration._
+
+case class AddressV1(street: String, city: String)
+case class AddressV2(street: String, city: String, zip: String)
+case class PersonWithAddressV1(name: String, address: AddressV1)
+case class PersonWithAddressV2(name: String, address: AddressV2)
+
+object AddressV1 { implicit val schema: Schema[AddressV1] = Schema.derived }
+object AddressV2 { implicit val schema: Schema[AddressV2] = Schema.derived }
+object PersonWithAddressV1 { implicit val schema: Schema[PersonWithAddressV1] = Schema.derived }
+object PersonWithAddressV2 { implicit val schema: Schema[PersonWithAddressV2] = Schema.derived }
+
+val addressMigration = Migration
+  .newBuilder[AddressV1, AddressV2]
+  .addField(_.zip, SchemaExpr.literal("00000"))
+  .build
+
+val personMigration = Migration
+  .newBuilder[PersonWithAddressV1, PersonWithAddressV2]
+  .migrateField(_.address, addressMigration)
+  .build
+```
+
+There is no `transformNested` builder method in the current API. Nested structural changes are expressed by building a migration for the nested type and applying it with `migrateField`.
+
+### Enum and collection operations
+
+- `renameCase(from, to)`
+- `transformCase(caseName)(...)`
+- `transformElements(_.items, expr)`
+- `transformKeys(_.data, expr)`
+- `transformValues(_.data, expr)`
+
+Collection and map transforms are map-like operations. The `SchemaExpr` is evaluated once per matched value:
+
+- `transformElements` evaluates against each collection element
+- `transformKeys` evaluates against each map key
+- `transformValues` evaluates against each map value
+
+For example, `transformElements(_.scores, expr)` runs `expr` separately for every element in `scores`, and `transformValues(_.metadata, expr)` runs it separately for every map value.
+
+## `build`
+
+```scala
+def build(using ev: MigrationComplete[A, B, Changeset]): Migration[A, B]
+```
+
+`build` runs macro validation. It checks that every field needed to transform `A` into `B` is either:
+
+- auto-mapped by identical source/target structure, or
+- explicitly handled by one of the builder operations
+
+If the migration is incomplete, the code fails to compile.
+
+This validated `build` step is the public completion path for `MigrationBuilder`. If you need unvalidated or dynamically assembled migrations, work at the `DynamicMigration` layer and wrap it with `Migration.fromDynamic`.
+
+## `DynamicMigration`
+
+`DynamicMigration` is the untyped runtime representation:
+
+```scala
+final case class DynamicMigration(actions: Chunk[MigrationAction]) {
+  def apply(value: DynamicValue): Either[SchemaError, DynamicValue]
+  def ++(that: DynamicMigration): DynamicMigration
+  def andThen(that: DynamicMigration): DynamicMigration
+  def reverse: DynamicMigration
+  def isEmpty: Boolean
+  def size: Int
+}
+```
+
+Use it when you need to inspect, serialize, or apply migrations without keeping the original Scala types around.
+
+## `MigrationAction`
+
+`MigrationAction` is the serializable ADT used by `DynamicMigration`.
+
+| Action | Purpose | Reverse behavior |
+|---|---|---|
+| `AddField` | add a field with a default expression | `DropField` |
+| `DropField` | remove a field | `AddField` using stored reverse default |
+| `RenameField` | rename a record field | structural inverse rename |
+| `TransformField` | replace a field by evaluating an expression against the current field value | `Irreversible` |
+| `MandateField` | convert `Option[A]` to `A` with default for `None` | `OptionalizeField` |
+| `OptionalizeField` | wrap a value in `Some` | `Irreversible` |
+| `ChangeFieldType` | replace a field with a converted value computed from the current field value | `Irreversible` |
+| `RenameCase` | rename enum case | inverse rename |
+| `TransformCase` | run actions inside a specific case | reverse inner actions in reverse order |
+| `MigrateField` | apply a nested `DynamicMigration` to a field | reverse nested migration |
+| `TransformElements` | replace each collection element by evaluating the expression on that element | `Irreversible` |
+| `TransformKeys` | replace each map key by evaluating the expression on that key | `Irreversible` |
+| `TransformValues` | replace each map value by evaluating the expression on that value | `Irreversible` |
+| `Irreversible` | explicit non-invertible sentinel | itself |
+
+## Errors
+
+Migration execution returns [`SchemaError`](./schema-error.md), with migration-specific kinds including:
+
+- path not found
+- type mismatch
+- missing default
+- transform failure
+- field/case not found
+- invalid value
+- mandate failure
+
+Errors carry a [`DynamicOptic`](./dynamic-optic.md) path so failures can be traced to the precise field or nested location that failed.
+
+## Relationship to Other Schema-Evolution APIs
+
+`Into` and `As` derive structural conversions automatically. `Migration` is lower-level and more explicit:
+
+- use [`Into`](./schema-evolution/into.md) for one-way derived schema evolution
+- use [`As`](./schema-evolution/as.md) for bidirectional derived evolution
+- use `Migration` when you need explicit, inspectable, serializable transformation steps
+
+:::tip
+If you want a practical migration walkthrough rather than a reference, start with [`MigrationSpec`](https://github.com/zio/zio-blocks/blob/main/schema/shared/src/test/scala/zio/blocks/schema/migration/MigrationSpec.scala) and [`DynamicMigrationSpec`](https://github.com/zio/zio-blocks/blob/main/schema/shared/src/test/scala/zio/blocks/schema/migration/DynamicMigrationSpec.scala) in the repository until a dedicated migration guide lands.
+:::

package/reference/{modifier.md → schema/modifier.md}

@@ -11,7 +11,7 @@ Modifiers are designed to be **pure data** values that can be serialized, making
 sealed trait Modifier extends StaticAnnotation
 object Modifier {
   sealed trait Term    extends Modifier
-  // ... term modifiers (transient, rename, alias, config) ...
+  // ... term modifiers (transient, encodeTransient, rename, alias, id, config) ...
   sealed trait Reflect extends Modifier
   // ... reflect modifiers (config) ...
 }
@@ -119,12 +119,14 @@ Modifiers are organized into two main categories:
 ```
 Modifier
  ├── Modifier.Term     (annotates record fields and variant cases)
- │    ├── transient()      : exclude from serialization
- │    ├── rename(name)     : change serialized name
- │    ├── alias(name)      : add alternative name
- │    └── config(key, val) : attach key-value metadata
+ │    ├── transient()        : exclude from serialization
+ │    ├── encodeTransient()  : exclude from encoding only
+ │    ├── rename(name)       : change serialized name
+ │    ├── alias(name)        : add alternative name
+ │    ├── id()               : mark a primary-key field
+ │    └── config(key, val)   : attach key-value metadata
  └── Modifier.Reflect  (annotates reflect values / types)
-      └── config(key, val) : attach key-value metadata
+      └── config(key, val)   : attach key-value metadata
 ```
 
 As you can see, `config` is the only modifier that extends both `Term` and `Reflect`, allowing it to be used on both fields and types.
@@ -197,6 +199,23 @@ With this configuration:
 
 This pattern is particularly useful when migrating data formats without breaking compatibility with existing data.
 
+### id
+
+The `id` modifier marks a record field as its primary key. It is interpreted by
+`zio-blocks-sql` when deriving a `Repo[E, ID]`, and takes precedence over
+name- and type-based ID discovery.
+
+```scala
+import zio.blocks.schema._
+
+case class User(
+  email: String,
+  @Modifier.id() externalKey: Long
+)
+```
+
+Mark exactly one field with `@Modifier.id()` for a derived repository.
+
 ### config
 
 The `config` modifier attaches arbitrary key-value metadata to a term (record fields or variant cases) or a type itself. The convention for keys is `<format>.<property>`, allowing format-specific configuration.
@@ -218,7 +237,7 @@ The `config` modifier extends both `Term` and `Reflect`, making it usable on bot
 
 ## Reflect Modifiers
 
-Reflect modifiers annotate reflect values (types themselves). Currently, only `config` is a reflect modifier.
+Reflect modifiers annotate reflect values (types themselves). Alongside `config`, JSON derivation also understands dedicated reflect modifiers for discriminator selection, field naming, case naming, and rejecting extra fields.
 
 ### config
 
@@ -236,6 +255,33 @@ object Person {
 }
 ```
 
+### JSON reflect modifiers
+
+These reflect modifiers are consumed by `JsonCodecDeriver` and let you keep JSON-specific settings next to the annotated type instead of passing them programmatically at every call site.
+
+```scala
+import zio.blocks.schema._
+
+@Modifier.discriminator("type")
+@Modifier.caseNaming("snake_case")
+sealed trait Event
+
+@Modifier.noExtraFields()
+@Modifier.fieldNaming("snake_case")
+case class UserProfile(
+  @Modifier.rename("userId") id: String,
+  @Modifier.encodeTransient() checksum: Int = 0,
+  displayName: String
+)
+```
+
+- `Modifier.discriminator(name)` overrides the JSON discriminator field for sealed hierarchies.
+- `Modifier.noExtraFields()` rejects unknown object fields during decoding.
+- `Modifier.fieldNaming(strategy)` applies a naming strategy to direct record fields.
+- `Modifier.caseNaming(strategy)` applies a naming strategy to variant case names.
+
+For field-level JSON behavior, `Modifier.encodeTransient()` skips a field during encoding but still accepts it during decoding; like `transient`, it requires a default value when the field is neither optional nor a collection.
+
 Or add multiple modifiers at once:
 
 ```scala
@@ -305,8 +351,13 @@ import zio.blocks.schema._
 
 // Schema instances for individual modifiers
 Schema[Modifier.transient]
+Schema[Modifier.encodeTransient]
 Schema[Modifier.rename]
 Schema[Modifier.alias]
+Schema[Modifier.discriminator]
+Schema[Modifier.noExtraFields]
+Schema[Modifier.fieldNaming]
+Schema[Modifier.caseNaming]
 Schema[Modifier.config]
 
 // Schema instances for modifier traits

package/reference/{optics.md → schema/optics.md}

@@ -6,7 +6,7 @@ title: "Optics"
 Optics are a fundamental feature of ZIO Blocks that enable type-safe, composable access and modification of nested data structures. What sets ZIO Blocks apart is its implementation of **reflective optics** — a novel construct that combines the operational capabilities of traditional optics with embedded structural metadata, enabling both data manipulation AND introspection.
 
 :::tip
-For a practical walkthrough of building query DSLs with optics, see the [Writing a Query DSL with Reified Optics](../guides/query-dsl-reified-optics.md) guide.
+For a practical walkthrough of building query DSLs with optics, see the [Writing a Query DSL with Reified Optics](../../guides/query-dsl-reified-optics.md) guide.
 :::
 
 ## What Are Optics?
@@ -148,7 +148,7 @@ val updatedAddress: Address = Address.street.replace(address, "456 Elm St")
 
 While manual lens construction gives you fine-grained control, ZIO Blocks provides macro-based derivation as the **preferred approach** for creating lenses.
 
-The `optic` macro inside the `CompanionOptics` trait creates a lens using intuitive selector syntax that mirrors standard Scala field access:
+The `optic` macro inside the `CompanionOptics` trait creates a lens using intuitive selector syntax that mirrors standard Scala field access. In the rest of the schema docs you may also see the symbolic `$(_.field)` form; it is the same optic-construction API presented with symbolic syntax instead of the named `optic(_.field)` helper.
 
 ```scala
 import zio.blocks.schema.optic

package/reference/{patch.md → schema/patch.md}

@@ -16,7 +16,7 @@ A `Patch[S]` represents a sequence of operations that transform a value of type
 - **Schema Evolution** — Patches work with the schema system, adapting as data structures evolve
 
 :::note
-For **untyped JSON patching** without a schema, use [`JsonPatch`](./json-patch.md) instead. `JsonPatch` is optimized for diff-and-apply workflows on raw JSON values and provides compact delta representations without requiring typed optics.
+For **untyped JSON patching** without a schema, use [`JsonPatch`](./built-in-codecs/json/json-patch.md) instead. `JsonPatch` is optimized for diff-and-apply workflows on raw JSON values and provides compact delta representations without requiring typed optics.
 :::
 
 ```scala