@zio.dev/zio-blocks 0.0.33 → 0.0.55

package/reference/{dynamic-optic.md → schema/dynamic-optic.md}

@@ -26,7 +26,7 @@ val result = data.get(path).one
 ```
 
 :::tip
-For a practical example of extracting column names from `DynamicOptic` paths to generate SQL, see the [Query DSL Part 2: SQL Generation](../guides/query-dsl-sql.md) guide.
+For a practical example of extracting column names from `DynamicOptic` paths to generate SQL, see the [Query DSL Part 2: SQL Generation](../../guides/query-dsl-sql.md) guide.
 :::
 
 ## Motivation
@@ -57,7 +57,7 @@ The relationship between typed and dynamic optics:
 └──────────────────┘             └──────────────────┘
 ```
 
-The `Optic[S, A]` and `DynamicOptic` types serve complementary roles in ZIO Blocks' optics system. `Optic[S, A]` provides compile-time type safety through macros or manual construction, operates on typed Scala values, and can be converted to a `DynamicOptic` via the `optic.toDynamic` method. In contrast, `DynamicOptic` performs runtime type checking and is constructed through a builder API or [path interpolator](../path-interpolator.md), operating directly on [DynamicValue](./dynamic-value.md), [Schema](./schema.md), and [Reflect](./reflect.md) representations. 
+The `Optic[S, A]` and `DynamicOptic` types serve complementary roles in ZIO Blocks' optics system. `Optic[S, A]` provides compile-time type safety through macros or manual construction, operates on typed Scala values, and can be converted to a `DynamicOptic` via the `optic.toDynamic` method. In contrast, `DynamicOptic` performs runtime type checking and is constructed through a builder API or [path interpolator](./path-interpolator.md), operating directly on [DynamicValue](./dynamic-value.md), [Schema](./schema.md), and [Reflect](./reflect.md) representations. 
 
 ## Design & Structure
 
@@ -157,7 +157,7 @@ Note: `.atKey` and `.atKeys` require an implicit `Schema[K]` to convert the type
 
 ### Path Interpolator
 
-The [`p"..."` path interpolator](../path-interpolator.md) provides a concise compile-time syntax for building
+The [`p"..."` path interpolator](./path-interpolator.md) provides a concise compile-time syntax for building
 `DynamicOptic` values:
 
 ```scala
@@ -181,7 +181,7 @@ val success = p".result<Success>.value"
 val complex = p""".groups[*].members[0].contacts{"email"}"""
 ```
 
-See [Path Interpolator](../path-interpolator.md) for the full syntax reference.
+See [Path Interpolator](./path-interpolator.md) for the full syntax reference.
 
 ### From Typed Optics
 
@@ -360,7 +360,7 @@ expected Email, but got Push
 
 `DynamicOptic` provides two string formats for different contexts:
 
-- **`toString`** — Compact path syntax matching the [`p"..."` interpolator](../path-interpolator.md) format
+- **`toString`** — Compact path syntax matching the [`p"..."` interpolator](./path-interpolator.md) format
 - **`toScalaString`** — Scala method call syntax used in error messages
 
 | Node                      | `toString`   | `toScalaString`     |
@@ -396,6 +396,197 @@ val serialized: DynamicValue = opticSchema.toDynamicValue(path)
 ```
 
 
+## Search Optics
+
+A **Search optic** recursively traverses a data structure to find **all occurrences** matching a type or schema. It produces a `Traversal[S, A]` that collects matches in depth-first, left-to-right order. This section covers the two DSLs for building a search path; for the `SearchTraversal` engine that executes one — `fold`/`modify`/`check`, composing a search with other optics, and the `SchemaMatch` predicate behind `#Pattern` matching — see [Schema Search and Update](./schema-search.md).
+
+### Motivation
+
+Search optics address scenarios where you need to:
+
+- **Find all values of a specific type** across deeply nested structures (e.g., all `String` fields in a nested record)
+- **Extract all data matching a structural pattern** (e.g., all records with `{ name: string, age: int }` schema)
+- **Transform all matching occurrences** in untyped or partially-typed data
+- **Query data without knowing the exact path** — the search discovers all paths automatically
+
+### Typed API: `.searchFor[T]`
+
+Use the `.searchFor[T]` extension method on a `CompanionOptics` to search for all values of type `T`:
+
+```scala
+import zio.blocks.schema._
+
+case class Person(name: String, age: Int)
+object Person {
+  implicit val schema: Schema[Person] = Schema.derived
+}
+
+case class Address(city: String)
+object Address {
+  implicit val schema: Schema[Address] = Schema.derived
+}
+
+case class Company(name: String, employees: List[Person], hq: Address)
+object Company extends CompanionOptics[Company] {
+  implicit val schema: Schema[Company] = Schema.derived
+}
+
+// Find all Person instances within a Company
+val personSearch: Traversal[Company, Person] = Company.optic(_.searchFor[Person])
+
+val company = Company(
+  "Acme",
+  List(Person("Alice", 30), Person("Bob", 25)),
+  Address("NYC")
+)
+
+// Modify all persons (increment their age)
+val updated: Company = personSearch.modify(company, p => p.copy(age = p.age + 1))
+// Company("Acme", List(Person("Alice", 31), Person("Bob", 26)), Address("NYC"))
+```
+
+### Dynamic API: Path Strings with `#` Prefix
+
+Use the `#` prefix in path strings to specify type or schema patterns:
+
+```scala
+import zio.blocks.schema._
+
+val data = DynamicValue.Record(
+  "company" -> DynamicValue.Record(
+    "name" -> DynamicValue.string("Acme"),
+    "employees" -> DynamicValue.Sequence(
+      DynamicValue.Record("name" -> DynamicValue.string("Alice")),
+      DynamicValue.Record("name" -> DynamicValue.string("Bob"))
+    )
+  )
+)
+
+// Find all strings in the data
+val allStrings = data.get(p"#string").toChunk
+// Chunk(
+//   DynamicValue.string("Acme"),
+//   DynamicValue.string("Alice"),
+//   DynamicValue.string("Bob")
+// )
+
+// Find all records matching a schema
+val recordsWithName = data.get(p"#record { name: string }").toChunk
+
+// Modify all matching values
+val modified = data.modify(p"#string")(dv =>
+  dv match {
+    case DynamicValue.Primitive(PrimitiveValue.String(s)) =>
+      DynamicValue.string(s.toUpperCase)
+    case other => other
+  }
+)
+```
+
+### The `SchemaRepr` Pattern Type
+
+Every `#Pattern` string above parses into a `SchemaRepr` — a plain, immutable ADT you can also build by hand, independent of the interpolator. `DynamicOptic#searchSchema(schemaRepr)` takes one directly, and `Node.SchemaSearch(schemaRepr)` is the node it produces, exactly the node `#Pattern` syntax builds under the hood:
+
+```scala
+import zio.blocks.schema._
+
+val pattern: SchemaRepr = SchemaRepr.Record(
+  IndexedSeq("name" -> SchemaRepr.Primitive("string"), "age" -> SchemaRepr.Primitive("int"))
+)
+```
+
+`SchemaRepr#toString` renders the same syntax `#Pattern` strings use, so a pattern built by hand and one parsed from a literal print identically:
+
+```scala
+pattern.toString
+// res13: String = "record { name: string, age: int }"
+
+val path: DynamicOptic = DynamicOptic.root.searchSchema(pattern)
+// path: DynamicOptic = DynamicOptic(
+//   IndexedSeq(
+//     SchemaSearch(
+//       Record(Vector(("name", Primitive("string")), ("age", Primitive("int"))))
+//     )
+//   )
+// )
+path == p"#record { name: string, age: int }"
+// res14: Boolean = true
+```
+
+The eight cases cover every shape the grammar recognizes:
+
+| Case | Renders as | Matches |
+|------|------------|---------|
+| `Nominal(name)`         | the bare name, e.g. `Person`         | a value of that Scala type — only against a `Reflect` tree, see below |
+| `Primitive(name)`       | the bare name, e.g. `string`         | a primitive of that kind (case-insensitive) |
+| `Record(fields)`        | `record { name: string, ... }`       | a record with at least those fields, by subset match |
+| `Variant(cases)`        | `variant { Case: type, ... }`        | a variant whose active case matches one of those entries |
+| `Sequence(element)`     | `list(...)`                          | a sequence whose elements all match — parses from `list(...)`, `set(...)`, or `vector(...)`, all three producing the same case |
+| `Map(key, value)`       | `map(key, value)`                    | a map whose entries all match both patterns |
+| `Optional(inner)`       | `option(...)`                        | `Null`, or a value matching the inner pattern |
+| `Wildcard`              | `_`                                   | anything |
+
+Parsing a `#Pattern` string is itself layered: the path interpolator recognizes the `#` prefix and hands the substring after it to a small recursive-descent parser, which reports a specific error (unexpected character, an empty `record {}`/`variant {}`, an unterminated expression) at the exact offset where parsing failed — that's what turns a typo like `#record { name: strnig }` into a compile-time interpolator error rather than a pattern that silently matches nothing.
+
+### Supported Patterns
+
+Search optics support both **type-based** and **schema-based** patterns:
+
+| Pattern                        | Matches                                | Example                    |
+|--------------------------------|----------------------------------------|---------------------------|
+| `#Nominal` (type name)         | Values of that type name               | `#Person`                  |
+| `#int`, `#string`, `#boolean`  | Specific primitive types (case-insensitive) | `#int`, `#string`, `#boolean` |
+| `#record { ... }`              | Records with matching fields           | `#record { name: string }` |
+| `#variant { ... }`             | Variant cases with matching content    | `#variant { Error: string }` |
+| `#list(...)`, `#set(...)`, `#vector(...)` | Sequences with a matching element type, regardless of which of the three keywords appears | `#list(string)` |
+| `#map(...)`                    | Maps with key and value types          | `#map(string, int)`        |
+| `#option(...)`                 | Optional values with inner type       | `#option(int)`             |
+
+### Traversal Order
+
+Results are collected in **depth-first, left-to-right order**:
+
+```scala
+import zio.blocks.schema._
+
+val nested = DynamicValue.Record(
+  "a" -> DynamicValue.int(1),
+  "b" -> DynamicValue.Record(
+    "c" -> DynamicValue.int(2),
+    "d" -> DynamicValue.int(3)
+  ),
+  "e" -> DynamicValue.int(4)
+)
+
+// Depth-first, left-to-right: 1, 2, 3, 4
+val allInts = nested.get(p"#int").toChunk
+// Chunk(1, 2, 3, 4)
+```
+
+### Known Limitation: Nominal Matching in Untyped Contexts
+
+`Nominal` pattern matching (e.g., `#Person`) returns `false` against a `DynamicValue` or `Json` value because these untyped structures carry no type identity. The one exception is searching a `Schema`/`Reflect` tree itself (as `DynamicSchema#get` does) rather than a value — there, `Nominal` matches by comparing against the tree's own `TypeId`, since a `Reflect` node still carries the type identity a plain `DynamicValue`/`Json` value never does. For searching values, use one of these approaches instead:
+
+- **Use the typed API**: `optic(_.searchFor[Person])`
+- **Or use structural patterns**: `p"#record { name: string, age: int }"`
+
+Here's how the two approaches differ in practice:
+
+```scala
+import zio.blocks.schema._
+
+val dynamicData: DynamicValue = ???
+
+// This won't match — no type identity available
+val nominalMatch = dynamicData.get(p"#Person").toChunk
+// Chunk()
+
+// This works — structural matching
+val structuralMatch = dynamicData.get(p"#record { name: string, age: int }").toChunk
+// Chunk(...matching records...)
+```
+
 ## See Also
 
 - [DynamicSchema](./dynamic-schema.md) — use `DynamicSchema#get` with a `DynamicOptic` to navigate schema trees and inspect nested type structures.
+- [Optics](./optics.md) — reference page for typed `Optic[S, A]` and reflective optics concepts.

package/reference/{dynamic-schema.md → schema/dynamic-schema.md}

@@ -41,7 +41,7 @@ DynamicSchema         ←── reconstructed from storage
 Schema[OrderPlaced]   ←── operational again, can encode and decode
 ```
 
-This pattern enables schema registries: the Checkout Service registers its event schema on startup; the Fulfillment Service fetches it and rebinds it against its own type definitions, guaranteeing it uses the exact schema that was in effect when the event was encoded. See [BindingResolver](./binding-resolver.md) for the complete rebinding API.
+This pattern enables schema registries: the Checkout Service registers its event schema on startup; the Fulfillment Service fetches it and rebinds it against its own type definitions, guaranteeing it uses the exact schema that was in effect when the event was encoded. See [BindingResolver](binding-resolver.md) for the complete rebinding API.
 
 ## Creating a DynamicSchema
 
@@ -83,7 +83,7 @@ dynamic
 //         value = Primitive(
 //           primitiveType = String(None),
 //           typeId = String,
-//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5ca5d848,
+//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5fb4c2ef,
 //           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
 //           modifiers = List(),
 //           storedDefaultValue = None,
@@ -97,7 +97,7 @@ dynamic
 //         value = Primitive(
 //           primitiveType = Int(None),
 //           typeId = Int,
-//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5ca5d848,
+//           primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5fb4c2ef,
 //           doc = Doc(blocks = IndexedSeq(), metadata = Map()),
 //           modifiers = List(),
 //           storedDefaultValue = None,
@@ -115,7 +115,7 @@ dynamic
 //               value = Primitive(
 //                 primitiveType = String(None),
 //                 typeId = String,
-//                 primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5ca5d848,
+//                 primitiveBinding = zio.blocks.schema.binding.NoBinding$$anon$1@5fb4c2ef,
 //                 doc = Doc(blocks = IndexedSeq(), metadata = Map()),
 //                 modifiers = List(),
 //                 storedDefaultValue = None,
@@ -232,7 +232,7 @@ dynSchema.check(valid)
 dynSchema.check(missing)
 // res8: Option[SchemaError] = Some(
 //   SchemaError(
-//     List(MissingField(source = DynamicOptic(IndexedSeq()), fieldName = "y"))
+//     List(MissingField(source = DynamicOptic(ArraySeq()), fieldName = "y"))
 //   )
 // )
 dynSchema.check(wrongType)
@@ -240,7 +240,7 @@ dynSchema.check(wrongType)
 //   SchemaError(
 //     List(
 //       ExpectationMismatch(
-//         source = DynamicOptic(IndexedSeq(Field("y"))),
+//         source = DynamicOptic(ArraySeq(Field("y"))),
 //         expectation = "Expected Int, got String"
 //       )
 //     )
@@ -339,7 +339,7 @@ rebound.fromDynamicValue(encoded)
 If `DynamicSchema#rebind` cannot find a binding for any type in the unbound schema tree, it throws a `RebindException` at runtime. Ensure the resolver covers every concrete type — records, variants, wrappers, primitives, and standard collections — that appears in the schema. `BindingResolver.defaults` covers all standard primitives, `java.time` types, `List`, `Map`, `Option`, and other standard collections.
 :::
 
-See [BindingResolver](./binding-resolver.md) for the full resolver API including `BindingResolver.reflection` for automatic binding via reflection.
+See [BindingResolver](binding-resolver.md) for the full resolver API including `BindingResolver.reflection` for automatic binding via reflection.
 
 ## Structural Navigation
 
@@ -546,7 +546,7 @@ gatewaySchema.fromDynamicValue(incoming)
 
 ### With `BindingResolver`
 
-The primary consumer of `DynamicSchema` in a type-safe context is `DynamicSchema#rebind`, which requires a `BindingResolver` to reattach runtime bindings. See [BindingResolver](./binding-resolver.md) for the complete rebinding API, including `BindingResolver.reflection` for automatic binding discovery.
+The primary consumer of `DynamicSchema` in a type-safe context is `DynamicSchema#rebind`, which requires a `BindingResolver` to reattach runtime bindings. See [BindingResolver](binding-resolver.md) for the complete rebinding API, including `BindingResolver.reflection` for automatic binding discovery.
 
 ### With `DynamicOptic`
 

package/reference/schema/format.md

@@ -0,0 +1,92 @@
+---
+id: format
+title: "Format Type"
+sidebar_label: "Format"
+---
+
+A `Format` is an abstraction that bundles together everything needed to serialize and deserialize data in a specific format (JSON, Avro, MessagePack, etc.). It unifies metadata related to serialization formats, such as MIME type and codec deriver, in a single place.
+
+## Overview
+
+The `Format` trait defines the structure for any serialization format. Each format specifies the types for decoding input, encoding output, the codec typeclass, MIME type, and the deriver used to generate codecs from schemas:
+
+```scala
+import zio.blocks.schema.codec.Codec
+import zio.blocks.schema.derive.Deriver
+
+trait Format {
+  type DecodeInput
+  type EncodeOutput
+  type TypeClass[A] <: Codec[DecodeInput, EncodeOutput, A]
+  def mimeType: String
+  def deriver: Deriver[TypeClass]
+}
+```
+
+This design allows for a consistent API across different formats when deriving codecs from schemas. Having MIME type information helps with runtime content negotiation and format routing, for example in HTTP servers or message queues.
+
+## Using a Format
+
+You can easily call [`Schema[A].derive(format)`](./type-class-derivation.md#using-the-deriver-to-derive-type-class-instances) for any format that implements the `Format` trait to obtain a codec that can encode and decode values of type `A` according to that format's rules:
+
+```scala
+import zio.blocks.schema._
+import zio.blocks.schema.toon._
+
+case class Person(name: String, age: Int)
+
+object Person {
+  implicit val schema: Schema[Person] = Schema.derived
+}
+
+val codec = Schema[Person].derive(ToonFormat)
+val bytes: Array[Byte] = codec.encode(Person("Alice", 30))
+val decoded: Either[SchemaError, Person] = codec.decode(bytes)
+```
+
+## Format Categories
+
+Formats are categorized into `BinaryFormat` and `TextFormat`, which specify the types of input and output for encoding and decoding:
+
+```scala
+sealed trait Format
+abstract class BinaryFormat[A] extends Format { }
+abstract class TextFormat[A] extends Format { }
+```
+
+**BinaryFormat** — Encodes and decodes data as bytes. Used for compact, efficient serialization (e.g., Avro, Thrift, MessagePack).
+
+**TextFormat** — Encodes and decodes data as text strings. Used for human-readable formats (e.g., JSON, YAML, TOON).
+
+### Example: JsonFormat
+
+The `JsonFormat` is a `BinaryFormat` that represents JSON serialization, where both the input for decoding and output for encoding are `ByteBuffer`, the MIME type is `application/json`, and the deriver for generating codecs from schemas is `JsonCodecDeriver`:
+
+```scala
+import zio.blocks.schema.json._
+
+object MyJsonFormat extends zio.blocks.schema.codec.BinaryFormat("application/json", JsonCodecDeriver)
+```
+
+## Defining a Custom Format
+
+To add a new serialization format, define a `BinaryFormat` (or `TextFormat`) singleton with a custom `Deriver`:
+
+```scala
+import zio.blocks.schema.codec.BinaryCodec
+
+// 1. Define your codec base class
+abstract class MyCodec[A] extends BinaryCodec[A]
+
+// 2. Implement a Deriver[MyCodec] (see Type-class Derivation docs)
+// val myDeriver: Deriver[MyCodec] = ...
+
+// 3. Create the format singleton
+// object MyFormat extends BinaryFormat[MyCodec]("application/x-myformat", myDeriver)
+```
+
+For details on implementing a `Deriver`, see [Type-class Derivation](./type-class-derivation.md).
+
+## Available Formats
+
+ZIO Blocks provides multiple built-in formats. See the [Built-in Formats and Codecs](./built-in-codecs/index.md) section for a complete list of supported serialization formats and their respective codec modules.

package/reference/schema/index.md

@@ -0,0 +1,52 @@
+---
+id: index
+title: "ZIO Blocks Schema"
+sidebar_label: "Schema"
+---
+
+## Introduction
+
+ZIO Blocks Schema is the core type system and serialization framework that provides reified structural metadata for Scala data types. It enables type-safe schema definition, validation, optics-based data access, multi-format serialization, and runtime introspection — all derived from a single `Schema` definition.
+
+**Core Type System:**
+- [`Schema`](./schema.md) — Primary data type containing reified structure of a Scala data type
+- [`Reflect`](./reflect.md) — Foundational data structure containing reified structural information
+- [`Binding`](./binding.md) — Operational machinery for constructing and deconstructing values
+- [`ReflectTransformer`](./reflect-transformer.md) — Generic mechanism for rewriting a Reflect tree's binding type, powering schema (de)serialization and derivation
+- [`Registers`](./registers.md) — Register-based design for zero-allocation construction and deconstruction
+- [`Structural Types`](./structural-types.md) — Duck typing with ZIO Blocks schemas
+
+**Dynamic & Runtime Data:**
+- [`DynamicValue`](./dynamic-value.md) — Schema-less, dynamically-typed representation of any structured value
+- [`DynamicSchema`](./dynamic-schema.md) — Type-erased schema container for serialization and transport
+- [`Lazy`](./lazy.md) — Deferred computation with monadic abstraction, memoization, and stack-safe evaluation
+
+**Navigation & Transformation:**
+- [`Optics`](./optics.md) — Reflective optics for type-safe, composable access to nested data structures
+- [`DynamicOptic`](./dynamic-optic.md) — Runtime path through nested data structures
+- [`Path Interpolator`](./path-interpolator.md) — Compile-time string interpolator `p"..."` for constructing `DynamicOptic` instances
+- [`Schema Search and Update`](./schema-search.md) — Executing type/schema-pattern searches with `SearchTraversal`, and rewriting schema metadata with `Reflect.Updater`/`Term.Updater`
+- [`SchemaExpr`](./schema-expr.md) — Schema-aware expressions for evaluation and query language translation
+- [`Patch`](./patch.md) — Type-safe, serializable transformations of data structures
+- [`Modifier`](./modifier.md) — Mechanism to attach metadata and configuration to schema elements
+
+**Serialization:**
+- [`Codec`](./codec.md) — Base abstraction for encoding and decoding values between formats
+- [`Format`](./format.md) — Unified abstraction bundling serialization and deserialization for a specific format
+- [`Type Class Derivation`](./type-class-derivation.md) — Automatic generation of type class instances from schemas
+- [`Syntax`](./syntax.md) — Extension methods for fluent JSON encoding/decoding and patching
+
+**Formats:**
+- [`JSON Codec`](./built-in-codecs/json/) — Complete JSON support: ADT for JSON values, fluent navigation, encoding/decoding, diffs, patches, and JSON Schema 2020-12 validation
+- [`Xml`](./built-in-codecs/xml.md) — Type-safe, immutable representation of XML document structures
+
+**Validation & Errors:**
+- [`Validation`](./validation.md) — Declarative constraints on primitive values
+- [`SchemaError`](./schema-error.md) — Structured error type for schema operations
+- [`Allows`](./allows.md) — Compile-time capability token proving a type satisfies a structural grammar
+
+## Overview
+
+The ZIO Blocks Schema module revolves around the `Schema` and `Reflect` types, which capture the complete structural description of Scala data types at runtime. From a single schema definition, you get automatic derivation of codecs (`Codec`, `Formats`), type-safe data access (`Optics`, `DynamicOptic`), structural validation (`Validation`, `SchemaError`), patching (`Patch`, `JsonPatch`), and serialization to multiple formats (`Json`, `Xml`, `JSON Schema`).
+
+The type system is powered by a register-based architecture (`Registers`) that eliminates boxing overhead, and supports both compile-time (`Allows`) and runtime (`DynamicValue`, `DynamicSchema`) type manipulation. The `Deriver` system (`Type Class Derivation`) enables automatic generation of any type class instance from schema metadata.