@zio.dev/zio-blocks 0.0.51 → 0.0.56
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/adr/2026-07-18-data-migration.md +123 -0
- package/guides/async-getting-started.md +687 -0
- package/guides/compile-time-resource-safety-with-scope.md +6 -0
- package/guides/getting-started-with-mux.md +0 -112
- package/guides/query-dsl-extending.md +1 -1
- package/guides/query-dsl-fluent-builder.md +1 -1
- package/guides/query-dsl-reified-optics.md +1 -1
- package/guides/query-dsl-sql.md +395 -1
- package/guides/sql-checked-interpolation.md +173 -0
- package/guides/sql-transactions.md +286 -0
- package/guides/telemetry-guide.md +131 -70
- package/guides/zio-schema-migration.md +6 -6
- package/index.md +200 -559
- package/package.json +1 -1
- package/reference/async.md +1379 -531
- package/reference/chunk.md +3 -3
- package/reference/codegen/index.md +1 -1
- package/reference/combinators.md +4 -4
- package/reference/config/config-decoder.md +460 -0
- package/reference/config/config-source.md +489 -0
- package/reference/config/errors.md +278 -0
- package/reference/config/flags.md +369 -0
- package/reference/config/formats.md +314 -0
- package/reference/config/index.md +304 -0
- package/reference/config/rollout.md +336 -0
- package/reference/context.md +6 -49
- package/reference/data-migration.md +269 -0
- package/reference/datastar/attributes.md +302 -0
- package/reference/datastar/events.md +234 -0
- package/reference/datastar/index.md +256 -0
- package/reference/datastar/signals.md +230 -0
- package/reference/datastar/sse.md +295 -0
- package/reference/datastar.md +2 -2
- package/reference/docs.md +2 -2
- package/reference/endpoint/bulk-creation.md +96 -0
- package/reference/endpoint/endpoint.md +1 -0
- package/reference/endpoint/index.md +9 -89
- package/reference/endpoint/path-codec.md +12 -24
- package/reference/endpoint/route-pattern.md +4 -6
- package/reference/endpoint/segment-codec.md +19 -32
- package/reference/html.md +313 -9
- package/reference/htmx/index.md +4 -52
- package/reference/htmx/response-headers.md +240 -0
- package/reference/http-model/headers.md +735 -0
- package/reference/http-model/index.md +3 -1
- package/reference/http-model/model.md +107 -71
- package/reference/http-model/schema-codecs.md +522 -0
- package/reference/http-model/schema.md +6 -3
- package/reference/http-model/server-sent-event.md +341 -0
- package/reference/jwt.md +195 -0
- package/reference/maybe.md +128 -11
- package/reference/media-type.md +2 -2
- package/reference/mux.mdx +7 -2
- package/reference/openapi.md +3 -3
- package/reference/projection.md +654 -0
- package/reference/resource-management/index.md +1 -1
- package/reference/resource-management/resource.md +2 -98
- package/reference/resource-management/scope.md +1 -209
- package/reference/resource-management/wire.md +4 -50
- package/reference/ringbuffer/advanced.mdx +1 -1
- package/reference/ringbuffer/index.mdx +3 -3
- package/reference/ringbuffer/mpmc.mdx +38 -4
- package/reference/ringbuffer/mpsc.mdx +36 -4
- package/reference/ringbuffer/spmc.mdx +1 -1
- package/reference/ringbuffer/spsc.mdx +87 -15
- package/reference/schema/allows.md +0 -96
- package/reference/schema/binding.md +2 -2
- package/reference/schema/built-in-codecs/avro.md +2 -2
- package/reference/schema/built-in-codecs/bson.md +50 -20
- package/reference/schema/built-in-codecs/csv.md +2 -2
- package/reference/schema/built-in-codecs/index.md +3 -3
- package/reference/schema/built-in-codecs/json/index.md +2 -2
- package/reference/schema/built-in-codecs/json/json.md +1 -0
- package/reference/schema/built-in-codecs/messagepack.md +3 -3
- package/reference/schema/built-in-codecs/thrift.md +2 -2
- package/reference/schema/built-in-codecs/toon.md +3 -3
- package/reference/schema/built-in-codecs/yaml.md +2 -2
- package/reference/schema/codec.md +11 -11
- package/reference/schema/dynamic-optic.md +48 -3
- package/reference/schema/dynamic-schema.md +3 -3
- package/reference/schema/index.md +2 -0
- package/reference/schema/path-interpolator.md +2 -0
- package/reference/schema/reflect-transformer.md +140 -0
- package/reference/schema/schema-evolution/as.md +4 -4
- package/reference/schema/schema-evolution/into.md +2 -2
- package/reference/schema/schema-expr.md +2 -2
- package/reference/schema/schema-search.md +263 -0
- package/reference/schema/schema.md +10 -2
- package/reference/schema/type-class-derivation.md +1 -1
- package/reference/smithy.md +502 -3
- package/reference/sql/db-codec-deriver.md +3 -3
- package/reference/sql/db-codec.md +22 -22
- package/reference/sql/db-con.md +4 -4
- package/reference/sql/db-connection.md +1 -1
- package/reference/sql/db-param.md +1 -1
- package/reference/sql/db-result-reader.md +4 -2
- package/reference/sql/db-tx.md +46 -14
- package/reference/sql/ddl.md +1 -1
- package/reference/sql/frag.md +44 -10
- package/reference/sql/index.md +7 -7
- package/reference/sql/repo.md +15 -15
- package/reference/sql/sql-dialect.md +1 -1
- package/reference/sql/sql-logger.md +1 -1
- package/reference/sql/sql-name-mapper.md +3 -3
- package/reference/sql/table-metadata.md +3 -3
- package/reference/sql/table.md +10 -10
- package/reference/sql/transactor-zio.md +1 -1
- package/reference/sql/transactor.md +21 -11
- package/reference/sql-zio.md +2 -2
- package/reference/streams/core/index.md +32 -0
- package/reference/streams/{pipeline.md → core/pipeline.md} +210 -74
- package/reference/streams/{sink.md → core/sink.md} +331 -353
- package/reference/streams/{stream.md → core/stream.md} +919 -209
- package/reference/streams/execution-and-compatibility/async-execution.md +822 -0
- package/reference/streams/execution-and-compatibility/index.md +35 -0
- package/reference/streams/execution-and-compatibility/platform-differences.md +297 -0
- package/reference/streams/execution-and-compatibility/scala-2-compatibility.md +88 -0
- package/reference/streams/execution-and-compatibility/zero-boxing.md +393 -0
- package/reference/streams/index.md +140 -67
- package/reference/streams/primitives/index.md +30 -0
- package/reference/streams/primitives/reader.md +1992 -0
- package/reference/streams/{writer.md → primitives/writer.md} +254 -98
- package/reference/telemetry/common/any-value.md +90 -0
- package/reference/telemetry/common/attribute-key.md +87 -0
- package/reference/telemetry/common/attributes.md +118 -0
- package/reference/telemetry/common/index.md +39 -0
- package/reference/telemetry/common/instrumentation-scope.md +24 -0
- package/reference/telemetry/common/resource.md +34 -0
- package/reference/telemetry/index.md +311 -0
- package/reference/telemetry/logging/index.md +197 -0
- package/reference/telemetry/logging/log-enrichment.md +72 -0
- package/reference/telemetry/logging/log-formatter.md +100 -0
- package/reference/telemetry/logging/log-record-processor.md +56 -0
- package/reference/telemetry/logging/log-record.md +44 -0
- package/reference/telemetry/logging/log-writer.md +64 -0
- package/reference/telemetry/logging/logger-provider.md +142 -0
- package/reference/telemetry/logging/logger.md +83 -0
- package/reference/telemetry/logging/severity.md +62 -0
- package/reference/telemetry/metrics/index.md +150 -0
- package/reference/telemetry/metrics/instruments.md +183 -0
- package/reference/telemetry/metrics/labeled-instruments.md +74 -0
- package/reference/telemetry/metrics/meter-provider.md +76 -0
- package/reference/telemetry/metrics/meter.md +98 -0
- package/reference/telemetry/metrics/metric-data.md +57 -0
- package/reference/telemetry/otel/custom-exporter.md +216 -0
- package/reference/telemetry/otel/index.md +212 -0
- package/reference/telemetry/tracing/index.md +155 -0
- package/reference/telemetry/tracing/sampler.md +89 -0
- package/reference/telemetry/tracing/span-builder.md +57 -0
- package/reference/telemetry/tracing/span-context.md +39 -0
- package/reference/telemetry/tracing/span-data.md +32 -0
- package/reference/telemetry/tracing/span-kind.md +55 -0
- package/reference/telemetry/tracing/span-processor.md +53 -0
- package/reference/telemetry/tracing/span-status.md +47 -0
- package/reference/telemetry/tracing/span.md +117 -0
- package/reference/telemetry/tracing/tracer-provider.md +91 -0
- package/reference/telemetry/tracing/tracer.md +52 -0
- package/reference/typeid.md +0 -64
- package/sidebars.js +365 -185
- package/undocumented-report.md +528 -270
- package/reference/config.md +0 -158
- package/reference/streams/concurrent-operators.md +0 -106
- package/reference/streams/reader.md +0 -1284
- package/reference/streams/scala-2-compatibility.md +0 -55
- package/reference/streams/zero-boxing.md +0 -275
- package/reference/telemetry.md +0 -693
|
@@ -1,27 +1,53 @@
|
|
|
1
1
|
---
|
|
2
2
|
id: stream
|
|
3
3
|
title: "Stream"
|
|
4
|
+
sidebar_label: "Stream"
|
|
5
|
+
description: "The Stream data type: construction, transformation, resource safety, and the cross-platform async and JVM-only blocking terminal families."
|
|
6
|
+
keywords:
|
|
7
|
+
- "Pull-Based Streams"
|
|
8
|
+
- "Stateful Transformations"
|
|
9
|
+
- "Async Operators"
|
|
10
|
+
- "Blocking Terminals"
|
|
11
|
+
- "Stream"
|
|
4
12
|
---
|
|
5
13
|
|
|
6
14
|
import Tabs from '@theme/Tabs';
|
|
7
15
|
import TabItem from '@theme/TabItem';
|
|
8
16
|
|
|
9
|
-
`Stream[+E, +A]` is a **lazy, pull-based, typed-error stream** of elements that may fail with an error of type `E`. Nothing executes until a terminal operation is
|
|
17
|
+
`Stream[+E, +A]` is a **lazy, pull-based, typed-error stream** of elements that may fail with an error of type `E`. Nothing executes until a terminal operation is driven. Cross-platform asynchronous terminals return `Async[Either[E, Z]]`; the JVM also provides the existing blocking terminal family returning `Either[E, Z]`. Typed errors surface as `Left(e)`, while defects and cleanup failures fail the outer `Async` or propagate from a JVM blocking terminal:
|
|
10
18
|
|
|
11
19
|
```scala
|
|
12
20
|
abstract class Stream[+E, +A] {
|
|
13
|
-
def
|
|
14
|
-
|
|
21
|
+
def runAsync[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
|
|
22
|
+
errorConcat: Concat.WithOut[E, ES, E3]
|
|
23
|
+
): Async[Either[E3, Z]]
|
|
24
|
+
def runCollectAsync: Async[Either[E, Chunk[A]]]
|
|
25
|
+
|
|
26
|
+
// JVM only
|
|
27
|
+
def run[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
|
|
28
|
+
errorConcat: Concat.WithOut[E, ES, E3]
|
|
29
|
+
): Either[E3, Z]
|
|
15
30
|
}
|
|
16
31
|
```
|
|
17
32
|
|
|
18
33
|
`Stream` is purely functional, referentially transparent, and resource-safe:
|
|
19
34
|
- **Lazy**: descriptions of pipelines, not eager computations
|
|
20
|
-
- **
|
|
35
|
+
- **Nonblocking across platforms**: `*Async` terminals drive synchronous or asynchronous readers without blocking JavaScript
|
|
36
|
+
- **JVM-compatible**: plain blocking terminals remain available on the JVM
|
|
21
37
|
- **Pull-based**: execution is driven from the sink backward through the pipeline
|
|
22
38
|
- **Typed errors**: distinguish recoverable errors (`E`) from untyped defects (`Throwable`)
|
|
23
39
|
- **Resource-safe**: RAII semantics ensure resources are released in all cases
|
|
24
40
|
|
|
41
|
+
### Asynchronous Source Constructors
|
|
42
|
+
|
|
43
|
+
The companion constructors whose names end in `Async` — `attemptAsync`, `attemptEvalAsync`, `deferAsync`, `evalAsync`, `fromAcquireReleaseAsync`, `fromIteratorAsync`, `fromReaderAsync`, and `unfoldAsync` — defer their `Async` thunk until the first reader operation is driven, and `Stream.unwrap` flattens an `Async[Stream[E, A]]` so that ordinary operators such as `flatMap`, `catchAll`, and `flatMapPar` compose with asynchronously produced streams. [Async Source Constructors](../execution-and-compatibility/async-execution.md#async-source-constructors) documents each of them, along with the laziness and error conventions they share; this page does not repeat them.
|
|
44
|
+
|
|
45
|
+
### Source Compatibility
|
|
46
|
+
|
|
47
|
+
`Reader` is an ordinary `abstract class`, not a sealed one, but every reader the library hands you is a `Reader.SyncReader[A]` or a `Reader.AsyncReader[A]`, so code that implements or accepts a reader must choose one kind or match both with a fallback case. Custom sinks cannot be written by subclassing `Sink`, whose two abstract drains are `private[streams]`; use `Sink.createAsync`, `Sink.createBoth`, or the JVM-only `Sink.create`. Plain terminals, `start`, `AsyncReader#toSync`, and `Sink.create` are JVM-only, so shared sources should migrate to `run*Async`, `startAsync`/`useReaderAsync`, and `createAsync`.
|
|
48
|
+
|
|
49
|
+
Async constructor callbacks remain lazy until the first drive, and managed/unmanaged names encode ownership. Do not compensate by eagerly opening a resource before constructing the stream. Cancellation closes an acquired reader and awaits its finalizer; `startAsync` is the exception because it explicitly transfers that responsibility to its caller.
|
|
50
|
+
|
|
25
51
|
## Motivation
|
|
26
52
|
|
|
27
53
|
Traditional eager sequences (like Scala `List`) fall short in **three critical dimensions**. Here's what `Stream[E, A]` solves for each:
|
|
@@ -227,6 +253,8 @@ val result = singleElement.runCollect
|
|
|
227
253
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(42))
|
|
228
254
|
```
|
|
229
255
|
|
|
256
|
+
The `Byte` overload remains a byte stream: `Stream.succeed(1.toByte)` has type `Stream[Nothing, Byte]`, uses the `Byte` representation lane, and compiles through `Reader.singleByte` rather than widening the element type to `Int`.
|
|
257
|
+
|
|
230
258
|
#### `Stream.fail[E]`
|
|
231
259
|
|
|
232
260
|
Creates a stream that fails immediately with a typed error:
|
|
@@ -273,7 +301,7 @@ val dieStream = Stream.die(new Exception("System failure"))
|
|
|
273
301
|
|
|
274
302
|
### From Collections
|
|
275
303
|
|
|
276
|
-
Streams can be created from existing collections and iterables, making it easy to convert `List`, `Array`, `Chunk
|
|
304
|
+
Streams can be created from existing collections and iterables, making it easy to convert `List`, `Array`, [`Chunk`](../../chunk.md), or custom iterables into lazy streams:
|
|
277
305
|
|
|
278
306
|
#### `Stream.apply[A]`
|
|
279
307
|
|
|
@@ -281,7 +309,7 @@ Wraps a variable number of arguments into a stream:
|
|
|
281
309
|
|
|
282
310
|
```scala
|
|
283
311
|
object Stream {
|
|
284
|
-
def apply[A](as: A*): Stream[Nothing, A]
|
|
312
|
+
def apply[A](as: A*)(implicit jt: JvmType.Infer[A]): Stream[Nothing, A]
|
|
285
313
|
}
|
|
286
314
|
```
|
|
287
315
|
|
|
@@ -302,7 +330,7 @@ Converts a `Chunk` into a stream. Chunks are immutable, indexed sequences optimi
|
|
|
302
330
|
|
|
303
331
|
```scala
|
|
304
332
|
object Stream {
|
|
305
|
-
def fromChunk[A](chunk: Chunk[A]): Stream[Nothing, A]
|
|
333
|
+
def fromChunk[A](chunk: Chunk[A])(implicit jt: JvmType.Infer[A]): Stream[Nothing, A]
|
|
306
334
|
}
|
|
307
335
|
```
|
|
308
336
|
|
|
@@ -326,7 +354,7 @@ Converts any `Iterable[A]` (List, Set, Vector, etc.) into a stream:
|
|
|
326
354
|
|
|
327
355
|
```scala
|
|
328
356
|
object Stream {
|
|
329
|
-
def fromIterable[A](it: Iterable[A]): Stream[Nothing, A]
|
|
357
|
+
def fromIterable[A](it: Iterable[A])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
|
|
330
358
|
}
|
|
331
359
|
```
|
|
332
360
|
|
|
@@ -349,7 +377,7 @@ Converts an `Iterator[A]` into a stream. The iterator is consumed lazily:
|
|
|
349
377
|
|
|
350
378
|
```scala
|
|
351
379
|
object Stream {
|
|
352
|
-
def fromIterator[A](it: => Iterator[A]): Stream[Nothing, A]
|
|
380
|
+
def fromIterator[A](it: => Iterator[A])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
|
|
353
381
|
}
|
|
354
382
|
```
|
|
355
383
|
|
|
@@ -424,7 +452,7 @@ Emits the same value infinitely:
|
|
|
424
452
|
|
|
425
453
|
```scala
|
|
426
454
|
object Stream {
|
|
427
|
-
def repeat[A](a: A): Stream[Nothing, A]
|
|
455
|
+
def repeat[A](a: A)(implicit jt: JvmType.Infer[A]): Stream[Nothing, A]
|
|
428
456
|
}
|
|
429
457
|
```
|
|
430
458
|
|
|
@@ -447,7 +475,7 @@ A stateful generator that emits elements based on a fold-like transition functio
|
|
|
447
475
|
|
|
448
476
|
```scala
|
|
449
477
|
object Stream {
|
|
450
|
-
def unfold[S, A](s: S)(f: S => Option[(A, S)]): Stream[Nothing, A]
|
|
478
|
+
def unfold[S, A](s: S)(f: S => Option[(A, S)])(implicit jtA: JvmType.Infer[A]): Stream[Nothing, A]
|
|
451
479
|
}
|
|
452
480
|
```
|
|
453
481
|
|
|
@@ -500,7 +528,7 @@ Wraps a potentially throwing computation, converting non-fatal `Throwable`s into
|
|
|
500
528
|
|
|
501
529
|
```scala
|
|
502
530
|
object Stream {
|
|
503
|
-
def attempt[A](f: => A): Stream[Throwable, A]
|
|
531
|
+
def attempt[A](f: => A)(implicit jtA: JvmType.Infer[A]): Stream[Throwable, A]
|
|
504
532
|
}
|
|
505
533
|
```
|
|
506
534
|
|
|
@@ -512,7 +540,7 @@ import zio.blocks.streams.*
|
|
|
512
540
|
def unsafeJsonParse(s: String): Int = s.toInt
|
|
513
541
|
|
|
514
542
|
val parsed = Stream.attempt(unsafeJsonParse("42"))
|
|
515
|
-
// parsed: Stream[Throwable, Int] = Stream.
|
|
543
|
+
// parsed: Stream[Throwable, Int] = Stream.attempt(...)
|
|
516
544
|
val result = parsed.runCollect
|
|
517
545
|
// result: Either[Throwable, Chunk[Int]] = Right(IndexedSeq(42))
|
|
518
546
|
```
|
|
@@ -541,7 +569,7 @@ val result = effect.runDrain
|
|
|
541
569
|
|
|
542
570
|
#### `Stream.defer[A]`
|
|
543
571
|
|
|
544
|
-
|
|
572
|
+
Creates an empty stream that lazily registers `f` as a release action. `f` runs exactly once when each materialization closes, including after failure, early termination, or cancellation; exceptions are defects:
|
|
545
573
|
|
|
546
574
|
```scala
|
|
547
575
|
object Stream {
|
|
@@ -549,15 +577,15 @@ object Stream {
|
|
|
549
577
|
}
|
|
550
578
|
```
|
|
551
579
|
|
|
552
|
-
|
|
580
|
+
Register a release action that runs when the stream closes:
|
|
553
581
|
|
|
554
582
|
```scala
|
|
555
583
|
import zio.blocks.streams.*
|
|
556
584
|
|
|
557
|
-
val deferred = Stream.defer(println("
|
|
585
|
+
val deferred = Stream.defer(println("Release action runs when the stream closes"))
|
|
558
586
|
// deferred: Stream[Nothing, Nothing] = Stream.defer(...)
|
|
559
587
|
val result = deferred.runDrain
|
|
560
|
-
//
|
|
588
|
+
// Release action runs when the stream closes
|
|
561
589
|
// result: Either[Nothing, Unit] = Right(())
|
|
562
590
|
```
|
|
563
591
|
|
|
@@ -594,7 +622,7 @@ Reads bytes from a Java `InputStream`, managing the resource:
|
|
|
594
622
|
|
|
595
623
|
```scala
|
|
596
624
|
object Stream {
|
|
597
|
-
def fromInputStream(is: java.io.InputStream): Stream[java.io.IOException,
|
|
625
|
+
def fromInputStream(is: java.io.InputStream): Stream[java.io.IOException, Byte]
|
|
598
626
|
}
|
|
599
627
|
```
|
|
600
628
|
|
|
@@ -605,7 +633,7 @@ import zio.blocks.streams.*
|
|
|
605
633
|
import java.io.ByteArrayInputStream
|
|
606
634
|
|
|
607
635
|
val data = new ByteArrayInputStream("Hello".getBytes)
|
|
608
|
-
// data: ByteArrayInputStream = java.io.ByteArrayInputStream@
|
|
636
|
+
// data: ByteArrayInputStream = java.io.ByteArrayInputStream@461f0ea1
|
|
609
637
|
val bytes = Stream.fromInputStream(data)
|
|
610
638
|
// bytes: Stream[IOException, Byte] = Stream.fromAcquireRelease(...)
|
|
611
639
|
val result = bytes.runCollect
|
|
@@ -631,7 +659,7 @@ import zio.blocks.streams.*
|
|
|
631
659
|
import java.io.StringReader
|
|
632
660
|
|
|
633
661
|
val reader = new StringReader("hello world")
|
|
634
|
-
// reader: StringReader = java.io.StringReader@
|
|
662
|
+
// reader: StringReader = java.io.StringReader@1f2290ef
|
|
635
663
|
val stream = Stream.fromJavaReader(reader)
|
|
636
664
|
// stream: Stream[IOException, Char] = Stream.fromAcquireRelease(...)
|
|
637
665
|
val result = stream.runCollect
|
|
@@ -646,7 +674,7 @@ Reads bytes from a Java `InputStream` without automatic resource management. The
|
|
|
646
674
|
|
|
647
675
|
```scala
|
|
648
676
|
object Stream {
|
|
649
|
-
def fromInputStreamUnmanaged(is: java.io.InputStream): Stream[java.io.IOException,
|
|
677
|
+
def fromInputStreamUnmanaged(is: java.io.InputStream): Stream[java.io.IOException, Byte]
|
|
650
678
|
}
|
|
651
679
|
```
|
|
652
680
|
|
|
@@ -697,8 +725,8 @@ These operations apply functions to stream elements one-by-one, applying the tra
|
|
|
697
725
|
Applies a function to each element:
|
|
698
726
|
|
|
699
727
|
```scala
|
|
700
|
-
|
|
701
|
-
def map[B](f: A => B): Stream[E, B]
|
|
728
|
+
abstract class Stream[+E, +A] {
|
|
729
|
+
def map[B](f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B]
|
|
702
730
|
}
|
|
703
731
|
```
|
|
704
732
|
|
|
@@ -715,15 +743,17 @@ val result = doubled.runCollect
|
|
|
715
743
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(2, 4, 6))
|
|
716
744
|
```
|
|
717
745
|
|
|
718
|
-
|
|
746
|
+
For bounded concurrency, [`Stream#mapPar`](#streammappar) applies a synchronous function with up to `n` applications active and [`Stream#mapParAsync`](#streammapparasync) keeps up to `n` `Async` callbacks in flight; both are unordered. See [Bounded Concurrency](#bounded-concurrency).
|
|
747
|
+
|
|
748
|
+
**Key point:** `Stream#map` is covariant in the output type because it preserves the error type and only transforms elements. Output-changing operations such as `map`, `collect`, `flatMap`, `mapAccum`, `scan`, and `zipWith` take `JvmType.Infer` evidence for their result type; that result evidence selects the physical output lane. Type-preserving operations retain the source's known lane, including when the static element type is widened.
|
|
719
749
|
|
|
720
750
|
#### `Stream#mapError[E2]`
|
|
721
751
|
|
|
722
752
|
Transforms typed errors without affecting elements:
|
|
723
753
|
|
|
724
754
|
```scala
|
|
725
|
-
|
|
726
|
-
|
|
755
|
+
abstract class Stream[+E, +A] {
|
|
756
|
+
def mapError[E2](f: E => E2): Stream[E2, A]
|
|
727
757
|
}
|
|
728
758
|
```
|
|
729
759
|
|
|
@@ -745,7 +775,7 @@ val mapped = mayFail.mapError(e => ServerError("Connection failed"))
|
|
|
745
775
|
Emits only elements that satisfy a predicate:
|
|
746
776
|
|
|
747
777
|
```scala
|
|
748
|
-
|
|
778
|
+
abstract class Stream[+E, +A] {
|
|
749
779
|
def filter(pred: A => Boolean): Stream[E, A]
|
|
750
780
|
}
|
|
751
781
|
```
|
|
@@ -768,8 +798,8 @@ val result = evens.runCollect
|
|
|
768
798
|
Applies a partial function, emitting only defined results:
|
|
769
799
|
|
|
770
800
|
```scala
|
|
771
|
-
|
|
772
|
-
def collect[B](pf: PartialFunction[A, B]): Stream[E, B]
|
|
801
|
+
abstract class Stream[+E, +A] {
|
|
802
|
+
def collect[B](pf: PartialFunction[A, B])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
|
|
773
803
|
}
|
|
774
804
|
```
|
|
775
805
|
|
|
@@ -795,8 +825,8 @@ These operations maintain internal state while processing elements, allowing you
|
|
|
795
825
|
Maintains state while transforming each element:
|
|
796
826
|
|
|
797
827
|
```scala
|
|
798
|
-
|
|
799
|
-
def mapAccum[S, B](init: S)(f: (S, A) => (S, B)): Stream[E, B]
|
|
828
|
+
abstract class Stream[+E, +A] {
|
|
829
|
+
def mapAccum[S, B](init: S)(f: (S, A) => (S, B))(implicit jtB: JvmType.Infer[B]): Stream[E, B]
|
|
800
830
|
}
|
|
801
831
|
```
|
|
802
832
|
|
|
@@ -808,20 +838,44 @@ import zio.blocks.streams.*
|
|
|
808
838
|
val nums = Stream(1, 2, 3)
|
|
809
839
|
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
|
|
810
840
|
val indexed = nums.mapAccum(0)((idx, x) => (idx + 1, (idx, x)))
|
|
811
|
-
// indexed: Stream[Nothing, Tuple2[Int, Int]] = Stream
|
|
841
|
+
// indexed: Stream[Nothing, Tuple2[Int, Int]] = Stream.suspend(...)
|
|
812
842
|
val result = indexed.runCollect
|
|
813
843
|
// result: Either[Nothing, Chunk[Tuple2[Int, Int]]] = Right(
|
|
814
844
|
// IndexedSeq((0, 1), (1, 2), (2, 3))
|
|
815
845
|
// )
|
|
816
846
|
```
|
|
817
847
|
|
|
848
|
+
#### `Stream#mapAccumAsync[S, B]`
|
|
849
|
+
|
|
850
|
+
The asynchronous twin of `mapAccum`: the step returns an `Async`, and the state is still threaded strictly in order.
|
|
851
|
+
|
|
852
|
+
```scala
|
|
853
|
+
abstract class Stream[+E, +A] {
|
|
854
|
+
def mapAccumAsync[S, B](init: S)(f: (S, A) => Async[(S, B)])(implicit
|
|
855
|
+
jtB: JvmType.Infer[B]
|
|
856
|
+
): Stream[E, B]
|
|
857
|
+
}
|
|
858
|
+
```
|
|
859
|
+
|
|
860
|
+
At most one invocation of `f` is active at a time, which is what keeps the accumulator meaningful — there is no concurrency here to reorder the steps or to hand two invocations the same state. A failure inside `f` is a defect, not a typed error. The implicit `JvmType.Infer[B]` records the physical lane of the new output type and is supplied by the compiler.
|
|
861
|
+
|
|
862
|
+
```scala
|
|
863
|
+
import zio.blocks.async.*
|
|
864
|
+
import zio.blocks.streams.*
|
|
865
|
+
|
|
866
|
+
val events = Stream("open", "write", "close")
|
|
867
|
+
val numbered = events.mapAccumAsync(0L)((seq, event) => Async.succeed((seq + 1, s"$seq:$event")))
|
|
868
|
+
```
|
|
869
|
+
|
|
870
|
+
This operator is also catalogued with the rest of the sequential asynchronous family in [Async Operators](../execution-and-compatibility/async-execution.md#streammapaccumasync), and [Stateful Asynchronous Operators](#stateful-asynchronous-operators) runs it end to end alongside `scanAsync`, `takeWhileAsync`, and `ensuringAsync`.
|
|
871
|
+
|
|
818
872
|
#### `Stream#scan[S]`
|
|
819
873
|
|
|
820
|
-
Like `mapAccum`, but
|
|
874
|
+
Like `mapAccum`, but emits the accumulator rather than a mapped value, starting with `init` — so the output stream has one more element than the input:
|
|
821
875
|
|
|
822
876
|
```scala
|
|
823
|
-
|
|
824
|
-
def scan[S](init: S)(f: (S, A) => S): Stream[E, S]
|
|
877
|
+
abstract class Stream[+E, +A] {
|
|
878
|
+
def scan[S](init: S)(f: (S, A) => S)(implicit jtS: JvmType.Infer[S]): Stream[E, S]
|
|
825
879
|
}
|
|
826
880
|
```
|
|
827
881
|
|
|
@@ -838,13 +892,38 @@ val result = cumsum.runCollect
|
|
|
838
892
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(0, 1, 3, 6, 10))
|
|
839
893
|
```
|
|
840
894
|
|
|
895
|
+
#### `Stream#scanAsync[S]`
|
|
896
|
+
|
|
897
|
+
The asynchronous twin of `scan`: the fold step returns an `Async`, and the accumulator is still emitted at each step.
|
|
898
|
+
|
|
899
|
+
```scala
|
|
900
|
+
abstract class Stream[+E, +A] {
|
|
901
|
+
def scanAsync[S](init: S)(f: (S, A) => Async[S])(implicit jtS: JvmType.Infer[S]): Stream[E, S]
|
|
902
|
+
}
|
|
903
|
+
```
|
|
904
|
+
|
|
905
|
+
The output stream carries one more element than the input, because `init` is emitted before the first step runs. As with `mapAccumAsync`, the steps are sequential and a failure inside `f` is a defect.
|
|
906
|
+
|
|
907
|
+
```scala
|
|
908
|
+
import zio.blocks.async.*
|
|
909
|
+
import zio.blocks.streams.*
|
|
910
|
+
|
|
911
|
+
val amounts = Stream(120, -40, 75)
|
|
912
|
+
val balances = amounts.scanAsync(0L)((balance, amount) => Async.succeed(balance + amount))
|
|
913
|
+
```
|
|
914
|
+
|
|
915
|
+
`balances` emits `0`, `120`, `80`, `155`. See [Async Operators](../execution-and-compatibility/async-execution.md#streamscanasync) for the same entry alongside the rest of the asynchronous family.
|
|
916
|
+
|
|
841
917
|
### Flat-Mapping (Nested Streams)
|
|
842
918
|
|
|
843
|
-
`flatMap[E2, B]` — Maps each element to a stream and flattens the results.:
|
|
919
|
+
`flatMap[E2, E3, B]` — Maps each element to a stream and flattens the results.:
|
|
844
920
|
|
|
845
921
|
```scala
|
|
846
|
-
|
|
847
|
-
def flatMap[E2, B](f: A => Stream[E2, B])
|
|
922
|
+
abstract class Stream[+E, +A] {
|
|
923
|
+
def flatMap[E2, E3, B](f: A => Stream[E2, B])(implicit
|
|
924
|
+
errorConcat: Concat.WithOut[E, E2, E3],
|
|
925
|
+
jtB: JvmType.Infer[B]
|
|
926
|
+
): Stream[E3, B]
|
|
848
927
|
}
|
|
849
928
|
```
|
|
850
929
|
|
|
@@ -863,13 +942,15 @@ val result = expanded.runCollect
|
|
|
863
942
|
// )
|
|
864
943
|
```
|
|
865
944
|
|
|
945
|
+
For the concurrent counterpart, which merges up to `n` inner streams at once in arrival order, see [`Stream#flatMapPar`](#streamflatmappar) in [Bounded Concurrency](#bounded-concurrency).
|
|
946
|
+
|
|
866
947
|
#### `Stream.flattenAll[E, A]`
|
|
867
948
|
|
|
868
949
|
Flattens a stream of streams into a single stream, processing them sequentially:
|
|
869
950
|
|
|
870
951
|
```scala
|
|
871
952
|
object Stream {
|
|
872
|
-
def flattenAll[E, A](streams: Stream[E, Stream[E, A]]): Stream[E, A]
|
|
953
|
+
def flattenAll[E, A](streams: Stream[E, Stream[E, A]])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
|
|
873
954
|
}
|
|
874
955
|
```
|
|
875
956
|
|
|
@@ -889,6 +970,8 @@ val result = flat.runCollect
|
|
|
889
970
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4))
|
|
890
971
|
```
|
|
891
972
|
|
|
973
|
+
To flatten a stream of streams concurrently instead of sequentially, the companion also offers [`Stream.mergeAll`](#streammergeall), which drains up to `maxOpen` inner streams at a time. See [Bounded Concurrency](#bounded-concurrency).
|
|
974
|
+
|
|
892
975
|
## Windowing
|
|
893
976
|
|
|
894
977
|
Streams can be grouped, sliced, and scanned to process data in temporal windows. These operations group elements into chunks and slide windows over the stream for batch processing:
|
|
@@ -898,7 +981,7 @@ Streams can be grouped, sliced, and scanned to process data in temporal windows.
|
|
|
898
981
|
Collects elements into fixed-size chunks:
|
|
899
982
|
|
|
900
983
|
```scala
|
|
901
|
-
|
|
984
|
+
abstract class Stream[+E, +A] {
|
|
902
985
|
def grouped(n: Int): Stream[E, Chunk[A]]
|
|
903
986
|
}
|
|
904
987
|
```
|
|
@@ -923,7 +1006,7 @@ val result = groups.runCollect
|
|
|
923
1006
|
Creates a sliding window of size `n`, optionally stepping by `step` elements:
|
|
924
1007
|
|
|
925
1008
|
```scala
|
|
926
|
-
|
|
1009
|
+
abstract class Stream[+E, +A] {
|
|
927
1010
|
def sliding(n: Int, step: Int = 1): Stream[E, Chunk[A]]
|
|
928
1011
|
}
|
|
929
1012
|
```
|
|
@@ -949,11 +1032,15 @@ Streams can be sequentially concatenated, zipped together, or merged:
|
|
|
949
1032
|
|
|
950
1033
|
### Sequential Concatenation
|
|
951
1034
|
|
|
952
|
-
`++[E2, A2]` or `concat[E2, A2]` — Emits all elements of the first stream, then all elements of the second stream:
|
|
1035
|
+
`++[E2, E3, A2, A3]` or `concat[E2, E3, A2, A3]` — Emits all elements of the first stream, then all elements of the second stream:
|
|
953
1036
|
|
|
954
1037
|
```scala
|
|
955
|
-
|
|
956
|
-
def ++[E2, A2](that: Stream[E2, A2])
|
|
1038
|
+
abstract class Stream[+E, +A] {
|
|
1039
|
+
final def ++[E2, E3, A2, A3](that: Stream[E2, A2])(implicit
|
|
1040
|
+
errorConcat: Concat.WithOut[E, E2, E3],
|
|
1041
|
+
valueConcat: Concat.WithOut[A, A2, A3],
|
|
1042
|
+
jtA3: JvmType.Infer[A3]
|
|
1043
|
+
): Stream[E3, A3] = concat(that)
|
|
957
1044
|
}
|
|
958
1045
|
```
|
|
959
1046
|
|
|
@@ -1029,12 +1116,12 @@ val right = Stream.succeed(true)
|
|
|
1029
1116
|
// right: Stream[Nothing, Boolean] = Stream.succeed(...)
|
|
1030
1117
|
|
|
1031
1118
|
left.runCollect
|
|
1032
|
-
//
|
|
1119
|
+
// res39: Either[LeftError, Chunk[String]] = Left(Boom("boom"))
|
|
1033
1120
|
|
|
1034
1121
|
val failed = left ++ (Stream.fail(Missing(404)): Stream[Missing, Boolean])
|
|
1035
1122
|
// failed: Stream[LeftError | Missing, String | Boolean] = Stream.fail(...) ++ Stream.fail(...)
|
|
1036
1123
|
failed.runCollect
|
|
1037
|
-
//
|
|
1124
|
+
// res40: Either[LeftError | Missing, Chunk[String | Boolean]] = Left(
|
|
1038
1125
|
// Boom("boom")
|
|
1039
1126
|
// )
|
|
1040
1127
|
```
|
|
@@ -1043,15 +1130,20 @@ There is no separate `choice` operator anymore. Use `++` / `concat` for all sequ
|
|
|
1043
1130
|
|
|
1044
1131
|
### Zipping
|
|
1045
1132
|
|
|
1046
|
-
Zips two streams together as tuples
|
|
1133
|
+
Zips two streams together as tuples:
|
|
1047
1134
|
|
|
1048
1135
|
```scala
|
|
1049
|
-
|
|
1050
|
-
def &&[E2, B, C](that: Stream[E2, B])(
|
|
1051
|
-
|
|
1052
|
-
|
|
1136
|
+
abstract class Stream[+E, +A] {
|
|
1137
|
+
def &&[E2, E3, B, C](that: Stream[E2, B])(implicit
|
|
1138
|
+
errorConcat: Concat.WithOut[E, E2, E3],
|
|
1139
|
+
zip: Stream.Zip[A, B, C],
|
|
1140
|
+
jtC: JvmType.Infer[C]
|
|
1141
|
+
): Stream[E3, C]
|
|
1142
|
+
}
|
|
1053
1143
|
```
|
|
1054
1144
|
|
|
1145
|
+
The error type `E3` is the `Concat` of the two error types, and the element type `C` is chosen by the `Stream.Zip` evidence, which flattens nested pairs so that `a && b && c` produces a `Stream` of `(A, B, C)`.
|
|
1146
|
+
|
|
1055
1147
|
The result streams have the same length as the shorter input:
|
|
1056
1148
|
|
|
1057
1149
|
```scala
|
|
@@ -1062,13 +1154,540 @@ val nums = Stream(1, 2, 3)
|
|
|
1062
1154
|
val chars = Stream('a', 'b')
|
|
1063
1155
|
// chars: Stream[Nothing, Char] = Stream(a, b)
|
|
1064
1156
|
val zipped = nums && chars
|
|
1065
|
-
// zipped: Stream[Nothing, Tuple2[Int, Char]] = Stream
|
|
1157
|
+
// zipped: Stream[Nothing, Tuple2[Int, Char]] = Stream(1, 2, 3) && Stream(a, b)
|
|
1066
1158
|
val result = zipped.runCollect
|
|
1067
1159
|
// result: Either[Nothing, Chunk[Tuple2[Int, Char]]] = Right(
|
|
1068
1160
|
// IndexedSeq((1, 'a'), (2, 'b'))
|
|
1069
1161
|
// )
|
|
1070
1162
|
```
|
|
1071
1163
|
|
|
1164
|
+
## Bounded Concurrency
|
|
1165
|
+
|
|
1166
|
+
Four operators bound the concurrency of a stream. `Stream#mapPar`, `Stream#flatMapPar`, and `Stream.mergeAll` take synchronous callbacks; `Stream#mapParAsync` takes a callback that returns an `Async`. All four run on either of two execution paths — a synchronous one backed by worker threads, or an asynchronous one — and all four carry an `n = 1` degradation guarantee.
|
|
1167
|
+
|
|
1168
|
+
The two execution paths are different engines with different bounds, and which one a program gets is decided by the kind of reader its pipeline compiles to — not by which operator it called.
|
|
1169
|
+
|
|
1170
|
+
### The Operators
|
|
1171
|
+
|
|
1172
|
+
| Operator | Element callback | What `n` bounds |
|
|
1173
|
+
|-------------------------------------|------------------------------------------------|--------------------------------|
|
|
1174
|
+
| `Stream#mapPar(n)(f)` | `A => B` | concurrent applications of `f` |
|
|
1175
|
+
| `Stream#mapParAsync(n)(f)` | `A => Async[B]` | callbacks in flight |
|
|
1176
|
+
| `Stream#flatMapPar(n)(f)` | `A => Stream[E1, B]` | open inner streams |
|
|
1177
|
+
| `Stream.mergeAll(maxOpen)(streams)` | none; `streams` is a `Stream[E, Stream[E, A]]` | open inner streams |
|
|
1178
|
+
|
|
1179
|
+
Each of the four begins with `require` on its parallelism argument, so passing zero or a negative number raises `IllegalArgumentException` at description time rather than producing an empty or sequential stream.
|
|
1180
|
+
|
|
1181
|
+
#### `Stream#mapPar`
|
|
1182
|
+
|
|
1183
|
+
```scala
|
|
1184
|
+
def mapPar[B](n: Int)(f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B]
|
|
1185
|
+
```
|
|
1186
|
+
|
|
1187
|
+
Applies `f` to each element with up to `n` applications active. Output is unordered: elements leave in the order their applications finish, not the order they entered. `f` is synchronous, so this is the operator for CPU-bound or blocking work on the JVM — and, as [Threading and Platform Behaviour](#threading-and-platform-behaviour) explains, the operator that overlaps nothing at all once the pipeline is on the asynchronous lane.
|
|
1188
|
+
|
|
1189
|
+
#### `Stream#mapParAsync`
|
|
1190
|
+
|
|
1191
|
+
```scala
|
|
1192
|
+
def mapParAsync[B](n: Int)(f: A => Async[B])(implicit jtB: JvmType.Infer[B]): Stream[E, B]
|
|
1193
|
+
```
|
|
1194
|
+
|
|
1195
|
+
Keeps at most `n` `Async` callbacks in flight and emits each result in completion order. It is the only member of the family whose callback can suspend: `f` returns a description, so the engine holds `n` unfinished effects rather than `n` busy threads. A failure inside the callback is a defect, not a typed error, and it fails the outer terminal effect.
|
|
1196
|
+
|
|
1197
|
+
Unlike `mapPar`, this operator has no synchronous materialization at all. It compiles to the shared asynchronous concurrent reader on both platforms, which is why its `n` counts suspended callbacks rather than workers.
|
|
1198
|
+
|
|
1199
|
+
#### `Stream#flatMapPar`
|
|
1200
|
+
|
|
1201
|
+
```scala
|
|
1202
|
+
def flatMapPar[E1 >: E, B](n: Int)(f: A => Stream[E1, B])(implicit jtB: JvmType.Infer[B]): Stream[E1, B]
|
|
1203
|
+
```
|
|
1204
|
+
|
|
1205
|
+
Applies `f` to each element to produce an inner stream, then merges up to `n` inner streams concurrently. It has no engine of its own. Past the `n = 1` branch, its body is a single delegation:
|
|
1206
|
+
|
|
1207
|
+
```scala
|
|
1208
|
+
Stream.mergeAll[E1, B](n)(
|
|
1209
|
+
this.asInstanceOf[Stream[E1, A]].map(f)(JvmType.Infer.boxed[Stream[E1, B]])
|
|
1210
|
+
)(jtB)
|
|
1211
|
+
```
|
|
1212
|
+
|
|
1213
|
+
One fan-in engine, not two. Everything below about slots, admission, ordering, and shutdown is stated for `mergeAll`, and `flatMapPar` inherits all of it unchanged.
|
|
1214
|
+
|
|
1215
|
+
#### `Stream.mergeAll`
|
|
1216
|
+
|
|
1217
|
+
```scala
|
|
1218
|
+
def mergeAll[E, A](maxOpen: Int)(streams: Stream[E, Stream[E, A]])(implicit jtA: JvmType.Infer[A]): Stream[E, A]
|
|
1219
|
+
```
|
|
1220
|
+
|
|
1221
|
+
Merges up to `maxOpen` inner streams concurrently into one output stream, with elements arriving in completion order. Note the argument: `streams` is a *stream of streams*, not a varargs list, so a fixed collection of sources is fed in through a constructor such as `Stream.fromIterable`.
|
|
1222
|
+
|
|
1223
|
+
```scala
|
|
1224
|
+
import zio.blocks.streams._
|
|
1225
|
+
|
|
1226
|
+
val sources: Stream[Nothing, Stream[Nothing, Int]] =
|
|
1227
|
+
Stream.fromIterable((0 until 10).map(i => Stream.range(i * 100, (i + 1) * 100)))
|
|
1228
|
+
|
|
1229
|
+
val merged: Stream[Nothing, Int] = Stream.mergeAll(4)(sources)
|
|
1230
|
+
```
|
|
1231
|
+
|
|
1232
|
+
### Semantics
|
|
1233
|
+
|
|
1234
|
+
#### What `n` Means
|
|
1235
|
+
|
|
1236
|
+
`n` is a count of selector slots, not of threads. The concurrent reader allocates one `AsyncSelector` with `n + 1` entries: `n` entries for inner work, and one final entry reserved for the outer source.
|
|
1237
|
+
|
|
1238
|
+
```
|
|
1239
|
+
┌────────────────────────────────────────────────────────────────────┐
|
|
1240
|
+
│ Async.selectorWithCapacity(n + 1) one selector, n + 1 entries │
|
|
1241
|
+
├────────────────────────────────────────────────────────────────────┤
|
|
1242
|
+
│ entry 0 inner work: callback in flight, or an open inner │
|
|
1243
|
+
│ entry 1 inner work: callback in flight, or an open inner │
|
|
1244
|
+
│ ... up to n of these; `active` counts the occupied ones │
|
|
1245
|
+
│ entry n-1 inner work: callback in flight, or an open inner │
|
|
1246
|
+
├────────────────────────────────────────────────────────────────────┤
|
|
1247
|
+
│ entry n the outer source re-armed only while active < n │
|
|
1248
|
+
└────────────────────────────────────────────────────────────────────┘
|
|
1249
|
+
```
|
|
1250
|
+
|
|
1251
|
+
The extra entry is what keeps the operator from pulling ahead. The source entry is re-armed only while `active < n`, so the reader stops asking upstream for elements the moment every inner slot is taken, and resumes the instant one frees. Nothing queues behind a full set of slots.
|
|
1252
|
+
|
|
1253
|
+
Whether a slot corresponds to a thread depends entirely on which engine materialized. On the JVM's synchronous lane each slot does have a worker thread behind it; on the asynchronous lane a slot is one pending `Async` and there are no threads involved.
|
|
1254
|
+
|
|
1255
|
+
#### Ordering
|
|
1256
|
+
|
|
1257
|
+
All four operators are unordered with respect to input position. An element leaves when its work finishes, so output is in arrival order. The property tests compare results with `.toSet` for exactly this reason: there is no input-order assertion available to make.
|
|
1258
|
+
|
|
1259
|
+
The single exception is `n = 1`, which is exactly sequential — and the `n = 1` tests do assert exact [`Chunk`](../../chunk.md) equality, because at that value the operator is not the concurrent engine at all. See [The `n = 1` Guarantee](#the-n-1-guarantee).
|
|
1260
|
+
|
|
1261
|
+
:::warning[Unordered means unordered on Scala.js too]
|
|
1262
|
+
Single-threaded execution does not restore input order. The concurrent engine runs on Scala.js with immediately-ready effects, and its arrival-order semantics are retained there. Code that depends on Scala.js emitting source order is a bug that will not reproduce on the JVM.
|
|
1263
|
+
:::
|
|
1264
|
+
|
|
1265
|
+
If input order is what you need, use sequential `map`, `mapAsync`, or `flatMap`. Sorting afterwards — `.runCollectAsync.map(_.map(_.sorted))` — recovers *a* total order, but not the input one unless the elements happen to sort that way.
|
|
1266
|
+
|
|
1267
|
+
#### Boundedness
|
|
1268
|
+
|
|
1269
|
+
Two separate things are bounded, and conflating them leads to the wrong buffer size.
|
|
1270
|
+
|
|
1271
|
+
The first is the number of open inner streams. Admission is guarded by an active count: `occupied` is a `Boolean` array of length `n` and `active` is the number of `true` entries, and a new element is accepted only into a free index. A full set of slots stops admission at the source rather than accumulating work anywhere.
|
|
1272
|
+
|
|
1273
|
+
The second is the number of buffered elements, and it exists only on the JVM's synchronous lane. `ConcurrentMapParReader` allocates `n` input and `n` output `SpscRingBuffer`s, each of `bufferSize` capacity, so a `mapPar(8)` with the default buffer holds at most 1024 elements in transit. The concurrent merge readers allocate one output ring per slot on the same basis.
|
|
1274
|
+
|
|
1275
|
+
The asynchronous engine has no rings of its own. `AsyncConcurrentReaders.mapPar` does not even take a buffer size: each slot holds exactly one unfinished effect, so in-flight work is bounded at `n` and nothing more. `AsyncConcurrentReaders.merge` does take one, but spends it on compiling each inner stream (`stream.compile(0, bufferSize)`), where it sizes whatever buffered stages that inner stream contains.
|
|
1276
|
+
|
|
1277
|
+
#### Admission and Replenishment
|
|
1278
|
+
|
|
1279
|
+
A slot is occupied before the work in it begins, and — for merge — freed only after that work has fully closed.
|
|
1280
|
+
|
|
1281
|
+
```
|
|
1282
|
+
free
|
|
1283
|
+
│ outer element arrives: occupied(i) = true, active += 1
|
|
1284
|
+
▼
|
|
1285
|
+
constructing the Async that builds the child runs HERE, in the slot
|
|
1286
|
+
│ child installed
|
|
1287
|
+
▼
|
|
1288
|
+
draining elements reach the consumer in arrival order
|
|
1289
|
+
│ MapWorkerEnd: slot entry replaced by the child's close effect
|
|
1290
|
+
▼
|
|
1291
|
+
closing slot still held; no successor may be admitted yet
|
|
1292
|
+
│ MapWorkerClosed: releaseSlot(i), then armSource()
|
|
1293
|
+
▼
|
|
1294
|
+
free
|
|
1295
|
+
```
|
|
1296
|
+
|
|
1297
|
+
That two-phase ending is the part worth remembering. When an inner stream reaches its end the engine does not free the slot; it replaces the slot's selector entry with the inner reader's own close effect, which resolves to a second signal. Only then does `releaseSlot` clear `occupied(i)`, decrement `active`, and re-arm the source. A slot is therefore never handed to a successor before its predecessor's finalizers have run to completion.
|
|
1298
|
+
|
|
1299
|
+
`mapPar` and `mapParAsync` have the shorter version of this, because a callback has no reader to close: the slot is released in the same step that hands the value to the consumer, and the source is re-armed there.
|
|
1300
|
+
|
|
1301
|
+
The "constructing" phase is where the slot accounting surprises people. With `flatMapPar(n)(a => Stream.unwrap(f(a)))`, the effect `f(a)` that *produces* the child runs inside the slot the child will later occupy — they share one of the `n`, they do not get one each. [Async children and slot accounting](#async-children-and-slot-accounting) below demonstrates this with a running program.
|
|
1302
|
+
|
|
1303
|
+
#### The `n = 1` Guarantee {#the-n-1-guarantee}
|
|
1304
|
+
|
|
1305
|
+
At `n = 1` each operator degrades to its sequential twin. This is a guarantee about the code path, not an optimization note: the branch sits in the public operator body, above every allocation.
|
|
1306
|
+
|
|
1307
|
+
```scala
|
|
1308
|
+
def mapPar[B](n: Int)(f: A => B)(implicit jtB: JvmType.Infer[B]): Stream[E, B] = {
|
|
1309
|
+
require(n >= 1, s"mapPar requires n >= 1, got $n")
|
|
1310
|
+
if (n == 1) map(f)
|
|
1311
|
+
else {
|
|
1312
|
+
jtB.jvmType
|
|
1313
|
+
new Stream.MapPar[E, A, B](this, n, f, elementRepresentation, jtB.jvmType)
|
|
1314
|
+
}
|
|
1315
|
+
}
|
|
1316
|
+
```
|
|
1317
|
+
|
|
1318
|
+
The other three are shaped identically: `mapParAsync(1)` returns `mapAsync(f)`, `flatMapPar(1)` returns `flatMap(f)`, and `mergeAll(1)` returns `streams.flatMap(identity)` — or, when the outer stream is already a `Mapped` node, the fused `mapped.self.flatMap(mapped.f)` that skips the intermediate stream entirely. No reader, no selector, no thread, and no ring is allocated in any of those cases, because the decision is made before the concurrent node is ever constructed.
|
|
1319
|
+
|
|
1320
|
+
The practical consequence is that `n` can be a configuration value that is allowed to be `1`. A deployment that dials concurrency down to one gets the sequential operator, with its exact input ordering, rather than a concurrent engine running at width one.
|
|
1321
|
+
|
|
1322
|
+
### Buffer Sizing
|
|
1323
|
+
|
|
1324
|
+
Concurrent readers on the synchronous lane use ring-buffer queues sized by the enclosing buffer-size region. The default is 64 (the library-internal `Stream.DefaultBufferSize`, which is not part of the public API).
|
|
1325
|
+
|
|
1326
|
+
```scala
|
|
1327
|
+
import zio.blocks.streams._
|
|
1328
|
+
|
|
1329
|
+
def heavyComputation(n: Int): Int = n * n
|
|
1330
|
+
|
|
1331
|
+
val sized: Stream[Nothing, Int] =
|
|
1332
|
+
Stream.bufferSize(256) {
|
|
1333
|
+
Stream.range(0, 1000000).mapPar(8)(heavyComputation)
|
|
1334
|
+
}
|
|
1335
|
+
```
|
|
1336
|
+
|
|
1337
|
+
`Stream.bufferSize(n)` requires a positive power of two and rejects anything else with `IllegalArgumentException`:
|
|
1338
|
+
|
|
1339
|
+
```scala
|
|
1340
|
+
require(n >= 1 && (n & (n - 1)) == 0, s"bufferSize must be a positive power of 2, got $n")
|
|
1341
|
+
```
|
|
1342
|
+
|
|
1343
|
+
Nested regions use the innermost size. Larger buffers absorb bursty producers; smaller ones cut memory when many slots are open at once. The default suits most workloads. On the asynchronous lane its reach is narrower: `mapPar` and `mapParAsync` ignore it entirely, because each slot holds exactly one unfinished effect and the bound is the slot count, while `mergeAll` and `flatMapPar` still pass it down, where it sizes the buffered stages inside each compiled inner stream.
|
|
1344
|
+
|
|
1345
|
+
`Pipeline.buffer(n)` is a different tool for a different job: it inserts a bounded buffer between two stages rather than resizing the queues inside one concurrent reader, and it participates in the asynchronous reader graph on both platforms.
|
|
1346
|
+
|
|
1347
|
+
### Error Behaviour
|
|
1348
|
+
|
|
1349
|
+
First failure wins. The concurrent reader commits a terminal state exactly once, and the first typed source or inner-stream error to reach that commit becomes the result; the fold short-circuits and the operation surfaces as `Left(e)`.
|
|
1350
|
+
|
|
1351
|
+
```scala
|
|
1352
|
+
import zio.blocks.streams._
|
|
1353
|
+
|
|
1354
|
+
// A typed error anywhere upstream terminates every worker.
|
|
1355
|
+
val fromUpstream: Stream[String, Int] =
|
|
1356
|
+
Stream
|
|
1357
|
+
.range(0, 1000)
|
|
1358
|
+
.flatMap(n => if (n == 500) Stream.fail("bad element") else Stream.succeed(n))
|
|
1359
|
+
.mapPar(4)(identity)
|
|
1360
|
+
|
|
1361
|
+
// A typed error in one inner stream terminates the merge.
|
|
1362
|
+
val fromInner: Stream[String, Int] =
|
|
1363
|
+
Stream.mergeAll(4)(
|
|
1364
|
+
Stream.fromIterable(
|
|
1365
|
+
List(Stream.range(0, 100), Stream.fail("inner error"), Stream.range(200, 300))
|
|
1366
|
+
)
|
|
1367
|
+
)
|
|
1368
|
+
```
|
|
1369
|
+
|
|
1370
|
+
Both of those collect to a `Left`. Elements that were already emitted stay emitted — ordering is arrival-based, so a consumer may well have seen output from other slots before the failing one reached the commit point.
|
|
1371
|
+
|
|
1372
|
+
Once a failure is committed, cleanup runs and every sibling is torn down. A failure *during* that cleanup is attached to the primary failure rather than substituted for it: the engine calls `StreamError.attachCleanupReplay(primary, secondary)`, so the error a caller sees is still the one that caused the termination, with the cleanup problem carried alongside it. The same rule holds on the successful path, where a cleanup failure with no primary becomes the failure via `StreamError.attachCleanup(null, cleanupFailure)`.
|
|
1373
|
+
|
|
1374
|
+
:::note[What the tests actually prove]
|
|
1375
|
+
`MergeInnerErrorSpec` asserts `result.isLeft` across the generic, `Int`, `Long`, `Float`, and `Double` lanes under a ten-second timeout, and again with four coordinated simultaneous failures per lane under a thirty-second one. That establishes the general shape — a failing inner terminates the merge promptly and does not hang — but it does not pin down *which* failure wins when several race at `n > 1`. Do not write code that depends on a particular one of several concurrent errors being the one reported.
|
|
1376
|
+
:::
|
|
1377
|
+
|
|
1378
|
+
A defect is different from a typed error. The `mapParAsync` callback failing, or an `ensuring` finalizer throwing, is a defect and fails the outer terminal effect rather than appearing in the `E` channel.
|
|
1379
|
+
|
|
1380
|
+
### Cancellation and Shutdown
|
|
1381
|
+
|
|
1382
|
+
Shutdown is cooperative and ordered, and it is driven from the consumer end. Closing, failing, or cancelling the reader runs the same owned-cleanup path.
|
|
1383
|
+
|
|
1384
|
+
For `mergeAll` and `flatMapPar` that path is three steps, in order: shut the selector down, close every installed inner reader, then close the outer source. The steps are joined rather than sequenced-and-abandoned, so a failure in one does not skip the others — each later close still runs, and its failure is attached to the first. For `mapPar` and `mapParAsync` there are no inner readers, so it is the selector shutdown followed by the upstream close.
|
|
1385
|
+
|
|
1386
|
+
Cancellation goes through the `Async` cancellation protocol rather than thread interruption: the in-flight child run is cancelled with cleanup, and the reader settles its completion from the cancellation's outcome. A cancellation that arrives after the reader is already closed is recognised as stale and does not turn a clean close into a failure.
|
|
1387
|
+
|
|
1388
|
+
Closing is idempotent and it waits. The reader will not report closed until the cleanup it owns has finished, which is the property the slot lifecycle above depends on — a slot's successor cannot start while the predecessor's finalizers are still running.
|
|
1389
|
+
|
|
1390
|
+
[Cancellation](../execution-and-compatibility/async-execution.md#cancellation) covers the protocol these readers participate in, and [Async.Running#cancel](../../async.md#runningcancel) documents the primitive underneath it.
|
|
1391
|
+
|
|
1392
|
+
### Threading and Platform Behaviour
|
|
1393
|
+
|
|
1394
|
+
Which engine a concurrent operator materializes depends on the kind of reader its upstream compiled to, and the two engines differ far more than the two platforms do.
|
|
1395
|
+
|
|
1396
|
+
| Materialization | JVM reader | Scala.js reader | What actually overlaps |
|
|
1397
|
+
|---------------------------------|-----------------------------------------|---------------------------------|--------------------------------------------|
|
|
1398
|
+
| `mapPar`, synchronous upstream | `ConcurrentMapParReader` family | `Reader.MappedInt` and siblings | one worker thread per slot |
|
|
1399
|
+
| `mapPar`, asynchronous upstream | `AsyncConcurrentReaders.mapPar` | `AsyncConcurrentReaders.mapPar` | nothing; `f` is wrapped in `Async.succeed` |
|
|
1400
|
+
| `mapParAsync`, always | `AsyncConcurrentReaders.mapPar` | `AsyncConcurrentReaders.mapPar` | whatever the `Async` callbacks suspend on |
|
|
1401
|
+
| `mergeAll` from a `Reader` | `AsyncConcurrentReaders.merge` | `AsyncConcurrentReaders.merge` | whatever the inner streams suspend on |
|
|
1402
|
+
| `mergeAll` via the interpreter | `IntConcurrentMergeReader` and siblings | `Reader.FlatMappedRef` | one drainer thread per slot on the JVM |
|
|
1403
|
+
|
|
1404
|
+
Read the second row before choosing an operator. Once the upstream compiles to an `AsyncReader`, `Platform.createMapParReaderFromReader` routes `mapPar` to the shared asynchronous engine with the mapping function wrapped as `Async.succeed(f(a))` — an already-complete effect. The slots and the selector are all still there, but there is nothing for them to overlap, on either platform. `mapParAsync` exists precisely because a callback that returns a real `Async` is the only way to get concurrency out of that lane.
|
|
1405
|
+
|
|
1406
|
+
#### The JVM
|
|
1407
|
+
|
|
1408
|
+
Workers on the synchronous lane run on virtual threads where the runtime provides them. `Platform.startVirtualThread` obtains `Thread.ofVirtual()` reflectively, so the module builds and runs on any supported JDK and uses virtual threads on JDK 21 and later; if the reflective lookup fails for any reason it starts a named daemon platform thread instead.
|
|
1409
|
+
|
|
1410
|
+
The threads are named, which makes them identifiable in a thread dump. `mapPar` names its workers `zio-blocks-mappar-worker-<n>-<index>` and its dispatcher `zio-blocks-mappar-coordinator-<n>`, where `<n>` counts reader instances and `<index>` identifies the worker within one reader; merge uses `zio-blocks-merge-drainer-<n>-<index>` and `zio-blocks-merge-coordinator-<n>` on the same scheme. Those strings are prefixes rather than final names: the virtual-thread builder is created with `Thread.ofVirtual().name(prefix, 0L)`, whose two-argument form appends a counter, so a virtual worker appears in a dump as `zio-blocks-mappar-worker-<n>-<index>0`. Only the platform-thread fallback, which calls `setName` directly, uses the name verbatim.
|
|
1411
|
+
|
|
1412
|
+
#### Scala.js
|
|
1413
|
+
|
|
1414
|
+
There is no parallelism on Scala.js either way — `Platform.supportsConcurrency` is `false` and `Platform.startVirtualThread` throws `UnsupportedOperationException`. But "no parallelism" resolves into two different mechanisms, and only one of them is sequential in the sense the scaladoc suggests.
|
|
1415
|
+
|
|
1416
|
+
When the pipeline compiles end to end to a `SyncReader`, the operator really is sequential: `Platform.createMapParReaderFromReader` builds an ordinary mapped reader (`Reader.MappedIntInt`, `Reader.MappedInt`, `Reader.MappedLong`, and the rest), and the synchronous merge path builds a `Reader.FlatMappedRef` — one that throws `UnsupportedOperationException` if an inner stream turns out to be asynchronous.
|
|
1417
|
+
|
|
1418
|
+
When the pipeline is on the asynchronous lane, the concurrent engine runs, with immediately-ready effects standing in for suspension. The effect is still sequential, but the *semantics* are the concurrent engine's: **unordered arrival is retained**.
|
|
1419
|
+
|
|
1420
|
+
:::warning[The scaladoc is a throughput claim, not an ordering claim]
|
|
1421
|
+
The scaladoc on `Stream#mapPar`, `Stream#flatMapPar`, and `Stream.mergeAll` says that on Scala.js each "degrades to sequential `map`" or "sequential `flatMap`". That describes the synchronous-lane case as though it were the whole story. Never read it as a promise about element order.
|
|
1422
|
+
:::
|
|
1423
|
+
|
|
1424
|
+
[Platform Differences](../execution-and-compatibility/platform-differences.md#concurrency-and-threading) states the same split from the platform side, including the full capability surface of `Platform`.
|
|
1425
|
+
|
|
1426
|
+
### Laws
|
|
1427
|
+
|
|
1428
|
+
`ConcurrentLawsSpec` asserts three equalities, all of them as set equality (`.toSet == .toSet`), since neither side of any of them is ordered:
|
|
1429
|
+
|
|
1430
|
+
- `Stream.mergeAll(1)(streams)` equals `streams.flatMap(identity)`
|
|
1431
|
+
- `stream.mapPar(1)(f)` equals `stream.map(f)`
|
|
1432
|
+
- `stream.flatMapPar(n)(f)` equals `Stream.mergeAll(n)(stream.map(f))`
|
|
1433
|
+
|
|
1434
|
+
The first two additionally hold as exact equality, and not because the engine happens to preserve order — at `n = 1` the operator body *returns the sequential operator itself*, so the two sides are the same description. The third holds only as a set equality: both sides run the concurrent engine at width `n`, so both are in arrival order and neither has an input order to compare against.
|
|
1435
|
+
|
|
1436
|
+
The third law is also a statement about the implementation rather than a coincidence, since `flatMapPar` is defined as that right-hand side. Reading it as "there is one fan-in engine" is more useful than reading it as a property that had to be checked.
|
|
1437
|
+
|
|
1438
|
+
### Bounded Concurrency Examples
|
|
1439
|
+
|
|
1440
|
+
#### `mapPar`, `mergeAll`, and `flatMapPar` on the JVM
|
|
1441
|
+
|
|
1442
|
+
The three snippets below use blocking terminals, which exist only on the JVM. In cross-platform code, swap the terminal for its `*Async` twin — `runCollectAsync`, `runFoldAsync` — and the operator itself is unchanged.
|
|
1443
|
+
|
|
1444
|
+
Expensive per-element work across eight slots:
|
|
1445
|
+
|
|
1446
|
+
```scala
|
|
1447
|
+
import zio.blocks.streams._
|
|
1448
|
+
|
|
1449
|
+
val doubled = Stream
|
|
1450
|
+
.range(0, 1000)
|
|
1451
|
+
.mapPar(8) { n =>
|
|
1452
|
+
Thread.sleep(1)
|
|
1453
|
+
n * 2
|
|
1454
|
+
}
|
|
1455
|
+
.runCollect
|
|
1456
|
+
```
|
|
1457
|
+
|
|
1458
|
+
`doubled` is a `Right` holding all thousand elements, in arrival order rather than `0, 2, 4, …`.
|
|
1459
|
+
|
|
1460
|
+
Ten sources drained four at a time:
|
|
1461
|
+
|
|
1462
|
+
```scala
|
|
1463
|
+
import zio.blocks.streams._
|
|
1464
|
+
|
|
1465
|
+
val sources = Stream.fromIterable((0 until 10).map(i => Stream.range(i * 100, (i + 1) * 100)))
|
|
1466
|
+
val summed = Stream.mergeAll(4)(sources).runFold(0L)(_ + _)
|
|
1467
|
+
```
|
|
1468
|
+
|
|
1469
|
+
A sum is order-insensitive, which is what makes it a safe thing to compute over an unordered merge: `summed` is `Right(499500)` on every run.
|
|
1470
|
+
|
|
1471
|
+
One sub-stream per element, eight drained at a time:
|
|
1472
|
+
|
|
1473
|
+
```scala
|
|
1474
|
+
import zio.blocks.streams._
|
|
1475
|
+
|
|
1476
|
+
val flattened = Stream
|
|
1477
|
+
.range(0, 50)
|
|
1478
|
+
.flatMapPar(8)(i => Stream.range(i * 20, (i + 1) * 20))
|
|
1479
|
+
.runFold(0L)(_ + _)
|
|
1480
|
+
```
|
|
1481
|
+
|
|
1482
|
+
Again `Right(499500)`: the same 1000 integers, reached through 50 inner streams instead of 10.
|
|
1483
|
+
|
|
1484
|
+
#### A Worked `mapParAsync`
|
|
1485
|
+
|
|
1486
|
+
The two runnable files below live in the `streams-examples` module. This one makes arrival order visible rather than asserting it: every callback hands back an unresolved `Completer`, and a driver thread then settles the four of them in reverse. The collected chunk comes back reversed with respect to the input.
|
|
1487
|
+
|
|
1488
|
+
```scala title="streams-examples/src/main/scala/stream/MapParAsyncExample.scala"
|
|
1489
|
+
/*
|
|
1490
|
+
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
1491
|
+
*
|
|
1492
|
+
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
1493
|
+
* you may not use this file except in compliance with the License.
|
|
1494
|
+
*/
|
|
1495
|
+
package stream
|
|
1496
|
+
|
|
1497
|
+
import java.util.concurrent.{ConcurrentHashMap, CountDownLatch}
|
|
1498
|
+
|
|
1499
|
+
import zio.blocks.async._
|
|
1500
|
+
import zio.blocks.chunk.Chunk
|
|
1501
|
+
import zio.blocks.streams.Stream
|
|
1502
|
+
|
|
1503
|
+
/**
|
|
1504
|
+
* `mapParAsync` keeps at most `n` `Async` callbacks in flight and emits each
|
|
1505
|
+
* result the moment it completes. Output is therefore in arrival order, not
|
|
1506
|
+
* input order.
|
|
1507
|
+
*
|
|
1508
|
+
* This example makes that visible rather than asserting it. Each callback hands
|
|
1509
|
+
* back a `Completer` instead of a value, so nothing completes on its own; a
|
|
1510
|
+
* driver thread then settles the four callbacks in reverse order. The collected
|
|
1511
|
+
* chunk comes back reversed with respect to the input.
|
|
1512
|
+
*/
|
|
1513
|
+
object MapParAsyncExample {
|
|
1514
|
+
def main(args: Array[String]): Unit = {
|
|
1515
|
+
val inputs = Chunk(1, 2, 3, 4)
|
|
1516
|
+
|
|
1517
|
+
// Every callback registers its completer and reports for duty. Nothing
|
|
1518
|
+
// resolves until the driver thread below decides that it should.
|
|
1519
|
+
val pending = new ConcurrentHashMap[Int, Completer[Int]]
|
|
1520
|
+
val allInFlight = new CountDownLatch(inputs.length)
|
|
1521
|
+
|
|
1522
|
+
val arrivals: Async[Either[Nothing, Chunk[Int]]] =
|
|
1523
|
+
Stream
|
|
1524
|
+
.fromChunk(inputs)
|
|
1525
|
+
.mapParAsync(4) { value =>
|
|
1526
|
+
val completer = new Completer[Int]
|
|
1527
|
+
pending.put(value, completer)
|
|
1528
|
+
allInFlight.countDown()
|
|
1529
|
+
completer
|
|
1530
|
+
}
|
|
1531
|
+
.runCollectAsync
|
|
1532
|
+
|
|
1533
|
+
// n = 4 and there are four elements, so all four callbacks are admitted
|
|
1534
|
+
// before any of them completes. The driver settles 4 first and 1 last.
|
|
1535
|
+
val driver = new Thread(() => {
|
|
1536
|
+
allInFlight.await()
|
|
1537
|
+
List(4, 3, 2, 1).foreach { value =>
|
|
1538
|
+
pending.get(value).succeed(value * 10)
|
|
1539
|
+
Thread.sleep(100L)
|
|
1540
|
+
}
|
|
1541
|
+
})
|
|
1542
|
+
driver.setDaemon(true)
|
|
1543
|
+
driver.start()
|
|
1544
|
+
|
|
1545
|
+
// `.block` is JVM-only; on Scala.js drive the Async with map/flatMap.
|
|
1546
|
+
val collected = arrivals.block
|
|
1547
|
+
|
|
1548
|
+
println(s"input order: ${inputs.map(_ * 10)}")
|
|
1549
|
+
println(s"arrival order: $collected")
|
|
1550
|
+
|
|
1551
|
+
// The elements are all there...
|
|
1552
|
+
require(
|
|
1553
|
+
collected.map(_.toSet) == Right(Set(10, 20, 30, 40)),
|
|
1554
|
+
s"unexpected elements: $collected"
|
|
1555
|
+
)
|
|
1556
|
+
// ...but not in the order they went in.
|
|
1557
|
+
require(
|
|
1558
|
+
collected != Right(inputs.map(_ * 10)),
|
|
1559
|
+
"mapParAsync emitted input order; arrival order was expected"
|
|
1560
|
+
)
|
|
1561
|
+
}
|
|
1562
|
+
}
|
|
1563
|
+
```
|
|
1564
|
+
|
|
1565
|
+
Run it with:
|
|
1566
|
+
|
|
1567
|
+
```bash
|
|
1568
|
+
sbt "streams-examples/runMain stream.MapParAsyncExample"
|
|
1569
|
+
```
|
|
1570
|
+
|
|
1571
|
+
It prints the two orders side by side:
|
|
1572
|
+
|
|
1573
|
+
```
|
|
1574
|
+
input order: Chunk(10,20,30,40)
|
|
1575
|
+
arrival order: Right(Chunk(40,30,20,10))
|
|
1576
|
+
```
|
|
1577
|
+
|
|
1578
|
+
#### Async Children and Slot Accounting
|
|
1579
|
+
|
|
1580
|
+
`flatMapPar(n)(a => Stream.unwrap(f(a)))` is the idiom for asynchronously produced children, and this example pins down what a slot covers. A gauge is incremented when child construction starts and decremented by the child's finalizer, so it counts exactly the elements occupying a slot; a `CyclicBarrier(2)` forces each construction to wait for a partner, so the program can only finish if two really are in flight at once.
|
|
1581
|
+
|
|
1582
|
+
```scala title="streams-examples/src/main/scala/stream/FlatMapParAsyncChildrenExample.scala"
|
|
1583
|
+
/*
|
|
1584
|
+
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
1585
|
+
*
|
|
1586
|
+
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
1587
|
+
* you may not use this file except in compliance with the License.
|
|
1588
|
+
*/
|
|
1589
|
+
package stream
|
|
1590
|
+
|
|
1591
|
+
import java.util.concurrent.CyclicBarrier
|
|
1592
|
+
import java.util.concurrent.atomic.AtomicInteger
|
|
1593
|
+
|
|
1594
|
+
import zio.blocks.async._
|
|
1595
|
+
import zio.blocks.streams.Stream
|
|
1596
|
+
|
|
1597
|
+
/**
|
|
1598
|
+
* `flatMapPar(n)(a => Stream.unwrap(f(a)))` is the idiom for asynchronously
|
|
1599
|
+
* produced children. The point of this example is the slot accounting: the
|
|
1600
|
+
* effect that *builds* the child and the child that is then *drained* share one
|
|
1601
|
+
* of the `n` slots, so a slot is occupied from the moment construction starts
|
|
1602
|
+
* until the child has finished closing.
|
|
1603
|
+
*
|
|
1604
|
+
* `inFlight` is incremented when construction starts and decremented by the
|
|
1605
|
+
* child's finalizer, so it counts exactly the elements occupying a slot. With
|
|
1606
|
+
* `n = 2` its peak is 2 and never 4, even though four outer elements are
|
|
1607
|
+
* available immediately: the engine re-arms the outer source only while
|
|
1608
|
+
* `active < n`, so it will not pull ahead to start a third construction.
|
|
1609
|
+
*
|
|
1610
|
+
* The `CyclicBarrier(2)` proves the lower bound from the other side. Each
|
|
1611
|
+
* construction waits for a partner, so the run can only finish if two
|
|
1612
|
+
* constructions really are in flight at once — and it pairs off exactly twice.
|
|
1613
|
+
*/
|
|
1614
|
+
object FlatMapParAsyncChildrenExample {
|
|
1615
|
+
private val inFlight = new AtomicInteger
|
|
1616
|
+
private val peak = new AtomicInteger
|
|
1617
|
+
private val pairUp = new CyclicBarrier(2)
|
|
1618
|
+
|
|
1619
|
+
/**
|
|
1620
|
+
* Produces a value on a thread of its own. Blocking inside an `Async` would
|
|
1621
|
+
* stall the driver that is running the stream, so the barrier wait has to
|
|
1622
|
+
* happen somewhere else.
|
|
1623
|
+
*/
|
|
1624
|
+
private def deferred[A](value: => A): Async[A] = {
|
|
1625
|
+
val completer = new Completer[A]
|
|
1626
|
+
val thread = new Thread(() => completer.succeed(value))
|
|
1627
|
+
thread.setDaemon(true)
|
|
1628
|
+
thread.start()
|
|
1629
|
+
completer
|
|
1630
|
+
}
|
|
1631
|
+
|
|
1632
|
+
/** The asynchronously produced child for one outer element. */
|
|
1633
|
+
private def child(value: Int): Async[Stream[Nothing, Int]] =
|
|
1634
|
+
deferred {
|
|
1635
|
+
val current = inFlight.incrementAndGet()
|
|
1636
|
+
peak.getAndUpdate(seen => if (current > seen) current else seen)
|
|
1637
|
+
pairUp.await()
|
|
1638
|
+
Stream(value * 10, value * 10 + 1).ensuring {
|
|
1639
|
+
inFlight.decrementAndGet()
|
|
1640
|
+
()
|
|
1641
|
+
}
|
|
1642
|
+
}
|
|
1643
|
+
|
|
1644
|
+
def main(args: Array[String]): Unit = {
|
|
1645
|
+
// `.block` is JVM-only; on Scala.js drive the Async with map/flatMap.
|
|
1646
|
+
val collected = Stream(1, 2, 3, 4)
|
|
1647
|
+
.flatMapPar(2)(value => Stream.unwrap(child(value)))
|
|
1648
|
+
.runCollectAsync
|
|
1649
|
+
.block
|
|
1650
|
+
|
|
1651
|
+
println(s"elements: ${collected.map(_.toSet.toList.sorted)}")
|
|
1652
|
+
println(s"peak slot occupancy: ${peak.get()} of 2")
|
|
1653
|
+
|
|
1654
|
+
require(
|
|
1655
|
+
collected.map(_.toSet) == Right(Set(10, 11, 20, 21, 30, 31, 40, 41)),
|
|
1656
|
+
s"unexpected elements: $collected"
|
|
1657
|
+
)
|
|
1658
|
+
// Two slots, so at most two elements are ever being constructed or drained.
|
|
1659
|
+
require(peak.get() == 2, s"peak occupancy was ${peak.get()}, expected 2")
|
|
1660
|
+
// Every slot was released, which means every child was closed.
|
|
1661
|
+
require(inFlight.get() == 0, s"${inFlight.get()} children were left open")
|
|
1662
|
+
}
|
|
1663
|
+
}
|
|
1664
|
+
```
|
|
1665
|
+
|
|
1666
|
+
Run it with:
|
|
1667
|
+
|
|
1668
|
+
```bash
|
|
1669
|
+
sbt "streams-examples/runMain stream.FlatMapParAsyncChildrenExample"
|
|
1670
|
+
```
|
|
1671
|
+
|
|
1672
|
+
```
|
|
1673
|
+
elements: Right(List(10, 11, 20, 21, 30, 31, 40, 41))
|
|
1674
|
+
peak slot occupancy: 2 of 2
|
|
1675
|
+
```
|
|
1676
|
+
|
|
1677
|
+
Peak occupancy is 2 and not 4, even though all four outer elements are available immediately, because the construction effect holds the slot its child will use. For sequential asynchronous children, `flatMap(a => Stream.unwrap(f(a)))` is the operator you want instead.
|
|
1678
|
+
|
|
1679
|
+
### Comparison with Other Libraries
|
|
1680
|
+
|
|
1681
|
+
The concurrent surface is small, and what distinguishes it is less the operator names than what a caller has to bring along to use them.
|
|
1682
|
+
|
|
1683
|
+
| Feature | ZIO Blocks Streams | fs2 | Kyo | Ox | Pekko |
|
|
1684
|
+
|----------------------|------------------------------------|--------------------|-------------------|-----------------------|----------------------------|
|
|
1685
|
+
| Concurrent operators | `mapPar`, `mergeAll`, `flatMapPar` | `parEvalMap` | `mapParUnordered` | `mapPar` | `mapAsync`, `flatMapMerge` |
|
|
1686
|
+
| Effect system | none required | cats-effect | Kyo | none; virtual threads | Akka |
|
|
1687
|
+
| Typed errors | `Either[E, Z]` | `ApplicativeError` | Kyo effects | exceptions | none |
|
|
1688
|
+
|
|
1689
|
+
The table compares contracts, not speed. Cross-provider throughput rankings require the specific benchmark classes that were built to compare like with like, and none of the numbers from those runs belong in a row next to a feature name.
|
|
1690
|
+
|
|
1072
1691
|
## Other Operations
|
|
1073
1692
|
|
|
1074
1693
|
Common utilities for deduplication, draining, and error recovery:
|
|
@@ -1082,8 +1701,8 @@ These operations remove duplicate elements, useful for deduplicating streams bef
|
|
|
1082
1701
|
Emits only unique elements (using a mutable `HashSet` internally):
|
|
1083
1702
|
|
|
1084
1703
|
```scala
|
|
1085
|
-
|
|
1086
|
-
def distinct
|
|
1704
|
+
abstract class Stream[+E, +A] {
|
|
1705
|
+
def distinct: Stream[E, A]
|
|
1087
1706
|
}
|
|
1088
1707
|
```
|
|
1089
1708
|
|
|
@@ -1095,7 +1714,7 @@ import zio.blocks.streams.*
|
|
|
1095
1714
|
val nums = Stream(1, 2, 2, 3, 3, 3)
|
|
1096
1715
|
// nums: Stream[Nothing, Int] = Stream(1, 2, 2, 3, 3, ...)
|
|
1097
1716
|
val unique = nums.distinct
|
|
1098
|
-
// unique: Stream[Nothing, Int] = Stream(
|
|
1717
|
+
// unique: Stream[Nothing, Int] = Stream.suspend(...)
|
|
1099
1718
|
val result = unique.runCollect
|
|
1100
1719
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3))
|
|
1101
1720
|
```
|
|
@@ -1105,8 +1724,8 @@ val result = unique.runCollect
|
|
|
1105
1724
|
Emits only elements whose key (computed by `f`) has not been seen before:
|
|
1106
1725
|
|
|
1107
1726
|
```scala
|
|
1108
|
-
|
|
1109
|
-
def distinctBy[K](f: A => K)
|
|
1727
|
+
abstract class Stream[+E, +A] {
|
|
1728
|
+
def distinctBy[K](f: A => K): Stream[E, A]
|
|
1110
1729
|
}
|
|
1111
1730
|
```
|
|
1112
1731
|
|
|
@@ -1137,7 +1756,7 @@ These operations skip or limit elements, allowing you to keep or drop unwanted p
|
|
|
1137
1756
|
Skips the first `n` elements:
|
|
1138
1757
|
|
|
1139
1758
|
```scala
|
|
1140
|
-
|
|
1759
|
+
abstract class Stream[+E, +A] {
|
|
1141
1760
|
def drop(n: Long): Stream[E, A]
|
|
1142
1761
|
}
|
|
1143
1762
|
```
|
|
@@ -1157,7 +1776,7 @@ val result = remaining.runCollect
|
|
|
1157
1776
|
Emits at most the first `n` elements, then stops:
|
|
1158
1777
|
|
|
1159
1778
|
```scala
|
|
1160
|
-
|
|
1779
|
+
abstract class Stream[+E, +A] {
|
|
1161
1780
|
def take(n: Long): Stream[E, A]
|
|
1162
1781
|
}
|
|
1163
1782
|
```
|
|
@@ -1182,7 +1801,7 @@ val result = first10.runCollect
|
|
|
1182
1801
|
Emits elements while a predicate is true, then stops:
|
|
1183
1802
|
|
|
1184
1803
|
```scala
|
|
1185
|
-
|
|
1804
|
+
abstract class Stream[+E, +A] {
|
|
1186
1805
|
def takeWhile(pred: A => Boolean): Stream[E, A]
|
|
1187
1806
|
}
|
|
1188
1807
|
```
|
|
@@ -1200,13 +1819,38 @@ val result = firstFive.runCollect
|
|
|
1200
1819
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(1, 2, 3, 4, 5))
|
|
1201
1820
|
```
|
|
1202
1821
|
|
|
1822
|
+
#### `Stream#takeWhileAsync`
|
|
1823
|
+
|
|
1824
|
+
The asynchronous twin of `takeWhile`, for a predicate that has to await something before it can answer:
|
|
1825
|
+
|
|
1826
|
+
```scala
|
|
1827
|
+
abstract class Stream[+E, +A] {
|
|
1828
|
+
def takeWhileAsync(pred: A => Async[Boolean]): Stream[E, A]
|
|
1829
|
+
}
|
|
1830
|
+
```
|
|
1831
|
+
|
|
1832
|
+
Elements are tested sequentially, and the first `false` closes upstream — so the element that failed the test is not emitted, and nothing beyond it is pulled. Predicate failure is a defect. No `JvmType.Infer` evidence appears here because the element type does not change.
|
|
1833
|
+
|
|
1834
|
+
```scala
|
|
1835
|
+
import zio.blocks.async.*
|
|
1836
|
+
import zio.blocks.streams.*
|
|
1837
|
+
|
|
1838
|
+
val feed = Stream(120, -40, 75, 0, 999)
|
|
1839
|
+
val untilZero = feed.takeWhileAsync(amount => Async.succeed(amount != 0))
|
|
1840
|
+
```
|
|
1841
|
+
|
|
1842
|
+
The same entry appears in [Async Operators](../execution-and-compatibility/async-execution.md#streamtakewhileasync).
|
|
1843
|
+
|
|
1203
1844
|
### Interspersing
|
|
1204
1845
|
|
|
1205
|
-
`intersperse[
|
|
1846
|
+
`intersperse[A2, A3]` — Inserts a separator value between every two elements.:
|
|
1206
1847
|
|
|
1207
1848
|
```scala
|
|
1208
|
-
|
|
1209
|
-
def intersperse[
|
|
1849
|
+
abstract class Stream[+E, +A] {
|
|
1850
|
+
def intersperse[A2, A3](sep: A2)(implicit
|
|
1851
|
+
valueConcat: Concat.WithOut[A, A2, A3],
|
|
1852
|
+
jtA3: JvmType.Infer[A3]
|
|
1853
|
+
): Stream[E, A3]
|
|
1210
1854
|
}
|
|
1211
1855
|
```
|
|
1212
1856
|
|
|
@@ -1227,10 +1871,10 @@ val result = separated.runCollect
|
|
|
1227
1871
|
|
|
1228
1872
|
### Repeating
|
|
1229
1873
|
|
|
1230
|
-
`repeated` —
|
|
1874
|
+
`repeated` — Rematerializes the stream after each clean completion, emitting the whole sequence again indefinitely. A typed error or a defect terminates the repetition.:
|
|
1231
1875
|
|
|
1232
1876
|
```scala
|
|
1233
|
-
|
|
1877
|
+
abstract class Stream[+E, +A] {
|
|
1234
1878
|
def repeated: Stream[E, A]
|
|
1235
1879
|
}
|
|
1236
1880
|
```
|
|
@@ -1253,8 +1897,8 @@ val result = repeated.runCollect
|
|
|
1253
1897
|
`tapEach` — Applies a function to each element for side effects, passing the element through unchanged.:
|
|
1254
1898
|
|
|
1255
1899
|
```scala
|
|
1256
|
-
|
|
1257
|
-
def tapEach(f: A => Unit)
|
|
1900
|
+
abstract class Stream[+E, +A] {
|
|
1901
|
+
def tapEach(f: A => Unit): Stream[E, A]
|
|
1258
1902
|
}
|
|
1259
1903
|
```
|
|
1260
1904
|
|
|
@@ -1266,7 +1910,7 @@ import zio.blocks.streams.*
|
|
|
1266
1910
|
val nums = Stream(1, 2, 3)
|
|
1267
1911
|
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
|
|
1268
1912
|
val logged = nums.tapEach(x => println(s"Element: $x"))
|
|
1269
|
-
// logged: Stream[Nothing, Int] = Stream(1, 2, 3).
|
|
1913
|
+
// logged: Stream[Nothing, Int] = Stream(1, 2, 3).filter(...)
|
|
1270
1914
|
val result = logged.runCollect
|
|
1271
1915
|
// Element: 1
|
|
1272
1916
|
// Element: 2
|
|
@@ -1278,7 +1922,7 @@ val result = logged.runCollect
|
|
|
1278
1922
|
|
|
1279
1923
|
Streams distinguish between recoverable business errors and unexpected exceptions, providing separate recovery mechanisms for each:
|
|
1280
1924
|
|
|
1281
|
-
### Typed Error
|
|
1925
|
+
### Typed Error Vs Untyped Defect
|
|
1282
1926
|
|
|
1283
1927
|
ZIO Blocks distinguishes two error channels:
|
|
1284
1928
|
|
|
@@ -1294,17 +1938,20 @@ This separation allows you to:
|
|
|
1294
1938
|
|
|
1295
1939
|
Streams distinguish between recoverable domain errors and fatal defects, with flexible recovery patterns:
|
|
1296
1940
|
|
|
1297
|
-
### Recovering
|
|
1941
|
+
### Recovering From Typed Errors
|
|
1298
1942
|
|
|
1299
1943
|
These operations handle typed errors gracefully by recovering with alternative streams:
|
|
1300
1944
|
|
|
1301
|
-
#### `Stream#catchAll[E2,
|
|
1945
|
+
#### `Stream#catchAll[E2, A2, A3]`
|
|
1302
1946
|
|
|
1303
1947
|
Recovers from any typed error by switching to a recovery stream:
|
|
1304
1948
|
|
|
1305
1949
|
```scala
|
|
1306
|
-
|
|
1307
|
-
def catchAll[E2,
|
|
1950
|
+
abstract class Stream[+E, +A] {
|
|
1951
|
+
def catchAll[E2, A2, A3](f: E => Stream[E2, A2])(implicit
|
|
1952
|
+
valueConcat: Concat.WithOut[A, A2, A3],
|
|
1953
|
+
jtA3: JvmType.Infer[A3]
|
|
1954
|
+
): Stream[E2, A3]
|
|
1308
1955
|
}
|
|
1309
1956
|
```
|
|
1310
1957
|
|
|
@@ -1324,13 +1971,16 @@ val result = recovered.runCollect
|
|
|
1324
1971
|
// result: Either[Nothing, Chunk[String]] = Right(IndexedSeq("default"))
|
|
1325
1972
|
```
|
|
1326
1973
|
|
|
1327
|
-
#### `Stream#orElse[E2,
|
|
1974
|
+
#### `Stream#orElse[E2, A2, A3]`
|
|
1328
1975
|
|
|
1329
1976
|
If this stream fails, tries the fallback stream. The fallback is evaluated lazily, only on error:
|
|
1330
1977
|
|
|
1331
1978
|
```scala
|
|
1332
|
-
|
|
1333
|
-
def orElse[E2,
|
|
1979
|
+
abstract class Stream[+E, +A] {
|
|
1980
|
+
def orElse[E2, A2, A3](that: => Stream[E2, A2])(implicit
|
|
1981
|
+
valueConcat: Concat.WithOut[A, A2, A3],
|
|
1982
|
+
jtA3: JvmType.Infer[A3]
|
|
1983
|
+
): Stream[E2, A3]
|
|
1334
1984
|
}
|
|
1335
1985
|
```
|
|
1336
1986
|
|
|
@@ -1347,15 +1997,19 @@ val result = (primary || fallback).runCollect
|
|
|
1347
1997
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(42))
|
|
1348
1998
|
```
|
|
1349
1999
|
|
|
1350
|
-
### Recovering
|
|
2000
|
+
### Recovering From Defects
|
|
1351
2001
|
|
|
1352
|
-
`catchDefect[
|
|
2002
|
+
`catchDefect[E2, E3, A2, A3]` — Catches untyped defects (exceptions not wrapped as typed errors) using a partial function.:
|
|
1353
2003
|
|
|
1354
2004
|
```scala
|
|
1355
|
-
|
|
1356
|
-
def catchDefect[
|
|
1357
|
-
f: PartialFunction[Throwable, Stream[
|
|
1358
|
-
)
|
|
2005
|
+
abstract class Stream[+E, +A] {
|
|
2006
|
+
def catchDefect[E2, E3, A2, A3](
|
|
2007
|
+
f: PartialFunction[Throwable, Stream[E2, A2]]
|
|
2008
|
+
)(implicit
|
|
2009
|
+
errorConcat: Concat.WithOut[E, E2, E3],
|
|
2010
|
+
valueConcat: Concat.WithOut[A, A2, A3],
|
|
2011
|
+
jtA3: JvmType.Infer[A3]
|
|
2012
|
+
): Stream[E3, A3]
|
|
1359
2013
|
}
|
|
1360
2014
|
```
|
|
1361
2015
|
|
|
@@ -1475,7 +2129,7 @@ val result = stream.runCollect
|
|
|
1475
2129
|
Adds a **cleanup action to any stream**, regardless of how it is created. The finalizer runs in a `finally` block:
|
|
1476
2130
|
|
|
1477
2131
|
```scala
|
|
1478
|
-
|
|
2132
|
+
abstract class Stream[+E, +A] {
|
|
1479
2133
|
def ensuring(finalizer: => Unit): Stream[E, A]
|
|
1480
2134
|
}
|
|
1481
2135
|
```
|
|
@@ -1504,9 +2158,39 @@ val managed = Stream(1, 2, 3)
|
|
|
1504
2158
|
val result = managed.runCollect
|
|
1505
2159
|
```
|
|
1506
2160
|
|
|
2161
|
+
### `Stream#ensuringAsync`
|
|
2162
|
+
|
|
2163
|
+
The asynchronous twin of `ensuring`, for cleanup that is itself an `Async` — closing a socket, flushing a remote session, releasing a lease:
|
|
2164
|
+
|
|
2165
|
+
```scala
|
|
2166
|
+
abstract class Stream[+E, +A] {
|
|
2167
|
+
def ensuringAsync(finalizer: => Async[Unit]): Stream[E, A]
|
|
2168
|
+
}
|
|
2169
|
+
```
|
|
2170
|
+
|
|
2171
|
+
The finalizer is registered lazily and awaited exactly once when the materialized stream closes: on normal completion, on failure, on early termination such as `take` or `takeWhileAsync` cutting the stream short, and on cancellation. Awaited, not merely started — the close does not complete until the finalizer does. A failure inside the finalizer is a defect, and when the stream had already failed, that defect is attached to the primary failure rather than replacing it.
|
|
2172
|
+
|
|
2173
|
+
```scala
|
|
2174
|
+
import zio.blocks.async.*
|
|
2175
|
+
import zio.blocks.streams.*
|
|
2176
|
+
|
|
2177
|
+
val session = Stream(1, 2, 3)
|
|
2178
|
+
.ensuringAsync(Async.succeed(println("session closed")))
|
|
2179
|
+
|
|
2180
|
+
val closed = session.runCollectAsync
|
|
2181
|
+
```
|
|
2182
|
+
|
|
2183
|
+
What drives that close is the terminal. Under `runCollectAsync` and every other terminal the library closes the reader for you, and so does `useReaderAsync`; under `startAsync` you own the reader, and the finalizer has not run until you await `close()`. [Resource Management](../execution-and-compatibility/async-execution.md#resource-management) works through the three cases.
|
|
2184
|
+
|
|
1507
2185
|
## Running Streams
|
|
1508
2186
|
|
|
1509
|
-
|
|
2187
|
+
There are two terminal families, and which one you reach for is a platform decision rather than a stylistic one.
|
|
2188
|
+
|
|
2189
|
+
The **cross-platform** family is the one whose names end in `Async`: `runAsync`, `runCollectAsync`, `runDrainAsync`, `runFoldAsync`, `runForeachAsync`/`foreachAsync`, `countAsync`, `existsAsync`, `findAsync`, `forallAsync`, `headAsync`, and `lastAsync`. Each returns `Async[Either[E, Z]]`, stays lazy until driven, and awaits reader cleanup on success, failure, or cancellation. It drives a synchronous and an asynchronous pipeline alike, and it is the only family that compiles for both targets. [Async Terminals](../execution-and-compatibility/async-execution.md#async-terminals) documents the family and the `Async[Either[E, Z]]` convention it follows.
|
|
2190
|
+
|
|
2191
|
+
The **JVM-only** family is everything documented in the rest of this section — `run`, `runCollect`, `runDrain`, `runFold`, `runForeach`/`foreach`, `count`, `exists`, `find`, `forall`, `head`, and `last` — together with `start`, covered under [Manual Pull via `start`](#manual-pull-via-start). Those sixteen members, counting `runFold`'s four overloads, are the entire JVM-only surface of `Stream`: the terminals among them park a thread and return a plain `Either[E, Z]`, and `start` hands back a blocking `Reader.SyncReader[A]`. They live in `StreamPlatformSpecific`, whose Scala.js copy has an empty body, so calling one from shared code fails to compile for the JavaScript target rather than failing at runtime. The [availability matrix](../execution-and-compatibility/platform-differences.md#availability-matrix) lists them member by member.
|
|
2192
|
+
|
|
2193
|
+
Each heading below therefore carries a one-line note naming its cross-platform form.
|
|
1510
2194
|
|
|
1511
2195
|
### Collecting Results
|
|
1512
2196
|
|
|
@@ -1514,10 +2198,12 @@ These operations accumulate or examine stream results, running the entire stream
|
|
|
1514
2198
|
|
|
1515
2199
|
#### `Stream#runCollect`
|
|
1516
2200
|
|
|
2201
|
+
JVM only. The cross-platform form is `runCollectAsync`.
|
|
2202
|
+
|
|
1517
2203
|
Collects all elements into a `Chunk[A]`:
|
|
1518
2204
|
|
|
1519
2205
|
```scala
|
|
1520
|
-
|
|
2206
|
+
abstract class Stream[+E, +A] {
|
|
1521
2207
|
def runCollect: Either[E, Chunk[A]]
|
|
1522
2208
|
}
|
|
1523
2209
|
```
|
|
@@ -1536,11 +2222,15 @@ val result = nums.runCollect
|
|
|
1536
2222
|
|
|
1537
2223
|
#### `Stream#run[E2, Z]`
|
|
1538
2224
|
|
|
2225
|
+
JVM only. The cross-platform form is `runAsync`.
|
|
2226
|
+
|
|
1539
2227
|
Runs the stream with a custom sink, producing result `Z`:
|
|
1540
2228
|
|
|
1541
2229
|
```scala
|
|
1542
|
-
|
|
1543
|
-
def run[
|
|
2230
|
+
abstract class Stream[+E, +A] {
|
|
2231
|
+
def run[ES, E3, Z](sink: Sink[ES, A, Z])(implicit
|
|
2232
|
+
errorConcat: Concat.WithOut[E, ES, E3]
|
|
2233
|
+
): Either[E3, Z]
|
|
1544
2234
|
}
|
|
1545
2235
|
```
|
|
1546
2236
|
|
|
@@ -1562,10 +2252,12 @@ These operations consume streams without collecting their elements, useful when
|
|
|
1562
2252
|
|
|
1563
2253
|
#### `Stream#runDrain`
|
|
1564
2254
|
|
|
2255
|
+
JVM only. The cross-platform form is `runDrainAsync`.
|
|
2256
|
+
|
|
1565
2257
|
Consumes all elements and discards them, returning `Unit`:
|
|
1566
2258
|
|
|
1567
2259
|
```scala
|
|
1568
|
-
|
|
2260
|
+
abstract class Stream[+E, +A] {
|
|
1569
2261
|
def runDrain: Either[E, Unit]
|
|
1570
2262
|
}
|
|
1571
2263
|
```
|
|
@@ -1578,7 +2270,7 @@ import zio.blocks.streams.*
|
|
|
1578
2270
|
val nums = Stream(1, 2, 3)
|
|
1579
2271
|
// nums: Stream[Nothing, Int] = Stream(1, 2, 3)
|
|
1580
2272
|
val sideEffect = nums.tapEach(x => println(s"Processing $x"))
|
|
1581
|
-
// sideEffect: Stream[Nothing, Int] = Stream(1, 2, 3).
|
|
2273
|
+
// sideEffect: Stream[Nothing, Int] = Stream(1, 2, 3).filter(...)
|
|
1582
2274
|
val result = sideEffect.runDrain
|
|
1583
2275
|
// Processing 1
|
|
1584
2276
|
// Processing 2
|
|
@@ -1588,10 +2280,12 @@ val result = sideEffect.runDrain
|
|
|
1588
2280
|
|
|
1589
2281
|
#### `Stream#runForeach`
|
|
1590
2282
|
|
|
2283
|
+
JVM only. The cross-platform forms are `runForeachAsync` and its alias `foreachAsync`.
|
|
2284
|
+
|
|
1591
2285
|
Applies a function to each element for side effects:
|
|
1592
2286
|
|
|
1593
2287
|
```scala
|
|
1594
|
-
|
|
2288
|
+
abstract class Stream[+E, +A] {
|
|
1595
2289
|
def runForeach(f: A => Unit): Either[E, Unit]
|
|
1596
2290
|
}
|
|
1597
2291
|
```
|
|
@@ -1611,11 +2305,13 @@ These operations reduce streams to single values, aggregating elements into resu
|
|
|
1611
2305
|
|
|
1612
2306
|
#### `Stream#runFold[Z]`
|
|
1613
2307
|
|
|
2308
|
+
JVM only. The cross-platform form is `runFoldAsync`.
|
|
2309
|
+
|
|
1614
2310
|
Folds all elements using an accumulator, returning the final result:
|
|
1615
2311
|
|
|
1616
2312
|
```scala
|
|
1617
|
-
|
|
1618
|
-
def runFold[Z](z: Z)(f: (Z, A) => Z): Either[E, Z]
|
|
2313
|
+
abstract class Stream[+E, +A] {
|
|
2314
|
+
def runFold[Z](z: Z)(f: (Z, A) => Z)(implicit jtZ: JvmType.Infer[Z]): Either[E, Z]
|
|
1619
2315
|
}
|
|
1620
2316
|
```
|
|
1621
2317
|
|
|
@@ -1640,10 +2336,12 @@ def runFold(z: Double)(f: (Double, A) => Double): Either[E, Double]
|
|
|
1640
2336
|
|
|
1641
2337
|
#### `Stream#count`
|
|
1642
2338
|
|
|
2339
|
+
JVM only. The cross-platform form is `countAsync`.
|
|
2340
|
+
|
|
1643
2341
|
Returns the number of elements:
|
|
1644
2342
|
|
|
1645
2343
|
```scala
|
|
1646
|
-
|
|
2344
|
+
abstract class Stream[+E, +A] {
|
|
1647
2345
|
def count: Either[E, Long]
|
|
1648
2346
|
}
|
|
1649
2347
|
```
|
|
@@ -1661,10 +2359,12 @@ val total = nums.count
|
|
|
1661
2359
|
|
|
1662
2360
|
#### `Stream#head`
|
|
1663
2361
|
|
|
2362
|
+
JVM only. The cross-platform form is `headAsync`.
|
|
2363
|
+
|
|
1664
2364
|
Returns the first element (or `None` if empty):
|
|
1665
2365
|
|
|
1666
2366
|
```scala
|
|
1667
|
-
|
|
2367
|
+
abstract class Stream[+E, +A] {
|
|
1668
2368
|
def head: Either[E, Option[A]]
|
|
1669
2369
|
}
|
|
1670
2370
|
```
|
|
@@ -1682,10 +2382,12 @@ val first = nums.head
|
|
|
1682
2382
|
|
|
1683
2383
|
#### `Stream#last`
|
|
1684
2384
|
|
|
2385
|
+
JVM only. The cross-platform form is `lastAsync`.
|
|
2386
|
+
|
|
1685
2387
|
Returns the last element (or `None` if empty):
|
|
1686
2388
|
|
|
1687
2389
|
```scala
|
|
1688
|
-
|
|
2390
|
+
abstract class Stream[+E, +A] {
|
|
1689
2391
|
def last: Either[E, Option[A]]
|
|
1690
2392
|
}
|
|
1691
2393
|
```
|
|
@@ -1703,10 +2405,12 @@ val last = nums.last
|
|
|
1703
2405
|
|
|
1704
2406
|
#### `Stream#find[A]`
|
|
1705
2407
|
|
|
2408
|
+
JVM only. The cross-platform form is `findAsync`.
|
|
2409
|
+
|
|
1706
2410
|
Returns the first element satisfying a predicate:
|
|
1707
2411
|
|
|
1708
2412
|
```scala
|
|
1709
|
-
|
|
2413
|
+
abstract class Stream[+E, +A] {
|
|
1710
2414
|
def find(pred: A => Boolean): Either[E, Option[A]]
|
|
1711
2415
|
}
|
|
1712
2416
|
```
|
|
@@ -1724,10 +2428,12 @@ val firstEven = nums.find(_ % 2 == 0)
|
|
|
1724
2428
|
|
|
1725
2429
|
#### `Stream#exists[A]`
|
|
1726
2430
|
|
|
2431
|
+
JVM only. The cross-platform form is `existsAsync`.
|
|
2432
|
+
|
|
1727
2433
|
Returns `true` if any element satisfies a predicate, short-circuiting:
|
|
1728
2434
|
|
|
1729
2435
|
```scala
|
|
1730
|
-
|
|
2436
|
+
abstract class Stream[+E, +A] {
|
|
1731
2437
|
def exists(pred: A => Boolean): Either[E, Boolean]
|
|
1732
2438
|
}
|
|
1733
2439
|
```
|
|
@@ -1745,10 +2451,12 @@ val hasLargeValue = nums.exists(_ > 35)
|
|
|
1745
2451
|
|
|
1746
2452
|
#### `Stream#forall[A]`
|
|
1747
2453
|
|
|
2454
|
+
JVM only. The cross-platform form is `forallAsync`.
|
|
2455
|
+
|
|
1748
2456
|
Returns `true` if all elements satisfy a predicate, short-circuiting:
|
|
1749
2457
|
|
|
1750
2458
|
```scala
|
|
1751
|
-
|
|
2459
|
+
abstract class Stream[+E, +A] {
|
|
1752
2460
|
def forall(pred: A => Boolean): Either[E, Boolean]
|
|
1753
2461
|
}
|
|
1754
2462
|
```
|
|
@@ -1773,7 +2481,7 @@ Streams compose with pipelines and sinks to form complete data processing flows:
|
|
|
1773
2481
|
`via[B]` — Applies a `Pipeline[A, B]` transformation to the stream.:
|
|
1774
2482
|
|
|
1775
2483
|
```scala
|
|
1776
|
-
|
|
2484
|
+
abstract class Stream[+E, +A] {
|
|
1777
2485
|
final def via[B](pipe: Pipeline[A, B]): Stream[E, B]
|
|
1778
2486
|
}
|
|
1779
2487
|
```
|
|
@@ -1785,8 +2493,8 @@ import zio.blocks.streams.*
|
|
|
1785
2493
|
|
|
1786
2494
|
val nums = Stream(1, 2, 3, 4, 5)
|
|
1787
2495
|
// nums: Stream[Nothing, Int] = Stream(1, 2, 3, 4, 5)
|
|
1788
|
-
val pipe = Pipeline.filter((x: Int) => x > 2).andThen(Pipeline.map(
|
|
1789
|
-
// pipe: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$Composed@
|
|
2496
|
+
val pipe = Pipeline.filter((x: Int) => x > 2).andThen(Pipeline.map((x: Int) => x * 10))
|
|
2497
|
+
// pipe: Pipeline[Int, Int] = zio.blocks.streams.Pipeline$Composed@7f645c3b
|
|
1790
2498
|
val result = nums.via(pipe).runCollect
|
|
1791
2499
|
// result: Either[Nothing, Chunk[Int]] = Right(IndexedSeq(30, 40, 50))
|
|
1792
2500
|
```
|
|
@@ -1812,7 +2520,7 @@ val result = positives.runCollect
|
|
|
1812
2520
|
A `Sink[+E, -A, +Z]` is a consumer of elements of type `A` that produces a result `Z` or fails with `E`. Sinks are contravariant in `A` (they can accept a supertype of what they expect). Common sinks include:
|
|
1813
2521
|
|
|
1814
2522
|
- `Sink.collectAll: Sink[Nothing, A, Chunk[A]]` — collects all elements
|
|
1815
|
-
- `Sink.drain: Sink[Nothing,
|
|
2523
|
+
- `Sink.drain: Sink[Nothing, Any, Unit]` — discards all elements
|
|
1816
2524
|
- `Sink.count: Sink[Nothing, Any, Long]` — counts elements
|
|
1817
2525
|
- `Sink.foldLeft: Sink[Nothing, A, Z]` — folds elements with an accumulator
|
|
1818
2526
|
- `Sink.head: Sink[Nothing, A, Option[A]]` — takes the first element
|
|
@@ -1822,37 +2530,26 @@ When you call `stream.run(sink)`, the stream is compiled to a `Reader` and the s
|
|
|
1822
2530
|
|
|
1823
2531
|
## Low-Level Pull with Reader
|
|
1824
2532
|
|
|
1825
|
-
`Reader[+Elem]` is the low-level, pull-based source that backs every stream at execution time.
|
|
2533
|
+
`Reader[+Elem]` is the low-level, pull-based source that backs every stream at execution time. Use cross-platform `startAsync` for a caller-owned `Reader.AsyncReader`, or `useReaderAsync` for bracketed access that awaits close on every outcome. The JVM additionally provides blocking `start` with a [`Scope`](../../resource-management/scope.md).
|
|
1826
2534
|
|
|
1827
2535
|
### Manual Pull via `start`
|
|
1828
2536
|
|
|
1829
|
-
`
|
|
2537
|
+
`startAsync` transfers ownership to its caller, which must await `close()`. Prefer `useReaderAsync` when ownership need not escape. On the JVM, `start` opens a blocking reader within a `Scope`, which closes it when the scope exits:
|
|
1830
2538
|
|
|
1831
2539
|
```scala
|
|
1832
|
-
|
|
1833
|
-
def
|
|
2540
|
+
abstract class Stream[+E, +A] {
|
|
2541
|
+
def startAsync: Async[Reader.AsyncReader[A]]
|
|
2542
|
+
def useReaderAsync[Z](f: Reader.AsyncReader[A] => Async[Z]): Async[Z]
|
|
2543
|
+
// JVM only
|
|
2544
|
+
def start(implicit scope: Scope): scope.$[Reader.SyncReader[A]]
|
|
1834
2545
|
}
|
|
1835
2546
|
```
|
|
1836
2547
|
|
|
2548
|
+
`start` is JVM only, and its result type is `Reader.SyncReader[A]` rather than an undifferentiated `Reader[A]`: a pipeline with asynchronous stages still works under it, because the JVM runtime bridges those boundaries by blocking. Shared code has no such member and must use `startAsync` or `useReaderAsync` instead, both of which hand back a `Reader.AsyncReader[A]`. [Manual Pull Across Platforms](../execution-and-compatibility/platform-differences.md#manual-pull-across-platforms) compares the three, and [`Stream#startAsync`](../execution-and-compatibility/async-execution.md#streamstartasync) covers the ownership rules that come with the asynchronous form.
|
|
2549
|
+
|
|
1837
2550
|
Use `start` to manually pull elements within a resource scope:
|
|
1838
2551
|
|
|
1839
2552
|
```scala title="streams-examples/src/main/scala/stream/ManualPullUsingStart.scala"
|
|
1840
|
-
/*
|
|
1841
|
-
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
1842
|
-
*
|
|
1843
|
-
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
1844
|
-
* you may not use this file except in compliance with the License.
|
|
1845
|
-
* You may obtain a copy of the License at
|
|
1846
|
-
*
|
|
1847
|
-
* http://www.apache.org/licenses/LICENSE-2.0
|
|
1848
|
-
*
|
|
1849
|
-
* Unless required by applicable law or agreed to in writing, software
|
|
1850
|
-
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
1851
|
-
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
1852
|
-
* See the License for the specific language governing permissions and
|
|
1853
|
-
* limitations under the License.
|
|
1854
|
-
*/
|
|
1855
|
-
|
|
1856
2553
|
package stream
|
|
1857
2554
|
|
|
1858
2555
|
import zio.blocks.streams.*
|
|
@@ -1864,7 +2561,7 @@ object ManualPullUsingStart extends App {
|
|
|
1864
2561
|
import scope.*
|
|
1865
2562
|
|
|
1866
2563
|
// Open a stream for manual pulling
|
|
1867
|
-
val reader: $[Reader[Int]] = Stream.range(1, 6).start(using scope)
|
|
2564
|
+
val reader: $[Reader.SyncReader[Int]] = Stream.range(1, 6).start(using scope)
|
|
1868
2565
|
|
|
1869
2566
|
$(reader) { r =>
|
|
1870
2567
|
// Iterate through reader values using the protocol directly
|
|
@@ -1880,7 +2577,7 @@ object ManualPullUsingStart extends App {
|
|
|
1880
2577
|
}
|
|
1881
2578
|
```
|
|
1882
2579
|
|
|
1883
|
-
Use
|
|
2580
|
+
Use these methods when you need element-by-element control rather than running through a Sink. Do not overlap asynchronous pulls: an async reader allows one active pull, and its lifecycle operations must be awaited.
|
|
1884
2581
|
|
|
1885
2582
|
### The Reader Protocol
|
|
1886
2583
|
|
|
@@ -1892,11 +2589,17 @@ The pull protocol uses a **sentinel value** to signal end-of-stream:
|
|
|
1892
2589
|
|
|
1893
2590
|
For primitive types, specialized methods avoid boxing:
|
|
1894
2591
|
|
|
2592
|
+
- `readBoolean(sentinel: Int): Int`
|
|
2593
|
+
- `readByte(): Int`
|
|
2594
|
+
- `readChar(sentinel: Int): Int`
|
|
2595
|
+
- `readShort(sentinel: Int): Int`
|
|
1895
2596
|
- `readInt(sentinel: Long): Long`
|
|
1896
2597
|
- `readLong(sentinel: Long): Long`
|
|
1897
2598
|
- `readFloat(sentinel: Double): Double`
|
|
1898
2599
|
- `readDouble(sentinel: Double): Double`
|
|
1899
2600
|
|
|
2601
|
+
These are the eight exact physical methods selected by `Reader.jvmType`. The tag describes the reader's actual representation and method contract, so a known primitive lane survives type-preserving wrappers and static widening. Operations that produce a different element type select their output lane from result `JvmType.Infer` evidence.
|
|
2602
|
+
|
|
1900
2603
|
:::note
|
|
1901
2604
|
Avoid holding references to a `Reader` obtained via `start` outside its `Scope`. The scope guarantees cleanup; escaping the reader defeats that guarantee.
|
|
1902
2605
|
:::
|
|
@@ -1907,13 +2610,13 @@ ZIO Blocks Streams achieves zero-boxing via compile-time type detection and dual
|
|
|
1907
2610
|
|
|
1908
2611
|
### JVM Primitive Specialization
|
|
1909
2612
|
|
|
1910
|
-
By default, Scala's type system boxes primitive values
|
|
2613
|
+
By default, Scala's type system boxes primitive values into objects, which wastes memory and is slower. ZIO Blocks specializes all eight JVM primitive representations: `Boolean`, `Byte`, `Char`, `Short`, `Int`, `Long`, `Float`, and `Double`. `JvmType.Infer[A]` records a result type's physical lane at construction and at output-changing operations; type-preserving operations carry an already-known lane forward.
|
|
1911
2614
|
|
|
1912
|
-
For example,
|
|
2615
|
+
For example, an `Int` pipeline uses `readInt(Long.MinValue)` instead of boxing. A `Long` or `Double` internal pull cannot use a collision-free scalar sentinel, so it calls `readLongs` or `readDoubles` with a length-one array and interprets the returned count as EOF/data status. This preserves every `Long` and `Double` value:
|
|
1913
2616
|
|
|
1914
2617
|
```scala
|
|
1915
2618
|
if (jvmType eq JvmType.Int) {
|
|
1916
|
-
val i = source.readInt(Long.MinValue)
|
|
2619
|
+
val i = source.readInt(Long.MinValue)
|
|
1917
2620
|
// ... unboxed, fast path
|
|
1918
2621
|
} else {
|
|
1919
2622
|
val o = reader.read(EndOfStream) // generic boxed path
|
|
@@ -1923,7 +2626,7 @@ if (jvmType eq JvmType.Int) {
|
|
|
1923
2626
|
|
|
1924
2627
|
This optimization is transparent: you write normal, high-level code, and the compiler and runtime automatically use the fast path for primitives.
|
|
1925
2628
|
|
|
1926
|
-
### Dual Compilation: Recursive
|
|
2629
|
+
### Dual Compilation: Recursive Vs Interpreter
|
|
1927
2630
|
|
|
1928
2631
|
Each stream node compiles in two ways:
|
|
1929
2632
|
|
|
@@ -1933,9 +2636,11 @@ Each stream node compiles in two ways:
|
|
|
1933
2636
|
|
|
1934
2637
|
The switch happens at `DepthCutoff = 100`. You should never see this in normal use, but it ensures that pipelines of any depth are safe.
|
|
1935
2638
|
|
|
2639
|
+
A second split sits alongside this one: a graph made only of synchronous nodes compiles to the synchronous engine, and a graph containing any asynchronous node compiles to a separate, heap-allocated asynchronous engine. That choice is made once per materialization, never per element, and never from an annotation you write. [Why two engines](../execution-and-compatibility/async-execution.md#why-two-engines) explains what it buys, and why a purely synchronous stream pays nothing for it.
|
|
2640
|
+
|
|
1936
2641
|
## Running the Examples
|
|
1937
2642
|
|
|
1938
|
-
All code from this guide is available as runnable examples in the `
|
|
2643
|
+
All code from this guide is available as runnable examples in the `streams-examples` module.
|
|
1939
2644
|
|
|
1940
2645
|
Clone the repository and navigate to the project:
|
|
1941
2646
|
|
|
@@ -1953,22 +2658,6 @@ cd zio-blocks
|
|
|
1953
2658
|
This example demonstrates constructing streams from collections, transforming elements with `Stream#map` and `Stream#filter`, and collecting results:
|
|
1954
2659
|
|
|
1955
2660
|
```scala title="streams-examples/src/main/scala/stream/StreamBasicUsageExample.scala"
|
|
1956
|
-
/*
|
|
1957
|
-
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
1958
|
-
*
|
|
1959
|
-
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
1960
|
-
* you may not use this file except in compliance with the License.
|
|
1961
|
-
* You may obtain a copy of the License at
|
|
1962
|
-
*
|
|
1963
|
-
* http://www.apache.org/licenses/LICENSE-2.0
|
|
1964
|
-
*
|
|
1965
|
-
* Unless required by applicable law or agreed to in writing, software
|
|
1966
|
-
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
1967
|
-
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
1968
|
-
* See the License for the specific language governing permissions and
|
|
1969
|
-
* limitations under the License.
|
|
1970
|
-
*/
|
|
1971
|
-
|
|
1972
2661
|
package stream
|
|
1973
2662
|
|
|
1974
2663
|
import zio.blocks.streams.Stream
|
|
@@ -2038,22 +2727,6 @@ sbt "streams-examples/runMain stream.StreamBasicUsageExample"
|
|
|
2038
2727
|
This example shows how `Stream#flatMap` sequences multiple streams and flattens the results:
|
|
2039
2728
|
|
|
2040
2729
|
```scala title="streams-examples/src/main/scala/stream/StreamFlatMapExample.scala"
|
|
2041
|
-
/*
|
|
2042
|
-
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
2043
|
-
*
|
|
2044
|
-
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
2045
|
-
* you may not use this file except in compliance with the License.
|
|
2046
|
-
* You may obtain a copy of the License at
|
|
2047
|
-
*
|
|
2048
|
-
* http://www.apache.org/licenses/LICENSE-2.0
|
|
2049
|
-
*
|
|
2050
|
-
* Unless required by applicable law or agreed to in writing, software
|
|
2051
|
-
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
2052
|
-
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
2053
|
-
* See the License for the specific language governing permissions and
|
|
2054
|
-
* limitations under the License.
|
|
2055
|
-
*/
|
|
2056
|
-
|
|
2057
2730
|
package stream
|
|
2058
2731
|
|
|
2059
2732
|
import zio.blocks.streams.Stream
|
|
@@ -2122,22 +2795,6 @@ sbt "streams-examples/runMain stream.StreamFlatMapExample"
|
|
|
2122
2795
|
This example demonstrates typed error recovery with `fail`, `catchAll`, and `orElse`:
|
|
2123
2796
|
|
|
2124
2797
|
```scala title="streams-examples/src/main/scala/stream/StreamErrorHandlingExample.scala"
|
|
2125
|
-
/*
|
|
2126
|
-
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
2127
|
-
*
|
|
2128
|
-
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
2129
|
-
* you may not use this file except in compliance with the License.
|
|
2130
|
-
* You may obtain a copy of the License at
|
|
2131
|
-
*
|
|
2132
|
-
* http://www.apache.org/licenses/LICENSE-2.0
|
|
2133
|
-
*
|
|
2134
|
-
* Unless required by applicable law or agreed to in writing, software
|
|
2135
|
-
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
2136
|
-
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
2137
|
-
* See the License for the specific language governing permissions and
|
|
2138
|
-
* limitations under the License.
|
|
2139
|
-
*/
|
|
2140
|
-
|
|
2141
2798
|
package stream
|
|
2142
2799
|
|
|
2143
2800
|
import zio.blocks.streams.Stream
|
|
@@ -2229,22 +2886,6 @@ sbt "streams-examples/runMain stream.StreamErrorHandlingExample"
|
|
|
2229
2886
|
This example shows how `fromAcquireRelease` and `ensuring` manage resources safely:
|
|
2230
2887
|
|
|
2231
2888
|
```scala title="streams-examples/src/main/scala/stream/StreamResourceExample.scala"
|
|
2232
|
-
/*
|
|
2233
|
-
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
2234
|
-
*
|
|
2235
|
-
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
2236
|
-
* you may not use this file except in compliance with the License.
|
|
2237
|
-
* You may obtain a copy of the License at
|
|
2238
|
-
*
|
|
2239
|
-
* http://www.apache.org/licenses/LICENSE-2.0
|
|
2240
|
-
*
|
|
2241
|
-
* Unless required by applicable law or agreed to in writing, software
|
|
2242
|
-
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
2243
|
-
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
2244
|
-
* See the License for the specific language governing permissions and
|
|
2245
|
-
* limitations under the License.
|
|
2246
|
-
*/
|
|
2247
|
-
|
|
2248
2889
|
package stream
|
|
2249
2890
|
|
|
2250
2891
|
import zio.blocks.streams.Stream
|
|
@@ -2418,22 +3059,6 @@ sbt "streams-examples/runMain stream.StreamResourceExample"
|
|
|
2418
3059
|
This example demonstrates `grouped`, `sliding`, and `scan` for windowing and stateful transformations:
|
|
2419
3060
|
|
|
2420
3061
|
```scala title="streams-examples/src/main/scala/stream/StreamWindowingExample.scala"
|
|
2421
|
-
/*
|
|
2422
|
-
* Copyright 2024-2026 John A. De Goes and the ZIO Contributors
|
|
2423
|
-
*
|
|
2424
|
-
* Licensed under the Apache License, Version 2.0 (the "License");
|
|
2425
|
-
* you may not use this file except in compliance with the License.
|
|
2426
|
-
* You may obtain a copy of the License at
|
|
2427
|
-
*
|
|
2428
|
-
* http://www.apache.org/licenses/LICENSE-2.0
|
|
2429
|
-
*
|
|
2430
|
-
* Unless required by applicable law or agreed to in writing, software
|
|
2431
|
-
* distributed under the License is distributed on an "AS IS" BASIS,
|
|
2432
|
-
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
2433
|
-
* See the License for the specific language governing permissions and
|
|
2434
|
-
* limitations under the License.
|
|
2435
|
-
*/
|
|
2436
|
-
|
|
2437
3062
|
package stream
|
|
2438
3063
|
|
|
2439
3064
|
import zio.blocks.streams.Stream
|
|
@@ -2524,3 +3149,88 @@ Run this example:
|
|
|
2524
3149
|
```bash
|
|
2525
3150
|
sbt "streams-examples/runMain stream.StreamWindowingExample"
|
|
2526
3151
|
```
|
|
3152
|
+
|
|
3153
|
+
### Stateful Asynchronous Operators
|
|
3154
|
+
|
|
3155
|
+
This example runs `takeWhileAsync`, `mapAccumAsync`, `scanAsync`, and `ensuringAsync` in a single pipeline, with every callback completing on another thread. It is JVM-only, because it ends in `.block`:
|
|
3156
|
+
|
|
3157
|
+
```scala title="streams-examples/src/main/scala/stream/StreamAsyncStatefulExample.scala"
|
|
3158
|
+
package stream
|
|
3159
|
+
|
|
3160
|
+
import java.util.concurrent.atomic.AtomicInteger
|
|
3161
|
+
|
|
3162
|
+
import zio.blocks.async._
|
|
3163
|
+
import zio.blocks.chunk.Chunk
|
|
3164
|
+
import zio.blocks.streams.Stream
|
|
3165
|
+
|
|
3166
|
+
/**
|
|
3167
|
+
* The stateful asynchronous operators in one pipeline: `takeWhileAsync`,
|
|
3168
|
+
* `mapAccumAsync`, `scanAsync`, and `ensuringAsync`.
|
|
3169
|
+
*
|
|
3170
|
+
* A day's ledger feed is cut at its end-of-day marker, numbered, folded into
|
|
3171
|
+
* running balances, and closed by an asynchronous finalizer. Every step
|
|
3172
|
+
* suspends on another thread, and every step is still sequential: at most one
|
|
3173
|
+
* invocation of each callback is active at a time, so the accumulator is never
|
|
3174
|
+
* shared across concurrent work.
|
|
3175
|
+
*
|
|
3176
|
+
* JVM only: the closing `.block` parks the calling thread until the `Async`
|
|
3177
|
+
* completes. On Scala.js, keep the `Async` and let the host drive it.
|
|
3178
|
+
*/
|
|
3179
|
+
object StreamAsyncStatefulExample {
|
|
3180
|
+
final case class Posting(seq: Long, amount: Int)
|
|
3181
|
+
|
|
3182
|
+
/** Completes on another thread, so each callback below really suspends. */
|
|
3183
|
+
def deferred[A](value: => A): Async[A] = {
|
|
3184
|
+
val completer = new Completer[A]
|
|
3185
|
+
val thread = new Thread(() => completer.succeed(value))
|
|
3186
|
+
thread.start()
|
|
3187
|
+
completer
|
|
3188
|
+
}
|
|
3189
|
+
|
|
3190
|
+
def main(args: Array[String]): Unit = {
|
|
3191
|
+
val sessionsClosed = new AtomicInteger
|
|
3192
|
+
|
|
3193
|
+
// `0` is the end-of-day marker, not an amount. Everything after it belongs
|
|
3194
|
+
// to the next day and must not be posted.
|
|
3195
|
+
val feed: Stream[Nothing, Int] = Stream(120, -40, 75, 0, 999)
|
|
3196
|
+
|
|
3197
|
+
val balances: Stream[Nothing, Long] = feed
|
|
3198
|
+
// Stops at the marker and closes upstream, so `999` is never pulled.
|
|
3199
|
+
.takeWhileAsync(amount => deferred(amount != 0))
|
|
3200
|
+
// Threads a sequence number through while numbering each entry.
|
|
3201
|
+
.mapAccumAsync(0L)((seq, amount) => deferred((seq + 1, Posting(seq + 1, amount))))
|
|
3202
|
+
// Emits the accumulator at each step, starting with `0L`, so the output
|
|
3203
|
+
// carries one more element than the input.
|
|
3204
|
+
.scanAsync(0L)((balance, posting) => deferred(balance + posting.amount))
|
|
3205
|
+
// Awaited exactly once when the materialized stream closes.
|
|
3206
|
+
.ensuringAsync(deferred { sessionsClosed.incrementAndGet(); () })
|
|
3207
|
+
|
|
3208
|
+
val result: Either[Nothing, Chunk[Long]] = balances.runCollectAsync.block
|
|
3209
|
+
|
|
3210
|
+
require(result == Right(Chunk(0L, 120L, 80L, 155L)), s"unexpected balances: $result")
|
|
3211
|
+
require(sessionsClosed.get() == 1, s"finalizer ran ${sessionsClosed.get()} times")
|
|
3212
|
+
|
|
3213
|
+
println(s"running balances -> $result")
|
|
3214
|
+
println(s"sessions closed -> ${sessionsClosed.get()}")
|
|
3215
|
+
}
|
|
3216
|
+
}
|
|
3217
|
+
```
|
|
3218
|
+
|
|
3219
|
+
Run this example:
|
|
3220
|
+
|
|
3221
|
+
```bash
|
|
3222
|
+
sbt "streams-examples/runMain stream.StreamAsyncStatefulExample"
|
|
3223
|
+
```
|
|
3224
|
+
|
|
3225
|
+
## Native Asynchronous Byte Readers
|
|
3226
|
+
|
|
3227
|
+
Each platform ships adapters that turn a native byte source into a `Reader.AsyncReader[Byte]`, which `Stream.fromReader` then lifts into a stream: `AsyncNioReaders.fromChannel` and `fromSocket` on the JVM, `ReadableStreamReaders.fromReadableStream` on Scala.js, each with an `Unmanaged` variant that leaves the native source caller-owned. [Reader](../primitives/reader.md#from-native-asynchronous-sources) documents them, together with their chunking and EOF rules and the way source failures reach the typed error channel.
|
|
3228
|
+
|
|
3229
|
+
## See Also
|
|
3230
|
+
|
|
3231
|
+
- [Asynchronous Stream Execution](../execution-and-compatibility/async-execution.md) — the full asynchronous constructor, operator, and terminal API, and how one `Stream` type describes both execution modes
|
|
3232
|
+
- [Platform Differences](../execution-and-compatibility/platform-differences.md) — which members exist on the JVM, which exist on Scala.js, and why the blocking family is JVM-only
|
|
3233
|
+
- [Reader](../primitives/reader.md) — the `SyncReader` / `AsyncReader` union that decides which engine materializes behind the bounded-concurrency operators, and the adapters behind the native asynchronous byte readers
|
|
3234
|
+
- [Async Reference](../../async.md) — `Async.promise` and `Completer` bridge callback-based APIs into async values that can feed stream sources; `Async.Running` carries a synchronous cancellation handle that complements stream resource management
|
|
3235
|
+
- [Async](../../async.md#asyncselector) — `AsyncSelector`, the primitive the bounded-concurrency slot machinery is built from
|
|
3236
|
+
- [Scope Reference](../../resource-management/scope.md) — compile-time resource safety for stream acquisition and release; `fromAcquireRelease` follows the same ownership rules as Scope-managed resources
|