penguiflow 1.0.3__py3-none-any.whl → 2.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Potentially problematic release.


This version of penguiflow might be problematic. Click here for more details.

@@ -0,0 +1,646 @@
1
+ Metadata-Version: 2.4
2
+ Name: penguiflow
3
+ Version: 2.1.0
4
+ Summary: Async agent orchestration primitives.
5
+ Author: PenguiFlow Team
6
+ License: MIT License
7
+
8
+ Copyright (c) 2025 hurtener
9
+
10
+ Permission is hereby granted, free of charge, to any person obtaining a copy
11
+ of this software and associated documentation files (the "Software"), to deal
12
+ in the Software without restriction, including without limitation the rights
13
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
14
+ copies of the Software, and to permit persons to whom the Software is
15
+ furnished to do so, subject to the following conditions:
16
+
17
+ The above copyright notice and this permission notice shall be included in all
18
+ copies or substantial portions of the Software.
19
+
20
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
21
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
22
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
23
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
24
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
25
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
26
+ SOFTWARE.
27
+
28
+ Project-URL: Homepage, https://github.com/penguiflow/penguiflow
29
+ Requires-Python: >=3.11
30
+ Description-Content-Type: text/markdown
31
+ License-File: LICENSE
32
+ Requires-Dist: pydantic>=2.6
33
+ Provides-Extra: dev
34
+ Requires-Dist: mypy>=1.8; extra == "dev"
35
+ Requires-Dist: pytest>=7.4; extra == "dev"
36
+ Requires-Dist: pytest-asyncio>=0.23; extra == "dev"
37
+ Requires-Dist: pytest-cov>=4.0; extra == "dev"
38
+ Requires-Dist: coverage[toml]>=7.0; extra == "dev"
39
+ Requires-Dist: ruff>=0.2; extra == "dev"
40
+ Requires-Dist: fastapi>=0.110; extra == "dev"
41
+ Requires-Dist: httpx>=0.27; extra == "dev"
42
+ Provides-Extra: a2a-server
43
+ Requires-Dist: fastapi>=0.110; extra == "a2a-server"
44
+ Dynamic: license-file
45
+
46
+ # PenguiFlow 🐧❄️
47
+
48
+ <p align="center">
49
+ <img src="asset/Penguiflow.png" alt="PenguiFlow logo" width="220">
50
+ </p>
51
+
52
+ <p align="center">
53
+ <a href="https://github.com/penguiflow/penguiflow/actions/workflows/ci.yml">
54
+ <img src="https://github.com/penguiflow/penguiflow/actions/workflows/ci.yml/badge.svg" alt="CI Status">
55
+ </a>
56
+ <a href="https://github.com/penguiflow/penguiflow">
57
+ <img src="https://img.shields.io/badge/coverage-85%25-brightgreen" alt="Coverage">
58
+ </a>
59
+ <a href="https://pypi.org/project/penguiflow/">
60
+ <img src="https://img.shields.io/pypi/v/penguiflow.svg" alt="PyPI version">
61
+ </a>
62
+ <a href="https://github.com/penguiflow/penguiflow/blob/main/LICENSE">
63
+ <img src="https://img.shields.io/badge/license-MIT-blue.svg" alt="License">
64
+ </a>
65
+ </p>
66
+
67
+ **Async-first orchestration library for multi-agent and data pipelines**
68
+
69
+ PenguiFlow is a **lightweight Python library** to orchestrate agent flows.
70
+ It provides:
71
+
72
+ * **Typed, async message passing** (Pydantic v2)
73
+ * **Concurrent fan-out / fan-in patterns**
74
+ * **Routing & decision points**
75
+ * **Retries, timeouts, backpressure**
76
+ * **Streaming chunks** (LLM-style token emission with `Context.emit_chunk`)
77
+ * **Dynamic loops** (controller nodes)
78
+ * **Runtime playbooks** (callable subflows with shared metadata)
79
+ * **Per-trace cancellation** (`PenguiFlow.cancel` with `TraceCancelled` surfacing in nodes)
80
+ * **Deadlines & budgets** (`Message.deadline_s`, `WM.budget_hops`, and `WM.budget_tokens` guardrails that you can leave unset/`None`)
81
+ * **Observability hooks** (`FlowEvent` callbacks for logging, MLflow, or custom metrics sinks)
82
+ * **Policy-driven routing** (optional policies steer routers without breaking existing flows)
83
+ * **Traceable exceptions** (`FlowError` captures node/trace metadata and optionally emits to Rookery)
84
+ * **Distribution hooks (opt-in)** — plug a `StateStore` to persist trace history and a
85
+ `MessageBus` to publish floe traffic for remote workers without changing existing flows.
86
+ * **Remote calls (opt-in)** — `RemoteNode` bridges the runtime to external agents through a
87
+ pluggable `RemoteTransport` interface (A2A-ready) while propagating streaming chunks and
88
+ cancellation.
89
+ * **A2A server adapter (opt-in)** — wrap a PenguiFlow graph in a FastAPI surface using
90
+ `penguiflow_a2a.A2AServerAdapter` so other agents can call `message/send`,
91
+ `message/stream`, and `tasks/cancel` while reusing the runtime's backpressure and
92
+ cancellation semantics.
93
+ * **Observability & ops polish** — remote calls emit structured metrics (latency, payload
94
+ sizes, cancel reasons) and the `penguiflow-admin` CLI replays trace history from any
95
+ configured `StateStore` for debugging.
96
+
97
+ Built on pure `asyncio` (no threads), PenguiFlow is small, predictable, and repo-agnostic.
98
+ Product repos only define **their models + node functions** — the core stays dependency-light.
99
+
100
+ ---
101
+
102
+ ## ✨ Why PenguiFlow?
103
+
104
+ * **Orchestration is everywhere.** Every Pengui service needs to connect LLMs, retrievers, SQL, or external APIs.
105
+ * **Stop rewriting glue.** This library gives you reusable primitives (nodes, flows, contexts) so you can focus on business logic.
106
+ * **Typed & safe.** Every hop validated with Pydantic.
107
+ * **Lightweight.** Only depends on asyncio + pydantic. No broker, no server, no threads.
108
+
109
+ ---
110
+
111
+ ## 🏗️ Core Concepts
112
+
113
+ ### Message
114
+
115
+ Every payload is wrapped in a `Message` with headers and metadata.
116
+
117
+ ```python
118
+ from pydantic import BaseModel
119
+ from penguiflow.types import Message, Headers
120
+
121
+ class QueryIn(BaseModel):
122
+ text: str
123
+
124
+ msg = Message(
125
+ payload=QueryIn(text="unique reach last 30 days"),
126
+ headers=Headers(tenant="acme")
127
+ )
128
+ msg.meta["request_id"] = "abc123"
129
+ ```
130
+
131
+ ### Node
132
+
133
+ A node is an async function wrapped with a `Node`.
134
+ It validates inputs/outputs (via `ModelRegistry`) and applies `NodePolicy` (timeout, retries, etc.).
135
+
136
+ ```python
137
+ from penguiflow.node import Node
138
+
139
+ class QueryOut(BaseModel):
140
+ topic: str
141
+
142
+ async def triage(msg: QueryIn, ctx) -> QueryOut:
143
+ return QueryOut(topic="metrics")
144
+
145
+ triage_node = Node(triage, name="triage")
146
+ ```
147
+
148
+ Node functions must always accept **two positional parameters**: the incoming payload and
149
+ the `Context` object. If a node does not use the context, name it `_` or `_ctx`, but keep
150
+ the parameter so the runtime can still inject it. Registering the node with
151
+ `ModelRegistry` ensures the payload is validated/cast to the expected Pydantic model;
152
+ setting `NodePolicy(validate="none")` skips that validation for hot paths.
153
+
154
+ ### Flow
155
+
156
+ A flow wires nodes together in a directed graph.
157
+ Edges are called **Floe**s, and flows have two invisible contexts:
158
+
159
+ * **OpenSea** 🌊 — ingress (start of the flow)
160
+ * **Rookery** 🐧 — egress (end of the flow)
161
+
162
+ ```python
163
+ from penguiflow.core import create
164
+
165
+ flow = create(
166
+ triage_node.to(packer_node)
167
+ )
168
+ ```
169
+
170
+ ### Running a Flow
171
+
172
+ ```python
173
+ from penguiflow.registry import ModelRegistry
174
+
175
+ registry = ModelRegistry()
176
+ registry.register("triage", QueryIn, QueryOut)
177
+ registry.register("packer", QueryOut, PackOut)
178
+
179
+ flow.run(registry=registry)
180
+
181
+ await flow.emit(msg) # emit into OpenSea
182
+ out = await flow.fetch() # fetch from Rookery
183
+ print(out.payload) # PackOut(...)
184
+ await flow.stop()
185
+ ```
186
+
187
+ > **Opt-in distribution:** pass `state_store=` and/or `message_bus=` when calling
188
+ > `penguiflow.core.create(...)` to persist trace history and publish floe traffic
189
+ > without changing node logic.
190
+
191
+ ---
192
+
193
+ ## 🧭 Design Principles
194
+
195
+ 1. **Async-only (`asyncio`).**
196
+
197
+ * Flows are orchestrators, mostly I/O-bound.
198
+ * Async tasks are cheap, predictable, and cancellable.
199
+ * Heavy CPU work should be offloaded inside a node (process pool, Ray, etc.), not in PenguiFlow itself.
200
+ * v1 intentionally stays in-process; scaling out or persisting state will arrive with future pluggable backends.
201
+
202
+ 2. **Typed contracts.**
203
+
204
+ * In/out models per node are defined with Pydantic.
205
+ * Validated at runtime via cached `TypeAdapter`s.
206
+ * `flow.run(registry=...)` verifies every validating node is registered so misconfigurations fail fast.
207
+
208
+ 3. **Reliability first.**
209
+
210
+ * Timeouts, retries with backoff, backpressure on queues.
211
+ * Nodes run inside error boundaries.
212
+
213
+ 4. **Minimal dependencies.**
214
+
215
+ * Only asyncio + pydantic.
216
+ * No broker, no server. Everything in-process.
217
+
218
+ 5. **Repo-agnostic.**
219
+
220
+ * Product repos declare their models + node funcs, register them, and run.
221
+ * No product-specific code in the library.
222
+
223
+ ---
224
+
225
+ ## 📦 Installation
226
+
227
+ ```bash
228
+ pip install -e ./penguiflow
229
+ ```
230
+
231
+ Requires **Python 3.11+**.
232
+
233
+ ---
234
+
235
+ ## 🛠️ Key capabilities
236
+
237
+ ### Streaming & incremental delivery
238
+
239
+ `Context.emit_chunk` (and `PenguiFlow.emit_chunk`) provide token-level streaming without
240
+ sacrificing backpressure or ordering guarantees. The helper wraps the payload in a
241
+ `StreamChunk`, mirrors routing metadata from the parent message, and automatically
242
+ increments per-stream sequence numbers. See `tests/test_streaming.py` and
243
+ `examples/streaming_llm/` for an end-to-end walk-through.
244
+
245
+ ### Remote orchestration
246
+
247
+ Phase 2 introduces `RemoteNode` and the `RemoteTransport` protocol so flows can delegate
248
+ work to remote agents (e.g., the A2A JSON-RPC/SSE ecosystem) without changing existing
249
+ nodes. The helper records remote bindings via the `StateStore`, mirrors streaming
250
+ partials back into the graph, and propagates per-trace cancellation to remote tasks via
251
+ `RemoteTransport.cancel`. See `tests/test_remote.py` for reference in-memory transports.
252
+
253
+ ### Exposing a flow over A2A
254
+
255
+ Install the optional extra to expose PenguiFlow as an A2A-compatible FastAPI service:
256
+
257
+ ```bash
258
+ pip install "penguiflow[a2a-server]"
259
+ ```
260
+
261
+ Create the adapter and mount the routes:
262
+
263
+ ```python
264
+ from penguiflow import Message, Node, create
265
+ from penguiflow_a2a import A2AAgentCard, A2AServerAdapter, A2ASkill, create_a2a_app
266
+
267
+ async def orchestrate(message: Message, ctx):
268
+ await ctx.emit_chunk(parent=message, text="thinking...")
269
+ return {"result": "done"}
270
+
271
+ node = Node(orchestrate, name="main")
272
+ flow = create(node.to())
273
+
274
+ card = A2AAgentCard(
275
+ name="Main Agent",
276
+ description="Primary entrypoint for orchestration",
277
+ version="2.1.0",
278
+ skills=[A2ASkill(name="orchestrate", description="Handles orchestration")],
279
+ )
280
+
281
+ adapter = A2AServerAdapter(
282
+ flow,
283
+ agent_card=card,
284
+ agent_url="https://agent.example",
285
+ )
286
+ app = create_a2a_app(adapter)
287
+ ```
288
+
289
+ The generated FastAPI app implements:
290
+
291
+ * `GET /agent` for discovery (Agent Card)
292
+ * `POST /message/send` for unary execution
293
+ * `POST /message/stream` for SSE streaming
294
+ * `POST /tasks/cancel` to mirror cancellation into PenguiFlow traces
295
+
296
+ `A2AServerAdapter` reuses the runtime's `StateStore` hooks, so bindings between trace IDs
297
+ and external `taskId`/`contextId` pairs are persisted automatically.
298
+
299
+ ### Reliability & guardrails
300
+
301
+ PenguiFlow enforces reliability boundaries out of the box:
302
+
303
+ * **Per-trace cancellation** (`PenguiFlow.cancel(trace_id)`) unwinds a single run while
304
+ other traces keep executing. Worker tasks observe `TraceCancelled` and clean up
305
+ resources; `tests/test_cancel.py` covers the behaviour.
306
+ * **Deadlines & budgets** let you keep loops honest. `Message.deadline_s` guards
307
+ wall-clock execution, while controller payloads (`WM`) track hop and token budgets.
308
+ Exhaustion short-circuits into terminal `FinalAnswer` messages as demonstrated in
309
+ `tests/test_budgets.py` and `examples/controller_multihop/`.
310
+ * **Retries & timeouts** live in `NodePolicy`. Exponential backoff, timeout enforcement,
311
+ and structured retry events are exercised heavily in the core test suite.
312
+
313
+ ### Metadata & observability
314
+
315
+ Every `Message` carries a mutable `meta` dictionary so nodes can propagate debugging
316
+ breadcrumbs, billing information, or routing hints without touching the payload. The
317
+ runtime clones metadata during streaming and playbook calls (`tests/test_metadata.py`).
318
+ Structured runtime events surface through `FlowEvent` objects; attach middlewares for
319
+ custom logging or metrics ingestion (`examples/mlflow_metrics/`).
320
+
321
+ ### Routing & dynamic policies
322
+
323
+ Branching flows stay flexible thanks to routers and optional policies. The
324
+ `predicate_router` and `union_router` helpers can consult a `RoutingPolicy` at runtime to
325
+ override or drop successors, while `DictRoutingPolicy` provides a config-driven
326
+ implementation ready for JSON/YAML/env inputs (`tests/test_routing_policy.py`,
327
+ `examples/routing_policy/`).
328
+
329
+ ### Traceable exceptions
330
+
331
+ When retries are exhausted or timeouts fire, PenguiFlow wraps the failure in a
332
+ `FlowError` that preserves the trace id, node metadata, and a stable error code.
333
+ Opt into `emit_errors_to_rookery=True` to receive these objects directly from
334
+ `flow.fetch()`—see `tests/test_errors.py` and `examples/traceable_errors/` for usage.
335
+
336
+ ### FlowTestKit
337
+
338
+ The new `penguiflow.testkit` module keeps unit tests tiny:
339
+
340
+ * `await testkit.run_one(flow, message)` boots a flow, emits a message, captures runtime
341
+ events, and returns the first Rookery payload.
342
+ * `testkit.assert_node_sequence(trace_id, [...])` asserts the order in which nodes ran.
343
+ * `testkit.simulate_error(...)` builds coroutine helpers that fail a configurable number
344
+ of times—perfect for retry scenarios.
345
+
346
+ The harness is covered by `tests/test_testkit.py` and demonstrated in
347
+ `examples/testkit_demo/`.
348
+
349
+
350
+ ## 🧭 Repo Structure
351
+
352
+ penguiflow/
353
+ __init__.py
354
+ core.py # runtime orchestrator, retries, controller helpers, playbooks
355
+ errors.py # FlowError / FlowErrorCode definitions
356
+ node.py
357
+ types.py
358
+ registry.py
359
+ patterns.py
360
+ middlewares.py
361
+ viz.py
362
+ README.md
363
+ pyproject.toml # build metadata
364
+ tests/ # pytest suite
365
+ examples/ # runnable flows (fan-out, routing, controller, playbooks)
366
+
367
+ ---
368
+
369
+ ## 🚀 Quickstart Example
370
+
371
+ ```python
372
+ from pydantic import BaseModel
373
+ from penguiflow import Headers, Message, ModelRegistry, Node, NodePolicy, create
374
+
375
+
376
+ class TriageIn(BaseModel):
377
+ text: str
378
+
379
+
380
+ class TriageOut(BaseModel):
381
+ text: str
382
+ topic: str
383
+
384
+
385
+ class RetrieveOut(BaseModel):
386
+ topic: str
387
+ docs: list[str]
388
+
389
+
390
+ class PackOut(BaseModel):
391
+ prompt: str
392
+
393
+
394
+ async def triage(msg: TriageIn, ctx) -> TriageOut:
395
+ topic = "metrics" if "metric" in msg.text else "general"
396
+ return TriageOut(text=msg.text, topic=topic)
397
+
398
+
399
+ async def retrieve(msg: TriageOut, ctx) -> RetrieveOut:
400
+ docs = [f"doc_{i}_{msg.topic}" for i in range(2)]
401
+ return RetrieveOut(topic=msg.topic, docs=docs)
402
+
403
+
404
+ async def pack(msg: RetrieveOut, ctx) -> PackOut:
405
+ prompt = f"[{msg.topic}] summarize {len(msg.docs)} docs"
406
+ return PackOut(prompt=prompt)
407
+
408
+
409
+ triage_node = Node(triage, name="triage", policy=NodePolicy(validate="both"))
410
+ retrieve_node = Node(retrieve, name="retrieve", policy=NodePolicy(validate="both"))
411
+ pack_node = Node(pack, name="pack", policy=NodePolicy(validate="both"))
412
+
413
+ registry = ModelRegistry()
414
+ registry.register("triage", TriageIn, TriageOut)
415
+ registry.register("retrieve", TriageOut, RetrieveOut)
416
+ registry.register("pack", RetrieveOut, PackOut)
417
+
418
+ flow = create(
419
+ triage_node.to(retrieve_node),
420
+ retrieve_node.to(pack_node),
421
+ )
422
+ flow.run(registry=registry)
423
+
424
+ message = Message(
425
+ payload=TriageIn(text="show marketing metrics"),
426
+ headers=Headers(tenant="acme"),
427
+ )
428
+
429
+ await flow.emit(message)
430
+ out = await flow.fetch()
431
+ print(out.prompt) # PackOut(prompt='[metrics] summarize 2 docs')
432
+
433
+ await flow.stop()
434
+ ```
435
+
436
+ ### Patterns Toolkit
437
+
438
+ PenguiFlow ships a handful of **composable patterns** to keep orchestration code tidy
439
+ without forcing you into a one-size-fits-all DSL. Each helper is opt-in and can be
440
+ stitched directly into a flow adjacency list:
441
+
442
+ - `map_concurrent(items, worker, max_concurrency=8)` — fan a single message out into
443
+ many in-memory tasks (e.g., batch document enrichment) while respecting a semaphore.
444
+ - `predicate_router(name, predicate, policy=None)` — route messages to successor nodes
445
+ based on simple boolean functions over payload or headers, optionally consulting a
446
+ runtime `policy` to override or filter the computed targets. Perfect for guardrails or
447
+ conditional tool invocation without rebuilding the flow.
448
+ - `union_router(name, discriminated_model)` — accept a Pydantic discriminated union and
449
+ forward each variant to the matching typed successor node. Keeps type-safety even when
450
+ multiple schema branches exist.
451
+ - `join_k(name, k)` — aggregate `k` messages per `trace_id` before resuming downstream
452
+ work. Useful for fan-out/fan-in batching, map-reduce style summarization, or consensus.
453
+ - `DictRoutingPolicy(mapping, key_getter=None)` — load routing overrides from
454
+ configuration and pair it with the router helpers via `policy=...` to switch routing at
455
+ runtime without modifying the flow graph.
456
+
457
+ All helpers are regular `Node` instances under the hood, so they inherit retries,
458
+ timeouts, and validation just like hand-written nodes.
459
+
460
+ ### Streaming Responses
461
+
462
+ PenguiFlow now supports **LLM-style streaming** with the `StreamChunk` model. Each
463
+ chunk carries `stream_id`, `seq`, `text`, optional `meta`, and a `done` flag. Use
464
+ `Context.emit_chunk(parent=message, text=..., done=...)` inside a node (or the
465
+ convenience wrapper `await flow.emit_chunk(...)` from outside a node) to push
466
+ chunks downstream without manually crafting `Message` envelopes:
467
+
468
+ ```python
469
+ await ctx.emit_chunk(parent=msg, text=token, done=done)
470
+ ```
471
+
472
+ - Sequence numbers auto-increment per `stream_id` (defaults to the parent trace).
473
+ - Backpressure is preserved; if the downstream queue is full the helper awaits just
474
+ like `Context.emit`.
475
+ - When `done=True`, the sequence counter resets so a new stream can reuse the same id.
476
+
477
+ Pair the producer with a sink node that consumes `StreamChunk` payloads and assembles
478
+ the final result when `done` is observed. See `examples/streaming_llm/` for a complete
479
+ mock LLM → SSE pipeline. For presentation layers, utilities like
480
+ `format_sse_event(chunk)` and `chunk_to_ws_json(chunk)` (both exported from the
481
+ package) will convert a `StreamChunk` into SSE-compatible text or WebSocket JSON payloads
482
+ without boilerplate.
483
+
484
+ ### Dynamic Controller Loops
485
+
486
+ Long-running agents often need to **think, plan, and act over multiple hops**. PenguiFlow
487
+ models this with a controller node that loops on itself:
488
+
489
+ 1. Define a controller `Node` with `allow_cycle=True` and wire `controller.to(controller)`.
490
+ 2. Emit a `Message` whose payload is a `WM` (working memory). PenguiFlow increments the
491
+ `hops` counter automatically and enforces `budget_hops` + `deadline_s` so controllers
492
+ cannot loop forever.
493
+ 3. The controller can attach intermediate `Thought` artifacts or emit `PlanStep`s for
494
+ transparency/debugging. When it is ready to finish, it returns a `FinalAnswer` which
495
+ is immediately forwarded to Rookery.
496
+
497
+ Deadlines and hop budgets turn into automated `FinalAnswer` error messages, making it
498
+ easy to surface guardrails to downstream consumers.
499
+
500
+ ---
501
+
502
+ ### Playbooks & Subflows
503
+
504
+ Sometimes a controller or router needs to execute a **mini flow** — for example,
505
+ retrieval → rerank → compress — without polluting the global topology.
506
+ `Context.call_playbook` spawns a brand-new `PenguiFlow` on demand and wires it into
507
+ the parent message context:
508
+
509
+ - Trace IDs and headers are reused so observability stays intact.
510
+ - The helper respects optional timeouts, mirrors cancellation to the subflow, and always
511
+ stops it (even on cancel).
512
+ - The first payload emitted to the playbook's Rookery is returned to the caller,
513
+ allowing you to treat subflows as normal async functions.
514
+
515
+ ```python
516
+ from penguiflow.types import Message
517
+
518
+ async def controller(msg: Message, ctx) -> Message:
519
+ playbook_result = await ctx.call_playbook(build_retrieval_playbook, msg)
520
+ return msg.model_copy(update={"payload": playbook_result})
521
+ ```
522
+
523
+ Playbooks are ideal for deploying frequently reused toolchains while keeping the main
524
+ flow focused on high-level orchestration logic.
525
+
526
+ ---
527
+
528
+ ### Visualization
529
+
530
+ Need a quick view of the flow topology? Call `flow_to_mermaid(flow)` to render the graph
531
+ as a Mermaid diagram ready for Markdown or docs tools, or `flow_to_dot(flow)` for a
532
+ Graphviz-friendly definition. Both outputs annotate controller loops and the synthetic
533
+ OpenSea/Rookery boundaries so you can spot ingress/egress paths at a glance:
534
+
535
+ ```python
536
+ from penguiflow import flow_to_dot, flow_to_mermaid
537
+
538
+ print(flow_to_mermaid(flow, direction="LR"))
539
+ print(flow_to_dot(flow, rankdir="LR"))
540
+ ```
541
+
542
+ See `examples/visualizer/` for a runnable script that exports Markdown and DOT files for
543
+ docs or diagramming pipelines.
544
+
545
+ ---
546
+
547
+ ## 🛡️ Reliability & Observability
548
+
549
+ * **NodePolicy**: set validation scope plus per-node timeout, retries, and backoff curves.
550
+ * **Per-trace metrics**: cancellation events include `trace_pending`, `trace_inflight`,
551
+ `q_depth_in`, `q_depth_out`, and node fan-out counts for richer observability.
552
+ * **Structured `FlowEvent`s**: every node event carries `{ts, trace_id, node_name, event,
553
+ latency_ms, q_depth_in, q_depth_out, attempt}` plus a mutable `extra` map for custom
554
+ annotations.
555
+ * **Remote call telemetry**: `RemoteNode` executions emit extra metrics (latency, request
556
+ and response bytes, context/task identifiers, cancel reasons) so remote hops can be
557
+ traced end-to-end.
558
+ * **Middleware hooks**: subscribe observers (e.g., MLflow) to the structured `FlowEvent`
559
+ stream. See `examples/mlflow_metrics/` for an MLflow integration and
560
+ `examples/reliability_middleware/` for a concrete timeout + retry walkthrough.
561
+ * **`penguiflow-admin` CLI**: inspect or replay stored trace history from any configured
562
+ `StateStore` (`penguiflow-admin history <trace>` or `penguiflow-admin replay <trace>`)
563
+ when debugging distributed runs.
564
+
565
+ ---
566
+
567
+ ## ⚠️ Current Constraints
568
+
569
+ - **In-process runtime**: there is no built-in distribution layer yet. Long-running CPU work should be delegated to your own pools or services.
570
+ - **Registry-driven typing**: nodes default to validation. Provide a `ModelRegistry` when calling `flow.run(...)` or set `validate="none"` explicitly for untyped hops.
571
+ - **Observability**: structured `FlowEvent` callbacks and the `penguiflow-admin` CLI power
572
+ local debugging; integrations with third-party stacks (OTel, Prometheus, Datadog) remain
573
+ DIY. See the MLflow middleware example for a lightweight pattern.
574
+ - **Roadmap**: follow-up releases focus on optional distributed backends, deeper observability integrations, and additional playbook patterns. Contributions and proposals are welcome!
575
+
576
+ ---
577
+
578
+ ## 📊 Benchmarks
579
+
580
+ Lightweight benchmarks live under `benchmarks/`. Run them via `uv run python benchmarks/<name>.py`
581
+ to capture baselines for fan-out throughput, retry/timeout overhead, and controller
582
+ playbook latency. Copy them into product repos to watch for regressions over time.
583
+
584
+ ---
585
+
586
+ ## 🔮 Roadmap
587
+
588
+ * **v2 (current)**: streaming, per-trace cancellation, deadlines/budgets, metadata propagation, observability hooks, visualizer, routing policies, traceable errors, and FlowTestKit.
589
+ * **Future**: optional distributed runners, richer third-party observability adapters, and opinionated playbook templates.
590
+
591
+ ---
592
+
593
+ ## 🧪 Testing
594
+
595
+ ```bash
596
+ pytest -q
597
+ ```
598
+
599
+ * Unit tests cover core runtime, type safety, routing, retries.
600
+ * Example flows under `examples/` are runnable end-to-end.
601
+
602
+ ---
603
+
604
+ ## 🐧 Naming Glossary
605
+
606
+ * **Node**: an async function + metadata wrapper.
607
+ * **Floe**: an edge (queue) between nodes.
608
+ * **Context**: context passed into each node to fetch/emit.
609
+ * **OpenSea** 🌊: ingress context.
610
+ * **Rookery** 🐧: egress context.
611
+
612
+ ---
613
+
614
+ ## 📖 Examples
615
+
616
+ * `examples/quickstart/`: hello world pipeline.
617
+ * `examples/routing_predicate/`: branching with predicates.
618
+ * `examples/routing_union/`: discriminated unions with typed branches.
619
+ * `examples/fanout_join/`: split work and join with `join_k`.
620
+ * `examples/map_concurrent/`: bounded fan-out work inside a node.
621
+ * `examples/controller_multihop/`: dynamic multi-hop agent loop.
622
+ * `examples/reliability_middleware/`: retries, timeouts, and middleware hooks.
623
+ * `examples/mlflow_metrics/`: structured `FlowEvent` export to MLflow (stdout fallback).
624
+ * `examples/playbook_retrieval/`: retrieval → rerank → compress playbook.
625
+ * `examples/trace_cancel/`: per-trace cancellation propagating into a playbook.
626
+ * `examples/streaming_llm/`: mock LLM emitting streaming chunks to an SSE sink.
627
+ * `examples/metadata_propagation/`: attaching and consuming `Message.meta` context.
628
+ * `examples/visualizer/`: exports Mermaid + DOT diagrams with loop/subflow annotations.
629
+
630
+ ---
631
+
632
+ ## 🤝 Contributing
633
+
634
+ * Keep the library **lightweight and generic**.
635
+ * Product-specific playbooks go into `examples/`, not core.
636
+ * Every new primitive requires:
637
+
638
+ * Unit tests in `tests/`
639
+ * Runnable example in `examples/`
640
+ * Docs update in README
641
+
642
+ ---
643
+
644
+ ## License
645
+
646
+ MIT
@@ -0,0 +1,25 @@
1
+ penguiflow/__init__.py,sha256=Ik7scO1qMwAI0iTL2ASII_z2jl7kpkUAhsQ2yAuUstI,1944
2
+ penguiflow/admin.py,sha256=093xFkE4bM_2ZhLrzhrEUKtmKHi_yVfMPyaGfwi1rcA,5382
3
+ penguiflow/bus.py,sha256=mb29509_n97A6zwC-6EDpYorfAWFSpwqsMu_WeZhLE8,732
4
+ penguiflow/core.py,sha256=LRE2TUkAXhiVEO6trD1T1NP_7PRwTFy4MLvzVW0pN24,52631
5
+ penguiflow/errors.py,sha256=mXpCqZ3zdvz7J7Dck_kcw2BGTIm9yrJAjxp_L8KMY7o,3419
6
+ penguiflow/metrics.py,sha256=RW76sYDebzix6Hf7mDemOj-4SwsnILu5YVEoUHNUKuA,3675
7
+ penguiflow/middlewares.py,sha256=4mQgkPowTs6II-_UvIFGyMUTvzhxwDYUicvJMpNNlPU,341
8
+ penguiflow/node.py,sha256=0NOs3rU6t1tHNNwwJopqzM2ufGcp82JpzhckynWBRqs,3563
9
+ penguiflow/patterns.py,sha256=qtzRSNRKxV5_qEPXhffd15PuCZs0YnoGF80nNUsrcxw,5512
10
+ penguiflow/policies.py,sha256=3w8ionnpTyuA0ZCc3jPpB011L7_i1qlbiO6escY024s,4385
11
+ penguiflow/registry.py,sha256=4lrGDMFjM7c8pfZFc_YG0YHg-F80JyF4c-j0UbAf150,1419
12
+ penguiflow/remote.py,sha256=0-2aW48P8OB8KLEC_7_F_RHtzVJk3huyAMBGdXjmWeA,16426
13
+ penguiflow/state.py,sha256=fBY5d_48hR4XHWVG08FraaQ7u4IVPJwooewfVLmzu1Q,1773
14
+ penguiflow/streaming.py,sha256=RKMm4VfaDA2ceEM_pB2Cuhmpwtdcjj7og-kjXQQDcbc,3863
15
+ penguiflow/testkit.py,sha256=wNPqLldHjoq6q7RItSEdKLveSHqXkNj1HAS_FTZDxxs,8581
16
+ penguiflow/types.py,sha256=Fl56-b7OwIEUbPMDD1CY09nbOG_tmBw3FUhioojeG5M,1503
17
+ penguiflow/viz.py,sha256=KbBb9kKoL223vj0NgJV_jo5ny-0RTc2gcSBACm0jG8w,5508
18
+ penguiflow-2.1.0.dist-info/licenses/LICENSE,sha256=JSvodvLXxSct_kI9IBsZOBpVKoESQTB_AGbkClwZ7HI,1065
19
+ penguiflow_a2a/__init__.py,sha256=JuK_ov06yS2H97D2OVXhgX8LcgdOqE3EujUPaDKaduc,342
20
+ penguiflow_a2a/server.py,sha256=VMBO-oGjB6Z9mtRBU0z7ZFGprDUC_kihZJukh3budbs,25932
21
+ penguiflow-2.1.0.dist-info/METADATA,sha256=ep-_5zifZ9G-25TSfCzctVMsiom1QdrZXx_b0tZAaog,24706
22
+ penguiflow-2.1.0.dist-info/WHEEL,sha256=_zCd3N1l69ArxyTb8rzEoP9TpbYXkqRFSNOD5OuxnTs,91
23
+ penguiflow-2.1.0.dist-info/entry_points.txt,sha256=F2KxANLEVGRbpWLmcHcvYrTVLWbKWdmk3VOe98a7t9I,59
24
+ penguiflow-2.1.0.dist-info/top_level.txt,sha256=K-fTwLA14n0u_LDxDBCV7FmeBnJffhTOtUbTtOymQns,26
25
+ penguiflow-2.1.0.dist-info/RECORD,,
@@ -0,0 +1,2 @@
1
+ [console_scripts]
2
+ penguiflow-admin = penguiflow.admin:main
@@ -0,0 +1,2 @@
1
+ penguiflow
2
+ penguiflow_a2a