py-data-engine 0.1.0__py3-none-any.whl

data_engine/docs/sphinx_source/guides/configuring-flows.md

@@ -0,0 +1,185 @@
+# Configuring Flows
+
+Per-flow configuration lives in the fluent `Flow` chain, not in TOML.
+
+That is an important design choice:
+
+- the runtime shape of a flow belongs in the authored `Flow(...)` definition
+- workspace-local TOML in `config/` is for step logic and runtime parameters consumed by your code
+- there is no separate "expand this flow into several configured variants" layer after `build()`
+
+## Core fields
+
+```python
+Flow(group="Claims")
+```
+
+`group` is author-defined. The flow-module filename provides the flow identity.
+
+Use `group` to cluster related flows in the UI and runtime model. A good rule of thumb is that a group should mean "these flows belong to the same operator-facing area of work."
+
+## Watching
+
+Single-file polling:
+
+```python
+Flow(group="Settings").watch(
+    mode="poll",
+    source="../../example_data/Settings/single_watch.xlsx",
+    interval="5s",
+).mirror(
+    root="../../example_data/Output/example_single_watch",
+)
+```
+
+Directory polling:
+
+```python
+Flow(group="Claims").watch(
+    mode="poll",
+    source="../../example_data/Input/claims_flat",
+    interval="5s",
+    extensions=[".xlsx", ".xls", ".xlsm"],
+    settle=1,
+).mirror(
+    root="../../example_data/Output/example_mirror",
+)
+```
+
+Scheduled batch runs:
+
+```python
+Flow(group="Analytics").watch(
+    mode="schedule",
+    run_as="batch",
+    interval="15m",
+    source="../../example_data/Input/claims_flat",
+).mirror(
+    root="../../example_data/Output/example_summary",
+)
+
+Flow(group="Settings").watch(
+    mode="schedule",
+    run_as="batch",
+    time="10:31",
+    source="../../example_data/Settings/single_watch.xlsx",
+).mirror(
+    root="../../example_data/Output/example_schedule",
+)
+
+Flow(group="Settings").watch(
+    mode="schedule",
+    run_as="batch",
+    time=["08:15", "14:45"],
+    source="../../example_data/Settings/single_watch.xlsx",
+)
+```
+
+What watching owns:
+
+- source selection
+- ledger-backed source freshness tracking
+- extension filtering for poll directory sources
+- settle/debounce behavior for poll flows
+- whether runtime executes per file or as one root-level batch via `run_as=`
+
+What watching does not own:
+
+- dataframe reads
+- dataframe transforms
+- database work
+- output writing
+
+That separation is what keeps `watch(...)` readable. It tells the engine when and why to run, not how to do the underlying data work.
+
+`watch(mode="schedule", ...)` accepts exactly one of:
+
+- `interval="10m"`
+- `time="HH:MM"`
+- `time=["08:15", "14:45"]`
+
+It may also bind an optional `source=...` path for recurring jobs.
+
+### `run_as`
+
+`run_as` controls what the runtime treats as one unit of work.
+
+Common values are:
+
+- `run_as="individual"`: one run per concrete source file
+- `run_as="batch"`: one run at the watched root
+
+Use `individual` when each source file should be processed independently.
+
+Use `batch` when the flow should reason about the watched source as one collection, such as "all current workbooks in this folder."
+
+### Poll-specific options
+
+`extensions=` limits which files in a polled directory participate in freshness checks and execution.
+
+`settle=` adds debounce behavior so the engine does not immediately react to a file that is still being written by another process.
+
+## Mirror bindings
+
+Use `mirror(root=...)` when a flow needs source-relative output routing.
+
+Inside steps:
+
+```python
+context.mirror.with_suffix(".parquet")
+context.mirror.file("summary.json")
+context.mirror.namespaced_file("open_claims.parquet")
+context.mirror.root_file("analytics.duckdb")
+```
+
+`mirror(...)` does not write files. It only defines the output namespace available at runtime.
+
+If a flow has no natural mirrored outputs, you do not need `mirror(...)`.
+
+If a flow writes several related outputs, `mirror(...)` is usually the cleanest way to keep them organized without scattering path math through your steps.
+
+## Batch workflows
+
+Use `collect(...)` and `map(...)` or `step_each(...)` together for folder-oriented processing:
+
+```python
+Flow(group="Analytics") \
+    .watch(mode="schedule", run_as="batch", interval="15m", source="../../example_data/Input/claims_flat") \
+    .collect([".xlsx"], save_as="claim_files") \
+    .map(read_claims, use="claim_files", save_as="claim_frames")
+```
+
+`map(...)` is the per-item stage in that pipeline, and `step_each(...)` is the equivalent alias. Both raise immediately when the batch is empty.
+
+This is the standard batch shape:
+
+1. watch a directory or scheduled source root
+2. collect matching files into a `Batch`
+3. map one callable across each file
+4. switch back to `step(...)` once you want to reason about the combined result
+
+## Configuring step labels and saved objects
+
+Flow configuration also includes the names and labels you assign in the chain.
+
+Examples:
+
+```python
+(
+    Flow(group="Claims")
+    .step(read_claims, save_as="raw_df")
+    .step(clean_claims, use="raw_df", save_as="clean_df")
+    .step(write_output, use="clean_df", label="Write Parquet")
+)
+```
+
+Those fields affect the authoring experience directly:
+
+- `save_as=` creates stable names for later steps and notebook previews
+- `use=` loads one of those saved names into `context.current`
+- `label=` controls the display name in the UI
+
+If you are deciding where a piece of information belongs:
+
+- if it shapes orchestration, put it in the `Flow` chain
+- if it shapes step logic, put it in your code or in `context.config`

data_engine/docs/sphinx_source/guides/core-concepts.md

@@ -0,0 +1,208 @@
+# Core Concepts
+
+## Flow
+
+A `Flow` is an immutable definition with:
+
+- `group`
+- an optional trigger via `watch(...)`
+- an optional mirrored output binding via `mirror(...)`
+- ordered generic steps
+
+```python
+from data_engine import Flow
+
+flow = Flow(group="Claims")
+```
+
+The flow-module filename is the flow identity used for discovery and runtime bookkeeping. `group` is the author-controlled grouping visible in the UI.
+
+## Runtime modes
+
+Manual:
+
+- no trigger configured
+- `run_once()` executes the steps once with `context.current = None`
+- useful for button-driven operator runs or preview-oriented flows
+
+Poll:
+
+- source-driven execution over either one file or a directory of files
+- the runtime compares the current source file signature against the persisted runtime ledger
+- the first step sees the active input through `context.source`
+- startup backlog handling is based on persisted ledger state for each source version
+- intermediate saved objects do not participate in staleness checks
+
+Schedule:
+
+- interval-driven via `watch(mode="schedule", interval="15m")`
+- or wall-clock via `watch(mode="schedule", time="10:31")`
+- `time` may also be a collection such as `["08:15", "14:45"]`
+- may optionally bind a `source=...` path for recurring jobs
+
+The distinction between poll and schedule is important:
+
+- poll is source freshness driven
+- schedule is time driven
+
+You can combine scheduled execution with a source binding when the flow should run on a schedule but still read from a known source root.
+
+## Step
+
+Each `step(...)` is one callable:
+
+```python
+def step(context) -> object:
+    ...
+```
+
+The return value always becomes `context.current`.
+
+This is the main design boundary:
+
+- the fluent API orchestrates runtime behavior
+- native libraries perform the actual data and file work
+
+That means Data Engine is intentionally not trying to replace Polars, DuckDB, pathlib, or your Python helper code. It coordinates them.
+
+## Saved objects
+
+Steps can save and reuse values:
+
+```python
+(
+    Flow(group="Docs")
+    .step(read_claims, save_as="raw_df")
+    .step(clean_claims, use="raw_df", save_as="clean_df")
+    .step(write_output, use="clean_df")
+)
+```
+
+- `use="name"` loads `context.objects["name"]` into `context.current`
+- `save_as="name"` stores the returned value into `context.objects["name"]`
+
+In notebooks, those saved names are also the easiest way to inspect intermediates:
+
+```python
+build().preview(use="clean_df").head(10)
+```
+
+This is one of the most useful parts of the authoring model:
+
+- `current` gives you the current object in the pipeline
+- `objects` gives you stable named waypoints
+
+That makes it easy to structure flows around a few explicit intermediate states rather than one long opaque chain.
+
+## Batch mapping
+
+`collect(...)` and `map(...)` are the batch-oriented authoring tools.
+
+```python
+def read_claims(file_ref):
+    return pl.read_excel(file_ref.path)
+
+
+def combine_claims(context):
+    return pl.concat(context.current, how="vertical_relaxed")
+
+
+flow = (
+    Flow(group="Analytics")
+    .watch(mode="schedule", run_as="batch", interval="15m", source="../../example_data/Input/claims_flat")
+    .collect([".xlsx"], save_as="claim_files")
+    .map(read_claims, use="claim_files", save_as="claim_frames")
+    .step(combine_claims, use="claim_frames")
+)
+```
+
+Use `map(...)` when the same callable should run once per batch item instead of once per whole flow. `map(...)` raises immediately when the batch is empty.
+
+Batch mapping is especially useful when you want to:
+
+- read many files into many dataframes
+- validate one file at a time
+- emit one lightweight record per source item before combining
+
+Use a normal `step(...)` when the callable should reason about the batch as a whole.
+
+## Source and mirror namespaces
+
+The runtime exposes two structured path namespaces:
+
+- `context.source`
+- `context.mirror`
+
+Examples:
+
+```python
+context.source.path
+context.source.with_extension(".json")
+context.source.with_suffix(".json")
+context.source.file("notes.json")
+context.source.namespaced_file("notes.json")
+context.source.root_file("lookup.csv")
+
+context.mirror.with_extension(".parquet")
+context.mirror.with_suffix(".parquet")
+context.mirror.file("open_claims.parquet")
+context.mirror.namespaced_file("open_claims.parquet")
+context.mirror.root_file("analytics.duckdb")
+```
+
+`context.source` resolves read-side paths. `context.mirror` resolves write-ready output paths.
+
+The important difference is:
+
+- `source` is about where the active input lives
+- `mirror` is about where outputs for that input should go
+
+That lets you keep path logic readable and source-aware without hand-building relative paths in every step.
+
+Examples of common patterns:
+
+- read a sidecar file beside the current source with `context.source.file("notes.json")`
+- write one mirrored parquet beside the source shape with `context.mirror.with_suffix(".parquet")`
+- write multiple outputs for the same source with `context.mirror.namespaced_file(...)`
+- write a stable root-level artifact such as a snapshot or DuckDB file with `context.mirror.root_file(...)`
+
+## Discovery
+
+The desktop UI and Python entrypoints discover flows from compiled flow modules.
+
+Each discovered flow module contributes:
+
+- a module name
+- optional `DESCRIPTION`
+- `build() -> Flow`
+
+The flow-module filename/module name is the flow identity surfaced in discovery and execution. The UI uses `Flow.label` when present, otherwise it derives a readable title from that internal name.
+
+That discovered `Flow` object is what the UI inspects for:
+
+- grouping
+- step labels
+- runtime mode
+- source and mirror bindings
+
+The app does not maintain a second hidden config layer that mutates flow behavior after discovery. The authored `Flow` is the real contract the runtime and UI are looking at.
+
+## Workspaces
+
+Flows do not exist in isolation. They are discovered from the currently selected authored workspace.
+
+An authored workspace typically contains:
+
+- `flow_modules/`
+- `flow_modules/flow_helpers/`
+- `config/`
+- `databases/`
+
+The desktop app binds to one workspace at a time. When the selected workspace changes, the app reloads:
+
+- discovered flows
+- local runtime state
+- daemon control state
+- visible runs and logs
+
+For the control and state model behind that, see [App Runtime and Workspaces](app-runtime-and-workspaces.md).

data_engine/docs/sphinx_source/guides/database-methods.md

@@ -0,0 +1,107 @@
+# Database Methods
+
+There is no first-class database sub-chain. Use DuckDB directly inside step callables, usually with a workspace-local database path from `context.database(...)`.
+
+If you want common warehouse-style shortcuts, see [DuckDB Helpers](duckdb-helpers.md). That helper layer covers several repeated patterns without taking over general SQL authoring.
+
+That is intentional. The core API deliberately avoids hiding connection ownership, transactions, or query semantics behind a special fluent DSL.
+
+In practice, that means:
+
+- Data Engine gives you a conventional path
+- your step opens and closes the database connection
+- normal DuckDB and Python rules apply
+
+## `context.database(...)`
+
+`context.database(name)` returns a path beneath the current authored workspace's `databases/` folder.
+
+Examples:
+
+```python
+context.database("analytics.duckdb")
+context.database("claims/analytics.duckdb")
+```
+
+Those resolve to:
+
+- `workspaces/<workspace_id>/databases/analytics.duckdb`
+- `workspaces/<workspace_id>/databases/claims/analytics.duckdb`
+
+Rules:
+
+- the path must be relative
+- parent directories are created automatically
+- the helper is only available for authored workspace flows
+- it does not create a connection for you
+
+That last point is important. Returning the path avoids hidden connection lifetime issues and keeps the behavior obvious.
+
+## Example
+
+```python
+import duckdb
+import polars as pl
+
+from data_engine import Flow
+
+
+def read_claims(file_ref):
+    return pl.read_excel(file_ref.path)
+
+
+def build_source(context):
+    return pl.concat(context.current, how="vertical_relaxed")
+
+
+def summarize(context):
+    conn = duckdb.connect(context.database("analytics.duckdb"))
+    try:
+        conn.register("input", context.current)
+        return conn.sql(
+            """
+            select workflow, count(*) as row_count
+            from input
+            group by workflow
+            """
+        ).pl()
+    finally:
+        conn.close()
+
+
+def build():
+    return (
+        Flow(group="Analytics")
+        .watch(
+            mode="schedule",
+            run_as="batch",
+            interval="15m",
+            source="../../example_data/Input/claims_flat",
+        )
+        .mirror(root="../../example_data/Output/example_summary")
+        .collect([".xlsx"], save_as="claim_files")
+        .map(read_claims, use="claim_files", save_as="claim_frames")
+        .step(build_source, use="claim_frames", save_as="raw_df")
+        .step(summarize, use="raw_df", save_as="summary_df")
+    )
+```
+
+This keeps the flow API small while still letting flow modules use native SQL and native DuckDB connections.
+
+## Good patterns
+
+- open the connection inside the step that needs it
+- close the connection in `finally:`
+- keep the path stable when you want incremental or append-oriented databases
+- use subfolders such as `claims/analytics.duckdb` when one workspace owns several related databases
+
+## A note on mirror vs database paths
+
+If the database is a durable workspace-local asset, prefer `context.database(...)`.
+
+If the database is really just another output artifact produced by one mirrored source flow, `context.mirror.root_file("analytics.duckdb")` can still be appropriate.
+
+The difference is mostly semantic:
+
+- `context.database(...)` says "this belongs to the workspace as a local database"
+- `context.mirror...` says "this belongs to this flow's output namespace"