py-data-engine 0.1.0__py3-none-any.whl

data_engine/docs/sphinx_source/guides/flow-methods.md

@@ -0,0 +1,206 @@
+# Flow Methods
+
+This page covers the small author-facing `Flow` surface.
+
+```python
+from data_engine import Flow
+```
+
+## `Flow(group)`
+
+Create a new immutable flow definition.
+
+```python
+flow = Flow(group="Claims")
+```
+
+Rules:
+
+- `group` must be a non-empty string
+- the flow-module filename provides the flow identity
+- the returned object is immutable, so each fluent call returns a new `Flow`
+
+Immutability matters because it keeps authoring predictable. Each chained call produces a new flow definition rather than mutating hidden shared state.
+
+## `watch(...)`
+
+Configure a runtime trigger for manual, poll, or schedule execution.
+
+```python
+flow = flow.watch(
+    mode="poll",
+    source="../../example_data/Input/claims_flat",
+    interval="5s",
+    extensions=[".xlsx", ".xlsm"],
+    settle=1,
+)
+```
+
+```python
+flow = flow.watch(
+    mode="poll",
+    source="../../example_data/Settings/single_watch.xlsx",
+    interval="5s",
+)
+```
+
+```python
+flow = flow.watch(mode="schedule", run_as="batch", interval="15m")
+flow = flow.watch(mode="schedule", run_as="batch", interval="15m", source="../../example_data/Input/claims_flat")
+flow = flow.watch(mode="schedule", run_as="batch", time="10:31", source="../../example_data/Settings/single_watch.xlsx")
+flow = flow.watch(mode="schedule", run_as="batch", time=["08:15", "14:45"])
+flow = flow.watch(mode="manual")
+```
+
+Rules:
+
+- `mode` must be one of `manual`, `poll`, or `schedule`
+- `run_as` defaults to `individual`
+- `run_as="individual"` means one run per concrete source file
+- `run_as="batch"` means one run at the watched root
+- poll flows require `source=` and `interval=`
+- schedule flows accept exactly one of `interval=` or `time=`
+- `time` accepts either one `HH:MM` string or a collection of `HH:MM` strings
+- `extensions` and `settle` are poll-only options
+- missing or bad paths fail now and recover later when the path becomes valid
+- poll freshness compares the current source file signature against the runtime ledger
+
+Practical guidance:
+
+- use `manual` for explicit button-driven flows
+- use `poll` when the source changing should be the trigger
+- use `schedule` when time should be the trigger
+- use `run_as="batch"` when the flow should reason about a folder or root as one unit
+- use `run_as="individual"` when each source file should become its own run
+
+`watch(...)` is where you describe orchestration intent, not transformation logic.
+
+## `mirror(root=...)`
+
+Bind a mirrored output namespace rooted at one directory.
+
+```python
+flow = flow.mirror(root="../../example_data/Output/example_mirror")
+```
+
+`mirror(...)` does not write files. It defines the output namespace exposed later through `context.mirror`.
+
+You can omit `mirror(...)` entirely if the flow has no need for a mirrored output namespace.
+
+## `step(fn, use=None, save_as=None, label=None)`
+
+Add one generic callable step.
+
+```python
+flow = flow.step(read_claims, save_as="raw_df")
+flow = flow.step(clean_claims, use="raw_df", save_as="clean_df")
+flow = flow.step(write_output, use="clean_df", label="Write Parquet")
+```
+
+Rules:
+
+- `fn` must be callable
+- `fn` must accept exactly one `context` parameter
+- `use=` selects a previously saved object
+- `save_as=` stores the returned object
+- `label=` overrides the UI display name
+
+The return value always becomes `context.current`.
+
+This is the default workhorse method. Most flows are easiest to read when they are mostly made of `step(...)` with occasional `collect(...)` and `map(...)` where batching is truly needed.
+
+## `map(fn, use=None, save_as=None, label=None)`
+
+Map one callable across the current batch.
+
+```python
+flow = flow.collect([".pdf"])
+flow = flow.map(validate_pdf)
+flow = flow.map(validate_pdf_with_context, label="Validate Pdf")
+```
+
+```python
+def validate_pdf(file_ref):
+    return {"name": file_ref.name, "ok": file_ref.exists()}
+
+
+def validate_pdf_with_context(context, file_ref):
+    return {"flow": context.flow_name, "name": file_ref.name}
+```
+
+Rules:
+
+- `map()` expects the current value to be iterable
+- `fn` may accept either `(item)` or `(context, item)`
+- the mapped results are returned as a `Batch`
+- `map()` raises when the current batch is empty
+- `use=`, `save_as=`, and `label=` work the same way they do for `step()`
+
+Reach for `map(...)` when the same callable should run once per collected item. If the callable should reason about the whole collection, switch back to a normal `step(...)`.
+
+## `step_each(fn, use=None, save_as=None, label=None)`
+
+`step_each(...)` is an alias for `map(...)`.
+
+Use whichever reads better in the flow module:
+
+```python
+flow = flow.map(read_claims)
+flow = flow.step_each(read_claims)
+```
+
+## `collect(extensions, root=None, recursive=False, use=None, save_as=None, label=None)`
+
+Collect matching files into a `Batch` of `FileRef` items.
+
+```python
+flow = flow.collect([".xlsx"])
+flow = flow.collect([".pdf"], recursive=True)
+```
+
+Behavior:
+
+- uses `root=` when provided
+- otherwise falls back to `context.source.root`
+- returns a `Batch`, not a raw list
+- each item exposes `.name`, `.path`, `.stem`, `.suffix`, and `.parent`
+
+If `root=` is omitted, the runtime falls back to the current source root. That is often the cleanest choice for poll or scheduled batch flows already bound to a source.
+
+## `run_once()`
+
+Run the flow one time and return the completed contexts.
+
+Use this when you want a one-off Python-driven execution rather than continuous watching.
+
+## `run()`
+
+Start continuous execution for watched poll or schedule flows.
+
+This is the entrypoint behind long-lived runtime behavior.
+
+## `preview(use=None)`
+
+Run one flow for notebook inspection and return a real object.
+
+```python
+build().preview()
+build().preview(use="raw_df").head(10)
+build().preview(use="claim_frames")
+```
+
+Behavior:
+
+- without `use=`, returns the final `context.current`
+- with `use="name"`, runs only until `save_as="name"` exists
+- returns the real saved object, so dataframe methods like `.head(10)` work naturally
+- avoids running later write/debug steps once the requested saved object is available
+- if a poll flow would have several startup source files, preview uses the first deterministic source candidate for notebook inspection rather than trying to preview every file at once
+
+`preview(...)` is especially useful while authoring notebook-backed flows because it lets you stop at a meaningful intermediate instead of running the whole flow to the final writer step every time.
+
+## `show()`
+
+Preview the single current result from a one-off flow.
+
+Use this for quick interactive inspection when the final current value itself is the thing you want to see.

data_engine/docs/sphinx_source/guides/getting-started.md

@@ -0,0 +1,271 @@
+# Getting Started
+
+This guide is for someone new to the code-defined Data Engine API and desktop app.
+
+By the end, you should understand:
+
+- what a flow is
+- where flow modules live
+- what a workspace contains
+- how discovery and runtime execution work at a high level
+- how to run a first flow end to end
+- how batch workflows fit into the model
+
+## The mental model
+
+Data Engine has one source of truth for per-flow behavior: the `Flow` returned by `build()`.
+
+In practice:
+
+- the flow module defines the flow name, group, runtime mode, and ordered steps
+- step functions do real work with native libraries such as Polars, DuckDB, and plain Python
+- the desktop app discovers those flow modules inside the selected workspace and shows them as configurable runnable flows
+
+The engine does not hide the real work behind a DSL. The fluent API owns orchestration, while the step callables own your actual business logic.
+
+## The basic workspace layout
+
+A typical authored workspace looks like this:
+
+```text
+workspaces/
+  example_workspace/
+    flow_modules/
+    flow_modules/flow_helpers/
+    config/
+    databases/
+    .workspace_state/
+```
+
+The parts you will usually author directly are:
+
+- `flow_modules/`: runnable flows in `.py` or `.ipynb`
+- `flow_modules/flow_helpers/`: reusable helper modules imported from flows
+- `config/`: workspace-local TOML files available through `context.config`
+- `databases/`: a conventional home for workspace-local databases used through `context.database(...)`
+
+The app can provision that shape for you without overwriting existing content.
+
+## Where flow module sources live
+
+Flow module sources are authored in:
+
+- `workspaces/<workspace_id>/flow_modules/<name>.ipynb`
+- `workspaces/<workspace_id>/flow_modules/<name>.py`
+
+Reusable helper modules live in:
+
+- `workspaces/<workspace_id>/flow_modules/flow_helpers/<name>.py`
+
+Compiled runtime modules are generated into machine-local artifacts rather than into the authored workspace itself.
+
+Each flow module should export:
+
+- optional `DESCRIPTION`
+- `build() -> Flow`
+
+Display titles come from `Flow(label=...)` when provided. Otherwise the UI derives them from the flow-module filename.
+
+## Your first flow
+
+A minimal scheduled flow can create data in memory and write it out:
+
+```python
+from data_engine import Flow
+import polars as pl
+
+
+def build_dates(context):
+    return pl.DataFrame({"day": [1, 2, 3]})
+
+
+def write_dates(context):
+    output = context.mirror.file("dates.parquet")
+    context.current.write_parquet(output)
+    return output
+
+
+def build():
+    return (
+        Flow(group="Reference")
+        .watch(mode="schedule", run_as="batch", interval="1h")
+        .mirror(root="../../example_data/Output/date_dimension")
+        .step(build_dates, save_as="dates_df")
+        .step(write_dates, use="dates_df", label="Write Parquet")
+    )
+```
+
+That example shows the full shape:
+
+1. create `Flow(group=...)`
+2. attach a runtime mode with `watch(...)`
+3. optionally attach `mirror(...)`
+4. add ordered `step(...)` callables
+5. return the built flow from `build()`
+
+The return value from each step becomes `context.current`, so later steps can keep operating on the current object or reach back to previously saved objects through `use=`.
+
+## What the app actually does with that flow
+
+Once the flow is discovered, the desktop app uses it for:
+
+- grouping and labels in the home view
+- deciding whether the flow is manual, poll, or schedule
+- deciding whether the flow participates in the engine
+- rendering step names and inspectable outputs
+- manual runs and engine runs for the selected workspace
+
+The app itself binds to one workspace at a time, so when you switch workspaces, the discovered flows, runtime ledger, daemon state, and visible runs all switch with it.
+
+## A starter-style polling flow
+
+This shape maps directly to starter flows such as `example_mirror` and `example_poll`:
+
+```python
+from data_engine import Flow
+import polars as pl
+
+
+def read_claims(context):
+    return pl.read_excel(context.source.path)
+
+
+def keep_open(context):
+    return context.current.filter(pl.col("status") == "OPEN")
+
+
+def write_target(context):
+    output = context.mirror.with_suffix(".parquet")
+    context.current.write_parquet(output)
+    return output
+
+
+def build():
+    return (
+        Flow(group="Claims")
+        .watch(
+            mode="poll",
+            source="../../example_data/Input/claims_dated",
+            interval="5s",
+            extensions=[".xlsx", ".xlsm"],
+            settle=1,
+        )
+        .mirror(root="../../example_data/Output/example_poll")
+        .step(read_claims, save_as="raw_df")
+        .step(keep_open, use="raw_df", save_as="filtered_df")
+        .step(write_target, use="filtered_df", label="Write Parquet")
+    )
+```
+
+This is a good first mental model for source-driven flows:
+
+- `watch(...)` tells the runtime what to listen to
+- `context.source` tells the step which concrete file is active
+- `mirror(...)` defines where mirrored outputs belong
+- returning the written path makes the result inspectable in the UI
+
+## Batch-oriented files
+
+When you want a folder of files as one runtime object, use `Flow.collect(...)` and either `Flow.map(...)` or `Flow.step_each(...)`.
+
+```python
+from data_engine import Flow
+
+
+def validate_pdf(file_ref):
+    return {"name": file_ref.name, "ok": file_ref.exists()}
+
+
+def summarize_results(context):
+    return tuple(item["name"] for item in context.current if item["ok"])
+
+
+def build():
+    return (
+        Flow(group="Claims")
+        .watch(mode="schedule", run_as="batch", interval="15m", source="../../example_data/Input/pdfs")
+        .collect([".pdf"], save_as="pdf_files")
+        .map(validate_pdf, use="pdf_files", save_as="pdf_results")
+        .step(summarize_results, use="pdf_results")
+    )
+```
+
+`Flow.collect(...)` returns a `Batch` of `FileRef` items.
+
+`Flow.map(...)` runs one callable per item and returns a new `Batch`.
+
+`Flow.step_each(...)` is the same operation with a name that can read more clearly in some flows.
+
+If the batch is empty, both forms raise immediately. That behavior is intentional so batch flows fail loudly instead of silently producing ambiguous "nothing happened" results.
+
+## Running flows from Python
+
+Load one discovered flow:
+
+```python
+from data_engine import load_flow
+
+built = load_flow("example_poll")
+results = built.run_once()
+```
+
+Discover everything the workspace exposes:
+
+```python
+from data_engine import discover_flows, run
+
+flows = discover_flows()
+run(*flows)
+```
+
+Notebook-authored flows also support preview-oriented authoring:
+
+```python
+build().preview()
+build().preview(use="raw_df")
+```
+
+That is often the fastest way to sanity-check a flow while you are still writing it.
+
+For poll flows that watch a folder, `preview(...)` uses one deterministic startup source as a representative notebook preview rather than trying to run every discovered file.
+
+## Manual, poll, and schedule at a glance
+
+### Manual
+
+- `watch(mode="manual")`
+- `context.current` starts as `None`
+- useful for ad hoc or UI-driven runs
+- does not require a source binding
+
+### Poll
+
+- `watch(mode="poll", ...)`
+- watches either one file or a directory of source files
+- the first step receives the active source through `context.source`
+- freshness compares the current source file signature against the runtime ledger
+- `extensions=` and `settle=` only apply here
+
+### Schedule
+
+- `watch(mode="schedule", ...)`
+- runs on an interval or on one or more wall-clock times
+- supports one `time="HH:MM"` value or a collection of times
+- often starts by building data in memory or loading from a known source root
+
+## A few good habits early
+
+- keep import-time code side-effect free
+- keep expensive work inside steps, not at module import
+- return output paths from writer steps when you want the UI `Inspect` action
+- move reusable SQL, parsing helpers, and constants into `flow_modules/flow_helpers/`
+- use `context.config` for workspace-local TOML configuration rather than inventing ad hoc config loading in every flow
+- use `context.database(...)` when you want a conventional workspace-local database path
+
+## Next steps
+
+- Read [Core Concepts](core-concepts.md)
+- Read [Authoring Flow Modules](authoring-flow-modules.md)
+- Read [Flow Methods](flow-methods.md)
+- Read [FlowContext](flow-context.md)
+- Read [App Runtime and Workspaces](app-runtime-and-workspaces.md)