jerry-thomas 0.3.0__py3-none-any.whl → 1.0.0__py3-none-any.whl

datapipeline/templates/plugin_skeleton/README.md

@@ -4,10 +4,10 @@ Minimal plugin skeleton for the Jerry Thomas (datapipeline) framework.
 
 Quick start
 - Initialize a plugin (already done if you’re reading this here):
-- `jerry plugin init --name {{DIST_NAME}}`
+- `jerry plugin init {{DIST_NAME}}`
 - Add a source via CLI (transport-specific placeholders are scaffolded):
-  - File data: `jerry source add -p <provider> -d <dataset> -t fs -f <csv|json|json-lines>`
-  - URL data: `jerry source add -p <provider> -d <dataset> -t url -f <json|json-lines|csv>`
+  - File data: `jerry source add <provider> <dataset> -t fs -f <csv|json|json-lines|pickle>`
+  - HTTP data: `jerry source add <provider>.<dataset> -t http -f <json|json-lines|csv>`
   - Synthetic: `jerry source add -p <provider> -d <dataset> -t synthetic`
 - Edit the generated `config/sources/*.yaml` to fill in the `path`, delimiter, etc.
 - Reinstall after EP changes (pyproject.toml) and restart Python processes:
@@ -15,12 +15,15 @@ Quick start
   - This plugin: `python -m pip install -e .`
 
 Folder layout
-- `config/`
-  - `sources/*.yaml` — raw source definitions (one file per source)
+- `example/`
+  - `project.yaml` — project root (paths, globals, cadence/split)
+  - `dataset.yaml` — feature/target declarations (uses `${group_by}` from globals)
+  - `postprocess.yaml` — postprocess transforms
   - `contracts/*.yaml` — canonical stream definitions
-  - `datasets/<name>/build.yaml` — build configuration (partitioned ids today, more artifacts later)
+  - `sources/*.yaml` — raw source definitions (one file per source)
+  - `tasks/*.yaml` — task specs (schema/scaler/metadata/serve)
 - Every dataset `project.yaml` declares a `name`; reference it via `${project_name}`
-  inside other config files (e.g., `paths.artifacts: ../../build/datasets/${project_name}`) to
+  inside other config files (e.g., `paths.artifacts: ../artifacts/${project_name}`) to
   avoid hard-coding per-dataset directories.
 - `src/{{PACKAGE_NAME}}/`
   - `sources/<provider>/<dataset>/dto.py` — DTO model for the source
@@ -30,52 +33,59 @@ Folder layout
   - `mappers/*.py` — map DTOs → domain records
 
 How loaders work
-- For fs/url, sources use the generic loader entry point:
-  - `loader.entrypoint: "{{COMPOSED_LOADER_EP}}"`
+- For fs/http, sources use the generic loader entry point:
+  - `loader.entrypoint: "{{DEFAULT_IO_LOADER_EP}}"`
 - `loader.args` include `transport`, `format`, and source-specific args (placeholders are provided):
     - fs: `path`, `glob`, `encoding`, plus `delimiter` for csv
-    - url: `url`, `headers`, `encoding`, optional `count_by_fetch`
+    - http: `url`, `headers`, `params`, `encoding`, optional `count_by_fetch`
 - Synthetic sources generate data in-process and keep a small loader stub.
 
 Run data flows
-- Build artifacts once: `jerry build --project config/datasets/default/project.yaml`
-- Preview records (stage 1): `jerry serve --project config/datasets/default/project.yaml --stage 1 --limit 100`
-- Preview features (stage 3): `jerry serve --project config/datasets/default/project.yaml --stage 3 --limit 100`
-- Preview vectors (stage 7): `jerry serve --project config/datasets/default/project.yaml --stage 7 --limit 100`
+- Build artifacts once: `jerry build --project example/project.yaml`
+- Preview records (stage 1): `jerry serve --project example/project.yaml --stage 1 --limit 100`
+- Preview features (stage 3): `jerry serve --project example/project.yaml --stage 3 --limit 100`
+- Preview vectors (stage 7): `jerry serve --project example/project.yaml --stage 7 --limit 100`
 
 Analyze vectors
-- `jerry inspect report   --project config/datasets/default/project.yaml` (console only)
-- `jerry inspect coverage --project config/datasets/default/project.yaml` (writes build/coverage.json)
-- `jerry inspect matrix   --project config/datasets/default/project.yaml --format html` (writes build/matrix.html)
-- `jerry inspect partitions --project config/datasets/default/project.yaml` (writes build/partitions.json)
+- `jerry inspect report   --project example/project.yaml` (console only)
+- `jerry inspect partitions --project example/project.yaml` (writes build/partitions.json)
+- `jerry inspect matrix   --project example/project.yaml --format html` (writes build/matrix.html)
+- `jerry inspect expected --project example/project.yaml` (writes build/expected.txt)
 - Use post-processing transforms in `postprocess.yaml` to keep coverage high
   (history/horizontal fills, constants, or drop rules) before serving vectors.
+  Add `payload: targets` inside a transform when you need to mutate label vectors.
 
 Train/Val/Test splits (deterministic)
 - Configure split mechanics once in your project file:
-  - Edit `config/datasets/default/project.yaml` and set:
+  - Edit `example/project.yaml` and set:
     ```yaml
     globals:
+      group_by: 10m          # dataset cadence; reused as contract cadence
       split:
         mode: hash            # hash|time
         key: group            # group or feature:<id> (entity-stable)
         seed: 42              # deterministic hash seed
         ratios: {train: 0.8, val: 0.1, test: 0.1}
     ```
-- Select the active slice via `config/datasets/default/run.yaml` (or `--keep`):
+- Select the active slice via `example/tasks/serve.<name>.yaml` (or `--keep`):
   ```yaml
-  version: 1
+  kind: serve
+  name: train               # defaults to filename stem when omitted
   keep: train               # any label defined in globals.split; null disables filtering
-  output: print             # serve output default (print|stream|/path)
+  output:
+    transport: stdout       # stdout | fs
+    format: print           # print | json-lines | json | csv | pickle
   limit: 100                # cap vectors per serve run (null = unlimited)
-  include_targets: false    # include dataset.targets when serving
   throttle_ms: null         # sleep between vectors (milliseconds)
+  # visuals: AUTO  # AUTO | TQDM | RICH | OFF
+  # progress: AUTO # AUTO | SPINNER | BARS | OFF
   ```
-- If you prefer separate configs per split, point `project.paths.run` at a folder (e.g., `config/datasets/default/runs/`),
-  drop `train.yaml`, `val.yaml`, etc. inside, and the CLI will run each file in order unless you pass `--run <name>`.
-- Serve examples (change run.yaml or pass `--keep val|test`):
-  - `jerry serve -p config/datasets/default/project.yaml -o stream > train.jsonl`
-  - `jerry serve -p config/datasets/default/project.yaml --keep val -o stream > val.jsonl`
+- Add additional `kind: serve` files (e.g., `serve.val.yaml`, `serve.test.yaml`) and the CLI will run each enabled file in order unless you pass `--run <name>`.
+- Serve examples (change the serve task or pass `--keep val|test`):
+  - `jerry serve -p example/project.yaml --out-transport stdout --out-format json-lines > train.jsonl`
+  - `jerry serve -p example/project.yaml --keep val --out-transport stdout --out-format json-lines > val.jsonl`
+  - Add `--visuals rich --progress bars` for a richer interactive UI; defaults to `AUTO`.
+- For shared workspace defaults (visual renderer, progress display, build mode), drop a `jerry.yaml` next to your workspace root and set `shared.visuals`, `shared.progress`, etc. CLI commands walk up from the current directory to find it.
 - The split is applied at the end (after postprocess transforms), and assignment
   is deterministic (hash-based) with a fixed seed; no overlap across runs.
 
@@ -84,13 +94,47 @@ Key selection guidance
 - `key: feature:<id>` hashes a specific feature value, e.g., `feature:entity_id` or `feature:station_id`, ensuring all vectors for that entity land in the same split (recommended to avoid leakage).
 
 Postprocess expected IDs
-- Build once with `jerry build --project config/datasets/default/project.yaml` (or run `jerry inspect expected …`) to materialize `<paths.artifacts>/expected.txt`.
+- Build once with `jerry build --project config/project.yaml` (or run `jerry inspect expected …`) to materialize `<paths.artifacts>/expected.txt`.
 - Bootstrap registers the artifact; postprocess transforms read it automatically. Per-transform `expected:` overrides are no longer required or supported — the build output is the single source of truth.
 
 Scaler statistics
-- Enable the scaler task in `build.yaml` (default `enabled: true`) to compute mean/std per feature using the configured training split.
+- Jerry computes scaler stats automatically. If you need custom paths or settings, add `tasks/scaler.yaml` and override the defaults.
 - The build writes `<paths.artifacts>/scaler.pkl`; runtime scaling requires this artifact. If it is missing, scaling transforms raise a runtime error.
 
 Tips
 - Keep parsers thin — mirror source schema and return DTOs; use the identity parser only if your loader already emits domain records.
 - Prefer small, composable configs over monolithic ones: one YAML per source is easier to review and reuse.
+
+Composed streams (engineered domains)
+- Declare engineered streams that depend on other canonical streams directly in contracts. The runtime builds each input to stage 4, stream‑aligns by partition+timestamp, runs your composer, and emits fresh records for the derived stream.
+
+```yaml
+# example/contracts/air_density.processed.yaml
+kind: composed
+id: air_density.processed
+inputs:
+  - p=pressure.processed
+  - t=temp_dry.processed
+partition_by: station_id
+sort_batch_size: 20000
+
+mapper:
+  entrypoint: {{PACKAGE_NAME}}.mappers.air_density:mapper
+  args:
+    driver: p   # optional; defaults to first input alias
+
+# Optional post‑compose policies (same as any stream):
+# record: [...]
+# stream: [...]
+# debug: [...]
+```
+
+Then reference the composed stream in your dataset:
+
+```yaml
+# example/dataset.yaml
+group_by: ${group_by}
+features:
+  - id: air_density
+    record_stream: air_density.processed
+```

datapipeline/templates/plugin_skeleton/example/contracts/time.ticks.hour_sin.yaml

@@ -0,0 +1,31 @@
+kind: ingest
+source: synthetic.ticks
+id: time.ticks.hour_sin # format: domain.dataset.(variant)
+
+# Fine-grained cadence for this stream. Defaults to the dataset group_by via project.globals.
+cadence: ${group_by}
+
+mapper:
+  entrypoint: encode_time
+  args: { mode: hour_sin }
+
+# partition_by: field you want to partition
+
+record:
+  - filter: { operator: ge, field: time, comparand: "${start_time}" }
+  - filter: { operator: le, field: time, comparand: "${end_time}" }
+  - floor_time: { cadence: "${cadence}" }
+  # - lag: { lag: "${cadence}" }
+
+stream:
+  - dedupe: {}
+  - granularity: { mode: first }
+  - ensure_cadence: { cadence: "${cadence}" }
+  # Optional: fill gaps before downstream transforms:
+  # - fill: { statistic: median, window: 24, min_samples: 4 }
+
+debug:
+  - lint: { mode: error, tick: "${cadence}" }
+
+# sort_batch_size: 100000
+

datapipeline/templates/plugin_skeleton/example/contracts/time.ticks.linear.yaml

@@ -0,0 +1,30 @@
+kind: ingest
+source: synthetic.ticks # raw source alias (see example/sources)
+id: time.ticks.linear # canonical stream id (format: domain.dataset.(variant))
+
+# Fine-grained cadence for this stream. Defaults to the dataset group_by via project.globals.
+cadence: ${group_by}
+
+mapper: # normalize/reshape DTO -> TemporalRecord
+  entrypoint: encode_time
+  args: { mode: linear }
+# partition_by: station_id      # optional: add partition suffixes to feature ids
+
+record: # record-level transforms
+  - filter: { operator: ge, field: time, comparand: "${start_time}" }
+  - filter: { operator: le, field: time, comparand: "${end_time}" }
+  - floor_time: { cadence: "${cadence}" }          # snap timestamps to cadence boundaries
+  # - lag: { lag: "${cadence}" }                      # optional: shift timestamps backwards
+
+stream:                       # per-feature stream transforms (input sorted by id,time)
+  - dedupe: {}                                # drop exact-duplicate records per tick
+  - granularity: { mode: first }              # aggregate duplicates within a tick
+  - ensure_cadence: { cadence: "${cadence}" }           # insert missing ticks (value=None)
+  # Consider adding a fill transform to impute None values before sequence/windowing:
+  # - fill: { statistic: median, window: 6, min_samples: 1 }
+
+debug:                        # optional validation-only transforms
+  - lint: { mode: error, tick: "${cadence}" }  # strict cadence/order; value issues handled by downstream transforms
+
+# sort_batch_size: 100000       # in-memory chunk size used by internal sorting
+

datapipeline/templates/plugin_skeleton/example/dataset.yaml

@@ -0,0 +1,18 @@
+group_by: ${group_by}
+
+features:
+  - id: time_linear
+    record_stream: time.ticks.linear
+    scale: true # optionally add with_mean/with_std overrides
+    # Sliding window over the regularized stream; cadence is enforced in the contract.
+    sequence: { size: 6, stride: 1 }
+
+  - id: time_hour_sin
+    record_stream: time.ticks.hour_sin
+
+  # - id: third_feature
+  #   record_stream: anotherstream
+# targets:
+#   - id: some_target
+#     record_stream: time.ticks.linear
+

datapipeline/templates/plugin_skeleton/example/postprocess.yaml

@@ -0,0 +1,29 @@
+#### example combination of postprocessing steps ######
+#### making sure data is complete after these combinations ######
+- drop: # example of dropping sparse partitions/vertical-axis for targets
+    axis: vertical
+    payload: targets
+    threshold: 0.9
+
+- drop: # example of dropping sparse partitions for features
+    axis: vertical
+    payload: features
+    threshold: 0.9
+
+- drop: # dropping vectors/horizontal-axis that has features which none
+    axis: horizontal
+    payload: features
+    threshold: 1
+
+- drop:
+    axis: horizontal
+    payload: targets
+    threshold: 1
+######
+# - fill:
+#     statistic: median
+#     window: 48
+#     min_samples: 6
+# - replace:
+#     payload: targets
+#     value: 0.0

datapipeline/templates/plugin_skeleton/{config/datasets/default → example}/project.yaml

@@ -1,18 +1,21 @@
 version: 1
-name: default
+name: example
 paths:
-  streams: ../../contracts
-  sources: ../../sources
+  streams: ./contracts
+  sources: ./sources
   dataset: dataset.yaml
   postprocess: postprocess.yaml
-  artifacts: ../../build/datasets/${project_name}
-  build: build.yaml
-  run: runs
+  artifacts: ../artifacts/${project_name}/v${version}
+  tasks: ./tasks
 globals:
+  # Globals to use in your .yaml files via ${var_name}.
+  # Primary dataset cadence; referenced from dataset.yaml (group_by)
+  # and contracts via ${group_by}.
+  group_by: 1h
   start_time: 2021-01-01T00:00:00Z
-  end_time: 2023-01-03T23:00:00Z
+  end_time: 2021-01-02T00:00:00Z
   # Configure deterministic dataset split here (applied at serve time, after postprocess).
-  # Adjust `ratios` as needed; the active split is selected via run.yaml or CLI.
+  # Adjust `ratios` as needed; the active split is selected via serve tasks or CLI.
   split:
     mode: hash # hash | time (time uses boundaries/labels)
     key: group # group | feature:<id> (entity-stable split)

datapipeline/templates/plugin_skeleton/example/sources/synthetic.ticks.yaml

@@ -0,0 +1,12 @@
+id: synthetic.ticks
+
+parser:
+  entrypoint: "core.synthetic.ticks"
+  args: {}
+loader:
+  entrypoint: "core.synthetic.ticks"
+  args:
+    start: "${start_time}"
+    end: "${end_time}"
+    frequency: "${group_by}"
+

datapipeline/templates/plugin_skeleton/example/tasks/metadata.yaml

@@ -0,0 +1,3 @@
+kind: metadata
+# window_mode: intersection # union|intersection|strict|relaxed (default: intersection)
+

datapipeline/templates/plugin_skeleton/example/tasks/scaler.yaml

@@ -0,0 +1,9 @@
+kind: scaler
+
+# Output path is relative to project.paths.artifacts; defaults to "scaler.pkl".
+# output: scaler.pkl
+
+# Split label to use when fitting scaler statistics.
+# Must match a label from globals.split.ratios.
+split_label: train
+

datapipeline/templates/plugin_skeleton/example/tasks/schema.yaml

@@ -0,0 +1,2 @@
+kind: schema
+

datapipeline/templates/plugin_skeleton/example/tasks/serve.test.yaml

@@ -0,0 +1,4 @@
+kind: serve
+name: test
+keep: test
+

datapipeline/templates/plugin_skeleton/example/tasks/serve.train.yaml

@@ -0,0 +1,28 @@
+kind: serve
+
+# Optional identifier for this serve task; defaults to filename stem.
+name: train
+
+# Active split label to serve; must match a label from globals.split.ratios.
+# Set to null to disable split filtering.
+keep: train
+#output:
+# transport: stdout | fs
+# format: print | json-lines | json | csv | pickle
+# When using fs transport, set a directory (and optionally filename) for outputs:
+# directory: artifacts/serve
+# filename: vectors.train
+
+# Default max number of vectors to emit (null = unlimited).
+# limit: 5
+# Optional pipeline stage preview (0-7); null lets the CLI decide.
+# stage: 7
+
+# Optional pacing between emitted vectors (milliseconds).
+# throttle_ms: null
+
+# Visuals/logging knobs (inherit CLI or jerry.yaml defaults when omitted):
+# visuals: AUTO   # AUTO | TQDM | RICH | OFF
+# progress: AUTO  # AUTO | SPINNER | BARS | OFF
+# log_level: INFO # CRITICAL | ERROR | WARNING | INFO | DEBUG
+

datapipeline/templates/plugin_skeleton/example/tasks/serve.val.yaml

@@ -0,0 +1,4 @@
+kind: serve
+name: val
+keep: val
+

datapipeline/templates/plugin_skeleton/jerry.yaml

@@ -0,0 +1,28 @@
+# Workspace defaults. Move this file to the root of your workspace if you serve
+# configs outside the plugin repo; the CLI searches upward from cwd for jerry.yaml.
+
+plugin_root: . # relative path to the plugin repo root
+
+datasets:
+  example: example/project.yaml
+  your-second-example-dataset: your-dataset/project.yaml
+
+default_dataset: example
+
+shared:
+  visuals: AUTO # AUTO | TQDM | RICH | OFF
+  progress: BARS # AUTO | SPINNER | BARS | OFF
+  log_level: INFO
+
+serve:
+  # log_level: INFO
+  limit: null
+  stage: null
+  output:
+    transport: stdout
+    format: print # set to fs + directory for file outputs
+    # directory: artifacts/serve
+
+build:
+  # log_level: INFO
+  mode: AUTO # AUTO | FORCE | OFF

datapipeline/templates/plugin_skeleton/your-dataset/contracts/time.ticks.hour_sin.yaml

@@ -0,0 +1,31 @@
+kind: ingest
+source: synthetic.ticks
+id: time.ticks.hour_sin # format: domain.dataset.(variant)
+
+# Fine-grained cadence for this stream. Defaults to the dataset group_by via project.globals.
+cadence: ${group_by}
+
+mapper:
+  entrypoint: encode_time
+  args: { mode: hour_sin }
+
+# partition_by: field you want to partition
+
+record:
+  - filter: { operator: ge, field: time, comparand: "${start_time}" }
+  - filter: { operator: le, field: time, comparand: "${end_time}" }
+  - floor_time: { cadence: "${cadence}" }
+  # - lag: { lag: "${cadence}" }
+
+stream:
+  - dedupe: {}
+  - granularity: { mode: first }
+  - ensure_cadence: { cadence: "${cadence}" }
+  # Optional: fill gaps before downstream transforms:
+  # - fill: { statistic: median, window: 24, min_samples: 4 }
+
+debug:
+  - lint: { mode: error, tick: "${cadence}" }
+
+# sort_batch_size: 100000
+

datapipeline/templates/plugin_skeleton/your-dataset/contracts/time.ticks.linear.yaml

@@ -0,0 +1,30 @@
+kind: ingest
+source: synthetic.ticks # raw source alias (see example/sources)
+id: time.ticks.linear # canonical stream id (format: domain.dataset.(variant))
+
+# Fine-grained cadence for this stream. Defaults to the dataset group_by via project.globals.
+cadence: ${group_by}
+
+mapper: # normalize/reshape DTO -> TemporalRecord
+  entrypoint: encode_time
+  args: { mode: linear }
+# partition_by: station_id      # optional: add partition suffixes to feature ids
+
+record: # record-level transforms
+  - filter: { operator: ge, field: time, comparand: "${start_time}" }
+  - filter: { operator: le, field: time, comparand: "${end_time}" }
+  - floor_time: { cadence: "${cadence}" }          # snap timestamps to cadence boundaries
+  # - lag: { lag: "${cadence}" }                      # optional: shift timestamps backwards
+
+stream:                       # per-feature stream transforms (input sorted by id,time)
+  - dedupe: {}                                # drop exact-duplicate records per tick
+  - granularity: { mode: first }              # aggregate duplicates within a tick
+  - ensure_cadence: { cadence: "${cadence}" }           # insert missing ticks (value=None)
+  # Consider adding a fill transform to impute None values before sequence/windowing:
+  # - fill: { statistic: median, window: 6, min_samples: 1 }
+
+debug:                        # optional validation-only transforms
+  - lint: { mode: error, tick: "${cadence}" }  # strict cadence/order; value issues handled by downstream transforms
+
+# sort_batch_size: 100000       # in-memory chunk size used by internal sorting
+

datapipeline/templates/plugin_skeleton/your-dataset/dataset.yaml

@@ -0,0 +1,18 @@
+group_by: ${group_by}
+
+features:
+  - id: time_linear
+    record_stream: time.ticks.linear
+    scale: true # optionally add with_mean/with_std overrides
+    # Sliding window over the regularized stream; cadence is enforced in the contract.
+    sequence: { size: 6, stride: 1 }
+
+  - id: time_hour_sin
+    record_stream: time.ticks.hour_sin
+
+  # - id: third_feature
+  #   record_stream: anotherstream
+# targets:
+#   - id: some_target
+#     record_stream: time.ticks.linear
+

datapipeline/templates/plugin_skeleton/your-dataset/postprocess.yaml

@@ -0,0 +1,29 @@
+#### example combination of postprocessing steps ######
+#### making sure data is complete after these combinations ######
+- drop: # example of dropping sparse partitions/vertical-axis for targets
+    axis: vertical
+    payload: targets
+    threshold: 0.9
+
+- drop: # example of dropping sparse partitions for features
+    axis: vertical
+    payload: features
+    threshold: 0.9
+
+- drop: # dropping vectors/horizontal-axis that has features which none
+    axis: horizontal
+    payload: features
+    threshold: 1
+
+- drop:
+    axis: horizontal
+    payload: targets
+    threshold: 1
+######
+# - fill:
+#     statistic: median
+#     window: 48
+#     min_samples: 6
+# - replace:
+#     payload: targets
+#     value: 0.0

datapipeline/templates/plugin_skeleton/your-dataset/project.yaml

@@ -0,0 +1,22 @@
+version: 1
+name: <your-dataset>
+paths:
+  streams: ./contracts
+  sources: ./sources
+  dataset: dataset.yaml
+  postprocess: postprocess.yaml
+  artifacts: ../artifacts/${project_name}/v${version}
+  tasks: ./tasks
+globals:
+  # Primary dataset cadence; referenced from dataset.yaml (group_by)
+  # and contracts via ${group_by}.
+  group_by: <your-bucket-cadence>
+  start_time: null #2021-01-01T00:00:00Z
+  end_time: null #2021-01-02T00:00:00Z
+  # Configure deterministic dataset split here (applied at serve time, after postprocess).
+  # Adjust `ratios` as needed; the active split is selected via serve tasks or CLI.
+  split:
+    mode: hash # hash | time (time uses boundaries/labels)
+    key: group # group | feature:<id> (entity-stable split)
+    seed: 42 # deterministic hash seed
+    ratios: { train: 0.8, val: 0.1, test: 0.1 }

datapipeline/templates/plugin_skeleton/your-dataset/sources/synthetic.ticks.yaml

@@ -0,0 +1,12 @@
+id: synthetic.ticks
+
+parser:
+  entrypoint: "core.synthetic.ticks"
+  args: {}
+loader:
+  entrypoint: "core.synthetic.ticks"
+  args:
+    start: "${start_time}"
+    end: "${end_time}"
+    frequency: "${group_by}"
+

datapipeline/templates/plugin_skeleton/your-dataset/tasks/metadata.yaml

@@ -0,0 +1,3 @@
+kind: metadata
+# window_mode: intersection # union|intersection|strict|relaxed (default: intersection)
+

datapipeline/templates/plugin_skeleton/your-dataset/tasks/scaler.yaml

@@ -0,0 +1,9 @@
+kind: scaler
+
+# Output path is relative to project.paths.artifacts; defaults to "scaler.pkl".
+# output: scaler.pkl
+
+# Split label to use when fitting scaler statistics.
+# Must match a label from globals.split.ratios.
+split_label: train
+

datapipeline/templates/plugin_skeleton/your-dataset/tasks/schema.yaml

@@ -0,0 +1,2 @@
+kind: schema
+

datapipeline/templates/plugin_skeleton/your-dataset/tasks/serve.test.yaml

@@ -0,0 +1,4 @@
+kind: serve
+name: test
+keep: test
+

datapipeline/templates/plugin_skeleton/your-dataset/tasks/serve.train.yaml

@@ -0,0 +1,28 @@
+kind: serve
+
+# Optional identifier for this serve task; defaults to filename stem.
+name: train
+
+# Active split label to serve; must match a label from globals.split.ratios.
+# Set to null to disable split filtering.
+keep: train
+#output:
+# transport: stdout | fs
+# format: print | json-lines | json | csv | pickle
+# When using fs transport, set a directory (and optionally filename) for outputs:
+# directory: artifacts/serve
+# filename: vectors.train
+
+# Default max number of vectors to emit (null = unlimited).
+# limit: 5
+# Optional pipeline stage preview (0-7); null lets the CLI decide.
+# stage: 7
+
+# Optional pacing between emitted vectors (milliseconds).
+# throttle_ms: null
+
+# Visuals/logging knobs (inherit CLI or jerry.yaml defaults when omitted):
+# visuals: AUTO   # AUTO | TQDM | RICH | OFF
+# progress: AUTO  # AUTO | SPINNER | BARS | OFF
+# log_level: INFO # CRITICAL | ERROR | WARNING | INFO | DEBUG
+

datapipeline/templates/plugin_skeleton/your-dataset/tasks/serve.val.yaml

@@ -0,0 +1,4 @@
+kind: serve
+name: val
+keep: val
+

datapipeline/templates/stubs/dto.py.j2

@@ -1,4 +1,6 @@
 from dataclasses import dataclass
+from datetime import datetime
+
 
 @dataclass
 class {{CLASS_NAME}}:

datapipeline/templates/stubs/mapper.py.j2

@@ -1,16 +1,17 @@
-from typing import Iterator, Any
-from {{PACKAGE_NAME}}.sources.{{ORIGIN}}.{{DATASET}}.dto import {{OriginDTO}}
+from typing import Any, Iterator
+
 from {{PACKAGE_NAME}}.domains.{{TARGET_DOMAIN}}.model import {{DomainRecord}}
+from {{PACKAGE_NAME}}.sources.{{ORIGIN}}.{{DATASET}}.dto import {{OriginDTO}}
 
 
 def {{FUNCTION_NAME}}(
     stream: Iterator[{{OriginDTO}}],
-    *args: Any,
+    **params: Any,
 ) -> Iterator[{{DomainRecord}}]:
     """Map raw {{ORIGIN}} DTOs to domain-level {{TARGET_DOMAIN}} records.
 
     - Required on domain record: time and value.
-    - Additional options may be passed via kwargs (e.g., mode="...").
+    - Additional options may be passed via kwargs (e.g., variant="..." or mode="...").
     """
     for dto in stream:
         # TODO: construct {{DomainRecord}} from dto fields