wowdata 0.1.0__tar.gz

wowdata-0.1.0/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright 2026 sci2pro
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

wowdata-0.1.0/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: wowdata
+Version: 0.1.0
+Summary: A human-centred, teachable data wrangling and pipeline framework.
+Author: WowData Contributors
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/sci2pro/wowdata
+Project-URL: Repository, https://github.com/sci2pro/wowdata
+Project-URL: Issues, https://github.com/sci2pro/wowdata/issues
+Keywords: data-engineering,etl,data-pipelines,education,data-wrangling
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: petl>=1.7
+Requires-Dist: frictionless>=5.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs; extra == "docs"
+Requires-Dist: mkdocs-material; extra == "docs"
+Dynamic: license-file
+
+# WowData™
+
+WowData™ is a human-centred data wrangling and pipeline framework designed to make real-world data cleanup understandable, teachable, and inspectable.
+
+## Install
+
+From PyPI:
+
+```bash
+pip install wowdata
+```
+
+From source (editable):
+
+```bash
+git clone https://github.com/sci2pro/wowdata.git
+cd wowdata
+pip install -e .
+```
+
+## Quick Start
+
+Create a small input CSV:
+
+```csv
+person_id,age,country
+1,30,KE
+2,17,UG
+3,41,KE
+```
+
+Run with Python API:
+
+```python
+from wowdata import Pipeline, Sink, Source, Transform
+
+pipe = (
+    Pipeline(Source("people.csv"))
+    .then(Transform("cast", params={"types": {"age": "integer"}, "on_error": "null"}))
+    .then(Transform("filter", params={"where": "age >= 18 and country == 'KE'"}))
+    .then(Sink("adults_ke.csv"))
+)
+
+pipe.run()
+```
+
+Run from YAML with CLI:
+
+```bash
+wow run pipeline.yaml
+```
+
+Fallback command if `wow` conflicts in your shell:
+
+```bash
+wowdata run pipeline.yaml
+```
+
+## Documentation
+
+- Philosophy: [docs/philosophy.md](docs/philosophy.md)
+- Examples: [EXAMPLES.md](EXAMPLES.md)
+- Reference: [REFERENCE.md](REFERENCE.md)
+- Docs site source: [docs/](docs/) + [mkdocs.yml](mkdocs.yml)
+
+To preview docs locally:
+
+```bash
+pip install -e .[docs]
+mkdocs serve
+```
+
+The same docs can be published to GitHub Pages (for `wowdata.github.io`).

wowdata-0.1.0/README.md

@@ -0,0 +1,73 @@
+# WowData™
+
+WowData™ is a human-centred data wrangling and pipeline framework designed to make real-world data cleanup understandable, teachable, and inspectable.
+
+## Install
+
+From PyPI:
+
+```bash
+pip install wowdata
+```
+
+From source (editable):
+
+```bash
+git clone https://github.com/sci2pro/wowdata.git
+cd wowdata
+pip install -e .
+```
+
+## Quick Start
+
+Create a small input CSV:
+
+```csv
+person_id,age,country
+1,30,KE
+2,17,UG
+3,41,KE
+```
+
+Run with Python API:
+
+```python
+from wowdata import Pipeline, Sink, Source, Transform
+
+pipe = (
+    Pipeline(Source("people.csv"))
+    .then(Transform("cast", params={"types": {"age": "integer"}, "on_error": "null"}))
+    .then(Transform("filter", params={"where": "age >= 18 and country == 'KE'"}))
+    .then(Sink("adults_ke.csv"))
+)
+
+pipe.run()
+```
+
+Run from YAML with CLI:
+
+```bash
+wow run pipeline.yaml
+```
+
+Fallback command if `wow` conflicts in your shell:
+
+```bash
+wowdata run pipeline.yaml
+```
+
+## Documentation
+
+- Philosophy: [docs/philosophy.md](docs/philosophy.md)
+- Examples: [EXAMPLES.md](EXAMPLES.md)
+- Reference: [REFERENCE.md](REFERENCE.md)
+- Docs site source: [docs/](docs/) + [mkdocs.yml](mkdocs.yml)
+
+To preview docs locally:
+
+```bash
+pip install -e .[docs]
+mkdocs serve
+```
+
+The same docs can be published to GitHub Pages (for `wowdata.github.io`).

wowdata-0.1.0/pyproject.toml

@@ -0,0 +1,95 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "wowdata"
+version = "0.1.0"
+description = "A human-centred, teachable data wrangling and pipeline framework."
+readme = "README.md"
+license = { text = "Apache-2.0" }
+authors = [
+  { name = "WowData Contributors" }
+]
+requires-python = ">=3.10"
+
+dependencies = [
+  "petl>=1.7",
+  "frictionless>=5.0",
+]
+
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Education",
+  "Intended Audience :: Developers",
+  "Topic :: Software Development :: Libraries",
+  "Topic :: Scientific/Engineering :: Information Analysis",
+  "License :: OSI Approved :: Apache Software License",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+]
+
+keywords = [
+  "data-engineering",
+  "etl",
+  "data-pipelines",
+  "education",
+  "data-wrangling",
+]
+
+[project.optional-dependencies]
+dev = [
+  "pytest>=7.0",
+  "pytest-cov",
+  "black",
+  "ruff",
+  "mypy",
+]
+
+docs = [
+  "mkdocs",
+  "mkdocs-material",
+]
+
+[project.urls]
+Homepage = "https://github.com/sci2pro/wowdata"
+Repository = "https://github.com/sci2pro/wowdata"
+Issues = "https://github.com/sci2pro/wowdata/issues"
+
+[project.scripts]
+wow = "wowdata.cli:main"
+wowdata = "wowdata.cli:main"
+
+[tool.setuptools]
+packages = ["wowdata"]
+
+[tool.setuptools.package-data]
+wowdata = ["py.typed"]
+
+[tool.pytest.ini_options]
+testpaths = ["wowdata/tests"]
+addopts = "-q"
+
+[tool.black]
+line-length = 88
+target-version = ["py310"]
+
+[tool.ruff]
+line-length = 88
+select = ["E", "F", "W", "I"]
+ignore = ["E501"]
+
+[tool.mypy]
+python_version = "3.10"
+strict = false
+warn_return_any = true
+warn_unused_configs = true
+
+[tools.coverage.run]
+omit = [
+  "wowdata/wowdata/tests/examples/",
+  "wowdata/wowdata/tests/",
+  "wowdata/experiments/"
+]

wowdata-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

wowdata-0.1.0/wowdata/__init__.py

@@ -0,0 +1,7 @@
+from wowdata.models.pipeline import Pipeline
+from wowdata.models.sinks import Sink
+from wowdata.models.sources import Source
+from wowdata.models.transforms import Transform
+from wowdata.errors import WowDataUserError
+
+__all__ = ["Pipeline", "Sink", "Source", "Transform", "WowDataUserError"]

wowdata-0.1.0/wowdata/cli.py

@@ -0,0 +1,211 @@
+from __future__ import annotations
+
+import argparse
+import json
+import sys
+from pathlib import Path
+from typing import Any, Dict, Optional, Sequence
+
+from wowdata import Pipeline, WowDataUserError
+
+EXIT_OK = 0
+EXIT_USAGE = 2
+EXIT_PIPELINE_PARSE = 3
+EXIT_PIPELINE_RUNTIME = 4
+
+
+def _default_locked_path(pipeline_path: str) -> Path:
+    p = Path(pipeline_path)
+    if p.suffix in {".yaml", ".yml"}:
+        return p.with_name(f"{p.stem}.locked{p.suffix}")
+    return p.with_name(f"{p.name}.locked.yaml")
+
+
+def _load_pipeline(path: str, base_dir: Optional[str]) -> Pipeline:
+    resolved_base_dir = Path(base_dir) if base_dir else None
+    return Pipeline.from_yaml(path, base_dir=resolved_base_dir)
+
+
+def _print_user_error(err: WowDataUserError) -> None:
+    print(str(err), file=sys.stderr)
+
+
+def _print_json(payload: Dict[str, Any]) -> None:
+    print(json.dumps(payload, indent=2, sort_keys=False))
+
+
+def _cmd_run(args: argparse.Namespace) -> int:
+    try:
+        pipe = _load_pipeline(args.pipeline, args.base_dir)
+    except WowDataUserError as e:
+        _print_user_error(e)
+        return EXIT_PIPELINE_PARSE
+
+    try:
+        ctx = pipe.run()
+    except WowDataUserError as e:
+        _print_user_error(e)
+        return EXIT_PIPELINE_RUNTIME
+
+    sinks = [s for s in pipe.steps if getattr(s, "uri", None) is not None and s.__class__.__name__ == "Sink"]
+
+    if args.json:
+        payload: Dict[str, Any] = {
+            "ok": True,
+            "command": "run",
+            "pipeline": args.pipeline,
+            "steps": len(pipe.steps),
+            "sinks": [s.uri for s in sinks],
+            "checkpoints": len(ctx.checkpoints),
+            "validations": ctx.validations,
+        }
+        if args.show_checkpoints:
+            payload["checkpoint_data"] = ctx.checkpoints
+        _print_json(payload)
+        return EXIT_OK
+
+    if not args.quiet:
+        print(f"Run complete: {args.pipeline}")
+        print(f"Steps: {len(pipe.steps)}")
+        if sinks:
+            print("Sinks:")
+            for s in sinks:
+                print(f"- {s.uri}")
+        print(f"Checkpoints: {len(ctx.checkpoints)}")
+        if ctx.validations:
+            print(f"Validations: {len(ctx.validations)}")
+
+    if args.show_checkpoints:
+        print(json.dumps(ctx.checkpoints, indent=2, default=str))
+
+    return EXIT_OK
+
+
+def _cmd_validate(args: argparse.Namespace) -> int:
+    try:
+        pipe = _load_pipeline(args.pipeline, args.base_dir)
+        pipe.preflight()
+    except WowDataUserError as e:
+        _print_user_error(e)
+        return EXIT_PIPELINE_PARSE
+
+    if args.json:
+        _print_json(
+            {
+                "ok": True,
+                "command": "validate",
+                "pipeline": args.pipeline,
+                "steps": len(pipe.steps),
+            }
+        )
+    else:
+        print(f"Pipeline is valid: {args.pipeline}")
+        print(f"Steps: {len(pipe.steps)}")
+
+    return EXIT_OK
+
+
+def _cmd_schema(args: argparse.Namespace) -> int:
+    try:
+        pipe = _load_pipeline(args.pipeline, args.base_dir)
+        sch = pipe.schema(sample_rows=args.sample_rows, force=args.force)
+    except WowDataUserError as e:
+        _print_user_error(e)
+        return EXIT_PIPELINE_PARSE
+
+    if args.json:
+        _print_json(
+            {
+                "ok": True,
+                "command": "schema",
+                "pipeline": args.pipeline,
+                "schema": sch,
+            }
+        )
+    else:
+        print(f"Schema for: {args.pipeline}")
+        print(json.dumps(sch, indent=2, sort_keys=False))
+
+    return EXIT_OK
+
+
+def _cmd_lock_schema(args: argparse.Namespace) -> int:
+    out_path = Path(args.output) if args.output else _default_locked_path(args.pipeline)
+
+    try:
+        pipe = _load_pipeline(args.pipeline, args.base_dir)
+        pipe.to_yaml(
+            path=out_path,
+            lock_schema=True,
+            sample_rows=args.sample_rows,
+            force=args.force,
+        )
+    except WowDataUserError as e:
+        _print_user_error(e)
+        return EXIT_PIPELINE_PARSE
+
+    if args.json:
+        _print_json(
+            {
+                "ok": True,
+                "command": "lock-schema",
+                "pipeline": args.pipeline,
+                "output": str(out_path),
+            }
+        )
+    else:
+        print(f"Schema-locked pipeline written: {out_path}")
+
+    return EXIT_OK
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    parser = argparse.ArgumentParser(
+        prog="wowdata",
+        description="WowData CLI for running YAML pipelines.",
+    )
+
+    subparsers = parser.add_subparsers(dest="command", required=True)
+
+    run_p = subparsers.add_parser("run", help="Run a YAML pipeline.")
+    run_p.add_argument("pipeline", help="Path to pipeline YAML file.")
+    run_p.add_argument("--base-dir", help="Base directory for relative paths in YAML.")
+    run_p.add_argument("--json", action="store_true", help="Print machine-readable JSON output.")
+    run_p.add_argument("--quiet", action="store_true", help="Suppress normal status output.")
+    run_p.add_argument("--show-checkpoints", action="store_true", help="Print checkpoint details after run.")
+    run_p.set_defaults(func=_cmd_run)
+
+    validate_p = subparsers.add_parser("validate", help="Validate pipeline YAML and preflight I/O paths.")
+    validate_p.add_argument("pipeline", help="Path to pipeline YAML file.")
+    validate_p.add_argument("--base-dir", help="Base directory for relative paths in YAML.")
+    validate_p.add_argument("--json", action="store_true", help="Print machine-readable JSON output.")
+    validate_p.set_defaults(func=_cmd_validate)
+
+    schema_p = subparsers.add_parser("schema", help="Infer output schema without full execution.")
+    schema_p.add_argument("pipeline", help="Path to pipeline YAML file.")
+    schema_p.add_argument("--base-dir", help="Base directory for relative paths in YAML.")
+    schema_p.add_argument("--sample-rows", type=int, default=200, help="Rows to sample for schema inference.")
+    schema_p.add_argument("--force", action="store_true", help="Recompute schema even if cached.")
+    schema_p.add_argument("--json", action="store_true", help="Print machine-readable JSON output.")
+    schema_p.set_defaults(func=_cmd_schema)
+
+    lock_p = subparsers.add_parser("lock-schema", help="Write a schema-locked YAML pipeline.")
+    lock_p.add_argument("pipeline", help="Path to pipeline YAML file.")
+    lock_p.add_argument("-o", "--output", help="Output path for locked YAML.")
+    lock_p.add_argument("--base-dir", help="Base directory for relative paths in YAML.")
+    lock_p.add_argument("--sample-rows", type=int, default=200, help="Rows to sample for schema inference.")
+    lock_p.add_argument("--force", action="store_true", help="Recompute schema even if cached.")
+    lock_p.add_argument("--json", action="store_true", help="Print machine-readable JSON output.")
+    lock_p.set_defaults(func=_cmd_lock_schema)
+
+    return parser
+
+
+def main(argv: Optional[Sequence[str]] = None) -> int:
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+    return int(args.func(args))
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())

wowdata-0.1.0/wowdata/errors.py

@@ -0,0 +1,23 @@
+from __future__ import annotations
+
+from typing import Optional
+
+
+class WowDataUserError(Exception):
+    """An instructional error intended for end users.
+
+    Use this for mistakes in the pipeline definition (invalid params, missing columns, etc.).
+    It carries a short error code and an optional hint to guide the user.
+    """
+
+    def __init__(self, code: str, message: str, *, hint: Optional[str] = None):
+        super().__init__(message)
+        self.code = code
+        self.message = message
+        self.hint = hint
+
+    def __str__(self) -> str:
+        base = f"[{self.code}] {self.message}"
+        if self.hint:
+            return base + f"\nHint: {self.hint}"
+        return base

wowdata-0.1.0/wowdata/schema.py

@@ -0,0 +1,254 @@
+from __future__ import annotations
+
+import os
+from pathlib import Path
+from typing import Dict, Any, Optional, List
+
+from wowdata.errors import WowDataUserError
+from wowdata.models.sources import Source
+from wowdata.models.sinks import Sink
+from wowdata.models.transforms import Transform
+from wowdata.util import _norm_path, FrictionlessSchema
+
+
+def _schema_field_names(schema: Optional[FrictionlessSchema]) -> List[str]:
+    if not schema or not isinstance(schema, dict):
+        return []
+    fields = schema.get("fields")
+    if not isinstance(fields, list):
+        return []
+    out: List[str] = []
+    for f in fields:
+        if isinstance(f, dict) and isinstance(f.get("name"), str):
+            out.append(f["name"])
+    return out
+
+
+def _source_to_ir(src: Source) -> Dict[str, Any]:
+    uri = os.fspath(src.uri)
+    d: Dict[str, Any] = {"uri": uri}
+    if src.type is not None:
+        d["type"] = src.type
+    if src.schema is not None:
+        d["schema"] = src.schema
+    if src.options:
+        d["options"] = dict(src.options)
+    return d
+
+
+def _sink_to_ir(sink: Sink) -> Dict[str, Any]:
+    uri = os.fspath(sink.uri)
+    d: Dict[str, Any] = {"uri": uri}
+    if sink.type is not None:
+        d["type"] = sink.type
+    if sink.options:
+        d["options"] = dict(sink.options)
+    return d
+
+
+def _transform_to_ir(t: Transform) -> Dict[str, Any]:
+    d: Dict[str, Any] = {"op": t.op}
+    if t.params:
+        d["params"] = dict(t.params)
+    if t.output_schema_override is not None:
+        d["output_schema"] = t.output_schema_override
+    return d
+
+
+def _source_from_ir(d: Dict[str, Any]) -> Source:
+    if not isinstance(d, dict):
+        raise WowDataUserError(
+            "E_IR_SOURCE",
+            "IR source must be a mapping.",
+            hint="Example: start: {uri: people.csv, type: csv}",
+        )
+    uri = d.get("uri")
+    if not isinstance(uri, str) or not uri:
+        raise WowDataUserError(
+            "E_IR_SOURCE",
+            "IR source requires a non-empty 'uri' string.",
+            hint="Example: start: {uri: people.csv}",
+        )
+    return Source(
+        uri,
+        type=d.get("type"),
+        schema=d.get("schema"),
+        options=d.get("options") or {},
+    )
+
+
+def _sink_from_ir(d: Dict[str, Any]) -> Sink:
+    if not isinstance(d, dict):
+        raise WowDataUserError(
+            "E_IR_SINK",
+            "IR sink must be a mapping.",
+            hint="Example: {sink: {uri: out.csv}}",
+        )
+    uri = d.get("uri")
+    if not isinstance(uri, str) or not uri:
+        raise WowDataUserError(
+            "E_IR_SINK",
+            "IR sink requires a non-empty 'uri' string.",
+            hint="Example: {sink: {uri: out.csv}}",
+        )
+    return Sink(
+        uri,
+        type=d.get("type"),
+        options=d.get("options") or {},
+    )
+
+
+def _normalize_ir(ir: Any, *, base_dir: Optional[Path]) -> Dict[str, Any]:
+    """Normalize IR structure and paths.
+
+    Guarantees:
+      - returns a dict with keys: wowdata, pipeline
+      - pipeline.start.uri is normalized
+      - any sink.uri is normalized
+      - any join params.right (string or descriptor.uri) is normalized
+      - missing/None options become {}
+      - missing transform params become {}
+
+    This does not change semantics; it makes the IR portable and deterministic.
+    """
+    if not isinstance(ir, dict):
+        raise WowDataUserError(
+            "E_IR_ROOT",
+            "IR must be a mapping at the root.",
+            hint="Expected keys: wowdata, pipeline.",
+        )
+
+    ir2: Dict[str, Any] = dict(ir)
+
+    # Default version
+    if ir2.get("wowdata") is None:
+        ir2["wowdata"] = 0
+
+    version = ir2.get("wowdata")
+    if version != 0:
+        raise WowDataUserError(
+            "E_IR_VERSION",
+            f"Unsupported IR version: {version!r}.",
+            hint="Supported: wowdata: 0",
+        )
+
+    pipe = ir2.get("pipeline")
+    if not isinstance(pipe, dict):
+        raise WowDataUserError(
+            "E_IR_PIPELINE",
+            "IR requires a 'pipeline' mapping.",
+            hint="Example: {wowdata: 0, pipeline: {start: {...}, steps: [...]}}",
+        )
+
+    pipe2: Dict[str, Any] = dict(pipe)
+
+    # Normalize start
+    start = pipe2.get("start")
+    if not isinstance(start, dict):
+        raise WowDataUserError(
+            "E_IR_SOURCE",
+            "IR pipeline.start must be a mapping.",
+            hint="Example: start: {uri: people.csv, type: csv}",
+        )
+    start2: Dict[str, Any] = dict(start)
+    u = start2.get("uri")
+    if isinstance(u, str):
+        start2["uri"] = _norm_path(u, base_dir=base_dir)
+    if "options" in start2 and start2["options"] is None:
+        start2["options"] = {}
+    pipe2["start"] = start2
+
+    # Normalize steps
+    steps = pipe2.get("steps")
+    if steps is None:
+        steps = []
+    if not isinstance(steps, list):
+        raise WowDataUserError(
+            "E_IR_STEPS",
+            "IR pipeline.steps must be a list.",
+            hint="Example: steps: [{transform: {...}}, {sink: {...}}]",
+        )
+
+    norm_steps: List[Dict[str, Any]] = []
+    for i, item in enumerate(steps):
+        if not isinstance(item, dict) or len(item) != 1:
+            raise WowDataUserError(
+                "E_IR_STEP",
+                f"IR step #{i} must be a mapping with exactly one key: 'transform' or 'sink'.",
+                hint=str(item),
+            )
+
+        if "sink" in item:
+            s = item["sink"]
+            if not isinstance(s, dict):
+                raise WowDataUserError(
+                    "E_IR_SINK",
+                    "IR sink must be a mapping.",
+                    hint="Example: - sink: {uri: out.csv}",
+                )
+            s2: Dict[str, Any] = dict(s)
+            su = s2.get("uri")
+            if isinstance(su, str):
+                s2["uri"] = _norm_path(su, base_dir=base_dir)
+            if "options" in s2 and s2["options"] is None:
+                s2["options"] = {}
+            norm_steps.append({"sink": s2})
+            continue
+
+        if "transform" in item:
+            t = item["transform"]
+            if not isinstance(t, dict):
+                raise WowDataUserError(
+                    "E_IR_TRANSFORM",
+                    "IR transform must be a mapping.",
+                    hint="Example: - transform: {op: select, params: {...}}",
+                )
+            t2: Dict[str, Any] = dict(t)
+            params = t2.get("params")
+            if params is None:
+                params = {}
+                t2["params"] = params
+            if not isinstance(params, dict):
+                raise WowDataUserError(
+                    "E_IR_TRANSFORM",
+                    "IR transform 'params' must be a mapping.",
+                    hint="Example: params: {where: \"age >= 30\"}",
+                )
+
+            # Join: normalize YAML 1.1 boolean key oddities and right-side path descriptors.
+            if t2.get("op") == "join":
+                # In YAML 1.1, unquoted key `on` may parse as boolean True.
+                # Normalize `{true: [...]}` to `{"on": [...]}` for user-friendly behavior.
+                if "on" not in params and True in params:
+                    params["on"] = params.pop(True)
+
+                if "right" in params:
+                    r = params.get("right")
+                    if isinstance(r, str):
+                        params["right"] = _norm_path(r, base_dir=base_dir)
+                    elif isinstance(r, dict):
+                        r2: Dict[str, Any] = dict(r)
+                        ru = r2.get("uri")
+                        if isinstance(ru, str):
+                            r2["uri"] = _norm_path(ru, base_dir=base_dir)
+                        if "options" in r2 and r2["options"] is None:
+                            r2["options"] = {}
+                        params["right"] = r2
+
+            # Cast: in YAML, unquoted `null` parses to None. Accept it as the "null" policy.
+            if t2.get("op") == "cast" and isinstance(params, dict):
+                if "on_error" in params and params.get("on_error") is None:
+                    params["on_error"] = "null"
+
+            norm_steps.append({"transform": t2})
+            continue
+
+        raise WowDataUserError(
+            "E_IR_STEP",
+            f"IR step #{i} must be 'transform' or 'sink'.",
+            hint="Example: {transform: {op: select, params: {...}}}",
+        )
+
+    pipe2["steps"] = norm_steps
+
+    return {"wowdata": 0, "pipeline": pipe2}

wowdata-0.1.0/wowdata/util.py

@@ -0,0 +1,72 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional, Dict, Any
+
+from wowdata.errors import WowDataUserError
+
+try:
+    from frictionless import Resource, Detector
+except Exception:  # pragma: no cover
+    Resource = None
+    Detector = None
+
+
+def _is_probably_url(s: str) -> bool:
+    return s.startswith("http://") or s.startswith("https://") or s.startswith("s3://")
+
+
+def _norm_path(p: str, *, base_dir: Optional[Path]) -> str:
+    """Normalize a URI/path relative to a base directory (when provided).
+
+    - Leaves absolute paths and URLs unchanged.
+    - If base_dir is provided and p is relative, returns an absolute resolved path.
+    """
+    if not isinstance(p, str) or not p:
+        return p
+    if _is_probably_url(p):
+        return p
+    try:
+        pp = Path(p)
+    except Exception:
+        return p
+    if pp.is_absolute():
+        return str(pp)
+    if base_dir is None:
+        return p
+    return str((base_dir / pp).resolve())
+
+
+def _infer_type_from_uri(uri: str) -> Optional[str]:
+    ext = Path(uri).suffix.lower()
+    if ext == ".csv":
+        return "csv"
+    # todo: extend later: .xlsx, .parquet, db urls, s3://, http(s)://, etc.
+    return None
+
+
+FrictionlessSchema = Dict[str, Any]
+
+
+def _normalize_inline_schema(schema: Dict[str, Any]) -> Dict[str, Any]:
+    """
+    Expect frictionless-ish schema:
+      {"fields":[{"name":"age","type":"integer"}, ...]}
+    We keep it permissive for now.
+    """
+    if not isinstance(schema, dict):
+        raise WowDataUserError(
+            "E_SCHEMA_INLINE_TYPE",
+            "Inline schema must be a mapping (dict).",
+            hint="Example: {'fields': [{'name': 'age', 'type': 'integer'}]}",
+        )
+    if "fields" not in schema:
+        # allow passing full frictionless resource descriptor later; for now keep simple
+        return {"fields": []}
+    if not isinstance(schema["fields"], list):
+        raise WowDataUserError(
+            "E_SCHEMA_INLINE_FIELDS",
+            "Inline schema['fields'] must be a list.",
+            hint="Example: {'fields': [{'name': 'age', 'type': 'integer'}]}",
+        )
+    return schema

wowdata-0.1.0/wowdata.egg-info/PKG-INFO

@@ -0,0 +1,109 @@
+Metadata-Version: 2.4
+Name: wowdata
+Version: 0.1.0
+Summary: A human-centred, teachable data wrangling and pipeline framework.
+Author: WowData Contributors
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/sci2pro/wowdata
+Project-URL: Repository, https://github.com/sci2pro/wowdata
+Project-URL: Issues, https://github.com/sci2pro/wowdata/issues
+Keywords: data-engineering,etl,data-pipelines,education,data-wrangling
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Education
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: petl>=1.7
+Requires-Dist: frictionless>=5.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Provides-Extra: docs
+Requires-Dist: mkdocs; extra == "docs"
+Requires-Dist: mkdocs-material; extra == "docs"
+Dynamic: license-file
+
+# WowData™
+
+WowData™ is a human-centred data wrangling and pipeline framework designed to make real-world data cleanup understandable, teachable, and inspectable.
+
+## Install
+
+From PyPI:
+
+```bash
+pip install wowdata
+```
+
+From source (editable):
+
+```bash
+git clone https://github.com/sci2pro/wowdata.git
+cd wowdata
+pip install -e .
+```
+
+## Quick Start
+
+Create a small input CSV:
+
+```csv
+person_id,age,country
+1,30,KE
+2,17,UG
+3,41,KE
+```
+
+Run with Python API:
+
+```python
+from wowdata import Pipeline, Sink, Source, Transform
+
+pipe = (
+    Pipeline(Source("people.csv"))
+    .then(Transform("cast", params={"types": {"age": "integer"}, "on_error": "null"}))
+    .then(Transform("filter", params={"where": "age >= 18 and country == 'KE'"}))
+    .then(Sink("adults_ke.csv"))
+)
+
+pipe.run()
+```
+
+Run from YAML with CLI:
+
+```bash
+wow run pipeline.yaml
+```
+
+Fallback command if `wow` conflicts in your shell:
+
+```bash
+wowdata run pipeline.yaml
+```
+
+## Documentation
+
+- Philosophy: [docs/philosophy.md](docs/philosophy.md)
+- Examples: [EXAMPLES.md](EXAMPLES.md)
+- Reference: [REFERENCE.md](REFERENCE.md)
+- Docs site source: [docs/](docs/) + [mkdocs.yml](mkdocs.yml)
+
+To preview docs locally:
+
+```bash
+pip install -e .[docs]
+mkdocs serve
+```
+
+The same docs can be published to GitHub Pages (for `wowdata.github.io`).

wowdata-0.1.0/wowdata.egg-info/SOURCES.txt

@@ -0,0 +1,14 @@
+LICENSE
+README.md
+pyproject.toml
+wowdata/__init__.py
+wowdata/cli.py
+wowdata/errors.py
+wowdata/schema.py
+wowdata/util.py
+wowdata.egg-info/PKG-INFO
+wowdata.egg-info/SOURCES.txt
+wowdata.egg-info/dependency_links.txt
+wowdata.egg-info/entry_points.txt
+wowdata.egg-info/requires.txt
+wowdata.egg-info/top_level.txt

wowdata-0.1.0/wowdata.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

wowdata-0.1.0/wowdata.egg-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+wow = wowdata.cli:main
+wowdata = wowdata.cli:main

wowdata-0.1.0/wowdata.egg-info/requires.txt

@@ -0,0 +1,13 @@
+petl>=1.7
+frictionless>=5.0
+
+[dev]
+pytest>=7.0
+pytest-cov
+black
+ruff
+mypy
+
+[docs]
+mkdocs
+mkdocs-material

wowdata-0.1.0/wowdata.egg-info/top_level.txt

@@ -0,0 +1 @@
+wowdata