streamwright 0.0.1__tar.gz → 0.1.2__tar.gz

streamwright-0.1.2/PKG-INFO

@@ -0,0 +1,77 @@
+Metadata-Version: 2.4
+Name: streamwright
+Version: 0.1.2
+Summary: Declarative data connectors — pull from APIs, files and databases with YAML, shape with SQL, write anywhere.
+Author: Karthick Jaganathan
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/karthick-jaganathan/streamwright
+Project-URL: Repository, https://github.com/karthick-jaganathan/streamwright
+Project-URL: Documentation, https://streamwright.web.app/docs/
+Keywords: data,pipeline,etl,connector,singer,yaml
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml==6.0.2
+Requires-Dist: requests<3,>=2.25
+Requires-Dist: duckdb>=1.5.2
+Requires-Dist: pytz
+Provides-Extra: dlt
+Requires-Dist: dlt>=1.0; extra == "dlt"
+Dynamic: license-file
+
+# StreamWright — the `streamwright` CLI and runtime
+
+**StreamWright turns a few YAML files into a working data pipeline.** You describe *what*
+to pull — from an API, a file store or a database — and StreamWright handles the *how*:
+signing in, paging, retries, shaping the data with DuckDB SQL, and writing it out.
+No per-connector code, and one command runs any source.
+
+The `streamwright` package is the heart of it: the `streamwright` command plus the runtime for
+`kind: source` sources. *(Formerly published as `adapt-core`.)*
+
+- **`streamwright run`** executes a source · **`streamwright validate`** checks one · **`streamwright connectors`** browses and installs connectors.
+- A source is a folder — `source.yaml` (`spec`, `auth`, `http`) and one file per stream in `streams/` — or a single YAML file.
+- Each stream **fetches** (HTTP requests / SDK connectors), **shapes** (sandboxed DuckDB SQL), and **writes** (files, DuckDB/DuckLake or dlt).
+
+## Install
+
+```bash
+pip install streamwright
+streamwright --help
+```
+
+This installs the `streamwright` CLI, the runtime and the connector hub — but no
+connectors yet. Add the ones you need on demand:
+
+```bash
+streamwright connectors list                 # browse the hub
+streamwright connectors install files        # a reader (files, s3, gcs, postgres)
+streamwright connectors install google_ads   # an API connector (pulls in its SDK)
+```
+
+## Quickstart
+
+```bash
+pip install streamwright
+streamwright connectors install files
+
+# the runnable examples live in the repo — clone it to try one end-to-end
+git clone https://github.com/karthick-jaganathan/streamwright.git
+streamwright validate streamwright/examples
+streamwright run streamwright/examples/sources/readers/files_demo \
+  --set data_root=streamwright/examples/sources/readers/files_demo/data \
+  --allow-connector files --output jsonl:out
+```
+
+📖 **Full documentation:** https://streamwright.web.app/docs/ — command line,
+streams, outputs, logging, connectors & readers and the Python API.
+
+- Architecture guide: https://streamwright.web.app/docs/architecture/
+- Connectors & readers: https://streamwright.web.app/docs/connectors/

streamwright-0.1.2/README.md

@@ -0,0 +1,49 @@
+# StreamWright — the `streamwright` CLI and runtime
+
+**StreamWright turns a few YAML files into a working data pipeline.** You describe *what*
+to pull — from an API, a file store or a database — and StreamWright handles the *how*:
+signing in, paging, retries, shaping the data with DuckDB SQL, and writing it out.
+No per-connector code, and one command runs any source.
+
+The `streamwright` package is the heart of it: the `streamwright` command plus the runtime for
+`kind: source` sources. *(Formerly published as `adapt-core`.)*
+
+- **`streamwright run`** executes a source · **`streamwright validate`** checks one · **`streamwright connectors`** browses and installs connectors.
+- A source is a folder — `source.yaml` (`spec`, `auth`, `http`) and one file per stream in `streams/` — or a single YAML file.
+- Each stream **fetches** (HTTP requests / SDK connectors), **shapes** (sandboxed DuckDB SQL), and **writes** (files, DuckDB/DuckLake or dlt).
+
+## Install
+
+```bash
+pip install streamwright
+streamwright --help
+```
+
+This installs the `streamwright` CLI, the runtime and the connector hub — but no
+connectors yet. Add the ones you need on demand:
+
+```bash
+streamwright connectors list                 # browse the hub
+streamwright connectors install files        # a reader (files, s3, gcs, postgres)
+streamwright connectors install google_ads   # an API connector (pulls in its SDK)
+```
+
+## Quickstart
+
+```bash
+pip install streamwright
+streamwright connectors install files
+
+# the runnable examples live in the repo — clone it to try one end-to-end
+git clone https://github.com/karthick-jaganathan/streamwright.git
+streamwright validate streamwright/examples
+streamwright run streamwright/examples/sources/readers/files_demo \
+  --set data_root=streamwright/examples/sources/readers/files_demo/data \
+  --allow-connector files --output jsonl:out
+```
+
+📖 **Full documentation:** https://streamwright.web.app/docs/ — command line,
+streams, outputs, logging, connectors & readers and the Python API.
+
+- Architecture guide: https://streamwright.web.app/docs/architecture/
+- Connectors & readers: https://streamwright.web.app/docs/connectors/

streamwright-0.1.2/pyproject.toml

@@ -0,0 +1,62 @@
+[build-system]
+requires = ["setuptools>=61.0.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "streamwright"
+version = "0.1.2"
+description = "Declarative data connectors — pull from APIs, files and databases with YAML, shape with SQL, write anywhere."
+readme = "README.md"
+license = "Apache-2.0"
+authors = [
+    {name = "Karthick Jaganathan"},
+]
+keywords = ["data", "pipeline", "etl", "connector", "singer", "yaml"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+requires-python = ">=3.10"
+dependencies = [
+    "pyyaml==6.0.2",
+    "requests>=2.25,<3",
+    # `select`: the SQL that shapes each stream's records runs in an embedded DuckDB (which needs pytz for
+    # timestamps with time zones)
+    "duckdb>=1.5.2",
+    "pytz",
+]
+
+[project.optional-dependencies]
+# --output dlt:DESTINATION; also install the destination's extra, e.g. "dlt[duckdb]" or "dlt[bigquery]"
+dlt = ["dlt>=1.0"]
+
+[project.urls]
+Homepage = "https://github.com/karthick-jaganathan/streamwright"
+Repository = "https://github.com/karthick-jaganathan/streamwright"
+Documentation = "https://streamwright.web.app/docs/"
+
+[project.scripts]
+# `streamwright run`, `streamwright validate`, `streamwright connectors` - one CLI (see streamwright.core.cli)
+streamwright = "streamwright.core.cli:main"
+
+[tool.setuptools]
+# src layout: `streamwright` is a namespace package shared with the connector distributions (streamwright.connectors.*)
+package-dir = {"" = "src"}
+packages = [
+    "streamwright.core",
+    "streamwright.core.engine",
+    "streamwright.core.outputs",
+    "streamwright.core.net",
+    "streamwright.core.config",
+    "streamwright.core.validation",
+    "streamwright.core.runtime",
+]
+
+[tool.setuptools.package-data]
+# the bundled connector catalog (default for `streamwright connectors`, refreshed from $STREAMWRIGHT_HUB_URL)
+"streamwright.core" = ["connectors.json"]

streamwright-0.1.2/setup.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+# /*************************************************************************
+# * Copyright 2025 Karthick Jaganathan
+# *
+# * Licensed under the Apache License, Version 2.0 (the "License");
+# * you may not use this file except in compliance with the License.
+# * You may obtain a copy of the License at
+# *
+# * https://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# **************************************************************************/
+
+from setuptools import setup
+
+# Configuration is now in pyproject.toml
+setup()

streamwright-0.1.2/src/streamwright/core/__init__.py

@@ -0,0 +1,18 @@
+#!/usr/bin/env python
+# /*************************************************************************
+# * Copyright 2025 Karthick Jaganathan
+# *
+# * Licensed under the Apache License, Version 2.0 (the "License");
+# * you may not use this file except in compliance with the License.
+# * You may obtain a copy of the License at
+# *
+# * https://www.apache.org/licenses/LICENSE-2.0
+# *
+# * Unless required by applicable law or agreed to in writing, software
+# * distributed under the License is distributed on an "AS IS" BASIS,
+# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# * See the License for the specific language governing permissions and
+# * limitations under the License.
+# **************************************************************************/
+
+"""Run-time StreamWright sources (`kind: source`). Entry point: the `streamwright` command (streamwright.core.cli)."""

streamwright-0.1.2/src/streamwright/core/catalog.py

@@ -0,0 +1,150 @@
+#!/usr/bin/env python
+"""The connector catalog: discover and install connectors from the StreamWright hub.
+
+A catalog is a static index (bundled with streamwright, optionally refreshed from a
+remote hub URL) that maps a connector key to how it's installed - a PyPI
+requirement, a pinned git URL, or an image. The key is the same name used in
+source YAML (`provider`/`sdk`); the install source is just a delivery detail, so
+a package can move from git to PyPI without changing what users type.
+
+Security: non-"official" connectors run third-party code, so `install` asks for
+confirmation (or `--yes`) and refuses non-interactive installs without it.
+"""
+import json
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+try:
+    from importlib import resources
+except ImportError:  # pragma: no cover
+    import importlib_resources as resources
+
+from streamwright.core.runtime import components
+
+HUB_ENV = "STREAMWRIGHT_HUB_URL"
+CACHE = Path(os.path.expanduser("~/.streamwright/connectors.json"))
+
+
+def _bundled():
+    text = resources.files("streamwright.core").joinpath("connectors.json").read_text()
+    return json.loads(text)
+
+
+def _remote(url):
+    import requests
+
+    response = requests.get(url, timeout=10)
+    response.raise_for_status()
+    data = response.json()
+    try:
+        CACHE.parent.mkdir(parents=True, exist_ok=True)
+        CACHE.write_text(json.dumps(data))
+    except OSError:
+        pass
+    return data
+
+
+def load(hub_url=None):
+    """The catalog: the remote hub if configured (cached), else the bundled default."""
+    url = hub_url or os.environ.get(HUB_ENV)
+    if url:
+        try:
+            return _remote(url)
+        except Exception:  # offline or hub down: fall back
+            pass
+    if CACHE.exists():
+        try:
+            return json.loads(CACHE.read_text())
+        except Exception:
+            pass
+    return _bundled()
+
+
+def connectors(hub_url=None):
+    return load(hub_url).get("connectors", {})
+
+
+def installed():
+    try:
+        return set(components.available())
+    except Exception:
+        return set()
+
+
+def search(text, hub_url=None):
+    text = (text or "").lower()
+    hits = {}
+    for key, entry in connectors(hub_url).items():
+        haystack = " ".join([key, entry.get("title", ""), entry.get("summary", ""),
+                             entry.get("family", "")]).lower()
+        if text in haystack:
+            hits[key] = entry
+    return hits
+
+
+def _source_of(install):
+    if "pip" in install:
+        return "PyPI: " + install["pip"]
+    if "git" in install:
+        return "git: " + install["git"]
+    if "image" in install:
+        return "image: " + install["image"]
+    return "unknown"
+
+
+def listing(hub_url=None):
+    """Lines for `streamwright connectors list`: every catalog connector with install state + trust."""
+    have = installed()
+    lines = []
+    by_family = {}
+    for key, entry in connectors(hub_url).items():
+        by_family.setdefault(entry.get("family", "other"), []).append((key, entry))
+    for family in sorted(by_family):
+        lines.append("%s:" % family)
+        for key, entry in sorted(by_family[family]):
+            mark = "[installed]" if key in have else "          "
+            trust = entry.get("trust", "community")
+            lines.append("  %s %-16s %-10s %s" % (mark, key, trust, entry.get("summary", "")))
+    return lines
+
+
+def install(key, assume_yes=False, hub_url=None, pip_args=()):
+    """Resolve `key` in the catalog and install it (pip for pip/git; images are run, not installed)."""
+    entry = connectors(hub_url).get(key)
+    if not entry:
+        sys.stderr.write("streamwright: unknown connector %r (try: streamwright connectors list)\n" % key)
+        return 2
+    inst = entry.get("install", {})
+    trust = entry.get("trust", "community")
+
+    if "image" in inst:
+        print("%r is distributed as an image, not a Python package:" % key)
+        print("  %s" % inst["image"])
+        print("Run it with the docker or k8s execution mode (it is not pip-installed).")
+        return 0
+
+    spec = inst.get("pip") or inst.get("git")
+    if not spec:
+        sys.stderr.write("streamwright: connector %r has no installable source\n" % key)
+        return 2
+
+    if trust != "official" and not assume_yes:
+        source = _source_of(inst)
+        if not sys.stdin.isatty():
+            sys.stderr.write(
+                "streamwright: refusing to install %s connector %r non-interactively; pass --yes to confirm (%s)\n"
+                % (trust, key, source))
+            return 2
+        sys.stderr.write(
+            "\n  WARNING: %r is a %s connector. Installing runs third-party code from:\n    %s\n"
+            % (key, trust, source))
+        answer = input("  Continue? [y/N] ").strip().lower()
+        if answer not in ("y", "yes"):
+            print("aborted")
+            return 1
+
+    cmd = [sys.executable, "-m", "pip", "install", *pip_args, spec]
+    print("+ " + " ".join(cmd))
+    return subprocess.call(cmd)