streamwright 0.1.4__tar.gz → 0.1.6__tar.gz

streamwright-0.1.6/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.4
+Name: streamwright
+Version: 0.1.6
+Summary: Declarative data connectors — pull from APIs, files and databases with YAML, shape with SQL, write anywhere.
+Author-email: Karthick Jaganathan <kjaganathan.sde@gmail.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/karthick-jaganathan/streamwright
+Project-URL: Repository, https://github.com/karthick-jaganathan/streamwright
+Project-URL: Documentation, https://streamwright.web.app/docs/
+Keywords: data,pipeline,etl,connector,singer,yaml,duckdb,rest-api
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pyyaml==6.0.2
+Requires-Dist: requests<3,>=2.25
+Requires-Dist: duckdb>=1.5.2
+Requires-Dist: pytz
+Provides-Extra: dlt
+Requires-Dist: dlt>=1.0; extra == "dlt"
+Dynamic: license-file
+
+# StreamWright — Declarative Data Pipelines
+
+**StreamWright turns a few YAML files into a working data pipeline.** You describe what to pull—from an API, a file store, or a database—and StreamWright handles the how: authentication, pagination, retries, shaping data with DuckDB SQL, and writing it to your destination.
+
+Build reusable data pipelines without writing per-connector code. Define your sources declaratively, install connectors on demand, and run everything through a single CLI command.
+
+## How It Works
+
+The `streamwright` package provides both the CLI tooling and execution runtime for declarative data sources.
+
+Each stream follows a three-stage pipeline:
+
+1. **Fetch** — Retrieve records through declarative REST HTTP requests, storage readers (S3, GCS, and local files), database integrations, or vendor SDK connectors.
+2. **Shape** — Filter, transform, and normalize records on the fly using embedded DuckDB SQL.
+3. **Write** — Deliver data to JSONL, CSV, TSV, and Parquet files; DuckDB and DuckLake; Singer-compatible message streams; or supported `dlt` destinations.
+
+## CLI Commands
+
+- **`streamwright run`** — Execute a source and stream records to the configured destination.
+- **`streamwright validate`** — Validate source definitions, authentication specifications, and schemas without executing network requests.
+- **`streamwright connectors`** — Browse, inspect, and install connectors from the StreamWright Hub.
+
+## Source Structure
+
+A source is defined declaratively as a directory containing a `source.yaml` file and a `streams/` directory, or as a standalone YAML file where supported.
+
+```text
+my_source/
+├── source.yaml
+└── streams/
+    ├── customers.yaml
+    └── orders.yaml
+```
+
+- **`source.yaml`** — Defines input parameters (`spec`), authentication (`auth`), and base HTTP configuration (`http`).
+- **`streams/*.yaml`** — Defines stream endpoints, pagination, rate limits, schemas, and stream-specific configuration.
+
+## Installation
+
+Install StreamWright using pip:
+
+```bash
+pip install streamwright
+streamwright --help
+```
+
+This installs the CLI, execution runtime, and connector hub client. Install additional connectors on demand:
+
+```bash
+# Browse available connectors
+streamwright connectors list
+
+# Install the generic REST API connector
+streamwright connectors install restapi
+
+# Install file and storage readers
+streamwright connectors install files
+
+# Install Google Ads API support
+streamwright connectors install google_ads
+
+# Install multiple connectors at once
+streamwright connectors install google_ads meta_ads microsoft_ads
+```
+
+The connector catalog is discovered from the live [StreamWright Hub](https://github.com/karthick-jaganathan/streamwright-hub) and cached locally in `~/.streamwright/connectors.json`.
+
+## Quickstart
+
+The repository includes runnable examples to help you get started.
+
+```bash
+# Install StreamWright and the required connector
+pip install streamwright
+streamwright connectors install files
+
+# Clone the repository
+git clone https://github.com/karthick-jaganathan/streamwright.git
+
+# Validate the example sources
+streamwright validate streamwright/examples
+
+# Run the file-reader example
+streamwright run streamwright/examples/sources/readers/files_demo \
+  --set data_root=streamwright/examples/sources/readers/files_demo/data \
+  --allow-connector files \
+  --output jsonl:out
+```
+
+## Documentation
+
+- **Full documentation:** https://streamwright.web.app/docs/
+- **Architecture Guide:** https://streamwright.web.app/docs/architecture/
+- **Connectors & Readers:** https://streamwright.web.app/docs/connectors/

streamwright-0.1.6/README.md

@@ -0,0 +1,94 @@
+# StreamWright — Declarative Data Pipelines
+
+**StreamWright turns a few YAML files into a working data pipeline.** You describe what to pull—from an API, a file store, or a database—and StreamWright handles the how: authentication, pagination, retries, shaping data with DuckDB SQL, and writing it to your destination.
+
+Build reusable data pipelines without writing per-connector code. Define your sources declaratively, install connectors on demand, and run everything through a single CLI command.
+
+## How It Works
+
+The `streamwright` package provides both the CLI tooling and execution runtime for declarative data sources.
+
+Each stream follows a three-stage pipeline:
+
+1. **Fetch** — Retrieve records through declarative REST HTTP requests, storage readers (S3, GCS, and local files), database integrations, or vendor SDK connectors.
+2. **Shape** — Filter, transform, and normalize records on the fly using embedded DuckDB SQL.
+3. **Write** — Deliver data to JSONL, CSV, TSV, and Parquet files; DuckDB and DuckLake; Singer-compatible message streams; or supported `dlt` destinations.
+
+## CLI Commands
+
+- **`streamwright run`** — Execute a source and stream records to the configured destination.
+- **`streamwright validate`** — Validate source definitions, authentication specifications, and schemas without executing network requests.
+- **`streamwright connectors`** — Browse, inspect, and install connectors from the StreamWright Hub.
+
+## Source Structure
+
+A source is defined declaratively as a directory containing a `source.yaml` file and a `streams/` directory, or as a standalone YAML file where supported.
+
+```text
+my_source/
+├── source.yaml
+└── streams/
+    ├── customers.yaml
+    └── orders.yaml
+```
+
+- **`source.yaml`** — Defines input parameters (`spec`), authentication (`auth`), and base HTTP configuration (`http`).
+- **`streams/*.yaml`** — Defines stream endpoints, pagination, rate limits, schemas, and stream-specific configuration.
+
+## Installation
+
+Install StreamWright using pip:
+
+```bash
+pip install streamwright
+streamwright --help
+```
+
+This installs the CLI, execution runtime, and connector hub client. Install additional connectors on demand:
+
+```bash
+# Browse available connectors
+streamwright connectors list
+
+# Install the generic REST API connector
+streamwright connectors install restapi
+
+# Install file and storage readers
+streamwright connectors install files
+
+# Install Google Ads API support
+streamwright connectors install google_ads
+
+# Install multiple connectors at once
+streamwright connectors install google_ads meta_ads microsoft_ads
+```
+
+The connector catalog is discovered from the live [StreamWright Hub](https://github.com/karthick-jaganathan/streamwright-hub) and cached locally in `~/.streamwright/connectors.json`.
+
+## Quickstart
+
+The repository includes runnable examples to help you get started.
+
+```bash
+# Install StreamWright and the required connector
+pip install streamwright
+streamwright connectors install files
+
+# Clone the repository
+git clone https://github.com/karthick-jaganathan/streamwright.git
+
+# Validate the example sources
+streamwright validate streamwright/examples
+
+# Run the file-reader example
+streamwright run streamwright/examples/sources/readers/files_demo \
+  --set data_root=streamwright/examples/sources/readers/files_demo/data \
+  --allow-connector files \
+  --output jsonl:out
+```
+
+## Documentation
+
+- **Full documentation:** https://streamwright.web.app/docs/
+- **Architecture Guide:** https://streamwright.web.app/docs/architecture/
+- **Connectors & Readers:** https://streamwright.web.app/docs/connectors/

{streamwright-0.1.4 → streamwright-0.1.6}/pyproject.toml

@@ -4,14 +4,14 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "streamwright"
-version = "0.1.4"
+version = "0.1.6"
 description = "Declarative data connectors — pull from APIs, files and databases with YAML, shape with SQL, write anywhere."
 readme = "README.md"
-license = "Apache-2.0"
+license = {text = "Apache-2.0"}
 authors = [
-    {name = "Karthick Jaganathan"},
+    {name = "Karthick Jaganathan", email = "kjaganathan.sde@gmail.com"},
 ]
-keywords = ["data", "pipeline", "etl", "connector", "singer", "yaml"]
+keywords = ["data", "pipeline", "etl", "connector", "singer", "yaml", "duckdb", "rest-api"]
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Intended Audience :: Developers",
@@ -44,6 +44,7 @@ Documentation = "https://streamwright.web.app/docs/"
 # `streamwright run`, `streamwright validate`, `streamwright connectors` - one CLI (see streamwright.core.cli)
 streamwright = "streamwright.core.cli:main"
 
+
 [tool.setuptools]
 # src layout: `streamwright` is a namespace package shared with the connector distributions (streamwright.connectors.*)
 package-dir = {"" = "src"}
@@ -57,6 +58,3 @@ packages = [
     "streamwright.core.runtime",
 ]
 
-[tool.setuptools.package-data]
-# the bundled connector catalog (default for `streamwright connectors`, refreshed from $STREAMWRIGHT_HUB_URL)
-"streamwright.core" = ["connectors.json"]

{streamwright-0.1.4 → streamwright-0.1.6}/src/streamwright/core/catalog.py

@@ -1,11 +1,13 @@
 #!/usr/bin/env python
 """The connector catalog: discover and install connectors from the StreamWright hub.
 
-A catalog is a static index (bundled with streamwright, optionally refreshed from a
-remote hub URL) that maps a connector key to how it's installed - a PyPI
-requirement, a pinned git URL, or an image. The key is the same name used in
-source YAML (`provider`/`sdk`); the install source is just a delivery detail, so
-a package can move from git to PyPI without changing what users type.
+A catalog is a static index that maps a connector key to how it's installed - a
+PyPI requirement, a pinned git URL, or an image. It is read from the live hub
+(DEFAULT_HUB, or $STREAMWRIGHT_HUB_URL / --hub-url); when the hub is unreachable,
+from the last copy fetched (~/.streamwright/connectors.json) with a warning.
+The key is the same name used in source YAML (`provider`/`sdk`); the install source
+is just a delivery detail, so a package can move from git to PyPI without changing
+what users type.
 
 Security: non-"official" connectors run third-party code, so `install` asks for
 confirmation (or `--yes`) and refuses non-interactive installs without it.
@@ -16,28 +18,26 @@ import subprocess
 import sys
 from pathlib import Path
 
-try:
-    from importlib import resources
-except ImportError:  # pragma: no cover
-    import importlib_resources as resources
-
 from streamwright.core.runtime import components
 
 HUB_ENV = "STREAMWRIGHT_HUB_URL"
+DEFAULT_HUB = "https://karthick-jaganathan.github.io/streamwright-hub/index.json"
 CACHE = Path(os.path.expanduser("~/.streamwright/connectors.json"))
+TIMEOUT_S = 5
 
 
-def _bundled():
-    text = resources.files("streamwright.core").joinpath("connectors.json").read_text()
-    return json.loads(text)
+def _valid(data):
+    return isinstance(data, dict) and isinstance(data.get("connectors"), dict)
 
 
 def _remote(url):
     import requests
 
-    response = requests.get(url, timeout=10)
+    response = requests.get(url, timeout=TIMEOUT_S)
     response.raise_for_status()
     data = response.json()
+    if not _valid(data):
+        raise ValueError("not a connector catalog (no `connectors` mapping)")
     try:
         CACHE.parent.mkdir(parents=True, exist_ok=True)
         CACHE.write_text(json.dumps(data))
@@ -47,19 +47,35 @@ def _remote(url):
 
 
 def load(hub_url=None):
-    """The catalog: the remote hub if configured (cached), else the bundled default."""
-    url = hub_url or os.environ.get(HUB_ENV)
-    if url:
-        try:
-            return _remote(url)
-        except Exception:  # offline or hub down: fall back
-            pass
+    """
+    The catalog: the live hub (`hub_url`, else $STREAMWRIGHT_HUB_URL, else DEFAULT_HUB; each fetch is cached). When the
+    hub cannot be read, the cached copy is used with a warning on stderr.
+    """
+    url = hub_url or os.environ.get(HUB_ENV) or DEFAULT_HUB
+    try:
+        return _remote(url)
+    except Exception as exc:  # offline, hub down or a bad publish: fall back, visibly
+        reason = type(exc).__name__  # network errors: the type is enough; an HTTP status or a bad index says why
+        status = getattr(getattr(exc, "response", None), "status_code", None)
+        if status:
+            reason = "HTTP %s" % status
+        elif isinstance(exc, ValueError):
+            reason += ": %s" % str(exc)[:100]
     if CACHE.exists():
         try:
-            return json.loads(CACHE.read_text())
+            data = json.loads(CACHE.read_text())
+            if _valid(data):
+                sys.stderr.write("streamwright: the connector hub %s is unreachable (%s); using the copy fetched on %s\n"
+                                 % (url, reason, _mtime(CACHE)))
+                return data
         except Exception:
             pass
-    return _bundled()
+    raise RuntimeError("the connector hub %s is unreachable (%s) and no cached catalog is available" % (url, reason))
+
+
+def _mtime(path):
+    import datetime
+    return datetime.datetime.fromtimestamp(path.stat().st_mtime).strftime("%Y-%m-%d %H:%M")
 
 
 def connectors(hub_url=None):

{streamwright-0.1.4 → streamwright-0.1.6}/src/streamwright/core/cli.py

@@ -37,6 +37,7 @@ Exit status: 0 success, 1 the run failed, 2 invalid source or inputs.
 
 import argparse
 import datetime
+import functools
 import json
 import logging
 import os
@@ -247,7 +248,8 @@ def _parser():
                       help="connector keys (install: one or more) or words (search)")
     conn.add_argument("--yes", action="store_true", help="skip the confirmation prompt when installing")
     conn.add_argument("--hub-url", metavar="URL",
-                      help="catalog URL to use (default: the bundled catalog, or $STREAMWRIGHT_HUB_URL)")
+                      help="the connector hub (default: $STREAMWRIGHT_HUB_URL, else the live StreamWright hub; when it is "
+                           "unreachable, the cached catalog is used, with a warning)")
 
     validate = commands.add_parser("validate", help="check configuration files: streamwright-validate, plus the checks of "
                                                      "the installed connectors and query builders", add_help=False)
@@ -507,8 +509,17 @@ def _log_error(message):
     return None
 
 
+@functools.lru_cache(maxsize=None)
+def _pkg_version(dist_name):
+    try:
+        from importlib import metadata
+        return metadata.version(dist_name)
+    except Exception:
+        return None
+
+
 def _connector_lines():
-    """`streamwright connectors`: installed connectors grouped by category, each with its description and SDK loggers."""
+    """`streamwright connectors`: installed connectors grouped by category using connector.spec."""
     groups = {}
     for name in components.available():
         try:
@@ -516,18 +527,31 @@ def _connector_lines():
         except components.ComponentLoadError as exc:
             groups.setdefault("other", []).append("  %s — %s" % (name, exc))
             continue
-        loggers = getattr(connector, "network_loggers", None) or ()
-        loggers = (loggers,) if isinstance(loggers, str) else tuple(loggers)
-        parts = [name]
-        summary = getattr(connector, "summary", "") or ""
-        if summary:
-            parts.append("— " + summary)
-        if loggers:
-            parts.append("(SDK loggers: %s)" % ", ".join(loggers))
-        category = getattr(connector, "category", "other") or "other"
-        groups.setdefault(category, []).append("  " + " ".join(parts))
+        spec = getattr(connector, "spec", None)
+        if spec is None:
+            spec = components.ConnectorSpec(name=name)
+        prefix = "  [installed]  "
+        conn_name = spec.name or name
+        title = spec.title
+        client_tag = spec.badge(_pkg_version)
+        loggers = spec.effective_loggers()
+
+        if client_tag and loggers:
+            logger_str = ", ".join(loggers)
+            line = "%s%-15s%-23s%-32s  [logger: %s]" % (prefix, conn_name, title, client_tag, logger_str)
+        elif client_tag:
+            line = "%s%-15s%-23s%s" % (prefix, conn_name, title, client_tag)
+        elif loggers:
+            logger_str = ", ".join(loggers)
+            line = "%s%-15s%-23s%-32s  [logger: %s]" % (prefix, conn_name, title, "", logger_str)
+        else:
+            line = "%s%-15s%s" % (prefix, conn_name, title)
+
+        groups.setdefault(spec.category, []).append(line)
     lines = []
     for category in sorted(groups):
+        if lines:
+            lines.append("")
         lines.append("%s:" % category)
         lines.extend(sorted(groups[category]))
     return lines

{streamwright-0.1.4 → streamwright-0.1.6}/src/streamwright/core/engine/runner.py

@@ -39,7 +39,9 @@ import requests
 from streamwright.core.net import downloads
 from streamwright.core.runtime import components, logs
 from streamwright.core.engine import queries, sql
-from streamwright.core.net.http import Authenticator, HttpClient, HttpError, RateLimiter, Redactor, error_excerpt, paginate
+from streamwright.core.net.http import Authenticator, HttpClient, HttpError, RateLimiter, Redactor, error_excerpt, paginate, merge_headers
+from streamwright.core.net.encoding import encode_params
+from streamwright.core.net.paginators import set_dotted_path
 from streamwright.core.config.inputs import as_date, parse_duration
 from streamwright.core.runtime.logs import RunMetrics
 from streamwright.core.runtime.components import ConnectorContext, ConnectorError, ComponentLoadError
@@ -236,6 +238,7 @@ class SourceRunner(object):
                  clock=time.monotonic, sleep=time.sleep, redact=None, allowed_connectors=None, metrics=None):
         self.source = source
         self.config = config
+        self.secrets = secrets
         self.output = output
         self.today = today or datetime.date.today()
         self.state = copy.deepcopy(state) if state else {}
@@ -273,12 +276,30 @@ class SourceRunner(object):
                 self.connector = components.load(self.provider, allowed_connectors)
             except ComponentLoadError as exc:
                 raise SourceError(str(exc))
-            self.connector_auth = render(dict((key, value) for key, value in auth.items() if key != "provider"),
-                                         dict(self.scopes(), secrets=secrets))
+            if getattr(self.connector, "transport", "sdk") == "http":
+                auth_block = dict((key, value) for key, value in auth.items() if key != "provider")
+                rendered = render(auth_block, dict(self.scopes(), secrets=secrets))
+                self.authenticator = Authenticator(rendered, self.session, self.redact)
+                self.connector_auth = None
+            else:
+                self.connector_auth = render(dict((key, value) for key, value in auth.items() if key != "provider"),
+                                             dict(self.scopes(), secrets=secrets))
+            if self.allowed_connectors is not None:
+                extra = getattr(self.connector, "query_builders", ())
+                if extra:
+                    self.allowed_connectors = list(self.allowed_connectors) + [b for b in extra if b not in self.allowed_connectors]
         else:
             rendered = render(auth, dict(self.scopes(), secrets=secrets))
             self.authenticator = Authenticator(rendered, self.session, self.redact)
-        self.http = render(source.get("http") or {}, self.scopes())
+        source_http = dict(source.get("http") or {})
+        headers_spec = source_http.pop("headers", None)
+        self.http = render(source_http, self.scopes())
+        if headers_spec is not None:
+            rendered_headers = render(headers_spec, dict(self.scopes(), secrets=secrets))
+            if isinstance(rendered_headers, dict):
+                self.http["headers"] = dict((k, v) for k, v in rendered_headers.items() if v not in ("", None))
+            else:
+                self.http["headers"] = rendered_headers
         # the source-level limit is shared by every stream; a stream's own `rate_limit` replaces it
         self.shared_limiter = self.limiter(self.http.get("rate_limit"))
 
@@ -554,9 +575,14 @@ class SourceRunner(object):
             if "async_job" in request and "sdk" not in (request["async_job"].get("submit") or {}):
                 problems.append("%s: an async job's `submit` must be an sdk request (its `poll` calls the same "
                                 "service)" % where)
-            if self.connector is not None and _http_requests(request):
-                problems.append("%s: http requests need a built-in `auth.type`; auth provider %r only authenticates "
-                                "sdk requests" % (where, self.provider))
+            if self.connector is not None:
+                if getattr(self.connector, "transport", "sdk") == "http":
+                    if calls:
+                        problems.append("%s: %s is a REST API connector: use an http request (http: {path, method, ...}), not sdk" % (where, self.provider))
+                else:
+                    if _http_requests(request):
+                        problems.append("%s: http requests need a built-in `auth.type`; auth provider %r only authenticates "
+                                        "sdk requests" % (where, self.provider))
         return problems
 
     def run_stream(self, stream):
@@ -586,9 +612,10 @@ class SourceRunner(object):
         client = self.client(stream, limiter)
         context = None
         if any(components.sdk_calls(components.request_body(item)) for item in stream.get("requests") or []):
-            context = ConnectorContext(self.connector, self.redact, retry=self.retry_policy(stream),
-                                       rate_limiter=limiter, sleep=self.sleep, metrics=self.metrics, clock=self.clock)
-            self.connect(context)
+            if self.connector and getattr(self.connector, "transport", "sdk") != "http":
+                context = ConnectorContext(self.connector, self.redact, retry=self.retry_policy(stream),
+                                           rate_limiter=limiter, sleep=self.sleep, metrics=self.metrics, clock=self.clock)
+                self.connect(context)
         return client, context
 
     def parameters(self, plan, step):
@@ -808,7 +835,17 @@ class SourceRunner(object):
     def request_select(self, request, response):
         """The records in one response of a request: its `records.path` (or the response itself) as a list."""
         path = (request.get("records") or {}).get("path")
-        data = response if not path else _lookup(response, path)
+        if not path:
+            data = response
+        else:
+            try:
+                data = get_path(response, path)
+            except (KeyError, TypeError, IndexError):
+                # a response without the path's first key means a wrong path; below it, an absent or null value
+                # (`{"data": null}`, an empty SOAP element) means no records
+                if isinstance(response, dict) and response and path.split(".")[0] not in response:
+                    raise SourceError("records path %r not found in the response" % (path,))
+                return []
         if data is None:
             return []
         if isinstance(data, dict):
@@ -824,13 +861,16 @@ class SourceRunner(object):
         (RunMetrics.read) that counts the pages.
         """
         body = components.request_body(request)
-        explode = (request.get("records") or {}).get("explode")
+        effective_records = request["records"] if "records" in request else self.http.get("records")
+        explode = (effective_records or {}).get("explode")
 
         def select(response):
-            return self.request_select(request, response)
+            return self.request_select({"records": effective_records}, response)
         if "http" in body:
             pages = self.http_pages(body["http"], request, client, scopes, where)
         elif "sdk" in body:
+            if self.connector and getattr(self.connector, "transport", "sdk") == "http":
+                raise SourceError("%s is a REST API connector: use an http request (http: {path, method, ...}), not sdk" % self.provider)
             pages = self.sdk_responses(body, context, scopes, where, select)
         else:
             pages = self.async_job(stream, body["async_job"], client, context, scopes, select, where)
@@ -840,22 +880,48 @@ class SourceRunner(object):
 
     def http_pages(self, request, item, client, scopes, where=None):
         """The pages of an http request; `item`: its request item (`paginator`, `records`)."""
-        http = self.build(request, render(request, scopes))
-
-        def send(params, url):
-            if url:
-                return client.request(http.get("method", "GET"), client.follow(url), headers=http.get("headers"),
-                                      fields=where)
-            merged = dict(http.get("params") or {}, **params)
-            return client.request(http.get("method", "GET"), http["path"], params=merged,
-                                  headers=http.get("headers"), json_body=http.get("json"), fields=where)
-
-        def paginator_for(response):
-            if item.get("paginator") is None:
+        effective_paginator_spec = item["paginator"] if "paginator" in item else self.http.get("paginator")
+        effective_records = item["records"] if "records" in item else self.http.get("records")
+        effective_encoding = request.get("params_encoding") or self.http.get("params_encoding", "plain")
+
+        unrendered_req = dict(request)
+        req_headers_spec = unrendered_req.pop("headers", None)
+        http = self.build(unrendered_req, render(unrendered_req, scopes))
+
+        rendered_req_headers = render(req_headers_spec, dict(scopes, secrets=self.secrets)) if req_headers_spec else {}
+        merged_headers = merge_headers(self.http.get("headers"), rendered_req_headers)
+
+        def send(patch):
+            if patch.url:
+                url = client.follow(patch.url)
+                res = client.request(http.get("method", "GET"), url, headers=merged_headers, fields=where)
+                send.last_headers = client.last_headers
+                return res
+
+            params = dict(http.get("params") or {})
+            if patch.params:
+                params.update(patch.params)
+            encoded_params = encode_params(params, effective_encoding)
+
+            json_body = http.get("json")
+            if patch.body:
+                json_copy = copy.deepcopy(json_body) if json_body is not None else {}
+                for path, val in patch.body.items():
+                    set_dotted_path(json_copy, path, val)
+                json_body = json_copy
+
+            res = client.request(http.get("method", "GET"), http["path"], params=encoded_params,
+                                 headers=merged_headers, json_body=json_body, fields=where)
+            send.last_headers = client.last_headers
+            return res
+
+        def paginator_for_resp(response):
+            if effective_paginator_spec is None:
                 return None
-            return render(item["paginator"], dict(scopes, response=response or {}))
+            return render(effective_paginator_spec, dict(scopes, response=response or {}))
 
-        for _, records in paginate(send, paginator_for, lambda response: self.request_select(item, response)):
+        select_item = {"records": effective_records}
+        for _, records in paginate(send, paginator_for_resp, lambda response: self.request_select(select_item, response)):
             yield records
 
     def build(self, unrendered, rendered):