afly 0.1.0__tar.gz

afly-0.1.0/CHANGELOG.md

@@ -0,0 +1,46 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-09-24
+
+First public release.
+
+### Added
+
+- `afly init <name>` — scaffold a project: `afly_project.yml`, `profiles.yml`
+  (credentials via `{{ env_var('…') }}` / `${…}`), `.env.example`, and three
+  starter extracts (`standard`, `facebook`, `yandex`) demonstrating the
+  `exclude_media_sources` ownership convention.
+- `afly run` — idempotent AppsFlyer aggregate Pull API → ClickHouse extraction:
+  - watermark-based windows with per-extract `lookback_days`, epoch-aligned
+    `chunk_days` chunks, `--from/--to/--full-refresh`;
+  - partition rebuild via staging + `REPLACE PARTITION` (or `DROP PARTITION`
+    when nothing remains): no duplicates, no `FINAL` for readers, rows that
+    vanish from AppsFlyer vanish from the table, no row-level dedup (rows
+    identical on every dimension within one pull are all kept);
+  - `partition_granularity: month | day` (default `month`);
+  - quota-aware scheduler: per-key spacing, daily long-call budgets,
+    rate-limited jobs deferred without blocking other apps, lookahead across
+    waves (`max_waves_in_flight`), `--max-calls/--max-minutes`;
+  - empty-response guard, per-table locks, `--dry-run`, `--json` summary.
+- One destination table for all apps of an account, snake_case columns with
+  `app_id`, Facebook campaign/adset/adgroup detail, in-app events as
+  `Map` columns, and a `currency` column (AppsFlyer reports money in each
+  app's own currency).
+- ClickHouse over the native protocol or HTTP (`protocol: native | http`),
+  tested on ClickHouse 22.11 and 26.3; optional separate `internal_database`
+  and `staging_database`.
+- `afly ls`, `afly validate`, `afly apps`, `afly debug` (connectivity, grants,
+  `--deep` partition-swap drill, `--pull` sample pull), `afly unlock`.
+- `afly init-claude` — Claude Code context (`CLAUDE.md` block,
+  `.claude/rules/afly/`, four skills), idempotent and refreshable.
+- Failure alerting to Mattermost/Slack/webhook, once per run, never on quota
+  or policy skips.
+- Documentation in `docs/`: getting started, configuration, extracts, formats
+  and the Facebook split, idempotency, quotas, scheduling, alerting.

afly-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 selishchev
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

afly-0.1.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE
+include README.md
+include CHANGELOG.md
+recursive-include afly/cli/assets *.md

afly-0.1.0/PKG-INFO

@@ -0,0 +1,177 @@
+Metadata-Version: 2.4
+Name: afly
+Version: 0.1.0
+Summary: Idempotent AppsFlyer aggregate Pull API → ClickHouse extractor with a dbt-style CLI
+Author: selishchev
+License: MIT
+Project-URL: Homepage, https://github.com/selishchev/afly
+Project-URL: Repository, https://github.com/selishchev/afly
+Project-URL: Changelog, https://github.com/selishchev/afly/blob/main/CHANGELOG.md
+Keywords: appsflyer,clickhouse,etl,marketing,cli
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Database
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click>=8.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.25.0
+Requires-Dist: clickhouse-driver>=0.2.6
+Requires-Dist: clickhouse-connect>=0.8
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == "dev"
+Requires-Dist: pytest-mock; extra == "dev"
+Requires-Dist: requests-mock; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: types-PyYAML; extra == "dev"
+Requires-Dist: types-requests; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Requires-Dist: detect-secrets; extra == "dev"
+Provides-Extra: integration
+Requires-Dist: testcontainers[clickhouse]>=4.0; extra == "integration"
+Dynamic: license-file
+
+# afly
+
+[![CI](https://github.com/selishchev/afly/actions/workflows/ci.yml/badge.svg)](https://github.com/selishchev/afly/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/afly.svg)](https://pypi.org/project/afly/)
+[![Python](https://img.shields.io/pypi/pyversions/afly.svg)](https://pypi.org/project/afly/)
+
+**Idempotent AppsFlyer aggregate reports → ClickHouse, with a dbt-style
+CLI.**
+
+`afly` pulls AppsFlyer's [aggregate Pull
+API](https://support.appsflyer.com/hc/en-us/articles/207034346-Pull-API-aggregate-data) reports and
+writes them into ClickHouse. It's a dbt/detectkit-style project: an
+`afly_project.yml` describes the project, `profiles.yml` holds credentials,
+and `extracts/*.yml` declare which reports to pull and where they land.
+
+
+## Why
+
+- **Idempotent by construction, not by convention.** Every run rebuilds
+  whole ClickHouse partitions atomically (`REPLACE PARTITION`) instead of
+  appending — re-running the same window, or recovering from a crash mid-run,
+  never duplicates a row and never needs `FINAL` on read.
+- **Quota-aware.** AppsFlyer enforces a per-minute limit and a small daily
+  budget for wider date ranges. afly's scheduler interleaves apps/report
+  types to stay under both automatically, and a backfill can be capped with
+  `--max-calls`/`--max-minutes` and safely resumed later.
+- **The Facebook split is a config concern, not a bug.** AppsFlyer only
+  returns campaign/adset/adgroup breakdown columns for a Facebook-scoped
+  pull — afly's scaffold ships a dedicated `facebook` extract alongside an
+  unfiltered one that explicitly excludes it, so the ownership is explicit
+  instead of accidentally double-counted.
+- **A `--dry-run` you can actually trust.** Before pulling anything, `afly
+  run --dry-run` prints the exact window/chunk plan and the AppsFlyer quota
+  totals it would spend, flagging anything that would exceed budget.
+
+## Install
+
+```bash
+pip install afly
+```
+
+Requires Python 3.10+.
+
+## 60-second quickstart
+
+```bash
+afly init my_project && cd my_project
+cp .env.example .env        # fill in AppsFlyer token + ClickHouse credentials
+set -a; source .env; set +a
+
+afly validate                       # config sanity check, no network calls
+afly debug                          # probe AppsFlyer/ClickHouse connectivity
+afly run --select "*" --dry-run     # print the plan, pull nothing
+afly run --select "*"               # pull for real
+```
+
+```sql
+SELECT date, media_source, sum(total_cost)
+FROM appsflyer.appsflyer_geo_by_date
+GROUP BY 1, 2
+ORDER BY 1, 2;
+```
+
+See [Quickstart](docs/getting-started/quickstart.md) for the full walkthrough.
+
+## The three-extract picture
+
+`afly init` scaffolds a realistic split that demonstrates the ownership
+convention every extra media-source extract should follow:
+
+```yaml
+# extracts/standard.yml — everything else
+name: standard
+report_type: geo_by_date_report
+exclude_media_sources: [Facebook Ads, yandexdirect_int]   # owned below
+table: appsflyer_geo_by_date
+
+# extracts/facebook.yml — needs its own scoped pull for adset/adgroup columns
+name: facebook
+media_source: facebook
+table: appsflyer_geo_by_date
+
+# extracts/yandex.yml — reports arrive late, so re-pull a wider window
+name: yandex
+media_source: yandexdirect_int
+lookback_days: 7
+table: appsflyer_geo_by_date
+```
+
+All three write the same table safely, because `standard` explicitly excludes
+what the other two own. See [Extracts
+guide](docs/guides/extracts.md) and [Facebook
+split](docs/guides/formats-and-facebook-split.md).
+
+## AI-native onboarding
+
+```bash
+afly init-claude
+```
+
+Scaffolds `CLAUDE.md` + `.claude/rules/afly/` + four skills
+(`afly-setup-project`, `afly-new-extract`, `afly-backfill`,
+`afly-debug-run`) into the project, so Claude Code (or another AI assistant)
+can configure extracts, size a backfill against the AppsFlyer quota, and
+debug a failing run with the real reference instead of guessing. Idempotent —
+safe to re-run after upgrading afly. See [Claude Code
+guide](docs/guides/claude-code.md).
+
+## Documentation
+
+- [Installation](docs/getting-started/installation.md)
+- [Quickstart](docs/getting-started/quickstart.md)
+- [Configuration](docs/guides/configuration.md)
+- [Extracts](docs/guides/extracts.md)
+- [Formats & the Facebook split](docs/guides/formats-and-facebook-split.md)
+- [Idempotency (the write path)](docs/guides/idempotency.md)
+- [Quotas & scheduling](docs/guides/quotas.md)
+- [Running on a schedule](docs/guides/scheduling.md)
+- [Alerting](docs/guides/alerting.md)
+- [Claude Code](docs/guides/claude-code.md)
+- [CLI reference](docs/reference/cli.md)
+- [Config reference](docs/reference/config.md)
+- [Tables reference](docs/reference/tables.md)
+- [Changelog](CHANGELOG.md)
+
+## Requirements
+
+- Python 3.10+
+- ClickHouse (tested against 22.11)
+- An AppsFlyer account with Pull API (API V2) access
+
+## License
+
+MIT License — see [LICENSE](LICENSE) for details.

afly-0.1.0/README.md

@@ -0,0 +1,134 @@
+# afly
+
+[![CI](https://github.com/selishchev/afly/actions/workflows/ci.yml/badge.svg)](https://github.com/selishchev/afly/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/afly.svg)](https://pypi.org/project/afly/)
+[![Python](https://img.shields.io/pypi/pyversions/afly.svg)](https://pypi.org/project/afly/)
+
+**Idempotent AppsFlyer aggregate reports → ClickHouse, with a dbt-style
+CLI.**
+
+`afly` pulls AppsFlyer's [aggregate Pull
+API](https://support.appsflyer.com/hc/en-us/articles/207034346-Pull-API-aggregate-data) reports and
+writes them into ClickHouse. It's a dbt/detectkit-style project: an
+`afly_project.yml` describes the project, `profiles.yml` holds credentials,
+and `extracts/*.yml` declare which reports to pull and where they land.
+
+
+## Why
+
+- **Idempotent by construction, not by convention.** Every run rebuilds
+  whole ClickHouse partitions atomically (`REPLACE PARTITION`) instead of
+  appending — re-running the same window, or recovering from a crash mid-run,
+  never duplicates a row and never needs `FINAL` on read.
+- **Quota-aware.** AppsFlyer enforces a per-minute limit and a small daily
+  budget for wider date ranges. afly's scheduler interleaves apps/report
+  types to stay under both automatically, and a backfill can be capped with
+  `--max-calls`/`--max-minutes` and safely resumed later.
+- **The Facebook split is a config concern, not a bug.** AppsFlyer only
+  returns campaign/adset/adgroup breakdown columns for a Facebook-scoped
+  pull — afly's scaffold ships a dedicated `facebook` extract alongside an
+  unfiltered one that explicitly excludes it, so the ownership is explicit
+  instead of accidentally double-counted.
+- **A `--dry-run` you can actually trust.** Before pulling anything, `afly
+  run --dry-run` prints the exact window/chunk plan and the AppsFlyer quota
+  totals it would spend, flagging anything that would exceed budget.
+
+## Install
+
+```bash
+pip install afly
+```
+
+Requires Python 3.10+.
+
+## 60-second quickstart
+
+```bash
+afly init my_project && cd my_project
+cp .env.example .env        # fill in AppsFlyer token + ClickHouse credentials
+set -a; source .env; set +a
+
+afly validate                       # config sanity check, no network calls
+afly debug                          # probe AppsFlyer/ClickHouse connectivity
+afly run --select "*" --dry-run     # print the plan, pull nothing
+afly run --select "*"               # pull for real
+```
+
+```sql
+SELECT date, media_source, sum(total_cost)
+FROM appsflyer.appsflyer_geo_by_date
+GROUP BY 1, 2
+ORDER BY 1, 2;
+```
+
+See [Quickstart](docs/getting-started/quickstart.md) for the full walkthrough.
+
+## The three-extract picture
+
+`afly init` scaffolds a realistic split that demonstrates the ownership
+convention every extra media-source extract should follow:
+
+```yaml
+# extracts/standard.yml — everything else
+name: standard
+report_type: geo_by_date_report
+exclude_media_sources: [Facebook Ads, yandexdirect_int]   # owned below
+table: appsflyer_geo_by_date
+
+# extracts/facebook.yml — needs its own scoped pull for adset/adgroup columns
+name: facebook
+media_source: facebook
+table: appsflyer_geo_by_date
+
+# extracts/yandex.yml — reports arrive late, so re-pull a wider window
+name: yandex
+media_source: yandexdirect_int
+lookback_days: 7
+table: appsflyer_geo_by_date
+```
+
+All three write the same table safely, because `standard` explicitly excludes
+what the other two own. See [Extracts
+guide](docs/guides/extracts.md) and [Facebook
+split](docs/guides/formats-and-facebook-split.md).
+
+## AI-native onboarding
+
+```bash
+afly init-claude
+```
+
+Scaffolds `CLAUDE.md` + `.claude/rules/afly/` + four skills
+(`afly-setup-project`, `afly-new-extract`, `afly-backfill`,
+`afly-debug-run`) into the project, so Claude Code (or another AI assistant)
+can configure extracts, size a backfill against the AppsFlyer quota, and
+debug a failing run with the real reference instead of guessing. Idempotent —
+safe to re-run after upgrading afly. See [Claude Code
+guide](docs/guides/claude-code.md).
+
+## Documentation
+
+- [Installation](docs/getting-started/installation.md)
+- [Quickstart](docs/getting-started/quickstart.md)
+- [Configuration](docs/guides/configuration.md)
+- [Extracts](docs/guides/extracts.md)
+- [Formats & the Facebook split](docs/guides/formats-and-facebook-split.md)
+- [Idempotency (the write path)](docs/guides/idempotency.md)
+- [Quotas & scheduling](docs/guides/quotas.md)
+- [Running on a schedule](docs/guides/scheduling.md)
+- [Alerting](docs/guides/alerting.md)
+- [Claude Code](docs/guides/claude-code.md)
+- [CLI reference](docs/reference/cli.md)
+- [Config reference](docs/reference/config.md)
+- [Tables reference](docs/reference/tables.md)
+- [Changelog](CHANGELOG.md)
+
+## Requirements
+
+- Python 3.10+
+- ClickHouse (tested against 22.11)
+- An AppsFlyer account with Pull API (API V2) access
+
+## License
+
+MIT License — see [LICENSE](LICENSE) for details.

afly-0.1.0/afly/__init__.py

@@ -0,0 +1,5 @@
+"""afly — idempotent AppsFlyer → ClickHouse extractor."""
+
+__version__ = "0.1.0"
+
+__all__ = ["__version__"]

afly-0.1.0/afly/alerting/__init__.py

@@ -0,0 +1,3 @@
+"""afly.alerting — run-failure notifications (project-level, not data-quality)."""
+
+from __future__ import annotations

afly-0.1.0/afly/alerting/webhook.py

@@ -0,0 +1,92 @@
+"""Failure-alert delivery over a webhook (Mattermost/Slack "attachments", or plain JSON).
+
+This only ever fires for a *run-level* failure (the pipeline itself broke —
+auth, ClickHouse, an aborted run) — see
+``afly.config.project_config.ErrorAlertingConfig`` and
+``afly.run.runner._send_alerts``. It is not a data-quality alert.
+"""
+
+from __future__ import annotations
+
+import logging
+from collections.abc import Callable, Sequence
+from typing import Any
+
+import requests
+
+from afly import __version__
+from afly.config.profile import AlertChannelConfig
+
+logger = logging.getLogger(__name__)
+
+_ATTACHMENT_COLOR = "#D63232"
+
+
+def send_failure_alert(
+    channel: AlertChannelConfig,
+    *,
+    project: str,
+    profile: str,
+    run_id: str,
+    title: str,
+    lines: Sequence[str],
+    mentions: Sequence[str] = (),
+    post: Callable[..., Any] = requests.post,
+) -> bool:
+    """POST a failure alert to *channel*. Never raises — returns ``False`` and
+    logs a warning on any failure (network error, non-2xx, bad channel config).
+
+    The webhook URL (which may itself embed a token, per Mattermost/Slack
+    convention) is only ever used as the POST target — nothing here echoes
+    any credential back into the message body.
+    """
+    try:
+        payload = _build_payload(
+            channel, project=project, run_id=run_id, title=title, lines=lines, mentions=mentions
+        )
+        response = post(channel.webhook_url, json=payload, timeout=channel.timeout)
+        response.raise_for_status()
+        return True
+    except Exception:
+        logger.warning(
+            "failed to send failure alert via %s channel for run %s",
+            channel.type,
+            run_id,
+            exc_info=True,
+        )
+        return False
+
+
+def _build_payload(
+    channel: AlertChannelConfig,
+    *,
+    project: str,
+    run_id: str,
+    title: str,
+    lines: Sequence[str],
+    mentions: Sequence[str],
+) -> dict[str, Any]:
+    if channel.type == "webhook":
+        return {"title": title, "text": "\n".join(lines), "project": project, "run_id": run_id}
+
+    # mattermost / slack: both speak the same incoming-webhook "attachments" shape.
+    payload: dict[str, Any] = {
+        "username": channel.username,
+        "text": " ".join(mentions),
+        "attachments": [
+            {
+                "color": _ATTACHMENT_COLOR,
+                "title": title,
+                "text": "\n".join(lines),
+                "footer": f"afly {__version__}",
+            }
+        ],
+    }
+    if channel.icon_emoji:
+        payload["icon_emoji"] = channel.icon_emoji
+    if channel.channel:
+        payload["channel"] = channel.channel
+    return payload
+
+
+__all__ = ["send_failure_alert"]

afly-0.1.0/afly/appsflyer/__init__.py

@@ -0,0 +1 @@
+"""AppsFlyer HTTP client: errors, retry policy, and the Pull/management APIs."""

afly-0.1.0/afly/appsflyer/client.py

@@ -0,0 +1,83 @@
+"""Thin HTTP wrapper around AppsFlyer's REST APIs.
+
+Deliberately does *not* interpret response status codes — that's
+``afly.appsflyer.errors.classify_response``'s job, applied by the caller
+(``pull_api``/``mng_api``) once it knows which endpoint's error semantics
+apply. This class only owns: auth headers, timeouts, redirect-following, and
+turning a network-level failure (not an HTTP error response) into a
+:class:`TransientError` the retry policy already knows how to handle.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import Any
+
+import requests
+
+from afly import __version__
+from afly.appsflyer.errors import TransientError
+
+_DEFAULT_BASE_URL = "https://hq1.appsflyer.com"
+
+
+class AppsFlyerClient:
+    """Authenticated HTTP client for one AppsFlyer account (one token)."""
+
+    def __init__(
+        self,
+        token: str,
+        base_url: str = _DEFAULT_BASE_URL,
+        timeout_seconds: int = 120,
+        user_agent: str | None = None,
+        session: requests.Session | None = None,
+    ) -> None:
+        self._token = token
+        self.base_url = base_url.rstrip("/")
+        self.timeout_seconds = timeout_seconds
+        self.user_agent = user_agent or f"afly/{__version__}"
+        self._session = session or requests.Session()
+        self.api_calls = 0
+
+    def reset_counter(self) -> None:
+        """Zero the ``api_calls`` counter (e.g. between extracts in a run)."""
+        self.api_calls = 0
+
+    def get(
+        self,
+        path: str,
+        params: Mapping[str, Any] | None = None,
+        accept: str = "application/json",
+    ) -> requests.Response:
+        """GET ``base_url + path``, following redirects, with afly's standard headers.
+
+        Increments ``api_calls`` exactly once per call — including a call
+        that raises, since the network round-trip still happened and counts
+        against AppsFlyer's rate limit either way.
+        """
+        url = f"{self.base_url}{path}"
+        headers = {
+            "authorization": f"Bearer {self._token}",
+            "accept": accept,
+            "user-agent": self.user_agent,
+        }
+        try:
+            response = self._session.get(
+                url,
+                params=params,
+                headers=headers,
+                timeout=self.timeout_seconds,
+                allow_redirects=True,
+            )
+        except requests.RequestException as exc:
+            self.api_calls += 1
+            raise TransientError(
+                f"network error calling AppsFlyer: {exc}", status=None, body=str(exc), url=url
+            ) from exc
+
+        self.api_calls += 1
+        return response
+
+    def __repr__(self) -> str:
+        masked = f"{self._token[:4]}…" if self._token else "(empty)"
+        return f"AppsFlyerClient(base_url={self.base_url!r}, token={masked!r})"