repster 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- repster/.agents/skills/repster/SKILL.md +65 -0
- repster/.agents/skills/repster/references/cli.md +59 -0
- repster/.agents/skills/repster/references/cloud-api.md +83 -0
- repster/.agents/skills/repster/references/configuration.md +55 -0
- repster/.agents/skills/repster/references/deploy-to-cloud.md +74 -0
- repster/.agents/skills/repster/references/how-repster-works.md +74 -0
- repster/.agents/skills/repster/references/index.md +104 -0
- repster/.agents/skills/repster/references/manage-api-keys.md +58 -0
- repster/.agents/skills/repster/references/manage-secrets.md +74 -0
- repster/.agents/skills/repster/references/manage-team.md +58 -0
- repster/.agents/skills/repster/references/open-core-model.md +52 -0
- repster/.agents/skills/repster/references/pricing.md +36 -0
- repster/.agents/skills/repster/references/quickstart.md +101 -0
- repster/.agents/skills/repster/references/schedule-a-flow.md +73 -0
- repster/.agents/skills/repster/references/use-with-coding-agents.md +47 -0
- repster/.agents/skills/repster/references/view-logs.md +45 -0
- repster/.agents/skills/repster/references/why-repster.md +58 -0
- repster/__init__.py +1 -0
- repster/cli.py +400 -0
- repster/cloud/__init__.py +0 -0
- repster/cloud/cli/__init__.py +4 -0
- repster/cloud/cli/_helpers.py +58 -0
- repster/cloud/cli/auth.py +44 -0
- repster/cloud/cli/deployments.py +49 -0
- repster/cloud/cli/interactive.py +144 -0
- repster/cloud/cli/keys.py +93 -0
- repster/cloud/cli/orgs.py +212 -0
- repster/cloud/cli/registration.py +18 -0
- repster/cloud/cli/secrets.py +77 -0
- repster/cloud/client.py +187 -0
- repster/cloud/config.py +105 -0
- repster/cloud/integration.py +112 -0
- repster/cloud/packaging.py +79 -0
- repster/cloud/service_runner.py +75 -0
- repster/config.py +87 -0
- repster/db.py +28 -0
- repster/decorator.py +33 -0
- repster/executor.py +153 -0
- repster/formatting.py +16 -0
- repster/logging.py +12 -0
- repster/runner.py +76 -0
- repster/scan.py +164 -0
- repster/scheduler.py +42 -0
- repster/store.py +101 -0
- repster/ui/__init__.py +0 -0
- repster/ui/app.py +29 -0
- repster/ui/data_source.py +170 -0
- repster/ui/routes.py +160 -0
- repster/ui/static/app.css +317 -0
- repster/ui/templates/404.html +9 -0
- repster/ui/templates/base.html +28 -0
- repster/ui/templates/flow_detail.html +67 -0
- repster/ui/templates/flows.html +43 -0
- repster/ui/templates/run_detail.html +49 -0
- repster/ui/templates/runs.html +43 -0
- repster-0.1.0.dist-info/METADATA +71 -0
- repster-0.1.0.dist-info/RECORD +60 -0
- repster-0.1.0.dist-info/WHEEL +4 -0
- repster-0.1.0.dist-info/entry_points.txt +3 -0
- repster-0.1.0.dist-info/licenses/LICENSE +202 -0
|
@@ -0,0 +1,65 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: repster
|
|
3
|
+
description: >
|
|
4
|
+
Reference for using Repster, a lightweight scheduling and observability layer for dlt pipelines, with an AI-ready CLI.
|
|
5
|
+
Use when writing, deploying, or debugging Repster flows, or when asked about rp commands, secrets, scheduling, or cloud execution.
|
|
6
|
+
Triggers: "repster", "rp", "rp deploy", "rp run", "rp flow", "dlt pipeline", "repster cloud", "schedule dlt", "AI agent dlt"
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Repster
|
|
10
|
+
|
|
11
|
+
Repster adds lightweight scheduling, run history, and a web UI to dlt pipelines. Define flows as code, run them locally, and deploy to managed serverless workers (one isolated worker per run). Every operation is an `rp` CLI command, so coding agents can operate all of it.
|
|
12
|
+
|
|
13
|
+
```bash
|
|
14
|
+
uv add repster
|
|
15
|
+
```
|
|
16
|
+
|
|
17
|
+
Quick pattern:
|
|
18
|
+
|
|
19
|
+
```python
|
|
20
|
+
import dlt
|
|
21
|
+
import repster as rp
|
|
22
|
+
|
|
23
|
+
@rp.flow(schedule="0 * * * *")
|
|
24
|
+
def my_pipeline():
|
|
25
|
+
pipeline = dlt.pipeline(destination="duckdb")
|
|
26
|
+
pipeline.run(my_source())
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
rp run my_pipeline # local
|
|
31
|
+
rp deploy && rp run --cloud my_pipeline # cloud
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Reference Index
|
|
35
|
+
|
|
36
|
+
### Tutorials
|
|
37
|
+
|
|
38
|
+
- `references/deploy-to-cloud.md`
|
|
39
|
+
- `references/quickstart.md`
|
|
40
|
+
|
|
41
|
+
### How-to guides
|
|
42
|
+
|
|
43
|
+
- `references/manage-api-keys.md`
|
|
44
|
+
- `references/manage-secrets.md`
|
|
45
|
+
- `references/manage-team.md`
|
|
46
|
+
- `references/schedule-a-flow.md`
|
|
47
|
+
- `references/use-with-coding-agents.md`
|
|
48
|
+
- `references/view-logs.md`
|
|
49
|
+
|
|
50
|
+
### Reference
|
|
51
|
+
|
|
52
|
+
- `references/cli.md`
|
|
53
|
+
- `references/cloud-api.md`
|
|
54
|
+
- `references/configuration.md`
|
|
55
|
+
|
|
56
|
+
### Explanation
|
|
57
|
+
|
|
58
|
+
- `references/how-repster-works.md`
|
|
59
|
+
- `references/open-core-model.md`
|
|
60
|
+
- `references/pricing.md`
|
|
61
|
+
- `references/why-repster.md`
|
|
62
|
+
|
|
63
|
+
### Other
|
|
64
|
+
|
|
65
|
+
- `references/index.md`
|
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
# CLI reference
|
|
2
|
+
|
|
3
|
+
All commands are available via `rp` (or `repster`). Cloud commands walk you through login/registration inline if you haven't authenticated yet; you never need to run `rp login` before your first `rp deploy`.
|
|
4
|
+
|
|
5
|
+
## Local commands
|
|
6
|
+
|
|
7
|
+
| Command | Description |
|
|
8
|
+
|---------|-------------|
|
|
9
|
+
| `rp dev` | Start the local scheduler; fires cron-scheduled flows according to their `@rp.flow(schedule=...)` expressions |
|
|
10
|
+
| `rp run <flow>` | Run a flow now (local execution) |
|
|
11
|
+
| `rp list` | List all discovered flows with their schedule and next run time |
|
|
12
|
+
| `rp status [<flow>] [--profile]` | Show last N runs and next scheduled run; merges local and cloud runs when a profile is configured |
|
|
13
|
+
| `rp logs [--flow <name>] [--run <id>] [--profile]` | View captured log output for a run |
|
|
14
|
+
| `rp retry [--flow <name>] [--run <id>] [--profile]` | Re-run a flow, linked to the run it retries |
|
|
15
|
+
|
|
16
|
+
Schedules are declared in `@rp.flow(schedule=...)` in code, not set via the CLI.
|
|
17
|
+
|
|
18
|
+
## Cloud commands
|
|
19
|
+
|
|
20
|
+
| Command | Description |
|
|
21
|
+
|---------|-------------|
|
|
22
|
+
| `rp login [--token rpk_...] [--profile NAME]` | Authenticate with Repster Cloud; interactive email-first flow: logs in if you have an account, registers if you don't. Writes a profile to `~/.repster/config.toml`. Pass `--token rpk_...` to skip interactive flow (for CI). |
|
|
23
|
+
| `rp deploy [--profile NAME]` | Package the project and push to the control plane; immutable and content-addressed; deploying the same code twice is a no-op |
|
|
24
|
+
| `rp run --cloud <flow> [--profile NAME]` | Trigger a cloud run of the latest deployment; fails with a helpful message if no deployment exists |
|
|
25
|
+
| `rp secrets push [--profile NAME]` | Read `.env` and store its values in the cloud secrets vault |
|
|
26
|
+
| `rp secrets list [--profile NAME]` | List secret key names stored in the vault for this project (values are never returned) |
|
|
27
|
+
| `rp secrets delete KEY [--yes] [--profile NAME]` | Delete a secret from the vault for this project; prompts for confirmation unless `--yes` |
|
|
28
|
+
|
|
29
|
+
## Org management
|
|
30
|
+
|
|
31
|
+
| Command | Description |
|
|
32
|
+
|---------|-------------|
|
|
33
|
+
| `rp org list [--profile]` | List all orgs you belong to (slug, name, role, joined date) |
|
|
34
|
+
| `rp org switch <slug>` | Set the default profile to a different org |
|
|
35
|
+
| `rp org invite <email> [--role member\|admin] [--profile]` | Invite a user; creates an account with a temporary password if needed |
|
|
36
|
+
| `rp org members [--profile]` | List members (email, role, joined date) |
|
|
37
|
+
| `rp org members promote <email> [--profile]` | Promote a member to admin |
|
|
38
|
+
| `rp org members remove <email> [--yes] [--profile]` | Remove a member; `--yes` skips confirmation |
|
|
39
|
+
|
|
40
|
+
## Key management
|
|
41
|
+
|
|
42
|
+
| Command | Description |
|
|
43
|
+
|---------|-------------|
|
|
44
|
+
| `rp keys list [--profile]` | List API keys for this org (prefix, name, status, last-used, created) |
|
|
45
|
+
| `rp keys create [--name NAME] [--profile]` | Create a new API key; raw key is printed once |
|
|
46
|
+
| `rp keys revoke <key-id-or-prefix> [--yes] [--profile]` | Revoke a key; `--yes` skips confirmation |
|
|
47
|
+
|
|
48
|
+
## Global flags
|
|
49
|
+
|
|
50
|
+
All commands that talk to the cloud accept:
|
|
51
|
+
|
|
52
|
+
| Flag | Description |
|
|
53
|
+
|------|-------------|
|
|
54
|
+
| `--profile NAME` | Use a named profile from `~/.repster/config.toml` |
|
|
55
|
+
| `--service-url URL` | Override the Repster Cloud URL for this invocation |
|
|
56
|
+
|
|
57
|
+
Environment variables `REPSTER_SERVICE_URL` and `REPSTER_API_TOKEN` are also accepted and override the profile.
|
|
58
|
+
|
|
59
|
+
See [Configuration](configuration.md) for the full resolution chain.
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
# Cloud API
|
|
2
|
+
|
|
3
|
+
The REST API exposed by Repster Cloud. Used by the Repster CLI and the Repster Cloud UI. You can call it directly for automation or integration.
|
|
4
|
+
|
|
5
|
+
**Base URL**: `https://repster.online/api` — the paths below are relative to it. (The browser UI is served at `https://repster.online/`; `GET /health` lives at the host root, outside `/api`.)
|
|
6
|
+
|
|
7
|
+
**Auth**: API key in the `Authorization: Bearer rpk_...` header. Keys are org-scoped, so all endpoints automatically scope to the calling key's org.
|
|
8
|
+
|
|
9
|
+
**Client header**: `X-Repster-Client: <version>`, sent automatically by the CLI.
|
|
10
|
+
|
|
11
|
+
**Async runs**: `POST /runs` returns a `run_id` immediately. Poll `GET /runs/{id}` or stream logs via `GET /runs/{id}/logs?follow=true`.
|
|
12
|
+
|
|
13
|
+
## Auth endpoints (no token required)
|
|
14
|
+
|
|
15
|
+
| Method | Path | Body | Returns |
|
|
16
|
+
|--------|------|------|---------|
|
|
17
|
+
| `POST` | `/auth/register` | `{email, password, org_slug, org_name}` | `{org, api_key}` (raw key shown once) |
|
|
18
|
+
| `POST` | `/auth/login` | `{email, password, org_slug?}` | `{org, api_key}` or `{needs_org: true, orgs: [...]}` |
|
|
19
|
+
| `POST` | `/auth/create-org` | `{email, password, org_slug, org_name}` | `{org, api_key}` |
|
|
20
|
+
|
|
21
|
+
When a user belongs to multiple orgs and `org_slug` is omitted from `POST /auth/login`, the response includes `orgs` listing all their orgs. Re-POST with `org_slug` to select one.
|
|
22
|
+
|
|
23
|
+
## User identity
|
|
24
|
+
|
|
25
|
+
| Method | Path | Returns |
|
|
26
|
+
|--------|------|---------|
|
|
27
|
+
| `GET` | `/users/me` | `{user_id, email, created_at}` |
|
|
28
|
+
| `GET` | `/users/me/orgs` | `[{org_id, slug, name, role, joined_at}]` |
|
|
29
|
+
|
|
30
|
+
## Org & key management
|
|
31
|
+
|
|
32
|
+
| Method | Path | Body | Returns |
|
|
33
|
+
|--------|------|------|---------|
|
|
34
|
+
| `GET` | `/orgs/me` | — | `{org_id, slug, name, created_at}` |
|
|
35
|
+
| `POST` | `/orgs` | `{slug, name}` | `{org, api_key}` |
|
|
36
|
+
| `POST` | `/orgs/{org_id}/keys` | `{name}` | `{key_id, raw_key, ...}` (raw key shown once) |
|
|
37
|
+
| `GET` | `/orgs/{org_id}/keys` | — | `[{key_id, name, key_prefix, created_at, last_used_at, revoked_at}]` |
|
|
38
|
+
| `DELETE` | `/orgs/{org_id}/keys/{key_id}` | — | `{ok: true}` |
|
|
39
|
+
|
|
40
|
+
## Member management (admin only)
|
|
41
|
+
|
|
42
|
+
| Method | Path | Body | Returns |
|
|
43
|
+
|--------|------|------|---------|
|
|
44
|
+
| `GET` | `/orgs/{org_id}/members` | — | `[{user_id, email, role, created_at}]` |
|
|
45
|
+
| `POST` | `/orgs/{org_id}/members` | `{email, role?}` | `{user_id, email, role, temp_password?}` (201) |
|
|
46
|
+
| `PATCH` | `/orgs/{org_id}/members/{user_id}` | `{role}` | updated member |
|
|
47
|
+
| `DELETE` | `/orgs/{org_id}/members/{user_id}` | — | `{ok: true}` |
|
|
48
|
+
|
|
49
|
+
`role` defaults to `"member"`. Valid values: `"admin"`, `"member"`. Cannot remove or demote the last admin (409). Cannot remove yourself (409).
|
|
50
|
+
|
|
51
|
+
## Deployments (admin only)
|
|
52
|
+
|
|
53
|
+
| Method | Path | Body | Returns |
|
|
54
|
+
|--------|------|------|---------|
|
|
55
|
+
| `POST` | `/deployments` | multipart: `manifest` + `archive` | `{deployment_id, code_digest}` |
|
|
56
|
+
| `GET` | `/deployments?project=` | — | deployments for the caller's org, latest first |
|
|
57
|
+
|
|
58
|
+
Deployments are immutable and content-addressed, so uploading the same archive twice returns the existing deployment.
|
|
59
|
+
|
|
60
|
+
## Runs
|
|
61
|
+
|
|
62
|
+
| Method | Path | Body / Params | Returns |
|
|
63
|
+
|--------|------|---------------|---------|
|
|
64
|
+
| `POST` | `/runs` | `{project, flow, trigger, retry_of?}` | `{run_id, status: "running"}` |
|
|
65
|
+
| `GET` | `/runs?project=&flow=&limit=` | — | run records for the caller's org |
|
|
66
|
+
| `GET` | `/runs/{id}` | — | `{run_id, status, trigger, deployment_id, started_at, finished_at, rows_loaded, summary, retry_of}` |
|
|
67
|
+
| `GET` | `/runs/{id}/logs` | `?follow=bool` | full log text; SSE stream when `follow=true` |
|
|
68
|
+
|
|
69
|
+
`POST /runs` returns 404 with a helpful message if the flow has never been deployed.
|
|
70
|
+
|
|
71
|
+
## Secrets (admin only)
|
|
72
|
+
|
|
73
|
+
| Method | Path | Body | Returns |
|
|
74
|
+
|--------|------|------|---------|
|
|
75
|
+
| `PUT` | `/projects/{project}/secrets` | `{KEY: VALUE, ...}` | `{ok: true}` |
|
|
76
|
+
| `GET` | `/projects/{project}/secrets` | — | key names only (values are never returned) |
|
|
77
|
+
| `DELETE` | `/projects/{project}/secrets/{key}` | — | `{ok: true}` |
|
|
78
|
+
|
|
79
|
+
## Health
|
|
80
|
+
|
|
81
|
+
| Method | Path | Returns |
|
|
82
|
+
|--------|------|---------|
|
|
83
|
+
| `GET` | `/health` | `{ok: true}` (no auth required) |
|
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Configuration
|
|
2
|
+
|
|
3
|
+
How repster knows which Repster Cloud endpoint to connect to, as whom, and about which project. Standard CLI pattern: global per-user credentials, per-repo non-secret pointers, env-var overrides for CI.
|
|
4
|
+
|
|
5
|
+
## Resolution chain
|
|
6
|
+
|
|
7
|
+
Settings are resolved in priority order (highest first):
|
|
8
|
+
|
|
9
|
+
1. **CLI flags**: `--profile`, `--service-url`
|
|
10
|
+
2. **Environment variables**: `REPSTER_SERVICE_URL`, `REPSTER_API_TOKEN`
|
|
11
|
+
3. **Project config**: `[tool.repster]` in `pyproject.toml` (non-secret; safe to commit)
|
|
12
|
+
4. **Global profiles**: `~/.repster/config.toml` (credentials; never commit this)
|
|
13
|
+
|
|
14
|
+
## Global credentials: `~/.repster/config.toml`
|
|
15
|
+
|
|
16
|
+
Written by `rp login`. Edit it directly if needed.
|
|
17
|
+
|
|
18
|
+
```toml
|
|
19
|
+
[profiles.default]
|
|
20
|
+
service_url = "https://repster.online"
|
|
21
|
+
token = "rpk_K7mXwQ3..."
|
|
22
|
+
|
|
23
|
+
[profiles.prod]
|
|
24
|
+
service_url = "https://repster.online"
|
|
25
|
+
token = "rpk_..."
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
Use `rp login --profile <name>` to add a named profile. Use `--profile <name>` on any cloud command to select it.
|
|
29
|
+
|
|
30
|
+
## Project config: `pyproject.toml`
|
|
31
|
+
|
|
32
|
+
Your project is already a uv project, so `pyproject.toml` exists. By default nothing needs configuring: the project name defaults to `[project].name` and the profile defaults to `default`.
|
|
33
|
+
|
|
34
|
+
Override when needed:
|
|
35
|
+
|
|
36
|
+
```toml
|
|
37
|
+
[tool.repster]
|
|
38
|
+
project = "acme-pipelines" # defaults to [project].name
|
|
39
|
+
profile = "prod" # defaults to "default"
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
Never put tokens in `pyproject.toml`; use `~/.repster/config.toml` or environment variables.
|
|
43
|
+
|
|
44
|
+
## Environment variables
|
|
45
|
+
|
|
46
|
+
| Variable | Description |
|
|
47
|
+
|----------|-------------|
|
|
48
|
+
| `REPSTER_SERVICE_URL` | Override the Repster Cloud URL |
|
|
49
|
+
| `REPSTER_API_TOKEN` | Override the API token (useful for CI) |
|
|
50
|
+
|
|
51
|
+
Example CI setup:
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
REPSTER_API_TOKEN=rpk_... rp deploy
|
|
55
|
+
```
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Deploy dlt pipelines to Repster Cloud with managed scheduling, secrets, and serverless workers, using the same code you ran locally.
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# Deploy to cloud
|
|
6
|
+
|
|
7
|
+
Run your flows on Repster Cloud: your schedules fire without your machine on, and each run gets its own isolated worker, created on demand. The pipeline code is exactly what you ran locally.
|
|
8
|
+
|
|
9
|
+
## Prerequisites
|
|
10
|
+
|
|
11
|
+
- A working local flow (see [Quickstart](quickstart.md))
|
|
12
|
+
- A Repster account (you'll create one during `rp login` if you don't have one yet)
|
|
13
|
+
|
|
14
|
+
## 1. Authenticate
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
rp login
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Repster walks you through account creation or login interactively. Your credentials are saved to `~/.repster/config.toml`. You never need to run `rp login` before deploying; any cloud command that needs auth will walk you through it inline.
|
|
21
|
+
|
|
22
|
+
## 2. Push secrets
|
|
23
|
+
|
|
24
|
+
If your pipeline uses environment variables (API keys, database credentials), push them before deploying:
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
rp secrets push
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
This reads your `.env` file and stores the values, encrypted, in the Repster Cloud vault. See [Manage secrets](../how-tos/manage-secrets.md) for details.
|
|
31
|
+
|
|
32
|
+
## 3. Deploy
|
|
33
|
+
|
|
34
|
+
```bash
|
|
35
|
+
rp deploy
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Repster packages your project (source files + `uv.lock`), uploads it, and registers your flows. Deployments are immutable; see [How Repster works](../explanation/how-repster-works.md#deployments).
|
|
39
|
+
|
|
40
|
+
## 4. Trigger a cloud run
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
rp run --cloud hackernews_top
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
The run is dispatched to an isolated cloud worker. You get the run ID immediately; the flow executes asynchronously.
|
|
47
|
+
|
|
48
|
+
## 5. Monitor the run
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
rp status # last N runs (local + cloud, combined)
|
|
52
|
+
rp logs --run <run-id> # full log output
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## 6. View in the UI
|
|
56
|
+
|
|
57
|
+
Log in to the Repster Cloud UI ([repster.online](https://repster.online)) to see runs, logs, deployments, and secrets in one place.
|
|
58
|
+
|
|
59
|
+
## Re-deploying
|
|
60
|
+
|
|
61
|
+
After changing your pipeline code:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
rp deploy # upload new version
|
|
65
|
+
rp run --cloud hackernews_top # runs the latest deployment
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
Old deployments are preserved; runs always reference the deployment they were dispatched against.
|
|
69
|
+
|
|
70
|
+
## Next steps
|
|
71
|
+
|
|
72
|
+
- **[Manage secrets →](../how-tos/manage-secrets.md)**
|
|
73
|
+
- **[Schedule a flow →](../how-tos/schedule-a-flow.md)**
|
|
74
|
+
- **[Manage API keys →](../how-tos/manage-api-keys.md)**
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Repster's execution model: flow discovery, local vs. cloud runs, immutable deployments, secrets injection, and observability.
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# How Repster works
|
|
6
|
+
|
|
7
|
+
Repster adds scheduling, secrets management, cloud execution, and observability on top of [dlt](https://dlthub.com), without changing how you write pipelines.
|
|
8
|
+
|
|
9
|
+
## The core idea
|
|
10
|
+
|
|
11
|
+
A Repster project is a Python package with flows:
|
|
12
|
+
|
|
13
|
+
```python
|
|
14
|
+
@rp.flow(schedule="0 * * * *")
|
|
15
|
+
def my_pipeline():
|
|
16
|
+
pipeline = dlt.pipeline(destination="duckdb")
|
|
17
|
+
pipeline.run(my_source())
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Repster discovers flows by scanning your project files for `@rp.flow()` decorators. Each flow is a callable that runs a dlt pipeline. The decorator adds a name, an optional schedule, and hooks into the Repster execution model.
|
|
21
|
+
|
|
22
|
+
## Local vs. cloud execution
|
|
23
|
+
|
|
24
|
+
Repster uses the same `@rp.flow()` code for both local and cloud execution; the runner is swapped, not the flow definition.
|
|
25
|
+
|
|
26
|
+
**Local**: `rp run <flow>` or `rp dev` (scheduler). Runs in your local Python environment, writes state to `.repster/` (SQLite).
|
|
27
|
+
|
|
28
|
+
**Cloud**: `rp run --cloud <flow>`. Dispatches to an isolated worker managed by Repster. The worker runs the same `@rp.flow()` code in an environment built from your `uv.lock`.
|
|
29
|
+
|
|
30
|
+
## Deployments
|
|
31
|
+
|
|
32
|
+
`rp deploy` packages your project into a source archive with your `uv.lock` and uploads it to the Repster control plane. Each deployment is:
|
|
33
|
+
|
|
34
|
+
- **Immutable**: the archive is content-addressed by `sha256`. Deploying the same code twice is a no-op.
|
|
35
|
+
- **Complete**: the lockfile is included so workers build a reproducible environment.
|
|
36
|
+
|
|
37
|
+
Cloud runs always execute against a specific deployment. `rp run --cloud` uses the latest.
|
|
38
|
+
|
|
39
|
+
## Secrets
|
|
40
|
+
|
|
41
|
+
Secrets (API keys, database credentials) live in `.env` locally and in the Repster secrets vault in the cloud.
|
|
42
|
+
|
|
43
|
+
`rp secrets push` reads `.env` and stores values encrypted in the vault. At run time, the worker receives the secrets as environment variables, never in logs or run records.
|
|
44
|
+
|
|
45
|
+
Repster uses dlt's env-var naming convention, so the same secrets work locally and in the cloud without any code changes.
|
|
46
|
+
|
|
47
|
+
## Observability
|
|
48
|
+
|
|
49
|
+
Every run produces a log stream captured from dlt's output. Logs include:
|
|
50
|
+
|
|
51
|
+
- dlt pipeline progress (rows loaded, tables updated)
|
|
52
|
+
- Any `print()` / `logging` output from your code
|
|
53
|
+
- Full stack traces on failure
|
|
54
|
+
|
|
55
|
+
Runs are stored in the control plane with status, timing, and row counts. Local runs are stored in `.repster/` and merged with cloud runs in `rp status`.
|
|
56
|
+
|
|
57
|
+
## What belongs in a flow
|
|
58
|
+
|
|
59
|
+
Keep the flow body a thin wrapper: construct a `dlt.pipeline(...)` and call `.run(...)`. Extraction logic (HTTP calls, pagination, parsing) belongs inside your `@dlt.resource`/`@dlt.source` functions, where dlt already expects arbitrary Python. One flow run should map to one observable dlt pipeline run.
|
|
60
|
+
|
|
61
|
+
Repster is not a general-purpose script or task runner. If you need to run arbitrary non-dlt workloads on a schedule, use a dedicated orchestrator (Dagster, Airflow, plain cron) instead of stretching a flow to cover it. See [Why Repster?](why-repster.md) for how the two fit together.
|
|
62
|
+
|
|
63
|
+
## What Repster owns and what you own
|
|
64
|
+
|
|
65
|
+
| Repster owns | You own |
|
|
66
|
+
|--------------|---------|
|
|
67
|
+
| Execution (cloud workers) | Pipeline code |
|
|
68
|
+
| Scheduling (cron dispatch) | Connectors and sources |
|
|
69
|
+
| Secrets injection | Destination schema |
|
|
70
|
+
| Log capture and retention | `.env` secrets |
|
|
71
|
+
| Run state and history | `pyproject.toml` config |
|
|
72
|
+
| Infrastructure | |
|
|
73
|
+
|
|
74
|
+
Your pipeline code runs unchanged. Repster adds the operational layer around it.
|
|
@@ -0,0 +1,104 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Repster adds lightweight scheduling, run history, and an AI-ready CLI to dlt pipelines. Run them locally for free, or on managed serverless workers.
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# Repster
|
|
6
|
+
|
|
7
|
+
**Lightweight scheduling and observability for dlt pipelines.**
|
|
8
|
+
|
|
9
|
+
[dlt](https://dlthub.com) loads your data; Repster runs it. Add one decorator to an existing dlt pipeline and get a scheduler, run history, and a web UI, all behind an AI-ready CLI that does everything, so your coding agent can drive it too.
|
|
10
|
+
|
|
11
|
+
```python
|
|
12
|
+
import dlt
|
|
13
|
+
import repster as rp # add this
|
|
14
|
+
|
|
15
|
+
@rp.flow() # add this
|
|
16
|
+
def flow_name():
|
|
17
|
+
""" Your dlt pipeline code goes here """
|
|
18
|
+
...
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
uv add repster
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
## How it works
|
|
26
|
+
|
|
27
|
+
Illustrative example: an hourly schedule around a dlt pipeline:
|
|
28
|
+
|
|
29
|
+
```python
|
|
30
|
+
import dlt
|
|
31
|
+
import repster as rp
|
|
32
|
+
|
|
33
|
+
@rp.flow(schedule="0 * * * *") # run hourly
|
|
34
|
+
def github_stars():
|
|
35
|
+
@dlt.resource
|
|
36
|
+
def stars():
|
|
37
|
+
yield from fetch_stars("my-org/my-repo")
|
|
38
|
+
|
|
39
|
+
pipeline = dlt.pipeline(destination="duckdb")
|
|
40
|
+
pipeline.run(stars())
|
|
41
|
+
```
|
|
42
|
+
|
|
43
|
+
Run it locally:
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
rp run github_stars # run now
|
|
47
|
+
rp dev # start the scheduler and local UI
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
## What you get
|
|
51
|
+
|
|
52
|
+
Everything here is open source (Apache 2.0) and runs entirely on your machine:
|
|
53
|
+
|
|
54
|
+
* **An AI ready CLI.** Everything is an `rp` command, from running flows and viewing logs to deploying, managing secrets, and more. The web UI is an alternative view, never the only way.
|
|
55
|
+
|
|
56
|
+
* **One decorator.** `@rp.flow()` wraps your pipeline while leaving dlt fully usable underneath. Its state, destinations, and tooling all work as before.
|
|
57
|
+
|
|
58
|
+
* **Just run `rp dev`.** Repster schedules your flows, records every run, and serves a web UI where you can trigger flows, watch logs, and inspect run history.
|
|
59
|
+
|
|
60
|
+
Wondering how this compares to cron, Airflow, or Airbyte? See **[Why Repster?](explanation/why-repster.md)**
|
|
61
|
+
|
|
62
|
+
Because every operation is available from the CLI, AI coding agents can drive the entire development loop. Write a flow, run it, inspect the logs, fix the code, retry the run, and deploy, all without needing a browser.
|
|
63
|
+
|
|
64
|
+
Repster also ships with an agent skill, and these docs are published as [llms.txt](https://docs.repster.online/llms.txt) for agent context windows. See **[Use Repster with AI coding agents](how-tos/use-with-coding-agents.md)**.
|
|
65
|
+
|
|
66
|
+
## Repster Cloud
|
|
67
|
+
|
|
68
|
+
When pipelines need to run without your laptop open, deploy them:
|
|
69
|
+
|
|
70
|
+
```bash
|
|
71
|
+
rp deploy
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
Then flows can run on a schedule or on-demand:
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
rp run --cloud github_stars
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
Repster Cloud gives you:
|
|
81
|
+
|
|
82
|
+
* **Serverless compute**: each run gets its own isolated worker, created on demand and gone when the run finishes. Nothing to provision, size, or keep running. Automatically scales to your workload, and you only pay for what you use.
|
|
83
|
+
* **Managed scheduling**: the same cron expressions from your code, fired by the cloud.
|
|
84
|
+
* **Teams and secrets**: authentication, org roles, and an encrypted secrets vault.
|
|
85
|
+
* **Transparent pricing**: when metered billing launches, you'll pay for raw compute: CPU seconds, memory, storage. No seats, no rows, no MTU.
|
|
86
|
+
|
|
87
|
+
Develop locally, then deploy with confidence. The same flow code you test on your machine runs in Repster Cloud, so moving from development to production is seamless.
|
|
88
|
+
|
|
89
|
+
**[Deploy to cloud →](tutorials/deploy-to-cloud.md)**
|
|
90
|
+
|
|
91
|
+
## Get started
|
|
92
|
+
|
|
93
|
+
**[Quickstart →](tutorials/quickstart.md)**: build and run your first flow in 5 minutes
|
|
94
|
+
|
|
95
|
+
**[Deploy to cloud →](tutorials/deploy-to-cloud.md)**: move from local to managed execution
|
|
96
|
+
|
|
97
|
+
## Sections
|
|
98
|
+
|
|
99
|
+
| Section | What you'll find |
|
|
100
|
+
| ------- | --------------- |
|
|
101
|
+
| [Tutorials](tutorials/quickstart.md) | Step-by-step guides for getting started |
|
|
102
|
+
| [How-to guides](how-tos/manage-secrets.md) | Recipes for specific tasks |
|
|
103
|
+
| [Reference](reference/cli.md) | CLI commands, configuration, and API |
|
|
104
|
+
| [Explanation](explanation/why-repster.md) | Concepts and background |
|
|
@@ -0,0 +1,58 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Create, list, and revoke Repster Cloud API keys (rpk_) for CI and programmatic access.
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# Manage API keys
|
|
6
|
+
|
|
7
|
+
API keys (`rpk_...`) authenticate CLI and programmatic access to Repster Cloud. Keys are org-scoped and can be created, listed, and revoked.
|
|
8
|
+
|
|
9
|
+
## List keys
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
rp keys list
|
|
13
|
+
rp keys list --profile prod
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Shows key prefix, name, status, last-used date, and creation date. Key values are never shown after creation.
|
|
17
|
+
|
|
18
|
+
## Create a key
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
rp keys create
|
|
22
|
+
rp keys create --name "ci-deploy-key"
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
The raw key is printed once; copy it immediately. It cannot be retrieved again.
|
|
26
|
+
|
|
27
|
+
## Revoke a key
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
rp keys revoke <key-id-or-prefix>
|
|
31
|
+
rp keys revoke rpk_abc123 --yes # skip confirmation prompt
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
Revoked keys are rejected immediately by Repster Cloud.
|
|
35
|
+
|
|
36
|
+
## Using keys in CI
|
|
37
|
+
|
|
38
|
+
Set the key as an environment variable; repster picks it up automatically:
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
REPSTER_API_TOKEN=rpk_... rp deploy
|
|
42
|
+
```
|
|
43
|
+
|
|
44
|
+
Or configure it in a named profile:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
rp login --token rpk_... --profile ci
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
Then use `--profile ci` in commands that need it.
|
|
51
|
+
|
|
52
|
+
## Key roles
|
|
53
|
+
|
|
54
|
+
Keys inherit the role of the user who created them (see the role table in [Manage your team](manage-team.md)). Create dedicated CI keys with the minimum role needed.
|
|
55
|
+
|
|
56
|
+
## Web UI
|
|
57
|
+
|
|
58
|
+
Keys can also be created and revoked in the Repster Cloud UI ([repster.online](https://repster.online)) under **Settings → API Keys**.
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
---
|
|
2
|
+
description: Store credentials for dlt pipelines in .env locally and push them to the encrypted Repster Cloud secrets vault.
|
|
3
|
+
---
|
|
4
|
+
|
|
5
|
+
# Manage secrets
|
|
6
|
+
|
|
7
|
+
Repster separates secrets from your pipeline code: `.env` locally, the encrypted Repster Cloud vault for cloud runs.
|
|
8
|
+
|
|
9
|
+
## Local development
|
|
10
|
+
|
|
11
|
+
For local runs, Repster reads secrets from a `.env` file at the root of your project. Use dlt's env-var naming convention:
|
|
12
|
+
|
|
13
|
+
```properties
|
|
14
|
+
SOURCES__GITHUB__ACCESS_TOKEN=ghp_...
|
|
15
|
+
DESTINATION__BIGQUERY__CREDENTIALS__PROJECT_ID=my-project
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Never commit `.env`; add it to `.gitignore`.
|
|
19
|
+
|
|
20
|
+
## Pushing secrets to cloud
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
rp secrets push
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Reads all key-value pairs from `.env` and stores them in the vault, encrypted at rest. At run time they're injected into the worker as environment variables, never written to logs or run records (see [How Repster works](../explanation/how-repster-works.md#secrets)).
|
|
27
|
+
|
|
28
|
+
To target a specific profile:
|
|
29
|
+
|
|
30
|
+
```bash
|
|
31
|
+
rp secrets push --profile prod
|
|
32
|
+
```
|
|
33
|
+
|
|
34
|
+
## Viewing stored secrets (key names only)
|
|
35
|
+
|
|
36
|
+
```bash
|
|
37
|
+
rp secrets list
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
The vault returns key names only; values are never retrievable after being stored.
|
|
41
|
+
|
|
42
|
+
## Rotating a secret
|
|
43
|
+
|
|
44
|
+
Push the updated value; Repster overwrites the existing entry:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
# update .env, then:
|
|
48
|
+
rp secrets push
|
|
49
|
+
```
|
|
50
|
+
|
|
51
|
+
## Removing a secret
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
rp secrets delete SOURCES__GITHUB__ACCESS_TOKEN
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Prompts for confirmation (pass `--yes` to skip). Secrets can also be deleted in the Repster Cloud UI ([repster.online](https://repster.online)).
|
|
58
|
+
|
|
59
|
+
## CI / non-interactive environments
|
|
60
|
+
|
|
61
|
+
For CI pipelines, skip the interactive login flow by passing a token directly:
|
|
62
|
+
|
|
63
|
+
```bash
|
|
64
|
+
REPSTER_API_TOKEN=rpk_... rp secrets push
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Or configure it in `pyproject.toml`:
|
|
68
|
+
|
|
69
|
+
```toml
|
|
70
|
+
[tool.repster]
|
|
71
|
+
profile = "ci"
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
And add the profile to `~/.repster/config.toml` on the CI machine.
|