flowstash-cli 0.1.0__py3-none-any.whl

flowstash/templates/AGENTS.md

@@ -0,0 +1,222 @@
+# flowstash Agents Guide
+
+This guide explains how to build AI Agents and robust integrations using flowstash. It covers the core concepts you need to know to bring data in, process it, and interact with external systems.
+
+## Core Concepts
+
+flowstash integrations revolve around the following core ideas:
+- **Integration Context**: Automatically tracks the current integration, pipeline, run, and context across distributed operations.
+- **Ingress**: How data enters your pipelines (via Webhooks or Polled schedules).
+- **Clients**: How you make authenticated requests to external APIs.
+- **Records Feed**: How you queue and deduplicate records for processing.
+- **Project Structure**: How to organize code between API, Worker, and Shared modules.
+
+## Bringing Data In: Ingress
+
+You can trigger your pipelines through two main ingress decorators provided by `flowstash_lib.pipelines.ingress`:
+
+### 1. Webhooks (`@ingress.webhook`)
+Use webhooks when an external system can push events to your application. This decorator registers the handler as part of the integration but leaves the exact handling logic to you.
+
+```python
+from flowstash_lib.pipelines.ingress import ingress
+
+@router.post("/webhook/slack")
+@ingress.webhook(pipeline="slack.messages", integration="slack")
+async def handle_slack_webhook(request):
+    data = await request.json()
+    # Read the webhook and process the data!
+```
+
+### 2. Polling (`@ingress.poll`)
+Use polling when you need to fetch data on a schedule. This decorator acts as a specialized scheduler entrypoint that injects a durable `state` dictionary into your function. This is perfect for remembering "watermarks" (like the last processed token or timestamp) to paginate stateful APIs.
+
+The state is automatically saved for you as long as the function executes without exceptions.
+
+```python
+from flowstash_lib.pipelines.ingress import ingress
+from datetime import datetime
+
+@ingress.poll(pipeline="slack.messages", integration="slack", schedule="*/5 * * * *")
+async def poll_slack(state: dict):
+    # Retrieve the watermark from the previous run
+    since = state.get("since")
+    
+    # Fetch new data using the watermark...
+    page = await fetch_messages_from_api(since=since)
+    
+    # Process or publish data
+    
+    # Update the watermark; it will be automatically saved!
+    state["since"] = datetime.now()
+```
+
+## Making External Requests: Clients
+
+flowstash makes it easy to interact with external APIs via Clients. They handle setting Base URLs, automatic Authentication headers, observability tracing, and defaults out of the box.
+
+### Adding a Client
+To configure a new REST client, you place a configuration file (like `slackClient.yaml`) in your shared clients folder. flowstash dynamically loads these clients if your configuration registry points to the folder. For example, if you have `_config/shared/clients.yaml` configured as follows:
+```yaml
+path: clients
+pattern: "*.yaml"
+```
+You can simply define `clients/slackClient.yaml` and the system will expose it to your tasks.
+
+### Using a Client
+Once configured, you can retrieve the standard HTTP client (from the `flowstash_clients` package) anywhere.
+
+```python
+from flowstash_clients import get_client
+from flowstash_lib.decorators import integration_task
+
+# Load the client configuration
+slack_client = get_client("slackClient")
+
+@integration_task(integration="slack", integration_pipeline="send_message")
+async def send_slack_message_task(message: str):
+    # Authorization and base URL logic are automatically handled here
+    await slack_client.request("POST", "/chat.postMessage", json={"text": message})
+```
+
+### Custom Clients
+If you have an API you interact with heavily, you can define a custom typed client. This encapsulates specific API routes for a better developer experience.
+
+```python
+from flowstash_lib.clients import HttpClient, client
+from typing import List
+
+# Extending HttpClient and registering with the @client decorator
+@client("DemoClient")
+class DemoClient(HttpClient):
+    """Custom client for the Demo API."""
+    
+    async def get_users(self) -> List[dict]:
+        response = await self.request("GET", "/users")
+        return response.json()
+        
+    async def get_user(self, user_id: int) -> dict:
+        response = await self.request("GET", f"/users/{user_id}")
+        return response.json()
+
+# You can then resolve this custom client by name:
+# demo = get_client("DemoClient")
+```
+
+## Processing Data: Records Feed
+
+When you receive payloads via Webhooks or Polling, you usually want to process them robustly and asynchronously on the backend. The `RecordsFeed` module provides an opinionated record queue with built-in deduplication that prevents overwhelming pipelines.
+
+### Publishing Records
+The framework resolves deduplication using a unique combination of `(integration, record_type, record_id)`. If multiple records arrive with the same identity, "latest-wins" semantics are applied (keeping the one with the latest timestamp).
+
+It also seamlessly supports feeding massive data payloads -- large objects (> 5KB) are automatically offloaded to a BlobStore.
+
+```python
+from flowstash_lib.pipelines.records_feed import RecordsFeed
+from flowstash_lib.pipelines.records_model import RecordData
+
+# Retrieve your specific feed
+feed = RecordsFeed.get(feed_id="slack.messages")
+
+# A common pattern is inserting data retrieved via an `@ingress.poll` into a feed
+await feed.publish(RecordData(
+    record_id="msg_123",
+    record_type="message",
+    data={"text": "Hello World", "user": "U123"}
+))
+```
+
+### Consuming Records
+To process the data asynchronously, decorate a function with `@feed_consumer`. This supports robust configurations like automatic batched receiving, delays, rate-limiting, and concurrency control.
+
+```python
+from flowstash_lib import feed_consumer, RecordData
+
+@feed_consumer(
+    feed_id="slack.messages", 
+    batch=True,
+    max_batch_size=50,
+    max_delay_ms=2000 # wait up to 2 seconds for the queue to fill
+)
+async def process_batch_messages(records: list[RecordData]):
+    """
+    Consumes a maximum of 50 records in a batch, waiting 
+    up to 2 seconds for the queue to fill.
+    """
+    print(f"Began processing {len(records)} records.")
+    for record in records:
+        print(f"Record: {record.record_id} of type {record.record_type}")
+        print(f"Data Payload: {record.data}")
+```
+
+```python
+@feed_consumer(
+    feed_id="slack.messages", 
+    batch=False
+)
+async def process_batch_messages(record: RecordData):
+    """
+    Consumes a single record at a time.
+    """
+    print(f"Began processing {record.record_id} of type {record.record_type}")
+    print(f"Data Payload: {record.data}")
+```
+## Project Structure
+
+A typical flowstash project is organized into three main areas: Configuration, Deployment, and Source Code.
+
+```text
+.
+├── config/                 # Application configuration
+│   ├── shared/             # Base configuration for all environments
+│   │   ├── backend.yaml    # Task backend settings (shared)
+│   │   ├── clients.yaml    # Points to the clients folder
+│   │   ├── clients/        # Shared client configurations
+│   │   │   └── fooClient.yaml
+│   │   └── .env            # Shared environment variables
+│   └── local/              # Environment-specific overrides (e.g., local, dev, prod)
+│       ├── backend.yaml    # Local-specific backend settings
+│       └── .env            # Local-specific environment variables
+├── deployment/             # deployment-related files
+│   ├── shared/             # Base Dockerfiles
+│   │   ├── api.Dockerfile
+│   │   └── worker.Dockerfile
+│   └── local/              # Local deployment configuration
+│       └── docker-compose.yaml
+├── src/                    # Source code for your integration
+│   ├── api/                # API-specific code (Routes, Webhooks)
+│   │   └── routes/
+│   │       └── webhooks.py # typical place for @ingress.webhook ... we can also add the webhooks into purpose specific files ie slack_webhooks.py / stripe_webhooks.py etc
+│   │       └── routes.py   # we can also add standard api routes ... like health check etc
+│   ├── worker/             # Worker-specific code (Tasks, Consumers)
+│   │   └── tasks/
+│   │       └── xyzTask.py  # place for feed consumers, polling tasks, integration tasks etc
+│   └── shared/             # Shared code (Models, Clients, Utils)
+│       ├── clients/        # here should to the custom clients
+│       └── models/         # here should to the custom models
+├── api_main.py             # API Entry point
+└── worker_main.py          # Worker Entry point
+```
+
+### Source Organization (`src/`)
+
+flowstash separates code based on where it executes to ensure clean isolation and efficient scaling:
+
+1.  **API (`src/api/`)**: Code that runs on the webserver. Its primary role is to handle incoming requests, typically defined using the `@ingress.webhook` decorator.
+2.  **Worker (`src/worker/`)**: Code that runs on the background worker. This is where the "heavy lifting" happens, including `@integration_task`, `@ingress.poll`, and `@feed_consumer` handlers.
+3.  **Shared (`src/shared/`)**: Logic used by both the API and Worker, such as custom typed Clients, Pydantic models, and utility functions.
+
+### Configuration & Environments
+
+flowstash uses a tiered configuration system that allows for seamless transitions between environments:
+
+- **Inheritance**: Configuration is loaded from `config/shared/` first, and then overridden by environment-specific files in `config/{env}/`.
+- **Backend Setup**: The `backend.yaml` file defines how tasks are executed (e.g., `asyncio` for local dev or `dramatiq` or `managed` for production).
+- **Environment Variables**: `.env` files in both the shared and environment folders are automatically resolved and injected into the application.
+
+### Execution Entry Points
+
+- **`api_main.py`**: The entry point for the web server. It sets up the FastAPI application and automatically imports modules from `src/api` to register routes.
+- **`worker_main.py`**: The entry point for the background process. It connects to the task backend (like Redis) and automatically imports modules from `src/worker` to register task signatures.
+

flowstash/templates/README.md

@@ -0,0 +1,135 @@
+# {project_name}
+
+flowstash application generated for {project_name}.
+
+## Structure
+
+- `api.py`: Entry point for the API service.
+- `worker.py`: Entry point for the worker service.
+- `src/shared/`: Shared helpers, constants, and client utilities.
+- `src/api/`: Webhook handlers and API startup.
+- `src/worker/`: Worker startup and task definitions (such as feed consumers).
+- `config/`: Environment-based configuration (e.g. `local/`, `dev/`, `prod/`, `shared/`).
+- `deployment/`: Docker Compose setup and Dockerfiles.
+
+---
+
+## Dependency Management
+
+This project defines dependencies using optional extras, allowing you to install only what is needed for a specific service.
+
+- **Base dependencies**: Required by both API and Worker.
+- **`api` extra**: Dependencies required only by the API service.
+- **`worker` extra**: Dependencies required only by the Worker service.
+- **`dev` extra** (optional): Development tooling.
+
+### Standard Installation (pip)
+
+Install for local development (recommended, installs everything):
+```bash
+pip install -e ".[api,worker,dev]"
+```
+
+Install API-only:
+```bash
+pip install -e ".[api]"
+```
+
+Install Worker-only:
+```bash
+pip install -e ".[worker]"
+```
+
+### Alternative Installation (Poetry)
+
+If you prefer managing your environment with Poetry:
+
+Install for local development (everything):
+```bash
+poetry install --extras "api worker dev"
+```
+
+Install API-only:
+```bash
+poetry install --extras "api"
+```
+
+Install Worker-only:
+```bash
+poetry install --extras "worker"
+```
+
+---
+
+## Running Locally
+
+### 1. Start supporting services (e.g., Redis)
+```bash
+docker compose -f deployment/local/docker-compose.yaml up -d
+```
+
+### 2. Set environment
+```bash
+export BEAMSTACK_ENV=local
+```
+
+### 3. Run API
+```bash
+python api.py
+```
+API will be available at [http://localhost:8000](http://localhost:8000).
+
+### 4. Run Worker
+```bash
+python worker.py
+```
+
+*(Alternatively, you can use the flowstash CLI to run your environments: `flowstash run local`)*
+
+---
+
+## flowstash Core Concepts Usage
+
+This project is built on `flowstash_lib` – a core library providing robust context management, queue abstractions, and observability. Here is how you can utilize its components in your application:
+
+### Integration Context
+`IntegrationContext` propagates automatically through all your integration operations. It manages the current `run_id`, `integration`, and `current_record_key` to ensure all logs and traces are correctly correlated without manual passing.
+
+### Ingress & Pipelines
+- **Webhook Ingress**: Use the `@ingress.webhook(pipeline="...", integration="...")` decorator on your FastAPI endpoints to attach integration context without manually wrapping your handlers.
+- **Polling Ingress**: Use `@ingress.poll(pipeline="...", integration="...", schedule="...")` to create durable, stateful scheduled tasks (such as API polling).
+
+### RecordsFeed & Consumers
+Extract and ingest data efficiently using **RecordsFeed**:
+```python
+from flowstash_lib.pipelines.records_feed import RecordsFeed, RecordData
+
+feed = RecordsFeed.get(feed_id="your.pipeline")
+feed.publish(RecordData(record_id="123", record_type="invoice", data={...}))
+```
+Records are automatically deduplicated and optionally ordered by timestamp. 
+
+Consume them robustly with **Feed Consumers**, which handle batching, rate-limiting, and parallel execution automatically:
+```python
+from flowstash_lib.pipelines.consumer import feed_consumer
+
+@feed_consumer(
+  feed_id="your.pipeline",
+  batch=True,
+  max_batch_size=100,
+  rate_limit_per_sec=50,
+)
+async def process_records(records: list[RecordData]):
+    for record in records:
+        # Insert your business logic here
+        pass
+```
+
+### Observability
+Use the provided `logger` to naturally correlate your logs with the active context. The framework tracks the lifecycle of records intrinsically (from publishing to consumption) without manual `record_id` logging.
+
+```python
+from flowstash_lib import logger
+
+logger.info("Processing the invoice batch", details="step 1 completed")
+```

flowstash/templates/_.dockerignore

@@ -0,0 +1,8 @@
+.git
+.venv
+__pycache__
+.pytest_cache
+*.pyc
+.flowstash
+README.md
+tests

flowstash/templates/_.flowstash

@@ -0,0 +1,4 @@
+project_name: {project_name}
+environments:
+  - name: dev
+    managed: false

flowstash/templates/_api_main.py

@@ -0,0 +1,21 @@
+from pathlib import Path
+from fastapi import Request
+from flowstash_lib.config.env_loader import load_config_dir
+from flowstash_runtime.ingress.app import create_fastapi_app
+import os
+env = os.getenv("ENVIRONMENT", "dev")
+# Load configuration relative to this file
+config = load_config_dir("config", environment=env)
+
+# Create FastAPI app
+app = create_fastapi_app(config, auto_import=[Path(__file__).parent / "src" / "api" / "routes" / "webhooks.py"])
+
+@app.get("/health")
+async def health():
+    """Simple status webhook."""
+    return {"status": "ok", "service": "{project_name}-api"}
+
+if __name__ == "__main__":
+    import uvicorn
+    port = int(os.getenv("PORT", 8000))
+    uvicorn.run(app, host="0.0.0.0", port=port)

flowstash/templates/_config/[env]/(backend-asyncio)/backend.yaml

@@ -0,0 +1,4 @@
+backend:
+  type: asyncio
+webhooks:
+  prefix: /webhooks

flowstash/templates/_config/[env]/(backend-dramatiq)/backend.yaml

@@ -0,0 +1,8 @@
+backend:
+  type: dramatiq
+
+
+webhooks:
+  prefix: /webhooks
+
+

flowstash/templates/_config/[env]/(backend-managed)/backend.yaml

@@ -0,0 +1,6 @@
+backend:
+  type: managed
+webhooks:
+  prefix: /webhooks
+
+

flowstash/templates/_config/[env]/_backend.yaml

@@ -0,0 +1,4 @@
+# Environment-specific overrides for backend
+backend:
+  # The type of backend can be 'dramatiq', 'asyncio', or 'managed'
+  type: managed | dramatiq | asyncio

flowstash/templates/_config/shared/.env

@@ -0,0 +1 @@
+flowstash_API_URL="http://localhost:8080"

flowstash/templates/_config/shared/backend.yaml

@@ -0,0 +1,4 @@
+webhooks:
+  prefix: /webhooks
+
+

flowstash/templates/_config/shared/clients/demoClient.yaml

@@ -0,0 +1,39 @@
+client_id: demoClient
+baseUrl: https://jsonplaceholder.typicode.com
+timeout: 10
+handleRedirects: false
+# Authentication configuration (OAuth2 example)
+auth:
+  type: oauth2
+  client_id: "${DEMO_CLIENT_ID}"
+  client_secret: "${DEMO_CLIENT_SECRET}"
+  token_url: "https://auth.example.com/token"
+  refresh_token: null
+  scopes: []
+  extra_params: {}
+
+# Authentication Alternatives (Commented out)
+# ------------------------------------------s
+# Basic Auth:
+# auth:
+#   type: basic
+#   username: "${DEMO_USERNAME}"
+#   password: "${DEMO_PASSWORD}"
+#
+# API Key Auth:
+# auth:
+#   type: api_key
+#   key: "X-API-Key"
+#   value: "${DEMO_API_KEY}"
+#   in: header # Supported values: header, query
+
+# Retry configuration
+retry:
+  maxRetries: 0
+  whitelist: [] # Keywords in response to retry (e.g., ["rate limit"])
+  blacklist: [] # Keywords in response NOT to retry (e.g., ["400"])
+
+# Extra parameters (e.g., global headers)
+extra:
+  headers:
+    User-Agent: DemoClient/1.0

flowstash/templates/_config/shared/clients.yaml

@@ -0,0 +1,3 @@
+path: clients
+pattern: "*.yaml"
+recursive: false

flowstash/templates/_deployment/[env]/(backend-asyncio)/docker-compose.yaml

@@ -0,0 +1,24 @@
+version: '3.8'
+services:
+  api:
+    build:
+      context: ../..
+      dockerfile: deployment/shared/api.Dockerfile
+    image: flowstash-{project_name_slug}-api:latest
+    ports:
+      - "8000:8000"
+    environment:
+      - ENVIRONMENT={env}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+
+  worker:
+    build:
+      context: ../..
+      dockerfile: deployment/shared/worker.Dockerfile
+    image: flowstash-{project_name_slug}-worker:latest
+    environment:
+      - ENVIRONMENT={env}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+

flowstash/templates/_deployment/[env]/(backend-dramatiq)/docker-compose.yaml

@@ -0,0 +1,34 @@
+version: '3.8'
+services:
+  api:
+    build:
+      context: ../..
+      dockerfile: deployment/shared/api.Dockerfile
+    image: flowstash-{project_name_slug}-api:latest
+    ports:
+      - "8000:8000"
+    environment:
+      - REDIS_URL=redis://redis:6379/0
+      - ENVIRONMENT={env}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    depends_on:
+      - redis
+
+  worker:
+    build:
+      context: ../..
+      dockerfile: deployment/shared/worker.Dockerfile
+    image: flowstash-{project_name_slug}-worker:latest
+    environment:
+      - REDIS_URL=redis://redis:6379/0
+      - ENVIRONMENT={env}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    depends_on:
+      - redis
+
+  redis:
+    image: redis:7-alpine
+    ports:
+      - "6379:6379"

flowstash/templates/_deployment/[env]/.env

@@ -0,0 +1,3 @@
+# Specific Envirment Variables for this enviroment belong here
+# i.e.
+# FOO_SEVICE_BASE_URL=dev/prod.rest.domain.com/v1

flowstash/templates/_deployment/shared/.env

@@ -0,0 +1,5 @@
+# Any ENV variables that should have the same value for any enviroment bellong here
+# i.e.
+# TEST_ENV_VAR=demo-value
+# DEMO_CLIENT_ID=123
+# DEMO_CLIENT_SECRET=456

flowstash/templates/_deployment/shared/api.Dockerfile

@@ -0,0 +1,18 @@
+# API Dockerfile for {project_name}
+FROM flowstash/flowstash-base:latest
+
+WORKDIR /app
+COPY . .
+
+# Build arg that chooses which extra(s) to install
+# e.g. SERVICE_EXTRAS="api" or "worker" or "api,worker"
+ARG SERVICE_EXTRAS="api"
+
+# Install base deps + extras into system site-packages
+# Base deps:
+RUN uv pip install --system -r pyproject.toml
+RUN uv pip install --system -r pyproject.toml --extra api || echo "no worker extra; skipping"
+RUN rm -rf src/worker
+EXPOSE 8000
+ENV PYTHONPATH=/app/src:/app
+CMD ["python", "api_main.py"]

flowstash/templates/_deployment/shared/worker.Dockerfile

@@ -0,0 +1,18 @@
+# Worker Dockerfile for {project_name}
+FROM flowstash/flowstash-base:latest
+
+WORKDIR /app
+COPY . .
+
+# Build arg that chooses which extra(s) to install
+# e.g. SERVICE_EXTRAS="api" or "worker" or "api,worker"
+ARG SERVICE_EXTRAS="worker"
+
+# Install base deps + extras into system site-packages
+# Base deps:
+RUN uv pip install --system -r pyproject.toml
+RUN uv pip install --system -r pyproject.toml --extra worker || echo "no worker extra; skipping"
+
+RUN rm -rf src/api
+ENV PYTHONPATH=/app/src:/app
+CMD ["python", "-u", "worker_main.py"]

flowstash/templates/_pyproject.toml

@@ -0,0 +1,40 @@
+[project]
+name = "{project_name}-app"
+version = "0.1.0"
+description = "flowstash application {project_name}"
+requires-python = ">=3.11"
+
+# Base dependencies (required by both API and Worker)
+dependencies = [
+    "pyyaml>=6.0",
+]
+
+[project.optional-dependencies]
+
+# API-only dependencies
+api = [
+    "fastapi>=0.100.0",
+    "uvicorn>=0.23.0",
+]
+
+# Worker-only dependencies
+worker = [
+    "dramatiq[redis]>=1.14.0",
+]
+
+# Development dependencies (optional)
+dev = [
+    "pytest>=7.0",
+    "ipython>=8.0",
+]
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+addopts = "-v -s"
+testpaths = ["tests"]
+python_files = "test_*.py"

flowstash/templates/_src/_api/__init__.py

@@ -0,0 +1 @@
+# API package

flowstash/templates/_src/_api/_routes/webhooks.py

@@ -0,0 +1,25 @@
+from flowstash_lib import ingress
+from flowstash_lib import RecordData
+from fastapi import Request
+from flowstash_lib.pipelines.records_feed import RecordsFeed
+
+@ingress.webhook(integration="demo", pipeline="process_user", path="/demo/process_user", method="POST")
+async def process_user_webhook(request: Request):
+    """Webhook that triggers a background worker task."""
+    try:
+        data = await request.json()
+    except Exception:
+        data = {}
+        
+    user_id = data.get("user_id")
+    
+    if not user_id:
+        return {"error": "user_id required"}, 400
+
+    feed = RecordsFeed.get("test_feed")
+    await feed.publish(RecordData(record_id=user_id, record_type="user", data={"user_id": user_id}))
+    
+    return {
+        "status": "submitted",
+        "user_id": user_id
+    }

flowstash/templates/_src/_shared/__init__.py

@@ -0,0 +1 @@
+# Shared package