@trigger.dev/sdk 4.5.0-rc.5 → 4.5.0-rc.7

package/docs/self-hosting/docker.mdx

@@ -0,0 +1,458 @@
+---
+title: "Docker compose"
+description: "You can self-host Trigger.dev on your own infrastructure using Docker."
+---
+
+The following instructions will use docker compose to spin up a Trigger.dev instance. Make sure to read the self-hosting [overview](/self-hosting/overview) first.
+
+As self-hosted deployments tend to have unique requirements and configurations, we don't provide specific advice for securing your deployment, scaling up, or improving reliability.
+
+Should the burden ever get too much, we'd be happy to see you on [Trigger.dev cloud](https://trigger.dev/pricing) where we deal with these concerns for you.
+
+**Warning:** This guide alone is unlikely to result in a production-ready deployment. Security, scaling, and reliability concerns are not fully addressed here.
+
+## What's new?
+
+Goodbye v3, hello v4! We made quite a few changes:
+
+- **Much simpler setup.** The provider and coordinator are now combined into a single supervisor. No more startup scripts, just `docker compose up`.
+- **Automatic container cleanup.** The supervisor will automatically clean up containers that are no longer needed.
+- **Support for multiple worker machines.** This is a big one, and we're very excited about it! You can now scale your workers horizontally as needed.
+- **Resource limits enforced by default.** This means that tasks will be limited to the total CPU and RAM of the machine preset, preventing noisy neighbours.
+- **No direct Docker socket access.** The compose file now comes with [Docker Socket Proxy](https://github.com/Tecnativa/docker-socket-proxy) by default. Yes, you want this.
+- **No host networking.** All containers are now running with network isolation, using only the network access they need.
+- **No checkpoint support.** This was only ever experimental when self-hosting and not recommended. It caused a bunch of issues. We decided to focus on the core features instead.
+- **Built-in container registry and object storage.** You can now deploy and execute tasks without needing third party services for this.
+- **Improved CLI commands.** You don't need any additional flags to deploy anymore, and there's a new command to easily `switch` between profiles.
+- **Whitelisting for GitHub OAuth.** Any whitelisted email addresses will now also apply to sign ins via GitHub, unlike v3 where they only applied to magic links.
+
+## Requirements
+
+These are the minimum requirements for running the webapp and worker components. They can run on the same, or on separate machines.
+
+It's fine to run everything on the same machine for testing. To be able to scale your workers, you will want to run them separately.
+
+### Prerequisites
+
+To run the webapp and worker components, you will need:
+
+- [Docker](https://docs.docker.com/get-docker/) 20.10.0+
+- [Docker Compose](https://docs.docker.com/compose/install/) 2.20.0+
+
+### Webapp
+
+This machine will host the webapp, postgres, redis, and related services.
+
+- 3+ vCPU
+- 6+ GB RAM
+
+### Worker
+
+This machine will host the supervisor and all of the runs.
+
+- 4+ vCPU
+- 8+ GB RAM
+
+How many workers and resources you need will depend on your workloads and concurrency requirements.
+
+For example:
+
+- 10 concurrency x `small-1x` (0.5 vCPU, 0.5 GB RAM) = 5 vCPU and 5 GB RAM
+- 20 concurrency x `small-1x` (0.5 vCPU, 0.5 GB RAM) = 10 vCPU and 10 GB RAM
+- 100 concurrency x `small-1x` (0.5 vCPU, 0.5 GB RAM) = 50 vCPU and 50 GB RAM
+- 100 concurrency x `small-2x` (1 vCPU, 1 GB RAM) = 100 vCPU and 100 GB RAM
+
+You may need to spin up multiple workers to handle peak concurrency. The good news is you don't have to know the exact numbers upfront. You can start with a single worker and add more as needed.
+
+## Setup
+
+### Webapp
+
+1. Clone the repository
+
+```bash
+git clone --depth=1 https://github.com/triggerdotdev/trigger.dev
+cd trigger.dev/hosting/docker
+```
+
+2. Create a `.env` file
+
+```bash
+cp .env.example .env
+```
+
+3. Start the webapp
+
+```bash
+cd webapp
+docker compose up -d
+```
+
+4. Configure the webapp using the [environment variables](/self-hosting/env/webapp) in your `.env` file, then apply the changes:
+
+```bash
+docker compose up -d
+```
+
+5. You should now be able to access the webapp at `http://localhost:8030`. When logging in, check the container logs for the magic link:
+
+```bash
+docker compose logs -f webapp
+```
+
+6. (optional) To initialize a new project, run the following command:
+
+```bash
+npx trigger.dev@latest init -p <project-ref> -a http://localhost:8030
+```
+
+### Worker
+
+1. Clone the repository
+
+```bash
+git clone --depth=1 https://github.com/triggerdotdev/trigger.dev
+cd trigger.dev/hosting/docker
+```
+
+2. Create a `.env` file
+
+```bash
+cp .env.example .env
+```
+
+3. Start the worker
+
+```bash
+cd worker
+docker compose up -d
+```
+
+4. Configure the supervisor using the [environment variables](/self-hosting/env/supervisor) in your `.env` file, including the [worker token](#worker-token).
+
+5. Apply the changes:
+
+```bash
+docker compose up -d
+```
+
+6. Repeat as needed for additional workers.
+
+### Combined
+
+If you want to run the webapp and worker on the same machine, just replace the `up` command with the following:
+
+```bash
+# Run this from the /hosting/docker directory
+docker compose -f webapp/docker-compose.yml -f worker/docker-compose.yml up -d
+```
+
+## Worker token
+
+When running the combined stack, worker bootstrap is handled automatically. When running the webapp and worker separately, you will need to manually set the worker token.
+
+On the first run, the webapp will generate a worker token and store it in a shared volume. It will also print the token to the console. It should look something like this:
+
+```bash
+==========================
+Trigger.dev Bootstrap - Worker Token
+
+WARNING: This will only be shown once. Save it now!
+
+Worker group:
+bootstrap
+
+Token:
+tr_wgt_fgfAEjsTmvl4lowBLTbP7Xo563UlnVa206mr9uW6
+
+If using docker compose, set:
+TRIGGER_WORKER_TOKEN=tr_wgt_fgfAEjsTmvl4lowBLTbP7Xo563UlnVa206mr9uW6
+
+Or, if using a file:
+TRIGGER_WORKER_TOKEN=file:///home/node/shared/worker_token
+
+==========================
+```
+
+You can then uncomment and set the `TRIGGER_WORKER_TOKEN` environment variable in your `.env` file.
+
+Don't forget to restart the worker container for the changes to take effect:
+
+```bash
+# Run this from the /hosting/docker/worker directory
+docker compose down
+docker compose up -d
+```
+
+### Creating additional worker groups
+
+To create additional worker groups beyond the bootstrap group, use the admin API endpoint. This requires admin privileges.
+
+**Making a user admin:**
+
+- **New users**: Set `ADMIN_EMAILS` environment variable (regex pattern) before user creation.
+- **Existing users**: Set `admin = true` in the `user` table in your database.
+
+**Creating a worker group:**
+
+```bash
+api_url=http://localhost:8030
+wg_name=my-worker
+admin_pat=tr_pat_...
+
+curl -X POST \
+  "$api_url/admin/api/v1/workers" \
+  -H "Authorization: Bearer $admin_pat" \
+  -H "Content-Type: application/json" \
+  -d "{\"name\": \"$wg_name\"}"
+```
+
+The response includes a `token` field if the worker group is newly created.
+
+## Registry setup
+
+The registry is used to store and pull deployment images. When testing the stack locally, the defaults should work out of the box.
+
+When deploying to production, you will need to set the correct URL and generate secure credentials for the registry.
+
+### Default settings
+
+The default settings for the registry are:
+
+- Registry: `localhost:5000`
+- Username: `registry-user`
+- Password: `very-secure-indeed`
+
+You should change these before deploying to production, especially the password. You can find more information about how to do this in the official [registry docs](https://github.com/distribution/distribution/blob/735c161b53e7faf81a21ba94c55ac9edee081cd9/docs/deploying.md#native-basic-auth).
+
+**Note:** This will require modifying the default `.htpasswd` file located at `./hosting/docker/registry/auth.htpasswd` of the repo root.
+
+### Logging in
+
+When self-hosting, builds run locally. You will have to login to the registry on every machine that runs the `deploy` command. You should only have to do this once:
+
+```bash
+docker login -u <username> <registry>
+```
+
+This will prompt for the password. Afterwards, the deploy command should work as expected.
+
+## Object storage
+
+This is mainly used for large payloads and outputs. There are a few simple steps to follow to get started.
+
+### Default settings
+
+The default settings for the object storage are:
+
+- Endpoint: `http://localhost:9000`
+- Username: `admin`
+- Password: `very-safe-password`
+
+You should change these before deploying to production, especially the password.
+
+### Setup
+
+<Note>
+  The `packets` bucket is created by default. In case this doesn't work, you can create it manually.
+</Note>
+
+1. Login to the dashboard: `http://localhost:9001`
+
+2. Create a bucket named `packets`.
+
+3. For production, you will want to set up a dedicated user and not use the root credentials above.
+
+## Authentication
+
+The specific set of variables required will depend on your choice of email transport or alternative login methods like GitHub OAuth.
+
+### Magic link
+
+By default, magic link auth is the only login option. If the `EMAIL_TRANSPORT` env var is not set, the magic links will be logged by the webapp container and not sent via email.
+
+#### Resend
+
+```bash
+EMAIL_TRANSPORT=resend
+FROM_EMAIL=
+REPLY_TO_EMAIL=
+RESEND_API_KEY=<your_resend_api_key>
+```
+
+#### SMTP
+
+Note that setting `SMTP_SECURE=false` does _not_ mean the email is sent insecurely.
+This simply means that the connection is secured using the modern STARTTLS protocol command instead of implicit TLS.
+You should only set this to true when the SMTP server host directs you to do so (generally when using port 465)
+
+```bash
+EMAIL_TRANSPORT=smtp
+FROM_EMAIL=
+REPLY_TO_EMAIL=
+SMTP_HOST=<your_smtp_server>
+SMTP_PORT=587
+SMTP_SECURE=false
+SMTP_USER=<your_smtp_username>
+SMTP_PASSWORD=<your_smtp_password>
+```
+
+#### AWS SES
+
+Credentials are to be supplied as with any other program using the AWS SDK.
+
+In this scenario, you would likely either supply the additional environment variables `AWS_REGION`, `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` or, when running on AWS, use credentials supplied by the EC2 IMDS.
+
+```bash
+EMAIL_TRANSPORT=aws-ses
+FROM_EMAIL=
+REPLY_TO_EMAIL=
+```
+
+### GitHub OAuth
+
+To authenticate with GitHub, you will need to set up a GitHub OAuth app. It needs a callback URL `https://<your_webapp_domain>/auth/github/callback` and you will have to set the following env vars:
+
+```bash
+AUTH_GITHUB_CLIENT_ID=<your_client_id>
+AUTH_GITHUB_CLIENT_SECRET=<your_client_secret>
+```
+
+### Restricting access
+
+All email addresses can sign up and log in this way. If you would like to restrict this, you can use the `WHITELISTED_EMAILS` env var. For example:
+
+```bash
+# every email that does not match this regex will be rejected
+WHITELISTED_EMAILS="^(authorized@yahoo\.com|authorized@gmail\.com)$"
+```
+
+This will apply to all auth methods including magic link and GitHub OAuth.
+
+## Version locking
+
+There are several reasons to lock the version of your Docker images:
+
+- **Backwards compatibility.** We try our best to maintain compatibility with older CLI versions, but it's not always possible. If you don't want to update your CLI, you can lock your Docker images to that specific version.
+- **Ensuring full feature support.** Sometimes, new CLI releases will also require new or updated platform features. Running unlocked images can make any issues difficult to debug. Using a specific tag can help here as well.
+
+By default, the images will point at the latest versioned release via the `latest` tag. You can override this by specifying a different tag in your `.env` file. For example:
+
+```bash
+TRIGGER_IMAGE_TAG=v4.0.0
+```
+
+## Task events
+
+By default, task events (timeline, logs, spans) are stored in PostgreSQL. For production deployments we recommend storing them in ClickHouse instead, it scales to much higher volumes and avoids unbounded growth of the `TaskEvent` table.
+
+To enable, set on the webapp in your `.env`:
+
+```bash
+EVENT_REPOSITORY_DEFAULT_STORE=clickhouse_v2
+```
+
+This only affects new runs; existing runs continue to read from wherever their events were originally stored.
+
+## Troubleshooting
+
+- **Deployment fails at the push step.** The machine running `deploy` needs registry access. See the [registry setup](#registry-setup) section for more details.
+
+- **Magic links don't arrive.** The webapp container needs to be able to send emails. You probably need to set up an email transport. See the [authentication](#authentication) section for more details.
+
+  You should check the logs of the webapp container to see the magic link:
+
+  ```bash
+  # Run this from the /hosting/docker/webapp directory
+  docker compose logs -f webapp
+  ```
+
+- **Deploy fails with `ERROR: schema "graphile_worker" does not exist`.** This error occurs when Graphile Worker migrations fail to run during webapp startup. Check the webapp logs for certificate-related errors like `self-signed certificate in certificate chain`. This is often caused by PostgreSQL SSL certificate issues when using an external PostgreSQL instance with SSL enabled. Ensure that both the webapp and supervisor containers have access to the same CA certificate used by your PostgreSQL instance. You can configure this by mounting the certificate file and setting the `NODE_EXTRA_CA_CERTS` environment variable to point to the certificate path. Once the certificate issue is resolved, the migrations will complete and create the required `graphile_worker` schema.
+
+- **ClickHouse migrations say "no migrations to run" but schema is missing.** The goose migration tracker is out of sync. Exec into the webapp container, set the GOOSE env vars (from webapp startup logs), and run `goose reset && goose up`.
+
+  <Warning>
+    **Data Loss Warning:** The `goose reset` command is destructive and will drop the entire schema.
+    Make sure to backup your data and confirm you are running this in a non-production environment
+    before executing this command.
+  </Warning>
+
+## CLI usage
+
+This section highlights some of the CLI commands and options that are useful when self-hosting. Please check the [CLI reference](/cli-introduction) for more in-depth documentation.
+
+### Login
+
+To avoid being redirected to [Trigger.dev Cloud](https://cloud.trigger.dev) when using the CLI, you need to specify the URL of your self-hosted instance with the `--api-url` or `-a` flag. For example:
+
+```bash
+npx trigger.dev@latest login -a http://trigger.example.com
+```
+
+Once you've logged in, you shouldn't have to specify the URL again with other commands.
+
+### Profiles
+
+You can specify a profile when logging in. This allows you to easily use the CLI with multiple instances of Trigger.dev. For example:
+
+```bash
+npx trigger.dev@latest login -a http://trigger.example.com \
+    --profile self-hosted
+```
+
+Logging in with a new profile will also make it the new default profile.
+
+To use a specific profile, you can use the `--profile` flag with other commands:
+
+```bash
+npx trigger.dev@latest dev --profile self-hosted
+```
+
+To list all your profiles, use the `list-profiles` command:
+
+```bash
+npx trigger.dev@latest list-profiles
+```
+
+To remove a profile, use the `logout` command:
+
+```bash
+npx trigger.dev@latest logout --profile self-hosted
+```
+
+To switch to a different profile, use the `switch` command:
+
+```bash
+# To run interactively
+npx trigger.dev@latest switch
+
+# To switch to a specific profile
+npx trigger.dev@latest switch self-hosted
+```
+
+### Whoami
+
+It can be useful to check you are logged into the correct instance. Running this will also show the API URL:
+
+```bash
+npx trigger.dev@latest whoami
+```
+
+## CI / GitHub Actions
+
+When running the CLI in a CI environment, your login profiles won't be available. Instead, you can use the `TRIGGER_API_URL` and `TRIGGER_ACCESS_TOKEN` environment
+variables to point at your self-hosted instance and authenticate.
+
+For more detailed instructions, see the [GitHub Actions guide](/github-actions).
+
+## Telemetry
+
+By default, the Trigger.dev webapp sends telemetry data to our servers. This data is used to improve the product and is not shared with third parties. If you would like to opt-out of this, you can set the `TRIGGER_TELEMETRY_DISABLED` environment variable on the webapp container. The value doesn't matter, it just can't be empty. For example:
+
+```yaml
+services:
+  webapp:
+    ...
+    environment:
+      TRIGGER_TELEMETRY_DISABLED: 1
+```

package/docs/self-hosting/env/supervisor.mdx

@@ -0,0 +1,74 @@
+---
+title: "Supervisor"
+description: "Environment variables for the supervisor container."
+sidebarTitle: "Supervisor"
+mode: "wide"
+---
+
+| Name                                        | Required | Default     | Description                                                 |
+| :------------------------------------------ | :------- | :---------- | :---------------------------------------------------------- |
+| **Required settings**                       |          |             |                                                             |
+| `TRIGGER_API_URL`                           | Yes      | —           | Trigger.dev API URL. Should point at the webapp.            |
+| `TRIGGER_WORKER_TOKEN`                      | Yes      | —           | Worker token (can be a file path with file://).             |
+| `MANAGED_WORKER_SECRET`                     | Yes      | —           | Managed worker secret. Needs to match webapp value.         |
+| `OTEL_EXPORTER_OTLP_ENDPOINT`               | Yes      | —           | OTel exporter endpoint. Point at: `<webapp-url>/otel`       |
+| **Worker instance**                         |          |             |                                                             |
+| `TRIGGER_WORKER_INSTANCE_NAME`              | No       | random UUID | Worker instance name. Set to `spec.nodeName` on k8s.        |
+| `TRIGGER_WORKER_HEARTBEAT_INTERVAL_SECONDS` | No       | 30          | Worker heartbeat interval (seconds).                        |
+| **Workload API settings**                   |          |             |                                                             |
+| `TRIGGER_WORKLOAD_API_ENABLED`              | No       | true        | Enable workload API. Runs use this to perform actions.      |
+| `TRIGGER_WORKLOAD_API_PROTOCOL`             | No       | http        | Workload API protocol (http/https).                         |
+| `TRIGGER_WORKLOAD_API_DOMAIN`               | No       | —           | Workload API domain. Keep empty for auto-detection.         |
+| `TRIGGER_WORKLOAD_API_HOST_INTERNAL`        | No       | 0.0.0.0     | Workload API internal host.                                 |
+| `TRIGGER_WORKLOAD_API_PORT_INTERNAL`        | No       | 8020        | Workload API internal port.                                 |
+| `TRIGGER_WORKLOAD_API_PORT_EXTERNAL`        | No       | 8020        | Workload API external port.                                 |
+| **Runner settings**                         |          |             |                                                             |
+| `RUNNER_HEARTBEAT_INTERVAL_SECONDS`         | No       | —           | Runner heartbeat interval (seconds).                        |
+| `RUNNER_SNAPSHOT_POLL_INTERVAL_SECONDS`     | No       | —           | Runner snapshot poll interval (seconds).                    |
+| `RUNNER_ADDITIONAL_ENV_VARS`                | No       | —           | Additional runner env vars (CSV).                           |
+| `RUNNER_PRETTY_LOGS`                        | No       | false       | Pretty logs for runner.                                     |
+| **Dequeue settings**                        |          |             |                                                             |
+| `TRIGGER_DEQUEUE_ENABLED`                   | No       | true        | Enable dequeue to pull runs from the queue.                 |
+| `TRIGGER_DEQUEUE_INTERVAL_MS`               | No       | 250         | Dequeue interval (ms).                                      |
+| `TRIGGER_DEQUEUE_IDLE_INTERVAL_MS`          | No       | 1000 (1s)   | Dequeue idle interval (ms).                                 |
+| `TRIGGER_DEQUEUE_MAX_RUN_COUNT`             | No       | 10          | Max dequeue run count.                                      |
+| `TRIGGER_DEQUEUE_MAX_CONSUMER_COUNT`        | No       | 1           | Max dequeue consumer count.                                 |
+| **Docker settings**                         |          |             |                                                             |
+| `DOCKER_API_VERSION`                        | No       | v1.41       | Docker API version. You should probably not touch this.     |
+| `DOCKER_STRIP_IMAGE_DIGEST`                 | No       | true        | Strip image digest in Docker. Turning off can cause issues. |
+| `DOCKER_ENFORCE_MACHINE_PRESETS`            | No       | true        | Enforce Docker machine cpu and memory limits.               |
+| `DOCKER_AUTOREMOVE_EXITED_CONTAINERS`       | No       | true        | Auto-remove exited containers.                              |
+| `DOCKER_RUNNER_NETWORKS`                    | No       | host        | Docker runner networks (CSV).                               |
+| **Registry auth**                           |          |             |                                                             |
+| `DOCKER_REGISTRY_URL`                       | No       | —           | Docker registry URL, e.g. `docker.io`.                      |
+| `DOCKER_REGISTRY_USERNAME`                  | No       | —           | Docker registry username.                                   |
+| `DOCKER_REGISTRY_PASSWORD`                  | No       | —           | Docker registry password.                                   |
+| **Kubernetes settings**                     |          |             |                                                             |
+| `KUBERNETES_FORCE_ENABLED`                  | No       | false       | Force Kubernetes mode.                                      |
+| `KUBERNETES_NAMESPACE`                      | No       | default     | The namespace that runs should be in.                       |
+| `KUBERNETES_WORKER_NODETYPE_LABEL`          | No       | v4-worker   | Nodes for runs need this label, e.g. `nodetype=v4-worker`.  |
+| `KUBERNETES_IMAGE_PULL_SECRETS`             | No       | —           | Image pull secrets (CSV).                                   |
+| `KUBERNETES_EPHEMERAL_STORAGE_SIZE_LIMIT`   | No       | 10Gi        | Ephemeral storage size limit. Applies to all runs.          |
+| `KUBERNETES_EPHEMERAL_STORAGE_SIZE_REQUEST` | No       | 2Gi         | Ephemeral storage size request. Applies to all runs.        |
+| **Metrics**                                 |          |             |                                                             |
+| `METRICS_ENABLED`                           | No       | true        | Enable metrics.                                             |
+| `METRICS_COLLECT_DEFAULTS`                  | No       | true        | Collect default metrics.                                    |
+| `METRICS_HOST`                              | No       | 127.0.0.1   | Metrics host.                                               |
+| `METRICS_PORT`                              | No       | 9090        | Metrics port.                                               |
+| **Pod cleaner**                             |          |             |                                                             |
+| `POD_CLEANER_ENABLED`                       | No       | true        | Enable pod cleaner.                                         |
+| `POD_CLEANER_INTERVAL_MS`                   | No       | 10000 (10s) | Pod cleaner interval (ms). Best not to touch this.          |
+| `POD_CLEANER_BATCH_SIZE`                    | No       | 500         | Pod cleaner batch size.                                     |
+| **Failed pod handler**                      |          |             |                                                             |
+| `FAILED_POD_HANDLER_ENABLED`                | No       | true        | Enable failed pod handler.                                  |
+| `FAILED_POD_HANDLER_RECONNECT_INTERVAL_MS`  | No       | 1000 (1s)   | Failed pod handler reconnect interval (ms).                 |
+| **Debug**                                   |          |             |                                                             |
+| `DEBUG`                                     | No       | false       | Enable debug logs.                                          |
+| `SEND_RUN_DEBUG_LOGS`                       | No       | false       | Send run debug logs to the platform.                        |
+| **Not used for self-hosting**               |          |             |                                                             |
+| `TRIGGER_WARM_START_URL`                    | No       | —           | Warm start URL.                                             |
+| `TRIGGER_CHECKPOINT_URL`                    | No       | —           | Checkpoint URL.                                             |
+| `TRIGGER_METADATA_URL`                      | No       | —           | Metadata URL.                                               |
+| `RESOURCE_MONITOR_ENABLED`                  | No       | false       | Enable resource monitor.                                    |
+| `RESOURCE_MONITOR_OVERRIDE_CPU_TOTAL`       | No       | —           | Override CPU total for resource monitor.                    |
+| `RESOURCE_MONITOR_OVERRIDE_MEMORY_TOTAL_GB` | No       | —           | Override memory total (GB) for resource monitor.            |