pdt-cli 0.1.0__tar.gz → 0.1.2__tar.gz

{pdt_cli-0.1.0 → pdt_cli-0.1.2}/.gitignore

@@ -15,6 +15,7 @@ dist/
 *.egg-info/
 .pytest_cache/
 .uv-cache/
+claude-task-report/
 
 # OS / editor noise
 .DS_Store

pdt_cli-0.1.0/README.md → pdt_cli-0.1.2/PKG-INFO

@@ -1,3 +1,34 @@
+Metadata-Version: 2.5
+Name: pdt-cli
+Version: 0.1.2
+Summary: Easily deploy your process to AWS, Azure, Google Cloud, or Windows.
+License-Expression: MIT
+Requires-Python: >=3.12
+Requires-Dist: argcomplete
+Requires-Dist: backoff
+Requires-Dist: google-auth
+Requires-Dist: google-auth-oauthlib
+Requires-Dist: msal
+Requires-Dist: python-dotenv
+Requires-Dist: pyyaml
+Requires-Dist: rich
+Requires-Dist: rich-argparse>=1.8.0
+Requires-Dist: shellingham
+Provides-Extra: apps
+Requires-Dist: adlfs; extra == 'apps'
+Requires-Dist: azure-identity; extra == 'apps'
+Requires-Dist: azure-keyvault-secrets; extra == 'apps'
+Requires-Dist: azure-storage-blob; extra == 'apps'
+Requires-Dist: boto3; extra == 'apps'
+Requires-Dist: cryptography; extra == 'apps'
+Requires-Dist: fsspec; extra == 'apps'
+Requires-Dist: gcsfs; extra == 'apps'
+Requires-Dist: google-cloud-secret-manager; extra == 'apps'
+Requires-Dist: google-cloud-storage; extra == 'apps'
+Requires-Dist: pyjwt; extra == 'apps'
+Requires-Dist: s3fs; extra == 'apps'
+Description-Content-Type: text/markdown
+
 # pdt
 
 Run scheduled jobs — reports, integrations, automations — and deploy them to AWS, Azure, Google Cloud, or Windows Task Scheduler with one command.
@@ -10,6 +41,8 @@ uv tool install pdt-cli
 
 That puts a `pdt` command on your PATH. To update it later, run `uv tool upgrade pdt-cli`.
 
+The first interactive `pdt` command turns on tab completion without a prompt. It works in every new terminal: Tab finishes pdt commands, their options, your app names, and example names for `new --from`. It supports bash, zsh, fish, and PowerShell. `pdt completion [SHELL]` sets up a second shell, and `pdt completion --script bash` prints the script instead of writing it, if you keep your own dotfiles.
+
 You can also clone this repository and run `./pdt` (or `.\pdt.bat` on Windows) instead. It installs `uv` for you if you do not have it. Both ways give you the same commands.
 
 ## Set up a project
@@ -20,7 +53,7 @@ A project is a folder holding `pdt.yml`. Each app is a folder inside it that con
 pdt init my-jobs
 ```
 
-`pdt init` asks where the project should live, which cloud you want, and which region. It warns you if you are about to create a project somewhere unwise, such as your home folder. Run `pdt init` with no folder name to use the current folder, or add `--yes` to take the defaults and answer nothing.
+`pdt init` asks where the project should live, which cloud you want, and which region. It warns you if you are about to create a project somewhere unwise, such as your home folder. The folder name is optional: `pdt init` with no name uses the current folder, and `pdt init DIR` uses `DIR`, creating it if it is not there yet. Add `--yes` to take the defaults and answer nothing.
 
 Region is the only setting it asks for. Your AWS account, Azure subscription, and Google Cloud project all come from your credentials the first time you deploy, and pdt writes the answer into `pdt.yml` so every later deploy checks against it.
 
@@ -84,17 +117,70 @@ pdt login my-report
 | `pdt init [DIR]` | create a project here, or in DIR |
 | `pdt examples` | list the example apps bundled with pdt |
 | `pdt new APP --from EXAMPLE` | add an app to the project |
-| `pdt list` | show every app with its schedule and provider |
+| `pdt list` | show every app with its schedule and provider; `--names` prints only the enabled app names |
 | `pdt validate` | check the config files and the required env vars |
 | `pdt run APP` | run an app on this machine |
 | `pdt deploy APP` | deploy an app to its configured platform |
 | `pdt destroy APP` | remove everything deploy created |
+| `pdt secrets APP` | show which `.env` values differ from the deployed app |
+| `pdt secrets APP save` | send your `.env` values to the deployed app; the next run uses them |
+| `pdt secrets APP get` | copy the deployed values into a `.env.<provider>` file |
+| `pdt secrets APP set NAME` | put one value, read from stdin, into the deployed app's secrets |
 | `pdt login APP` | sign in again to the app's cloud provider |
+| `pdt storage APP ls|get|query|destroy` | look at, fetch, query, or delete the app's stored files |
 | `pdt az ...` | run the Azure CLI that pdt installs |
 | `pdt gcloud ...` | run the Google Cloud CLI that pdt installs |
+| `pdt completion [SHELL]` | turn on tab completion for a shell |
+
+Leave `APP` off `run`, `deploy`, `destroy`, `secrets`, or `login`, or mistype it, and pdt lists the apps in the project so you can pick one.
 
 `pdt az` and `pdt gcloud` hand your arguments straight to the cloud tool, and install it first if it is missing. For example, `pdt az account list`.
 
+## Keeping files between runs
+
+Your app runs, writes some files, and stops. Then the computer it ran on is thrown away, and the files go with it. So pdt gives every app a folder in your cloud account that stays. Deploy the app, destroy it, deploy it again: the folder and everything in it is still there.
+
+pdt creates one bucket per cloud account, named `pdt-data-` plus a short code, and gives each app its own folder inside it. When your app runs in the cloud, `PDT_STORAGE_URL` points at that folder. When you run `pdt run APP` on your own computer, it points at `.pdt/storage/APP/` inside your project instead. The app code is the same in both places.
+
+Write and read files with `pdt.utils.storage`:
+
+```python
+from pdt.utils import storage
+
+store = storage.store()
+with store.open("report.csv", "wb") as f:
+    f.write(b"name,count\n")
+```
+
+`store.fs()` gives you the full [fsspec](https://filesystem-spec.readthedocs.io/) filesystem, rooted at your app's folder. Output from one run goes under `store.run_folder()`, which names a new `runs/<time>-<id>/` folder each run.
+
+Some apps need files from the last run: a Meltano bookmark, a database. Keep those in one local folder, and copy it down before the work and up after:
+
+```python
+from pathlib import Path
+from pdt.utils import storage
+
+store = storage.store()
+lease = store.pull("state/", Path(".pdt-state"))
+# do the work; keep anything the next run needs inside .pdt-state/
+store.push(Path(".pdt-state"), "state/", lease)
+```
+
+`pull` locks the folder, so a second copy of your app cannot run at the same time and mix up the files. `push` checks that nobody else changed them, saves them, and unlocks. If a run crashes, the next one takes over the lock after 30 minutes.
+
+From your own computer, `pdt storage APP ls`, `get`, and `query` read the files with your own cloud sign-in. `pdt storage APP destroy` is the only command that deletes them, and it asks first. An app that needs none of this sets `storage: false` in its `config.yml`.
+
+An app that is not ready sets `enabled: false` in its `config.yml`. `pdt list` still shows it, and every other command acts as if the app is not there. `uv run run.py` in the app folder still runs it.
+
+## Running pdt from CI
+
+Every command reads its values from the environment it runs in. A build server sets them as CI variables, and no `.env` file is needed.
+
+- A `.env` file is a convenience for a person working on their own machine. A value that is already in the environment wins over the same name in a `.env` file.
+- `PDT_ENV_JSON` holds every value as one JSON object, for a CI system that keeps one secret instead of many. For example, `PDT_ENV_JSON={"PDT_TOKEN": "abc", "PDT_SMTP_USER": "reports@example.com"}`.
+- pdt creates no `.env` file on a build server. It looks for the `CI` variable that build systems set, and for a missing terminal. Email authorization must therefore be done first: run `pdt run APP` once on a machine with a browser, then copy `PDT_SMTP_OAUTH_CACHE_B64` (or `PDT_GRAPH_MAIL_CACHE_B64` for Microsoft Graph) from your `.env` into the CI variables.
+- `pdt deploy APP` and `pdt destroy APP` ask before they change anything. Add `--yes` so they proceed without asking.
+
 ## Choosing where jobs run
 
 Set `platform:` in `pdt.yml` for every app, or in an app's own `config.yml` for one app. An app's own file wins.
@@ -107,12 +193,14 @@ platform:
   subscription: 00000000-0000-0000-0000-000000000000
   region: eastus
   resource_group: pdt
-  # Options: functions (default), container_apps
-  runtime: functions
+  # Optional. An environment you already own, as <resource-group>/<name>.
+  # environment: my-group/my-environment
 ```
 
 `subscription` is optional. When it is missing or wrong, the deploy asks you to choose one.
 
+Every pdt project in a subscription runs its jobs in one shared Container Apps environment per region, `pdt-<region>` in the resource group `pdt-shared`, because a subscription allows only a few environments. Destroying the last app that uses the environment removes it, and removes `pdt-shared` once it holds no environment. Set `environment` to use an environment you already have; pdt then never creates, changes, or deletes it.
+
 ### AWS
 
 ```yaml
@@ -121,10 +209,12 @@ platform:
   region: us-east-1
   # Written for you on the first deploy, from your credentials.
   account: "123456789012"
-  # lambda (default) or fargate
-  runtime: lambda
+  # Optional. Which profile in ~/.aws to use. Written for you if pdt has to ask.
+  profile: work
 ```
 
+`profile` is optional. When it is missing, pdt uses `AWS_PROFILE`, or the only profile on your computer. When there are several and none is chosen, or the one in the file is not on this computer, the deploy lists them and asks you to choose, then writes your answer here.
+
 ### Google Cloud
 
 ```yaml
@@ -136,6 +226,25 @@ platform:
 
 `project` is optional. When it is missing or wrong, the deploy lists your projects and asks you to choose one, then writes your answer here.
 
+### Bringing your own Dockerfile
+
+Every cloud provider runs a job as a container: AWS on Fargate, Azure on Container Apps Jobs, Google Cloud on Cloud Run Jobs. The deploy builds an image for each app from a generated Dockerfile. It copies the app folder and `pdt.yml` into `/workspace`, installs the script-header dependencies of `run.py` with `uv sync --script`, and runs `run.py` as the entrypoint.
+
+Put a `Dockerfile` in the app folder to build the image your own way, for example to add system packages or to install a heavy tool at build time instead of on every run. The build context is the same as the generated one: the app folder under its own name, next to `pdt.yml`. Start from the generated file:
+
+```dockerfile
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
+COPY . /workspace
+WORKDIR /workspace/my-app
+ENV PDT_PROJECT=/workspace
+RUN uv sync --script run.py
+ENTRYPOINT ["uv", "run", "--script", "run.py"]
+```
+
+A `.dockerignore` in the app folder keeps files out of the image. Write its patterns relative to the app folder (`.meltano`, `output`, `*.csv`); pdt moves them to the root of the build context for you. `.env` and the other secret files never reach the context, with or without a `.dockerignore`.
+
+The Windows provider never builds an image, so `pdt validate` reports a Dockerfile in an app that uses it.
+
 ### Windows Task Scheduler
 
 ```yaml
@@ -195,7 +304,7 @@ An app is a folder with a `run.py` that has a `main()` function. It declares its
 ```python
 # /// script
 # requires-python = ">=3.12"
-# dependencies = ["pdt-cli[apps]==0.1.0"]
+# dependencies = ["pdt-cli[apps]==0.1.2"]
 # ///
 from pathlib import Path
 
@@ -305,3 +414,15 @@ Leave the two SES key variables empty to use the standard AWS credential chain (
 Required environment variables:
 
 - PDT_RESEND_API_KEY
+
+## Logging
+
+`pdt.utils.log` writes one log line per event:
+
+```python
+from pdt.utils.log import die, log
+
+log("info", "processed", rows=42)
+```
+
+Levels are `debug`, `info`, `warning`, and `error`. Locally a line is human-readable text on stdout; with `LOG_FORMAT=json` it is one JSON object per line, which is also the default on Cloud Run.

pdt_cli-0.1.0/PKG-INFO → pdt_cli-0.1.2/README.md

@@ -1,21 +1,3 @@
-Metadata-Version: 2.5
-Name: pdt-cli
-Version: 0.1.0
-Summary: Validate, run, and deploy small scheduled jobs to AWS, Azure, Google Cloud, or Windows.
-License-Expression: MIT
-Requires-Python: >=3.12
-Requires-Dist: google-auth
-Requires-Dist: google-auth-oauthlib
-Requires-Dist: msal
-Requires-Dist: python-dotenv
-Requires-Dist: pyyaml
-Provides-Extra: apps
-Requires-Dist: boto3; extra == 'apps'
-Requires-Dist: cryptography; extra == 'apps'
-Requires-Dist: google-cloud-secret-manager; extra == 'apps'
-Requires-Dist: pyjwt; extra == 'apps'
-Description-Content-Type: text/markdown
-
 # pdt
 
 Run scheduled jobs — reports, integrations, automations — and deploy them to AWS, Azure, Google Cloud, or Windows Task Scheduler with one command.
@@ -28,6 +10,8 @@ uv tool install pdt-cli
 
 That puts a `pdt` command on your PATH. To update it later, run `uv tool upgrade pdt-cli`.
 
+The first interactive `pdt` command turns on tab completion without a prompt. It works in every new terminal: Tab finishes pdt commands, their options, your app names, and example names for `new --from`. It supports bash, zsh, fish, and PowerShell. `pdt completion [SHELL]` sets up a second shell, and `pdt completion --script bash` prints the script instead of writing it, if you keep your own dotfiles.
+
 You can also clone this repository and run `./pdt` (or `.\pdt.bat` on Windows) instead. It installs `uv` for you if you do not have it. Both ways give you the same commands.
 
 ## Set up a project
@@ -38,7 +22,7 @@ A project is a folder holding `pdt.yml`. Each app is a folder inside it that con
 pdt init my-jobs
 ```
 
-`pdt init` asks where the project should live, which cloud you want, and which region. It warns you if you are about to create a project somewhere unwise, such as your home folder. Run `pdt init` with no folder name to use the current folder, or add `--yes` to take the defaults and answer nothing.
+`pdt init` asks where the project should live, which cloud you want, and which region. It warns you if you are about to create a project somewhere unwise, such as your home folder. The folder name is optional: `pdt init` with no name uses the current folder, and `pdt init DIR` uses `DIR`, creating it if it is not there yet. Add `--yes` to take the defaults and answer nothing.
 
 Region is the only setting it asks for. Your AWS account, Azure subscription, and Google Cloud project all come from your credentials the first time you deploy, and pdt writes the answer into `pdt.yml` so every later deploy checks against it.
 
@@ -102,17 +86,70 @@ pdt login my-report
 | `pdt init [DIR]` | create a project here, or in DIR |
 | `pdt examples` | list the example apps bundled with pdt |
 | `pdt new APP --from EXAMPLE` | add an app to the project |
-| `pdt list` | show every app with its schedule and provider |
+| `pdt list` | show every app with its schedule and provider; `--names` prints only the enabled app names |
 | `pdt validate` | check the config files and the required env vars |
 | `pdt run APP` | run an app on this machine |
 | `pdt deploy APP` | deploy an app to its configured platform |
 | `pdt destroy APP` | remove everything deploy created |
+| `pdt secrets APP` | show which `.env` values differ from the deployed app |
+| `pdt secrets APP save` | send your `.env` values to the deployed app; the next run uses them |
+| `pdt secrets APP get` | copy the deployed values into a `.env.<provider>` file |
+| `pdt secrets APP set NAME` | put one value, read from stdin, into the deployed app's secrets |
 | `pdt login APP` | sign in again to the app's cloud provider |
+| `pdt storage APP ls|get|query|destroy` | look at, fetch, query, or delete the app's stored files |
 | `pdt az ...` | run the Azure CLI that pdt installs |
 | `pdt gcloud ...` | run the Google Cloud CLI that pdt installs |
+| `pdt completion [SHELL]` | turn on tab completion for a shell |
+
+Leave `APP` off `run`, `deploy`, `destroy`, `secrets`, or `login`, or mistype it, and pdt lists the apps in the project so you can pick one.
 
 `pdt az` and `pdt gcloud` hand your arguments straight to the cloud tool, and install it first if it is missing. For example, `pdt az account list`.
 
+## Keeping files between runs
+
+Your app runs, writes some files, and stops. Then the computer it ran on is thrown away, and the files go with it. So pdt gives every app a folder in your cloud account that stays. Deploy the app, destroy it, deploy it again: the folder and everything in it is still there.
+
+pdt creates one bucket per cloud account, named `pdt-data-` plus a short code, and gives each app its own folder inside it. When your app runs in the cloud, `PDT_STORAGE_URL` points at that folder. When you run `pdt run APP` on your own computer, it points at `.pdt/storage/APP/` inside your project instead. The app code is the same in both places.
+
+Write and read files with `pdt.utils.storage`:
+
+```python
+from pdt.utils import storage
+
+store = storage.store()
+with store.open("report.csv", "wb") as f:
+    f.write(b"name,count\n")
+```
+
+`store.fs()` gives you the full [fsspec](https://filesystem-spec.readthedocs.io/) filesystem, rooted at your app's folder. Output from one run goes under `store.run_folder()`, which names a new `runs/<time>-<id>/` folder each run.
+
+Some apps need files from the last run: a Meltano bookmark, a database. Keep those in one local folder, and copy it down before the work and up after:
+
+```python
+from pathlib import Path
+from pdt.utils import storage
+
+store = storage.store()
+lease = store.pull("state/", Path(".pdt-state"))
+# do the work; keep anything the next run needs inside .pdt-state/
+store.push(Path(".pdt-state"), "state/", lease)
+```
+
+`pull` locks the folder, so a second copy of your app cannot run at the same time and mix up the files. `push` checks that nobody else changed them, saves them, and unlocks. If a run crashes, the next one takes over the lock after 30 minutes.
+
+From your own computer, `pdt storage APP ls`, `get`, and `query` read the files with your own cloud sign-in. `pdt storage APP destroy` is the only command that deletes them, and it asks first. An app that needs none of this sets `storage: false` in its `config.yml`.
+
+An app that is not ready sets `enabled: false` in its `config.yml`. `pdt list` still shows it, and every other command acts as if the app is not there. `uv run run.py` in the app folder still runs it.
+
+## Running pdt from CI
+
+Every command reads its values from the environment it runs in. A build server sets them as CI variables, and no `.env` file is needed.
+
+- A `.env` file is a convenience for a person working on their own machine. A value that is already in the environment wins over the same name in a `.env` file.
+- `PDT_ENV_JSON` holds every value as one JSON object, for a CI system that keeps one secret instead of many. For example, `PDT_ENV_JSON={"PDT_TOKEN": "abc", "PDT_SMTP_USER": "reports@example.com"}`.
+- pdt creates no `.env` file on a build server. It looks for the `CI` variable that build systems set, and for a missing terminal. Email authorization must therefore be done first: run `pdt run APP` once on a machine with a browser, then copy `PDT_SMTP_OAUTH_CACHE_B64` (or `PDT_GRAPH_MAIL_CACHE_B64` for Microsoft Graph) from your `.env` into the CI variables.
+- `pdt deploy APP` and `pdt destroy APP` ask before they change anything. Add `--yes` so they proceed without asking.
+
 ## Choosing where jobs run
 
 Set `platform:` in `pdt.yml` for every app, or in an app's own `config.yml` for one app. An app's own file wins.
@@ -125,12 +162,14 @@ platform:
   subscription: 00000000-0000-0000-0000-000000000000
   region: eastus
   resource_group: pdt
-  # Options: functions (default), container_apps
-  runtime: functions
+  # Optional. An environment you already own, as <resource-group>/<name>.
+  # environment: my-group/my-environment
 ```
 
 `subscription` is optional. When it is missing or wrong, the deploy asks you to choose one.
 
+Every pdt project in a subscription runs its jobs in one shared Container Apps environment per region, `pdt-<region>` in the resource group `pdt-shared`, because a subscription allows only a few environments. Destroying the last app that uses the environment removes it, and removes `pdt-shared` once it holds no environment. Set `environment` to use an environment you already have; pdt then never creates, changes, or deletes it.
+
 ### AWS
 
 ```yaml
@@ -139,10 +178,12 @@ platform:
   region: us-east-1
   # Written for you on the first deploy, from your credentials.
   account: "123456789012"
-  # lambda (default) or fargate
-  runtime: lambda
+  # Optional. Which profile in ~/.aws to use. Written for you if pdt has to ask.
+  profile: work
 ```
 
+`profile` is optional. When it is missing, pdt uses `AWS_PROFILE`, or the only profile on your computer. When there are several and none is chosen, or the one in the file is not on this computer, the deploy lists them and asks you to choose, then writes your answer here.
+
 ### Google Cloud
 
 ```yaml
@@ -154,6 +195,25 @@ platform:
 
 `project` is optional. When it is missing or wrong, the deploy lists your projects and asks you to choose one, then writes your answer here.
 
+### Bringing your own Dockerfile
+
+Every cloud provider runs a job as a container: AWS on Fargate, Azure on Container Apps Jobs, Google Cloud on Cloud Run Jobs. The deploy builds an image for each app from a generated Dockerfile. It copies the app folder and `pdt.yml` into `/workspace`, installs the script-header dependencies of `run.py` with `uv sync --script`, and runs `run.py` as the entrypoint.
+
+Put a `Dockerfile` in the app folder to build the image your own way, for example to add system packages or to install a heavy tool at build time instead of on every run. The build context is the same as the generated one: the app folder under its own name, next to `pdt.yml`. Start from the generated file:
+
+```dockerfile
+FROM ghcr.io/astral-sh/uv:python3.12-bookworm-slim
+COPY . /workspace
+WORKDIR /workspace/my-app
+ENV PDT_PROJECT=/workspace
+RUN uv sync --script run.py
+ENTRYPOINT ["uv", "run", "--script", "run.py"]
+```
+
+A `.dockerignore` in the app folder keeps files out of the image. Write its patterns relative to the app folder (`.meltano`, `output`, `*.csv`); pdt moves them to the root of the build context for you. `.env` and the other secret files never reach the context, with or without a `.dockerignore`.
+
+The Windows provider never builds an image, so `pdt validate` reports a Dockerfile in an app that uses it.
+
 ### Windows Task Scheduler
 
 ```yaml
@@ -213,7 +273,7 @@ An app is a folder with a `run.py` that has a `main()` function. It declares its
 ```python
 # /// script
 # requires-python = ">=3.12"
-# dependencies = ["pdt-cli[apps]==0.1.0"]
+# dependencies = ["pdt-cli[apps]==0.1.2"]
 # ///
 from pathlib import Path
 
@@ -323,3 +383,15 @@ Leave the two SES key variables empty to use the standard AWS credential chain (
 Required environment variables:
 
 - PDT_RESEND_API_KEY
+
+## Logging
+
+`pdt.utils.log` writes one log line per event:
+
+```python
+from pdt.utils.log import die, log
+
+log("info", "processed", rows=42)
+```
+
+Levels are `debug`, `info`, `warning`, and `error`. Locally a line is human-readable text on stdout; with `LOG_FORMAT=json` it is one JSON object per line, which is also the default on Cloud Run.