arc-kernel 0.1.1__tar.gz → 0.1.2__tar.gz

{arc_kernel-0.1.1 → arc_kernel-0.1.2}/Install.md

@@ -43,7 +43,7 @@ Then: [verify](#verify-everything), [keep it running and put it on the internet]
 
 You do not need a compiler or any `-dev` libraries: every Python dependency installs from a prebuilt wheel.
 
-**How this guide was checked.** The steps below were run from an empty directory against a freshly initialised, private PostgreSQL 14.24 and Redis 6.0.16, on Ubuntu 22.04 with Python 3.12, Node 20.20.2 and Yarn 1.22.22 — ending in a real login and successful calls to the Admin API. In that run the repositories were cloned from local copies confirmed to be at the same commits as GitHub's `main`. Step 3 was then checked separately against the real PyPI package: `pip install arc-kernel` inside a Python 3.12 virtualenv, `uv tool install arc-kernel`, and the failure quoted there on Python 3.10; then `arc init` with the kernel cloned straight from GitHub, `arc install` of `pgdb` from GitHub, and `arc doctor` (the later steps were not repeated on that second project). Things that were **not** run are marked *(not run)*: the `arc deploy` commands, the server-wide `setup.sh`, Valkey, and every OS other than Ubuntu.
+**How this guide was checked.** The steps below were run from an empty directory against a freshly initialised, private PostgreSQL 14.24 and Redis 6.0.16, on Ubuntu 22.04 with Python 3.12, Node 20.20.2 and Yarn 1.22.22 — ending in a real login and successful calls to the Admin API. In that run the repositories were cloned from local copies confirmed to be at the same commits as GitHub's `main`. Step 3 was then checked separately against the published PyPI package, version 0.1.1: `pip install arc-kernel` inside a Python 3.12 virtualenv, `uv tool install arc-kernel`, and the failure quoted there on Python 3.10; then `arc init` with the kernel cloned straight from GitHub (it wrote `.vscode/settings.json` and `.gitignore` excluded the settings store), `arc install` of `pgdb` from GitHub, and `arc doctor` (the later steps were not repeated on that second project). Things that were **not** run are marked *(not run)*: `arc deploy prod` (system-wide units, nginx and certbot — its units were checked with `systemd-analyze verify` and unit tests only), the server-wide `setup.sh`, Valkey, and every OS other than Ubuntu. `arc deploy dev` was run against a scratch project and driven through crash tests under a real `systemd --user` manager, and `arc deploy dev --enable` was used to switch a running server from the old single `arc run` unit (about 9 seconds of downtime for the switch).
 
 ---
 
@@ -105,7 +105,7 @@ arc --help
 **The `pip` has to belong to Python 3.12 or newer** — the package declares `Requires-Python >=3.12`. Check with `pip --version`: it should end in `(python 3.12)` or later. On Ubuntu 22.04, whose default Python is 3.10, pip refuses with:
 
 ```
-ERROR: Ignored the following versions that require a different python version: 0.1.0 Requires-Python >=3.12
+ERROR: Ignored the following versions that require a different python version: 0.1.0 Requires-Python >=3.12; 0.1.1 Requires-Python >=3.12
 ERROR: Could not find a version that satisfies the requirement arc-kernel (from versions: none)
 ERROR: No matching distribution found for arc-kernel
 ```
@@ -124,7 +124,9 @@ uv tool install arc-kernel
 
 If your shell says `arc: command not found` after (b), run `uv tool update-shell` and open a new shell. Avoid `sudo pip install arc-kernel` into the system Python: recent distributions (Ubuntu 24.04, Debian 12) mark it "externally managed" and refuse global `pip install`s *(not run)*, which is what the virtualenv in (a) sidesteps.
 
-This global `arc` only bootstraps projects. `arc init` (step 5) clones the kernel into your project, and from then on the project's own copy is the `arc` you use. So the published package can lag behind the kernel on GitHub without harm, apart from one small gap that step 5 shows how to close.
+Use version **0.1.1 or newer** (`pip show arc-kernel`; upgrade with `pip install -U arc-kernel`). 0.1.0 is older than the kernel `arc init` clones, and its `arc init` writes a `.gitignore` that does not exclude `.arc/arc.store.db`, the encrypted settings store.
+
+This global `arc` only bootstraps projects. `arc init` (step 5) clones the kernel into your project, and from then on the project's own copy is the `arc` you use.
 
 > Setting up a shared server where several people will each create projects? `setup.sh` in this repository installs `uv`, Python 3.12 and `arc` system-wide for every user *(not run)*:
 > `sudo ARC_KERNEL_REPO=https://github.com/vignesh-ramsh/arc.git ./setup.sh`
@@ -178,8 +180,6 @@ Run this as your normal user, not with `sudo`. To avoid typing the URL every tim
 | `pyproject.toml`, `.venv/` | a [uv workspace](https://docs.astral.sh/uv/concepts/projects/workspaces/) tying the kernel and every plugin into one virtual environment |
 | `.gitignore` | already excludes the master key, the settings store, `.venv/`, `arc/` and `plugins/*/` |
 
-> **Look for `Wrote …/.vscode/settings.json` in the output.** If it is missing, the `arc` you installed from PyPI is older than the kernel it just cloned. An older `arc init` does not yet write that file (it enables the red-underline checks for table and field names in VS Code), and its `.gitignore` leaves out `.arc/arc.store.db`, the encrypted settings store, so `git add .` would commit it. With the project's virtualenv active (as above), run `arc init .` once more — it is safe to repeat: it keeps your master key and existing files and only adds what is missing.
-
 > **Back up `.arc/arc.mkey`.** It encrypts every secret ARC stores, including your database password. Lose it and those secrets cannot be recovered — ARC will report `Could not decrypt secret … wrong master key, or the store is corrupt`. `arc init` keeps it out of git on purpose; keep a copy somewhere safe.
 
 ## 6. Install the plugins
@@ -315,6 +315,7 @@ arc health
 
 - `arc doctor` prints the plugin load order and ends with **`Boot resolution OK.`** Two lines starting `warning: optional ordering skipped` are normal and expected — they describe a harmless dependency cycle, not a problem.
 - `arc pgdb status` and `arc redix status` should each print `OK` with a latency.
+- `arc status` (once you have deployed with `arc deploy`) shows every process that should be running and whether it is; it exits 1 if anything is off.
 - `arc health` lists each capability with `ok: yes`. For a deeper check that really opens connections and reports pool sizes and settings drift, use `arc doctor --deep`.
 
 From another terminal, a quick end-to-end check:
@@ -330,23 +331,92 @@ A `403` on `/whoami` before logging in is correct, not a fault.
 
 ## Keep it running and put it on the internet
 
-`arc run` stops when you close the terminal. These two commands turn it into a service *(not run — read from the commands' own `--help`)*.
+`arc run` stops when you close the terminal, and if one of its processes keeps crashing it stops everything. For a server people use, let systemd supervise **each process on its own**. Both commands below install one unit per process — the log writer, the gateway, the background-job workers (one set per queue), the scheduler — grouped under `arc-<project>.target`. systemd restarts exactly the process that died; a process that keeps crashing ends as one `failed` unit (5 starts in 120 s) while the rest keep running. In testing against a real `systemd --user` manager, killing a job worker, a web worker, the gateway or the log writer never took the site down for longer than the gateway's own restart (about 4.5 s when the gateway master itself dies), and the other units' restart counters stayed at 0. The user-level commands were run; the system-wide ones need root, so they were checked with systemd's own `systemd-analyze verify` and unit tests, but not run *(not run)*.
 
-**A per-user systemd service** — no root needed:
+**Development or a small server: a per-user service** — no root needed:
 
 ```bash
 arc deploy dev --enable
 ```
 
-This installs a `systemd --user` unit whose command is `arc run`, starts it, keeps it running across logouts, and sets `restart_command` so `arc restart` works.
+This runs **2 web workers, 1 worker per queue and 1 scheduler**, keeps them running across logouts and reboots, and sets `restart_command` so `arc restart` works. Without `--enable` it only writes the units and never changes what is running.
 
-**nginx and HTTPS in front of it** — needs root:
+**Production: system-wide units plus nginx and HTTPS** — needs root:
 
 ```bash
 sudo "$(command -v arc)" deploy prod --domain app.example.com
 ```
 
-This writes an nginx site for that domain that proxies to `arc run`, and obtains a Let's Encrypt certificate with certbot. Use `--no-ssl` for plain HTTP, and `--port` if you changed `arc run`'s port. It only touches its own nginx site file. Run it from inside your project with the virtualenv active: `sudo` does not inherit your `PATH`, so a bare `sudo arc` usually cannot find a user-installed `arc`, and `"$(command -v arc)"` hands `sudo` the full path to the project's copy.
+This writes an nginx site for that domain that proxies to the app and obtains a Let's Encrypt certificate with certbot (`--no-ssl` for plain HTTP; `--port` to change the app's internal port, default 8001). It then installs the same units under `/etc/systemd/system`, run as the project's owner (never root), started on boot, **sized from the machine**: as many web workers as the CPU and memory allow (never more than 8 automatically), and the same number of job workers in total, shared out over the queues with at least one each. The app starts last, so it comes up with the final settings. `arc restart` runs `sudo systemctl restart arc-<project>.target`, so the project's owner needs passwordless sudo for exactly that command. It only touches this project's own units and nginx site. Run it from inside your project with the virtualenv active: `sudo` does not inherit your `PATH`, and `"$(command -v arc)"` hands it the full path to the project's copy.
+
+**Choosing how many workers.** The defaults are only a starting point. Set what you want, then run the same `arc deploy` command again — the counts are read when it runs, and it removes the units of workers you no longer have:
+
+| Setting | Meaning |
+|---|---|
+| `gateway_workers` | web worker processes (any number; yours wins over the default in both modes) |
+| `lineup_workers_per_queue` | job workers for **every** queue |
+| `lineup_workers_<queue>` | job workers for one queue, e.g. `arc settings set lineup_workers_mail 3` |
+
+The scheduler is always exactly one: more would fire every cron job twice.
+
+**Database connections.** Every process holds its own connection pool, so a deployment opens *(processes × pool size)* connections, and PostgreSQL refuses the ones past its limit. You do not have to work that out: `arc deploy` (and `arc run`) ask PostgreSQL what this application may open, keep a few free for yourself, and split the rest over the processes. It reads the real limit — `max_connections` minus the superuser reserve, or the role's or database's own `CONNECTION LIMIT` if that is lower — never an assumed 100. For the dev sizing on a default server it prints:
+
+```
+Database connections: 97 usable (max_connections 100 − 3 reserved for superusers), 10 kept free
+  gateway-worker    2 × 10 = 20
+  lineup-worker     2 × 5  = 10
+  lineup-scheduler  1 × 2  = 2
+  32 of 87 available for the application
+```
+
+If the processes would not fit even at the smallest useful pool (2 per web or job worker, 1 for the scheduler), the command stops before it writes anything and says by how much; run fewer workers or raise `max_connections`. If the pools only fit by shrinking, they are all shrunk together, and what is left over goes back to the web workers first.
+
+| Setting | Meaning |
+|---|---|
+| `pgdb_connection_reserve` | Connections kept free for migrations, `arc console`, backups and your own `psql` (default 10) |
+| `pgdb_connection_budget` | Total connections ARC may use, when PostgreSQL's own numbers are not the real limit — behind PgBouncer, or on a managed database that hides them. Default `0` = ask the server. *(PgBouncer itself was not run; the setting replaces the number read from the server and nothing else.)* |
+| `pgdb_pool_max_size_<role>` | Fix one kind of process's pool yourself — `gateway_worker`, `lineup_worker` or `lineup_scheduler` (e.g. `pgdb_pool_max_size_lineup_worker`). Counted first; a deployment never changes it |
+| `lineup_concurrency` | How many jobs one worker process runs at once. Default `0` = as many as its pool has connections |
+
+A job worker runs no more jobs at once than it has connections, so a burst of jobs waits in the queue instead of piling up inside the workers. If PostgreSQL's limit is lowered after you deployed, every process notices when it starts and shrinks its pool to a share that fits (it logs a warning); run `arc deploy` again to make the new numbers official. If a request still finds every connection busy for `pgdb_pool_acquire_timeout_ms` (10 s), it gets **HTTP 503 with `Retry-After`**, not a 500 and not a traceback.
+
+Measured on one machine, PostgreSQL's default 100 connections, 8 web workers and 8 job workers: 600 queued jobs each holding a connection for 2 s, while 64 clients read records for 30 s. Before: connections peaked at 97, PostgreSQL turned away 83 attempts and **1,438 requests failed with HTTP 500**. After: the plan was 82 connections, the peak was 81, and **all 119,790 requests returned 200**; every job ran. With a deliberately tiny pool, the requests that could not get a connection all got 503 + `Retry-After: 2` (95,245 of them in two runs) and none a 500.
+
+**Adding a queue later** (a new plugin that declares one): run the `arc deploy` command again, or nothing consumes it and its jobs just wait. `arc doctor --deep` tells you if that happened: *"queue(s) mail have no worker unit, so jobs for them are never consumed."*
+
+**Checking it.** `arc status` is the one command that answers *how many web workers, which queues, how many schedulers are running*, and whether that is what was deployed:
+
+```bash
+arc status
+```
+
+```
+arc-desk — per-user systemd units, dev sizing
+┏━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
+┃ process               ┃ expected ┃ running ┃ restarts ┃ configured by    ┃
+│ gateway (web workers) │        2 │       2 │        - │ dev default      │
+│ queue default         │        1 │       1 │        - │ dev default      │
+│ queue mail            │        1 │       1 │        - │ dev default      │
+│ scheduler             │        1 │       1 │        - │ fixed (one only) │
+│ log writer            │        1 │       1 │        - │ fixed            │
+units: 6 of 6 active, 0 failed · restarts since they started: 0
+database connections: 6 open now · PostgreSQL lets this application open 97, 10 kept free
+  plan: 2 gateway-worker × 10 + 1 lineup-scheduler × 2 + 2 lineup-worker × 5 = 32 of the 87 available when it was made
+Everything matches.
+```
+
+*Expected* is read from the unit files `arc deploy` wrote; *running* comes from systemd and from the process list `arc ps` shows. It flags a failed unit (with the command to recover it), a count that does not match, a leftover `arc run` running workers a second time, and a setting you changed but have not applied yet (*"settings give 4 web worker(s) but the units run 2 — run `arc deploy dev` to apply"*). It also checks the connection plan against PostgreSQL as it is *now*: a plan made for other worker counts, or one larger than a limit that was lowered since (*"the plan uses 32 connections but PostgreSQL now allows 20 — run `arc deploy dev` to re-plan"*). It is read-only, does not boot the kernel (it opens one short database connection to read the limits), and exits 1 when anything is wrong, so it also works as a monitoring check; `--json` gives the same thing for scripts. `arc ps` alone lists the processes; `arc doctor --deep` additionally compares the queues your plugins declare with the queues that have workers.
+
+**Running it.** For a production install drop `--user` and use `sudo` where needed:
+
+```bash
+systemctl --user list-units 'arc-*'              # everything, and which are failed
+systemctl --user status arc-<project>-gateway    # one process
+journalctl --user -u arc-<project>-worker-mail-1 # its output
+systemctl --user reset-failed arc-<project>-worker-mail-1 && systemctl --user start arc-<project>-worker-mail-1
+```
+
+An older install that used a single `arc run` unit is switched over by running the new command with `--enable` (it stops and disables the old unit, and leaves its file in place).
 
 **Before going public**, review these settings (`arc settings set <key> <value>`):
 
@@ -356,6 +426,10 @@ This writes an nginx site for that domain that proxies to `arc run`, and obtains
 | `gateway_trusted_proxies` | The IP of your reverse proxy, so the real client IP is read from `X-Forwarded-For` rather than being nginx's address. Login rate limiting is keyed on the pair (client IP, account), so without this ARC sees nginx's address for everyone and that per-source separation is lost. |
 | `gateway_cors_origins` | Left empty, CORS is disabled — the safe default. Only list origins that genuinely need cross-origin access. |
 | `gateway_metrics_token` | Left empty, `/metrics` is open to anyone who can reach the port. Set a token if it is reachable. |
+| `gateway_access_log` | Where each request's access line goes. `file` (the default) writes it to `logs/gateway.jsonl` only; `console` also prints it through `arc run` (handy on a dev machine, but at thousands of requests per second that relay becomes the bottleneck); `off` disables the line. Request metrics are kept in every mode. Restart to apply. |
+| `log_max_bytes`, `log_backup_count` | Size of each `logs/<category>.jsonl` before it rotates (default 10 MB) and how many rotated copies are kept (default 5). The access log is one ~370-byte line per request, so at several thousand requests per second the defaults hold only tens of seconds of history. Raise both if you need longer. |
+| `log_collector` | `true` (default): `arc run` starts one writer for `logs/*.jsonl`, so no line is lost to rotation races between workers (8 processes at ~8,000 requests per second: every line kept). `false`: every process writes the files itself. If the collector goes away while the stack is running, each process keeps logging to files of its own, `logs/<category>.fallback-<pid>.jsonl`, and never to the shared ones. Restart to apply. |
+| `log_flush_ms` | How long (ms) each process accumulates log lines before sending them to the collector, and how long the collector accumulates before one write (default 10). Lower means lines reach disk sooner; higher means fewer writes. |
 | `arc_server_timezone` | Defaults to `UTC`; set an IANA name such as `Asia/Kolkata` if you want naive date-times interpreted in local time. |
 
 Keep `arc run` on `127.0.0.1` (the default) and let nginx be the only public listener. Back up `.arc/` (especially `arc.mkey`) and dump the database with `pg_dump`.
@@ -394,9 +468,18 @@ nginx is up but nothing is listening on the port it proxies to — `arc run` is
 **Port 8001 is already in use**
 `arc run --port 9000`. If you use `arc deploy prod`, pass the same `--port`.
 
+**`Connection budget: ... connections are needed at the very least`** from `arc deploy` or `arc run`
+The workers you asked for cannot fit in the connections PostgreSQL allows (the message says how many are needed and how many exist). Run fewer workers (`gateway_workers`, `lineup_workers_per_queue`), raise `max_connections` in `postgresql.conf` (then restart PostgreSQL), or lower `pgdb_connection_reserve`. If a pooler such as PgBouncer sits in front, set `pgdb_connection_budget` to what it can serve.
+
+**Requests fail with `503` and `Retry-After`, log says `no database connection available`**
+Every connection in that process's pool was busy for `pgdb_pool_acquire_timeout_ms`. It is capacity, not a bug, and clients can retry. Check `arc status` for the pool sizes and how many connections are open; the usual fixes are fewer concurrent jobs (`lineup_concurrency`), a slow query holding connections (`slow_query_threshold_ms` logs it), or more connections (see *Database connections* above).
+
 **`uv sync` removed all my plugins**
 A bare `uv sync` at the project root uninstalls every workspace member. Always use `uv sync --all-packages`. ARC's own commands already do.
 
+**`lineup scheduler: cannot reach Redis`**
+The scheduler keeps running and retries (every 1 s, backing off to every 10 s) until Redis answers, then takes the leader lock again; it does not exit, so systemd never marks it failed because of an outage. A second scheduler waits as a standby (`another scheduler is the leader — standing by`) and takes over when the first stops. Check `redis-cli ping`. If a scheduler that is leading can no longer confirm its lock (a renewal failed), it stops dispatching at once and says so (`no longer sure it is the leader`), then contends again — so two schedulers never fire the same cron job. Any other error that ends a scheduler is printed and exits with code 1.
+
 **`redix: FAILED to connect`**
 Check `redis-cli ping`, then re-run `arc redix setup`. If Redis has a password, enter it there.
 

{arc_kernel-0.1.1 → arc_kernel-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: arc-kernel
-Version: 0.1.1
+Version: 0.1.2
 Summary: ARC kernel — capability-based Python platform
 Project-URL: Homepage, https://github.com/vignesh-ramsh/arc
 Project-URL: Repository, https://github.com/vignesh-ramsh/arc

{arc_kernel-0.1.1 → arc_kernel-0.1.2}/arc/__init__.py

@@ -28,7 +28,7 @@ from .kernel import (  # noqa: F401
 from .resolver import BootPlan, PluginSpec, ResolutionError  # noqa: F401
 from .runtime import BootError, boot, find_project_root, shutdown  # noqa: F401
 
-__version__ = "0.1.1"
+__version__ = "0.1.2"
 
 
 def __getattr__(name: str) -> Any: