@gigzen/populace 0.1.0 → 1.0.0

package/README.md

@@ -3,6 +3,11 @@
 **A simulated population that uses your app through its real API — so you can
 test what needs more than one person.**
 
+```bash
+npm install -g @gigzen/populace
+populace demo
+```
+
 **Site:** https://shakhtar-sankur.github.io/populace/  · 
 **Test report:** [the full engineering record](https://shakhtar-sankur.github.io/gigzen/test-report.html)
  ·  A [Gigzen](https://shakhtar-sankur.github.io/gigzen/) product
@@ -22,13 +27,13 @@ applying**. Then it hands you a report on what broke.
 
 ## It has done this to a real, finished app
 
-![Populace report for Buzz Buzz, 9 August 2026 — no failures across 400 API calls](docs/buzzbuzz-run-2026-08-09.svg)
+![Populace report for Buzz, 21 August 2026 — no failures across 932,455 API calls from 200 simulated drivers across 20 cities in 11 countries](docs/buzz-run-2026-08-21.svg)
 
 That is the *second* run. The first one is the interesting one.
 
 ### What happened, in plain English
 
-On **9 August 2026** we pointed Populace at **Buzz Buzz** — a gig-worker platform
+On **9 August 2026** we pointed Populace at **Buzz** — a gig-worker platform
 on Android with a live Postgres backend, 17 tables and 48 row-level-security
 policies. It was finished. It was signed. It had been through a full manual test
 of every screen by the person who wrote it, and it had passed.
@@ -97,15 +102,20 @@ within seconds to six users signing up at once.
 
 ### After the fixes
 
-All five were fixed and the run repeated: the report at the top of this page.
-**No failures across 400 API calls.** Six accounts created, six accounts deleted,
-nothing left behind.
+All five were fixed, and the runs since have grown by three orders of magnitude.
+The report at the top of this page is the largest that was **clean end to end**:
+**no failures across 932,455 API calls**, 200 concurrent drivers over 20 cities in
+11 countries, 200 accounts created and 200 removed.
 
-**What this does not claim.** Six users for three minutes is a **correctness run,
-not a load test**, and it ran against that project while it was still empty. The
-latencies above are what six concurrent users saw and nothing more. Populace has
-been pointed at exactly one real backend so far, and that backend was ours — the
-next one should be someone else's.
+A later run put **300 drivers through 1,401,435 calls with zero API failures** — but
+one call never reached the server, a socket exhausted on the test machine, so
+Populace marked it *inconclusive* rather than clean. That distinction is the point:
+a verdict that is never withheld is worth nothing when it is given.
+
+**What this does not claim.** These runs are against a **local backend over
+loopback**, so no network is in the latencies — the same calls cost about 175 ms
+against a hosted project. Three hundred drivers is where throughput stops scaling,
+not where the app breaks; that is still unfound.
 
 ---
 
@@ -170,6 +180,213 @@ your API for what were really expired tokens — and the run cannot even delete
 its own accounts, stranding simulated users in your environment. See it happen:
 `node examples/token-expiry/expiry-demo.mjs`
 
+
+### And to an app we did not write
+
+Pointed at a local [Gitea](https://gitea.io) instance — a git forge, not a social
+app, and nobody here has touched its source:
+
+```
+POPULACE REPORT — Gitea
+✔ No failures across 1106 API calls.
+
+  recentPostsByOthers   382   0   364ms   879ms
+  like                  382   0   186ms   520ms
+  post                  173   0   575ms    1.2s
+  comment               139   0   405ms   771ms
+  createUser             10   0   116ms   183ms
+  setProfile             10   0    59ms    77ms
+  deleteUser             10   0   336ms   410ms
+
+  NOT TESTED — adapter implements 8/13
+✔ Cleanup complete — 10 accounts removed.
+```
+
+Gitea has no location tracking, no direct messages and no group joins, so five
+methods have no equivalent. They are **absent from the adapter rather than
+stubbed**, and the report says which and what each would have covered — because
+a stub returning a fake success turns "we did not test this" into "this works".
+Posting is opening an issue; the feed is listing other people's; liking is a
+reaction.
+
+Ten accounts created through Gitea's own signup, driven under Gitea's own
+permissions with per-user tokens, and ten removed — checked inside Gitea
+afterwards, not taken from this tool's own report.
+
+**It found five defects on the way in, all of them in Populace.** Gitea's OpenAPI
+description has 482 operations against the 18 in the fixture the adapter
+generator was built on, and at that scale it mismatched four methods whose
+correct endpoint was right there in the spec: it summed path-word hits instead of
+taking the best, penalised path depth too weakly, matched `"registration"` inside
+a CI-runner token endpoint, and — the good one — matched `"dm"` inside
+`"admin"`. All fixed; the confidence markers are what stopped the wrong guesses
+from looking finished.
+
+## It tells you what to do about it
+
+A report that says `duplicate key value violates unique constraint` has told you
+what happened and nothing about what to do. Every run now ends with the part
+that matters:
+
+```
+  WHAT TO DO
+    12 failures in the application.
+
+    [YOUR APP]  like × 12
+      The database refused the write on permissions, not on data.
+      A row-level-security policy rejected this call for the signed-in role.
+      The request was well formed; the policy did not allow it. This is
+      invisible to whoever wrote the policy, because their own account
+      usually satisfies it.
+      Fix:
+        Check the policy for this table against the role the app
+        authenticates as, and confirm there is a policy for this specific
+        command — a table with only a SELECT policy refuses every INSERT.
+```
+
+**The first judgement is the one that matters: was this your application at
+all?** A socket that never opened and a 500 look equally red in a terminal and
+mean completely different things. Every failure is attributed to your app, the
+platform, or the test client, and the summary line says so before any detail:
+
+```
+No application failures — 21 never reached the server.
+```
+
+A failure Populace cannot recognise is reported as unrecognised. It is never
+counted as "not your app", because that would turn *we could not tell* into an
+all-clear.
+
+### When no rule fits
+
+Rules cover the shapes a simulated population usually provokes. For anything
+else, `populace explain` can ask a model:
+
+```bash
+export ANTHROPIC_API_KEY=sk-ant-...
+populace explain
+```
+
+Rules run first and the model only ever sees what they could not name — so most
+runs never call it at all, and the ones that do send a handful of short strings.
+Explanations that came from the model are labelled as such.
+
+**What is sent, exactly:** the method name, the error text, how many times it
+happened, and that method's p50/p95. Nothing else — no target URL, no keys, no
+tokens, no request bodies, no simulated users' content. Run with `--verbose` to
+print it before it leaves.
+
+```
+  Sending to the model — this is everything, nothing else leaves this machine:
+    [ { "method": "post", "error": "quorum lost on shard 3 …",
+        "occurrences": 9, "p50ms": 40, "p95ms": 120 } ]
+```
+
+The model layer can never break a run. No key, a rejected key, a rate limit, a
+refusal, a malformed reply — each prints one line and falls back to the rule
+output. `populace explain` is a separate command for the same reason: a run
+should not wait on somebody else's network, and a report is complete without it.
+
+## Start from your API description
+
+Writing the adapter is the slow part: thirteen methods against your endpoints.
+If your API has an OpenAPI description, Populace can fill most of it in:
+
+```bash
+populace init --from-openapi openapi.json
+```
+
+It reads every operation, matches them against the thirteen contract methods by
+path, verb and description, and writes the adapter with each confidence marked:
+
+```
+  Read 18 operations from openapi.json and matched 13 of 13.
+
+    ✔ createUser            POST /auth/signup
+    ✔ like                  POST /posts/{postId}/likes
+    ? joinGroup             POST /groups/{id}/members
+        low confidence — check this one
+    ✖ reportLocation        left as the template default
+```
+
+Path parameters become real template literals — `` `/posts/${postId}/likes` ``,
+not the literal string — so the adapter runs rather than requesting a URL with
+braces in it.
+
+**This is a guess and the generated file says so.** Request bodies, field names
+and response shapes are still the template's defaults. It removes the half hour
+of looking endpoints up; it does not finish the job. Run `populace smoke` next —
+it calls every method once and names the first one that is wrong.
+
+A verb match alone is never enough to claim a method: a spec containing only
+`/health` matches nothing, rather than guessing that a lone `GET` is your feed.
+
+JSON only. A YAML parser would be Populace's first runtime dependency, and
+`npx js-yaml openapi.yaml > openapi.json` covers it.
+
+## Run it in CI
+
+Populace is a GitHub Action, so a population can run against your staging
+environment on every pull request:
+
+```yaml
+- uses: Shakhtar-Sankur/populace@v1
+  with:
+    agents: 30
+    minutes: 10
+  env:
+    MY_API_URL: ${{ secrets.STAGING_URL }}
+    MY_API_KEY: ${{ secrets.STAGING_KEY }}
+```
+
+Your credentials stay in your repository's secrets and are read by your own
+adapter. **Populace never sees them and nothing is sent anywhere** — the run
+happens on your runner, against your server.
+
+The job fails when the run is not clean, and the per-method table is written to
+the job summary, so the result is visible in the pull request without
+downloading an artifact.
+
+| input | default | |
+|---|---|---|
+| `config` | `populace.config.mjs` | path to your config |
+| `agents` `minutes` `tick` `engagement` `cities` | from the config | overrides |
+| `report` | `populace-report.json` | the HTML lands beside it |
+| `working-directory` | `.` | |
+| `fail-on-problems` | `true` | set `false` to record without breaking the build |
+
+Outputs `verdict`, `calls`, `api-failures`, `transport-failures`,
+`report-json` and `report-html` for later steps:
+
+```yaml
+- uses: actions/upload-artifact@v4
+  if: always()
+  with:
+    name: populace-report
+    path: ${{ steps.populace.outputs.report-html }}
+```
+
+**`api-failures` is the number about your app.** `transport-failures` counts
+calls that never reached your API at all — a flaky runner, not your code — and
+Populace reports a run with any of those as *inconclusive* rather than clean,
+because it cannot vouch for a call it could not make.
+
+## Staying current
+
+```bash
+populace update
+```
+
+Checks the npm registry and tells you if there is a newer release. A run mentions
+it at most once a day, **after** the report rather than before it, and says
+nothing at all if the registry cannot be reached — a version check is never worth
+an error message at the end of a good run.
+
+It is a plain anonymous GET to the public registry: no identifiers, no telemetry,
+nothing about your app or your runs. Off entirely with `POPULACE_NO_UPDATE_CHECK=1`,
+and off by default under `CI`, because a build server should not make an outbound
+call nobody asked for.
+
 ## The safety guard
 
 Populace creates real accounts and writes real rows. Pointed at production it

package/action.yml

@@ -0,0 +1,145 @@
+name: Populace
+description: Bring a population of simulated users to life against your app's own API, and report what broke.
+author: Gigzen
+
+branding:
+  icon: users
+  color: green
+
+# Why a composite action rather than a Docker or JavaScript one.
+#
+# Docker would add a build and a pull to every job, for a tool whose entire
+# claim is that it has nothing to install. A JavaScript action would need
+# `@actions/core` bundled in, which is a runtime dependency in a package that
+# advertises zero. Composite keeps the promise: this runs the same
+# `node src/cli.mjs` a person runs locally, and nothing else.
+
+inputs:
+  config:
+    description: Path to populace.config.mjs, relative to working-directory.
+    required: false
+    default: populace.config.mjs
+  agents:
+    description: How many simulated users. Overrides the config.
+    required: false
+  minutes:
+    description: How long the population runs for. Overrides the config.
+    required: false
+  tick:
+    description: Simulated seconds per step. Lower means each person decides more often.
+    required: false
+  engagement:
+    description: How busy people are. 1 is normal, 5 is relentless.
+    required: false
+  cities:
+    description: Comma-separated city keys. Overrides the config.
+    required: false
+  report:
+    description: Where to write the JSON report. The HTML lands beside it.
+    required: false
+    default: populace-report.json
+  working-directory:
+    description: Directory to run in. Defaults to the repository root.
+    required: false
+    default: "."
+  fail-on-problems:
+    description: >-
+      Fail the job when the run is not clean. Set to false to record the result
+      without breaking the build - useful while you are still wiring an adapter up.
+    required: false
+    default: "true"
+  summary:
+    description: Write the per-method table to the job summary page.
+    required: false
+    default: "true"
+
+outputs:
+  verdict:
+    description: clean, problems-found, or inconclusive.
+    value: ${{ steps.report.outputs.verdict }}
+  calls:
+    description: Total API calls made.
+    value: ${{ steps.report.outputs.calls }}
+  api-failures:
+    description: Calls your API answered with an error. This is the number about your app.
+    value: ${{ steps.report.outputs.api-failures }}
+  transport-failures:
+    description: Calls that never reached your API. These say nothing about your app.
+    value: ${{ steps.report.outputs.transport-failures }}
+  report-json:
+    description: Path to the JSON report.
+    value: ${{ steps.report.outputs.report-json }}
+  report-html:
+    description: Path to the self-contained HTML report.
+    value: ${{ steps.report.outputs.report-html }}
+
+runs:
+  using: composite
+  steps:
+    - name: Run the population
+      id: run
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      env:
+        POPULACE_CONFIG: ${{ inputs.config }}
+        POPULACE_AGENTS: ${{ inputs.agents }}
+        POPULACE_MINUTES: ${{ inputs.minutes }}
+        POPULACE_TICK: ${{ inputs.tick }}
+        POPULACE_ENGAGEMENT: ${{ inputs.engagement }}
+        POPULACE_CITIES: ${{ inputs.cities }}
+        POPULACE_REPORT: ${{ inputs.report }}
+      run: |
+        # Inputs arrive through the environment rather than being interpolated
+        # into this script. A city list or a path is attacker-controllable in a
+        # fork PR, and ${{ }} pasted straight into bash is a shell injection.
+        args=(run --config "$POPULACE_CONFIG" --report "$POPULACE_REPORT")
+        [ -n "$POPULACE_AGENTS" ]     && args+=(--agents "$POPULACE_AGENTS")
+        [ -n "$POPULACE_MINUTES" ]    && args+=(--minutes "$POPULACE_MINUTES")
+        [ -n "$POPULACE_TICK" ]       && args+=(--tick "$POPULACE_TICK")
+        [ -n "$POPULACE_ENGAGEMENT" ] && args+=(--engagement "$POPULACE_ENGAGEMENT")
+        [ -n "$POPULACE_CITIES" ]     && args+=(--cities "$POPULACE_CITIES")
+
+        set +e
+        node "$GITHUB_ACTION_PATH/src/cli.mjs" "${args[@]}"
+        code=$?
+        set -e
+        echo "exit-code=$code" >> "$GITHUB_OUTPUT"
+        echo "Populace exited $code"
+
+    - name: Read the report
+      id: report
+      if: always()
+      shell: bash
+      working-directory: ${{ inputs.working-directory }}
+      env:
+        POPULACE_REPORT: ${{ inputs.report }}
+        POPULACE_SUMMARY: ${{ inputs.summary }}
+      run: node "$GITHUB_ACTION_PATH/src/github-summary.mjs"
+
+    - name: Decide the job
+      if: always()
+      shell: bash
+      env:
+        EXIT_CODE: ${{ steps.run.outputs.exit-code }}
+        VERDICT: ${{ steps.report.outputs.verdict }}
+        API_FAILURES: ${{ steps.report.outputs.api-failures }}
+        FAIL_ON_PROBLEMS: ${{ inputs.fail-on-problems }}
+      run: |
+        if [ "$EXIT_CODE" = "0" ]; then
+          echo "Clean run."
+          exit 0
+        fi
+        if [ "$FAIL_ON_PROBLEMS" != "true" ]; then
+          echo "::warning::Populace reported '$VERDICT' ($API_FAILURES API failures), but fail-on-problems is false."
+          exit 0
+        fi
+        # An inconclusive run is called out separately: it means Populace could
+        # not make every call, which is usually the runner's network rather than
+        # the app under test. Saying so here saves someone hunting a bug that is
+        # not in their code.
+        if [ "$VERDICT" = "inconclusive" ]; then
+          echo "::error::Populace could not complete the run: $API_FAILURES API failures, but some calls never reached your API. This is usually the network between the runner and your server, not your app. See the job summary."
+        else
+          echo "::error::Populace found problems: $API_FAILURES API failures. See the job summary."
+        fi
+        exit 1

package/adapters/buzzbuzz.mjs

@@ -1,4 +1,4 @@
-// Adapter: Buzz Buzz — a gig-driver tracking app on Supabase.
+// Adapter: Buzz — a gig-driver tracking app on Supabase.
 //
 // Populace's first customer, and the reference implementation. Read this
 // alongside contract.md to see what a complete adapter looks like: it is ~140

package/adapters/template-rest.mjs

@@ -19,6 +19,22 @@
  */
 
 export function createAdapter(target, config) {
+  /**
+   * Fail with a sentence, not a TypeError.
+   *
+   * A fresh scaffold has `url: process.env.MY_APP_TEST_URL`, which is
+   * undefined until you set it — so the very first `populace doctor` a new
+   * user runs went straight into `target.url.replace(...)` and printed
+   * "Cannot read properties of undefined (reading 'replace')". Technically a
+   * correct refusal, and a terrible first five minutes.
+   */
+  if (!target?.url) {
+    throw new Error(
+      "No target URL. Set `target.url` in populace.config.mjs, or export the " +
+      "environment variable it reads. It must point at your TEST environment.",
+    );
+  }
+
   const base = target.url.replace(/\/$/, "");
 
   /**