mercury-agent 0.20.0 → 0.22.0

package/container/Dockerfile

@@ -111,6 +111,7 @@ COPY src/cli/mrctl.ts /app/src/cli/mrctl.ts
 COPY src/cli/mrctl-http.ts /app/src/cli/mrctl-http.ts
 COPY src/extensions/reserved.ts /app/src/extensions/reserved.ts
 COPY src/extensions/permission-guard.ts /app/src/extensions/permission-guard.ts
+COPY src/extensions/bash-timeout.ts /app/src/extensions/bash-timeout.ts
 COPY src/text/sanitize-text.ts /app/src/text/sanitize-text.ts
 COPY src/types.ts /app/src/types.ts
 COPY resources/ /app/resources/

package/container/Dockerfile.base

@@ -80,6 +80,7 @@ COPY src/cli/mrctl.ts /app/src/cli/mrctl.ts
 COPY src/cli/mrctl-http.ts /app/src/cli/mrctl-http.ts
 COPY src/extensions/reserved.ts /app/src/extensions/reserved.ts
 COPY src/extensions/permission-guard.ts /app/src/extensions/permission-guard.ts
+COPY src/extensions/bash-timeout.ts /app/src/extensions/bash-timeout.ts
 COPY src/text/sanitize-text.ts /app/src/text/sanitize-text.ts
 COPY src/types.ts /app/src/types.ts
 COPY resources/ /app/resources/

package/docs/behavior-layers.md

@@ -145,8 +145,10 @@ prompt", and it runs as the caller who created it.
 ## 4. The history window
 
 - `context.mode` is a per-space key. `main` and DM auto-spaces are seeded
-  `context`; **any other space defaults to `clear`** (no history at all — only
-  the reply chain when someone swipe-replies). Set it explicitly.
+  with the host default (`contextMode` / `MERCURY_CONTEXT_MODE`, `context`
+  unless set); **any space without a row runs on that same host default**.
+  `clear` means no history at all — only the reply chain when someone
+  swipe-replies. Set the key explicitly on a space that should differ.
 - `context.window_size` (default 10 — `config.ts contextWindowSize`) counts
   **user turns**: `getRecentTurns` takes the newest `turnCount*5` non-ambient
   rows after the compact boundary and cuts at the Nth user row. Assistant rows

package/docs/configuration.md

@@ -199,6 +199,21 @@ It compares path *segments*, never string prefixes, so `mercury-shadowy` is
 not inside `mercury-shadow`. A live process (`MERCURY_SHADOW` unset) is never
 checked and runs exactly the code it ran before.
 
+**A shadow never fetches an image.** `ensureImage` does not pull a missing
+base image and does not re-pull a floating tag whose host version moved (the
+release-gate R0.4 refresh), and `ensureDerivedImage` does not build: a cache
+miss is fatal at boot, naming the tag. A rehearsal must run the image the live
+bot is running, and a shadow that fell back to the base image would be
+rehearsing a bot with none of the owner's extension CLIs.
+
+**Every container carries `mercury.instance=<id>`**, derived from the resolved
+data dir and settable by nothing, and the orphan sweep filters on it — no id,
+no sweep. Two Mercury processes on one Docker daemon is the rehearsal shape,
+and without this each one's boot sweep would force-remove the other's running
+container. `/health` reports the same id in its `instance` field, always, so an
+external probe can tell which process answered. `MERCURY_AGENT_ID` still
+narrows further; it never replaces this.
+
 Snapshot directories are built by `mercury shadow snapshot`, which generates
 the shadow's `.env` and `mercury.yaml` — nothing here needs to be set by
 hand. See [live-testing.md](live-testing.md) §2 and §7.
@@ -287,6 +302,48 @@ Container env passthrough: explicit `all` override — these vars reach every sp
 
 For secrets that only host-side hooks and jobs need, prefer `mercury.env({ from: "…", hostOnly: true })`, which keeps them out of containers in either mode. For credentials the agent should never hold at all, use a host-side capability handler (`mercury.capability()`), which runs the privileged call on the host and returns only the result.
 
+## Container shared memory (`agent.container_shm_size`)
+
+How big `/dev/shm` is inside every agent container, in Docker's `--shm-size` spelling:
+
+```yaml
+agent:
+  container_shm_size: 512m   # 512m (default) | 1g | any Docker size
+```
+
+Env: `MERCURY_CONTAINER_SHM_SIZE`. Applies to both runtimes — `/dev/shm` is Docker's, not gVisor's.
+
+Docker's own default is **64 MB**, and that is the wrong size for what these containers actually do. Chromium is the workload: the `web-browser` extension drives it, the `poster` extension renders through it, and an agent asked to screenshot a page runs it directly. A large frame (a 1856x2304 poster, say) exhausts 64 MB, and Chromium's response is not an error — it **hangs**. The turn then sits idle until the host kills the container at `container_timeout_ms`, and the user gets no reply, only the time-limit marker. Three customer-facing runs were lost that way on 2026-09-10 before the cause was found.
+
+The default is deliberately generous rather than minimal. A tmpfs is allocated as it is used, so a container that renders nothing pays nothing for the headroom, and under `container_runtime: runsc` the total is bounded by the container's `--memory` limit anyway. Lower it only on a host that is genuinely short of RAM, and expect large renders to hang again if you go back to 64 MB.
+
+Two related things worth knowing:
+
+- Passing `--disable-dev-shm-usage` to Chromium sidesteps `/dev/shm` entirely and works at any size. The `poster` extension already does this, which is why it kept working while direct Chromium calls hung.
+- `mercury preflight`'s round-trip probe spawns with the same value, so the shape it vouches for is the shape a real turn gets.
+
+## Bash call timeout (`agent.container_bash_timeout_seconds`)
+
+The longest a single `bash` tool call may run inside an agent container, in seconds:
+
+```yaml
+agent:
+  container_bash_timeout_seconds: 300   # 300 (default)
+```
+
+Env: `MERCURY_CONTAINER_BASH_TIMEOUT_SECONDS`. It is both a default and a ceiling: a call that names no `timeout` gets this one, and a call that asks for more is clamped down to it. The agent can only ask for *less*.
+
+pi's `bash` tool takes an optional `timeout` and arms no timer at all when the model omits it — its own parameter description says "optional, no default timeout". A command that hangs rather than failing therefore blocks the turn with nothing watching it, until the host kills the whole container at `container_timeout_ms` and the user gets no reply, only the time-limit marker. That is the other half of the 2026-09-10 incident described under `container_shm_size`: the hanging Chromium was the cause, and the missing bound is what turned each hang into a lost 20-minute run instead of a recoverable error.
+
+When the bound fires, pi kills the command's whole process tree and returns `Command timed out after N seconds` as a normal tool error, so the turn can retry differently, fall back, or explain — while it still has budget left to reply.
+
+Two things to keep in mind when changing it:
+
+- **Keep it comfortably below `container_timeout_ms`.** A bound at or above the container limit buys nothing, because the SIGKILL arrives first. The defaults (300 s against 5 minutes) leave that ratio to whoever tunes both.
+- **Long jobs need a different shape, not a bigger number.** Raising this to cover a slow build makes every hang that much more expensive. Prefer backgrounding the work and polling a file for its result.
+
+The value in force is stated to the model in its system prompt, so it can plan around the ceiling instead of meeting it as a dead tool call.
+
 ## Agent run traces (`agent.trace_runs`)
 
 Captures pi's raw output from inside each agent container, with arrival timestamps, and copies it to `<dataDir>/traces/` before the container is reaped:

package/docs/container-lifecycle.md

@@ -100,6 +100,12 @@ When a container exceeds the timeout:
 4. A marker assistant message ("[System: this run was killed at the container time limit before replying. …]") is recorded in chat history, so the next run's agent knows the previous run was cut off and can check the workspace for partial work
 5. Queue unblocks, next message can proceed
 
+A container that hits the timeout was not necessarily busy. The commonest way to burn the whole budget is a child process that hangs rather than fails, and the commonest such child is Chromium on a too-small `/dev/shm`. Every spawn therefore sizes it explicitly — see `container_shm_size` in [configuration.md](configuration.md#container-shared-memory-agentcontainer_shm_size).
+
+| Config | Env Var | Default | Applies to |
+|--------|---------|---------|------------|
+| `containerShmSize` | `MERCURY_CONTAINER_SHM_SIZE` | `512m` | both runtimes |
+
 The host always injects a resolved **model chain** into the container (after `MERCURY_*` passthrough) so retries and fallbacks use the same policy Mercury loaded at startup:
 
 | In-container env | Source (host) | Purpose |

package/docs/deployment.md

@@ -135,6 +135,27 @@ first:
 Mercury's own CLI does not load the WhatsApp stack at startup, so running
 `mercury upgrade` never locks the tree it is replacing.
 
+### Disk: prune the build cache after every upgrade
+
+Each upgrade pulls a new base image (~2 GB of layers) and Mercury rebuilds the
+derived `mercury-agent-ext` image on the next start. Removing the old base with
+`docker rmi` / `docker image prune` does **not** free its layers: the derived
+build leaves BuildKit cache records that keep the old base layers alive, so
+overlay2 grows by roughly 2 GB per release and `docker system df` under-reports
+it. A 38 GB box hit 93% after five releases (2026-09-08).
+
+Once `/health` reports the new version and the journal shows `Built derived
+agent image`, run:
+
+```bash
+docker builder prune -af
+```
+
+Safe with the bot running: it touches no image, container or data. The only
+cost is that the next derived-image build starts cold, which it does after a
+base change anyway. Never `docker image prune -a` on an idle box — with no
+container running it removes the derived image and the base too.
+
 ## Auto-Restart Behavior
 
 Both systemd and launchd are configured to automatically restart Mercury if it crashes:

package/docs/extensions.md

@@ -279,6 +279,39 @@ mercury.widget({
 
 Widgets render HTML fragments in the dashboard overview. Errors show a placeholder — never crash the dashboard.
 
+### `mercury.capability(name, handler)`
+
+Register a host-side handler the agent reaches from inside the container with
+`mrctl capability <name> <action> '<json>'` → `POST /api/capability/:name/:action`.
+Credentials the handler uses stay on the host and never enter the container.
+
+```typescript
+// in extensions/rooms/index.ts
+mercury.permission({ defaultRoles: ["admin"] });
+mercury.capability("rooms", async (req, ctx) => {
+  if (req.action === "book") return { data: await bookRoom(req.callerId, req.body) };
+  return { status: 400, data: { error: "unknown action" } };
+});
+```
+
+- **`name` must equal the extension's own name** — registering under any other
+  name throws at load. The broker authorizes a call with `checkPerm(<name>)`,
+  so the equality is what makes that the *extension's* permission: the same
+  predicate the env-var injection rule and the sensitive-connection guard use.
+  A capability named after some other extension's permission would be reachable
+  by callers those two legs believe cannot reach it.
+- **The extension must also call `mercury.permission()`.** Without a *registered*
+  permission of that name, `checkPerm` refuses every role — including `admin`,
+  whose grant is the enumerated set of built-in plus registered permissions, not
+  a wildcard — so the capability is unreachable, silently.
+- An extension named after a **built-in** permission (`prompt`, `stop`, `clear`,
+  …) cannot register a capability at all: it could not own that permission
+  (`mercury.permission()` refuses a built-in name) while the broker would still
+  authorize the handler with it, and `prompt` is one every member holds.
+- One capability per extension; branch on `req.action` for sub-verbs.
+- The name is resolved host-wide, so two extensions cannot own it — a
+  collision is rejected at load time.
+
 ### `mercury.store`
 
 Scoped key-value store for persistent state.

package/docs/goals/feed-watch-social-sources/blocked-sources-research.md

@@ -0,0 +1,289 @@
+# Blocked sources: what a personal poller can still read (research, 2026-09-06)
+
+**Goal**: [feed-watch-social-sources](goal.md)
+**Status**: research note — feeds the ladder in `goal.md` and the rung-3 goal; decides nothing by itself
+**Asked**: the owner, 2026-09-06 — "search for methods to bypass the restrictions and use Facebook / Instagram / X / Yad2 / Madlan or any other sources that are blocking … specifically Facebook is a very important source and I think the email notifications won't be good enough."
+
+The question is answered per platform: which read paths exist, what each
+costs in money, latency, completeness, account risk and terms, and which
+one this goal should build. The line drawn throughout: paths that use the
+owner's **own account on their own data**, an **official alert channel**, a
+**licensed data provider**, or an **open mirror** are candidates. Defeating
+a bot-protection challenge on a site whose terms forbid automated access
+(Yad2, Madlan) is recorded as "what exists", not planned — see the Yad2
+section for why the sanctioned channels are the better deal anyway.
+
+## Probes run from the WSL box (residential IP, one GET each, 2026-09-06 ~19:00 IDT)
+
+Plain `curl`, Chrome desktop UA unless noted. `mercury-feed-watch/1.0` is the
+UA `feeds.ts` sends today.
+
+| Endpoint | Result | What it means |
+|----------|--------|---------------|
+| `bsky.app/profile/theathletic.com/rss` | 200, `application/xml`, items with `<pubDate>` (newest 2026-09-06 06:49 UTC), post text in `<description>` | **Bluesky is rung 0, not rung 2**: per-account RSS with no auth, readable by today's `rss` source type with zero code |
+| `public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=theathletic.com` | 200 JSON, `indexedAt` today | The richer API path also works unauthenticated; `searchPosts` returns 403 (auth) |
+| `public.api.bsky.app/xrpc/app.bsky.actor.searchActors?q=…` | 200; finds `skysportspl.bsky.social` (1 post — parked), several English football journalists, a handful of Hebrew-language football accounts (small, personal) | Who is *on* Bluesky decides its value. Romano is not (`fabrizioromano.bsky.social` is a 2023 squat). Sky Sports PL is parked. The Athletic is live |
+| `xcancel.com/FabrizioRomano/rss` (Chrome UA) | 400 "This URL only works inside an RSS client" | The instance gates RSS on the User-Agent |
+| `xcancel.com/FabrizioRomano/rss` (UA `mercury-feed-watch/1.0`) | 200 `application/rss+xml`, one placeholder item: "RSS reader not yet whitelisted! … send an email to rss [AT] xcancel [DOT] com with the reason … and this ID: …" | Whitelisted by the *instance operator*, not by X: the instance is under X Corp's cease-and-desist of 2026-08-24 and its terms position is the one in row X6 below. Recorded as what exists, not as a path to build |
+| `nitter.net/…/rss` | 410 Gone | Dead |
+| `nitter.privacydev.net` | connection refused | Dead |
+| `syndication.twitter.com/srv/timeline-profile/screen-name/…` | 429 "Rate limit exceeded" on the first request | Not usable from this IP without more work; not pursued |
+| `cdn.syndication.twimg.com/timeline/profile?screen_name=…` | 200, empty body | Deprecated |
+| `threads.net/@fabrizioromano` → `threads.com` | 200, 268 KB, no post text, no `taken_at`, no post codes in the HTML | Client-rendered behind a login prompt; not readable by fetch |
+| `instagram.com/fabrizioromano/` | 200, 618 KB, no captions, no `shortcode`, no timeline data | Same |
+| `i.instagram.com/api/v1/users/web_profile_info/?username=…` with `x-ig-app-id` | 401 `require_login: true` | The 2023-era unauthenticated endpoint is closed |
+| `facebook.com/groups/telavivapartments/` (public group) | 400, 1.5 KB "Error" page | Meta answers non-browser clients with an error, not a login page |
+| `m.facebook.com/groups/…` | 400 via redirect to `www` | The mobile site no longer exists as a separate surface |
+| `mbasic.facebook.com/groups/…` | 200 → `mbasic.facebook.com/login.php?next=…` (Hebrew login page, 6.5 KB) | **mbasic still serves** a login page in 2026; whether a logged-in session gets a group feed there is unverified (probe 9 in the open list) |
+| `t.me/s/nester_rent_telaviv` | 200, 4 message blocks, last `<time datetime>` 2026-09-06 16:01 UTC | Live; the M1 fixture candidate posted three hours before the probe |
+| `homeless.co.il/rent/`, `rss.app`, `fetchrss.com` | 200 | Reachable |
+
+Host tooling on the box: no Chromium, no node on the WSL host; `docker`
+runs without sudo; the derived image `mercury-agent-ext` carries
+Playwright + Chromium (`container/Dockerfile` line 51) and pinchtab. A
+headless page load can therefore run in a throwaway container from the
+home IP without installing anything on the host.
+
+Not probed: Yad2 and Madlan (the 2026-09-06 survey in `goal.md` already
+holds their answers for plain fetches, and a real-browser load was not
+run — see the Yad2 section).
+
+
+
+## Facebook groups
+
+The owner's premise holds: the notification-email bridge (rung 1c) is
+*compliant* but not *complete*. Notifications are algorithmically
+selected and bundled into digests, so a listing posted at 14:03 in a
+53 K-member group may never produce a mail. For the apartment adopter,
+where a good listing lasts ten minutes to an hour, that is the wrong
+tool. What is left is a choice between three complete-feed paths, none of
+them sanctioned by Meta.
+
+**Terms, stated once.** Meta's Terms §3.2.3 (effective 2025-01-01) forbid
+automated access or collection "even if that automated access or
+collection happens while you're logged into your Facebook account". The
+2024 Bright Data ruling (Meta lost on summary judgment, waived appeal)
+covers logged-out public scraping only and gives a logged-in reader
+nothing. No case was found 2024–2026 of Meta pursuing an individual for
+low-volume read-only automation; every action targets commercial
+scraping-for-hire. The realistic exposure is the **account**: temporary
+blocks and ID checkpoints, decided by behaviour and browser fingerprint,
+not by who the account is. A checkpoint on the owner's personal account
+locks them out of the same groups they are trying to read — which is why
+every path below is stated with *which account* it burns.
+
+### Paths, ranked for "private groups I am a member of"
+
+| # | Path | How | Complete? | Latency | Cost | Account at risk | Maintenance |
+|---|------|-----|-----------|---------|------|-----------------|-------------|
+| F1 | **Own session in Playwright, reading Facebook's feed JSON** | A persistent Chromium profile logged in as the reader account opens `facebook.com/groups/<id>/?sorting_setting=CHRONOLOGICAL` (desktop only; the parameter still works, the UI button is hidden); a response listener captures the JSON of Facebook's own `GroupsCometFeedRegularStoriesPaginationQuery` instead of parsing the DOM. Precedents: `viseshrp/fbn` (persistent profile, floor 15 min, default 1–3 h randomised) and the `browser-act` skill `facebook-groups-scrape-posts` (updated Aug 2026: 2–5 s between groups, never parallel). The `doc_id` changes on every Meta deploy, which is why the page's own request is *observed*, never constructed | yes, the full chronological feed | poll interval (15–30 min floor) | none | the reader account | medium: Meta frontend churn; the intercept absorbs most of it |
+| F2 | **Groups Watcher Chrome extension → local webhook** | Free extension (v5.13, 2026-06-17, ~1,000 users) in the owner's *everyday* Chrome on the *existing* session; polls every 3–7 min; with keywords disabled it POSTs every new post in batches to a URL: `{group_id, poster_name, post_url, post_id, body, images, timestamp}`. Needs the extension page open in a running Chrome — the same desktop Mercury already runs on | yes | 3–7 min | none | the owner's personal account, in a real browser fingerprint (lower risk than F1) | low, but the feed dies whenever Chrome does, and a 1,000-user extension holds the session and an outbound hook |
+| F3 | **Third-party Apify actor with exported cookies** (`whoareyouanas/facebook-group-scraper`) | Takes `c_user`+`xs`, `onlyPostsNewerThan`, chronological sort; ~20–40 posts/min; `maxConcurrency ≤ 2` | yes | per run | ~$3–10 per 1,000 posts (the page contradicts itself) | **higher**: the session replays from a foreign datacenter ASN, a classic checkpoint trigger | none |
+| F4 | **Israeli aggregators that already ingest the Hebrew groups** (apartments only) | Scoutr (`scoutrbot.com/city/tel-aviv`, no sign-up, plain HTML — 12,217 rentals, 2,596 attributed to Facebook, +191 in 24 h at research time); Dorin (₪19.90/week) and Agent Jeremy (₪29.99/week) Telegram bots claiming hundreds of groups | their coverage, unverifiable; Scoutr's sampled "Facebook" links pointed at Marketplace items, not group posts — check before relying on it | minutes to hours | free to ₪30/week | none | none; it is their scraper |
+| F5 | **Notification mirroring** | A spare Android phone or emulator with the Facebook app and group notifications on, plus a notification-listener app that POSTs to a local webhook (`BigShoots/NotificationWebhookApp`, `ItsAzni/NotificationForwarder`) | **no** — same algorithmic filter as email, just faster | seconds | ~none | none; nothing automates Facebook | low |
+| 1c | Email "All posts" → IMAP (already on the ladder) | — | **no** | minutes to hours | none | none | low |
+
+Not viable for private groups: rss.app, FetchRSS, Feedspot (public only by
+construction); Zapier, Make, Zoho, Buffer, Hootsuite (Groups apps removed
+2024-04-22 with the API); Apify's *official* `apify/facebook-groups-scraper`
+(refuses cookies and private groups by policy); Bright Data, ZenRows,
+ScrapingBee, Data365, Scrapfly (logged-out content only); Meta Content
+Library (academic/NGO enclave, $371/month + setup); "Download your
+information" (the owner's own content, no API).
+
+### Public groups and pages (football)
+
+`apify/facebook-groups-scraper` at $2.60–5.00 per 1,000 posts with
+`CHRONOLOGICAL` and `onlyPostsNewerThan`; Bright Data's groups scraper
+with 5,000 records/month free; rss.app / FetchRSS produce a feed URL for
+some public pages and groups ($0–15/month; FetchRSS deletes a free feed
+after 7 unread days). Logged-out browser scraping hits a login wall after
+roughly 1,000 posts per session, so for public content the managed actors
+beat engineering.
+
+### Corrections to what the goal recorded from the Gemini conversation
+
+- `rebrowser-playwright` is abandoned (last commit 2024-09). The 2026
+  maintained options are **Patchright** (2026-08-05) and **Camoufox**
+  (2026-08-12), which patch at the CDP layer; `nodriver` benchmarked best.
+- The *official* Apify actor does not take cookies; private-group support
+  is third-party only. The free credit is $5/month and does not roll over.
+- "All posts notifications are filtered" is asserted mainly by Groups
+  Watcher, which sells the fix. Probable, unconfirmed by any Meta source.
+- `mbasic.facebook.com` still serves (a login page, probe above); whether a
+  logged-in session gets a light group feed there is unverified.
+- No Tel Aviv apartment group mirrors itself to Telegram or WhatsApp;
+  `nester_rent_telaviv` carries Nester's own landlord submissions.
+  Secret Tel Aviv has no classifieds board, RSS or API.
+
+### What this means for the plan
+
+- The email bridge stays on the ladder as the compliant floor, demoted from
+  "the Facebook path" to "a Facebook supplement".
+- The rung-3 Facebook goal has a concrete primary design, **F1**, whose
+  tooling the *agent image* already ships (Playwright + Chromium + pinchtab
+  in `mercury-agent-ext`) — with one cost the goal must carry: feed-watch
+  polls **host-side**, the agent container is **per-run**, and F1 needs a
+  **persistent** logged-in Chromium profile (its precedent `fbn` is built
+  on one). So F1 is either a long-lived container with a mounted profile or
+  Chromium on the host, which today has neither Chromium nor node. **F2** is
+  the zero-code experiment the owner can run *today* to learn the real post
+  rate and whether their account gets checkpointed, before any Mercury code
+  exists.
+- The policy question the goal already flagged is now sharper: F1 and F2
+  both put a Facebook account at risk, and a burner account has to be
+  *admitted* to each private group by its admins. That is a decision for
+  the owner, not the plan.
+
+## X (Twitter)
+
+The football adopter's want list is a fixed set of ~10–20 accounts:
+clubs, a few journalists (Romano, Ornstein, Pedullà), Israeli outlets.
+The finding that shapes everything: **clubs are easy, journalists are the
+hard part**. Every free path covers the club accounts and fails for exactly
+the journalists and Israeli outlets.
+
+| # | Path | How | Complete? | Latency | Cost / month (20 accounts, hourly) | Risk | Terms |
+|---|------|-----|-----------|---------|------------------------------------|------|-------|
+| X1 | **twitterapi.io** (or SocialData search monitors) | One HTTP call per account per poll; pay-as-you-go, $0.15 per 1,000 tweets, no minimum (pricing page 2026-09-06) | full timelines | minutes | ~$2–3 if only returned tweets bill, ~$20 worst case; SocialData monitors ~$5–6 | vendor-side: the vendor gets sued, not the owner — a live risk after X's 2026-08 cease-and-desist to Nitter | the vendor violates, the owner does not |
+| X2 | **Official X API, pay-per-use** | Free tier closed to new sign-ups 2026-02-06; Basic auto-migrated 2026-06-01. `GET /2/users/:id/tweets` with `since_id`; Posts:Read $0.005 per resource, **deduplicated within a 24 h UTC window** (so hourly polling does not multiply cost), 3 M reads/month cap | full, official | minutes | **$50–90** | none | clean — the only fully sanctioned path |
+| X3 | **`syndication.twitter.com/srv/timeline-profile/screen-name/<u>`** (embedded timeline) | Unauthenticated HTML with `__NEXT_DATA__` → `timeline.entries`. **Two modes, deterministic per handle**: ~20 entries = a live reverse-chronological widget (ManUtd, Arsenal, realmadrid, LFC, FCBarcelona all fresh on 2026-09-06); ~100 entries = a **frozen engagement-ranked cache** (FabrizioRomano and Alfredopedulla 2026-09-01, ChampionsLeague 09-05, **David_Ornstein 2025-10-23, sport5il 2025-10-30, MaccabiTLVFC 2025-11-06**); ONE_Sport 0 entries. Intermittent 429 from the same IP (this session's own probe) | clubs only | minutes for clubs, **months** for the stale mode | none | low | grey |
+| X4 | Guest-token GraphQL (`guest/activate.json` → `UserTweets`) | Still issues a token and returns 200 — with the **identical 101 top-ranked tweets** as X3's stale mode. Not a separate data plane | same as X3 | same | none | medium | grey |
+| X5 | **Bluesky** (RSS per handle, open API) | Probe table above. Live: `david-ornstein.bsky.social` (2026-09-01), `arsenalfc.bsky.social` (09-03), The Athletic. Dead or squatted: Romano (abandoned 2024-09-11, bio says "Follow me on X"), `liverpoolfc.bsky.social`, `fcbarcelona.bsky.social`, Sky Sports PL parked. One live hobbyist mirror of Romano: `fabrizioromano.yopro20.com` (22,521 posts, newest 2026-09-06 17:27 UTC, working RSS); the sibling mirror farms died 2026-04 and 2025-02 | partial | minutes | none | none | clean |
+| X6 | **xcancel** (Nitter instance) RSS | Answered this session's probe today with a UA-whitelist request (probe table). X Corp sent Nitter and xcancel a cease-and-desist on **2026-08-24**; both went offline 08-25 and the Nitter repository was archived. Whatever is answering now is on borrowed time; self-hosting Nitter needs a donor account's session tokens and is named in the same letter | full while it lasts | minutes | none | the instance vanishes; a donor account for self-hosting is lost | violates |
+| — | RSSHub `/twitter/user`, RSS-Bridge TwitterBridge | Need a real account's `auth_token`+`ct0`; RSSHub issue #19420 (2025-06) reports empty results with no fix; RSS-Bridge guest-token breakage unresolved since 2023 | — | — | — | account loss | violates |
+| — | Own-session Playwright on x.com | Login wall, hardened fingerprint binding in 2025–26; no report of low-frequency read-only automation surviving long-term | full | — | none | **the owner's account** | violates |
+
+Dead ends: Google News `site:x.com`, X e-mail digests via IMAP (no
+evidence either yields timelines). RapidAPI Twitter tiers: pricing pages
+returned nothing, unverified.
+
+**A trap for feed-watch, independent of the path chosen.** X3 returns HTTP
+200 with plausible tweets that are months old. A source adapter that
+trusts status codes would post David Ornstein's 2025-10 tweets as new and
+never fail loudly. This is the same failure shape as the dead-but-healthy
+Telegram channel in `goal.md`: **source liveness must assert the newest
+item's timestamp, not the fetch's success.** That moves the "source
+liveness" candidate from optional to required before any source kind whose
+page can serve a stale body with HTTP 200 — Telegram included; until it
+ships, M1.5's per-channel precondition (newest post inside 7 days) stands
+in by hand.
+
+**What this means for the plan.** X gets a rung-2 API adapter after all,
+but for a *paid* vendor (X1 at ~$2–3/month typical, ~$20 worst case, or X2
+at ~$50–90/month if the owner wants a clean-terms path), keyed by a host-side secret, not a
+scraper. Free supplements: Bluesky RSS today (rung 0, zero code) for the
+handles that are live, and X3 for the club handles that demonstrably
+return the 20-entry live widget — verified per handle, with the freshness
+assertion. Nothing on X uses the owner's account.
+
+## Instagram
+
+| # | Path | How | Complete? | Cost / month | Risk | Terms |
+|---|------|-----|-----------|--------------|------|-------|
+| I1 | **Business Discovery** (official Graph API) | `?fields=business_discovery.username(<target>){media{caption,timestamp,permalink}}`. Needs the *"Instagram API with Facebook Login"* product (the newer "with Instagram Login" product lacks it), the owner's own Business/Creator account linked to a Facebook Page, `instagram_business_basic` (old scopes deprecated 2025-01-27) and Standard App Review. **The target must also be a Professional account** — clubs and outlets mostly are, individual journalists often are not. 200 calls/hour, ample. Hashtag search: 30 unique hashtags per 7-day window, last 24 h only | Professional targets only | none | none | clean |
+| I2 | **Apify Instagram Scraper** | `apify/instagram-scraper` $1.50 per 1,000 posts (2026-05); ~3,000 posts/month for 20 accounts ≈ $4.50, but the free plan is unusable so the $19 Starter floor applies | full | ~$19–24 | vendor-side | vendor violates |
+| I3 | `i.instagram.com/api/v1/users/web_profile_info/` with `x-ig-app-id` | Reported working from residential IPs by a 2026-08-28 guide; **this session's probe from the box got 401 `require_login`**. `?__a=1&__d=dis` is 404. The `graphql/query` `doc_id` path works but Meta rotates `doc_id` every 2–4 weeks | while it lasts | none | 429 per IP | grey |
+| I4 | **Android push → webhook** | Bell notifications are push-only (no e-mail digest exists for followed accounts). A notification-listener app (MacroDroid webhook trigger, `NotificationForwarder`, `NotificationWebhookApp`) POSTs each one to Mercury | misses whenever the phone sleeps or the listener is killed | none | none | clean |
+| — | Picuki, gramhir, RSSHub `picuki` route | Picuki dropped Instagram in 2025 (TikTok pivot), which killed the most-cited RSSHub route; gramhir dead; Imginn/Dumpor flaky; Pixwox/Picnob/Iganony attested only by affiliate blogs | — | — | — | — |
+| — | RSSHub `private-api`, instaloader with a session file, RSS-Bridge InstagramBridge | Stored credentials, no 2FA; documented temporary locks and forced password resets from frequent polling | full | none | **account** | violates |
+| — | rss.app Instagram feeds | From $8.32/month annual; the free tier refreshes every 24 h | partial | ~$8+ | none | grey |
+
+**What this means for the plan.** Instagram is the lowest-value platform
+for both adopters (club posts duplicate X; apartments are not on
+Instagram in volume). If it is ever built: I1 for the club/outlet accounts
+that are Professional, I2 for the rest, and nothing that logs in as the
+owner. It stays in the rung-3 goal.
+
+## Yad2, Madlan and the Tel Aviv rental market
+
+The premise to unwind first: Yad2 does not need to be read *from Yad2*.
+Two sanctioned channels carry its listings, one of them as plain RSS.
+
+### Realta — an aggregator with a listing-level RSS feed (verified from the box)
+
+`https://realta.co.il/feed.xml`: 200, `application/rss+xml`, **50 items,
+newest `pubDate` 2026-09-06 17:29 GMT, oldest of the 50 at 15:27** — the
+window covers about two hours of national volume at Saturday-evening
+rates. Each item is one listing: title `2 rooms in Jerusalem, Nahlaot —
+5,500 ₪/mo`, description with type / m² / street, permalink `guid`
+(`realta.co.il/en/<city>/<neighbourhood>/<id>/`). No per-city feed (the
+Tel Aviv path 404s); 7 of the 50 were Tel Aviv, so the term filter does
+the city. `robots.txt` allows everything but `/api/`, build assets and
+`*.json`, and advertises the sitemap — polling the feed is what the site
+invites. The Tel Aviv listing page carries a source icon per row; in one
+page load: **Madlan 7, Yad2 6, Facebook 6, Komo 4, Homeless 1**. The
+listing page links the original via `go.realta.co.il/go/<id>?url=<original>`,
+so the Yad2 or Facebook URL is recoverable from the query string without
+following the redirect. Realta also runs a Telegram channel,
+`t.me/s/realta_rent_il`, live (20 blocks, last 17:30 UTC).
+
+What that buys: **Yad2, Madlan and Facebook-group listings arrive as an
+`rss` source today, with zero code**, at rung 0. What it costs: a single
+point of failure run by a small operator whose legal position is the
+interesting one — Yad2 sent Keyz a demand to stop "collecting and
+republishing" its listings in June 2026 (TheMarker 2026-06-07). If Realta
+is next, the feed goes dark and the second leg below is the fallback.
+Unverified: whether Realta's "Facebook" rows are group posts or
+Marketplace items, and how much of the market it misses.
+
+### Yad2's own alerts: a fast alert within 15 minutes
+
+Yad2's app listing (v16.0.0, updated 2026-09-06) states that saved-search
+alerts go to e-mail either daily or as a **fast alert within 15 minutes**
+("התראה מהירה (עד 15 דק')"). That is inside the goal's "minutes" target
+with **zero blocking risk and zero terms exposure** — a logged-in user
+receiving mail they asked for. Mercury already has IMAP (`yahoo-mail`) and
+Gmail (`gws`). Unverified and cheap to settle: whether the mail body
+carries per-listing id, price, rooms, address and link, or only "N new
+results" with a link. One saved search and one raw MIME read answer it.
+This is the same rung-1c adapter the Facebook notifications need, with a
+second mail parser. No official Yad2 Telegram bot, WhatsApp/SMS alert, RSS
+or partner API exists.
+
+### Paths compared
+
+| # | Path | Coverage | Latency | Cost | Blocking risk | Terms / legal | Maintenance |
+|---|------|----------|---------|------|---------------|---------------|-------------|
+| Y1 | **Realta `feed.xml`** as an `rss` source | Yad2 + Madlan + Homeless + Komo + OnMap + Facebook; ~3,989 Tel Aviv listings on site | minutes | none | low; the feed is advertised | Realta's exposure, not the owner's | none; one dependency |
+| Y2 | **Yad2 e-mail alerts → IMAP** (rung 1c) | Yad2, exactly the saved search | ≤15 min stated | none | none | **clean** | low; mail format may change silently |
+| Y3 | Commercial Yad2 API (`swerve/yad2-scraper` $5 per 1,000, 96.4 % run success; `solidcode/yad2-scraper` $4 per 1,000; ScrapingBee $19–599/month) | Yad2 complete | per run | $4–5 per 1,000 results | the vendor's; they route through Israeli residential IPs with a real browser fingerprint and a fresh session per retry | the vendor breaches Yad2 §7; the owner is the buyer | none |
+| Y4 | Public Telegram channels | `nester_rent_telaviv` (live, Nester's own no-broker stock, ~4–5 posts/h, appears nowhere else); `realta_rent_il` (live, duplicates Y1). **`ILRentsTLV` is dead: 8 blocks, last 2023-04-12** — this session's probe confirms the survey and corrects a research claim that it was live | narrow, distinct | minutes | none | low | grey | channels die |
+| Y5 | homeless.co.il / komo.co.il direct (rung 1b) | thin slice, both inside Y1 | poll rate | none | 200 from the box, 403 from datacenters | unchecked | medium |
+| — | Own browser against Radware | Yad2 complete | — | proxies | high, adversarial, permanent | **breaches §7 in its own words**; Yad2 has enforced twice (a 2009 demand, litigated in the Petah Tikva Magistrates' Court in 2011; the Keyz demand, June 2026) — always against commercial republishers, never an individual | high, forever |
+| — | Google Alerts RSS | ~none: Yad2 listing pages have no crawl path; alerts fire after indexing | days to never | — | — | fine | — |
+| — | Madlan direct | Cloudflare 403 on HTML; sitemaps serve but are day-granular and over 10 MB; no reachable terms page (404). Madlan is Roztel Knowledge Systems, not Yad2 | — | — | — | — | — |
+| — | Craigslist Tel Aviv | 30 listings total; RSS 403 to datacenter fetchers | — | — | — | — | — |
+
+Corrections to the survey in `goal.md`: Yad2's parent is Apax (2026), not
+Coral/Axel Springer; "Yad2 v. Homeless" and "Yad2 v. Madlan" do not
+exist; no Israeli judgment applies the Computers Law §4 to reading an
+unauthenticated public site, and none holds that automated access in
+breach of terms is by itself unlawful — exposure runs through unjust
+enrichment and copyright in arrangement, and the Privacy Law's Amendment
+13 (in force 2025-08-14) carves out collections for personal,
+non-business use. Two commercial vendors report the Radware challenge is
+passable at 96–100 % with residential IPs and real browser fingerprints;
+the "heavily blocked" quote in circulation is from a 2019 freelancer post.
+
+### What this means for the plan
+
+- **Yad2 and Madlan leave the "out of scope" row.** Direct reads stay
+  out; their *listings* arrive through Y1 at rung 0 today and through Y2
+  at rung 1c. The Radware bypass is not built: it is the one path that is
+  high-maintenance, in breach of §7 verbatim, and unnecessary.
+- The Google Alerts spike is dropped from the roadmap.
+- Two empirical checks precede any apartment profile: how many minutes
+  the 50-item Realta window covers on a weekday morning (poll at 5 min or
+  less until measured), and what a Yad2 fast-alert mail body looks like.
+
+## Summary: the ladder after this research
+
+| Platform | Build | Do not build | Owner decision needed |
+|----------|-------|--------------|------------------------|
+| **Facebook groups** | rung 3 goal with **F1** (own session in Playwright reading Facebook's feed JSON; the image already has the tooling) as primary; **F2** (Groups Watcher → webhook) as the zero-code experiment to run first; 1c e-mail as the compliant floor; Realta's Facebook rows for apartments | anything replaying the session from a foreign datacenter (F3 as a poller) | which account F1/F2 burn — the owner's, or a second account admitted to each group |
+| **X** | rung 2 adapter against a **paid vendor** (twitterapi.io ~$2–3/month typical and ~$20 worst case, or the official pay-per-use API ~$50–90/month for a clean-terms path); Bluesky RSS at rung 0 for live handles; syndication for club handles that pass the freshness check | Nitter self-hosting, RSSHub with cookies, own-session Playwright | vendor vs official API; the monthly budget |
+| **Instagram** | I1 Business Discovery for Professional targets, I2 Apify for the rest, if ever | anything logged in as the owner | whether Instagram is worth building at all |
+| **Yad2 / Madlan** | Y1 Realta `rss` at rung 0; Y2 Yad2 fast alerts at rung 1c | a Radware bypass | none: both legs are sanctioned |
+| **All of them** | **source liveness that asserts the newest item's timestamp** — X's stale-but-200 cache and Telegram's dead-but-served channel are the same bug; required before any kind that can serve a stale 200, Telegram included, with M1.5's 7-day precondition as the manual stand-in | — | whether it joins Milestone 1 |