mercury-agent 0.20.0 → 0.21.0

package/docs/behavior-layers.md

@@ -145,8 +145,10 @@ prompt", and it runs as the caller who created it.
 ## 4. The history window
 
 - `context.mode` is a per-space key. `main` and DM auto-spaces are seeded
-  `context`; **any other space defaults to `clear`** (no history at all — only
-  the reply chain when someone swipe-replies). Set it explicitly.
+  with the host default (`contextMode` / `MERCURY_CONTEXT_MODE`, `context`
+  unless set); **any space without a row runs on that same host default**.
+  `clear` means no history at all — only the reply chain when someone
+  swipe-replies. Set the key explicitly on a space that should differ.
 - `context.window_size` (default 10 — `config.ts contextWindowSize`) counts
   **user turns**: `getRecentTurns` takes the newest `turnCount*5` non-ambient
   rows after the compact boundary and cuts at the Nth user row. Assistant rows

package/docs/configuration.md

@@ -199,6 +199,21 @@ It compares path *segments*, never string prefixes, so `mercury-shadowy` is
 not inside `mercury-shadow`. A live process (`MERCURY_SHADOW` unset) is never
 checked and runs exactly the code it ran before.
 
+**A shadow never fetches an image.** `ensureImage` does not pull a missing
+base image and does not re-pull a floating tag whose host version moved (the
+release-gate R0.4 refresh), and `ensureDerivedImage` does not build: a cache
+miss is fatal at boot, naming the tag. A rehearsal must run the image the live
+bot is running, and a shadow that fell back to the base image would be
+rehearsing a bot with none of the owner's extension CLIs.
+
+**Every container carries `mercury.instance=<id>`**, derived from the resolved
+data dir and settable by nothing, and the orphan sweep filters on it — no id,
+no sweep. Two Mercury processes on one Docker daemon is the rehearsal shape,
+and without this each one's boot sweep would force-remove the other's running
+container. `/health` reports the same id in its `instance` field, always, so an
+external probe can tell which process answered. `MERCURY_AGENT_ID` still
+narrows further; it never replaces this.
+
 Snapshot directories are built by `mercury shadow snapshot`, which generates
 the shadow's `.env` and `mercury.yaml` — nothing here needs to be set by
 hand. See [live-testing.md](live-testing.md) §2 and §7.

package/docs/goals/feed-watch-social-sources/blocked-sources-research.md

@@ -0,0 +1,289 @@
+# Blocked sources: what a personal poller can still read (research, 2026-09-06)
+
+**Goal**: [feed-watch-social-sources](goal.md)
+**Status**: research note — feeds the ladder in `goal.md` and the rung-3 goal; decides nothing by itself
+**Asked**: the owner, 2026-09-06 — "search for methods to bypass the restrictions and use Facebook / Instagram / X / Yad2 / Madlan or any other sources that are blocking … specifically Facebook is a very important source and I think the email notifications won't be good enough."
+
+The question is answered per platform: which read paths exist, what each
+costs in money, latency, completeness, account risk and terms, and which
+one this goal should build. The line drawn throughout: paths that use the
+owner's **own account on their own data**, an **official alert channel**, a
+**licensed data provider**, or an **open mirror** are candidates. Defeating
+a bot-protection challenge on a site whose terms forbid automated access
+(Yad2, Madlan) is recorded as "what exists", not planned — see the Yad2
+section for why the sanctioned channels are the better deal anyway.
+
+## Probes run from the WSL box (residential IP, one GET each, 2026-09-06 ~19:00 IDT)
+
+Plain `curl`, Chrome desktop UA unless noted. `mercury-feed-watch/1.0` is the
+UA `feeds.ts` sends today.
+
+| Endpoint | Result | What it means |
+|----------|--------|---------------|
+| `bsky.app/profile/theathletic.com/rss` | 200, `application/xml`, items with `<pubDate>` (newest 2026-09-06 06:49 UTC), post text in `<description>` | **Bluesky is rung 0, not rung 2**: per-account RSS with no auth, readable by today's `rss` source type with zero code |
+| `public.api.bsky.app/xrpc/app.bsky.feed.getAuthorFeed?actor=theathletic.com` | 200 JSON, `indexedAt` today | The richer API path also works unauthenticated; `searchPosts` returns 403 (auth) |
+| `public.api.bsky.app/xrpc/app.bsky.actor.searchActors?q=…` | 200; finds `skysportspl.bsky.social` (1 post — parked), several English football journalists, a handful of Hebrew-language football accounts (small, personal) | Who is *on* Bluesky decides its value. Romano is not (`fabrizioromano.bsky.social` is a 2023 squat). Sky Sports PL is parked. The Athletic is live |
+| `xcancel.com/FabrizioRomano/rss` (Chrome UA) | 400 "This URL only works inside an RSS client" | The instance gates RSS on the User-Agent |
+| `xcancel.com/FabrizioRomano/rss` (UA `mercury-feed-watch/1.0`) | 200 `application/rss+xml`, one placeholder item: "RSS reader not yet whitelisted! … send an email to rss [AT] xcancel [DOT] com with the reason … and this ID: …" | Whitelisted by the *instance operator*, not by X: the instance is under X Corp's cease-and-desist of 2026-08-24 and its terms position is the one in row X6 below. Recorded as what exists, not as a path to build |
+| `nitter.net/…/rss` | 410 Gone | Dead |
+| `nitter.privacydev.net` | connection refused | Dead |
+| `syndication.twitter.com/srv/timeline-profile/screen-name/…` | 429 "Rate limit exceeded" on the first request | Not usable from this IP without more work; not pursued |
+| `cdn.syndication.twimg.com/timeline/profile?screen_name=…` | 200, empty body | Deprecated |
+| `threads.net/@fabrizioromano` → `threads.com` | 200, 268 KB, no post text, no `taken_at`, no post codes in the HTML | Client-rendered behind a login prompt; not readable by fetch |
+| `instagram.com/fabrizioromano/` | 200, 618 KB, no captions, no `shortcode`, no timeline data | Same |
+| `i.instagram.com/api/v1/users/web_profile_info/?username=…` with `x-ig-app-id` | 401 `require_login: true` | The 2023-era unauthenticated endpoint is closed |
+| `facebook.com/groups/telavivapartments/` (public group) | 400, 1.5 KB "Error" page | Meta answers non-browser clients with an error, not a login page |
+| `m.facebook.com/groups/…` | 400 via redirect to `www` | The mobile site no longer exists as a separate surface |
+| `mbasic.facebook.com/groups/…` | 200 → `mbasic.facebook.com/login.php?next=…` (Hebrew login page, 6.5 KB) | **mbasic still serves** a login page in 2026; whether a logged-in session gets a group feed there is unverified (probe 9 in the open list) |
+| `t.me/s/nester_rent_telaviv` | 200, 4 message blocks, last `<time datetime>` 2026-09-06 16:01 UTC | Live; the M1 fixture candidate posted three hours before the probe |
+| `homeless.co.il/rent/`, `rss.app`, `fetchrss.com` | 200 | Reachable |
+
+Host tooling on the box: no Chromium, no node on the WSL host; `docker`
+runs without sudo; the derived image `mercury-agent-ext` carries
+Playwright + Chromium (`container/Dockerfile` line 51) and pinchtab. A
+headless page load can therefore run in a throwaway container from the
+home IP without installing anything on the host.
+
+Not probed: Yad2 and Madlan (the 2026-09-06 survey in `goal.md` already
+holds their answers for plain fetches, and a real-browser load was not
+run — see the Yad2 section).
+
+
+
+## Facebook groups
+
+The owner's premise holds: the notification-email bridge (rung 1c) is
+*compliant* but not *complete*. Notifications are algorithmically
+selected and bundled into digests, so a listing posted at 14:03 in a
+53 K-member group may never produce a mail. For the apartment adopter,
+where a good listing lasts ten minutes to an hour, that is the wrong
+tool. What is left is a choice between three complete-feed paths, none of
+them sanctioned by Meta.
+
+**Terms, stated once.** Meta's Terms §3.2.3 (effective 2025-01-01) forbid
+automated access or collection "even if that automated access or
+collection happens while you're logged into your Facebook account". The
+2024 Bright Data ruling (Meta lost on summary judgment, waived appeal)
+covers logged-out public scraping only and gives a logged-in reader
+nothing. No case was found 2024–2026 of Meta pursuing an individual for
+low-volume read-only automation; every action targets commercial
+scraping-for-hire. The realistic exposure is the **account**: temporary
+blocks and ID checkpoints, decided by behaviour and browser fingerprint,
+not by who the account is. A checkpoint on the owner's personal account
+locks them out of the same groups they are trying to read — which is why
+every path below is stated with *which account* it burns.
+
+### Paths, ranked for "private groups I am a member of"
+
+| # | Path | How | Complete? | Latency | Cost | Account at risk | Maintenance |
+|---|------|-----|-----------|---------|------|-----------------|-------------|
+| F1 | **Own session in Playwright, reading Facebook's feed JSON** | A persistent Chromium profile logged in as the reader account opens `facebook.com/groups/<id>/?sorting_setting=CHRONOLOGICAL` (desktop only; the parameter still works, the UI button is hidden); a response listener captures the JSON of Facebook's own `GroupsCometFeedRegularStoriesPaginationQuery` instead of parsing the DOM. Precedents: `viseshrp/fbn` (persistent profile, floor 15 min, default 1–3 h randomised) and the `browser-act` skill `facebook-groups-scrape-posts` (updated Aug 2026: 2–5 s between groups, never parallel). The `doc_id` changes on every Meta deploy, which is why the page's own request is *observed*, never constructed | yes, the full chronological feed | poll interval (15–30 min floor) | none | the reader account | medium: Meta frontend churn; the intercept absorbs most of it |
+| F2 | **Groups Watcher Chrome extension → local webhook** | Free extension (v5.13, 2026-06-17, ~1,000 users) in the owner's *everyday* Chrome on the *existing* session; polls every 3–7 min; with keywords disabled it POSTs every new post in batches to a URL: `{group_id, poster_name, post_url, post_id, body, images, timestamp}`. Needs the extension page open in a running Chrome — the same desktop Mercury already runs on | yes | 3–7 min | none | the owner's personal account, in a real browser fingerprint (lower risk than F1) | low, but the feed dies whenever Chrome does, and a 1,000-user extension holds the session and an outbound hook |
+| F3 | **Third-party Apify actor with exported cookies** (`whoareyouanas/facebook-group-scraper`) | Takes `c_user`+`xs`, `onlyPostsNewerThan`, chronological sort; ~20–40 posts/min; `maxConcurrency ≤ 2` | yes | per run | ~$3–10 per 1,000 posts (the page contradicts itself) | **higher**: the session replays from a foreign datacenter ASN, a classic checkpoint trigger | none |
+| F4 | **Israeli aggregators that already ingest the Hebrew groups** (apartments only) | Scoutr (`scoutrbot.com/city/tel-aviv`, no sign-up, plain HTML — 12,217 rentals, 2,596 attributed to Facebook, +191 in 24 h at research time); Dorin (₪19.90/week) and Agent Jeremy (₪29.99/week) Telegram bots claiming hundreds of groups | their coverage, unverifiable; Scoutr's sampled "Facebook" links pointed at Marketplace items, not group posts — check before relying on it | minutes to hours | free to ₪30/week | none | none; it is their scraper |
+| F5 | **Notification mirroring** | A spare Android phone or emulator with the Facebook app and group notifications on, plus a notification-listener app that POSTs to a local webhook (`BigShoots/NotificationWebhookApp`, `ItsAzni/NotificationForwarder`) | **no** — same algorithmic filter as email, just faster | seconds | ~none | none; nothing automates Facebook | low |
+| 1c | Email "All posts" → IMAP (already on the ladder) | — | **no** | minutes to hours | none | none | low |
+
+Not viable for private groups: rss.app, FetchRSS, Feedspot (public only by
+construction); Zapier, Make, Zoho, Buffer, Hootsuite (Groups apps removed
+2024-04-22 with the API); Apify's *official* `apify/facebook-groups-scraper`
+(refuses cookies and private groups by policy); Bright Data, ZenRows,
+ScrapingBee, Data365, Scrapfly (logged-out content only); Meta Content
+Library (academic/NGO enclave, $371/month + setup); "Download your
+information" (the owner's own content, no API).
+
+### Public groups and pages (football)
+
+`apify/facebook-groups-scraper` at $2.60–5.00 per 1,000 posts with
+`CHRONOLOGICAL` and `onlyPostsNewerThan`; Bright Data's groups scraper
+with 5,000 records/month free; rss.app / FetchRSS produce a feed URL for
+some public pages and groups ($0–15/month; FetchRSS deletes a free feed
+after 7 unread days). Logged-out browser scraping hits a login wall after
+roughly 1,000 posts per session, so for public content the managed actors
+beat engineering.
+
+### Corrections to what the goal recorded from the Gemini conversation
+
+- `rebrowser-playwright` is abandoned (last commit 2024-09). The 2026
+  maintained options are **Patchright** (2026-08-05) and **Camoufox**
+  (2026-08-12), which patch at the CDP layer; `nodriver` benchmarked best.
+- The *official* Apify actor does not take cookies; private-group support
+  is third-party only. The free credit is $5/month and does not roll over.
+- "All posts notifications are filtered" is asserted mainly by Groups
+  Watcher, which sells the fix. Probable, unconfirmed by any Meta source.
+- `mbasic.facebook.com` still serves (a login page, probe above); whether a
+  logged-in session gets a light group feed there is unverified.
+- No Tel Aviv apartment group mirrors itself to Telegram or WhatsApp;
+  `nester_rent_telaviv` carries Nester's own landlord submissions.
+  Secret Tel Aviv has no classifieds board, RSS or API.
+
+### What this means for the plan
+
+- The email bridge stays on the ladder as the compliant floor, demoted from
+  "the Facebook path" to "a Facebook supplement".
+- The rung-3 Facebook goal has a concrete primary design, **F1**, whose
+  tooling the *agent image* already ships (Playwright + Chromium + pinchtab
+  in `mercury-agent-ext`) — with one cost the goal must carry: feed-watch
+  polls **host-side**, the agent container is **per-run**, and F1 needs a
+  **persistent** logged-in Chromium profile (its precedent `fbn` is built
+  on one). So F1 is either a long-lived container with a mounted profile or
+  Chromium on the host, which today has neither Chromium nor node. **F2** is
+  the zero-code experiment the owner can run *today* to learn the real post
+  rate and whether their account gets checkpointed, before any Mercury code
+  exists.
+- The policy question the goal already flagged is now sharper: F1 and F2
+  both put a Facebook account at risk, and a burner account has to be
+  *admitted* to each private group by its admins. That is a decision for
+  the owner, not the plan.
+
+## X (Twitter)
+
+The football adopter's want list is a fixed set of ~10–20 accounts:
+clubs, a few journalists (Romano, Ornstein, Pedullà), Israeli outlets.
+The finding that shapes everything: **clubs are easy, journalists are the
+hard part**. Every free path covers the club accounts and fails for exactly
+the journalists and Israeli outlets.
+
+| # | Path | How | Complete? | Latency | Cost / month (20 accounts, hourly) | Risk | Terms |
+|---|------|-----|-----------|---------|------------------------------------|------|-------|
+| X1 | **twitterapi.io** (or SocialData search monitors) | One HTTP call per account per poll; pay-as-you-go, $0.15 per 1,000 tweets, no minimum (pricing page 2026-09-06) | full timelines | minutes | ~$2–3 if only returned tweets bill, ~$20 worst case; SocialData monitors ~$5–6 | vendor-side: the vendor gets sued, not the owner — a live risk after X's 2026-08 cease-and-desist to Nitter | the vendor violates, the owner does not |
+| X2 | **Official X API, pay-per-use** | Free tier closed to new sign-ups 2026-02-06; Basic auto-migrated 2026-06-01. `GET /2/users/:id/tweets` with `since_id`; Posts:Read $0.005 per resource, **deduplicated within a 24 h UTC window** (so hourly polling does not multiply cost), 3 M reads/month cap | full, official | minutes | **$50–90** | none | clean — the only fully sanctioned path |
+| X3 | **`syndication.twitter.com/srv/timeline-profile/screen-name/<u>`** (embedded timeline) | Unauthenticated HTML with `__NEXT_DATA__` → `timeline.entries`. **Two modes, deterministic per handle**: ~20 entries = a live reverse-chronological widget (ManUtd, Arsenal, realmadrid, LFC, FCBarcelona all fresh on 2026-09-06); ~100 entries = a **frozen engagement-ranked cache** (FabrizioRomano and Alfredopedulla 2026-09-01, ChampionsLeague 09-05, **David_Ornstein 2025-10-23, sport5il 2025-10-30, MaccabiTLVFC 2025-11-06**); ONE_Sport 0 entries. Intermittent 429 from the same IP (this session's own probe) | clubs only | minutes for clubs, **months** for the stale mode | none | low | grey |
+| X4 | Guest-token GraphQL (`guest/activate.json` → `UserTweets`) | Still issues a token and returns 200 — with the **identical 101 top-ranked tweets** as X3's stale mode. Not a separate data plane | same as X3 | same | none | medium | grey |
+| X5 | **Bluesky** (RSS per handle, open API) | Probe table above. Live: `david-ornstein.bsky.social` (2026-09-01), `arsenalfc.bsky.social` (09-03), The Athletic. Dead or squatted: Romano (abandoned 2024-09-11, bio says "Follow me on X"), `liverpoolfc.bsky.social`, `fcbarcelona.bsky.social`, Sky Sports PL parked. One live hobbyist mirror of Romano: `fabrizioromano.yopro20.com` (22,521 posts, newest 2026-09-06 17:27 UTC, working RSS); the sibling mirror farms died 2026-04 and 2025-02 | partial | minutes | none | none | clean |
+| X6 | **xcancel** (Nitter instance) RSS | Answered this session's probe today with a UA-whitelist request (probe table). X Corp sent Nitter and xcancel a cease-and-desist on **2026-08-24**; both went offline 08-25 and the Nitter repository was archived. Whatever is answering now is on borrowed time; self-hosting Nitter needs a donor account's session tokens and is named in the same letter | full while it lasts | minutes | none | the instance vanishes; a donor account for self-hosting is lost | violates |
+| — | RSSHub `/twitter/user`, RSS-Bridge TwitterBridge | Need a real account's `auth_token`+`ct0`; RSSHub issue #19420 (2025-06) reports empty results with no fix; RSS-Bridge guest-token breakage unresolved since 2023 | — | — | — | account loss | violates |
+| — | Own-session Playwright on x.com | Login wall, hardened fingerprint binding in 2025–26; no report of low-frequency read-only automation surviving long-term | full | — | none | **the owner's account** | violates |
+
+Dead ends: Google News `site:x.com`, X e-mail digests via IMAP (no
+evidence either yields timelines). RapidAPI Twitter tiers: pricing pages
+returned nothing, unverified.
+
+**A trap for feed-watch, independent of the path chosen.** X3 returns HTTP
+200 with plausible tweets that are months old. A source adapter that
+trusts status codes would post David Ornstein's 2025-10 tweets as new and
+never fail loudly. This is the same failure shape as the dead-but-healthy
+Telegram channel in `goal.md`: **source liveness must assert the newest
+item's timestamp, not the fetch's success.** That moves the "source
+liveness" candidate from optional to required before any source kind whose
+page can serve a stale body with HTTP 200 — Telegram included; until it
+ships, M1.5's per-channel precondition (newest post inside 7 days) stands
+in by hand.
+
+**What this means for the plan.** X gets a rung-2 API adapter after all,
+but for a *paid* vendor (X1 at ~$2–3/month typical, ~$20 worst case, or X2
+at ~$50–90/month if the owner wants a clean-terms path), keyed by a host-side secret, not a
+scraper. Free supplements: Bluesky RSS today (rung 0, zero code) for the
+handles that are live, and X3 for the club handles that demonstrably
+return the 20-entry live widget — verified per handle, with the freshness
+assertion. Nothing on X uses the owner's account.
+
+## Instagram
+
+| # | Path | How | Complete? | Cost / month | Risk | Terms |
+|---|------|-----|-----------|--------------|------|-------|
+| I1 | **Business Discovery** (official Graph API) | `?fields=business_discovery.username(<target>){media{caption,timestamp,permalink}}`. Needs the *"Instagram API with Facebook Login"* product (the newer "with Instagram Login" product lacks it), the owner's own Business/Creator account linked to a Facebook Page, `instagram_business_basic` (old scopes deprecated 2025-01-27) and Standard App Review. **The target must also be a Professional account** — clubs and outlets mostly are, individual journalists often are not. 200 calls/hour, ample. Hashtag search: 30 unique hashtags per 7-day window, last 24 h only | Professional targets only | none | none | clean |
+| I2 | **Apify Instagram Scraper** | `apify/instagram-scraper` $1.50 per 1,000 posts (2026-05); ~3,000 posts/month for 20 accounts ≈ $4.50, but the free plan is unusable so the $19 Starter floor applies | full | ~$19–24 | vendor-side | vendor violates |
+| I3 | `i.instagram.com/api/v1/users/web_profile_info/` with `x-ig-app-id` | Reported working from residential IPs by a 2026-08-28 guide; **this session's probe from the box got 401 `require_login`**. `?__a=1&__d=dis` is 404. The `graphql/query` `doc_id` path works but Meta rotates `doc_id` every 2–4 weeks | while it lasts | none | 429 per IP | grey |
+| I4 | **Android push → webhook** | Bell notifications are push-only (no e-mail digest exists for followed accounts). A notification-listener app (MacroDroid webhook trigger, `NotificationForwarder`, `NotificationWebhookApp`) POSTs each one to Mercury | misses whenever the phone sleeps or the listener is killed | none | none | clean |
+| — | Picuki, gramhir, RSSHub `picuki` route | Picuki dropped Instagram in 2025 (TikTok pivot), which killed the most-cited RSSHub route; gramhir dead; Imginn/Dumpor flaky; Pixwox/Picnob/Iganony attested only by affiliate blogs | — | — | — | — |
+| — | RSSHub `private-api`, instaloader with a session file, RSS-Bridge InstagramBridge | Stored credentials, no 2FA; documented temporary locks and forced password resets from frequent polling | full | none | **account** | violates |
+| — | rss.app Instagram feeds | From $8.32/month annual; the free tier refreshes every 24 h | partial | ~$8+ | none | grey |
+
+**What this means for the plan.** Instagram is the lowest-value platform
+for both adopters (club posts duplicate X; apartments are not on
+Instagram in volume). If it is ever built: I1 for the club/outlet accounts
+that are Professional, I2 for the rest, and nothing that logs in as the
+owner. It stays in the rung-3 goal.
+
+## Yad2, Madlan and the Tel Aviv rental market
+
+The premise to unwind first: Yad2 does not need to be read *from Yad2*.
+Two sanctioned channels carry its listings, one of them as plain RSS.
+
+### Realta — an aggregator with a listing-level RSS feed (verified from the box)
+
+`https://realta.co.il/feed.xml`: 200, `application/rss+xml`, **50 items,
+newest `pubDate` 2026-09-06 17:29 GMT, oldest of the 50 at 15:27** — the
+window covers about two hours of national volume at Saturday-evening
+rates. Each item is one listing: title `2 rooms in Jerusalem, Nahlaot —
+5,500 ₪/mo`, description with type / m² / street, permalink `guid`
+(`realta.co.il/en/<city>/<neighbourhood>/<id>/`). No per-city feed (the
+Tel Aviv path 404s); 7 of the 50 were Tel Aviv, so the term filter does
+the city. `robots.txt` allows everything but `/api/`, build assets and
+`*.json`, and advertises the sitemap — polling the feed is what the site
+invites. The Tel Aviv listing page carries a source icon per row; in one
+page load: **Madlan 7, Yad2 6, Facebook 6, Komo 4, Homeless 1**. The
+listing page links the original via `go.realta.co.il/go/<id>?url=<original>`,
+so the Yad2 or Facebook URL is recoverable from the query string without
+following the redirect. Realta also runs a Telegram channel,
+`t.me/s/realta_rent_il`, live (20 blocks, last 17:30 UTC).
+
+What that buys: **Yad2, Madlan and Facebook-group listings arrive as an
+`rss` source today, with zero code**, at rung 0. What it costs: a single
+point of failure run by a small operator whose legal position is the
+interesting one — Yad2 sent Keyz a demand to stop "collecting and
+republishing" its listings in June 2026 (TheMarker 2026-06-07). If Realta
+is next, the feed goes dark and the second leg below is the fallback.
+Unverified: whether Realta's "Facebook" rows are group posts or
+Marketplace items, and how much of the market it misses.
+
+### Yad2's own alerts: a fast alert within 15 minutes
+
+Yad2's app listing (v16.0.0, updated 2026-09-06) states that saved-search
+alerts go to e-mail either daily or as a **fast alert within 15 minutes**
+("התראה מהירה (עד 15 דק')"). That is inside the goal's "minutes" target
+with **zero blocking risk and zero terms exposure** — a logged-in user
+receiving mail they asked for. Mercury already has IMAP (`yahoo-mail`) and
+Gmail (`gws`). Unverified and cheap to settle: whether the mail body
+carries per-listing id, price, rooms, address and link, or only "N new
+results" with a link. One saved search and one raw MIME read answer it.
+This is the same rung-1c adapter the Facebook notifications need, with a
+second mail parser. No official Yad2 Telegram bot, WhatsApp/SMS alert, RSS
+or partner API exists.
+
+### Paths compared
+
+| # | Path | Coverage | Latency | Cost | Blocking risk | Terms / legal | Maintenance |
+|---|------|----------|---------|------|---------------|---------------|-------------|
+| Y1 | **Realta `feed.xml`** as an `rss` source | Yad2 + Madlan + Homeless + Komo + OnMap + Facebook; ~3,989 Tel Aviv listings on site | minutes | none | low; the feed is advertised | Realta's exposure, not the owner's | none; one dependency |
+| Y2 | **Yad2 e-mail alerts → IMAP** (rung 1c) | Yad2, exactly the saved search | ≤15 min stated | none | none | **clean** | low; mail format may change silently |
+| Y3 | Commercial Yad2 API (`swerve/yad2-scraper` $5 per 1,000, 96.4 % run success; `solidcode/yad2-scraper` $4 per 1,000; ScrapingBee $19–599/month) | Yad2 complete | per run | $4–5 per 1,000 results | the vendor's; they route through Israeli residential IPs with a real browser fingerprint and a fresh session per retry | the vendor breaches Yad2 §7; the owner is the buyer | none |
+| Y4 | Public Telegram channels | `nester_rent_telaviv` (live, Nester's own no-broker stock, ~4–5 posts/h, appears nowhere else); `realta_rent_il` (live, duplicates Y1). **`ILRentsTLV` is dead: 8 blocks, last 2023-04-12** — this session's probe confirms the survey and corrects a research claim that it was live | narrow, distinct | minutes | none | low | grey | channels die |
+| Y5 | homeless.co.il / komo.co.il direct (rung 1b) | thin slice, both inside Y1 | poll rate | none | 200 from the box, 403 from datacenters | unchecked | medium |
+| — | Own browser against Radware | Yad2 complete | — | proxies | high, adversarial, permanent | **breaches §7 in its own words**; Yad2 has enforced twice (a 2009 demand, litigated in the Petah Tikva Magistrates' Court in 2011; the Keyz demand, June 2026) — always against commercial republishers, never an individual | high, forever |
+| — | Google Alerts RSS | ~none: Yad2 listing pages have no crawl path; alerts fire after indexing | days to never | — | — | fine | — |
+| — | Madlan direct | Cloudflare 403 on HTML; sitemaps serve but are day-granular and over 10 MB; no reachable terms page (404). Madlan is Roztel Knowledge Systems, not Yad2 | — | — | — | — | — |
+| — | Craigslist Tel Aviv | 30 listings total; RSS 403 to datacenter fetchers | — | — | — | — | — |
+
+Corrections to the survey in `goal.md`: Yad2's parent is Apax (2026), not
+Coral/Axel Springer; "Yad2 v. Homeless" and "Yad2 v. Madlan" do not
+exist; no Israeli judgment applies the Computers Law §4 to reading an
+unauthenticated public site, and none holds that automated access in
+breach of terms is by itself unlawful — exposure runs through unjust
+enrichment and copyright in arrangement, and the Privacy Law's Amendment
+13 (in force 2025-08-14) carves out collections for personal,
+non-business use. Two commercial vendors report the Radware challenge is
+passable at 96–100 % with residential IPs and real browser fingerprints;
+the "heavily blocked" quote in circulation is from a 2019 freelancer post.
+
+### What this means for the plan
+
+- **Yad2 and Madlan leave the "out of scope" row.** Direct reads stay
+  out; their *listings* arrive through Y1 at rung 0 today and through Y2
+  at rung 1c. The Radware bypass is not built: it is the one path that is
+  high-maintenance, in breach of §7 verbatim, and unnecessary.
+- The Google Alerts spike is dropped from the roadmap.
+- Two empirical checks precede any apartment profile: how many minutes
+  the 50-item Realta window covers on a weekday morning (poll at 5 min or
+  less until measured), and what a Yad2 fast-alert mail body looks like.
+
+## Summary: the ladder after this research
+
+| Platform | Build | Do not build | Owner decision needed |
+|----------|-------|--------------|------------------------|
+| **Facebook groups** | rung 3 goal with **F1** (own session in Playwright reading Facebook's feed JSON; the image already has the tooling) as primary; **F2** (Groups Watcher → webhook) as the zero-code experiment to run first; 1c e-mail as the compliant floor; Realta's Facebook rows for apartments | anything replaying the session from a foreign datacenter (F3 as a poller) | which account F1/F2 burn — the owner's, or a second account admitted to each group |
+| **X** | rung 2 adapter against a **paid vendor** (twitterapi.io ~$2–3/month typical and ~$20 worst case, or the official pay-per-use API ~$50–90/month for a clean-terms path); Bluesky RSS at rung 0 for live handles; syndication for club handles that pass the freshness check | Nitter self-hosting, RSSHub with cookies, own-session Playwright | vendor vs official API; the monthly budget |
+| **Instagram** | I1 Business Discovery for Professional targets, I2 Apify for the rest, if ever | anything logged in as the owner | whether Instagram is worth building at all |
+| **Yad2 / Madlan** | Y1 Realta `rss` at rung 0; Y2 Yad2 fast alerts at rung 1c | a Radware bypass | none: both legs are sanctioned |
+| **All of them** | **source liveness that asserts the newest item's timestamp** — X's stale-but-200 cache and Telegram's dead-but-served channel are the same bug; required before any kind that can serve a stale 200, Telegram included, with M1.5's 7-day precondition as the manual stand-in | — | whether it joins Milestone 1 |

package/docs/goals/feed-watch-social-sources/decisions.md

@@ -0,0 +1,112 @@
+# Decisions: feed-watch source adapters — social and messaging platforms as a ladder
+
+**Goal**: [feed-watch-social-sources](goal.md)
+
+> Inherits every decision in
+> [`whatsapp-bot-hardening/decisions.md`](../whatsapp-bot-hardening/decisions.md)
+> (in particular **D-014**: one instruction, one layer; countable rules) and
+> in [`football-reporter-profile/decisions.md`](../football-reporter-profile/decisions.md)
+> (in particular **D-001**: a space profile is a repo folder applied by
+> `scripts/space-profile.ts`, so `feed-watch.sources` deploys through the
+> manifest and a hand edit on the box is drift). Bare `D-0NN` ids below mean
+> *this* goal's sequence.
+>
+> The goal was decomposed autonomously by `/f-feature-roadmap` on 2026-09-06
+> and approved on 2026-09-07; D-001 to D-004 were recorded when Milestone 1's
+> architecture was filled the same day. No conflict with either inherited
+> goal's tech choices — this goal adds no storage, framework or dependency.
+
+## D-001: The parser seam is a closed `SourceKind` union in `feeds.ts`, dispatched by one exhaustive switch in `index.ts`
+- **Category:** architecture
+- **Decided:** `feeds.ts` exports `type SourceKind = "xml" | …` and stays a
+  leaf module; `FetchTarget.kind: SourceKind` is **required**; `fetchFeed`
+  gains optional `parse` (a body parser, default `parseFeed`) and `redirect`
+  (a `RequestRedirect`, default follow) options; `index.ts` implements
+  `PollDeps.fetchFeed` as `switch (target.kind)` with a `never` default, and
+  every case returns the **whole fetch** (`Promise<FeedItem[]>`).
+- **Alternatives considered:** (a) an exported `parserFor` table keyed by
+  kind — a runtime registry the compiler cannot check, and a fourth mechanism
+  for a two-entry table; (b) `SourceKind` on `FetchTarget` in `watch.ts` —
+  an import cycle once `feeds.ts` needs it; (c) `kind?: SourceKind` with an
+  `?? "xml"` default — an optional discriminator whose absent case allows;
+  (d) a `parse` option only, no `kind` — closes the door on a non-HTTP kind
+  (the rung-1c mailbox bridge has no URL to fetch).
+- **Reasoning:** the switch is the single place a parser is chosen, guaranteed
+  at compile time; a later kind is one union member and one case; the RSS
+  path is byte-identical because the `xml` case is today's call verbatim.
+- **Revisit if:** a third HTTP kind wants the same `parse`/`redirect` pair
+  with a different transport (per-target user-agent, auth header) — then the
+  options bag grows, not the switch.
+- **Used by:** feed-watch-parser-seam, feed-watch-telegram-parser,
+  feed-watch-telegram-source-config
+
+## D-002: Fixtures count toward the 500-line story budget
+- **Category:** pattern
+- **Decided:** a fixture is real markup cut to its head plus the 4–6 blocks
+  the tests need, with a leading comment naming the source, the capture date
+  and the cut; its line count is part of the story's diff budget.
+- **Alternatives considered:** a verbatim page (~958 lines for `t.me/s/…`,
+  twice the budget on its own); a hand-written minimal fixture (agrees with
+  the parser by construction and proves nothing about the real page).
+- **Reasoning:** the existing feed-watch fixtures are real bodies for the same
+  reason; a cut keeps the parser honest against real markup while keeping the
+  story reviewable in one session.
+- **Revisit if:** a source's page cannot be cut without losing a shape the
+  parser must handle — then split the story rather than grow the fixture.
+- **Used by:** feed-watch-telegram-parser, feed-watch-telegram-source-config
+
+## D-003: `MAX_SOURCES` stays at 20 for Milestone 1
+- **Category:** scope
+- **Decided:** the football manifest holds 14 entries; M1.1 may add at most 3
+  and M1.5 at most 3. The cap constant, its validator and its description do
+  not move in this milestone.
+- **Alternatives considered:** raising the constant now (a cap the owner has
+  not asked to raise, with the `gnews` fan-out warning behind it); a per-space
+  override (new config surface for a need nobody has yet).
+- **Reasoning:** the budget fits; the owner ranks the rung-0 candidates into
+  3 slots rather than the code absorbing an unranked list.
+- **Revisit if:** the apartment profile or a second football channel list
+  needs more than 6 new entries — then a `config.ts` constant change with a
+  test, decided as its own story.
+- **Used by:** feed-watch-rung0-manifest-sources,
+  feed-watch-telegram-football-deploy
+
+## D-004: A validator change deploys restart-first, apply-second
+- **Category:** pattern
+- **Decided:** when a story changes what `isValidSourceEntry` accepts, the
+  deploy order is sync → restart in a cron gap → verify the **installed copy**
+  under `~/whatsapp-bot/.mercury/extensions/feed-watch/` → `space-profile
+  apply` → `check`. One restart per milestone, carried by the last story.
+- **Alternatives considered:** apply first and restart after (the manifest
+  is stored, the running validator rejects it, and `readSpaceConfig` blanks
+  the whole `sources` key on every poll until the restart — all 14 sources
+  dark); a config-key version guard (more machinery than the rule it
+  replaces).
+- **Reasoning:** `space-profile apply` writes the key without the
+  extension's validator and `check` compares the exact string, so nothing on
+  the write path stops a shape the running code does not know. M1.4 asserts
+  the whole-key fallback with a test so the reason stays countable.
+- **Revisit if:** the profile script starts running extension validators, or
+  the reader falls back per entry instead of per key.
+- **Used by:** feed-watch-telegram-source-config,
+  feed-watch-telegram-football-deploy
+
+## Open — for the owner, raised by the 2026-09-06 blocked-sources research
+
+Recorded in [blocked-sources-research.md](blocked-sources-research.md); none
+is needed for Milestone 1 and each becomes a numbered decision when its rung
+is decomposed:
+
+- **which Facebook account** a rung-3 reader burns — the owner's own
+  (already a member of every group, checkpoint locks them out of the same
+  groups) or a second account that each group's admins must admit;
+- **X: paid vendor or official API** — twitterapi.io at ~$2–3/month typical
+  (~$20 worst case) with a vendor whose own terms position is exposed, or the
+  official API at ~$50–90/month, clean;
+- **accepting Realta as a single dependency** for the apartment adopter's
+  Yad2/Madlan/Facebook coverage, with Yad2's own alert mail as the fallback
+  leg;
+- **no bot-protection bypass is built for any site whose terms forbid it**
+  (Yad2 §7, Madlan) — proposed as a standing rule; the approved plan builds
+  nothing that violates it, and it is recorded as D-005 the first time a
+  story would otherwise have to.