mercury-agent 0.20.0 → 0.22.0

package/docs/goals/feed-watch-social-sources/decisions.md

@@ -0,0 +1,332 @@
+# Decisions: feed-watch source adapters — social and messaging platforms as a ladder
+
+**Goal**: [feed-watch-social-sources](goal.md)
+
+> Inherits every decision in
+> [`whatsapp-bot-hardening/decisions.md`](../whatsapp-bot-hardening/decisions.md)
+> (in particular **D-014**: one instruction, one layer; countable rules) and
+> in [`football-reporter-profile/decisions.md`](../football-reporter-profile/decisions.md)
+> (in particular **D-001**: a space profile is a repo folder applied by
+> `scripts/space-profile.ts`, so `feed-watch.sources` deploys through the
+> manifest and a hand edit on the box is drift). Bare `D-0NN` ids below mean
+> *this* goal's sequence.
+>
+> The goal was decomposed autonomously by `/f-feature-roadmap` on 2026-09-06
+> and approved on 2026-09-07; D-001 to D-004 were recorded when Milestone 1's
+> architecture was filled the same day. No conflict with either inherited
+> goal's tech choices — this goal adds no storage, framework or dependency.
+
+## D-001: The parser seam is a closed `SourceKind` union in `feeds.ts`, dispatched by one exhaustive switch in `index.ts`
+- **Category:** architecture
+- **Decided:** `feeds.ts` exports `type SourceKind = "xml" | …` and stays a
+  leaf module; `FetchTarget.kind: SourceKind` is **required**; `fetchFeed`
+  gains optional `parse` (a body parser, default `parseFeed`) and `redirect`
+  (a `RequestRedirect`, default follow) options; `index.ts` implements
+  `PollDeps.fetchFeed` as `switch (target.kind)` with a `never` default, and
+  every case returns the **whole fetch** (`Promise<FeedItem[]>`).
+- **Alternatives considered:** (a) an exported `parserFor` table keyed by
+  kind — a runtime registry the compiler cannot check, and a fourth mechanism
+  for a two-entry table; (b) `SourceKind` on `FetchTarget` in `watch.ts` —
+  an import cycle once `feeds.ts` needs it; (c) `kind?: SourceKind` with an
+  `?? "xml"` default — an optional discriminator whose absent case allows;
+  (d) a `parse` option only, no `kind` — closes the door on a non-HTTP kind
+  (the rung-1c mailbox bridge has no URL to fetch).
+- **Reasoning:** the switch is the single place a parser is chosen, guaranteed
+  at compile time; a later kind is one union member and one case; the RSS
+  path is byte-identical because the `xml` case is today's call verbatim.
+- **Revisit if:** a third HTTP kind wants the same `parse`/`redirect` pair
+  with a different transport (per-target user-agent, auth header) — then the
+  options bag grows, not the switch.
+- **Used by:** feed-watch-parser-seam, feed-watch-telegram-parser,
+  feed-watch-telegram-source-config
+
+## D-002: Fixtures count toward the 500-line story budget
+- **Category:** pattern
+- **Decided:** a fixture is real markup cut to its head plus the 4–6 blocks
+  the tests need, with a leading comment naming the source, the capture date
+  and the cut; its line count is part of the story's diff budget.
+- **Alternatives considered:** a verbatim page (~958 lines for `t.me/s/…`,
+  twice the budget on its own); a hand-written minimal fixture (agrees with
+  the parser by construction and proves nothing about the real page).
+- **Reasoning:** the existing feed-watch fixtures are real bodies for the same
+  reason; a cut keeps the parser honest against real markup while keeping the
+  story reviewable in one session.
+- **Revisit if:** a source's page cannot be cut without losing a shape the
+  parser must handle — then split the story rather than grow the fixture.
+- **Used by:** feed-watch-telegram-parser, feed-watch-telegram-source-config
+
+## D-003: `MAX_SOURCES` stays at 20 for Milestone 1
+- **Category:** scope — **superseded 2026-09-10 by D-006 and D-007**: the
+  cap is 150 and the rationing it implied is gone. Kept as written; the
+  revisit clause it names is exactly what fired.
+- **Decided:** the football manifest holds 14 entries; M1.1 may add at most 3
+  and M1.5 at most 3. The cap constant, its validator and its description do
+  not move in this milestone.
+- **Alternatives considered:** raising the constant now (a cap the owner has
+  not asked to raise, with the `gnews` fan-out warning behind it); a per-space
+  override (new config surface for a need nobody has yet).
+- **Reasoning:** the budget fits; the owner ranks the rung-0 candidates into
+  3 slots rather than the code absorbing an unranked list.
+- **Revisit if:** the apartment profile or a second football channel list
+  needs more than 6 new entries — then a `config.ts` constant change with a
+  test, decided as its own story.
+- **Used by:** feed-watch-rung0-manifest-sources,
+  feed-watch-telegram-football-deploy
+
+## D-004: A validator change deploys restart-first, apply-second
+- **Category:** pattern
+- **Decided:** when a story changes what `isValidSourceEntry` accepts, the
+  deploy order is sync → restart in a cron gap → verify the **installed copy**
+  under `~/whatsapp-bot/.mercury/extensions/feed-watch/` → `space-profile
+  apply` → `check`. One restart per milestone, carried by the last story.
+- **Alternatives considered:** apply first and restart after (the manifest
+  is stored, the running validator rejects it, and `readSpaceConfig` blanks
+  the whole `sources` key on every poll until the restart — all 14 sources
+  dark); a config-key version guard (more machinery than the rule it
+  replaces).
+- **Reasoning:** `space-profile apply` writes the key without the
+  extension's validator and `check` compares the exact string, so nothing on
+  the write path stops a shape the running code does not know. M1.4 asserts
+  the whole-key fallback with a test so the reason stays countable.
+- **Revisit if:** the profile script starts running extension validators, or
+  the reader falls back per entry instead of per key.
+- **Used by:** feed-watch-telegram-source-config,
+  feed-watch-telegram-football-deploy
+
+## D-006: `MAX_SOURCES` moves and the poll fetches concurrently, as one story
+- **Category:** scope (revisits D-003; D-005 is reserved for the no-bypass
+  rule named under "Open" below, so this is D-006)
+- **Decided:** 2026-09-10, by the owner: the football space is to watch
+  many sources, and `football-match-day-mode` D-010 dropped its fixtures
+  provider in favour of watched channels, so D-003's revisit clause fires.
+  One story, `feed-watch-source-cap-and-concurrent-poll` (M1.6 on the
+  roadmap; idea filed at
+  `docs/ideas/feed-watch-source-cap-and-concurrent-poll.md`; architecture
+  via `/f-feature-planning`), raises the constant and its description, makes
+  `pollSpace` fetch targets through a bounded pool instead of one after
+  another, and measures the tick against the poll interval at the new cap.
+  The number is decided in that story from the measurement, not here.
+- **Alternatives considered:** rationing the six free slots between the
+  two goals (the 2026-09-07 shape); a per-space override of the cap (a
+  config surface for a bound that should simply be higher).
+- **Reasoning:** the cap guards the tick, not the bill — sequential fetches
+  at `FETCH_TIMEOUT_MS = 5000` inside a 3-minute interval on the host's
+  single thread, and `gnews` fan-out per term. `verify_max_items`,
+  `digest_max_items`, `BUFFER_CAP` and `max_per_hour` bound everything that
+  costs tokens, so more sources cannot make a run dearer. A concurrent fetch
+  removes the reason the bound was low.
+- **Revisit if:** the measured tick at the new cap still overruns the
+  interval — then the interval, not the cap, is the next knob.
+- **Used by:** feed-watch-source-cap-and-concurrent-poll,
+  feed-watch-rung0-manifest-sources (its "stay within 20" clause relaxes
+  once M1.6 lands), football-match-day-mode M1.7
+
+## D-007: 150 sources, a pool of 8, and one Reddit fetch per tick
+- **Category:** implementation (settles the numbers D-006 left to the story)
+- **Decided:** 2026-09-10, in `feed-watch-source-cap-and-concurrent-poll`.
+  `MAX_SOURCES = 150`; `FETCH_POOL_SIZE = 8`, a module constant and not a
+  config key; `BUFFER_CAP = 1500`; hosts in `ONE_PER_TICK_HOSTS` (the three
+  Reddit spellings) get **one** fetch per tick, round-robin from
+  `host_cursor:<spaceId>`, the rest counted as `hostDeferred`.
+- **Alternatives considered:** a pool of 6 (the idea file's first guess — the
+  ratio to the interval is what matters, and 8 makes the worst case a third
+  of it rather than half); making the pool a config key (a property of the
+  host process, not of a space, and no admin has the measurement to set it);
+  spacing same-host fetches 60 s apart *inside* the tick instead of rationing
+  across ticks (three Reddit feeds would put the tick's floor at 120 s, past
+  the slow-tick threshold, so every tick would report itself slow); giving
+  Reddit its own `SourceKind` (a rate limit is a property of the host, not of
+  the parser).
+- **Reasoning:** the cap always guarded the tick, and the pool is what moves
+  the tick's worst case from `due x 5 s` to `ceil(due / 8) x 5 s`. 150 clears
+  the round's 85-entry manifest (§7.I of
+  `docs/notes/2026-09-10-feed-watch-round.md`) with room for the bench tier.
+  1,500 buffered items is a *digest* bound: the daily article is built from
+  the buffer, so an overflow silently drops the oldest matched items of a
+  match day. Reddit's 60 s per-IP ceiling was measured on 2026-09-10 from the
+  box's own network, for both user-agents.
+- **Revisit if:** the `feed-watch: slow poll tick` line appears on a healthy
+  box (then the interval or the pool, in that order, not the cap); or a
+  second space starts watching (see D-008).
+- **Used by:** feed-watch-source-cap-and-concurrent-poll,
+  football-match-day-sources, feed-watch-rung0-manifest-sources
+
+## D-008: `inFlight` stays process-wide, and the tick's cost is now readable
+- **Category:** scope
+- **Decided:** 2026-09-10. The `inFlight` guard in `index.ts` keeps its
+  present meaning — **one tick at a time across every space** — and does not
+  become per space in this story. The reader that would justify changing it
+  ships instead: `feed-watch: slow poll tick` at `info` past half the
+  interval, and an unconditional `last_tick` store row carrying `lastTickMs`
+  and `lastTickAt`.
+- **Alternatives considered:** a per-space `inFlight` map (one slow space
+  would stop costing the others their poll — but the fetch phase is now
+  concurrent *within* a space, so the sum of ticks is far smaller than it
+  was, and a per-space guard would let N spaces put N pools on the host's
+  single thread with nothing bounding the total).
+- **Reasoning:** the football space is the only watcher on the box, so the
+  guard has never actually fired for the reason a per-space version would
+  fix. Splitting it now would trade a bound nobody has hit for an unbounded
+  total, and would do it without a measurement. The measurement is what this
+  story adds.
+- **Revisit if:** a second space enables `feed-watch.enabled`, or the
+  `lastTickMs` row on the box climbs past half the interval.
+- **Used by:** feed-watch-source-cap-and-concurrent-poll
+
+## D-009: liveness is `newestItemAt` plus a once-per-streak pair, and nothing else
+- **Category:** implementation
+- **Decided:** 2026-09-10, in `feed-watch-source-liveness` (M1.7).
+  `SourceStatus` gains `newestItemAt` — the running **maximum** of every
+  item's `publishedAt` a source has served — and `silentAt`. Past
+  `STALE_AFTER_MS = 7 days` with the fetch still succeeding, one
+  `feed-watch: source silent` at `warn`; a fresh item writes one
+  `feed-watch: source active again`, also at `warn`. Both fields are carried
+  through the failure branch untouched.
+- **Alternatives considered:** recording only this tick's newest date (a feed
+  that rotates its window would read as freshly stale every time it dropped
+  its newest item); a boolean `silent` flag (the *time* the silence started is
+  the fact an operator wants, and a boolean cannot be read as a duration);
+  `info` for the recovery line (the pair it mirrors is both-at-warn, and two
+  levels for two ends of one streak makes `grep -c` return a number that reads
+  as unresolved incidents).
+- **Reasoning:** `lastOkAt` is about the fetch, so a page serving a 2023 body
+  with HTTP 200 writes the same row as a live one — `t.me/s/ILRentsTLV` does
+  exactly this, and X's syndication widget did it for months. At 14 sources a
+  person notices; at the ~85 the round ships, the only symptom is an article
+  missing something nobody knew to expect.
+- **Revisit if:** a low-volume source flaps silent/active often enough to be
+  noise — then a per-source threshold, which is config surface nobody needs
+  yet.
+- **Used by:** feed-watch-source-liveness
+
+## D-010: a stale source is **not** treated as failed, for any kind
+- **Category:** scope (answers the roadmap's "Source liveness" row, which
+  asked for stale-as-failed "for kinds known to serve stale caches")
+- **Decided:** 2026-09-10. No backoff, no `nextAttemptAt`, no removal, no
+  exception for Telegram or for any other kind. Visibility only.
+- **Alternatives considered:** treating a stale Telegram channel as failed
+  (the row's own wording); a per-kind flag on `SourceKind` saying whether its
+  transport can serve a stale 200.
+- **Reasoning:** the roadmap row's premise does not survive contact with the
+  data. A *quiet* channel and a *dead* one are byte-identical from outside —
+  the same 200, the same well-formed body, the same old newest post — so
+  "known to serve stale caches" describes the transport, not the source, and
+  every Telegram channel would qualify including the ones that simply post
+  twice a month. Backing one off costs the story it eventually publishes, and
+  at the 96-minute backoff ceiling it would miss the return as well. The
+  failure the row is really about is *nobody noticing*, and a log line plus a
+  store field closes that without touching the poll.
+- **Consequence:** the manual per-channel precondition in
+  `feed-watch-telegram-football-deploy` (M1.5 — "newest `<time datetime>`
+  inside the last 7 days", checked by hand before adding a channel) is
+  **redundant** the moment this deploys: the same week, checked every tick, by
+  the running code. The edit to that document is **not** made here —
+  `football-match-day-sources` owns it this round — and is owed to whoever
+  lands it.
+- **Revisit if:** a watched kind appears whose transport can serve a stale
+  body *and* whose real publication rate is known to be daily, so that
+  silence is unambiguous.
+- **Used by:** feed-watch-source-liveness, feed-watch-telegram-football-deploy,
+  football-match-day-sources
+
+> **Numbering note (2026-09-10).** These three were written as D-007 to D-009
+> by S2 of the 2026-09-10 feed-watch round while S1 claimed D-007 to D-010 in
+> a parallel worktree; the coordinator renumbered them to D-011 to D-013 on
+> landing, together with the citations in `feed-watch-roundup-lane`'s spec.
+
+## D-011: The digest is lane-addressed, and the default lane is the unnamed one
+
+- **Category:** architecture
+- **Decided:** `[feed-watch:digest]` keeps its exact behaviour and its exact
+  cursor key `last_digest_at:<spaceId>`. A second reader carries
+  `[feed-watch:digest:<lane>]` (`lane` matching `[a-z0-9-]{1,32}`) and reads
+  and advances `last_digest_at:<spaceId>:<lane>`, with its own pending mark
+  `digest_until:<spaceId>:<lane>`. The block of a named lane carries
+  `lane="…"`; the default block is byte-for-byte unchanged, so the article's
+  prompt does not move. A marker whose lane will not parse yields **no**
+  digest, no mark and no cursor movement — never a fallback to the default
+  lane — and is warned about once per hook.
+- **Alternatives considered:** a second task on the plain marker (the reason
+  this story exists: `handleAfterContainer` advances the one cursor for any
+  run carrying the marker, so the 09:00 article would see only what arrived
+  since the last roundup — silently, reading as a thin news day); a per-task
+  cursor keyed on the task id (the id is a live row that does not exist until
+  the task is created, so the prompt could not name its own cursor, and a
+  re-created task would silently start from the beginning of the buffer); a
+  config key listing lanes (a second surface to keep in step with the prompts
+  that actually carry the markers).
+- **Reasoning:** the marker is already the contract between a task prompt and
+  the extension, and it is the only thing both hooks can see. Extending it is
+  a suffix, not a mechanism. Making the unnamed lane the default is what keeps
+  the migration empty: nothing on the box has to be renamed.
+- **Revisit if:** a lane ever needs to *read without consuming* (a preview
+  lane). Today every lane advances its own cursor, and that is stated as the
+  point rather than an accident.
+- **Used by:** feed-watch-roundup-lane
+
+## D-012: Each lane owns its window file and its archive
+
+- **Category:** architecture
+- **Decided:** the default lane keeps `knowledge/feed-digest-window.md` and
+  `knowledge/feed-digest.md`; a named lane writes
+  `knowledge/feed-digest-window.<lane>.md` and
+  `knowledge/feed-digest.<lane>.md`. The trailer names the lane's own path.
+- **Alternatives considered:** one shared pair of files with the lane written
+  inside each section. Rejected on two counts: the window file is *named to
+  the model in prose*, so it has to hold the window that model was handed
+  rather than whichever lane wrote last; and the archive is trimmed to
+  `DIGEST_FILE_WINDOWS = 30` sections, so six writers a day would cut the
+  article's readable history from thirty days to five without anything saying
+  so.
+- **Reasoning:** the cost is one string; the failure it removes is silent.
+- **Revisit if:** a space runs enough lanes for the workspace listing to
+  matter — then a `knowledge/feed-digest/<lane>.md` subdirectory, decided as
+  its own change.
+- **Used by:** feed-watch-roundup-lane
+
+## D-013: The roundup releases what has not been reported, on five slots a day
+
+- **Category:** scope
+- **Decided:** the football space's roundup runs
+  `0 10,13,16,19,22 * * *` `Asia/Jerusalem` — five slots, all outside
+  `feed-watch.quiet_hours` (`00:30-07:30`) and none at 09:00, which is the
+  article's. Its window **overlaps** what the immediate verify runs already
+  posted, and that overlap is resolved by the editorial rule that already
+  exists (`AGENTS.md § מניעת כפילויות`: an item already in the notebook's
+  `## Current State` is not reported again unless a concrete detail changed),
+  not by a second cursor. The roundup is not subject to
+  `feed-watch.max_per_hour`, which governs verify runs only.
+- **Alternatives considered:** a cursor that skips items a verify run
+  consumed (feed-watch does not record which items a run posted — only which
+  it was handed — so this would be a new bookkeeping surface for a rule the
+  standard already states); hourly slots (five was chosen against a `NO_UPDATE`
+  run's own cost, below); making the roundup obey `max_per_hour` (it is a
+  `tasks` row, not a batch, and a scheduled release nobody can predict is
+  worse than one that says nothing).
+- **Reasoning:** cost is `runs/day × prefix × $2.50/M` and a run that answers
+  `NO_UPDATE` still pays its whole prefix — five slots is roughly
+  $0.75–1.75/day against the space's $16–20/day, and the slot count is the
+  lever if the 50-source manifest pushes the window up.
+- **Revisit if:** the real group finds five too many or too few after the
+  link — the cron is a live row, so changing it needs no deploy.
+- **Used by:** feed-watch-roundup-lane
+
+## Open — for the owner, raised by the 2026-09-06 blocked-sources research
+
+Recorded in [blocked-sources-research.md](blocked-sources-research.md); none
+is needed for Milestone 1 and each becomes a numbered decision when its rung
+is decomposed:
+
+- **which Facebook account** a rung-3 reader burns — the owner's own
+  (already a member of every group, checkpoint locks them out of the same
+  groups) or a second account that each group's admins must admit;
+- **X: paid vendor or official API** — twitterapi.io at ~$2–3/month typical
+  (~$20 worst case) with a vendor whose own terms position is exposed, or the
+  official API at ~$50–90/month, clean;
+- **accepting Realta as a single dependency** for the apartment adopter's
+  Yad2/Madlan/Facebook coverage, with Yad2's own alert mail as the fallback
+  leg;
+- **no bot-protection bypass is built for any site whose terms forbid it**
+  (Yad2 §7, Madlan) — proposed as a standing rule; the approved plan builds
+  nothing that violates it, and it is recorded as D-005 the first time a
+  story would otherwise have to.

package/docs/goals/feed-watch-social-sources/goal.md

@@ -0,0 +1,217 @@
+# feed-watch source adapters — social and messaging platforms as a ladder, not a feature
+
+**Status**: Active — approved by the owner on 2026-09-07 after two critic rounds (decomposed by `/f-feature-roadmap` on 2026-09-06)
+**Slug**: feed-watch-social-sources
+**Created**: 2026-09-06
+**Last updated**: 2026-09-07
+
+---
+
+## Goal Statement
+
+> Autonomously decomposed from the idea `docs/ideas/feed-watch-social-sources.md` (deleted on approval, 2026-09-07; its content lives in the survey below). Approved by the owner on 2026-09-07 after two critic rounds.
+
+> "Track and search social media and Telegram groups — useful for other
+> profiles too, but a big feature to defer." — the owner, 2026-09-01, on the
+> football space. And on 2026-09-06: "I also plan on using the social-sources
+> ability for finding apartments in Tel Aviv."
+
+`feed-watch` knows two source types, `rss` and `gnews`. For the football
+space that misses where the news appears first — Israeli clubs and
+journalists post on Instagram, Facebook and public Telegram channels before
+any outlet writes it up. For an apartment hunter in Tel Aviv the gap is the
+whole market: listings live on Facebook groups, WhatsApp groups, Telegram
+channels and a handful of listing sites, none of which is an RSS feed.
+
+The goal is **source adapters, one rung at a time**: each adapter is a new
+`type` in `feed-watch.sources` that yields the same `{title, link,
+publishedAt, source}` items the RSS adapter yields, so the seen-set, the term
+match, the batching, the source status and the verify run are untouched and
+every profile gets every rung for free. Milestone 1 opens the seam and climbs
+the two cheapest rungs — plain-RSS sources that need only config, and public
+Telegram channels through their server-rendered preview page — with the
+football space as the first adopter. The apartment-hunting profile is the
+**second adopter** and is what makes the later rungs (a page watcher with a
+first-seen clock, a notification-email bridge for Facebook groups) worth
+their place on the roadmap; it is not built in this milestone, but nothing
+in this milestone may close a door it needs.
+
+Two research inputs shaped the plan and are summarised under **Sources**
+below: the owner's 2026-09-05 Gemini conversation on polling Facebook groups
+(printed to PDF, kept outside the repo), and a 2026-09-06 web survey of the
+Tel Aviv rental sources a host-side poller can actually read.
+
+## Constraints
+
+- **No new dependencies.** `feeds.ts` is dependency-free by design; every
+  adapter parses with regex tag extraction over raw text, no cheerio/jsdom.
+  `feeds.ts` stays a leaf module (watch.ts and index.ts import it; it imports
+  neither).
+- **The governing rule of `feeds.ts` holds for every adapter:** a malformed
+  body yields zero items and never throws; a transport failure (non-2xx,
+  including a manual-redirect 3xx, or a timeout) throws so `pollSpace`
+  backs the source off; an **oversize body is truncated, not thrown** — it
+  is cut to the head and reported once per key through the `bytes` warn,
+  because a throw would drop a feed that grew past the cap out of the
+  watchlist permanently (the comment in `feeds.ts` says so, and two football
+  feeds sit within 2× of the cap). A source `type` the validator does not
+  know is rejected, not ignored.
+- **Read-only, public, no account.** Milestone 1 touches nothing that needs a
+  login, a cookie, a burner account, a proxy or a scraping API. Those belong
+  to the rung-3 goal with its own account, credential and stored-data policy.
+- **Profile-owned keys deploy through the manifest.** `feed-watch.sources`
+  lands in `examples/profiles/football-reporter/config.yaml` and deploys with
+  `space-profile apply`; a hand edit on the box is drift. `feed-watch.terms`
+  and `feed-watch.enabled` are chat-seeded and are not touched.
+- **Deploy order for a validator change is fixed:** sync → restart in a gap of
+  the active task crons → verify the installed copy → `space-profile apply` →
+  `space-profile check`. Applying a manifest the running validator rejects
+  makes `readSpaceConfig` fall back to `[]` for the whole key — all 14
+  existing sources go dark until the restart. One restart per milestone.
+- **`MAX_SOURCES = 20` is a hard cap with 14 slots used.** The milestone fits
+  in the remaining 6 or explicitly decides to move the cap.
+- **Every rule is countable by a test** (`whatsapp-bot-hardening` D-014).
+  Each story < 500 lines of change **including tests and fixtures** — a
+  verbatim Telegram preview page is ~958 lines, so the fixture is a real page
+  cut to its head plus 4–6 untouched message blocks.
+- **Verification is against the running bot, never the diff.** The only
+  operator-visible status surfaces today are the extension store row
+  `source_status:<spaceId>` (keyed by `target.key`; fields `failures`,
+  `lastOkAt`, `nextAttemptAt`, `lastError`; table `extension_state`) and
+  the warn lines `feed-watch: source failed, backing off` /
+  `feed-watch: source recovered` / `feed-watch: source hit a cap, older
+  items dropped`. The `feed-watch: poll tick` line with its `failed` count
+  is **debug** and the live box runs `log_level: info`, so it is not a
+  live check. There is **no source-status widget or dashboard view**; no
+  story in this milestone adds one.
+- **Extension deploys are a service restart plus the startup whole-tree sync
+  of `.mercury/extensions/feed-watch/`**, not an image rebuild. The
+  Dockerfile does `COPY examples/extensions/` into a temp path, but the
+  `RUN` after it installs only the names in
+  `resources/builtin-extensions.txt` (`yahoo-mail`, `gws`, `tradestation`)
+  — `feed-watch` is not among them, so nothing of it lives in the image.
+  Verified by grepping the installed copy under `~/whatsapp-bot`.
+
+## Preferences
+
+- Regex-over-raw-text parsing in the house style of `feeds.ts` (required —
+  the header comment is explicit about not pulling a parser in).
+- One transport-and-parser dispatch on `FetchTarget.kind` in `index.ts`, an
+  exhaustive `switch` with a `never` default (preferred — it is the single
+  place a parser is chosen today, and the closed union is the guarantee
+  actually available; the plan review rejected a runtime table).
+- The dispatch chooses a **whole fetch**, not only a body parser, so a later
+  kind that is not an HTTP GET (the notification-email bridge reads an IMAP
+  folder, not a URL) plugs into the same switch (preferred).
+- A per-channel `outlet` label on the Telegram source, so a journalist's own
+  channel and a repost channel can be told apart at the attribution rule
+  (preferred — the idea's own open question, adopted by the plan).
+- Restart the live bot once per milestone, in a cron gap (required —
+  `CLAUDE.local.md`: a restart kills the running container and loses the
+  in-memory retry).
+
+## Target Users
+
+- **The football-space owner and admins** (first adopter, Milestone 1): the
+  space is a one-member test bed until the real group is linked, so a noisy
+  new source costs ledger rows only while it is tuned. They name the Telegram
+  channels and the rung-0 feeds.
+- **An apartment hunter in Tel Aviv** (second adopter, later milestones — the
+  owner, 2026-09-06): wants new rental posts from Telegram channels, listing
+  sites and (eventually) Facebook groups in the buffer within minutes, with
+  price, rooms and neighbourhood extracted from free Hebrew text by the verify
+  run. This is a new profile, not feed-watch code, except for the rungs it
+  needs: the page watcher (1b) with a first-seen clock, and the
+  notification-email bridge (1c).
+- **Any other profile** that watches a topic: every rung is a `sources` entry,
+  so the value compounds across profiles (inferred).
+
+## Success Criteria
+
+1. A public Telegram channel can be watched by configuration alone —
+   `{"type":"telegram","channel":"<name>","outlet":"<label>"}` in
+   `feed-watch.sources` — and a matched post reaches the buffer and a verify
+   run with the channel's own publication time and the `https://t.me/<ch>/<id>`
+   permalink, never the poll time. Countable in tests over a real trimmed
+   fixture and observed once on the running bot.
+2. A dead or private channel backs off alone: a 302 from `t.me/s/<name>` is a
+   source failure for that key only, the other sources stay fresh and the
+   service stays up. Countable in a `pollSpace` test and drilled once live.
+3. The RSS/gnews path is byte-identical after the seam opens: every existing
+   fixture test passes unchanged, and every rss/gnews target carries
+   `kind: "xml"`.
+4. Rung 0 is measured, not assumed: Reddit's unauthenticated RSS and the
+   hard-coded `user-agent` are probed from the box and the verdict is written
+   in the manifest comment before any entry is committed.
+5. Nothing in Milestone 1 closes a door the apartment adopter needs: the
+   dispatch chooses a whole fetch per kind, `FetchTarget.url` is not
+   load-bearing outside HTTP kinds, and `seenAt` remains a first-class item
+   field (the page watcher's only clock).
+
+## Sources
+
+### The owner's Gemini conversation, 2026-09-05 ("Automated Facebook Group Polling")
+
+Read from the PDF the owner printed on 2026-09-06 (kept outside the repo).
+What it established, and what the plan took from it:
+
+| Finding | Effect on this goal |
+|---------|---------------------|
+| Meta removed the Groups API (all versions, 2024-04-22); there is no official read path for group posts, public or private. Group RSS is long gone. | Facebook groups stay **rung 3**, a separate goal. Nothing in Milestone 1 pretends otherwise. |
+| Unofficial reads are browser automation (Playwright with stealth patches such as `rebrowser-playwright`, a dedicated "burner" account, saved `storageState`, home IP, throttled to 15–30 min) or a managed scraper API (Apify Facebook Group Scraper — ~$5/month free credit, "a check every 30 minutes on 3–5 groups"; Scrape.do / ZenRows / ScraperAPI, 1,000–5,000 free requests/month). Free public proxy lists are near 0 % effective against Meta. `kevinzg/facebook-scraper` is unmaintained (last release 2022), cookie-walled and not recommended. | The rung-3 goal has two candidate designs to weigh — **managed API as a host-side capability** (delegates anti-bot, usage-billed, external dependency, clean JSON) versus **own-session Playwright in the container** (the image already ships Playwright + Chromium; the `pinchtab` extension drives it) — and a policy question it must answer first: which account, what is stored, and Meta's terms §3.2 (no automated collection without permission). Recorded here so the later goal does not re-derive it. *Corrected 2026-09-06:* `rebrowser-playwright` is abandoned (last commit 2024-09); the maintained 2026 options are Patchright and Camoufox; Apify's official groups actor refuses cookies and private groups, so managed APIs read public groups only; the maintained precedent for the own-session design captures Facebook's own feed query instead of parsing the DOM. |
+| **"All Posts" group notifications → email → an IMAP parser** is a zero-risk read of groups the account is already a member of. Limits: new posts only, no history, and Facebook bundles high-volume groups into digests or skips mails. | A new rung the idea did not have, **1c — notification-email bridge**: a `sources` entry that reads a mailbox folder and yields one item per notification (post link, text, timestamp). Mercury already has host-side IMAP (`examples/extensions/yahoo-mail/lib/ymail.ts`, `imapflow`) and Gmail API scope (`gws`). This is the apartment adopter's cheapest Facebook path and the reason Milestone 1's dispatch must be able to choose a non-HTTP transport. Not built in M1. |
+| Historical search is a different problem from live tracking: `facebook.com/groups/<id>/search/?q=…` under a browser, or an Apify historical run, backfills; notifications catch new posts. | Confirms the idea's split: **"track"** is feed-watch's job; **"search on demand"** (a member asks the bot to search a platform) is a separate tool, out of this goal. |
+| Unstructured post text: price, rooms and neighbourhood are extracted by regex or an LLM; dedupe on the post id; notify with a direct link. | Maps one-to-one onto feed-watch: substring terms select, the verify run extracts and judges, the seen-set dedupes on the permalink. No new matching engine. |
+
+### Tel Aviv rental sources a host-side poller can read (web survey, 2026-09-06)
+
+| Source | Verdict | Rung |
+|--------|---------|------|
+| **Yad2** | Unreachable *directly*: Radware "Verifying your browser" JS challenge even from a residential IP with a Chrome UA (plain fetch; commercial vendors report real browsers on Israeli residential IPs pass at 96–100 %); the feed gateway 302s to a ShieldSquare validator; sitemaps list only search pages; no RSS. Terms §7 forbids "תוכנות אוטומטיות (ובכלל זאת רובוטים, spiders, scrapers)" verbatim, and Yad2 has enforced against commercial republishers (2009, and Keyz in June 2026). Sanctioned signals: **Yad2's own saved-search e-mail alerts, "fast" within 15 minutes** (app listing, 2026-09-06), and **Realta's aggregator RSS** below. | Direct reads **out of scope**; listings via **0** (Realta) and **1c** (alert mail). See [blocked-sources-research.md](blocked-sources-research.md). |
+| **Realta** (`realta.co.il/feed.xml`) | Listing-level RSS, 50 items ≈ two hours of national volume, `pubDate` minutes old, permalink `guid`, title with rooms / neighbourhood / price; sources per row on the Tel Aviv page: Madlan 7, Yad2 6, Facebook 6, Komo 4, Homeless 1; original URL recoverable from the `go.realta.co.il` redirect's query string; `robots.txt` invites crawling. Single small operator — the same exposure Yad2 just pressed on Keyz. | **0**, zero code, the apartment adopter's first source. Measure the window on a weekday before trusting it. |
+| **Facebook groups** (e.g. `telavivapartments`, 53 K members) | Non-browser clients get a 400 error page, not a login wall; API removed; terms §3.2.3 (2025-01-01) forbid automated collection *even while logged in*. "All posts" notifications are algorithm-filtered, not exhaustive (vendor claim, unverified). Complete reads exist only through an account's own session: Playwright reading Facebook's feed JSON (F1), or the Groups Watcher extension posting every new post to a webhook (F2) — see the research note. Realta already carries Facebook-sourced apartment listings. | 3 for a complete feed; 1c and Realta meanwhile. |
+| **Telegram channels** | `t.me/s/nester_rent_telaviv` serves message blocks (title "תל אביב דירות להשכרה ללא תיווך", ISO `<time datetime>`, `data-post`, price and rooms in the text, outbound link; Nester's own no-broker stock, ~4–5 posts/hour, appears nowhere else). `t.me/s/ILRentsTLV` also serves blocks **but its last post is 2023-04-12** — directory-listed yet dead (re-confirmed 2026-09-06 19:00 IDT). `t.me/s/realta_rent_il` is live and duplicates Realta's RSS. `t.me/s/plotlv` (a *group*) returns the landing page: `/s/` is channel-only. Scoutr, Shushu and Jeremy deliver via *bots*, which have no public preview. | **1** — and a live channel is a per-channel fact; see the liveness item under future milestones. `nester_rent_telaviv` is a candidate fixture with real RTL Hebrew. |
+| **homeless.co.il** | Server-rendered; 200 from the WSL box with any UA (403 from Anthropic egress); 53 `viewad,<id>.aspx` links per city page with a per-row `dd/mm/yyyy` date. No RSS. | **1b** — has a per-item date. |
+| **komo.co.il** | Server-rendered, no block; 999 items with price/rooms/m²/floor; permalink `?modaaNum=<id>`; **no per-item date**. | **1b** — first-seen is the only clock. |
+| **madlan.co.il** | Cloudflare 403 on HTML regardless of UA; sitemaps serve but are day-granular and over 10 MB; no reachable terms page; Apify reads its GraphQL API. Madlan is Roztel Knowledge Systems, not Yad2. | None directly; its listings are the largest share of Realta's Tel Aviv rows. |
+| **onmap.co.il** | JS-rendered ("0 apartments" in the HTML). **WinWin** closed 2020. | None. |
+| **Google News RSS** | Works as RSS but carries no listings — `site:yad2.co.il` returns landing pages, Hebrew queries return newspaper articles. | 0, useless for listings; the football success does not generalise. |
+| **Google Alerts RSS** | "Deliver to RSS feed" still exists (2026-06-17), but Yad2 listing pages have no crawl path and alerts fire only after indexing: days to never. | **Dropped** — Realta's feed does what the spike hoped for. |
+| **WhatsApp groups** | The bot *is* a WhatsApp client — a group the owner's number joins arrives as ordinary inbound messages. Scoutr (2026-04): WhatsApp is "the primary market", a good listing lasts 10 min–1 h. | **In-band, not a feed-watch source.** If wanted, matching must accept inbound messages, not poll. |
+| **Reddit r/telaviv** | `/new/.rss` 200 from a residential IP with a custom UA; 403 from datacenter IPs is widely reported. Low volume. | 0, fragile — the same UA/rate question as r/soccer. |
+| **Instagram** | Login wall on hashtag pages; profile HTML carries no post data; the 2023-era `web_profile_info` endpoint answers 401 from the box. Official Business Discovery reads Professional accounts only. | 3, lowest value of the three. |
+| **X** | Nitter and xcancel received X Corp's cease-and-desist on 2026-08-24 (an xcancel host answered the box on 09-06 with a whitelist-by-e-mail placeholder — borrowed time); the unauthenticated syndication widget serves a live 20-entry timeline for big clubs and a **months-stale engagement cache with HTTP 200** for journalists and Israeli outlets (Ornstein 2025-10, sport5il 2025-10, Maccabi 2025-11); the official API is pay-per-use only since 2026-02 (~$50–90/month for 20 accounts hourly, reads deduplicated per 24 h); twitterapi.io ~$5/month. Romano is on none of the free surfaces; one hobbyist Bluesky mirror of him is live. | **2, paid vendor**; Bluesky RSS at 0 for the handles that exist. |
+
+Not verified by the survey: homeless's date-column semantics (update vs.
+entry date), Telegram's terms on reading previews, the body format of a
+Yad2 fast-alert mail, how many minutes Realta's 50-item window covers on a
+weekday, whether Realta's "Facebook" rows are group posts or Marketplace.
+
+### Blocked platforms, researched 2026-09-06 at the owner's request
+
+The owner asked for every way past the walls on Facebook, Instagram, X,
+Yad2 and Madlan, Facebook first, because the notification-email bridge
+"won't be good enough". The answer — per platform, with probes from the
+box, costs, completeness, account risk and terms — is in
+[blocked-sources-research.md](blocked-sources-research.md). The ladder
+below already reflects it: Bluesky and Realta move to rung 0, X becomes a
+paid-vendor rung 2, Facebook's rung-3 goal gets a primary design that
+uses the owner's own session in the image's Chromium, Yad2 and Madlan
+listings arrive through Realta and Yad2's own 15-minute alerts, and no
+bot-protection bypass is built anywhere. One cross-cutting rule came out
+of it: **a source is live when its newest item is recent, not when its
+fetch returned 200** — X's stale cache and Telegram's dead channel fail
+the same way.
+
+## The ladder, revised
+
+| Rung | What | Cost | Status |
+|------|------|------|--------|
+| 0 | Reddit `/new/.rss`, YouTube channel feeds, **Bluesky per-account RSS** (`bsky.app/profile/<handle>/rss`, verified live for The Athletic, Ornstein, Arsenal), and for the apartment adopter **Realta's listing feed** (`realta.co.il/feed.xml` — Yad2, Madlan, Homeless, Komo and Facebook-group listings as one RSS, verified 2026-09-06) — already RSS; config plus a probe of UA and rate | config | **M1.1** (football); Realta waits for the apartment profile |
+| 1 | Public Telegram channels via `t.me/s/<channel>` — server-rendered, no account, no dependency | one adapter + fixture | **M1.2–M1.5** |
+| 1b | Generic page watcher — poll a server-rendered page, diff its link list against the seen-set; `seenAt` is the clock when the page has no per-item date (homeless.co.il has one, komo.co.il has none) | one adapter | future milestone, demoted: both sites are inside Realta's feed |
+| 1c | Notification-email bridge — read a mailbox folder over IMAP (or Gmail API) and yield one item per Facebook "All Posts" notification and per **Yad2 fast alert** (Yad2 states e-mail alerts within 15 minutes for a saved search — a sanctioned, complete Yad2 read); member's own groups, new posts only, digest bundling as a known gap on the Facebook side | one adapter on existing host-side mail code, two mail parsers | future milestone |
+| 2 | **X through a paid vendor** (twitterapi.io at $0.15 per 1,000 tweets — ~$2–3/month typical, ~$20 worst case; or the official pay-per-use API at ~$50–90/month for a clean-terms path), keyed by a host-side secret. Free paths cover **club handles only**: the unauthenticated syndication widget's live 20-entry mode, verified per handle behind the liveness check. Journalists and Israeli outlets get a months-stale cache with HTTP 200, Nitter is under a cease-and-desist (2026-08-24), and own-session automation burns an account. Bluesky moved to rung 0 | one API adapter, plus the syndication sub-item | future milestone, after the owner picks vendor and budget |
+| 3 | Facebook groups (and Instagram, if ever) — the [blocked-sources research](blocked-sources-research.md) gives the rung-3 goal a primary design: the owner's **own session in Playwright reading Facebook's feed JSON** in the image that already ships Chromium (F1), with the Groups Watcher extension → webhook (F2) as the zero-code experiment to run first, and 1c as the compliant floor. The tooling is in the *agent* image, but feed-watch polls host-side and that container is per-run, while F1 needs a persistent logged-in profile — a long-lived container with a mounted profile, or Chromium on the host, is part of the goal's cost. Which account is put at risk is the owner's decision, not the plan's | large | **separate goal**, not this one |
+| — | WhatsApp groups | none | in-band; not a source |
+| — | Yad2, Madlan *direct* | — | out of scope: a Radware bypass breaches Yad2's terms verbatim and is unnecessary — their listings arrive through rung 0 (Realta) and rung 1c (Yad2's own alerts) |