fetchkit-agents 0.2.0__tar.gz → 0.3.0__tar.gz

{fetchkit_agents-0.2.0 → fetchkit_agents-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fetchkit-agents
-Version: 0.2.0
+Version: 0.3.0
 Summary: A YAML-configured data fetching library for agentic applications.
 Author-email: mete morris <morrismete@gmail.com>
 License: MIT
@@ -50,15 +50,16 @@ Dynamic: license-file
 A YAML-configured data-fetching library for agentic applications.
 
 `fetchkit` collects posts and comments from sources (Hacker News, RSS/Atom, arXiv,
-GitHub, Lobsters) into a single canonical `Post` model, with de-duplication,
+GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon) into a single canonical `Post`
+model, with de-duplication,
 deterministic sorting, and a shared HTTP client with retries and rate limiting. It is
 designed as the data-collection layer for LLM/agent pipelines — feed it configs, get
 back clean typed data.
 
 - **YAML-first** configuration with strict validation: unknown or misspelled keys
   are rejected up front, so a bad config fails loudly instead of silently.
-- **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, and Lobsters — all
-  zero-auth.
+- **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange,
+  Bluesky, and Mastodon — all zero-auth.
 - **Relative time windows**: say `window: "last 6 hours"` instead of computing
   timestamps.
 - **Open `metadata`**: each `Post` carries a `metadata` dict for source-specific
@@ -73,6 +74,9 @@ back clean typed data.
   no Python required.
 - **RSS feed discovery**: `fetchkit discover "<use case>"` maps a natural-language
   query onto real RSS feeds an agent can fetch — closing the "which feed URL?" gap.
+- **Per-source discovery**: `fetchkit suggest <source>` lists the selectable knobs
+  for any source (tags, sites, arXiv categories, trending hashtags, Bluesky feeds) —
+  all zero-auth — so an agent can fill a config without guessing.
 
 ## Install
 
@@ -107,6 +111,15 @@ fetchers:
     repos: ["python/cpython", "pydantic/pydantic"]
   - type: lobsters
     listing: hottest
+  - type: stackexchange
+    site: stackoverflow
+    tagged: ["python", "asyncio"]
+  - type: bluesky
+    resource: search
+    query: "large language models"
+  - type: mastodon
+    instance: mastodon.social
+    tag: ai
   - type: rss
     feeds:
       - url: "https://feeds.bbci.co.uk/news/rss.xml"
@@ -159,7 +172,7 @@ settings, every builtin fetcher's typed config, and the canonical `Post` output
 so a config can be written without knowing the YAML format out of band:
 
 ```bash
-fetchkit schema | jq '.fetchers | keys'     # ["arxiv","github","hackernews","lobsters","rss"]
+fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","github","hackernews","lobsters","mastodon","rss","stackexchange"]
 fetchkit schema -o schema.json              # write to a file; supports --compact too
 ```
 
@@ -194,11 +207,12 @@ result = collect_all(config)
 
 ## Discovering RSS feeds
 
-Every other source has a finite, nameable set of options (`fetchkit schema`
-enumerates them). RSS is the exception: a feed is an *arbitrary URL*, so an agent
-that wants "central-bank policy" or "rust programming" has no way to know which
-feeds exist. The optional `discovery` module closes that gap — give it a
+RSS is the hardest source to discover for: a feed is an *arbitrary URL*, so an
+agent that wants "central-bank policy" or "rust programming" has no way to know
+which feeds exist. The optional `discovery` module closes that gap — give it a
 use case, get back ranked feeds you can drop straight into an `rss` fetcher.
+(The other sources have their own, simpler discovery surface — see
+[Discoverability for every source](#discoverability-for-every-source) below.)
 
 ```python
 from fetchkit.discovery import discover, to_rss_config
@@ -255,6 +269,40 @@ not the live article stream — for the latter, fetch the feed with the `rss`
 fetcher and match individual posts. Catalog quality is the main lever on result
 quality; see `src/fetchkit/discovery/data/CATALOG.md` to extend it.
 
+## Discoverability for every source
+
+`discover` solves RSS's "which feed URL?" problem. The other sources have a
+*different* discoverability gap — *which tag / site / instance / feed do I put in
+the config?* — answered by `fetchkit suggest <source>` (and `run_suggester()` in
+Python). Each suggester is **no-auth** and returns JSON-ready rows:
+
+```bash
+fetchkit suggest lobsters                                  # all Lobsters tags
+fetchkit suggest stackexchange --site stackoverflow        # popular SO tags
+fetchkit suggest stackexchange --what sites                # available SE sites
+fetchkit suggest arxiv --query vision                      # matching arXiv categories
+fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to watch
+fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
+fetchkit suggest bluesky                                   # popular custom feeds
+fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
+fetchkit suggest rss --query "AI safety news"              # delegates to discover()
+```
+
+| Source | `suggest` returns | Fills config field |
+|--------|-------------------|--------------------|
+| `hackernews`    | sort orders (static — HN has no tags)         | `posts.order` |
+| `rss`           | ranked feeds (delegates to `discover`)        | `feeds` |
+| `arxiv`         | category codes + names (static taxonomy)      | `categories` |
+| `github`        | popular `owner/name` repos for a query        | `repos` |
+| `lobsters`      | all tags (from `/tags.json`)                  | `tag` |
+| `stackexchange` | popular tags, or sites (`--what sites`)       | `tagged` / `site` |
+| `bluesky`       | popular feeds, or actors (`--what actors`)    | `actor` |
+| `mastodon`      | trending hashtags on an instance              | `tag` / `instance` |
+
+> Most suggesters call the source's live API (all no-auth); `hackernews`/`arxiv`
+> are static, and `rss` reuses the offline catalog ranker. Network suggesters fail
+> gracefully (the CLI reports the error on stderr and exits non-zero).
+
 ## Configuration reference
 
 ### Top-level (`FetchKitConfig`)
@@ -352,6 +400,47 @@ The lobste.rs public JSON endpoints (no auth). Tags are preserved in `post.metad
 | `tag`      | null    | Restrict to a single tag (uses `/t/<tag>.json`) |
 | `max_items`| 50      | 1–200                                           |
 
+### Stack Exchange (`type: stackexchange`)
+
+The Stack Exchange API (no auth; anonymous access is capped at 300 requests/day/IP).
+Questions become `Post`s; with `comments.fetch: true`, top answers are attached as
+`Comment`s. Tags, answer count, and the accepted-answer id are kept in `post.metadata`.
+
+| Field      | Default        | Notes                                                       |
+|------------|----------------|-------------------------------------------------------------|
+| `site`     | stackoverflow  | API site, e.g. `serverfault`, `superuser`, `askubuntu`      |
+| `tagged`   | `[]`           | Questions carrying ALL of these tags (joined with `;`)      |
+| `query`    | null           | Free-text search (uses `/search/advanced`)                  |
+| `posts`    | —              | `max_items` (1–500) and `order` (`top`→votes, `new`→creation) |
+| `comments` | —              | `fetch`/`max_items` — answers attached as comments          |
+
+### Bluesky (`type: bluesky`)
+
+The public Bluesky AppView (`public.api.bsky.app`), unauthenticated by design.
+`uri`, `cid`, and `langs` are kept in `post.metadata`; likes map to `score` and
+replies to `comment_count`.
+
+| Field      | Default | Notes                                                        |
+|------------|---------|--------------------------------------------------------------|
+| `resource` | search  | `search` (full-text) or `author_feed`                        |
+| `query`    | null    | Search query — required for `resource: search`               |
+| `actor`    | null    | Handle/DID — required for `resource: author_feed`            |
+| `max_items`| 50      | 1–500 (paginated, 100/page)                                  |
+
+### Mastodon (`type: mastodon`)
+
+Public and hashtag timelines on any instance (no auth, when the instance keeps
+public preview enabled — otherwise requests return 401). HTML content is reduced to
+plain text; tags, instance, and visibility are kept in `post.metadata`.
+
+| Field      | Default          | Notes                                                  |
+|------------|------------------|--------------------------------------------------------|
+| `instance` | mastodon.social  | Instance host without scheme, e.g. `fosstodon.org`     |
+| `resource` | tag              | `tag` (`/timelines/tag/<tag>`) or `public`             |
+| `tag`      | null             | Hashtag without `#` — required for `resource: tag`     |
+| `local`    | false            | Restrict to statuses originating on this instance      |
+| `max_items`| 50               | 1–200 (paginated, 40/page)                             |
+
 ### HTTP (`http:`)
 
 | Field                 | Default                  | Notes                                              |
@@ -384,6 +473,15 @@ validation, and tests. Add a new source via a PR or a local fork in four steps:
 
 3. Import the module in `src/fetchkit/fetchers/__init__.py` so it registers on import.
 4. Add tests (mock HTTP with the `responses` library — see `tests/fetchers/`).
+5. *(Optional)* register a discovery helper so agents can find your source's knobs:
+
+   ```python
+   from fetchkit.fetchers.suggest_registry import register_suggester
+
+   @register_suggester("mysource")
+   def suggest(*, query=None, limit=20, **kwargs) -> list[dict]:
+       return [{"tag": "…"}]   # JSON-ready rows; surfaced via `fetchkit suggest mysource`
+   ```
 
 Use `Post.metadata` for any source-specific fields that don't map to the canonical
 columns.
@@ -394,6 +492,7 @@ columns.
 class Post(BaseModel):
     id: str                       # unique within source
     source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
+                                  #   | "stackexchange" | "bluesky" | "mastodon"
     title: str | None
     text: str | None              # body / content
     url: str | None               # external link
@@ -410,8 +509,9 @@ All datetimes are normalized to UTC. Posts are deduplicated by `(source, id)` an
 sorted descending by `(created_at, id)` for deterministic output.
 
 > **`score` is source-relative.** Each source defines it differently — Hacker News
-> points, Lobsters score, GitHub stars (for `search_repos`), and `None` for arXiv
-> and RSS. The values are **not comparable across sources**, so don't rank a mixed
+> points, Lobsters score, GitHub stars (for `search_repos`), Stack Exchange question
+> score, Bluesky likes, Mastodon favourites, and `None` for arXiv and RSS. The values
+> are **not comparable across sources**, so don't rank a mixed
 > feed by `score` directly. Compare within a single `source`, or use a source-aware
 > ranking of your own. Output is ordered by recency (`created_at`), not by `score`.
 

{fetchkit_agents-0.2.0 → fetchkit_agents-0.3.0}/README.md

@@ -8,15 +8,16 @@
 A YAML-configured data-fetching library for agentic applications.
 
 `fetchkit` collects posts and comments from sources (Hacker News, RSS/Atom, arXiv,
-GitHub, Lobsters) into a single canonical `Post` model, with de-duplication,
+GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon) into a single canonical `Post`
+model, with de-duplication,
 deterministic sorting, and a shared HTTP client with retries and rate limiting. It is
 designed as the data-collection layer for LLM/agent pipelines — feed it configs, get
 back clean typed data.
 
 - **YAML-first** configuration with strict validation: unknown or misspelled keys
   are rejected up front, so a bad config fails loudly instead of silently.
-- **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, and Lobsters — all
-  zero-auth.
+- **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange,
+  Bluesky, and Mastodon — all zero-auth.
 - **Relative time windows**: say `window: "last 6 hours"` instead of computing
   timestamps.
 - **Open `metadata`**: each `Post` carries a `metadata` dict for source-specific
@@ -31,6 +32,9 @@ back clean typed data.
   no Python required.
 - **RSS feed discovery**: `fetchkit discover "<use case>"` maps a natural-language
   query onto real RSS feeds an agent can fetch — closing the "which feed URL?" gap.
+- **Per-source discovery**: `fetchkit suggest <source>` lists the selectable knobs
+  for any source (tags, sites, arXiv categories, trending hashtags, Bluesky feeds) —
+  all zero-auth — so an agent can fill a config without guessing.
 
 ## Install
 
@@ -65,6 +69,15 @@ fetchers:
     repos: ["python/cpython", "pydantic/pydantic"]
   - type: lobsters
     listing: hottest
+  - type: stackexchange
+    site: stackoverflow
+    tagged: ["python", "asyncio"]
+  - type: bluesky
+    resource: search
+    query: "large language models"
+  - type: mastodon
+    instance: mastodon.social
+    tag: ai
   - type: rss
     feeds:
       - url: "https://feeds.bbci.co.uk/news/rss.xml"
@@ -117,7 +130,7 @@ settings, every builtin fetcher's typed config, and the canonical `Post` output
 so a config can be written without knowing the YAML format out of band:
 
 ```bash
-fetchkit schema | jq '.fetchers | keys'     # ["arxiv","github","hackernews","lobsters","rss"]
+fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","github","hackernews","lobsters","mastodon","rss","stackexchange"]
 fetchkit schema -o schema.json              # write to a file; supports --compact too
 ```
 
@@ -152,11 +165,12 @@ result = collect_all(config)
 
 ## Discovering RSS feeds
 
-Every other source has a finite, nameable set of options (`fetchkit schema`
-enumerates them). RSS is the exception: a feed is an *arbitrary URL*, so an agent
-that wants "central-bank policy" or "rust programming" has no way to know which
-feeds exist. The optional `discovery` module closes that gap — give it a
+RSS is the hardest source to discover for: a feed is an *arbitrary URL*, so an
+agent that wants "central-bank policy" or "rust programming" has no way to know
+which feeds exist. The optional `discovery` module closes that gap — give it a
 use case, get back ranked feeds you can drop straight into an `rss` fetcher.
+(The other sources have their own, simpler discovery surface — see
+[Discoverability for every source](#discoverability-for-every-source) below.)
 
 ```python
 from fetchkit.discovery import discover, to_rss_config
@@ -213,6 +227,40 @@ not the live article stream — for the latter, fetch the feed with the `rss`
 fetcher and match individual posts. Catalog quality is the main lever on result
 quality; see `src/fetchkit/discovery/data/CATALOG.md` to extend it.
 
+## Discoverability for every source
+
+`discover` solves RSS's "which feed URL?" problem. The other sources have a
+*different* discoverability gap — *which tag / site / instance / feed do I put in
+the config?* — answered by `fetchkit suggest <source>` (and `run_suggester()` in
+Python). Each suggester is **no-auth** and returns JSON-ready rows:
+
+```bash
+fetchkit suggest lobsters                                  # all Lobsters tags
+fetchkit suggest stackexchange --site stackoverflow        # popular SO tags
+fetchkit suggest stackexchange --what sites                # available SE sites
+fetchkit suggest arxiv --query vision                      # matching arXiv categories
+fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to watch
+fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
+fetchkit suggest bluesky                                   # popular custom feeds
+fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
+fetchkit suggest rss --query "AI safety news"              # delegates to discover()
+```
+
+| Source | `suggest` returns | Fills config field |
+|--------|-------------------|--------------------|
+| `hackernews`    | sort orders (static — HN has no tags)         | `posts.order` |
+| `rss`           | ranked feeds (delegates to `discover`)        | `feeds` |
+| `arxiv`         | category codes + names (static taxonomy)      | `categories` |
+| `github`        | popular `owner/name` repos for a query        | `repos` |
+| `lobsters`      | all tags (from `/tags.json`)                  | `tag` |
+| `stackexchange` | popular tags, or sites (`--what sites`)       | `tagged` / `site` |
+| `bluesky`       | popular feeds, or actors (`--what actors`)    | `actor` |
+| `mastodon`      | trending hashtags on an instance              | `tag` / `instance` |
+
+> Most suggesters call the source's live API (all no-auth); `hackernews`/`arxiv`
+> are static, and `rss` reuses the offline catalog ranker. Network suggesters fail
+> gracefully (the CLI reports the error on stderr and exits non-zero).
+
 ## Configuration reference
 
 ### Top-level (`FetchKitConfig`)
@@ -310,6 +358,47 @@ The lobste.rs public JSON endpoints (no auth). Tags are preserved in `post.metad
 | `tag`      | null    | Restrict to a single tag (uses `/t/<tag>.json`) |
 | `max_items`| 50      | 1–200                                           |
 
+### Stack Exchange (`type: stackexchange`)
+
+The Stack Exchange API (no auth; anonymous access is capped at 300 requests/day/IP).
+Questions become `Post`s; with `comments.fetch: true`, top answers are attached as
+`Comment`s. Tags, answer count, and the accepted-answer id are kept in `post.metadata`.
+
+| Field      | Default        | Notes                                                       |
+|------------|----------------|-------------------------------------------------------------|
+| `site`     | stackoverflow  | API site, e.g. `serverfault`, `superuser`, `askubuntu`      |
+| `tagged`   | `[]`           | Questions carrying ALL of these tags (joined with `;`)      |
+| `query`    | null           | Free-text search (uses `/search/advanced`)                  |
+| `posts`    | —              | `max_items` (1–500) and `order` (`top`→votes, `new`→creation) |
+| `comments` | —              | `fetch`/`max_items` — answers attached as comments          |
+
+### Bluesky (`type: bluesky`)
+
+The public Bluesky AppView (`public.api.bsky.app`), unauthenticated by design.
+`uri`, `cid`, and `langs` are kept in `post.metadata`; likes map to `score` and
+replies to `comment_count`.
+
+| Field      | Default | Notes                                                        |
+|------------|---------|--------------------------------------------------------------|
+| `resource` | search  | `search` (full-text) or `author_feed`                        |
+| `query`    | null    | Search query — required for `resource: search`               |
+| `actor`    | null    | Handle/DID — required for `resource: author_feed`            |
+| `max_items`| 50      | 1–500 (paginated, 100/page)                                  |
+
+### Mastodon (`type: mastodon`)
+
+Public and hashtag timelines on any instance (no auth, when the instance keeps
+public preview enabled — otherwise requests return 401). HTML content is reduced to
+plain text; tags, instance, and visibility are kept in `post.metadata`.
+
+| Field      | Default          | Notes                                                  |
+|------------|------------------|--------------------------------------------------------|
+| `instance` | mastodon.social  | Instance host without scheme, e.g. `fosstodon.org`     |
+| `resource` | tag              | `tag` (`/timelines/tag/<tag>`) or `public`             |
+| `tag`      | null             | Hashtag without `#` — required for `resource: tag`     |
+| `local`    | false            | Restrict to statuses originating on this instance      |
+| `max_items`| 50               | 1–200 (paginated, 40/page)                             |
+
 ### HTTP (`http:`)
 
 | Field                 | Default                  | Notes                                              |
@@ -342,6 +431,15 @@ validation, and tests. Add a new source via a PR or a local fork in four steps:
 
 3. Import the module in `src/fetchkit/fetchers/__init__.py` so it registers on import.
 4. Add tests (mock HTTP with the `responses` library — see `tests/fetchers/`).
+5. *(Optional)* register a discovery helper so agents can find your source's knobs:
+
+   ```python
+   from fetchkit.fetchers.suggest_registry import register_suggester
+
+   @register_suggester("mysource")
+   def suggest(*, query=None, limit=20, **kwargs) -> list[dict]:
+       return [{"tag": "…"}]   # JSON-ready rows; surfaced via `fetchkit suggest mysource`
+   ```
 
 Use `Post.metadata` for any source-specific fields that don't map to the canonical
 columns.
@@ -352,6 +450,7 @@ columns.
 class Post(BaseModel):
     id: str                       # unique within source
     source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
+                                  #   | "stackexchange" | "bluesky" | "mastodon"
     title: str | None
     text: str | None              # body / content
     url: str | None               # external link
@@ -368,8 +467,9 @@ All datetimes are normalized to UTC. Posts are deduplicated by `(source, id)` an
 sorted descending by `(created_at, id)` for deterministic output.
 
 > **`score` is source-relative.** Each source defines it differently — Hacker News
-> points, Lobsters score, GitHub stars (for `search_repos`), and `None` for arXiv
-> and RSS. The values are **not comparable across sources**, so don't rank a mixed
+> points, Lobsters score, GitHub stars (for `search_repos`), Stack Exchange question
+> score, Bluesky likes, Mastodon favourites, and `None` for arXiv and RSS. The values
+> are **not comparable across sources**, so don't rank a mixed
 > feed by `score` directly. Compare within a single `source`, or use a source-aware
 > ranking of your own. Output is ordered by recency (`created_at`), not by `score`.
 

{fetchkit_agents-0.2.0 → fetchkit_agents-0.3.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 # Distribution name on PyPI. The importable package and CLI are still `fetchkit`;
 # the `fetchkit` PyPI name was already taken by an unrelated project.
 name = "fetchkit-agents"
-version = "0.2.0"
+version = "0.3.0"
 description = "A YAML-configured data fetching library for agentic applications."
 readme = "README.md"
 requires-python = ">=3.10"

{fetchkit_agents-0.2.0 → fetchkit_agents-0.3.0}/src/fetchkit/__init__.py

@@ -2,9 +2,9 @@
 fetchkit — a YAML-configured data fetching library for agentic applications.
 
 Fetch posts and comments from sources (Hacker News, RSS/Atom, arXiv, GitHub,
-Lobsters) into a single canonical ``Post`` model, with deduplication, sorting,
-and a typed config. Designed as a data-collection layer for agentic / LLM
-applications.
+Lobsters, Stack Exchange, Bluesky, Mastodon) into a single canonical ``Post``
+model, with deduplication, sorting, and a typed config. Designed as a
+data-collection layer for agentic / LLM applications.
 
 Quick start::
 
@@ -19,7 +19,7 @@ See the README for the full configuration reference and the guide to adding
 fetchers.
 """
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 from typing import TYPE_CHECKING, Any
 
@@ -27,6 +27,12 @@ from fetchkit.collector import collect_all
 from fetchkit.config_loader import load_config, ConfigError
 from fetchkit.fetchers.base import Fetcher, FetcherResult
 from fetchkit.fetchers.registry import register_fetcher, get_fetcher
+from fetchkit.fetchers.suggest_registry import (
+    register_suggester,
+    get_suggester,
+    list_suggesters,
+    run_suggester,
+)
 from fetchkit.http import (
     HttpClient,
     RateLimiter,
@@ -46,6 +52,9 @@ from fetchkit.schemas.fetcher import (
     ArxivFetchConfig,
     GitHubFetchConfig,
     LobstersFetchConfig,
+    StackExchangeFetchConfig,
+    BlueskyFetchConfig,
+    MastodonFetchConfig,
     FetcherConfig,
 )
 from fetchkit.schemas.collector import CollectorResult
@@ -85,6 +94,10 @@ __all__ = [
     "FetcherResult",
     "register_fetcher",
     "get_fetcher",
+    "register_suggester",
+    "get_suggester",
+    "list_suggesters",
+    "run_suggester",
     # HTTP
     "HttpClient",
     "RateLimiter",
@@ -105,6 +118,9 @@ __all__ = [
     "ArxivFetchConfig",
     "GitHubFetchConfig",
     "LobstersFetchConfig",
+    "StackExchangeFetchConfig",
+    "BlueskyFetchConfig",
+    "MastodonFetchConfig",
     "FetcherConfig",
     "CollectorResult",
     "FetchKitConfig",

{fetchkit_agents-0.2.0 → fetchkit_agents-0.3.0}/src/fetchkit/cli.py

@@ -10,8 +10,9 @@ YAML spec and parse structured JSON from stdout::
     fetchkit schema                          # JSON Schema for every config/output model
     fetchkit discover "AI safety news"       # find RSS feeds for a use case
     fetchkit find-feeds https://example.com  # autodiscover feeds from a site
+    fetchkit suggest lobsters                # list what you can query for a source
 
-stdout carries only JSON (for ``run``, ``schema``, ``discover``, and
+stdout carries only JSON (for ``run``, ``schema``, ``discover``, ``suggest``, and
 ``find-feeds``) so it is safe to pipe into ``jq`` or parse programmatically.
 Diagnostics and ``--verbose`` logging go to stderr. ``schema`` lets an agent
 discover the available fetchers and their options without being told the YAML
@@ -163,6 +164,31 @@ def _cmd_discover(args: argparse.Namespace) -> int:
     return 0
 
 
+def _cmd_suggest(args: argparse.Namespace) -> int:
+    """List what you can query for a source (tags, sites, categories, feeds); emit JSON."""
+    from fetchkit.fetchers import run_suggester
+
+    params: dict[str, Any] = {"limit": args.limit}
+    for key in ("query", "site", "instance", "what"):
+        value = getattr(args, key)
+        if value is not None:
+            params[key] = value
+
+    try:
+        suggestions = run_suggester(args.source, **params)
+    except ValueError as exc:
+        print(str(exc), file=sys.stderr)
+        return 2
+    except Exception as exc:  # network/HTTP failure — report, don't crash
+        print(f"suggest failed for {args.source}: {exc}", file=sys.stderr)
+        return 2
+
+    indent = None if args.compact else args.indent
+    payload = {"source": args.source, "count": len(suggestions), "suggestions": suggestions}
+    _emit_json(payload, indent, args.output)
+    return 0
+
+
 def _cmd_find_feeds(args: argparse.Namespace) -> int:
     """Autodiscover RSS/Atom feeds from one or more site URLs; emit JSON."""
     from fetchkit.discovery import find_feeds
@@ -294,6 +320,52 @@ def build_parser() -> argparse.ArgumentParser:
     )
     discover.set_defaults(func=_cmd_discover)
 
+    # `suggest` is the cross-source analog of `discover` (which is RSS-only): it
+    # lists the selectable knobs for any source — SE sites/tags, arXiv categories,
+    # Lobsters tags, Mastodon trending tags, Bluesky feeds/actors, RSS feeds.
+    from fetchkit.fetchers import list_suggesters
+
+    suggest = sub.add_parser(
+        "suggest",
+        help="List what you can query for a source (tags, sites, categories, feeds).",
+    )
+    suggest.add_argument(
+        "source", choices=list_suggesters(),
+        help="Source to get discovery suggestions for.",
+    )
+    suggest.add_argument(
+        "--query", default=None, metavar="Q",
+        help="Filter/seed term (e.g. RSS use case, GitHub repo search, Bluesky actor search).",
+    )
+    suggest.add_argument(
+        "--site", default=None,
+        help="Stack Exchange site for tag suggestions (default: stackoverflow).",
+    )
+    suggest.add_argument(
+        "--instance", default=None,
+        help="Mastodon instance host for trending tags (default: mastodon.social).",
+    )
+    suggest.add_argument(
+        "--what", default=None,
+        help="Sub-selection: stackexchange 'sites'|'tags'; bluesky 'feeds'|'actors'.",
+    )
+    suggest.add_argument(
+        "--limit", type=int, default=20, help="Max suggestions to return (default: 20).",
+    )
+    suggest.add_argument(
+        "-o", "--output", default=None, metavar="PATH",
+        help="Write JSON to this file instead of stdout.",
+    )
+    suggest.add_argument(
+        "--indent", type=int, default=2,
+        help="JSON indentation for pretty output (default: 2).",
+    )
+    suggest.add_argument(
+        "--compact", action="store_true",
+        help="Emit single-line JSON (overrides --indent).",
+    )
+    suggest.set_defaults(func=_cmd_suggest)
+
     find_feeds = sub.add_parser(
         "find-feeds",
         help="Autodiscover RSS/Atom feeds from one or more site URLs.",

fetchkit_agents-0.3.0/src/fetchkit/fetchers/__init__.py

@@ -0,0 +1,28 @@
+"""Fetcher package exports and builtin fetcher registration side effects."""
+
+from .registry import get_fetcher, register_fetcher
+from .suggest_registry import (
+    get_suggester,
+    list_suggesters,
+    register_suggester,
+    run_suggester,
+)
+# Import built-in fetchers to ensure they register themselves (fetchers and
+# suggesters) on import.
+from . import hackernews
+from . import rss
+from . import arxiv
+from . import github
+from . import lobsters
+from . import stackexchange
+from . import bluesky
+from . import mastodon
+
+__all__ = [
+    "get_fetcher",
+    "register_fetcher",
+    "get_suggester",
+    "list_suggesters",
+    "register_suggester",
+    "run_suggester",
+]

{fetchkit_agents-0.2.0 → fetchkit_agents-0.3.0}/src/fetchkit/fetchers/arxiv.py

@@ -18,6 +18,7 @@ from fetchkit.schemas.post import Post, Source
 from fetchkit.schemas.fetcher import ArxivFetchConfig, FetcherConfig
 from fetchkit.fetchers.base import FetcherResult
 from fetchkit.fetchers.registry import register_fetcher
+from fetchkit.fetchers.suggest_registry import register_suggester
 
 logger = logging.getLogger(__name__)
 
@@ -25,6 +26,39 @@ ARXIV_API_URL = "https://export.arxiv.org/api/query"
 SOURCE_NAME = Source.ARXIV
 DEFAULT_TIMEOUT_S = 30  # arXiv can be slow
 
+# arXiv's category taxonomy is fixed and well-known, so discoverability is a static
+# lookup (no network/auth). A representative cross-discipline subset of the most-used
+# categories — enough for an agent to pick a `categories` value without guessing.
+ARXIV_CATEGORIES: dict[str, str] = {
+    "cs.AI": "Artificial Intelligence",
+    "cs.CL": "Computation and Language (NLP)",
+    "cs.CV": "Computer Vision and Pattern Recognition",
+    "cs.LG": "Machine Learning",
+    "cs.CR": "Cryptography and Security",
+    "cs.DC": "Distributed, Parallel, and Cluster Computing",
+    "cs.DS": "Data Structures and Algorithms",
+    "cs.HC": "Human-Computer Interaction",
+    "cs.IR": "Information Retrieval",
+    "cs.NE": "Neural and Evolutionary Computing",
+    "cs.RO": "Robotics",
+    "cs.SE": "Software Engineering",
+    "cs.SY": "Systems and Control",
+    "stat.ML": "Machine Learning (Statistics)",
+    "stat.ME": "Methodology (Statistics)",
+    "math.OC": "Optimization and Control",
+    "math.PR": "Probability",
+    "math.ST": "Statistics Theory",
+    "eess.AS": "Audio and Speech Processing",
+    "eess.IV": "Image and Video Processing",
+    "eess.SP": "Signal Processing",
+    "physics.comp-ph": "Computational Physics",
+    "q-bio.NC": "Neurons and Cognition",
+    "q-bio.QM": "Quantitative Methods",
+    "q-fin.CP": "Computational Finance",
+    "q-fin.PM": "Portfolio Management",
+    "econ.EM": "Econometrics",
+}
+
 
 def _build_search_query(config: ArxivFetchConfig) -> str:
     """Combine categories and free-text query into an arXiv ``search_query``."""
@@ -122,6 +156,16 @@ def fetch_posts(config: ArxivFetchConfig) -> list[Post]:
     return posts
 
 
+@register_suggester("arxiv")
+def suggest(*, query: Optional[str] = None, limit: int = 100, **kwargs: Any) -> list[dict[str, Any]]:
+    """Discoverability for arXiv: list selectable categories (filtered by ``query``)."""
+    items = [{"category": code, "name": name} for code, name in ARXIV_CATEGORIES.items()]
+    if query:
+        q = query.lower()
+        items = [it for it in items if q in it["category"].lower() or q in it["name"].lower()]
+    return items[:limit]
+
+
 @register_fetcher("arxiv")
 def fetch(config: FetcherConfig) -> FetcherResult:
     """Fetcher protocol implementation for arXiv."""