fetchkit-agents 0.3.0__tar.gz → 0.4.0__tar.gz

{fetchkit_agents-0.3.0 → fetchkit_agents-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fetchkit-agents
-Version: 0.3.0
+Version: 0.4.0
 Summary: A YAML-configured data fetching library for agentic applications.
 Author-email: mete morris <morrismete@gmail.com>
 License: MIT
@@ -50,7 +50,8 @@ Dynamic: license-file
 A YAML-configured data-fetching library for agentic applications.
 
 `fetchkit` collects posts and comments from sources (Hacker News, RSS/Atom, arXiv,
-GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon) into a single canonical `Post`
+GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT,
+SEC EDGAR, Polymarket, Manifold) into a single canonical `Post`
 model, with de-duplication,
 deterministic sorting, and a shared HTTP client with retries and rate limiting. It is
 designed as the data-collection layer for LLM/agent pipelines — feed it configs, get
@@ -59,7 +60,8 @@ back clean typed data.
 - **YAML-first** configuration with strict validation: unknown or misspelled keys
   are rejected up front, so a bad config fails loudly instead of silently.
 - **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange,
-  Bluesky, and Mastodon — all zero-auth.
+  Bluesky, Mastodon, Reddit, Google News, GDELT (global news firehose), SEC EDGAR
+  (securities filings), and Polymarket/Manifold (prediction markets) — all zero-auth.
 - **Relative time windows**: say `window: "last 6 hours"` instead of computing
   timestamps.
 - **Open `metadata`**: each `Post` carries a `metadata` dict for source-specific
@@ -120,6 +122,20 @@ fetchers:
   - type: mastodon
     instance: mastodon.social
     tag: ai
+  - type: edgar
+    companies: ["AAPL", "MSFT"]
+    forms: ["8-K", "10-Q"]
+  - type: reddit
+    subreddits: ["news", "wallstreetbets"]
+    listing: new
+  - type: googlenews
+    resource: search
+    query: "nvidia earnings when:6h"
+  - type: gdelt
+    query: '"interest rates"'
+    languages: ["english"]
+  - type: polymarket
+    query: "fed"
   - type: rss
     feeds:
       - url: "https://feeds.bbci.co.uk/news/rss.xml"
@@ -156,33 +172,49 @@ Python. `run` prints **only** JSON to stdout (diagnostics go to stderr), so it p
 cleanly into `jq`:
 
 ```bash
-fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "errors"}
+fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "sources", "errors"}
 fetchkit run config.yaml -o out.json        # write JSON to a file (stdout stays empty)
 fetchkit run config.yaml --window "6h"      # override the time window at runtime
 fetchkit run config.yaml --compact          # single-line JSON
 fetchkit run config.yaml --fail-on-error    # exit 1 if any source failed
-fetchkit validate config.yaml               # validate a config without fetching
+echo "$CONFIG_YAML" | fetchkit run -        # read the config from stdin
+fetchkit validate config.yaml               # validate a config without fetching (also accepts -)
 fetchkit schema                             # JSON Schema for every config/output model
 python -m fetchkit run config.yaml          # module form, identical behavior
 ```
 
+`fetchkit run --help` and `fetchkit validate --help` print the YAML config format
+inline (top-level keys, a minimal example, and the fetcher type list), so simple
+configs need no schema round-trip.
+
 `fetchkit schema` lets an agent discover what it can fetch — it prints the JSON
 Schema (with field descriptions) for the top-level config, the shared HTTP
 settings, every builtin fetcher's typed config, and the canonical `Post` output —
-so a config can be written without knowing the YAML format out of band:
+so a config can be written without knowing the YAML format out of band. Pass a
+section name to print just that part instead of the full ~50 KB document:
 
 ```bash
-fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","github","hackernews","lobsters","mastodon","rss","stackexchange"]
+fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","edgar","gdelt","github","googlenews","hackernews","lobsters","manifold","mastodon","polymarket","reddit","rss","stackexchange"]
+fetchkit schema config                      # just the top-level config schema
+fetchkit schema rss                         # just one fetcher's config schema
+fetchkit schema post                        # just the output Post model
 fetchkit schema -o schema.json              # write to a file; supports --compact too
 ```
 
-Output shape:
+Output shape — `sources` reports per-fetcher success/failure (one entry per
+enabled fetcher, in config order), so partial failures are visible without
+diffing domains against `posts`. A source can partially fail, e.g. one dead
+feed among many: it keeps its posts, records the error, and reports `ok: false`.
 
 ```json
 {
   "count": 2,
   "posts": [ { "id": "...", "source": "rss", "title": "...", "url": "...", "created_at": "..." } ],
-  "errors": [ { "source": "hackernews", "error": "..." } ]
+  "sources": [
+    { "source": "rss", "name": null, "ok": false, "posts": 2,
+      "errors": ["Failed to fetch feed https://dead.example/feed.xml: ..."] }
+  ],
+  "errors": [ { "source": "rss", "error": "Failed to fetch feed https://dead.example/feed.xml: ..." } ]
 }
 ```
 
@@ -269,6 +301,14 @@ not the live article stream — for the latter, fetch the feed with the `rss`
 fetcher and match individual posts. Catalog quality is the main lever on result
 quality; see `src/fetchkit/discovery/data/CATALOG.md` to extend it.
 
+Because discovery only *ranks* candidates it already holds, a niche query (a
+specific region, product, or community) can come back filled with generic feeds
+that matched only the query's broad terms. The CLI flags this instead of letting
+it pass: the JSON payload carries `unmatched_terms` (query terms no returned
+feed mentions) plus a `hint`, and the same hint is printed to stderr. When you
+see it, feed discovery better candidates — `--from-urls` with sites from your
+own web search, `fetchkit find-feeds <site-url>`, or `--external`.
+
 ## Discoverability for every source
 
 `discover` solves RSS's "which feed URL?" problem. The other sources have a
@@ -285,6 +325,12 @@ fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to
 fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
 fetchkit suggest bluesky                                   # popular custom feeds
 fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
+fetchkit suggest edgar --query apple                       # tickers/CIKs for an edgar config
+fetchkit suggest reddit --query finance                    # subreddits for a topic
+fetchkit suggest gdelt --query country                     # GDELT languages/countries/operators
+fetchkit suggest googlenews                                # Google News topics + query operators
+fetchkit suggest polymarket --query fed                    # most active prediction markets
+fetchkit suggest manifold --query "rate cut"               # Manifold markets for a topic
 fetchkit suggest rss --query "AI safety news"              # delegates to discover()
 ```
 
@@ -298,6 +344,12 @@ fetchkit suggest rss --query "AI safety news"              # delegates to discov
 | `stackexchange` | popular tags, or sites (`--what sites`)       | `tagged` / `site` |
 | `bluesky`       | popular feeds, or actors (`--what actors`)    | `actor` |
 | `mastodon`      | trending hashtags on an instance              | `tag` / `instance` |
+| `edgar`         | company tickers/CIKs matching a query         | `companies` |
+| `reddit`        | subreddits matching a query                   | `subreddits` |
+| `gdelt`         | languages, country codes, query operators (static) | `languages` / `countries` / `query` |
+| `googlenews`    | topic sections + query operators (static)     | `topic` / `query` |
+| `polymarket`    | most active live markets (by 24h volume)      | `query` |
+| `manifold`      | live markets matching a query                 | `query` |
 
 > Most suggesters call the source's live API (all no-auth); `hackernews`/`arxiv`
 > are static, and `rss` reuses the offline catalog ranker. Network suggesters fail
@@ -319,7 +371,9 @@ Set the time window with **either** a relative `window` **or** both
 `start_time`/`end_time` — the two are mutually exclusive. **If you specify none of
 them, the window defaults to the last 24 hours.** `start_time <= end_time` is
 enforced, and unknown top-level keys are rejected. Per-fetcher
-`start_time`/`end_time` default to `None` and **inherit** the global window at runtime.
+`start_time`/`end_time` default to `None` and **inherit** the run window from these
+top-level keys at runtime (there is no `global:` key — the top level *is* the
+global scope).
 
 #### Relative windows
 
@@ -441,6 +495,111 @@ plain text; tags, instance, and visibility are kept in `post.metadata`.
 | `local`    | false            | Restrict to statuses originating on this instance      |
 | `max_items`| 50               | 1–200 (paginated, 40/page)                             |
 
+### SEC EDGAR (`type: edgar`)
+
+Securities filings from the SEC's free, keyless EDGAR APIs. `resource: filings`
+pulls recent filings per company from `data.sec.gov/submissions` (acceptance
+timestamps are near-real-time — new 8-Ks appear within minutes); `resource: search`
+runs EDGAR full-text search across all filers (date granularity only; the run
+window is passed to the API as a date range). Form type, CIK, accession number,
+and 8-K item codes are kept in `post.metadata`.
+
+The SEC's [fair-access policy](https://www.sec.gov/os/accessing-edgar-data) asks
+clients to identify themselves with contact info and stay under 10 req/s — set
+`user_agent` and consider `http.rate_limit_per_host`.
+
+| Field        | Default  | Notes                                                        |
+|--------------|----------|--------------------------------------------------------------|
+| `resource`   | filings  | `filings` (per-company recent) or `search` (full-text)       |
+| `companies`  | `[]`     | Tickers or CIKs — required for `resource: filings`           |
+| `forms`      | `[]`     | Form types to include, e.g. `["8-K", "10-Q"]` (empty = all)  |
+| `query`      | null     | Full-text query — required for `resource: search`            |
+| `user_agent` | null     | Descriptive UA with contact info, e.g. `myapp me@example.com`|
+| `max_items`  | 50       | 1–500                                                        |
+
+### Reddit (`type: reddit`)
+
+Reddit's public `.json` endpoints with a descriptive User-Agent. `listing: new`
+surfaces posts seconds after submission (engagement is near zero at that point by
+definition); `hot`/`top` carry engagement at the cost of recency. Score,
+comment count, and upvote ratio are captured as of fetch time. Subreddit, flair,
+and upvote ratio are kept in `post.metadata`.
+
+> **Terms note:** unauthenticated access is tolerated by Reddit at low volume
+> (~10 requests/minute per IP) but sits in a gray area of Reddit's Data API
+> terms, which formally expect registered OAuth clients. fetchkit issues one
+> request per fetch; keep polling rates low (`http.rate_limit_per_host`).
+> Reddit also blocks many datacenter/cloud IPs outright (403 regardless of
+> User-Agent), so this fetcher may work from a residential connection but not
+> from CI or a cloud host.
+
+| Field         | Default | Notes                                                       |
+|---------------|---------|-------------------------------------------------------------|
+| `subreddits`  | `[]`    | Without `r/` prefix; combined into one listing request      |
+| `listing`     | new     | `new`, `hot`, `top`, or `rising` (when `query` unset)       |
+| `query`       | null    | Search; restricted to `subreddits` when both are set        |
+| `time_filter` | day     | For `listing: top` and searches: `hour`…`all`               |
+| `max_items`   | 50      | 1–100 (single request)                                      |
+
+### GDELT (`type: gdelt`)
+
+The GDELT Project's DOC 2.0 API — machine-reads worldwide news media in 65
+languages, re-indexed every 15 minutes. A zero-auth global news firehose. Hits
+carry URL, title, timestamp, source domain/country/language (kept in
+`post.metadata`); there is no author, body text, or engagement, so those fields
+are `None`. The run window maps to the API's `startdatetime`/`enddatetime`.
+
+| Field       | Default | Notes                                                          |
+|-------------|---------|----------------------------------------------------------------|
+| `query`     | —       | Required; keywords/"phrases" plus operators (`domain:`, `tone<-5`) |
+| `languages` | `[]`    | Source languages, e.g. `["english", "spanish"]`                |
+| `countries` | `[]`    | Source countries (GDELT/FIPS codes), e.g. `["US", "UK"]`       |
+| `max_items` | 75      | 1–250                                                          |
+
+### Google News (`type: googlenews`)
+
+Google News' public RSS endpoints: top stories, search, or a topic section.
+Minutes-fresh headlines aggregated from thousands of publishers. The query
+language supports `when:2h` (recency), `site:`, quoted phrases, `-exclusions`,
+and `intitle:`. Links are Google News redirect URLs (they resolve to the
+publisher); the publisher name maps to `author`.
+
+| Field       | Default | Notes                                                          |
+|-------------|---------|----------------------------------------------------------------|
+| `resource`  | top     | `top`, `search` (needs `query`), or `topic` (needs `topic`)    |
+| `query`     | null    | e.g. `'nvidia earnings when:2h'`                               |
+| `topic`     | null    | `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE`, `HEALTH` |
+| `language`  | en-US   | UI language (`hl=`)                                            |
+| `country`   | US      | Edition (`gl=`/`ceid=`)                                        |
+| `max_items` | 50      | 1–100                                                          |
+
+### Polymarket (`type: polymarket`) and Manifold (`type: manifold`)
+
+Prediction markets as posts: the market question is the `title`, and live odds,
+volume, and close date land in `post.metadata` (`odds`/`probability`,
+`volume_24h`, …). Odds moving is often a signal about breaking news before
+articles are written. Both APIs are public read-only, no auth.
+
+> **Snapshot semantics:** markets are live state, not timestamped events, so
+> these two fetchers deliberately ignore the run window — an active market
+> created months ago is still current. `created_at` is the market's creation
+> time, which also means snapshot posts sort below newer article-shaped posts
+> in a mixed feed.
+
+| Field (`polymarket`) | Default | Notes                                            |
+|----------------------|---------|--------------------------------------------------|
+| `query`              | null    | Substring filter on the question, applied client-side over the top ~100 markets for the chosen `order` (not a full search) |
+| `order`              | volume  | `volume` (24h), `liquidity`, or `new`            |
+| `closed`             | false   | Include closed/resolved markets                  |
+| `max_items`          | 50      | 1–100                                            |
+
+| Field (`manifold`) | Default | Notes                                              |
+|--------------------|---------|----------------------------------------------------|
+| `query`            | null    | Full-text search; omit for most recent markets     |
+| `market_filter`    | open    | `open`, `closed`, `resolved`, `all` (searches)     |
+| `sort`             | score   | `score`, `newest`, `liquidity`, `24-hour-vol`      |
+| `max_items`        | 50      | 1–100                                              |
+
 ### HTTP (`http:`)
 
 | Field                 | Default                  | Notes                                              |
@@ -492,7 +651,8 @@ columns.
 class Post(BaseModel):
     id: str                       # unique within source
     source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
-                                  #   | "stackexchange" | "bluesky" | "mastodon"
+                                  #   | "stackexchange" | "bluesky" | "mastodon" | "edgar"
+                                  #   | "reddit" | "gdelt" | "googlenews" | "polymarket" | "manifold"
     title: str | None
     text: str | None              # body / content
     url: str | None               # external link
@@ -510,7 +670,9 @@ sorted descending by `(created_at, id)` for deterministic output.
 
 > **`score` is source-relative.** Each source defines it differently — Hacker News
 > points, Lobsters score, GitHub stars (for `search_repos`), Stack Exchange question
-> score, Bluesky likes, Mastodon favourites, and `None` for arXiv and RSS. The values
+> score, Bluesky likes, Mastodon favourites, Reddit score (fuzzed by Reddit, and
+> near zero on brand-new posts), and `None` for arXiv, RSS, Google News, GDELT,
+> EDGAR, and the prediction-market sources. The values
 > are **not comparable across sources**, so don't rank a mixed
 > feed by `score` directly. Compare within a single `source`, or use a source-aware
 > ranking of your own. Output is ordered by recency (`created_at`), not by `score`.

{fetchkit_agents-0.3.0 → fetchkit_agents-0.4.0}/README.md

@@ -8,7 +8,8 @@
 A YAML-configured data-fetching library for agentic applications.
 
 `fetchkit` collects posts and comments from sources (Hacker News, RSS/Atom, arXiv,
-GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon) into a single canonical `Post`
+GitHub, Lobsters, Stack Exchange, Bluesky, Mastodon, Reddit, Google News, GDELT,
+SEC EDGAR, Polymarket, Manifold) into a single canonical `Post`
 model, with de-duplication,
 deterministic sorting, and a shared HTTP client with retries and rate limiting. It is
 designed as the data-collection layer for LLM/agent pipelines — feed it configs, get
@@ -17,7 +18,8 @@ back clean typed data.
 - **YAML-first** configuration with strict validation: unknown or misspelled keys
   are rejected up front, so a bad config fails loudly instead of silently.
 - **Builtin fetchers**: Hacker News, RSS/Atom, arXiv, GitHub, Lobsters, Stack Exchange,
-  Bluesky, and Mastodon — all zero-auth.
+  Bluesky, Mastodon, Reddit, Google News, GDELT (global news firehose), SEC EDGAR
+  (securities filings), and Polymarket/Manifold (prediction markets) — all zero-auth.
 - **Relative time windows**: say `window: "last 6 hours"` instead of computing
   timestamps.
 - **Open `metadata`**: each `Post` carries a `metadata` dict for source-specific
@@ -78,6 +80,20 @@ fetchers:
   - type: mastodon
     instance: mastodon.social
     tag: ai
+  - type: edgar
+    companies: ["AAPL", "MSFT"]
+    forms: ["8-K", "10-Q"]
+  - type: reddit
+    subreddits: ["news", "wallstreetbets"]
+    listing: new
+  - type: googlenews
+    resource: search
+    query: "nvidia earnings when:6h"
+  - type: gdelt
+    query: '"interest rates"'
+    languages: ["english"]
+  - type: polymarket
+    query: "fed"
   - type: rss
     feeds:
       - url: "https://feeds.bbci.co.uk/news/rss.xml"
@@ -114,33 +130,49 @@ Python. `run` prints **only** JSON to stdout (diagnostics go to stderr), so it p
 cleanly into `jq`:
 
 ```bash
-fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "errors"}
+fetchkit run config.yaml                    # pretty JSON: {"count", "posts", "sources", "errors"}
 fetchkit run config.yaml -o out.json        # write JSON to a file (stdout stays empty)
 fetchkit run config.yaml --window "6h"      # override the time window at runtime
 fetchkit run config.yaml --compact          # single-line JSON
 fetchkit run config.yaml --fail-on-error    # exit 1 if any source failed
-fetchkit validate config.yaml               # validate a config without fetching
+echo "$CONFIG_YAML" | fetchkit run -        # read the config from stdin
+fetchkit validate config.yaml               # validate a config without fetching (also accepts -)
 fetchkit schema                             # JSON Schema for every config/output model
 python -m fetchkit run config.yaml          # module form, identical behavior
 ```
 
+`fetchkit run --help` and `fetchkit validate --help` print the YAML config format
+inline (top-level keys, a minimal example, and the fetcher type list), so simple
+configs need no schema round-trip.
+
 `fetchkit schema` lets an agent discover what it can fetch — it prints the JSON
 Schema (with field descriptions) for the top-level config, the shared HTTP
 settings, every builtin fetcher's typed config, and the canonical `Post` output —
-so a config can be written without knowing the YAML format out of band:
+so a config can be written without knowing the YAML format out of band. Pass a
+section name to print just that part instead of the full ~50 KB document:
 
 ```bash
-fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","github","hackernews","lobsters","mastodon","rss","stackexchange"]
+fetchkit schema | jq '.fetchers | keys'     # ["arxiv","bluesky","edgar","gdelt","github","googlenews","hackernews","lobsters","manifold","mastodon","polymarket","reddit","rss","stackexchange"]
+fetchkit schema config                      # just the top-level config schema
+fetchkit schema rss                         # just one fetcher's config schema
+fetchkit schema post                        # just the output Post model
 fetchkit schema -o schema.json              # write to a file; supports --compact too
 ```
 
-Output shape:
+Output shape — `sources` reports per-fetcher success/failure (one entry per
+enabled fetcher, in config order), so partial failures are visible without
+diffing domains against `posts`. A source can partially fail, e.g. one dead
+feed among many: it keeps its posts, records the error, and reports `ok: false`.
 
 ```json
 {
   "count": 2,
   "posts": [ { "id": "...", "source": "rss", "title": "...", "url": "...", "created_at": "..." } ],
-  "errors": [ { "source": "hackernews", "error": "..." } ]
+  "sources": [
+    { "source": "rss", "name": null, "ok": false, "posts": 2,
+      "errors": ["Failed to fetch feed https://dead.example/feed.xml: ..."] }
+  ],
+  "errors": [ { "source": "rss", "error": "Failed to fetch feed https://dead.example/feed.xml: ..." } ]
 }
 ```
 
@@ -227,6 +259,14 @@ not the live article stream — for the latter, fetch the feed with the `rss`
 fetcher and match individual posts. Catalog quality is the main lever on result
 quality; see `src/fetchkit/discovery/data/CATALOG.md` to extend it.
 
+Because discovery only *ranks* candidates it already holds, a niche query (a
+specific region, product, or community) can come back filled with generic feeds
+that matched only the query's broad terms. The CLI flags this instead of letting
+it pass: the JSON payload carries `unmatched_terms` (query terms no returned
+feed mentions) plus a `hint`, and the same hint is printed to stderr. When you
+see it, feed discovery better candidates — `--from-urls` with sites from your
+own web search, `fetchkit find-feeds <site-url>`, or `--external`.
+
 ## Discoverability for every source
 
 `discover` solves RSS's "which feed URL?" problem. The other sources have a
@@ -243,6 +283,12 @@ fetchkit suggest github --query "language:rust stars:>5000"  # popular repos to
 fetchkit suggest mastodon --instance fosstodon.org         # trending hashtags
 fetchkit suggest bluesky                                   # popular custom feeds
 fetchkit suggest bluesky --what actors --query "ai"        # accounts for an author_feed
+fetchkit suggest edgar --query apple                       # tickers/CIKs for an edgar config
+fetchkit suggest reddit --query finance                    # subreddits for a topic
+fetchkit suggest gdelt --query country                     # GDELT languages/countries/operators
+fetchkit suggest googlenews                                # Google News topics + query operators
+fetchkit suggest polymarket --query fed                    # most active prediction markets
+fetchkit suggest manifold --query "rate cut"               # Manifold markets for a topic
 fetchkit suggest rss --query "AI safety news"              # delegates to discover()
 ```
 
@@ -256,6 +302,12 @@ fetchkit suggest rss --query "AI safety news"              # delegates to discov
 | `stackexchange` | popular tags, or sites (`--what sites`)       | `tagged` / `site` |
 | `bluesky`       | popular feeds, or actors (`--what actors`)    | `actor` |
 | `mastodon`      | trending hashtags on an instance              | `tag` / `instance` |
+| `edgar`         | company tickers/CIKs matching a query         | `companies` |
+| `reddit`        | subreddits matching a query                   | `subreddits` |
+| `gdelt`         | languages, country codes, query operators (static) | `languages` / `countries` / `query` |
+| `googlenews`    | topic sections + query operators (static)     | `topic` / `query` |
+| `polymarket`    | most active live markets (by 24h volume)      | `query` |
+| `manifold`      | live markets matching a query                 | `query` |
 
 > Most suggesters call the source's live API (all no-auth); `hackernews`/`arxiv`
 > are static, and `rss` reuses the offline catalog ranker. Network suggesters fail
@@ -277,7 +329,9 @@ Set the time window with **either** a relative `window` **or** both
 `start_time`/`end_time` — the two are mutually exclusive. **If you specify none of
 them, the window defaults to the last 24 hours.** `start_time <= end_time` is
 enforced, and unknown top-level keys are rejected. Per-fetcher
-`start_time`/`end_time` default to `None` and **inherit** the global window at runtime.
+`start_time`/`end_time` default to `None` and **inherit** the run window from these
+top-level keys at runtime (there is no `global:` key — the top level *is* the
+global scope).
 
 #### Relative windows
 
@@ -399,6 +453,111 @@ plain text; tags, instance, and visibility are kept in `post.metadata`.
 | `local`    | false            | Restrict to statuses originating on this instance      |
 | `max_items`| 50               | 1–200 (paginated, 40/page)                             |
 
+### SEC EDGAR (`type: edgar`)
+
+Securities filings from the SEC's free, keyless EDGAR APIs. `resource: filings`
+pulls recent filings per company from `data.sec.gov/submissions` (acceptance
+timestamps are near-real-time — new 8-Ks appear within minutes); `resource: search`
+runs EDGAR full-text search across all filers (date granularity only; the run
+window is passed to the API as a date range). Form type, CIK, accession number,
+and 8-K item codes are kept in `post.metadata`.
+
+The SEC's [fair-access policy](https://www.sec.gov/os/accessing-edgar-data) asks
+clients to identify themselves with contact info and stay under 10 req/s — set
+`user_agent` and consider `http.rate_limit_per_host`.
+
+| Field        | Default  | Notes                                                        |
+|--------------|----------|--------------------------------------------------------------|
+| `resource`   | filings  | `filings` (per-company recent) or `search` (full-text)       |
+| `companies`  | `[]`     | Tickers or CIKs — required for `resource: filings`           |
+| `forms`      | `[]`     | Form types to include, e.g. `["8-K", "10-Q"]` (empty = all)  |
+| `query`      | null     | Full-text query — required for `resource: search`            |
+| `user_agent` | null     | Descriptive UA with contact info, e.g. `myapp me@example.com`|
+| `max_items`  | 50       | 1–500                                                        |
+
+### Reddit (`type: reddit`)
+
+Reddit's public `.json` endpoints with a descriptive User-Agent. `listing: new`
+surfaces posts seconds after submission (engagement is near zero at that point by
+definition); `hot`/`top` carry engagement at the cost of recency. Score,
+comment count, and upvote ratio are captured as of fetch time. Subreddit, flair,
+and upvote ratio are kept in `post.metadata`.
+
+> **Terms note:** unauthenticated access is tolerated by Reddit at low volume
+> (~10 requests/minute per IP) but sits in a gray area of Reddit's Data API
+> terms, which formally expect registered OAuth clients. fetchkit issues one
+> request per fetch; keep polling rates low (`http.rate_limit_per_host`).
+> Reddit also blocks many datacenter/cloud IPs outright (403 regardless of
+> User-Agent), so this fetcher may work from a residential connection but not
+> from CI or a cloud host.
+
+| Field         | Default | Notes                                                       |
+|---------------|---------|-------------------------------------------------------------|
+| `subreddits`  | `[]`    | Without `r/` prefix; combined into one listing request      |
+| `listing`     | new     | `new`, `hot`, `top`, or `rising` (when `query` unset)       |
+| `query`       | null    | Search; restricted to `subreddits` when both are set        |
+| `time_filter` | day     | For `listing: top` and searches: `hour`…`all`               |
+| `max_items`   | 50      | 1–100 (single request)                                      |
+
+### GDELT (`type: gdelt`)
+
+The GDELT Project's DOC 2.0 API — machine-reads worldwide news media in 65
+languages, re-indexed every 15 minutes. A zero-auth global news firehose. Hits
+carry URL, title, timestamp, source domain/country/language (kept in
+`post.metadata`); there is no author, body text, or engagement, so those fields
+are `None`. The run window maps to the API's `startdatetime`/`enddatetime`.
+
+| Field       | Default | Notes                                                          |
+|-------------|---------|----------------------------------------------------------------|
+| `query`     | —       | Required; keywords/"phrases" plus operators (`domain:`, `tone<-5`) |
+| `languages` | `[]`    | Source languages, e.g. `["english", "spanish"]`                |
+| `countries` | `[]`    | Source countries (GDELT/FIPS codes), e.g. `["US", "UK"]`       |
+| `max_items` | 75      | 1–250                                                          |
+
+### Google News (`type: googlenews`)
+
+Google News' public RSS endpoints: top stories, search, or a topic section.
+Minutes-fresh headlines aggregated from thousands of publishers. The query
+language supports `when:2h` (recency), `site:`, quoted phrases, `-exclusions`,
+and `intitle:`. Links are Google News redirect URLs (they resolve to the
+publisher); the publisher name maps to `author`.
+
+| Field       | Default | Notes                                                          |
+|-------------|---------|----------------------------------------------------------------|
+| `resource`  | top     | `top`, `search` (needs `query`), or `topic` (needs `topic`)    |
+| `query`     | null    | e.g. `'nvidia earnings when:2h'`                               |
+| `topic`     | null    | `WORLD`, `NATION`, `BUSINESS`, `TECHNOLOGY`, `ENTERTAINMENT`, `SPORTS`, `SCIENCE`, `HEALTH` |
+| `language`  | en-US   | UI language (`hl=`)                                            |
+| `country`   | US      | Edition (`gl=`/`ceid=`)                                        |
+| `max_items` | 50      | 1–100                                                          |
+
+### Polymarket (`type: polymarket`) and Manifold (`type: manifold`)
+
+Prediction markets as posts: the market question is the `title`, and live odds,
+volume, and close date land in `post.metadata` (`odds`/`probability`,
+`volume_24h`, …). Odds moving is often a signal about breaking news before
+articles are written. Both APIs are public read-only, no auth.
+
+> **Snapshot semantics:** markets are live state, not timestamped events, so
+> these two fetchers deliberately ignore the run window — an active market
+> created months ago is still current. `created_at` is the market's creation
+> time, which also means snapshot posts sort below newer article-shaped posts
+> in a mixed feed.
+
+| Field (`polymarket`) | Default | Notes                                            |
+|----------------------|---------|--------------------------------------------------|
+| `query`              | null    | Substring filter on the question, applied client-side over the top ~100 markets for the chosen `order` (not a full search) |
+| `order`              | volume  | `volume` (24h), `liquidity`, or `new`            |
+| `closed`             | false   | Include closed/resolved markets                  |
+| `max_items`          | 50      | 1–100                                            |
+
+| Field (`manifold`) | Default | Notes                                              |
+|--------------------|---------|----------------------------------------------------|
+| `query`            | null    | Full-text search; omit for most recent markets     |
+| `market_filter`    | open    | `open`, `closed`, `resolved`, `all` (searches)     |
+| `sort`             | score   | `score`, `newest`, `liquidity`, `24-hour-vol`      |
+| `max_items`        | 50      | 1–100                                              |
+
 ### HTTP (`http:`)
 
 | Field                 | Default                  | Notes                                              |
@@ -450,7 +609,8 @@ columns.
 class Post(BaseModel):
     id: str                       # unique within source
     source: str                   # "hackernews" | "rss" | "arxiv" | "github" | "lobsters"
-                                  #   | "stackexchange" | "bluesky" | "mastodon"
+                                  #   | "stackexchange" | "bluesky" | "mastodon" | "edgar"
+                                  #   | "reddit" | "gdelt" | "googlenews" | "polymarket" | "manifold"
     title: str | None
     text: str | None              # body / content
     url: str | None               # external link
@@ -468,7 +628,9 @@ sorted descending by `(created_at, id)` for deterministic output.
 
 > **`score` is source-relative.** Each source defines it differently — Hacker News
 > points, Lobsters score, GitHub stars (for `search_repos`), Stack Exchange question
-> score, Bluesky likes, Mastodon favourites, and `None` for arXiv and RSS. The values
+> score, Bluesky likes, Mastodon favourites, Reddit score (fuzzed by Reddit, and
+> near zero on brand-new posts), and `None` for arXiv, RSS, Google News, GDELT,
+> EDGAR, and the prediction-market sources. The values
 > are **not comparable across sources**, so don't rank a mixed
 > feed by `score` directly. Compare within a single `source`, or use a source-aware
 > ranking of your own. Output is ordered by recency (`created_at`), not by `score`.

{fetchkit_agents-0.3.0 → fetchkit_agents-0.4.0}/pyproject.toml

@@ -6,7 +6,7 @@ build-backend = "setuptools.build_meta"
 # Distribution name on PyPI. The importable package and CLI are still `fetchkit`;
 # the `fetchkit` PyPI name was already taken by an unrelated project.
 name = "fetchkit-agents"
-version = "0.3.0"
+version = "0.4.0"
 description = "A YAML-configured data fetching library for agentic applications."
 readme = "README.md"
 requires-python = ">=3.10"

{fetchkit_agents-0.3.0 → fetchkit_agents-0.4.0}/src/fetchkit/__init__.py

@@ -19,7 +19,7 @@ See the README for the full configuration reference and the guide to adding
 fetchers.
 """
 
-__version__ = "0.3.0"
+__version__ = "0.4.0"
 
 from typing import TYPE_CHECKING, Any
 
@@ -55,6 +55,12 @@ from fetchkit.schemas.fetcher import (
     StackExchangeFetchConfig,
     BlueskyFetchConfig,
     MastodonFetchConfig,
+    EdgarFetchConfig,
+    RedditFetchConfig,
+    GdeltFetchConfig,
+    GoogleNewsFetchConfig,
+    PolymarketFetchConfig,
+    ManifoldFetchConfig,
     FetcherConfig,
 )
 from fetchkit.schemas.collector import CollectorResult
@@ -121,6 +127,12 @@ __all__ = [
     "StackExchangeFetchConfig",
     "BlueskyFetchConfig",
     "MastodonFetchConfig",
+    "EdgarFetchConfig",
+    "RedditFetchConfig",
+    "GdeltFetchConfig",
+    "GoogleNewsFetchConfig",
+    "PolymarketFetchConfig",
+    "ManifoldFetchConfig",
     "FetcherConfig",
     "CollectorResult",
     "FetchKitConfig",