@pencil-agent/nano-pencil 2.0.0-beta.8 → 2.0.0

package/dist/extensions/builtin/browser/agent-workspace/domain-skills/dev-to/scraping.md

@@ -1,323 +1,323 @@
-# DEV Community (dev.to) — Data Extraction
-
-`https://dev.to` — developer blogging platform. Everything useful is available via a public REST API with no auth required. No browser needed for any read task.
-
-## Do this first
-
-**Use the REST API — it returns clean JSON in ~150–250ms with no browser, no login, no JS rendering.**
-
-```python
-import json
-articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&tag=python"))
-# Each article: id, title, description, url, cover_image, tag_list, tags,
-#               published_at, published_timestamp, readable_publish_date,
-#               reading_time_minutes, positive_reactions_count,
-#               public_reactions_count, comments_count, user, organization,
-#               flare_tag, collection_id, slug, path, canonical_url,
-#               social_image, language, subforem_id
-```
-
-The API serves **V0 (beta) by default** and emits a `Warning: 299` header on every response. Suppress it silently with the V1 `Accept` header (same data, no deprecated warning):
-
-```python
-import json
-import urllib.request, gzip
-
-def dev_get(url):
-    h = {
-        "User-Agent": "Mozilla/5.0",
-        "Accept-Encoding": "gzip",
-        "Accept": "application/vnd.forem.api-v1+json",
-    }
-    with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=20) as r:
-        data = r.read()
-        if r.headers.get("Content-Encoding") == "gzip":
-            data = gzip.decompress(data)
-        return data.decode()
-
-articles = json.loads(dev_get("https://dev.to/api/articles?per_page=10&tag=python"))
-```
-
-Or just use `http_get` directly if you don't care about the warning header noise.
-
----
-
-## Common workflows
-
-### Articles by tag
-
-```python
-import json
-articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&tag=python"))
-# Paginate with &page=2, &page=3 etc. (1-indexed)
-for a in articles:
-    print(a['id'], a['positive_reactions_count'], a['title'][:60])
-```
-
-Confirmed working tags: `python`, `javascript`, `typescript`, `rust`, `go`, `webdev`, `tutorial`, `react`, `devops`, `ai`, `beginners`.
-
-### Top articles by time window
-
-```python
-import json
-# top=N means "top articles from the last N days"
-top_day   = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=1"))
-top_week  = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=7"))
-top_month = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=30"))
-top_year  = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=365"))
-```
-
-### Articles by username
-
-```python
-import json
-articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&username=ben"))
-# Paginates cleanly: page=1, page=2 etc. Return distinct IDs, no overlap.
-```
-
-### New and rising articles
-
-```python
-import json
-fresh  = json.loads(http_get("https://dev.to/api/articles?per_page=10&state=fresh"))   # very new
-rising = json.loads(http_get("https://dev.to/api/articles?per_page=10&state=rising"))  # gaining traction
-# state=all returns 0 results (requires auth, not useful unauthenticated)
-```
-
-### Single article by ID (adds body_html and body_markdown)
-
-```python
-import json
-article = json.loads(http_get("https://dev.to/api/articles/3442047"))
-# Full article adds two fields not in list response:
-#   body_html     — rendered HTML (safe to display directly)
-#   body_markdown — raw Markdown source
-print(len(article['body_html']), len(article['body_markdown']))
-```
-
-### Single article by username/slug
-
-```python
-import json
-# path field from list response is "/username/slug"
-article = json.loads(http_get("https://dev.to/api/articles/ben/some-article-slug"))
-```
-
-### Tags — popular list with colors
-
-```python
-import json
-tags = json.loads(http_get("https://dev.to/api/tags?per_page=10"))
-# Fields: id, name, bg_color_hex, text_color_hex, short_summary
-# Sorted by popularity. Paginate with &page=2 etc.
-for t in tags:
-    print(t['name'], t['bg_color_hex'], t['text_color_hex'])
-# e.g. webdev  #562765  #ffffff
-#      javascript  #f7df1e  #000000
-#      ai  #17fd1a  #ffffff
-```
-
-### User profile
-
-```python
-import json
-user = json.loads(http_get("https://dev.to/api/users/by_username?url=ben"))
-# Fields: type_of, id, username, name, twitter_username, github_username,
-#         summary, location, website_url, joined_at, profile_image
-print(user['id'], user['username'], user['summary'])
-# e.g. 1  ben  "A Canadian software developer who thinks he's funny."
-```
-
-`joined_at` is a human string like `"Dec 27, 2015"` — not ISO 8601. Parse with `datetime.strptime(user['joined_at'], "%b %d, %Y")`.
-
-### Comments on an article
-
-```python
-import json
-comments = json.loads(http_get("https://dev.to/api/comments?a_id=3442047"))
-# Returns top-level comments only (replies nested under children key)
-# Fields per comment: id_code (string, not int!), type_of, body_html,
-#                     created_at, user (dict), children (list of same shape)
-for c in comments:
-    print(c['id_code'], c['user']['username'], c['created_at'])
-    for reply in c.get('children', []):
-        print("  reply:", reply['id_code'], reply['user']['username'])
-```
-
-### Single comment by id_code
-
-```python
-import json
-comment = json.loads(http_get("https://dev.to/api/comments/36lnc"))
-# Same fields as above: id_code, body_html, created_at, user, children
-```
-
-### Bulk tag fetch (parallel)
-
-```python
-import json
-from concurrent.futures import ThreadPoolExecutor
-
-tags = ['python', 'javascript', 'typescript', 'rust', 'go',
-        'devops', 'webdev', 'tutorial', 'productivity', 'react']
-
-def fetch_tag(tag):
-    data = json.loads(http_get(f"https://dev.to/api/articles?per_page=5&tag={tag}"))
-    return tag, data
-
-with ThreadPoolExecutor(max_workers=3) as ex:
-    results = dict(ex.map(lambda t: fetch_tag(t), tags))
-# 10 tags × 5 articles each: ~0.67s total with max_workers=3
-```
-
----
-
-## Endpoint reference
-
-| Endpoint | Auth | Key params | Latency |
-|----------|------|-----------|---------|
-| `GET /api/articles` | None | `tag`, `username`, `top`, `state`, `page`, `per_page` | ~200ms |
-| `GET /api/articles/{id}` | None | — | ~80ms |
-| `GET /api/articles/{username}/{slug}` | None | — | ~200ms |
-| `GET /api/tags` | None | `page`, `per_page` | ~190ms |
-| `GET /api/users/by_username?url={username}` | None | — | ~190ms |
-| `GET /api/comments?a_id={article_id}` | None | — | ~160ms |
-| `GET /api/comments/{id_code}` | None | — | ~150ms |
-| `GET /api/listings` | None | `category`, `page`, `per_page` | ~260ms (returns 0) |
-
-**Listings endpoint returns 0 results.** The `/api/listings` endpoint is documented but returns an empty array for all categories (`jobs`, `forsale`, `education`, `cfp`) without auth. Skip it.
-
----
-
-## Pagination
-
-All list endpoints paginate with `page=` (1-indexed) and `per_page=`:
-
-```python
-import json
-
-def get_all_articles_by_tag(tag, max_pages=5):
-    results = []
-    for page in range(1, max_pages + 1):
-        batch = json.loads(http_get(
-            f"https://dev.to/api/articles?per_page=30&tag={tag}&page={page}"
-        ))
-        if not batch:
-            break
-        results.extend(batch)
-    return results
-```
-
-- `per_page` supports up to **1000** (confirmed). No documented max, but 1000 works in testing.
-- No `total_count` field in list responses — you paginate until an empty array.
-- Page ordering is consistent — confirmed no ID overlap between page 1 and page 2.
-
----
-
-## Article field reference
-
-All fields returned in list responses (single article adds `body_html` and `body_markdown`):
-
-```
-id                      int    — article ID, stable, use for single-article fetch
-title                   str
-description             str    — auto-excerpt, never null
-slug                    str    — URL slug component
-path                    str    — "/username/slug"
-url                     str    — full canonical URL
-canonical_url           str    — same as url for native posts; author's site URL for cross-posts
-cover_image             str|null — CDN URL or null (~30% of articles have no cover image)
-social_image            str    — always present (generated if no cover_image)
-tag_list                list   — e.g. ['python', 'ai', 'tutorial']  ← use this for code
-tags                    str    — same tags as comma-separated string "python, ai, tutorial"
-published_at            str    — ISO 8601 UTC e.g. "2026-04-18T03:49:36Z"
-published_timestamp     str    — identical to published_at
-readable_publish_date   str    — human string e.g. "Apr 18"
-reading_time_minutes    int
-positive_reactions_count int   — hearts/likes count
-public_reactions_count  int    — total reactions (usually same as positive_reactions_count)
-comments_count          int
-user                    dict   — name, username, twitter_username, github_username,
-                                 user_id, website_url, profile_image, profile_image_90
-organization            dict|null — present when posted under an org: name, username, slug,
-                                     profile_image, profile_image_90
-flare_tag               dict|null — {name, bg_color_hex, text_color_hex} — discussion/challenge badge
-collection_id           int|null  — series/collection ID if part of a series
-language                str    — e.g. "en"
-subforem_id             int|null
-crossposted_at          str|null — ISO datetime if cross-posted
-edited_at               str|null
-last_comment_at         str|null
-created_at              str    — ISO 8601
-type_of                 str    — always "article"
-```
-
----
-
-## Rate limits
-
-- **Burst limit: ~6 rapid sequential requests**, then HTTP 429.
-- **Recovery: `Retry-After: 1` second** — wait 1s after a 429 and you're good again.
-- No `X-RateLimit-*` headers in 200 responses — you only see `Retry-After` on the 429 itself.
-- With `ThreadPoolExecutor(max_workers=3)`, 10 concurrent requests succeed without hitting the limit.
-- No difference in limits between V0 (default) and V1 (`Accept` header) — same underlying rate limit.
-- **No auth token tested** — all endpoints above work without `api_key`. Authenticated requests likely have higher limits.
-
-Safe pattern for bulk fetching:
-
-```python
-import json, time
-from concurrent.futures import ThreadPoolExecutor
-
-def safe_fetch(url):
-    for attempt in range(3):
-        try:
-            return json.loads(http_get(url))
-        except Exception as e:
-            if '429' in str(e):
-                time.sleep(1)   # Retry-After is 1s
-                continue
-            raise
-    return []
-
-urls = [
-    f"https://dev.to/api/articles?per_page=10&tag={tag}"
-    for tag in ['python', 'javascript', 'typescript', 'rust']
-]
-with ThreadPoolExecutor(max_workers=3) as ex:
-    results = list(ex.map(safe_fetch, urls))
-```
-
----
-
-## Gotchas
-
-- **`tag_list` (list) vs `tags` (string)** — both fields always present. `tag_list` is a Python list; `tags` is the same data as a comma-separated string. Use `tag_list` in code.
-
-- **Comments have `id_code`, not `id`** — comment identifiers are alphanumeric strings like `"36lnc"`, not integers. The integer `id` field is absent from comment objects. Use `id_code` to fetch a specific comment via `GET /api/comments/{id_code}`.
-
-- **Comments endpoint returns top-level only** — replies are nested under `children` recursively, not returned as a flat list. A thread with 100 total comments may only show 60 top-level objects; walk `children` recursively to count all.
-
-- **`cover_image` can be null** — ~30% of articles have no cover image. Always guard: `a.get('cover_image') or a['social_image']` for a guaranteed image URL.
-
-- **`flare_tag` is null for most articles** — only discussion/challenge posts carry it. It's a dict `{name, bg_color_hex, text_color_hex}` when present.
-
-- **`published_at` == `published_timestamp`** — both fields contain identical ISO 8601 UTC strings. `readable_publish_date` is human-only (`"Apr 18"`, no year).
-
-- **`joined_at` on user profile is not ISO** — it's `"Dec 27, 2015"`. Parse: `datetime.strptime(u['joined_at'], "%b %d, %Y")`.
-
-- **`state=all` returns 0 results unauthenticated** — it's for the authenticated user's own feed. `state=fresh` and `state=rising` work without auth.
-
-- **`top=N` means last N days** — `top=1` is last 24h, `top=7` is last week, `top=30` is last month, `top=365` is last year. Results differ from the `state=` param.
-
-- **V0 warning header on every response** — `Warning: 299 - This endpoint is part of the V0 (beta) API…` appears on all responses without the `Accept` header. It's harmless but noisy. Suppress with `"Accept": "application/vnd.forem.api-v1+json"`.
-
-- **No `total_count` in list responses** — paginate until an empty array. There is no way to know upfront how many total results exist.
-
-- **Listings endpoint returns empty** — `GET /api/listings` and all category variants return `[]` without auth. Documented but non-functional publicly.
-
-- **`/api/articles/{id}/comments` returns 404** — comments must be fetched via `GET /api/comments?a_id={id}`, not as a sub-resource of articles.
-
-- **`canonical_url` may point off-site** — for cross-posted articles, `canonical_url` is the author's original blog URL, not dev.to. Use `url` for the dev.to link.
-
-- **`organization` field is null for personal posts** — only present when the article was posted under an org account. Check before accessing sub-fields.
+# DEV Community (dev.to) — Data Extraction
+
+`https://dev.to` — developer blogging platform. Everything useful is available via a public REST API with no auth required. No browser needed for any read task.
+
+## Do this first
+
+**Use the REST API — it returns clean JSON in ~150–250ms with no browser, no login, no JS rendering.**
+
+```python
+import json
+articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&tag=python"))
+# Each article: id, title, description, url, cover_image, tag_list, tags,
+#               published_at, published_timestamp, readable_publish_date,
+#               reading_time_minutes, positive_reactions_count,
+#               public_reactions_count, comments_count, user, organization,
+#               flare_tag, collection_id, slug, path, canonical_url,
+#               social_image, language, subforem_id
+```
+
+The API serves **V0 (beta) by default** and emits a `Warning: 299` header on every response. Suppress it silently with the V1 `Accept` header (same data, no deprecated warning):
+
+```python
+import json
+import urllib.request, gzip
+
+def dev_get(url):
+    h = {
+        "User-Agent": "Mozilla/5.0",
+        "Accept-Encoding": "gzip",
+        "Accept": "application/vnd.forem.api-v1+json",
+    }
+    with urllib.request.urlopen(urllib.request.Request(url, headers=h), timeout=20) as r:
+        data = r.read()
+        if r.headers.get("Content-Encoding") == "gzip":
+            data = gzip.decompress(data)
+        return data.decode()
+
+articles = json.loads(dev_get("https://dev.to/api/articles?per_page=10&tag=python"))
+```
+
+Or just use `http_get` directly if you don't care about the warning header noise.
+
+---
+
+## Common workflows
+
+### Articles by tag
+
+```python
+import json
+articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&tag=python"))
+# Paginate with &page=2, &page=3 etc. (1-indexed)
+for a in articles:
+    print(a['id'], a['positive_reactions_count'], a['title'][:60])
+```
+
+Confirmed working tags: `python`, `javascript`, `typescript`, `rust`, `go`, `webdev`, `tutorial`, `react`, `devops`, `ai`, `beginners`.
+
+### Top articles by time window
+
+```python
+import json
+# top=N means "top articles from the last N days"
+top_day   = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=1"))
+top_week  = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=7"))
+top_month = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=30"))
+top_year  = json.loads(http_get("https://dev.to/api/articles?per_page=10&top=365"))
+```
+
+### Articles by username
+
+```python
+import json
+articles = json.loads(http_get("https://dev.to/api/articles?per_page=10&username=ben"))
+# Paginates cleanly: page=1, page=2 etc. Return distinct IDs, no overlap.
+```
+
+### New and rising articles
+
+```python
+import json
+fresh  = json.loads(http_get("https://dev.to/api/articles?per_page=10&state=fresh"))   # very new
+rising = json.loads(http_get("https://dev.to/api/articles?per_page=10&state=rising"))  # gaining traction
+# state=all returns 0 results (requires auth, not useful unauthenticated)
+```
+
+### Single article by ID (adds body_html and body_markdown)
+
+```python
+import json
+article = json.loads(http_get("https://dev.to/api/articles/3442047"))
+# Full article adds two fields not in list response:
+#   body_html     — rendered HTML (safe to display directly)
+#   body_markdown — raw Markdown source
+print(len(article['body_html']), len(article['body_markdown']))
+```
+
+### Single article by username/slug
+
+```python
+import json
+# path field from list response is "/username/slug"
+article = json.loads(http_get("https://dev.to/api/articles/ben/some-article-slug"))
+```
+
+### Tags — popular list with colors
+
+```python
+import json
+tags = json.loads(http_get("https://dev.to/api/tags?per_page=10"))
+# Fields: id, name, bg_color_hex, text_color_hex, short_summary
+# Sorted by popularity. Paginate with &page=2 etc.
+for t in tags:
+    print(t['name'], t['bg_color_hex'], t['text_color_hex'])
+# e.g. webdev  #562765  #ffffff
+#      javascript  #f7df1e  #000000
+#      ai  #17fd1a  #ffffff
+```
+
+### User profile
+
+```python
+import json
+user = json.loads(http_get("https://dev.to/api/users/by_username?url=ben"))
+# Fields: type_of, id, username, name, twitter_username, github_username,
+#         summary, location, website_url, joined_at, profile_image
+print(user['id'], user['username'], user['summary'])
+# e.g. 1  ben  "A Canadian software developer who thinks he's funny."
+```
+
+`joined_at` is a human string like `"Dec 27, 2015"` — not ISO 8601. Parse with `datetime.strptime(user['joined_at'], "%b %d, %Y")`.
+
+### Comments on an article
+
+```python
+import json
+comments = json.loads(http_get("https://dev.to/api/comments?a_id=3442047"))
+# Returns top-level comments only (replies nested under children key)
+# Fields per comment: id_code (string, not int!), type_of, body_html,
+#                     created_at, user (dict), children (list of same shape)
+for c in comments:
+    print(c['id_code'], c['user']['username'], c['created_at'])
+    for reply in c.get('children', []):
+        print("  reply:", reply['id_code'], reply['user']['username'])
+```
+
+### Single comment by id_code
+
+```python
+import json
+comment = json.loads(http_get("https://dev.to/api/comments/36lnc"))
+# Same fields as above: id_code, body_html, created_at, user, children
+```
+
+### Bulk tag fetch (parallel)
+
+```python
+import json
+from concurrent.futures import ThreadPoolExecutor
+
+tags = ['python', 'javascript', 'typescript', 'rust', 'go',
+        'devops', 'webdev', 'tutorial', 'productivity', 'react']
+
+def fetch_tag(tag):
+    data = json.loads(http_get(f"https://dev.to/api/articles?per_page=5&tag={tag}"))
+    return tag, data
+
+with ThreadPoolExecutor(max_workers=3) as ex:
+    results = dict(ex.map(lambda t: fetch_tag(t), tags))
+# 10 tags × 5 articles each: ~0.67s total with max_workers=3
+```
+
+---
+
+## Endpoint reference
+
+| Endpoint | Auth | Key params | Latency |
+|----------|------|-----------|---------|
+| `GET /api/articles` | None | `tag`, `username`, `top`, `state`, `page`, `per_page` | ~200ms |
+| `GET /api/articles/{id}` | None | — | ~80ms |
+| `GET /api/articles/{username}/{slug}` | None | — | ~200ms |
+| `GET /api/tags` | None | `page`, `per_page` | ~190ms |
+| `GET /api/users/by_username?url={username}` | None | — | ~190ms |
+| `GET /api/comments?a_id={article_id}` | None | — | ~160ms |
+| `GET /api/comments/{id_code}` | None | — | ~150ms |
+| `GET /api/listings` | None | `category`, `page`, `per_page` | ~260ms (returns 0) |
+
+**Listings endpoint returns 0 results.** The `/api/listings` endpoint is documented but returns an empty array for all categories (`jobs`, `forsale`, `education`, `cfp`) without auth. Skip it.
+
+---
+
+## Pagination
+
+All list endpoints paginate with `page=` (1-indexed) and `per_page=`:
+
+```python
+import json
+
+def get_all_articles_by_tag(tag, max_pages=5):
+    results = []
+    for page in range(1, max_pages + 1):
+        batch = json.loads(http_get(
+            f"https://dev.to/api/articles?per_page=30&tag={tag}&page={page}"
+        ))
+        if not batch:
+            break
+        results.extend(batch)
+    return results
+```
+
+- `per_page` supports up to **1000** (confirmed). No documented max, but 1000 works in testing.
+- No `total_count` field in list responses — you paginate until an empty array.
+- Page ordering is consistent — confirmed no ID overlap between page 1 and page 2.
+
+---
+
+## Article field reference
+
+All fields returned in list responses (single article adds `body_html` and `body_markdown`):
+
+```
+id                      int    — article ID, stable, use for single-article fetch
+title                   str
+description             str    — auto-excerpt, never null
+slug                    str    — URL slug component
+path                    str    — "/username/slug"
+url                     str    — full canonical URL
+canonical_url           str    — same as url for native posts; author's site URL for cross-posts
+cover_image             str|null — CDN URL or null (~30% of articles have no cover image)
+social_image            str    — always present (generated if no cover_image)
+tag_list                list   — e.g. ['python', 'ai', 'tutorial']  ← use this for code
+tags                    str    — same tags as comma-separated string "python, ai, tutorial"
+published_at            str    — ISO 8601 UTC e.g. "2026-04-18T03:49:36Z"
+published_timestamp     str    — identical to published_at
+readable_publish_date   str    — human string e.g. "Apr 18"
+reading_time_minutes    int
+positive_reactions_count int   — hearts/likes count
+public_reactions_count  int    — total reactions (usually same as positive_reactions_count)
+comments_count          int
+user                    dict   — name, username, twitter_username, github_username,
+                                 user_id, website_url, profile_image, profile_image_90
+organization            dict|null — present when posted under an org: name, username, slug,
+                                     profile_image, profile_image_90
+flare_tag               dict|null — {name, bg_color_hex, text_color_hex} — discussion/challenge badge
+collection_id           int|null  — series/collection ID if part of a series
+language                str    — e.g. "en"
+subforem_id             int|null
+crossposted_at          str|null — ISO datetime if cross-posted
+edited_at               str|null
+last_comment_at         str|null
+created_at              str    — ISO 8601
+type_of                 str    — always "article"
+```
+
+---
+
+## Rate limits
+
+- **Burst limit: ~6 rapid sequential requests**, then HTTP 429.
+- **Recovery: `Retry-After: 1` second** — wait 1s after a 429 and you're good again.
+- No `X-RateLimit-*` headers in 200 responses — you only see `Retry-After` on the 429 itself.
+- With `ThreadPoolExecutor(max_workers=3)`, 10 concurrent requests succeed without hitting the limit.
+- No difference in limits between V0 (default) and V1 (`Accept` header) — same underlying rate limit.
+- **No auth token tested** — all endpoints above work without `api_key`. Authenticated requests likely have higher limits.
+
+Safe pattern for bulk fetching:
+
+```python
+import json, time
+from concurrent.futures import ThreadPoolExecutor
+
+def safe_fetch(url):
+    for attempt in range(3):
+        try:
+            return json.loads(http_get(url))
+        except Exception as e:
+            if '429' in str(e):
+                time.sleep(1)   # Retry-After is 1s
+                continue
+            raise
+    return []
+
+urls = [
+    f"https://dev.to/api/articles?per_page=10&tag={tag}"
+    for tag in ['python', 'javascript', 'typescript', 'rust']
+]
+with ThreadPoolExecutor(max_workers=3) as ex:
+    results = list(ex.map(safe_fetch, urls))
+```
+
+---
+
+## Gotchas
+
+- **`tag_list` (list) vs `tags` (string)** — both fields always present. `tag_list` is a Python list; `tags` is the same data as a comma-separated string. Use `tag_list` in code.
+
+- **Comments have `id_code`, not `id`** — comment identifiers are alphanumeric strings like `"36lnc"`, not integers. The integer `id` field is absent from comment objects. Use `id_code` to fetch a specific comment via `GET /api/comments/{id_code}`.
+
+- **Comments endpoint returns top-level only** — replies are nested under `children` recursively, not returned as a flat list. A thread with 100 total comments may only show 60 top-level objects; walk `children` recursively to count all.
+
+- **`cover_image` can be null** — ~30% of articles have no cover image. Always guard: `a.get('cover_image') or a['social_image']` for a guaranteed image URL.
+
+- **`flare_tag` is null for most articles** — only discussion/challenge posts carry it. It's a dict `{name, bg_color_hex, text_color_hex}` when present.
+
+- **`published_at` == `published_timestamp`** — both fields contain identical ISO 8601 UTC strings. `readable_publish_date` is human-only (`"Apr 18"`, no year).
+
+- **`joined_at` on user profile is not ISO** — it's `"Dec 27, 2015"`. Parse: `datetime.strptime(u['joined_at'], "%b %d, %Y")`.
+
+- **`state=all` returns 0 results unauthenticated** — it's for the authenticated user's own feed. `state=fresh` and `state=rising` work without auth.
+
+- **`top=N` means last N days** — `top=1` is last 24h, `top=7` is last week, `top=30` is last month, `top=365` is last year. Results differ from the `state=` param.
+
+- **V0 warning header on every response** — `Warning: 299 - This endpoint is part of the V0 (beta) API…` appears on all responses without the `Accept` header. It's harmless but noisy. Suppress with `"Accept": "application/vnd.forem.api-v1+json"`.
+
+- **No `total_count` in list responses** — paginate until an empty array. There is no way to know upfront how many total results exist.
+
+- **Listings endpoint returns empty** — `GET /api/listings` and all category variants return `[]` without auth. Documented but non-functional publicly.
+
+- **`/api/articles/{id}/comments` returns 404** — comments must be fetched via `GET /api/comments?a_id={id}`, not as a sub-resource of articles.
+
+- **`canonical_url` may point off-site** — for cross-posted articles, `canonical_url` is the author's original blog URL, not dev.to. Use `url` for the dev.to link.
+
+- **`organization` field is null for personal posts** — only present when the article was posted under an org account. Check before accessing sub-fields.