threadsense 0.2.1__tar.gz → 0.3.0__tar.gz

{threadsense-0.2.1 → threadsense-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: threadsense
-Version: 0.2.1
+Version: 0.3.0
 Summary: Evidence-first discussion intelligence tooling.
 License: MIT License
         
@@ -41,7 +41,7 @@ Scrapers give you raw payloads. AI summarizers give you prose. Neither gives you
 
 ThreadSense keeps the evidence chain intact at every stage:
 
-1. **Acquire** — source connectors fetch threads from Reddit, Hacker News, and GitHub Discussions
+1. **Acquire** — source connectors fetch threads from Reddit, Hacker News, GitHub Discussions, and GitHub Gists
 2. **Normalize** — source-specific payloads are mapped into a canonical thread model with provenance metadata
 3. **Analyze** — deterministic extraction identifies issues, feature requests, themes, and sentiment — each linked to the comment that produced it
 4. **Synthesize** — optional local-model inference adds summaries on top of the deterministic evidence layer
@@ -63,6 +63,14 @@ uv run threadsense run reddit \
   --summary-required
 ```
 
+Works with GitHub Gists too — flat-comment discussions with code context:
+
+```bash
+uv run threadsense run gist \
+  "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f" \
+  --format markdown
+```
+
 ### Cross-thread research
 
 Search a topic across multiple subreddits. ThreadSense deterministically selects, ranks, and analyzes matching threads, then synthesizes a corpus-level report.
@@ -98,10 +106,10 @@ fetch → normalize → analyze → [optional inference] → report
 
 ## Sources and Discovery
 
-| Capability      | Reddit | Hacker News | GitHub Discussions |
-| --------------- | :----: | :---------: | :----------------: |
-| Thread analysis |  yes   |     yes     |        yes         |
-| Topic research  |  yes   |      —      |         —          |
+| Capability      | Reddit | Hacker News | GitHub Discussions | GitHub Gists |
+| --------------- | :----: | :---------: | :----------------: | :----------: |
+| Thread analysis |  yes   |     yes     |        yes         |     yes      |
+| Topic research  |  yes   |      —      |         —          |      —       |
 
 ## Output Modes
 
@@ -137,7 +145,11 @@ uv run threadsense preflight
 uv run threadsense run reddit \
   "https://www.reddit.com/r/ClaudeCode/comments/1ro0qbl/anyone_actually_built_a_second_brain_that_isnt/"
 
-# 4. Research a topic across subreddits
+# 4. Analyze a GitHub Gist
+uv run threadsense run gist \
+  "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f"
+
+# 5. Research a topic across subreddits
 uv run threadsense research reddit \
   --query "second brain OR agentic PKM" \
   --subreddit ClaudeCode \
@@ -213,6 +225,8 @@ uv run pytest
 - Topic research is implemented for Reddit; other source discovery workflows are planned
 - Reddit research queries support `OR`/`|` clause unions only (intentionally narrow for deterministic alignment)
 - Corpus reports are Markdown only
+- GitHub Gist comments are flat (no threading); analysis uses temporal/semantic clustering
+- GitHub Gist connector does not retry on transient API errors (502/503)
 - The local API is a trusted local surface, not a hardened public service
 
 ## Direction

{threadsense-0.2.1 → threadsense-0.3.0}/README.md

@@ -8,7 +8,7 @@ Scrapers give you raw payloads. AI summarizers give you prose. Neither gives you
 
 ThreadSense keeps the evidence chain intact at every stage:
 
-1. **Acquire** — source connectors fetch threads from Reddit, Hacker News, and GitHub Discussions
+1. **Acquire** — source connectors fetch threads from Reddit, Hacker News, GitHub Discussions, and GitHub Gists
 2. **Normalize** — source-specific payloads are mapped into a canonical thread model with provenance metadata
 3. **Analyze** — deterministic extraction identifies issues, feature requests, themes, and sentiment — each linked to the comment that produced it
 4. **Synthesize** — optional local-model inference adds summaries on top of the deterministic evidence layer
@@ -30,6 +30,14 @@ uv run threadsense run reddit \
   --summary-required
 ```
 
+Works with GitHub Gists too — flat-comment discussions with code context:
+
+```bash
+uv run threadsense run gist \
+  "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f" \
+  --format markdown
+```
+
 ### Cross-thread research
 
 Search a topic across multiple subreddits. ThreadSense deterministically selects, ranks, and analyzes matching threads, then synthesizes a corpus-level report.
@@ -65,10 +73,10 @@ fetch → normalize → analyze → [optional inference] → report
 
 ## Sources and Discovery
 
-| Capability      | Reddit | Hacker News | GitHub Discussions |
-| --------------- | :----: | :---------: | :----------------: |
-| Thread analysis |  yes   |     yes     |        yes         |
-| Topic research  |  yes   |      —      |         —          |
+| Capability      | Reddit | Hacker News | GitHub Discussions | GitHub Gists |
+| --------------- | :----: | :---------: | :----------------: | :----------: |
+| Thread analysis |  yes   |     yes     |        yes         |     yes      |
+| Topic research  |  yes   |      —      |         —          |      —       |
 
 ## Output Modes
 
@@ -104,7 +112,11 @@ uv run threadsense preflight
 uv run threadsense run reddit \
   "https://www.reddit.com/r/ClaudeCode/comments/1ro0qbl/anyone_actually_built_a_second_brain_that_isnt/"
 
-# 4. Research a topic across subreddits
+# 4. Analyze a GitHub Gist
+uv run threadsense run gist \
+  "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f"
+
+# 5. Research a topic across subreddits
 uv run threadsense research reddit \
   --query "second brain OR agentic PKM" \
   --subreddit ClaudeCode \
@@ -180,6 +192,8 @@ uv run pytest
 - Topic research is implemented for Reddit; other source discovery workflows are planned
 - Reddit research queries support `OR`/`|` clause unions only (intentionally narrow for deterministic alignment)
 - Corpus reports are Markdown only
+- GitHub Gist comments are flat (no threading); analysis uses temporal/semantic clustering
+- GitHub Gist connector does not retry on transient API errors (502/503)
 - The local API is a trusted local surface, not a hardened public service
 
 ## Direction

{threadsense-0.2.1 → threadsense-0.3.0}/docs/artifacts.md

@@ -25,6 +25,7 @@ Examples:
 - `.threadsense/raw/reddit/<thread-id>.json`
 - `.threadsense/raw/hackernews/<story-id>.json`
 - `.threadsense/raw/github_discussions/<thread-id>.json`
+- `.threadsense/raw/github_gist/<gist-id>.json`
 
 Purpose:
 
@@ -41,6 +42,7 @@ Examples:
 - `.threadsense/normalized/reddit/<thread-id>.json`
 - `.threadsense/normalized/hackernews/<thread-id>.json`
 - `.threadsense/normalized/github_discussions/<thread-id>.json`
+- `.threadsense/normalized/github_gist/<gist-id>.json`
 
 Purpose:
 
@@ -57,6 +59,7 @@ Examples:
 - `.threadsense/analysis/reddit/<thread-id>.json`
 - `.threadsense/analysis/hackernews/<thread-id>.json`
 - `.threadsense/analysis/github_discussions/<thread-id>.json`
+- `.threadsense/analysis/github_gist/<gist-id>.json`
 
 Purpose:
 

{threadsense-0.2.1 → threadsense-0.3.0}/docs/overview.md

@@ -6,7 +6,7 @@ ThreadSense is an evidence-first discussion intelligence system for turning publ
 
 The implemented system supports:
 
-- source connectors for Reddit, Hacker News, and GitHub Discussions
+- source connectors for Reddit, Hacker News, GitHub Discussions, and GitHub Gists
 - canonical normalization into a shared thread model
 - deterministic analysis with evidence-linked findings
 - optional local-model synthesis for thread and corpus summaries
@@ -90,9 +90,10 @@ ThreadSense separates source-specific acquisition from canonical normalization.
 
 Current source support:
 
-- Reddit
-- Hacker News
-- GitHub Discussions
+- Reddit — threaded comments via JSON API with "more" expansion
+- Hacker News — recursive tree-based item fetching via Firebase API
+- GitHub Discussions — GraphQL with cursor-based pagination for comments and replies
+- GitHub Gists — REST API with paginated flat comments; authentication optional for public gists
 
 Each connector preserves enough source metadata for traceability, while normalization maps the result into a canonical `Thread` object with:
 
@@ -182,7 +183,7 @@ JSON is for machines. Human mode is for operators. Both should expose the same u
 ThreadSense is best described as:
 
 ```text
-multi-source thread analysis + corpus synthesis + subreddit topic research
+multi-source discussion intelligence + corpus synthesis + subreddit topic research
 ```
 
 It is not just a scraper, and it is not just a summarizer. It is a reproducible pipeline from source thread to inspectable intelligence.

{threadsense-0.2.1 → threadsense-0.3.0}/docs/system-design.md

@@ -49,6 +49,7 @@ subreddit search -> candidate ranking -> selected thread fetch/analyze
 - `src/threadsense/connectors/reddit.py`
 - `src/threadsense/connectors/hackernews.py`
 - `src/threadsense/connectors/github_discussions.py`
+- `src/threadsense/connectors/github_gist.py`
 
 Responsibilities:
 
@@ -58,11 +59,11 @@ Responsibilities:
 - preserve source-specific thread metadata
 - return raw artifacts for later normalization
 
-Special note for Reddit:
+Special notes:
 
-- direct thread fetch is URL-based
-- topic research uses explicit subreddit search
-- exact time-window filtering is applied after Reddit retrieval
+- Reddit: direct thread fetch is URL-based; topic research uses explicit subreddit search with exact time-window filtering after retrieval
+- GitHub Discussions: GraphQL with cursor-based pagination for comments (100/page) and replies; requires authentication token
+- GitHub Gists: REST API with paginated flat comments; authentication optional for public gists; comments have no threading (all depth=0)
 
 ### Canonical Model
 

{threadsense-0.2.1 → threadsense-0.3.0}/docs/usage.md

@@ -43,7 +43,7 @@ Top-level commands:
 ### Single-thread run
 
 ```bash
-uv run threadsense run reddit <reddit-url> \
+uv run threadsense run [<source>] <url> \
   [--format markdown|html|json] \
   [--with-summary] \
   [--summary-required] \
@@ -51,7 +51,9 @@ uv run threadsense run reddit <reddit-url> \
   [--flat]
 ```
 
-Example:
+Source can be `reddit`, `hn`, `github-discussions`, `gist`, or omitted for auto-detection.
+
+Examples:
 
 ```bash
 uv run threadsense run reddit \
@@ -61,6 +63,12 @@ uv run threadsense run reddit \
   --summary-required
 ```
 
+```bash
+uv run threadsense run gist \
+  "https://gist.github.com/karpathy/442a6bf555914893e9891c11519de94f" \
+  --format markdown
+```
+
 ### Topic research across subreddits
 
 ```bash
@@ -125,6 +133,20 @@ uv run threadsense fetch hn <hackernews-item-url> [--output path/to/raw.json]
 uv run threadsense fetch github-discussions <discussion-url> [--output path/to/raw.json]
 ```
 
+### `fetch github-gist`
+
+```bash
+uv run threadsense fetch github-gist <gist-url> [--output path/to/raw.json]
+```
+
+Aliases: `fetch gist`
+
+Notes:
+
+- Authentication is optional. Public gists work without a token (60 req/hr rate limit).
+- Set `THREADSENSE_GITHUB_GIST_TOKEN` for authenticated access (5,000 req/hr).
+- Comments are paginated at 100/page automatically.
+
 ## `normalize`
 
 Normalize one raw artifact into the canonical thread schema.
@@ -147,6 +169,19 @@ uv run threadsense normalize hn --input path/to/raw.json [--output path/to/norma
 uv run threadsense normalize github-discussions --input path/to/raw.json [--output path/to/normalized.json]
 ```
 
+### `normalize github-gist`
+
+```bash
+uv run threadsense normalize github-gist --input path/to/raw.json [--output path/to/normalized.json]
+```
+
+Aliases: `normalize gist`
+
+Notes:
+
+- Gist comments are flat — all normalized to `depth=0` with no `parent_comment_id`.
+- Multi-file gists are concatenated into a single body, each file prefixed with its filename.
+
 ## `analyze`
 
 Run deterministic analysis for one canonical thread artifact.

{threadsense-0.2.1 → threadsense-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "threadsense"
-version = "0.2.1"
+version = "0.3.0"
 description = "Evidence-first discussion intelligence tooling."
 readme = "README.md"
 license = { file = "LICENSE" }

{threadsense-0.2.1 → threadsense-0.3.0}/src/threadsense/__init__.py

@@ -2,4 +2,4 @@ from __future__ import annotations
 
 __all__ = ["__version__"]
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

{threadsense-0.2.1 → threadsense-0.3.0}/src/threadsense/cli.py

@@ -49,8 +49,8 @@ from threadsense.workflows import (
     normalize_source_thread,
     report_analysis,
     report_corpus,
-    run_reddit_research,
     run_reddit_pipeline,
+    run_reddit_research,
     run_source_pipeline,
     search_corpora,
 )
@@ -188,6 +188,20 @@ def _build_fetch_parser(subparsers: argparse._SubParsersAction[argparse.Argument
     )
     _add_config_argument(gh_parser)
     _add_no_cache_argument(gh_parser)
+    gist_parser = fetch_subparsers.add_parser(
+        "github-gist",
+        aliases=["gist"],
+        help="Fetch one GitHub Gist with comments.",
+    )
+    gist_parser.add_argument("url", help="Full GitHub Gist URL")
+    gist_parser.add_argument(
+        "--output",
+        "-o",
+        type=Path,
+        help="Raw artifact output path. Defaults to the configured raw store path.",
+    )
+    _add_config_argument(gist_parser)
+    _add_no_cache_argument(gist_parser)
 
 
 def _build_normalize_parser(
@@ -251,6 +265,24 @@ def _build_normalize_parser(
         help="Normalized artifact output path. Defaults to the configured normalized store path.",
     )
     _add_config_argument(normalize_gh_parser)
+    normalize_gist_parser = normalize_subparsers.add_parser(
+        "github-gist",
+        aliases=["gist"],
+        help="Normalize one GitHub Gist raw artifact.",
+    )
+    normalize_gist_parser.add_argument(
+        "--input",
+        type=Path,
+        required=True,
+        help="Raw GitHub Gist artifact path.",
+    )
+    normalize_gist_parser.add_argument(
+        "--output",
+        "-o",
+        type=Path,
+        help="Normalized artifact output path. Defaults to the configured normalized store path.",
+    )
+    _add_config_argument(normalize_gist_parser)
 
 
 def _build_analyze_parser(subparsers: argparse._SubParsersAction[argparse.ArgumentParser]) -> None:
@@ -1355,6 +1387,8 @@ def parse_run_target(target: list[str]) -> tuple[str | None, str]:
             source_name = "hackernews"
         if source_name in {"github-discussions", "gh-discussions"}:
             source_name = "github_discussions"
+        if source_name in {"github-gist", "gist"}:
+            source_name = "github_gist"
         return source_name, target[1]
     raise _CommandDispatchError("run")
 
@@ -1395,13 +1429,27 @@ def _dispatch_command(args: argparse.Namespace) -> int:
                 source_name="github_discussions",
                 no_cache=args.no_cache,
             )
+        case ("fetch", "github-gist" | "gist", _, _, _):
+            return run_source_fetch(
+                config_path=args.config,
+                url=args.url,
+                output_path=args.output,
+                source_name="github_gist",
+                no_cache=args.no_cache,
+            )
         case ("normalize", "reddit", _, _, _):
             return run_reddit_normalize(
                 config_path=args.config,
                 input_path=args.input,
                 output_path=args.output,
             )
-        case ("normalize", "hn" | "hackernews" | "github-discussions" | "gh-discussions", _, _, _):
+        case (
+            "normalize",
+            "hn" | "hackernews" | "github-discussions" | "gh-discussions" | "github-gist" | "gist",
+            _,
+            _,
+            _,
+        ):
             return run_source_normalize(
                 config_path=args.config,
                 input_path=args.input,

{threadsense-0.2.1 → threadsense-0.3.0}/src/threadsense/config.py

@@ -50,7 +50,7 @@ class RuntimeConfig(BaseModel):
 class SourcePolicyConfig(BaseModel):
     model_config = ConfigDict(frozen=True)
 
-    enabled_sources: tuple[str, ...] = ("reddit", "hackernews")
+    enabled_sources: tuple[str, ...] = ("reddit", "hackernews", "github_discussions", "github_gist")
 
     @field_validator("enabled_sources", mode="before")
     @classmethod
@@ -97,6 +97,30 @@ class GitHubConfig(BaseModel):
         return v.strip()
 
 
+class GitHubGistConfig(BaseModel):
+    model_config = ConfigDict(frozen=True)
+
+    base_url: str = "https://api.github.com"
+    token: str = ""
+    timeout_seconds: float = 15
+    comments_per_page: int = 100
+    request_delay_seconds: float = 0.5
+    max_retries: int = 2
+    backoff_seconds: float = 1.0
+
+    @field_validator("token")
+    @classmethod
+    def validate_token(cls, v: str) -> str:
+        return v.strip()
+
+    @field_validator("comments_per_page")
+    @classmethod
+    def validate_comments_per_page(cls, v: int) -> int:
+        if v < 1 or v > 100:
+            raise ValueError("comments_per_page must be between 1 and 100")
+        return v
+
+
 class CacheConfig(BaseModel):
     model_config = ConfigDict(frozen=True)
 
@@ -194,6 +218,7 @@ class AppConfig(BaseModel):
     reddit: RedditConfig = RedditConfig()
     hackernews: HackerNewsConfig = HackerNewsConfig()
     github: GitHubConfig = GitHubConfig()
+    github_gist: GitHubGistConfig = GitHubGistConfig()
     cache: CacheConfig = CacheConfig()
     storage: StorageConfig = StorageConfig()
     batch: BatchConfig = BatchConfig()
@@ -227,6 +252,11 @@ _ENV_MAP: dict[str, tuple[str, ...]] = {
     "THREADSENSE_GITHUB_BASE_URL": ("github", "base_url"),
     "THREADSENSE_GITHUB_TOKEN": ("github", "token"),
     "THREADSENSE_GITHUB_TIMEOUT": ("github", "timeout_seconds"),
+    "THREADSENSE_GITHUB_GIST_BASE_URL": ("github_gist", "base_url"),
+    "THREADSENSE_GITHUB_GIST_TOKEN": ("github_gist", "token"),
+    "THREADSENSE_GITHUB_GIST_TIMEOUT": ("github_gist", "timeout_seconds"),
+    "THREADSENSE_GITHUB_GIST_COMMENTS_PER_PAGE": ("github_gist", "comments_per_page"),
+    "THREADSENSE_GITHUB_GIST_REQUEST_DELAY": ("github_gist", "request_delay_seconds"),
     "THREADSENSE_CACHE_ENABLED": ("cache", "enabled"),
     "THREADSENSE_CACHE_TTL_SECONDS": ("cache", "ttl_seconds"),
     "THREADSENSE_CACHE_DIR": ("cache", "cache_dir"),
@@ -262,6 +292,7 @@ _TOML_SECTION_MAP: dict[str, str] = {
     "reddit": "reddit",
     "hackernews": "hackernews",
     "github": "github",
+    "github_gist": "github_gist",
     "cache": "cache",
     "storage": "storage",
     "batch": "batch",