ietf-notebook 0.1.0__tar.gz

ietf_notebook-0.1.0/LICENSE.md

@@ -0,0 +1,19 @@
+Copyright (c) Mark Nottingham
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

ietf_notebook-0.1.0/MANIFEST.in

@@ -0,0 +1 @@
+include ietf_notebook/py.typed

ietf_notebook-0.1.0/PKG-INFO

@@ -0,0 +1,85 @@
+Metadata-Version: 2.4
+Name: ietf-notebook
+Version: 0.1.0
+Summary: Gather resources for an IETF Working Group for use in NotebookLM
+Author-email: Mark Nottingham <mnot@mnot.net>
+License: Copyright (c) Mark Nottingham
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in
+        all copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+        THE SOFTWARE.
+        
+Project-URL: homepage, https://github.com/mnot/ietf-notebook
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 4 - Beta
+Classifier: License :: OSI Approved :: MIT License
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE.md
+Requires-Dist: requests
+Requires-Dist: beautifulsoup4
+Provides-Extra: dev
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: pylint; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-md; extra == "dev"
+Requires-Dist: validate-pyproject; extra == "dev"
+Requires-Dist: build; extra == "dev"
+Requires-Dist: types-requests; extra == "dev"
+Requires-Dist: types-beautifulsoup4; extra == "dev"
+Dynamic: license-file
+
+# ietf-notebook
+
+Automate gathering of [NotebookLM](https://notebooklm.google.com/)-ready documents for an [IETF](https://www.ietf.org/) Working Group.
+
+This tool gathers Working Group charters, drafts, meeting minutes, PDF slides, mailing list archives, and GitHub issues into a set of clean text files and PDFs suitable for ingestion into NotebookLM.
+
+## Installation
+
+```bash
+pipx install ietf-notebook
+```
+
+## Usage
+
+```bash
+ietf-notebook [wg_shortname] _OPTIONS_
+```
+
+### Options
+
+- `wg_shortname`: IETF Working Group short name (e.g., `httpbis`).
+- `--destination`: Folder to save files in (default: current directory).
+- `--github`: GitHub org/repo for issues (e.g., `ietf-wg-httpbis/wg-materials`).
+- `--months`: Number of months of mailing list history to fetch (default: all).
+- `--force`: Force re-downloading of existing files. By default, the tool skips files that already exist in the destination.
+- `--quiet`: No messages except for errors and the final resource summary.
+- `--verbose`: Detailed progress reporting.
+
+### Default Behavior
+
+- **Charters, Meetings, and Mbox**: Existing files are skipped unless `--force` is used.
+- **Mailing List Discovery**: The tool automatically finds the mailing list for the WG from the Datatracker.
+- **IMAP Retrieval**: Mailing list archives are fetched via IMAP from `imap.ietf.org` and cached locally in `.imap-cache/`.
+- **GitHub Strategy**: The tool first checks for `archive.json` on the `gh-pages` branch (common in repos using [Martin Thomson's template](https://github.com/martinthomson/internet-draft-template)).
+- **GitHub Auth**: To avoid rate limits when fetching from the API, set the `GITHUB_TOKEN` environment variable.
+
+## Contributing
+
+Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

ietf_notebook-0.1.0/README.md

@@ -0,0 +1,39 @@
+# ietf-notebook
+
+Automate gathering of [NotebookLM](https://notebooklm.google.com/)-ready documents for an [IETF](https://www.ietf.org/) Working Group.
+
+This tool gathers Working Group charters, drafts, meeting minutes, PDF slides, mailing list archives, and GitHub issues into a set of clean text files and PDFs suitable for ingestion into NotebookLM.
+
+## Installation
+
+```bash
+pipx install ietf-notebook
+```
+
+## Usage
+
+```bash
+ietf-notebook [wg_shortname] _OPTIONS_
+```
+
+### Options
+
+- `wg_shortname`: IETF Working Group short name (e.g., `httpbis`).
+- `--destination`: Folder to save files in (default: current directory).
+- `--github`: GitHub org/repo for issues (e.g., `ietf-wg-httpbis/wg-materials`).
+- `--months`: Number of months of mailing list history to fetch (default: all).
+- `--force`: Force re-downloading of existing files. By default, the tool skips files that already exist in the destination.
+- `--quiet`: No messages except for errors and the final resource summary.
+- `--verbose`: Detailed progress reporting.
+
+### Default Behavior
+
+- **Charters, Meetings, and Mbox**: Existing files are skipped unless `--force` is used.
+- **Mailing List Discovery**: The tool automatically finds the mailing list for the WG from the Datatracker.
+- **IMAP Retrieval**: Mailing list archives are fetched via IMAP from `imap.ietf.org` and cached locally in `.imap-cache/`.
+- **GitHub Strategy**: The tool first checks for `archive.json` on the `gh-pages` branch (common in repos using [Martin Thomson's template](https://github.com/martinthomson/internet-draft-template)).
+- **GitHub Auth**: To avoid rate limits when fetching from the API, set the `GITHUB_TOKEN` environment variable.
+
+## Contributing
+
+Pull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.

ietf_notebook-0.1.0/ietf_notebook/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.1.0"

ietf_notebook-0.1.0/ietf_notebook/__main__.py

@@ -0,0 +1,120 @@
+import argparse
+import os
+from .mbox import sync_mailing_list
+from .github import download_github_issues, process_github_issues
+from .meetings import process_meetings
+from .charter import process_charter
+from .drafts import process_drafts
+from .utils import Verbosity, LogLevel, log
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Automate creation of NotebookLM-ready documents for an IETF Working Group."
+    )
+    parser.add_argument("wg", help="IETF Working Group short name (e.g., 'httpbis')")
+    parser.add_argument(
+        "--github", help="GitHub owner/repo (e.g., 'ietf-wg-httpbis/wg-materials')"
+    )
+    parser.add_argument(
+        "--months",
+        type=int,
+        default=None,
+        help="Number of months of mailing list archives to fetch (default: all)",
+    )
+    parser.add_argument(
+        "--destination",
+        default=".",
+        help="Destination folder (default: current directory)",
+    )
+    parser.add_argument(
+        "--force",
+        action="store_true",
+        help="Overwrite resources that already exist in the destination",
+    )
+    parser.add_argument("--quiet", "-q", action="store_true", help="Only output errors")
+    parser.add_argument(
+        "--verbose", "-v", action="store_true", help="Detailed progress reporting"
+    )
+
+    args = parser.parse_args()
+
+    if not os.path.exists(args.destination):
+        os.makedirs(args.destination)
+
+    verbosity = Verbosity.STATUS
+    if args.quiet:
+        verbosity = Verbosity.QUIET
+    elif args.verbose:
+        verbosity = Verbosity.VERBOSE
+
+    if verbosity != Verbosity.QUIET:
+        print(f"Processing WG: {args.wg}")
+        print(f"Destination: {args.destination}")
+        if args.force:
+            print("Force mode: overwriting existing files.")
+        else:
+            print("Default mode: skipping existing files (except GitHub issues).")
+        print("-" * 40)
+
+    results = []
+
+    # 1. Charter
+    charter_file = os.path.join(args.destination, f"{args.wg}-charter.txt")
+    if not args.force and os.path.exists(charter_file):
+        log(
+            f"Skipping charter: {charter_file} already exists.",
+            verbosity,
+            level=LogLevel.PROGRESS,
+        )
+    else:
+        results.extend(process_charter(args.wg, charter_file, verbose=verbosity))
+
+    # 2. Meetings
+    results.extend(
+        process_meetings(args.wg, args.destination, force=args.force, verbose=verbosity)
+    )
+
+    # 3. Mailing List
+    results.extend(
+        sync_mailing_list(
+            args.wg, args.destination, months=args.months, verbose=verbosity
+        )
+    )
+
+    # 4. Drafts
+    results.extend(
+        process_drafts(args.wg, args.destination, force=args.force, verbose=verbosity)
+    )
+
+    # 5. GitHub Issues
+    if args.github:
+        gh_json = os.path.join(args.destination, f"{args.wg}-github-issues.json")
+        gh_txt = os.path.join(args.destination, f"{args.wg}-github-issues.txt")
+
+        if download_github_issues(args.github, gh_json, verbose=verbosity):
+            results.extend(process_github_issues(gh_json, gh_txt, verbose=verbosity))
+            try:
+                os.remove(gh_json)
+            except OSError as err:
+                log(f"Error cleaning up {gh_json}: {err}", verbosity, level=LogLevel.ERROR)
+    else:
+        log(
+            "Skip GitHub issues: no GitHub repo provided.",
+            verbosity,
+            level=LogLevel.PROGRESS,
+        )
+
+    if verbosity != Verbosity.QUIET:
+        print("-" * 40)
+        print("All tasks completed.")
+
+    if results:
+        print("\n## Updated Resources")
+        for res in sorted(list(set(results))):
+            rel_path = os.path.relpath(res, os.getcwd())
+            print(f"- {rel_path}")
+
+
+if __name__ == "__main__":
+    main()

ietf_notebook-0.1.0/ietf_notebook/charter.py

@@ -0,0 +1,64 @@
+from typing import List
+from bs4 import BeautifulSoup
+from .utils import Verbosity, LogLevel, clean_html, fetch_resource, log
+
+
+def process_charter(
+    wg_name: str, output_file: str, verbose: Verbosity = Verbosity.STATUS
+) -> List[str]:
+    """Fetch the WG charter and write to output_file. Returns list of updated files."""
+    url = f"https://datatracker.ietf.org/doc/charter-ietf-{wg_name}/"
+    log(f"Fetching charter for {wg_name}...", verbose, level=LogLevel.STATUS)
+
+    # Try fetching as markdown first
+    res = fetch_resource(url, headers={"Accept": "text/markdown"})
+    if not res:
+        log(f"Error: Could not fetch charter from {url}", verbose, level=LogLevel.ERROR)
+        return []
+
+    charter_text = ""
+    if "text/markdown" in res.headers.get("Content-Type", ""):
+        charter_text = res.text
+    else:
+        # Fallback to HTML cleaning
+        html = res.text
+        bs_soup = BeautifulSoup(html, "html.parser")
+
+        # The charter text is usually in a div with class 'card-body' on the datatracker.
+        charter_div = bs_soup.find("div", class_="card-body")
+
+        if not charter_div:
+            # Fallback to charter-text or similar
+            charter_div = bs_soup.find("div", class_="charter-text")
+
+        if not charter_div:
+            # Fallback to looking for the "Charter" heading
+            heading = None
+            for h2 in bs_soup.find_all("h2"):
+                if h2.string and "Charter" in h2.string:
+                    heading = h2
+                    break
+            if heading:
+                charter_div = heading.find_next("div")
+
+        if charter_div:
+            charter_text = clean_html(str(charter_div))
+        else:
+            # Last resort: clean the whole page but it might be noisy
+            log(
+                "Warning: Could not isolate charter text, cleaning entire page.",
+                verbose,
+                level=LogLevel.PROGRESS,
+            )
+            charter_text = clean_html(html)
+
+    if charter_text:
+        with open(output_file, "w", encoding="utf-8") as out_fh:
+            out_fh.write(f"Working Group Charter: {wg_name}\n")
+            out_fh.write(f"Source: {url}\n")
+            out_fh.write("=" * 80 + "\n\n")
+            out_fh.write(charter_text + "\n")
+
+        log(f"Done! Charter written to {output_file}.", verbose, level=LogLevel.STATUS)
+        return [output_file]
+    return []

ietf_notebook-0.1.0/ietf_notebook/drafts.py

@@ -0,0 +1,94 @@
+import os
+import re
+from typing import List, Dict, Any
+from bs4 import BeautifulSoup
+from .utils import LogLevel, Verbosity, log, fetch_resource
+
+
+def get_adopted_drafts(
+    wg_name: str, verbose: Verbosity = Verbosity.STATUS
+) -> List[Dict[str, Any]]:
+    """Scrape WG documents page for active drafts."""
+    url = f"https://datatracker.ietf.org/wg/{wg_name}/documents/"
+    log(f"Finding adopted drafts for {wg_name}...", verbose, level=LogLevel.STATUS)
+    res = fetch_resource(url)
+    if not res:
+        return []
+
+    soup = BeautifulSoup(res.text, "html.parser")
+    drafts = []
+
+    # Strategy: look for links starting with /doc/draft-ietf-{wg_name}-
+    # The link text contains the draft name + version
+    pattern = f"/doc/draft-ietf-{wg_name}-"
+    for a_tag in soup.find_all("a", href=True):
+        href = a_tag.get("href")
+        if not isinstance(href, str) or not href.startswith(pattern):
+            continue
+
+        text = a_tag.get_text(strip=True)
+        # Text usually looks like "draft-ietf-aipref-vocab-05"
+        if text.startswith(f"draft-ietf-{wg_name}-"):
+            # Use regex to find "draft-..." + extension
+            match = re.search(
+                r"(draft-ietf-" + re.escape(wg_name) + r"-.*?)-(\d+)$", text
+            )
+            if match:
+                draft_name = match.group(1)
+                try:
+                    current_rev = int(match.group(2))
+                    drafts.append({"name": draft_name, "max_rev": current_rev})
+                except ValueError:
+                    continue
+
+    # De-duplicate and keep the highest revision found
+    unique_drafts: Dict[str, int] = {}
+    for draft_entry in drafts:
+        name = str(draft_entry["name"])
+        rev = int(draft_entry["max_rev"])
+        if name not in unique_drafts or rev > unique_drafts[name]:
+            unique_drafts[name] = rev
+
+    result = [{"name": name, "max_rev": rev} for name, rev in unique_drafts.items()]
+    return result
+
+
+def process_drafts(
+    wg_name: str,
+    destination: str,
+    force: bool = False,
+    verbose: Verbosity = Verbosity.STATUS,
+) -> List[str]:
+    """Download all revisions of WG drafts as text."""
+    updated = []
+    drafts = get_adopted_drafts(wg_name, verbose)
+    if not drafts:
+        log(f"No adopted drafts found for {wg_name}.", verbose, level=LogLevel.STATUS)
+        return []
+
+    for draft in drafts:
+        name = draft["name"]
+        max_rev = draft["max_rev"]
+        log(
+            f"Processing draft: {name} (revs 00 to {max_rev:02d})",
+            verbose,
+            level=LogLevel.STATUS,
+        )
+
+        for rev in range(max_rev + 1):
+            rev_str = f"{rev:02d}"
+            filename = f"{name}-{rev_str}.txt"
+            filepath = os.path.join(destination, filename)
+
+            if not force and os.path.exists(filepath):
+                continue
+
+            url = f"https://www.ietf.org/archive/id/{name}-{rev_str}.txt"
+            log(f"Downloading {filename}...", verbose, level=LogLevel.PROGRESS)
+            res = fetch_resource(url)
+            if res:
+                with open(filepath, "w", encoding="utf-8") as out_fh:
+                    out_fh.write(str(res.text))
+                updated.append(filepath)
+
+    return updated

ietf_notebook-0.1.0/ietf_notebook/github.py

@@ -0,0 +1,242 @@
+import json
+import os
+from datetime import datetime
+from typing import Optional, List, Dict, Any
+import requests
+from .utils import LogLevel, Verbosity, log
+
+
+def format_date(iso_date: Optional[str]) -> str:
+    """Convert ISO date to a more readable format."""
+    if not iso_date:
+        return "(Unknown Date)"
+    try:
+        dt = datetime.fromisoformat(iso_date.replace("Z", "+00:00"))
+        return dt.strftime("%Y-%m-%d %H:%M:%S %Z")
+    except (ValueError, TypeError):
+        return iso_date
+
+
+def process_github_issues(
+    input_file: str, output_file: str, verbose: Verbosity = Verbosity.STATUS
+) -> List[str]:
+    """Process a GitHub issues JSON archive and write cleaned text to output_file."""
+    log(f"Opening {input_file}...", verbose, level=LogLevel.PROGRESS)
+    try:
+        with open(input_file, "r", encoding="utf-8") as json_fh:
+            data = json.load(json_fh)
+    except (json.JSONDecodeError, OSError) as err:
+        log(f"Error parsing GitHub JSON: {err}", verbose, level=LogLevel.ERROR)
+        return []
+
+    issues = data.get("issues", [])
+    repo_name = data.get("repo", "Unknown Repo")
+
+    with open(output_file, "w", encoding="utf-8") as out_fh:
+        out_fh.write(f"Repository: {repo_name}\n")
+        out_fh.write(f"Archive Export Date: {format_date(data.get('timestamp'))}\n")
+        out_fh.write("=" * 80 + "\n\n")
+
+        for issue in issues:
+            title = issue.get("title", "(No Title)")
+            number = issue.get("number", "?")
+            state = issue.get("state", "(Unknown State)")
+            author = issue.get("author", "(Unknown Author)")
+            created_at = format_date(issue.get("createdAt"))
+            labels = ", ".join(issue.get("labels", []))
+            body = (issue.get("body") or "").strip()
+
+            out_fh.write(f"Issue #{number}: {title}\n")
+            out_fh.write(f"State: {state}\n")
+            out_fh.write(f"Date: {created_at}\n")
+            out_fh.write(f"Author: {author}\n")
+            if labels:
+                out_fh.write(f"Labels: {labels}\n")
+            out_fh.write("\n")
+
+            out_fh.write((body or "(No description provided)") + "\n")
+
+            comments = issue.get("comments", [])
+            if comments:
+                out_fh.write("\n" + "-" * 40 + "\n")
+                out_fh.write(f"Comments ({len(comments)}):\n\n")
+                for comment in comments:
+                    c_author = comment.get("author", "(Unknown)")
+                    c_date = format_date(comment.get("createdAt"))
+                    c_body = (comment.get("body") or "").strip()
+
+                    out_fh.write(f"--- Comment by {c_author} on {c_date} ---\n")
+                    out_fh.write(c_body + "\n\n")
+
+            out_fh.write("=" * 80 + "\n\n")
+
+    log(
+        f"Done! Extracted {len(issues)} issues to {output_file}.",
+        verbose,
+        level=LogLevel.STATUS,
+    )
+    return [output_file]
+
+
+def download_github_issues(
+    repo_short: str,
+    dest_path: str,
+    token: Optional[str] = None,
+    verbose: Verbosity = Verbosity.STATUS,
+) -> bool:
+    """Download GitHub issues JSON using the API from 'owner/repo' short name."""
+    if repo_short.startswith("http"):
+        log(
+            f"Direct downloading GitHub issues from {repo_short}...",
+            verbose,
+            level=LogLevel.STATUS,
+        )
+        try:
+            response = requests.get(repo_short, timeout=60)
+            response.raise_for_status()
+            with open(dest_path, "w", encoding="utf-8") as json_file:
+                json_file.write(response.text)
+            return True
+        except (requests.RequestException, OSError) as err:
+            log(
+                f"Error downloading GitHub issues: {err}", verbose, level=LogLevel.ERROR
+            )
+            return False
+
+    # Expecting owner/repo
+    if "/" not in repo_short:
+        log(
+            f"Invalid GitHub short name: {repo_short}. Expected 'owner/repo'.",
+            verbose,
+            level=LogLevel.ERROR,
+        )
+        return False
+    owner, repo = repo_short.split("/", 1)
+    archive_url = f"https://{owner}.github.io/{repo}/archive.json"
+
+    log(
+        f"Checking for GitHub archive at {archive_url}...",
+        verbose,
+        level=LogLevel.STATUS,
+    )
+    try:
+        response = requests.get(archive_url, timeout=30)
+        if response.status_code == 200:
+            log("Archive found; downloading...", verbose, level=LogLevel.STATUS)
+            try:
+                archive_data = response.json()
+                # Ensure it's in our expected format (dict with 'issues' key)
+                if isinstance(archive_data, list):
+                    archive_data = {
+                        "repo": f"{owner}/{repo}",
+                        "timestamp": datetime.now().isoformat(),
+                        "issues": archive_data,
+                    }
+                elif "issues" not in archive_data:
+                    # If it's a dict but missing 'issues', we might still want to wrap it
+                    # or handle it differently. For now, assume it might be a single issue
+                    # or some other format and wrap if it's not our expected schema.
+                    archive_data = {
+                        "repo": f"{owner}/{repo}",
+                        "timestamp": datetime.now().isoformat(),
+                        "issues": [archive_data],
+                    }
+                with open(dest_path, "w", encoding="utf-8") as json_fh:
+                    json.dump(archive_data, json_fh, indent=2)
+                return True
+            except (json.JSONDecodeError, TypeError) as err:
+                log(f"Error parsing archive JSON: {err}", Verbosity.VERBOSE, level=LogLevel.STATUS)
+        log("No archive found on gh-pages.", verbose, level=LogLevel.PROGRESS)
+    except (requests.RequestException, OSError) as err:
+        log(
+            f"Error checking gh-pages archive: {err}",
+            verbose,
+            level=LogLevel.STATUS,
+        )
+
+    log(
+        f"Fetching GitHub issues via API for {owner}/{repo}...",
+        verbose,
+        level=LogLevel.STATUS,
+    )
+    headers = {"Accept": "application/vnd.github.v3+json"}
+    github_token = token or os.environ.get("GITHUB_TOKEN")
+    if github_token:
+        headers["Authorization"] = f"token {github_token}"
+
+    try:
+        all_issues = _fetch_all_issues(owner, repo, headers, verbose)
+        export_data = {
+            "repo": f"{owner}/{repo}",
+            "timestamp": datetime.now().isoformat(),
+            "issues": all_issues,
+        }
+        with open(dest_path, "w", encoding="utf-8") as json_fh:
+            json.dump(export_data, json_fh, indent=2)
+        return True
+    except (requests.RequestException, OSError) as err:
+        log(f"Error fetching GitHub issues: {err}", verbose, level=LogLevel.ERROR)
+        return False
+
+
+def _fetch_all_issues(
+    owner: str, repo_name: str, headers: Dict[str, str], verbose: Verbosity
+) -> List[Dict[str, Any]]:
+    """Fetch all issues and their comments from GitHub API."""
+    all_issues = []
+    page = 1
+    while True:
+        api_url = (
+            f"https://api.github.com/repos/{owner}/{repo_name}/issues"
+            f"?state=all&page={page}&per_page=100"
+        )
+        res = requests.get(api_url, headers=headers, timeout=60)
+        res.raise_for_status()
+        issues = res.json()
+        if not issues:
+            break
+
+        for issue in issues:
+            # GitHub API returns both issues and PRs (PRs have a 'pull_request' key)
+            if "pull_request" in issue:
+                continue
+
+            issue_data = {
+                "number": issue.get("number"),
+                "title": issue.get("title"),
+                "state": issue.get("state"),
+                "author": issue.get("user", {}).get("login"),
+                "createdAt": issue.get("created_at"),
+                "labels": [l.get("name") for l in issue.get("labels", [])],
+                "body": issue.get("body"),
+                "comments": [],
+            }
+            if issue.get("comments", 0) > 0:
+                issue_data["comments"] = _fetch_issue_comments(
+                    issue.get("comments_url"), headers
+                )
+
+            all_issues.append(issue_data)
+
+        page += 1
+        if len(issues) < 100:
+            break
+    return all_issues
+
+
+def _fetch_issue_comments(
+    comments_url: str, headers: Dict[str, str]
+) -> List[Dict[str, Any]]:
+    """Fetch comments for a specific issue."""
+    c_res = requests.get(comments_url, headers=headers, timeout=30)
+    if c_res.status_code == 200:
+        comments = c_res.json()
+        return [
+            {
+                "author": comment.get("user", {}).get("login"),
+                "createdAt": comment.get("created_at"),
+                "body": comment.get("body"),
+            }
+            for comment in comments
+        ]
+    return []