topicscout 0.1.0__tar.gz

topicscout-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Adecubed
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

topicscout-0.1.0/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.4
+Name: topicscout
+Version: 0.1.0
+Summary: Find the GitHub repos on a topic, and only the new ones next time. No dependencies, token optional.
+Author: Adecubed
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/adecubed/topicscout
+Keywords: github,scraper,search,discovery,topics
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# topicscout
+
+[![ci](https://github.com/adecubed/topicscout/actions/workflows/ci.yml/badge.svg)](https://github.com/adecubed/topicscout/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/topicscout)](https://pypi.org/project/topicscout/)
+
+Find the GitHub repositories on a topic, and next time only the new ones.
+
+```bash
+pip install topicscout        # or run it without installing: uvx topicscout run github-scrapers
+topicscout run github-scrapers
+```
+
+```
+19 found, 17 kept, 17 new -> scout-github-scrapers/new.md
+```
+
+`new.md` is a table of what appeared since the last run, most stars first: stars,
+last push, language, license, a guess at the interface from the README (MCP, pip,
+npm, REST, Docker...) and the description. `candidates.json` keeps everything seen,
+so a weekly run is a short list, not the same 300 repos again.
+
+No dependencies, Python 3.11+. `GITHUB_TOKEN` is optional: without it the searches
+are paced to stay under GitHub's 10 per minute and READMEs are capped at 40 per run.
+
+## Why
+
+GitHub search is good at "repos with this topic" and bad at "what's new since I last
+looked, minus the forks, the awesome-lists, the abandoned ones and the repos that
+only tagged themselves". topicscout runs a fixed set of searches, applies your
+filters, remembers what it has seen and what you already have.
+
+It was the discovery step of [adebench](https://github.com/adecubed/adebench), a
+benchmark for AI memory systems, which uses it to find new memories to measure.
+
+## Your own topic
+
+Copy a built-in profile (they are in [`topicscout/profiles/`](topicscout/profiles/)),
+change the queries and the words a hit must contain, and run it:
+
+```bash
+topicscout run my-topic.json
+```
+
+Run it again next week: `new.md` lists only what appeared in between. Put the repos
+you already know or rejected in `scout-my-topic/known.txt`.
+
+## From a script or an agent
+
+`--json` prints the result on stdout, progress stays on stderr:
+
+```bash
+topicscout run github-scrapers --json -q
+```
+
+```json
+{"profile": "github-scrapers", "found": 19, "kept": 17, "new": [{"full_name": "yusufkaraaslan/Skill_Seekers", "stars": 15052, "interface": ["cli", "pip", "action"], "...": "..."}, "..."], "stopped": null, "out": "scout-github-scrapers"}
+```
+
+It is a command-line tool and a Python library (`from topicscout.core import Profile, run`),
+not an MCP server; an agent calls it like any other command.
+
+## Commands
+
+```bash
+topicscout run PROFILE [--out DIR] [--min-stars N] [--days N] [--readme-max N]
+                       [--known-from GLOB ...] [--json] [-q]
+topicscout profiles    # built-in profiles
+topicscout doctor      # token and remaining GitHub quota
+```
+
+- `PROFILE` is a built-in name or a path to your own JSON profile.
+- `--out` is the state folder (default `scout-<profile>`): `candidates.json`,
+  `new.md`, and your `known.txt`.
+- `--json` prints the summary and the new repos as JSON on stdout; progress and
+  errors always go to stderr, so scripts and agents can read stdout as is.
+
+## What counts as known
+
+Repos you already have are marked `known` and never listed as new:
+
+- `known.txt` in the state folder: one `owner/repo` per line, `#` comments. Put
+  rejected repos here too.
+- `--known-from "adapters/*.py"`: the GitHub URLs in the first 20 lines of those
+  files. adebench points it at its adapters, so a memory with an adapter drops out.
+
+## Profiles
+
+Built in: `ai-memory` (memory systems for AI agents and LLMs) and `github-scrapers`
+(tools that scrape, crawl or mine GitHub). A profile is a JSON file:
+
+```json
+{
+  "name": "ai-memory",
+  "description": "Memory systems for AI agents and LLMs",
+  "queries": ["topic:agent-memory", "\"llm memory\" in:name,description"],
+  "exclude": "(?i)awesome|curated list|\\bpapers\\b",
+  "require": [
+    {"pattern": "(?i)memor|\\bmem\\b", "own_words": true},
+    {"pattern": "(?i)\\b(agents?|llms?|ai|mcp)\\b"}
+  ],
+  "interfaces": [["mcp", "(?i)\\bmcp\\b"], ["pip", "(?i)\\bpip install\\b"]],
+  "min_stars": 10,
+  "days": 90
+}
+```
+
+- `queries`: [GitHub repository search](https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories)
+  queries. Each one always gets `fork:false archived:false stars:>=min_stars
+  pushed:>=today-days`; up to 300 results per query.
+- `exclude`: a regex on name, description and topics; a match drops the repo.
+- `require`: every pattern must match. With `own_words` it must be in the name or
+  description, not only in a topic: a database that tagged itself `agent-memory`
+  is not a memory. Hyphens and underscores count as spaces.
+- `interfaces`: `[label, regex]` pairs tried on the README of each new repo.
+  Leave it out and no README is fetched.
+- `require_language` (default true) drops repos with no language, which are
+  usually lists and docs.
+
+Patterns are Python regexes; add `(?i)` for case-insensitive.
+
+## Rate limits
+
+On a 403 or 429 it waits for the reset if that is under 70 seconds; otherwise it
+stops, saves what it found and says so in `new.md` and on stderr. The next run
+picks up the READMEs it did not get to.
+
+## How it compares
+
+- [ghcrawl](https://github.com/pwrdrvr/ghcrawl) goes deep into one repository
+  (issues and PRs, embeddings, clusters); topicscout goes wide across GitHub for a
+  topic.
+- [top-github-scraper](https://github.com/khuyentran1401/top-github-scraper)
+  lists the top repos for a keyword, once; topicscout adds filters, profiles and
+  memory across runs.
+
+## License
+
+MIT

topicscout-0.1.0/README.md

@@ -0,0 +1,138 @@
+# topicscout
+
+[![ci](https://github.com/adecubed/topicscout/actions/workflows/ci.yml/badge.svg)](https://github.com/adecubed/topicscout/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/topicscout)](https://pypi.org/project/topicscout/)
+
+Find the GitHub repositories on a topic, and next time only the new ones.
+
+```bash
+pip install topicscout        # or run it without installing: uvx topicscout run github-scrapers
+topicscout run github-scrapers
+```
+
+```
+19 found, 17 kept, 17 new -> scout-github-scrapers/new.md
+```
+
+`new.md` is a table of what appeared since the last run, most stars first: stars,
+last push, language, license, a guess at the interface from the README (MCP, pip,
+npm, REST, Docker...) and the description. `candidates.json` keeps everything seen,
+so a weekly run is a short list, not the same 300 repos again.
+
+No dependencies, Python 3.11+. `GITHUB_TOKEN` is optional: without it the searches
+are paced to stay under GitHub's 10 per minute and READMEs are capped at 40 per run.
+
+## Why
+
+GitHub search is good at "repos with this topic" and bad at "what's new since I last
+looked, minus the forks, the awesome-lists, the abandoned ones and the repos that
+only tagged themselves". topicscout runs a fixed set of searches, applies your
+filters, remembers what it has seen and what you already have.
+
+It was the discovery step of [adebench](https://github.com/adecubed/adebench), a
+benchmark for AI memory systems, which uses it to find new memories to measure.
+
+## Your own topic
+
+Copy a built-in profile (they are in [`topicscout/profiles/`](topicscout/profiles/)),
+change the queries and the words a hit must contain, and run it:
+
+```bash
+topicscout run my-topic.json
+```
+
+Run it again next week: `new.md` lists only what appeared in between. Put the repos
+you already know or rejected in `scout-my-topic/known.txt`.
+
+## From a script or an agent
+
+`--json` prints the result on stdout, progress stays on stderr:
+
+```bash
+topicscout run github-scrapers --json -q
+```
+
+```json
+{"profile": "github-scrapers", "found": 19, "kept": 17, "new": [{"full_name": "yusufkaraaslan/Skill_Seekers", "stars": 15052, "interface": ["cli", "pip", "action"], "...": "..."}, "..."], "stopped": null, "out": "scout-github-scrapers"}
+```
+
+It is a command-line tool and a Python library (`from topicscout.core import Profile, run`),
+not an MCP server; an agent calls it like any other command.
+
+## Commands
+
+```bash
+topicscout run PROFILE [--out DIR] [--min-stars N] [--days N] [--readme-max N]
+                       [--known-from GLOB ...] [--json] [-q]
+topicscout profiles    # built-in profiles
+topicscout doctor      # token and remaining GitHub quota
+```
+
+- `PROFILE` is a built-in name or a path to your own JSON profile.
+- `--out` is the state folder (default `scout-<profile>`): `candidates.json`,
+  `new.md`, and your `known.txt`.
+- `--json` prints the summary and the new repos as JSON on stdout; progress and
+  errors always go to stderr, so scripts and agents can read stdout as is.
+
+## What counts as known
+
+Repos you already have are marked `known` and never listed as new:
+
+- `known.txt` in the state folder: one `owner/repo` per line, `#` comments. Put
+  rejected repos here too.
+- `--known-from "adapters/*.py"`: the GitHub URLs in the first 20 lines of those
+  files. adebench points it at its adapters, so a memory with an adapter drops out.
+
+## Profiles
+
+Built in: `ai-memory` (memory systems for AI agents and LLMs) and `github-scrapers`
+(tools that scrape, crawl or mine GitHub). A profile is a JSON file:
+
+```json
+{
+  "name": "ai-memory",
+  "description": "Memory systems for AI agents and LLMs",
+  "queries": ["topic:agent-memory", "\"llm memory\" in:name,description"],
+  "exclude": "(?i)awesome|curated list|\\bpapers\\b",
+  "require": [
+    {"pattern": "(?i)memor|\\bmem\\b", "own_words": true},
+    {"pattern": "(?i)\\b(agents?|llms?|ai|mcp)\\b"}
+  ],
+  "interfaces": [["mcp", "(?i)\\bmcp\\b"], ["pip", "(?i)\\bpip install\\b"]],
+  "min_stars": 10,
+  "days": 90
+}
+```
+
+- `queries`: [GitHub repository search](https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories)
+  queries. Each one always gets `fork:false archived:false stars:>=min_stars
+  pushed:>=today-days`; up to 300 results per query.
+- `exclude`: a regex on name, description and topics; a match drops the repo.
+- `require`: every pattern must match. With `own_words` it must be in the name or
+  description, not only in a topic: a database that tagged itself `agent-memory`
+  is not a memory. Hyphens and underscores count as spaces.
+- `interfaces`: `[label, regex]` pairs tried on the README of each new repo.
+  Leave it out and no README is fetched.
+- `require_language` (default true) drops repos with no language, which are
+  usually lists and docs.
+
+Patterns are Python regexes; add `(?i)` for case-insensitive.
+
+## Rate limits
+
+On a 403 or 429 it waits for the reset if that is under 70 seconds; otherwise it
+stops, saves what it found and says so in `new.md` and on stderr. The next run
+picks up the READMEs it did not get to.
+
+## How it compares
+
+- [ghcrawl](https://github.com/pwrdrvr/ghcrawl) goes deep into one repository
+  (issues and PRs, embeddings, clusters); topicscout goes wide across GitHub for a
+  topic.
+- [top-github-scraper](https://github.com/khuyentran1401/top-github-scraper)
+  lists the top repos for a keyword, once; topicscout adds filters, profiles and
+  memory across runs.
+
+## License
+
+MIT

topicscout-0.1.0/pyproject.toml

@@ -0,0 +1,26 @@
+[project]
+name = "topicscout"
+version = "0.1.0"
+description = "Find the GitHub repos on a topic, and only the new ones next time. No dependencies, token optional."
+readme = "README.md"
+requires-python = ">=3.11"
+license = "MIT"
+authors = [{name = "Adecubed"}]
+dependencies = []
+keywords = ["github", "scraper", "search", "discovery", "topics"]
+
+[project.urls]
+Homepage = "https://github.com/adecubed/topicscout"
+
+[project.scripts]
+topicscout = "topicscout.cli:main"
+
+[build-system]
+requires = ["setuptools>=77"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = ["topicscout", "topicscout.profiles"]
+
+[tool.setuptools.package-data]
+"topicscout.profiles" = ["*.json"]

topicscout-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

topicscout-0.1.0/tests/test_topicscout.py

@@ -0,0 +1,198 @@
+"""topicscout with fake HTTP (no network)."""
+from __future__ import annotations
+
+import json
+from datetime import date
+from urllib.parse import parse_qs, urlparse
+
+import pytest
+
+from topicscout import cli, core
+from topicscout.core import Profile
+
+TODAY = date(2026, 9, 29)
+MEM = Profile.load("ai-memory")
+
+
+def repo(name, stars=50, pushed="2026-09-20T10:00:00Z", fork=False, archived=False,
+         language="Python", description="Long-term memory for agents", topics=("agent-memory",)):
+    return {"full_name": name, "html_url": f"https://github.com/{name}", "stargazers_count": stars,
+            "pushed_at": pushed, "fork": fork, "archived": archived, "language": language,
+            "description": description, "license": {"spdx_id": "MIT"}, "topics": list(topics)}
+
+
+class FakeGitHub:
+    """Every search query returns the same items; READMEs from a dict."""
+
+    def __init__(self, items, readmes=None, limited_after=None):
+        self.items, self.readmes = items, readmes or {}
+        self.limited_after = limited_after
+        self.searches, self.readme_calls = [], []
+
+    def __call__(self, url, accept):
+        path = urlparse(url).path
+        if path == "/search/repositories":
+            if self.limited_after is not None and len(self.searches) >= self.limited_after:
+                return 403, {"x-ratelimit-remaining": "0", "x-ratelimit-reset": "9999999999"}, "{}"
+            self.searches.append(parse_qs(urlparse(url).query)["q"][0])
+            return 200, {}, json.dumps({"items": self.items})
+        if path.endswith("/readme"):
+            name = path[len("/repos/"):-len("/readme")]
+            self.readme_calls.append(name)
+            if name in self.readmes:
+                return 200, {}, self.readmes[name]
+            return 404, {}, ""
+        raise AssertionError(url)
+
+
+def run(tmp_path, gh, profile=MEM, **kw):
+    return core.run(profile, tmp_path / "scout", gh, today=TODAY, sleep=lambda s: None, **kw)
+
+
+def load(tmp_path):
+    return json.loads((tmp_path / "scout" / "candidates.json").read_text(encoding="utf-8"))["repos"]
+
+
+def one_query(**extra):
+    return Profile({"name": "t", "queries": ["topic:a"], **extra})
+
+
+def test_queries_carry_the_filters():
+    gh = FakeGitHub([])
+    core.search(gh, ["topic:agent-memory"], min_stars=10, days=90, today=TODAY, sleep=lambda s: None)
+    q = gh.searches[0]
+    assert "topic:agent-memory" in q and "fork:false" in q and "archived:false" in q
+    assert "stars:>=10" in q and "pushed:>=2026-07-01" in q
+
+
+def test_ai_memory_drops_forks_archived_stale_small_lists(tmp_path):
+    items = [repo("a/good"), repo("a/fork", fork=True), repo("a/old", archived=True),
+             repo("a/stale", pushed="2026-01-01T00:00:00Z"), repo("a/tiny", stars=3),
+             repo("a/nolang", language=None), repo("a/awesome-agent-memory"),
+             repo("a/papers", description="A curated list of memory papers"),
+             repo("a/db", description="Git for data, for agents", topics=("agent-memory", "database")),
+             repo("a/claude-mem", description="Persistent context across sessions for every agent")]
+    run(tmp_path, FakeGitHub(items))
+    assert set(load(tmp_path)) == {"a/good", "a/claude-mem"}
+
+
+def test_github_scrapers_profile():
+    gs = Profile.load("github-scrapers")
+    cutoff = date(2025, 9, 29)
+    ok = repo("n/github-scraper", description="crawl GitHub web pages", topics=())
+    spam = repo("x/OnlySnap", description="Scrape OnlyFans, a GitHub mirror", topics=())
+    other = repo("y/web-scraper", description="scrape any website", topics=())
+    assert core.keep(ok, gs, min_stars=5, cutoff=cutoff)
+    assert not core.keep(spam, gs, min_stars=5, cutoff=cutoff)
+    assert not core.keep(other, gs, min_stars=5, cutoff=cutoff)
+
+
+def test_require_own_words_ignores_topics():
+    p = one_query(require=[{"pattern": "(?i)memory", "own_words": True}])
+    tagged = repo("a/db", description="Git for data", topics=("agent-memory",))
+    said = repo("a/mem", description="memory for agents", topics=())
+    cutoff = date(2026, 1, 1)
+    assert not core.keep(tagged, p, min_stars=1, cutoff=cutoff)
+    assert core.keep(said, p, min_stars=1, cutoff=cutoff)
+
+
+def test_require_language_can_be_off():
+    p = one_query(require_language=False)
+    assert core.keep(repo("a/docs", language=None), p, min_stars=1, cutoff=date(2026, 1, 1))
+
+
+def test_interface_guess_from_readme():
+    g = lambda s: core.guess_interface(s, MEM)  # noqa: E731
+    assert g("Add it to Claude: an MCP server (modelcontextprotocol)") == ["mcp"]
+    assert g("pip install foo\n\ncurl http://localhost:8080/v1/memories") == ["pip", "rest"]
+    assert g("npm install -g bar; docker compose up") == ["npm", "docker"]
+    assert g("A Claude Code plugin: /plugin install x") == ["plugin"]
+    assert g("") == []
+
+
+def test_no_interfaces_no_readmes(tmp_path):
+    gh = FakeGitHub([repo("a/one")])
+    run(tmp_path, gh, profile=one_query())
+    assert gh.readme_calls == [] and load(tmp_path)["a/one"]["interface"] is None
+
+
+def test_known_from_files_and_known_txt(tmp_path):
+    ad = tmp_path / "adapters"
+    ad.mkdir()
+    (ad / "x.py").write_text('"""Adapter for X (https://github.com/Owner/X-Mem)."""\n', encoding="utf-8")
+    (ad / "late.py").write_text("\n" * 30 + "# https://github.com/not/this\n", encoding="utf-8")
+    out = tmp_path / "scout"
+    out.mkdir()
+    (out / "known.txt").write_text("# rejected\nother/thing\n", encoding="utf-8")
+    known_from = [str(ad / "*.py")]
+    assert core.known_repos(out / "known.txt", known_from) == {"owner/x-mem", "other/thing"}
+    run(tmp_path, FakeGitHub([repo("owner/X-Mem"), repo("other/thing"), repo("new/one")]), known_from=known_from)
+    repos = load(tmp_path)
+    assert repos["owner/X-Mem"]["status"] == "known" and repos["new/one"]["status"] == "new"
+    md = (out / "new.md").read_text(encoding="utf-8")
+    assert "new/one" in md and "owner/X-Mem" not in md
+
+
+def test_state_across_runs(tmp_path):
+    gh = FakeGitHub([repo("a/one", stars=20)], readmes={"a/one": "pip install one"})
+    first = run(tmp_path, gh)
+    assert [r["full_name"] for r in first["new"]] == ["a/one"] and gh.readme_calls == ["a/one"]
+    gh.items = [repo("a/one", stars=25), repo("b/two", stars=90)]
+    second = run(tmp_path, gh)
+    assert [r["full_name"] for r in second["new"]] == ["b/two"]
+    assert gh.readme_calls == ["a/one", "b/two"]  # README only for first sightings
+    repos = load(tmp_path)
+    assert repos["a/one"]["status"] == "seen" and repos["a/one"]["stars"] == 25
+    assert repos["a/one"]["first_seen"] == "2026-09-29" and repos["a/one"]["interface"] == ["pip"]
+
+
+def test_new_md_most_stars_first(tmp_path):
+    run(tmp_path, FakeGitHub([repo("a/small", stars=11), repo("a/big", stars=900)]))
+    md = (tmp_path / "scout" / "new.md").read_text(encoding="utf-8")
+    assert md.startswith("# New repos for ai-memory") and md.index("a/big") < md.index("a/small")
+
+
+def test_readme_cap(tmp_path):
+    gh = FakeGitHub([repo(f"a/r{i}") for i in range(5)])
+    run(tmp_path, gh, readme_max=2)
+    assert len(gh.readme_calls) == 2
+
+
+def test_rate_limit_saves_what_was_found(tmp_path):
+    gh = FakeGitHub([repo("a/one")], limited_after=1)
+    p = one_query()
+    p.queries = ["topic:a", "topic:b", "topic:c"]
+    summary = run(tmp_path, gh, profile=p)
+    assert summary["stopped"] and "rate limit" in summary["stopped"]
+    assert set(load(tmp_path)) == {"a/one"}
+
+
+def test_profile_from_path_and_unknown(tmp_path):
+    f = tmp_path / "mine.json"
+    f.write_text(json.dumps({"queries": ["topic:x"]}), encoding="utf-8")
+    p = Profile.load(str(f))
+    assert p.name == "mine" and p.min_stars == 10 and p.days == 90
+    with pytest.raises(FileNotFoundError):
+        Profile.load("no-such-profile")
+    with pytest.raises(ValueError):
+        Profile({"queries": []})
+
+
+def test_builtin_profiles_all_load():
+    names = core.builtin_profiles()
+    assert {"ai-memory", "github-scrapers"} <= set(names)
+    for n in names:
+        Profile.load(n)
+
+
+def test_cli_run_json(tmp_path, monkeypatch, capsys):
+    monkeypatch.setattr(core, "fetch_github", FakeGitHub([repo("a/one")]))
+    monkeypatch.setattr(core.time, "sleep", lambda s: None)
+    rc = cli.main(["run", "ai-memory", "--out", str(tmp_path / "o"), "--json", "-q"])
+    out = json.loads(capsys.readouterr().out)
+    assert rc == 0 and out["profile"] == "ai-memory" and [r["full_name"] for r in out["new"]] == ["a/one"]
+
+
+def test_cli_unknown_profile(capsys):
+    assert cli.main(["run", "nope"]) == 2
+    assert "no profile" in capsys.readouterr().err

topicscout-0.1.0/topicscout/__init__.py

@@ -0,0 +1,2 @@
+"""topicscout: find the GitHub repos on a topic, and only the new ones next time."""
+__version__ = "0.1.0"

topicscout-0.1.0/topicscout/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())

topicscout-0.1.0/topicscout/cli.py

@@ -0,0 +1,103 @@
+"""topicscout run | profiles | doctor.
+
+Results go to stdout (a summary line, or JSON with --json); progress and
+errors go to stderr, so a script or an agent can read stdout as is.
+"""
+from __future__ import annotations
+
+import argparse
+import json
+import os
+import sys
+from datetime import datetime, timezone
+from pathlib import Path
+
+from . import __version__
+from .core import Profile, _stderr, builtin_profiles, rate_limit, run
+
+
+def _run(args) -> int:
+    try:
+        profile = Profile.load(args.profile)
+    except (FileNotFoundError, ValueError, KeyError) as e:
+        _stderr(f"topicscout: {e}")
+        return 2
+    out = Path(args.out or f"scout-{profile.name}")
+    log = (lambda m: None) if args.quiet else _stderr
+    if not os.environ.get("GITHUB_TOKEN"):
+        log("no GITHUB_TOKEN: searches paced to 10 per minute, READMEs capped at 40")
+    s = run(profile, out, min_stars=args.min_stars, days=args.days, readme_max=args.readme_max,
+            known_from=args.known_from, log=log)
+    if args.json:
+        print(json.dumps({**s, "out": str(out)}, indent=2, ensure_ascii=False))
+    else:
+        print(f"{s['found']} found, {s['kept']} kept, {len(s['new'])} new -> {out / 'new.md'}")
+    if s["stopped"]:
+        _stderr(f"stopped early: {s['stopped']}")
+    return 0
+
+
+def _profiles(args) -> int:
+    ps = builtin_profiles()
+    if args.json:
+        print(json.dumps(ps, indent=2))
+    else:
+        for name, desc in ps.items():
+            print(f"{name:20} {desc}")
+    return 0
+
+
+def _doctor(args) -> int:
+    token = bool(os.environ.get("GITHUB_TOKEN"))
+    report = {"version": __version__, "token": token}
+    try:
+        res = rate_limit()
+    except Exception as e:  # network, bad token: say it, don't crash
+        report["error"] = str(e)
+    else:
+        for key in ("search", "core"):
+            r = res[key]
+            reset = datetime.fromtimestamp(r["reset"], timezone.utc).strftime("%H:%M:%S UTC")
+            report[key] = {"remaining": r["remaining"], "limit": r["limit"], "reset": reset}
+    if args.json:
+        print(json.dumps(report, indent=2))
+    else:
+        print(f"topicscout {__version__}")
+        print(f"GITHUB_TOKEN: {'set' if token else 'not set (works, but slower and READMEs capped)'}")
+        if "error" in report:
+            print(f"GitHub: unreachable ({report['error']})")
+        else:
+            for key, label in (("search", "search API"), ("core", "READMEs (core API)")):
+                r = report[key]
+                print(f"{label}: {r['remaining']}/{r['limit']} left, resets {r['reset']}")
+    return 1 if "error" in report else 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    ap = argparse.ArgumentParser(prog="topicscout",
+                                 description="Find the GitHub repos on a topic, and only the new ones next time.")
+    ap.add_argument("--version", action="version", version=f"topicscout {__version__}")
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    r = sub.add_parser("run", help="search GitHub with a profile and list what is new")
+    r.add_argument("profile", help="built-in profile name (see `topicscout profiles`) or path to a JSON profile")
+    r.add_argument("--out", help="state folder: candidates.json, new.md, known.txt (default scout-<profile>)")
+    r.add_argument("--min-stars", type=int, help="override the profile's min_stars")
+    r.add_argument("--days", type=int, help="override the profile's days: drop repos with no push in this many days")
+    r.add_argument("--readme-max", type=int, help="READMEs read per run (default 40, 300 with GITHUB_TOKEN)")
+    r.add_argument("--known-from", action="append", default=[], metavar="GLOB",
+                   help="files whose first 20 lines name repos you already have (repeatable)")
+    r.add_argument("--json", action="store_true", help="print the summary and the new repos as JSON")
+    r.add_argument("-q", "--quiet", action="store_true", help="no progress on stderr")
+    r.set_defaults(func=_run)
+
+    p = sub.add_parser("profiles", help="list the built-in profiles")
+    p.add_argument("--json", action="store_true")
+    p.set_defaults(func=_profiles)
+
+    d = sub.add_parser("doctor", help="token and GitHub quota")
+    d.add_argument("--json", action="store_true")
+    d.set_defaults(func=_doctor)
+
+    args = ap.parse_args(argv)
+    return args.func(args)

topicscout-0.1.0/topicscout/core.py

@@ -0,0 +1,274 @@
+"""Search, filter, remember: the whole scout, with no dependencies.
+
+A profile says what to look for (GitHub search queries), what a real hit
+must say about itself (require), what to drop (exclude) and how to guess
+the interface from the README. The state is kept across runs, so new.md
+holds only what appeared since the last one.
+"""
+from __future__ import annotations
+
+import glob
+import json
+import os
+import re
+import sys
+import time
+import urllib.error
+import urllib.request
+from datetime import date, datetime, timedelta, timezone
+from pathlib import Path
+from urllib.parse import urlencode
+
+API = "https://api.github.com"
+GITHUB_URL = re.compile(r"github\.com/([\w.-]+)/([\w.-]+)")
+
+
+class RateLimited(Exception):
+    pass
+
+
+def fetch_github(url: str, accept: str) -> tuple[int, dict, str]:
+    """GET with the optional GITHUB_TOKEN; returns (status, lower-case headers, body)."""
+    headers = {"Accept": accept, "User-Agent": "topicscout", "X-GitHub-Api-Version": "2022-11-28"}
+    if os.environ.get("GITHUB_TOKEN"):
+        headers["Authorization"] = f"Bearer {os.environ['GITHUB_TOKEN']}"
+    req = urllib.request.Request(url, headers=headers)
+    try:
+        with urllib.request.urlopen(req, timeout=30) as r:
+            return r.status, {k.lower(): v for k, v in r.headers.items()}, r.read().decode("utf-8", "replace")
+    except urllib.error.HTTPError as e:
+        return e.code, {k.lower(): v for k, v in e.headers.items()}, e.read().decode("utf-8", "replace")
+
+
+def _stderr(msg: str) -> None:
+    print(msg, file=sys.stderr, flush=True)
+
+
+# ---------------------------------------------------------------- profiles
+
+PROFILES = Path(__file__).resolve().parent / "profiles"
+
+
+def builtin_profiles() -> dict[str, str]:
+    """name -> description of the profiles shipped with the package."""
+    out = {}
+    for p in sorted(PROFILES.glob("*.json")):
+        out[p.stem] = json.loads(p.read_text(encoding="utf-8")).get("description", "")
+    return out
+
+
+class Profile:
+    """A compiled profile: queries, require/exclude patterns, interface guesses."""
+
+    def __init__(self, data: dict):
+        self.name = data.get("name", "profile")
+        self.description = data.get("description", "")
+        self.queries = list(data["queries"])
+        if not self.queries:
+            raise ValueError("a profile needs at least one query")
+        self.exclude = re.compile(data["exclude"]) if data.get("exclude") else None
+        # own_words: the pattern must be in the repo's name or description, not only in a topic
+        self.require = [(re.compile(r["pattern"]), bool(r.get("own_words"))) for r in data.get("require", [])]
+        self.interfaces = [(n, re.compile(p)) for n, p in data.get("interfaces", [])]
+        self.min_stars = int(data.get("min_stars", 10))
+        self.days = int(data.get("days", 90))
+        self.require_language = bool(data.get("require_language", True))
+
+    @classmethod
+    def load(cls, name_or_path: str) -> "Profile":
+        path = Path(name_or_path)
+        if not path.is_file():
+            path = PROFILES / f"{name_or_path}.json"
+        if not path.is_file():
+            raise FileNotFoundError(f"no profile {name_or_path!r}: not a file, and built-in ones are "
+                                    f"{', '.join(builtin_profiles())}")
+        data = json.loads(path.read_text(encoding="utf-8"))
+        data.setdefault("name", path.stem)
+        return cls(data)
+
+
+# ---------------------------------------------------------------- search
+
+def _get(fetch, url, accept, sleep):
+    """One request; waits out a short rate limit once, raises RateLimited on a long one."""
+    for attempt in (0, 1):
+        status, headers, body = fetch(url, accept)
+        limited = status == 429 or (status == 403 and (headers.get("x-ratelimit-remaining") == "0"
+                                                       or "retry-after" in headers))
+        if not limited:
+            return status, body
+        if "retry-after" in headers:
+            wait = int(headers["retry-after"])
+        else:
+            wait = int(headers.get("x-ratelimit-reset", "0")) - int(time.time())
+        if attempt or wait > 70:
+            raise RateLimited(f"GitHub rate limit, reset in {max(wait, 0)} s")
+        sleep(max(wait, 0) + 1)
+    raise AssertionError("unreachable")
+
+
+def search(fetch, queries, *, min_stars, days, today, sleep, pause=0.0, pages=3,
+           log=lambda m: None) -> tuple[dict, str | None]:
+    """Run the queries; returns ({full_name: item}, reason it stopped early or None)."""
+    since = (today - timedelta(days=days)).isoformat()
+    found: dict[str, dict] = {}
+    first = True
+    for i, q in enumerate(queries, 1):
+        full = f"{q} fork:false archived:false stars:>={min_stars} pushed:>={since}"
+        for page in range(1, pages + 1):
+            if not first and pause:
+                sleep(pause)
+            first = False
+            url = f"{API}/search/repositories?" + urlencode(
+                {"q": full, "sort": "stars", "order": "desc", "per_page": 100, "page": page})
+            try:
+                status, body = _get(fetch, url, "application/vnd.github+json", sleep)
+            except RateLimited as e:
+                return found, str(e)
+            if status != 200:
+                return found, f"search failed ({status}) on {q!r}: {body[:200]}"
+            items = json.loads(body).get("items", [])
+            for it in items:
+                found.setdefault(it["full_name"], it)
+            log(f"[{i}/{len(queries)}] {q} page {page}: {len(items)} ({len(found)} so far)")
+            if len(items) < 100:
+                break
+    return found, None
+
+
+def keep(item: dict, profile: Profile, *, min_stars: int, cutoff: date) -> bool:
+    if item.get("fork") or item.get("archived"):
+        return False
+    if profile.require_language and not item.get("language"):
+        return False
+    if item.get("stargazers_count", 0) < min_stars:
+        return False
+    if datetime.fromisoformat(item["pushed_at"].replace("Z", "+00:00")).date() < cutoff:
+        return False
+    own = f"{item['full_name']} {item.get('description') or ''}"
+    text = f"{own} {' '.join(item.get('topics') or [])}"
+    if profile.exclude and profile.exclude.search(text):
+        return False
+    own_n, text_n = (s.replace("-", " ").replace("_", " ") for s in (own, text))
+    return all(rx.search(own_n if own_words else text_n) for rx, own_words in profile.require)
+
+
+def guess_interface(readme: str, profile: Profile) -> list[str]:
+    return [name for name, rx in profile.interfaces if rx.search(readme)]
+
+
+def known_repos(known_file: Path, known_from: list[str] = (), head: int = 20) -> set[str]:
+    """owner/repo (lower case) from known.txt plus the GitHub URLs in the first
+    `head` lines of the files matching the known_from globs."""
+    known: set[str] = set()
+    for pattern in known_from:
+        for f in glob.glob(pattern, recursive=True):
+            p = Path(f)
+            if not p.is_file():
+                continue
+            top = "\n".join(p.read_text(encoding="utf-8", errors="replace").splitlines()[:head])
+            for owner, name in GITHUB_URL.findall(top):
+                name = name.rstrip(".").removesuffix(".git")
+                known.add(f"{owner}/{name}".lower())
+    if known_file.is_file():
+        for line in known_file.read_text(encoding="utf-8").splitlines():
+            line = line.split("#", 1)[0].strip()
+            if line:
+                known.add(line.lower())
+    return known
+
+
+# ---------------------------------------------------------------- run
+
+def _record(item: dict) -> dict:
+    return {"full_name": item["full_name"], "url": item["html_url"], "stars": item["stargazers_count"],
+            "pushed": item["pushed_at"][:10], "license": (item.get("license") or {}).get("spdx_id"),
+            "language": item.get("language"), "description": item.get("description") or "",
+            "topics": item.get("topics") or []}
+
+
+def _new_md(new: list[dict], profile: Profile, today: date, stopped: str | None) -> str:
+    lines = [f"# New repos for {profile.name}, {today.isoformat()}", ""]
+    if stopped:
+        lines += [f"> Stopped early: {stopped}. The list is partial.", ""]
+    if not new:
+        return "\n".join(lines + ["Nothing new since the last run.", ""])
+    lines += [f"{len(new)} new.", "", "| repo | stars | last push | language | license | interface | description |",
+              "|---|---:|---|---|---|---|---|"]
+    for r in new:
+        iface = ", ".join(r["interface"]) if r.get("interface") else ("?" if r.get("interface") is None else "-")
+        desc = r["description"].replace("|", "\\|").replace("\n", " ")[:160]
+        lines.append(f"| [{r['full_name']}]({r['url']}) | {r['stars']} | {r['pushed']} | {r['language'] or '-'} | "
+                     f"{r['license'] or '-'} | {iface} | {desc} |")
+    return "\n".join(lines + [""])
+
+
+def run(profile: Profile, out: Path, fetch=None, *, today: date | None = None, sleep=None,
+        min_stars: int | None = None, days: int | None = None, readme_max: int | None = None,
+        pause: float | None = None, known_from: list[str] = (), log=lambda m: None) -> dict:
+    fetch = fetch or fetch_github
+    sleep = sleep or time.sleep
+    today = today or datetime.now(timezone.utc).date()
+    min_stars = profile.min_stars if min_stars is None else min_stars
+    days = profile.days if days is None else days
+    token = bool(os.environ.get("GITHUB_TOKEN"))
+    if pause is None:
+        pause = 2.0 if token else 7.0  # search API: 30/min with a token, 10/min without
+    if readme_max is None:
+        readme_max = 300 if token else 40
+    out.mkdir(parents=True, exist_ok=True)
+    state_file = out / "candidates.json"
+    state = json.loads(state_file.read_text(encoding="utf-8")) if state_file.is_file() else {"repos": {}}
+    repos: dict[str, dict] = state["repos"]
+    known = known_repos(out / "known.txt", known_from)
+
+    found, stopped = search(fetch, profile.queries, min_stars=min_stars, days=days, today=today,
+                            sleep=sleep, pause=pause, log=log)
+    cutoff = today - timedelta(days=days)
+    new_names = []
+    for name, item in found.items():
+        if not keep(item, profile, min_stars=min_stars, cutoff=cutoff):
+            continue
+        rec = _record(item)
+        old = repos.get(name)
+        rec["first_seen"] = old["first_seen"] if old else today.isoformat()
+        rec["last_seen"] = today.isoformat()
+        rec["interface"] = old.get("interface") if old else None
+        if name.lower() in known:
+            rec["status"] = "known"
+        elif old:
+            rec["status"] = "seen"
+        else:
+            rec["status"] = "new"
+            new_names.append(name)
+        repos[name] = rec
+
+    # READMEs: first sightings first, then older ones never read (cap reached last time)
+    if profile.interfaces:
+        todo = new_names + [n for n, r in repos.items() if r["status"] == "seen" and r.get("interface") is None]
+        for name in todo[:readme_max]:
+            try:
+                status, body = _get(fetch, f"{API}/repos/{name}/readme", "application/vnd.github.raw", sleep)
+            except RateLimited as e:
+                stopped = stopped or f"{e} (while reading READMEs)"
+                break
+            repos[name]["interface"] = guess_interface(body, profile) if status == 200 else []
+        log(f"READMEs read: {min(len(todo), readme_max)} of {len(todo)}")
+
+    state = {"profile": profile.name, "updated": today.isoformat(),
+             "repos": dict(sorted(repos.items(), key=lambda kv: kv[0].lower()))}
+    state_file.write_text(json.dumps(state, indent=2, ensure_ascii=False) + "\n", encoding="utf-8")
+    new = sorted((repos[n] for n in new_names), key=lambda r: -r["stars"])
+    (out / "new.md").write_text(_new_md(new, profile, today, stopped), encoding="utf-8")
+    return {"profile": profile.name, "found": len(found),
+            "kept": sum(1 for r in repos.values() if r["last_seen"] == today.isoformat()),
+            "new": new, "stopped": stopped}
+
+
+def rate_limit(fetch=None) -> dict:
+    """GitHub's own view of the quota; this call does not count against it."""
+    fetch = fetch or fetch_github
+    status, _, body = fetch(f"{API}/rate_limit", "application/vnd.github+json")
+    if status != 200:
+        raise RuntimeError(f"rate_limit failed ({status}): {body[:200]}")
+    return json.loads(body)["resources"]

topicscout-0.1.0/topicscout/profiles/ai-memory.json

@@ -0,0 +1,59 @@
+{
+  "name": "ai-memory",
+  "description": "Memory systems for AI agents and LLMs (the adebench list)",
+  "queries": [
+    "topic:agent-memory",
+    "topic:ai-memory",
+    "topic:llm-memory",
+    "topic:memory-mcp",
+    "topic:mcp-memory",
+    "topic:long-term-memory",
+    "topic:agentic-memory",
+    "topic:memory-layer",
+    "topic:memory-system",
+    "\"agent memory\" in:name,description",
+    "\"llm memory\" in:name,description",
+    "\"memory layer\" in:name,description",
+    "\"mcp memory\" in:name,description",
+    "\"long-term memory\" in:description",
+    "\"persistent memory\" in:description"
+  ],
+  "exclude": "(?i)awesome|curated list|reading list|\\bpapers\\b|\\bsurvey\\b",
+  "require": [
+    {
+      "pattern": "(?i)memor|\\bmem\\b|remember",
+      "own_words": true
+    },
+    {
+      "pattern": "(?i)\\b(agents?|agentic|llms?|ai|mcp|claude|gpt|rag|assistants?|chatbots?)\\b"
+    }
+  ],
+  "interfaces": [
+    [
+      "mcp",
+      "(?i)\\bmcp\\b|modelcontextprotocol"
+    ],
+    [
+      "pip",
+      "(?i)\\bpip3? install\\b|\\buv add\\b|\\bpypi\\b"
+    ],
+    [
+      "npm",
+      "(?i)\\bnpm (i|install)\\b|\\bnpx\\b|\\bbun add\\b|\\bpnpm add\\b"
+    ],
+    [
+      "rest",
+      "\\bcurl\\b[^\n]*https?://|\\bREST\\b|\\bopenapi\\b|/v1/"
+    ],
+    [
+      "docker",
+      "(?i)\\bdocker\\b"
+    ],
+    [
+      "plugin",
+      "(?i)/plugin install|\\.?claude-plugin|claude code plugin"
+    ]
+  ],
+  "min_stars": 10,
+  "days": 90
+}

topicscout-0.1.0/topicscout/profiles/github-scrapers.json

@@ -0,0 +1,55 @@
+{
+  "name": "github-scrapers",
+  "description": "Tools that scrape, crawl or mine GitHub itself",
+  "queries": [
+    "topic:github-scraper",
+    "topic:github-crawler",
+    "topic:github-scraping",
+    "topic:github-mining",
+    "\"github scraper\" in:name,description",
+    "\"github crawler\" in:name,description",
+    "\"scrape github\" in:name,description",
+    "\"github scraping\" in:name,description",
+    "\"scrapes github\" in:description",
+    "\"crawl github\" in:name,description",
+    "\"github repository scraper\" in:description",
+    "\"github repo scraper\" in:description"
+  ],
+  "exclude": "(?i)awesome|curated list|onlyfans",
+  "require": [
+    {
+      "pattern": "(?i)scrap|crawl|mining|miner|harvest"
+    },
+    {
+      "pattern": "(?i)github"
+    }
+  ],
+  "interfaces": [
+    [
+      "cli",
+      "(?i)\\busage:|\\$ \\w+ --|\\bcommand line\\b|\\bcli\\b"
+    ],
+    [
+      "pip",
+      "(?i)\\bpip3? install\\b|\\buv add\\b|\\bpypi\\b"
+    ],
+    [
+      "npm",
+      "(?i)\\bnpm (i|install)\\b|\\bnpx\\b"
+    ],
+    [
+      "action",
+      "(?i)uses: [\\w.-]+/[\\w.-]+@|github action"
+    ],
+    [
+      "docker",
+      "(?i)\\bdocker\\b"
+    ],
+    [
+      "token",
+      "(?i)GITHUB_TOKEN|personal access token"
+    ]
+  ],
+  "min_stars": 5,
+  "days": 365
+}

topicscout-0.1.0/topicscout.egg-info/PKG-INFO

@@ -0,0 +1,151 @@
+Metadata-Version: 2.4
+Name: topicscout
+Version: 0.1.0
+Summary: Find the GitHub repos on a topic, and only the new ones next time. No dependencies, token optional.
+Author: Adecubed
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/adecubed/topicscout
+Keywords: github,scraper,search,discovery,topics
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# topicscout
+
+[![ci](https://github.com/adecubed/topicscout/actions/workflows/ci.yml/badge.svg)](https://github.com/adecubed/topicscout/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/topicscout)](https://pypi.org/project/topicscout/)
+
+Find the GitHub repositories on a topic, and next time only the new ones.
+
+```bash
+pip install topicscout        # or run it without installing: uvx topicscout run github-scrapers
+topicscout run github-scrapers
+```
+
+```
+19 found, 17 kept, 17 new -> scout-github-scrapers/new.md
+```
+
+`new.md` is a table of what appeared since the last run, most stars first: stars,
+last push, language, license, a guess at the interface from the README (MCP, pip,
+npm, REST, Docker...) and the description. `candidates.json` keeps everything seen,
+so a weekly run is a short list, not the same 300 repos again.
+
+No dependencies, Python 3.11+. `GITHUB_TOKEN` is optional: without it the searches
+are paced to stay under GitHub's 10 per minute and READMEs are capped at 40 per run.
+
+## Why
+
+GitHub search is good at "repos with this topic" and bad at "what's new since I last
+looked, minus the forks, the awesome-lists, the abandoned ones and the repos that
+only tagged themselves". topicscout runs a fixed set of searches, applies your
+filters, remembers what it has seen and what you already have.
+
+It was the discovery step of [adebench](https://github.com/adecubed/adebench), a
+benchmark for AI memory systems, which uses it to find new memories to measure.
+
+## Your own topic
+
+Copy a built-in profile (they are in [`topicscout/profiles/`](topicscout/profiles/)),
+change the queries and the words a hit must contain, and run it:
+
+```bash
+topicscout run my-topic.json
+```
+
+Run it again next week: `new.md` lists only what appeared in between. Put the repos
+you already know or rejected in `scout-my-topic/known.txt`.
+
+## From a script or an agent
+
+`--json` prints the result on stdout, progress stays on stderr:
+
+```bash
+topicscout run github-scrapers --json -q
+```
+
+```json
+{"profile": "github-scrapers", "found": 19, "kept": 17, "new": [{"full_name": "yusufkaraaslan/Skill_Seekers", "stars": 15052, "interface": ["cli", "pip", "action"], "...": "..."}, "..."], "stopped": null, "out": "scout-github-scrapers"}
+```
+
+It is a command-line tool and a Python library (`from topicscout.core import Profile, run`),
+not an MCP server; an agent calls it like any other command.
+
+## Commands
+
+```bash
+topicscout run PROFILE [--out DIR] [--min-stars N] [--days N] [--readme-max N]
+                       [--known-from GLOB ...] [--json] [-q]
+topicscout profiles    # built-in profiles
+topicscout doctor      # token and remaining GitHub quota
+```
+
+- `PROFILE` is a built-in name or a path to your own JSON profile.
+- `--out` is the state folder (default `scout-<profile>`): `candidates.json`,
+  `new.md`, and your `known.txt`.
+- `--json` prints the summary and the new repos as JSON on stdout; progress and
+  errors always go to stderr, so scripts and agents can read stdout as is.
+
+## What counts as known
+
+Repos you already have are marked `known` and never listed as new:
+
+- `known.txt` in the state folder: one `owner/repo` per line, `#` comments. Put
+  rejected repos here too.
+- `--known-from "adapters/*.py"`: the GitHub URLs in the first 20 lines of those
+  files. adebench points it at its adapters, so a memory with an adapter drops out.
+
+## Profiles
+
+Built in: `ai-memory` (memory systems for AI agents and LLMs) and `github-scrapers`
+(tools that scrape, crawl or mine GitHub). A profile is a JSON file:
+
+```json
+{
+  "name": "ai-memory",
+  "description": "Memory systems for AI agents and LLMs",
+  "queries": ["topic:agent-memory", "\"llm memory\" in:name,description"],
+  "exclude": "(?i)awesome|curated list|\\bpapers\\b",
+  "require": [
+    {"pattern": "(?i)memor|\\bmem\\b", "own_words": true},
+    {"pattern": "(?i)\\b(agents?|llms?|ai|mcp)\\b"}
+  ],
+  "interfaces": [["mcp", "(?i)\\bmcp\\b"], ["pip", "(?i)\\bpip install\\b"]],
+  "min_stars": 10,
+  "days": 90
+}
+```
+
+- `queries`: [GitHub repository search](https://docs.github.com/en/search-github/searching-on-github/searching-for-repositories)
+  queries. Each one always gets `fork:false archived:false stars:>=min_stars
+  pushed:>=today-days`; up to 300 results per query.
+- `exclude`: a regex on name, description and topics; a match drops the repo.
+- `require`: every pattern must match. With `own_words` it must be in the name or
+  description, not only in a topic: a database that tagged itself `agent-memory`
+  is not a memory. Hyphens and underscores count as spaces.
+- `interfaces`: `[label, regex]` pairs tried on the README of each new repo.
+  Leave it out and no README is fetched.
+- `require_language` (default true) drops repos with no language, which are
+  usually lists and docs.
+
+Patterns are Python regexes; add `(?i)` for case-insensitive.
+
+## Rate limits
+
+On a 403 or 429 it waits for the reset if that is under 70 seconds; otherwise it
+stops, saves what it found and says so in `new.md` and on stderr. The next run
+picks up the READMEs it did not get to.
+
+## How it compares
+
+- [ghcrawl](https://github.com/pwrdrvr/ghcrawl) goes deep into one repository
+  (issues and PRs, embeddings, clusters); topicscout goes wide across GitHub for a
+  topic.
+- [top-github-scraper](https://github.com/khuyentran1401/top-github-scraper)
+  lists the top repos for a keyword, once; topicscout adds filters, profiles and
+  memory across runs.
+
+## License
+
+MIT

topicscout-0.1.0/topicscout.egg-info/SOURCES.txt

@@ -0,0 +1,15 @@
+LICENSE
+README.md
+pyproject.toml
+tests/test_topicscout.py
+topicscout/__init__.py
+topicscout/__main__.py
+topicscout/cli.py
+topicscout/core.py
+topicscout.egg-info/PKG-INFO
+topicscout.egg-info/SOURCES.txt
+topicscout.egg-info/dependency_links.txt
+topicscout.egg-info/entry_points.txt
+topicscout.egg-info/top_level.txt
+topicscout/profiles/ai-memory.json
+topicscout/profiles/github-scrapers.json

topicscout-0.1.0/topicscout.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

topicscout-0.1.0/topicscout.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+topicscout = topicscout.cli:main

topicscout-0.1.0/topicscout.egg-info/top_level.txt

@@ -0,0 +1 @@
+topicscout