usenet 0.1.0a2__tar.gz

usenet-0.1.0a2/LICENSE

@@ -0,0 +1,19 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.
+
+   Copyright 2026 JarbasAi
+
+   Full text: https://www.apache.org/licenses/LICENSE-2.0.txt

usenet-0.1.0a2/PKG-INFO

@@ -0,0 +1,106 @@
+Metadata-Version: 2.4
+Name: usenet
+Version: 0.1.0a2
+Summary: NNTP client and newsgroup harvesting toolkit for Python.
+Author-email: JarbasAi <jarbasai@mailfence.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/JarbasAl/usenet
+Keywords: usenet,nntp,newsgroups,scraper,corpus,dataset
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Topic :: Communications :: Usenet News
+Classifier: Topic :: Software Development :: Libraries
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: dateparser
+Requires-Dist: standard-nntplib; python_version >= "3.13"
+Provides-Extra: scrape
+Requires-Dist: unblock_requests; extra == "scrape"
+Provides-Extra: anon
+Requires-Dist: anon_requests; extra == "anon"
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Requires-Dist: pytest-cov; extra == "test"
+Dynamic: license-file
+
+# usenet
+
+NNTP client and newsgroup-harvesting toolkit for Python. Read and post articles,
+discover public servers, and harvest groups into a text corpus.
+
+Works on Python 3.9–3.13: `nntplib` was removed from the standard library in
+3.13 (PEP 594), so the `standard-nntplib` backport is pulled in automatically
+there.
+
+## Install
+
+```bash
+pip install usenet
+# optional: anti-bot + Wayback transport for the legacy server-list scrapers
+pip install usenet[scrape]
+```
+
+## Quickstart
+
+Read a group:
+
+```python
+from datetime import timedelta
+from usenet import UsenetServer
+
+with UsenetServer("news.eternal-september.org") as server:
+    for article in server.get_new_news("comp.lang.python", since=timedelta(days=7)):
+        print(article.subject, article.author, article.date)
+        print(article.text)
+```
+
+Post an article (most servers need a free account):
+
+```python
+from usenet import UsenetServer
+
+with UsenetServer("news.eternal-september.org", user="login", pswd="secret") as server:
+    server.post("this is a test", subject="hello", group="misc.test")
+```
+
+## Server discovery
+
+A curated, offline list ships with the package:
+
+```python
+from usenet import get_known_servers
+
+for s in get_known_servers():
+    print(s.url, s._can_post)
+```
+
+The legacy directory scrapers in `usenet.scrappers` are a secondary "refresh
+once" path; their source sites are mostly dead, so with `usenet[scrape]`
+installed requests fall back to the Wayback Machine. See `examples/`.
+
+## Dataset
+
+`dataset.py` harvests a newsgroup into a JSONL corpus (one article per line) for
+publishing to the Hugging Face Hub. See [docs/dataset.md](docs/dataset.md).
+
+```bash
+python dataset.py comp.lang.python --days 30 --out comp.lang.python.jsonl
+```
+
+## Testing
+
+```bash
+pip install -e .[test]
+pytest test/
+```
+
+The unit tests are offline; they exercise article parsing, post framing, the
+bundled server list, and scraper HTML parsing against fixtures.
+
+## License
+
+Apache-2.0

usenet-0.1.0a2/README.md

@@ -0,0 +1,77 @@
+# usenet
+
+NNTP client and newsgroup-harvesting toolkit for Python. Read and post articles,
+discover public servers, and harvest groups into a text corpus.
+
+Works on Python 3.9–3.13: `nntplib` was removed from the standard library in
+3.13 (PEP 594), so the `standard-nntplib` backport is pulled in automatically
+there.
+
+## Install
+
+```bash
+pip install usenet
+# optional: anti-bot + Wayback transport for the legacy server-list scrapers
+pip install usenet[scrape]
+```
+
+## Quickstart
+
+Read a group:
+
+```python
+from datetime import timedelta
+from usenet import UsenetServer
+
+with UsenetServer("news.eternal-september.org") as server:
+    for article in server.get_new_news("comp.lang.python", since=timedelta(days=7)):
+        print(article.subject, article.author, article.date)
+        print(article.text)
+```
+
+Post an article (most servers need a free account):
+
+```python
+from usenet import UsenetServer
+
+with UsenetServer("news.eternal-september.org", user="login", pswd="secret") as server:
+    server.post("this is a test", subject="hello", group="misc.test")
+```
+
+## Server discovery
+
+A curated, offline list ships with the package:
+
+```python
+from usenet import get_known_servers
+
+for s in get_known_servers():
+    print(s.url, s._can_post)
+```
+
+The legacy directory scrapers in `usenet.scrappers` are a secondary "refresh
+once" path; their source sites are mostly dead, so with `usenet[scrape]`
+installed requests fall back to the Wayback Machine. See `examples/`.
+
+## Dataset
+
+`dataset.py` harvests a newsgroup into a JSONL corpus (one article per line) for
+publishing to the Hugging Face Hub. See [docs/dataset.md](docs/dataset.md).
+
+```bash
+python dataset.py comp.lang.python --days 30 --out comp.lang.python.jsonl
+```
+
+## Testing
+
+```bash
+pip install -e .[test]
+pytest test/
+```
+
+The unit tests are offline; they exercise article parsing, post framing, the
+bundled server list, and scraper HTML parsing against fixtures.
+
+## License
+
+Apache-2.0

usenet-0.1.0a2/pyproject.toml

@@ -0,0 +1,54 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "usenet"
+dynamic = ["version"]
+description = "NNTP client and newsgroup harvesting toolkit for Python."
+readme = "README.md"
+requires-python = ">=3.9"
+license = {text = "Apache-2.0"}
+authors = [{name = "JarbasAi", email = "jarbasai@mailfence.com"}]
+keywords = ["usenet", "nntp", "newsgroups", "scraper", "corpus", "dataset"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: Apache Software License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3 :: Only",
+    "Topic :: Communications :: Usenet News",
+    "Topic :: Software Development :: Libraries",
+]
+dependencies = [
+    "dateparser",
+    # nntplib was removed from the stdlib in Python 3.13 (PEP 594); the
+    # backport keeps `import nntplib` working there. On 3.9-3.12 the stdlib
+    # module is used and this dependency is not installed.
+    "standard-nntplib; python_version >= '3.13'",
+]
+
+[project.optional-dependencies]
+# Anti-bot transport for the (mostly archival) server-list scrapers; dead
+# sites resolve transparently via the Wayback fallback.
+scrape = ["unblock_requests"]
+# IP rotation layer, composed with the scrape transport.
+anon = ["anon_requests"]
+test = ["pytest>=7", "pytest-cov"]
+
+[project.urls]
+Homepage = "https://github.com/JarbasAl/usenet"
+
+[tool.setuptools.dynamic]
+version = {attr = "usenet.version.__version__"}
+
+[tool.setuptools.packages.find]
+include = ["usenet*"]
+exclude = ["test*", "examples*", "docs*"]
+
+[tool.setuptools.package-data]
+"usenet.data" = ["*.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["test"]
+addopts = "-q"

usenet-0.1.0a2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

usenet-0.1.0a2/test/test_article.py

@@ -0,0 +1,84 @@
+"""Offline tests for usenet.article_entry.Article."""
+from datetime import datetime
+
+from usenet.article_entry import Article
+
+HEADERS = [
+    b"From: Alice <alice@example.com>",
+    b"Subject: Hello World",
+    b"Date: Mon, 5 May 2025 10:00:00 +0000",
+    b"Content-Language: en",
+    b"Newsgroups: comp.lang.python",
+]
+BODY = [b"first line", b"second line"]
+
+
+def test_headers_parsed():
+    art = Article("1", headers=HEADERS, body=BODY)
+    h = art.headers
+    assert h["From"] == "Alice <alice@example.com>"
+    assert h["Subject"] == "Hello World"
+    assert h["Content-Language"] == "en"
+
+
+def test_subject_author_language():
+    art = Article("1", headers=HEADERS, body=BODY)
+    assert art.subject == "Hello World"
+    assert art.author == "Alice <alice@example.com>"
+    assert art.language == "en"
+
+
+def test_date_parsed():
+    art = Article("1", headers=HEADERS, body=BODY)
+    assert isinstance(art.date, datetime)
+    assert art.date.year == 2025
+
+
+def test_date_with_rfc2822_utc_comment():
+    # the obsolete "-0000 (UTC)" comment form that dateparser returns None for
+    art = Article("1", headers=[b"Date: Thu, 14 May 2026 01:06:51 -0000 (UTC)"],
+                  body=[])
+    assert isinstance(art.date, datetime)
+    assert (art.date.year, art.date.month, art.date.day) == (2026, 5, 14)
+
+
+def test_text_join():
+    art = Article("1", headers=HEADERS, body=BODY)
+    assert art.text == "first line\nsecond line"
+
+
+def test_missing_headers_are_safe():
+    art = Article("1", headers=[b"X-Foo: bar"], body=[])
+    assert art.subject == ""
+    assert art.author == ""
+    assert art.language is None
+    assert art.date is None
+    assert art.text == ""
+
+
+def test_binary_body_falls_back():
+    art = Article("1", headers=HEADERS, body=[b"\xff\xfe\x00bad"])
+    assert "<failed to decode line, binary data?>" in art.text
+
+
+def test_str_lines_pass_through():
+    art = Article("1", headers=["Subject: already str"], body=["plain"])
+    assert art.subject == "already str"
+    assert art.text == "plain"
+
+
+def test_missing_article_is_tolerated():
+    # a cancelled/expired article number raises 4xx on head/body; must not crash
+    import nntplib
+
+    class GoneConn:
+        def head(self, article_id):
+            raise nntplib.NNTPTemporaryError("423 no such article")
+
+        def body(self, article_id):
+            raise nntplib.NNTPTemporaryError("423 no such article")
+
+    art = Article("999", connection=GoneConn())
+    assert art.subject == ""
+    assert art.text == ""
+    assert art.date is None

usenet-0.1.0a2/test/test_known_servers.py

@@ -0,0 +1,23 @@
+"""Offline tests for the bundled public-server list."""
+from usenet import get_known_servers, load_server_records
+from usenet.models import ServerRecord
+from usenet.server_entry import UsenetServer
+
+
+def test_records_load():
+    records = load_server_records()
+    assert isinstance(records, list)
+    assert records, "bundled servers.json should not be empty"
+    assert all(isinstance(r, ServerRecord) for r in records)
+    for rec in records:
+        assert rec.host
+        assert isinstance(rec.port, int)
+
+
+def test_get_known_servers_yields_server_objects():
+    servers = list(get_known_servers())  # validate=False -> no network
+    assert servers
+    assert all(isinstance(s, UsenetServer) for s in servers)
+    # the posting hint is carried over from the JSON
+    hosts = {s.url for s in servers}
+    assert "news.eternal-september.org" in hosts

usenet-0.1.0a2/test/test_models.py

@@ -0,0 +1,55 @@
+"""Offline tests for the dataclass value objects."""
+from dataclasses import asdict
+
+from usenet.article_entry import Article
+from usenet.models import ArticleRecord, ServerRecord
+from usenet.server_entry import UsenetServer
+
+
+def test_server_record_from_dict_defaults():
+    rec = ServerRecord.from_dict({"host": "news.example.org"})
+    assert rec.host == "news.example.org"
+    assert rec.port == 119
+    assert rec.tls_port is None
+    assert rec.post is None
+    assert rec.auth == "none"
+
+
+def test_server_record_to_server_carries_post_hint():
+    rec = ServerRecord.from_dict({"host": "news.example.org", "post": False})
+    server = rec.to_server()
+    assert isinstance(server, UsenetServer)
+    assert server.url == "news.example.org"
+    assert server._can_post is False
+
+
+def test_server_record_is_frozen():
+    rec = ServerRecord(host="h")
+    try:
+        rec.host = "other"  # type: ignore[misc]
+    except Exception as exc:
+        assert exc.__class__.__name__ == "FrozenInstanceError"
+    else:
+        raise AssertionError("ServerRecord should be immutable")
+
+
+def test_article_record_from_article():
+    headers = [
+        b"From: Bob <bob@example.com>",
+        b"Subject: Hi",
+        b"Date: Mon, 5 May 2025 10:00:00 +0000",
+        b"Content-Language: en",
+    ]
+    art = Article("42", headers=headers, body=[b"hello"])
+    rec = ArticleRecord.from_article(art, group="comp.lang.python")
+    assert rec.group == "comp.lang.python"
+    assert rec.message_id == "42"
+    assert rec.subject == "Hi"
+    assert rec.author == "Bob <bob@example.com>"
+    assert rec.language == "en"
+    assert rec.date.startswith("2025-05-05")
+    assert rec.text == "hello"
+    # serializes cleanly for JSONL
+    assert set(asdict(rec)) == {
+        "group", "message_id", "subject", "author", "date", "language", "text",
+    }

usenet-0.1.0a2/test/test_scrappers.py

@@ -0,0 +1,29 @@
+"""Offline tests for the directory scrapers (HTML parsing only)."""
+import usenet.scrappers as scrappers
+
+
+def test_elfqrin_parses_hosts(monkeypatch):
+    html = 'var x; c[i]="news.example.com"; i++; c[i]="news2.example.com"; i++;'
+    monkeypatch.setattr(scrappers, "get_html", lambda url, **kw: html)
+    hosts = [s.url for s in scrappers.get_elfqrin(validate=False)]
+    assert hosts == ["news.example.com", "news2.example.com"]
+
+
+def test_news_url_table_parses_sok(monkeypatch):
+    html = (
+        "<table>"
+        '<tr><th>name</th></tr>'
+        '<tr><td><a href="news://news.alpha.org/">alpha</a></td></tr>'
+        '<tr><td><a href="news://news.beta.net/">beta</a></td></tr>'
+        "</table>"
+    )
+    monkeypatch.setattr(scrappers, "get_html", lambda url, **kw: html)
+    hosts = [s.url for s in scrappers.get_sok(validate=False)]
+    assert hosts == ["news.alpha.org", "news.beta.net"]
+
+
+def test_scraper_skips_blank_hosts(monkeypatch):
+    html = 'c[i]=""; i++; c[i]="news.good.org"; i++;'
+    monkeypatch.setattr(scrappers, "get_html", lambda url, **kw: html)
+    hosts = [s.url for s in scrappers.get_elfqrin(validate=False)]
+    assert hosts == ["news.good.org"]

usenet-0.1.0a2/test/test_server.py

@@ -0,0 +1,82 @@
+"""Offline tests for usenet.server_entry.UsenetServer (no network)."""
+from datetime import date, timedelta
+
+from usenet.server_entry import UsenetServer
+
+
+class FakeConn:
+    """Records arguments instead of talking to a real server."""
+
+    def __init__(self, group_range=None):
+        self.posted = None
+        self.newnews_args = None
+        self._group_range = group_range  # (count, first, last)
+
+    def post(self, body):
+        self.posted = body
+        return "240 article posted"
+
+    def newnews(self, group, since):
+        self.newnews_args = (group, since)
+        return "230 list follows", []
+
+    def group(self, name):
+        count, first, last = self._group_range
+        return ("211 group selected", count, first, last, name)
+
+
+def _server_with(conn):
+    server = UsenetServer("fake.invalid")
+    server._connection = conn
+    server._dead = False
+    return server
+
+
+def test_post_frames_headers_and_body():
+    conn = FakeConn()
+    server = _server_with(conn)
+    server.post("hello body", subject="hi", group="misc.test")
+    raw = conn.posted.decode("utf-8")
+    assert "Subject: hi\r\n" in raw
+    assert "Newsgroups: misc.test\r\n" in raw
+    assert raw.endswith("\r\n\r\nhello body")
+    # default anonymous sender injected
+    assert "From: Anonymous User <anonymous@example.com>\r\n" in raw
+
+
+def test_post_respects_from_address():
+    conn = FakeConn()
+    server = _server_with(conn)
+    server.post("x", subject="s", group="g", from_address="me@example.com")
+    assert "From: me@example.com\r\n" in conn.posted.decode("utf-8")
+
+
+def test_get_new_news_converts_timedelta_to_date():
+    conn = FakeConn()
+    server = _server_with(conn)
+    server.get_new_news("comp.lang.python", since=timedelta(days=3))
+    group, since = conn.newnews_args
+    assert group == "comp.lang.python"
+    assert since == date.today() - timedelta(days=3)
+
+
+def test_get_new_news_defaults_to_five_days():
+    conn = FakeConn()
+    server = _server_with(conn)
+    server.get_new_news("comp.lang.python")
+    _, since = conn.newnews_args
+    assert since == date.today() - timedelta(days=5)
+
+
+def test_get_articles_returns_newest_first():
+    conn = FakeConn(group_range=(16, 10, 25))
+    server = _server_with(conn)
+    arts = server.get_articles("comp.lang.python", limit=5)
+    assert [a.article_id for a in arts] == [25, 24, 23, 22, 21]
+
+
+def test_get_articles_clamps_to_group_start():
+    conn = FakeConn(group_range=(2, 1, 2))
+    server = _server_with(conn)
+    arts = server.get_articles("misc.test", limit=10)
+    assert [a.article_id for a in arts] == [2, 1]

usenet-0.1.0a2/usenet/__init__.py

@@ -0,0 +1,19 @@
+from usenet.server_entry import UsenetServer
+from usenet.article_entry import Article
+from usenet.models import ServerRecord, ArticleRecord
+from usenet.known_servers import get_known_servers, load_server_records
+from usenet.probe import ServerProbe, probe_server, probe_many
+from usenet.version import __version__
+
+__all__ = [
+    "UsenetServer",
+    "Article",
+    "ServerRecord",
+    "ArticleRecord",
+    "get_known_servers",
+    "load_server_records",
+    "ServerProbe",
+    "probe_server",
+    "probe_many",
+    "__version__",
+]

usenet-0.1.0a2/usenet/article_entry.py

@@ -0,0 +1,109 @@
+"""A single Usenet article, with lazy header/body retrieval."""
+from datetime import datetime
+from email.utils import parsedate_to_datetime
+from typing import Dict, List, Optional
+
+import dateparser
+import nntplib
+
+
+class Article:
+    def __init__(self, article_id, headers=None, body=None, connection=None):
+        self.article_id = article_id
+        self._headers = headers or {}
+        self._body = body
+        self.connection = connection
+
+    def bind(self, connection) -> None:
+        self.connection = connection
+
+    @property
+    def headers(self) -> Dict[str, str]:
+        headers: Dict[str, str] = {}
+        decoded = self._decode_headers()
+        for idx, line in enumerate(decoded):
+            if not line:
+                continue
+            # fold continuation lines (a header value wrapped onto the next line)
+            next_line = decoded[idx + 1] if idx + 1 < len(decoded) else None
+            if next_line and ": " not in next_line:
+                line = line + next_line
+            if ": " in line:
+                field, _, val = line.partition(": ")
+                headers[field] = val
+        return headers
+
+    @property
+    def text(self) -> str:
+        return "\n".join(self._decode_body())
+
+    @property
+    def subject(self) -> str:
+        return self.headers.get('Subject', '').replace("\n", " ")
+
+    @property
+    def author(self) -> str:
+        return self.headers.get('From', '')
+
+    @property
+    def date(self) -> Optional[datetime]:
+        raw = self.headers.get('Date')
+        if not raw:
+            return None
+        # email.utils handles RFC-2822 news/mail dates, including the obsolete
+        # "-0000 (UTC)" comment form that trips dateparser.
+        try:
+            return parsedate_to_datetime(raw)
+        except (TypeError, ValueError):
+            return dateparser.parse(raw)
+
+    @property
+    def language(self) -> Optional[str]:
+        return self.headers.get('Content-Language')
+
+    # internal
+    def _decode_body(self) -> List[str]:
+        if self.connection and not self._body:
+            # lazy get
+            self._get_body()
+        if not self._body:
+            return []
+        return self._decode(self._body)
+
+    def _decode_headers(self) -> List[str]:
+        if self.connection and not self._headers:
+            # lazy get
+            self._get_headers()
+        if not self._headers:
+            return []
+        return self._decode(self._headers)
+
+    @staticmethod
+    def _decode(lines) -> List[str]:
+        decoded_lines = []
+        for line in lines:
+            if isinstance(line, str):
+                decoded_lines.append(line)
+                continue
+            try:
+                decoded_lines.append(line.decode("utf-8"))
+            except (UnicodeDecodeError, AttributeError):
+                decoded_lines.append("<failed to decode line, binary data?>")
+        return decoded_lines
+
+    def _get_body(self, connection=None) -> None:
+        connection = connection or self.connection
+        # an article in a GROUP range may be cancelled/expired (4xx) — tolerate it
+        try:
+            response, article = connection.body(self.article_id)
+            self._body = article.lines if article else []
+        except (nntplib.NNTPError, OSError):
+            self._body = []
+
+    def _get_headers(self, connection=None) -> None:
+        connection = connection or self.connection
+        try:
+            response, article = connection.head(self.article_id)
+            self._headers = article.lines if article else []
+        except (nntplib.NNTPError, OSError):
+            self._headers = []

usenet-0.1.0a2/usenet/data/servers.json

@@ -0,0 +1,13 @@
+{
+  "description": "Curated list of public, text-oriented NNTP servers. `anon_read`/`anon_post` were verified live with usenet.probe (reachability and policy drift over time; re-probe to refresh). Binary servers are intentionally excluded.",
+  "servers": [
+    {"host": "paganini.bofh.team", "port": 119, "tls_port": 563, "anon_read": true, "anon_post": true, "post": true, "auth": "none", "notes": "BOFH.team; accepts anonymous posts to test groups (verified)."},
+    {"host": "news.tcpreset.net", "port": 119, "tls_port": 563, "anon_read": true, "anon_post": true, "post": true, "auth": "none", "notes": "tcpreset; accepts anonymous posts to test groups (verified)."},
+    {"host": "news.neodome.net", "port": 119, "tls_port": 563, "anon_read": true, "anon_post": false, "post": false, "auth": "free account to post", "notes": "Neodome; anonymous reading, carries alt.anonymous.messages; posting needs an account."},
+    {"host": "news.samoylyk.net", "port": 119, "tls_port": 563, "anon_read": true, "anon_post": false, "post": false, "auth": "free account to post", "notes": "Samoylyk public feed; anonymous reading."},
+    {"host": "freenews.netfront.net", "port": 119, "tls_port": null, "anon_read": true, "anon_post": false, "post": false, "auth": "free account to post", "notes": "NetFront free news; anonymous reading."},
+    {"host": "news.bbs.nz", "port": 119, "tls_port": 563, "anon_read": true, "anon_post": false, "post": false, "auth": "free account to post", "notes": "BBS.nz; anonymous reading."},
+    {"host": "news.dizum.net", "port": 119, "tls_port": null, "anon_read": true, "anon_post": false, "post": false, "auth": "free account to post", "notes": "Dizum (sewer); anonymous reading; also runs a mail2news gateway."},
+    {"host": "news.eternal-september.org", "port": 119, "tls_port": 563, "anon_read": false, "anon_post": false, "post": false, "auth": "free account", "notes": "Eternal September; free registration required to read and post."}
+  ]
+}