ladon-hackernews 0.0.1__tar.gz

ladon_hackernews-0.0.1/LICENSE

@@ -0,0 +1,179 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship made available under
+      the License, as indicated by a copyright notice that is included in
+      or attached to the work (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other
+      transformations represent, as a whole, an original work of authorship.
+      For the purposes of this License, Derivative Works shall not include
+      works that remain separable from, or merely link (or bind by name)
+      to the interfaces of, the Work and Derivative Works thereof.
+
+      "Contribution" shall mean, as submitted to the Licensor for inclusion
+      in the Work by the copyright owner or by an individual or Legal Entity
+      authorized to submit on behalf of the copyright owner. For the purposes
+      of this definition, "submitted" means any form of electronic, verbal,
+      or written communication sent to the Licensor or its representatives,
+      including but not limited to communication on electronic mailing lists,
+      source code control systems, and issue tracking systems that are managed
+      by, or on behalf of, the Licensor for the purpose of discussing and
+      improving the Work, but excluding communication that is conspicuously
+      marked or designated in writing by the copyright owner as "Not a
+      Contribution."
+
+      "Contributor" shall mean Licensor and any Legal Entity on behalf of
+      whom a Contribution has been received by the Licensor and included
+      within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by the combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a cross-claim
+      or counterclaim in a lawsuit) alleging that the Work or any
+      Contribution embodied within the Work constitutes direct or
+      contributory patent infringement, then any patent licenses granted to
+      You under this License for that Work shall terminate as of the date
+      such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or Derivative
+          Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Work
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file, You must include a
+          readable copy of the attribution notices contained within such
+          NOTICE file, in at least one of the following places: within a
+          NOTICE text file distributed as part of the Derivative Works;
+          within the Source form or documentation, if provided along with
+          the Derivative Works; or, within a display generated by the
+          Derivative Works, if and wherever such third-party notices
+          normally appear. The contents of the NOTICE file are for
+          informational purposes only and do not modify the License. You
+          may add Your own attribution notices within Derivative Works
+          that You distribute, alongside or as an addendum to the NOTICE
+          text from the Work, provided that such additional attribution
+          notices cannot be construed as modifying the License.
+
+      You may add Your own license statement for Your modifications and
+      may provide additional grant of rights to use, copy, modify, merge,
+      publish, distribute, sublicense, and/or sell copies of the
+      Derivative Works, as separate terms and conditions for use,
+      reproduction, or distribution of Your modifications, or for such
+      Derivative Works as a whole, subject to the terms and conditions
+      of this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or reproducing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or exemplary damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (even if such Contributor has been advised of the possibility
+      of such damages).
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may offer only
+      conditions consistent with the terms of this License.
+
+   END OF TERMS AND CONDITIONS
+
+   Copyright 2026 MoonyFringers
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

ladon_hackernews-0.0.1/PKG-INFO

@@ -0,0 +1,282 @@
+Metadata-Version: 2.4
+Name: ladon-hackernews
+Version: 0.0.1
+Summary: Hacker News adapter for the Ladon crawler framework.
+Author-email: Alessio Pascucci <feeder81@gmail.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/MoonyFringers/ladon-hackernews
+Project-URL: Repository, https://github.com/MoonyFringers/ladon-hackernews
+Project-URL: Framework, https://github.com/MoonyFringers/ladon
+Project-URL: Issues, https://github.com/MoonyFringers/ladon-hackernews/issues
+Project-URL: Documentation, https://moonyfringers.github.io/ladon
+Keywords: web-crawler,crawler,hackernews,hacker-news,ladon,ladon-adapter,duckdb,data-pipeline
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: ladon-crawl>=0.0.1
+Requires-Dist: duckdb>=1.0.0
+Requires-Dist: pytz>=2023.3
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pre-commit; extra == "dev"
+Dynamic: license-file
+
+# ladon-hackernews
+
+[![CI](https://github.com/MoonyFringers/ladon-hackernews/actions/workflows/test.yml/badge.svg)](https://github.com/MoonyFringers/ladon-hackernews/actions/workflows/test.yml)
+[![License: Apache 2.0](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11%2B-blue)](https://www.python.org/downloads/)
+
+Hacker News adapter for the [Ladon](https://github.com/MoonyFringers/ladon) crawler framework.
+
+Crawls the HN top-stories list, expands each story into its direct comments,
+and persists everything to a DuckDB database — ready to export as Parquet for
+LLM training pipelines or downstream analysis.
+
+## Quick start
+
+```bash
+pip install ladon-hackernews
+ladon-hackernews --top 30 --out hn.db
+```
+
+No authentication. No external server.
+
+## What you get
+
+Each run writes two tables to `hn.db`:
+
+**`hn_comments`** — one row per HN comment (the leaf record):
+
+| column | type | description |
+|---|---|---|
+| `id` | INTEGER | HN comment ID |
+| `story_id` | INTEGER | Parent story ID |
+| `parent_id` | INTEGER | Immediate parent (story or comment) |
+| `by` | TEXT | Author username |
+| `text` | TEXT | Raw HTML comment body |
+| `time` | TIMESTAMPTZ | Comment timestamp (UTC) |
+| `run_id` | TEXT | UUID of the crawl run that wrote this row |
+
+**`ladon_runs`** — one row per story crawled (upserted twice: at start and finish):
+
+| column | type | description |
+|---|---|---|
+| `run_id` | TEXT | UUID for this crawl run |
+| `plugin_name` | TEXT | Always `"hackernews"` |
+| `top_ref` | TEXT | HN item URL that was the root of this run |
+| `started_at` | TIMESTAMPTZ | When the run started (UTC) |
+| `finished_at` | TIMESTAMPTZ | When the run finished; NULL while running |
+| `status` | TEXT | `done`, `partial`, `not_ready`, `failed`, or `running` |
+| `leaves_consumed` | INTEGER | Comments for which `sink.consume()` succeeded |
+| `leaves_persisted` | INTEGER | Comments successfully written to `hn_comments` |
+| `leaves_failed` | INTEGER | Comments that failed to fetch or persist |
+| `branch_errors` | INTEGER | Expander-level errors (branch could not be expanded) |
+| `errors` | TEXT | JSON array of error message strings |
+
+### Sample DuckDB query
+
+```sql
+-- Top commenters across all crawled stories
+SELECT "by", COUNT(*) AS comments
+FROM hn_comments
+GROUP BY "by"
+ORDER BY comments DESC
+LIMIT 10;
+```
+
+HN comments are structured, human-authored, and high signal-to-noise — a useful
+corpus for instruction tuning and dialogue modelling. A typical pipeline looks like:
+
+```
+ladon-hackernews --top 500 --out hn.db
+    → export_parquet("hn.db", "hn.parquet")
+        → training pipeline
+```
+
+## Export to Parquet
+
+```python
+from ladon_hackernews import export_parquet
+
+export_parquet("hn.db", "hn.parquet")
+```
+
+## CLI reference
+
+```
+ladon-hackernews [--top N] [--out PATH] [--verbose]
+```
+
+| flag | default | description |
+|---|---|---|
+| `--top N` | `30` | Number of top stories to crawl (range: 1–500) |
+| `--out PATH` | `hn.db` | Output DuckDB database path |
+| `--verbose`, `-v` | off | Show DEBUG-level framework messages (leaf warnings, HTTP timings) |
+
+In default mode the terminal shows one progress line per story and a summary at
+the end. Framework-level noise (`leaf unavailable`, `expander branch failed`) is
+suppressed; partial stories print a `↳` hint instead. Pass `--verbose` to expose
+the raw framework log messages.
+
+---
+
+## Use as a library
+
+The CLI is the simplest way to run a crawl, but `HNPlugin` and
+`HNDuckDBRepository` can be used directly as a library — useful when you need
+custom scheduling, batching, or integration into a larger pipeline.
+
+```python
+import uuid
+from datetime import datetime, timezone
+
+from ladon.networking.client import HttpClient
+from ladon.networking.config import HttpClientConfig
+from ladon.persistence import RunAudit, RunRecord
+from ladon.plugins.errors import ExpansionNotReadyError
+from ladon.plugins.models import Ref
+from ladon.runner import RunConfig, run_crawl
+from ladon_hackernews import HNPlugin, HNDuckDBRepository
+
+plugin = HNPlugin(top=10)
+config = RunConfig()
+client_config = HttpClientConfig(user_agent="my-bot/1.0")
+
+with HNDuckDBRepository("hn.db") as repo, HttpClient(client_config) as client:
+    for story_ref in plugin.source.discover(client):
+        if not isinstance(story_ref, Ref):
+            raise TypeError(f"unexpected type {type(story_ref).__name__}")
+        run_id = str(uuid.uuid4())
+        run = RunRecord(
+            run_id=run_id,
+            plugin_name=plugin.name,
+            top_ref=story_ref.url,
+            started_at=datetime.now(tz=timezone.utc),
+            status="running",
+        )
+        if isinstance(repo, RunAudit):
+            repo.record_run(run)
+
+        try:
+            result = run_crawl(
+                story_ref, plugin, client, config,
+                # Default-argument capture binds run_id to each lambda.
+                on_leaf=lambda rec, _, _id=run_id: repo.write_leaf(rec, _id),
+            )
+            run.branch_errors = sum(
+                1 for e in result.errors if e.startswith("expander branch")
+            )
+            run.status = (
+                "partial"
+                if result.leaves_failed
+                or result.leaves_consumed > result.leaves_persisted
+                or run.branch_errors
+                else "done"
+            )
+            run.leaves_consumed = result.leaves_consumed
+            run.leaves_persisted = result.leaves_persisted
+            run.leaves_failed = result.leaves_failed
+            run.errors = result.errors
+        except ExpansionNotReadyError:
+            run.status = "not_ready"
+        except Exception as exc:
+            run.status = "failed"
+            run.errors = (str(exc),)
+        finally:
+            run.finished_at = datetime.now(tz=timezone.utc)
+            if isinstance(repo, RunAudit):
+                repo.record_run(run)
+```
+
+## How it works
+
+This adapter implements the Ladon [SES (Source / Expander / Sink)](https://github.com/MoonyFringers/ladon/blob/main/docs/decisions/adr-004-ses-protocol-design.md) protocol against the [HN Firebase API](https://github.com/HackerNews/API).
+
+### Pipeline
+
+```mermaid
+flowchart TB
+    subgraph plugin ["SES Plugin"]
+        direction LR
+        SRC["HNSource\ndiscover()"] -- "Ref × N" --> EXP["HNExpander\nexpand()"] -- "StoryRecord\nRef × M" --> SNK["HNSink\nconsume()"]
+    end
+    subgraph persistence ["Persistence"]
+        direction LR
+        REPO["HNDuckDBRepository"] --> DB[("hn.db")] -- "export_parquet()" --> PQ[("hn.parquet")]
+    end
+    SNK -- "CommentRecord" --> REPO
+```
+
+### Domain records
+
+```mermaid
+flowchart LR
+    SR["StoryRecord\n─────────────\nid · title · url\nby · score · time\ndescendants · comment_ids"]
+    CR["CommentRecord\n─────────────\nid · story_id · parent_id\nby · text · time"]
+
+    SR -- "expands into" --> CR
+```
+
+### SES class map
+
+| Layer | Class | HN API call |
+|---|---|---|
+| `Source` | `HNSource` | `GET /v0/topstories.json` → story ID list |
+| `Expander` | `HNExpander` | `GET /v0/item/{story_id}.json` → comment refs |
+| `Sink` | `HNSink` | `GET /v0/item/{comment_id}.json` → `CommentRecord` |
+
+`HNDuckDBRepository` implements both `Repository` (leaf persistence) and
+`RunAudit` (run history) from [`ladon.persistence`](https://github.com/MoonyFringers/ladon/blob/main/docs/decisions/adr-006-persistence-layer.md)
+— structurally, with no Ladon base class imported.
+
+Each story is one independent run. The run audit trail lets you resume from
+the last successful crawl:
+
+```python
+last = repo.get_last_run("hackernews")  # most recent "done" run
+```
+
+## Writing your own adapter
+
+`ladon-hackernews` is the canonical reference for building a Ladon adapter.
+See the [Ladon documentation](https://moonyfringers.github.io/ladon/) and
+[ADR-003](https://github.com/MoonyFringers/ladon/blob/main/docs/decisions/adr-003-plugin-adapter-interface.md)
+for the full adapter authoring guide.
+
+Key pattern: adapters implement Ladon protocols **structurally** — no
+inheritance from any Ladon base class is required. Only `RunRecord` needs
+to be imported for `RunAudit` implementations.
+
+## Development
+
+```bash
+git clone https://github.com/MoonyFringers/ladon-hackernews
+cd ladon-hackernews
+pip install -e ".[dev]"
+pre-commit install
+pytest tests/ -v
+```
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for full guidelines.
+
+## License
+
+Apache-2.0 — see [LICENSE](LICENSE).
+
+The [Ladon](https://github.com/MoonyFringers/ladon) core framework is
+AGPL-3.0-only. `ladon-hackernews` is Apache-2.0 but has a runtime
+dependency on Ladon core; review the AGPL terms if you plan to distribute
+or run this as a network service.