toapi 2.2.0__tar.gz → 2.2.1__tar.gz

toapi-2.2.1/.claude/settings.local.json

@@ -0,0 +1,9 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(uv run *)",
+      "Bash(uv sync *)",
+      "Bash(uv build *)"
+    ]
+  }
+}

{toapi-2.2.0 → toapi-2.2.1}/.gitignore

@@ -1,4 +1,5 @@
 .idea/
+.omc/
 # Byte-compiled / optimized / DLL files
 __pycache__/
 *.py[cod]

toapi-2.2.1/PKG-INFO

@@ -0,0 +1,145 @@
+Metadata-Version: 2.4
+Name: toapi
+Version: 2.2.1
+Summary: Every web site provides APIs.
+Project-URL: homepage, https://github.com/gaojiuli/toapi
+Project-URL: repository, https://github.com/gaojiuli/toapi
+Project-URL: documentation, https://gaojiuli.github.io/toapi/
+Author-email: Elliot Gao <gaojiuli@gmail.com>
+License: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: charset-normalizer>=3.3
+Requires-Dist: click>=8.1
+Requires-Dist: colorama>=0.4.6
+Requires-Dist: cssselect>=1.2
+Requires-Dist: flask>=3.0
+Requires-Dist: htmlfetcher>=0.0.6
+Requires-Dist: htmlparsing>=0.1.5
+Requires-Dist: requests>=2.32
+Description-Content-Type: text/markdown
+
+# toapi
+
+[![CI](https://github.com/elliotgao2/toapi/actions/workflows/ci.yml/badge.svg)](https://github.com/elliotgao2/toapi/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/toapi.svg)](https://pypi.org/project/toapi/)
+[![Python](https://img.shields.io/pypi/pyversions/toapi.svg)](https://pypi.org/project/toapi/)
+[![License](https://img.shields.io/pypi/l/toapi.svg)](https://pypi.org/project/toapi/)
+
+> Turn any website into a JSON API — declaratively.
+
+`toapi` lets you point at a web page, declare the fields you want with CSS
+selectors, and get back a clean JSON API. No crawler to babysit, no database to
+maintain — pages are fetched and parsed on demand, with built‑in caching.
+
+## Install
+
+```bash
+pip install toapi
+```
+
+Requires Python 3.10+.
+
+## Quickstart
+
+```python
+from htmlparsing import Attr, Text
+from toapi import Api, Item
+
+api = Api()
+
+
+@api.site("https://news.ycombinator.com")
+@api.list(".athing")
+@api.route("/posts", "/news")
+@api.route("/posts?page={page}", "/news?p={page}")
+class Post(Item):
+    title = Text(".titleline > a")
+    url = Attr(".titleline > a", "href")
+
+
+api.run(host="127.0.0.1", port=5000)
+```
+
+Run it:
+
+```bash
+python app.py
+```
+
+Then visit <http://127.0.0.1:5000/posts> and you get:
+
+```json
+{
+  "Post": [
+    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://www.quantamagazine.org/..."},
+    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://jalopnik.com/..."}
+  ]
+}
+```
+
+## How it works
+
+```
+   ┌────────────┐    ┌────────────┐    ┌────────────┐
+   │  /posts    │ ─▶ │  fetch     │ ─▶ │  parse     │ ─▶  JSON
+   │  (route)   │    │  (cache)   │    │  (Item)    │
+   └────────────┘    └────────────┘    └────────────┘
+```
+
+1. **Route** — `@api.route("/posts", "/news")` maps your API path to a source URL.
+2. **Fetch** — pages are fetched with `requests` (or a headless browser if you pass `browser=`) and cached in memory.
+3. **Parse** — each `Item` extracts fields with CSS selectors via `htmlparsing`.
+4. **Serve** — Flask returns the result as JSON; subsequent calls hit the cache.
+
+## Features
+
+- **Declarative** — describe data, not scraping logic.
+- **Routes** — map clean API paths to messy source URLs with `{param}` placeholders.
+- **Multi-site** — merge several websites behind one API.
+- **Cleaning hooks** — define `clean_<field>` methods to post-process values.
+- **Caching** — pages and parsed results are cached automatically.
+- **Headless browser** — pass `Api(browser="/path/to/geckodriver")` for JS-heavy sites.
+
+## Cleaning values
+
+Add a `clean_<fieldname>` method on the Item to transform a value before it's
+returned:
+
+```python
+@api.site("https://news.ycombinator.com")
+@api.route("/posts", "/news")
+class Page(Item):
+    next_page = Attr(".morelink", "href")
+
+    def clean_next_page(self, value):
+        return f"/posts?{value.split('?', 1)[1]}"
+```
+
+## Development
+
+```bash
+git clone https://github.com/elliotgao2/toapi.git
+cd toapi
+uv sync          # install deps into .venv
+uv run pytest    # run tests
+uv run ruff check .
+```
+
+We use [uv](https://github.com/astral-sh/uv) for packaging and
+[ruff](https://github.com/astral-sh/ruff) for lint + format. Pre-commit hooks
+keep both clean:
+
+```bash
+uv run pre-commit install
+```
+
+## Contributing
+
+Pull requests are welcome. For non-trivial changes, please open an issue first
+to discuss what you'd like to change. Make sure `uv run pytest` and
+`uv run ruff check .` pass before submitting.
+
+## License
+
+[MIT](LICENSE) © Elliot Gao

toapi-2.2.1/README.md

@@ -0,0 +1,124 @@
+# toapi
+
+[![CI](https://github.com/elliotgao2/toapi/actions/workflows/ci.yml/badge.svg)](https://github.com/elliotgao2/toapi/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/toapi.svg)](https://pypi.org/project/toapi/)
+[![Python](https://img.shields.io/pypi/pyversions/toapi.svg)](https://pypi.org/project/toapi/)
+[![License](https://img.shields.io/pypi/l/toapi.svg)](https://pypi.org/project/toapi/)
+
+> Turn any website into a JSON API — declaratively.
+
+`toapi` lets you point at a web page, declare the fields you want with CSS
+selectors, and get back a clean JSON API. No crawler to babysit, no database to
+maintain — pages are fetched and parsed on demand, with built‑in caching.
+
+## Install
+
+```bash
+pip install toapi
+```
+
+Requires Python 3.10+.
+
+## Quickstart
+
+```python
+from htmlparsing import Attr, Text
+from toapi import Api, Item
+
+api = Api()
+
+
+@api.site("https://news.ycombinator.com")
+@api.list(".athing")
+@api.route("/posts", "/news")
+@api.route("/posts?page={page}", "/news?p={page}")
+class Post(Item):
+    title = Text(".titleline > a")
+    url = Attr(".titleline > a", "href")
+
+
+api.run(host="127.0.0.1", port=5000)
+```
+
+Run it:
+
+```bash
+python app.py
+```
+
+Then visit <http://127.0.0.1:5000/posts> and you get:
+
+```json
+{
+  "Post": [
+    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://www.quantamagazine.org/..."},
+    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://jalopnik.com/..."}
+  ]
+}
+```
+
+## How it works
+
+```
+   ┌────────────┐    ┌────────────┐    ┌────────────┐
+   │  /posts    │ ─▶ │  fetch     │ ─▶ │  parse     │ ─▶  JSON
+   │  (route)   │    │  (cache)   │    │  (Item)    │
+   └────────────┘    └────────────┘    └────────────┘
+```
+
+1. **Route** — `@api.route("/posts", "/news")` maps your API path to a source URL.
+2. **Fetch** — pages are fetched with `requests` (or a headless browser if you pass `browser=`) and cached in memory.
+3. **Parse** — each `Item` extracts fields with CSS selectors via `htmlparsing`.
+4. **Serve** — Flask returns the result as JSON; subsequent calls hit the cache.
+
+## Features
+
+- **Declarative** — describe data, not scraping logic.
+- **Routes** — map clean API paths to messy source URLs with `{param}` placeholders.
+- **Multi-site** — merge several websites behind one API.
+- **Cleaning hooks** — define `clean_<field>` methods to post-process values.
+- **Caching** — pages and parsed results are cached automatically.
+- **Headless browser** — pass `Api(browser="/path/to/geckodriver")` for JS-heavy sites.
+
+## Cleaning values
+
+Add a `clean_<fieldname>` method on the Item to transform a value before it's
+returned:
+
+```python
+@api.site("https://news.ycombinator.com")
+@api.route("/posts", "/news")
+class Page(Item):
+    next_page = Attr(".morelink", "href")
+
+    def clean_next_page(self, value):
+        return f"/posts?{value.split('?', 1)[1]}"
+```
+
+## Development
+
+```bash
+git clone https://github.com/elliotgao2/toapi.git
+cd toapi
+uv sync          # install deps into .venv
+uv run pytest    # run tests
+uv run ruff check .
+```
+
+We use [uv](https://github.com/astral-sh/uv) for packaging and
+[ruff](https://github.com/astral-sh/ruff) for lint + format. Pre-commit hooks
+keep both clean:
+
+```bash
+uv run pre-commit install
+```
+
+## Contributing
+
+Pull requests are welcome. For non-trivial changes, please open an issue first
+to discuss what you'd like to change. Make sure `uv run pytest` and
+`uv run ruff check .` pass before submitting.
+
+## License
+
+[MIT](LICENSE) © Elliot Gao

toapi-2.2.1/docs/about/contributing.md

@@ -0,0 +1,51 @@
+# Contributing
+
+Thanks for your interest in improving `toapi`! Bug reports, feature ideas,
+documentation tweaks, and pull requests are all welcome.
+
+## Reporting an issue
+
+Open an issue on [GitHub](https://github.com/elliotgao2/toapi/issues) with:
+
+- What you tried
+- What you expected to happen
+- What actually happened (including the full error and traceback)
+- Your Python version and `toapi` version
+
+## Setting up a development environment
+
+We use [uv](https://github.com/astral-sh/uv) for packaging and
+[ruff](https://github.com/astral-sh/ruff) for lint and format.
+
+```bash
+git clone https://github.com/elliotgao2/toapi.git
+cd toapi
+uv sync
+```
+
+Install the pre-commit hooks so ruff runs on every commit:
+
+```bash
+uv run pre-commit install
+```
+
+## Running the checks
+
+```bash
+uv run pytest               # tests
+uv run ruff check .         # lint
+uv run ruff format --check . # format
+```
+
+CI runs the same checks on Python 3.10, 3.11, and 3.12.
+
+## Submitting a pull request
+
+1. Fork the repo and create a topic branch.
+2. Make your change. Keep diffs focused — one concern per PR.
+3. Add or update tests when the behavior changes.
+4. Make sure `pytest` and `ruff check` pass locally.
+5. Open the PR with a short description of *what* changed and *why*.
+
+For non-trivial changes, please open an issue first so we can discuss the
+approach before you spend time on it.

toapi-2.2.1/docs/about/installation.md

@@ -0,0 +1,49 @@
+# Installation
+
+## Requirements
+
+- Python 3.10 or newer
+- pip (or [uv](https://github.com/astral-sh/uv),
+  [pipx](https://pipx.pypa.io/), [Poetry](https://python-poetry.org/) — any
+  modern installer)
+
+Check your Python version:
+
+```bash
+python --version
+```
+
+## Install from PyPI
+
+```bash
+pip install toapi
+```
+
+Or with uv:
+
+```bash
+uv add toapi
+```
+
+## Verify
+
+```bash
+python -c "import toapi; print(toapi.__version__)"
+```
+
+## Upgrade
+
+```bash
+pip install -U toapi
+```
+
+## Install from source
+
+```bash
+git clone https://github.com/elliotgao2/toapi.git
+cd toapi
+uv sync
+```
+
+This drops you in a working development environment with all dependencies
+and dev tools.

toapi-2.2.1/docs/about/license.md

@@ -0,0 +1,27 @@
+# License
+
+`toapi` is released under the MIT License.
+
+```
+MIT License
+
+Copyright (c) 2021 Elliot Gao
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to
+deal in the Software without restriction, including without limitation the
+rights to use, copy, modify, merge, publish, distribute, sublicense, and/or
+sell copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS
+IN THE SOFTWARE.
+```

toapi-2.2.1/docs/about/release-notes.md

@@ -0,0 +1,39 @@
+# Release Notes
+
+## Upgrading
+
+```bash
+pip install -U toapi
+```
+
+Or with uv:
+
+```bash
+uv add toapi@latest
+```
+
+## Changelog
+
+### 2.2.0 (2026-05-22)
+
+- Switched packaging from Poetry to [uv](https://github.com/astral-sh/uv)
+  (PEP 621 + hatchling).
+- Raised the minimum Python version to 3.10.
+- Replaced the abandoned `cchardet` dependency with `charset-normalizer`.
+- Bumped Flask 2 → 3, plus `requests`, `click`, `colorama`, and `cssselect`
+  to current majors.
+- Replaced black + isort + flake8 + `pytest-pep8` with a single
+  [ruff](https://github.com/astral-sh/ruff) toolchain.
+- Replaced Travis CI with GitHub Actions on a 3.10 / 3.11 / 3.12 matrix.
+- Replaced the `ItemType` metaclass with `__init_subclass__` — same
+  behavior, half the code.
+- `__version__` is now sourced from package metadata, fixing an import
+  error in `toapi.cli`.
+
+### 2.1.x
+
+- Maintenance releases on the old Poetry / Python 3.8 stack.
+
+### 1.0.0 (2017-12-26)
+
+- Initial release.

toapi-2.2.1/docs/index.md

@@ -0,0 +1,49 @@
+# toapi
+
+> Turn any website into a JSON API — declaratively.
+
+`toapi` lets you point at a web page, declare the fields you want with CSS
+selectors, and get a clean JSON API back. No crawler to babysit, no database
+to maintain — pages are fetched and parsed on demand, with built-in caching.
+
+## A 10-line example
+
+```python
+from htmlparsing import Attr, Text
+from toapi import Api, Item
+
+api = Api()
+
+
+@api.site("https://news.ycombinator.com")
+@api.list(".athing")
+@api.route("/posts", "/news")
+class Post(Item):
+    title = Text(".titleline > a")
+    url = Attr(".titleline > a", "href")
+
+
+api.run(host="127.0.0.1", port=5000)
+```
+
+Visit `http://127.0.0.1:5000/posts` and you get a JSON list of every story
+on the front page.
+
+## How it works
+
+1. **Route** — `@api.route("/posts", "/news")` maps your API path to a source
+   URL.
+2. **Fetch** — pages are fetched with `requests` (or a headless browser if
+   you pass `browser=`) and cached in memory.
+3. **Parse** — each `Item` extracts fields with CSS selectors via
+   `htmlparsing`.
+4. **Serve** — Flask returns the result as JSON; subsequent calls hit the
+   cache.
+
+## Next steps
+
+- [Quickstart](quickstart.md) — a complete walk-through with two routes and a
+  clean method.
+- [Api](topics/api.md) — the `Api` class and its decorators.
+- [Item](topics/item.md) — how to declare data shapes.
+- [Selectors](topics/selector.md) — picking values out of HTML.

toapi-2.2.1/docs/quickstart.md

@@ -0,0 +1,84 @@
+# Quickstart
+
+Build a small API in front of Hacker News. By the end you'll have two routes,
+a list of posts, and a cleaned `next_page` URL that loops back into your own
+API.
+
+## 1. Install
+
+```bash
+pip install toapi
+```
+
+Requires Python 3.10+.
+
+## 2. Write `app.py`
+
+```python
+from flask import request
+from htmlparsing import Attr, Text
+from toapi import Api, Item
+
+api = Api()
+
+
+@api.site("https://news.ycombinator.com")
+@api.list(".athing")
+@api.route("/posts", "/news")
+@api.route("/posts?page={page}", "/news?p={page}")
+class Post(Item):
+    title = Text(".titleline > a")
+    url = Attr(".titleline > a", "href")
+
+
+@api.site("https://news.ycombinator.com")
+@api.route("/posts", "/news")
+@api.route("/posts?page={page}", "/news?p={page}")
+class Page(Item):
+    next_page = Attr(".morelink", "href")
+
+    def clean_next_page(self, value):
+        return api.convert_string(
+            "/" + value,
+            "/news?p={page}",
+            request.host_url.strip("/") + "/posts?page={page}",
+        )
+
+
+api.run(host="127.0.0.1", port=5000)
+```
+
+## 3. Run
+
+```bash
+python app.py
+```
+
+Then open <http://127.0.0.1:5000/posts>:
+
+```json
+{
+  "Post": [
+    {"title": "Mathematicians Crack the Cursed Curve", "url": "https://..."},
+    {"title": "Stuffing a Tesla Drivetrain into a 1981 Honda Accord", "url": "https://..."}
+  ],
+  "Page": {
+    "next_page": "http://127.0.0.1:5000/posts?page=2"
+  }
+}
+```
+
+## What just happened?
+
+- `@api.site(...)` told the item which website to scrape from.
+- `@api.list(".athing")` said *this item repeats* — each `.athing` element on
+  the page becomes one entry.
+- `@api.route(api_path, source_path)` mapped the path your users hit to the
+  path on the source site. `{page}` is a placeholder passed through both
+  directions.
+- `Text(...)` and `Attr(...)` are CSS selectors that pull a value out of each
+  matched element.
+- `clean_next_page(self, value)` runs after parsing and rewrites the source
+  pagination link to point back at our own API.
+
+That's the whole framework. See [Topics](topics/api.md) for the details.

toapi-2.2.1/docs/topics/api.md

@@ -0,0 +1,83 @@
+# Api
+
+`Api` is the entry point. It owns the Flask app, the cache, and the registry
+of routes.
+
+```python
+from toapi import Api
+
+api = Api()
+```
+
+## Constructor
+
+```python
+Api(site: str = "", browser: str | None = None)
+```
+
+- **`site`** — a default base URL prefix appended in front of every Item's
+  source path. Most users leave this blank and put the site on the Item with
+  `@api.site(...)`.
+- **`browser`** — path to a headless-browser driver (e.g. `geckodriver`).
+  When set, pages are fetched through the browser instead of plain
+  `requests`. Useful for JavaScript-heavy sites.
+
+## Decorators
+
+Decorators are stacked on an `Item` class to declare *what* to scrape,
+*where* it lives, and *which URLs* expose it.
+
+### `@api.site(url)`
+
+Sets the source website for an Item.
+
+```python
+@api.site("https://news.ycombinator.com")
+class Post(Item): ...
+```
+
+### `@api.list(selector)`
+
+Marks the Item as a *list item* — the parser will return one entry per
+element matched by `selector` on the source page.
+
+```python
+@api.list(".athing")
+class Post(Item): ...
+```
+
+Without `@api.list`, the Item is a *detail item* — it parses a single record
+from the page.
+
+### `@api.route(api_path, source_path)`
+
+Maps a path on your API to a path on the source site. Placeholders like
+`{page}` are passed through both directions.
+
+```python
+@api.route("/posts?page={page}", "/news?p={page}")
+@api.route("/posts", "/news")
+class Post(Item): ...
+```
+
+Multiple `@api.route` decorators may be stacked on the same Item.
+
+## `api.run(host, port, **flask_options)`
+
+Starts the Flask development server.
+
+```python
+api.run(host="0.0.0.0", port=5000, debug=True)
+```
+
+For production, mount `api.app` (a plain Flask app) under your WSGI server of
+choice — gunicorn, uWSGI, waitress.
+
+## Caching
+
+Two in-memory caches are populated automatically:
+
+- **Page cache** (`api._storage`) — keyed by source URL, stores raw HTML.
+- **Result cache** (`api._cache`) — keyed by API path, stores parsed JSON.
+
+Both live for the lifetime of the process. Restart to clear.