diffbot 2.0.0__tar.gz → 3.1.0__tar.gz

diffbot-3.1.0/.gitignore

@@ -0,0 +1,11 @@
+# Python
+.venv
+__pycache__
+.pytest_cache
+.env
+dist/
+build/
+*.egg-info/
+
+# Claude
+.claude/settings.local.json

diffbot-3.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Diffbot
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

diffbot-3.1.0/PKG-INFO

@@ -0,0 +1,423 @@
+Metadata-Version: 2.5
+Name: diffbot
+Version: 3.1.0
+Summary: Python client library for Diffbot APIs
+Project-URL: Homepage, https://github.com/diffbot/diffbot-python
+Project-URL: Documentation, https://github.com/diffbot/diffbot-python#readme
+Project-URL: Repository, https://github.com/diffbot/diffbot-python
+Project-URL: Issues, https://github.com/diffbot/diffbot-python/issues
+Author-email: Jerome Choo <jerome@diffbot.com>, Mike Tung <miket@diffbot.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: api-client,crawler,diffbot,extract,knowledge-graph,llm,nlp,web-scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Requires-Dist: click>=8.1.0
+Requires-Dist: httpx>=0.27.0
+Requires-Dist: rich>=13.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Diffbot Python Library
+
+> ⚠️ **The PyPI package is now [`diffbot`](https://pypi.org/project/diffbot/)** (`pip install diffbot`).
+> `diffbot-python` is legacy and its final release (0.3.0) will never be updated.
+> See [Installation](#installation) to migrate.
+
+Python client library for [Diffbot](https://www.diffbot.com) APIs.
+
+
+## Installation
+
+Install the [standalone CLI binary](#standalone-binary) for [agentic use](#how-to-use-with-an-agent):
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/diffbot/diffbot-python/main/install.sh | sh
+```
+
+If you prefer, the full Python library can also be installed with pip:
+
+```bash
+python3 -m pip install diffbot
+```
+
+> This package was formerly published as `diffbot-python`, whose final release (0.3.0) gets no further updates.
+> If you're upgrading from it, run `pip uninstall diffbot-python && pip install --force-reinstall diffbot`
+> and replace `diffbot-python` with `diffbot` in your requirements.
+
+For local development:
+
+```bash
+pip install -e ".[dev]"
+```
+
+## Usage
+
+### Authentication
+
+The CLI and the library can share a single credential. The token always has to be
+passed to the client explicitly, but `resolve_token()` gives you the same lookup the
+CLI uses, in this order:
+
+1. An explicit token passed to `resolve_token(token)`.
+2. The `DIFFBOT_API_TOKEN` environment variable.
+3. A `DIFFBOT_API_TOKEN=...` line in `~/.diffbot/credentials`.
+
+Set it once and it works for both the CLI and your scripts. Either export it:
+
+```bash
+export DIFFBOT_API_TOKEN=<TOKEN>
+```
+
+…or write it to the shared credentials file (handy for keeping it out of your shell environment):
+
+```bash
+mkdir -p ~/.diffbot
+printf 'DIFFBOT_API_TOKEN=%s\n' '<TOKEN>' > ~/.diffbot/credentials
+chmod 600 ~/.diffbot/credentials
+```
+
+With either in place, resolve the token and pass it to the client:
+
+```python
+from diffbot import Diffbot, resolve_token
+
+db = Diffbot(token=resolve_token())  # from env var or ~/.diffbot/credentials
+data = db.extract("https://www.example.com")
+```
+
+### Client configuration
+
+`Diffbot` and `DiffbotAsync` take the same keyword arguments — the client is the single
+place to configure the SDK.
+
+| Argument | Default | Used by |
+|----------|---------|---------|
+| `token` | — (required) | all |
+| `timeout` | `30.0` (seconds) | all |
+| `transport` | httpx default | all |
+| `analyze_url` | `https://api.diffbot.com/v3` | `extract` |
+| `crawler_url` | `https://api.diffbot.com/v3/crawl` | `crawl`, `crawl_list_jobs`, `crawl_get_job`, `crawl_delete_job` |
+| `llm_url` | `https://llm.diffbot.com/rag/v1/chat/completions` | `ask`, `ask_json` |
+| `web_search_url` | `https://llm.diffbot.com/api/v1/web_search` | `web_search` |
+| `nlp_url` | `https://nl.diffbot.com/v1/` | `entities` |
+| `dql_url` | `https://kg.diffbot.com/kg/v3/dql` | `dql`, `dql_parallel` |
+| `ontology_url` | `https://kg.diffbot.com/kg/ontology` | `dql_fetch_ontology`, `dql_refresh_ontology` |
+
+```python
+from diffbot import Diffbot
+
+db = Diffbot(
+    token="YOUR_TOKEN",
+    timeout=60.0,
+    dql_url="http://localhost:8080/kg/v3/dql",
+)
+```
+
+`analyze_url` is a base the SDK appends `/{api}` to, and `crawler_url` is used as given
+for job management and with `/data` appended for crawl results. The rest are complete
+endpoints, used as given.
+
+Passing `transport` replaces the httpx transport, which is the hook for retries, proxies,
+or mocking in tests. `Diffbot` takes an `httpx.BaseTransport`, `DiffbotAsync` an
+`httpx.AsyncBaseTransport`:
+
+```python
+import httpx
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN", transport=httpx.HTTPTransport(retries=3))
+```
+
+### Extract structured content
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+data = db.extract("https://www.example.com")
+```
+
+### Ask Diffbot LLM
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+for chunk in db.ask([{"role": "user", "content": "What's the capital of France?"}]):
+    print(chunk, end="")
+```
+
+### Structured output
+Pass a JSON Schema to constrain the answer. The model is held to the schema by a grammar during decoding, so the result parses reliably even though the answer is retrieved live from the web.
+
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+schema = {
+    "type": "object",
+    "properties": {
+        "country": {"type": "string"},
+        "capital": {"type": "string"},
+    },
+    "required": ["country", "capital"],
+}
+answer = db.ask_json([{"role": "user", "content": "What's the capital of France?"}], schema)
+print(answer["capital"])
+```
+
+Omit the schema to let the model choose the shape, or use `ask` with `response_format` to stream a constrained answer:
+
+```python
+from diffbot import Diffbot, json_schema_format
+
+db = Diffbot(token="YOUR_TOKEN")
+answer = db.ask_json([{"role": "user", "content": "What's the capital of France?"}])
+
+for chunk in db.ask(
+    [{"role": "user", "content": "What's the capital of France?"}],
+    response_format=json_schema_format(schema),
+):
+    print(chunk, end="")
+```
+
+> **Avoid `response_format={"type": "json_object"}`.** The endpoint accepts it, but applies no grammar to it — the RAG loop's internal tool call is itself a JSON object, so it can be returned as the final answer. This is reproducible whenever the request includes a system message. `ask_json` therefore defaults to a permissive JSON Schema instead, and raises `ValidationError` if it ever sees a tool call come back.
+
+### Crawl a site for structured content
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+for event in db.crawl("https://www.example.com", hops=1):
+    print(event)
+```
+
+### Query the Knowledge Graph
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+results = db.dql('type:Organization name:"Diffbot"')
+```
+
+### Web Search
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+results = db.web_search("diffbot knowledge graph")
+for r in results["search_results"]:
+    print(r["score"], r["title"], r["pageUrl"])
+    print(r["content"])
+
+# snippet_length controls how much of each page comes back in "content":
+# "short" (~1-1.5 KB per result), "medium", "long" (default), or "full"
+results = db.web_search("diffbot knowledge graph", num_results=5, snippet_length="short")
+
+# "full" with a url: query returns the whole stored page
+page = db.web_search("url:https://en.wikipedia.org/wiki/Knowledge_graph", snippet_length="full")
+```
+
+### Entities (NLP)
+```python
+from diffbot import Diffbot
+
+db = Diffbot(token="YOUR_TOKEN")
+result = db.entities("Apple CEO Tim Cook announced record quarterly earnings.")
+for entity in result["entities"]:
+    print(entity["name"], entity.get("type"), entity.get("id"))
+print("sentiment:", result.get("sentiment"))
+```
+
+## Async Usage
+
+### Extract structured content
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        data = await db.extract("https://www.example.com")
+        print(data)
+
+asyncio.run(main())
+```
+
+### Ask Diffbot LLM
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        async for chunk in db.ask([{"role": "user", "content": "What's the capital of France?"}]):
+            print(chunk, end="")
+
+asyncio.run(main())
+```
+
+### Structured output
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+schema = {
+    "type": "object",
+    "properties": {
+        "country": {"type": "string"},
+        "capital": {"type": "string"},
+    },
+    "required": ["country", "capital"],
+}
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        answer = await db.ask_json(
+            [{"role": "user", "content": "What's the capital of France?"}], schema
+        )
+        print(answer["capital"])
+
+asyncio.run(main())
+```
+
+### Crawl a site for structured content
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        async for event in db.crawl("https://www.example.com", hops=1):
+            print(event)
+
+asyncio.run(main())
+```
+
+### Query the Knowledge Graph
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        results = await db.dql('type:Organization name:"Diffbot"')
+        print(results)
+
+asyncio.run(main())
+```
+
+### Web Search
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        results = await db.web_search("diffbot knowledge graph")
+        for r in results["search_results"]:
+            print(r["score"], r["title"], r["pageUrl"])
+            print(r["content"])
+
+asyncio.run(main())
+```
+
+### Entities (NLP)
+```python
+import asyncio
+from diffbot import DiffbotAsync
+
+async def main():
+    async with DiffbotAsync(token="YOUR_TOKEN") as db:
+        result = await db.entities("Apple CEO Tim Cook announced record quarterly earnings.")
+        for entity in result["entities"]:
+            print(entity["name"], entity.get("type"), entity.get("id"))
+        print("sentiment:", result.get("sentiment"))
+
+asyncio.run(main())
+```
+
+## CLI
+
+This library also includes a CLI exposed as the `db` command.
+
+To make `db` available from anywhere, install it as an isolated tool with [uv](https://docs.astral.sh/uv/):
+
+```bash
+uv tool install .
+```
+
+This drops a `db` executable into `~/.local/bin` (ensure it is on your `PATH`). Use `--force` to reinstall or upgrade after changes, or `--editable` to have source edits take effect immediately. Alternatively, a plain `pip install .` (or `pip install -e .`) also installs the `db` entry point into the active environment.
+
+### Standalone binary
+
+Every release also ships a self-contained `db` binary for Linux (x86_64 and aarch64) and macOS (Apple Silicon) as a Python-free option. The installer detects your platform, verifies the SHA256 checksum, and installs (or upgrades) `db` into `~/.local/bin`:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/diffbot/diffbot-python/main/install.sh | sh
+```
+
+Pin a specific release or install location with flags (or the `DB_VERSION` / `DB_INSTALL_DIR` environment variables); re-running the installer upgrades an existing install in place:
+
+```bash
+curl -fsSL https://raw.githubusercontent.com/diffbot/diffbot-python/main/install.sh | sh -s -- --version v0.2.1 --bin-dir ~/bin
+```
+
+### How to use
+
+```bash
+export DIFFBOT_API_TOKEN=your-token-here
+
+db extract https://www.example.com
+db ask "What's the capital of France?"
+db ask "What's the capital of France?" --json
+db ask "What's the capital of France?" --schema capital.json
+db crawl https://www.example.com --hops 1
+db crawl-list-jobs
+db crawl-delete-job crawl-1234567890
+db web-search "diffbot knowledge graph"
+db web-search "diffbot knowledge graph" -n 5 -f json
+db web-search "diffbot knowledge graph" -n 5 -s short
+db web-search "url:https://en.wikipedia.org/wiki/Knowledge_graph" -s full -f json
+db entities "Apple CEO Tim Cook announced record quarterly earnings."
+db entities "Apple CEO Tim Cook announced record quarterly earnings." -f dql
+```
+
+### How to use with an agent
+Once installed, this library will work alongside [`diffbot-skills`](https://github.com/diffbot/diffbot-skills) to enable your agent full access to structuring web knowledge with Diffbot. Diffbot Agent Skills even unlocks some additional skills like crafting DQL from natural language. 
+
+`diffbot-skills` will pick up or install this library automatically. 
+
+
+## Tests
+
+Run the mock test suite:
+```bash
+python -m pytest
+```
+
+Run live integration tests against the real API (requires a valid token).
+The token is resolved the same way as everywhere else — the `DIFFBOT_API_TOKEN`
+environment variable or `~/.diffbot/credentials`:
+```bash
+DIFFBOT_API_TOKEN=your_token python -m pytest -m live
+```
+
+## Acknowledgements
+
+Thanks to [@attilaolah](https://github.com/attilaolah), who published the original `diffbot` package in 2012 and looked after the name for 14 years.