crewai-specstream 0.1.0__tar.gz

crewai_specstream-0.1.0/.gitignore

@@ -0,0 +1,14 @@
+node_modules/
+dist/
+build/
+*.egg-info/
+__pycache__/
+.venv/
+.pytest_cache/
+*.tsbuildinfo
+*.tgz
+*.whl
+.env
+.env.*
+!.env.example
+.DS_Store

crewai_specstream-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 SpecStream SAS
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

crewai_specstream-0.1.0/PKG-INFO

@@ -0,0 +1,149 @@
+Metadata-Version: 2.5
+Name: crewai-specstream
+Version: 0.1.0
+Summary: Give a CrewAI agent any list from the live web: ask in plain words, get every matching row with the page it came from, within a price you set in code.
+Project-URL: Homepage, https://specstream.ai
+Project-URL: Documentation, https://specstream.ai/docs/crewai
+Project-URL: Repository, https://github.com/Specstreamai/specstream-tools/tree/main/packages/crewai-specstream
+Project-URL: Issues, https://github.com/Specstreamai/specstream-tools/issues
+Project-URL: Pricing, https://specstream.ai/pricing
+Author-email: SpecStream <support@specstream.ai>
+Maintainer-email: SpecStream <support@specstream.ai>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,crewai,crewai-tools,lead-lists,live-web,research,specstream,tools,web-data
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Internet :: WWW/HTTP
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: <3.14,>=3.10
+Requires-Dist: crewai<2.0.0,>=1.0.0
+Provides-Extra: test
+Requires-Dist: pytest>=8; extra == 'test'
+Description-Content-Type: text/markdown
+
+# crewai-specstream
+
+Most web tools for CrewAI hand your agent pages to read, and the agent builds the list itself, one search and one scrape at a time. This one hands it the finished list. The agent asks in plain words – "dental clinics in Lisbon with online booking", "US startups that filed a $5M+ Form D after October 8, 2026" – and gets back every matching row, each with the page it was read from. The list starts only within a price you set in code, and the model can't raise it.
+
+```bash
+pip install -U crewai-specstream
+export SPECSTREAM_API_KEY=ss_live_...   # Agents & API in your SpecStream workspace
+```
+
+A free account comes with $10 of work to use within 30 days and needs no card: [specstream.ai](https://specstream.ai/signup). Works with CrewAI 1.x on Python 3.10 – 3.13.
+
+## A crew that can find lists
+
+```python
+from crewai import Agent, Crew, Task
+from crewai_specstream import specstream_tools
+
+analyst = Agent(
+    role="Funding analyst",
+    goal="Answer questions about who is raising money, with a source for every claim",
+    backstory="You work from lists, not from memory. You cite the page each row came from.",
+    tools=specstream_tools(max_price_usd=5),
+)
+
+task = Task(
+    description=(
+        "Find US startups that raised $5M or more in a Form D filed after October 8, 2026. "
+        "Which industries lead? Link the filing for each company you name."
+    ),
+    expected_output="The three leading industries, with the companies and their filing links.",
+    agent=analyst,
+)
+
+print(Crew(agents=[analyst], tasks=[task]).kickoff().raw)
+```
+
+The agent writes the request and picks how many rows. You decide what one list may cost. The agent uses your default model; pass `llm="..."` to `Agent` to choose another.
+
+## What comes back
+
+One flat row per match: the list's own columns, then where the row was read. This is a real row from the request above.
+
+```json
+{
+  "list_id": "fnd_gk5y3d0r97bvyz0t08zzh75kr6",
+  "position": 26,
+  "Organization": "RapidCanvas Inc",
+  "Published address": "801 BARTON SPRINGS RD, AUSTIN, TEXAS, 78704",
+  "Source industry classification": "Other Technology",
+  "Amount sold (USD)": "28999988",
+  "Funding filing date": "2026-10-14",
+  "source_url": "https://www.sec.gov/Archives/edgar/data/2052031/000205203126000006/primary_doc.xml",
+  "checked_at": "2026-10-14T19:56:02.016693Z",
+  "list_link": "https://specstream.ai/workspace/collections/fnd_gk5y3d0r97bvyz0t08zzh75kr6"
+}
+```
+
+Columns follow the request. Ask for "breweries in Portland, Oregon that host events, with their website" and the rows carry the brewery and its website instead. A fact a source doesn't publish comes back empty, never guessed.
+
+## The two tools
+
+| Tool | The model gives | What happens | Cost |
+| --- | --- | --- | --- |
+| `specstream_find_list` | `request`, `max_results` (1 – 1000, default 25) | Reads the request, prices it, starts it only within `max_price_usd`, waits up to `wait_seconds` (7 minutes by default) and returns every row | Only the rows that match |
+| `specstream_get_list_results` | `list_id` | Every row found so far, for a list that was still running | Free |
+
+When the price is above your maximum, nothing runs: the tool returns the price, so the agent can ask for fewer rows or tell the user. When a request is too vague to price ("clinics"), it returns how SpecStream read it, so the agent can add the place or the conditions.
+
+Use the classes directly when you want different settings per tool:
+
+```python
+from crewai_specstream import SpecStreamFindListTool, SpecStreamGetListResultsTool
+
+find = SpecStreamFindListTool(max_price_usd=20, wait_seconds=600)
+results = SpecStreamGetListResultsTool()
+```
+
+Both read `SPECSTREAM_API_KEY`; pass `api_key="..."` to use another. The key is stored as a secret, so it stays out of reprs and dumps, and it is sent only in the `Authorization` header.
+
+## In a crew project
+
+Give the tools to an agent in your `@CrewBase` class the way you would any other:
+
+```python
+from crewai import Agent
+from crewai.project import CrewBase, agent
+from crewai_specstream import specstream_tools
+
+@CrewBase
+class LeadCrew:
+    @agent
+    def researcher(self) -> Agent:
+        return Agent(config=self.agents_config["researcher"], tools=specstream_tools(max_price_usd=5))
+```
+
+The tools also implement CrewAI's async hook (`_arun`) and run their requests in a worker thread, so an async crew's event loop isn't blocked while a list runs.
+
+## Money, plainly
+
+- `max_price_usd` lives on the tool, in your code. The model never sees it and can't change it.
+- You pay for rows that match. Candidates SpecStream couldn't confirm are left out and aren't billed.
+- A retried step asking for the same list within the hour gets the list it already started, not a second bill. That covers CrewAI retrying a failed tool call.
+- CrewAI caches tool calls by their arguments. These tools cache a list only once it has finished, so a list that was still running is read again instead of replayed.
+- Rates and plans: [specstream.ai/pricing](https://specstream.ai/pricing).
+
+## When to reach for it
+
+When the answer is many things that meet conditions: companies in a city, clinics with a service, funding filings, open grants, tenders, repositories, people in a role. For a single fact or a summary of one page, a search tool fits better.
+
+## More
+
+- [SpecStream docs](https://specstream.ai/docs) and the [CrewAI page](https://specstream.ai/docs/crewai)
+- Agents outside CrewAI: the [MCP server](https://specstream.ai/docs/mcp) at `https://api.specstream.ai/mcp`
+- The [Python SDK](https://pypi.org/project/specstream-client/) for everything else in the API
+- Questions: support@specstream.ai
+
+MIT licensed. Made by [SpecStream](https://specstream.ai), the live web data engine for teams and agents.

crewai_specstream-0.1.0/README.md

@@ -0,0 +1,117 @@
+# crewai-specstream
+
+Most web tools for CrewAI hand your agent pages to read, and the agent builds the list itself, one search and one scrape at a time. This one hands it the finished list. The agent asks in plain words – "dental clinics in Lisbon with online booking", "US startups that filed a $5M+ Form D after October 8, 2026" – and gets back every matching row, each with the page it was read from. The list starts only within a price you set in code, and the model can't raise it.
+
+```bash
+pip install -U crewai-specstream
+export SPECSTREAM_API_KEY=ss_live_...   # Agents & API in your SpecStream workspace
+```
+
+A free account comes with $10 of work to use within 30 days and needs no card: [specstream.ai](https://specstream.ai/signup). Works with CrewAI 1.x on Python 3.10 – 3.13.
+
+## A crew that can find lists
+
+```python
+from crewai import Agent, Crew, Task
+from crewai_specstream import specstream_tools
+
+analyst = Agent(
+    role="Funding analyst",
+    goal="Answer questions about who is raising money, with a source for every claim",
+    backstory="You work from lists, not from memory. You cite the page each row came from.",
+    tools=specstream_tools(max_price_usd=5),
+)
+
+task = Task(
+    description=(
+        "Find US startups that raised $5M or more in a Form D filed after October 8, 2026. "
+        "Which industries lead? Link the filing for each company you name."
+    ),
+    expected_output="The three leading industries, with the companies and their filing links.",
+    agent=analyst,
+)
+
+print(Crew(agents=[analyst], tasks=[task]).kickoff().raw)
+```
+
+The agent writes the request and picks how many rows. You decide what one list may cost. The agent uses your default model; pass `llm="..."` to `Agent` to choose another.
+
+## What comes back
+
+One flat row per match: the list's own columns, then where the row was read. This is a real row from the request above.
+
+```json
+{
+  "list_id": "fnd_gk5y3d0r97bvyz0t08zzh75kr6",
+  "position": 26,
+  "Organization": "RapidCanvas Inc",
+  "Published address": "801 BARTON SPRINGS RD, AUSTIN, TEXAS, 78704",
+  "Source industry classification": "Other Technology",
+  "Amount sold (USD)": "28999988",
+  "Funding filing date": "2026-10-14",
+  "source_url": "https://www.sec.gov/Archives/edgar/data/2052031/000205203126000006/primary_doc.xml",
+  "checked_at": "2026-10-14T19:56:02.016693Z",
+  "list_link": "https://specstream.ai/workspace/collections/fnd_gk5y3d0r97bvyz0t08zzh75kr6"
+}
+```
+
+Columns follow the request. Ask for "breweries in Portland, Oregon that host events, with their website" and the rows carry the brewery and its website instead. A fact a source doesn't publish comes back empty, never guessed.
+
+## The two tools
+
+| Tool | The model gives | What happens | Cost |
+| --- | --- | --- | --- |
+| `specstream_find_list` | `request`, `max_results` (1 – 1000, default 25) | Reads the request, prices it, starts it only within `max_price_usd`, waits up to `wait_seconds` (7 minutes by default) and returns every row | Only the rows that match |
+| `specstream_get_list_results` | `list_id` | Every row found so far, for a list that was still running | Free |
+
+When the price is above your maximum, nothing runs: the tool returns the price, so the agent can ask for fewer rows or tell the user. When a request is too vague to price ("clinics"), it returns how SpecStream read it, so the agent can add the place or the conditions.
+
+Use the classes directly when you want different settings per tool:
+
+```python
+from crewai_specstream import SpecStreamFindListTool, SpecStreamGetListResultsTool
+
+find = SpecStreamFindListTool(max_price_usd=20, wait_seconds=600)
+results = SpecStreamGetListResultsTool()
+```
+
+Both read `SPECSTREAM_API_KEY`; pass `api_key="..."` to use another. The key is stored as a secret, so it stays out of reprs and dumps, and it is sent only in the `Authorization` header.
+
+## In a crew project
+
+Give the tools to an agent in your `@CrewBase` class the way you would any other:
+
+```python
+from crewai import Agent
+from crewai.project import CrewBase, agent
+from crewai_specstream import specstream_tools
+
+@CrewBase
+class LeadCrew:
+    @agent
+    def researcher(self) -> Agent:
+        return Agent(config=self.agents_config["researcher"], tools=specstream_tools(max_price_usd=5))
+```
+
+The tools also implement CrewAI's async hook (`_arun`) and run their requests in a worker thread, so an async crew's event loop isn't blocked while a list runs.
+
+## Money, plainly
+
+- `max_price_usd` lives on the tool, in your code. The model never sees it and can't change it.
+- You pay for rows that match. Candidates SpecStream couldn't confirm are left out and aren't billed.
+- A retried step asking for the same list within the hour gets the list it already started, not a second bill. That covers CrewAI retrying a failed tool call.
+- CrewAI caches tool calls by their arguments. These tools cache a list only once it has finished, so a list that was still running is read again instead of replayed.
+- Rates and plans: [specstream.ai/pricing](https://specstream.ai/pricing).
+
+## When to reach for it
+
+When the answer is many things that meet conditions: companies in a city, clinics with a service, funding filings, open grants, tenders, repositories, people in a role. For a single fact or a summary of one page, a search tool fits better.
+
+## More
+
+- [SpecStream docs](https://specstream.ai/docs) and the [CrewAI page](https://specstream.ai/docs/crewai)
+- Agents outside CrewAI: the [MCP server](https://specstream.ai/docs/mcp) at `https://api.specstream.ai/mcp`
+- The [Python SDK](https://pypi.org/project/specstream-client/) for everything else in the API
+- Questions: support@specstream.ai
+
+MIT licensed. Made by [SpecStream](https://specstream.ai), the live web data engine for teams and agents.

crewai_specstream-0.1.0/crewai_specstream/__init__.py

@@ -0,0 +1,7 @@
+"""SpecStream for CrewAI: ask for any list on the live web in plain words, within a price you set."""
+
+from crewai_specstream._api import SpecStreamError
+from crewai_specstream.tools import SpecStreamFindListTool, SpecStreamGetListResultsTool, specstream_tools
+
+__all__ = ["SpecStreamError", "SpecStreamFindListTool", "SpecStreamGetListResultsTool", "specstream_tools"]
+__version__ = "0.1.0"

crewai_specstream-0.1.0/crewai_specstream/_api.py

@@ -0,0 +1,206 @@
+"""The SpecStream REST calls the tools share: read a request, price it, start it within a maximum, read its rows."""
+
+from __future__ import annotations
+
+import hashlib
+import json
+import time
+import urllib.error
+import urllib.parse
+import urllib.request
+from dataclasses import dataclass
+from typing import Any, Callable
+
+DEFAULT_BASE_URL = "https://api.specstream.ai"
+USER_AGENT = "crewai-specstream/0.1.0"
+# A list runs for up to about five minutes, so the tools wait a little longer than that by default.
+DEFAULT_WAIT_SECONDS = 420.0
+FINISHED = {"completed", "partial"}
+STOPPED = {"failed", "cancelled", "canceled", "expired"}
+# Fields every row carries; a column with the same name is renamed so it can't overwrite them.
+RESERVED = {"list_id", "list_link", "position", "source_url", "checked_at"}
+
+Opener = Callable[[urllib.request.Request, float], Any]
+
+
+class SpecStreamError(RuntimeError):
+    """SpecStream refused a call or answered with something this package can't read."""
+
+
+def _dollars(cents: int) -> str:
+    return f"${cents / 100:.2f}"
+
+
+def list_link(find_id: str) -> str:
+    return f"https://specstream.ai/workspace/collections/{urllib.parse.quote(find_id, safe='')}"
+
+
+def flatten_row(find_id: str, record: dict[str, Any]) -> dict[str, Any]:
+    """One row as flat fields: each column's label holds what the list shows, plus where the row was read."""
+    row: dict[str, Any] = {"list_id": find_id, "position": record.get("ordinal")}
+    source: dict[str, Any] | None = None
+    for cell in record.get("details") or record.get("cells") or []:
+        label = str(cell.get("label") or cell.get("column_id") or "").strip()
+        if not label:
+            continue
+        if label in RESERVED:
+            label = f"{label} (column)"
+        if cell.get("absence"):
+            value: Any = ""
+        elif cell.get("display") is not None:
+            value = cell["display"]
+        else:
+            value = cell.get("value") if cell.get("value") is not None else ""
+        row[label] = value
+        provenance = cell.get("provenance") or {}
+        if source is None and str(provenance.get("source_url") or "").startswith("http"):
+            source = provenance
+    row["source_url"] = source.get("source_url", "") if source else ""
+    row["checked_at"] = source.get("observed_at", "") if source else ""
+    row["list_link"] = list_link(find_id)
+    return row
+
+
+@dataclass
+class SpecStreamAPI:
+    api_key: str
+    base_url: str = DEFAULT_BASE_URL
+    timeout: float = 30.0
+    opener: Opener | None = None
+    sleep: Callable[[float], None] = time.sleep
+    clock: Callable[[], float] = time.monotonic
+
+    def request(self, method: str, path: str, body: dict[str, Any] | None = None, *, allow_404: bool = False) -> dict[str, Any] | None:
+        key = (self.api_key or "").strip()
+        if not key:
+            raise SpecStreamError("Set SPECSTREAM_API_KEY or pass api_key. Create a key under Agents & API in your SpecStream workspace.")
+        request = urllib.request.Request(
+            self.base_url.rstrip("/") + path,
+            data=None if body is None else json.dumps(body).encode(),
+            method=method,
+            headers={
+                "Authorization": key if key.lower().startswith("bearer ") else f"Bearer {key}",
+                "Accept": "application/json",
+                "Content-Type": "application/json",
+                "User-Agent": USER_AGENT,
+            },
+        )
+        open_ = self.opener or (lambda req, timeout: urllib.request.urlopen(req, timeout=timeout))
+        try:
+            response = open_(request, self.timeout)
+            try:
+                raw = response.read()
+            finally:
+                response.close()
+        except urllib.error.HTTPError as error:
+            if allow_404 and error.code == 404:
+                return None
+            raise SpecStreamError(_error_message(error)) from None
+        except urllib.error.URLError as error:
+            raise SpecStreamError(f"Couldn't reach SpecStream: {error.reason}") from None
+        try:
+            payload = json.loads(raw or b"{}")
+        except ValueError:
+            payload = None
+        if not isinstance(payload, dict) or not isinstance(payload.get("data"), dict):
+            raise SpecStreamError("SpecStream returned a response this package couldn't read.")
+        return payload["data"]
+
+    def results(self, find_id: str) -> dict[str, Any]:
+        """Every row found so far for a list, following the pages."""
+        rows: list[dict[str, Any]] = []
+        cursor: str | None = None
+        status = "running"
+        while True:
+            query = {"limit": "100", **({"cursor": cursor} if cursor else {})}
+            page = self.request("GET", f"/v1/finds/{urllib.parse.quote(find_id, safe='')}/rows?{urllib.parse.urlencode(query)}") or {}
+            status = page.get("status") or status
+            rows.extend(flatten_row(find_id, record) for record in page.get("rows") or [])
+            cursor = page.get("next_cursor")
+            if not cursor:
+                break
+        return {"list_id": find_id, "status": status, "list_link": list_link(find_id), "row_count": len(rows), "rows": rows}
+
+    def find_list(
+        self,
+        request: str,
+        *,
+        max_results: int,
+        max_price_usd: float,
+        wait_seconds: float = DEFAULT_WAIT_SECONDS,
+        poll_interval: float = 3.0,
+        locale: str = "en-US",
+        key_prefix: str = "crewai",
+    ) -> dict[str, Any]:
+        q = (request or "").strip()
+        if not q:
+            raise ValueError("Describe the list to find.")
+        if not isinstance(max_results, int) or max_results < 1:
+            raise ValueError("max_results must be 1 or more.")
+        max_cents = round(float(max_price_usd) * 100)
+        if max_cents <= 0:
+            raise ValueError("max_price_usd must be greater than 0.")
+
+        # The same request with the same limits gets the run it already started in this hour or the one before,
+        # so a retried agent step never buys the list twice, even across the turn of the hour.
+        def key_for(at: float) -> str:
+            hour = time.strftime("%Y-%m-%dT%H", time.gmtime(at))
+            return f"{key_prefix}-" + hashlib.sha256(f"{q}|{max_results}|{max_cents}|{hour}".encode()).hexdigest()[:32]
+
+        now = time.time()
+        request_key = key_for(now)
+        existing = None
+        for candidate in (request_key, key_for(now - 3600)):
+            existing = self.request("GET", f"/v1/capacity/requests/{urllib.parse.quote(candidate, safe='')}", allow_404=True)
+            if existing and existing.get("find_id"):
+                break
+        if existing and existing.get("find_id"):
+            find_id = existing["find_id"]
+            maximum_cents = int(existing.get("maximum_cents") or max_cents)
+        else:
+            reading = self.request("POST", "/v1/parse", {"q": q, "locale": locale}) or {}
+            if reading.get("status") != "runnable":
+                read_as = reading.get("canonical_sentence") or q
+                return {
+                    "status": "needs_edit",
+                    "request": q,
+                    "message": f'SpecStream needs a clearer request before it can price it. It read: "{read_as}". Add the place or what every row must meet.',
+                }
+            quote = self.request(
+                "POST", "/v1/capacity/quotes", {"ast_hash": reading["ast_hash"], "operation": "find", "requested_units": max_results}
+            ) or {}
+            maximum_cents = int(quote.get("maximum_cents") or 0)
+            if maximum_cents > max_cents:
+                return {
+                    "status": "over_maximum",
+                    "request": q,
+                    "maximum_price_usd": maximum_cents / 100,
+                    "message": f"Nothing ran: this list can cost up to {_dollars(maximum_cents)}, above the maximum of {_dollars(max_cents)}. Ask for fewer results or raise max_price_usd.",
+                }
+            run = self.request(
+                "POST",
+                "/v1/capacity/executions",
+                {"quote_id": quote["quote_id"], "quote_hash": quote["quote_hash"], "confirm_max_cents": maximum_cents, "idempotency_key": request_key},
+            ) or {}
+            find_id = run["find_id"]
+
+        deadline = self.clock() + max(0.0, wait_seconds)
+        while True:
+            result = self.results(find_id)
+            if result["status"] in FINISHED | STOPPED or self.clock() >= deadline:
+                break
+            self.sleep(poll_interval)
+        result.update({"request": q, "maximum_price_usd": maximum_cents / 100})
+        if result["status"] not in FINISHED | STOPPED:
+            result["message"] = "Still running. Call specstream_get_list_results with this list_id for the rest."
+        return result
+
+
+def _error_message(error: urllib.error.HTTPError) -> str:
+    try:
+        payload = json.loads(error.read() or b"{}")
+    except ValueError:
+        payload = {}
+    detail = payload.get("error") if isinstance(payload, dict) else None
+    message = detail.get("message") if isinstance(detail, dict) else detail if isinstance(detail, str) else None
+    return f"SpecStream answered {error.code}: {message or error.reason}"

crewai_specstream-0.1.0/crewai_specstream/tools.py

@@ -0,0 +1,162 @@
+"""CrewAI tools that ask SpecStream for a list on the live web and read its rows."""
+
+import asyncio
+import json
+import os
+from typing import Any
+
+from crewai.tools import BaseTool, EnvVar
+from pydantic import BaseModel, Field, SecretStr, field_validator
+
+from crewai_specstream._api import DEFAULT_BASE_URL, DEFAULT_WAIT_SECONDS, FINISHED, STOPPED, SpecStreamAPI
+
+KEY_HELP = "Set SPECSTREAM_API_KEY or pass api_key. Create a key under Agents & API in your SpecStream workspace."
+MAX_RESULTS = 1000
+
+
+class FindListInput(BaseModel):
+    """What the model fills in. The price ceiling is not here: the developer sets it on the tool."""
+
+    request: str = Field(
+        ...,
+        description=(
+            "The list in plain words: what to find, where, what every row must meet, and the facts to bring back. "
+            'Example: "Dental clinics in Lisbon with online booking, with their website and phone".'
+        ),
+    )
+    max_results: int = Field(
+        default=25,
+        ge=1,
+        le=MAX_RESULTS,
+        description="How many rows to find, from 1 to 1000. Ask for what the task needs; the price grows with the matches.",
+    )
+
+
+class GetListResultsInput(BaseModel):
+    list_id: str = Field(
+        ...,
+        description='The list_id that specstream_find_list returned, like "fnd_gk5y3d0r97bvyz0t08zzh75kr6".',
+    )
+
+
+def _key_from_env() -> SecretStr:
+    key = os.environ.get("SPECSTREAM_API_KEY", "").strip()
+    if not key:
+        raise ValueError(KEY_HELP)
+    return SecretStr(key)
+
+
+def _cache_finished(_args: Any = None, result: Any = None) -> bool:
+    """Cache a list only once it is done; a running, vague or over-the-maximum answer should be asked again."""
+    try:
+        return json.loads(result).get("status") in FINISHED | STOPPED
+    except (TypeError, ValueError, AttributeError):
+        return False
+
+
+def _never_cache(_args: Any = None, _result: Any = None) -> bool:
+    return False
+
+
+class _SpecStreamTool(BaseTool):
+    api_key: SecretStr = Field(default_factory=_key_from_env, repr=False)
+    """A SpecStream API key. Read from SPECSTREAM_API_KEY when not passed."""
+    base_url: str = DEFAULT_BASE_URL
+    env_vars: list[EnvVar] = Field(
+        default_factory=lambda: [
+            EnvVar(name="SPECSTREAM_API_KEY", description="SpecStream API key (Agents & API in your workspace)", required=True)
+        ]
+    )
+
+    @field_validator("api_key", mode="after")
+    @classmethod
+    def _key_is_not_empty(cls, value: SecretStr) -> SecretStr:
+        if not value.get_secret_value().strip():
+            raise ValueError(KEY_HELP)
+        return value
+
+    def _api(self) -> SpecStreamAPI:
+        return SpecStreamAPI(self.api_key.get_secret_value(), self.base_url)
+
+
+class SpecStreamFindListTool(_SpecStreamTool):
+    """Ask for a list in plain words and get every matching row, each with the page it was read from.
+
+    The list starts only if its price is within ``max_price_usd``, which you set here, in code. The model
+    chooses the request and how many rows; it can't raise the price ceiling.
+
+    Example:
+        .. code-block:: python
+
+            from crewai_specstream import SpecStreamFindListTool
+
+            tool = SpecStreamFindListTool(max_price_usd=5)
+    """
+
+    name: str = "specstream_find_list"
+    description: str = (
+        "Find a list of real things on the live web from a plain-English request and get back every matching row, "
+        "each with the page it was read from. Use it when the answer is many things that meet conditions: companies "
+        "in a city, local businesses with a service, funding filings, open grants, tenders, clinical trials, "
+        "repositories, people in a role. For one fact or a summary of one page, use a search tool instead. Rows that "
+        "couldn't be confirmed are left out and aren't charged. "
+        "Returns JSON with status, list_id, list_link and rows (one object per match: its columns, source_url and "
+        'checked_at). status "over_maximum" means nothing ran because the price was above the cap: ask for fewer rows '
+        'or tell the user. status "needs_edit" means the request was too vague to price: add the place or the '
+        "conditions and call again. If the list is still running, call specstream_get_list_results with its list_id "
+        "for the rest."
+    )
+    args_schema: type[BaseModel] = FindListInput
+
+    max_price_usd: float = Field(gt=0)
+    """The most one list may cost, in US dollars. If the price is higher, nothing runs."""
+    wait_seconds: float = DEFAULT_WAIT_SECONDS
+    """How long to wait for the list to finish before returning the rows found so far."""
+    poll_interval: float = 3.0
+    locale: str = "en-US"
+
+    def __init__(self, **kwargs: Any) -> None:
+        kwargs.setdefault("cache_function", _cache_finished)
+        super().__init__(**kwargs)
+
+    def _run(self, request: str, max_results: int = 25) -> str:
+        result = self._api().find_list(
+            request,
+            max_results=max(1, min(int(max_results), MAX_RESULTS)),
+            max_price_usd=self.max_price_usd,
+            wait_seconds=self.wait_seconds,
+            poll_interval=self.poll_interval,
+            locale=self.locale,
+        )
+        return json.dumps(result, ensure_ascii=False, default=str)
+
+    async def _arun(self, request: str, max_results: int = 25) -> str:
+        return await asyncio.to_thread(self._run, request, max_results)
+
+
+class SpecStreamGetListResultsTool(_SpecStreamTool):
+    """Read every row found so far for a list. Reading a list you already have costs nothing."""
+
+    name: str = "specstream_get_list_results"
+    description: str = (
+        "Read every row found so far for a SpecStream list, such as one that was still running. Use it with the "
+        "list_id from specstream_find_list. Reading is free. Returns JSON with status, list_id, list_link and rows, "
+        "each with its columns, source_url and checked_at."
+    )
+    args_schema: type[BaseModel] = GetListResultsInput
+
+    def __init__(self, **kwargs: Any) -> None:
+        kwargs.setdefault("cache_function", _never_cache)
+        super().__init__(**kwargs)
+
+    def _run(self, list_id: str) -> str:
+        return json.dumps(self._api().results(list_id.strip()), ensure_ascii=False, default=str)
+
+    async def _arun(self, list_id: str) -> str:
+        return await asyncio.to_thread(self._run, list_id)
+
+
+def specstream_tools(max_price_usd: float, **kwargs: Any) -> list[BaseTool]:
+    """Both tools, sharing one key and base URL: the usual way to give an agent SpecStream."""
+    shared = {key: kwargs.pop(key) for key in ("api_key", "base_url") if key in kwargs}
+    return [SpecStreamFindListTool(max_price_usd=max_price_usd, **shared, **kwargs), SpecStreamGetListResultsTool(**shared)]

crewai_specstream-0.1.0/pyproject.toml

@@ -0,0 +1,49 @@
+[project]
+name = "crewai-specstream"
+version = "0.1.0"
+description = "Give a CrewAI agent any list from the live web: ask in plain words, get every matching row with the page it came from, within a price you set in code."
+readme = "README.md"
+requires-python = ">=3.10,<3.14"
+license = "MIT"
+license-files = ["LICENSE"]
+authors = [{ name = "SpecStream", email = "support@specstream.ai" }]
+maintainers = [{ name = "SpecStream", email = "support@specstream.ai" }]
+keywords = ["crewai", "crewai-tools", "specstream", "agents", "tools", "web-data", "live-web", "lead-lists", "research"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Intended Audience :: Developers",
+  "Operating System :: OS Independent",
+  "Programming Language :: Python :: 3",
+  "Programming Language :: Python :: 3 :: Only",
+  "Programming Language :: Python :: 3.10",
+  "Programming Language :: Python :: 3.11",
+  "Programming Language :: Python :: 3.12",
+  "Programming Language :: Python :: 3.13",
+  "Topic :: Scientific/Engineering :: Artificial Intelligence",
+  "Topic :: Internet :: WWW/HTTP",
+  "Typing :: Typed",
+]
+dependencies = ["crewai>=1.0.0,<2.0.0"]
+
+[project.optional-dependencies]
+test = ["pytest>=8"]
+
+[project.urls]
+Homepage = "https://specstream.ai"
+Documentation = "https://specstream.ai/docs/crewai"
+Repository = "https://github.com/Specstreamai/specstream-tools/tree/main/packages/crewai-specstream"
+Issues = "https://github.com/Specstreamai/specstream-tools/issues"
+Pricing = "https://specstream.ai/pricing"
+
+[build-system]
+requires = ["hatchling>=1.27,<2"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["crewai_specstream"]
+
+[tool.hatch.build.targets.sdist]
+include = ["crewai_specstream", "tests", "README.md", "LICENSE"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

crewai_specstream-0.1.0/tests/test_crewai_specstream.py

@@ -0,0 +1,342 @@
+import asyncio
+import io
+import json
+import urllib.error
+import urllib.parse
+
+import pytest
+from pydantic import ValidationError
+
+from crewai_specstream import (
+    SpecStreamError,
+    SpecStreamFindListTool,
+    SpecStreamGetListResultsTool,
+    specstream_tools,
+)
+
+KEY = "ss_live_test_key"
+
+
+def cell(label, value, source=None):
+    provenance = {"source_url": source, "observed_at": "2026-10-14T08:00:00Z"} if source else None
+    return {"label": label, "value": value, "display": value, "absence": None, "provenance": provenance}
+
+
+ROWS = [
+    {"ordinal": 1, "cells": [cell("Organization", "Clínica Sorriso", "https://sorriso.pt/marcacoes"), cell("Website", "https://sorriso.pt")]},
+    {"ordinal": 2, "cells": [cell("Organization", "Dentes de Lisboa", "https://dentesdelisboa.pt"), cell("Website", None) | {"absence": "Not published"}]},
+]
+
+
+class FakeSpecStream:
+    """Answers like the SpecStream API: the request, its quote, the run, then rows over two polls and two pages."""
+
+    def __init__(self, maximum_cents=250, runnable=True, existing=None):
+        self.maximum_cents = maximum_cents
+        self.runnable = runnable
+        self.existing = existing
+        self.calls = []
+        self.polls = 0
+        self.misses = 0
+
+    def __call__(self, request, timeout=None):
+        url = urllib.parse.urlsplit(request.full_url)
+        body = json.loads(request.data) if request.data else None
+        self.calls.append((request.get_method(), url.path, body, dict(request.header_items())))
+        query = dict(urllib.parse.parse_qsl(url.query))
+        if url.path.startswith("/v1/capacity/requests/"):
+            if self.misses:
+                self.misses -= 1
+                raise urllib.error.HTTPError(request.full_url, 404, "Not Found", {}, io.BytesIO(b"{}"))
+            if self.existing:
+                return self.reply(self.existing)
+            raise urllib.error.HTTPError(request.full_url, 404, "Not Found", {}, io.BytesIO(b"{}"))
+        if url.path == "/v1/parse":
+            return self.reply({"status": "runnable" if self.runnable else "needs_input", "ast_hash": "a" * 64, "canonical_sentence": "25 dental clinics in Lisbon"})
+        if url.path == "/v1/capacity/quotes":
+            return self.reply({"quote_id": "cqt_1", "quote_hash": "b" * 64, "maximum_cents": self.maximum_cents})
+        if url.path == "/v1/capacity/executions":
+            return self.reply({"find_id": "fnd_1", "state": "running"})
+        if url.path == "/v1/finds/fnd_1/rows":
+            if query.get("cursor") == "page-2":
+                return self.reply({"status": "completed", "rows": ROWS[1:], "next_cursor": None})
+            self.polls += 1
+            if self.polls == 1:
+                return self.reply({"status": "running", "rows": [], "next_cursor": None})
+            return self.reply({"status": "completed", "rows": ROWS[:1], "next_cursor": "page-2"})
+        raise AssertionError(f"unexpected call {request.get_method()} {url.path}")
+
+    @staticmethod
+    def reply(data):
+        return io.BytesIO(json.dumps({"data": data}).encode())
+
+    def paths(self):
+        return [path for _, path, _, _ in self.calls]
+
+
+@pytest.fixture
+def api(monkeypatch):
+    def install(**kwargs):
+        fake = FakeSpecStream(**kwargs)
+        monkeypatch.setattr("urllib.request.urlopen", fake)
+        return fake
+
+    return install
+
+
+def test_finds_a_list_within_the_maximum_and_returns_every_row(api):
+    fake = api()
+    tool = SpecStreamFindListTool(api_key=KEY, max_price_usd=5, poll_interval=0)
+    result = json.loads(tool.run(request="Dental clinics in Lisbon with online booking", max_results=25))
+
+    assert result["status"] == "completed"
+    assert result["row_count"] == 2
+    assert result["list_link"] == "https://specstream.ai/workspace/collections/fnd_1"
+    first, second = result["rows"]
+    assert first["Organization"] == "Clínica Sorriso"
+    assert first["source_url"] == "https://sorriso.pt/marcacoes"
+    assert first["checked_at"] == "2026-10-14T08:00:00Z"
+    assert second["Website"] == ""
+    run = next(body for method, path, body, _ in fake.calls if path == "/v1/capacity/executions")
+    assert run["confirm_max_cents"] == 250
+    assert run["idempotency_key"].startswith("crewai-")
+
+
+def test_the_result_is_a_json_string_the_agent_can_read(api):
+    api()
+    output = SpecStreamFindListTool(api_key=KEY, max_price_usd=5, poll_interval=0).run(request="Dental clinics in Lisbon")
+
+    assert isinstance(output, str)
+    assert "Clínica Sorriso" in output
+
+
+def test_nothing_runs_when_the_price_is_above_the_maximum(api):
+    fake = api(maximum_cents=900)
+    result = json.loads(SpecStreamFindListTool(api_key=KEY, max_price_usd=5).run(request="Dental clinics in Lisbon"))
+
+    assert result["status"] == "over_maximum"
+    assert "$9.00" in result["message"] and "$5.00" in result["message"]
+    assert "/v1/capacity/executions" not in fake.paths()
+
+
+def test_an_unclear_request_comes_back_with_how_it_was_read(api):
+    fake = api(runnable=False)
+    result = json.loads(SpecStreamFindListTool(api_key=KEY, max_price_usd=5).run(request="clinics"))
+
+    assert result["status"] == "needs_edit"
+    assert "25 dental clinics in Lisbon" in result["message"]
+    assert "/v1/capacity/quotes" not in fake.paths()
+
+
+def test_the_model_cannot_set_the_price():
+    tool = SpecStreamFindListTool(api_key=KEY, max_price_usd=2)
+    assert set(tool.args_schema.model_fields) == {"request", "max_results"}
+    assert "max_price_usd" not in json.dumps(tool.args_schema.model_json_schema())
+    assert "max_price_usd" not in tool.description
+
+
+def test_the_price_cap_is_required_and_must_be_positive():
+    with pytest.raises(ValidationError):
+        SpecStreamFindListTool(api_key=KEY)
+    with pytest.raises(ValidationError):
+        SpecStreamFindListTool(api_key=KEY, max_price_usd=0)
+
+
+def test_max_results_is_clamped_to_1_through_1000(api):
+    fake = api()
+    tool = SpecStreamFindListTool(api_key=KEY, max_price_usd=5, poll_interval=0)
+    units = []
+    for asked in (5000, 0, -3):
+        tool._run("Dental clinics in Lisbon", asked)
+        quote = [body for _, path, body, _ in fake.calls if path == "/v1/capacity/quotes"][-1]
+        units.append(quote["requested_units"])
+        fake.polls = 0
+
+    assert units == [1000, 1, 1]
+
+
+def test_the_schema_shows_the_model_the_same_bounds():
+    field = SpecStreamFindListTool(api_key=KEY, max_price_usd=5).args_schema.model_json_schema()["properties"]["max_results"]
+    assert field["minimum"] == 1 and field["maximum"] == 1000 and field["default"] == 25
+
+
+def test_a_retried_step_reuses_the_run_it_already_started(api):
+    fake = api(existing={"find_id": "fnd_1", "maximum_cents": 250})
+    result = json.loads(SpecStreamFindListTool(api_key=KEY, max_price_usd=5, poll_interval=0).run(request="Dental clinics in Lisbon"))
+
+    assert result["row_count"] == 2
+    assert "/v1/parse" not in fake.paths() and "/v1/capacity/executions" not in fake.paths()
+
+
+def test_a_retry_after_the_turn_of_the_hour_reuses_the_run_from_the_hour_before(api):
+    fake = api(existing={"find_id": "fnd_1", "maximum_cents": 250})
+    fake.misses = 1
+    result = json.loads(SpecStreamFindListTool(api_key=KEY, max_price_usd=5, poll_interval=0).run(request="Dental clinics in Lisbon"))
+
+    lookups = [path for path in fake.paths() if path.startswith("/v1/capacity/requests/")]
+    assert len(lookups) == 2 and lookups[0] != lookups[1]
+    assert result["row_count"] == 2
+    assert "/v1/capacity/executions" not in fake.paths()
+
+
+def test_a_list_still_running_points_to_the_free_read(api):
+    api()
+    result = json.loads(SpecStreamFindListTool(api_key=KEY, max_price_usd=5, wait_seconds=0).run(request="Dental clinics in Lisbon"))
+
+    assert result["status"] == "running"
+    assert "specstream_get_list_results" in result["message"]
+
+
+def test_reading_results_follows_every_page(api):
+    fake = api()
+    fake.polls = 1
+    result = json.loads(SpecStreamGetListResultsTool(api_key=KEY).run(list_id="fnd_1"))
+
+    assert [row["position"] for row in result["rows"]] == [1, 2]
+    assert result["row_count"] == 2
+    pages = [path for path in fake.paths() if path == "/v1/finds/fnd_1/rows"]
+    assert len(pages) == 2
+
+
+def test_reading_results_costs_nothing(api):
+    fake = api()
+    fake.polls = 1
+    SpecStreamGetListResultsTool(api_key=KEY).run(list_id="fnd_1")
+
+    assert all(method == "GET" for method, _, _, _ in fake.calls)
+
+
+def test_the_key_goes_in_the_authorization_header_and_never_in_the_result(api):
+    fake = api()
+    tool = SpecStreamFindListTool(api_key=KEY, max_price_usd=5, poll_interval=0)
+    output = tool.run(request="Dental clinics in Lisbon")
+
+    headers = fake.calls[0][3]
+    assert headers["Authorization"] == f"Bearer {KEY}"
+    assert headers["User-agent"].startswith("crewai-specstream/")
+    assert KEY not in output
+    assert KEY not in repr(tool) and KEY not in str(tool)
+    assert KEY not in str(tool.model_dump(exclude={"args_schema", "cache_function"}))
+
+
+def test_the_key_is_read_from_the_environment(api, monkeypatch):
+    fake = api()
+    monkeypatch.setenv("SPECSTREAM_API_KEY", "ss_live_from_env")
+    SpecStreamGetListResultsTool().run(list_id="fnd_1")
+
+    assert fake.calls[0][3]["Authorization"] == "Bearer ss_live_from_env"
+
+
+def test_a_missing_key_says_where_to_get_one(monkeypatch):
+    monkeypatch.delenv("SPECSTREAM_API_KEY", raising=False)
+    with pytest.raises(ValueError, match="SPECSTREAM_API_KEY"):
+        SpecStreamFindListTool(max_price_usd=5)
+    with pytest.raises(ValueError, match="Agents & API"):
+        SpecStreamGetListResultsTool()
+    with pytest.raises(ValueError, match="SPECSTREAM_API_KEY"):
+        SpecStreamGetListResultsTool(api_key="  ")
+
+
+def test_a_refused_call_raises_with_the_reason(api, monkeypatch):
+    def refuse(request, timeout=None):
+        raise urllib.error.HTTPError(request.full_url, 401, "Unauthorized", {}, io.BytesIO(b'{"error": {"message": "Bad key"}}'))
+
+    monkeypatch.setattr("urllib.request.urlopen", refuse)
+    with pytest.raises(SpecStreamError, match="401: Bad key"):
+        SpecStreamGetListResultsTool(api_key=KEY).run(list_id="fnd_1")
+
+
+def test_both_tools_share_the_key_and_the_cap(api):
+    find, results = specstream_tools(max_price_usd=5, api_key=KEY, wait_seconds=60)
+    assert find.name == "specstream_find_list" and results.name == "specstream_get_list_results"
+    assert find.max_price_usd == 5 and find.wait_seconds == 60
+    assert results.api_key.get_secret_value() == KEY
+
+
+def test_the_tools_declare_their_environment_variable():
+    for tool in specstream_tools(max_price_usd=5, api_key=KEY):
+        assert [(var.name, var.required) for var in tool.env_vars] == [("SPECSTREAM_API_KEY", True)]
+
+
+def test_an_agent_calls_the_tool_through_its_structured_form(api):
+    api()
+    find, _ = specstream_tools(max_price_usd=5, api_key=KEY, poll_interval=0)
+    structured = find.to_structured_tool()
+    result = json.loads(structured.invoke({"request": "Dental clinics in Lisbon", "max_results": 10}))
+
+    assert structured.name == "specstream_find_list"
+    assert result["row_count"] == 2
+
+
+def test_the_tools_run_without_blocking_an_event_loop(api):
+    api()
+    find, results = specstream_tools(max_price_usd=5, api_key=KEY, poll_interval=0)
+
+    async def go():
+        found = json.loads(await find._arun("Dental clinics in Lisbon", 25))
+        again = json.loads(await results._arun("fnd_1"))
+        return found, again
+
+    found, again = asyncio.run(go())
+    assert found["row_count"] == 2 and again["row_count"] == 2
+
+
+def test_only_a_finished_list_is_cached():
+    find = SpecStreamFindListTool(api_key=KEY, max_price_usd=5)
+    done = json.dumps({"status": "completed"})
+    for waiting in (json.dumps({"status": "running"}), json.dumps({"status": "over_maximum"}), json.dumps({"status": "needs_edit"}), "not json"):
+        assert find.cache_function({}, waiting) is False
+    assert find.cache_function({}, done) is True
+    assert SpecStreamGetListResultsTool(api_key=KEY).cache_function({}, done) is False
+
+
+def test_an_agent_accepts_both_tools():
+    from crewai import LLM, Agent
+
+    agent = Agent(
+        role="Researcher",
+        goal="Find lists",
+        backstory="Finds lists.",
+        tools=specstream_tools(max_price_usd=5, api_key=KEY),
+        llm=LLM(model="gpt-4o-mini", api_key="sk-test"),
+    )
+    assert [tool.name for tool in agent.tools] == ["specstream_find_list", "specstream_get_list_results"]
+
+
+SCRIPTED = []
+SEEN = []
+
+
+def test_a_crew_finds_a_list_end_to_end_with_a_scripted_model(api):
+    """A real Agent, Task and Crew run the agent loop; only the model's replies are scripted."""
+    from crewai import Agent, BaseLLM, Crew, Task
+
+    class Scripted(BaseLLM):
+        def call(self, messages, tools=None, callbacks=None, available_functions=None, **kwargs):
+            SEEN.append(messages)
+            return SCRIPTED.pop(0)
+
+        def supports_function_calling(self):
+            return False
+
+        def supports_stop_words(self):
+            return True
+
+        def get_context_window_size(self):
+            return 8000
+
+    fake = api()
+    SEEN.clear()
+    SCRIPTED[:] = [
+        'Thought: I need the list.\nAction: specstream_find_list\nAction Input: {"request": "Dental clinics in Lisbon", "max_results": 10}',
+        "Thought: I have the rows.\nFinal Answer: Found 2 clinics.",
+    ]
+    find, results = specstream_tools(max_price_usd=5, api_key=KEY, poll_interval=0)
+    agent = Agent(role="Researcher", goal="Find lists", backstory="Finds lists.", tools=[find, results], llm=Scripted(model="scripted"))
+    task = Task(description="Find dental clinics in Lisbon.", expected_output="A short answer.", agent=agent)
+    output = Crew(agents=[agent], tasks=[task], verbose=False).kickoff()
+
+    assert output.raw == "Found 2 clinics."
+    assert "Clínica Sorriso" in json.dumps(SEEN[-1], ensure_ascii=False)
+    assert KEY not in json.dumps(SEEN, ensure_ascii=False)
+    assert "/v1/capacity/executions" in fake.paths()