crewai-webmetadata-extractor 0.1.0__tar.gz

crewai_webmetadata_extractor-0.1.0/.gitignore

@@ -0,0 +1,6 @@
+.venv/
+dist/
+build/
+*.egg-info/
+__pycache__/
+.pytest_cache/

crewai_webmetadata_extractor-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 JosejuX
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

crewai_webmetadata_extractor-0.1.0/PKG-INFO

@@ -0,0 +1,81 @@
+Metadata-Version: 2.5
+Name: crewai-webmetadata-extractor
+Version: 0.1.0
+Summary: CrewAI tools for the Web Metadata, OpenGraph & Contact Extractor API — give your crew URL-to-structured-data extraction, AI-ready Markdown, contact discovery, and SEO auditing.
+Project-URL: Homepage, https://github.com/JosejuX/rapidapi-metadata-extractor
+Project-URL: RapidAPI Listing, https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor
+Project-URL: PyPI SDK, https://pypi.org/project/webmetadata-extractor/
+Author-email: Juanjo <josejuanjo.coding@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agent,crewai,crewai-tool,metadata-extractor,rag,seo,web-scraping
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.10
+Requires-Dist: crewai>=0.80.0
+Requires-Dist: pydantic>=2
+Requires-Dist: webmetadata-extractor>=0.1.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == 'dev'
+Requires-Dist: responses>=0.23; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# crewai-webmetadata-extractor
+
+CrewAI tools for the [Web Metadata, OpenGraph & Contact Extractor API](https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor) — give a crew the ability to turn any URL into structured web intelligence, clean Markdown for RAG, contact signals for lead-gen, or a graded SEO audit.
+
+```bash
+pip install crewai-webmetadata-extractor
+```
+
+## Quick start
+
+Get a free API key on [RapidAPI](https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor) (1,000 requests/month, no credit card required).
+
+```python
+from crewai import Agent
+from crewai_webmetadata_extractor import get_tools
+
+tools = get_tools(api_key="YOUR_RAPIDAPI_KEY")
+# or leave api_key out and set the WEBMETADATA_API_KEY environment variable instead
+
+researcher = Agent(
+    role="Web Researcher",
+    goal="Extract structured intelligence from URLs",
+    tools=tools,
+    ...
+)
+```
+
+## Tools included
+
+| Tool | Use it when the agent needs to... |
+|---|---|
+| `WebMetadataExtractTool` | Get a broad, all-in-one picture of a page (SEO, contacts, tech stack, security, links) |
+| `WebMarkdownTool` | Ingest a page's actual content into a RAG pipeline as clean Markdown |
+| `WebContactsTool` | Find public emails/phones/social links for outreach or lead-gen workflows |
+| `WebSEOAuditTool` | Grade a page's SEO with a score, warnings, and readability metrics |
+
+Every tool returns a JSON string — API errors come back as `{"error": true, "status_code": ..., "detail": ...}` instead of raising, so a crew can react to them without crashing.
+
+## Configuration
+
+Pass `api_key` explicitly per tool, or set it once via the `WEBMETADATA_API_KEY` environment variable and omit it everywhere:
+
+```bash
+export WEBMETADATA_API_KEY=your-rapidapi-key
+```
+
+## Links
+
+- [Python SDK this wraps](https://pypi.org/project/webmetadata-extractor/) (`webmetadata-extractor`)
+- [Also available for LangChain](https://pypi.org/project/langchain-webmetadata-extractor/) (`langchain-webmetadata-extractor`)
+- [Full API documentation](https://github.com/JosejuX/rapidapi-metadata-extractor)
+- [Live interactive demo](https://rapidapi-metadata-extractor.onrender.com) — no signup needed
+
+## License
+
+MIT

crewai_webmetadata_extractor-0.1.0/README.md

@@ -0,0 +1,56 @@
+# crewai-webmetadata-extractor
+
+CrewAI tools for the [Web Metadata, OpenGraph & Contact Extractor API](https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor) — give a crew the ability to turn any URL into structured web intelligence, clean Markdown for RAG, contact signals for lead-gen, or a graded SEO audit.
+
+```bash
+pip install crewai-webmetadata-extractor
+```
+
+## Quick start
+
+Get a free API key on [RapidAPI](https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor) (1,000 requests/month, no credit card required).
+
+```python
+from crewai import Agent
+from crewai_webmetadata_extractor import get_tools
+
+tools = get_tools(api_key="YOUR_RAPIDAPI_KEY")
+# or leave api_key out and set the WEBMETADATA_API_KEY environment variable instead
+
+researcher = Agent(
+    role="Web Researcher",
+    goal="Extract structured intelligence from URLs",
+    tools=tools,
+    ...
+)
+```
+
+## Tools included
+
+| Tool | Use it when the agent needs to... |
+|---|---|
+| `WebMetadataExtractTool` | Get a broad, all-in-one picture of a page (SEO, contacts, tech stack, security, links) |
+| `WebMarkdownTool` | Ingest a page's actual content into a RAG pipeline as clean Markdown |
+| `WebContactsTool` | Find public emails/phones/social links for outreach or lead-gen workflows |
+| `WebSEOAuditTool` | Grade a page's SEO with a score, warnings, and readability metrics |
+
+Every tool returns a JSON string — API errors come back as `{"error": true, "status_code": ..., "detail": ...}` instead of raising, so a crew can react to them without crashing.
+
+## Configuration
+
+Pass `api_key` explicitly per tool, or set it once via the `WEBMETADATA_API_KEY` environment variable and omit it everywhere:
+
+```bash
+export WEBMETADATA_API_KEY=your-rapidapi-key
+```
+
+## Links
+
+- [Python SDK this wraps](https://pypi.org/project/webmetadata-extractor/) (`webmetadata-extractor`)
+- [Also available for LangChain](https://pypi.org/project/langchain-webmetadata-extractor/) (`langchain-webmetadata-extractor`)
+- [Full API documentation](https://github.com/JosejuX/rapidapi-metadata-extractor)
+- [Live interactive demo](https://rapidapi-metadata-extractor.onrender.com) — no signup needed
+
+## License
+
+MIT

crewai_webmetadata_extractor-0.1.0/pyproject.toml

@@ -0,0 +1,36 @@
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project]
+name = "crewai-webmetadata-extractor"
+version = "0.1.0"
+description = "CrewAI tools for the Web Metadata, OpenGraph & Contact Extractor API — give your crew URL-to-structured-data extraction, AI-ready Markdown, contact discovery, and SEO auditing."
+readme = "README.md"
+license = "MIT"
+authors = [{ name = "Juanjo", email = "josejuanjo.coding@gmail.com" }]
+requires-python = ">=3.10"
+keywords = ["crewai", "crewai-tool", "agent", "rag", "web-scraping", "seo", "metadata-extractor"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Topic :: Software Development :: Libraries :: Python Modules",
+]
+dependencies = [
+    "webmetadata-extractor>=0.1.0",
+    "crewai>=0.80.0",
+    "pydantic>=2",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=7.0", "responses>=0.23"]
+
+[project.urls]
+Homepage = "https://github.com/JosejuX/rapidapi-metadata-extractor"
+"RapidAPI Listing" = "https://rapidapi.com/josejuanjocoding/api/web-metadata-and-contact-extractor"
+"PyPI SDK" = "https://pypi.org/project/webmetadata-extractor/"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/crewai_webmetadata_extractor"]

crewai_webmetadata_extractor-0.1.0/src/crewai_webmetadata_extractor/__init__.py

@@ -0,0 +1,11 @@
+from .tools import WebContactsTool, WebMarkdownTool, WebMetadataExtractTool, WebSEOAuditTool, get_tools
+
+__all__ = [
+    "WebMetadataExtractTool",
+    "WebMarkdownTool",
+    "WebContactsTool",
+    "WebSEOAuditTool",
+    "get_tools",
+]
+
+__version__ = "0.1.0"

crewai_webmetadata_extractor-0.1.0/src/crewai_webmetadata_extractor/tools.py

@@ -0,0 +1,102 @@
+from __future__ import annotations
+
+import json
+import os
+from typing import Optional, Type
+
+from crewai.tools import BaseTool
+from pydantic import BaseModel, ConfigDict, Field
+
+from webmetadata_extractor import WebMetadataClient
+from webmetadata_extractor.exceptions import WebMetadataAPIError
+
+
+class UrlInput(BaseModel):
+    """Input schema shared by every tool in this package."""
+
+    url: str = Field(description="The fully-qualified URL to fetch and analyze, e.g. https://example.com")
+
+
+def _dumps(result: dict) -> str:
+    return json.dumps(result, ensure_ascii=False)
+
+
+class _BaseWebMetadataTool(BaseTool):
+    """Shared config and dispatch logic. Not meant to be used directly.
+
+    Reads the RapidAPI key from the `api_key` constructor argument, falling back
+    to the `WEBMETADATA_API_KEY` environment variable if not given.
+    """
+
+    model_config = ConfigDict(arbitrary_types_allowed=True)
+    args_schema: Type[BaseModel] = UrlInput
+    client_method: str = ""
+    api_key: Optional[str] = Field(default=None, exclude=True)
+
+    def _run(self, url: str) -> str:
+        client = WebMetadataClient(api_key=self.api_key or os.environ.get("WEBMETADATA_API_KEY"))
+        try:
+            return _dumps(getattr(client, self.client_method)(url))
+        except WebMetadataAPIError as exc:
+            return _dumps({"error": True, "status_code": exc.status_code, "detail": exc.body})
+        finally:
+            client.close()
+
+
+class WebMetadataExtractTool(_BaseWebMetadataTool):
+    """Full extraction: SEO/OpenGraph metadata, contacts, tech stack, schema.org, security, links."""
+
+    name: str = "web_metadata_extract"
+    description: str = (
+        "Extract the full structured web intelligence payload for a URL: SEO/OpenGraph metadata, "
+        "public contact info (emails/phones/social links), detected tech stack, Schema.org product data, "
+        "a graded security-headers audit, and internal/external links. Use this when you need a broad "
+        "picture of a page rather than one specific facet. Input is a single URL."
+    )
+    client_method: str = "extract"
+
+
+class WebMarkdownTool(_BaseWebMetadataTool):
+    """Clean, AI/LLM-ready Markdown of a page's main content — for RAG ingestion pipelines."""
+
+    name: str = "web_markdown_reader"
+    description: str = (
+        "Fetch a URL and return its main article content as clean, LLM-ready Markdown (boilerplate, nav, "
+        "and ads stripped). Use this to ingest web pages into a RAG pipeline or to let an agent read a "
+        "page's actual content. Input is a single URL."
+    )
+    client_method: str = "markdown"
+
+
+class WebContactsTool(_BaseWebMetadataTool):
+    """Public contact discovery (emails, phone numbers, social links) — a raw signal for lead-gen crews."""
+
+    name: str = "web_contacts_extract"
+    description: str = (
+        "Find public contact signals on a URL: email addresses, phone numbers, and social media profile "
+        "links present in the page's HTML. This is a raw signal for a lead-generation or outreach workflow, "
+        "not verified company/people intelligence. Input is a single URL."
+    )
+    client_method: str = "contacts"
+
+
+class WebSEOAuditTool(_BaseWebMetadataTool):
+    """14-point automated SEO diagnostic with a warnings list and readability metrics."""
+
+    name: str = "web_seo_audit"
+    description: str = (
+        "Run a 14-point automated SEO audit on a URL: overall score, a list of warnings with severity and "
+        "evidence, and readability metrics (heading structure, sentence/paragraph counts). Use this when "
+        "asked to review, grade, or diagnose a page's SEO. Input is a single URL."
+    )
+    client_method: str = "seo_audit"
+
+
+def get_tools(api_key: Optional[str] = None) -> list[BaseTool]:
+    """Convenience helper: instantiate all four tools with a shared api_key, ready to add to an Agent."""
+    return [
+        WebMetadataExtractTool(api_key=api_key),
+        WebMarkdownTool(api_key=api_key),
+        WebContactsTool(api_key=api_key),
+        WebSEOAuditTool(api_key=api_key),
+    ]

crewai_webmetadata_extractor-0.1.0/tests/test_tools.py

@@ -0,0 +1,84 @@
+import json
+
+import responses
+from webmetadata_extractor.client import DEFAULT_BASE_URL
+
+from crewai_webmetadata_extractor import (
+    WebContactsTool,
+    WebMarkdownTool,
+    WebMetadataExtractTool,
+    WebSEOAuditTool,
+    get_tools,
+)
+
+
+@responses.activate
+def test_extract_tool_run_returns_json_string():
+    responses.add(
+        responses.GET,
+        f"{DEFAULT_BASE_URL}/api/v1/extract",
+        json={"title": "GitHub"},
+        status=200,
+    )
+
+    tool = WebMetadataExtractTool(api_key="fake-key")
+    result = tool._run("https://github.com")
+
+    assert json.loads(result) == {"title": "GitHub"}
+    assert responses.calls[0].request.headers["X-RapidAPI-Key"] == "fake-key"
+
+
+@responses.activate
+def test_markdown_tool_hits_markdown_endpoint():
+    responses.add(
+        responses.GET,
+        f"{DEFAULT_BASE_URL}/api/v1/markdown",
+        json={"markdown_content": "# Hello"},
+        status=200,
+    )
+
+    tool = WebMarkdownTool(api_key="fake-key")
+    result = tool._run("https://example.com")
+
+    assert json.loads(result) == {"markdown_content": "# Hello"}
+
+
+@responses.activate
+def test_contacts_tool_returns_error_payload_on_api_error_instead_of_raising():
+    responses.add(
+        responses.GET,
+        f"{DEFAULT_BASE_URL}/api/v1/contacts",
+        json={"detail": "Invalid URL"},
+        status=400,
+    )
+
+    tool = WebContactsTool(api_key="fake-key")
+    result = tool._run("not-a-url")
+
+    parsed = json.loads(result)
+    assert parsed["error"] is True
+    assert parsed["status_code"] == 400
+
+
+@responses.activate
+def test_seo_audit_tool_reads_env_var_when_no_api_key_passed(monkeypatch):
+    monkeypatch.setenv("WEBMETADATA_API_KEY", "env-key")
+    responses.add(
+        responses.GET, f"{DEFAULT_BASE_URL}/api/v1/seo-audit", json={"seo_score_percentage": 91.7}, status=200
+    )
+
+    tool = WebSEOAuditTool()
+    tool._run("https://example.com")
+
+    assert responses.calls[0].request.headers["X-RapidAPI-Key"] == "env-key"
+
+
+def test_get_tools_returns_all_four_with_shared_key():
+    tools = get_tools(api_key="shared-key")
+    assert {t.name for t in tools} == {
+        "web_metadata_extract",
+        "web_markdown_reader",
+        "web_contacts_extract",
+        "web_seo_audit",
+    }
+    assert all(t.api_key == "shared-key" for t in tools)