symbolicai 1.4.0__tar.gz → 1.5.0__tar.gz

{symbolicai-1.4.0 → symbolicai-1.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: symbolicai
-Version: 1.4.0
+Version: 1.5.0
 Summary: A Neurosymbolic Perspective on Large Language Models
 Author-email: Marius-Constantin Dinu <marius@extensity.ai>, Leoveanu-Condrei Claudiu <leo@extensity.ai>
 License: BSD 3-Clause License
@@ -113,6 +113,7 @@ Requires-Dist: openai-whisper>=20240930; extra == "whisper"
 Requires-Dist: numba>=0.62.1; extra == "whisper"
 Requires-Dist: llvmlite>=0.45.1; extra == "whisper"
 Provides-Extra: search
+Requires-Dist: firecrawl-py>=4.12.0; extra == "search"
 Requires-Dist: parallel-web>=0.3.3; extra == "search"
 Provides-Extra: serpapi
 Requires-Dist: google_search_results>=2.4.2; extra == "serpapi"

symbolicai-1.5.0/docs/source/ENGINES/scrape_engine.md

@@ -0,0 +1,143 @@
+# Scrape Engine
+
+## Naive Scrape
+
+To access data from the web, we can use the `naive_scrape` interface. The engine underneath is very lightweight and can be used to scrape data from websites. It is based on the `requests` library, as well as `trafilatura` for output formatting, and `bs4` for HTML parsing. `trafilatura` currently supports the following output formats: `json`, `csv`, `html`, `markdown`, `text`, `xml`
+
+```python
+from symai.interfaces import Interface
+
+scraper = Interface("naive_scrape")
+url = "https://docs.astral.sh/uv/guides/scripts/#next-steps"
+res = scraper(url)
+```
+
+## Parallel (Parallel.ai)
+
+The Parallel.ai integration routes scrape calls through the official `parallel-web` SDK and can handle PDFs, JavaScript-heavy feeds, and standard HTML pages in the same workflow. Instantiate the Parallel interface and call `.scrape(...)` with the target URL. The engine detects scrape requests automatically whenever a URL is supplied.
+
+```python
+from symai.extended import Interface
+
+scraper = Interface("parallel")
+article = scraper.scrape(
+    "https://trafilatura.readthedocs.io/en/latest/crawls.html",
+    full_content=True,           # optional: request full document text
+    excerpts=True,               # optional: default True, retain excerpt snippets
+    objective="Summarize crawl guidance for internal notes."
+)
+print(str(article))
+```
+
+Configuration requires a Parallel API key and the Parallel model token. Add the following to your settings:
+
+```bash
+{
+    …
+    "SEARCH_ENGINE_API_KEY": "…",
+    "SEARCH_ENGINE_MODEL": "parallel"
+    …
+}
+```
+
+When invoked with a URL, the engine hits Parallel's Extract API and returns an `ExtractResult`. The result string joins excerpts or the full content if requested. Because processing is offloaded to Parallel's hosted infrastructure, the engine remains reliable on dynamic pages that the naive scraper cannot render. Install the dependency with `pip install parallel-web` before enabling this engine.
+
+## Firecrawl
+
+Firecrawl.dev specializes in reliable web scraping with automatic handling of JavaScript-rendered content, proxies, and anti-bot mechanisms. It converts web pages into clean formats suitable for LLM consumption and supports advanced features like actions, caching, and location-based scraping.
+
+### Examples
+
+```python
+from symai.extended import Interface
+
+scraper = Interface("firecrawl")
+
+# Example 1: Basic webpage scraping
+content = scraper.scrape(
+    "https://docs.firecrawl.dev/introduction",
+    formats=["markdown"]
+)
+print(content)
+
+# Example 2: PDF scraping with content extraction and trimming
+pdf_full = scraper.scrape(
+    "https://pmc.ncbi.nlm.nih.gov/articles/PMC7231600",
+    only_main_content=True,
+    formats=["markdown"],
+    proxy="auto"
+)
+# Trim locally if needed
+pdf_trimmed = str(pdf_full)[:100]
+
+# Note: JS-heavy sites like Twitter/LinkedIn are currently not fully supported
+# They typically return 403 Forbidden errors (may vary by subscription tier)
+```
+
+### Configuration
+
+Enable the engine by configuring Firecrawl credentials:
+
+```bash
+{
+    "SEARCH_ENGINE_API_KEY": "fc-your-api-key",
+    "SEARCH_ENGINE_MODEL": "firecrawl"
+}
+```
+
+### JSON Schema Extraction
+
+Firecrawl supports structured data extraction using JSON schemas. This is useful for extracting specific fields from web pages using LLM-powered extraction:
+
+```python
+from pydantic import Field
+from symai.extended import Interface
+from symai.models import LLMDataModel
+
+class MetadataModel(LLMDataModel):
+    """Bibliographic metadata extracted from a source document."""
+    title: str = Field(description="Title of the source.")
+    year: str = Field(description="Publication year (4 digits) or Unknown.")
+    authors: list[str] = Field(default_factory=list, description="List of authors.")
+    doi: str | None = Field(default=None, description="DOI if available.")
+
+# Build JSON format config from Pydantic schema
+schema = MetadataModel.model_json_schema()
+json_format = {
+    "type": "json",
+    "prompt": "Extract bibliographic metadata from this academic paper.",
+    "schema": schema,
+}
+
+scraper = Interface("firecrawl")
+result = scraper.scrape(
+    "https://journals.physiology.org/doi/full/10.1152/ajpregu.00051.2002",
+    formats=[json_format],
+    proxy="auto"
+)
+
+# Access extracted data as dict
+extracted = result.raw["json"]
+metadata = MetadataModel(**extracted)
+print(metadata.model_dump())
+
+# Or as JSON string
+print(str(result))
+```
+
+### Supported Parameters
+
+The engine supports many parameters (passed as kwargs). Common ones include:
+- **formats**: Output formats (["markdown"], ["html"], ["rawHtml"])
+- **only_main_content**: Extract main content only (boolean)
+- **proxy**: Proxy mode ("basic", "stealth", "auto")
+- **location**: Geographic location object with optional country and languages
+  - Example: `{"country": "US"}` or `{"country": "RO", "languages": ["ro"]}`
+- **maxAge**: Cache duration in seconds (integer)
+- **storeInCache**: Enable caching (boolean)
+- **actions**: Page interactions before scraping (list of action objects)
+  - Example: `[{"type": "wait", "milliseconds": 2000}]`
+  - Example: `[{"type": "click", "selector": ".button"}]`
+  - Example: `[{"type": "scroll", "direction": "down", "amount": 500}]`
+
+Check the Firecrawl v2 API documentation for the complete list of available parameters and action types.

{symbolicai-1.4.0 → symbolicai-1.5.0}/docs/source/ENGINES/search_engine.md

@@ -152,3 +152,75 @@ Here's how to configure the OpenAI search engine:
 ```
 
 This engine calls the OpenAI Responses API under the hood. When you target a reasoning-capable model, pass a `reasoning` dictionary matching the Responses payload schema (for example `{"effort": "low", "summary": "auto"}`). If omitted, the engine falls back to the default effort/summary settings shown above.
+
+## Firecrawl
+Firecrawl.dev provides web scraping and search capabilities with built-in handling of dynamic JavaScript content and anti-bot mechanisms. The engine converts web pages into clean markdown and can perform web searches across multiple sources with advanced filtering and content extraction.
+
+### Comprehensive Search Example
+
+```python
+from symai.extended import Interface
+
+engine = Interface("firecrawl")
+
+# Example 1: Location-aware search with language, scraping, and citations
+result = engine.search(
+    "who is nicusor dan",
+    limit=5,
+    location="Romania",
+    lang="ro",
+    sources=["web"],
+    formats=["markdown"],
+    only_main_content=True,
+    proxy="stealth"
+)
+
+# Access structured citations (similar to parallel.ai)
+citations = result.get_citations()
+for citation in citations:
+    print(f"[{citation.id}] {citation.title}: {citation.url}")
+
+# Example 2: Domain-filtered search with character limits
+domains = ["arxiv.org", "nature.com"]
+filters = " OR ".join(f"site:{domain}" for domain in domains)
+query = f"({filters}) what is thermodynamic computing"
+
+result = engine.search(
+    query,
+    limit=10,
+    max_chars_per_result=500,
+    categories=["research"],
+    formats=["markdown"],
+    proxy="basic"
+)
+print(result)
+```
+
+### Configuration
+
+Enable the engine by configuring Firecrawl credentials:
+
+```bash
+{
+    "SEARCH_ENGINE_API_KEY": "fc-your-api-key",
+    "SEARCH_ENGINE_MODEL": "firecrawl"
+}
+```
+
+### Supported Parameters
+
+The engine supports many parameters (passed as kwargs). Common ones include:
+- **limit**: Max number of results
+- **location**: Country code string for search (e.g., "Romania", "Germany")
+- **lang**: Language code string for search (e.g., "ro", "es") - hint, not enforcement
+- **sources**: List of sources (["web"], ["news"], ["images"])
+- **categories**: Content types (["research"], ["github"], ["pdf"])
+- **tbs**: Time-based filter (e.g., "qdr:d" for past day)
+- **formats**: Output formats for scraped content (["markdown"], ["html"])
+- **only_main_content**: Extract main content only when scraping (boolean)
+- **max_chars_per_result**: Truncate results locally (integer)
+- **proxy**: Proxy mode for scraping ("basic", "stealth", "auto")
+- **scrape_location**: Location object for scraping with optional country and languages
+  - Example: `{"country": "US"}` or `{"country": "RO", "languages": ["ro"]}`
+
+Check the Firecrawl v2 API documentation for the complete list of available parameters.

{symbolicai-1.4.0 → symbolicai-1.5.0}/pyproject.toml

@@ -78,7 +78,7 @@ scrape       = ["beautifulsoup4>=4.12.3", "trafilatura>=2.0.0", "pdfminer.six",
 llama_cpp    = ["llama-cpp-python[server]>=0.3.7"] # handle separately since this dependency may not compile and require special maintenance
 wolframalpha = ["wolframalpha>=5.0.0"]
 whisper      = ["openai-whisper>=20240930", "numba>=0.62.1", "llvmlite>=0.45.1"]
-search       = ["parallel-web>=0.3.3"]
+search       = ["firecrawl-py>=4.12.0", "parallel-web>=0.3.3"]
 serpapi      = ["google_search_results>=2.4.2"]
 services     = ["fastapi>=0.110.0", "redis>=5.0.2", "uvicorn>=0.27.1"]
 solver       = ["z3-solver>=4.12.6.0"]

{symbolicai-1.4.0 → symbolicai-1.5.0}/symai/__init__.py

@@ -33,7 +33,7 @@ os.environ["TOKENIZERS_PARALLELISM"] = "false"
 # Create singleton instance
 config_manager = settings.SymAIConfig()
 
-SYMAI_VERSION = "1.4.0"
+SYMAI_VERSION = "1.5.0"
 __version__ = SYMAI_VERSION
 __root_dir__ = config_manager.config_dir
 

{symbolicai-1.4.0 → symbolicai-1.5.0}/symai/backend/engines/scrape/engine_requests.py

@@ -9,6 +9,7 @@ service disruption.
 
 import io
 import logging
+import random
 import re
 from typing import Any, ClassVar
 from urllib.parse import parse_qsl, urlencode, urljoin, urlparse, urlunparse
@@ -17,7 +18,9 @@ import requests
 import trafilatura
 from bs4 import BeautifulSoup
 from pdfminer.high_level import extract_text
+from requests.adapters import HTTPAdapter
 from requests.structures import CaseInsensitiveDict
+from urllib3.util.retry import Retry
 
 from ....symbol import Result
 from ....utils import UserMessage
@@ -80,24 +83,49 @@ class RequestsEngine(Engine):
         "none": "None",
     }
 
-    def __init__(self, timeout=15, verify_ssl=True, user_agent=None):
+    USER_AGENT_POOL: ClassVar[list[str]] = [
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+        "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36",
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0",
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:121.0) Gecko/20100101 Firefox/121.0",
+        "Mozilla/5.0 (X11; Linux x86_64; rv:121.0) Gecko/20100101 Firefox/121.0",
+        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.2 Safari/605.1.15",
+        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36 Edg/120.0.0.0",
+    ]
+
+    def __init__(self, timeout=15, verify_ssl=True, user_agent=None, retries=3, backoff_factor=0.5, retry_status_codes=(500, 502, 503, 504)):
         """
         Args:
             timeout: Seconds to wait for network operations before aborting.
             verify_ssl: Toggle for TLS certificate verification.
-            user_agent: Optional override for the default desktop Chrome UA.
+            user_agent: Optional override for user agent rotation.
+            retries: Number of retries for failed requests (default: 3).
+            backoff_factor: Multiplier for exponential backoff (default: 0.5).
+            retry_status_codes: HTTP status codes to retry on (default: 500, 502, 503, 504).
         """
         super().__init__()
         self.timeout = timeout
         self.verify_ssl = verify_ssl
         self.name = self.__class__.__name__
-
-        headers = dict(self.DEFAULT_HEADERS)
-        if user_agent:
-            headers["User-Agent"] = user_agent
+        self._user_agent_override = user_agent
 
         self.session = requests.Session()
-        self.session.headers.update(headers)
+        self.session.headers.update({k: v for k, v in self.DEFAULT_HEADERS.items() if k != "User-Agent"})
+
+        retry_strategy = Retry(
+            total=retries,
+            backoff_factor=backoff_factor,
+            status_forcelist=retry_status_codes,
+            allowed_methods=["GET", "HEAD"],
+        )
+        adapter = HTTPAdapter(max_retries=retry_strategy)
+        self.session.mount("http://", adapter)
+        self.session.mount("https://", adapter)
+
+    def _get_user_agent(self) -> str:
+        """Return user agent: override if set, otherwise random from pool."""
+        return self._user_agent_override or random.choice(self.USER_AGENT_POOL)
 
     def _maybe_set_bypass_cookies(self, url: str):
         netloc = urlparse(url).hostname
@@ -232,7 +260,7 @@ class RequestsEngine(Engine):
         # Avoid loops
         if target == resp.url:
             return resp
-        return self.session.get(target, timeout=timeout, allow_redirects=True)
+        return self.session.get(target, timeout=timeout, allow_redirects=True, headers={"User-Agent": self._get_user_agent()})
 
     def _fetch_with_playwright(
         self,
@@ -259,7 +287,7 @@ class RequestsEngine(Engine):
 
         timeout_seconds = timeout if timeout is not None else self.timeout
         timeout_ms = max(int(timeout_seconds * 1000), 0)
-        user_agent = self.session.headers.get("User-Agent")
+        user_agent = self._get_user_agent()
 
         parsed = urlparse(url)
         hostname = parsed.hostname or ""
@@ -348,7 +376,8 @@ class RequestsEngine(Engine):
             )
         else:
             resp = self.session.get(
-                clean_url, timeout=self.timeout, allow_redirects=True, verify=self.verify_ssl
+                clean_url, timeout=self.timeout, allow_redirects=True, verify=self.verify_ssl,
+                headers={"User-Agent": self._get_user_agent()}
             )
         resp.raise_for_status()
 

symbolicai-1.5.0/symai/backend/engines/search/__init__.py

@@ -0,0 +1,13 @@
+from .engine_firecrawl import FirecrawlEngine
+from .engine_parallel import ParallelEngine
+
+SEARCH_ENGINE_MAPPING = {
+    "firecrawl": FirecrawlEngine,
+    "parallel": ParallelEngine,
+}
+
+__all__ = [
+    "SEARCH_ENGINE_MAPPING",
+    "FirecrawlEngine",
+    "ParallelEngine",
+]