perplexity-webui-scraper 0.3.4__tar.gz → 0.3.6__tar.gz

{perplexity_webui_scraper-0.3.4 → perplexity_webui_scraper-0.3.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: perplexity-webui-scraper
-Version: 0.3.4
+Version: 0.3.6
 Summary: Python scraper to extract AI responses from Perplexity's web interface.
 Keywords: perplexity,ai,scraper,webui,api,client
 Author: henrique-coder
@@ -20,14 +20,18 @@ Classifier: Topic :: Internet :: WWW/HTTP
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
 Requires-Dist: curl-cffi>=0.14.0
+Requires-Dist: loguru>=0.7.3
 Requires-Dist: orjson>=3.11.5
 Requires-Dist: pydantic>=2.12.5
+Requires-Dist: tenacity>=9.1.2
+Requires-Dist: fastmcp>=2.14.2 ; extra == 'mcp'
 Requires-Python: >=3.10
 Project-URL: Changelog, https://github.com/henrique-coder/perplexity-webui-scraper/releases
 Project-URL: Documentation, https://github.com/henrique-coder/perplexity-webui-scraper#readme
 Project-URL: Homepage, https://github.com/henrique-coder/perplexity-webui-scraper
 Project-URL: Issues, https://github.com/henrique-coder/perplexity-webui-scraper/issues
 Project-URL: Repository, https://github.com/henrique-coder/perplexity-webui-scraper.git
+Provides-Extra: mcp
 Description-Content-Type: text/markdown
 
 <div align="center">
@@ -47,7 +51,8 @@ Python scraper to extract AI responses from [Perplexity's](https://www.perplexit
 ## Installation
 
 ```bash
-uv pip install perplexity-webui-scraper
+uv pip install perplexity-webui-scraper  # from PyPI (stable)
+uv pip install git+https://github.com/henrique-coder/perplexity-webui-scraper.git@dev  # from GitHub (development)
 ```
 
 ## Requirements
@@ -197,18 +202,103 @@ conversation.ask("Latest AI research", files=["paper.pdf"])
 | `timezone`        | `None`        | Timezone           |
 | `coordinates`     | `None`        | Location (lat/lng) |
 
-## CLI Tools
+## Exceptions
 
-### Session Token Generator
+The library provides specific exception types for better error handling:
+
+| Exception                          | Description                                                  |
+| ---------------------------------- | ------------------------------------------------------------ |
+| `PerplexityError`                  | Base exception for all library errors                        |
+| `AuthenticationError`              | Session token is invalid or expired (HTTP 403)               |
+| `RateLimitError`                   | Rate limit exceeded (HTTP 429)                               |
+| `FileUploadError`                  | File upload failed                                           |
+| `FileValidationError`              | File validation failed (size, type, etc.)                    |
+| `ResearchClarifyingQuestionsError` | Research mode is asking clarifying questions (not supported) |
+| `ResponseParsingError`             | API response could not be parsed                             |
+| `StreamingError`                   | Error during streaming response                              |
+
+### Handling Research Mode Clarifying Questions
+
+When using Research mode (`Models.RESEARCH`), the API may ask clarifying questions before providing an answer. Since programmatic interaction is not supported, the library raises a `ResearchClarifyingQuestionsError` with the questions:
+
+```python
+from perplexity_webui_scraper import (
+    Perplexity,
+    ResearchClarifyingQuestionsError,
+)
+
+try:
+    conversation.ask("Research this topic", model=Models.RESEARCH)
+except ResearchClarifyingQuestionsError as error:
+    print("The AI needs clarification:")
+    for question in error.questions:
+        print(f"  - {question}")
+    # Consider rephrasing your query to be more specific
+```
+
+## MCP Server (Model Context Protocol)
+
+The library includes an MCP server that allows AI assistants (like Claude) to search using Perplexity AI directly.
+
+### Installation
 
 ```bash
-get-perplexity-session-token
+uv pip install perplexity-webui-scraper[mcp]
+```
+
+### Running the Server
+
+```bash
+# Set your session token
+export PERPLEXITY_SESSION_TOKEN="your_token_here"  # For Linux/Mac
+set PERPLEXITY_SESSION_TOKEN="your_token_here"  # For Windows
+
+# Run with FastMCP
+uv run fastmcp run src/perplexity_webui_scraper/mcp/server.py
+
+# Or test with the dev inspector
+uv run fastmcp dev src/perplexity_webui_scraper/mcp/server.py
+```
+
+### Claude Desktop Configuration
+
+Add to `~/.config/claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "perplexity": {
+      "command": "uv",
+      "args": [
+        "run",
+        "fastmcp",
+        "run",
+        "path/to/perplexity_webui_scraper/mcp/server.py"
+      ],
+      "env": {
+        "PERPLEXITY_SESSION_TOKEN": "your_token_here"
+      }
+    }
+  }
+}
 ```
 
-Interactive tool to automatically obtain your Perplexity session token via email authentication. The token can be automatically saved to your `.env` file for immediate use.
+### Available Tool
+
+| Tool             | Description                                                                 |
+| ---------------- | --------------------------------------------------------------------------- |
+| `perplexity_ask` | Ask questions and get AI-generated answers with real-time data from the web |
+
+**Parameters:**
+
+| Parameter      | Type  | Default  | Description                                                   |
+| -------------- | ----- | -------- | ------------------------------------------------------------- |
+| `query`        | `str` | -        | Question to ask (required)                                    |
+| `model`        | `str` | `"best"` | AI model (`best`, `research`, `gpt52`, `claude_sonnet`, etc.) |
+| `source_focus` | `str` | `"web"`  | Source type (`web`, `academic`, `social`, `finance`, `all`)   |
 
 ## Disclaimer
 
-This is an **unofficial** library. It uses internal APIs that may change without notice. Use at your own risk. Not for production use.
+This is an **unofficial** library. It uses internal APIs that may change without notice. Use at your own risk.
 
 By using this library, you agree to Perplexity AI's Terms of Service.

{perplexity_webui_scraper-0.3.4 → perplexity_webui_scraper-0.3.6}/README.md

@@ -15,7 +15,8 @@ Python scraper to extract AI responses from [Perplexity's](https://www.perplexit
 ## Installation
 
 ```bash
-uv pip install perplexity-webui-scraper
+uv pip install perplexity-webui-scraper  # from PyPI (stable)
+uv pip install git+https://github.com/henrique-coder/perplexity-webui-scraper.git@dev  # from GitHub (development)
 ```
 
 ## Requirements
@@ -165,18 +166,103 @@ conversation.ask("Latest AI research", files=["paper.pdf"])
 | `timezone`        | `None`        | Timezone           |
 | `coordinates`     | `None`        | Location (lat/lng) |
 
-## CLI Tools
+## Exceptions
 
-### Session Token Generator
+The library provides specific exception types for better error handling:
+
+| Exception                          | Description                                                  |
+| ---------------------------------- | ------------------------------------------------------------ |
+| `PerplexityError`                  | Base exception for all library errors                        |
+| `AuthenticationError`              | Session token is invalid or expired (HTTP 403)               |
+| `RateLimitError`                   | Rate limit exceeded (HTTP 429)                               |
+| `FileUploadError`                  | File upload failed                                           |
+| `FileValidationError`              | File validation failed (size, type, etc.)                    |
+| `ResearchClarifyingQuestionsError` | Research mode is asking clarifying questions (not supported) |
+| `ResponseParsingError`             | API response could not be parsed                             |
+| `StreamingError`                   | Error during streaming response                              |
+
+### Handling Research Mode Clarifying Questions
+
+When using Research mode (`Models.RESEARCH`), the API may ask clarifying questions before providing an answer. Since programmatic interaction is not supported, the library raises a `ResearchClarifyingQuestionsError` with the questions:
+
+```python
+from perplexity_webui_scraper import (
+    Perplexity,
+    ResearchClarifyingQuestionsError,
+)
+
+try:
+    conversation.ask("Research this topic", model=Models.RESEARCH)
+except ResearchClarifyingQuestionsError as error:
+    print("The AI needs clarification:")
+    for question in error.questions:
+        print(f"  - {question}")
+    # Consider rephrasing your query to be more specific
+```
+
+## MCP Server (Model Context Protocol)
+
+The library includes an MCP server that allows AI assistants (like Claude) to search using Perplexity AI directly.
+
+### Installation
 
 ```bash
-get-perplexity-session-token
+uv pip install perplexity-webui-scraper[mcp]
+```
+
+### Running the Server
+
+```bash
+# Set your session token
+export PERPLEXITY_SESSION_TOKEN="your_token_here"  # For Linux/Mac
+set PERPLEXITY_SESSION_TOKEN="your_token_here"  # For Windows
+
+# Run with FastMCP
+uv run fastmcp run src/perplexity_webui_scraper/mcp/server.py
+
+# Or test with the dev inspector
+uv run fastmcp dev src/perplexity_webui_scraper/mcp/server.py
+```
+
+### Claude Desktop Configuration
+
+Add to `~/.config/claude/claude_desktop_config.json`:
+
+```json
+{
+  "mcpServers": {
+    "perplexity": {
+      "command": "uv",
+      "args": [
+        "run",
+        "fastmcp",
+        "run",
+        "path/to/perplexity_webui_scraper/mcp/server.py"
+      ],
+      "env": {
+        "PERPLEXITY_SESSION_TOKEN": "your_token_here"
+      }
+    }
+  }
+}
 ```
 
-Interactive tool to automatically obtain your Perplexity session token via email authentication. The token can be automatically saved to your `.env` file for immediate use.
+### Available Tool
+
+| Tool             | Description                                                                 |
+| ---------------- | --------------------------------------------------------------------------- |
+| `perplexity_ask` | Ask questions and get AI-generated answers with real-time data from the web |
+
+**Parameters:**
+
+| Parameter      | Type  | Default  | Description                                                   |
+| -------------- | ----- | -------- | ------------------------------------------------------------- |
+| `query`        | `str` | -        | Question to ask (required)                                    |
+| `model`        | `str` | `"best"` | AI model (`best`, `research`, `gpt52`, `claude_sonnet`, etc.) |
+| `source_focus` | `str` | `"web"`  | Source type (`web`, `academic`, `social`, `finance`, `all`)   |
 
 ## Disclaimer
 
-This is an **unofficial** library. It uses internal APIs that may change without notice. Use at your own risk. Not for production use.
+This is an **unofficial** library. It uses internal APIs that may change without notice. Use at your own risk.
 
 By using this library, you agree to Perplexity AI's Terms of Service.

{perplexity_webui_scraper-0.3.4 → perplexity_webui_scraper-0.3.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "perplexity-webui-scraper"
-version = "0.3.4"
+version = "0.3.6"
 description = "Python scraper to extract AI responses from Perplexity's web interface."
 authors = [{ name = "henrique-coder", email = "henriquemoreira10fk@gmail.com" }]
 license = "MIT"
@@ -24,8 +24,10 @@ classifiers = [
 ]
 dependencies = [
     "curl-cffi>=0.14.0",
+    "loguru>=0.7.3",
     "orjson>=3.11.5",
     "pydantic>=2.12.5",
+    "tenacity>=9.1.2",
 ]
 
 [dependency-groups]
@@ -33,16 +35,23 @@ dev = [
     "beautifulsoup4>=4.14.3",
     "jsbeautifier>=1.15.4",
     "lxml>=6.0.2",
+    "prek>=0.2.25",
     "python-dotenv>=1.2.1",
     "rich>=14.2.0",
 ]
 lint = [
     "ruff>=0.14.10",
+    "ty>=0.0.9",
 ]
 tests = [
     "pytest>=9.0.2",
 ]
 
+[project.optional-dependencies]
+mcp = [
+    "fastmcp>=2.14.2",
+]
+
 [project.urls]
 Homepage = "https://github.com/henrique-coder/perplexity-webui-scraper"
 Documentation = "https://github.com/henrique-coder/perplexity-webui-scraper#readme"
@@ -103,6 +112,7 @@ skip-magic-trailing-comma = false   # Preserve trailing commas as formatting hin
 
 [project.scripts]
 get-perplexity-session-token = "perplexity_webui_scraper.cli.get_perplexity_session_token:get_token"
+perplexity-webui-scraper-mcp = "perplexity_webui_scraper.mcp:run_server"
 
 [build-system]
 requires = ["uv_build"]

{perplexity_webui_scraper-0.3.4 → perplexity_webui_scraper-0.3.6}/src/perplexity_webui_scraper/__init__.py

@@ -1,36 +1,27 @@
-"""Extract AI responses from Perplexity's web interface."""
+"""
+Extract AI responses from Perplexity's web interface.
+"""
 
 from importlib import metadata
 
 from .config import ClientConfig, ConversationConfig
 from .core import Conversation, Perplexity
-from .enums import CitationMode, SearchFocus, SourceFocus, TimeRange
-from .exceptions import (
-    AuthenticationError,
-    FileUploadError,
-    FileValidationError,
-    PerplexityError,
-    RateLimitError,
-)
+from .enums import CitationMode, LogLevel, SearchFocus, SourceFocus, TimeRange
 from .models import Model, Models
 from .types import Coordinates, Response, SearchResultItem
 
 
 __version__: str = metadata.version("perplexity-webui-scraper")
 __all__: list[str] = [
-    "AuthenticationError",
     "CitationMode",
     "ClientConfig",
     "Conversation",
     "ConversationConfig",
     "Coordinates",
-    "FileUploadError",
-    "FileValidationError",
+    "LogLevel",
     "Model",
     "Models",
     "Perplexity",
-    "PerplexityError",
-    "RateLimitError",
     "Response",
     "SearchFocus",
     "SearchResultItem",

{perplexity_webui_scraper-0.3.4 → perplexity_webui_scraper-0.3.6}/src/perplexity_webui_scraper/cli/get_perplexity_session_token.py

@@ -1,4 +1,6 @@
-"""CLI utility for secure Perplexity authentication and session extraction."""
+"""
+CLI utility for secure Perplexity authentication and session extraction.
+"""
 
 from __future__ import annotations
 
@@ -57,7 +59,9 @@ def update_env(token: str) -> bool:
 
 
 def _initialize_session() -> tuple[Session, str]:
-    """Initialize session and obtain CSRF token."""
+    """
+    Initialize session and obtain CSRF token.
+    """
 
     session = Session(impersonate="chrome", headers={"Referer": BASE_URL, "Origin": BASE_URL})
 
@@ -73,7 +77,9 @@ def _initialize_session() -> tuple[Session, str]:
 
 
 def _request_verification_code(session: Session, csrf: str, email: str) -> None:
-    """Send verification code to user's email."""
+    """
+    Send verification code to user's email.
+    """
 
     with console.status("[bold green]Sending verification code...", spinner="dots"):
         r = session.post(
@@ -92,7 +98,9 @@ def _request_verification_code(session: Session, csrf: str, email: str) -> None:
 
 
 def _validate_and_get_redirect_url(session: Session, email: str, user_input: str) -> str:
-    """Validate user input (OTP or magic link) and return redirect URL."""
+    """
+    Validate user input (OTP or magic link) and return redirect URL.
+    """
 
     with console.status("[bold green]Validating...", spinner="dots"):
         if user_input.startswith("http"):
@@ -120,7 +128,9 @@ def _validate_and_get_redirect_url(session: Session, email: str, user_input: str
 
 
 def _extract_session_token(session: Session, redirect_url: str) -> str:
-    """Extract session token from cookies after authentication."""
+    """
+    Extract session token from cookies after authentication.
+    """
 
     session.get(redirect_url)
     token = session.cookies.get("__Secure-next-auth.session-token")
@@ -132,7 +142,9 @@ def _extract_session_token(session: Session, redirect_url: str) -> str:
 
 
 def _display_and_save_token(token: str) -> None:
-    """Display token and optionally save to .env file."""
+    """
+    Display token and optionally save to .env file.
+    """
 
     console.print("\n[bold green]✅ Token generated successfully![/bold green]")
     console.print(f"\n[bold white]Your session token:[/bold white]\n[green]{token}[/green]\n")
@@ -147,7 +159,9 @@ def _display_and_save_token(token: str) -> None:
 
 
 def _show_header() -> None:
-    """Display welcome header."""
+    """
+    Display welcome header.
+    """
 
     console.print(
         Panel(
@@ -161,7 +175,9 @@ def _show_header() -> None:
 
 
 def _show_exit_message() -> None:
-    """Display security note and wait for user to exit."""
+    """
+    Display security note and wait for user to exit.
+    """
 
     console.print("\n[bold yellow]⚠️ Security Note:[/bold yellow]")
     console.print("Press [bold white]ENTER[/bold white] to clear screen and exit.")

perplexity_webui_scraper-0.3.6/src/perplexity_webui_scraper/config.py

@@ -0,0 +1,65 @@
+"""
+Configuration classes.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import TYPE_CHECKING
+
+from .enums import CitationMode, LogLevel, SearchFocus, SourceFocus, TimeRange
+
+
+if TYPE_CHECKING:
+    from pathlib import Path
+
+    from .models import Model
+    from .types import Coordinates
+
+
+@dataclass(slots=True)
+class ConversationConfig:
+    """
+    Default settings for a conversation. Can be overridden per message.
+    """
+
+    model: Model | None = None
+    citation_mode: CitationMode = CitationMode.CLEAN
+    save_to_library: bool = False
+    search_focus: SearchFocus = SearchFocus.WEB
+    source_focus: SourceFocus | list[SourceFocus] = SourceFocus.WEB
+    time_range: TimeRange = TimeRange.ALL
+    language: str = "en-US"
+    timezone: str | None = None
+    coordinates: Coordinates | None = None
+
+
+@dataclass(frozen=True, slots=True)
+class ClientConfig:
+    """
+    HTTP client settings.
+
+    Attributes:
+        timeout: Request timeout in seconds.
+        impersonate: Browser to impersonate (e.g., "chrome", "edge", "safari").
+        max_retries: Maximum retry attempts for failed requests.
+        retry_base_delay: Initial delay in seconds before first retry.
+        retry_max_delay: Maximum delay between retries.
+        retry_jitter: Random jitter factor (0-1) to add to delays.
+        requests_per_second: Rate limit for requests (0 to disable).
+        rotate_fingerprint: Whether to rotate browser fingerprint on retries.
+        logging_level: Logging verbosity level. Default is DISABLED.
+        log_file: Optional file path for persistent logging. If set, logs go to file only.
+                  If None, logs go to console. All logs are appended.
+    """
+
+    timeout: int = 3600
+    impersonate: str = "chrome"
+    max_retries: int = 3
+    retry_base_delay: float = 1.0
+    retry_max_delay: float = 60.0
+    retry_jitter: float = 0.5
+    requests_per_second: float = 0.5
+    rotate_fingerprint: bool = True
+    logging_level: LogLevel = LogLevel.DISABLED
+    log_file: str | Path | None = None

{perplexity_webui_scraper-0.3.4 → perplexity_webui_scraper-0.3.6}/src/perplexity_webui_scraper/constants.py

@@ -1,4 +1,6 @@
-"""Constants and values for the Perplexity internal API and HTTP interactions."""
+"""
+Constants and values for the Perplexity internal API and HTTP interactions.
+"""
 
 from __future__ import annotations
 
@@ -8,20 +10,30 @@ from typing import Final
 
 # API Configuration
 API_VERSION: Final[str] = "2.18"
-"""Current API version used by Perplexity WebUI."""
+"""
+Current API version used by Perplexity WebUI.
+"""
 
 API_BASE_URL: Final[str] = "https://www.perplexity.ai"
-"""Base URL for all API requests."""
+"""
+Base URL for all API requests.
+"""
 
 # API Endpoints
 ENDPOINT_ASK: Final[str] = "/rest/sse/perplexity_ask"
-"""SSE endpoint for sending prompts."""
+"""
+SSE endpoint for sending prompts.
+"""
 
 ENDPOINT_SEARCH_INIT: Final[str] = "/search/new"
-"""Endpoint to initialize a search session."""
+"""
+Endpoint to initialize a search session.
+"""
 
 ENDPOINT_UPLOAD: Final[str] = "/rest/uploads/batch_create_upload_urls"
-"""Endpoint for file upload URL generation."""
+"""
+Endpoint for file upload URL generation.
+"""
 
 # API Fixed Parameters
 SEND_BACK_TEXT: Final[bool] = True
@@ -33,10 +45,14 @@ False = API sends delta chunks only (accumulate mode).
 """
 
 USE_SCHEMATIZED_API: Final[bool] = False
-"""Whether to use the schematized API format."""
+"""
+Whether to use the schematized API format.
+"""
 
 PROMPT_SOURCE: Final[str] = "user"
-"""Source identifier for prompts."""
+"""
+Source identifier for prompts.
+"""
 
 # Regex Patterns (Pre-compiled for performance in streaming parsing)
 CITATION_PATTERN: Final[Pattern[str]] = compile(r"\[(\d{1,2})\]")
@@ -47,7 +63,9 @@ Uses word boundary to avoid matching things like [123].
 """
 
 JSON_OBJECT_PATTERN: Final[Pattern[str]] = compile(r"^\{.*\}$")
-"""Pattern to detect JSON object strings."""
+"""
+Pattern to detect JSON object strings.
+"""
 
 # HTTP Headers
 DEFAULT_HEADERS: Final[dict[str, str]] = {
@@ -61,4 +79,6 @@ Referer and Origin are added dynamically based on BASE_URL.
 """
 
 SESSION_COOKIE_NAME: Final[str] = "__Secure-next-auth.session-token"
-"""Name of the session cookie used for authentication."""
+"""
+Name of the session cookie used for authentication.
+"""