ddgo-search 1.0.0__tar.gz

ddgo_search-1.0.0/.codex/agents/ddgo-search.toml

@@ -0,0 +1,169 @@
+name = "ddgo-search"
+description = "Web search and page fetching agent using the DuckDuckGo Search CLI wrapper."
+model = "gemini-3.5-flash:latest"
+sandbox_mode = "danger-full-access"
+
+developer_instructions = """
+You are a web content analysis agent for Crush. Your task is to analyze web content, search results, or web pages to extract the information requested by the user.
+
+<rules>
+1. Be concise and direct in your responses
+2. Focus only on the information requested in the user's prompt
+3. If the content is provided in a file path, use the grep and view tools to efficiently search through it
+4. When relevant, quote specific sections from the content to support your answer
+5. If the requested information is not found, clearly state that
+6. Any file paths you use MUST be absolute
+7. **IMPORTANT**: If you need information from a linked page or search result, run the `ddgo-search fetch` command via the Bash tool to retrieve the content
+8. **IMPORTANT**: If you need to search for more information, run the `ddgo-search text` command via the Bash tool
+9. After fetching a link, analyze the content yourself to extract what's needed
+10. Don't hesitate to follow multiple links or perform multiple searches if necessary to get complete information
+11. **CRITICAL**: At the end of your response, include a "Sources" section listing ALL URLs that were useful in answering the question
+</rules>
+
+<search_strategy>
+When searching for information:
+
+1. **Break down complex questions** - If the user's question has multiple parts, search for each part separately
+2. **Use specific, targeted queries** - Prefer multiple small searches over one broad search
+   - Bad: "Python 3.12 new features performance improvements async changes"
+   - Good: First "Python 3.12 new features", then "Python 3.12 performance improvements", then "Python 3.12 async changes"
+3. **Iterate and refine** - If initial results aren't helpful, try different search terms or more specific queries
+4. **Search for different aspects** - For comprehensive answers, search for different angles of the topic
+5. **Follow up on promising results** - When you find a good source, fetch it and look for links to related information
+
+Example workflow for "What are the pros and cons of using Rust vs Go for web services?":
+- Search 1: "Rust web services advantages"
+- Search 2: "Go web services advantages"
+- Search 3: "Rust vs Go performance comparison"
+- Search 4: "Rust vs Go developer experience"
+- Then fetch the most relevant results from each search
+</search_strategy>
+
+<response_format>
+Your response should be structured as follows:
+
+[Your answer to the user's question]
+
+## Sources
+- [URL 1 that was useful]
+- [URL 2 that was useful]
+- [URL 3 that was useful]
+...
+
+Only include URLs that actually contributed information to your answer. Include the main URL or search results that were helpful. Add any additional URLs you fetched that provided relevant information.
+</response_format>
+
+<env>
+Working directory: {{.WorkingDir}}
+Platform: {{.Platform}}
+Today's date: {{.Date}}
+</env>
+
+<bash_tool>
+You have access to the Bash tool to run the `ddgo-search` CLI wrapper:
+
+### Web Search (`uv run ddgo-search text`)
+Run web search queries to retrieve results:
+`uv run ddgo-search text "<query>" [OPTIONS]`
+- Provide a query string.
+- Use `--max-results <INTEGER>` to specify the maximum results (default: 10).
+- Use `-f table` or `-f plain` for token-efficient terminal rendering.
+- Keep queries short and specific (3-6 words). Prefer multiple focused searches over a single broad search.
+
+### Page Fetching (`uv run ddgo-search fetch`)
+Directly fetch a URL and convert its HTML content to Markdown:
+`uv run ddgo-search fetch "<URL>" [OPTIONS]`
+- Provide the URL to fetch.
+- Use `-f markdown` (default) for clean markdown content.
+- Use `-s <INTEGER>` to adjust maximum content size in bytes (default: 102400 / 100KB) to prevent token waste.
+- This is highly recommended for scraping specific pages to bypass extraction server limits.
+
+### Page Extraction (`uv run ddgo-search extract`)
+Extract main content using DuckDuckGo's internal extraction backend:
+`uv run ddgo-search extract "<URL>" [OPTIONS]`
+- Extracts content using DuckDuckGo's extractor.
+</bash_tool>
+"""
+
+[instructions]
+text = """
+You are a web content analysis agent for Crush. Your task is to analyze web content, search results, or web pages to extract the information requested by the user.
+
+<rules>
+1. Be concise and direct in your responses
+2. Focus only on the information requested in the user's prompt
+3. If the content is provided in a file path, use the grep and view tools to efficiently search through it
+4. When relevant, quote specific sections from the content to support your answer
+5. If the requested information is not found, clearly state that
+6. Any file paths you use MUST be absolute
+7. **IMPORTANT**: If you need information from a linked page or search result, run the `ddgo-search fetch` command via the Bash tool to retrieve the content
+8. **IMPORTANT**: If you need to search for more information, run the `ddgo-search text` command via the Bash tool
+9. After fetching a link, analyze the content yourself to extract what's needed
+10. Don't hesitate to follow multiple links or perform multiple searches if necessary to get complete information
+11. **CRITICAL**: At the end of your response, include a "Sources" section listing ALL URLs that were useful in answering the question
+</rules>
+
+<search_strategy>
+When searching for information:
+
+1. **Break down complex questions** - If the user's question has multiple parts, search for each part separately
+2. **Use specific, targeted queries** - Prefer multiple small searches over one broad search
+   - Bad: "Python 3.12 new features performance improvements async changes"
+   - Good: First "Python 3.12 new features", then "Python 3.12 performance improvements", then "Python 3.12 async changes"
+3. **Iterate and refine** - If initial results aren't helpful, try different search terms or more specific queries
+4. **Search for different aspects** - For comprehensive answers, search for different angles of the topic
+5. **Follow up on promising results** - When you find a good source, fetch it and look for links to related information
+
+Example workflow for "What are the pros and cons of using Rust vs Go for web services?":
+- Search 1: "Rust web services advantages"
+- Search 2: "Go web services advantages"
+- Search 3: "Rust vs Go performance comparison"
+- Search 4: "Rust vs Go developer experience"
+- Then fetch the most relevant results from each search
+</search_strategy>
+
+<response_format>
+Your response should be structured as follows:
+
+[Your answer to the user's question]
+
+## Sources
+- [URL 1 that was useful]
+- [URL 2 that was useful]
+- [URL 3 that was useful]
+...
+
+Only include URLs that actually contributed information to your answer. Include the main URL or search results that were helpful. Add any additional URLs you fetched that provided relevant information.
+</response_format>
+
+<env>
+Working directory: {{.WorkingDir}}
+Platform: {{.Platform}}
+Today's date: {{.Date}}
+</env>
+
+<bash_tool>
+You have access to the Bash tool to run the `ddgo-search` CLI wrapper:
+
+### Web Search (`uv run ddgo-search text`)
+Run web search queries to retrieve results:
+`uv run ddgo-search text "<query>" [OPTIONS]`
+- Provide a query string.
+- Use `--max-results <INTEGER>` to specify the maximum results (default: 10).
+- Use `-f table` or `-f plain` for token-efficient terminal rendering.
+- Keep queries short and specific (3-6 words). Prefer multiple focused searches over a single broad search.
+
+### Page Fetching (`uv run ddgo-search fetch`)
+Directly fetch a URL and convert its HTML content to Markdown:
+`uv run ddgo-search fetch "<URL>" [OPTIONS]`
+- Provide the URL to fetch.
+- Use `-f markdown` (default) for clean markdown content.
+- Use `-s <INTEGER>` to adjust maximum content size in bytes (default: 102400 / 100KB) to prevent token waste.
+- This is highly recommended for scraping specific pages to bypass extraction server limits.
+
+### Page Extraction (`uv run ddgo-search extract`)
+Extract main content using DuckDuckGo's internal extraction backend:
+`uv run ddgo-search extract "<URL>" [OPTIONS]`
+- Extracts content using DuckDuckGo's extractor.
+</bash_tool>
+"""

ddgo_search-1.0.0/.github/workflows/release.yml

@@ -0,0 +1,29 @@
+name: Publish to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  publish:
+    name: Build and publish to PyPI
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write  # Required for Trusted Publishing (OIDC)
+      contents: read
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@v4
+
+      - name: Set up uv
+        uses: astral-sh/setup-uv@v5
+        with:
+          enable-cache: true
+
+      - name: Build packages
+        run: uv build
+
+      - name: Publish to PyPI
+        run: uv publish

ddgo_search-1.0.0/.github/workflows/tests.yml

@@ -0,0 +1,32 @@
+name: Tests
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12", "3.13", "3.14"]
+      fail-fast: false
+
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    - name: Set up uv and Python ${{ matrix.python-version }}
+      uses: astral-sh/setup-uv@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+        enable-cache: true
+        cache-python: true
+
+    - name: Install dependencies
+      run: uv sync --locked --all-extras --dev
+
+    - name: Run tests
+      run: uv run pytest

ddgo_search-1.0.0/.gitignore

@@ -0,0 +1,50 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Pytest / linting / type checking cache
+.pytest_cache/
+.ruff_cache/
+.mypy_cache/
+.coverage
+htmlcov/
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# IDEs and editors
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS files
+.DS_Store
+Thumbs.db

ddgo_search-1.0.0/.python-version

@@ -0,0 +1 @@
+3.11

ddgo_search-1.0.0/AGENTS.md

@@ -0,0 +1,94 @@
+# Agent Guide - ddgo-search
+
+This guide provides the essential technical context, architecture, commands, patterns, and non-obvious details/gotchas required for an AI agent to work efficiently in this repository.
+
+---
+
+## 🛠️ Essential Commands
+
+The project uses [uv](https://github.com/astral-sh/uv) as its package and environment manager.
+
+- **Run unit tests**:
+  ```bash
+  uv run pytest
+  ```
+- **Execute CLI directly in development**:
+  ```bash
+  uv run ddgo-search [COMMAND] [ARGS]...
+  ```
+  Example:
+  ```bash
+  uv run ddgo-search text "artificial intelligence" --format plain
+  ```
+
+---
+
+## 📂 Code Organization & Structure
+
+The repository has a clean, standard Python layout:
+
+```text
+├── .python-version      # Target Python version (>=3.11)
+├── pyproject.toml       # Build metadata, CLI entry points, and dependencies
+├── uv.lock              # Lockfile for the uv environment manager
+├── src/
+│   └── ddgo_search/
+│       ├── __init__.py  # Package metadata (defines __version__)
+│       ├── cli.py       # Typer CLI application structure, parameters, commands
+│       └── utils.py     # Resiliency, rate limiting, and output formatting helpers
+└── tests/
+    └── test_cli.py      # Pytest unit tests utilizing extensive mocking
+```
+
+---
+
+## 🏗️ Architecture & Data Flow
+
+The project is a resilient CLI wrapper around the Python `ddgs` (DuckDuckGo Search) library.
+
+### Control Flow
+1. **Invocation**: The user invokes `ddgo-search`.
+2. **Context Setup**: `main_callback()` is triggered, parsing global options (`--proxy`, `--timeout`, `--verify`, `--max-retries`). It builds a `Config` object and attaches it to the Typer context (`ctx.obj`).
+3. **Command Routing**: Typer routes execution to a specific subcommand handler (`text`, `images`, `videos`, `news`, `books`, `extract`, `fetch`).
+4. **Execution & Resiliency**: The subcommand invokes `execute_with_retry()`, supplying a search function matching the category.
+5. **Rate Limiting**: Within `execute_with_retry()`, `ensure_rate_limit()` is called. It checks the global `ddgo_search_rate.json` file to ensure a randomized gap of **1.5 to 3.0 seconds** has elapsed since the last request across any process.
+6. **Query & Proxy Rotation**: The wrapper tries to execute the query. If a proxy was provided (single, comma-separated, or file path), the query uses it. On failure, it performs exponential backoff with jitter and rotates to the next proxy.
+7. **Formatting**: If successful, results are passed to `display_results()` which maps formatting functions (`json`, `csv`, `plain`, `table`) according to the active query category.
+
+---
+
+## 🎨 Design & Style Patterns
+
+- **CLI Framework**: [Typer](https://typer.tiangolo.com/) is used for defining commands, arguments, option groups, and help text. Subcommands match `ddgs` methods directly.
+- **Terminal output**: [Rich](https://github.com/Textualize/rich) is used for rendering plain and markdown outputs.
+- **Output Formats**:
+  - `table`: Custom space-efficient ASCII table. To minimize token waste, a custom ASCII generator `format_simple_table()` is used instead of heavy tables.
+  - `json`: Standard JSON-serialized dump.
+  - `csv`: Standard CSV format.
+  - `plain`: Clean colored output optimized for terminal readability.
+
+---
+
+## 🧪 Testing Approach
+
+- **Resiliency Mocking**: Since live queries to DuckDuckGo are prone to rate-limiting and external network failures, **all CLI tests must mock the `ddgs.DDGS` class**.
+- **Instant Test Runs**: The `tests/test_cli.py` module defines an autouse fixture `mock_rate_limit` which patches `ddgo_search.utils.ensure_rate_limit` to instantly bypass rate limits, speeding up test suite execution.
+- **CliRunner**: Use Typer's `CliRunner` for invoking and verifying standard outputs and exit codes.
+
+---
+
+## ⚠️ Non-Obvious Gotchas & Quirks
+
+1. **Typo in the `ddgs` library (Video Resolution)**:
+   - The third-party `ddgs` library expects `"standart"` as the value for standard resolution instead of `"standard"`.
+   - **Do not "fix" this spelling** in `cli.py` or `VideoResolution` enum. It is defined as `STANDARD = "standart"` to correctly interface with the underlying library.
+2. **Rate Limiting Persistence**:
+   - Rate limiting relies on a file named `ddgo_search_rate.json` written to the system's temporary directory (`tempfile.gettempdir()`). If writing/reading to/from this file fails, the application fails silently to avoid blocking execution.
+3. **Proxy Input Formats**:
+   - The `--proxy` option accepts a single proxy URL, a comma-separated list of proxy URLs, or a **file path** containing proxy URLs (one per line). The parser `parse_proxies()` detects local files dynamically.
+4. **Markdown Cleaning**:
+   - Web extraction results can contain bloated markup. The `clean_markdown` utility collapses three or more consecutive blank lines into exactly two and strips trailing whitespace from all lines.
+5. **Direct Fetch (`fetch`) vs DDG Extract (`extract`)**:
+   - The `extract` command uses DuckDuckGo's internal extraction backend via `ddgs.extract`.
+   - The `fetch` command mimics Charmbracelet Crush's direct fetch tool by directly requesting the target URL via `httpx` and parsing it locally with `BeautifulSoup` and `markdownify` into `text`, `markdown`, or `html`. It also respects a `--max-size` limit (default 100KB) and truncates any content exceeding it.
+

ddgo_search-1.0.0/PKG-INFO

@@ -0,0 +1,177 @@
+Metadata-Version: 2.4
+Name: ddgo-search
+Version: 1.0.0
+Summary: Add your description here
+Requires-Python: >=3.11
+Requires-Dist: beautifulsoup4>=4.14.3
+Requires-Dist: ddgs>=9.14.4
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: markdownify>=1.2.2
+Requires-Dist: rich>=15.0.0
+Requires-Dist: typer>=0.26.6
+Description-Content-Type: text/markdown
+
+# ddgo-search
+
+A highly resilient, token-efficient, and feature-rich Command Line Interface (CLI) wrapper around the DuckDuckGo Search (`ddgs`) Python library. It features built-in proxy rotation, rate-limiting, custom token-saving ASCII rendering, webpage extraction, and direct content fetching.
+
+---
+
+## ✨ Features
+
+- **🌐 Comprehensive Query Support**: Subcommands for `text`, `images`, `videos`, `news`, `books`, and web page extraction/fetching.
+- **🔄 Resilient Proxy Rotation**: Accepts single proxy URLs, comma-separated lists, or files containing lists of proxies. Automatically rotates proxy servers sequentially on failure.
+- **⏱️ Process-Safe Rate Limiting**: Randomised delays (between 1.5s to 3.0s) are tracked globally using a system-level temporary file to ensure safe execution even when multiple commands are run concurrently.
+- **⚡ Direct Web Fetching (`fetch`)**: Inspired by Charmbracelet's `crush` tool. Directly fetches and converts webpages using `httpx`, `BeautifulSoup`, and `markdownify` into beautiful plain text, markdown, or HTML, with auto-truncation limits (e.g., 100KB) to preserve context windows.
+- **📊 Token-Efficient ASCII Layouts**: Formats console lists using space-padded ASCII dividers rather than heavy Unicode grids to save context window tokens when using LLM agents.
+
+---
+
+## 🚀 Installation
+
+Install and run using [uv](https://github.com/astral-sh/uv):
+
+```bash
+# Clone the repository
+git clone <repository-url>
+cd ddgo-search
+
+# Install dependencies and sync virtual environment
+uv sync
+```
+
+---
+
+## 📖 CLI Usage
+
+Invoke `ddgo-search` directly using `uv`:
+
+```bash
+uv run ddgo-search [GLOBAL-OPTIONS] COMMAND [ARGS]...
+```
+
+### Global Options
+
+These options must be passed *before* any subcommand:
+
+- `-p, --proxy TEXT`: Proxy URL, comma-separated list of proxy URLs, or file path containing proxies (one per line). Falls back to the `DDGS_PROXY` environment variable.
+- `-t, --timeout INTEGER`: Request timeout in seconds (default: `10`).
+- `--verify / --no-verify`: Enable/disable SSL certification verification (default: `--verify`).
+- `-r, --max-retries INTEGER`: Maximum retries upon server failures or timeouts (default: `3`).
+
+---
+
+### Commands
+
+#### 1. Text Search (`text`)
+Search the web for text results with custom formatting.
+```bash
+uv run ddgo-search text "artificial intelligence" --format plain
+uv run ddgo-search text "python programming" --format table --max-results 5
+```
+
+#### 2. Image Search (`images`)
+Query and filter DuckDuckGo images.
+```bash
+uv run ddgo-search images "cute kittens" --size Large --color Monochrome
+uv run ddgo-search images "space nebula" --layout Wide --format json
+```
+
+#### 3. Video Search (`videos`)
+Search for videos with specific duration, resolution, or license filters.
+```bash
+uv run ddgo-search videos "golang tutorial" --resolution high --duration short
+```
+*Note: The third-party library standard resolution uses the spelling `"standart"`. The CLI enum automatically handles this mapping.*
+
+#### 4. News Search (`news`)
+Query recent news.
+```bash
+uv run ddgo-search news "climate change" --timelimit w --format csv
+```
+
+#### 5. Book Search (`books`)
+Search DuckDuckGo books.
+```bash
+uv run ddgo-search books "machine learning" --max-results 10
+```
+
+#### 6. Web Page Extract (`extract`)
+Fetch and extract webpage content using DuckDuckGo's internal extraction backend.
+```bash
+uv run ddgo-search extract "https://example.com" --format text_markdown
+```
+
+#### 7. Direct Web Page Fetch (`fetch`)
+Directly fetch a URL via `httpx` and convert its HTML content locally to Markdown, clean text (excluding scripts, styles, headers, footers), or HTML. Includes auto-truncation.
+```bash
+# Direct fetch and convert to Markdown
+uv run ddgo-search fetch "https://example.com" --format markdown
+
+# Direct fetch and extract readable plain text
+uv run ddgo-search fetch "https://example.com" --format text
+
+# Direct fetch and write to file
+uv run ddgo-search fetch "https://example.com" --format markdown --output doc.md
+
+# Set custom truncation limit (e.g. 5KB)
+uv run ddgo-search fetch "https://example.com" --max-size 5120
+```
+
+---
+
+## 🤖 Codex Subagent & Skill Integration
+
+You can integrate `ddgo-search` as a custom subagent in Codex to handle all web search and page fetching tasks.
+
+### 1. Global Installation (Recommended)
+
+To allow Codex to automatically call the `ddgo-search` subagent across all your projects:
+
+1. **Install the CLI globally** so it is available from any workspace directory:
+   ```bash
+   pipx install .
+   # Or install into your global python environment
+   pip install .
+   ```
+2. **Install the Skill globally** by copying the skill directory to your user-level Codex skills folder:
+   ```bash
+   mkdir -p ~/.codex/skills/
+   cp -r skills/ddgo-search-skill ~/.codex/skills/
+   ```
+3. **Install the Subagent Configuration globally**:
+   ```bash
+   mkdir -p ~/.codex/agents/
+   cp .codex/agents/ddgo-search.toml ~/.codex/agents/
+   ```
+
+### 2. Project-level Installation
+
+If you only want this subagent available inside this project directory:
+
+1. Ensure the project is trusted in your `~/.codex/config.toml`:
+   ```toml
+   [projects."/absolute/path/to/ddgo-search"]
+   trust_level = "trusted"
+   ```
+2. The local configurations are already set up under:
+   - Skill: [SKILL.md](file:///Users/2342184/programs/ddgs-search/skills/ddgo-search-skill/SKILL.md)
+   - Subagent Config: [.codex/agents/ddgo-search.toml](file:///Users/2342184/programs/ddgs-search/.codex/agents/ddgo-search.toml)
+
+### 3. Usage
+
+Once the skill and subagent are installed globally, Codex can delegate searches automatically when prompted. You can trigger it explicitly by prompting:
+
+> "使用 `ddgo-search` 子智能体帮我查找关于..."
+
+---
+
+## 🧪 Development & Testing
+
+Run the comprehensive unit test suite:
+
+```bash
+uv run pytest
+```
+
+Our tests mock the `ddgs.DDGS` library as well as network activities to ensure the test suite runs instantly, robustly, and offline.