search-parser 0.0.1__tar.gz

search_parser-0.0.1/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,30 @@
+---
+name: Bug Report
+about: Report a bug to help us improve
+title: "[BUG] "
+labels: bug
+assignees: ''
+---
+
+**Describe the bug**
+A clear description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce:
+1. Parse HTML from '...'
+2. Use output format '...'
+3. See error
+
+**Expected behavior**
+What you expected to happen.
+
+**HTML Sample**
+If applicable, provide a minimal HTML snippet that reproduces the issue.
+
+**Environment:**
+ - OS: [e.g., Ubuntu 22.04]
+ - Python version: [e.g., 3.11]
+ - Package version: [e.g., 0.1.0]
+
+**Additional context**
+Any other context about the problem.

search_parser-0.0.1/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: Feature Request
+about: Suggest an idea for this project
+title: "[FEATURE] "
+labels: enhancement
+assignees: ''
+---
+
+**Is your feature request related to a problem?**
+A clear description of the problem. Ex. I'm frustrated when [...]
+
+**Describe the solution you'd like**
+What you want to happen.
+
+**Describe alternatives you've considered**
+Any alternative solutions or features you've considered.
+
+**Additional context**
+Any other context or screenshots about the feature request.

search_parser-0.0.1/.github/dependabot.yml

@@ -0,0 +1,16 @@
+# To get started with Dependabot version updates, you'll need to specify which
+# package ecosystems to update and where the package manifests are located.
+# Please see the documentation for all configuration options:
+# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
+
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "weekly"

search_parser-0.0.1/.github/pull_request_template.md

@@ -0,0 +1,23 @@
+## Description
+
+Brief description of the changes.
+
+## Type of Change
+
+- [ ] Bug fix
+- [ ] New feature
+- [ ] Breaking change
+- [ ] Documentation update
+
+## Checklist
+
+- [ ] My code follows the project's style guidelines
+- [ ] I have added tests that prove my fix/feature works
+- [ ] All new and existing tests pass (`uv run pytest`)
+- [ ] Linting passes (`uv run ruff check .`)
+- [ ] Type checking passes (`uv run mypy src/search_engine_parser`)
+- [ ] I have updated documentation as needed
+
+## Test Plan
+
+How were these changes tested?

search_parser-0.0.1/.github/workflows/codeql.yml

@@ -0,0 +1,101 @@
+# For most projects, this workflow file will not need changing; you simply need
+# to commit it to your repository.
+#
+# You may wish to alter this file to override the set of languages analyzed,
+# or to provide custom queries or build logic.
+#
+# ******** NOTE ********
+# We have attempted to detect the languages in your repository. Please check
+# the `language` matrix defined below to confirm you have the correct set of
+# supported CodeQL languages.
+#
+name: "CodeQL Advanced"
+
+on:
+  push:
+    branches: [ "main" ]
+  pull_request:
+    branches: [ "main" ]
+  schedule:
+    - cron: '35 10 * * 1'
+
+jobs:
+  analyze:
+    name: Analyze (${{ matrix.language }})
+    # Runner size impacts CodeQL analysis time. To learn more, please see:
+    #   - https://gh.io/recommended-hardware-resources-for-running-codeql
+    #   - https://gh.io/supported-runners-and-hardware-resources
+    #   - https://gh.io/using-larger-runners (GitHub.com only)
+    # Consider using larger runners or machines with greater resources for possible analysis time improvements.
+    runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
+    permissions:
+      # required for all workflows
+      security-events: write
+
+      # required to fetch internal or private CodeQL packs
+      packages: read
+
+      # only required for workflows in private repositories
+      actions: read
+      contents: read
+
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+        - language: actions
+          build-mode: none
+        - language: python
+          build-mode: none
+        # CodeQL supports the following values keywords for 'language': 'actions', 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'rust', 'swift'
+        # Use `c-cpp` to analyze code written in C, C++ or both
+        # Use 'java-kotlin' to analyze code written in Java, Kotlin or both
+        # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
+        # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
+        # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
+        # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
+        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
+    steps:
+    - name: Checkout repository
+      uses: actions/checkout@v4
+
+    # Add any setup steps before running the `github/codeql-action/init` action.
+    # This includes steps like installing compilers or runtimes (`actions/setup-node`
+    # or others). This is typically only required for manual builds.
+    # - name: Setup runtime (example)
+    #   uses: actions/setup-example@v1
+
+    # Initializes the CodeQL tools for scanning.
+    - name: Initialize CodeQL
+      uses: github/codeql-action/init@v4
+      with:
+        languages: ${{ matrix.language }}
+        build-mode: ${{ matrix.build-mode }}
+        # If you wish to specify custom queries, you can do so here or in a config file.
+        # By default, queries listed here will override any specified in a config file.
+        # Prefix the list here with "+" to use these queries and those in the config file.
+
+        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
+        # queries: security-extended,security-and-quality
+
+    # If the analyze step fails for one of the languages you are analyzing with
+    # "We were unable to automatically build your code", modify the matrix above
+    # to set the build mode to "manual" for that language. Then modify this step
+    # to build your code.
+    # ℹ️ Command-line programs to run using the OS shell.
+    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
+    - name: Run manual build steps
+      if: matrix.build-mode == 'manual'
+      shell: bash
+      run: |
+        echo 'If you are using a "manual" build mode for one or more of the' \
+          'languages you are analyzing, replace this with the commands to build' \
+          'your code, for example:'
+        echo '  make bootstrap'
+        echo '  make release'
+        exit 1
+
+    - name: Perform CodeQL Analysis
+      uses: github/codeql-action/analyze@v4
+      with:
+        category: "/language:${{matrix.language}}"

search_parser-0.0.1/.github/workflows/coverage-badge.yml

@@ -0,0 +1,40 @@
+name: Coverage Badge
+
+on:
+  push:
+    branches: [main]
+
+jobs:
+  coverage:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Install dependencies
+        run: uv sync --all-extras
+
+      - name: Run tests with coverage
+        run: |
+          uv run pytest --cov=search_engine_parser --cov-report=term --cov-report=json
+          COVERAGE=$(python -c "import json; print(int(float(json.load(open('coverage.json'))['totals']['percent_covered_display'])))")
+          echo "COVERAGE=${COVERAGE}%" >> $GITHUB_ENV
+          if [ "$COVERAGE" -ge 90 ]; then echo "COVERAGE_COLOR=brightgreen" >> $GITHUB_ENV;
+          elif [ "$COVERAGE" -ge 70 ]; then echo "COVERAGE_COLOR=yellow" >> $GITHUB_ENV;
+          else echo "COVERAGE_COLOR=red" >> $GITHUB_ENV; fi
+
+      - name: Create coverage badge
+        uses: schneegans/dynamic-badges-action@v1.7.0
+        with:
+          auth: ${{ secrets.GIST_SECRET }}
+          gistID: ${{ vars.GIST_ID }}
+          filename: search-parser-coverage.json
+          label: coverage
+          message: ${{ env.COVERAGE }}
+          color: ${{ env.COVERAGE_COLOR }}
+          namedLogo: python

search_parser-0.0.1/.github/workflows/lint.yml

@@ -0,0 +1,31 @@
+name: Lint
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Install dependencies
+        run: uv sync --all-extras
+
+      - name: Run ruff check
+        run: uv run ruff check .
+
+      - name: Run ruff format check
+        run: uv run ruff format --check .
+
+      - name: Run mypy
+        run: uv run mypy src/search_engine_parser

search_parser-0.0.1/.github/workflows/publish.yml

@@ -0,0 +1,26 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    permissions:
+      id-token: write
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+
+      - name: Set up Python
+        run: uv python install 3.11
+
+      - name: Build package
+        run: uv build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

search_parser-0.0.1/.github/workflows/test.yml

@@ -0,0 +1,47 @@
+name: Tests
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9", "3.10", "3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v6
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v7
+        with:
+          enable-cache: true
+
+      - name: Set up Python ${{ matrix.python-version }}
+        run: uv python install ${{ matrix.python-version }}
+
+      - name: Install dependencies
+        run: uv sync --all-extras
+
+      - name: Run tests with coverage
+        run: uv run pytest --cov=search_engine_parser --cov-report=xml --cov-report=term --cov-report=html
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v5
+        if: matrix.python-version == '3.11'
+        with:
+          file: ./coverage.xml
+          fail_ci_if_error: true
+          token: ${{ secrets.CODECOV_TOKEN }}
+
+      - name: Upload coverage HTML artifact
+        uses: actions/upload-artifact@v6
+        if: matrix.python-version == '3.11'
+        with:
+          name: coverage-report
+          path: htmlcov/
+          retention-days: 30

search_parser-0.0.1/.gitignore

@@ -0,0 +1,49 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+*.egg-info/
+*.egg
+dist/
+build/
+eggs/
+*.whl
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+.env
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+
+# Testing
+htmlcov/
+.coverage
+.coverage.*
+coverage.xml
+*.cover
+.pytest_cache/
+
+# mypy
+.mypy_cache/
+
+# ruff
+.ruff_cache/
+
+# Documentation
+site/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# uv
+
+search_parser_prompt.md

search_parser-0.0.1/.pre-commit-config.yaml

@@ -0,0 +1,18 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.1.0
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.7.0
+    hooks:
+      - id: mypy
+        additional_dependencies:
+          - pydantic>=2.0.0
+          - types-beautifulsoup4
+          - lxml-stubs
+        args: [--strict]
+        files: ^src/

search_parser-0.0.1/CHANGELOG.md

@@ -0,0 +1,28 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2025-01-01
+
+### Added
+
+- Initial release of `search-parser`.
+- Core parsing framework with `BaseParser` and `SearchResult` data model.
+- Google search results parser with auto-detection.
+- Bing search results parser with auto-detection.
+- DuckDuckGo search results parser with auto-detection.
+- Auto-detection of search engine from raw HTML content.
+- Three output formats: JSON, Markdown, and Python dict.
+- Command-line interface (`search-parser parse`).
+- Comprehensive test suite with HTML fixtures.
+- Documentation site using MkDocs with Material theme.
+- CI/CD workflows for testing, linting, coverage, and publishing.
+- Pre-commit hooks for ruff and mypy.
+
+[Unreleased]: https://github.com/getlinksc/search-parser/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/getlinksc/search-parser/releases/tag/v0.1.0

search_parser-0.0.1/CLAUDE.md

@@ -0,0 +1,40 @@
+# CLAUDE.md - Search Engine Parser
+
+## Project Overview
+Python library for parsing search engine HTML results into structured data (JSON, Markdown, dict).
+
+## Architecture
+- **Strategy Pattern**: Each search engine has its own parser implementing `BaseParser`
+- **Plugin Architecture**: New engines added by creating a parser class and registering it
+- **Layers**: Detection → Parsing → Formatting
+
+## Package Manager
+Using `uv` for fast, modern Python package management.
+
+## Common Commands
+```bash
+uv sync --all-extras          # Install/sync all dependencies
+uv run pytest                 # Run tests
+uv run pytest --cov=search_engine_parser --cov-report=term  # Tests with coverage
+uv run ruff check .           # Lint code
+uv run ruff format .          # Format code
+uv run mypy src/search_engine_parser  # Type check
+```
+
+## Adding a New Search Engine Parser
+1. Create `src/search_engine_parser/parsers/engine_name.py`
+2. Implement class extending `BaseParser` with `engine_name`, `parse()`, and `can_parse()` methods
+3. Register in `src/search_engine_parser/parsers/__init__.py`
+4. Add HTML test fixtures in `tests/fixtures/engine_name/`
+5. Write unit tests in `tests/unit/test_engine_name_parser.py`
+
+## Coding Standards
+- Full type hints on all functions (mypy --strict)
+- Google-style docstrings on public APIs
+- `ruff check` + `ruff format` (line-length 100)
+- Functions under 15 lines when possible
+
+## Testing
+- pytest with pytest-cov, target >90% coverage
+- HTML fixtures in tests/fixtures/ for each engine
+- Unit tests per parser, integration tests for full workflows

search_parser-0.0.1/CODE_OF_CONDUCT.md

@@ -0,0 +1,40 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+We as members, contributors, and leaders pledge to make participation in our
+community a harassment-free experience for everyone, regardless of age, body
+size, visible or invisible disability, ethnicity, sex characteristics, gender
+identity and expression, level of experience, education, socio-economic status,
+nationality, personal appearance, race, caste, color, religion, or sexual
+identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to a positive environment:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior:
+
+* The use of sexualized language or imagery, and sexual attention or advances of any kind
+* Trolling, insulting or derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be
+reported to the project maintainers. All complaints will be reviewed and
+investigated promptly and fairly.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant](https://www.contributor-covenant.org/),
+version 2.1, available at
+https://www.contributor-covenant.org/version/2/1/code_of_conduct.html.