toolkitx 0.0.4__tar.gz → 0.0.7__tar.gz

{toolkitx-0.0.4 → toolkitx-0.0.7}/.github/workflows/python-publish.yml

@@ -10,6 +10,9 @@ on:
     branches:
       - main   # 在向 main 分支发起 Pull Request 时运行测试
 
+env:
+  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: true
+
 jobs:
   test:
     runs-on: ubuntu-latest
@@ -24,8 +27,8 @@ jobs:
     - name: Install uv
       uses: astral-sh/setup-uv@v5
       with:
-        # Install a specific version of uv.
-        version: "0.7.3"
+        # Install the latest stable version of uv
+        enable-cache: true
 
     - name: "Set up Python"
       uses: actions/setup-python@v5
@@ -48,7 +51,7 @@ jobs:
     permissions:
       # 对于受信任的发布 (Trusted Publishing) 至 PyPI，此权限是必需的
       id-token: write
-      contents: read # 检出仓库需要
+      contents: write # 检出仓库并创建 Release 需要写权限
 
     steps:
     - name: Checkout repository
@@ -57,8 +60,7 @@ jobs:
     - name: Install uv
       uses: astral-sh/setup-uv@v5
       with:
-        # Install a specific version of uv.
-        version: "0.7.3"
+        enable-cache: true
 
     - name: "Set up Python"
       uses: actions/setup-python@v5

{toolkitx-0.0.4 → toolkitx-0.0.7}/.gitignore

@@ -145,6 +145,7 @@ venv.bak/
 
 # mkdocs documentation
 /site
+/docs/reference/
 
 # mypy
 .mypy_cache/

toolkitx-0.0.7/.readthedocs.yaml

@@ -0,0 +1,16 @@
+version: 2
+
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.12"
+
+mkdocs:
+  configuration: mkdocs.yml
+
+python:
+  install:
+    - method: uv
+      command: sync
+      extras:
+        - docs

toolkitx-0.0.7/.vscode/settings.json

@@ -0,0 +1,10 @@
+{
+    "workbench.editorAssociations": {
+        "*.copilotmd": "vscode.markdown.preview.editor",
+        "*.ipynb": "jupyter.notebook.ipynb",
+        "*.parquet": "duckdb.dataFileViewer",
+        "*.csv": "duckdb.dataFileViewer",
+        "*.tsv": "duckdb.dataFileViewer",
+        "*.xlsx": "duckdb.dataFileViewer"
+    }
+}

toolkitx-0.0.7/PKG-INFO

@@ -0,0 +1,154 @@
+Metadata-Version: 2.4
+Name: toolkitx
+Version: 0.0.7
+Summary: A personal Python toolkit for common tasks
+Project-URL: Homepage, https://github.com/ider-zh/toolkitx
+Project-URL: Bug Tracker, https://github.com/ider-zh/toolkitx/issues
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: beautifulsoup4>=4.12.0
+Requires-Dist: diskcache>=5.6.3
+Requires-Dist: httpx>=0.28.1
+Requires-Dist: markdownify>=0.14.0
+Requires-Dist: polars>=1.38.1
+Requires-Dist: pydantic>=2.12.5
+Requires-Dist: tencentcloud-sdk-python<3.1.106,>=3.1.50
+Requires-Dist: tqdm>=4.67.3
+Provides-Extra: dev
+Requires-Dist: pytest-dotenv>=0.5.2; extra == 'dev'
+Requires-Dist: pytest>=9.0.2; extra == 'dev'
+Requires-Dist: ruff>=0.9.0; extra == 'dev'
+Provides-Extra: docs
+Requires-Dist: mkdocs-gen-files>=0.5.0; extra == 'docs'
+Requires-Dist: mkdocs-literate-nav>=0.6.0; extra == 'docs'
+Requires-Dist: mkdocs-material>=9.5.0; extra == 'docs'
+Requires-Dist: mkdocs-section-index>=0.3.0; extra == 'docs'
+Requires-Dist: mkdocs<2.0.0,>=1.6.0; extra == 'docs'
+Requires-Dist: mkdocstrings[python]>=0.25.0; extra == 'docs'
+Description-Content-Type: text/markdown
+
+# ToolkitX
+
+[![Documentation Status](https://readthedocs.org/projects/toolkitx/badge/?version=latest)](https://toolkitx.readthedocs.io/en/latest/?badge=latest)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+
+A personal Python toolkit for common tasks. This package provides robust utility functions to simplify common development workflows, focusing on text processing, HTML conversion, and task resilience.
+
+📖 **Full Documentation**: [https://toolkitx.readthedocs.io/en/latest/](https://toolkitx.readthedocs.io/en/latest/)
+
+## Features
+
+*   **HTML Utilities** (`toolkitx.html_utils`):
+    *   `html_to_markdown`: Robust HTML to Markdown conversion. Handles complex tables (colspan/rowspan) by expansion and serializes nested tables to JSON for better LLM/Agent understanding. Automatically promotes the first row to header if missing.
+    
+*   **Text Utilities** (`toolkitx.text_utils`):
+    *   `truncate_text_smart`: Smartly truncates text by characters or words, attempting to preserve sentence or word boundaries with configurable tolerance.
+    *   `split_text_by_word_count`: Splits long text into overlapping chunks based on word count.
+    
+*   **Task Utilities** (`toolkitx.task_utils`):
+    *   `with_resilience`: A decorator for API resilience with rate limiting (QPS), exponential backoff retry, and jitter.
+    *   `PersistentTaskQueue`: A persistent task queue with SQLite backend, supporting concurrent processing, automatic retry, crash recovery, and graceful shutdown.
+    
+*   **Experimental Translator** (`toolkitx.lab.translator`):
+    *   `Translator`: A class providing translation capabilities using Baidu or Tencent translation APIs, with disk-based caching.
+
+## Installation
+
+We recommend using [uv](https://github.com/astral-sh/uv) for fast and reliable dependency management.
+
+```bash
+# Clone the repository
+git clone https://github.com/ider-zh/toolkitx.git
+cd toolkitx
+
+# Install with development dependencies
+uv pip install -e ".[dev,docs]"
+```
+
+## Usage
+
+### HTML to Markdown (Robust Table Support)
+
+```python
+from toolkitx import html_to_markdown
+
+# Handles merged cells (colspan/rowspan) and nested tables
+html = """
+<table>
+  <tr><td colspan="2">Merged Header</td></tr>
+  <tr><td>Cell 1</td><td>Cell 2</td></tr>
+  <tr>
+    <td>Outer</td>
+    <td>
+      <table><tr><td>Nested</td></tr></table>
+    </td>
+  </tr>
+</table>
+"""
+
+md = html_to_markdown(html)
+print(md)
+```
+
+### Text Smart Truncation
+
+```python
+from toolkitx import truncate_text_smart
+
+text = "Hello World. This is a long sentence that should be truncated smartly."
+# Strips trailing punctuation automatically
+truncated = truncate_text_smart(text, limit=12) 
+print(truncated) # Output: 'Hello World...'
+```
+
+### Task Resilience Decorator
+
+```python
+from toolkitx import with_resilience
+
+@with_resilience(qps=2.0, max_retries=3)
+def fetch_data(url):
+    # This function will be rate-limited and retried automatically
+    pass
+```
+
+## Development
+
+### Running Tests
+```bash
+# Run unit tests
+make test
+
+# Run documentation tests (verify examples in docstrings)
+make test-docs
+```
+
+### Documentation
+```bash
+# Preview documentation locally
+make docs-serve
+
+# Build static documentation site
+make docs-build
+```
+
+## Changelog
+
+### v0.0.7 (2026-07-09)
+- **Bug Fix**: `html_to_markdown` no longer crashes (`IndexError: list index out of range`) when expanding tables that contain empty cells (e.g. `<td rowspan="2"></td>`). Empty cells now render as zero-width placeholders while preserving colspan/rowspan expansion.
+
+### v0.0.6 (2026-07-09)
+- **Dependency**: Pinned `tencentcloud-sdk-python` to `<3.1.106`. The Tencent SDK removed the TMT `TextTranslate` API (incl. `TextTranslateRequest`) starting in `3.1.106`, which would break `toolkitx.lab.translator`. The text-translation engine now stays on the last compatible release.
+
+### v0.0.5 (2026-05-30)
+- **New Feature**: Added `html_utils` with robust `html_to_markdown` converter.
+- **Improved**: `truncate_text_smart` now strips trailing punctuation before appending suffix.
+- **Documentation**: Established full automated documentation system with MkDocs, Material theme, and Read the Docs integration.
+- **Verifiable Docs**: Added `doctest` examples to all core functions and a `make test-docs` target.
+- **Workflow**: Integrated `ruff` for linting and formatting.
+- **Dependency Management**: Fully transitioned to `uv` and pinned `mkdocs` for stability.
+
+### v0.0.4 (2026-03-07)
+- Added `task_utils` module with `with_resilience` decorator and `PersistentTaskQueue`.
+- Added `polars`, `pydantic`, and `tqdm` as dependencies.

toolkitx-0.0.7/README.md

@@ -0,0 +1,125 @@
+# ToolkitX
+
+[![Documentation Status](https://readthedocs.org/projects/toolkitx/badge/?version=latest)](https://toolkitx.readthedocs.io/en/latest/?badge=latest)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+
+A personal Python toolkit for common tasks. This package provides robust utility functions to simplify common development workflows, focusing on text processing, HTML conversion, and task resilience.
+
+📖 **Full Documentation**: [https://toolkitx.readthedocs.io/en/latest/](https://toolkitx.readthedocs.io/en/latest/)
+
+## Features
+
+*   **HTML Utilities** (`toolkitx.html_utils`):
+    *   `html_to_markdown`: Robust HTML to Markdown conversion. Handles complex tables (colspan/rowspan) by expansion and serializes nested tables to JSON for better LLM/Agent understanding. Automatically promotes the first row to header if missing.
+    
+*   **Text Utilities** (`toolkitx.text_utils`):
+    *   `truncate_text_smart`: Smartly truncates text by characters or words, attempting to preserve sentence or word boundaries with configurable tolerance.
+    *   `split_text_by_word_count`: Splits long text into overlapping chunks based on word count.
+    
+*   **Task Utilities** (`toolkitx.task_utils`):
+    *   `with_resilience`: A decorator for API resilience with rate limiting (QPS), exponential backoff retry, and jitter.
+    *   `PersistentTaskQueue`: A persistent task queue with SQLite backend, supporting concurrent processing, automatic retry, crash recovery, and graceful shutdown.
+    
+*   **Experimental Translator** (`toolkitx.lab.translator`):
+    *   `Translator`: A class providing translation capabilities using Baidu or Tencent translation APIs, with disk-based caching.
+
+## Installation
+
+We recommend using [uv](https://github.com/astral-sh/uv) for fast and reliable dependency management.
+
+```bash
+# Clone the repository
+git clone https://github.com/ider-zh/toolkitx.git
+cd toolkitx
+
+# Install with development dependencies
+uv pip install -e ".[dev,docs]"
+```
+
+## Usage
+
+### HTML to Markdown (Robust Table Support)
+
+```python
+from toolkitx import html_to_markdown
+
+# Handles merged cells (colspan/rowspan) and nested tables
+html = """
+<table>
+  <tr><td colspan="2">Merged Header</td></tr>
+  <tr><td>Cell 1</td><td>Cell 2</td></tr>
+  <tr>
+    <td>Outer</td>
+    <td>
+      <table><tr><td>Nested</td></tr></table>
+    </td>
+  </tr>
+</table>
+"""
+
+md = html_to_markdown(html)
+print(md)
+```
+
+### Text Smart Truncation
+
+```python
+from toolkitx import truncate_text_smart
+
+text = "Hello World. This is a long sentence that should be truncated smartly."
+# Strips trailing punctuation automatically
+truncated = truncate_text_smart(text, limit=12) 
+print(truncated) # Output: 'Hello World...'
+```
+
+### Task Resilience Decorator
+
+```python
+from toolkitx import with_resilience
+
+@with_resilience(qps=2.0, max_retries=3)
+def fetch_data(url):
+    # This function will be rate-limited and retried automatically
+    pass
+```
+
+## Development
+
+### Running Tests
+```bash
+# Run unit tests
+make test
+
+# Run documentation tests (verify examples in docstrings)
+make test-docs
+```
+
+### Documentation
+```bash
+# Preview documentation locally
+make docs-serve
+
+# Build static documentation site
+make docs-build
+```
+
+## Changelog
+
+### v0.0.7 (2026-07-09)
+- **Bug Fix**: `html_to_markdown` no longer crashes (`IndexError: list index out of range`) when expanding tables that contain empty cells (e.g. `<td rowspan="2"></td>`). Empty cells now render as zero-width placeholders while preserving colspan/rowspan expansion.
+
+### v0.0.6 (2026-07-09)
+- **Dependency**: Pinned `tencentcloud-sdk-python` to `<3.1.106`. The Tencent SDK removed the TMT `TextTranslate` API (incl. `TextTranslateRequest`) starting in `3.1.106`, which would break `toolkitx.lab.translator`. The text-translation engine now stays on the last compatible release.
+
+### v0.0.5 (2026-05-30)
+- **New Feature**: Added `html_utils` with robust `html_to_markdown` converter.
+- **Improved**: `truncate_text_smart` now strips trailing punctuation before appending suffix.
+- **Documentation**: Established full automated documentation system with MkDocs, Material theme, and Read the Docs integration.
+- **Verifiable Docs**: Added `doctest` examples to all core functions and a `make test-docs` target.
+- **Workflow**: Integrated `ruff` for linting and formatting.
+- **Dependency Management**: Fully transitioned to `uv` and pinned `mkdocs` for stability.
+
+### v0.0.4 (2026-03-07)
+- Added `task_utils` module with `with_resilience` decorator and `PersistentTaskQueue`.
+- Added `polars`, `pydantic`, and `tqdm` as dependencies.

toolkitx-0.0.7/docs/index.md

@@ -0,0 +1,14 @@
+# Welcome to ToolkitX
+
+ToolkitX is a personal Python toolkit for common tasks, focusing on resilience, text processing, and lab tools like translators.
+
+## Modules
+
+- **Task Utils**: Resilience decorators and persistent task queues.
+- **Text Utils**: Smart text truncation and word-count based splitting.
+- **HTML Utils**: Robust HTML to Markdown conversion with advanced table support.
+- **Lab**: Experimental tools including Baidu and Tencent translators.
+
+## Getting Started
+
+Check the [API Reference](reference/toolkitx/index.md) for detailed documentation of all modules.

toolkitx-0.0.7/docs/superpowers/plans/2026-05-30-auto-docs.md

@@ -0,0 +1,225 @@
+# Automated Python API Documentation Implementation Plan
+
+> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
+
+**Goal:** Establish a zero-maintenance documentation system using MkDocs and mkdocstrings that automatically generates API reference from source code.
+
+**Architecture:**
+- **MkDocs + Material Theme:** Modern UI for documentation.
+- **mkdocstrings:** Extracts documentation directly from Python code and docstrings.
+- **mkdocs-gen-files:** Automatically generates Markdown files for every module in `toolkitx/`.
+- **mkdocs-literate-nav + mkdocs-section-index:** Automates navigation sidebar generation.
+
+**Tech Stack:** Python, uv, MkDocs, Material for MkDocs, mkdocstrings, mkdocs-gen-files.
+
+---
+
+### Task 1: Setup Dependencies
+
+**Files:**
+- Modify: `pyproject.toml`
+
+- [ ] **Step 1: Add documentation dependencies to `pyproject.toml`**
+
+Add `docs` optional dependencies:
+```toml
+[project.optional-dependencies]
+dev = [
+    "pytest>=9.0.2",
+    "pytest-dotenv>=0.5.2",
+    "ruff>=0.9.0",
+]
+docs = [
+    "mkdocs>=1.6.0",
+    "mkdocs-material>=9.5.0",
+    "mkdocstrings[python]>=0.25.0",
+    "mkdocs-gen-files>=0.5.0",
+    "mkdocs-literate-nav>=0.6.0",
+    "mkdocs-section-index>=0.3.0",
+]
+```
+
+- [ ] **Step 2: Update lockfile**
+
+Run: `uv lock`
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add pyproject.toml uv.lock
+git commit -m "chore: add documentation dependencies"
+```
+
+### Task 2: Configure MkDocs
+
+**Files:**
+- Create: `mkdocs.yml`
+- Create: `docs/index.md`
+
+- [ ] **Step 1: Create `mkdocs.yml`**
+
+```yaml
+site_name: ToolkitX Documentation
+site_url: https://github.com/ider-zh/toolkitx
+repo_url: https://github.com/ider-zh/toolkitx
+edit_uri: edit/main/docs/
+
+theme:
+  name: material
+  palette:
+    - scheme: default
+      primary: indigo
+      accent: indigo
+      toggle:
+        icon: material/brightness-7
+        name: Switch to dark mode
+    - scheme: slate
+      primary: indigo
+      accent: indigo
+      toggle:
+        icon: material/brightness-4
+        name: Switch to light mode
+  features:
+    - navigation.tabs
+    - navigation.sections
+    - navigation.top
+    - search.suggest
+    - search.highlight
+    - content.code.copy
+
+plugins:
+  - search
+  - gen-files:
+      scripts:
+        - scripts/gen_ref_pages.py
+  - literate-nav:
+      nav_file: SUMMARY.md
+  - section-index
+  - mkdocstrings:
+      handlers:
+        python:
+          paths: [.]
+          options:
+            docstring_style: google
+            show_source: true
+            show_root_heading: true
+            show_category_heading: true
+
+markdown_extensions:
+  - admonition
+  - pymdownx.highlight:
+      anchor_linenums: true
+  - pymdownx.inlinehilite
+  - pymdownx.snippets
+  - pymdownx.superfences
+  - pymdownx.tabbed:
+      alternate_style: true
+```
+
+- [ ] **Step 2: Create `docs/index.md`**
+
+```markdown
+# Welcome to ToolkitX
+
+ToolkitX is a personal Python toolkit for common tasks, focusing on resilience, text processing, and lab tools like translators.
+
+## Modules
+
+- **Task Utils**: Resilience decorators and persistent task queues.
+- **Text Utils**: Smart text truncation and word-count based splitting.
+- **Lab**: Experimental tools including Baidu and Tencent translators.
+
+## Getting Started
+
+Check the [API Reference](reference/toolkitx/index.md) for detailed documentation of all modules.
+```
+
+- [ ] **Step 3: Commit**
+
+```bash
+git add mkdocs.yml docs/index.md
+git commit -m "chore: configure mkdocs and create index page"
+```
+
+### Task 3: Implement Automated API Reference Generation
+
+**Files:**
+- Create: `scripts/gen_ref_pages.py`
+
+- [ ] **Step 1: Create `scripts/gen_ref_pages.py`**
+
+```python
+"""Generate the code reference pages and navigation."""
+
+from pathlib import Path
+import mkdocs_gen_files
+
+nav = mkdocs_gen_files.Nav()
+
+root = Path(__file__).parent.parent
+src = root / "toolkitx"
+
+for path in sorted(src.rglob("*.py")):
+    module_path = path.relative_to(root).with_suffix("")
+    doc_path = path.relative_to(root).with_suffix(".md")
+    full_doc_path = Path("reference", doc_path)
+
+    parts = tuple(module_path.parts)
+
+    if parts[-1] == "__init__":
+        parts = parts[:-1]
+        doc_path = doc_path.with_name("index.md")
+        full_doc_path = full_doc_path.with_name("index.md")
+    elif parts[-1] == "__main__":
+        continue
+
+    nav[parts] = doc_path.as_posix()
+
+    with mkdocs_gen_files.open(full_doc_path, "w") as fd:
+        ident = ".".join(parts)
+        fd.write(f"::: {ident}")
+
+    mkdocs_gen_files.set_edit_path(full_doc_path, path.relative_to(root))
+
+with mkdocs_gen_files.open("reference/SUMMARY.md", "w") as nav_file:
+    nav_file.writelines(nav.build_literate_nav())
+```
+
+- [ ] **Step 2: Commit**
+
+```bash
+mkdir -p scripts
+git add scripts/gen_ref_pages.py
+git commit -m "feat: add automatic api reference generation script"
+```
+
+### Task 4: Update Makefile and Verify
+
+**Files:**
+- Modify: `makefile`
+
+- [ ] **Step 1: Add documentation targets to `makefile`**
+
+```makefile
+docs-serve:
+	uv run --extra docs mkdocs serve
+
+docs-build:
+	uv run --extra docs mkdocs build
+```
+
+- [ ] **Step 2: Verify local documentation server**
+
+Run: `make docs-serve` (and check if it starts successfully, manually stop it after check)
+
+- [ ] **Step 3: Verify build**
+
+Run: `make docs-build`
+Expected: `site/` directory created with HTML files.
+
+- [ ] **Step 4: Commit**
+
+```bash
+git add makefile
+git commit -m "chore: add documentation targets to makefile"
+```