@ngxtm/devkit 3.0.3 → 3.2.0

package/skills/python-packaging/resources/implementation-playbook.md

@@ -0,0 +1,869 @@
+# Python Packaging Implementation Playbook
+
+This file contains detailed patterns, checklists, and code samples referenced by the skill.
+
+# Python Packaging
+
+Comprehensive guide to creating, structuring, and distributing Python packages using modern packaging tools, pyproject.toml, and publishing to PyPI.
+
+## When to Use This Skill
+
+- Creating Python libraries for distribution
+- Building command-line tools with entry points
+- Publishing packages to PyPI or private repositories
+- Setting up Python project structure
+- Creating installable packages with dependencies
+- Building wheels and source distributions
+- Versioning and releasing Python packages
+- Creating namespace packages
+- Implementing package metadata and classifiers
+
+## Core Concepts
+
+### 1. Package Structure
+- **Source layout**: `src/package_name/` (recommended)
+- **Flat layout**: `package_name/` (simpler but less flexible)
+- **Package metadata**: pyproject.toml, setup.py, or setup.cfg
+- **Distribution formats**: wheel (.whl) and source distribution (.tar.gz)
+
+### 2. Modern Packaging Standards
+- **PEP 517/518**: Build system requirements
+- **PEP 621**: Metadata in pyproject.toml
+- **PEP 660**: Editable installs
+- **pyproject.toml**: Single source of configuration
+
+### 3. Build Backends
+- **setuptools**: Traditional, widely used
+- **hatchling**: Modern, opinionated
+- **flit**: Lightweight, for pure Python
+- **poetry**: Dependency management + packaging
+
+### 4. Distribution
+- **PyPI**: Python Package Index (public)
+- **TestPyPI**: Testing before production
+- **Private repositories**: JFrog, AWS CodeArtifact, etc.
+
+## Quick Start
+
+### Minimal Package Structure
+
+```
+my-package/
+├── pyproject.toml
+├── README.md
+├── LICENSE
+├── src/
+│   └── my_package/
+│       ├── __init__.py
+│       └── module.py
+└── tests/
+    └── test_module.py
+```
+
+### Minimal pyproject.toml
+
+```toml
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "my-package"
+version = "0.1.0"
+description = "A short description"
+authors = [{name = "Your Name", email = "you@example.com"}]
+readme = "README.md"
+requires-python = ">=3.8"
+dependencies = [
+    "requests>=2.28.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "black>=22.0",
+]
+```
+
+## Package Structure Patterns
+
+### Pattern 1: Source Layout (Recommended)
+
+```
+my-package/
+├── pyproject.toml
+├── README.md
+├── LICENSE
+├── .gitignore
+├── src/
+│   └── my_package/
+│       ├── __init__.py
+│       ├── core.py
+│       ├── utils.py
+│       └── py.typed          # For type hints
+├── tests/
+│   ├── __init__.py
+│   ├── test_core.py
+│   └── test_utils.py
+└── docs/
+    └── index.md
+```
+
+**Advantages:**
+- Prevents accidentally importing from source
+- Cleaner test imports
+- Better isolation
+
+**pyproject.toml for source layout:**
+```toml
+[tool.setuptools.packages.find]
+where = ["src"]
+```
+
+### Pattern 2: Flat Layout
+
+```
+my-package/
+├── pyproject.toml
+├── README.md
+├── my_package/
+│   ├── __init__.py
+│   └── module.py
+└── tests/
+    └── test_module.py
+```
+
+**Simpler but:**
+- Can import package without installing
+- Less professional for libraries
+
+### Pattern 3: Multi-Package Project
+
+```
+project/
+├── pyproject.toml
+├── packages/
+│   ├── package-a/
+│   │   └── src/
+│   │       └── package_a/
+│   └── package-b/
+│       └── src/
+│           └── package_b/
+└── tests/
+```
+
+## Complete pyproject.toml Examples
+
+### Pattern 4: Full-Featured pyproject.toml
+
+```toml
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "my-awesome-package"
+version = "1.0.0"
+description = "An awesome Python package"
+readme = "README.md"
+requires-python = ">=3.8"
+license = {text = "MIT"}
+authors = [
+    {name = "Your Name", email = "you@example.com"},
+]
+maintainers = [
+    {name = "Maintainer Name", email = "maintainer@example.com"},
+]
+keywords = ["example", "package", "awesome"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+]
+
+dependencies = [
+    "requests>=2.28.0,<3.0.0",
+    "click>=8.0.0",
+    "pydantic>=2.0.0",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0.0",
+    "pytest-cov>=4.0.0",
+    "black>=23.0.0",
+    "ruff>=0.1.0",
+    "mypy>=1.0.0",
+]
+docs = [
+    "sphinx>=5.0.0",
+    "sphinx-rtd-theme>=1.0.0",
+]
+all = [
+    "my-awesome-package[dev,docs]",
+]
+
+[project.urls]
+Homepage = "https://github.com/username/my-awesome-package"
+Documentation = "https://my-awesome-package.readthedocs.io"
+Repository = "https://github.com/username/my-awesome-package"
+"Bug Tracker" = "https://github.com/username/my-awesome-package/issues"
+Changelog = "https://github.com/username/my-awesome-package/blob/main/CHANGELOG.md"
+
+[project.scripts]
+my-cli = "my_package.cli:main"
+awesome-tool = "my_package.tools:run"
+
+[project.entry-points."my_package.plugins"]
+plugin1 = "my_package.plugins:plugin1"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+zip-safe = false
+
+[tool.setuptools.packages.find]
+where = ["src"]
+include = ["my_package*"]
+exclude = ["tests*"]
+
+[tool.setuptools.package-data]
+my_package = ["py.typed", "*.pyi", "data/*.json"]
+
+# Black configuration
+[tool.black]
+line-length = 100
+target-version = ["py38", "py39", "py310", "py311"]
+include = '\.pyi?$'
+
+# Ruff configuration
+[tool.ruff]
+line-length = 100
+target-version = "py38"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "N", "W", "UP"]
+
+# MyPy configuration
+[tool.mypy]
+python_version = "3.8"
+warn_return_any = true
+warn_unused_configs = true
+disallow_untyped_defs = true
+
+# Pytest configuration
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+python_files = ["test_*.py"]
+addopts = "-v --cov=my_package --cov-report=term-missing"
+
+# Coverage configuration
+[tool.coverage.run]
+source = ["src"]
+omit = ["*/tests/*"]
+
+[tool.coverage.report]
+exclude_lines = [
+    "pragma: no cover",
+    "def __repr__",
+    "raise AssertionError",
+    "raise NotImplementedError",
+]
+```
+
+### Pattern 5: Dynamic Versioning
+
+```toml
+[build-system]
+requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "my-package"
+dynamic = ["version"]
+description = "Package with dynamic version"
+
+[tool.setuptools.dynamic]
+version = {attr = "my_package.__version__"}
+
+# Or use setuptools-scm for git-based versioning
+[tool.setuptools_scm]
+write_to = "src/my_package/_version.py"
+```
+
+**In __init__.py:**
+```python
+# src/my_package/__init__.py
+__version__ = "1.0.0"
+
+# Or with setuptools-scm
+from importlib.metadata import version
+__version__ = version("my-package")
+```
+
+## Command-Line Interface (CLI) Patterns
+
+### Pattern 6: CLI with Click
+
+```python
+# src/my_package/cli.py
+import click
+
+@click.group()
+@click.version_option()
+def cli():
+    """My awesome CLI tool."""
+    pass
+
+@cli.command()
+@click.argument("name")
+@click.option("--greeting", default="Hello", help="Greeting to use")
+def greet(name: str, greeting: str):
+    """Greet someone."""
+    click.echo(f"{greeting}, {name}!")
+
+@cli.command()
+@click.option("--count", default=1, help="Number of times to repeat")
+def repeat(count: int):
+    """Repeat a message."""
+    for i in range(count):
+        click.echo(f"Message {i + 1}")
+
+def main():
+    """Entry point for CLI."""
+    cli()
+
+if __name__ == "__main__":
+    main()
+```
+
+**Register in pyproject.toml:**
+```toml
+[project.scripts]
+my-tool = "my_package.cli:main"
+```
+
+**Usage:**
+```bash
+pip install -e .
+my-tool greet World
+my-tool greet Alice --greeting="Hi"
+my-tool repeat --count=3
+```
+
+### Pattern 7: CLI with argparse
+
+```python
+# src/my_package/cli.py
+import argparse
+import sys
+
+def main():
+    """Main CLI entry point."""
+    parser = argparse.ArgumentParser(
+        description="My awesome tool",
+        prog="my-tool"
+    )
+
+    parser.add_argument(
+        "--version",
+        action="version",
+        version="%(prog)s 1.0.0"
+    )
+
+    subparsers = parser.add_subparsers(dest="command", help="Commands")
+
+    # Add subcommand
+    process_parser = subparsers.add_parser("process", help="Process data")
+    process_parser.add_argument("input_file", help="Input file path")
+    process_parser.add_argument(
+        "--output", "-o",
+        default="output.txt",
+        help="Output file path"
+    )
+
+    args = parser.parse_args()
+
+    if args.command == "process":
+        process_data(args.input_file, args.output)
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+def process_data(input_file: str, output_file: str):
+    """Process data from input to output."""
+    print(f"Processing {input_file} -> {output_file}")
+
+if __name__ == "__main__":
+    main()
+```
+
+## Building and Publishing
+
+### Pattern 8: Build Package Locally
+
+```bash
+# Install build tools
+pip install build twine
+
+# Build distribution
+python -m build
+
+# This creates:
+# dist/
+#   my-package-1.0.0.tar.gz (source distribution)
+#   my_package-1.0.0-py3-none-any.whl (wheel)
+
+# Check the distribution
+twine check dist/*
+```
+
+### Pattern 9: Publishing to PyPI
+
+```bash
+# Install publishing tools
+pip install twine
+
+# Test on TestPyPI first
+twine upload --repository testpypi dist/*
+
+# Install from TestPyPI to test
+pip install --index-url https://test.pypi.org/simple/ my-package
+
+# If all good, publish to PyPI
+twine upload dist/*
+```
+
+**Using API tokens (recommended):**
+```bash
+# Create ~/.pypirc
+[distutils]
+index-servers =
+    pypi
+    testpypi
+
+[pypi]
+username = __token__
+password = pypi-...your-token...
+
+[testpypi]
+username = __token__
+password = pypi-...your-test-token...
+```
+
+### Pattern 10: Automated Publishing with GitHub Actions
+
+```yaml
+# .github/workflows/publish.yml
+name: Publish to PyPI
+
+on:
+  release:
+    types: [created]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Set up Python
+        uses: actions/setup-python@v4
+        with:
+          python-version: "3.11"
+
+      - name: Install dependencies
+        run: |
+          pip install build twine
+
+      - name: Build package
+        run: python -m build
+
+      - name: Check package
+        run: twine check dist/*
+
+      - name: Publish to PyPI
+        env:
+          TWINE_USERNAME: __token__
+          TWINE_PASSWORD: ${{ secrets.PYPI_API_TOKEN }}
+        run: twine upload dist/*
+```
+
+## Advanced Patterns
+
+### Pattern 11: Including Data Files
+
+```toml
+[tool.setuptools.package-data]
+my_package = [
+    "data/*.json",
+    "templates/*.html",
+    "static/css/*.css",
+    "py.typed",
+]
+```
+
+**Accessing data files:**
+```python
+# src/my_package/loader.py
+from importlib.resources import files
+import json
+
+def load_config():
+    """Load configuration from package data."""
+    config_file = files("my_package").joinpath("data/config.json")
+    with config_file.open() as f:
+        return json.load(f)
+
+# Python 3.9+
+from importlib.resources import files
+
+data = files("my_package").joinpath("data/file.txt").read_text()
+```
+
+### Pattern 12: Namespace Packages
+
+**For large projects split across multiple repositories:**
+
+```
+# Package 1: company-core
+company/
+└── core/
+    ├── __init__.py
+    └── models.py
+
+# Package 2: company-api
+company/
+└── api/
+    ├── __init__.py
+    └── routes.py
+```
+
+**Do NOT include __init__.py in the namespace directory (company/):**
+
+```toml
+# company-core/pyproject.toml
+[project]
+name = "company-core"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["company.core*"]
+
+# company-api/pyproject.toml
+[project]
+name = "company-api"
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["company.api*"]
+```
+
+**Usage:**
+```python
+# Both packages can be imported under same namespace
+from company.core import models
+from company.api import routes
+```
+
+### Pattern 13: C Extensions
+
+```toml
+[build-system]
+requires = ["setuptools>=61.0", "wheel", "Cython>=0.29"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+ext-modules = [
+    {name = "my_package.fast_module", sources = ["src/fast_module.c"]},
+]
+```
+
+**Or with setup.py:**
+```python
+# setup.py
+from setuptools import setup, Extension
+
+setup(
+    ext_modules=[
+        Extension(
+            "my_package.fast_module",
+            sources=["src/fast_module.c"],
+            include_dirs=["src/include"],
+        )
+    ]
+)
+```
+
+## Version Management
+
+### Pattern 14: Semantic Versioning
+
+```python
+# src/my_package/__init__.py
+__version__ = "1.2.3"
+
+# Semantic versioning: MAJOR.MINOR.PATCH
+# MAJOR: Breaking changes
+# MINOR: New features (backward compatible)
+# PATCH: Bug fixes
+```
+
+**Version constraints in dependencies:**
+```toml
+dependencies = [
+    "requests>=2.28.0,<3.0.0",  # Compatible range
+    "click~=8.1.0",              # Compatible release (~= 8.1.0 means >=8.1.0,<8.2.0)
+    "pydantic>=2.0",             # Minimum version
+    "numpy==1.24.3",             # Exact version (avoid if possible)
+]
+```
+
+### Pattern 15: Git-Based Versioning
+
+```toml
+[build-system]
+requires = ["setuptools>=61.0", "setuptools-scm>=8.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "my-package"
+dynamic = ["version"]
+
+[tool.setuptools_scm]
+write_to = "src/my_package/_version.py"
+version_scheme = "post-release"
+local_scheme = "dirty-tag"
+```
+
+**Creates versions like:**
+- `1.0.0` (from git tag)
+- `1.0.1.dev3+g1234567` (3 commits after tag)
+
+## Testing Installation
+
+### Pattern 16: Editable Install
+
+```bash
+# Install in development mode
+pip install -e .
+
+# With optional dependencies
+pip install -e ".[dev]"
+pip install -e ".[dev,docs]"
+
+# Now changes to source code are immediately reflected
+```
+
+### Pattern 17: Testing in Isolated Environment
+
+```bash
+# Create virtual environment
+python -m venv test-env
+source test-env/bin/activate  # Linux/Mac
+# test-env\Scripts\activate  # Windows
+
+# Install package
+pip install dist/my_package-1.0.0-py3-none-any.whl
+
+# Test it works
+python -c "import my_package; print(my_package.__version__)"
+
+# Test CLI
+my-tool --help
+
+# Cleanup
+deactivate
+rm -rf test-env
+```
+
+## Documentation
+
+### Pattern 18: README.md Template
+
+```markdown
+# My Package
+
+[![PyPI version](https://badge.fury.io/py/my-package.svg)](https://pypi.org/project/my-package/)
+[![Python versions](https://img.shields.io/pypi/pyversions/my-package.svg)](https://pypi.org/project/my-package/)
+[![Tests](https://github.com/username/my-package/workflows/Tests/badge.svg)](https://github.com/username/my-package/actions)
+
+Brief description of your package.
+
+## Installation
+
+```bash
+pip install my-package
+```
+
+## Quick Start
+
+```python
+from my_package import something
+
+result = something.do_stuff()
+```
+
+## Features
+
+- Feature 1
+- Feature 2
+- Feature 3
+
+## Documentation
+
+Full documentation: https://my-package.readthedocs.io
+
+## Development
+
+```bash
+git clone https://github.com/username/my-package.git
+cd my-package
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT
+```
+
+## Common Patterns
+
+### Pattern 19: Multi-Architecture Wheels
+
+```yaml
+# .github/workflows/wheels.yml
+name: Build wheels
+
+on: [push, pull_request]
+
+jobs:
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - name: Build wheels
+        uses: pypa/cibuildwheel@v2.16.2
+
+      - uses: actions/upload-artifact@v3
+        with:
+          path: ./wheelhouse/*.whl
+```
+
+### Pattern 20: Private Package Index
+
+```bash
+# Install from private index
+pip install my-package --index-url https://private.pypi.org/simple/
+
+# Or add to pip.conf
+[global]
+index-url = https://private.pypi.org/simple/
+extra-index-url = https://pypi.org/simple/
+
+# Upload to private index
+twine upload --repository-url https://private.pypi.org/ dist/*
+```
+
+## File Templates
+
+### .gitignore for Python Packages
+
+```gitignore
+# Build artifacts
+build/
+dist/
+*.egg-info/
+*.egg
+.eggs/
+
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+
+# Virtual environments
+venv/
+env/
+ENV/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+
+# Distribution
+*.whl
+*.tar.gz
+```
+
+### MANIFEST.in
+
+```
+# MANIFEST.in
+include README.md
+include LICENSE
+include pyproject.toml
+
+recursive-include src/my_package/data *.json
+recursive-include src/my_package/templates *.html
+recursive-exclude * __pycache__
+recursive-exclude * *.py[co]
+```
+
+## Checklist for Publishing
+
+- [ ] Code is tested (pytest passing)
+- [ ] Documentation is complete (README, docstrings)
+- [ ] Version number updated
+- [ ] CHANGELOG.md updated
+- [ ] License file included
+- [ ] pyproject.toml is complete
+- [ ] Package builds without errors
+- [ ] Installation tested in clean environment
+- [ ] CLI tools work (if applicable)
+- [ ] PyPI metadata is correct (classifiers, keywords)
+- [ ] GitHub repository linked
+- [ ] Tested on TestPyPI first
+- [ ] Git tag created for release
+
+## Resources
+
+- **Python Packaging Guide**: https://packaging.python.org/
+- **PyPI**: https://pypi.org/
+- **TestPyPI**: https://test.pypi.org/
+- **setuptools documentation**: https://setuptools.pypa.io/
+- **build**: https://pypa-build.readthedocs.io/
+- **twine**: https://twine.readthedocs.io/
+
+## Best Practices Summary
+
+1. **Use src/ layout** for cleaner package structure
+2. **Use pyproject.toml** for modern packaging
+3. **Pin build dependencies** in build-system.requires
+4. **Version appropriately** with semantic versioning
+5. **Include all metadata** (classifiers, URLs, etc.)
+6. **Test installation** in clean environments
+7. **Use TestPyPI** before publishing to PyPI
+8. **Document thoroughly** with README and docstrings
+9. **Include LICENSE** file
+10. **Automate publishing** with CI/CD