rust-crate-pipeline 1.3.6__tar.gz → 1.4.1__tar.gz

rust_crate_pipeline-1.4.1/CHANGELOG_v1.4.0.md

@@ -0,0 +1,21 @@
+# Changelog v1.4.0
+
+## [1.4.0] - 2025-06-28
+
+### Added
+- Robust Ed25519 and RSA cryptographic signing for RAG database
+- Automated provenance and signature validation workflows
+- GitHub Actions for signature/hash validation and RAG auto-update
+- Docker image and compose updates for new version
+
+### Fixed
+- Signature validation for Ed25519 keys in both scripts and CI
+- Public key tracking in git, private key protection
+- Workflow reliability for PyPI and Docker builds
+
+### Changed
+- Version bump to 1.4.0 (minor release)
+- All version references updated
+- RAG and provenance now reflect new version
+
+--- 

{rust_crate_pipeline-1.3.6 → rust_crate_pipeline-1.4.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: rust-crate-pipeline
-Version: 1.3.6
+Version: 1.4.1
 Summary: A comprehensive pipeline for analyzing Rust crates with AI enrichment and enhanced scraping
 Home-page: https://github.com/SigilDERG/rust-crate-pipeline
 Author: SigilDERG Team
@@ -386,6 +386,13 @@ docker run -it -v $(pwd):/app rust-crate-pipeline
 
 ## Recent Improvements
 
+### Version 1.4.0
+- **Security**: Robust Ed25519/RSA cryptographic signing and provenance
+- **Automation**: Automated RAG and provenance workflows
+- **CI/CD**: Improved GitHub Actions for validation and publishing
+- **Docker**: Updated Docker image and compose for new version
+- **Bug Fixes**: Workflow and validation fixes for Ed25519
+
 ### Version 1.3.6
 - **Python 3.12+ Requirement**: Updated to use modern type annotations and language features
 - **Type Safety**: Enhanced type annotations throughout the codebase with modern syntax
@@ -453,4 +460,56 @@ Or, text attribution:
 
 ```
 This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
-``` 
+```
+
+## 🚀 Unified, Cross-Platform, Multi-Provider LLM Support
+
+This project supports **all major LLM providers** (cloud and local) on **Mac, Linux, and Windows** using a single, unified interface. All LLM calls are routed through the `UnifiedLLMProcessor` and `LLMConfig` abstractions, ensuring:
+
+- **One code path for all providers:** Azure OpenAI, OpenAI, Anthropic, Google, Cohere, HuggingFace, Ollama, LM Studio, and any OpenAI-compatible endpoint.
+- **Cross-platform compatibility:** Works out of the box on Mac, Linux, and Windows.
+- **Configurable via CLI and config files:** Select provider, model, API key, endpoint, and provider-specific options at runtime.
+- **Easy extensibility:** Add new providers by updating your config or CLI arguments—no code changes needed.
+
+### 📖 Provider Setup & Usage
+- See [`README_LLM_PROVIDERS.md`](./README_LLM_PROVIDERS.md) for full details, setup instructions, and usage examples for every supported provider.
+- Run `python run_pipeline_with_llm.py --help` for CLI options and provider-specific arguments.
+
+### 🧩 Example Usage
+```bash
+# Azure OpenAI
+python run_pipeline_with_llm.py --llm-provider azure --llm-model gpt-4o --crates tokio
+
+# Ollama (local)
+python run_pipeline_with_llm.py --llm-provider ollama --llm-model llama2 --crates serde
+
+# OpenAI API
+python run_pipeline_with_llm.py --llm-provider openai --llm-model gpt-4 --llm-api-key YOUR_KEY --crates tokio
+
+# Anthropic Claude
+python run_pipeline_with_llm.py --llm-provider anthropic --llm-model claude-3-sonnet --llm-api-key YOUR_KEY --crates serde
+```
+
+### 🔒 Security & Best Practices
+- Store API keys as environment variables.
+- Use local providers (Ollama, LM Studio) for full privacy—no data leaves your machine.
+- All LLM calls are routed through a single, auditable interface for maximum maintainability and security.
+
+### 🧪 Testing
+- Run `python test_unified_llm.py` to verify provider support and configuration.
+
+For more, see [`README_LLM_PROVIDERS.md`](./README_LLM_PROVIDERS.md) and the CLI help output. 
+
+## Public RAG Database Hash Verification
+
+The canonical hash of the RAG SQLite database (`sigil_rag_cache.db`) is stored in the public file `sigil_rag_cache.hash`.
+
+- **Purpose:** Anyone can verify the integrity of the RAG database by comparing its SHA256 hash to the value in `sigil_rag_cache.hash`.
+- **How to verify:**
+
+```sh
+python audits/validate_db_hash.py --db sigil_rag_cache.db --expected-hash "$(cat sigil_rag_cache.hash)"
+```
+
+- **CI/CD:** The GitHub Actions workflow `.github/workflows/validate-db-hash.yml` automatically checks this on every push.
+- **No secrets required:** The hash is public and verifiable by anyone.

{rust_crate_pipeline-1.3.6 → rust_crate_pipeline-1.4.1}/README.md

@@ -339,6 +339,13 @@ docker run -it -v $(pwd):/app rust-crate-pipeline
 
 ## Recent Improvements
 
+### Version 1.4.0
+- **Security**: Robust Ed25519/RSA cryptographic signing and provenance
+- **Automation**: Automated RAG and provenance workflows
+- **CI/CD**: Improved GitHub Actions for validation and publishing
+- **Docker**: Updated Docker image and compose for new version
+- **Bug Fixes**: Workflow and validation fixes for Ed25519
+
 ### Version 1.3.6
 - **Python 3.12+ Requirement**: Updated to use modern type annotations and language features
 - **Type Safety**: Enhanced type annotations throughout the codebase with modern syntax
@@ -406,4 +413,56 @@ Or, text attribution:
 
 ```
 This project uses Crawl4AI (https://github.com/unclecode/crawl4ai) for web data extraction.
-``` 
+```
+
+## 🚀 Unified, Cross-Platform, Multi-Provider LLM Support
+
+This project supports **all major LLM providers** (cloud and local) on **Mac, Linux, and Windows** using a single, unified interface. All LLM calls are routed through the `UnifiedLLMProcessor` and `LLMConfig` abstractions, ensuring:
+
+- **One code path for all providers:** Azure OpenAI, OpenAI, Anthropic, Google, Cohere, HuggingFace, Ollama, LM Studio, and any OpenAI-compatible endpoint.
+- **Cross-platform compatibility:** Works out of the box on Mac, Linux, and Windows.
+- **Configurable via CLI and config files:** Select provider, model, API key, endpoint, and provider-specific options at runtime.
+- **Easy extensibility:** Add new providers by updating your config or CLI arguments—no code changes needed.
+
+### 📖 Provider Setup & Usage
+- See [`README_LLM_PROVIDERS.md`](./README_LLM_PROVIDERS.md) for full details, setup instructions, and usage examples for every supported provider.
+- Run `python run_pipeline_with_llm.py --help` for CLI options and provider-specific arguments.
+
+### 🧩 Example Usage
+```bash
+# Azure OpenAI
+python run_pipeline_with_llm.py --llm-provider azure --llm-model gpt-4o --crates tokio
+
+# Ollama (local)
+python run_pipeline_with_llm.py --llm-provider ollama --llm-model llama2 --crates serde
+
+# OpenAI API
+python run_pipeline_with_llm.py --llm-provider openai --llm-model gpt-4 --llm-api-key YOUR_KEY --crates tokio
+
+# Anthropic Claude
+python run_pipeline_with_llm.py --llm-provider anthropic --llm-model claude-3-sonnet --llm-api-key YOUR_KEY --crates serde
+```
+
+### 🔒 Security & Best Practices
+- Store API keys as environment variables.
+- Use local providers (Ollama, LM Studio) for full privacy—no data leaves your machine.
+- All LLM calls are routed through a single, auditable interface for maximum maintainability and security.
+
+### 🧪 Testing
+- Run `python test_unified_llm.py` to verify provider support and configuration.
+
+For more, see [`README_LLM_PROVIDERS.md`](./README_LLM_PROVIDERS.md) and the CLI help output. 
+
+## Public RAG Database Hash Verification
+
+The canonical hash of the RAG SQLite database (`sigil_rag_cache.db`) is stored in the public file `sigil_rag_cache.hash`.
+
+- **Purpose:** Anyone can verify the integrity of the RAG database by comparing its SHA256 hash to the value in `sigil_rag_cache.hash`.
+- **How to verify:**
+
+```sh
+python audits/validate_db_hash.py --db sigil_rag_cache.db --expected-hash "$(cat sigil_rag_cache.hash)"
+```
+
+- **CI/CD:** The GitHub Actions workflow `.github/workflows/validate-db-hash.yml` automatically checks this on every push.
+- **No secrets required:** The hash is public and verifiable by anyone.

{rust_crate_pipeline-1.3.6 → rust_crate_pipeline-1.4.1}/README_LLM_PROVIDERS.md

@@ -1,3 +1,12 @@
+> **All LLM usage in this project is unified and cross-platform.**
+>
+> - All LLM calls are routed through the `UnifiedLLMProcessor` and `LLMConfig` abstractions.
+> - This ensures support for all major providers (cloud and local) on Mac, Linux, and Windows.
+> - **All new LLM features must use this pattern.**
+> - The project is future-proof: as [LiteLLM](https://github.com/BerriAI/litellm) adds new providers, you can use them immediately by updating your config/CLI—no code changes needed.
+
+---
+
 # Unified LLM Provider Support for Rust Crate Pipeline
 
 This document describes the comprehensive LLM provider support in the Rust Crate Pipeline, allowing you to use any LiteLLM-compatible provider for AI-powered crate analysis.

{rust_crate_pipeline-1.3.6 → rust_crate_pipeline-1.4.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "rust-crate-pipeline"
-version = "1.3.6"
+version = "1.4.1"
 authors = [
     {name = "SigilDERG Team", email = "sigilderg@example.com"}
 ]