ski-audit-ledger 3.0.3__tar.gz

ski_audit_ledger-3.0.3/PKG-INFO

@@ -0,0 +1,581 @@
+Metadata-Version: 2.4
+Name: ski-audit-ledger
+Version: 3.0.3
+Summary: Audit ledger management tool for the SKI Framework.
+Author-email: KpiFinity <hello@kpifinity.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/kpifinity/ski-framework
+Project-URL: Documentation, https://kpifinity.github.io/ski-framework/
+Project-URL: Issues, https://github.com/kpifinity/ski-framework/issues
+Keywords: ski-framework,audit,compliance,ledger
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: System Administrators
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: psycopg[binary]<4,>=3.1.18
+Requires-Dist: sqlalchemy<3,>=2.0.27
+Requires-Dist: click<9,>=8.1.7
+Requires-Dist: pydantic<3,>=2.13.4
+Requires-Dist: python-dotenv<2,>=1.2.2
+Requires-Dist: jinja2<4,>=3.1.6
+Requires-Dist: pyyaml<7,>=6.0.1
+Requires-Dist: httpx<0.29,>=0.27
+Requires-Dist: tabulate<1,>=0.9
+
+# audit-ledger
+
+> Install from PyPI: `pip install ski-audit-ledger` (publishing starts with the first release after June 2026).
+
+> **⚠ STATUS: EARLY ALPHA (v0.1.0a0).** Alpha-quality tooling. See the
+> repo root `README.md` for the project-wide status.
+
+Manage and verify the immutable audit ledger for SKI Framework compliance monitoring.
+
+## v2.1 highlights
+
+- **Real `verify_integrity()`.** Recomputes every entry's hash from the
+  documented canonical serialization. Catches in-place tampering, not
+  just chain-linkage breakage.
+- **Real `backup`.** Invokes `pg_dump` and verifies the dump with
+  `pg_restore --list`. No more stub.
+- **Five-verdict taxonomy.** `--verdict-filter` now accepts
+  `NULL_UNMAPPED` and `NULL_STALE` (the pre-v2.1 `NULL` is gone).
+- **No `confidence_level`.** Removed entirely from the data model.
+- **Append-only enforcement** lives at the database layer
+  (`reference-implementation/src/ledger/append_only.sql`). UPDATE,
+  DELETE, and TRUNCATE on `ledger_entries` are refused by triggers.
+
+## Overview
+
+The audit ledger is a hash-chained, immutable record of all compliance
+verdicts generated by the SKI Model runtime. This tool provides
+utilities for:
+
+- **Verification** — Verify ledger integrity and detect tampering
+- **Backup** — Create secure backups with cryptographic verification
+- **Export** — Extract ledger data for analysis and reporting
+- **Reporting** — Generate compliance reports and summaries
+
+## Features
+
+### Ledger Integrity Verification
+
+Verify the complete hash chain of ledger entries to ensure no records have been tampered with:
+
+```bash
+audit-ledger verify --ledger-db postgresql://user:pass@localhost/ledger
+```
+
+Output:
+```
+Ledger Integrity Verification
+├── Total entries: 4,327
+├── Chain integrity: ✓ VERIFIED
+├── Hash verification: ✓ 4,327/4,327 valid
+├── Timestamp order: ✓ VALID
+└── Last verified: 2026-05-21T15:30:00Z
+```
+
+### Secure Backup
+
+Create verified backups with integrity checking:
+
+```bash
+audit-ledger backup \
+  --source postgresql://user:pass@localhost/ledger \
+  --output ./ledger-backup-2026-05-21.sql \
+  --verify
+```
+
+### Data Export
+
+Export ledger entries for analysis and reporting:
+
+```bash
+# Export to JSON
+audit-ledger export \
+  --source postgresql://user:pass@localhost/ledger \
+  --output ledger-data.json \
+  --format json
+
+# Export to CSV
+audit-ledger export \
+  --source postgresql://user:pass@localhost/ledger \
+  --output ledger-data.csv \
+  --format csv \
+  --fields "id,timestamp,verdict,rule_id,telemetry_id"
+```
+
+### Compliance Reporting
+
+Generate formatted compliance reports:
+
+```bash
+audit-ledger report \
+  --source postgresql://user:pass@localhost/ledger \
+  --start-date 2026-01-01 \
+  --end-date 2026-05-31 \
+  --output compliance-report.html
+```
+
+## Installation
+
+### From Source
+
+```bash
+cd tools/audit-ledger
+pip install -e .
+```
+
+### Requirements
+
+- Python 3.8+
+- PostgreSQL 12+ (for ledger database)
+- psycopg2 (PostgreSQL adapter)
+- sqlalchemy (ORM)
+
+## Quick Start
+
+### 1. Verify Ledger Integrity
+
+After running compliance monitoring for a period, verify the ledger has not been tampered with:
+
+```bash
+audit-ledger verify \
+  --ledger-db postgresql://compliance:securepass@localhost:5432/ski_ledger \
+  --verbose
+```
+
+### 2. Create a Backup
+
+Before major system changes, create a verified backup:
+
+```bash
+audit-ledger backup \
+  --source postgresql://compliance:securepass@localhost:5432/ski_ledger \
+  --output ./backups/ledger-$(date +%Y%m%d).sql.gz \
+  --compress \
+  --verify
+```
+
+### 3. Generate Monthly Report
+
+At month-end, generate a compliance summary:
+
+```bash
+audit-ledger report \
+  --source postgresql://compliance:securepass@localhost:5432/ski_ledger \
+  --start-date 2026-05-01 \
+  --end-date 2026-05-31 \
+  --output monthly-report-2026-05.html \
+  --include-verdicts \
+  --include-violations
+```
+
+### 4. Export for Audit
+
+Export ledger data for external audit:
+
+```bash
+audit-ledger export \
+  --source postgresql://compliance:securepass@localhost:5432/ski_ledger \
+  --output audit-export.json \
+  --format json \
+  --date-range 2026-01-01,2026-05-31
+```
+
+## Command Reference
+
+### verify
+
+Verify ledger integrity and hash chain validity.
+
+```bash
+audit-ledger verify [OPTIONS]
+
+Options:
+  --ledger-db TEXT          PostgreSQL connection string [required]
+  --verbose                 Show detailed verification output
+  --check-timestamps        Verify timestamp ordering
+  --repair                  Attempt to repair minor issues (use with caution)
+  --output TEXT             Write results to file
+  --help                    Show help message
+```
+
+### backup
+
+Create a backup of the ledger database.
+
+```bash
+audit-ledger backup [OPTIONS]
+
+Options:
+  --source TEXT             PostgreSQL connection string [required]
+  --output TEXT             Output file path [required]
+  --compress                Gzip compress the backup
+  --verify                  Verify backup integrity after creation
+  --encryption-key TEXT     Encrypt backup (optional)
+  --help                    Show help message
+```
+
+### export
+
+Export ledger entries for analysis.
+
+```bash
+audit-ledger export [OPTIONS]
+
+Options:
+  --source TEXT             PostgreSQL connection string [required]
+  --output TEXT             Output file path [required]
+  --format TEXT             json|csv|jsonl (default: json)
+  --fields TEXT             Comma-separated field names to include
+  --date-range TEXT         Start,end dates (YYYY-MM-DD,YYYY-MM-DD)
+  --verdict-filter TEXT     Filter by verdict (CLEAR|FLAG|NULL|DISCRETIONARY)
+  --rule-id TEXT            Filter by rule ID
+  --limit INTEGER           Maximum entries to export
+  --help                    Show help message
+```
+
+### report
+
+Generate a compliance report from ledger data.
+
+```bash
+audit-ledger report [OPTIONS]
+
+Options:
+  --source TEXT             PostgreSQL connection string [required]
+  --output TEXT             Output HTML file path
+  --start-date TEXT         Report start date (YYYY-MM-DD)
+  --end-date TEXT           Report end date (YYYY-MM-DD)
+  --include-verdicts        Include verdict summary statistics
+  --include-violations      Include violation details
+  --include-timeline        Include timeline visualization
+  --include-audit-trail     Include audit trail verification
+  --title TEXT              Report title
+  --organization TEXT       Organization name for report
+  --help                    Show help message
+```
+
+## Ledger Structure
+
+The audit ledger stores compliance verdicts in a hash-chained format:
+
+```sql
+CREATE TABLE ledger_entries (
+    id BIGSERIAL PRIMARY KEY,
+    sequence_number BIGINT UNIQUE NOT NULL,
+    previous_hash CHAR(64) NOT NULL,
+    entry_hash CHAR(64) NOT NULL UNIQUE,
+    timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
+    verdict TEXT NOT NULL CHECK (verdict IN (
+        'CLEAR','FLAG','NULL_UNMAPPED','NULL_STALE','DISCRETIONARY'
+    )),
+    telemetry_id TEXT NOT NULL,
+    telemetry_hash CHAR(64) NOT NULL,
+    rule_id TEXT,
+    knowledge_graph_version TEXT,
+    ski_model_version TEXT NOT NULL,
+    reasoning TEXT,
+    track TEXT CHECK (track IS NULL OR track IN ('symbolic','llm')),
+    escalation_status TEXT,
+    escalation_notes TEXT,
+    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
+);
+```
+
+> **v2.1 schema notes.** `milm_version` is renamed to `ski_model_version`.
+> `confidence_level` has been removed entirely — B3.1 prohibits confidence
+> scores in the ledger. The verdict CHECK constraint enforces the
+> five-verdict taxonomy.
+
+Each entry's `entry_hash` is `SHA-256` of the canonical serialization
+documented in
+[`src/audit_ledger/canonical.py`](./src/audit_ledger/canonical.py). The
+canonical payload includes `sequence_number`, `previous_hash`, `timestamp`,
+`verdict`, `telemetry_id`, `telemetry_hash`, `rule_id`, `kg_version`,
+`ski_model_version`, `reasoning`, and `track`. Modifying any one of
+these fields invalidates that entry's `entry_hash`.
+
+Append-only enforcement is in
+[`reference-implementation/src/ledger/append_only.sql`](../../reference-implementation/src/ledger/append_only.sql)
+— `BEFORE UPDATE`, `BEFORE DELETE`, and `BEFORE TRUNCATE` triggers
+refuse any modification at the database layer.
+
+## Integrity verification process
+
+The `verify` command checks, for every row:
+
+1. **Chain linkage** — each row's `previous_hash` equals the prior row's `entry_hash`.
+2. **Entry-hash recomputation** — re-derive `entry_hash` from the
+   canonical payload and compare with what's stored. Catches in-place
+   tampering that chain-linkage-only checks miss.
+3. **Sequence continuity** — `sequence_number` has no gaps.
+4. **Timestamp ordering** — timestamps are monotonic.
+
+Example verification output:
+
+```
+Ledger Integrity Verification Report
+====================================
+
+Database: postgresql://localhost/ski_ledger
+Verification Date: 2026-05-21T15:30:00Z
+
+Chain Analysis:
+  Total Entries: 4,327
+  Sequence Range: 1 - 4,327
+  Time Range: 2026-01-01 to 2026-05-21
+
+Integrity Checks:
+  ✓ Chain continuity: All 4,327 entries form valid chain
+  ✓ Hash verification: 4,327/4,327 entries (100%)
+  ✓ Timestamp ordering: All timestamps chronologically valid
+  ✓ Data consistency: All references valid
+
+Verdict Distribution:
+  CLEAR: 3,821 (88.3%)
+  FLAG: 412 (9.5%)
+  DISCRETIONARY: 94 (2.2%)
+  NULL: 0 (0.0%)
+
+Risk Assessment:
+  ✓ No signs of tampering
+  ✓ No missing entries
+  ✓ No invalid references
+  Status: VERIFIED
+
+Recommendation: Ledger integrity confirmed. Safe for regulatory reporting.
+```
+
+## Backup & Recovery
+
+### Creating a Backup
+
+```bash
+# Standard backup
+audit-ledger backup \
+  --source postgresql://compliance:pass@localhost/ski_ledger \
+  --output ./ledger-backup-2026-05-21.sql
+
+# Compressed with verification
+audit-ledger backup \
+  --source postgresql://compliance:pass@localhost/ski_ledger \
+  --output ./ledger-backup-2026-05-21.sql.gz \
+  --compress \
+  --verify
+```
+
+### Verifying a Backup
+
+```bash
+# Restore to test database
+psql test_ledger < ledger-backup-2026-05-21.sql
+
+# Verify the restored ledger
+audit-ledger verify \
+  --ledger-db postgresql://user:pass@localhost/test_ledger
+```
+
+## Export Examples
+
+### Export All Violations
+
+```bash
+audit-ledger export \
+  --source postgresql://user:pass@localhost/ski_ledger \
+  --output violations.json \
+  --format json \
+  --verdict-filter FLAG
+```
+
+Output:
+```json
+{
+  "export_date": "2026-05-21T15:30:00Z",
+  "entry_count": 412,
+  "entries": [
+    {
+      "id": 1543,
+      "timestamp": "2026-05-15T10:30:00Z",
+      "verdict": "FLAG",
+      "rule_id": "e001",
+      "telemetry_id": "TEL_2026_05_001",
+      "reasoning": "SO2 emissions 125 ppm exceeds limit of 100 ppm"
+    },
+    ...
+  ]
+}
+```
+
+### Export Monthly Audit Trail
+
+```bash
+audit-ledger export \
+  --source postgresql://user:pass@localhost/ski_ledger \
+  --output may-2026-audit.csv \
+  --format csv \
+  --date-range 2026-05-01,2026-05-31 \
+  --fields "timestamp,verdict,rule_id,reasoning,escalation_status"
+```
+
+## Reporting
+
+### Generate Monthly Compliance Report
+
+```bash
+audit-ledger report \
+  --source postgresql://user:pass@localhost/ski_ledger \
+  --start-date 2026-05-01 \
+  --end-date 2026-05-31 \
+  --output may-2026-compliance.html \
+  --include-verdicts \
+  --include-violations \
+  --include-timeline \
+  --organization "Acme Energy Corp"
+```
+
+Report includes:
+- Executive summary of compliance status
+- Verdict distribution charts
+- Violations identified and status
+- Remediation actions taken
+- Timeline of escalations
+- Audit trail verification status
+- Regulatory filing readiness
+
+## Security Considerations
+
+### Database Security
+
+The PostgreSQL ledger database should be:
+- Behind a firewall (not internet-accessible)
+- Using strong authentication (not default credentials)
+- Requiring SSL/TLS for all connections
+- Backed up regularly
+- Monitored for unauthorized access
+
+### Backup Security
+
+Backups should be:
+- Encrypted if stored off-site
+- Stored securely (not in version control)
+- Verified before relying on them
+- Tested for recovery capability
+
+### Access Control
+
+Ledger access should be restricted to:
+- Compliance officers
+- System administrators
+- Authorized auditors
+
+## Troubleshooting
+
+### Ledger Verification Fails
+
+If verification reports issues:
+
+```bash
+# Get detailed diagnostic output
+audit-ledger verify \
+  --ledger-db postgresql://user:pass@localhost/ski_ledger \
+  --verbose
+
+# Attempt automatic repair (use with caution)
+audit-ledger verify \
+  --ledger-db postgresql://user:pass@localhost/ski_ledger \
+  --repair
+```
+
+### Export Takes Too Long
+
+For large ledgers, use filters:
+
+```bash
+# Export only recent violations
+audit-ledger export \
+  --source postgresql://user:pass@localhost/ski_ledger \
+  --output recent-flags.json \
+  --verdict-filter FLAG \
+  --date-range 2026-05-01,2026-05-31
+```
+
+### Backup Verification Fails
+
+Restore to a test database and verify:
+
+```bash
+# Create test database
+createdb test_ledger
+
+# Restore backup
+psql test_ledger < backup.sql
+
+# Verify restoration
+audit-ledger verify --ledger-db postgresql://user:pass@localhost/test_ledger
+```
+
+## API Usage
+
+### Python API
+
+```python
+from audit_ledger import Ledger
+
+# Connect to ledger
+ledger = Ledger("postgresql://user:pass@localhost/ski_ledger")
+
+# Verify integrity
+result = ledger.verify_integrity()
+print(f"Ledger valid: {result.is_valid}")
+print(f"Total entries: {result.total_entries}")
+
+# Export entries
+entries = ledger.export_entries(
+    start_date="2026-05-01",
+    end_date="2026-05-31",
+    verdict_filter="FLAG"
+)
+
+# Generate report
+report = ledger.generate_report(
+    start_date="2026-05-01",
+    end_date="2026-05-31",
+    include_verdicts=True,
+    include_violations=True
+)
+ledger.save_report(report, "compliance-report.html")
+```
+
+## Production Checklist
+
+- [ ] Configure PostgreSQL with strong authentication
+- [ ] Enable SSL/TLS for database connections
+- [ ] Set up automated backup schedule
+- [ ] Verify backup recovery procedures
+- [ ] Document ledger access controls
+- [ ] Train staff on ledger commands
+- [ ] Set up monitoring for ledger database
+- [ ] Plan audit trail retention policy
+- [ ] Test export/reporting workflows
+- [ ] Schedule regular integrity verification
+- [ ] Document escalation procedures for verification failures
+
+## Support
+
+For issues or questions:
+- Check individual command help: `audit-ledger COMMAND --help`
+- Review ledger structure documentation
+- Open an issue on GitHub
+- Contact your compliance team
+
+---
+
+**Note:** The audit ledger is critical for regulatory compliance. All operations should be performed by authorized personnel only. Always verify backups and test recovery procedures before relying on them.