quark-trace 0.1.0__tar.gz

quark_trace-0.1.0/PKG-INFO

@@ -0,0 +1,258 @@
+Metadata-Version: 2.4
+Name: quark-trace
+Version: 0.1.0
+Summary: Quark Trace — ML traceability and audit trail library. Part of the Quark suite.
+Author-email: Mohammed <mohammed.alwedaei@outlook.com>
+License: MIT
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests
+Requires-Dist: python-dotenv
+Provides-Extra: dev
+Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: types-PyYAML; extra == "dev"
+
+# Quark Trace
+
+A Python library for ML traceability. Provides structured logging of machine learning project metadata, experiment history, and audit trails across interchangeable storage backends.
+
+Designed to integrate with federated learning frameworks and other distributed ML pipelines.
+
+---
+
+## Status
+
+Active development. Core modules — `FactSheet`, `TraceLog`, `Project`, and the JSON storage backend — are implemented.
+
+---
+
+## Architecture
+
+```
+user-defined YAML file
+        |
+        v
+Pydantic schema (validation)
+        |
+        v
+FactSheet (identity + amendment trail)
+        |
+        v
+Project (unified entry point)
+        |
+        v
+TraceLog (append-only trace records)
+        |
+        v
+StorageBackend (interchangeable: JSON, SQL, IPFS, Blockchain)
+```
+
+The library is organized around three layers:
+
+- **Identity layer** — `FactSheet` defines the static project identity, loaded from YAML and validated by Pydantic. Supports controlled amendments with a full version trail.
+- **Trace layer** — `TraceLog` maintains an append-only log of discrete lifecycle events. Each call to `project.log()` produces one immutable `TraceRecord`.
+- **Storage layer** — `StorageBackend` is an abstract interface. All persistence is delegated to a backend. No component is coupled to a specific storage mechanism.
+
+---
+
+## Module Structure
+
+```
+quark_trace/
+    project.py                  # Unified entry point
+    fact_sheet/
+        __init__.py
+        schema.py               # Pydantic validation models
+        fact_sheet.py           # FactSheet class
+        loader.py               # YAML -> Pydantic -> FactSheet
+    trace/
+        __init__.py
+        record.py               # TraceRecord — single immutable entry
+        trace_log.py            # TraceLog — append-only log
+        backends/
+            __init__.py
+            base.py             # Abstract StorageBackend
+            json_backend.py     # File system backend (JSONL + JSON)
+```
+
+---
+
+## Components
+
+### Project
+
+The single object the consuming framework interacts with. Binds a `FactSheet` and a `TraceLog` under one interface.
+
+```python
+from quark_trace.project import Project
+from quark_trace.trace.backends.json_backend import JsonBackend
+
+# First run — loads YAML, persists fact sheet, starts trace log
+project = Project.load(
+    fact_sheet_path="fact_sheet.yaml",
+    backend=JsonBackend(path="logs/")
+)
+
+# Resume an existing project without re-loading the YAML
+project = Project.resume(
+    project_id="fl-project-001",
+    backend=JsonBackend(path="logs/")
+)
+```
+
+### Logging
+
+All trace logging goes through a single method on `Project`:
+
+```python
+project.log(stage="experiment_start", rounds=10, clients=5)
+project.log(stage="client_round", round=1, client_id="client_03", loss=0.21)
+project.log(stage="aggregation_round", round=1, aggregated_loss=0.19)
+project.log(stage="experiment_end", final_loss=0.11, duration_seconds=342)
+```
+
+### Retrieving History
+
+```python
+records = project.history()
+```
+
+---
+
+### FactSheet
+
+Defines and tracks the static identity of an ML project. Loaded from a user-defined YAML file. Supports controlled amendments with a full version trail.
+
+**YAML template:**
+
+```yaml
+project_id: "my-project-001"   # optional — auto-assigned if omitted
+
+purpose: "Detect fraudulent transactions in real-time"
+domain: "Financial Services"
+ml_type: "supervised"
+
+algorithm:
+  - name: "XGBoost"
+    variant: "XGBClassifier"
+
+input:
+  - name: "transaction_features"
+    type: "tabular"
+    description: "Normalized transaction records"
+
+output:
+  - name: "fraud_label"
+    type: "label"
+    description: "Binary fraud classification"
+
+performance_metrics:
+  - "accuracy"
+  - "precision"
+  - "recall"
+  - "f1"
+
+bias:
+  type: "historical"
+  affected_group: "low-income demographics"
+  severity: "medium"
+  notes: "Training data reflects prior biased approval patterns"
+
+stakeholders:
+  - name: "Jane Doe"
+    role: "ML Engineer"
+    contact: "jane@example.com"
+```
+
+**Schema:**
+
+| Field | Type | Description |
+|---|---|---|
+| `sheet_id` | `str` | Unique identifier for the fact sheet |
+| `project_id` | `str` | Parent project identifier |
+| `version` | `int` | Increments on each amendment |
+| `created_at` | `str` | ISO-8601 timestamp of initial creation |
+| `amended_at` | `str` | ISO-8601 timestamp of last amendment |
+| `amendment_log` | `list` | Full history of all amendments |
+| `purpose` | `str` | Description of the project's objective |
+| `domain` | `str` | Application domain |
+| `ml_type` | `str` | supervised, unsupervised, semi-supervised, self-supervised, reinforcement |
+| `algorithm` | `list[dict]` | Algorithm name and optional variant |
+| `input` | `list[dict]` | Input modalities and types |
+| `output` | `list[dict]` | Output types and descriptions |
+| `performance_metrics` | `list[str]` | Metric names tracked in this project |
+| `bias` | `dict` | Structured bias declaration with type, affected group, severity, and notes |
+| `stakeholders` | `list[dict]` | Named stakeholders, roles, and contacts |
+
+---
+
+### TraceRecord
+
+A single immutable trace entry. Frozen at the object level — no field can be modified after creation.
+
+| Field | Type | Description |
+|---|---|---|
+| `record_id` | `str` | Unique identifier for this record |
+| `project_id` | `str` | Parent project identifier |
+| `stage` | `str` | Lifecycle stage label |
+| `timestamp` | `str` | ISO-8601 UTC timestamp |
+| `payload` | `dict` | Arbitrary stage-specific data |
+
+---
+
+### Storage Backends
+
+All backends implement the `StorageBackend` abstract interface:
+
+| Method | Description |
+|---|---|
+| `save(record)` | Persist a single trace record |
+| `load_all(project_id)` | Retrieve all trace records for a project |
+| `save_fact_sheet(fact_sheet)` | Persist the fact sheet for a project |
+| `load_fact_sheet(project_id)` | Retrieve the fact sheet for a project |
+
+**JSON Backend** stores data as two files per project:
+
+| File | Format | Description |
+|---|---|---|
+| `{project_id}.jsonl` | Newline-delimited JSON | Append-only trace records |
+| `{project_id}.fact.json` | JSON | Fact sheet |
+
+---
+
+## Design Principles
+
+- The `Project` object is the single interface for consuming frameworks. Internal components are not exposed.
+- Storage backends are interchangeable. Switching from JSON to SQL or IPFS requires no changes to `Project`, `FactSheet`, or `TraceLog`.
+- The fact sheet is written once and amended with a version trail — never silently overwritten.
+- Trace records are strictly append-only and immutable at the object level.
+- All structures are JSON-serializable by design.
+- YAML is the primary interface for fact sheet definition. Direct construction is not the intended path.
+
+---
+
+## Roadmap
+
+- [x] `FactSheet` class with amendment trail
+- [x] Pydantic validation schema
+- [x] YAML loader
+- [x] `TraceRecord` — immutable trace entry
+- [x] `TraceLog` — append-only log
+- [x] `StorageBackend` abstract interface
+- [x] `JsonBackend` — file system implementation
+- [x] `Project` — unified entry point
+- [ ] `SqlBackend`
+- [ ] `IpfsBackend`
+- [ ] `BlockchainBackend`
+- [ ] Stage schema validation layer
+- [ ] Query and filtering API for trace history
+
+---
+
+## License
+
+To be defined.

quark_trace-0.1.0/README.md

@@ -0,0 +1,240 @@
+# Quark Trace
+
+A Python library for ML traceability. Provides structured logging of machine learning project metadata, experiment history, and audit trails across interchangeable storage backends.
+
+Designed to integrate with federated learning frameworks and other distributed ML pipelines.
+
+---
+
+## Status
+
+Active development. Core modules — `FactSheet`, `TraceLog`, `Project`, and the JSON storage backend — are implemented.
+
+---
+
+## Architecture
+
+```
+user-defined YAML file
+        |
+        v
+Pydantic schema (validation)
+        |
+        v
+FactSheet (identity + amendment trail)
+        |
+        v
+Project (unified entry point)
+        |
+        v
+TraceLog (append-only trace records)
+        |
+        v
+StorageBackend (interchangeable: JSON, SQL, IPFS, Blockchain)
+```
+
+The library is organized around three layers:
+
+- **Identity layer** — `FactSheet` defines the static project identity, loaded from YAML and validated by Pydantic. Supports controlled amendments with a full version trail.
+- **Trace layer** — `TraceLog` maintains an append-only log of discrete lifecycle events. Each call to `project.log()` produces one immutable `TraceRecord`.
+- **Storage layer** — `StorageBackend` is an abstract interface. All persistence is delegated to a backend. No component is coupled to a specific storage mechanism.
+
+---
+
+## Module Structure
+
+```
+quark_trace/
+    project.py                  # Unified entry point
+    fact_sheet/
+        __init__.py
+        schema.py               # Pydantic validation models
+        fact_sheet.py           # FactSheet class
+        loader.py               # YAML -> Pydantic -> FactSheet
+    trace/
+        __init__.py
+        record.py               # TraceRecord — single immutable entry
+        trace_log.py            # TraceLog — append-only log
+        backends/
+            __init__.py
+            base.py             # Abstract StorageBackend
+            json_backend.py     # File system backend (JSONL + JSON)
+```
+
+---
+
+## Components
+
+### Project
+
+The single object the consuming framework interacts with. Binds a `FactSheet` and a `TraceLog` under one interface.
+
+```python
+from quark_trace.project import Project
+from quark_trace.trace.backends.json_backend import JsonBackend
+
+# First run — loads YAML, persists fact sheet, starts trace log
+project = Project.load(
+    fact_sheet_path="fact_sheet.yaml",
+    backend=JsonBackend(path="logs/")
+)
+
+# Resume an existing project without re-loading the YAML
+project = Project.resume(
+    project_id="fl-project-001",
+    backend=JsonBackend(path="logs/")
+)
+```
+
+### Logging
+
+All trace logging goes through a single method on `Project`:
+
+```python
+project.log(stage="experiment_start", rounds=10, clients=5)
+project.log(stage="client_round", round=1, client_id="client_03", loss=0.21)
+project.log(stage="aggregation_round", round=1, aggregated_loss=0.19)
+project.log(stage="experiment_end", final_loss=0.11, duration_seconds=342)
+```
+
+### Retrieving History
+
+```python
+records = project.history()
+```
+
+---
+
+### FactSheet
+
+Defines and tracks the static identity of an ML project. Loaded from a user-defined YAML file. Supports controlled amendments with a full version trail.
+
+**YAML template:**
+
+```yaml
+project_id: "my-project-001"   # optional — auto-assigned if omitted
+
+purpose: "Detect fraudulent transactions in real-time"
+domain: "Financial Services"
+ml_type: "supervised"
+
+algorithm:
+  - name: "XGBoost"
+    variant: "XGBClassifier"
+
+input:
+  - name: "transaction_features"
+    type: "tabular"
+    description: "Normalized transaction records"
+
+output:
+  - name: "fraud_label"
+    type: "label"
+    description: "Binary fraud classification"
+
+performance_metrics:
+  - "accuracy"
+  - "precision"
+  - "recall"
+  - "f1"
+
+bias:
+  type: "historical"
+  affected_group: "low-income demographics"
+  severity: "medium"
+  notes: "Training data reflects prior biased approval patterns"
+
+stakeholders:
+  - name: "Jane Doe"
+    role: "ML Engineer"
+    contact: "jane@example.com"
+```
+
+**Schema:**
+
+| Field | Type | Description |
+|---|---|---|
+| `sheet_id` | `str` | Unique identifier for the fact sheet |
+| `project_id` | `str` | Parent project identifier |
+| `version` | `int` | Increments on each amendment |
+| `created_at` | `str` | ISO-8601 timestamp of initial creation |
+| `amended_at` | `str` | ISO-8601 timestamp of last amendment |
+| `amendment_log` | `list` | Full history of all amendments |
+| `purpose` | `str` | Description of the project's objective |
+| `domain` | `str` | Application domain |
+| `ml_type` | `str` | supervised, unsupervised, semi-supervised, self-supervised, reinforcement |
+| `algorithm` | `list[dict]` | Algorithm name and optional variant |
+| `input` | `list[dict]` | Input modalities and types |
+| `output` | `list[dict]` | Output types and descriptions |
+| `performance_metrics` | `list[str]` | Metric names tracked in this project |
+| `bias` | `dict` | Structured bias declaration with type, affected group, severity, and notes |
+| `stakeholders` | `list[dict]` | Named stakeholders, roles, and contacts |
+
+---
+
+### TraceRecord
+
+A single immutable trace entry. Frozen at the object level — no field can be modified after creation.
+
+| Field | Type | Description |
+|---|---|---|
+| `record_id` | `str` | Unique identifier for this record |
+| `project_id` | `str` | Parent project identifier |
+| `stage` | `str` | Lifecycle stage label |
+| `timestamp` | `str` | ISO-8601 UTC timestamp |
+| `payload` | `dict` | Arbitrary stage-specific data |
+
+---
+
+### Storage Backends
+
+All backends implement the `StorageBackend` abstract interface:
+
+| Method | Description |
+|---|---|
+| `save(record)` | Persist a single trace record |
+| `load_all(project_id)` | Retrieve all trace records for a project |
+| `save_fact_sheet(fact_sheet)` | Persist the fact sheet for a project |
+| `load_fact_sheet(project_id)` | Retrieve the fact sheet for a project |
+
+**JSON Backend** stores data as two files per project:
+
+| File | Format | Description |
+|---|---|---|
+| `{project_id}.jsonl` | Newline-delimited JSON | Append-only trace records |
+| `{project_id}.fact.json` | JSON | Fact sheet |
+
+---
+
+## Design Principles
+
+- The `Project` object is the single interface for consuming frameworks. Internal components are not exposed.
+- Storage backends are interchangeable. Switching from JSON to SQL or IPFS requires no changes to `Project`, `FactSheet`, or `TraceLog`.
+- The fact sheet is written once and amended with a version trail — never silently overwritten.
+- Trace records are strictly append-only and immutable at the object level.
+- All structures are JSON-serializable by design.
+- YAML is the primary interface for fact sheet definition. Direct construction is not the intended path.
+
+---
+
+## Roadmap
+
+- [x] `FactSheet` class with amendment trail
+- [x] Pydantic validation schema
+- [x] YAML loader
+- [x] `TraceRecord` — immutable trace entry
+- [x] `TraceLog` — append-only log
+- [x] `StorageBackend` abstract interface
+- [x] `JsonBackend` — file system implementation
+- [x] `Project` — unified entry point
+- [ ] `SqlBackend`
+- [ ] `IpfsBackend`
+- [ ] `BlockchainBackend`
+- [ ] Stage schema validation layer
+- [ ] Query and filtering API for trace history
+
+---
+
+## License
+
+To be defined.

quark_trace-0.1.0/pyproject.toml

@@ -0,0 +1,42 @@
+[build-system]
+requires = ["setuptools>=72"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "quark-trace"
+version = "0.1.0"
+description = "Quark Trace — ML traceability and audit trail library. Part of the Quark suite."
+license = { text = "MIT" }
+authors = [
+    { name = "Mohammed", email = "mohammed.alwedaei@outlook.com" }
+]
+readme = "README.md"
+requires-python = ">=3.12"
+dependencies = [
+    "pydantic>=2.0",
+    "pyyaml>=6.0",
+    "requests",
+    "python-dotenv"
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=8.0",
+    "pytest-cov",
+    "ruff",
+    "types-PyYAML",
+]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+
+[tool.setuptools.packages.find]
+where = ["."]
+include = ["quark_trace*"]
+
+[tool.ruff]
+line-length = 88
+target-version = "py312"
+
+[tool.ruff.lint]
+select = ["E", "F", "I"]

quark_trace-0.1.0/quark_trace/__init__.py

@@ -0,0 +1,24 @@
+from quark_trace.exceptions import (BackendReadError, BackendWriteError,
+                                    FactSheetAlreadyExistsError,
+                                    FactSheetNotFoundError,
+                                    ImmutableFieldError, InvalidAmendmentError,
+                                    InvalidStageError, QuarkLensError,
+                                    RecordSerializationError)
+from quark_trace.fact_sheet.fact_sheet import FactSheet
+from quark_trace.project import Project
+from quark_trace.trace.backends.json_backend import JsonBackend
+
+__all__ = [
+    "Project",
+    "JsonBackend",
+    "FactSheet",
+    "QuarkLensError",
+    "FactSheetAlreadyExistsError",
+    "FactSheetNotFoundError",
+    "ImmutableFieldError",
+    "InvalidAmendmentError",
+    "InvalidStageError",
+    "RecordSerializationError",
+    "BackendReadError",
+    "BackendWriteError",
+]

quark_trace-0.1.0/quark_trace/exceptions.py

@@ -0,0 +1,122 @@
+class QuarkLensError(Exception):
+    """Base exception for all quark_trace errors."""
+    pass
+
+
+# --- Fact Sheet ---
+
+class FactSheetError(QuarkLensError):
+    """Base exception for fact sheet errors."""
+    pass
+
+
+class FactSheetAlreadyExistsError(FactSheetError):
+    """
+    Raised when attempting to save a fact sheet for a project
+    that already has one persisted in the backend.
+    """
+    def __init__(self, project_id: str) -> None:
+        super().__init__(
+            f"A fact sheet for project '{project_id}' already exists. "
+            f"Use amend() to modify it."
+        )
+        self.project_id = project_id
+
+
+class FactSheetNotFoundError(FactSheetError):
+    """
+    Raised when a fact sheet cannot be found for the given project ID.
+    """
+    def __init__(self, project_id: str) -> None:
+        super().__init__(
+            f"No fact sheet found for project '{project_id}'."
+        )
+        self.project_id = project_id
+
+
+class ImmutableFieldError(FactSheetError):
+    """
+    Raised when amend() attempts to modify a field that is
+    declared immutable on the FactSheet.
+    """
+    def __init__(self, field: str) -> None:
+        super().__init__(
+            f"Field '{field}' is immutable and cannot be amended."
+        )
+        self.field = field
+
+
+class InvalidAmendmentError(FactSheetError):
+    """
+    Raised when amend() is called with no valid fields to update.
+    """
+    def __init__(self) -> None:
+        super().__init__(
+            "Amendment contains no valid fields. "
+            "Ensure field names match FactSheet attributes."
+        )
+
+
+# --- Trace ---
+
+class TraceError(QuarkLensError):
+    """Base exception for trace errors."""
+    pass
+
+
+class InvalidStageError(TraceError):
+    """
+    Raised when log() is called with an empty or invalid stage value.
+    """
+    def __init__(self, stage: str) -> None:
+        super().__init__(
+            f"Invalid stage value: '{stage}'. Stage must be a non-empty string."
+        )
+        self.stage = stage
+
+
+class RecordSerializationError(TraceError):
+    """
+    Raised when a TraceRecord cannot be serialized to or
+    deserialized from JSON.
+    """
+    def __init__(self, record_id: str, reason: str) -> None:
+        super().__init__(
+            f"Failed to serialize record '{record_id}': {reason}"
+        )
+        self.record_id = record_id
+
+
+# --- Backend ---
+
+class BackendError(QuarkLensError):
+    """Base exception for storage backend errors."""
+    pass
+
+
+class BackendReadError(BackendError):
+    """
+    Raised when a backend read operation fails.
+    """
+    def __init__(self, reason: str) -> None:
+        super().__init__(f"Backend read failed: {reason}")
+
+
+class BackendWriteError(BackendError):
+    """
+    Raised when a backend write operation fails.
+    """
+    def __init__(self, reason: str) -> None:
+        super().__init__(f"Backend write failed: {reason}")
+
+class NodeVerificationError(QuarkLensError):
+    """Raised when the node cannot be verified against the registry."""
+    pass
+
+class NodeAuthenticationError(QuarkLensError):
+    """Raised when the server rejects the node's API key."""
+    pass
+
+class NodeScopeError(QuarkLensError):
+    """Raised when the node is not authorized for the target institution."""
+    pass

quark_trace-0.1.0/quark_trace/fact_sheet/__init__.py

@@ -0,0 +1,7 @@
+from quark_trace.fact_sheet.fact_sheet import FactSheet
+from quark_trace.fact_sheet.loader import load_fact_sheet
+
+__all__ = [
+    "FactSheet",
+    "load_fact_sheet",
+]