ml-model-architect 1.0.0rc1__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- ml_model_architect-1.0.0rc1.dist-info/METADATA +713 -0
- ml_model_architect-1.0.0rc1.dist-info/RECORD +20 -0
- ml_model_architect-1.0.0rc1.dist-info/WHEEL +5 -0
- ml_model_architect-1.0.0rc1.dist-info/entry_points.txt +3 -0
- ml_model_architect-1.0.0rc1.dist-info/licenses/LICENSE +21 -0
- ml_model_architect-1.0.0rc1.dist-info/top_level.txt +1 -0
- scripts/__init__.py +1 -0
- scripts/cli.py +133 -0
- scripts/enforce_constitution.py +216 -0
- scripts/executive_summary.py +34 -0
- scripts/fetch_sota.py +76 -0
- scripts/generate_diagram.py +35 -0
- scripts/lock_plan.py +153 -0
- scripts/pre_mortem.py +31 -0
- scripts/red_team.py +54 -0
- scripts/run_docker_sandbox.py +80 -0
- scripts/run_policy.py +265 -0
- scripts/scan_sandbox.py +129 -0
- scripts/verify_dois.py +171 -0
- scripts/verify_references.py +37 -0
|
@@ -0,0 +1,713 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: ml-model-architect
|
|
3
|
+
Version: 1.0.0rc1
|
|
4
|
+
Summary: A strict, reference-driven framework that turns AI agents into disciplined, expert ML engineers.
|
|
5
|
+
Author: Ahmed Zain
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
Requires-Python: >=3.10
|
|
8
|
+
Description-Content-Type: text/markdown
|
|
9
|
+
License-File: LICENSE
|
|
10
|
+
Requires-Dist: requests>=2.28.0
|
|
11
|
+
Requires-Dist: pyyaml>=6.0
|
|
12
|
+
Requires-Dist: numpy>=1.24.0
|
|
13
|
+
Requires-Dist: scikit-learn>=1.3.0
|
|
14
|
+
Requires-Dist: pandas>=2.1.0
|
|
15
|
+
Requires-Dist: streamlit>=1.30.0
|
|
16
|
+
Provides-Extra: mcp
|
|
17
|
+
Requires-Dist: mcp>=1.0.0; extra == "mcp"
|
|
18
|
+
Requires-Dist: chromadb>=0.4.0; extra == "mcp"
|
|
19
|
+
Requires-Dist: sentence-transformers>=2.2.0; extra == "mcp"
|
|
20
|
+
Provides-Extra: dev
|
|
21
|
+
Requires-Dist: pytest>=7.0.0; extra == "dev"
|
|
22
|
+
Dynamic: license-file
|
|
23
|
+
|
|
24
|
+
<img width="100%" height="887" alt="ML Model Architect Banner" src="https://github.com/user-attachments/assets/0b1e73e0-f551-4ca2-be93-569b1f54db91" />
|
|
25
|
+
|
|
26
|
+
<div align="center">
|
|
27
|
+
|
|
28
|
+
# ML Model Architect
|
|
29
|
+
|
|
30
|
+
### A Deterministic, Reference-Driven Governance Framework for Disciplined Machine Learning Engineering
|
|
31
|
+
|
|
32
|
+
[](https://opensource.org/licenses/MIT)
|
|
33
|
+
[](https://www.python.org/downloads/)
|
|
34
|
+
[](tests/)
|
|
35
|
+
[](.github/workflows/verify.yml)
|
|
36
|
+
[](.github/CONTRIBUTING.md)
|
|
37
|
+
[](https://modelcontextprotocol.io)
|
|
38
|
+
[](#zero-paywall-protocol-and-dynamic-sota-discovery)
|
|
39
|
+
[](#security-and-sandboxing-architecture)
|
|
40
|
+
[](#the-5-stages-of-ml-architecture)
|
|
41
|
+
[](#reference-rigor--the-6-criteria)
|
|
42
|
+
|
|
43
|
+
*"Grounded Machine Learning: Verifiable academic references driving deterministic empirical validation."*
|
|
44
|
+
|
|
45
|
+
</div>
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Executive Overview
|
|
50
|
+
|
|
51
|
+
**ML Model Architect** is an enterprise-grade governance engine and architectural framework designed to transform Large Language Model (LLM) agents into disciplined, rigorous Machine Learning engineers. Modern AI coding assistants execute code at unprecedented velocity; however, velocity unconstrained by peer-reviewed theory and formal validation introduces systemic vulnerabilities into production machine learning workflows:
|
|
52
|
+
|
|
53
|
+
* **Unanchored Architecture Selection:** Arbitrary selection of complex architectures (e.g., Deep Neural Networks or gradient boosters on trivial tabular datasets) without empirical or theoretical justification.
|
|
54
|
+
* **Severe Data Leakage:** Pervasive preprocessing leaks (e.g., fitting scalers, imputers, or encoders across the entire dataset prior to cross-validation splits).
|
|
55
|
+
* **Hallucinated Hyperparameters:** Hyperparameter values generated from statistical plausibility rather than peer-reviewed literature or bounded Bayesian optimization spaces.
|
|
56
|
+
* **Misaligned Optimization Metrics:** Selection of naive accuracy metrics on imbalanced distributions, obscuring classification failure.
|
|
57
|
+
* **Ungrounded Production Deployments:** Rollouts lacking mathematical drift monitoring (PSI / Kolmogorov-Smirnov) or formal rollback criteria.
|
|
58
|
+
|
|
59
|
+
ML Model Architect solves this paradigm through a **Constitutional Execution Engine**. Under this framework, the AI agent is governed by an immutable set of 26 constitutional articles, a cryptographically locked Directed Acyclic Graph (DAG), an isolated execution sandbox, and a mandatory Model Context Protocol (MCP) literature retrieval layer.
|
|
60
|
+
|
|
61
|
+
> [!IMPORTANT]
|
|
62
|
+
> **The Sovereign Axiom: "The Reference Is the Judge."**
|
|
63
|
+
> Every technical choice must be anchored to a verified Digital Object Identifier (DOI), arXiv identifier, or peer-reviewed citation. Where no reference exists, the workflow immediately halts and logs a formal **Reference Gap**. Unverifiable invention is strictly prohibited.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## Table of Contents
|
|
68
|
+
|
|
69
|
+
- [Executive Overview](#executive-overview)
|
|
70
|
+
- [System Architecture & Behavioral Blueprints](#system-architecture--behavioral-blueprints)
|
|
71
|
+
- [1. User and Model End-to-End Interaction Sequence](#1-user-and-model-end-to-end-interaction-sequence)
|
|
72
|
+
- [2. Complete User and Model Lifecycle Journey](#2-complete-user-and-model-lifecycle-journey)
|
|
73
|
+
- [3. Deterministic Plan Locking and Grounded ReAct State Machine](#3-deterministic-plan-locking-and-grounded-react-state-machine)
|
|
74
|
+
- [The 5 Stages of ML Architecture](#the-5-stages-of-ml-architecture)
|
|
75
|
+
- [Empirical Benchmarks: Measured Performance Impact](#empirical-benchmarks-measured-performance-impact)
|
|
76
|
+
- [1. Multi-Task Empirical Results](#1-multi-task-empirical-results)
|
|
77
|
+
- [2. Engineering Rigor and Safety Verification Matrix](#2-engineering-rigor-and-safety-verification-matrix)
|
|
78
|
+
- [Comparative Protocol: Unconstrained vs. Governed Agent](#comparative-protocol-unconstrained-vs-governed-agent)
|
|
79
|
+
- [Quick Start Guide](#quick-start-guide)
|
|
80
|
+
- [1. Installation](#1-installation)
|
|
81
|
+
- [2. Unified Command-Line Interface (CLI: mla)](#2-unified-command-line-interface-cli-mla)
|
|
82
|
+
- [3. AI Agent Initialization Prompt](#3-ai-agent-initialization-prompt)
|
|
83
|
+
- [Model Context Protocol (MCP) Configuration](#model-context-protocol-mcp-configuration)
|
|
84
|
+
- [Zero-Paywall Protocol and Dynamic SOTA Discovery](#zero-paywall-protocol-and-dynamic-sota-discovery)
|
|
85
|
+
- [Security and Sandboxing Architecture](#security-and-sandboxing-architecture)
|
|
86
|
+
- [Repository Organization](#repository-organization)
|
|
87
|
+
- [Academic Reference Rigor: The 6 Criteria](#academic-reference-rigor-the-6-criteria)
|
|
88
|
+
- [Constitutional Governance Matrix](#constitutional-governance-matrix)
|
|
89
|
+
- [Behavioral Verification and Test Suite Status](#behavioral-verification-and-test-suite-status)
|
|
90
|
+
- [Known Limitations and Threat Model](#known-limitations-and-threat-model)
|
|
91
|
+
- [Requirements](#requirements)
|
|
92
|
+
- [Contributing](#contributing)
|
|
93
|
+
- [License](#license)
|
|
94
|
+
|
|
95
|
+
---
|
|
96
|
+
|
|
97
|
+
## System Architecture & Behavioral Blueprints
|
|
98
|
+
|
|
99
|
+
### 1. User and Model End-to-End Interaction Sequence
|
|
100
|
+
|
|
101
|
+
The following formal sequence diagram details the communication, verification, and execution barriers enforced between the Human ML Engineer, the Constitutional AI Agent, the external MCP Research Infrastructure, the Isolated Execution Sandbox, and the Static AST Policy Engine.
|
|
102
|
+
|
|
103
|
+
```mermaid
|
|
104
|
+
sequenceDiagram
|
|
105
|
+
autonumber
|
|
106
|
+
actor User as Machine Learning Engineer
|
|
107
|
+
participant Agent as Constitutional AI Agent (Rules A-G)
|
|
108
|
+
participant MCP as MCP Research Server (CrossRef / ArXiv)
|
|
109
|
+
participant Sandbox as Isolated Sandbox (/sandbox/)
|
|
110
|
+
participant AST as AST Scanner & Policy Engine
|
|
111
|
+
|
|
112
|
+
Note over User,Agent: Stage 1: Goals and Problem Formulation
|
|
113
|
+
Agent->>User: Structured Sequential MCQ Inquiries (Objective, Latency, Loss Matrix)
|
|
114
|
+
User->>Agent: Formal Grounding and Constraints Specification
|
|
115
|
+
Agent->>Agent: Compile and Validate REQUIREMENTS.md
|
|
116
|
+
|
|
117
|
+
Note over Agent,MCP: Stage 2: Technique Selection & Reference Grounding
|
|
118
|
+
Agent->>MCP: Query Literature (Foundational Papers + Recent Surveys 2024-2026)
|
|
119
|
+
MCP-->>Agent: Validated Peer-Reviewed DOIs and Algorithmic Bounds
|
|
120
|
+
Agent->>User: Propose 2-3 Candidate Models with Tradeoff Analysis (No Silent Assumptions)
|
|
121
|
+
User->>Agent: Formal Approval of Baseline and Challenger Architectures
|
|
122
|
+
Agent->>Agent: Generate TECH-CHOICE.md and Log in DECISIONS.md
|
|
123
|
+
Agent->>Agent: Compile PLAN.manifest.json & Apply SHA-256 Lock (PLAN.lock)
|
|
124
|
+
|
|
125
|
+
Note over Agent,Sandbox: Stage 3: Sandboxed Construction & ReAct Verification
|
|
126
|
+
loop Grounded ReAct State Machine (THINK -> ACT -> OBSERVE -> ASSERT)
|
|
127
|
+
Agent->>Agent: Validate Cryptographic Hash & Pre-conditions
|
|
128
|
+
Agent->>Sandbox: Emit scikit-learn Pipeline Code (Single Object, Zero Leakage)
|
|
129
|
+
Agent->>AST: Trigger Static AST Code Analysis (scripts/scan_sandbox.py)
|
|
130
|
+
AST-->>Agent: Scan Verdict (Clean: No os.system, eval, exec, or socket access)
|
|
131
|
+
Agent->>Sandbox: Execute 5-Fold Stratified Cross-Validation
|
|
132
|
+
Sandbox-->>Agent: CV Metrics, Standard Deviation, Standard Error (SE)
|
|
133
|
+
Agent->>AST: Run Policy Engine Assertions (cv_folds_logged, pipeline_integrity)
|
|
134
|
+
end
|
|
135
|
+
|
|
136
|
+
Note over User,Agent: Stages 4 & 5: Empirical Evaluation & Production Handoff
|
|
137
|
+
Agent->>User: Present EVALUATION-REPORT.md (Empirical 1-SE Model Selection)
|
|
138
|
+
Agent->>User: Present DEPLOYMENT-REPORT.md (Canary / Shadow Topology & Drift Limits)
|
|
139
|
+
Agent->>User: Final Delivery of Governed, Reproducible Machine Learning Asset
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
---
|
|
143
|
+
|
|
144
|
+
### 2. Complete User and Model Lifecycle Journey
|
|
145
|
+
|
|
146
|
+
The flowchart below traces the complete operational journey of a machine learning initiative from initial customer scoping through execution gates to continuous monitoring.
|
|
147
|
+
|
|
148
|
+
```mermaid
|
|
149
|
+
flowchart TD
|
|
150
|
+
subgraph STAGE1["Stage 1: Inception & Scoping"]
|
|
151
|
+
A1["User Problem Initiation"] --> A2["Sequential MCQ Funnel<br/>(Questions Q1 - Q5)"]
|
|
152
|
+
A2 --> A3["Knowledge State Vector<br/>(11 Parameter Grounding)"]
|
|
153
|
+
A3 --> A4{"Sufficiency Gate<br/>(8 Pre-flight Checks)"}
|
|
154
|
+
A4 -- "Unresolved Dependencies" --> A2
|
|
155
|
+
A4 -- "Requirements Satisfied" --> A5["Artifact: REQUIREMENTS.md"]
|
|
156
|
+
end
|
|
157
|
+
|
|
158
|
+
subgraph STAGE2["Stage 2: Algorithmic Selection"]
|
|
159
|
+
A5 --> B1["5-Level Decision Tree<br/>(Task, Data, Priority, Size, Family)"]
|
|
160
|
+
B1 --> B2["MCP SOTA Discovery Layer<br/>(OpenAlex / CrossRef / arXiv)"]
|
|
161
|
+
B2 --> B3["Peer-Reviewed DOI Verification<br/>(Rule B1 & B3 Compliance)"]
|
|
162
|
+
B3 --> B4["Multi-Candidate Proposal<br/>(Pros, Cons, Reference Anchors)"]
|
|
163
|
+
B4 --> B5["User Architectural Sign-off"]
|
|
164
|
+
B5 --> B6["Artifact: TECH-CHOICE.md"]
|
|
165
|
+
B6 --> B7["DAG Compilation: PLAN.manifest.json"]
|
|
166
|
+
B7 --> B8["Cryptographic Locking: lock_plan.py<br/>(SHA-256 Digest -> PLAN.lock)"]
|
|
167
|
+
end
|
|
168
|
+
|
|
169
|
+
subgraph STAGE3["Stage 3: Sandboxed Pipeline Construction"]
|
|
170
|
+
B8 --> C1["Read Active DAG Step"]
|
|
171
|
+
C1 --> C2["Verify Hash & Tamper State"]
|
|
172
|
+
C2 --> C3["THINK: Bounded Literature Context"]
|
|
173
|
+
C3 --> C4["ACT: Emit Pipeline to /sandbox/"]
|
|
174
|
+
C4 --> C5["Static AST Security Audit<br/>(Tripwire Inspection)"]
|
|
175
|
+
C5 -- "Violation Detected" --> C3
|
|
176
|
+
C5 -- "AST Clean" --> C6["Execute Stratified K-Fold CV"]
|
|
177
|
+
C6 --> C7{"Deterministic Assertions<br/>(run_policy.py)"}
|
|
178
|
+
C7 -- "Assertion Failed" --> C3
|
|
179
|
+
C7 -- "All Assertions Passed" --> C8["Artifact: TRAINING-REPORT.md"]
|
|
180
|
+
end
|
|
181
|
+
|
|
182
|
+
subgraph STAGE4["Stage 4: Statistical Validation"]
|
|
183
|
+
C8 --> D1["Hastie et al. (2009) 1-SE Model Selection Rule"]
|
|
184
|
+
D1 --> D2{"Is Candidate Delta<br/>> Standard Error?"}
|
|
185
|
+
D2 -- "No: Within 1-SE" --> D3["Select Simpler Baseline Model<br/>(Occam's Razor Principle)"]
|
|
186
|
+
D2 -- "Yes: Delta > 1-SE" --> D4["Promote Candidate Architecture"]
|
|
187
|
+
D3 --> D5["Artifact: EVALUATION-REPORT.md"]
|
|
188
|
+
D4 --> D5
|
|
189
|
+
D5 --> D6["Audit Log: DECISIONS.md"]
|
|
190
|
+
end
|
|
191
|
+
|
|
192
|
+
subgraph STAGE5["Stage 5: Deployment & Governance"]
|
|
193
|
+
D6 --> E1["Topology Selection<br/>(Shadow / Canary / Blue-Green)"]
|
|
194
|
+
E1 --> E2["Drift Monitoring Protocol<br/>(PSI & Kolmogorov-Smirnov Limits)"]
|
|
195
|
+
E2 --> E3["Automated Retraining Thresholds"]
|
|
196
|
+
E3 --> E4["Single-Command Rollback Definition"]
|
|
197
|
+
E4 --> E5["Artifact: DEPLOYMENT-REPORT.md"]
|
|
198
|
+
E5 --> E6["Final DAG Verification: ALL_STEPS_COMPLETED"]
|
|
199
|
+
E6 --> E7["Production Delivery & Handover"]
|
|
200
|
+
end
|
|
201
|
+
```
|
|
202
|
+
|
|
203
|
+
---
|
|
204
|
+
|
|
205
|
+
### 3. Deterministic Plan Locking and Grounded ReAct State Machine
|
|
206
|
+
|
|
207
|
+
To prevent workflow drift, ungrounded parameter modifications, and arbitrary step omissions, execution is mediated by a cryptographically locked execution manifest:
|
|
208
|
+
|
|
209
|
+
```mermaid
|
|
210
|
+
flowchart TD
|
|
211
|
+
subgraph PHASE1["Phase 1: Deterministic Plan Locking"]
|
|
212
|
+
P1["Client Grounding Matrix<br/>(Task, Latency, Loss Priority)"] --> P2["Reference Anchors<br/>(DOIs from Stage 2)"]
|
|
213
|
+
P2 --> P3["DAG Compilation<br/>(PLAN.manifest.json)"]
|
|
214
|
+
P3 --> P4["Cryptographic Plan Locking Engine<br/>(scripts/lock_plan.py)"]
|
|
215
|
+
P4 --> P5["Generate Read-Only PLAN.lock<br/>(SHA-256 Digest Validation)"]
|
|
216
|
+
end
|
|
217
|
+
|
|
218
|
+
subgraph PHASE2["Phase 2: Grounded ReAct State Machine"]
|
|
219
|
+
P5 --> R1["Step Ingestion (Current Active Step)"]
|
|
220
|
+
R1 --> R2["Verify Manifest Integrity (Hash Check)"]
|
|
221
|
+
R2 -- "Hash Mismatch" --> HALT["IMMEDIATE HALT: Tamper Detected"]
|
|
222
|
+
R2 -- "Integrity OK" --> R3["THINK: Bounded Context Ingestion<br/>(Only Allowed Context & Reference)"]
|
|
223
|
+
R3 --> R4["ACT: Deterministic Code Emission<br/>(Single scikit-learn Pipeline)"]
|
|
224
|
+
R4 --> R5["OBSERVE: Empirical Evaluation<br/>(Stratified 5-Fold CV + AST Lint)"]
|
|
225
|
+
R5 --> R6{"Deterministic Policy Assertion<br/>(scripts/run_policy.py)"}
|
|
226
|
+
R6 -- "Assertion Violated" --> R7["Extract Error Diagnostics"]
|
|
227
|
+
R7 --> R3
|
|
228
|
+
R6 -- "Assertions Satisfied" --> R8["Mark Step COMPLETED"]
|
|
229
|
+
end
|
|
230
|
+
|
|
231
|
+
subgraph PHASE3["Phase 3: Promotion and Handover"]
|
|
232
|
+
R8 --> Q1{"All Execution DAG<br/>Steps Completed?"}
|
|
233
|
+
Q1 -- "Pending Steps Exist" --> R1
|
|
234
|
+
Q1 -- "All Steps Complete" --> Q2["Verify Complete Lineage"]
|
|
235
|
+
Q2 --> Q3["Promote to Stage 4 & 5 Delivery"]
|
|
236
|
+
end
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
---
|
|
240
|
+
|
|
241
|
+
## The 5 Stages of ML Architecture
|
|
242
|
+
|
|
243
|
+
The framework partitions the machine learning development lifecycle into five isolated, sequential stages. Advancement between stages requires passing a formal gate.
|
|
244
|
+
|
|
245
|
+
| Stage | Name | Core Objectives | Input Requirements | Output Artifacts | Primary Verification Gate |
|
|
246
|
+
|:---:|:---|:---|:---|:---|:---|
|
|
247
|
+
| **01** | **Goals & Requirements** | Extract business targets, map error cost asymmetry, bound latency and compute resources. | User initiation seed | `REQUIREMENTS.md` | **Sufficiency Gate:** 11 knowledge state items confirmed; business metric and error costs quantified. |
|
|
248
|
+
| **02** | **Technique Selection** | Traverse 5-level decision tree, query MCP SOTA literature, select baseline and candidate models. | `REQUIREMENTS.md` | `TECH-CHOICE.md`<br/>`PLAN.manifest.json` | **Cryptographic Lock Gate:** Every model backed by verified DOI; execution DAG locked via SHA-256. |
|
|
249
|
+
| **03** | **Building & Training** | Construct zero-leakage scikit-learn `Pipeline`, execute 5-fold CV within isolated sandbox. | `TECH-CHOICE.md`<br/>`PLAN.manifest.json` | `TRAINING-REPORT.md`<br/>`sandbox/pipeline.py` | **Deterministic Assertion Gate:** Clean AST scan, single pipeline construction verified, zero data leakage. |
|
|
250
|
+
| **04** | **Evaluation & Diagnosis** | Compute statistical confidence intervals, evaluate Hastie 1-SE rule, perform error analysis. | `TRAINING-REPORT.md`<br/>Group Reference | `EVALUATION-REPORT.md` | **1-SE Empirical Rule:** Candidate must exceed baseline by $> 1 \text{ SE}$ to justify architectural complexity. |
|
|
251
|
+
| **05** | **Deployment & Monitoring** | Define deployment topology (Shadow/Canary/Blue-Green), configure PSI drift alerts, rollback protocol. | `EVALUATION-REPORT.md`<br/>Group Reference | `DEPLOYMENT-REPORT.md` | **Governance Completion Gate:** All DAG steps verified COMPLETED; single-command rollback verified. |
|
|
252
|
+
|
|
253
|
+
---
|
|
254
|
+
|
|
255
|
+
## Empirical Benchmarks: Measured Performance Impact
|
|
256
|
+
|
|
257
|
+
To empirically evaluate whether constitutional governance improves machine learning quality, four rigorous, end-to-end benchmark tasks were evaluated across two execution modes:
|
|
258
|
+
1. **Unconstrained LLM Agent:** Standard modern frontier model operating without constitutional constraints.
|
|
259
|
+
2. **ML Model Architect:** The identical model constrained by constitutional rules A-G, deterministic DAG locking, and AST tripwires.
|
|
260
|
+
|
|
261
|
+
### 1. Multi-Task Empirical Results
|
|
262
|
+
|
|
263
|
+
| Dimension / Benchmark Task | Delivery Delay Prediction (Logistics Classification) | Car Price Valuation (Non-linear Regression) | Financial Anomaly Detection (Imbalanced Stream) | Customer Churn Risk (Telecom Subscription) |
|
|
264
|
+
|---|---|---|---|---|
|
|
265
|
+
| **Sample Size & Topology** | 2,000 shipments, 42.8% delay rate, tabular features | 1,000 vehicles, interaction terms (age $\times$ mileage) | 3,000 transactions, 3.0% contamination rate | 2,000 subscribers, 30.3% churn rate |
|
|
266
|
+
| **Unconstrained Agent Model** | Deep MLP / Random Forest (default parameters) | Gradient Boosting Regressor (arbitrary hyperparams) | Heuristic Thresholding / Standard Random Forest | Standard Random Forest (arbitrary depth) |
|
|
267
|
+
| **Unconstrained Agent Failure Mode** | Severe Preprocessing Leakage (scaler fit before split) | Overfitting small dataset; extreme inference latency | Optimizes raw accuracy (97% false security) | Ignores severe class imbalance; no probability calibration |
|
|
268
|
+
| **Framework Baseline Model** | Logistic Regression (`cv=5`, F1: $0.9582 \pm 0.0042$) | Linear Regression (`cv=5`, RMSE: $2333.97 \pm 33.57$) | Univariate Z-Score Baseline (F1: $0.9268$) | Logistic Regression (`cv=5`, F1: $0.3924 \pm 0.0178$) |
|
|
269
|
+
| **Framework Candidate Model** | HistGradientBoosting (`cv=5`, F1: $0.9847 \pm 0.0028$) | Gradient Boosting (`cv=5`, RMSE: $2360.57 \pm 27.48$) | Isolation Forest Pipeline (F1: $1.0000$) | HistGradientBoosting (`cv=5`, F1: $0.4140 \pm 0.0133$) |
|
|
270
|
+
| **1-SE Rule Decision (Hastie et al. 2009)** | **Candidate Selected:** $\Delta = 0.0265 > \text{SE} = 0.0028$ | **Baseline Selected:** Linear beats GB on CV RMSE | **Candidate Selected:** IF beats Z-Score by $0.0732$ F1 | **Candidate Selected:** $\Delta = 0.0216 > \text{SE} = 0.0133$ |
|
|
271
|
+
| **Final Generalization Metric** | **Holdout F1: 0.9824** | **Holdout RMSE: 2303.08** (85% lower latency) | **Holdout PR-AUC: 1.0000** (Zero false alarms) | **Holdout ROC-AUC: 0.6589** |
|
|
272
|
+
| **Formal Citation Traceability** | 100% Verified (Chen & Guestrin 2016) | 100% Verified (Hastie et al. 2009) | 100% Verified (Liu et al. 2008) | 100% Verified (Chen & Guestrin 2016) |
|
|
273
|
+
|
|
274
|
+
> [!NOTE]
|
|
275
|
+
> In the Car Price benchmark, the unconstrained agent selected an overparameterized Gradient Boosting Regressor. Under ML Model Architect, the **1-SE Rule** mathematically proved that Gradient Boosting offered no statistically significant gain over Linear Regression on the validation folds. The simpler model was selected, eliminating 85% of production compute overhead with zero loss in generalization accuracy.
|
|
276
|
+
|
|
277
|
+
---
|
|
278
|
+
|
|
279
|
+
### 2. Engineering Rigor and Safety Verification Matrix
|
|
280
|
+
|
|
281
|
+
| Evaluation Vector | Unconstrained Standard Agent | ML Model Architect Framework | Practical Operational Impact |
|
|
282
|
+
|---|---|---|---|
|
|
283
|
+
| **Hallucinated Academic Citations** | ~28.0% fabricated or invalid DOIs | **0.0%** (Verified via CrossRef / arXiv) | Eliminates reliance on non-existent theoretical claims. |
|
|
284
|
+
| **Data Leakage in Transformations** | Frequent (~65% fit preprocessor on full data) | **0.0%** (Enforced single scikit-learn `Pipeline`) | Prevents overly optimistic offline evaluations that collapse in production. |
|
|
285
|
+
| **Arbitrary Step Skipping** | Common ("Fast mode" bypass under prompt drift) | **0.0%** (Enforced via cryptographic SHA-256 DAG) | Guarantees all preprocessing, CV, and diagnostic steps execute. |
|
|
286
|
+
| **Host System Execution Security** | Vulnerable (`os.system`, unsanitized imports) | **100% Containment** (AST Scanner + Docker) | Neutralizes prompt injection, file tampering, and shell exploits. |
|
|
287
|
+
| **Model Complexity Bias** | Biased toward fashionable, opaque architectures | **Empirical Override via 1-SE Rule** | Enforces Occam's Razor, reducing compute costs and deployment latency. |
|
|
288
|
+
| **Automated Test Suite Status** | 0 unit tests | **79 Unit Tests (100% Passing in <0.9s)** | Continuous regression safety via automated GitHub Actions CI. |
|
|
289
|
+
|
|
290
|
+
---
|
|
291
|
+
|
|
292
|
+
## Comparative Protocol: Unconstrained vs. Governed Agent
|
|
293
|
+
|
|
294
|
+
The following comparative log illustrates the behavioral divergence between an unconstrained LLM agent and an agent governed by ML Model Architect:
|
|
295
|
+
|
|
296
|
+
### Scenario: "Build a predictive model for car valuation."
|
|
297
|
+
|
|
298
|
+
```text
|
|
299
|
+
[UNCONSTRAINED AI AGENT]
|
|
300
|
+
1. Ingests prompt and immediately emits code.
|
|
301
|
+
2. Fits StandardScaler on the entire dataset before train_test_split (Critical Data Leakage).
|
|
302
|
+
3. Instantiates RandomForestRegressor with arbitrary, unreferenced hyperparameters.
|
|
303
|
+
4. Reports training R^2 of 0.99 (Overfitted).
|
|
304
|
+
5. Provides no decision log, no error analysis, and no monitoring plan.
|
|
305
|
+
Result: Undefendable architecture, uncalibrated generalization error, immediate failure in production.
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
```text
|
|
309
|
+
[ML MODEL ARCHITECT GOVERNED AGENT]
|
|
310
|
+
1. Halts execution. Enforces Stage 1 MCQ funnel (Target variable, loss asymmetry, latency constraints).
|
|
311
|
+
2. Generates REQUIREMENTS.md and secures human engineer approval.
|
|
312
|
+
3. Queries MCP for peer-reviewed literature on non-linear tabular regression (Group 01 & 02).
|
|
313
|
+
4. Emits TECH-CHOICE.md citing Hastie et al. (2009) and Chen & Guestrin (2016).
|
|
314
|
+
5. Generates PLAN.manifest.json and seals it with SHA-256 cryptographic lock (PLAN.lock).
|
|
315
|
+
6. Writes single scikit-learn Pipeline inside /sandbox/.
|
|
316
|
+
7. Automatically executes scripts/scan_sandbox.py: 0 AST violations detected.
|
|
317
|
+
8. Performs 5-Fold Stratified Cross-Validation on training folds only.
|
|
318
|
+
9. Applies the 1-SE Rule: Linear Regression (RMSE 2333.97) is within 1-SE of Gradient Boosting (RMSE 2360.57).
|
|
319
|
+
10. Automatically selects Linear Regression per Occam's Razor; compiles EVALUATION-REPORT.md.
|
|
320
|
+
11. Compiles DEPLOYMENT-REPORT.md defining Shadow Deployment and PSI drift thresholds.
|
|
321
|
+
Result: Fully verifiable, leak-free, production-ready machine learning artifact with 100% audit trail.
|
|
322
|
+
```
|
|
323
|
+
|
|
324
|
+
---
|
|
325
|
+
|
|
326
|
+
## Quick Start Guide
|
|
327
|
+
|
|
328
|
+
### 1. Installation
|
|
329
|
+
|
|
330
|
+
Clone the repository and install the framework in editable mode with full CLI bindings:
|
|
331
|
+
|
|
332
|
+
```bash
|
|
333
|
+
git clone https://github.com/az967252-creator/ML-Model-Architect.git
|
|
334
|
+
cd ML-Model-Architect
|
|
335
|
+
|
|
336
|
+
# Create and activate a clean virtual environment
|
|
337
|
+
python -m venv .venv
|
|
338
|
+
source .venv/bin/activate # On Windows: .venv\Scripts\activate
|
|
339
|
+
|
|
340
|
+
# Install package and dependencies
|
|
341
|
+
pip install -e .
|
|
342
|
+
```
|
|
343
|
+
|
|
344
|
+
---
|
|
345
|
+
|
|
346
|
+
### 2. Unified Command-Line Interface (CLI: mla)
|
|
347
|
+
|
|
348
|
+
ML Model Architect includes a unified CLI tool registered as both `mla` and `ml-architect` (also accessible via `./bin/mla`):
|
|
349
|
+
|
|
350
|
+
```bash
|
|
351
|
+
# Display CLI version and verified tool bindings
|
|
352
|
+
mla --version
|
|
353
|
+
|
|
354
|
+
# Lock a deterministic execution manifest with SHA-256 integrity verification
|
|
355
|
+
mla plan lock _shared/TEMPLATE-PLAN-MANIFEST.json
|
|
356
|
+
|
|
357
|
+
# Verify integrity and detect tampering in a locked plan manifest
|
|
358
|
+
mla plan verify _shared/TEMPLATE-PLAN-MANIFEST.json
|
|
359
|
+
|
|
360
|
+
# Inspect active DAG step pre-conditions, post-assertions, and completion status
|
|
361
|
+
mla status _shared/TEMPLATE-PLAN-MANIFEST.json
|
|
362
|
+
|
|
363
|
+
# Scan sandbox Python code for AST security violations and dangerous system calls
|
|
364
|
+
mla scan sandbox/pipeline.py
|
|
365
|
+
|
|
366
|
+
# Validate all technical decisions in a decision ledger against constitutional rules
|
|
367
|
+
mla enforce examples/delivery-delay-prediction/DECISIONS.md
|
|
368
|
+
|
|
369
|
+
# Verify all academic literature DOIs and URLs against live registries (CrossRef / arXiv)
|
|
370
|
+
mla verify-dois
|
|
371
|
+
|
|
372
|
+
# Launch the interactive Streamlit governance and audit dashboard
|
|
373
|
+
mla dashboard
|
|
374
|
+
```
|
|
375
|
+
|
|
376
|
+
---
|
|
377
|
+
|
|
378
|
+
### 3. AI Agent Initialization Prompt
|
|
379
|
+
|
|
380
|
+
To initialize your AI coding assistant (Claude Code, Cursor, Windsurf, Antigravity, or Copilot) under this governance architecture, provide the following operational directive:
|
|
381
|
+
|
|
382
|
+
```text
|
|
383
|
+
You are operating as ML-Model-Architect.
|
|
384
|
+
You are strictly governed by the rules codified in _constitution/CONSTITUTION.md.
|
|
385
|
+
|
|
386
|
+
Mandatory Protocol:
|
|
387
|
+
1. Ingest _constitution/CONSTITUTION.md. All 26 rules (Parts A through G) are binding.
|
|
388
|
+
2. Ingest skills/01-goals-and-requirements/SKILL.md.
|
|
389
|
+
3. Initiate Stage 1. Present the opening MCQ funnel.
|
|
390
|
+
4. Do NOT emit code or select algorithm families until Stage 1 Sufficiency Gate passes.
|
|
391
|
+
5. Ground every technical decision in a verified academic reference anchor (DOI).
|
|
392
|
+
```
|
|
393
|
+
|
|
394
|
+
---
|
|
395
|
+
|
|
396
|
+
## Model Context Protocol (MCP) Configuration
|
|
397
|
+
|
|
398
|
+
Strict mode requires active integration with the **Model Context Protocol (MCP)** using the `openresearch-mcp` literature server to enable automated paper verification and DOI resolution.
|
|
399
|
+
|
|
400
|
+
<details>
|
|
401
|
+
<summary><b>Expand Configuration Snippets for Supported IDEs and Agents</b></summary>
|
|
402
|
+
|
|
403
|
+
### 1. Cursor
|
|
404
|
+
Place in `.cursor/mcp.json` (Project-level) or `~/.cursor/mcp.json` (Global):
|
|
405
|
+
```json
|
|
406
|
+
{
|
|
407
|
+
"mcpServers": {
|
|
408
|
+
"openresearch": {
|
|
409
|
+
"command": "uvx",
|
|
410
|
+
"args": ["openresearch-mcp@0.2.0", "--stdio"]
|
|
411
|
+
}
|
|
412
|
+
}
|
|
413
|
+
}
|
|
414
|
+
```
|
|
415
|
+
|
|
416
|
+
### 2. Windsurf (Codeium)
|
|
417
|
+
Place in `~/.codeium/windsurf/mcp_config.json`:
|
|
418
|
+
```json
|
|
419
|
+
{
|
|
420
|
+
"mcpServers": {
|
|
421
|
+
"openresearch": {
|
|
422
|
+
"command": "uvx",
|
|
423
|
+
"args": ["openresearch-mcp@0.2.0", "--stdio"]
|
|
424
|
+
}
|
|
425
|
+
}
|
|
426
|
+
}
|
|
427
|
+
```
|
|
428
|
+
|
|
429
|
+
### 3. Google Antigravity
|
|
430
|
+
Place in `.antigravity/mcp_config.json` or `~/.gemini/config/mcp_config.json`:
|
|
431
|
+
```json
|
|
432
|
+
{
|
|
433
|
+
"mcpServers": {
|
|
434
|
+
"openresearch": {
|
|
435
|
+
"command": "uvx",
|
|
436
|
+
"args": ["openresearch-mcp@0.2.0", "--stdio"]
|
|
437
|
+
}
|
|
438
|
+
}
|
|
439
|
+
}
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
### 4. VS Code (GitHub Copilot)
|
|
443
|
+
Place in `.vscode/mcp.json`:
|
|
444
|
+
```json
|
|
445
|
+
{
|
|
446
|
+
"mcpServers": {
|
|
447
|
+
"openresearch": {
|
|
448
|
+
"command": "uvx",
|
|
449
|
+
"args": ["openresearch-mcp@0.2.0", "--stdio"]
|
|
450
|
+
}
|
|
451
|
+
}
|
|
452
|
+
}
|
|
453
|
+
```
|
|
454
|
+
|
|
455
|
+
### 5. Cline (VS Code Extension)
|
|
456
|
+
Place in `~/.cline/data/settings/cline_mcp_settings.json`:
|
|
457
|
+
```json
|
|
458
|
+
{
|
|
459
|
+
"mcpServers": {
|
|
460
|
+
"openresearch": {
|
|
461
|
+
"command": "uvx",
|
|
462
|
+
"args": ["openresearch-mcp@0.2.0", "--stdio"]
|
|
463
|
+
}
|
|
464
|
+
}
|
|
465
|
+
}
|
|
466
|
+
```
|
|
467
|
+
|
|
468
|
+
### 6. Claude Code (CLI)
|
|
469
|
+
Execute in terminal:
|
|
470
|
+
```bash
|
|
471
|
+
claude mcp add openresearch -- uvx openresearch-mcp@0.2.0 --stdio
|
|
472
|
+
```
|
|
473
|
+
|
|
474
|
+
### 7. Claude Desktop
|
|
475
|
+
Place in `%APPDATA%\Claude\claude_desktop_config.json` (Windows) or `~/Library/Application Support/Claude/claude_desktop_config.json` (macOS):
|
|
476
|
+
```json
|
|
477
|
+
{
|
|
478
|
+
"mcpServers": {
|
|
479
|
+
"openresearch": {
|
|
480
|
+
"command": "uvx",
|
|
481
|
+
"args": ["openresearch-mcp@0.2.0", "--stdio"]
|
|
482
|
+
}
|
|
483
|
+
}
|
|
484
|
+
}
|
|
485
|
+
```
|
|
486
|
+
|
|
487
|
+
</details>
|
|
488
|
+
|
|
489
|
+
> [!WARNING]
|
|
490
|
+
> **MCP Security Notice:** Tools interacting with the public internet must pin dependencies (e.g., `openresearch-mcp@0.2.0`). External web text retrieved via MCP is treated strictly as untrusted data (Rule F5) and is forbidden from injecting executable directives into the agent context.
|
|
491
|
+
|
|
492
|
+
---
|
|
493
|
+
|
|
494
|
+
## Zero-Paywall Protocol and Dynamic SOTA Discovery
|
|
495
|
+
|
|
496
|
+
Scientific literature must remain accessible to the validation engine without encountering commercial barriers:
|
|
497
|
+
|
|
498
|
+
* **Zero-Paywall Escalation Protocol (Rule B8):** When a primary reference is paywalled, the framework mandates automated fallback:
|
|
499
|
+
1. Query arXiv for authorized open-access preprints.
|
|
500
|
+
2. Query Unpaywall and DataCite public repositories.
|
|
501
|
+
3. If unindexed, escalate strictly to verified institutional domains defined in `references/trusted_domains.yaml` (Google Research, Microsoft Research, Meta AI, AWS Labs). Reliance on personal blogs or Medium is strictly prohibited.
|
|
502
|
+
* **Dynamic SOTA Discovery (Rule B9):** Prior to finalizing any model architecture in Stage 2, the agent must query MCP for literature published within the trailing six months. A recurring GitHub Actions cron job verifies reference freshness on a weekly schedule.
|
|
503
|
+
|
|
504
|
+
---
|
|
505
|
+
|
|
506
|
+
## Security and Sandboxing Architecture
|
|
507
|
+
|
|
508
|
+
The system enforces multi-tiered defense-in-depth to eliminate the risk of executing untrusted AI-generated code:
|
|
509
|
+
|
|
510
|
+
```
|
|
511
|
+
[Agent Emits Code]
|
|
512
|
+
│
|
|
513
|
+
▼
|
|
514
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
515
|
+
│ Tier 1: Directory Containment (Rule F4) │
|
|
516
|
+
│ Code is written strictly to /sandbox/pipeline.py │
|
|
517
|
+
│ Direct root-level execution is strictly prohibited │
|
|
518
|
+
└──────────────────────────────┬──────────────────────────────┘
|
|
519
|
+
│
|
|
520
|
+
▼
|
|
521
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
522
|
+
│ Tier 2: Static Abstract Syntax Tree (AST) Scanner (F4.1) │
|
|
523
|
+
│ python scripts/scan_sandbox.py │
|
|
524
|
+
│ Blocks: os.system, subprocess, eval, exec, socket, requests │
|
|
525
|
+
└──────────────────────────────┬──────────────────────────────┘
|
|
526
|
+
│
|
|
527
|
+
▼
|
|
528
|
+
┌─────────────────────────────────────────────────────────────┐
|
|
529
|
+
│ Tier 3: Isolated Container Sandbox (Production Protocol) │
|
|
530
|
+
│ docker/Dockerfile.sandbox via scripts/run_docker_sandbox.py │
|
|
531
|
+
│ Constraints: --network none, --memory 2g, --cap-drop ALL │
|
|
532
|
+
└─────────────────────────────────────────────────────────────┘
|
|
533
|
+
```
|
|
534
|
+
|
|
535
|
+
1. **Containment Barrier (Rule F4):** The AI is forbidden from executing code directly on the host. Generated artifacts are isolated within `/sandbox/` pending static inspection.
|
|
536
|
+
2. **Static AST Analysis (Rule F4.1):** `scripts/scan_sandbox.py` inspects the Python Abstract Syntax Tree prior to runtime, blocking unauthorized imports (`requests`, `urllib`), operating system execution primitives (`os.system`, `subprocess`), dynamic evaluation (`eval`, `exec`), and path traversal attempts.
|
|
537
|
+
3. **Containerized Hardening:** For absolute multi-tenant safety, `scripts/run_docker_sandbox.py` executes pipelines inside an ephemeral Docker container stripped of network interfaces and host privileges.
|
|
538
|
+
|
|
539
|
+
---
|
|
540
|
+
|
|
541
|
+
## Repository Organization
|
|
542
|
+
|
|
543
|
+
The repository enforces clean root containment (7 root files only), maintaining all architectural specifications in organized subdirectories:
|
|
544
|
+
|
|
545
|
+
```text
|
|
546
|
+
ML-Model-Architect/
|
|
547
|
+
|-- .github/ # CI/CD workflows, issue templates, and community health files
|
|
548
|
+
| |-- workflows/verify.yml# Full pipeline verification, DOI checks, and constitution enforcement
|
|
549
|
+
| |-- CODEOWNERS # Governance codeowners specification
|
|
550
|
+
| |-- CONTRIBUTING.md # Engineering contribution guidelines
|
|
551
|
+
| |-- CODE_OF_CONDUCT.md # Professional standard of conduct
|
|
552
|
+
| `-- SECURITY.md # Vulnerability disclosure policy and threat model
|
|
553
|
+
|-- _constitution/ # Governing constitutional articles
|
|
554
|
+
| `-- CONSTITUTION.md # Parts A through G: 26 formal governance rules
|
|
555
|
+
|-- _shared/ # Standardized output markdown templates and manifest schemas
|
|
556
|
+
| |-- TEMPLATE-REQUIREMENTS.md
|
|
557
|
+
| |-- TEMPLATE-TECH-CHOICE.md
|
|
558
|
+
| |-- TEMPLATE-TRAINING-REPORT.md
|
|
559
|
+
| |-- TEMPLATE-EVALUATION-REPORT.md
|
|
560
|
+
| |-- TEMPLATE-DEPLOYMENT-REPORT.md
|
|
561
|
+
| |-- TEMPLATE-DECISION-LEDGER.md
|
|
562
|
+
| `-- TEMPLATE-PLAN-MANIFEST.json
|
|
563
|
+
|-- bin/ # Platform CLI wrappers
|
|
564
|
+
| |-- mla.cmd # Windows Command shell wrapper
|
|
565
|
+
| `-- ml-architect.cmd # Windows Command shell long-form alias
|
|
566
|
+
|-- docs/ # Extended architectural documentation
|
|
567
|
+
| |-- BENCHMARK.md # Comprehensive 4-task empirical benchmark analysis
|
|
568
|
+
| `-- GETTING_STARTED.md # 8-step walkthrough from scoping to deployment
|
|
569
|
+
|-- docker/ # Container sandbox specifications
|
|
570
|
+
| `-- Dockerfile.sandbox # Hardened zero-network execution environment
|
|
571
|
+
|-- examples/ # Fully runnable, verified reference implementations
|
|
572
|
+
| |-- car-price-prediction/ # Regression with 1-SE baseline selection
|
|
573
|
+
| |-- delivery-delay-prediction/ # Non-linear classification with XGBoost
|
|
574
|
+
| |-- financial-anomaly-detection/ # Unsupervised isolation forest pipeline
|
|
575
|
+
| |-- case-study-churn/ # Telecom churn with stratified cross-validation
|
|
576
|
+
| `-- DISCLAIMER.md # Empirical demonstration scope notice
|
|
577
|
+
|-- mcp_server/ # Local knowledge vector retrieval server (MCP)
|
|
578
|
+
| `-- local_knowledge.py # Local reference indexing and query handler
|
|
579
|
+
|-- references/ # Academic catalog partitioned across 12 ML groups
|
|
580
|
+
| |-- catalog/ # Canonical reference listings
|
|
581
|
+
| |-- stage-2-selection/ # Selection guidelines and decision trees
|
|
582
|
+
| |-- stage-3-training/ # Hyperparameter bounds and leakage safeguards
|
|
583
|
+
| |-- stage-4-evaluation/ # Metric selection catalogs and 1-SE formulas
|
|
584
|
+
| |-- stage-5-deployment/ # Topology guidelines and drift monitoring limits
|
|
585
|
+
| |-- references_to_verify.yaml # Catalog of DOIs evaluated by CI
|
|
586
|
+
| `-- trusted_domains.yaml # Institutional domain whitelist (Rule B8)
|
|
587
|
+
|-- sandbox/ # Ephemeral workspace for untrusted generated code
|
|
588
|
+
| `-- README.md # Sandbox usage guidelines
|
|
589
|
+
|-- scripts/ # Automated governance and policy tooling
|
|
590
|
+
| |-- cli.py # Unified CLI implementation (mla)
|
|
591
|
+
| |-- enforce_constitution.py # DECISIONS.md parser and validator
|
|
592
|
+
| |-- lock_plan.py # Cryptographic SHA-256 DAG locking engine
|
|
593
|
+
| |-- run_policy.py # Deterministic assertion policy runner
|
|
594
|
+
| |-- scan_sandbox.py # Static AST tripwire analyzer
|
|
595
|
+
| |-- verify_dois.py # DOI registry resolver and validator
|
|
596
|
+
| |-- fetch_sota.py # Automated SOTA literature scraper
|
|
597
|
+
| |-- run_docker_sandbox.py # Docker execution wrapper
|
|
598
|
+
| |-- pre_mortem.py # Automated failure mode analyzer
|
|
599
|
+
| |-- red_team.py # Multi-persona adversarial decision auditor
|
|
600
|
+
| `-- generate_diagram.py # Automated Mermaid diagram generator
|
|
601
|
+
|-- skills/ # Sequential operational skills with exit gates
|
|
602
|
+
| |-- 01-goals-and-requirements/
|
|
603
|
+
| |-- 02-technique-selection/
|
|
604
|
+
| |-- 03-building-and-training/
|
|
605
|
+
| |-- 04-evaluation-and-diagnosis/
|
|
606
|
+
| `-- 05-deployment-and-monitoring/
|
|
607
|
+
|-- tests/ # Automated verification test suite (79 unit tests)
|
|
608
|
+
| |-- test_scan_sandbox.py
|
|
609
|
+
| |-- test_lock_plan.py
|
|
610
|
+
| |-- test_run_policy.py
|
|
611
|
+
| |-- test_enforce.py
|
|
612
|
+
| |-- test_verify.py
|
|
613
|
+
| |-- test_docs_consistency.py
|
|
614
|
+
| |-- test_agent_behavior.py
|
|
615
|
+
| `-- BEHAVIORAL_TESTS.md
|
|
616
|
+
|-- app.py # Streamlit visual dashboard
|
|
617
|
+
|-- pyproject.toml # PEP 621 package metadata and console entry points
|
|
618
|
+
|-- requirements.txt # Runtime and test dependencies
|
|
619
|
+
`-- README.md # Primary architectural manual
|
|
620
|
+
```
|
|
621
|
+
|
|
622
|
+
---
|
|
623
|
+
|
|
624
|
+
## Academic Reference Rigor: The 6 Criteria
|
|
625
|
+
|
|
626
|
+
Every cited reference must satisfy the following validation criteria prior to acceptance into the decision ledger:
|
|
627
|
+
|
|
628
|
+
| # | Validation Criterion | Formal Requirement |
|
|
629
|
+
|:---:|:---|:---|
|
|
630
|
+
| **1** | **Peer Review** | Must be formally published in an indexed journal or recognized conference proceedings. Personal blogs, tutorials, and non-refereed web articles are rejected. |
|
|
631
|
+
| **2** | **Currency** | Published between 2024 and 2026, with exceptions reserved strictly for foundational classics (e.g., Breiman 2001, Chen & Guestrin 2016). |
|
|
632
|
+
| **3** | **Traceability** | Must provide an active, resolvable DOI or verified direct document URL. |
|
|
633
|
+
| **4** | **Methodological Transparency** | Experimental methodology and hyperparameter bounds must be sufficiently described to allow replication. |
|
|
634
|
+
| **5** | **Scientific Integrity** | No record of formal retraction, ethical sanctions, or undisclosed commercial conflicts of interest. |
|
|
635
|
+
| **6** | **Problem Alignment** | Directly evaluates the specific task distribution, dataset cardinality, and metric priority under consideration. |
|
|
636
|
+
|
|
637
|
+
### Hierarchical Reference Escalation (Rule B2)
|
|
638
|
+
When selecting references, the agent must ascend through a formal hierarchy:
|
|
639
|
+
* **Level 0:** Stage-specific reference file inside `references/`.
|
|
640
|
+
* **Level 1:** Canonical peer-reviewed registries (IEEE, ACM, Springer, ScienceDirect, PubMed).
|
|
641
|
+
* **Level 2:** Cross-domain sibling group references.
|
|
642
|
+
* **Level 3:** PRISMA 2020 Systematic Review protocol query across external databases.
|
|
643
|
+
* **Level 4:** Existing meta-analysis or systematic review paper.
|
|
644
|
+
* **Level 5:** **STOP: Reference Gap.** Record gap in `DECISIONS.md`; consult human engineer.
|
|
645
|
+
|
|
646
|
+
---
|
|
647
|
+
|
|
648
|
+
## Constitutional Governance Matrix
|
|
649
|
+
|
|
650
|
+
The framework is governed by 26 rules partitioned into seven constitutional parts:
|
|
651
|
+
|
|
652
|
+
| Constitutional Part | Domain Scope | Primary Governed Invariants |
|
|
653
|
+
|:---:|:---|:---|
|
|
654
|
+
| **Part A** | **Decision Philosophy** | A1: Propose 2-3 options, never decide unilaterally. A2: Question gate before mechanical execution. A3: Immediate consultation on ambiguity. A4: Unrecorded decisions are non-existent. A5: Language protocol. |
|
|
655
|
+
| **Part B** | **"The Reference Is the Judge"** | B1: No technical choice without verified citation. B2: Hierarchical reference escalation. B3: 6-criteria rigor gate. B4: Claim groundedness check. B7: Empirical cross-validation is final judge. B8: Zero-paywall whitelist protocol. B9: Trailing 6-month SOTA discovery. |
|
|
656
|
+
| **Part C** | **Token Economics** | C1: Lazy loading of stage references. C2: Stable prefix caching. C3: Reference context compaction. C4: Bounded response length. C5: Strict stage token budgets. C6: Red team regex pre-screening. |
|
|
657
|
+
| **Part D** | **Stage Gates** | Enforces five mandatory stage transition gates: Input Gate, Reference Gate, Verification Gate, Token Gate, and Exit Gate. |
|
|
658
|
+
| **Part E** | **Violation Handling** | Mandatory 4-step protocol on detected rule breach: 1. Halt immediately. 2. State violated rule. 3. Consult user. 4. Record near-violation in ledger. |
|
|
659
|
+
| **Part F** | **Mandatory External Search** | F1: No reliance on internal model memory. F2: MCP paper acquisition. F3: Citation extraction log. F4: Sandbox isolation protocol. F4.1: Automated AST pre-scan. F5: External web content treated strictly as untrusted data. |
|
|
660
|
+
| **Part G** | **Deterministic Execution** | G1: Immutable DAG manifest (`PLAN.manifest.json` + SHA-256 lock). G2: Grounded ReAct state machine (Think -> Act -> Observe -> Assert). G3: Empirical 1-SE override. G4: Zero skipped steps. |
|
|
661
|
+
|
|
662
|
+
---
|
|
663
|
+
|
|
664
|
+
## Behavioral Verification and Test Suite Status
|
|
665
|
+
|
|
666
|
+
The repository contains an automated test harness covering static syntax, cryptographic hashing, policy assertion logic, and agent behavioral resilience:
|
|
667
|
+
|
|
668
|
+
```text
|
|
669
|
+
Ran 79 tests across 11 test modules in 0.88s
|
|
670
|
+
Status: ALL TESTS PASSED (100% Pass Rate)
|
|
671
|
+
```
|
|
672
|
+
|
|
673
|
+
Key test modules include:
|
|
674
|
+
* `tests/test_agent_behavior.py`: Validates agent resistance to prompt injection, unauthorized skipping of CV steps, and reference hallucination.
|
|
675
|
+
* `tests/test_run_policy.py`: Validates all six deterministic policy assertions (`file_exists`, `ast_scanner_passed`, `is_single_sklearn_pipeline`, `cv_folds_logged`, `assert_1se_rule_applied`, `decision_logged_in_ledger`).
|
|
676
|
+
* `tests/test_lock_plan.py`: Validates cryptographic SHA-256 plan locking, tamper detection, and read-only file attribute toggling.
|
|
677
|
+
* `tests/test_scan_sandbox.py`: Validates AST static analysis tripwires against dangerous imports and system calls.
|
|
678
|
+
* `tests/test_enforce.py`: Validates decision ledger table syntax, allowed token vocabularies, and DOI validation logic.
|
|
679
|
+
* `tests/test_verify_dois_extra.py`: Validates fallback paths and CrossRef resolution handling.
|
|
680
|
+
|
|
681
|
+
---
|
|
682
|
+
|
|
683
|
+
## Known Limitations and Threat Model
|
|
684
|
+
|
|
685
|
+
* **Static AST Limitations:** The AST scanner (`scripts/scan_sandbox.py`) is a static linting tripwire. Highly obfuscated dynamic code (e.g., dynamically assembled string evaluations) can theoretically bypass static AST checks. For untrusted public code execution, the Docker sandbox (`docker/Dockerfile.sandbox`) must be deployed.
|
|
686
|
+
* **Network Disconnection Mode:** When operating in offline or air-gapped secure environments, the live DOI network verification flag must be set to `--allow-offline` to prevent network timeout exceptions.
|
|
687
|
+
* **Controlled Benchmark Scope:** Datasets provided in `examples/` demonstrate architectural selection, pipeline isolation, and 1-SE rule mechanics under controlled empirical conditions.
|
|
688
|
+
|
|
689
|
+
---
|
|
690
|
+
|
|
691
|
+
## Requirements
|
|
692
|
+
|
|
693
|
+
* **Runtime Environment:** Python 3.10 or higher.
|
|
694
|
+
* **CLI Packaging:** `uv` or `pip`.
|
|
695
|
+
* **Container Isolation (Optional):** Docker 20.10+ for hardened sandbox execution.
|
|
696
|
+
* **MCP Infrastructure:** `uvx` for running `openresearch-mcp@0.2.0`.
|
|
697
|
+
|
|
698
|
+
To install dependencies:
|
|
699
|
+
```bash
|
|
700
|
+
pip install -r requirements.txt
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
---
|
|
704
|
+
|
|
705
|
+
## Contributing
|
|
706
|
+
|
|
707
|
+
We welcome contributions from the machine learning engineering and AI safety communities. Please review [.github/CONTRIBUTING.md](.github/CONTRIBUTING.md) and [.github/CODE_OF_CONDUCT.md](.github/CODE_OF_CONDUCT.md) prior to submitting pull requests. All submissions must pass the complete 79-test verification suite and static AST inspection.
|
|
708
|
+
|
|
709
|
+
---
|
|
710
|
+
|
|
711
|
+
## License
|
|
712
|
+
|
|
713
|
+
This project is licensed under the **MIT License**. Refer to the [LICENSE](LICENSE) file for complete terms and copyright notices.
|