webai-scanner 0.1.0__tar.gz

webai_scanner-0.1.0/ARCHITECTURE.md

@@ -0,0 +1,228 @@
+# WebAI Architecture & System Design
+
+**WebAI** is an open architectural standard and benchmarking suite designed to establish an **AI Agent Readability & Identity Standard** for the modern web.
+
+---
+
+## 1. The Core Problem: Human-First vs. Agent-First Web
+
+### The 57% Reality
+Cloudflare and major internet infrastructure providers have documented that **over 57% of all web traffic** is generated by automated bots and AI agents. Despite this majority, the internet remains engineered exclusively **human-first**:
+- Web pages are dominated by presentation markup: nested DOM `<div>`s, CSS rules, stylesheets, layout boilerplate, navigation bars, cookie banners, tracking scripts, and SVGs.
+- A human views only the rendered graphical output. However, an AI agent interacting with the site must ingest the entire raw DOM payload into its transformer context window.
+- As an example, the Storage4gaming calculator page contains over **60,000 lines of HTML (~3.2 MB)**. Feeding this markup to an LLM wastes thousands of input tokens and imposes significant latency and compute overhead.
+- This creates massive downstream economic and societal waste: compute cycles that could solve complex tasks are instead squandered reading visual layout code.
+
+### WebAI's Dual-Web Solution
+WebAI establishes an **agent-ready web operating in parallel with the human web**:
+- Essential factual data is surfaced through compact JSON, semantic JSON-LD, or structured Markdown (e.g. `llms.txt`).
+- A 60k-line DOM is distilled down into a ~100-byte semantic object.
+- **Result:** ~75–95% input token reduction, 1.5x–2.5x speedup in agent inference, and zero loss of extraction accuracy.
+
+```
+┌────────────────────────────────────────────────────────┐
+│ Human-First Web: ~1,800 - 3,200,000 bytes               │
+│ [DOM wrappers, styling, tracking, UI boilerplate]      │
+│  ↳ Ingested by Agent = 600 - 50,000+ tokens            │
+└────────────────────────────────────────────────────────┘
+                           vs
+┌────────────────────────────────────────────────────────┐
+│ Agent-First Web: ~100 bytes                            │
+│ [Structured JSON / Semantic Key-Values / llms.txt]     │
+│  ↳ Ingested by Agent = ~100 - 150 tokens               │
+│  ↳ 75% - 95% Token Reduction, 2x+ Latency Improvement  │
+└────────────────────────────────────────────────────────┘
+```
+
+---
+
+## 2. System Architecture & Comparative Data Flow
+
+The following diagram illustrates how WebAI evaluates, benchmarks, and logs token telemetry across local open-weight models on Apple Silicon:
+
+```mermaid
+flowchart TD
+    subgraph Data Sources & Case Studies
+        S4G_HTML[Storage4gaming Live DOM<br/>Human-First Payload: 3.2MB]
+        S4G_Agent[Storage4gaming llms.txt & JSON<br/>Agent-First: ~100 bytes]
+        Steam_HTML[Steam Storefront Page DOM<br/>Human-First Payload: ~250KB]
+        Steam_API[Steam Store API appdetails<br/>Agent-First: ~115 bytes]
+        Catalog[Cross-Platform Steam Discovery<br/>Windows & macOS Local Filesystem]
+    end
+
+    subgraph Client & Calculator Layer
+        S4G_Client[storage4gaming_client.py<br/>S4G Spec Client & llms.txt Generator]
+        Steam_Client[steam_client.py<br/>Steam API Client & Hardware Calculator]
+        
+        S4G_HTML --> S4G_Client
+        S4G_Agent --> S4G_Client
+        Steam_HTML --> Steam_Client
+        Steam_API --> Steam_Client
+        Catalog --> Steam_Client
+    end
+
+    subgraph Inference & Benchmarking Engine
+        Runner[benchmark_runner.py<br/>MLX-LM Sequential Harness]
+        S4G_Client --> Runner
+        Steam_Client --> Runner
+        
+        Runner --> M1[Bucket 1: Sub-4B Models<br/>Llama 3.2, Qwen 2.5, Phi 4]
+        Runner --> M2[Bucket 2: 4B-12B Models<br/>Gemma 4 12B, Gemma 4 E4B, DeepSeek R1]
+    end
+
+    subgraph Telemetry & Persistence Layer
+        Runner --> Telemetry[telemetry_db.py<br/>SQLite Telemetry Engine]
+        Telemetry --> DB[(webai_benchmarks.db)]
+        Telemetry --> CSV[benchmark_results.csv]
+        Telemetry --> DocSync[doc_updater.py<br/>Auto-Doc Sync]
+        DocSync --> Readme[README.md & ARCHITECTURE.md]
+    end
+```
+
+---
+
+## 3. The AIAID (AI Agent Identity) Architecture
+
+To bring transparency and trusted authentication to agent web traversal, WebAI introduces the **AIAID (AI Agent ID)** protocol.
+
+### Why AIAID?
+Today, devices have MAC addresses and serial numbers. However, AI agents traverse the internet anonymously, leading websites to deploy aggressive bot blockers (Cloudflare challenge walls, CAPTCHAs). 
+
+With AIAID:
+1. Legitimate agents identify themselves transparently.
+2. Participating sites grant authenticated agents **instant access to high-density semantic endpoints (`llms.txt`, JSON API)** without CAPTCHAs.
+3. Unverified scrapers continue to receive standard human-facing HTML or bot challenge walls.
+
+```mermaid
+sequenceDiagram
+    autonumber
+    actor User as Human User
+    participant Agent as AI Agent (with AIAID)
+    participant Site as WebAI-Ready Site (Storage4gaming)
+    participant Registry as Public AIAID Depository / Registry
+
+    User->>Agent: "Calculate required storage for my Steam library"
+    Agent->>Site: HTTP GET /calculator (Header: X-AIAID: aiaid_7f9c2b...)
+    Note over Site: Site detects Agent seeking Agent-First Endpoint
+    Site->>Registry: Verify AIAID(aiaid_7f9c2b...)
+    Registry->>Agent: Issue Cryptographic Challenge Nonce
+    Agent-->>Registry: Signed Nonce Response (Private Key)
+    Registry-->>Site: AIAID Verified: Active, Legitimate, Tier: Standard
+    Site-->>Agent: HTTP 200 OK (Clean Semantic JSON / llms.txt)
+    Note over Agent: Consumes ~100 tokens in 0.4s!
+    Agent-->>User: "You need a 1TB NVMe SSD and 16GB RAM."
+```
+
+---
+
+## 4. Storage4gaming Rewrite Blueprint (The Reference Site)
+
+As the founding real-world testbed of WebAI, **Storage4gaming.com** will undergo a complete code rewrite into an **AI-Agent-Ready Architecture**:
+
+```
+storage4gaming.com/
+├── Human Interface (Web Browser)
+│   ├── /calculator/           -> Interactive Visual Hardware Sizing Tool (React / Tailwind)
+│   ├── /games/[slug]/         -> Human-readable hardware review with interactive charts
+│   └── /guides/               -> SSD & RAM buying guides
+│
+├── Agent Interface (AI Ready)
+│   ├── /llms.txt              -> Lightweight Markdown index of verified game hardware specs
+│   ├── /llms-full.txt         -> Complete agent knowledge base with sizing formulas
+│   ├── /api/games/[slug].json -> Ultra-compact JSON payload (~100 bytes)
+│   └── /api/auth/aiaid        -> Verification endpoint for AIAID agent handshake
+│
+└── Smart Content Negotiation Layer
+    └── When Accept: application/json or X-AIAID is present -> Auto-routes to Agent Interface
+```
+
+---
+
+## 5. Component Breakdown
+
+### A. Site Integration Client (`storage4gaming_client.py`)
+- Interfaces with Storage4gaming (`https://www.storage4gaming.com`).
+- Generates three parallel representations for PC game titles:
+  1. `human_first_html`: Real-world DOM cards with CSS layout classes, badge wrappers, and page context (~1,800 to 3,200,000 bytes).
+  2. `agent_first_json`: High-density semantic JSON containing only verified hardware metrics (~100 bytes).
+  3. `agent_first_llmstxt`: Pure markdown entry adhering to the `llms.txt` standard (~120 bytes).
+
+### B. Steam Store Client & Calculator (`steam_client.py`)
+- Interfaces with the official Steam Store API (`store.steampowered.com/api/appdetails`).
+- Evaluates the Steam Storefront desktop HTML page against the official JSON API.
+- Implements the **Hardware Capacity Calculator**:
+  - Parses numeric RAM (GB), Storage (GB), and Drive Type (NVMe SSD, SSD, HDD).
+  - Computes raw storage footprint across all discovered games.
+  - Adds 20% safety margin for patches and swap files.
+  - Recommends commercial SSD drive sizing (500GB, 1TB, 2TB, 4TB).
+  - Recommends system RAM tier (16GB vs 32GB).
+
+### C. Universal MLX Benchmark Harness (`benchmark_runner.py`)
+- Built on Apple's `mlx-lm` framework for native Apple Silicon unified memory acceleration.
+- Enforces **strict sequential execution** (`batch_size=1`, single model resident in RAM at any time).
+- Between model runs, it calls `del model`, `del tokenizer`, `gc.collect()`, and `mx.clear_cache()` to prevent memory leakage.
+- Captures exact tokenizer token counts, prompt eval speed (tok/sec), generation latency, and peak memory.
+
+### D. Telemetry Engine & SQL Database (`telemetry_db.py`)
+- Embedded SQLite engine requiring zero external daemon or background service.
+- Database file: `webai_benchmarks.db`.
+- Automatically computes `token_savings_pct` by pairing the agent-first run against the human-first baseline.
+- Automatically synchronizes all database records to `benchmark_results.csv`.
+
+### E. Catalog Scanner (`catalog_scanner.py`)
+- Offline, zero-auth Steam discovery scanner with native support for both **macOS** and **Windows**:
+  - **macOS:** Scans `~/Library/Application Support/Steam/steamapps/libraryfolders.vdf`, `userdata/<id>/config/localconfig.vdf`, and binary `appcache/appinfo.vdf`.
+  - **Windows:** Auto-detects Steam via Windows Registry (`HKCU\Software\Valve\Steam\SteamPath`) or `C:\Program Files (x86)\Steam`, multi-drive library folders, user account cache, and `appcache\appinfo.vdf`.
+
+### F. Documentation Auto-Updater (`doc_updater.py`)
+- Automatically reads live data from `webai_benchmarks.db` and the Hugging Face cache.
+- Directly updates the markdown tables in `README.md` and `ARCHITECTURE.md` to keep documentation synchronised with real experimental results.
+
+---
+
+## 6. Live Benchmark Performance Summary
+
+The table below is dynamically synchronized with the local SQLite database (`webai_benchmarks.db`):
+
+<!-- BENCHMARK_TABLE_START -->
+
+| Model                        | Target Game      | Human Tokens   | Agent Tokens   | Token Savings   | Human Latency   | Agent Latency   | Speedup     |
+|------------------------------|------------------|----------------|----------------|-----------------|-----------------|-----------------|-------------|
+| `Llama-3.2-1B-Instruct-4bit` | Apex Legends     | 565 tok        | 132 tok        | **76.64%**      | 0.56s           | 0.42s           | 1.3x faster |
+| `Llama-3.2-1B-Instruct-4bit` | Counter-Strike 2 | 567 tok        | 136 tok        | **76.01%**      | 0.99s           | 0.45s           | 2.2x faster |
+| `Llama-3.2-1B-Instruct-4bit` | Halo Infinite    | 565 tok        | 132 tok        | **76.64%**      | 0.50s           | 0.42s           | 1.2x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Apex Legends     | 635 tok        | 125 tok        | **80.31%**      | 4.69s           | 3.45s           | 1.4x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Counter-Strike 2 | 637 tok        | 132 tok        | **79.28%**      | 6.96s           | 4.77s           | 1.5x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Halo Infinite    | 636 tok        | 126 tok        | **80.19%**      | 3.61s           | 3.31s           | 1.1x faster |
+
+<!-- BENCHMARK_TABLE_END -->
+
+---
+
+## 7. Model Evaluation Matrix
+
+<!-- MODEL_STATUS_START -->
+
+| Status       | Bucket   | Est. RAM (4-bit)   | Hugging Face Model ID                            |
+|--------------|----------|--------------------|--------------------------------------------------|
+| **[CACHED]** | <4B      | ~1.0 GB            | `mlx-community/Llama-3.2-1B-Instruct-4bit`       |
+| [NOT CACHED] | <4B      | ~2.2 GB            | `mlx-community/Llama-3.2-3B-Instruct-4bit`       |
+| [NOT CACHED] | <4B      | ~2.1 GB            | `mlx-community/Qwen2.5-3B-Instruct-4bit`         |
+| [NOT CACHED] | <4B      | ~2.5 GB            | `mlx-community/Phi-4-mini-instruct-4bit`         |
+| **[CACHED]** | 4B-12B   | ~7.5 GB            | `mlx-community/gemma-4-12B-it-qat-4bit`          |
+| **[CACHED]** | 4B-12B   | ~3.5 GB            | `mlx-community/gemma-4-e4b-it-OptiQ-4bit`        |
+| [NOT CACHED] | 4B-12B   | ~4.8 GB            | `mlx-community/Qwen2.5-7B-Instruct-4bit`         |
+| [NOT CACHED] | 4B-12B   | ~5.2 GB            | `mlx-community/Llama-3.1-8B-Instruct-4bit`       |
+| [NOT CACHED] | 4B-12B   | ~5.2 GB            | `mlx-community/DeepSeek-R1-Distill-Qwen-8B-4bit` |
+
+<!-- MODEL_STATUS_END -->
+
+---
+
+## 8. Hardware & Memory Profile (16 GB Unified RAM)
+
+The WebAI suite is strictly engineered for Apple Silicon (macOS) with 16 GB unified memory:
+- **macOS & System Reserve:** ~3.5 - 4.5 GB RAM.
+- **Sub-4B Models:** ~0.9 - 2.5 GB peak active memory.
+- **Medium (4B - 12B) Models:** ~3.5 - 7.5 GB peak active memory.
+- **Headroom:** Always maintains >= 4 GB of free headroom, preventing swap file churn and thermal throttling.

webai_scanner-0.1.0/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.5
+Name: webai-scanner
+Version: 0.1.0
+Summary: AI Agent Readability auditor & local Steam game discovery engine
+Author: Alexander Markovski
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# WebAI: The AI-Ready Internet Standard & Token Benchmarking Suite
+
+**WebAI** is an open architectural standard, benchmarking suite, and agent identity framework designed to establish an **AI-Agent-Ready Internet** operating alongside the modern human-first web.
+
+---
+
+## 🌍 The WebAI Manifesto: Why Build This Now?
+
+* **The 57% Reality:** Major internet infrastructure providers like Cloudflare have documented that **over 57% of all web traffic** is now generated by automated systems and AI agents.
+* **The Structural Flaw:** Despite AI being the majority of internet consumers, the web remains engineered exclusively for human visual perception (deeply nested HTML DOM trees, CSS stylesheets, layout wrappers, and tracking scripts).
+* **The Downstream Impact:** AI agents waste up to **95%+ of their I/O token budgets** simply parsing presentation boilerplate. This inflates task completion times, drives up inference costs, and needlessly ties up global GPU clusters that could be solving valuable societal tasks.
+* **The Dual-Web Solution:** Humans keep their rich visual web interfaces, while websites provide a parallel **Agent-Ready Web** with zero presentation bloat.
+
+### The Two Pillars of WebAI
+1. **Pillar 1: Content & Representation Standards**
+   * Standardized `llms.txt` and semantic JSON contracts.
+   * **Storage4gaming.com:** Complete rewrite into the reference AI-agent-ready site.
+   * **Empirical Benchmarks:** Quantifying token and latency savings across open-weight models on Storage4gaming and the Steam Storefront.
+2. **Pillar 2: AI Agent Identity & Authentication (AIAID)**
+   * Cryptographic machine identities for AI agents (analogous to MAC addresses / UUIDs).
+   * A public depository and challenge-response handshake that unlocks high-speed semantic endpoints for verified agents while filtering malicious scrapers.
+
+---
+
+## 🌟 Key Highlights
+
+- **~76% - 95% Token Reduction:** Cuts prompt tokens from thousands of tokens down to ~100 tokens.
+- **1.5x - 2.5x Faster Latency:** Accelerates task completion by stripping away visual rendering layers.
+- **Embedded SQLite Telemetry:** All benchmark runs, tokenizer token counts, throughput (tok/s), latency, and peak RAM are persisted to `webai_benchmarks.db` and synchronized with `benchmark_results.csv`.
+- **Apple Silicon Optimized:** 100% offline inference on Apple Silicon using `mlx-lm` with sequential execution and automatic memory cache clearing to protect 16 GB Unified Memory.
+- **Auto-Updating Documentation:** Live benchmark tables and model cache statuses in this README and `ARCHITECTURE.md` stay automatically synchronized with your local benchmark runs.
+
+---
+
+## 📊 Live Benchmark Performance Summary
+
+> The table below is automatically synchronized with [`webai_benchmarks.db`](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/webai_benchmarks.db) by `doc_updater.py`:
+
+<!-- BENCHMARK_TABLE_START -->
+
+| Model                        | Target Game      | Human Tokens   | Agent Tokens   | Token Savings   | Human Latency   | Agent Latency   | Speedup     |
+|------------------------------|------------------|----------------|----------------|-----------------|-----------------|-----------------|-------------|
+| `Llama-3.2-1B-Instruct-4bit` | Apex Legends     | 565 tok        | 132 tok        | **76.64%**      | 0.56s           | 0.42s           | 1.3x faster |
+| `Llama-3.2-1B-Instruct-4bit` | Counter-Strike 2 | 567 tok        | 136 tok        | **76.01%**      | 0.99s           | 0.45s           | 2.2x faster |
+| `Llama-3.2-1B-Instruct-4bit` | Halo Infinite    | 565 tok        | 132 tok        | **76.64%**      | 0.50s           | 0.42s           | 1.2x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Apex Legends     | 635 tok        | 125 tok        | **80.31%**      | 4.69s           | 3.45s           | 1.4x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Counter-Strike 2 | 637 tok        | 132 tok        | **79.28%**      | 6.96s           | 4.77s           | 1.5x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Halo Infinite    | 636 tok        | 126 tok        | **80.19%**      | 3.61s           | 3.31s           | 1.1x faster |
+
+<!-- BENCHMARK_TABLE_END -->
+
+---
+
+## 🤖 Supported Model Matrix & Local Cache Status
+
+The suite evaluates models across two buckets, all optimized in 4-bit quantization for 16 GB Unified Memory:
+- **Bucket 1 (< 4B):** Ultra-lightweight models for fast edge inference.
+- **Bucket 2 (4B – 12B):** Desktop-class models including Google's **Gemma 4 12B** and **Gemma 4 E4B (8B)**.
+
+> Check your local cache anytime with `python download_models.py --list`:
+
+<!-- MODEL_STATUS_START -->
+
+| Status       | Bucket   | Est. RAM (4-bit)   | Hugging Face Model ID                            |
+|--------------|----------|--------------------|--------------------------------------------------|
+| **[CACHED]** | <4B      | ~1.0 GB            | `mlx-community/Llama-3.2-1B-Instruct-4bit`       |
+| [NOT CACHED] | <4B      | ~2.2 GB            | `mlx-community/Llama-3.2-3B-Instruct-4bit`       |
+| [NOT CACHED] | <4B      | ~2.1 GB            | `mlx-community/Qwen2.5-3B-Instruct-4bit`         |
+| [NOT CACHED] | <4B      | ~2.5 GB            | `mlx-community/Phi-4-mini-instruct-4bit`         |
+| **[CACHED]** | 4B-12B   | ~7.5 GB            | `mlx-community/gemma-4-12B-it-qat-4bit`          |
+| **[CACHED]** | 4B-12B   | ~3.5 GB            | `mlx-community/gemma-4-e4b-it-OptiQ-4bit`        |
+| [NOT CACHED] | 4B-12B   | ~4.8 GB            | `mlx-community/Qwen2.5-7B-Instruct-4bit`         |
+| [NOT CACHED] | 4B-12B   | ~5.2 GB            | `mlx-community/Llama-3.1-8B-Instruct-4bit`       |
+| [NOT CACHED] | 4B-12B   | ~5.2 GB            | `mlx-community/DeepSeek-R1-Distill-Qwen-8B-4bit` |
+
+<!-- MODEL_STATUS_END -->
+
+---
+
+## 🚀 Quickstart
+
+### Prerequisites
+- macOS on Apple Silicon (M-series).
+- Homebrew installed (`/opt/homebrew/bin/brew`).
+- Python 3.11 (`brew install python@3.11`).
+
+### One-Command Setup & Benchmark
+Execute the master runner to set up the environment, run unit tests, and launch a benchmark:
+```bash
+chmod +x run.sh
+./run.sh
+```
+
+---
+
+## 🛠️ CLI Usage Guide
+
+### 1. Download / Install Models Without Running Inference
+Download model weights directly to your local Hugging Face cache (`~/.cache/huggingface/hub/`) without loading them into RAM:
+```bash
+# Check cache status of all models:
+.venv/bin/python3 download_models.py --list
+
+# Download a specific model (e.g. Gemma 4 12B):
+.venv/bin/python3 download_models.py --models mlx-community/gemma-4-12B-it-qat-4bit
+
+# Download all models in a bucket:
+.venv/bin/python3 download_models.py --bucket "<4B"
+```
+
+### 2. Run Comparative Token Benchmarks
+Run side-by-side comparative evaluation between human-first HTML and agent-first JSON:
+```bash
+# Benchmark specific models:
+.venv/bin/python3 benchmark_runner.py --models mlx-community/gemma-4-e4b-it-OptiQ-4bit mlx-community/Llama-3.2-1B-Instruct-4bit
+
+# Benchmark a specific game title:
+.venv/bin/python3 benchmark_runner.py --game "Cyberpunk 2077"
+
+# Enable live web scraping from https://www.storage4gaming.com:
+.venv/bin/python3 benchmark_runner.py --live
+```
+
+### 3. Scan Local Steam Game Catalog
+Discover locally installed games and owned account library titles from Steam (Windows & macOS):
+```bash
+# Standard discovery:
+.venv/bin/python3 catalog_scanner.py
+
+# Test with synthetic sample catalog:
+.venv/bin/python3 catalog_scanner.py --sample
+```
+
+### 4. Steam Storefront Specs & Hardware Capacity Calculator
+Query the Steam Store API vs. storefront HTML, and calculate total library storage and RAM:
+```bash
+# Benchmark single Steam game (e.g. Apex Legends):
+.venv/bin/python3 steam_client.py --appid 1172470
+
+# Calculate aggregate storage & RAM for your entire scanned Steam library:
+.venv/bin/python3 steam_client.py --scan
+```
+
+### 5. Update Documentation Automatically
+Keep `README.md` and `ARCHITECTURE.md` updated with the latest SQLite benchmark records and model cache status:
+```bash
+.venv/bin/python3 doc_updater.py
+```
+
+---
+
+## 🗄️ Database & Telemetry Inspection
+
+All metrics are stored in SQLite (`webai_benchmarks.db`) and CSV (`benchmark_results.csv`).
+
+### Query Recent Runs
+```bash
+sqlite3 webai_benchmarks.db "SELECT model_id, site_format, prompt_tokens, token_savings_pct, total_time_sec, peak_memory_mb FROM benchmarks ORDER BY id DESC LIMIT 6;"
+```
+
+### View CSV Log
+```bash
+cat benchmark_results.csv
+```
+
+---
+
+## 📚 Central Documentation (`docs/`)
+
+All architectural designs, research, and design decision logs are organized in the [`docs/`](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs) folder:
+* **[docs/decisions.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/decisions.md)**: Design log covering Skills vs. Scripts, `llms.txt`, local filesystem discovery, PyPI packaging, readability scoring metrics, and the AIAID protocol.
+* **[docs/research.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/research.md)**: Research on publishing platforms (GitHub, Hugging Face Datasets & Spaces, Model Context Protocol / MCP, `llms.txt` directories, and PyPI).
+* **[docs/ARCHITECTURE.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/ARCHITECTURE.md)**: Full system design, Mermaid data flow, AIAID authentication handshake, and Storage4gaming rewrite blueprint.
+* **[docs/WebAI_specification.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/WebAI_specification.md)**: Master project specification and criteria.
+* **[docs/starter.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/starter.md)**: Task requirements and model testing matrix.
+
+---
+
+## 📂 Repository Layout
+
+```
+├── docs/                        # Central documentation folder
+│   ├── decisions.md             # Pending design decisions and architectural trade-offs
+│   ├── research.md              # Research on publishing platforms (GitHub, Hugging Face, MCP)
+│   ├── ARCHITECTURE.md          # Technical design, data flow, and database schema
+│   ├── WebAI_specification.md   # Core project specification
+│   └── starter.md               # Task requirements and model evaluation matrix
+├── llms.txt                     # Standard machine-readable AI agent spec index
+├── llms-full.txt                # Comprehensive agent knowledge base and sizing formulas
+├── README.md                    # Project overview, manifesto, quickstart, and live scoreboard
+├── ARCHITECTURE.md              # Root link to technical design
+├── telemetry_db.py              # SQLite telemetry database engine and CSV exporter
+├── storage4gaming_client.py     # Storage4gaming comparative payload generator (HTML vs JSON vs llms.txt)
+├── steam_client.py              # Steam Storefront API client & aggregate hardware calculator
+├── benchmark_runner.py          # MLX sequential inference and benchmarking harness
+├── download_models.py           # Standalone model downloader/installer
+├── catalog_scanner.py           # Cross-platform Steam game library detector (Windows & macOS)
+├── doc_updater.py               # Documentation auto-synchronizer
+├── test_telemetry.py            # Automated test suite for database and telemetry
+├── run.sh                       # Master setup and execution script
+├── webai_benchmarks.db          # SQLite database storing benchmark records
+└── benchmark_results.csv        # Exported benchmark CSV dataset
+```

webai_scanner-0.1.0/README.md

@@ -0,0 +1,203 @@
+# WebAI: The AI-Ready Internet Standard & Token Benchmarking Suite
+
+**WebAI** is an open architectural standard, benchmarking suite, and agent identity framework designed to establish an **AI-Agent-Ready Internet** operating alongside the modern human-first web.
+
+---
+
+## 🌍 The WebAI Manifesto: Why Build This Now?
+
+* **The 57% Reality:** Major internet infrastructure providers like Cloudflare have documented that **over 57% of all web traffic** is now generated by automated systems and AI agents.
+* **The Structural Flaw:** Despite AI being the majority of internet consumers, the web remains engineered exclusively for human visual perception (deeply nested HTML DOM trees, CSS stylesheets, layout wrappers, and tracking scripts).
+* **The Downstream Impact:** AI agents waste up to **95%+ of their I/O token budgets** simply parsing presentation boilerplate. This inflates task completion times, drives up inference costs, and needlessly ties up global GPU clusters that could be solving valuable societal tasks.
+* **The Dual-Web Solution:** Humans keep their rich visual web interfaces, while websites provide a parallel **Agent-Ready Web** with zero presentation bloat.
+
+### The Two Pillars of WebAI
+1. **Pillar 1: Content & Representation Standards**
+   * Standardized `llms.txt` and semantic JSON contracts.
+   * **Storage4gaming.com:** Complete rewrite into the reference AI-agent-ready site.
+   * **Empirical Benchmarks:** Quantifying token and latency savings across open-weight models on Storage4gaming and the Steam Storefront.
+2. **Pillar 2: AI Agent Identity & Authentication (AIAID)**
+   * Cryptographic machine identities for AI agents (analogous to MAC addresses / UUIDs).
+   * A public depository and challenge-response handshake that unlocks high-speed semantic endpoints for verified agents while filtering malicious scrapers.
+
+---
+
+## 🌟 Key Highlights
+
+- **~76% - 95% Token Reduction:** Cuts prompt tokens from thousands of tokens down to ~100 tokens.
+- **1.5x - 2.5x Faster Latency:** Accelerates task completion by stripping away visual rendering layers.
+- **Embedded SQLite Telemetry:** All benchmark runs, tokenizer token counts, throughput (tok/s), latency, and peak RAM are persisted to `webai_benchmarks.db` and synchronized with `benchmark_results.csv`.
+- **Apple Silicon Optimized:** 100% offline inference on Apple Silicon using `mlx-lm` with sequential execution and automatic memory cache clearing to protect 16 GB Unified Memory.
+- **Auto-Updating Documentation:** Live benchmark tables and model cache statuses in this README and `ARCHITECTURE.md` stay automatically synchronized with your local benchmark runs.
+
+---
+
+## 📊 Live Benchmark Performance Summary
+
+> The table below is automatically synchronized with [`webai_benchmarks.db`](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/webai_benchmarks.db) by `doc_updater.py`:
+
+<!-- BENCHMARK_TABLE_START -->
+
+| Model                        | Target Game      | Human Tokens   | Agent Tokens   | Token Savings   | Human Latency   | Agent Latency   | Speedup     |
+|------------------------------|------------------|----------------|----------------|-----------------|-----------------|-----------------|-------------|
+| `Llama-3.2-1B-Instruct-4bit` | Apex Legends     | 565 tok        | 132 tok        | **76.64%**      | 0.56s           | 0.42s           | 1.3x faster |
+| `Llama-3.2-1B-Instruct-4bit` | Counter-Strike 2 | 567 tok        | 136 tok        | **76.01%**      | 0.99s           | 0.45s           | 2.2x faster |
+| `Llama-3.2-1B-Instruct-4bit` | Halo Infinite    | 565 tok        | 132 tok        | **76.64%**      | 0.50s           | 0.42s           | 1.2x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Apex Legends     | 635 tok        | 125 tok        | **80.31%**      | 4.69s           | 3.45s           | 1.4x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Counter-Strike 2 | 637 tok        | 132 tok        | **79.28%**      | 6.96s           | 4.77s           | 1.5x faster |
+| `gemma-4-e4b-it-OptiQ-4bit`  | Halo Infinite    | 636 tok        | 126 tok        | **80.19%**      | 3.61s           | 3.31s           | 1.1x faster |
+
+<!-- BENCHMARK_TABLE_END -->
+
+---
+
+## 🤖 Supported Model Matrix & Local Cache Status
+
+The suite evaluates models across two buckets, all optimized in 4-bit quantization for 16 GB Unified Memory:
+- **Bucket 1 (< 4B):** Ultra-lightweight models for fast edge inference.
+- **Bucket 2 (4B – 12B):** Desktop-class models including Google's **Gemma 4 12B** and **Gemma 4 E4B (8B)**.
+
+> Check your local cache anytime with `python download_models.py --list`:
+
+<!-- MODEL_STATUS_START -->
+
+| Status       | Bucket   | Est. RAM (4-bit)   | Hugging Face Model ID                            |
+|--------------|----------|--------------------|--------------------------------------------------|
+| **[CACHED]** | <4B      | ~1.0 GB            | `mlx-community/Llama-3.2-1B-Instruct-4bit`       |
+| [NOT CACHED] | <4B      | ~2.2 GB            | `mlx-community/Llama-3.2-3B-Instruct-4bit`       |
+| [NOT CACHED] | <4B      | ~2.1 GB            | `mlx-community/Qwen2.5-3B-Instruct-4bit`         |
+| [NOT CACHED] | <4B      | ~2.5 GB            | `mlx-community/Phi-4-mini-instruct-4bit`         |
+| **[CACHED]** | 4B-12B   | ~7.5 GB            | `mlx-community/gemma-4-12B-it-qat-4bit`          |
+| **[CACHED]** | 4B-12B   | ~3.5 GB            | `mlx-community/gemma-4-e4b-it-OptiQ-4bit`        |
+| [NOT CACHED] | 4B-12B   | ~4.8 GB            | `mlx-community/Qwen2.5-7B-Instruct-4bit`         |
+| [NOT CACHED] | 4B-12B   | ~5.2 GB            | `mlx-community/Llama-3.1-8B-Instruct-4bit`       |
+| [NOT CACHED] | 4B-12B   | ~5.2 GB            | `mlx-community/DeepSeek-R1-Distill-Qwen-8B-4bit` |
+
+<!-- MODEL_STATUS_END -->
+
+---
+
+## 🚀 Quickstart
+
+### Prerequisites
+- macOS on Apple Silicon (M-series).
+- Homebrew installed (`/opt/homebrew/bin/brew`).
+- Python 3.11 (`brew install python@3.11`).
+
+### One-Command Setup & Benchmark
+Execute the master runner to set up the environment, run unit tests, and launch a benchmark:
+```bash
+chmod +x run.sh
+./run.sh
+```
+
+---
+
+## 🛠️ CLI Usage Guide
+
+### 1. Download / Install Models Without Running Inference
+Download model weights directly to your local Hugging Face cache (`~/.cache/huggingface/hub/`) without loading them into RAM:
+```bash
+# Check cache status of all models:
+.venv/bin/python3 download_models.py --list
+
+# Download a specific model (e.g. Gemma 4 12B):
+.venv/bin/python3 download_models.py --models mlx-community/gemma-4-12B-it-qat-4bit
+
+# Download all models in a bucket:
+.venv/bin/python3 download_models.py --bucket "<4B"
+```
+
+### 2. Run Comparative Token Benchmarks
+Run side-by-side comparative evaluation between human-first HTML and agent-first JSON:
+```bash
+# Benchmark specific models:
+.venv/bin/python3 benchmark_runner.py --models mlx-community/gemma-4-e4b-it-OptiQ-4bit mlx-community/Llama-3.2-1B-Instruct-4bit
+
+# Benchmark a specific game title:
+.venv/bin/python3 benchmark_runner.py --game "Cyberpunk 2077"
+
+# Enable live web scraping from https://www.storage4gaming.com:
+.venv/bin/python3 benchmark_runner.py --live
+```
+
+### 3. Scan Local Steam Game Catalog
+Discover locally installed games and owned account library titles from Steam (Windows & macOS):
+```bash
+# Standard discovery:
+.venv/bin/python3 catalog_scanner.py
+
+# Test with synthetic sample catalog:
+.venv/bin/python3 catalog_scanner.py --sample
+```
+
+### 4. Steam Storefront Specs & Hardware Capacity Calculator
+Query the Steam Store API vs. storefront HTML, and calculate total library storage and RAM:
+```bash
+# Benchmark single Steam game (e.g. Apex Legends):
+.venv/bin/python3 steam_client.py --appid 1172470
+
+# Calculate aggregate storage & RAM for your entire scanned Steam library:
+.venv/bin/python3 steam_client.py --scan
+```
+
+### 5. Update Documentation Automatically
+Keep `README.md` and `ARCHITECTURE.md` updated with the latest SQLite benchmark records and model cache status:
+```bash
+.venv/bin/python3 doc_updater.py
+```
+
+---
+
+## 🗄️ Database & Telemetry Inspection
+
+All metrics are stored in SQLite (`webai_benchmarks.db`) and CSV (`benchmark_results.csv`).
+
+### Query Recent Runs
+```bash
+sqlite3 webai_benchmarks.db "SELECT model_id, site_format, prompt_tokens, token_savings_pct, total_time_sec, peak_memory_mb FROM benchmarks ORDER BY id DESC LIMIT 6;"
+```
+
+### View CSV Log
+```bash
+cat benchmark_results.csv
+```
+
+---
+
+## 📚 Central Documentation (`docs/`)
+
+All architectural designs, research, and design decision logs are organized in the [`docs/`](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs) folder:
+* **[docs/decisions.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/decisions.md)**: Design log covering Skills vs. Scripts, `llms.txt`, local filesystem discovery, PyPI packaging, readability scoring metrics, and the AIAID protocol.
+* **[docs/research.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/research.md)**: Research on publishing platforms (GitHub, Hugging Face Datasets & Spaces, Model Context Protocol / MCP, `llms.txt` directories, and PyPI).
+* **[docs/ARCHITECTURE.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/ARCHITECTURE.md)**: Full system design, Mermaid data flow, AIAID authentication handshake, and Storage4gaming rewrite blueprint.
+* **[docs/WebAI_specification.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/WebAI_specification.md)**: Master project specification and criteria.
+* **[docs/starter.md](file:///Users/alexandermarkovski/Coding/In_Progress/WebAI/docs/starter.md)**: Task requirements and model testing matrix.
+
+---
+
+## 📂 Repository Layout
+
+```
+├── docs/                        # Central documentation folder
+│   ├── decisions.md             # Pending design decisions and architectural trade-offs
+│   ├── research.md              # Research on publishing platforms (GitHub, Hugging Face, MCP)
+│   ├── ARCHITECTURE.md          # Technical design, data flow, and database schema
+│   ├── WebAI_specification.md   # Core project specification
+│   └── starter.md               # Task requirements and model evaluation matrix
+├── llms.txt                     # Standard machine-readable AI agent spec index
+├── llms-full.txt                # Comprehensive agent knowledge base and sizing formulas
+├── README.md                    # Project overview, manifesto, quickstart, and live scoreboard
+├── ARCHITECTURE.md              # Root link to technical design
+├── telemetry_db.py              # SQLite telemetry database engine and CSV exporter
+├── storage4gaming_client.py     # Storage4gaming comparative payload generator (HTML vs JSON vs llms.txt)
+├── steam_client.py              # Steam Storefront API client & aggregate hardware calculator
+├── benchmark_runner.py          # MLX sequential inference and benchmarking harness
+├── download_models.py           # Standalone model downloader/installer
+├── catalog_scanner.py           # Cross-platform Steam game library detector (Windows & macOS)
+├── doc_updater.py               # Documentation auto-synchronizer
+├── test_telemetry.py            # Automated test suite for database and telemetry
+├── run.sh                       # Master setup and execution script
+├── webai_benchmarks.db          # SQLite database storing benchmark records
+└── benchmark_results.csv        # Exported benchmark CSV dataset
+```