dialectic 0.1.0

package/examples/kata1/architect-perf-solution_2-rounds.json

@@ -0,0 +1,234 @@
+{
+  "id": "deb-20251029-223256-rlpd",
+  "problem": "A university has greatly expanded its CS course and wants to be able to automate the grading of simple programming assignments.\r\n\r\nUsers: 300+ students per year, plus staff and admin.\r\n\r\nRequirements: Students must be able to upload their source code, which will be run and graded. Grades and runs must be persistent and auditable. There must be a plagiarism detection system involving comparing with other submissions and also submitting to a web-based service (TurnItIn). There must be some level of integration with the University's learning management system (LMS)",
+  "status": "completed",
+  "currentRound": 2,
+  "rounds": [
+    {
+      "roundNumber": 1,
+      "contributions": [
+        {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "type": "proposal",
+          "content": "## 1. Overall Architectural Style  \n\n| Goal | Chosen pattern | Why it fits |\n|------|----------------|-------------|\n| **Isolation of untrusted code** | **Container‑based sandbox** (Docker/Podman) + **resource‑capped execution** | Guarantees that a student’s program cannot affect the rest of the system, and makes it easy to enforce CPU / memory / time limits. |\n| **High‑throughput, bursty submissions** | **Event‑driven, queue‑backed micro‑services** (Kafka / RabbitMQ + worker pool) | Decouples the UI from the heavy‑weight grading work, lets us scale workers independently, and provides natural retry / dead‑letter handling. |\n| **Auditability & persistence** | **Append‑only immutable logs** (Kafka log + write‑once DB) + **relational store for query** | Guarantees a tamper‑evident record of every submission, run, and grade while still supporting reporting dashboards. |\n| **Plagiarism detection** | **Batch‑oriented processing pipeline** (Spark / Flink) + **external TurnItIn API** | Large‑scale similarity checks are CPU‑intensive; a batch framework can parallelise them and schedule periodic runs. |\n| **LMS integration** | **Standardised REST/GraphQL + LTI (Learning Tools Interoperability)** | Most university LMSs (Canvas, Blackboard, Moodle) already support LTI, giving a clean, standards‑based way to push grades and pull roster data. |\n| **Operational simplicity** | **Infrastructure‑as‑Code + Kubernetes** | Provides self‑healing, rolling upgrades, and easy horizontal scaling of all services. |\n\nThe resulting system is a **set of loosely‑coupled services** that communicate primarily through **asynchronous messages** and **well‑defined APIs**.  Synchronous calls are limited to UI‑to‑gateway and LMS‑to‑gateway interactions.\n\n---\n\n## 2. Key Components & Their Responsibilities  \n\n| Component | Primary responsibilities | Main interfaces | Deployment notes |\n|-----------|--------------------------|-----------------|------------------|\n| **Web Front‑End (Student Portal)** | - Auth (SSO via university IdP) <br>- Upload source files (zip, single file) <br>- Show submission history, grades, logs | - HTTPS (REST/GraphQL) to **API Gateway** <br>- LTI launch endpoint for LMS embedding | React/Angular SPA served behind CDN; stateless, can be scaled horizontally. |\n| **API Gateway / Auth Proxy** | - Central entry point <br>- Enforce auth/authorization (role‑based: student, TA, admin) <br>- Rate‑limit uploads | - Public HTTPS <br>- Internal gRPC/REST to downstream services | Envoy/Traefik + OIDC integration; runs in Kubernetes as a Deployment. |\n| **Submission Service** | - Validate payload (size, file type) <br>- Store raw archive in **Object Store** (S3‑compatible) <br>- Emit **SubmissionCreated** event | - Consumes from **API Gateway** <br>- Publishes to **Message Bus** (Kafka topic `submissions`) | Stateless workers; can be autoscaled. |\n| **Execution Sandbox Service** | - Pull submission archive <br>- Spin up a sandbox container with language‑specific image <br>- Run tests (unit‑style test harness) <br>- Capture stdout/stderr, exit code, resource usage <br>- Emit **RunResult** event | - Subscribes to `submissions` <br>- Publishes to `run-results` | Container images are version‑controlled; sandbox pods run in a dedicated node pool with strict security policies. |\n| **Grading Engine** | - Translate test results into a numeric score & feedback <br>- Apply weighting, late‑penalty rules <br>- Persist **Grade** record (PostgreSQL) <br>- Emit **GradeCreated** event | - Subscribes to `run-results` <br>- Writes to DB <br>- Publishes to `grades` | Stateless, can be scaled; business rules live in a config store (e.g., Consul). |\n| **Plagiarism Service** | - Periodically fetch recent submissions (batch) <br>- Run pairwise similarity (MOSS‑style, JPlag, custom tokenisation) <br>- Call TurnItIn API for external check <br>- Store similarity scores & flagged pairs | - Reads from **Object Store** and **DB** <br>- Writes to **Plagiarism DB** (PostgreSQL) <br>- Emits `plagiarism‑alert` events | Runs as a scheduled Spark/Flink job (e.g., nightly). TurnItIn integration uses its REST API with OAuth. |\n| **LMS Integration Service** | - Consume `grades` events <br>- Push final grades to LMS via LTI **Outcome Service** or LMS‑specific API <br>- Pull roster / assignment metadata from LMS (sync) | - LTI / LMS REST <br>- Subscribes to `grades` | Runs as a small worker; retries on LMS failures with exponential back‑off. |\n| **Audit & Logging Service** | - Persist every event (submission, run, grade, plagiarism) to an **append‑only log** (Kafka + immutable object store) <br>- Provide tamper‑evident proof (hash chain) <br>- Expose query API for auditors | - Reads all topics <br>- Writes to **Audit Store** (e.g., Cloud Object Store with versioning) | Separate from primary DB to avoid performance impact. |\n| **Reporting Dashboard (Staff/Admin UI)** | - Show class‑wide statistics, per‑student history, plagiarism reports, grade distributions <br>- Export CSV / PDF | - REST API to **Reporting Service** (read‑only DB replicas) | Can be a separate SPA or integrated into the main portal with role‑based views. |\n| **Infrastructure Services** | - **Object Store** (S3‑compatible) for source archives & logs <br>- **Relational DB** (PostgreSQL) for grades, users, assignment metadata <br>- **Message Bus** (Kafka) for events <br>- **Kubernetes** cluster for orchestration <br>- **CI/CD** pipelines for image builds | - Internal only | All services are provisioned via Terraform/Helm; monitoring via Prometheus + Grafana. |\n\n---\n\n## 3. Data Flow (Typical “Student Submits → Grade Sent to LMS”)\n\n1. **Student** logs in via university SSO → receives JWT from IdP.  \n2. SPA posts a **multipart/form‑data** upload to **API Gateway** with JWT.  \n3. **Gateway** validates JWT, checks role, forwards to **Submission Service**.  \n4. **Submission Service** stores the zip in **Object Store** (key = `submission/{assignmentId}/{studentId}/{timestamp}.zip`).  \n5. Service publishes **`SubmissionCreated`** (metadata + object‑store location) to **Kafka**.  \n6. **Execution Sandbox Service** consumes the event, pulls the archive, creates a sandbox container (language‑specific image), mounts the archive read‑only, runs the pre‑defined test harness (e.g., `pytest` for Python, `JUnit` for Java).  \n7. Sandbox streams stdout/stderr back to the service, which writes a **run log** to the object store and publishes **`RunResult`** (exit code, test pass/fail, resource usage).  \n8. **Grading Engine** consumes `RunResult`, applies the assignment’s grading rubric, writes a **Grade** row (`studentId, assignmentId, score, feedback, timestamp`) to PostgreSQL, and emits **`GradeCreated`**.  \n9. **LMS Integration Service** receives `GradeCreated`, calls the LMS Outcome Service (LTI) to push the numeric grade and optional feedback.  \n10. **Audit Service** records every event to an immutable log; the hash of each event is chained to the previous one, providing tamper‑evidence.  \n11. **Reporting Dashboard** reads from the read‑replica of PostgreSQL (or directly from the audit log) to display up‑to‑date statistics.  \n\n*Plagiarism detection* runs independently (nightly). It pulls all recent submissions from the object store, runs similarity algorithms, stores flagged pairs, and notifies staff via the dashboard and optional email alerts.\n\n---\n\n## 4. Scalability & Performance Considerations  \n\n| Concern | Mitigation |\n|---------|------------|\n| **Burst of submissions (e.g., deadline spike)** | - **Kafka** buffers events; workers can be auto‑scaled based on consumer lag. <br>- **Object Store** scales horizontally, no bottleneck for uploads. |\n| **Heavy sandbox CPU usage** | - Separate node pool with **CPU limits**; schedule workers via Kubernetes **resource quotas**. <br>- Use **pre‑warmed container pools** (e.g., `kata-runtime` or `gvisor`) to reduce container start latency. |\n| **Database write contention** | - Grade writes are low‑volume compared to submissions; use **partitioned tables** by semester/assignment. <br>- Read replicas for reporting. |\n| **Plagiarism similarity O(N²)** | - Use **MinHash / LSH** to prune candidate pairs before full comparison. <br>- Run in a distributed batch engine (Spark) that can scale out. |\n| **TurnItIn API rate limits** | - Queue TurnItIn requests separately, respect throttling, and cache results for a configurable TTL. |\n| **Network I/O for large archives** | - Enforce a reasonable size limit (e.g., 10 MiB). <br>- Use **multipart upload** directly to object store (presigned URLs) to offload the gateway. |\n| **Audit log growth** | - Store logs in **cold‑storage tier** after a retention period (e.g., 2 years) while keeping index metadata in a searchable service (Elastic). |\n\n---\n\n## 5. Failure Modes & Resilience  \n\n| Failure scenario | Detection & Recovery |\n|------------------|----------------------|\n| **Sandbox container crashes / hangs** | - Enforce **cgroup timeouts**; Kubernetes kills after `timeout`. <br>- Worker marks run as *failed* and publishes a `RunResult` with error code. |\n| **Message bus outage** | - Kafka persists to disk; producers retry with exponential back‑off. <br>- If lag exceeds threshold, alert is raised; system can fall back to a **local queue** (persisted on disk) until Kafka recovers. |\n| **Object Store unavailable** | - Use **multi‑zone replication**; fallback to secondary bucket. <br>- Uploads are retried; UI shows “temporary storage issue, please retry”. |\n| **Database outage** | - Primary/replica HA (Patroni or Cloud‑managed). <br>- Grade writes are queued (via a dead‑letter topic) and replayed when DB is back. |\n| **TurnItIn API failure** | - Mark submission as “pending external plagiarism check”. <br>- Scheduler retries later; staff can manually trigger re‑check. |\n| **LMS push fails (network or auth)** | - Store pending outcomes in a **retry table**; exponential back‑off with max attempts. <br>- Notify staff if persistent failure. |\n| **Security breach (malicious code escaping sandbox)** | - Run containers with **unprivileged user**, **seccomp**, **AppArmor** profiles. <br>- Use **hardware‑isolated runtimes** (gVisor/Kata) for extra protection. <br>- Regular penetration testing and audit of sandbox images. |\n\n---\n\n## 6. Operational & Governance Aspects  \n\n| Area | Practices |\n|------|-----------|\n| **Observability** | - **Prometheus** metrics per service (queue lag, sandbox CPU, DB latency). <br>- **Grafana** dashboards for SLA monitoring (e.g., 95th‑percentile grading latency < 2 min). <br>- Centralised **ELK** for logs; alerts via PagerDuty. |\n| **CI/CD** | - All container images built from **immutable Dockerfiles**; scanned with Trivy for vulnerabilities. <br>- Deployments via **ArgoCD** (GitOps) ensuring reproducible roll‑outs. |\n| **Security** | - All traffic TLS‑terminated at the gateway. <br>- JWT validation against university IdP (SAML/OIDC). <br>- Least‑privilege IAM for object store (students can only write to their own prefix). |\n| **Data Retention & Auditing** | - Raw submissions kept for **X years** (policy driven). <br>- Audit log is append‑only; hash chain stored in a **WORM** bucket. |\n| **Compliance** | - GDPR / FERPA: personal data (student IDs) stored encrypted at rest; access logged. |\n| **Capacity Planning** | - Baseline: 300 students × 2 assignments × 2 submissions each ≈ 1 200 uploads per term. <br>- Design for **10×** peak (e.g., 12 000 uploads in a 2‑hour window). |\n| **Support** | - Self‑service UI for resubmission, view logs. <br>- Staff portal to manually re‑grade or override grades (audit‑logged). |\n\n---\n\n## 7. Justification of Choices  \n\n1. **Micro‑service + Event‑driven** – isolates heavy grading work from the UI, enables independent scaling, and provides natural retry semantics.  \n2. **Container sandbox** – the safest, most portable way to run arbitrary student code while still allowing us to support many languages (Java, Python, C++, etc.) by swapping the base image.  \n3. **Kafka + immutable audit log** – guarantees that every step is recorded in order, satisfying the “persistent and auditable” requirement without coupling to the primary relational DB.  \n4. **LTI integration** – the de‑facto standard for university LMSs; avoids custom adapters per LMS and makes future migrations trivial.  \n5. **Batch plagiarism pipeline** – similarity detection is computationally heavy; a scheduled Spark job can be sized appropriately and run off‑peak, while TurnItIn provides an external, legally‑compliant check.  \n6. **Kubernetes + IaC** – gives us automated roll‑outs, self‑healing, and the ability to spin up extra worker nodes for deadline spikes without manual intervention.  \n\nOverall, the architecture balances **security**, **scalability**, **auditability**, and **operational simplicity**, while staying within the technology stack commonly available in university IT environments.\n\n--- \n\n### Next Steps for Implementation  \n\n1. **Prototype** the upload → sandbox → grade pipeline with a single language (e.g., Python) to validate end‑to‑end flow and latency.  \n2. **Define the grading rubric schema** and store it in a version‑controlled config repo.  \n3. **Set up the LTI launch** with a test LMS instance and verify grade push.  \n4. **Run a small‑scale plagiarism batch** (e.g., 50 submissions) to benchmark similarity algorithms and TurnItIn API costs.  \n5. **Iterate on scaling policies** (Kubernetes HPA, Kafka consumer lag thresholds) based on observed load during a pilot semester.  \n\nWith these foundations in place, the system can be rolled out to the full student cohort and extended to additional courses or assignment types as needed.",
+          "metadata": {
+            "latencyMs": 34868,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 3771
+          }
+        },
+        {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "type": "proposal",
+          "content": "## 1.  High‑level Goal  \n\nProvide a **low‑latency, high‑throughput, auditable** service that can:\n\n* Accept ≈ 300 + student submissions per week (≈ 5 – 10 submissions / second at peak)  \n* Run each submission in a **secure sandbox**, compare its output with the instructor‑provided test‑suite and produce a grade  \n* Store every run (source, compiled artefacts, console output, grade, timestamps) for later audit  \n* Run **plagiarism detection** against all prior submissions and an external TurnItIn API  \n* Expose a **stable API** that the university LMS can call to push/pull grades and status  \n\nAll of the above must stay **resource‑efficient** (CPU‑bound work only when needed, minimal idle containers) and be **observable** for capacity planning.\n\n---\n\n## 2.  Quantitative Performance Requirements  \n\n| Metric | Target (95 th percentile) | Rationale |\n|--------|---------------------------|-----------|\n| **Submission‑to‑grade latency** | ≤ 30 s | Students expect near‑real‑time feedback; grading pipelines must finish before the next class period. |\n| **Peak throughput** | 200 submissions / min (≈ 3 /s) sustained, 500 /min burst | Covers exam weeks when many students submit simultaneously. |\n| **Plagiarism detection latency** | ≤ 2 min per batch of 100 submissions | TurnItIn API is slower; batch the work and run asynchronously. |\n| **Storage write latency** | ≤ 10 ms per record (object store) | Guarantees audit logs are persisted quickly. |\n| **API response latency** | ≤ 200 ms for status/grade queries | LMS integration must feel snappy. |\n| **CPU utilisation per sandbox** | ≤ 80 % of allocated quota (no oversubscription) | Prevents noisy‑neighbor effects. |\n| **Memory per sandbox** | 256 MiB (adjustable per language) | Keeps node density high. |\n\n---\n\n## 3.  Architectural Overview (text diagram)\n\n```\n+-------------------+      +-------------------+      +-------------------+\n|  LMS / Front‑End  |<---->|   API Gateway     |<---->|   Auth / Rate‑Lim |\n+-------------------+      +-------------------+      +-------------------+\n                                 |\n                                 v\n                         +-------------------+\n                         |   Submission Svc  |\n                         +-------------------+\n                                 |\n                                 v\n                         +-------------------+\n                         |   Message Queue   |  (Kafka / RabbitMQ)\n                         +-------------------+\n                                 |\n               +-----------------+-----------------+\n               |                                   |\n               v                                   v\n   +-------------------+                 +-------------------+\n   |   Grader Workers  |                 |  Plagiarism Workers|\n   +-------------------+                 +-------------------+\n   (sandboxed Docker)                     (batch, external API)\n               |                                   |\n               v                                   v\n   +-------------------+                 +-------------------+\n   |   Result Store    |<--------------->|   Plagiarism DB   |\n   +-------------------+   (audit log)   +-------------------+\n               |\n               v\n   +-------------------+\n   |   Object Store    |  (S3‑compatible, immutable)\n   +-------------------+\n```\n\n*All components are containerised and orchestrated by Kubernetes (or an equivalent managed service).*\n\n---\n\n## 4.  Component‑level Latency / Throughput Design  \n\n### 4.1 API Gateway & Auth  \n\n* **Stateless reverse‑proxy** (Envoy/NGINX) with **TLS termination**.  \n* **Rate‑limit** per user (e.g., 5 submissions / minute) to protect downstream services.  \n* **Cache** static metadata (assignment description, test‑case list) in an in‑memory store (Redis) – TTL ≈ 5 min – reduces DB round‑trips for every upload.\n\n### 4.2 Submission Service  \n\n* Accepts multipart upload, validates file type, stores the raw source **once** in the object store (key = `<course>/<assignment>/<student>/<timestamp>`).  \n* Immediately **writes a lightweight audit record** (submission ID, user, timestamp) to a write‑optimized relational DB (e.g., PostgreSQL with `INSERT … RETURNING id`).  \n* Publishes a **single message** (`submission_id`) to the queue; the message payload is tiny (≈ 100 B) → **high throughput** and low network overhead.\n\n### 4.3 Message Queue  \n\n* **Kafka** (or RabbitMQ) provides **ordered, durable** delivery and can be horizontally scaled.  \n* Partition key = `assignment_id` → ensures that all submissions for the same assignment are processed by the same set of grader workers (helps cache test‑cases per partition).  \n\n### 4.4 Grader Workers (Sandbox Execution)  \n\n| Concern | Optimisation |\n|---------|--------------|\n| **Cold‑start latency** | Pre‑warm a pool of **idle containers** per language (e.g., 5 per node). Use **Kubernetes pod autoscaler** with a low target utilization (e.g., 30 %). |\n| **Compilation reuse** | Cache compiled artefacts (object files, binaries) in **Redis** keyed by `<source_hash, language, compiler_version>`. Cache hit eliminates compile step → 30 % latency reduction. |\n| **Test‑case loading** | Store test‑case data (input files, expected output) in a **read‑through cache** (Redis) with a TTL of the assignment’s active period. Workers fetch from cache instead of DB or object store. |\n| **Resource isolation** | Run each submission in a **Docker container** limited by cgroups (CPU‑quota, memory‑limit). Use **gVisor** or **Kata Containers** for additional security without large performance penalty. |\n| **Parallelism** | Each worker can process **multiple submissions concurrently** (e.g., 2‑3) if the node has spare cores; the scheduler respects the per‑container CPU quota. |\n| **Result aggregation** | After execution, the worker writes a **single row** to the Result Store (grade, stdout, stderr, execution time) and pushes a **completion event** back to the queue for downstream notifications. |\n\n### 4.5 Result Store (Auditable DB)  \n\n* **Append‑only** tables (no UPDATE/DELETE) – ensures auditability.  \n* Partitioned by `assignment_id` and `submission_date` → improves query performance for reporting.  \n* Use **read replicas** for LMS queries; primary handles only writes from graders.  \n\n### 4.6 Object Store  \n\n* Immutable blobs (source, compiled binary, execution logs).  \n* **Lifecycle policy**: move objects older than 1 year to **cold storage** (e.g., Glacier) – cost optimisation.  \n\n### 4.7 Plagiarism Workers  \n\n* Run **asynchronously** in batches (e.g., every 5 min or when 100 new submissions accumulate).  \n* **Local similarity engine** (MOSS‑style) works on the cached source hashes; results stored in a **dedicated Plagiarism DB** (graph of similarity edges).  \n* **TurnItIn integration**: submit batch of source files via their API, store the returned similarity report URL. Because TurnItIn is external and slower, the worker marks the submission as “Plagiarism pending” and updates the status when the report arrives.  \n\n### 4.8 LMS Integration  \n\n* Expose **RESTful endpoints** (`GET /grades/{student}`, `POST /submissions`) that are **cached** (Redis) for read‑heavy queries (e.g., “show all grades for a class”).  \n* Use **Webhooks** to push grade‑available events to the LMS, reducing polling latency.  \n\n---\n\n## 5.  Caching Strategy Summary  \n\n| Layer | What is cached | Where | TTL / Invalidation |\n|-------|----------------|------|--------------------|\n| **API Gateway** | Assignment metadata, test‑case list | Redis (cluster) | 5 min (or on assignment change) |\n| **Grader Workers** | Compiled binaries, interpreter images | Redis (binary cache) | 24 h or until source hash changes |\n| **Test‑case data** | Input/expected output files | Redis (read‑through) | Assignment end date |\n| **Result queries** | Recent grades (last 24 h) | Redis (LRU) | 10 min |\n| **Plagiarism similarity graph** | Pairwise similarity scores | In‑memory graph cache (optional) | Recomputed per batch |\n| **LMS grade view** | Student’s grade list | Redis (per‑student key) | 2 min or on grade write |\n\nAll caches are **write‑through**: a write to the DB also updates the cache, guaranteeing consistency for the 95 % latency target.\n\n---\n\n## 6.  Resource‑Efficiency Measures  \n\n1. **Auto‑scaling node pool** – Kubernetes Cluster Autoscaler adds nodes when pod pending queue > N, removes idle nodes after a cool‑down.  \n2. **Container pooling** – Maintain a small “warm” pool of language‑specific containers to avoid image pull & container creation on every submission.  \n3. **CPU‑quota per sandbox** – Ensures a single malicious or infinite‑loop program cannot starve other jobs.  \n4. **Memory‑limit & OOM‑killer** – If a submission exceeds its allocation, the container is killed and the run is marked “resource exceeded”.  \n5. **Batch plagiarism** – Runs only once per batch, not per submission, drastically reducing external API calls.  \n6. **Cold‑storage lifecycle** – Older objects moved to cheaper storage, freeing primary bucket I/O bandwidth.  \n\n---\n\n## 7.  Observability & Metrics  \n\n| Metric | Collection | Alert Threshold |\n|--------|------------|-----------------|\n| **Submission queue depth** | Kafka consumer lag | > 500 (scale workers) |\n| **Avg grading latency** | Prometheus histogram (submission → grade) | > 25 s |\n| **Sandbox CPU utilisation** | cAdvisor / kube‑state‑metrics | > 80 % for > 5 min |\n| **Cache hit‑rate (Redis)** | Redis INFO stats | < 70 % (investigate cold starts) |\n| **Plagiarism batch duration** | Worker logs | > 3 min per 100 submissions |\n| **Object store write latency** | S3 SDK metrics | > 15 ms |\n| **API error rate** | Istio/Envoy metrics | > 1 % (investigate) |\n| **Audit log growth** | DB size per day | > 5 GB (plan archiving) |\n\nAll metrics are visualised in Grafana dashboards; alerts are routed to the on‑call team via PagerDuty.\n\n---\n\n## 8.  Performance Testing Plan  \n\n1. **Synthetic Load Generation** – Use **k6** or **Locust** to simulate 300 students uploading concurrently (spike of 500 req/s).  \n2. **End‑to‑End Latency** – Measure time from `POST /submissions` to grade availability (including queue, sandbox, DB write).  \n3. **Stress Test** – Gradually increase submission rate until 95 th percentile latency exceeds 30 s; record the point of saturation (expected ~ 800 req/min).  \n4. **Plagiarism Batch Test** – Submit 1 000 new sources, verify batch processing completes within 2 min and TurnItIn API throttling is respected.  \n5. **Resource Utilisation** – Collect CPU/Memory per node; verify that autoscaler adds nodes before saturation.  \n6. **Cache Effectiveness** – Run the same test twice; compare cache‑hit rates and latency improvements.  \n\nResults feed back into capacity planning (e.g., “X nodes of type m5.large sustain Y submissions per minute”).\n\n---\n\n## 9.  Security & Auditing (Brief)\n\n* **TLS everywhere** – API, internal service‑to‑service (mutual TLS).  \n* **Immutable audit log** – Append‑only tables + write‑once object store; use **WORM** bucket policy for compliance.  \n* **Sandbox hardening** – No network access, limited syscalls, read‑only filesystem except `/tmp`.  \n* **Role‑based access** – Students can only POST their own submission; staff can read all; admin can purge.  \n\n---\n\n## 10.  Summary  \n\n* **Latency‑focused pipeline** – tiny messages, pre‑warmed containers, cached compilation & test data keep the *submission‑to‑grade* path under 30 s.  \n* **Throughput‑ready design** – Kafka‑driven decoupling, horizontal worker scaling, batch plagiarism detection handle exam‑week spikes.  \n* **Caching at every tier** (metadata, compiled artefacts, test cases, recent grades) raises cache‑hit rates > 80 % and reduces DB/I/O load.  \n* **Resource efficiency** – cgroup limits, autoscaling node pool, warm container pool, cold‑storage lifecycle keep cost proportional to actual usage.  \n* **Observability & testing** – Comprehensive metrics, alerts, and load‑testing framework guarantee that SLA targets are met and that capacity can be forecasted.  \n\nImplementing the architecture above gives the university a **robust, scalable, and auditable automated grading platform** that delivers fast feedback to students while staying economical and secure.",
+          "metadata": {
+            "latencyMs": 41046,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 3632
+          }
+        },
+        {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "type": "critique",
+          "content": "**Overall impression**  \nThe proposal is a solid, well‑structured “pipeline‑first” design that hits the headline goals (low latency, high throughput, auditability, and cost‑efficiency).  It follows a classic event‑driven, Kubernetes‑native pattern and makes good use of caching, batching, and observability.  Below is a systematic review that highlights what works, where the design could be tightened, and what risks need to be mitigated before a production rollout.\n\n---\n\n## 1. Strengths\n\n| Area | What the design does well |\n|------|---------------------------|\n| **Clear separation of concerns** | API‑gateway → submission service → durable queue → independent grader / plagiarism workers → immutable storage.  This makes each component independently scalable and testable. |\n| **Latency‑focused pipeline** | Tiny messages on the queue, pre‑warmed containers, and Redis caches for compiled artefacts and test data keep the *submission‑to‑grade* path well under the 30 s target. |\n| **Scalable messaging** | Kafka (or RabbitMQ) gives ordered, durable delivery and natural horizontal scaling. Partitioning by `assignment_id` is a smart way to keep test‑case caches warm on a per‑assignment basis. |\n| **Resource isolation & security** | Use of cgroups + gVisor/Kata containers, read‑only filesystem, and no network access inside the sandbox satisfies the security‑audit requirement while keeping the performance impact modest. |\n| **Observability baked in** | Prometheus‑based histograms, consumer‑lag alerts, cache‑hit metrics, and Grafana dashboards give operators the data they need to keep the SLA in check. |\n| **Cost‑aware lifecycle** | Warm‑container pool, autoscaling node pool, and object‑store lifecycle policies keep idle resources low and long‑term storage cheap. |\n| **Auditable data model** | Append‑only tables, immutable blobs, and WORM bucket policy guarantee a tamper‑evident audit trail. |\n| **Batch plagiarism** | Decoupling TurnItIn calls from the grading path prevents external latency from breaking the 30 s feedback loop. |\n\n---\n\n## 2. Weaknesses & Potential Bottlenecks\n\n| Concern | Why it matters | Suggested mitigation |\n|---------|----------------|----------------------|\n| **Cold‑start pool sizing** | The proposal suggests “5 idle containers per node” but does not tie this number to measured arrival rates or node capacity.  Under a burst (500 req/s) the pool could be exhausted, causing a spike in latency. | Run a short‑term load‑test to derive the *minimum* pool size that keeps 95‑th‑percentile latency < 30 s.  Make the pool size a function of node CPU cores (e.g., `idle_per_core = 0.5`).  Use a *warm‑up* job that periodically refreshes the pool to avoid image‑pull stalls. |\n| **Kafka consumer lag handling** | The alert threshold (`> 500`) is static.  If a single assignment receives a sudden surge, lag can grow quickly and the system may keep queuing submissions even after the alert fires. | Implement *back‑pressure* to the API layer: when consumer lag exceeds a configurable threshold, the submission service should return a `429 Too Many Requests` with a retry‑after header.  Also consider using Kafka’s *pause/resume* API per partition to throttle specific hot assignments. |\n| **Redis as a single point of failure** | Multiple critical caches (metadata, compiled binaries, test cases, recent grades) rely on a single Redis cluster.  A Redis outage would degrade latency dramatically and could break cache‑write‑through consistency. | Deploy Redis in a highly‑available mode (cluster with at least three master nodes and replicas).  Enable persistence (AOF) and configure health‑checks that trigger a fail‑over.  Consider a secondary “fallback” cache (e.g., Memcached) for the most latency‑sensitive path (compiled binary cache). |\n| **Result store write contention** | All grader workers write to the same append‑only table.  At peak load (≈ 500 req/min) the write throughput could become a bottleneck, especially if each row contains large JSON blobs (stdout, stderr). | Partition the result table not only by `assignment_id` but also by a time‑bucket (e.g., hour).  Store large logs in the object store and keep only a reference (URL) in the DB row.  Use `COPY`‑style bulk inserts for batched writes from each worker to reduce transaction overhead. |\n| **Plagiarism batch latency** | The spec allows up to 2 min per batch of 100 submissions, but the batch size is fixed.  If a burst creates 300 new submissions, the system will need three batches, potentially delaying some reports beyond the SLA. | Make the batch size *dynamic*: process up to `N` submissions or `T` seconds, whichever comes first.  For high‑traffic periods, increase the batch size or spin up additional plagiarism workers. |\n| **Network egress limits** | TurnItIn API calls and object‑store writes both consume outbound bandwidth.  A sudden exam‑week spike could saturate the egress quota of the VPC, throttling both plagiarism and audit‑log persistence. | Reserve a dedicated NAT gateway or egress‑optimized subnet for external API calls.  Rate‑limit TurnItIn requests per minute and implement exponential back‑off.  Monitor egress metrics and set alerts before hitting provider limits. |\n| **Testing plan coverage** | The performance testing plan focuses on load and latency but does not address *failure injection* (e.g., node loss, Kafka broker failure, Redis outage).  Resilience under partial failure is a core requirement for an audit‑grade service. | Add chaos‑engineering experiments (using tools like Gremlin or Chaos Mesh) to verify that the system continues to meet latency targets when a subset of workers, a broker, or a cache node disappears. |\n| **Security surface – image supply chain** | The sandbox containers are pulled from a registry at start‑up.  If an image is compromised, every submission could be executed in a malicious environment. | Sign container images (Docker Content Trust / Notary) and enforce verification in the pod spec.  Keep a minimal, reproducible base image per language and rebuild it on a regular cadence. |\n| **Audit‑log growth & retention** | The proposal mentions a 5 GB/day growth alert but does not define a retention policy beyond “cold storage after 1 year”.  Auditable data may need to be retained for longer (e.g., 5‑7 years for accreditation). | Clarify legal retention requirements and design a tiered archive: hot (30 days, fast‑access), warm (1 year, cheaper S3), cold (≥ 5 years, Glacier Deep Archive).  Provide a periodic “purge‑and‑verify” job that validates WORM compliance before moving data. |\n\n---\n\n## 3. Architectural Gaps & Recommendations\n\n1. **Service‑to‑service authentication**  \n   *Current*: “mutual TLS” is mentioned but not detailed.  \n   *Recommendation*: Adopt a zero‑trust framework such as **SPIFFE/SPIRE** to issue short‑lived workload identities.  Store the identity in a Kubernetes secret and rotate automatically.  This eliminates the need for static certificates and simplifies revocation.\n\n2. **Idempotency of submission processing**  \n   Submissions may be retried by the client or re‑queued after a worker crash.  The design does not guarantee that duplicate messages won’t produce duplicate grades.  \n   *Recommendation*: Make the `submission_id` the *primary key* of the result table and enforce `INSERT … ON CONFLICT DO NOTHING`.  Workers should be written to be *idempotent* (check if a result already exists before executing).\n\n3. **Graceful degradation path**  \n   If the grading pipeline is saturated, the system should still accept submissions and guarantee eventual grading.  The current flow will block the API response until the message is enqueued, which could fail under high load.  \n   *Recommendation*: Decouple the API from the queue by using a *write‑ahead log* (e.g., DynamoDB Streams or a small “ingress” table) that always succeeds, then a background job forwards entries to Kafka.  This guarantees a 200 OK to the client even when the queue is temporarily full.\n\n4. **Metrics for “sandbox health”**  \n   The proposal tracks CPU utilisation but not *system‑call* or *network* violations inside the sandbox.  A malicious submission could attempt a denial‑of‑service by exhausting kernel resources.  \n   *Recommendation*: Export sandbox‑level metrics (e.g., `syscalls_per_sec`, `fs_events`, `oom_kill`) via a side‑car exporter (cAdvisor + custom eBPF probes).  Alert on abnormal patterns.\n\n5. **Versioning of test‑case data**  \n   Test cases are cached per assignment, but the design does not address the scenario where an instructor updates a test suite after some submissions have already been graded.  This could lead to inconsistent grading.  \n   *Recommendation*: Store a *test‑suite version* identifier with each submission and make the grader fetch the exact version from the cache or object store.  Invalidate the cache only when a new version is published.\n\n6. **Data‑locality for high‑throughput nodes**  \n   Workers on the same node may compete for the same Redis cache and object‑store endpoints, causing network contention.  \n   *Recommendation*: Use a *node‑local* cache (e.g., `tmpfs` or a local Redis instance) for compiled artefacts that are frequently reused on that node.  Periodically sync the local cache with the central Redis to keep it coherent.\n\n7. **Capacity planning for the object store**  \n   The design assumes S3‑compatible storage but does not discuss request‑rate limits (e.g., 3,500 PUT / second per prefix).  A burst of 500 submissions could generate many concurrent PUTs for source files, logs, and binaries.  \n   *Recommendation*: Use *hashed prefixes* (e.g., `/<hash>/submission/...`) to distribute load across partitions, or enable *S3 Transfer Acceleration* if the latency budget is tight.\n\n---\n\n## 4. Prioritized Action List\n\n| Priority | Action | Rationale |\n|----------|--------|-----------|\n| **1** | **Validate warm‑container pool sizing** with a realistic burst test; make pool size dynamic based on node resources. | Direct impact on the 30 s latency SLA. |\n| **2** | **Implement idempotent submission handling** (unique key, upsert semantics). | Prevents duplicate grades and simplifies recovery after failures. |\n| **3** | **Hard‑enforce Redis HA** (cluster mode, replicas, automatic fail‑over). | Removes a single point of failure for multiple critical caches. |\n| **4** | **Add back‑pressure to the API** (429 on high consumer lag) and an ingress write‑ahead log. | Guarantees graceful degradation under overload. |\n| **5** | **Introduce chaos‑testing** for Kafka, Redis, and worker node loss. | Verifies resilience claims before production. |\n| **6** | **Define test‑suite versioning** and cache invalidation rules. | Ensures grading consistency when instructors modify assignments. |\n| **7** | **Secure the container image supply chain** (image signing, automated rebuilds). | Reduces risk of sandbox compromise. |\n| **8** | **Add sandbox‑health metrics** (syscalls, OOM, network violations). | Early detection of malicious or buggy submissions. |\n| **9** | **Plan long‑term audit‑log retention** with tiered storage and compliance checks. | Meets accreditation requirements and controls storage cost. |\n| **10** | **Distribute object‑store prefixes** to avoid request‑rate throttling. | Prevents hidden bottlenecks during exam spikes. |\n\n---\n\n## 5. Closing remarks\n\nThe architecture is fundamentally sound and aligns well with modern cloud‑native best practices.  By tightening the points above—especially around **idempotency, back‑pressure, cache resilience, and failure injection**—the system will not only meet the stated latency/throughput targets but also survive the inevitable spikes and component failures that occur during high‑stakes exam periods.  Once those refinements are in place, the platform will be a robust, auditable, and cost‑effective foundation for automated grading at scale.",
+          "metadata": {
+            "latencyMs": 17226,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 5933
+          },
+          "targetAgentId": "agent-performance"
+        },
+        {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "type": "critique",
+          "content": "## 1. What works well from a performance‑engineer’s view  \n\n| Area | Why it is a strength |\n|------|----------------------|\n| **Event‑driven, queue‑backed micro‑services** | Decouples the UI from the heavy grading work, gives natural back‑pressure (Kafka lag) and makes it easy to autoscale workers based on measurable metrics. |\n| **Container‑based sandbox with resource caps** | Guarantees that a single student’s program cannot starve the host; CPU‑ and memory‑limits give a predictable upper bound on per‑submission resource consumption. |\n| **Separate object store for archives & logs** | S3‑compatible storage scales horizontally, eliminates a classic I/O bottleneck on the relational DB and lets you off‑load large binary payloads from the request path. |\n| **Immutable audit log (Kafka + write‑once store)** | Guarantees ordered, durable recording without adding write pressure to the primary DB; the hash‑chain gives tamper‑evidence for compliance audits. |\n| **Kubernetes + IaC** | Provides self‑healing, rolling upgrades and a built‑in horizontal‑pod‑autoscaler (HPA) that can react to CPU, memory, or custom metrics (e.g., consumer lag). |\n| **Observability stack (Prometheus/Grafana + ELK)** | Gives you the data you need to spot latency spikes, queue build‑up, or resource saturation before they become SLA breaches. |\n\nThese choices already give you a solid foundation for high‑throughput, burst‑y workloads (e.g., deadline spikes) while keeping the system auditable and secure.\n\n---\n\n## 2. Potential performance bottlenecks & why they matter  \n\n| Component | Likely bottleneck | Impact on SLA (e.g., 95‑pct grading latency < 2 min) |\n|-----------|-------------------|------------------------------------------------------|\n| **Sandbox container start‑up** | Pulling the language image, creating the container, and mounting the archive can take 1‑3 s per submission. In a deadline surge this adds up and can push latency beyond the target. |\n| **Kafka consumer lag** | If the worker pool cannot keep up, the `submissions` topic backlog grows. Lag > 30 s translates directly into grading latency. |\n| **Object‑store upload path** | All uploads currently pass through the API gateway. Large archives (≈10 MiB) can saturate the gateway’s outbound bandwidth and increase request‑time for the student UI. |\n| **Database write contention** | Grades are written to a single PostgreSQL instance. Even though volume is lower than submissions, a burst of 10 k grades in a few minutes can cause lock contention on the `grades` table, inflating write latency and hurting the reporting UI. |\n| **Plagiarism O(N²) similarity** | A naïve pairwise comparison of *N* submissions grows quadratically. Even with Spark, the job can become CPU‑bound and extend the nightly window, delaying detection of cheating. |\n| **TurnItIn API throttling** | External rate limits can cause a backlog of pending plagiarism checks, leading to “pending” status for many recent submissions. |\n| **Audit‑log growth** | Persisting every event to an immutable bucket without tiering can increase storage costs and, if the index is kept in a hot store, degrade query performance for the dashboard. |\n| **Network I/O for test harness output** | Streaming full stdout/stderr for every run can saturate the internal network, especially for languages that produce verbose logs (e.g., Java). |\n\n---\n\n## 3. Concrete performance‑focused improvements  \n\n### 3.1 Reduce sandbox latency  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Warm‑pool of pre‑created containers** (or micro‑VMs such as Firecracker/gVisor) | Eliminates image pull & container creation on each request; you only need to reset the filesystem between runs. |\n| **Cache language images locally on the node pool** (e.g., using a private registry with node‑level image pull‑through) | Guarantees O(1) image fetch time, even after a node restart. |\n| **Parallel test execution inside the container** (e.g., `pytest -n auto`) | For assignments with many unit tests, parallelism can cut execution time by 2‑4× without extra CPU if the node pool has spare cores. |\n| **Limit stdout/stderr size** and optionally truncate logs after a configurable threshold (e.g., 1 MiB) | Prevents a single runaway process from flooding the network and storage. |\n\n### 3.2 Improve queue handling & autoscaling  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Custom HPA metric: Kafka consumer lag** (exposed via Prometheus exporter) | Workers scale out only when the backlog grows, keeping latency bounded while avoiding over‑provisioning. |\n| **Separate “fast‑track” topic for submissions that hit the deadline** | Gives priority to the last‑minute surge; workers can be configured with a higher CPU quota for this topic. |\n| **Back‑pressure to the API gateway** – reject uploads with HTTP 429 when lag > X seconds | Prevents the system from being overwhelmed and gives the UI a clear signal to retry later. |\n\n### 3.3 Off‑load upload traffic  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Presigned‑URL direct‑to‑object‑store upload** (client‑side) | The gateway only validates the JWT and returns a URL; the bulk data bypasses the gateway, reducing its CPU & bandwidth usage. |\n| **Multipart upload with resumable chunks** | Improves reliability for flaky student connections and keeps the average request size small for the gateway. |\n\n### 3.4 Database write scalability  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Partition the `grades` table by semester/assignment** (PostgreSQL native partitioning) | Each partition is a smaller index, reducing lock contention and improving insert throughput. |\n| **Batch insert grades** (e.g., collect up to 100 results before writing) | Reduces per‑row overhead and improves disk I/O efficiency. |\n| **Read‑replica for reporting** (already mentioned) – ensure the replica is kept warm with a low‑latency streaming replication to avoid stale dashboards. |\n\n### 3.5 Plagiarism detection efficiency  \n\n| Action | Reasoning |\n|--------|-----------|\n| **MinHash / Locality‑Sensitive Hashing (LSH) pre‑filter** before full pairwise comparison | Cuts the candidate set from O(N²) to O(N·k) where *k* is the average number of near‑duplicates, dramatically reducing Spark job runtime. |\n| **Incremental nightly runs** – only compare new submissions against the existing corpus, not all‑vs‑all each night. | Keeps the nightly window constant as the course grows. |\n| **Cache TurnItIn results** (keyed by submission hash) with a TTL of 24 h | Avoids repeated external calls for the same code version. |\n\n### 3.6 External API throttling mitigation  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Dedicated “TurnItIn worker pool” with its own rate‑limit token bucket** | Guarantees you never exceed the provider’s limits and isolates back‑pressure from the main grading pipeline. |\n| **Exponential back‑off + persistent dead‑letter queue** | Guarantees eventual consistency without dropping checks. |\n\n### 3.7 Audit‑log storage optimisation  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Hot‑cold tiering** – keep the most recent month in a low‑latency object store (e.g., S3 Standard) and move older logs to Glacier‑compatible cold storage. | Reduces cost and keeps query latency low for the dashboard that typically looks at recent data. |\n| **Index only metadata** (event type, timestamp, hash) in a searchable service (Elasticsearch) while the full payload stays in cold storage. | Enables fast audit queries without pulling large blobs. |\n\n### 3.8 Network & log streaming control  \n\n| Action | Reasoning |\n|--------|-----------|\n| **Compress log streams** (gzip) before writing to the object store | Cuts bandwidth and storage by ~70 % for verbose languages. |\n| **Configurable log‑level per language** – default to `INFO` and allow students to request `DEBUG` only for a limited time. | Prevents accidental log explosion. |\n\n---\n\n## 4. Performance requirements & measurable SLAs  \n\n| Metric | Target (95‑pct) | Measurement point |\n|--------|----------------|-------------------|\n| **End‑to‑end grading latency** (submission → grade persisted) | **≤ 2 min** | Timestamp at API gateway → `GradeCreated` event |\n| **Sandbox start‑up time** | **≤ 1 s** (warm pool) | Container creation log |\n| **Queue lag** (`submissions` topic) | **≤ 30 s** | Kafka consumer lag exporter |\n| **Upload response time (client‑visible)** | **≤ 5 s** for ≤ 10 MiB archive | API gateway latency histogram |\n| **Grade DB write latency** | **≤ 100 ms** per row | PostgreSQL pg_stat_statements |\n| **Plagiarism nightly job duration** | **≤ 2 h** for 10 k submissions | Spark job duration metric |\n| **TurnItIn request success rate** | **≥ 99 %** (within provider limits) | TurnItIn worker success counter |\n| **Audit‑log write latency** | **≤ 200 ms** (append only) | Kafka produce latency |\n\nAll metrics should be exported to Prometheus and visualised on Grafana dashboards with alerts (e.g., “grading latency > 2 min for 5 min”).\n\n---\n\n## 5. Testing & validation plan  \n\n1. **Synthetic load generator** (Locust or k6) that mimics the real submission pattern:  \n   *Burst phase* – 10 k submissions in 2 h (deadline spike).  \n   *Steady phase* – 300 submissions per hour.  \n   Capture queue lag, container start‑up, and end‑to‑end latency.\n\n2. **Container‑warm‑pool benchmark** – compare cold vs. warm start times across all supported languages; verify the warm‑pool size needed to keep 95‑pct start‑up ≤ 1 s.\n\n3. **Object‑store direct‑upload test** – measure gateway CPU & bandwidth before/after presigned‑URL implementation.\n\n4. **Database write stress test** – bulk insert grades (batch size 100) while the grading workers are running; monitor lock wait time and replication lag.\n\n5. **Plagiarism pipeline scaling test** – generate N = 20 k synthetic submissions, run the MinHash‑LSH pre‑filter, then full comparison on a Spark cluster; verify that job duration stays under the 2 h target.\n\n6. **Failure‑injection drills**:  \n   - Kill a Kafka broker → verify producer retry and consumer lag recovery.  \n   - Simulate sandbox OOM → ensure the worker marks the run as failed and continues.  \n   - Block TurnItIn endpoint → confirm exponential back‑off and dead‑letter handling.\n\n7. **Observability sanity check** – inject known latency spikes and confirm that alerts fire (e.g., latency > 2 min, queue lag > 30 s).\n\nAll test results should be recorded in a performance baseline document and used to size the initial Kubernetes node pools, Kafka broker count, and object‑store throughput limits.\n\n---\n\n## 6. Summary of recommendations  \n\n| Area | Immediate win | Longer‑term investment |\n|------|---------------|------------------------|\n| **Sandbox latency** | Warm‑pool + local image cache (quick to implement) | Move to micro‑VMs (Firecracker) for stronger isolation & faster start‑up |\n| **Queue & autoscaling** | Export consumer lag, hook to HPA | Separate fast‑track topic for deadline spikes |\n| **Upload path** | Add presigned‑URL flow | Add CDN edge‑cache for static assets (SPA) |\n| **DB writes** | Partition `grades` table, batch inserts | Consider event‑sourced write‑model (grades as events) to further decouple from relational DB |\n| **Plagiarism** | Add MinHash/LSH pre‑filter | Incremental similarity detection (only compare new vs. existing) |\n| **External API** | Dedicated TurnItIn worker pool with token bucket | Negotiate higher rate limits or cache results longer |\n| **Audit log** | Hot/cold tiering + metadata index | Provide a tamper‑evident Merkle‑tree proof service for compliance audits |\n| **Observability** | Deploy Prometheus exporters for Kafka lag, container start‑up, DB latency | Add SLO‑based alerting (error‑budget burn‑rate) |\n\nImplementing the “quick wins” (warm sandbox pool, direct upload, consumer‑lag‑driven autoscaling, and DB partitioning) will immediately bring the system within the 2‑minute grading latency target for typical loads. The longer‑term investments (micro‑VM isolation, smarter plagiarism pipelines, and hardened audit‑log querying) will keep the platform scalable as course enrollment grows and as more language runtimes are added.",
+          "metadata": {
+            "latencyMs": 33979,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 7004
+          },
+          "targetAgentId": "agent-architect"
+        },
+        {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "type": "refinement",
+          "content": "## Refined Architecture – “Secure, Auditable, High‑Throughput Grading Platform”\n\nBelow is the original design sharpened with the performance‑engineer’s observations.  \nThe changes are grouped by **problem → concrete mitigation → impact** and then re‑presented as an updated high‑level view, component diagram, data‑flow, and operational checklist.\n\n---\n\n### 1.  Core Architectural Style (unchanged)\n\n| Goal | Pattern | Why it still fits |\n|------|---------|-------------------|\n| Isolation of untrusted code | **Container‑/micro‑VM sandbox** (Docker + gVisor/Firecracker) | Guarantees strong security while allowing fast start‑up (warm‑pool). |\n| Burst‑y submissions | **Event‑driven, queue‑backed micro‑services** (Kafka) | Decouples UI from heavy work, provides natural back‑pressure, and lets us autoscale workers on lag. |\n| Auditability | **Append‑only immutable log** (Kafka + write‑once object store) | Guarantees ordered, tamper‑evident records without stressing the primary DB. |\n| Plagiarism detection | **Batch pipeline with MinHash‑LSH pre‑filter** (Spark/Flink) + TurnItIn API | Reduces O(N²) work to near‑linear, keeping nightly jobs within a 2 h window. |\n| LMS integration | **LTI / REST** | De‑facto university standard, no per‑LMS adapters. |\n| Operational simplicity | **IaC + Kubernetes** | Self‑healing, rolling upgrades, and HPA driven by custom metrics (Kafka lag, sandbox latency). |\n\n---\n\n### 2.  Updated Component Landscape\n\n| Component | New / Modified Behaviour | Key Interfaces |\n|-----------|--------------------------|----------------|\n| **Web Front‑End (SPA)** | Uploads now use **presigned‑URL** flow; UI obtains a short‑lived URL from the gateway and streams the archive directly to the object store. | HTTPS (REST) → API Gateway (JWT + presign request) |\n| **API Gateway / Auth Proxy** | 1️⃣ Validate JWT 2️⃣ Issue presigned URL (S3‑compatible) 3️⃣ Return URL to SPA 4️⃣ Accept a lightweight “submission‑metadata” POST (no binary payload). | Public HTTPS (REST) → internal gRPC/REST |\n| **Submission Service** | Receives only metadata (assignmentId, studentId, object‑store key). Stores a **SubmissionCreated** event. | Consumes from Gateway → Publishes to Kafka (`submissions`) |\n| **Sandbox Service** | **Warm‑pool of pre‑created containers / micro‑VMs** (e.g., Firecracker VMs kept idle). On a new submission the pool hands over a ready sandbox, mounts the archive, runs the test harness, then **re‑images** the VM (fast reset). | Subscribes `submissions` → Publishes `run‑results` |\n| **Grading Engine** | **Batch‑insert** grades (collect up to 100 results) into a **partitioned** `grades` table (partitioned by semester‑assignment). | Subscribes `run‑results` → Writes to PostgreSQL → Publishes `grades` |\n| **Plagiarism Service** | **MinHash‑LSH pre‑filter** to generate candidate pairs, then full similarity only on those. Runs **incrementally** (new submissions vs. existing corpus). Caches TurnItIn results keyed by submission hash. | Reads from Object Store & DB → Writes to Plagiarism DB → Emits `plagiarism‑alert` |\n| **TurnItIn Worker Pool** | Dedicated pool with **token‑bucket rate limiter**; retries with exponential back‑off; results cached for 24 h. | Consumes `plagiarism‑jobs` → Calls TurnItIn API |\n| **LMS Integration Service** | Same as before, but now **idempotent** outcome pushes (store last‑sent token). | Subscribes `grades` → Calls LTI Outcome Service |\n| **Audit & Logging Service** | **Hot‑cold tiering** – recent month in S3‑Standard, older months moved to Glacier‑compatible storage. Metadata indexed in Elasticsearch for fast queries. | Reads all topics → Writes to hot store & index |\n| **Reporting Dashboard** | Queries Elasticsearch for recent audit data; reads grades from a **read‑replica** of PostgreSQL. | REST API → Reporting UI |\n| **Infrastructure** | Same core services (Kafka, PostgreSQL, S3‑compatible store, K8s) plus **Prometheus exporters** for custom metrics (container‑warm‑pool size, consumer lag). | – |\n\n---\n\n### 3.  Data‑Flow – “Student Submits → Grade Sent to LMS” (with refinements)\n\n1. **Auth** – Student logs in via university IdP → receives JWT.  \n2. **Presigned URL** – SPA calls **Gateway** → receives a short‑lived PUT URL for the object store.  \n3. **Direct Upload** – Browser streams the zip directly to the object store (no gateway bandwidth consumption).  \n4. **Metadata POST** – SPA posts a lightweight JSON payload (`assignmentId`, `studentId`, `objectKey`) to the **Gateway**.  \n5. **Submission Service** stores the key, emits **`SubmissionCreated`**.  \n6. **Sandbox Warm‑Pool** – A ready container/VM is assigned, the archive is mounted read‑only, tests run, logs are streamed to the object store (compressed, size‑capped).  \n7. **RunResult** – Sandbox publishes **`RunResult`** (exit code, test summary, resource usage).  \n8. **Grading Engine** batches results, writes a **grade row** into a **partitioned** table, emits **`GradeCreated`**.  \n9. **LMS Integration** pushes the grade via LTI Outcome Service (idempotent).  \n10. **Audit Service** appends every event to the **hot log**; after 30 days the log segment is moved to cold storage.  \n11. **Reporting UI** reads recent audit metadata from Elasticsearch and grades from the read‑replica to display dashboards.\n\n*Plagiarism* runs nightly on the **incremental** candidate set, stores flagged pairs, and notifies staff via the dashboard and optional email.\n\n---\n\n### 4.  Addressing the Performance Bottlenecks\n\n| Bottleneck (original) | Refined Mitigation | Expected Effect |\n|-----------------------|--------------------|-----------------|\n| **Sandbox start‑up (image pull, container creation)** | Warm‑pool of pre‑created containers / Firecracker micro‑VMs + node‑level image cache | 95 % of runs start ≤ 1 s; eliminates image‑pull latency even after node churn. |\n| **Kafka consumer lag during deadline spikes** | Export **consumer lag** via Prometheus; HPA scales workers on lag; separate **fast‑track** topic for last‑minute submissions; back‑pressure (HTTP 429) when lag > 30 s | Grading latency stays ≤ 2 min; system gracefully rejects excess load instead of queuing indefinitely. |\n| **Gateway bandwidth for large uploads** | Direct upload via presigned URLs; multipart, resumable uploads | Gateway CPU & network usage drops > 80 %; UI latency improves to ≤ 5 s for 10 MiB archives. |\n| **DB write contention on grades** | Partitioned `grades` table (semester/assignment) + **batch inserts** (≤ 100 rows) | Insert throughput > 5 k grades/min; lock wait time < 5 ms; reporting replica stays fresh. |\n| **Plagiarism O(N²) similarity** | MinHash‑LSH pre‑filter + incremental nightly runs; cache TurnItIn results | Candidate pairs reduced to < 5 % of N²; nightly Spark job ≤ 2 h for 20 k submissions. |\n| **TurnItIn API throttling** | Dedicated worker pool with token‑bucket limiter; persistent cache of results | No more than 1 % of checks pending; back‑off handled automatically. |\n| **Audit‑log growth & query latency** | Hot‑cold tiering + Elasticsearch metadata index; only recent month kept hot | Write latency ≤ 200 ms; audit queries for last 30 days return < 1 s; storage cost reduced > 60 %. |\n| **Verbose stdout/stderr flooding network** | Compress logs (gzip) before storing; truncate logs > 1 MiB; configurable log level per language | Network usage cut ~70 %; sandbox container memory pressure reduced. |\n\n---\n\n### 5.  Updated SLAs & Observability\n\n| Metric | Target (95 pct) | Measurement Point |\n|--------|----------------|-------------------|\n| End‑to‑end grading latency | ≤ 2 min | Timestamp at **metadata POST** → `GradeCreated` event |\n| Sandbox warm‑start latency | ≤ 1 s | Container/VM hand‑off log |\n| Kafka consumer lag (`submissions`) | ≤ 30 s | Prometheus exporter `kafka_consumer_lag` |\n| Upload response time (client‑visible) | ≤ 5 s for ≤ 10 MiB | Gateway latency histogram (presign + metadata POST) |\n| Grade DB write latency | ≤ 100 ms per batch | PostgreSQL `pg_stat_statements` |\n| Plagiarism nightly job duration | ≤ 2 h for 20 k submissions | Spark job duration metric |\n| TurnItIn request success rate | ≥ 99 % (within provider limits) | TurnItIn worker success counter |\n| Audit‑log append latency | ≤ 200 ms | Kafka produce latency metric |\n\nAll metrics are exported to **Prometheus**, visualised on **Grafana**, and have alerting rules (PagerDuty) for breach of any SLA.\n\n---\n\n### 6.  Operational & Governance Enhancements\n\n| Area | Refined Practice |\n|------|-------------------|\n| **CI/CD** | GitOps via **ArgoCD**; container images scanned with **Trivy**; sandbox images version‑locked and stored in a private registry. |\n| **Security** | Unprivileged sandbox users, **seccomp** & **AppArmor** profiles, optional **gVisor/Firecracker** for extra isolation; JWT validation against university IdP; IAM policies restrict object‑store prefixes per student. |\n| **Capacity Planning** | Baseline: 300 students × 2 assignments × 2 submissions ≈ 1.2 k uploads/term. System sized for **10×** peak (≈ 12 k uploads in 2 h). Warm‑pool size = 2 × expected concurrent submissions during peak. |\n| **Disaster Recovery** | Kafka replicated across 3 zones; PostgreSQL HA with Patroni; object‑store multi‑zone replication; nightly backup of audit‑log metadata index. |\n| **Compliance** | GDPR / FERPA: student identifiers encrypted at rest; audit‑log hash‑chain stored in WORM bucket; retention policy configurable (e.g., 5 years). |\n| **Support** | Self‑service UI for resubmission, log view, and grade appeal; staff portal to manually override grades (audit‑logged). |\n| **Testing** | Load‑test with **k6** simulating deadline spikes; chaos‑testing (kill Kafka broker, sandbox node) to verify auto‑recovery; benchmark warm‑pool start‑up times per language. |\n\n---\n\n### 7.  Summary of the Refined Solution\n\n| Concern | How the refined design resolves it |\n|---------|------------------------------------|\n| **Sandbox latency** | Warm‑pool + micro‑VMs + local image cache → ≤ 1 s start‑up. |\n| **Queue back‑pressure** | Consumer‑lag‑driven HPA, fast‑track topic, HTTP 429 back‑pressure → grading stays within SLA. |\n| **Upload bottleneck** | Presigned‑URL direct upload → gateway no longer a bandwidth choke point. |\n| **DB contention** | Partitioned, batch‑insert grades → high write throughput, low lock contention. |\n| **Plagiarism scalability** | MinHash‑LSH pre‑filter + incremental nightly runs → near‑linear runtime, fits nightly window. |\n| **External API throttling** | Dedicated rate‑limited worker pool + caching → no backlog, high success rate. |\n| **Audit‑log growth** | Hot/cold tiering + Elasticsearch metadata index → fast queries, cost‑effective storage. |\n| **Observability & SLA enforcement** | Exported custom metrics, Grafana dashboards, alerting on lag/latency → proactive operations. |\n| **Security & compliance** | Hardened sandbox, IAM‑scoped object store, tamper‑evident log, encryption → meets university policies. |\n\nImplementing the **quick‑win** items (presigned uploads, warm sandbox pool, consumer‑lag autoscaling, DB partitioning) will bring the system comfortably within the 2‑minute grading latency target for typical and burst loads. The **long‑term** investments (micro‑VM isolation, sophisticated plagiarism pipeline, audit‑log indexing) ensure the platform remains performant, secure, and auditable as enrollment scales and new languages are added.",
+          "metadata": {
+            "latencyMs": 27274,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 9396
+          }
+        },
+        {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "type": "refinement",
+          "content": "## Refined Design – Low‑Latency, High‑Throughput Automated Grading Service  \n*Performance‑engineer’s response to the architectural critique*  \n\n---\n\n### 1.  High‑level Goal (unchanged)\n\nProvide a **low‑latency, high‑throughput, auditable** grading pipeline that can:\n\n* Accept ~300 + student submissions per week (≈ 5‑10 /s at peak, up to 500 /min during exams).  \n* Execute each submission in a **secure sandbox**, compare its output with the instructor‑provided test‑suite and return a grade.  \n* Persist every artefact (source, compiled binary, console output, grade, timestamps) for immutable audit.  \n* Run **plagiarism detection** against all prior submissions and the external TurnItIn API.  \n* Expose a **stable, versioned API** for LMS integration.  \n\nAll components must stay **resource‑efficient** and **observable** so that the 95 th‑percentile latency target (≤ 30 s) is reliably met even under burst load.\n\n---\n\n### 2.  Updated Quantitative Performance Requirements  \n\n| Metric (95 th pct) | Target | Rationale |\n|--------------------|--------|-----------|\n| Submission‑to‑grade latency | ≤ 30 s | Real‑time feedback for students. |\n| Peak sustained throughput | 200 submissions / min (≈ 3 /s) | Normal operation. |\n| Burst throughput | 500 submissions / min (≈ 8 /s) | Exam‑week spikes. |\n| Plagiarism batch latency | ≤ 2 min per 100‑submission batch | TurnItIn latency bound. |\n| API response latency (status/grade) | ≤ 200 ms | LMS feels snappy. |\n| Storage write latency (object store) | ≤ 10 ms per object | Audit log freshness. |\n| Sandbox CPU utilisation | ≤ 80 % of quota | Prevent noisy‑neighbor effects. |\n| Sandbox memory | 256 MiB (adjustable per language) | High node density. |\n| Cache‑hit rate (critical caches) | ≥ 80 % | Keeps DB/I/O load low. |\n| Kafka consumer lag | ≤ 500 messages per partition | Guarantees timely grading. |\n\n---\n\n### 3.  Architectural Overview (with refinements)\n\n```\n+-------------------+   +-------------------+   +-------------------+\n|  LMS / Front‑End  |<->|   API Gateway     |<->|   Auth / Rate‑Lim |\n+-------------------+   +-------------------+   +-------------------+\n                               |\n                               v\n                       +-------------------+\n                       |  Submission Svc   |\n                       +-------------------+\n                               |\n                               v\n                       +-------------------+\n                       |   Ingress Log DB  |   ← write‑ahead table (always succeeds)\n                       +-------------------+\n                               |\n                               v\n                       +-------------------+\n                       |   Message Queue   |  (Kafka, HA, 3‑replica)\n                       +-------------------+\n               ┌───────────────┬─────────────────────┐\n               │               │                     │\n               v               v                     v\n   +-------------------+ +-------------------+ +-------------------+\n   | Grader Workers    | | Plagiarism Workers| |  Metrics Exporter |\n   +-------------------+ +-------------------+ +-------------------+\n               │               │                     │\n               v               v                     v\n   +-------------------+ +-------------------+ +-------------------+\n   | Result Store (DB) | | Plagiarism DB    | |  Observability    |\n   +-------------------+ +-------------------+ +-------------------+\n               │               │\n               v               v\n   +-------------------+ +-------------------+\n   | Immutable Object  | | Object Store (S3) |\n   | Store (source,   | | (hashed prefixes) |\n   | binaries, logs) | +-------------------+\n   +-------------------+\n```\n\n**Key refinements**\n\n| Area | Change | Why it matters |\n|------|--------|----------------|\n| **Ingress reliability** | Add a tiny *write‑ahead* table (`IngressLog`) that records every accepted submission before publishing to Kafka. | Guarantees a `200 OK` response even if the queue is temporarily full; enables replay to Kafka on recovery. |\n| **Idempotency** | Use the `submission_id` (generated at ingest) as the primary key in the Result Store; `INSERT … ON CONFLICT DO NOTHING`. | Prevents duplicate grades when a message is redelivered after a worker crash. |\n| **Warm‑container pool** | Dynamically size the idle‑container pool per node: `idle_per_core = 0.5`.  A background “pool‑maintainer” job monitors queue lag and scales the pool up/down. | Keeps latency under 30 s during bursts; avoids static under‑provisioning. |\n| **Back‑pressure** | The Submission Service checks Kafka consumer lag (via JMX) before accepting a new upload. If lag > 500, it returns `429 Too Many Requests` with `Retry‑After`. | Stops the pipeline from being overwhelmed and protects the latency SLA. |\n| **Redis HA** | Deploy Redis as a **cluster** with three master nodes and replicas, enable AOF persistence, and configure automatic fail‑over. | Removes a single point of failure for all critical caches (metadata, compiled artefacts, test‑case data). |\n| **Result Store scaling** | Partition the audit table by `assignment_id` **and** by hour (`YYYYMMDDHH`).  Large stdout/stderr are stored as objects; the DB row only keeps a reference URL. | Reduces write contention and row size, enabling > 1 k writes / sec without hot‑spotting. |\n| **Plagiarism batch sizing** | Make the batch size *adaptive*: process up to `N` submissions **or** after `T` seconds (whichever comes first).  During exam spikes, increase `N` and spin up extra workers. | Guarantees that no submission waits longer than the 2‑min SLA even when the arrival rate spikes. |\n| **Egress protection** | Allocate a dedicated NAT gateway for all external calls (TurnItIn, object‑store writes).  Rate‑limit TurnItIn requests per minute and apply exponential back‑off. | Prevents network‑bandwidth saturation that would otherwise delay both plagiarism and audit persistence. |\n| **Chaos‑engineering** | Introduce a scheduled chaos‑mesh run that kills random pods, disconnects a Kafka broker, or shuts down a Redis replica.  Verify that latency and error‑rate remain within thresholds. | Validates resilience claims before production. |\n| **Image supply‑chain security** | Enforce **Docker Content Trust** (image signing) and require the orchestrator to verify signatures at pod creation.  Rebuild base images weekly from a minimal, reproducible Dockerfile. | Guarantees that sandbox images cannot be compromised silently. |\n| **Test‑suite versioning** | Store a `test_suite_version` identifier with each assignment.  Grader workers request the exact version from the cache; a new version invalidates only the relevant cache entries. | Prevents inconsistent grading when instructors update test cases after some submissions have already been processed. |\n| **Node‑local cache** | Each node runs a lightweight, in‑memory cache (e.g., `tmpfs`‑backed) for compiled artefacts that are reused frequently on that node.  Periodic sync with the central Redis keeps it coherent. | Reduces cross‑node network traffic and improves cache‑hit latency for hot languages. |\n| **Object‑store prefix sharding** | Use a **hashed prefix** (`/<hash>/submission/...`) for every object key.  This spreads PUT/GET traffic across many S3 partitions, staying well below the 3,500 PUT / sec per prefix limit. | Avoids hidden throttling during exam‑week bursts. |\n| **Audit‑log lifecycle** | Define a three‑tier retention policy:  \n  * Hot (0‑30 days) – standard S3, fast access.  \n  * Warm (30 days‑1 year) – S3 IA.  \n  * Cold (≥ 1 year) – Glacier Deep Archive.  \n  A nightly verification job checks WORM bucket settings before moving data. | Meets accreditation requirements (≥ 5 years) while controlling storage cost. |\n\n---\n\n### 4.  Component‑level Optimisations (Performance‑focused)\n\n| Component | Optimisation | Expected Impact |\n|-----------|--------------|-----------------|\n| **API Gateway** | TLS termination + HTTP/2, keep‑alive connections to downstream services, Redis‑backed static metadata cache (TTL 5 min). | Sub‑ms request routing, reduces DB round‑trips. |\n| **Submission Service** | Streaming upload directly to object store (multipart), immediate lightweight audit row, publish only `submission_id` to Kafka. | Minimal processing time per upload (< 200 ms). |\n| **Ingress Log DB** | Single‑row insert with `INSERT … RETURNING id`; primary key = `submission_id`. | Guarantees durability before queueing. |\n| **Kafka** | 3‑replica topic, partition key = `assignment_id`, consumer lag metrics exposed via JMX. | Ordered processing per assignment, easy scaling. |\n| **Grader Workers** | • Pre‑warmed language‑specific containers (pool). <br>• Compile‑cache in Redis (key = `<src_hash, lang, compiler>`). <br>• Test‑case read‑through cache (TTL = assignment end). <br>• Run up to 2‑3 submissions concurrently per pod, respecting cgroup quotas. | Cold‑start < 2 s, compile‑skip ≈ 30 % latency reduction, overall grading latency ≈ 20 s median. |\n| **Result Store** | Append‑only tables, hourly partitions, object‑reference for logs, bulk `COPY` inserts from each worker. | Sustains > 1 k writes / sec without lock contention. |\n| **Plagiarism Workers** | Adaptive batch size, local similarity engine (MOSS‑style) on cached source hashes, asynchronous TurnItIn submission with status polling. | Guarantees ≤ 2 min per batch, isolates external latency. |\n| **Metrics Exporter** | Prometheus client in each pod, cAdvisor for container‑level CPU/memory, custom sandbox‑health exporter (syscalls, OOM, network violations). | Fine‑grained alerts before SLA breach. |\n| **Observability Stack** | Grafana dashboards for latency histograms, consumer lag, cache‑hit rates, node‑local cache utilisation, audit‑log growth. | Enables capacity planning and rapid incident response. |\n\n---\n\n### 5.  Caching Strategy (refined)\n\n| Cache | Data | Location | TTL / Invalidation |\n|-------|------|----------|--------------------|\n| **API‑gateway metadata** | Assignment description, test‑case list | Redis cluster (replicated) | 5 min or on assignment update |\n| **Compiled artefact cache** | Binary / bytecode per `<src_hash, lang, compiler>` | Redis (binary‑cache) + node‑local `tmpfs` mirror | 24 h or until source hash changes |\n| **Test‑case data** | Input/expected‑output files | Redis read‑through | Assignment end date |\n| **Recent grades** | Student‑grade list (last 24 h) | Redis LRU | 10 min |\n| **Plagiarism similarity graph** | Pairwise similarity scores (in‑memory) | Worker‑local cache (evicted per batch) | Recomputed each batch |\n| **LMS grade view** | Per‑student grade summary | Redis per‑student key | 2 min or on grade write |\n| **Ingress‑log deduplication** | Already‑processed `submission_id` | Redis set (TTL 1 day) | Auto‑expire after 24 h |\n\nAll caches are **write‑through**: a successful DB write triggers an immediate cache update, guaranteeing strong consistency for the 95 th‑percentile latency target.\n\n---\n\n### 6.  Resource‑Efficiency Measures (enhanced)\n\n1. **Dynamic warm‑container pool** – size = `0.5 × CPU‑cores` per node; pool‑maintainer scales based on queue lag.  \n2. **Kubernetes Cluster Autoscaler** – adds nodes when pending pod count > N, removes idle nodes after 10 min of zero utilisation.  \n3. **CPU‑quota & memory‑limit per sandbox** – enforced by cgroups; OOM kills are recorded and reported as “resource‑exceeded”.  \n4. **Node‑local cache** – reduces cross‑node Redis traffic for hot artefacts.  \n5. **Batch‑size adaptive plagiarism** – scales workers automatically when batch queue length exceeds threshold.  \n6. **Object‑store lifecycle** – hot → IA → Glacier Deep Archive, driven by object age tags.  \n\nThese measures keep the cost proportional to actual usage while preserving the latency SLA.\n\n---\n\n### 7.  Observability & Metrics (expanded)\n\n| Metric | Collector | Alert Threshold |\n|--------|-----------|-----------------|\n| Submission‑to‑grade latency (histogram) | Prometheus (worker exporter) | 95 pct > 25 s |\n| Kafka consumer lag per partition | JMX → Prometheus | > 500 msgs |\n| Redis cache‑hit rate (overall) | Redis INFO → Prometheus | < 70 % |\n| Sandbox CPU utilisation (cAdvisor) | Prometheus | > 80 % for > 5 min |\n| Sandbox syscalls / OOM events | Custom eBPF exporter | Spike > 2× baseline |\n| Plagiarism batch duration | Worker logs → Prometheus | > 3 min per 100 subs |\n| Object‑store write latency | S3 SDK metrics → Prometheus | > 15 ms |\n| API error rate (5xx) | Envoy/Istio metrics | > 1 % |\n| Audit‑log growth (DB size/day) | DB stats exporter | > 5 GB |\n| Node‑local cache eviction rate | Node exporter | > 30 % per hour (indicates undersizing) |\n\nAll alerts feed into PagerDuty with severity based on impact (latency vs. availability). Dashboards in Grafana show real‑time trends and historical capacity planning data.\n\n---\n\n### 8.  Performance & Resilience Testing Plan (updated)\n\n| Test | Goal | Toolset |\n|------|------|---------|\n| **Synthetic load** | 500 req/min sustained, 1 k req/min burst; verify ≤ 30 s latency. | k6 / Locust driving the API, measuring end‑to‑end time. |\n| **Back‑pressure validation** | Submit until consumer lag > 500, ensure API returns 429 and system remains stable. | Custom script that monitors Kafka lag via JMX. |\n| **Warm‑pool sizing** | Determine minimal idle‑container count that keeps 95 pct latency < 30 s under burst. | Load test with varying pool sizes, capture cold‑start times. |\n| **Cache‑hit impact** | Run identical workload twice; compare cache‑hit rates and latency reduction. | Prometheus query + Grafana diff. |\n| **Result‑store write contention** | Spike to 1 k writes / sec; verify no DB lock timeouts. | pgbench with bulk `COPY` simulation. |\n| **Plagiarism batch scaling** | Inject 1 000 new submissions; confirm batch processing finishes within 2 min and TurnItIn rate limits respected. | Mock TurnItIn endpoint with latency injection. |\n| **Chaos engineering** | Randomly kill pods, disconnect a Kafka broker, shut down a Redis replica; ensure latency stays within SLA and no data loss. | Chaos Mesh / Gremlin. |\n| **Egress throttling** | Simulate NAT gateway bandwidth limit; verify that external calls are rate‑limited without affecting grading pipeline. | tc (traffic control) on test cluster. |\n| **Security validation** | Verify image signature enforcement, sandbox syscall restrictions, and network isolation. | Trivy for image scanning, Falco for runtime policy enforcement. |\n| **Retention & lifecycle** | Populate audit logs for > 1 year, trigger lifecycle policies; confirm objects move to Glacier and remain immutable. | S3 lifecycle simulation scripts. |\n\nResults feed directly into capacity‑planning spreadsheets (e.g., “X m5.large nodes sustain Y submissions/min”) and into the autoscaling policy definitions.\n\n---\n\n### 9.  Security & Auditing (strengthened)\n\n| Aspect | Enhancement |\n|--------|-------------|\n| **Transport security** | Mutual TLS everywhere (API‑gateway ↔ services, service‑to‑service). |\n| **Workload identity** | SPIFFE/SPIRE issued short‑lived certificates for each pod; automatic rotation. |\n| **Sandbox hardening** | gVisor + seccomp profile, read‑only root filesystem, no network, limited syscalls. |\n| **Image integrity** | Docker Content Trust (Notary) enforced at pod creation; weekly rebuilds from minimal Dockerfiles. |\n| **Immutable audit trail** | Append‑only DB tables, WORM bucket policy on object store, daily checksum verification job. |\n| **Role‑based access** | Students → own submissions only; staff → read all; admin → purge/archival. |\n| **Compliance** | Retention policy ≥ 5 years, periodic audit of WORM settings, GDPR‑compatible data‑subject access tooling. |\n\n---\n\n### 10.  Summary & Next Steps\n\n* **Latency guarantee** is now backed by a dynamic warm‑container pool, back‑pressure at the API edge, and idempotent processing.  \n* **Throughput scalability** is achieved through Kafka‑driven decoupling, adaptive plagiarism batching, and autoscaling node pools.  \n* **Reliability** is reinforced with Redis cluster HA, write‑ahead ingress logging, and a formal chaos‑testing regimen.  \n* **Observability** covers every critical path (queue lag, sandbox health, cache hit rates) and feeds automated alerts.  \n* **Security & auditability** meet university compliance requirements via immutable storage, signed images, and workload identity.  \n\n**Immediate actions**  \n\n1. Implement the write‑ahead `IngressLog` table and back‑pressure check.  \n2. Deploy Redis in HA cluster mode and configure fail‑over.  \n3. Build the pool‑maintainer controller to size idle containers per core.  \n4. Add the custom sandbox‑health exporter to Prometheus.  \n5. Run the updated performance test suite, capture baseline metrics, and tune autoscaling thresholds.  \n\nWith these refinements the system will reliably deliver sub‑30‑second grading feedback even during exam‑week spikes, while staying cost‑effective, secure, and fully auditable.",
+          "metadata": {
+            "latencyMs": 45720,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 10097
+          }
+        }
+      ],
+      "timestamp": "2025-10-29T20:32:56.385Z"
+    },
+    {
+      "roundNumber": 2,
+      "contributions": [
+        {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "type": "proposal",
+          "content": "## Refined Architecture – “Secure, Auditable, High‑Throughput Grading Platform”\n\nBelow is the original design sharpened with the performance‑engineer’s observations.  \nThe changes are grouped by **problem → concrete mitigation → impact** and then re‑presented as an updated high‑level view, component diagram, data‑flow, and operational checklist.\n\n---\n\n### 1.  Core Architectural Style (unchanged)\n\n| Goal | Pattern | Why it still fits |\n|------|---------|-------------------|\n| Isolation of untrusted code | **Container‑/micro‑VM sandbox** (Docker + gVisor/Firecracker) | Guarantees strong security while allowing fast start‑up (warm‑pool). |\n| Burst‑y submissions | **Event‑driven, queue‑backed micro‑services** (Kafka) | Decouples UI from heavy work, provides natural back‑pressure, and lets us autoscale workers on lag. |\n| Auditability | **Append‑only immutable log** (Kafka + write‑once object store) | Guarantees ordered, tamper‑evident records without stressing the primary DB. |\n| Plagiarism detection | **Batch pipeline with MinHash‑LSH pre‑filter** (Spark/Flink) + TurnItIn API | Reduces O(N²) work to near‑linear, keeping nightly jobs within a 2 h window. |\n| LMS integration | **LTI / REST** | De‑facto university standard, no per‑LMS adapters. |\n| Operational simplicity | **IaC + Kubernetes** | Self‑healing, rolling upgrades, and HPA driven by custom metrics (Kafka lag, sandbox latency). |\n\n---\n\n### 2.  Updated Component Landscape\n\n| Component | New / Modified Behaviour | Key Interfaces |\n|-----------|--------------------------|----------------|\n| **Web Front‑End (SPA)** | Uploads now use **presigned‑URL** flow; UI obtains a short‑lived URL from the gateway and streams the archive directly to the object store. | HTTPS (REST) → API Gateway (JWT + presign request) |\n| **API Gateway / Auth Proxy** | 1️⃣ Validate JWT 2️⃣ Issue presigned URL (S3‑compatible) 3️⃣ Return URL to SPA 4️⃣ Accept a lightweight “submission‑metadata” POST (no binary payload). | Public HTTPS (REST) → internal gRPC/REST |\n| **Submission Service** | Receives only metadata (assignmentId, studentId, object‑store key). Stores a **SubmissionCreated** event. | Consumes from Gateway → Publishes to Kafka (`submissions`) |\n| **Sandbox Service** | **Warm‑pool of pre‑created containers / micro‑VMs** (e.g., Firecracker VMs kept idle). On a new submission the pool hands over a ready sandbox, mounts the archive, runs the test harness, then **re‑images** the VM (fast reset). | Subscribes `submissions` → Publishes `run‑results` |\n| **Grading Engine** | **Batch‑insert** grades (collect up to 100 results) into a **partitioned** `grades` table (partitioned by semester‑assignment). | Subscribes `run‑results` → Writes to PostgreSQL → Publishes `grades` |\n| **Plagiarism Service** | **MinHash‑LSH pre‑filter** to generate candidate pairs, then full similarity only on those. Runs **incrementally** (new submissions vs. existing corpus). Caches TurnItIn results keyed by submission hash. | Reads from Object Store & DB → Writes to Plagiarism DB → Emits `plagiarism‑alert` |\n| **TurnItIn Worker Pool** | Dedicated pool with **token‑bucket rate limiter**; retries with exponential back‑off; results cached for 24 h. | Consumes `plagiarism‑jobs` → Calls TurnItIn API |\n| **LMS Integration Service** | Same as before, but now **idempotent** outcome pushes (store last‑sent token). | Subscribes `grades` → Calls LTI Outcome Service |\n| **Audit & Logging Service** | **Hot‑cold tiering** – recent month in S3‑Standard, older months moved to Glacier‑compatible storage. Metadata indexed in Elasticsearch for fast queries. | Reads all topics → Writes to hot store & index |\n| **Reporting Dashboard** | Queries Elasticsearch for recent audit data; reads grades from a **read‑replica** of PostgreSQL. | REST API → Reporting UI |\n| **Infrastructure** | Same core services (Kafka, PostgreSQL, S3‑compatible store, K8s) plus **Prometheus exporters** for custom metrics (container‑warm‑pool size, consumer lag). | – |\n\n---\n\n### 3.  Data‑Flow – “Student Submits → Grade Sent to LMS” (with refinements)\n\n1. **Auth** – Student logs in via university IdP → receives JWT.  \n2. **Presigned URL** – SPA calls **Gateway** → receives a short‑lived PUT URL for the object store.  \n3. **Direct Upload** – Browser streams the zip directly to the object store (no gateway bandwidth consumption).  \n4. **Metadata POST** – SPA posts a lightweight JSON payload (`assignmentId`, `studentId`, `objectKey`) to the **Gateway**.  \n5. **Submission Service** stores the key, emits **`SubmissionCreated`**.  \n6. **Sandbox Warm‑Pool** – A ready container/VM is assigned, the archive is mounted read‑only, tests run, logs are streamed to the object store (compressed, size‑capped).  \n7. **RunResult** – Sandbox publishes **`RunResult`** (exit code, test summary, resource usage).  \n8. **Grading Engine** batches results, writes a **grade row** into a **partitioned** table, emits **`GradeCreated`**.  \n9. **LMS Integration** pushes the grade via LTI Outcome Service (idempotent).  \n10. **Audit Service** appends every event to the **hot log**; after 30 days the log segment is moved to cold storage.  \n11. **Reporting UI** reads recent audit metadata from Elasticsearch and grades from the read‑replica to display dashboards.\n\n*Plagiarism* runs nightly on the **incremental** candidate set, stores flagged pairs, and notifies staff via the dashboard and optional email.\n\n---\n\n### 4.  Addressing the Performance Bottlenecks\n\n| Bottleneck (original) | Refined Mitigation | Expected Effect |\n|-----------------------|--------------------|-----------------|\n| **Sandbox start‑up (image pull, container creation)** | Warm‑pool of pre‑created containers / Firecracker micro‑VMs + node‑level image cache | 95 % of runs start ≤ 1 s; eliminates image‑pull latency even after node churn. |\n| **Kafka consumer lag during deadline spikes** | Export **consumer lag** via Prometheus; HPA scales workers on lag; separate **fast‑track** topic for last‑minute submissions; back‑pressure (HTTP 429) when lag > 30 s | Grading latency stays ≤ 2 min; system gracefully rejects excess load instead of queuing indefinitely. |\n| **Gateway bandwidth for large uploads** | Direct upload via presigned URLs; multipart, resumable uploads | Gateway CPU & network usage drops > 80 %; UI latency improves to ≤ 5 s for 10 MiB archives. |\n| **DB write contention on grades** | Partitioned `grades` table (semester/assignment) + **batch inserts** (≤ 100 rows) | Insert throughput > 5 k grades/min; lock wait time < 5 ms; reporting replica stays fresh. |\n| **Plagiarism O(N²) similarity** | MinHash‑LSH pre‑filter + incremental nightly runs; cache TurnItIn results | Candidate pairs reduced to < 5 % of N²; nightly Spark job ≤ 2 h for 20 k submissions. |\n| **TurnItIn API throttling** | Dedicated worker pool with token‑bucket limiter; persistent cache of results | No more than 1 % of checks pending; back‑off handled automatically. |\n| **Audit‑log growth & query latency** | Hot‑cold tiering + Elasticsearch metadata index; only recent month kept hot | Write latency ≤ 200 ms; audit queries for last 30 days return < 1 s; storage cost reduced > 60 %. |\n| **Verbose stdout/stderr flooding network** | Compress logs (gzip) before storing; truncate logs > 1 MiB; configurable log level per language | Network usage cut ~70 %; sandbox container memory pressure reduced. |\n\n---\n\n### 5.  Updated SLAs & Observability\n\n| Metric | Target (95 pct) | Measurement Point |\n|--------|----------------|-------------------|\n| End‑to‑end grading latency | ≤ 2 min | Timestamp at **metadata POST** → `GradeCreated` event |\n| Sandbox warm‑start latency | ≤ 1 s | Container/VM hand‑off log |\n| Kafka consumer lag (`submissions`) | ≤ 30 s | Prometheus exporter `kafka_consumer_lag` |\n| Upload response time (client‑visible) | ≤ 5 s for ≤ 10 MiB | Gateway latency histogram (presign + metadata POST) |\n| Grade DB write latency | ≤ 100 ms per batch | PostgreSQL `pg_stat_statements` |\n| Plagiarism nightly job duration | ≤ 2 h for 20 k submissions | Spark job duration metric |\n| TurnItIn request success rate | ≥ 99 % (within provider limits) | TurnItIn worker success counter |\n| Audit‑log append latency | ≤ 200 ms | Kafka produce latency metric |\n\nAll metrics are exported to **Prometheus**, visualised on **Grafana**, and have alerting rules (PagerDuty) for breach of any SLA.\n\n---\n\n### 6.  Operational & Governance Enhancements\n\n| Area | Refined Practice |\n|------|-------------------|\n| **CI/CD** | GitOps via **ArgoCD**; container images scanned with **Trivy**; sandbox images version‑locked and stored in a private registry. |\n| **Security** | Unprivileged sandbox users, **seccomp** & **AppArmor** profiles, optional **gVisor/Firecracker** for extra isolation; JWT validation against university IdP; IAM policies restrict object‑store prefixes per student. |\n| **Capacity Planning** | Baseline: 300 students × 2 assignments × 2 submissions ≈ 1.2 k uploads/term. System sized for **10×** peak (≈ 12 k uploads in 2 h). Warm‑pool size = 2 × expected concurrent submissions during peak. |\n| **Disaster Recovery** | Kafka replicated across 3 zones; PostgreSQL HA with Patroni; object‑store multi‑zone replication; nightly backup of audit‑log metadata index. |\n| **Compliance** | GDPR / FERPA: student identifiers encrypted at rest; audit‑log hash‑chain stored in WORM bucket; retention policy configurable (e.g., 5 years). |\n| **Support** | Self‑service UI for resubmission, log view, and grade appeal; staff portal to manually override grades (audit‑logged). |\n| **Testing** | Load‑test with **k6** simulating deadline spikes; chaos‑testing (kill Kafka broker, sandbox node) to verify auto‑recovery; benchmark warm‑pool start‑up times per language. |\n\n---\n\n### 7.  Summary of the Refined Solution\n\n| Concern | How the refined design resolves it |\n|---------|------------------------------------|\n| **Sandbox latency** | Warm‑pool + micro‑VMs + local image cache → ≤ 1 s start‑up. |\n| **Queue back‑pressure** | Consumer‑lag‑driven HPA, fast‑track topic, HTTP 429 back‑pressure → grading stays within SLA. |\n| **Upload bottleneck** | Presigned‑URL direct upload → gateway no longer a bandwidth choke point. |\n| **DB contention** | Partitioned, batch‑insert grades → high write throughput, low lock contention. |\n| **Plagiarism scalability** | MinHash‑LSH pre‑filter + incremental nightly runs → near‑linear runtime, fits nightly window. |\n| **External API throttling** | Dedicated rate‑limited worker pool + caching → no backlog, high success rate. |\n| **Audit‑log growth** | Hot/cold tiering + Elasticsearch metadata index → fast queries, cost‑effective storage. |\n| **Observability & SLA enforcement** | Exported custom metrics, Grafana dashboards, alerting on lag/latency → proactive operations. |\n| **Security & compliance** | Hardened sandbox, IAM‑scoped object store, tamper‑evident log, encryption → meets university policies. |\n\nImplementing the **quick‑win** items (presigned uploads, warm sandbox pool, consumer‑lag autoscaling, DB partitioning) will bring the system comfortably within the 2‑minute grading latency target for typical and burst loads. The **long‑term** investments (micro‑VM isolation, sophisticated plagiarism pipeline, audit‑log indexing) ensure the platform remains performant, secure, and auditable as enrollment scales and new languages are added.",
+          "metadata": {
+            "tokensUsed": 0,
+            "latencyMs": 0,
+            "model": "openai/gpt-oss-120b:exacto"
+          }
+        },
+        {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "type": "proposal",
+          "content": "## Refined Design – Low‑Latency, High‑Throughput Automated Grading Service  \n*Performance‑engineer’s response to the architectural critique*  \n\n---\n\n### 1.  High‑level Goal (unchanged)\n\nProvide a **low‑latency, high‑throughput, auditable** grading pipeline that can:\n\n* Accept ~300 + student submissions per week (≈ 5‑10 /s at peak, up to 500 /min during exams).  \n* Execute each submission in a **secure sandbox**, compare its output with the instructor‑provided test‑suite and return a grade.  \n* Persist every artefact (source, compiled binary, console output, grade, timestamps) for immutable audit.  \n* Run **plagiarism detection** against all prior submissions and the external TurnItIn API.  \n* Expose a **stable, versioned API** for LMS integration.  \n\nAll components must stay **resource‑efficient** and **observable** so that the 95 th‑percentile latency target (≤ 30 s) is reliably met even under burst load.\n\n---\n\n### 2.  Updated Quantitative Performance Requirements  \n\n| Metric (95 th pct) | Target | Rationale |\n|--------------------|--------|-----------|\n| Submission‑to‑grade latency | ≤ 30 s | Real‑time feedback for students. |\n| Peak sustained throughput | 200 submissions / min (≈ 3 /s) | Normal operation. |\n| Burst throughput | 500 submissions / min (≈ 8 /s) | Exam‑week spikes. |\n| Plagiarism batch latency | ≤ 2 min per 100‑submission batch | TurnItIn latency bound. |\n| API response latency (status/grade) | ≤ 200 ms | LMS feels snappy. |\n| Storage write latency (object store) | ≤ 10 ms per object | Audit log freshness. |\n| Sandbox CPU utilisation | ≤ 80 % of quota | Prevent noisy‑neighbor effects. |\n| Sandbox memory | 256 MiB (adjustable per language) | High node density. |\n| Cache‑hit rate (critical caches) | ≥ 80 % | Keeps DB/I/O load low. |\n| Kafka consumer lag | ≤ 500 messages per partition | Guarantees timely grading. |\n\n---\n\n### 3.  Architectural Overview (with refinements)\n\n```\n+-------------------+   +-------------------+   +-------------------+\n|  LMS / Front‑End  |<->|   API Gateway     |<->|   Auth / Rate‑Lim |\n+-------------------+   +-------------------+   +-------------------+\n                               |\n                               v\n                       +-------------------+\n                       |  Submission Svc   |\n                       +-------------------+\n                               |\n                               v\n                       +-------------------+\n                       |   Ingress Log DB  |   ← write‑ahead table (always succeeds)\n                       +-------------------+\n                               |\n                               v\n                       +-------------------+\n                       |   Message Queue   |  (Kafka, HA, 3‑replica)\n                       +-------------------+\n               ┌───────────────┬─────────────────────┐\n               │               │                     │\n               v               v                     v\n   +-------------------+ +-------------------+ +-------------------+\n   | Grader Workers    | | Plagiarism Workers| |  Metrics Exporter |\n   +-------------------+ +-------------------+ +-------------------+\n               │               │                     │\n               v               v                     v\n   +-------------------+ +-------------------+ +-------------------+\n   | Result Store (DB) | | Plagiarism DB    | |  Observability    |\n   +-------------------+ +-------------------+ +-------------------+\n               │               │\n               v               v\n   +-------------------+ +-------------------+\n   | Immutable Object  | | Object Store (S3) |\n   | Store (source,   | | (hashed prefixes) |\n   | binaries, logs) | +-------------------+\n   +-------------------+\n```\n\n**Key refinements**\n\n| Area | Change | Why it matters |\n|------|--------|----------------|\n| **Ingress reliability** | Add a tiny *write‑ahead* table (`IngressLog`) that records every accepted submission before publishing to Kafka. | Guarantees a `200 OK` response even if the queue is temporarily full; enables replay to Kafka on recovery. |\n| **Idempotency** | Use the `submission_id` (generated at ingest) as the primary key in the Result Store; `INSERT … ON CONFLICT DO NOTHING`. | Prevents duplicate grades when a message is redelivered after a worker crash. |\n| **Warm‑container pool** | Dynamically size the idle‑container pool per node: `idle_per_core = 0.5`.  A background “pool‑maintainer” job monitors queue lag and scales the pool up/down. | Keeps latency under 30 s during bursts; avoids static under‑provisioning. |\n| **Back‑pressure** | The Submission Service checks Kafka consumer lag (via JMX) before accepting a new upload. If lag > 500, it returns `429 Too Many Requests` with `Retry‑After`. | Stops the pipeline from being overwhelmed and protects the latency SLA. |\n| **Redis HA** | Deploy Redis as a **cluster** with three master nodes and replicas, enable AOF persistence, and configure automatic fail‑over. | Removes a single point of failure for all critical caches (metadata, compiled artefacts, test‑case data). |\n| **Result Store scaling** | Partition the audit table by `assignment_id` **and** by hour (`YYYYMMDDHH`).  Large stdout/stderr are stored as objects; the DB row only keeps a reference URL. | Reduces write contention and row size, enabling > 1 k writes / sec without hot‑spotting. |\n| **Plagiarism batch sizing** | Make the batch size *adaptive*: process up to `N` submissions **or** after `T` seconds (whichever comes first).  During exam spikes, increase `N` and spin up extra workers. | Guarantees that no submission waits longer than the 2‑min SLA even when the arrival rate spikes. |\n| **Egress protection** | Allocate a dedicated NAT gateway for all external calls (TurnItIn, object‑store writes).  Rate‑limit TurnItIn requests per minute and apply exponential back‑off. | Prevents network‑bandwidth saturation that would otherwise delay both plagiarism and audit persistence. |\n| **Chaos‑engineering** | Introduce a scheduled chaos‑mesh run that kills random pods, disconnects a Kafka broker, or shuts down a Redis replica.  Verify that latency and error‑rate remain within thresholds. | Validates resilience claims before production. |\n| **Image supply‑chain security** | Enforce **Docker Content Trust** (image signing) and require the orchestrator to verify signatures at pod creation.  Rebuild base images weekly from a minimal, reproducible Dockerfile. | Guarantees that sandbox images cannot be compromised silently. |\n| **Test‑suite versioning** | Store a `test_suite_version` identifier with each assignment.  Grader workers request the exact version from the cache; a new version invalidates only the relevant cache entries. | Prevents inconsistent grading when instructors update test cases after some submissions have already been processed. |\n| **Node‑local cache** | Each node runs a lightweight, in‑memory cache (e.g., `tmpfs`‑backed) for compiled artefacts that are reused frequently on that node.  Periodic sync with the central Redis keeps it coherent. | Reduces cross‑node network traffic and improves cache‑hit latency for hot languages. |\n| **Object‑store prefix sharding** | Use a **hashed prefix** (`/<hash>/submission/...`) for every object key.  This spreads PUT/GET traffic across many S3 partitions, staying well below the 3,500 PUT / sec per prefix limit. | Avoids hidden throttling during exam‑week bursts. |\n| **Audit‑log lifecycle** | Define a three‑tier retention policy:  \n  * Hot (0‑30 days) – standard S3, fast access.  \n  * Warm (30 days‑1 year) – S3 IA.  \n  * Cold (≥ 1 year) – Glacier Deep Archive.  \n  A nightly verification job checks WORM bucket settings before moving data. | Meets accreditation requirements (≥ 5 years) while controlling storage cost. |\n\n---\n\n### 4.  Component‑level Optimisations (Performance‑focused)\n\n| Component | Optimisation | Expected Impact |\n|-----------|--------------|-----------------|\n| **API Gateway** | TLS termination + HTTP/2, keep‑alive connections to downstream services, Redis‑backed static metadata cache (TTL 5 min). | Sub‑ms request routing, reduces DB round‑trips. |\n| **Submission Service** | Streaming upload directly to object store (multipart), immediate lightweight audit row, publish only `submission_id` to Kafka. | Minimal processing time per upload (< 200 ms). |\n| **Ingress Log DB** | Single‑row insert with `INSERT … RETURNING id`; primary key = `submission_id`. | Guarantees durability before queueing. |\n| **Kafka** | 3‑replica topic, partition key = `assignment_id`, consumer lag metrics exposed via JMX. | Ordered processing per assignment, easy scaling. |\n| **Grader Workers** | • Pre‑warmed language‑specific containers (pool). <br>• Compile‑cache in Redis (key = `<src_hash, lang, compiler>`). <br>• Test‑case read‑through cache (TTL = assignment end). <br>• Run up to 2‑3 submissions concurrently per pod, respecting cgroup quotas. | Cold‑start < 2 s, compile‑skip ≈ 30 % latency reduction, overall grading latency ≈ 20 s median. |\n| **Result Store** | Append‑only tables, hourly partitions, object‑reference for logs, bulk `COPY` inserts from each worker. | Sustains > 1 k writes / sec without lock contention. |\n| **Plagiarism Workers** | Adaptive batch size, local similarity engine (MOSS‑style) on cached source hashes, asynchronous TurnItIn submission with status polling. | Guarantees ≤ 2 min per batch, isolates external latency. |\n| **Metrics Exporter** | Prometheus client in each pod, cAdvisor for container‑level CPU/memory, custom sandbox‑health exporter (syscalls, OOM, network violations). | Fine‑grained alerts before SLA breach. |\n| **Observability Stack** | Grafana dashboards for latency histograms, consumer lag, cache‑hit rates, node‑local cache utilisation, audit‑log growth. | Enables capacity planning and rapid incident response. |\n\n---\n\n### 5.  Caching Strategy (refined)\n\n| Cache | Data | Location | TTL / Invalidation |\n|-------|------|----------|--------------------|\n| **API‑gateway metadata** | Assignment description, test‑case list | Redis cluster (replicated) | 5 min or on assignment update |\n| **Compiled artefact cache** | Binary / bytecode per `<src_hash, lang, compiler>` | Redis (binary‑cache) + node‑local `tmpfs` mirror | 24 h or until source hash changes |\n| **Test‑case data** | Input/expected‑output files | Redis read‑through | Assignment end date |\n| **Recent grades** | Student‑grade list (last 24 h) | Redis LRU | 10 min |\n| **Plagiarism similarity graph** | Pairwise similarity scores (in‑memory) | Worker‑local cache (evicted per batch) | Recomputed each batch |\n| **LMS grade view** | Per‑student grade summary | Redis per‑student key | 2 min or on grade write |\n| **Ingress‑log deduplication** | Already‑processed `submission_id` | Redis set (TTL 1 day) | Auto‑expire after 24 h |\n\nAll caches are **write‑through**: a successful DB write triggers an immediate cache update, guaranteeing strong consistency for the 95 th‑percentile latency target.\n\n---\n\n### 6.  Resource‑Efficiency Measures (enhanced)\n\n1. **Dynamic warm‑container pool** – size = `0.5 × CPU‑cores` per node; pool‑maintainer scales based on queue lag.  \n2. **Kubernetes Cluster Autoscaler** – adds nodes when pending pod count > N, removes idle nodes after 10 min of zero utilisation.  \n3. **CPU‑quota & memory‑limit per sandbox** – enforced by cgroups; OOM kills are recorded and reported as “resource‑exceeded”.  \n4. **Node‑local cache** – reduces cross‑node Redis traffic for hot artefacts.  \n5. **Batch‑size adaptive plagiarism** – scales workers automatically when batch queue length exceeds threshold.  \n6. **Object‑store lifecycle** – hot → IA → Glacier Deep Archive, driven by object age tags.  \n\nThese measures keep the cost proportional to actual usage while preserving the latency SLA.\n\n---\n\n### 7.  Observability & Metrics (expanded)\n\n| Metric | Collector | Alert Threshold |\n|--------|-----------|-----------------|\n| Submission‑to‑grade latency (histogram) | Prometheus (worker exporter) | 95 pct > 25 s |\n| Kafka consumer lag per partition | JMX → Prometheus | > 500 msgs |\n| Redis cache‑hit rate (overall) | Redis INFO → Prometheus | < 70 % |\n| Sandbox CPU utilisation (cAdvisor) | Prometheus | > 80 % for > 5 min |\n| Sandbox syscalls / OOM events | Custom eBPF exporter | Spike > 2× baseline |\n| Plagiarism batch duration | Worker logs → Prometheus | > 3 min per 100 subs |\n| Object‑store write latency | S3 SDK metrics → Prometheus | > 15 ms |\n| API error rate (5xx) | Envoy/Istio metrics | > 1 % |\n| Audit‑log growth (DB size/day) | DB stats exporter | > 5 GB |\n| Node‑local cache eviction rate | Node exporter | > 30 % per hour (indicates undersizing) |\n\nAll alerts feed into PagerDuty with severity based on impact (latency vs. availability). Dashboards in Grafana show real‑time trends and historical capacity planning data.\n\n---\n\n### 8.  Performance & Resilience Testing Plan (updated)\n\n| Test | Goal | Toolset |\n|------|------|---------|\n| **Synthetic load** | 500 req/min sustained, 1 k req/min burst; verify ≤ 30 s latency. | k6 / Locust driving the API, measuring end‑to‑end time. |\n| **Back‑pressure validation** | Submit until consumer lag > 500, ensure API returns 429 and system remains stable. | Custom script that monitors Kafka lag via JMX. |\n| **Warm‑pool sizing** | Determine minimal idle‑container count that keeps 95 pct latency < 30 s under burst. | Load test with varying pool sizes, capture cold‑start times. |\n| **Cache‑hit impact** | Run identical workload twice; compare cache‑hit rates and latency reduction. | Prometheus query + Grafana diff. |\n| **Result‑store write contention** | Spike to 1 k writes / sec; verify no DB lock timeouts. | pgbench with bulk `COPY` simulation. |\n| **Plagiarism batch scaling** | Inject 1 000 new submissions; confirm batch processing finishes within 2 min and TurnItIn rate limits respected. | Mock TurnItIn endpoint with latency injection. |\n| **Chaos engineering** | Randomly kill pods, disconnect a Kafka broker, shut down a Redis replica; ensure latency stays within SLA and no data loss. | Chaos Mesh / Gremlin. |\n| **Egress throttling** | Simulate NAT gateway bandwidth limit; verify that external calls are rate‑limited without affecting grading pipeline. | tc (traffic control) on test cluster. |\n| **Security validation** | Verify image signature enforcement, sandbox syscall restrictions, and network isolation. | Trivy for image scanning, Falco for runtime policy enforcement. |\n| **Retention & lifecycle** | Populate audit logs for > 1 year, trigger lifecycle policies; confirm objects move to Glacier and remain immutable. | S3 lifecycle simulation scripts. |\n\nResults feed directly into capacity‑planning spreadsheets (e.g., “X m5.large nodes sustain Y submissions/min”) and into the autoscaling policy definitions.\n\n---\n\n### 9.  Security & Auditing (strengthened)\n\n| Aspect | Enhancement |\n|--------|-------------|\n| **Transport security** | Mutual TLS everywhere (API‑gateway ↔ services, service‑to‑service). |\n| **Workload identity** | SPIFFE/SPIRE issued short‑lived certificates for each pod; automatic rotation. |\n| **Sandbox hardening** | gVisor + seccomp profile, read‑only root filesystem, no network, limited syscalls. |\n| **Image integrity** | Docker Content Trust (Notary) enforced at pod creation; weekly rebuilds from minimal Dockerfiles. |\n| **Immutable audit trail** | Append‑only DB tables, WORM bucket policy on object store, daily checksum verification job. |\n| **Role‑based access** | Students → own submissions only; staff → read all; admin → purge/archival. |\n| **Compliance** | Retention policy ≥ 5 years, periodic audit of WORM settings, GDPR‑compatible data‑subject access tooling. |\n\n---\n\n### 10.  Summary & Next Steps\n\n* **Latency guarantee** is now backed by a dynamic warm‑container pool, back‑pressure at the API edge, and idempotent processing.  \n* **Throughput scalability** is achieved through Kafka‑driven decoupling, adaptive plagiarism batching, and autoscaling node pools.  \n* **Reliability** is reinforced with Redis cluster HA, write‑ahead ingress logging, and a formal chaos‑testing regimen.  \n* **Observability** covers every critical path (queue lag, sandbox health, cache hit rates) and feeds automated alerts.  \n* **Security & auditability** meet university compliance requirements via immutable storage, signed images, and workload identity.  \n\n**Immediate actions**  \n\n1. Implement the write‑ahead `IngressLog` table and back‑pressure check.  \n2. Deploy Redis in HA cluster mode and configure fail‑over.  \n3. Build the pool‑maintainer controller to size idle containers per core.  \n4. Add the custom sandbox‑health exporter to Prometheus.  \n5. Run the updated performance test suite, capture baseline metrics, and tune autoscaling thresholds.  \n\nWith these refinements the system will reliably deliver sub‑30‑second grading feedback even during exam‑week spikes, while staying cost‑effective, secure, and fully auditable.",
+          "metadata": {
+            "tokensUsed": 0,
+            "latencyMs": 0,
+            "model": "openai/gpt-oss-120b:exacto"
+          }
+        },
+        {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "type": "critique",
+          "content": "## High‑level Impression  \n\nThe design is a **well‑structured, event‑driven pipeline** that isolates untrusted code, decouples heavy work from the API, and adds a number of safety nets (write‑ahead log, back‑pressure, idempotency, HA caches).  \nFrom an architectural standpoint it already hits most of the classic “secure‑sandbox‑at‑scale” checklist and it is **observable‑first**, which is essential for a 95 th‑percentile latency SLA.\n\nBelow is a systematic critique that looks at the **core functional flow**, **scalability & performance**, **operational complexity**, **data consistency & durability**, **security & compliance**, and **cost/efficiency**. For each area I point out what works, where the design is fragile or over‑engineered, and concrete ways to tighten it.\n\n---\n\n## 1. Functional Flow & Component Boundaries  \n\n| Component | What’s good | Potential issues |\n|----------|-------------|------------------|\n| **API Gateway → Submission Service** | Direct presigned‑URL upload removes the gateway from the data path; the lightweight metadata POST is fast. | The *IngressLog* table adds a synchronous DB write before the message is on Kafka. If the DB becomes a hotspot, the “200 OK” guarantee can be lost, defeating the back‑pressure goal. |\n| **Kafka as the backbone** | Strong decoupling, replayability, and ordering per `assignment_id`. | Using a single topic for *all* submissions can create hot partitions when a large exam is released (many submissions share the same `assignment_id`). Partition key must be chosen carefully (e.g., hash of `submission_id`) to avoid skew. |\n| **Grader Workers (warm‑container pool)** | Warm‑pool dramatically cuts cold‑start latency; pool‑maintainer reacts to lag. | The pool‑maintainer itself becomes a control‑plane component that must be highly reliable. If it mis‑estimates the needed pool size, latency spikes or over‑provisioning can occur. |\n| **Result Store (PostgreSQL) + Object Store** | Storing only URLs in the DB keeps rows small; hourly partitions avoid hot‑spot writes. | Hourly partitions mean a new table is created every hour – operationally heavy (vacuum, statistics, backup). A more conventional **time‑range partitioned table** (e.g., daily) may be sufficient and easier to manage. |\n| **Plagiarism Workers** | Adaptive batch size and isolation from the grading path keep the main SLA safe. | The batch‑size logic is tied to a time‑window (`T` seconds). If the arrival rate is very bursty, a batch could sit idle for the full `T`, violating the 2‑minute SLA for some submissions. A *sliding‑window* or *max‑wait* policy would be safer. |\n| **LTI/Outcome Service** | Separate service for LMS pushes keeps the core pipeline pure. | No mention of **idempotent delivery guarantees** (Canvas can reject duplicate posts). A small “outbox” table with retry state would make the integration more robust. |\n\n**Recommendation** – Keep the current separation but **simplify the write‑ahead path**: use a *Kafka‑producer‑side transaction* (or exactly‑once semantics) instead of a separate DB table, or at least make the `IngressLog` a *lightweight, append‑only* log (e.g., a Kafka compacted topic) that can survive a DB outage without blocking the API.\n\n---\n\n## 2. Scalability & Performance  \n\n### 2.1 Ingestion & Back‑pressure  \n\n* **Strength** – The API checks consumer lag and returns `429` when the pipeline is saturated.  \n* **Weakness** – Lag is measured **after** the message is produced (Kafka JMX). If the producer is blocked on the DB write, the lag check never runs, and the system can still be overwhelmed.  \n\n**Improvement** – Perform the lag check **before** the DB insert, using a *fast cache* (e.g., Redis) that mirrors the latest consumer lag. The cache can be refreshed every few seconds from JMX. This guarantees the API can reject early, protecting the DB and the write‑ahead table.\n\n### 2.2 Warm‑Container Pool  \n\n* **Strength** – Dynamic sizing per‑core (`idle_per_core = 0.5`) is a good heuristic.  \n* **Weakness** – The pool size is static per node; during an exam the *global* queue may be short on one node while another node is saturated, leading to uneven latency.  \n\n**Improvement** – Make the pool **cluster‑wide**: a central controller (could be the same pool‑maintainer) monitors per‑node lag and can *steal* idle containers from under‑utilised nodes (e.g., via a DaemonSet that spawns pods on demand). This reduces the need to over‑provision every node.\n\n### 2.3 Kafka Partitioning  \n\n* **Issue** – Partition key = `assignment_id` works for normal operation but creates a *single hot partition* for a large exam.  \n* **Solution** – Use a **composite key**: `hash(assignment_id || submission_id) % N` where `N` is the number of partitions (e.g., 30‑50). This spreads load while still allowing per‑assignment ordering if needed (ordering can be re‑established downstream by sorting on `submission_timestamp`).  \n\n### 2.4 Result Store Write Path  \n\n* **Current** – Workers do `INSERT … ON CONFLICT DO NOTHING` per submission, plus bulk `COPY` for logs.  \n* **Potential bottleneck** – Even with hourly partitions, the *primary key* (`submission_id`) is globally unique, causing a single index to be hit for every write.  \n\n**Alternative** – Use a **UUIDv7** (time‑ordered) as the primary key and **clustered index** on `assignment_id, created_at`. This keeps inserts localized to the recent partition and reduces index contention.\n\n### 2.5 Cache Layer  \n\n* **Strength** – Multi‑level cache (Redis cluster + node‑local `tmpfs`) reduces cross‑node traffic.  \n* **Weakness** – The design assumes **strong consistency** (write‑through) but does not address *cache invalidation* for compiled artefacts when a language version changes.  \n\n**Improvement** – Version the cache key with the **compiler version** and **sandbox image digest**. When an instructor upgrades the test suite or a new compiler is rolled out, the key automatically changes, avoiding stale binaries.\n\n### 2.6 Object‑store Prefix Sharding  \n\n* **Good** – Hashed prefixes avoid S3 per‑prefix throttling.  \n* **Missing** – No mention of **multipart upload concurrency limits** on the client side. During a burst, many workers may try to upload large logs simultaneously, hitting the S3 request‑rate limit per account.  \n\n**Mitigation** – Enable **S3 Transfer Acceleration** or use a **gateway cache** (e.g., MinIO in front of S3) that can buffer spikes locally before flushing to the remote store.\n\n---\n\n## 3. Operational Complexity  \n\n| Area | Complexity introduced | Suggested simplification |\n|------|-----------------------|--------------------------|\n| **Redis HA (3 masters + replicas)** | Requires Sentinel or Redis‑Cluster, careful slot rebalancing, and monitoring of fail‑over. | If the cache is *non‑critical* (i.e., can be rebuilt from DB), a **single‑master with persistence (AOF)** plus a **read‑replica** may be sufficient. The extra masters add little value for a cache that can be repopulated. |\n| **Write‑ahead IngressLog DB** | Extra table, extra transaction, extra backup/restore. | Replace with a **compact‑ed Kafka topic** (`ingress-log`) that stores `submission_id` and minimal metadata. Consumers can replay from this topic if needed, and the API can produce to it in the same transaction as the main `submissions` topic (using exactly‑once semantics). |\n| **Hourly DB partitions** | Need to create/drop tables each hour, manage statistics, vacuum. | Use **PostgreSQL native time‑range partitioning** (daily or weekly) and rely on **partition pruning** for queries. The overhead of a few extra partitions per day is negligible compared to the operational cost of hourly rotation. |\n| **Pool‑maintainer controller** | Custom controller, scaling logic, health checks. | Leverage **Kubernetes Horizontal Pod Autoscaler (HPA)** with a custom **external metric** (Kafka lag) to scale the *grader‑worker* deployment directly. The HPA can increase replica count, which implicitly creates more warm containers, removing the need for a separate pool manager. |\n| **Chaos‑mesh test harness** | Requires a separate test environment, coordination of failure injection. | Keep it, but **automate** as part of the CI pipeline (e.g., a nightly GitHub Action) so that the team does not have to run it manually. The value outweighs the added setup cost. |\n\n---\n\n## 4. Data Consistency, Durability & Auditing  \n\n* **Append‑only DB tables + WORM bucket** – solid for compliance.  \n* **Potential gap** – The system stores *large logs* in S3 and only a URL in the DB. If the S3 object is accidentally deleted (e.g., a lifecycle rule bug), the DB row becomes a dangling reference.  \n\n**Mitigation** – Enable **S3 Object Lock** (WORM) on the bucket from day 0, not just after a retention period. Also, run a **daily checksum verification job** that reads the object and compares it to a stored hash in the DB; any mismatch triggers an alert.\n\n* **Idempotency** – The design uses `INSERT … ON CONFLICT DO NOTHING`. However, if a worker crashes after writing the result but before publishing the `RunResult` event, the downstream `GradeCreated` may never happen.  \n\n**Solution** – Adopt an **outbox pattern**: after the DB write, the worker inserts a row into a local *outbox* table (same transaction). A separate *outbox‑publisher* reads pending rows and sends the Kafka event. This guarantees exactly‑once delivery without relying on the worker’s process lifetime.\n\n---\n\n## 5. Security & Compliance  \n\n| Strength | Observation |\n|----------|--------------|\n| **gVisor + seccomp + read‑only FS** | Provides strong isolation. |\n| **Docker Content Trust** | Prevents rogue images. |\n| **mTLS & SPIFFE** | Good for service‑to‑service auth. |\n| **WORM bucket** | Meets audit requirements. |\n\n**Remaining concerns**  \n\n1. **Network egress control** – The design mentions a dedicated NAT gateway, but does not specify **egress firewall rules**. TurnItIn, object‑store, and internal services share the same egress IP; a compromised sandbox could exfiltrate data.  \n   *Recommendation*: Use **per‑pod egress policies** (Kubernetes NetworkPolicy) that allow only the required destinations (object store, TurnItIn) and block everything else.  \n\n2. **Secret management** – No explicit mention of a secret store (e.g., Vault, KMS). API keys for TurnItIn, S3 credentials, and TLS certs need rotation and audit.  \n   *Recommendation*: Store all secrets in **Kubernetes Secrets encrypted at rest** backed by an external KMS, and rotate them automatically every 30 days.  \n\n3. **Sandbox syscall monitoring** – The proposal adds a custom sandbox‑health exporter, but does not define a **baseline** for allowed syscalls per language.  \n   *Recommendation*: Create **language‑specific seccomp profiles** derived from a minimal set (e.g., `read`, `write`, `exit`, `rt_sigreturn`). Any deviation should trigger an immediate kill and a high‑severity alert.  \n\n---\n\n## 6. Cost & Resource Efficiency  \n\n* **Dynamic autoscaling** (cluster autoscaler + pool‑maintainer) is a good way to keep cost proportional to load.  \n* **Node‑local cache** reduces cross‑node traffic but consumes RAM that could otherwise be used for more pods. The cache TTL of 24 h may be longer than needed for most assignments.  \n\n**Optimization** – Make the node‑local cache **size‑aware**: each node advertises its free memory, and the cache evicts aggressively when memory pressure rises. This avoids OOM situations that would otherwise kill sandbox pods.\n\n* **Object‑store lifecycle** – The three‑tier policy is appropriate, but the **IA and Glacier transition** can add retrieval latency if a student requests a historic log.  \n\n**Policy tweak** – Keep the most recent **90 days** in the *Standard* tier (fast retrieval) and move older data to IA. Only move to Glacier after 1 year, which aligns with most audit requirements while still offering cheap storage.\n\n---\n\n## 7. Summary of Key Action Items  \n\n| Category | Action | Rationale |\n|----------|--------|-----------|\n| **Ingress reliability** | Replace the DB `IngressLog` with a compacted Kafka topic (exactly‑once producer). | Removes a synchronous DB bottleneck and simplifies replay. |\n| **Back‑pressure** | Cache consumer lag in Redis (or a fast in‑memory store) and check it **before** DB insert. | Guarantees early rejection when the pipeline is saturated. |\n| **Kafka partitioning** | Use a hash of `submission_id` (or a composite key) to spread load across many partitions. | Prevents hot‑partition during large exams. |\n| **Warm‑container scaling** | Leverage HPA with external metric (Kafka lag) instead of a custom pool‑maintainer; allow cross‑node container stealing. | Reduces custom control‑plane code and improves burst handling. |\n| **Result store** | Switch to time‑range (daily) partitions and a clustered index on `assignment_id, created_at`. | Simplifies DB ops while keeping write locality. |\n| **Outbox pattern** | Add an outbox table for publishing `RunResult`/`GradeCreated` events. | Guarantees exactly‑once delivery even after worker crash. |\n| **Cache versioning** | Include compiler version, sandbox image digest, and test‑suite version in cache keys. | Avoids stale artefacts when any component changes. |\n| **Egress security** | Enforce per‑pod NetworkPolicies and use a dedicated egress IP per external service. | Limits data exfiltration risk. |\n| **Secret management** | Centralize all secrets in a KMS‑backed Vault or sealed‑Secrets solution. | Enables rotation, audit, and reduces leakage. |\n| **Lifecycle policy** | Keep 90 days in Standard tier, then IA, then Glacier after 1 year. | Balances cost vs. retrieval latency for audit requests. |\n| **Observability** | Add the outbox‑publisher metrics, sandbox‑syscall counters, and cache‑eviction rates to Prometheus dashboards. | Gives full visibility into the new reliability mechanisms. |\n| **Chaos testing** | Automate the chaos‑mesh scenarios in CI nightly runs. | Ensures the new components (outbox, HPA, cross‑node pool) survive failures. |\n\n---\n\n## Closing Thought  \n\nThe proposal already demonstrates a **mature, production‑grade mindset**—it anticipates latency spikes, isolates untrusted code, and builds a rich audit trail. The remaining work is mostly about **reducing unnecessary moving parts**, **tightening the back‑pressure loop**, and **ensuring that every asynchronous hand‑off is exactly‑once**. By applying the refinements above, the system will be easier to operate, cheaper at scale, and more robust against the inevitable bursts that exam periods bring.",
+          "metadata": {
+            "latencyMs": 37857,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 8143
+          },
+          "targetAgentId": "agent-performance"
+        },
+        {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "type": "critique",
+          "content": "## Performance‑Engineer's Review of the Refined Architecture  \n\nBelow is a **strength‑focused audit** followed by a **bottleneck analysis** and a set of **concrete, measurable improvements**.  All recommendations are expressed in terms of latency, throughput, resource utilisation, caching, algorithmic complexity and testability – the levers that directly affect the SLA targets.\n\n---\n\n### 1.  What Works Well (Strengths)\n\n| Area | Why it helps the SLA / Ops | Evidence of good practice |\n|------|----------------------------|---------------------------|\n| **Presigned‑URL direct upload** | Removes the API gateway from the data‑plane, eliminating a classic bandwidth and CPU choke point.  Upload latency now depends only on the client‑to‑object‑store path (≤ 5 s for 10 MiB). | Proven pattern in high‑throughput ingestion pipelines (e.g., S3 multipart). |\n| **Warm‑pool of containers / Firecracker VMs** | Guarantees sub‑second sandbox start‑up for > 95 % of runs, keeping the “submission‑to‑grade” latency budget well under the 2 min target. | Warm‑pool + image cache is the industry‑standard for low‑latency function‑as‑a‑service. |\n| **Kafka‑driven back‑pressure + HTTP 429** | Prevents unbounded queue growth during deadline spikes; the consumer‑lag metric becomes a natural scaling signal. | The “lag‑driven HPA” ties resource allocation to the actual work queue, a best‑practice for event‑driven systems. |\n| **Partitioned, batch‑insert grades** | Reduces row‑level lock contention and DB I/O, delivering > 5 k grade writes/min with < 5 ms lock wait. | Partitioning by semester‑assignment aligns write patterns with natural query boundaries. |\n| **MinHash‑LSH pre‑filter for plagiarism** | Turns an O(N²) similarity problem into near‑linear work; nightly Spark jobs stay within the 2 h window even at 20 k submissions. | LSH is the de‑facto algorithm for scalable similarity detection. |\n| **Hot‑cold audit‑log tiering + Elasticsearch index** | Keeps recent audit queries < 1 s while moving older data to cheap storage, reducing cost > 60 %. | Separation of write‑optimized hot store from read‑optimized index is a classic log‑analytics pattern. |\n| **Observability stack (Prometheus + Grafana + OpenTelemetry)** | End‑to‑end latency, consumer lag, sandbox start‑up, DB write latency are all measurable; alerts fire before SLA breach. | Metric‑driven SLO management is essential for a 95 % SLA. |\n| **Security hardening (seccomp, AppArmor, gVisor/Firecracker)** | Guarantees isolation without sacrificing the warm‑pool latency budget. | Aligns with FERPA/GDPR compliance while keeping the performance envelope. |\n\nOverall, the design already hits the major performance levers: **data‑plane off‑load, warm execution environments, back‑pressure, batched persistence, and algorithmic pruning**.\n\n---\n\n### 2.  Where the Design May Still Bottleneck  \n\n| Potential Bottleneck | Why it matters (performance impact) | Current mitigation (if any) | Gap / Risk |\n|----------------------|--------------------------------------|-----------------------------|------------|\n| **Warm‑pool sizing & churn** | If the pool is undersized during a deadline spike, containers fall back to cold start (≈ 5 s) → grading latency spikes.  Over‑provisioning wastes memory (each sandbox ≈ 256 MiB). | Fixed “≈ 0.5 × CPU‑cores per node” pool; scaling on consumer lag. | No predictive scaling based on *future* deadline traffic; no per‑language pool differentiation (some languages need larger images). |\n| **Image cache consistency** | Node‑local image cache must stay in sync when a new language version is released.  Stale images cause compile failures and re‑pull latency (seconds). | “Node‑level image cache” mentioned, but no eviction/refresh policy. | Risk of cache thrash after a release; no cache‑warm‑up for newly added nodes. |\n| **Kafka consumer lag metric granularity** | Lag is measured per topic, but a single “submissions” topic may contain heterogeneous payloads (large archives vs. tiny quizzes).  Large messages can artificially inflate lag, causing premature 429 responses. | Lag‑driven HPA, fast‑track topic for last‑minute submissions. | Fast‑track topic is a separate queue, but the main topic still mixes payload sizes; no per‑partition lag weighting. |\n| **TurnItIn worker pool & external throttling** | TurnItIn imposes per‑institution request limits.  A burst of new submissions could saturate the token bucket, leading to a backlog that spills over into the grading pipeline (delayed grades). | Dedicated pool + token‑bucket limiter + 24 h cache. | Cache TTL of 24 h may be too short for repeated checks; no fallback path if the provider is temporarily unavailable (e.g., circuit‑breaker). |\n| **Log compression & truncation** | Compressing logs *after* they are streamed to object store still consumes network bandwidth and sandbox memory.  Very verbose language runtimes (e.g., Java) can exceed the 1 MiB truncation limit, losing useful debugging info. | gzip before store, truncate > 1 MiB, configurable level. | No adaptive compression (e.g., LZ4 for low‑latency) and no tiered log retention (full logs for staff‑only view). |\n| **Elasticsearch indexing cost** | Indexing every audit event (potentially millions per day) can become CPU‑bound, especially during peak submission windows, causing indexing lag that defeats the “< 1 s query” promise. | Hot tier for recent month, index metadata only. | No bulk‑indexing window or back‑pressure from the indexing pipeline; risk of index queue overflow. |\n| **DB read‑replica lag for reporting** | Grades are written in batches; replication lag can be several seconds, causing the reporting UI to show stale grades during a burst. | Read‑replica used for dashboard. | No explicit SLA on replica lag; no “read‑after‑write” guarantee for the student UI. |\n| **Resource contention on sandbox nodes** | Each node runs a pool of containers + image cache + Redis + Prometheus exporters.  Memory pressure can cause OOM kills, especially when many languages with large runtimes (e.g., JVM) are active. | Sandbox quota 256 MiB per container, node‑level pool sizing. | No per‑node resource‑budget model; no cgroup‑level QoS to protect the host. |\n| **Testing realism** | Load‑tests with k6 are mentioned, but the test plan does not explicitly model *deadline‑driven spikes* (e.g., 10 min before deadline, 5 × normal rate). | Load‑test with k6, chaos‑testing. | Without a realistic “deadline‑burst” pattern, capacity planning may be optimistic. |\n\n---\n\n### 3.  Concrete, Measurable Improvements  \n\nBelow are **actionable items** grouped by the bottleneck they address.  Each includes a **metric** that can be added to the existing Prometheus‑Grafana stack to verify impact.\n\n#### 3.1 Warm‑Pool & Image Cache  \n\n| Action | Detail | Metric to watch |\n|--------|--------|-----------------|\n| **Dynamic pool sizing** | Implement a controller that watches `kafka_consumer_lag_seconds` *and* the upcoming deadline calendar (e.g., 30 min before a known deadline, pre‑scale pool to `expected_concurrent_submissions * 1.2`). | `sandbox_pool_size_target`, `sandbox_pool_actual`, `sandbox_cold_start_seconds`. |\n| **Per‑language pool** | Keep separate pools for “heavy” runtimes (JVM, .NET) vs. “light” (Python, C).  Size each pool based on historic language‑mix. | `sandbox_pool_by_lang{lang}`. |\n| **Cache warm‑up on node join** | When a new node joins the cluster, pre‑pull the most‑used language images (top‑3) before the node becomes a consumer. | `image_pull_latency_seconds`, `image_cache_hits_total`. |\n| **Cache eviction policy** | Use an LRU policy with a TTL (e.g., 48 h) to evict stale images; trigger a pull on first use. | `image_cache_evictions_total`. |\n\n**Expected impact:** Cold‑starts reduced from 5 s to < 0.5 s even under a 10× burst; memory usage stays bounded because heavy images are only kept for the most active languages.\n\n#### 3.2 Kafka Lag & Back‑Pressure  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **Weighted lag per message size** | Compute `lag_seconds_weighted = Σ(message_size * lag_seconds) / Σ(message_size)`.  Use this for HPA scaling instead of raw lag count. | `kafka_weighted_lag_seconds`. |\n| **Separate “large‑payload” topic** | Route submissions with archive > 5 MiB to a dedicated topic (`submissions_large`).  Scale a dedicated consumer group for those only when `large_topic_lag` exceeds a threshold. | `kafka_large_topic_lag`. |\n| **Graceful 429 with retry‑budget** | Return `Retry-After` based on current lag *and* a token bucket that limits the number of 429 responses per minute (prevents client‑side hammering). | `http_429_rate`, `http_429_retry_after_seconds`. |\n\n**Expected impact:** More accurate scaling, fewer unnecessary 429s for small submissions, and a smoother latency curve during mixed‑size bursts.\n\n#### 3.3 TurnItIn Integration  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **Circuit‑breaker** | If the TurnItIn error rate exceeds 2 % over a 5‑minute window, pause new jobs and fall back to “cached‑only” mode. | `turnitin_circuit_breaker_state`. |\n| **Extended cache TTL for repeat submissions** | Store results for 7 days (instead of 24 h) keyed by `submission_hash`.  Most students resubmit the same code within a short window. | `turnitin_cache_hit_ratio`. |\n| **Back‑off exponential with jitter** | Already mentioned, but enforce a max back‑off of 30 s to avoid long stalls. | `turnitin_retry_delay_seconds`. |\n\n**Expected impact:** TurnItIn failures no longer cascade into grading latency; cache hit ratio climbs from ~70 % to > 90 % for repeat submissions.\n\n#### 3.4 Log Handling  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **In‑sandbox log streaming with LZ4** | Replace gzip (CPU‑heavy) with LZ4 for on‑the‑fly compression; decompress only on demand. | `sandbox_log_compression_latency_seconds`. |\n| **Tiered log retention** | Keep full logs for 24 h in a “staff‑only” bucket (restricted IAM); after 24 h, replace with truncated version for audit. | `log_full_retention_seconds`. |\n| **Log size budgeting** | Enforce a per‑run log size limit (e.g., 2 MiB) via a side‑car that drops excess lines, preventing OOM. | `sandbox_log_dropped_bytes_total`. |\n\n**Expected impact:** Network usage for log upload drops ~30 % more, sandbox memory pressure reduced, and staff still have access to full logs when needed.\n\n#### 3.5 Elasticsearch Indexing  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **Bulk indexing windows** | Accumulate audit events in 500‑ms batches before sending to ES; use the `_bulk` API with `refresh=false`. | `es_bulk_batch_size`, `es_bulk_latency_seconds`. |\n| **Back‑pressure from ES** | If the ES indexing queue depth exceeds a threshold, pause the audit‑service producer (Kafka) for a short interval. | `es_queue_depth`. |\n| **Hot‑index rollover** | Create a new daily index for the hot tier; close old indices after 7 days to reduce segment count. | `es_active_shards`. |\n\n**Expected impact:** Indexing latency stays < 200 ms per batch, preventing audit‑log lag spikes during deadline bursts.\n\n#### 3.6 DB Replication & Reporting  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **Synchronous write‑through for recent grades** | For the most recent 5 min of grades, write to the replica synchronously (or use `pglogical` to stream). | `replica_lag_seconds`. |\n| **Materialized view for “latest grade per student”** | Refresh every 5 s; UI reads from the view, avoiding heavy joins on the raw table. | `materialized_view_refresh_time_seconds`. |\n| **Read‑after‑write verification** | After a batch insert, poll the replica for the inserted IDs; if missing, trigger a retry. | `grade_write_verification_latency_seconds`. |\n\n**Expected impact:** Student UI sees their grade within ≤ 5 s of completion, eliminating perceived latency even while the reporting dashboard continues to use the replica.\n\n#### 3.7 Resource Budgeting & Isolation  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **cgroup QoS tiers** | Assign “high‑priority” sandbox containers to `cpu.shares=1024`, “low‑priority” to `512`.  Prevent a flood of heavy JVM runs from starving the pool. | `sandbox_cpu_shares_utilization`. |\n| **Node‑level resource model** | Define a hard limit: `max_containers = floor(node_memory / (sandbox_memory + overhead))`.  Enforce via the scheduler (K8s `resourceQuota`). | `node_sandbox_capacity`. |\n| **Memory pressure alerts** | Trigger when `node_memory_available < 10%`. | `node_memory_pressure`. |\n\n**Expected impact:** OOM events drop to zero, and overall node utilisation stays in the 70‑80 % range, giving headroom for spikes.\n\n#### 3.8 Load‑Testing & Chaos Engineering  \n\n| Action | Detail | Metric |\n|--------|--------|--------|\n| **Deadline‑burst scenario** | Use k6 to ramp up to 8 × normal submission rate during a 10‑minute window, then drop back.  Measure end‑to‑end latency, 429 rate, and pool utilisation. | `deadline_burst_latency_p95_seconds`. |\n| **Failure injection** | Randomly kill a Kafka broker, a sandbox node, and a PostgreSQL replica during the burst; verify auto‑recovery and SLA adherence. | `service_recovery_time_seconds`. |\n| **Continuous performance regression** | Store baseline metrics in a time‑series DB; fail the CI pipeline if any SLA metric degrades > 5 % from baseline. | `ci_performance_regression`. |\n\n**Expected impact:** Capacity planning is validated against realistic worst‑case traffic, and the system’s resilience to component failures is quantified.\n\n---\n\n### 4.  Revised SLA Monitoring Blueprint  \n\n| SLA | Current Target | New Metric to Enforce | Alert Threshold (95 pct) |\n|-----|----------------|-----------------------|--------------------------|\n| **Submission‑to‑grade** | ≤ 2 min | `submission_to_grade_seconds` (timestamp at metadata POST → `GradeCreated`) | > 120 s |\n| **Sandbox warm‑start** | ≤ 1 s | `sandbox_warm_start_seconds` | > 1.5 s |\n| **Kafka consumer lag** | ≤ 30 s | `kafka_weighted_lag_seconds` | > 30 s |\n| **Upload response** | ≤ 5 s | `gateway_upload_latency_seconds` | > 5 s |\n| **Grade DB write** | ≤ 100 ms per batch | `grade_batch_write_latency_seconds` | > 0.1 s |\n| **Plagiarism nightly job** | ≤ 2 h | `plagiarism_job_duration_seconds` | > 7200 s |\n| **TurnItIn success** | ≥ 99 % | `turnitin_success_rate` | < 0.99 |\n| **Audit‑log append** | ≤ 200 ms | `audit_log_produce_latency_seconds` | > 0.2 s |\n| **Replica lag** | ≤ 5 s for recent grades | `replica_lag_seconds` | > 5 s |\n\nAll metrics should be exported via **Prometheus** with **SLO‑based alerts** routed to PagerDuty.  The dashboard must show the *trend* of each metric, not just the current value, to spot creeping latency before it breaches the SLA.\n\n---\n\n### 5.  Summary & Prioritisation  \n\n| Priority | Quick‑win (≤ 2 weeks) | Medium‑term (1‑2 months) | Long‑term (3 + months) |\n|----------|-----------------------|--------------------------|------------------------|\n| **1** | Implement weighted Kafka lag & fast‑track topic; add `submission_to_grade_seconds` metric. | Dynamic warm‑pool controller with deadline‑aware scaling. | Full micro‑VM (Firecracker) isolation with CRIU checkpoint/restore for ultra‑fast reset. |\n| **2** | Add LZ4 log streaming and truncation policy; expose `sandbox_log_compression_latency_seconds`. | Per‑language sandbox pools + image cache warm‑up on node join. | Hierarchical audit‑log tiering with automated lifecycle policies and ES segment optimisation. |\n| **3** | TurnItIn cache TTL extension + circuit‑breaker; expose `turnitin_cache_hit_ratio`. | Bulk ES indexing with back‑pressure; materialised view for latest grades. | Predictive capacity planning using machine‑learning on historic deadline spikes. |\n\nBy **addressing the identified bottlenecks** with the concrete actions above, the platform will not only meet the current SLA targets under normal load but will also retain **predictable latency and high throughput** during the inevitable deadline‑driven traffic spikes that characterize university grading systems.",
+          "metadata": {
+            "latencyMs": 43782,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 7810
+          },
+          "targetAgentId": "agent-architect"
+        },
+        {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "type": "refinement",
+          "content": "## Refined Design – Performance‑Centric Response to the Critique  \n\nBelow is a tightened version of the original architecture that directly addresses the weaknesses highlighted in the critique while preserving the latency, throughput and audit guarantees that the SLA demands.  Every change is justified in terms of **resource utilisation, algorithmic simplicity, cache effectiveness and observability**.\n\n---\n\n### 1.  End‑to‑end Flow (simplified)\n\n1. **LMS → API‑Gateway** – client receives a presigned URL and streams the source file straight to the object store (multipart, no gateway buffering).  \n2. **Metadata POST** – a tiny JSON payload (assignment‑id, language, student‑id, checksum) is posted to the *Submission Service*.  \n3. **Fast‑path ingest** – the service **(a)** checks a *cached consumer‑lag metric* (see §2.1), **(b)** writes a single row to a **compact‑ed Kafka topic** (`ingress‑log`) that contains the metadata and a generated `submission_id`, and **(c)** returns `200 OK`.  \n4. **Kafka backbone** – the same `submission_id` is produced to the *grading* topic using **exactly‑once semantics** (transactional producer).  \n5. **Grader Workers** – pull from the grading topic, acquire a warm container from the node‑local pool, run the compile‑test‑grade cycle, write the result to the *Result Store* **and** insert a row into a local *outbox* table (same DB transaction).  \n6. **Outbox Publisher** – a lightweight side‑car reads pending outbox rows and emits the `GradeCreated` event to a downstream *notification* topic.  \n7. **Result Store** – append‑only, time‑range partitioned (daily) table that stores only metadata and a URL to the immutable object store.  \n8. **Plagiarism Workers** – consume a separate *plagiarism‑request* topic, batch adaptively (max‑wait = 30 s, max‑size = 200) and invoke TurnItIn asynchronously.  \n9. **Observability & Alerts** – every component exports Prometheus metrics; Grafana dashboards drive autoscaling and SLA alerts.\n\nThe diagram is unchanged in spirit, but the **control‑plane** now consists of only three well‑known Kubernetes primitives (HPA, Deployment, CronJob) plus a tiny *outbox* side‑car, eliminating the bespoke “pool‑maintainer” controller.\n\n---\n\n### 2.  Performance‑Focused Improvements  \n\n| Area | Original Issue | Refined Solution | Expected Performance Impact |\n|------|----------------|------------------|-----------------------------|\n| **Ingress reliability** | Synchronous DB write (`IngressLog`) could become a hotspot, breaking the 200 ms “fast‑ack”. | Replace the table with a **compact‑ed Kafka topic** (`ingress‑log`). The producer writes the metadata in a single network round‑trip; the topic is configured for **log compaction**, so duplicate submissions are automatically deduped. | Eliminates a DB write path, reduces 95 pct latency to < 150 ms even under peak ingest. |\n| **Back‑pressure** | Lag check performed after DB insert; still vulnerable to DB overload. | Maintain a **Redis‑backed lag cache** refreshed every 2 s from Kafka JMX. The Submission Service reads this cache *before* publishing to `ingress‑log`. If lag > 500, it returns `429 Retry‑After`. | Guarantees early rejection, protects both the object store and the Kafka producer from overload. |\n| **Kafka partitioning** | Single key (`assignment_id`) creates hot partitions during large exams. | Use a **hash‑based composite key**: `hash(assignment_id || submission_id) % P` where `P` = 30‑50 partitions. Ordering per assignment is re‑established downstream by sorting on `submission_timestamp`. | Distributes load evenly, prevents any partition from exceeding the 500‑msg lag threshold. |\n| **Warm‑container pool** | Custom controller can mis‑size the pool; per‑node static sizing leads to uneven latency. | Leverage **Kubernetes HPA** with an **external metric** (Kafka lag). Scaling the *grader‑worker* Deployment automatically creates more pods, each of which starts with a pre‑warmed container (init‑container pulls the language image). No separate pool‑maintainer is needed. | Guarantees that the 95 pct latency stays ≤ 30 s for any burst; reduces operational code surface. |\n| **Result Store writes** | Hourly partitions cause heavy DDL churn; global primary‑key index creates contention. | Switch to **daily time‑range partitions** (PostgreSQL native) and a **clustered index** on `(assignment_id, created_at)`. Use **UUIDv7** (time‑ordered) as the primary key, which naturally groups recent inserts. | Removes hourly table‑creation overhead, lowers index contention, sustains > 1 k writes / sec. |\n| **Idempotent event emission** | Worker crash after DB write could lose the downstream `GradeCreated` event. | Adopt the **outbox pattern**: the DB transaction that writes the grade also inserts a row into an *outbox* table. A side‑car reads pending rows and publishes them transactionally to Kafka. | Guarantees exactly‑once delivery without relying on worker process lifetime. |\n| **Cache versioning** | Compiled‑artifact cache could serve stale binaries after a compiler or test‑suite change. | Encode **compiler version, sandbox image digest, and test‑suite version** into the cache key (`<src_hash,lang,compiler,image,test_version>`). When any component is upgraded, the key changes automatically, forcing a fresh compile. | Prevents silent grading regressions; cache‑hit rate remains ≥ 80 % for unchanged assignments. |\n| **Node‑local cache sizing** | Fixed 24 h TTL may waste RAM on rarely reused artefacts. | Make the cache **size‑aware**: each node advertises free memory; the cache evicts LRU entries when usage exceeds a configurable threshold (e.g., 70 % of pod memory). TTL is still 24 h but eviction can happen earlier under pressure. | Avoids OOM kills, keeps sandbox pods healthy while preserving most hot‑path hits. |\n| **Object‑store prefix sharding** | No explicit handling of S3 request‑rate limits during bursts. | Continue using **hashed prefixes** (`/<hash>/submission/...`) and enable **S3 Transfer Acceleration** (or a local MinIO gateway) to buffer spikes before flushing to the remote store. | Guarantees < 10 ms write latency even when 500 submissions/min are uploaded. |\n| **Redis HA** | Three‑master cluster adds operational burden for a cache that can be rebuilt. | Deploy **Redis in a single‑master with AOF persistence** plus one read‑replica. The cache is *ephemeral*; on master failure the replica promotes quickly, and any missing entries are recomputed on demand. | Reduces cluster‑management overhead while still providing sub‑millisecond reads and durability for critical metadata. |\n\n---\n\n### 3.  Caching Strategy (concrete refinements)\n\n| Cache | What is stored | Location | Key composition | TTL / Invalidation |\n|-------|----------------|----------|-----------------|--------------------|\n| **API‑gateway metadata** | Assignment description, test‑case list | Redis cluster (replicated) | `assignment:{id}` | 5 min or on assignment update |\n| **Compiled artefacts** | Binary / bytecode | Redis (binary cache) + node‑local `tmpfs` mirror | `<src_hash>:<lang>:<compiler_version>:<image_digest>:<test_suite_version>` | 24 h or LRU eviction |\n| **Test‑case data** | Input / expected output files | Redis read‑through | `testcase:{assignment_id}` | Until assignment end |\n| **Recent grades** | Student‑grade list (last 24 h) | Redis LRU | `grades:{student_id}` | 10 min |\n| **Plagiarism similarity graph** | Pairwise scores for the current batch | Worker‑local memory | `plag_batch:{batch_id}` | Recomputed each batch |\n| **LMS grade push outbox** | Pending grade notifications | PostgreSQL outbox table | `outbox:{grade_id}` | Processed by outbox‑publisher, then deleted |\n| **Ingress‑log deduplication** | Already‑processed `submission_id` | Compact‑ed Kafka topic (`ingress‑log`) | `submission_id` (key) | Log compaction removes old entries automatically |\n\nAll caches are **write‑through**: a successful DB write triggers an immediate cache update via the same service call, guaranteeing strong consistency for the 95 pct latency target.\n\n---\n\n### 4.  Resource‑Efficiency & Autoscaling  \n\n| Resource | Policy | Rationale |\n|----------|--------|-----------|\n| **Kubernetes nodes** | **Cluster Autoscaler** + **HPA** on grader‑worker Deployment (external metric = Kafka lag). | Nodes are added only when pending pods exceed the node capacity; pods are added when lag rises, keeping latency bounded. |\n| **Sandbox quotas** | cgroup limits: CPU ≤ 80 % of quota, memory = 256 MiB (adjustable per language). | Prevents noisy‑neighbor effects; OOM events are captured and reported as “resource‑exceeded”. |\n| **Node‑local cache** | Size‑aware eviction at 70 % of pod memory. | Keeps hot artefacts in RAM without starving the sandbox. |\n| **Object‑store lifecycle** | 0‑90 days → Standard, 90‑365 days → IA, > 1 yr → Glacier Deep Archive; **Object Lock** enabled from day 0. | Meets audit‑retention (≥ 5 yr) while controlling cost; lock guarantees immutability. |\n| **Redis persistence** | AOF with fsync = every second; one read‑replica. | Guarantees that a crash does not lose cache entries needed for ongoing grading. |\n| **Kafka retention** | 48 h for grading topic, 7 d for ingress‑log topic. | Sufficient for replay while limiting storage cost. |\n\n---\n\n### 5.  Observability – Metrics that Matter  \n\n| Metric | Collector | SLA‑related Alert |\n|--------|-----------|-------------------|\n| **Submission‑to‑grade latency (p95)** | Worker exporter → Prometheus | > 25 s |\n| **Kafka consumer lag (per partition)** | JMX exporter → Prometheus | > 500 msgs |\n| **Redis cache‑hit rate (overall)** | Redis INFO → Prometheus | < 70 % |\n| **Sandbox CPU utilisation** | cAdvisor → Prometheus | > 80 % for > 5 min |\n| **Sandbox syscall violations** | Falco (eBPF) → Prometheus | Any violation |\n| **Outbox backlog size** | DB exporter → Prometheus | > 1000 pending events |\n| **Object‑store write latency** | S3 SDK metrics → Prometheus | > 15 ms |\n| **API 5xx error rate** | Envoy/Istio → Prometheus | > 1 % |\n| **Node‑local cache eviction rate** | Node exporter → Prometheus | > 30 %/hour |\n| **Plagiarism batch duration** | Worker logs → Prometheus | > 3 min per 100 submissions |\n\nAll alerts feed into PagerDuty with severity tiers (latency → P1, availability → P2).  Grafana dashboards display latency histograms, lag heat‑maps, cache‑hit trends and resource utilisation, enabling rapid capacity planning.\n\n---\n\n### 6.  Security & Auditing Enhancements  \n\n| Concern | Strengthened Control |\n|---------|----------------------|\n| **Ingress & egress** | **NetworkPolicy** per pod: only object‑store (S3) and TurnItIn endpoints are allowed outbound; all other egress blocked.  A dedicated NAT gateway provides a single source IP for external services, simplifying firewall rules. |\n| **Secret management** | All API keys, TLS certs and DB passwords stored in **Kubernetes Secrets** encrypted with an external KMS (e.g., Cloud KMS).  Automatic rotation every 30 days via a CI job. |\n| **Image integrity** | **Docker Content Trust** enforced by the orchestrator; images are signed with Notary and verified at pod creation.  Base images are rebuilt weekly from a minimal Dockerfile and scanned with Trivy. |\n| **Sandbox syscall profile** | Language‑specific **seccomp** profiles generated from a whitelist of allowed syscalls (read, write, exit, rt_sigreturn, futex, clock_gettime).  Any deviation triggers a Falco alert and immediate container termination. |\n| **Immutable audit trail** | Result Store tables are **append‑only**; a nightly job computes SHA‑256 checksums of each object and stores them in a verification table.  S3 **Object Lock** (WORM) is enabled from day 0, preventing accidental deletion. |\n| **Idempotent LMS push** | Outbox table includes a `delivery_attempt` counter and a `last_status` field.  The notification worker retries with exponential back‑off until a 2xx response, then marks the row as `delivered`. |\n| **Compliance** | Retention policy (Standard → IA → Glacier) enforced by S3 lifecycle rules; quarterly audit of bucket policies and WORM settings; GDPR‑compatible data‑subject access tooling built on top of the outbox and audit tables. |\n\n---\n\n### 7.  Testing & Validation Plan (Performance‑first)\n\n| Test | Goal | Method |\n|------|------|--------|\n| **Peak ingest** | 500 req/min sustained, 1 k req/min burst, 95 pct latency ≤ 30 s | k6 script drives the presigned‑URL upload + metadata POST; monitors lag cache, Redis hit‑rate, Kafka lag. |\n| **Back‑pressure correctness** | Verify API returns 429 before DB is touched when lag > 500 | Custom probe reads Redis lag cache, forces lag > 500, then sends POSTs; asserts 429 response time < 50 ms. |\n| **HPA scaling** | Determine minimal pod count that keeps latency < 30 s under burst | Load test with varying lag thresholds; capture HPA reaction time and resulting pod count. |\n| **Cache‑hit impact** | Run identical workload with cache warm vs cold | Compare latency histograms; expect ≥ 20 % reduction when cache‑hit ≥ 80 %. |\n| **Result‑store contention** | Spike to 1 k writes / sec, verify no lock timeouts | pgbench‑style bulk `COPY` simulation; monitor transaction latency and vacuum activity. |\n| **Outbox reliability** | Crash a grader worker after DB write, before event publish | Verify that the outbox row is still processed by the side‑car and the `GradeCreated` event appears. |\n| **Plagiarism batch SLA** | 1 000 new submissions, ensure each batch finishes ≤ 2 min | Mock TurnItIn endpoint with latency injection; monitor batch‑completion metric. |\n| **Chaos engineering** | Random pod kill, Kafka broker disconnect, Redis replica loss | Chaos Mesh runs nightly; asserts latency ≤ 30 s and no data loss. |\n| **Egress throttling** | Simulate NAT bandwidth limit, verify rate‑limited TurnItIn calls do not affect grading pipeline | `tc` traffic‑control on test cluster; monitor TurnItIn request latency vs grading latency. |\n| **Security validation** | Verify image signatures, seccomp enforcement, network policies | Trivy scan, Falco alerts, `kubectl netpol` dry‑run. |\n| **Lifecycle verification** | Populate audit data > 1 yr, trigger S3 lifecycle, confirm objects move to Glacier and remain immutable | S3 lifecycle simulation scripts, checksum verification after transition. |\n\nResults feed directly into capacity‑planning spreadsheets (e.g., “X m5.large nodes sustain Y submissions/min”) and into the HPA threshold definitions.\n\n---\n\n### 8.  Summary of the Refined Solution  \n\n| SLA Target | How the Design Meets It |\n|------------|--------------------------|\n| **Submission‑to‑grade ≤ 30 s (95 pct)** | Early lag‑check, exact‑once Kafka ingest, HPA‑driven grader scaling, warm containers, outbox‑driven event delivery. |\n| **API status/grade ≤ 200 ms** | Presigned‑URL upload bypasses gateway; metadata POST hits Redis‑cached lag and writes to compacted Kafka in < 150 ms. |\n| **Peak throughput 200 sub/min, burst 500 sub/min** | Partitioned Kafka with hash key, HPA scaling, node‑local cache, S3 prefix sharding. |\n| **Plagiarism ≤ 2 min per 100 sub** | Adaptive batch with max‑wait = 30 s, parallel TurnItIn workers, out‑of‑band similarity engine. |\n| **Cache‑hit ≥ 80 %** | Multi‑level cache (Redis + node‑local), versioned keys, write‑through updates. |\n| **Sandbox CPU ≤ 80 % quota, memory 256 MiB** | cgroup limits, Falco syscall monitoring, OOM alerts. |\n| **Kafka lag ≤ 500 msgs/partition** | Lag cache, HPA scaling, back‑pressure at API edge. |\n| **Object‑store write ≤ 10 ms** | Direct multipart upload to S3, hashed prefixes, optional Transfer Acceleration. |\n| **Audit immutability** | Append‑only DB, WORM S3 bucket, daily checksum verification. |\n\nAll **new components** (compact‑ed ingress topic, outbox side‑car, HPA‑driven scaling) are built from well‑tested open‑source primitives, reducing operational complexity while delivering measurable performance gains.  The design now has **fewer moving parts**, **stronger back‑pressure**, **exactly‑once guarantees**, and **tight security controls**, positioning the grading service to reliably meet its SLA even during the most demanding exam weeks.",
+          "metadata": {
+            "latencyMs": 9422,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 12084
+          }
+        },
+        {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "type": "refinement",
+          "content": "## Refined Architecture – “Scalable, Low‑Latency Grading Platform”  \n\nBelow is the updated design that directly answers the performance‑engineer’s observations.  The changes are grouped by **problem → concrete mitigation → measurable impact** and are woven back into the overall system view, data‑flow, and operational playbook.\n\n---\n\n### 1.  Core Architectural Style (unchanged)\n\n| Goal | Pattern | Why it still fits |\n|------|---------|-------------------|\n| Isolation of untrusted code | Container / micro‑VM sandbox (Docker + gVisor / Firecracker) | Strong security while keeping start‑up sub‑second. |\n| Burst‑y submissions | Event‑driven, Kafka‑backed micro‑services | Natural back‑pressure, independent scaling of each stage. |\n| Auditability | Immutable Kafka log + write‑once object store | Tamper‑evident history without stressing the primary DB. |\n| Plagiarism detection | MinHash‑LSH pre‑filter + incremental Spark jobs | Near‑linear runtime even at tens of thousands of submissions. |\n| LMS integration | LTI / REST outcome service | De‑facto university standard, idempotent pushes. |\n| Operations | GitOps‑driven K8s, Prometheus‑Grafana, OpenTelemetry | Self‑healing, observable, easy to roll‑out changes. |\n\n---\n\n### 2.  Updated Component Landscape  \n\n| Component | New / Modified Behaviour | Key Interfaces |\n|-----------|--------------------------|----------------|\n| **Web Front‑End (SPA)** | Obtains a presigned PUT URL, streams the archive directly to the object store, then posts a tiny JSON metadata record. | HTTPS → API Gateway |\n| **API Gateway / Auth Proxy** | 1️⃣ Validate university JWT 2️⃣ Issue a short‑lived presigned URL 3️⃣ Accept metadata POST (no binary payload). | Public HTTPS (REST) → internal gRPC/REST |\n| **Submission Service** | Stores only the metadata (assignment, student, object‑key) and emits **`SubmissionCreated`**. | Consumes from Gateway → Publishes to Kafka (`submissions`) |\n| **Warm‑Pool Manager** *(new)* | Continuously watches **Kafka consumer lag** *and* the academic calendar.  Before a known deadline it scales the sandbox pool to `expected_concurrent_submissions × 1.2`.  It also maintains **per‑language sub‑pools** (heavy runtimes vs. light runtimes). | Exposes metrics `sandbox_pool_target`, `sandbox_pool_actual` |\n| **Sandbox Service** | Pulls a ready container / Firecracker VM from the appropriate sub‑pool, mounts the archive read‑only, runs the language‑specific harness, streams compressed logs to object storage, then **re‑images** the VM (CRIU‑style fast reset). | Subscribes `submissions` → Publishes `run‑results` |\n| **Image Cache Service** *(new)* | Node‑local image cache with LRU eviction and a **48 h TTL**.  When a node joins the cluster it pre‑pulls the top‑3 most‑used language images.  Cache misses trigger an asynchronous pull that is recorded as a metric. | Consumed by Sandbox Service; metrics `image_pull_latency`, `image_cache_hit` |\n| **Grading Engine** | Batches up to 100 results, writes them to a **partitioned** `grades` table (partition key = semester‑assignment), and emits `GradeCreated`. | Subscribes `run‑results` → Writes to PostgreSQL → Publishes `grades` |\n| **Plagiarism Service** | Incremental nightly job: (1) MinHash‑LSH generates candidate pairs, (2) Full similarity runs only on those, (3) TurnItIn checks are performed by a **rate‑limited worker pool** that caches results for **7 days**. | Reads from DB & object store → Writes to Plagiarism DB → Emits `plagiarism‑alert` |\n| **TurnItIn Worker Pool** | Token‑bucket limiter, exponential back‑off with jitter, and a **circuit‑breaker** that pauses new checks if error‑rate > 2 % over 5 min. | Consumes `plagiarism‑jobs` → Calls TurnItIn API |\n| **LTI Outcome Service** | Idempotent pushes; stores the last‑sent token per student‑assignment to guarantee exactly‑once semantics. | Subscribes `grades` → Calls LMS |\n| **Audit & Logging Service** | Hot tier (S3‑Standard) holds the most recent month; older months are moved to a Glacier‑compatible bucket.  Only **metadata** (event ID, timestamps, object‑key) is indexed in **Elasticsearch**; full logs stay in object storage. | Reads all Kafka topics → Writes to hot store & ES |\n| **Reporting Dashboard** | Queries Elasticsearch for audit data and a **read‑replica** of PostgreSQL for grades.  A materialised view (`latest_grade_per_student`) is refreshed every 5 s for ultra‑low UI latency. | REST API → UI |\n| **Infrastructure** | Core services (Kafka, PostgreSQL, S3‑compatible store, K8s) plus **Prometheus exporters** for custom metrics (sandbox pool size, weighted lag, image‑cache health). | – |\n\n---\n\n### 3.  End‑to‑End Data Flow (with refinements)\n\n1. **Auth** – Student logs in via university IdP → receives JWT.  \n2. **Presigned URL** – SPA calls **Gateway** → receives a short‑lived PUT URL.  \n3. **Direct Upload** – Browser streams the zip straight to the object store (multipart, resumable).  \n4. **Metadata POST** – SPA posts JSON (`assignmentId`, `studentId`, `objectKey`) to **Gateway**.  \n5. **Submission Service** stores the record, emits **`SubmissionCreated`**.  \n6. **Warm‑Pool Manager** sees the event, selects the appropriate language sub‑pool, hands a ready sandbox to **Sandbox Service**.  \n7. **Sandbox** mounts the archive read‑only, runs the test harness, streams **LZ4‑compressed** logs (truncated at 2 MiB) to object storage, then resets the VM.  \n8. **RunResult** – Sandbox publishes `RunResult` (exit code, test summary, resource usage).  \n9. **Grading Engine** batches results, writes to the **partitioned grades table**, emits `GradeCreated`.  \n10. **LTI Outcome Service** pushes the grade to the LMS (idempotent).  \n11. **Audit Service** appends every event to the hot Kafka log, writes metadata to Elasticsearch, moves older logs to cold storage.  \n12. **Reporting UI** reads recent audit data from Elasticsearch and grades from the materialised view on the read‑replica.\n\n*Plagiarism* runs nightly on the incremental candidate set, stores flagged pairs, and notifies staff via the dashboard and optional email.\n\n---\n\n### 4.  Targeted Mitigations & Expected Impact  \n\n| Concern | Mitigation (implemented) | Metric Added | Expected Quantitative Impact |\n|---------|--------------------------|--------------|------------------------------|\n| **Warm‑pool undersizing during deadline spikes** | Dynamic pool sizing based on **consumer lag** *and* calendar‑driven pre‑scale; per‑language sub‑pools. | `sandbox_pool_target`, `sandbox_cold_start_seconds` | Cold‑starts < 0.5 s for 95 % of runs even at 10× normal load; memory utilisation stays ≤ 80 % of node capacity. |\n| **Stale container images** | Node‑local LRU image cache with 48 h TTL; auto‑pull on first use; pre‑pull top‑3 images on node join. | `image_cache_hit_total`, `image_pull_latency_seconds` | Cache‑hit rate > 95 %; image‑pull latency contribution to run time < 200 ms. |\n| **Kafka lag mis‑representation (mixed payload sizes)** | Weighted lag = Σ(messageSize × lag) / Σ(messageSize); separate `submissions_large` topic for archives > 5 MiB. | `kafka_weighted_lag_seconds`, `kafka_large_topic_lag` | Scaling decisions reflect true processing pressure; 429 responses drop by ~30 % for small submissions. |\n| **TurnItIn throttling & failures** | Dedicated worker pool with token‑bucket limiter, 7‑day result cache, circuit‑breaker that pauses checks on > 2 % error rate. | `turnitin_success_rate`, `turnitin_cache_hit_ratio`, `turnitin_circuit_breaker_state` | Success rate ≥ 99.5 %; cache hit ratio rises to > 90 % for repeat submissions; no backlog spills into grading pipeline. |\n| **Log upload overhead** | In‑sandbox LZ4 streaming, truncation at 2 MiB, tiered retention (full logs 24 h, then truncated). | `sandbox_log_compression_latency_seconds`, `sandbox_log_dropped_bytes_total` | Network usage for logs ↓ ≈ 40 %; sandbox memory pressure eliminated for verbose runtimes. |\n| **Elasticsearch indexing latency under burst** | Bulk‑indexing windows (500 ms batches) with `refresh=false`; back‑pressure from ES queue depth; daily hot‑index rollover. | `es_bulk_latency_seconds`, `es_queue_depth`, `es_active_shards` | Indexing latency ≤ 200 ms per batch; no indexing queue overflow during deadline spikes. |\n| **Read‑after‑write latency for grades** | Materialised view `latest_grade_per_student` refreshed every 5 s; synchronous write‑through for the most recent 5 min of grades. | `replica_lag_seconds`, `materialized_view_refresh_time_seconds` | Student UI sees grade within ≤ 5 s of completion; dashboard latency unchanged. |\n| **Resource contention on sandbox nodes** | cgroup QoS tiers (high‑priority vs. low‑priority containers), explicit `resourceQuota` for max containers per node, node‑level capacity model (`max_containers = floor(node_memory / (sandbox_mem + overhead)`). | `sandbox_cpu_shares_utilization`, `node_memory_pressure` | OOM events eliminated; node utilisation stays in 70‑80 % range, leaving headroom for spikes. |\n| **Load‑test realism** | New k6 script that ramps to 8× normal rate during a 10‑minute “deadline‑burst”, then drops; chaos tests that kill a Kafka broker, a sandbox node, and a PostgreSQL replica during the burst. | `deadline_burst_latency_p95_seconds`, `service_recovery_time_seconds` | Capacity plan validated; SLA breach probability < 1 % under worst‑case burst. |\n\n---\n\n### 5.  Revised SLA & Observability Blueprint  \n\n| SLA | Target (95 pct) | Measurement Point | Alert Threshold |\n|-----|----------------|-------------------|-----------------|\n| **End‑to‑end grading latency** | ≤ 2 min | `submission_to_grade_seconds` (metadata POST → `GradeCreated`) | > 120 s |\n| **Sandbox warm‑start** | ≤ 1 s | `sandbox_warm_start_seconds` | > 1.5 s |\n| **Weighted Kafka consumer lag** | ≤ 30 s | `kafka_weighted_lag_seconds` | > 30 s |\n| **Upload response time** | ≤ 5 s (≤ 10 MiB) | `gateway_upload_latency_seconds` | > 5 s |\n| **Grade DB batch write** | ≤ 100 ms | `grade_batch_write_latency_seconds` | > 0.1 s |\n| **Plagiarism nightly job** | ≤ 2 h | `plagiarism_job_duration_seconds` | > 7200 s |\n| **TurnItIn success rate** | ≥ 99 % | `turnitin_success_rate` | < 0.99 |\n| **Audit‑log append** | ≤ 200 ms | `audit_log_produce_latency_seconds` | > 0.2 s |\n| **Replica lag for recent grades** | ≤ 5 s | `replica_lag_seconds` | > 5 s |\n| **Elasticsearch indexing latency** | ≤ 200 ms per bulk | `es_bulk_latency_seconds` | > 0.2 s |\n\nAll metrics are exported via **Prometheus**, visualised on **Grafana**, and have PagerDuty alerts attached.  The dashboard shows *trends* (rolling 1‑hour windows) to catch creeping latency before an SLA breach.\n\n---\n\n### 6.  Operational & Governance Enhancements  \n\n| Area | Refined Practice |\n|------|------------------|\n| **CI/CD** | GitOps with **ArgoCD**; container images scanned (Trivy) and signed; sandbox images version‑locked and stored in a private registry. |\n| **Security** | Unprivileged sandbox users, **seccomp** & **AppArmor** profiles, optional **gVisor/Firecracker** for extra isolation; IAM policies restrict object‑store prefixes per student; student identifiers encrypted at rest (AES‑256). |\n| **Capacity Planning** | Baseline: 1.2 k uploads/term. System sized for **10×** peak (≈ 12 k uploads in 2 h). Warm‑pool size = `expected_concurrent_submissions × 1.2`; per‑language pool split 70 % light / 30 % heavy. |\n| **Disaster Recovery** | Kafka replicated across 3 zones; PostgreSQL HA with Patroni; object‑store multi‑zone replication; nightly backup of audit‑log metadata index; restore‑test drill monthly. |\n| **Compliance** | GDPR / FERPA: student IDs hashed, audit‑log hash‑chain stored in WORM bucket, retention policy configurable (default 5 years). |\n| **Support** | Self‑service UI for resubmission, log view (full logs for staff only), grade appeal workflow (audit‑logged). |\n| **Testing** | Load‑test with deadline‑burst pattern; chaos‑testing (kill Kafka broker, sandbox node, DB replica); benchmark warm‑pool start‑up per language; regression guardrails on SLA metrics in CI. |\n| **Observability** | OpenTelemetry instrumentation across all services; custom Prometheus exporters for sandbox pool, weighted lag, image‑cache health; Grafana alerts with multi‑stage escalation. |\n\n---\n\n### 7.  Implementation Roadmap  \n\n| Phase | Scope | Duration | Success Criteria |\n|-------|-------|----------|------------------|\n| **Phase 1 – Quick Wins (≤ 2 weeks)** | Weighted Kafka lag, fast‑track large‑payload topic, add `submission_to_grade_seconds` metric, enable 429 back‑off with `Retry‑After`. | 429 rate drops < 5 % for mixed‑size loads; latency metric visible. |\n| **Phase 2 – Warm‑Pool & Image Cache (1‑2 months)** | Deploy Warm‑Pool Manager, per‑language sub‑pools, Image Cache Service with pre‑pull on node join. | Cold‑start ≤ 0.5 s for 95 % of runs under 10× load; image‑cache hit > 95 %. |\n| **Phase 3 – Plagiarism & TurnItIn Enhancements (2‑3 months)** | Rate‑limited worker pool, 7‑day cache, circuit‑breaker, incremental Spark job with MinHash‑LSH. | TurnItIn success ≥ 99.5 %; nightly job ≤ 2 h for 20 k submissions. |\n| **Phase 4 – Logging, Indexing & DB Replication (3‑4 months)** | LZ4 streaming, tiered log retention, bulk ES indexing with back‑pressure, materialised view for latest grades. | Log upload bandwidth ↓ ≈ 40 %; grade UI latency ≤ 5 s; ES indexing latency ≤ 200 ms. |\n| **Phase 5 – Full Load‑Test & Chaos Validation (4‑5 months)** | Deadline‑burst k6 script, chaos experiments, SLA regression guardrails in CI. | System sustains 8× normal load with SLA compliance; automated alerts fire on any breach. |\n\n---\n\n### 8.  Summary of the Strengthened Solution  \n\n| Concern | Refined Resolution |\n|---------|--------------------|\n| **Warm‑pool sizing & churn** | Dynamic, deadline‑aware pool manager + per‑language sub‑pools; pre‑pull of popular images. |\n| **Image cache consistency** | LRU cache with TTL, node‑join pre‑pull, explicit pull‑latency metrics. |\n| **Kafka lag granularity** | Weighted lag metric, separate large‑payload topic, fast‑track for last‑minute submissions. |\n| **TurnItIn throttling** | Token‑bucket limiter, 7‑day cache, circuit‑breaker, exponential back‑off with jitter. |\n| **Log handling** | LZ4 streaming, truncation, tiered retention (full 24 h, then truncated). |\n| **Elasticsearch indexing** | Bulk windows, back‑pressure, daily hot‑index rollover, queue‑depth monitoring. |\n| **DB replication lag** | Materialised view refreshed every 5 s, synchronous write‑through for recent grades. |\n| **Resource contention** | cgroup QoS tiers, explicit `resourceQuota` per node, capacity model for max containers. |\n| **Testing realism** | Deadline‑burst load test, chaos injection, CI SLA regression guardrails. |\n| **Observability & SLA enforcement** | Full set of Prometheus metrics, Grafana dashboards, PagerDuty alerts, trend‑based monitoring. |\n\nWith these refinements the platform now **guarantees sub‑second sandbox start‑up, predictable end‑to‑end grading latency under extreme deadline spikes, and robust isolation and auditability**.  The architecture remains modular, observable, and ready for future growth (new languages, larger cohorts, additional LMS integrations) while staying within the operational complexity budget of a typical university‑run Kubernetes cluster.",
+          "metadata": {
+            "latencyMs": 42851,
+            "model": "openai/gpt-oss-120b:exacto",
+            "tokensUsed": 11354
+          }
+        }
+      ],
+      "timestamp": "2025-10-29T20:34:57.158Z",
+      "summaries": {
+        "agent-architect": {
+          "agentId": "agent-architect",
+          "agentRole": "architect",
+          "summary": "**Architectural Core** – A Kubernetes‑hosted, event‑driven micro‑service stack isolates untrusted student code in a **container/micro‑VM sandbox** (Docker + gVisor/Firecracker) backed by a **warm‑pool** to keep start‑up ≤ 1 s. All heavy work is decoupled from the UI via **Kafka** topics (`submissions`, `run‑results`, `grades`).  \n\n**Upload Path** – Students obtain a **presigned S3‑compatible URL** from the API Gateway, stream the archive directly to the object store, then POST lightweight metadata. This removes the gateway’s bandwidth bottleneck and lets the **Submission Service** emit a `SubmissionCreated` event.  \n\n**Grading Pipeline** – The **Sandbox Service** consumes the event, mounts the archive read‑only, runs language‑specific test harnesses, writes compressed logs to object storage and publishes `RunResult`. The **Grading Engine** batches results, writes to a **partitioned PostgreSQL grades table** (by semester/assignment), and emits `GradeCreated`.  \n\n**LMS Integration** – A dedicated LTI/Outcome Service consumes `GradeCreated` and pushes idempotent grades to Canvas/Blackboard/etc.  \n\n**Plagiarism** – Nightly Spark/Flink jobs first apply **MinHash‑LSH** to prune candidate pairs, then run full similarity checks and option",
+          "metadata": {
+            "beforeChars": 25094,
+            "afterChars": 1250,
+            "method": "length-based",
+            "timestamp": "2025-10-29T20:35:03.838Z",
+            "latencyMs": 6673,
+            "tokensUsed": 7163,
+            "model": "openai/gpt-oss-120b:exacto",
+            "temperature": 0.2,
+            "provider": "openrouter"
+          }
+        },
+        "agent-performance": {
+          "agentId": "agent-performance",
+          "agentRole": "performance",
+          "summary": "**Performance‑focused summary (≈1 200 chars)**  \n\n**SLA targets (95 pct):**  \n- Submission‑to‑grade ≤ 30 s (median ≈ 20 s).  \n- API status/grade ≤ 200 ms.  \n- Peak sustained 200 sub /min (≈ 3 /s); burst 500 sub /min (≈ 8 /s).  \n- Plagiarism batch ≤ 2 min per 100 submissions.  \n- Object‑store write ≤ 10 ms; cache‑hit ≥ 80 %.  \n- Kafka consumer lag ≤ 500 msgs/partition; sandbox CPU ≤ 80 % of quota, memory 256 MiB.\n\n**Key architectural refinements**  \n1. **Write‑ahead IngressLog DB** records every accepted upload before queuing; enables immediate 200 OK and replay on failure.  \n2. **Back‑pressure at API edge**: Submission Service checks Kafka lag; returns 429 with Retry‑After when lag > 500, protecting latency.  \n3. **Dynamic warm‑container pool**: pool‑maintainer keeps idle language‑specific containers ≈ 0.5 × CPU‑cores per node; scales with queue lag to keep cold‑start < 2 s.  \n4. **Redis HA cluster** (3 masters + replicas) stores critical caches: assignment metadata, compiled artefacts (`<src_hash,lang,compiler>`), test‑case data, recent grades. Write‑through updates guarantee consistency.  \n5. **Compiled‑artifact cache** + node‑local `tmpfs` mirror eliminates up to 30 % of compile latency.  \n6. **Test‑case read‑through cache** (T",
+          "metadata": {
+            "beforeChars": 29317,
+            "afterChars": 1250,
+            "method": "length-based",
+            "timestamp": "2025-10-29T20:35:11.548Z",
+            "latencyMs": 7699,
+            "tokensUsed": 8234,
+            "model": "openai/gpt-oss-120b:exacto",
+            "temperature": 0.2,
+            "provider": "openrouter"
+          }
+        }
+      }
+    }
+  ],
+  "createdAt": "2025-10-29T20:32:56.382Z",
+  "updatedAt": "2025-10-29T20:38:29.807Z",
+  "judgeSummary": {
+    "agentId": "judge-main",
+    "agentRole": "generalist",
+    "summary": "# Debate Summary: Automated Grading Platform\n\n## Core Architecture (Consensus)\n- **Event-driven microservices** with Kafka backbone for burst handling and back-pressure\n- **Sandboxed execution** (Docker + gVisor/Firecracker) for security isolation\n- **Immutable audit trail** (Kafka + WORM object store) for compliance\n- **Direct upload via presigned URLs** to bypass gateway bottlenecks\n- **MinHash-LSH pre-filtering** for plagiarism detection scalability\n\n## Key Evolution Through Debate\n\n**Initial → Refined:**\n1. **Ingress reliability**: Synchronous DB write → Compacted Kafka topic for faster ack (<150ms)\n2. **Warm-pool management**: Static sizing → Dynamic HPA-driven scaling based on consumer lag + calendar-aware pre-scaling\n3. **Kafka partitioning**: Single assignment_id key → Hash-based composite key to prevent hot partitions\n4. **Result persistence**: Hourly partitions → Daily partitions with UUIDv7 for reduced DDL churn\n5. **Event delivery**: Worker-managed → Outbox pattern for exactly-once guarantees\n6. **Cache keys**: Simple hash → Versioned keys including compiler/image/test-suite versions\n\n## Critical Trade-offs\n- **Redis HA**: Three-master cluster vs single-master+replica (chose simpler single-master; cache is rebuildable)",
+    "metadata": {
+      "beforeChars": 59815,
+      "afterChars": 1250,
+      "method": "length-based",
+      "timestamp": "2025-10-29T20:36:55.223Z",
+      "latencyMs": 17006,
+      "tokensUsed": 18236,
+      "model": "anthropic/claude-sonnet-4.5",
+      "temperature": 0.3,
+      "provider": "openrouter"
+    }
+  },
+  "finalSolution": {
+    "description": "# Synthesized Solution: Automated Programming Assignment Grading Platform\n\n## Executive Summary\n\nAfter analyzing the comprehensive proposals and refinements from both the architect and performance engineer, I present a unified solution that combines the strongest architectural patterns with proven performance optimizations. This synthesis resolves the key tensions between architectural elegance and operational pragmatism.\n\n**Confidence Score: 92/100**\n\nThe high confidence stems from:\n- Convergence on core patterns (event-driven, sandboxed execution, immutable audit)\n- Concrete, measurable performance targets with validation plans\n- Elimination of speculative components in favor of proven primitives\n- Clear operational model with well-defined SLAs\n\n---\n\n## 1. Core Architecture\n\n### 1.1 Architectural Style\n**Event-Driven Microservices with Immutable Audit Trail**\n\n```\n┌─────────────┐    ┌──────────────┐    ┌─────────────────┐\n│   LMS/SPA   │───▶│ API Gateway  │───▶│ Auth/Rate Limit │\n└─────────────┘    └──────────────┘    └─────────────────┘\n                           │\n                           ▼\n                   ┌──────────────┐\n                   │ Submission   │──┐\n                   │   Service    │  │ Presigned URL\n                   └──────────────┘  │ (Direct Upload)\n                           │         │\n                           ▼         ▼\n                   ┌──────────────────────┐\n                   │  Kafka (3-replica)   │\n                   │  - submissions       │\n                   │  - run-results       │\n                   │  - grades            │\n                   └──────────────────────┘\n                    │         │         │\n        ┌───────────┘         │         └──────────┐\n        ▼                     ▼                    ▼\n┌──────────────┐    ┌──────────────┐    ┌──────────────┐\n│   Sandbox    │    │   Grading    │    │ Plagiarism   │\n│   Workers    │    │   Engine     │    │   Service    │\n└──────────────┘    └──────────────┘    └──────────────┘\n        │                     │                    │\n        └─────────────────────┴────────────────────┘\n                              │\n                              ▼\n                   ┌──────────────────────┐\n                   │  Immutable Storage   │\n                   │  - PostgreSQL (HA)   │\n                   │  - S3 (WORM)         │\n                   │  - Elasticsearch     │\n                   └──────────────────────┘\n```\n\n### 1.2 Key Design Decisions\n\n| Decision | Rationale | Trade-off Accepted |\n|----------|-----------|-------------------|\n| **Presigned URL uploads** | Eliminates gateway bandwidth bottleneck; 80%+ reduction in gateway load | Slightly more complex client logic |\n| **Compacted Kafka topic for ingress** | Replaces custom IngressLog DB table; automatic deduplication | Requires Kafka expertise |\n| **Warm container pool with HPA** | Sub-second start-up without custom controller complexity | ~10% idle resource overhead |\n| **Outbox pattern for events** | Guarantees exactly-once delivery without distributed transactions | Additional side-car process |\n| **MinHash-LSH for plagiarism** | Reduces O(N²) to near-linear; fits 2h nightly window | Initial implementation complexity |\n| **Daily partitioned grades table** | High write throughput without hourly DDL churn | Slightly coarser partition granularity |\n\n---\n\n## 2. Component Specifications\n\n### 2.1 Upload & Ingestion Flow\n\n**API Gateway**\n- **Function**: JWT validation, presigned URL generation, rate limiting\n- **Technology**: Envoy/Istio with custom auth filter\n- **Performance**: \n  - Presigned URL generation: <50ms (p95)\n  - Rate limit check (Redis): <10ms\n  - Back-pressure trigger: Consumer lag >500 msgs → HTTP 429\n\n**Submission Service**\n- **Function**: Metadata persistence, Kafka event emission\n- **Storage**: Compacted Kafka topic `ingress-log` (key=submission_id)\n- **Performance**: \n  - Metadata POST: <150ms (p95)\n  - Exactly-once semantics via transactional producer\n  - Deduplication automatic via log compaction\n\n### 2.2 Execution Pipeline\n\n**Sandbox Service**\n- **Isolation**: Docker + gVisor (or Firecracker for higher security)\n- **Resource Limits**: \n  - CPU: 80% of 1 core quota\n  - Memory: 256 MiB (adjustable per language)\n  - Network: Blocked except object store\n- **Warm Pool Sizing**: \n  ```\n  pool_size = expected_concurrent × 1.2\n  per_language_split = {python: 40%, java: 30%, cpp: 20%, other: 10%}\n  ```\n- **Image Cache**: \n  - Node-local LRU with 48h TTL\n  - Pre-pull top-3 images on node join\n  - Target cache hit: >95%\n\n**Grading Engine**\n- **Batch Processing**: Up to 100 results per DB transaction\n- **Partitioning**: Daily time-range partitions on `(assignment_id, created_at)`\n- **Primary Key**: UUIDv7 (time-ordered) for natural clustering\n- **Outbox Pattern**: Side-car publishes `GradeCreated` events transactionally\n\n### 2.3 Plagiarism Detection\n\n**Architecture**:\n1. **Pre-filter**: MinHash-LSH generates candidate pairs (reduces to <5% of N²)\n2. **Similarity Engine**: Full comparison only on candidates\n3. **TurnItIn Integration**: \n   - Dedicated worker pool with token-bucket limiter\n   - 7-day result cache (keyed by submission hash)\n   - Circuit breaker: pause on >2% error rate over 5min\n   - Exponential backoff with jitter\n\n**Performance**:\n- Nightly batch: ≤2h for 20k submissions\n- Incremental processing: new submissions vs. existing corpus\n- TurnItIn success rate: ≥99.5%\n\n### 2.4 Audit & Compliance\n\n**Immutable Storage**:\n- **Source/binaries/logs**: S3-compatible with Object Lock (WORM)\n- **Metadata**: PostgreSQL append-only tables\n- **Event stream**: Kafka with 7-day retention\n\n**Lifecycle Policy**:\n- 0-90 days: S3 Standard (hot)\n- 90-365 days: S3 IA (warm)\n- >1 year: Glacier Deep Archive (cold)\n- Retention: 5 years minimum (accreditation requirement)\n\n**Indexing**:\n- Elasticsearch for metadata queries (last 30 days hot)\n- Bulk indexing: 500ms windows, `refresh=false`\n- Daily index rollover\n\n---\n\n## 3. Performance Specifications\n\n### 3.1 SLA Targets (95th percentile)\n\n| Metric | Target | Measurement | Alert Threshold |\n|--------|--------|-------------|-----------------|\n| **End-to-end grading** | ≤2 min | submission_to_grade_seconds | >120s |\n| **Sandbox warm-start** | ≤1s | sandbox_warm_start_seconds | >1.5s |\n| **Upload response** | ≤5s (≤10 MiB) | gateway_upload_latency_seconds | >5s |\n| **API status query** | ≤200ms | api_query_latency_seconds | >0.2s |\n| **Grade DB write** | ≤100ms/batch | grade_batch_write_latency_seconds | >0.1s |\n| **Kafka consumer lag** | ≤30s weighted | kafka_weighted_lag_seconds | >30s |\n| **Plagiarism batch** | ≤2h/20k subs | plagiarism_job_duration_seconds | >7200s |\n\n### 3.2 Throughput Targets\n\n- **Normal operation**: 200 submissions/min (≈3/s)\n- **Burst capacity**: 500 submissions/min (≈8/s)\n- **Peak design**: 10× baseline (12k uploads in 2h)\n\n### 3.3 Key Optimizations\n\n**Weighted Kafka Lag**:\n```\nweighted_lag = Σ(message_size × lag) / Σ(message_size)\n```\n- Separate topic for large payloads (>5 MiB)\n- HPA scales on weighted lag, not message count\n\n**Multi-Level Caching**:\n| Cache | Hit Target | Invalidation |\n|-------|-----------|--------------|\n| Compiled artifacts | >80% | Version-aware key: `<src_hash>:<lang>:<compiler>:<image>:<test_suite>` |\n| Test cases | >95% | Assignment end date |\n| API metadata | >90% | 5min TTL or update event |\n| Node-local images | >95% | 48h TTL + LRU eviction |\n\n**Log Handling**:\n- LZ4 streaming compression (≈40% bandwidth reduction)\n- Truncation at 2 MiB\n- Tiered retention: full 24h, then truncated\n\n---\n\n## 4. Security & Compliance\n\n### 4.1 Sandbox Hardening\n- **Syscall filtering**: Language-specific seccomp profiles\n- **Runtime monitoring**: Falco (eBPF) for violation detection\n- **Network isolation**: NetworkPolicy blocks all egress except object store\n- **Image integrity**: Docker Content Trust with Notary signatures\n\n### 4.2 Data Protection\n- **Encryption at rest**: AES-256 for student identifiers\n- **Encryption in transit**: mTLS everywhere (SPIFFE/SPIRE)\n- **Access control**: IAM policies restrict object-store prefixes per student\n- **Audit trail**: SHA-256 checksums verified nightly\n\n### 4.3 Compliance (GDPR/FERPA)\n- Student IDs hashed in logs\n- Data subject access tooling\n- Configurable retention (default 5 years)\n- WORM bucket prevents tampering\n\n---\n\n## 5. Operational Model\n\n### 5.1 Deployment\n- **Platform**: Kubernetes (managed or self-hosted)\n- **GitOps**: ArgoCD for declarative deployments\n- **CI/CD**: Image scanning (Trivy), automated testing, canary releases\n\n### 5.2 Observability Stack\n- **Metrics**: Prometheus + custom exporters\n- **Visualization**: Grafana dashboards\n- **Tracing**: OpenTelemetry (optional for deep debugging)\n- **Alerting**: PagerDuty with severity tiers (P1: latency, P2: availability)\n\n### 5.3 Capacity Planning\n**Baseline Calculation**:\n```\nstudents = 300\nassignments_per_term = 2\nresubmissions = 2\ntotal_submissions = 300 × 2 × 2 = 1,200/term\n\npeak_window = 2h (deadline)\npeak_rate = 1,200 / 2h ≈ 10/min normal\ndesign_peak = 10× = 100/min = 1.67/s\n```\n\n**Resource Sizing** (for 10× peak):\n- **Sandbox nodes**: 5× m5.2xlarge (8 vCPU, 32 GiB each)\n- **Kafka**: 3× m5.large (2 vCPU, 8 GiB)\n- **PostgreSQL**: 1× m5.xlarge primary + 1× replica\n- **Redis**: 1× m5.large (single master + replica)\n- **Object store**: S3-compatible with unlimited scaling\n\n### 5.4 Disaster Recovery\n- **Kafka**: 3-zone replication, 7-day retention\n- **PostgreSQL**: Patroni HA, nightly backups, PITR\n- **Object store**: Multi-zone replication\n- **RTO**: <1h for full service restoration\n- **RPO**: <5min (Kafka commit interval)\n\n---\n\n## 6. Testing & Validation\n\n### 6.1 Performance Testing\n**Load Test Scenarios**:\n1. **Sustained load**: 200 req/min for 1h\n2. **Deadline burst**: Ramp to 500 req/min over 10min, sustain 30min\n3. **Mixed payload**: 70% small (<1 MiB), 30% large (5-10 MiB)\n\n**Chaos Engineering**:\n- Random pod kills during burst\n- Kafka broker disconnect\n- PostgreSQL replica failure\n- Redis master failover\n\n**Success Criteria**:\n- All SLAs met during sustained load\n- <1% SLA breach during burst\n- Automatic recovery within 2min\n\n### 6.2 Security Testing\n- Image signature verification\n- Seccomp profile enforcement\n- Network policy validation\n- Penetration testing of sandbox escape vectors\n\n---\n\n## 7. Implementation Roadmap\n\n### Phase 1: Foundation (Weeks 1-4)\n- [ ] Deploy Kafka cluster with monitoring\n- [ ] Implement presigned URL upload flow\n- [ ] Build Submission Service with compacted topic\n- [ ] Add weighted lag metric and 429 back-pressure\n\n**Success**: Upload latency <5s, 429 rate <5%\n\n### Phase 2: Execution Pipeline (Weeks 5-10)\n- [ ] Deploy sandbox service with warm pool\n- [ ] Implement HPA on weighted lag\n- [ ] Add node-local image cache\n- [ ] Build grading engine with outbox pattern\n\n**Success**: Cold-start <1s, grading latency <2min\n\n### Phase 3: Plagiarism & Audit (Weeks 11-16)\n- [ ] Implement MinHash-LSH pre-filter\n- [ ] Deploy TurnItIn worker pool with circuit breaker\n- [ ] Configure S3 lifecycle policies\n- [ ] Set up Elasticsearch indexing\n\n**Success**: Nightly job <2h, audit queries <1s\n\n### Phase 4: Production Hardening (Weeks 17-20)\n- [ ] Full load testing with deadline burst\n- [ ] Chaos engineering validation\n- [ ] Security audit and penetration testing\n- [ ] Runbook and on-call training\n\n**Success**: All SLAs met under 10× load, zero critical security findings\n\n---\n\n## 8. Risk Mitigation\n\n| Risk | Likelihood | Impact | Mitigation |\n|------|-----------|--------|------------|\n| **Kafka cluster failure** | Low | High | 3-zone replication, automated failover, 7-day retention for replay |\n| **Sandbox escape** | Low | Critical | gVisor isolation, seccomp profiles, Falco monitoring, regular security audits |\n| **TurnItIn API outage** | Medium | Medium | 7-day cache, circuit breaker, graceful degradation (internal similarity only) |\n| **Deadline spike exceeds capacity** | Medium | High | HPA with 10× headroom, 429 back-pressure, pre-scaling based on calendar |\n| **Object store throttling** | Low | Medium | Hashed prefixes, Transfer Acceleration, monitoring of S3 request rates |\n| **PostgreSQL write contention** | Low | Medium | Daily partitions, batch inserts, read replica for queries |\n\n---\n\n## 9. Key Recommendations\n\n### 9.1 Must-Have (Non-Negotiable)\n1. **Presigned URL uploads** – Critical for gateway scalability\n2. **Warm container pool** – Only way to meet <1s start-up SLA\n3. **Outbox pattern** – Guarantees exactly-once grade delivery\n4. **Weighted Kafka lag** – Prevents mis-scaling on mixed payloads\n5. **WORM object storage** – Compliance requirement\n\n### 9.2 Should-Have (High Value)\n1. **MinHash-LSH plagiarism** – Enables nightly batch within 2h window\n2. **Node-local image cache** – Reduces network traffic, improves cache hit\n3. **Circuit breaker for TurnItIn** – Prevents cascade failures\n4. **Daily partitioned grades** – Sustains high write throughput\n5. **Chaos testing** – Validates resilience claims\n\n### 9.3 Nice-to-Have (Future Enhancements)\n1. **Firecracker micro-VMs** – Stronger isolation than gVisor (if needed)\n2. **Multi-region deployment** – For global university partnerships\n3. **ML-based plagiarism** – Semantic similarity beyond text matching\n4. **Real-time grading dashboard** – WebSocket updates for students\n5. **A/B testing framework** – For test-suite optimization\n\n---\n\n## 10. Confidence Assessment\n\n**Overall Confidence: 92/100**\n\n**Strengths** (+):\n- Proven patterns (event-driven, outbox, warm pools) ✓\n- Concrete SLAs with measurable metrics ✓\n- Comprehensive testing plan ✓\n- Clear operational model ✓\n- Security and compliance addressed ✓\n\n**Remaining Uncertainties** (-):\n- Exact warm pool sizing requires empirical tuning (-3)\n- MinHash-LSH implementation complexity (-2)\n- TurnItIn API behavior under load unknown (-2)\n- University-specific LMS integration quirks (-1)\n\n**Mitigation**:\n- Phase 1-2 includes load testing to calibrate pool size\n- MinHash-LSH has open-source implementations (e.g., datasketch)\n- TurnItIn circuit breaker provides graceful degradation\n- LTI standard reduces LMS-specific risk\n\n---\n\n## Conclusion\n\nThis synthesized solution delivers a **production-ready, scalable, secure automated grading platform** that meets all stated requirements while staying within the operational complexity budget of a typical university IT department. The architecture balances architectural elegance with performance pragmatism, using proven patterns and avoiding speculative components.\n\nThe phased implementation plan allows for iterative validation of assumptions, with clear success criteria at each stage. The comprehensive testing strategy, including chaos engineering, provides high confidence in the system's ability to meet SLAs even under extreme deadline spikes.\n\n**Recommendation**: Proceed with Phase 1 implementation immediately, with a target of production deployment within 20 weeks.",
+    "tradeoffs": [],
+    "recommendations": [],
+    "confidence": 75,
+    "synthesizedBy": "judge-main"
+  },
+  "promptSources": {
+    "agents": [
+      {
+        "agentId": "agent-architect",
+        "role": "architect",
+        "source": "built-in"
+      },
+      {
+        "agentId": "agent-performance",
+        "role": "performance",
+        "source": "built-in"
+      }
+    ],
+    "judge": {
+      "id": "judge-main",
+      "source": "built-in",
+      "summarySource": "built-in"
+    }
+  }
+}