clawed 2.3.5__tar.gz → 2.3.8__tar.gz

{clawed-2.3.5 → clawed-2.3.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: clawed
-Version: 2.3.5
+Version: 2.3.8
 Summary: Claw-ED — personal AI teaching agent. Learns your voice, works while you sleep.
 Project-URL: Homepage, https://github.com/SirhanMacx/Claw-ED
 Project-URL: Documentation, https://github.com/SirhanMacx/Claw-ED#readme
@@ -18,14 +18,13 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Topic :: Education
 Requires-Python: >=3.10
-Requires-Dist: anthropic<1.0,>=0.40.0
 Requires-Dist: apscheduler<4.0,>=3.10.0
 Requires-Dist: fastapi<1.0,>=0.110.0
 Requires-Dist: httpx<1.0,>=0.25.0
 Requires-Dist: jinja2>=3.1.0
 Requires-Dist: json-repair>=0.30.0
+Requires-Dist: lxml>=4.9.0
 Requires-Dist: mcp>=1.0.0
-Requires-Dist: openai>=1.0.0
 Requires-Dist: pydantic<3.0,>=2.0.0
 Requires-Dist: pymupdf>=1.23.0
 Requires-Dist: python-docx>=1.0.0
@@ -41,6 +40,7 @@ Provides-Extra: all
 Requires-Dist: faster-whisper>=0.10.0; extra == 'all'
 Requires-Dist: keyring>=24.0.0; extra == 'all'
 Requires-Dist: onnxruntime>=1.16.0; extra == 'all'
+Requires-Dist: qrcode[pil]>=7.0; extra == 'all'
 Requires-Dist: textual>=0.56.0; extra == 'all'
 Requires-Dist: uvicorn[standard]>=0.27.0; extra == 'all'
 Provides-Extra: dev
@@ -59,6 +59,8 @@ Provides-Extra: memory
 Requires-Dist: onnxruntime>=1.16.0; extra == 'memory'
 Provides-Extra: pdf
 Requires-Dist: weasyprint>=60.0; extra == 'pdf'
+Provides-Extra: qr
+Requires-Dist: qrcode[pil]>=7.0; extra == 'qr'
 Provides-Extra: tui
 Requires-Dist: textual>=0.56.0; extra == 'tui'
 Provides-Extra: voice
@@ -80,23 +82,35 @@ Built on the OpenClaw agent framework. Open source. MIT license.
 
 ---
 
-## What's new in v2.3.5
+## What's new in v2.3.8
 
-**Master Content Track.** One LLM call generates a single `MasterContent` object. Three output documents are compiled mechanically from the same source of truth -- no content drift between what the teacher sees, what students hold, and what's on screen:
+**No more silent failures.** Every step of lesson generation now reports what happened — persona loading, material search, quality review, voice matching. If something fails, you'll know exactly what and why, with structured NLAH failure codes. Quality review runs automatically on every generated lesson and fails closed: if the review itself crashes, it reports failure instead of silently passing.
 
-1. **Teacher DOCX** — Full answer keys, scripted teacher language, guided notes with answers filled in, station answer keys, differentiation notes. Observation-ready.
-2. **Student DOCX** — Same structure, blanks instead of answers. Guided notes with fill-in lines, station directions without answer keys, exit ticket without expected responses. Print and hand out.
-3. **Slideshow PPTX** — Widescreen academic slides: title, vocabulary cards, instruction sections with images, primary source analysis, station overview, stimulus-based exit ticket.
+**Stricter quality gates.** Lessons now require at least 6 guided notes, 3 exit ticket questions, and 2 primary sources with actual text. Topic drift is caught automatically. Voice match scoring compares generated lessons against your teaching persona.
 
-**Zero silent failures.** All 11 generators use `safe_generate_json()` with automatic retry on validation errors. Post-generation validators catch empty outputs, topic drift, and delegation phrases. Quality review fails closed. CLI shows warnings, not raw tracebacks.
+**Safer onboarding.** Teacher names and subjects are validated and truncated. Invalid grade levels get a re-prompt instead of being silently accepted. Demo mode can be forced with `CLAWED_DEMO=1` for presentations.
 
-**Stimulus-based assessment.** Every question -- Do Now, guided notes, stations, exit ticket -- must be anchored to a stimulus (source text, data, diagram, scenario). Bare recall questions are banned at the prompt level.
+**Async cleanup.** Background ingestion no longer risks crashing on Python 3.12+ from nested event loops. The fix is shared across all async call sites.
 
-**Identity protection.** Onboarding only triggers on explicit `/setup`. Profile fields are validated and truncated. SOUL.md writes are audit-logged and capped at 500 chars.
+---
+
+## What's new in v2.3.7
+
+**Real images in every lesson.** Image specs are now required for every primary source and instruction section across all subjects. The LLM generates specific search queries ("Thomas Nast Boss Tweed political cartoon 1871") instead of leaving the field blank. Teacher images are found first using a three-stage progressive search (full query, individual keywords, subject fallback) with filename-weighted scoring across up to 150 candidates. External sources (Library of Congress, Wikimedia Commons, Unsplash) fill in the rest with subject-aware routing.
+
+**12 new file formats.** Your old `.doc`, `.ppt`, `.xls`, `.xlsx`, `.csv`, `.rtf`, `.html`, `.odt`, and `.odp` files are now parsed and indexed. Previously only 8 formats were supported -- teachers' archives spanning decades of file formats were 93% invisible to search. Now they're searchable.
+
+**Search actually works.** Three fixes to the search pipeline: cross-transport teacher ID fallback (files ingested via CLI now appear in Telegram searches), asset search errors are logged instead of silently swallowed, and the agent is explicitly instructed to surface results to you. Topic tags are auto-extracted from filenames and content for better matching.
+
+**Background file ingestion.** Send your files and keep chatting. The bot acknowledges immediately, processes everything in a background thread, sends progress updates ("Indexed 50/200 documents..."), and a summary when done. Max 3 concurrent ingestions, individual file failures don't abort the batch.
+
+**DEEP-tier model for lesson generation.** MasterContent routes to the DEEP tier. With a capable model (Claude Sonnet 4.6, GPT-4o), lesson quality improves dramatically.
+
+**Security hardened.** Path traversal protection on all file-reading tools. XSS escaping on the web dashboard. Thread-safe tool definitions. ZIP bomb protection. Debug info no longer leaked to users. Ingest paths restricted to home directory.
 
-**Your files are first-class.** Teacher materials are searched (AssetRegistry + CurriculumKB) before every generation -- not just in the bundle tool, but in standalone lesson and unit generation too.
+**50 MB lighter.** Removed unused `anthropic` and `openai` SDK dependencies. API key resolution unified across all code paths (env var + keyring + secrets file).
 
-**Parallel image pipeline.** All image specs from the MasterContent are fetched in parallel with configurable timeout and local caching.
+**Everything from v2.3.5 still applies:** Master Content Track, stimulus-based assessment, zero silent failures, parallel image pipeline, identity protection.
 
 ---
 
@@ -113,7 +127,7 @@ Everything runs on your own computer. Your files never leave your machine unless
 ## How it works
 
 ```
-Your files (PDFs, DOCX, PPTX, TXT)
+Your files (PDF, DOCX, PPTX, DOC, PPT, XLS, XLSX, CSV, RTF, HTML, ODT, TXT, and more)
         |
         v
 Claw-ED learns your teaching style

{clawed-2.3.5 → clawed-2.3.8}/README.md

@@ -13,23 +13,35 @@ Built on the OpenClaw agent framework. Open source. MIT license.
 
 ---
 
-## What's new in v2.3.5
+## What's new in v2.3.8
 
-**Master Content Track.** One LLM call generates a single `MasterContent` object. Three output documents are compiled mechanically from the same source of truth -- no content drift between what the teacher sees, what students hold, and what's on screen:
+**No more silent failures.** Every step of lesson generation now reports what happened — persona loading, material search, quality review, voice matching. If something fails, you'll know exactly what and why, with structured NLAH failure codes. Quality review runs automatically on every generated lesson and fails closed: if the review itself crashes, it reports failure instead of silently passing.
 
-1. **Teacher DOCX** — Full answer keys, scripted teacher language, guided notes with answers filled in, station answer keys, differentiation notes. Observation-ready.
-2. **Student DOCX** — Same structure, blanks instead of answers. Guided notes with fill-in lines, station directions without answer keys, exit ticket without expected responses. Print and hand out.
-3. **Slideshow PPTX** — Widescreen academic slides: title, vocabulary cards, instruction sections with images, primary source analysis, station overview, stimulus-based exit ticket.
+**Stricter quality gates.** Lessons now require at least 6 guided notes, 3 exit ticket questions, and 2 primary sources with actual text. Topic drift is caught automatically. Voice match scoring compares generated lessons against your teaching persona.
 
-**Zero silent failures.** All 11 generators use `safe_generate_json()` with automatic retry on validation errors. Post-generation validators catch empty outputs, topic drift, and delegation phrases. Quality review fails closed. CLI shows warnings, not raw tracebacks.
+**Safer onboarding.** Teacher names and subjects are validated and truncated. Invalid grade levels get a re-prompt instead of being silently accepted. Demo mode can be forced with `CLAWED_DEMO=1` for presentations.
 
-**Stimulus-based assessment.** Every question -- Do Now, guided notes, stations, exit ticket -- must be anchored to a stimulus (source text, data, diagram, scenario). Bare recall questions are banned at the prompt level.
+**Async cleanup.** Background ingestion no longer risks crashing on Python 3.12+ from nested event loops. The fix is shared across all async call sites.
 
-**Identity protection.** Onboarding only triggers on explicit `/setup`. Profile fields are validated and truncated. SOUL.md writes are audit-logged and capped at 500 chars.
+---
+
+## What's new in v2.3.7
+
+**Real images in every lesson.** Image specs are now required for every primary source and instruction section across all subjects. The LLM generates specific search queries ("Thomas Nast Boss Tweed political cartoon 1871") instead of leaving the field blank. Teacher images are found first using a three-stage progressive search (full query, individual keywords, subject fallback) with filename-weighted scoring across up to 150 candidates. External sources (Library of Congress, Wikimedia Commons, Unsplash) fill in the rest with subject-aware routing.
+
+**12 new file formats.** Your old `.doc`, `.ppt`, `.xls`, `.xlsx`, `.csv`, `.rtf`, `.html`, `.odt`, and `.odp` files are now parsed and indexed. Previously only 8 formats were supported -- teachers' archives spanning decades of file formats were 93% invisible to search. Now they're searchable.
+
+**Search actually works.** Three fixes to the search pipeline: cross-transport teacher ID fallback (files ingested via CLI now appear in Telegram searches), asset search errors are logged instead of silently swallowed, and the agent is explicitly instructed to surface results to you. Topic tags are auto-extracted from filenames and content for better matching.
+
+**Background file ingestion.** Send your files and keep chatting. The bot acknowledges immediately, processes everything in a background thread, sends progress updates ("Indexed 50/200 documents..."), and a summary when done. Max 3 concurrent ingestions, individual file failures don't abort the batch.
+
+**DEEP-tier model for lesson generation.** MasterContent routes to the DEEP tier. With a capable model (Claude Sonnet 4.6, GPT-4o), lesson quality improves dramatically.
+
+**Security hardened.** Path traversal protection on all file-reading tools. XSS escaping on the web dashboard. Thread-safe tool definitions. ZIP bomb protection. Debug info no longer leaked to users. Ingest paths restricted to home directory.
 
-**Your files are first-class.** Teacher materials are searched (AssetRegistry + CurriculumKB) before every generation -- not just in the bundle tool, but in standalone lesson and unit generation too.
+**50 MB lighter.** Removed unused `anthropic` and `openai` SDK dependencies. API key resolution unified across all code paths (env var + keyring + secrets file).
 
-**Parallel image pipeline.** All image specs from the MasterContent are fetched in parallel with configurable timeout and local caching.
+**Everything from v2.3.5 still applies:** Master Content Track, stimulus-based assessment, zero silent failures, parallel image pipeline, identity protection.
 
 ---
 
@@ -46,7 +58,7 @@ Everything runs on your own computer. Your files never leave your machine unless
 ## How it works
 
 ```
-Your files (PDFs, DOCX, PPTX, TXT)
+Your files (PDF, DOCX, PPTX, DOC, PPT, XLS, XLSX, CSV, RTF, HTML, ODT, TXT, and more)
         |
         v
 Claw-ED learns your teaching style

{clawed-2.3.5 → clawed-2.3.8}/clawed/__init__.py

@@ -17,7 +17,7 @@ if hasattr(sys.stderr, "reconfigure"):
     except Exception:
         pass
 
-__version__ = "2.3.5"
+__version__ = "2.3.8"
 __author__ = "Jon Maccarello & Claw-ED contributors"
 __description__ = "Personal AI teaching agent. Learns your voice, works while you sleep."
 

{clawed-2.3.5 → clawed-2.3.8}/clawed/_legacy_gateway.py

@@ -94,7 +94,8 @@ class Gateway:
         self._model_switch = ModelSwitchHandler()
 
     async def handle(self, message: str, teacher_id: str,
-                     files: list[Path] | None = None) -> GatewayResponse:
+                     files: list[Path] | None = None,
+                     progress_callback=None) -> GatewayResponse:
         """Process any message from any transport."""
         self._stats.messages_today += 1
         self.active_sessions[teacher_id] = {
@@ -112,31 +113,29 @@ class Gateway:
             if not has_config():
                 return await self._onboard.step(teacher_id, message)
 
-            return await self._dispatch(message, teacher_id, files)
+            return await self._dispatch(message, teacher_id, files,
+                                         progress_callback=progress_callback)
 
         except Exception as e:
             logger.warning("Gateway error: %s", e)
+            logger.debug("Gateway error detail: %s: %s", type(e).__name__, e)
             self._stats.errors_today += 1
             await self.emit("error", {"teacher_id": teacher_id, "message": str(e)})
 
             err = str(e).lower()
-            debug_hint = f"\n\n[Debug: {type(e).__name__}: {str(e)[:200]}]"
             if "401" in err or "unauthorized" in err or "api key" in err:
                 return GatewayResponse(
                     text="Your AI provider key doesn't seem to be working. "
                          "Run `clawed setup --reset` to reconfigure it."
-                         + debug_hint
                 )
             if "connection" in err or "connect" in err or "timeout" in err:
                 return GatewayResponse(
                     text="Can't connect to your AI provider right now. "
                          "Check your internet connection and try again."
-                         + debug_hint
                 )
             return GatewayResponse(
                 text="Something went wrong. Try again, or run "
                      "`clawed setup --reset` to reconfigure."
-                     + debug_hint
             )
 
     async def handle_callback(self, callback_data: str, teacher_id: str) -> GatewayResponse:
@@ -177,13 +176,16 @@ class Gateway:
         }
 
     async def _dispatch(self, message: str, teacher_id: str,
-                        files: list[Path] | None = None) -> GatewayResponse:
+                        files: list[Path] | None = None,
+                        progress_callback=None) -> GatewayResponse:
         """Route a message to the appropriate handler based on intent."""
         if files:
-            return await self._ingest.handle(teacher_id, files)
+            return await self._ingest.handle(teacher_id, files,
+                                             progress_callback=progress_callback)
 
         if self._looks_like_path(message):
-            return await self._ingest.handle(teacher_id, path=message.strip())
+            return await self._ingest.handle(teacher_id, path=message.strip(),
+                                             progress_callback=progress_callback)
 
         # NOTE: parse_intent() is keyword/regex-based (zero cost).
         # When upgraded to LLM-based detection, use:

{clawed-2.3.5 → clawed-2.3.8}/clawed/agent_core/context.py

@@ -1,12 +1,15 @@
 """Core data types for the agent system."""
 from __future__ import annotations
 
+import logging
 from dataclasses import dataclass, field
 from pathlib import Path
-from typing import Any
+from typing import Any, Callable, Optional
 
 from clawed.models import AppConfig
 
+logger = logging.getLogger(__name__)
+
 
 @dataclass
 class AgentContext:
@@ -19,6 +22,15 @@ class AgentContext:
     session_history: list[dict[str, Any]]
     improvement_context: str
     agent_name: str = "Claw-ED"
+    progress_callback: Optional[Callable[[str], None]] = None
+
+    def notify_progress(self, message: str) -> None:
+        """Send a progress update to the user if a callback is registered."""
+        if self.progress_callback:
+            try:
+                self.progress_callback(message)
+            except Exception as e:
+                logger.debug("Progress notification failed: %s", e)
 
 
 @dataclass

{clawed-2.3.5 → clawed-2.3.8}/clawed/agent_core/core.py

@@ -10,6 +10,7 @@ from __future__ import annotations
 import asyncio
 import json
 import logging
+import threading
 import time
 from datetime import datetime
 from pathlib import Path
@@ -28,6 +29,9 @@ from clawed.models import AppConfig
 logger = logging.getLogger(__name__)
 
 
+_tool_lock = threading.Lock()
+
+
 class _LLMClientAdapter:
     """Adapts the existing clawed.agent module's LLM calling to LLMInterface.
 
@@ -45,28 +49,24 @@ class _LLMClientAdapter:
         tools: list[dict[str, Any]] | None = None,
         system: str = "",
     ) -> dict[str, Any]:
-        # WARNING: This monkey-patches a module-level variable, which is NOT
-        # safe under concurrent requests. For v1.0 (hosted/multi-teacher),
-        # refactor the legacy agent functions to accept tool definitions as
-        # a parameter instead of reading from the module global.
-        #
         # The legacy agent functions operate on the global TOOL_DEFINITIONS.
-        # We temporarily monkey-patch them so the registry schemas are used
-        # instead. Since these functions read TOOL_DEFINITIONS at call time,
-        # we swap the module-level list.
+        # We temporarily swap them under a lock so concurrent requests don't
+        # clobber each other's tool definitions.
         import clawed.agent as _agent_mod
         from clawed.agent import _call_with_native_tools, _call_with_ollama_tools
         from clawed.models import LLMProvider
 
-        original_defs = _agent_mod.TOOL_DEFINITIONS
-        _agent_mod.TOOL_DEFINITIONS = tools or []
+        with _tool_lock:
+            original_defs = _agent_mod.TOOL_DEFINITIONS
+            _agent_mod.TOOL_DEFINITIONS = tools or []
         try:
             if self._config.provider in (LLMProvider.ANTHROPIC, LLMProvider.OPENAI):
                 return await _call_with_native_tools(messages, system, self._config)
             else:
                 return await _call_with_ollama_tools(messages, system, self._config)
         finally:
-            _agent_mod.TOOL_DEFINITIONS = original_defs
+            with _tool_lock:
+                _agent_mod.TOOL_DEFINITIONS = original_defs
 
 
 class Gateway:
@@ -116,6 +116,7 @@ class Gateway:
         message: str,
         teacher_id: str,
         files: list[Path] | None = None,
+        progress_callback: Any = None,
     ) -> GatewayResponse:
         """Process any message from any transport."""
         self._stats.messages_today += 1
@@ -128,9 +129,11 @@ class Gateway:
         })
 
         try:
-            # 1. Files → ingest (deterministic, no LLM)
+            # 1. Files → ingest (deterministic, no LLM, runs in background)
             if files:
-                return await self._ingest.handle(teacher_id, files)
+                return await self._ingest.handle(
+                    teacher_id, files, progress_callback=progress_callback
+                )
 
             # 2. Onboarding state machine (deterministic, no LLM)
             if self._onboard.is_onboarding(teacher_id):
@@ -147,38 +150,32 @@ class Gateway:
                 )
 
             # 4. Natural-language → agent loop
-            return await self._agent_loop(message, teacher_id)
+            return await self._agent_loop(message, teacher_id, progress_callback=progress_callback)
 
         except Exception as e:
-            logger.debug("Gateway error: %s", e)
+            logger.error("Agent error for teacher %s: %s", teacher_id, e, exc_info=True)
             self._stats.errors_today += 1
             await self.emit("error", {"teacher_id": teacher_id, "message": str(e)})
 
-            # Teacher-friendly error messages (include debug info for troubleshooting)
+            # Teacher-friendly error messages (no internal details exposed)
             err = str(e).lower()
-            debug_hint = f"\n\n[Debug: {type(e).__name__}: {str(e)[:200]}]"
             if "401" in err or "unauthorized" in err or "api key" in err:
                 return GatewayResponse(
                     text="Your AI provider key doesn't seem to be working. "
                          "Run `clawed setup --reset` to reconfigure it."
-                         + debug_hint
                 )
             if "connection" in err or "connect" in err or "timeout" in err:
                 return GatewayResponse(
                     text="Can't connect to your AI provider right now. "
                          "Check your internet connection and try again."
-                         + debug_hint
                 )
             if "rate limit" in err or "429" in err:
                 return GatewayResponse(
                     text="Your AI provider is temporarily overloaded. "
                          "Wait a minute and try again."
-                         + debug_hint
                 )
             return GatewayResponse(
-                text="Something went wrong. Try again, or run "
-                     "`clawed setup --reset` to reconfigure."
-                     + debug_hint
+                text="Something went wrong. Please try again."
             )
 
     async def handle_callback(self, callback_data: str, teacher_id: str) -> GatewayResponse:
@@ -299,7 +296,7 @@ class Gateway:
     # Agent loop — the core reasoning path
     # ------------------------------------------------------------------
 
-    async def _agent_loop(self, message: str, teacher_id: str) -> GatewayResponse:
+    async def _agent_loop(self, message: str, teacher_id: str, progress_callback: Any = None) -> GatewayResponse:
         """Load context, build prompt, and run the agent tool-use loop."""
         # 1. Load teacher context from canonical sources
         teacher_profile = self._load_teacher_profile()
@@ -342,7 +339,9 @@ class Gateway:
         # 2a. Load SOUL.md if available
         soul_context = ""
         try:
-            soul_path = Path.home() / ".eduagent" / "workspace" / "SOUL.md"
+            import os
+            data_dir = os.environ.get("EDUAGENT_DATA_DIR", str(Path.home() / ".eduagent"))
+            soul_path = Path(data_dir) / "workspace" / "SOUL.md"
             if soul_path.exists():
                 soul_context = soul_path.read_text(encoding="utf-8")[:2000]
         except Exception:
@@ -393,6 +392,7 @@ class Gateway:
             session_history=session_history,
             improvement_context=memory_ctx["improvement_context"],
             agent_name=agent_name,
+            progress_callback=progress_callback,
         )
 
         # 4. Get or create LLM adapter

{clawed-2.3.5 → clawed-2.3.8}/clawed/agent_core/memory/curriculum_kb.py

@@ -138,10 +138,13 @@ class CurriculumKB:
 
         with sqlite3.connect(self._db_path) as conn:
             conn.row_factory = sqlite3.Row
+            # Fetch up to 5000 chunks for scoring. This trades higher memory
+            # for better recall — teachers with large file collections may have
+            # thousands of chunks, and a 2000 cap was silently dropping results.
             rows = conn.execute(
                 "SELECT doc_title, source_path, chunk_text, embedding, metadata, created_at "
                 "FROM chunks WHERE teacher_id = ? "
-                "LIMIT 2000",
+                "LIMIT 5000",
                 (teacher_id,),
             ).fetchall()
 
@@ -166,6 +169,47 @@ class CurriculumKB:
         scored.sort(key=lambda x: x["similarity"], reverse=True)
         return scored[:top_k]
 
+    def search_all_teachers(
+        self,
+        query: str,
+        top_k: int = 10,
+    ) -> list[dict[str, Any]]:
+        """Fallback search across ALL teachers when teacher_id doesn't match.
+
+        This handles cross-transport mismatches (e.g. files ingested via
+        Telegram numeric ID, searched via CLI 'local-teacher').
+        """
+        query_embedding = self._embedder.embed(query)
+
+        with sqlite3.connect(self._db_path) as conn:
+            conn.row_factory = sqlite3.Row
+            # Search all chunks regardless of teacher_id — capped for safety
+            rows = conn.execute(
+                "SELECT doc_title, source_path, chunk_text, embedding, metadata, created_at "
+                "FROM chunks LIMIT 5000",
+            ).fetchall()
+
+        if not rows:
+            return []
+
+        scored = []
+        for row in rows:
+            stored_embedding = json.loads(row["embedding"])
+            sim = self._embedder.cosine_similarity(query_embedding, stored_embedding)
+            scored.append({
+                "doc_title": row["doc_title"],
+                "source_path": row["source_path"],
+                "chunk_text": row["chunk_text"],
+                "metadata": json.loads(row["metadata"]),
+                "created_at": row["created_at"],
+                "similarity": sim,
+            })
+
+        scored = [s for s in scored if s["similarity"] > 0.05]
+        logger.debug("KB fallback search '%s': %d chunks scored, %d above threshold", query, len(rows), len(scored))
+        scored.sort(key=lambda x: x["similarity"], reverse=True)
+        return scored[:top_k]
+
     def stats(self, teacher_id: str) -> dict[str, Any]:
         """Return stats about the teacher's curriculum knowledge base."""
         with sqlite3.connect(self._db_path) as conn:

{clawed-2.3.5 → clawed-2.3.8}/clawed/agent_core/memory/embeddings.py

@@ -60,7 +60,13 @@ class OllamaEmbedder:
 
 
 class TFIDFEmbedder:
-    """TF-IDF with bigrams — no dependencies, always available."""
+    """TF-IDF with bigrams — no dependencies, always available.
+
+    Vocabulary is capped at MAX_VOCAB tokens to bound vector dimensionality
+    and prevent unbounded memory growth during large ingestion runs.
+    """
+
+    MAX_VOCAB = 10_000
 
     def __init__(self) -> None:
         self._vocab: dict[str, int] = {}
@@ -78,12 +84,15 @@ class TFIDFEmbedder:
     def embed(self, text: str) -> list[float]:
         tokens = self._tokenize(text)
         for t in tokens:
-            if t not in self._vocab:
+            if t not in self._vocab and self._next_idx < self.MAX_VOCAB:
                 self._vocab[t] = self._next_idx
                 self._next_idx += 1
-        vec = [0.0] * len(self._vocab)
+        dim = min(len(self._vocab), self.MAX_VOCAB)
+        vec = [0.0] * dim
         for t in tokens:
-            vec[self._vocab[t]] += 1.0
+            idx = self._vocab.get(t)
+            if idx is not None and idx < dim:
+                vec[idx] += 1.0
         norm = math.sqrt(sum(x * x for x in vec)) or 1.0
         return [x / norm for x in vec]
 

{clawed-2.3.5 → clawed-2.3.8}/clawed/agent_core/prompt.py

@@ -96,6 +96,9 @@ def build_system_prompt(
         "The teacher has uploaded materials — if you skip this step, you will "
         "generate generic content instead of building on their prior work. "
         "Tell the teacher what you found before generating.\n"
+        "   IMPORTANT: If search_my_materials returns results, you MUST list them "
+        "for the teacher. NEVER say 'I didn't find anything' if the tool returned "
+        "materials. Always surface what was found, even if it's not an exact match.\n"
         "3. Generate complete packages (lesson plan + student handout + slideshow) "
         "using generate_lesson_bundle\n"
         "4. Never ask 'want me to create materials?' -- just create them\n"

{clawed-2.3.5 → clawed-2.3.8}/clawed/agent_core/tools/generate_lesson_bundle.py

@@ -6,6 +6,7 @@ from pathlib import Path
 from typing import Any
 
 from clawed.agent_core.context import AgentContext, ToolResult
+from clawed.failure_codes import FailureCode
 
 logger = logging.getLogger(__name__)
 
@@ -83,14 +84,24 @@ class GenerateLessonBundleTool:
         activity_type = params.get("activity_type", "general")
         include_images = params.get("include_images", True)
 
+        # ── Notify user we're starting ────────────────────────────────
+        context.notify_progress(
+            f"Working on your lesson materials for \"{topic}\" now — "
+            f"this usually takes 2-4 minutes. I'll send everything when it's ready!"
+        )
+
+        from clawed.generation_report import GenerationReport
+        report = GenerationReport()
+
         # ── Load config & persona from context ───────────────────────
         config = context.config
         persona = TeacherPersona()
         if context.persona:
             try:
                 persona = TeacherPersona(**context.persona)
-            except Exception:
-                pass
+            except Exception as e:
+                logger.warning("NLAH_FAILURE=%s: %s", FailureCode.PERSONA_PARSE_ERROR, e)
+                report.warnings.append(f"[{FailureCode.PERSONA_PARSE_ERROR}] Could not parse persona: {e}")
 
         # ── Load state standards if teacher profile has a state ───────
         state = ""
@@ -121,7 +132,8 @@ class GenerateLessonBundleTool:
                     len(assets), len(yt_links), topic,
                 )
         except Exception as e:
-            logger.debug("Asset search failed: %s", e)
+            logger.warning("NLAH_FAILURE=%s: %s", FailureCode.ASSET_SEARCH_FAILED, e)
+            report.warnings.append(f"[{FailureCode.ASSET_SEARCH_FAILED}] Asset search failed: {e}")
 
         # KB chunk-level search (text excerpts)
         try:
@@ -156,7 +168,8 @@ class GenerateLessonBundleTool:
                         )
                     logger.info("KB search found %d relevant chunks for '%s'", len(kb_parts), topic)
         except Exception as e:
-            logger.debug("KB search failed: %s", e)
+            logger.warning("NLAH_FAILURE=%s: %s", FailureCode.KB_SEARCH_FAILED, e)
+            report.warnings.append(f"[{FailureCode.KB_SEARCH_FAILED}] KB search failed: {e}")
 
         # ── Build a UnitPlan with standards ──────────────────────────
         description = f"Introduction to {topic}"
@@ -199,14 +212,12 @@ class GenerateLessonBundleTool:
                 teacher_materials=kb_prompt_section,
             )
         except Exception as e:
-            return ToolResult(text=f"Failed to generate lesson: {e}")
+            logger.error("NLAH_FAILURE=%s: %s", FailureCode.API_FAILURE, e)
+            return ToolResult(text=f"[{FailureCode.API_FAILURE}] Failed to generate lesson: {type(e).__name__}")
 
         # ── Validate ──────────────────────────────────────────────────
-        from clawed.generation_report import GenerationReport
         from clawed.validation import check_self_contained, validate_alignment, validate_master_content
 
-        report = GenerationReport()
-
         mc_errors = validate_master_content(master, topic)
         for err in mc_errors:
             report.warnings.append(err)
@@ -277,6 +288,45 @@ class GenerateLessonBundleTool:
             logger.error("Slides compile failed: %s", e)
             errors.append(f"Slideshow PPTX failed: {e}")
 
+        # ── Quality review (NLAH Stage 4 — non-blocking) ────────────
+        try:
+            from clawed.llm import LLMClient
+            from clawed.quality import score_voice_match
+
+            llm = LLMClient(config=config)
+            master_json = master.model_dump_json(indent=2)[:3000]
+            review = await llm.review_lesson_package(
+                lesson_json=master_json,
+                standards_present=bool(standards_list),
+                has_handout=len(generated_files) >= 2,
+                has_slideshow=len(generated_files) >= 3,
+            )
+            report.quality_review_passed = review.get("passed", False)
+            report.quality_review_issues = review.get("issues", [])
+            if not report.quality_review_passed:
+                for issue in report.quality_review_issues:
+                    report.warnings.append(f"[REVIEW] {issue}")
+                logger.info("Quality review: FAILED — %d issues", len(report.quality_review_issues))
+            else:
+                logger.info("Quality review: PASSED")
+
+            # Voice match scoring
+            persona_ctx = persona.to_prompt_context() if persona else ""
+            all_text = " ".join(s.content for s in master.direct_instruction)
+            voice_score = await score_voice_match(all_text, persona_ctx, llm)
+            report.voice_check_passed = voice_score >= 3.0
+            if voice_score < 3.0:
+                report.warnings.append(
+                    f"[{FailureCode.VOICE_MISMATCH}] Voice match score: {voice_score:.1f}/5.0"
+                )
+                logger.warning("Voice match: %.1f/5.0 — below threshold", voice_score)
+            else:
+                logger.info("Voice match: %.1f/5.0", voice_score)
+        except Exception as e:
+            logger.warning("Quality review/voice check failed: %s", e)
+            report.quality_review_passed = False
+            report.quality_review_issues = [f"Review failed: {type(e).__name__}"]
+
         # ── Build response ─────────────────────────────────────────────
         lines = []