pystdoc 0.7.2__tar.gz → 0.8.0__tar.gz

{pystdoc-0.7.2/src/pystdoc.egg-info → pystdoc-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pystdoc
-Version: 0.7.2
+Version: 0.8.0
 Summary: Python Structural & Topological Documentation Engine (pystdoc): High-precision, LLM-powered hierarchical codebase and architecture documentation generator.
 Author: tab4moji
 License: MIT
@@ -78,21 +78,25 @@ your_project/
    - Evaluates call graphs in $O(V+E)$ linear time.
    - Automatically breaks cyclic mutual recursions and organizes code symbols into dependency-safe execution levels.
    - Level-by-level parallel LLM execution guarantees context-rich bottom-up summaries without race conditions.
-2. **3-in-1 Unified Documentation Pipeline**:
+2. **High-Speed Incremental Sync & Semantic Change Propagation (v0.8.0)**:
+   - **Selective Bottom-Up Propagation**: Detects AST and logic-level code differences. If regenerated specifications remain semantically unchanged (evaluated by deterministic Jaccard / token-overlap metrics), change propagation terminates immediately without triggering cascading LLM re-analysis across upper modules.
+   - **Content-Defined Map-Reduce Caching**: Intermediate summaries in architecture synthesis (`designgen`) are chunked using stable hash boundaries and cached per group in SQLite, ensuring that only groups containing edited functions are re-summarized.
+   - **Architectural Entry-Point Filtering**: For system execution modeling (`execution_model.md`), key architectural functions (entry points, CLI routers, and core dispatchers) are automatically filtered and passed directly into the LLM context, bypassing heavy multi-step Map-Reduce loops and speeding up generation by up to 90%+.
+3. **3-in-1 Unified Documentation Pipeline**:
    - `docgen`: Bottom-up & top-down symbol-level documentation with SHA-256 and SQLite caching (`.pystdoc/documents/`).
    - `designgen`: Map-Reduce architectural synthesis (`.pystdoc/design/`).
    - `reportgen` / `pystdoc`: Executive summary README (`.pystdoc/README.md`) answering *"What does this project actually do?"*
-3. **Multi-Language Codebase Deep Understanding (C/C++, Python, Java, Kotlin, Shell)**:
+4. **Multi-Language Codebase Deep Understanding (C/C++, Python, Java, Kotlin, Shell)**:
    - **C/C++**: `compile_commands.json` integration, include path resolution, and macro expansion via `libclang`.
    - **Java / Kotlin**: AST parsing via `javalang` & `kopyt` with class hierarchy, companion objects, and data classes.
    - **Python & Shell**: AST visitor and generic regex parser with import/call extraction.
    - **Fully Qualified Domain Names (FQDN)**: Disambiguates identical symbol names across large monorepos.
-4. **Feature Location & UI Component Pinpointing (`locate` / `pystdoc_locate_feature`)**:
+5. **Feature Location & UI Component Pinpointing (`locate` / `pystdoc_locate_feature`)**:
    - Pinpoint exact files, functions, UI elements, and line ranges (`Lines X-Y`) from high-level natural language queries (e.g., *"where is the debug notification button"*, *"add auth header"*) without running brute-force `Glob`/`Grep` across the entire project.
    - Built-in multi-lingual token scoring with automatic boosting for UI elements (`@Composable`, `Button`, `Dialog`, `Activity`, `Modifier`, etc.).
-5. **Impact & Inbound/Outbound Dependency Analysis (`impact` / `pystdoc_trace_impact`)**:
+6. **Impact & Inbound/Outbound Dependency Analysis (`impact` / `pystdoc_trace_impact`)**:
    - Instant 360-degree dependency tracing: inspect where a symbol is called from (*Inbound Callers / References*) and what it uses (*Outbound Dependencies*) before making breaking code changes or deletions.
-6. **Standard LLM Options & Multi-Language Support**:
+7. **Standard LLM Options & Multi-Language Support**:
    - Works with **Ollama, LiteRT-LM, vLLM, and OpenAI API**.
    - Supports `--host`, `--model`, `--token` / `--api-key`, `--context-size`, and `--language` (defaults to terminal language, e.g. `Japanese`, `English`, `日本語`).
 

{pystdoc-0.7.2 → pystdoc-0.8.0}/README.md

@@ -49,21 +49,25 @@ your_project/
    - Evaluates call graphs in $O(V+E)$ linear time.
    - Automatically breaks cyclic mutual recursions and organizes code symbols into dependency-safe execution levels.
    - Level-by-level parallel LLM execution guarantees context-rich bottom-up summaries without race conditions.
-2. **3-in-1 Unified Documentation Pipeline**:
+2. **High-Speed Incremental Sync & Semantic Change Propagation (v0.8.0)**:
+   - **Selective Bottom-Up Propagation**: Detects AST and logic-level code differences. If regenerated specifications remain semantically unchanged (evaluated by deterministic Jaccard / token-overlap metrics), change propagation terminates immediately without triggering cascading LLM re-analysis across upper modules.
+   - **Content-Defined Map-Reduce Caching**: Intermediate summaries in architecture synthesis (`designgen`) are chunked using stable hash boundaries and cached per group in SQLite, ensuring that only groups containing edited functions are re-summarized.
+   - **Architectural Entry-Point Filtering**: For system execution modeling (`execution_model.md`), key architectural functions (entry points, CLI routers, and core dispatchers) are automatically filtered and passed directly into the LLM context, bypassing heavy multi-step Map-Reduce loops and speeding up generation by up to 90%+.
+3. **3-in-1 Unified Documentation Pipeline**:
    - `docgen`: Bottom-up & top-down symbol-level documentation with SHA-256 and SQLite caching (`.pystdoc/documents/`).
    - `designgen`: Map-Reduce architectural synthesis (`.pystdoc/design/`).
    - `reportgen` / `pystdoc`: Executive summary README (`.pystdoc/README.md`) answering *"What does this project actually do?"*
-3. **Multi-Language Codebase Deep Understanding (C/C++, Python, Java, Kotlin, Shell)**:
+4. **Multi-Language Codebase Deep Understanding (C/C++, Python, Java, Kotlin, Shell)**:
    - **C/C++**: `compile_commands.json` integration, include path resolution, and macro expansion via `libclang`.
    - **Java / Kotlin**: AST parsing via `javalang` & `kopyt` with class hierarchy, companion objects, and data classes.
    - **Python & Shell**: AST visitor and generic regex parser with import/call extraction.
    - **Fully Qualified Domain Names (FQDN)**: Disambiguates identical symbol names across large monorepos.
-4. **Feature Location & UI Component Pinpointing (`locate` / `pystdoc_locate_feature`)**:
+5. **Feature Location & UI Component Pinpointing (`locate` / `pystdoc_locate_feature`)**:
    - Pinpoint exact files, functions, UI elements, and line ranges (`Lines X-Y`) from high-level natural language queries (e.g., *"where is the debug notification button"*, *"add auth header"*) without running brute-force `Glob`/`Grep` across the entire project.
    - Built-in multi-lingual token scoring with automatic boosting for UI elements (`@Composable`, `Button`, `Dialog`, `Activity`, `Modifier`, etc.).
-5. **Impact & Inbound/Outbound Dependency Analysis (`impact` / `pystdoc_trace_impact`)**:
+6. **Impact & Inbound/Outbound Dependency Analysis (`impact` / `pystdoc_trace_impact`)**:
    - Instant 360-degree dependency tracing: inspect where a symbol is called from (*Inbound Callers / References*) and what it uses (*Outbound Dependencies*) before making breaking code changes or deletions.
-6. **Standard LLM Options & Multi-Language Support**:
+7. **Standard LLM Options & Multi-Language Support**:
    - Works with **Ollama, LiteRT-LM, vLLM, and OpenAI API**.
    - Supports `--host`, `--model`, `--token` / `--api-key`, `--context-size`, and `--language` (defaults to terminal language, e.g. `Japanese`, `English`, `日本語`).
 

{pystdoc-0.7.2 → pystdoc-0.8.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "pystdoc"
-version = "0.7.2"
+version = "0.8.0"
 description = "Python Structural & Topological Documentation Engine (pystdoc): High-precision, LLM-powered hierarchical codebase and architecture documentation generator."
 readme = "README.md"
 authors = [
@@ -51,3 +51,45 @@ filterwarnings = [
     "ignore::ResourceWarning",
     "ignore::pytest.PytestUnraisableExceptionWarning",
 ]
+
+[tool.pylint.main]
+# Third-party libs without static member info (javalang) / libclang.
+extension-pkg-allow-list = []
+
+[tool.pylint.messages_control]
+# Intentional design choices of this code base:
+#  - broad-exception-caught: parsers must never crash a whole sync run
+#  - import-outside-toplevel: optional/heavy deps are imported lazily
+#  - complexity/size metrics, duplicate-code, cyclic-import (adapter registry)
+#  - invalid-name: ast.NodeVisitor visit_* methods and module singletons
+disable = [
+    "broad-exception-caught",
+    "import-outside-toplevel",
+    "too-many-locals",
+    "too-many-branches",
+    "too-many-statements",
+    "too-many-arguments",
+    "too-many-positional-arguments",
+    "too-many-return-statements",
+    "too-many-instance-attributes",
+    "too-many-nested-blocks",
+    "too-many-lines",
+    "too-few-public-methods",
+    "duplicate-code",
+    "cyclic-import",
+    "no-else-return",
+    "no-else-break",
+    "invalid-name",
+    "disallowed-name",
+    "global-statement",
+    "protected-access",
+    "unnecessary-ellipsis",
+    "missing-function-docstring",
+    "missing-class-docstring",
+    "missing-module-docstring",
+    "consider-using-with",
+    "attribute-defined-outside-init",
+    "cell-var-from-loop",
+    "unused-argument",
+    "no-member",
+]

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/__init__.py

@@ -1,3 +1,3 @@
 """pystdoc: Python Structural & Topological Documentation Engine (pystdoc)."""
 
-__version__ = "0.7.2"
+__version__ = "0.8.0"

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/db.py

@@ -6,6 +6,7 @@ import threading
 from pathlib import Path
 from typing import Any, Dict, List, Optional, Tuple
 
+from pystdoc.similarity import is_semantically_same
 from pystdoc.symbols import Symbol
 
 
@@ -71,9 +72,16 @@ class DocgenDB:
                         target_key TEXT PRIMARY KEY,
                         input_hash TEXT NOT NULL,
                         content TEXT NOT NULL,
+                        input_text TEXT,
                         updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
                     );
                 """)
+                try:
+                    self.conn.execute(
+                        "ALTER TABLE design_cache ADD COLUMN input_text TEXT;"
+                    )
+                except Exception:
+                    pass
 
                 self.conn.execute("""
                     CREATE TABLE IF NOT EXISTS symbols_metadata (
@@ -231,41 +239,63 @@ class DocgenDB:
             return None
 
     def load_design_cache(
-        self, target_key: str, current_input_hash: str
+        self,
+        target_key: str,
+        current_input_hash: str,
+        input_text: Optional[str] = None,
+        similarity_threshold: float = 0.85,
     ) -> Optional[str]:
-        """Load cached design document if input hash matches."""
+        """Load cached design document if the input is unchanged.
+
+        Exact hash match is a hit. If ``input_text`` is given, a stored
+        input that is semantically similar (>= threshold) is also a hit,
+        so that wording-only changes do not cascade upward.
+        """
         with self.lock:
             cur = self.conn.cursor()
             cur.execute(
                 """
-                SELECT content FROM design_cache
-                WHERE target_key = ? AND input_hash = ?
+                SELECT content, input_hash, input_text FROM design_cache
+                WHERE target_key = ?
             """,
-                (target_key, current_input_hash),
+                (target_key,),
             )
             row = cur.fetchone()
-            if row:
-                return row["content"]
+        if not row:
             return None
+        if row["input_hash"] == current_input_hash:
+            return row["content"]
+        if input_text is not None and row["input_text"]:
+            if is_semantically_same(
+                row["input_text"], input_text, similarity_threshold
+            ):
+                return row["content"]
+        return None
 
     def save_design_cache(
-        self, target_key: str, input_hash: str, content: str
+        self,
+        target_key: str,
+        input_hash: str,
+        content: str,
+        input_text: Optional[str] = None,
     ) -> None:
-        """Save design doc content and input hash with immediate commit."""
+        """Save design doc content, input hash and text (immediate commit)."""
         with self.lock:
             with self.conn:
                 self.conn.execute(
                     """
                     INSERT INTO design_cache (
-                        target_key, input_hash, content, updated_at
+                        target_key, input_hash, content, input_text,
+                        updated_at
                     )
-                    VALUES (?, ?, ?, CURRENT_TIMESTAMP)
+                    VALUES (?, ?, ?, ?, CURRENT_TIMESTAMP)
                     ON CONFLICT(target_key) DO UPDATE SET
                         input_hash=excluded.input_hash,
                         content=excluded.content,
+                        input_text=excluded.input_text,
                         updated_at=CURRENT_TIMESTAMP;
                 """,
-                    (target_key, input_hash, content),
+                    (target_key, input_hash, content, input_text),
                 )
 
     def save_symbol_metadata(

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/design_engine.py

@@ -169,6 +169,43 @@ def chunk_list_by_size(
     return chunks
 
 
+def chunk_list_stable(
+    items: List[str], max_items: int = 15, max_chars: int = 3500
+) -> List[List[str]]:
+    """Split items into chunks whose boundaries are content-defined.
+
+    A boundary is placed after an item whose hash hits a target value, so
+    inserting/removing/editing one item only changes the chunk(s) around
+    it; the rest of the chunks stay byte-identical (cache friendly).
+    Size limits are still enforced as hard boundaries.
+    """
+    divisor = max(2, max_items // 2)
+    chunks: List[List[str]] = []
+    current_chunk: List[str] = []
+    current_length = 0
+
+    for item in items:
+        item_len = len(item)
+        if current_chunk and (
+            len(current_chunk) >= max_items
+            or current_length + item_len > max_chars
+        ):
+            chunks.append(current_chunk)
+            current_chunk = []
+            current_length = 0
+        current_chunk.append(item)
+        current_length += item_len
+        digest = hashlib.sha1(item.encode("utf-8")).hexdigest()
+        if int(digest[:8], 16) % divisor == 0:
+            chunks.append(current_chunk)
+            current_chunk = []
+            current_length = 0
+
+    if current_chunk:
+        chunks.append(current_chunk)
+    return chunks
+
+
 def hierarchical_reduce_summaries(
     items: List[str],
     category_title: str,
@@ -178,8 +215,15 @@ def hierarchical_reduce_summaries(
     max_items_per_chunk: int = 15,
     max_chars_per_chunk: int = 3500,
     depth: int = 0,
+    db: Optional[DocgenDB] = None,
+    force: bool = False,
 ) -> str:
-    """Recursively reduce descriptions into a concise summary."""
+    """Recursively reduce descriptions into a concise summary.
+
+    When ``db`` is given, each group summary is cached by the hash of its
+    input text, and groups are formed by content-defined chunking. Thus
+    only the groups that contain changed items are sent to the LLM again.
+    """
     if not items:
         return "(None)"
 
@@ -189,15 +233,25 @@ def hierarchical_reduce_summaries(
     ) or depth > 5:
         return "\n".join(items)
 
-    chunks = chunk_list_by_size(
-        items, max_items=max_items_per_chunk, max_chars=max_chars_per_chunk
-    )
+    if db is not None:
+        chunks = chunk_list_stable(
+            sorted(items),
+            max_items=max_items_per_chunk,
+            max_chars=max_chars_per_chunk,
+        )
+    else:
+        chunks = chunk_list_by_size(
+            items,
+            max_items=max_items_per_chunk,
+            max_chars=max_chars_per_chunk,
+        )
     if len(chunks) <= 1 and depth > 0:
         return "\n".join(items)
 
     intermediate_summaries: List[str] = []
 
     for idx, chunk in enumerate(chunks, 1):
+        InterruptionState.check_interrupted()
         chunk_text = "\n".join(chunk)
         if not llm_client:
             summary_item = (
@@ -207,6 +261,26 @@ def hierarchical_reduce_summaries(
             intermediate_summaries.append(summary_item)
             continue
 
+        group_cache_key = ""
+        group_hash = ""
+        if db is not None:
+            group_hash = hashlib.sha256(
+                f"{language}\n{category_title}\n{chunk_text}".encode("utf-8")
+            ).hexdigest()
+            group_cache_key = (
+                f"reduce::{language}::{category_title}::{group_hash}"
+            )
+            if not force:
+                cached_summary = db.load_design_cache(
+                    group_cache_key, group_hash
+                )
+                if cached_summary:
+                    # Header without index keeps upper levels stable.
+                    intermediate_summaries.append(
+                        f"#### Group Summary\n{cached_summary.strip()}"
+                    )
+                    continue
+
         grp_info = f"(Group {idx}/{len(chunks)})"
         prompt = f"""Please analyze {category_title} {grp_info}
 and create a concise, factual summary (200-400 words) in {language}.
@@ -231,9 +305,17 @@ Output as Markdown bullet points in {language}:"""
 
         try:
             summary = llm_client.chat_completion(messages)
-            intermediate_summaries.append(
-                f"#### Group {idx} Summary\n{summary.strip()}"
-            )
+            if db is not None:
+                db.save_design_cache(
+                    group_cache_key, group_hash, summary.strip()
+                )
+                intermediate_summaries.append(
+                    f"#### Group Summary\n{summary.strip()}"
+                )
+            else:
+                intermediate_summaries.append(
+                    f"#### Group {idx} Summary\n{summary.strip()}"
+                )
         except Exception as e:
             if not allow_fallback:
                 raise LLMError(
@@ -257,6 +339,8 @@ Output as Markdown bullet points in {language}:"""
             max_items_per_chunk=max_items_per_chunk,
             max_chars_per_chunk=max_chars_per_chunk,
             depth=depth + 1,
+            db=db,
+            force=force,
         )
 
     return "\n\n".join(intermediate_summaries)
@@ -292,7 +376,9 @@ def generate_data_models_doc(
     out_file = target_dir / ".pystdoc" / "design" / "data_models.md"
 
     if db and not force and out_file.exists():
-        cached_content = db.load_design_cache(cache_key, input_hash)
+        cached_content = db.load_design_cache(
+            cache_key, input_hash, combined_input
+        )
         if cached_content:
             if tracker:
                 tracker.advance(1, extra="[Cached]: data_models.md")
@@ -305,20 +391,37 @@ def generate_data_models_doc(
     if tracker:
         tracker.render_current(extra="data_models.md")
 
-    reduced_types_summary = hierarchical_reduce_summaries(
-        types_raw,
-        "Type Definitions",
-        llm_client,
-        language=norm_lang,
-        allow_fallback=allow_fallback,
-    )
-    reduced_vars_summary = hierarchical_reduce_summaries(
-        vars_raw,
-        "Variables and Fields",
-        llm_client,
-        language=norm_lang,
-        allow_fallback=allow_fallback,
-    )
+    total_types_len = sum(len(it) for it in types_raw)
+    if total_types_len <= 15000 or len(types_raw) <= 50:
+        reduced_types_summary = "\n".join(types_raw) if types_raw else "(None)"
+    else:
+        reduced_types_summary = hierarchical_reduce_summaries(
+            types_raw,
+            "Type Definitions",
+            llm_client,
+            language=norm_lang,
+            allow_fallback=allow_fallback,
+            max_items_per_chunk=30,
+            max_chars_per_chunk=8000,
+            db=db,
+            force=force,
+        )
+
+    total_vars_len = sum(len(it) for it in vars_raw)
+    if total_vars_len <= 15000 or len(vars_raw) <= 50:
+        reduced_vars_summary = "\n".join(vars_raw) if vars_raw else "(None)"
+    else:
+        reduced_vars_summary = hierarchical_reduce_summaries(
+            vars_raw,
+            "Variables and Fields",
+            llm_client,
+            language=norm_lang,
+            allow_fallback=allow_fallback,
+            max_items_per_chunk=30,
+            max_chars_per_chunk=8000,
+            db=db,
+            force=force,
+        )
 
     prompt = f"""Please analyze aggregated types and variables/fields,
 and create a factual "Data Models & Data Structure Design" doc in {norm_lang}.
@@ -405,10 +508,48 @@ Structure the Markdown as follows:
 
     write_flushed_text(out_file, content.strip() + "\n")
     if db:
-        db.save_design_cache(cache_key, input_hash, content)
+        db.save_design_cache(
+            cache_key, input_hash, content, combined_input
+        )
     return content
 
 
+def select_architectural_functions(
+    fn_docs: List[Dict[str, Any]], max_count: int = 60
+) -> List[Dict[str, Any]]:
+    """Select key architectural functions (entry points, pipelines).
+
+    Instead of passing hundreds of utility/test/leaf functions, we prioritize:
+    1. Functions with 'main', 'cli', 'run', 'start', 'handler', 'app', 'serve'.
+    2. Functions with UI interaction roles.
+    3. Functions that call many others (pipeline orchestrators).
+    4. Functions with no callers inside the project (root entry points).
+    """
+    if len(fn_docs) <= max_count:
+        return fn_docs
+
+    def score(d: Dict[str, Any]) -> int:
+        fname = d["file_name"].lower()
+        s = 0
+        if any(
+            k in fname
+            for k in (
+                "main", "cli", "run", "start", "handler", "app", "serve",
+                "pipeline", "engine", "orchestrat"
+            )
+        ):
+            s += 20
+        if d.get("ui_role"):
+            s += 15
+        s += len(d.get("callees") or []) * 2
+        if not d.get("referencing_funcs"):
+            s += 5
+        return s
+
+    scored = sorted(fn_docs, key=score, reverse=True)
+    return scored[:max_count]
+
+
 def generate_execution_model_doc(
     fn_docs: List[Dict[str, Any]],
     llm_client: Optional[LLMClient],
@@ -420,11 +561,15 @@ def generate_execution_model_doc(
     ui_info: Optional[Dict[str, Any]] = None,
     tracker: Optional[PhaseProgressTracker] = None,
 ) -> str:
-    """Step 2: Generate execution_model.md with SQLite cache & Map-Reduce."""
+    """Step 2: Generate execution_model.md with filtered functions."""
     norm_lang = normalize_language(language)
+
+    # 1. Filter to core entry points and high-level callers
+    arch_funcs = select_architectural_functions(fn_docs, max_count=60)
+
     funcs_raw = []
-    for d in fn_docs:
-        callees_str = ", ".join(d["callees"][:3]) if d["callees"] else "None"
+    for d in arch_funcs:
+        callees_str = ", ".join(d["callees"][:4]) if d["callees"] else "None"
         ui_extra = f" / UI Role: {d['ui_role']}" if d.get("ui_role") else ""
         line_desc = (
             f"- Function `{d['file_name']}` (Signature: `{d['signature']}`): "
@@ -438,7 +583,9 @@ def generate_execution_model_doc(
     out_file = target_dir / ".pystdoc" / "design" / "execution_model.md"
 
     if db and not force and out_file.exists():
-        cached_content = db.load_design_cache(cache_key, input_hash)
+        cached_content = db.load_design_cache(
+            cache_key, input_hash, combined_input
+        )
         if cached_content:
             if tracker:
                 tracker.advance(
@@ -453,13 +600,22 @@ def generate_execution_model_doc(
     if tracker:
         tracker.render_current(extra="execution_model.md")
 
-    reduced_funcs_summary = hierarchical_reduce_summaries(
-        funcs_raw,
-        "Function Call Structures",
-        llm_client,
-        language=norm_lang,
-        allow_fallback=allow_fallback,
-    )
+    # If items fit comfortably in a single LLM context, directly use them
+    total_raw_len = sum(len(it) for it in funcs_raw)
+    if total_raw_len <= 18000 or len(funcs_raw) <= 60:
+        reduced_funcs_summary = "\n".join(funcs_raw)
+    else:
+        reduced_funcs_summary = hierarchical_reduce_summaries(
+            funcs_raw,
+            "Function Call Structures",
+            llm_client,
+            language=norm_lang,
+            allow_fallback=allow_fallback,
+            max_items_per_chunk=30,
+            max_chars_per_chunk=8000,
+            db=db,
+            force=force,
+        )
 
     ui_context_text = ""
     if ui_info and ui_info.get("has_ui"):
@@ -554,7 +710,9 @@ Structure the Markdown as follows:
 
     write_flushed_text(out_file, content.strip() + "\n")
     if db:
-        db.save_design_cache(cache_key, input_hash, content)
+        db.save_design_cache(
+            cache_key, input_hash, content, combined_input
+        )
     return content
 
 
@@ -592,7 +750,9 @@ def generate_module_docs(
         )
 
         if db and not force and out_file.exists():
-            cached_content = db.load_design_cache(cache_key, input_hash)
+            cached_content = db.load_design_cache(
+                cache_key, input_hash, combined_input
+            )
             if cached_content:
                 if tracker:
                     tracker.advance(
@@ -615,6 +775,8 @@ def generate_module_docs(
             llm_client,
             language=norm_lang,
             allow_fallback=allow_fallback,
+            db=db,
+            force=force,
         )
 
         prompt = f"""Please analyze elements in module `{mod_name}`
@@ -700,7 +862,9 @@ Structure the Markdown as follows:
 
         write_flushed_text(out_file, content.strip() + "\n")
         if db:
-            db.save_design_cache(cache_key, input_hash, content)
+            db.save_design_cache(
+                cache_key, input_hash, content, combined_input
+            )
         module_summaries[mod_name] = content
 
     return module_summaries
@@ -740,7 +904,9 @@ def generate_overview_doc(
     out_file = target_dir / ".pystdoc" / "design" / "overview.md"
 
     if db and not force and out_file.exists():
-        cached_content = db.load_design_cache(cache_key, input_hash)
+        cached_content = db.load_design_cache(
+            cache_key, input_hash, combined_input
+        )
         if cached_content:
             if tracker:
                 tracker.advance(1, extra="[Cached]: overview.md")
@@ -758,6 +924,8 @@ def generate_overview_doc(
         llm_client,
         language=norm_lang,
         allow_fallback=allow_fallback,
+        db=db,
+        force=force,
     )
 
     ui_overview_text = ""
@@ -852,7 +1020,9 @@ Requirements:
 
     write_flushed_text(out_file, content.strip() + "\n")
     if db:
-        db.save_design_cache(cache_key, input_hash, content)
+        db.save_design_cache(
+            cache_key, input_hash, content, combined_input
+        )
     return content
 
 

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/engine.py

@@ -46,6 +46,7 @@ from pystdoc.progress import (
 )
 from pystdoc.interrupt import InterruptionState
 from pystdoc.perf import PerfProfileManager
+from pystdoc.similarity import similarity
 
 
 class FileLock:
@@ -58,7 +59,7 @@ class FileLock:
 
     def __enter__(self):
         self.lock_file.parent.mkdir(parents=True, exist_ok=True)
-        self.fd = open(self.lock_file, "w")
+        self.fd = open(self.lock_file, "w", encoding="utf-8")
         try:
             fcntl.flock(self.fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
         except BlockingIOError:
@@ -130,18 +131,6 @@ def check_external_interface_changed(
     if old_cache is None:
         return True
 
-    def get_tokens(text: Optional[str]) -> Set[str]:
-        clean = re.sub(
-            r"[.,。、:：\-_\(\)（）「」『』`\"'\s\r\n\t]+", " ", text or ""
-        )
-        words = set(clean.lower().split())
-        # Add 2-grams for CJK strings
-        compact = "".join(clean.lower().split())
-        if len(compact) >= 2:
-            for i in range(len(compact) - 1):
-                words.add(compact[i:i+2])
-        return words
-
     def normalize_spec_text(text: Optional[str]) -> str:
         if not text:
             return ""
@@ -160,28 +149,22 @@ def check_external_interface_changed(
     if old_in != new_in or old_out != new_out:
         if bool(old_in) != bool(new_in) or bool(old_out) != bool(new_out):
             return True
-        t_in1 = get_tokens(old_cache.get("inputs_note"))
-        t_in2 = get_tokens(new_sym.inputs_note)
-        if t_in1 and t_in2:
-            jaccard_in = len(t_in1 & t_in2) / max(len(t_in1 | t_in2), 1)
-            if jaccard_in < 0.5:
-                return True
-        t_out1 = get_tokens(old_cache.get("outputs_note"))
-        t_out2 = get_tokens(new_sym.outputs_note)
-        if t_out1 and t_out2:
-            jaccard_out = len(t_out1 & t_out2) / max(len(t_out1 | t_out2), 1)
-            if jaccard_out < 0.5:
-                return True
+        if old_in and new_in and similarity(
+            old_cache.get("inputs_note"), new_sym.inputs_note
+        ) < 0.5:
+            return True
+        if old_out and new_out and similarity(
+            old_cache.get("outputs_note"), new_sym.outputs_note
+        ) < 0.5:
+            return True
 
     # Check purpose
     if old_p == new_p:
         return False
-    t1 = get_tokens(old_cache.get("purpose"))
-    t2 = get_tokens(new_sym.purpose)
-    if t1 and t2:
-        jaccard = len(t1 & t2) / max(len(t1 | t2), 1)
-        if jaccard >= 0.5:
-            return False
+    if old_p and new_p and similarity(
+        old_cache.get("purpose"), new_sym.purpose
+    ) >= 0.5:
+        return False
 
     return True
 
@@ -924,7 +907,7 @@ def run_docgen(
                 )
 
         # Process Level by Level
-        for lvl_idx, lvl_nodes in enumerate(level_groups):
+        for lvl_nodes in level_groups:
             InterruptionState.check_interrupted()
             executor = concurrent.futures.ThreadPoolExecutor(
                 max_workers=concurrency,
@@ -1093,18 +1076,18 @@ def run_docgen(
                 parent_container_info is not None
                 and any(
                     t.symbol.name == parent_container_info["name"]
-                    and t.unique_id in changed_upstream_ids
+                    and t.unique_id in interface_changed_symbol_ids
                     for t in type_nodes
                     if t.rel_path == v_rel
                 )
             )
             is_parent_func_updated = (
                 any(
-                    caller_id in changed_upstream_ids
+                    caller_id in interface_changed_symbol_ids
                     for caller_id in v_node.direct_caller_ids
                 )
                 or any(
-                    f_node.unique_id in changed_upstream_ids
+                    f_node.unique_id in interface_changed_symbol_ids
                     for f_node in fn_nodes
                     if f_node.rel_path == v_rel
                     and (
@@ -1182,10 +1165,14 @@ def run_docgen(
                     or top_down_res.get("significance")
                     or top_down_res.get("top_down_summary")
                 )
-                if raw_v_role:
-                    v_sym.top_down_context = sanitize_architectural_context(
-                        raw_v_role, fallback_text=v_sym.purpose or ""
-                    )
+                v_sym.top_down_context = sanitize_architectural_context(
+                    raw_v_role,
+                    fallback_text=(
+                        v_sym.top_down_context
+                        or v_sym.purpose
+                        or v_sym.name
+                    ),
+                )
                 if top_down_res.get("purpose"):
                     v_sym.purpose = top_down_res["purpose"]
                 if top_down_res.get("overview"):

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/llm_client.py

@@ -15,8 +15,6 @@ from pystdoc.interrupt import InterruptionState, interruptible_sleep
 class LLMError(RuntimeError):
     """Exception raised when LLM generation fails in strict mode."""
 
-    pass
-
 
 def normalize_host_url(host: Optional[str]) -> str:
     """Normalize user-provided host string into a standard base URL."""
@@ -244,7 +242,8 @@ class LLMClient:
         """Detect WSL default gateway host IP via ip route or resolv.conf."""
         try:
             res = subprocess.run(
-                ["ip", "route"], capture_output=True, text=True, timeout=1
+                ["ip", "route"], capture_output=True, text=True, timeout=1,
+                check=False
             )
             for line in res.stdout.splitlines():
                 if line.startswith("default via"):
@@ -253,7 +252,7 @@ class LLMClient:
             pass
 
         try:
-            resolv = Path("/etc/resolv.conf").read_text()
+            resolv = Path("/etc/resolv.conf").read_text(encoding="utf-8")
             for line in resolv.splitlines():
                 if line.strip().startswith("nameserver"):
                     return line.split()[1].strip()

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/mcp_server.py

@@ -401,7 +401,6 @@ def create_mcp_server(
 
         # Capture output or format response
         from io import StringIO
-        import sys
         old_stdout = sys.stdout
         old_stderr = sys.stderr
         capture_out = StringIO()
@@ -439,7 +438,6 @@ def create_mcp_server(
             return _missing_docgen_msg(target_dir)
 
         from io import StringIO
-        import sys
         old_stdout = sys.stdout
         old_stderr = sys.stderr
         capture = StringIO()
@@ -547,7 +545,6 @@ def create_mcp_server(
         if not docgen_dir.exists():
             return _missing_docgen_msg(target_dir)
         from io import StringIO
-        import sys
         old_stdout = sys.stdout
         capture = StringIO()
         try:

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/perf.py

@@ -266,7 +266,7 @@ class PerfProfileManager:
 
             if not coeffs:
                 all_cats = self.data.get(key, {}).get("categories", {})
-                for c_k, c_v in all_cats.items():
+                for c_v in all_cats.values():
                     if c_v.get("coeffs"):
                         coeffs = c_v["coeffs"]
                         break
@@ -300,7 +300,7 @@ class PerfProfileManager:
             chars_coeffs = cat_data.get("chars_coeffs")
             if not chars_coeffs:
                 all_cats = self.data.get(key, {}).get("categories", {})
-                for c_k, c_v in all_cats.items():
+                for c_v in all_cats.values():
                     if c_v.get("chars_coeffs"):
                         chars_coeffs = c_v["chars_coeffs"]
                         break

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/query.py

@@ -456,7 +456,7 @@ def locate_features(target_dir: Path, query: str, limit: int = 10) -> str:
             (
                 uid, sname, skind, rpath,
                 lstart, lend, fqdn, sig,
-                ref_funcs_raw, callees_raw,
+                _ref_funcs_raw, _callees_raw,
                 purp, ovw
             ) = row
             sname_l = (sname or "").lower()

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/report_engine.py

@@ -142,7 +142,9 @@ def generate_readme_doc(
     cache_key = f"report::readme::{norm_lang}"
 
     if db and not force and out_file.exists():
-        cached_content = db.load_design_cache(cache_key, input_hash)
+        cached_content = db.load_design_cache(
+            cache_key, input_hash, combined_input
+        )
         if cached_content:
             tracker.advance(1, extra="[Cached]: .pystdoc/README.md")
             tracker.finish()
@@ -347,5 +349,7 @@ topological dependency mapping.
 
     write_flushed_text(out_file, content.strip() + "\n")
     if db:
-        db.save_design_cache(cache_key, input_hash, content)
+        db.save_design_cache(
+            cache_key, input_hash, content, combined_input
+        )
     return content

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc/scanner.py

@@ -58,7 +58,7 @@ def scan_files(target_dir: Path) -> List[Path]:
                 rel_path = file_path.relative_to(target_dir)
                 matched_files.append(rel_path)
 
-    matched_files.sort(key=lambda p: str(p))
+    matched_files.sort(key=str)
     return matched_files
 
 

pystdoc-0.8.0/src/pystdoc/similarity.py

@@ -0,0 +1,45 @@
+"""Lightweight, deterministic semantic-similarity helpers (no LLM calls).
+
+Used to decide whether a regenerated description is "essentially the same"
+as the previous one, so that change propagation can stop early.
+"""
+
+import re
+from typing import Optional, Set
+
+_PUNCT_RE = re.compile(r"[.,。、:：;；!?！？\-_()（）「」『』`\"'\[\]{}*#>|/\\\s]+")
+
+
+def get_tokens(text: Optional[str]) -> Set[str]:
+    """Tokenize into lowercase words plus character 2-grams (for CJK)."""
+    clean = _PUNCT_RE.sub(" ", (text or "").lower()).strip()
+    words = set(clean.split())
+    compact = clean.replace(" ", "")
+    for i in range(len(compact) - 1):
+        words.add(compact[i:i + 2])
+    return words
+
+
+def similarity(a: Optional[str], b: Optional[str]) -> float:
+    """Return the max of Jaccard and overlap coefficient (0.0 - 1.0).
+
+    The overlap coefficient tolerates elaboration (one text containing the
+    other) while Jaccard penalizes unrelated text.
+    """
+    t1, t2 = get_tokens(a), get_tokens(b)
+    if not t1 and not t2:
+        return 1.0
+    if not t1 or not t2:
+        return 0.0
+    inter = len(t1 & t2)
+    jaccard = inter / len(t1 | t2)
+    overlap = inter / min(len(t1), len(t2))
+    # overlap alone is too lenient for very different sizes; damp it.
+    size_ratio = min(len(t1), len(t2)) / max(len(t1), len(t2))
+    return max(jaccard, overlap * (0.5 + 0.5 * size_ratio))
+
+
+def is_semantically_same(
+    a: Optional[str], b: Optional[str], threshold: float = 0.85
+) -> bool:
+    return similarity(a, b) >= threshold

{pystdoc-0.7.2 → pystdoc-0.8.0/src/pystdoc.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pystdoc
-Version: 0.7.2
+Version: 0.8.0
 Summary: Python Structural & Topological Documentation Engine (pystdoc): High-precision, LLM-powered hierarchical codebase and architecture documentation generator.
 Author: tab4moji
 License: MIT
@@ -78,21 +78,25 @@ your_project/
    - Evaluates call graphs in $O(V+E)$ linear time.
    - Automatically breaks cyclic mutual recursions and organizes code symbols into dependency-safe execution levels.
    - Level-by-level parallel LLM execution guarantees context-rich bottom-up summaries without race conditions.
-2. **3-in-1 Unified Documentation Pipeline**:
+2. **High-Speed Incremental Sync & Semantic Change Propagation (v0.8.0)**:
+   - **Selective Bottom-Up Propagation**: Detects AST and logic-level code differences. If regenerated specifications remain semantically unchanged (evaluated by deterministic Jaccard / token-overlap metrics), change propagation terminates immediately without triggering cascading LLM re-analysis across upper modules.
+   - **Content-Defined Map-Reduce Caching**: Intermediate summaries in architecture synthesis (`designgen`) are chunked using stable hash boundaries and cached per group in SQLite, ensuring that only groups containing edited functions are re-summarized.
+   - **Architectural Entry-Point Filtering**: For system execution modeling (`execution_model.md`), key architectural functions (entry points, CLI routers, and core dispatchers) are automatically filtered and passed directly into the LLM context, bypassing heavy multi-step Map-Reduce loops and speeding up generation by up to 90%+.
+3. **3-in-1 Unified Documentation Pipeline**:
    - `docgen`: Bottom-up & top-down symbol-level documentation with SHA-256 and SQLite caching (`.pystdoc/documents/`).
    - `designgen`: Map-Reduce architectural synthesis (`.pystdoc/design/`).
    - `reportgen` / `pystdoc`: Executive summary README (`.pystdoc/README.md`) answering *"What does this project actually do?"*
-3. **Multi-Language Codebase Deep Understanding (C/C++, Python, Java, Kotlin, Shell)**:
+4. **Multi-Language Codebase Deep Understanding (C/C++, Python, Java, Kotlin, Shell)**:
    - **C/C++**: `compile_commands.json` integration, include path resolution, and macro expansion via `libclang`.
    - **Java / Kotlin**: AST parsing via `javalang` & `kopyt` with class hierarchy, companion objects, and data classes.
    - **Python & Shell**: AST visitor and generic regex parser with import/call extraction.
    - **Fully Qualified Domain Names (FQDN)**: Disambiguates identical symbol names across large monorepos.
-4. **Feature Location & UI Component Pinpointing (`locate` / `pystdoc_locate_feature`)**:
+5. **Feature Location & UI Component Pinpointing (`locate` / `pystdoc_locate_feature`)**:
    - Pinpoint exact files, functions, UI elements, and line ranges (`Lines X-Y`) from high-level natural language queries (e.g., *"where is the debug notification button"*, *"add auth header"*) without running brute-force `Glob`/`Grep` across the entire project.
    - Built-in multi-lingual token scoring with automatic boosting for UI elements (`@Composable`, `Button`, `Dialog`, `Activity`, `Modifier`, etc.).
-5. **Impact & Inbound/Outbound Dependency Analysis (`impact` / `pystdoc_trace_impact`)**:
+6. **Impact & Inbound/Outbound Dependency Analysis (`impact` / `pystdoc_trace_impact`)**:
    - Instant 360-degree dependency tracing: inspect where a symbol is called from (*Inbound Callers / References*) and what it uses (*Outbound Dependencies*) before making breaking code changes or deletions.
-6. **Standard LLM Options & Multi-Language Support**:
+7. **Standard LLM Options & Multi-Language Support**:
    - Works with **Ollama, LiteRT-LM, vLLM, and OpenAI API**.
    - Supports `--host`, `--model`, `--token` / `--api-key`, `--context-size`, and `--language` (defaults to terminal language, e.g. `Japanese`, `English`, `日本語`).
 

{pystdoc-0.7.2 → pystdoc-0.8.0}/src/pystdoc.egg-info/SOURCES.txt

@@ -26,6 +26,7 @@ src/pystdoc/progress.py
 src/pystdoc/query.py
 src/pystdoc/report_engine.py
 src/pystdoc/scanner.py
+src/pystdoc/similarity.py
 src/pystdoc/symbols.py
 src/pystdoc/ui_detector.py
 src/pystdoc/watcher.py
@@ -63,6 +64,7 @@ tests/test_pep8.py
 tests/test_perf.py
 tests/test_progress.py
 tests/test_query.py
+tests/test_reduce_cache.py
 tests/test_reportgen.py
 tests/test_ui_detector.py
 tests/test_watcher.py

pystdoc-0.8.0/tests/test_reduce_cache.py

@@ -0,0 +1,78 @@
+"""Tests for group-level caching in hierarchical_reduce_summaries."""
+
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock
+
+from pystdoc.db import DocgenDB
+from pystdoc.design_engine import (
+    chunk_list_stable,
+    hierarchical_reduce_summaries,
+)
+
+
+def _items(n, changed=None):
+    out = []
+    for i in range(n):
+        text = f"- Function `mod{i:04d}.fn.f{i}.md`: purpose of item {i} " * 3
+        if changed is not None and i == changed:
+            text += " CHANGED"
+        out.append(text)
+    return out
+
+
+class TestReduceCache(unittest.TestCase):
+    def setUp(self):
+        self.tmp = Path(tempfile.mkdtemp())
+        self.db = DocgenDB(self.tmp / "index.db")
+
+    def tearDown(self):
+        self.db.close()
+
+    def test_stable_chunking_localizes_change(self):
+        a = chunk_list_stable(_items(300))
+        b = chunk_list_stable(_items(300, changed=150))
+        changed = sum(1 for c in b if c not in a)
+        self.assertGreater(len(a), 5)
+        self.assertLessEqual(changed, 2)
+        self.assertEqual(sum(len(c) for c in a), 300)
+
+    def test_group_cache_only_recomputes_changed_groups(self):
+        llm = MagicMock()
+        llm.chat_completion.side_effect = lambda m: "summary"
+        hierarchical_reduce_summaries(
+            _items(300), "Funcs", llm, language="Japanese", db=self.db
+        )
+        first_calls = llm.chat_completion.call_count
+        self.assertGreater(first_calls, 5)
+
+        llm.chat_completion.reset_mock()
+        hierarchical_reduce_summaries(
+            _items(300), "Funcs", llm, language="Japanese", db=self.db
+        )
+        self.assertEqual(llm.chat_completion.call_count, 0)
+
+        llm.chat_completion.reset_mock()
+        hierarchical_reduce_summaries(
+            _items(300, changed=150), "Funcs", llm,
+            language="Japanese", db=self.db,
+        )
+        self.assertLessEqual(llm.chat_completion.call_count, 4)
+        self.assertGreater(llm.chat_completion.call_count, 0)
+
+    def test_force_bypasses_group_cache(self):
+        llm = MagicMock()
+        llm.chat_completion.side_effect = lambda m: "summary"
+        hierarchical_reduce_summaries(
+            _items(100), "Funcs", llm, db=self.db
+        )
+        llm.chat_completion.reset_mock()
+        hierarchical_reduce_summaries(
+            _items(100), "Funcs", llm, db=self.db, force=True
+        )
+        self.assertGreater(llm.chat_completion.call_count, 0)
+
+
+if __name__ == "__main__":
+    unittest.main()