claude-sql 0.4.0__tar.gz → 0.5.0__tar.gz

{claude_sql-0.4.0 → claude_sql-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.3
 Name: claude-sql
-Version: 0.4.0
+Version: 0.5.0
 Summary: Zero-copy SQL + semantic search + LLM analytics over ~/.claude/ transcripts.
 Keywords: claude,claude-code,anthropic,duckdb,sql,semantic-search,embeddings,bedrock,transcripts,analytics,observability
 Author: Laith Al-Saadoon
@@ -23,8 +23,9 @@ Requires-Dist: boto3>=1.42.91
 Requires-Dist: cyclopts>=4.10.2
 Requires-Dist: duckdb>=1.5.2
 Requires-Dist: hdbscan>=0.8.40
+Requires-Dist: igraph>=1.0.0,<2.0
+Requires-Dist: leidenalg>=0.11.0,<0.12
 Requires-Dist: loguru>=0.7.3
-Requires-Dist: networkx>=3.4
 Requires-Dist: numpy>=2.4.4
 Requires-Dist: packaging>=26.2
 Requires-Dist: polars>=1.40.0
@@ -122,7 +123,7 @@ flowchart LR
     L --> PA["session_classifications/, message_trajectory/,<br/>session_conflicts/, user_friction/<br/>(sharded part-*.parquet)"]
     P --> C["claude-sql cluster<br/>(UMAP + HDBSCAN)"]
     C --> PC["clusters + cluster_terms<br/>(c-TF-IDF)"]
-    P --> CM["claude-sql community<br/>(Louvain over centroids)"]
+    P --> CM["claude-sql community<br/>(Leiden + CPM over mutual-kNN centroids)"]
     CM --> PM["session_communities<br/>parquet"]
     PA --> AV[analytics views + macros]
     PC --> AV
@@ -272,7 +273,7 @@ Commands that spend real Bedrock money default to `--dry-run`.
 | `conflicts` | Per-session stance-conflict detection |
 | `friction` | Regex + Sonnet 4.6 → status pings, unmet expectations, confusion, etc. |
 | `cluster` | UMAP → HDBSCAN → c-TF-IDF over message embeddings |
-| `community` | Louvain over session centroids |
+| `community` | Leiden + CPM over mutual-kNN session centroids; emits medoid + coherence + resolution profile + `--neighbors-of` |
 | `skills sync` | Walk `~/.claude/skills/` + `~/.claude/plugins/cache/` → seedable skills catalog |
 | `skills ls` | List catalog entries, filterable by `--kind` and `--plugin` |
 | `analyze` | Run the whole pipeline in dependency order |
@@ -325,7 +326,8 @@ Commands that spend real Bedrock money default to `--dry-run`.
 | `session_conflicts` | per-session stance conflicts | `stance_a`, `stance_b`, `resolution` |
 | `message_clusters` | cluster id + 2d viz coords | `cluster_id`, `x`, `y`, `is_noise` |
 | `cluster_terms` | c-TF-IDF top terms per cluster | `cluster_id`, `term`, `weight`, `rank` |
-| `session_communities` | Louvain community per session | `community_id`, `size` |
+| `session_communities` | Leiden+CPM community per session | `community_id`, `size`, `is_medoid`, `coherence`, `gamma_used` |
+| `community_profile` | Resolution-profile sidecar (auto-γ runs only) | `gamma`, `n_communities`, `quality`, `plateau_length` |
 | `user_friction` | one row per classified short user message | `label` (7-way), `rationale`, `source` (`regex` / `llm` / `refused`), `confidence` |
 | `skills_catalog` | one row per known skill / slash command (seed by `claude-sql skills sync`) | `skill_id`, `name`, `plugin`, `plugin_version`, `source_kind` (`user-skill` / `plugin-skill` / `plugin-command` / `builtin`), `description` |
 | `skill_usage` | `skill_invocations` ⟕ `skills_catalog` | `source`, `skill_id`, `skill_name`, `plugin`, `is_builtin`, `description` |
@@ -380,7 +382,15 @@ Every option is configurable via `CLAUDE_SQL_*`:
 | `CLAUDE_SQL_SKILLS_CATALOG_PARQUET_PATH` | `~/.claude/skills_catalog.parquet` | Skills catalog parquet |
 | `CLAUDE_SQL_USER_SKILLS_DIR` | `~/.claude/skills` | Root scanned for user-installed skills |
 | `CLAUDE_SQL_PLUGINS_CACHE_DIR` | `~/.claude/plugins/cache` | Root scanned for plugin skills + commands |
-| `CLAUDE_SQL_SEED` | `42` | UMAP / HDBSCAN / Louvain determinism |
+| `CLAUDE_SQL_SEED` | `42` | UMAP / HDBSCAN / Leiden determinism |
+| `CLAUDE_SQL_LEIDEN_KNN_K` | `15` | Mutual-kNN k for the session-centroid graph |
+| `CLAUDE_SQL_LEIDEN_EDGE_FLOOR` | `0.3` | Cosine floor below which edges are dropped |
+| `CLAUDE_SQL_LEIDEN_MIN_COMMUNITY_SIZE` | `3` | Communities below this collapse to noise (`-1`) |
+| `CLAUDE_SQL_LEIDEN_RESOLUTION` | unset (auto) | Explicit CPM γ; skips the resolution profile |
+| `CLAUDE_SQL_LEIDEN_RESOLUTION_RANGE_LO` | `0.05` | Lower bound for `Optimiser.resolution_profile` bisection |
+| `CLAUDE_SQL_LEIDEN_RESOLUTION_RANGE_HI` | `0.95` | Upper bound for the bisection |
+| `CLAUDE_SQL_LEIDEN_N_ITERATIONS` | `-1` | Iterate to convergence; `2` is leidenalg's default |
+| `CLAUDE_SQL_COMMUNITY_PROFILE_PARQUET_PATH` | `~/.claude/community_profile.parquet` | Resolution-profile sidecar path |
 
 ## Development
 
@@ -497,12 +507,25 @@ See `docs/adr/0015-stack-modernization.md` and
   with adaptive thinking on. Pydantic v2 schemas are flattened (inline
   `$ref`, inject `additionalProperties: false`, strip the numeric /
   string constraints the validator rejects from Draft 2020-12).
-- **Determinism.** UMAP, HDBSCAN, and Louvain all seed from
+- **Determinism.** UMAP, HDBSCAN, and Leiden all seed from
   `CLAUDE_SQL_SEED=42` (default) so cluster IDs and community IDs are
-  stable across reruns.
-- **Louvain = `networkx`.** `networkx.community.louvain_communities`,
-  built into `networkx >= 3.4`. The abandoned `python-louvain` package
-  is not used.
+  stable across reruns. The Leiden seed flows into both
+  `leidenalg.find_partition(seed=...)` and `Optimiser.set_rng_seed(...)`
+  for the resolution-profile bisection — same seed + same input ⇒
+  byte-equal parquets across runs.
+- **Communities = `leidenalg` + CPM.** Reference Leiden implementation
+  (`leidenalg>=0.11.0`) over an `igraph>=1.0` mutual-kNN cosine graph
+  (k=15, edge floor 0.3) of session centroids. CPM γ has direct cosine
+  semantics (internal density ≥ γ, external ≤ γ); auto-γ via
+  `Optimiser.resolution_profile` + longest-plateau picker; warn-only
+  connectivity check. Output is signal-rich: `is_medoid`, `coherence`,
+  `gamma_used` per row, plus a `community_profile` sidecar with one row
+  per γ tested, so an LLM agent can ask "what γ would yield 50
+  communities?" without rerunning Leiden. Top terms per community come
+  from the live `community_top_topics(cid, n)` macro composed from
+  `cluster_terms`, not a frozen column. See
+  [`docs/research_notes.md`](docs/research_notes.md) for the
+  Louvain → Leiden+CPM swap rationale.
 - **Hybrid friction pipeline.** A hand-curated regex bank catches the
   unambiguous `status_ping` / `interruption` / `correction` cases at
   zero Bedrock cost; the ambiguous class — especially

{claude_sql-0.4.0 → claude_sql-0.5.0}/README.md

@@ -77,7 +77,7 @@ flowchart LR
     L --> PA["session_classifications/, message_trajectory/,<br/>session_conflicts/, user_friction/<br/>(sharded part-*.parquet)"]
     P --> C["claude-sql cluster<br/>(UMAP + HDBSCAN)"]
     C --> PC["clusters + cluster_terms<br/>(c-TF-IDF)"]
-    P --> CM["claude-sql community<br/>(Louvain over centroids)"]
+    P --> CM["claude-sql community<br/>(Leiden + CPM over mutual-kNN centroids)"]
     CM --> PM["session_communities<br/>parquet"]
     PA --> AV[analytics views + macros]
     PC --> AV
@@ -227,7 +227,7 @@ Commands that spend real Bedrock money default to `--dry-run`.
 | `conflicts` | Per-session stance-conflict detection |
 | `friction` | Regex + Sonnet 4.6 → status pings, unmet expectations, confusion, etc. |
 | `cluster` | UMAP → HDBSCAN → c-TF-IDF over message embeddings |
-| `community` | Louvain over session centroids |
+| `community` | Leiden + CPM over mutual-kNN session centroids; emits medoid + coherence + resolution profile + `--neighbors-of` |
 | `skills sync` | Walk `~/.claude/skills/` + `~/.claude/plugins/cache/` → seedable skills catalog |
 | `skills ls` | List catalog entries, filterable by `--kind` and `--plugin` |
 | `analyze` | Run the whole pipeline in dependency order |
@@ -280,7 +280,8 @@ Commands that spend real Bedrock money default to `--dry-run`.
 | `session_conflicts` | per-session stance conflicts | `stance_a`, `stance_b`, `resolution` |
 | `message_clusters` | cluster id + 2d viz coords | `cluster_id`, `x`, `y`, `is_noise` |
 | `cluster_terms` | c-TF-IDF top terms per cluster | `cluster_id`, `term`, `weight`, `rank` |
-| `session_communities` | Louvain community per session | `community_id`, `size` |
+| `session_communities` | Leiden+CPM community per session | `community_id`, `size`, `is_medoid`, `coherence`, `gamma_used` |
+| `community_profile` | Resolution-profile sidecar (auto-γ runs only) | `gamma`, `n_communities`, `quality`, `plateau_length` |
 | `user_friction` | one row per classified short user message | `label` (7-way), `rationale`, `source` (`regex` / `llm` / `refused`), `confidence` |
 | `skills_catalog` | one row per known skill / slash command (seed by `claude-sql skills sync`) | `skill_id`, `name`, `plugin`, `plugin_version`, `source_kind` (`user-skill` / `plugin-skill` / `plugin-command` / `builtin`), `description` |
 | `skill_usage` | `skill_invocations` ⟕ `skills_catalog` | `source`, `skill_id`, `skill_name`, `plugin`, `is_builtin`, `description` |
@@ -335,7 +336,15 @@ Every option is configurable via `CLAUDE_SQL_*`:
 | `CLAUDE_SQL_SKILLS_CATALOG_PARQUET_PATH` | `~/.claude/skills_catalog.parquet` | Skills catalog parquet |
 | `CLAUDE_SQL_USER_SKILLS_DIR` | `~/.claude/skills` | Root scanned for user-installed skills |
 | `CLAUDE_SQL_PLUGINS_CACHE_DIR` | `~/.claude/plugins/cache` | Root scanned for plugin skills + commands |
-| `CLAUDE_SQL_SEED` | `42` | UMAP / HDBSCAN / Louvain determinism |
+| `CLAUDE_SQL_SEED` | `42` | UMAP / HDBSCAN / Leiden determinism |
+| `CLAUDE_SQL_LEIDEN_KNN_K` | `15` | Mutual-kNN k for the session-centroid graph |
+| `CLAUDE_SQL_LEIDEN_EDGE_FLOOR` | `0.3` | Cosine floor below which edges are dropped |
+| `CLAUDE_SQL_LEIDEN_MIN_COMMUNITY_SIZE` | `3` | Communities below this collapse to noise (`-1`) |
+| `CLAUDE_SQL_LEIDEN_RESOLUTION` | unset (auto) | Explicit CPM γ; skips the resolution profile |
+| `CLAUDE_SQL_LEIDEN_RESOLUTION_RANGE_LO` | `0.05` | Lower bound for `Optimiser.resolution_profile` bisection |
+| `CLAUDE_SQL_LEIDEN_RESOLUTION_RANGE_HI` | `0.95` | Upper bound for the bisection |
+| `CLAUDE_SQL_LEIDEN_N_ITERATIONS` | `-1` | Iterate to convergence; `2` is leidenalg's default |
+| `CLAUDE_SQL_COMMUNITY_PROFILE_PARQUET_PATH` | `~/.claude/community_profile.parquet` | Resolution-profile sidecar path |
 
 ## Development
 
@@ -452,12 +461,25 @@ See `docs/adr/0015-stack-modernization.md` and
   with adaptive thinking on. Pydantic v2 schemas are flattened (inline
   `$ref`, inject `additionalProperties: false`, strip the numeric /
   string constraints the validator rejects from Draft 2020-12).
-- **Determinism.** UMAP, HDBSCAN, and Louvain all seed from
+- **Determinism.** UMAP, HDBSCAN, and Leiden all seed from
   `CLAUDE_SQL_SEED=42` (default) so cluster IDs and community IDs are
-  stable across reruns.
-- **Louvain = `networkx`.** `networkx.community.louvain_communities`,
-  built into `networkx >= 3.4`. The abandoned `python-louvain` package
-  is not used.
+  stable across reruns. The Leiden seed flows into both
+  `leidenalg.find_partition(seed=...)` and `Optimiser.set_rng_seed(...)`
+  for the resolution-profile bisection — same seed + same input ⇒
+  byte-equal parquets across runs.
+- **Communities = `leidenalg` + CPM.** Reference Leiden implementation
+  (`leidenalg>=0.11.0`) over an `igraph>=1.0` mutual-kNN cosine graph
+  (k=15, edge floor 0.3) of session centroids. CPM γ has direct cosine
+  semantics (internal density ≥ γ, external ≤ γ); auto-γ via
+  `Optimiser.resolution_profile` + longest-plateau picker; warn-only
+  connectivity check. Output is signal-rich: `is_medoid`, `coherence`,
+  `gamma_used` per row, plus a `community_profile` sidecar with one row
+  per γ tested, so an LLM agent can ask "what γ would yield 50
+  communities?" without rerunning Leiden. Top terms per community come
+  from the live `community_top_topics(cid, n)` macro composed from
+  `cluster_terms`, not a frozen column. See
+  [`docs/research_notes.md`](docs/research_notes.md) for the
+  Louvain → Leiden+CPM swap rationale.
 - **Hybrid friction pipeline.** A hand-curated regex bank catches the
   unambiguous `status_ping` / `interruption` / `correction` cases at
   zero Bedrock cost; the ambiguous class — especially

{claude_sql-0.4.0 → claude_sql-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "claude-sql"
-version = "0.4.0"
+version = "0.5.0"
 description = "Zero-copy SQL + semantic search + LLM analytics over ~/.claude/ transcripts."
 readme = "README.md"
 license = { text = "Apache-2.0" }
@@ -33,8 +33,9 @@ dependencies = [
     "cyclopts>=4.10.2",
     "duckdb>=1.5.2",
     "hdbscan>=0.8.40",
+    "igraph>=1.0.0,<2.0",
+    "leidenalg>=0.11.0,<0.12",
     "loguru>=0.7.3",
-    "networkx>=3.4",
     "numpy>=2.4.4",
     "packaging>=26.2",
     "polars>=1.40.0",

{claude_sql-0.4.0 → claude_sql-0.5.0}/src/claude_sql/cli.py

@@ -50,7 +50,11 @@ from claude_sql import (
     ungrounded_worker as _ungrounded_worker,
 )
 from claude_sql.cluster_worker import run_clustering
-from claude_sql.community_worker import run_communities
+from claude_sql.community_worker import (
+    ResolutionLevel,
+    neighbors_of,
+    run_communities,
+)
 from claude_sql.config import Settings
 from claude_sql.embed_worker import embed_query, run_backfill
 from claude_sql.friction_worker import detect_user_friction
@@ -96,7 +100,7 @@ Surfaces at a glance
   embed / search                  Cohere Embed v4 + HNSW cosine search
   classify / trajectory /         Sonnet 4.6 analytics -- each defaults to
   conflicts / friction            --dry-run; pass --no-dry-run to spend
-  cluster / terms / community     UMAP+HDBSCAN, c-TF-IDF, Louvain
+  cluster / terms / community     UMAP+HDBSCAN, c-TF-IDF, Leiden+CPM
   analyze                         composite pipeline over every stage above
 
 Flag placement (important for agents)
@@ -678,6 +682,7 @@ def list_cache(*, common: Common | None = None) -> None:
         _describe_cache_entry("message_clusters", settings.clusters_parquet_path),
         _describe_cache_entry("cluster_terms", settings.cluster_terms_parquet_path),
         _describe_cache_entry("session_communities", settings.communities_parquet_path),
+        _describe_cache_entry("community_profile", settings.community_profile_parquet_path),
         _describe_cache_entry("user_friction", settings.user_friction_parquet_path),
         _describe_cache_entry("skills_catalog", settings.skills_catalog_parquet_path),
         _describe_checkpoint_entry(settings.checkpoint_db_path),
@@ -1480,33 +1485,137 @@ def terms(*, force: bool = False, common: Common | None = None) -> None:
 
 
 @app.command
-def community(*, force: bool = False, common: Common | None = None) -> None:
-    """Session-level Louvain community detection over a cosine-similarity graph.
+def community(
+    *,
+    force: bool = False,
+    gamma: float | None = None,
+    resolution: ResolutionLevel = "medium",
+    neighbors_of_session: Annotated[str | None, Parameter(name=["--neighbors-of"])] = None,
+    top_k: int = 15,
+    dry_run: bool = False,
+    common: Common | None = None,
+) -> None:
+    """Session-level Leiden+CPM community detection over a mutual-kNN cosine graph.
 
     Prereq: ``embed`` (needs the embeddings parquet).
 
     Output columns (``session_communities`` view)
     ---------------------------------------------
-    session_id, community_id (int; -1 = isolated).
-
-    Method: build a session-centroid-cosine KNN graph, then run
-    ``networkx.community.louvain_communities`` (networkx ≥3.4).
-    Cost: zero. Seeded by ``CLAUDE_SQL_SEED=42``.
+    session_id, community_id (int; -1 = noise / sub-min-size),
+    size (int), is_medoid (bool — best representative session of its
+    community), coherence (float — mean intra-community cosine),
+    gamma_used (float — the CPM γ used at run time).
+
+    Sidecar (``community_profile`` view, written when auto-γ runs)
+    --------------------------------------------------------------
+    gamma, n_communities, quality, plateau_length — one row per γ tested
+    by ``leidenalg.Optimiser.resolution_profile``.  Lets the agent ask
+    "what γ would yield 50 communities?" without rerunning Leiden.
+
+    Method: build a session-centroid mutual-kNN graph (k=15, edge floor 0.3
+    by default), then ``leidenalg.find_partition`` with
+    ``CPMVertexPartition``.  CPM γ is auto-picked from the resolution
+    profile via the longest-plateau heuristic (Traag et al.); the
+    ``--resolution {coarse, medium, fine}`` flag picks alternate plateaus
+    of the same profile (no extra Leiden runs).
+
+    Cost: zero (CPU only).  Seeded by ``CLAUDE_SQL_SEED=42``.  For top
+    terms per community, run
+    ``claude-sql query "SELECT * FROM community_top_topics(<cid>, 10)"``.
 
     Flags
     -----
-    --force   Re-detect even if session_communities.parquet exists.
+    --force                 Re-detect even if session_communities.parquet exists.
+    --gamma FLOAT           Explicit CPM γ; skips the resolution profile + sidecar.
+                            Mutually exclusive with --resolution / --force / --neighbors-of.
+    --resolution {coarse,medium,fine}
+                            Pick a γ plateau without specifying a value.
+                            Default 'medium' = longest plateau.  Ignored if --gamma set.
+    --neighbors-of SID      Early-return path: skip Leiden, return top-k cosine
+                            neighbors of SID.  Reads centroids on the fly +
+                            joins session_communities.parquet if it exists.
+    --top-k N               Used with --neighbors-of (default 15).
+    --dry-run               Plan-only: count candidate sessions via SQL, do not
+                            run Leiden.  Honors agent JSON output for free.
+
+    Exit codes
+    ----------
+    0   success
+    64  invalid input (e.g., --neighbors-of combined with partition flags)
     """
     _configure(common)
     settings = _resolve_settings(common)
+    fmt = _fmt(common)
+
+    if neighbors_of_session is not None and (gamma is not None or force or dry_run):
+        err = ClassifiedError(
+            kind="invalid_input",
+            exit_code=EXIT_CODES["invalid_input"],
+            message=(
+                "--neighbors-of is mutually exclusive with --gamma, --force, "
+                "and --dry-run; pass only --neighbors-of and --top-k."
+            ),
+            hint="Run `claude-sql community --neighbors-of <sid> --top-k 15` alone.",
+        )
+        emit_error(err, fmt)
+        sys.exit(err.exit_code)
+
     con = _open_connection(settings)
     try:
-        stats = run_communities(con, settings, force=force)
+        if neighbors_of_session is not None:
+            df = neighbors_of(con, settings, neighbors_of_session, top_k=top_k)
+            emit_dataframe(df, fmt, table_rows=top_k)
+            return
+
+        if dry_run:
+            row = con.execute(
+                """
+                SELECT COUNT(DISTINCT m.session_id) AS candidate_sessions
+                  FROM read_parquet(?) e
+                  JOIN messages m
+                    ON CAST(m.uuid AS VARCHAR) = e.uuid
+                """,
+                [str(settings.embeddings_parquet_path)],
+            ).fetchone()
+            n = int(row[0]) if row else 0
+            plan: dict[str, object] = {
+                "pipeline": "community",
+                "candidate_sessions": n,
+                "knn_k": settings.leiden_knn_k,
+                "edge_floor": settings.leiden_edge_floor,
+                "min_community_size": settings.leiden_min_community_size,
+                "gamma": gamma if gamma is not None else "auto",
+                "resolution": resolution,
+                "would_write": [
+                    str(settings.communities_parquet_path),
+                ]
+                + ([] if gamma is not None else [str(settings.community_profile_parquet_path)]),
+                "dry_run": True,
+            }
+            emit_json(plan, fmt)
+            return
+
+        stats = run_communities(
+            con,
+            settings,
+            force=force,
+            gamma=gamma,
+            resolution=resolution,
+        )
+        import math
+
+        quality_val = stats["quality"]
+        quality_log = (
+            quality_val if isinstance(quality_val, float) and not math.isnan(quality_val) else 0.0
+        )
         logger.info(
-            "community: {} sessions grouped into {} communities",
+            "community: {} sessions, {} communities (γ={:.4f}, quality={:.4f})",
             stats["sessions"],
             stats["communities"],
+            stats["gamma_used"],
+            quality_log,
         )
+        _emit_worker_result(stats, common, pipeline="community")
     finally:
         con.close()
 
@@ -1538,7 +1647,7 @@ def analyze(
     1. embed        (Bedrock Cohere Embed v4; honors --dry-run)
     2. cluster      (UMAP+HDBSCAN; zero-cost; --force_cluster to rebuild)
     3. terms        (c-TF-IDF labels for clusters; zero-cost)
-    4. community    (Louvain; zero-cost; --force_community to rebuild)
+    4. community    (Leiden+CPM; zero-cost; --force-community to rebuild)
     5. classify     (Sonnet 4.6; honors --dry-run)
     6. trajectory   (Sonnet 4.6; honors --dry-run)
     7. conflicts    (Sonnet 4.6; honors --dry-run)