topiclayers 0.2.1__tar.gz → 0.3.0__tar.gz

{topiclayers-0.2.1 → topiclayers-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topiclayers
-Version: 0.2.1
+Version: 0.3.0
 Summary: From any collection of posts to topical multilayer networks — hardened and social-scientist-friendly
 License: MIT
 License-File: LICENSE
@@ -103,7 +103,7 @@ Stage ETA: ~35m 20s
 Total elapsed: 21m 05s
 ```
 
-Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/cache/stage_timings.json`.
+Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/stage_timings.json`. While running, the terminal itself shows a live progress line (spinner, counter, ETA, elapsed time) that keeps ticking even in stages without a counter, plus a per-stage duration summary when the run finishes.
 
 ## Topic labeling with an LLM (optional)
 
@@ -180,12 +180,7 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
 <output_dir>/
 ├── run_manifest.json              # Reproducibility metadata
 ├── run_status.json                # Live stage/ETA tracking (while running)
-├── cache/
-│   └── data/
-│       ├── tweets_<name>.pkl/.csv          # Full tweet table
-│       ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
-│       └── manifest_<name>.json            # Cache validity key
-│   └── stage_timings.json         # Per-stage durations for ETA prediction
+├── stage_timings.json             # Per-stage durations for ETA prediction
 └── networks/
     ├── <name>_retweet.gml                  # Single-layer retweet network
     ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
@@ -194,13 +189,28 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
         └── <name>__prj_<topic>.gml         # Per-topic projected networks
 ```
 
+Reusable artifacts (processed data, embeddings, the BERTopic model) go to the
+**cache**, not the output dir. By default the cache lives in a `cache/` folder
+**next to the input data file**, so it persists across runs regardless of
+`output_dir` and can be shared between configs (files are prefixed with the
+dataset `name`). Override the location with `cache_dir:` in the config:
+
+```
+<cache_dir>/
+├── data/
+│   ├── tweets_<name>.pkl/.csv          # Full tweet table
+│   ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
+│   └── manifest_<name>.json            # Cache validity key
+└── tm/<embedder_name>/
+    ├── embeddings_<name>.pkl           # Document embeddings
+    ├── topics_<name>.csv               # Topic table (with LLM labels)
+    └── model_<name>/                   # BERTopic safetensors dir
+```
+
 ## Requirements
 
 - Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
 - Optional: Ollama + `topiclayers[labeling]` extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)
 
-## Planned: JOSS software paper
 
-Once the API stabilises and the PLOS ONE core paper results are regenerated with this library,
-a short JOSS (Journal of Open Source Software) paper will be submitted with a Zenodo DOI.
 

{topiclayers-0.2.1 → topiclayers-0.3.0}/README.md

@@ -67,7 +67,7 @@ Stage ETA: ~35m 20s
 Total elapsed: 21m 05s
 ```
 
-Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/cache/stage_timings.json`.
+Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/stage_timings.json`. While running, the terminal itself shows a live progress line (spinner, counter, ETA, elapsed time) that keeps ticking even in stages without a counter, plus a per-stage duration summary when the run finishes.
 
 ## Topic labeling with an LLM (optional)
 
@@ -144,12 +144,7 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
 <output_dir>/
 ├── run_manifest.json              # Reproducibility metadata
 ├── run_status.json                # Live stage/ETA tracking (while running)
-├── cache/
-│   └── data/
-│       ├── tweets_<name>.pkl/.csv          # Full tweet table
-│       ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
-│       └── manifest_<name>.json            # Cache validity key
-│   └── stage_timings.json         # Per-stage durations for ETA prediction
+├── stage_timings.json             # Per-stage durations for ETA prediction
 └── networks/
     ├── <name>_retweet.gml                  # Single-layer retweet network
     ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
@@ -158,12 +153,27 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
         └── <name>__prj_<topic>.gml         # Per-topic projected networks
 ```
 
+Reusable artifacts (processed data, embeddings, the BERTopic model) go to the
+**cache**, not the output dir. By default the cache lives in a `cache/` folder
+**next to the input data file**, so it persists across runs regardless of
+`output_dir` and can be shared between configs (files are prefixed with the
+dataset `name`). Override the location with `cache_dir:` in the config:
+
+```
+<cache_dir>/
+├── data/
+│   ├── tweets_<name>.pkl/.csv          # Full tweet table
+│   ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
+│   └── manifest_<name>.json            # Cache validity key
+└── tm/<embedder_name>/
+    ├── embeddings_<name>.pkl           # Document embeddings
+    ├── topics_<name>.csv               # Topic table (with LLM labels)
+    └── model_<name>/                   # BERTopic safetensors dir
+```
+
 ## Requirements
 
 - Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
 - Optional: Ollama + `topiclayers[labeling]` extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)
 
-## Planned: JOSS software paper
 
-Once the API stabilises and the PLOS ONE core paper results are regenerated with this library,
-a short JOSS (Journal of Open Source Software) paper will be submitted with a Zenodo DOI.

{topiclayers-0.2.1 → topiclayers-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "topiclayers"
-version = "0.2.1"
+version = "0.3.0"
 description = "From any collection of posts to topical multilayer networks — hardened and social-scientist-friendly"
 authors = ["alessiogandelli <alessiogandelli99@gmail.com>"]
 readme = "README.md"

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/cli.py

@@ -45,14 +45,16 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
 
     status.stage("load")
     fmt, loader = resolve_adapter(input_path)
-    print(f"Loading posts via '{fmt}' adapter from {input_path}")
+    status.log(f"Loading posts via '{fmt}' adapter from {input_path}")
     posts = loader(input_path)
-    print(f"  {len(posts)} posts loaded")
+    status.log(f"  {len(posts)} posts loaded")
 
     row_counts = {"raw_posts": len(posts)}
     tm = None
 
-    cache_dir = config.output_dir / "cache"
+    cache_dir = config.cache_dir or (input_path.parent / "cache")
+    cache_dir.mkdir(parents=True, exist_ok=True)
+    status.log(f"Cache: {cache_dir.resolve()}")
 
     status.stage("data", total=len(posts))
     ds = Dataset(posts, n_cop=config.name, file_user=config.input.users)
@@ -60,11 +62,11 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
     row_counts["tweets"] = len(ds.df_tweets)
     row_counts["original"] = len(ds.df_original)
     row_counts["retweets"] = len(ds.df_retweets)
-    print(f"  {len(ds.df_original)} original, {len(ds.df_retweets)} retweets, "
-          f"{len(ds.df_quotes)} quotes, {len(ds.df_reply)} replies")
+    status.log(f"  {len(ds.df_original)} original, {len(ds.df_retweets)} retweets, "
+               f"{len(ds.df_quotes)} quotes, {len(ds.df_reply)} replies")
 
     if len(ds.df_original) == 0:
-        print("  No original posts found — skipping topic modeling. All retweets will be assigned topic -1.")
+        status.log("  No original posts found — skipping topic modeling. All retweets will be assigned topic -1.")
         df_labeled = ds.df_original.copy()
         df_labeled["topic"] = -1
         df_labeled["topic_prob"] = None
@@ -88,15 +90,15 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
                 umap_init=config.topic_model.umap_init,
             )
         except RuntimeError as e:
-            print(f"  Topic modeling failed: {e}")
-            print("  Assigning topic -1 to all originals and continuing.")
+            status.log(f"  Topic modeling failed: {e}")
+            status.log("  Assigning topic -1 to all originals and continuing.")
             df_labeled = ds.df_original.copy()
             df_labeled["topic"] = -1
             df_labeled["topic_prob"] = None
         row_counts["labeled_originals"] = len(df_labeled)
         topic_counts = df_labeled["topic"].value_counts().to_dict()
         topic_dist = {str(k): int(v) for k, v in topic_counts.items()}
-        print(f"  {len(topic_dist)} topics found (including -1 outlier)")
+        status.log(f"  {len(topic_dist)} topics found (including -1 outlier)")
 
     status.stage("label")
     if config.topic_model.label_model and tm is not None and tm.model is not None:
@@ -104,8 +106,9 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
             model=config.topic_model.label_model,
             api_base=config.topic_model.label_api_base,
             label_context=config.topic_model.label_context,
+            reporter=status,
         )
-    df_labeled = ds.update_df(df_labeled)
+    df_labeled = ds.update_df(df_labeled, reporter=status)
     ds._save_labeled_dataframe(cache_dir)
     row_counts["retweets_labeled"] = len(df_labeled)
 
@@ -120,7 +123,7 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
         try:
             nw.create_retweet_ml(reporter=status)
         except ValueError as e:
-            print(f"  Skipping multilayer network: {e}")
+            status.log(f"  Skipping multilayer network: {e}")
     elif network_type == "retweet":
         nw.create_retweet_network(reporter=status)
     elif network_type == "ttn":
@@ -130,11 +133,11 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
         try:
             nw.create_retweet_ml(reporter=status)
         except ValueError as e:
-            print(f"  Skipping multilayer network: {e}")
+            status.log(f"  Skipping multilayer network: {e}")
         nw.create_ttnetwork(project=config.network.project_ttn)
     else:
-        print(f"Unknown network type: '{network_type}'. "
-              f"Use one of: multilayer_repost, retweet, ttn, all.", file=sys.stderr)
+        status.log(f"Unknown network type: '{network_type}'. "
+                    f"Use one of: multilayer_repost, retweet, ttn, all.")
         sys.exit(1)
 
     status.stage("manifest")
@@ -167,7 +170,6 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
     print(f"\nDone. Output in {config.output_dir.resolve()}")
     print(f"  Run manifest: {config.output_dir / 'run_manifest.json'}")
 
-
 def _cmd_init(dest: Path) -> None:
     import shutil
     from importlib.resources import as_file, files

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/config.py

@@ -57,6 +57,12 @@ class RunConfig:
     topic_model: TopicModelConfig = field(default_factory=TopicModelConfig)
     network: NetworkConfig = field(default_factory=NetworkConfig)
     output_dir: Path = Path("./out")
+    # Where reusable artifacts (processed data, embeddings, models) live.
+    # Defaults to a 'cache/' folder next to the input data, so the cache
+    # survives output_dir changes and is shared across runs. Independent
+    # datasets can share one cache dir: all cache files are prefixed with
+    # the dataset `name`.
+    cache_dir: Optional[Path] = None
 
     @classmethod
     def from_yaml(cls, path: Path) -> "RunConfig":
@@ -75,10 +81,21 @@ class RunConfig:
                     p = (path.parent / p).resolve()
                 input_raw[key] = p
 
+        cache_dir = raw.get("cache_dir")
+        if cache_dir:
+            cache_dir = Path(cache_dir)
+            if not cache_dir.is_absolute():
+                cache_dir = (path.parent / cache_dir).resolve()
+        else:
+            # default: keep the cache next to the input data
+            posts = input_raw.get("posts")
+            cache_dir = posts.parent / "cache" if posts else Path("./cache")
+
         return cls(
             input=InputConfig(**input_raw),
             name=raw.get("name", "default"),
             topic_model=TopicModelConfig(**raw.get("topic_model", {})),
             network=NetworkConfig(**raw.get("network", {})),
             output_dir=Path(raw.get("output_dir", "./out")),
+            cache_dir=cache_dir,
         )

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/core/data.py

@@ -10,6 +10,7 @@ from typing import Optional
 import pandas as pd
 
 from topiclayers.support.cache import cache_key
+from topiclayers.status import log_via
 from topiclayers.schema import InteractionType, Post
 
 _INTERACTION_REVERSE_MAP = {
@@ -82,8 +83,8 @@ class Dataset:
                 self.df_users = pd.read_pickle(users_pkl)
         else:
             if tweets_pkl.exists():
-                print("Cache key mismatch — regenerating data cache.")
-            users_dict = self._load_user_dict() if self.file_user is not None else {}
+                log_via(reporter, "Cache key mismatch — regenerating data cache.")
+            users_dict = self._load_user_dict(reporter=reporter) if self.file_user is not None else {}
             tweets_dict = {}
             for i, post in enumerate(self.posts):
                 if reporter is not None:
@@ -135,13 +136,13 @@ class Dataset:
 
         self._create_dataframes()
 
-    def _load_user_dict(self) -> dict:
+    def _load_user_dict(self, reporter=None) -> dict:
         import json
 
         if self.file_user is None:
             return {}
         if not self.file_user.exists():
-            print(f"User file not found: {self.file_user} — skipping user metadata.")
+            log_via(reporter, f"User file not found: {self.file_user} — skipping user metadata.")
             return {}
 
         users = {}
@@ -167,7 +168,7 @@ class Dataset:
         self.df_quotes = df[df["referenced_type"] == "quoted"].copy()
         self.df_reply = df[df["referenced_type"] == "replied_to"].copy()
 
-    def update_df(self, df_labeled: pd.DataFrame) -> pd.DataFrame:
+    def update_df(self, df_labeled: pd.DataFrame, reporter=None) -> pd.DataFrame:
         if "id" in df_labeled.columns and df_labeled.index.name != "id":
             df_labeled = df_labeled.set_index("id")
 
@@ -197,9 +198,9 @@ class Dataset:
             resolved[idx] = val
 
         self.df_retweets_labeled["topic"] = self.df_retweets_labeled.index.map(resolved)
-        print(f"Unresolved references: {unresolved_count}")
+        log_via(reporter, f"Unresolved references: {unresolved_count}")
         if unresolved_count > 0:
-            print(f"  {unresolved_count} retweets could not be resolved to a topic. These will be excluded from the labeled output.")
+            log_via(reporter, f"  {unresolved_count} retweets could not be resolved to a topic. These will be excluded from the labeled output.")
 
         self.df_retweets_labeled = self.df_retweets_labeled[
             self.df_retweets_labeled["topic"].apply(lambda x: not isinstance(x, str))

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/core/network.py

@@ -5,6 +5,8 @@ from pathlib import Path
 import networkx as nx
 import pandas as pd
 
+from topiclayers.status import log_via
+
 
 class NetworkCreator:
     def __init__(self, df: pd.DataFrame, name: str, output_dir: Path):
@@ -25,7 +27,7 @@ class NetworkCreator:
         )
 
         if missing_refs:
-            print(f"  {missing_refs} retweets reference tweets not in dataset — skipped.")
+            log_via(reporter, f"  {missing_refs} retweets reference tweets not in dataset — skipped.")
 
         self._save_graph(G, "retweet")
         return G
@@ -101,7 +103,7 @@ class NetworkCreator:
             ml.add_nx_layer(ml_network, G, str(topic))
 
         if dropped_edges_total:
-            print(f"  {dropped_edges_total} edges dropped (referenced tweet missing in topic slice).")
+            log_via(reporter, f"  {dropped_edges_total} edges dropped (referenced tweet missing in topic slice).")
 
         self._save_multilayer(ml_network)
         self.ml_network = ml_network

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/core/topic.py

@@ -11,6 +11,7 @@ import numpy as np
 import pandas as pd
 
 from topiclayers.support.cache import cache_key
+from topiclayers.status import log_via
 
 
 class TopicModeler:
@@ -77,7 +78,7 @@ class TopicModeler:
                 except Exception:
                     pass
             if cache_valid:
-                print("using cached topics")
+                log_via(reporter, "using cached topics")
                 self.df = pd.read_pickle(self.df_labeled_path)
                 from bertopic import BERTopic
                 self.model = BERTopic.load(self.model_path)
@@ -85,7 +86,7 @@ class TopicModeler:
                     self.docs = self.df["new_text"].tolist()
                 return self.df
 
-        print("running topic modeling")
+        log_via(reporter, "running topic modeling")
         docs = self._preprocess()
 
         if self.embeddings_path.exists():
@@ -101,7 +102,7 @@ class TopicModeler:
                 except Exception:
                     pass
             if use_cached:
-                print("  loading embeddings from cache")
+                log_via(reporter, "  loading embeddings from cache")
                 with open(self.embeddings_path, "rb") as f:
                     self.embeddings = pickle.load(f)
             else:
@@ -214,6 +215,11 @@ class TopicModeler:
         repro_mode: str = "strict",
         umap_init: str = "pca",
     ) -> None:
+        # mark the stage before importing: torch/umap/bertopic imports take a
+        # while and would otherwise be billed to the previous stage
+        if reporter is not None:
+            reporter.stage("topic_model")
+
         import random
 
         from umap import UMAP
@@ -222,9 +228,6 @@ class TopicModeler:
         from bertopic import BERTopic
         from bertopic.vectorizers import ClassTfidfTransformer
 
-        if reporter is not None:
-            reporter.stage("topic_model")
-
         np.random.seed(random_state)
         random.seed(random_state)
         try:
@@ -293,7 +296,9 @@ class TopicModeler:
             # large corpora); the default computes the probability of the
             # assigned topic only — topics themselves are identical
             calculate_probabilities=exact_probabilities,
-            verbose=True,
+            # False: BERTopic's per-step INFO logging would clobber the live
+            # progress line; our reporter provides the progress feedback
+            verbose=False,
         )
 
         try:
@@ -350,7 +355,7 @@ class TopicModeler:
     def _save_qdrant(self, docs: list[str], reporter=None) -> None:
         qdrant_url = os.getenv("QDRANT_URL")
         if not qdrant_url:
-            print("Qdrant not configured — skipping. Set QDRANT_URL in .env to enable vector search.")
+            log_via(reporter, "Qdrant not configured — skipping. Set QDRANT_URL in .env to enable vector search.")
             return
 
         try:
@@ -386,8 +391,8 @@ class TopicModeler:
             if reporter is not None:
                 reporter.update(len(points), total=len(points))
         except Exception as e:
-            print(f"Qdrant upload failed (non-fatal): {type(e).__name__}: {e}")
-            print("The pipeline continues — networks and topic labels are unaffected.")
+            log_via(reporter, f"Qdrant upload failed (non-fatal): {type(e).__name__}: {e}")
+            log_via(reporter, "The pipeline continues — networks and topic labels are unaffected.")
 
     def _save_files(self) -> None:
         self.df_labeled_path.parent.mkdir(parents=True, exist_ok=True)
@@ -409,6 +414,7 @@ class TopicModeler:
         batch_size: int = 40,
         label_context: str = "",
         max_topics: int = 0,
+        reporter=None,
     ) -> dict[int, str]:
         """Label topics with an LLM, batching many topics per request.
 
@@ -443,18 +449,19 @@ class TopicModeler:
         api_base = api_base or os.getenv("LABEL_API_BASE", "")
 
         if self.model is None:
-            print("  No fitted topic model — skipping LLM labeling.")
+            log_via(reporter, "  No fitted topic model — skipping LLM labeling.")
             return {}
 
         topic_info = self.model.get_topic_info()
         if "LLM_label" in topic_info.columns and not overwrite:
-            print("  LLM labels already present — skipping (use overwrite=True).")
+            log_via(reporter, "  LLM labels already present — skipping (use overwrite=True).")
             return {}
 
         try:
             import litellm  # noqa: F401
         except ImportError:
-            print(
+            log_via(
+                reporter,
                 "  LLM labeling skipped: 'litellm' is not installed.\n"
                 "    Install it with: pip install 'topiclayers[labeling]'\n"
                 "    Topics keep their c-TF-IDF keyword labels."
@@ -470,7 +477,8 @@ class TopicModeler:
                 conn.getresponse().read()
                 conn.close()
             except (OSError, socket.timeout):
-                print(
+                log_via(
+                    reporter,
                     f"  LLM labeling skipped: Ollama not reachable at {host}:{port}.\n"
                     f"    Start it with: ollama serve   (then pull the model:\n"
                     f"    ollama pull {model.split('/', 1)[1]})\n"
@@ -486,7 +494,8 @@ class TopicModeler:
                 or os.getenv("OPENAI_API_KEY")
             )
             if not api_key:
-                print(
+                log_via(
+                    reporter,
                     "  LLM labeling skipped: custom api_base set but no API key found.\n"
                     "    Set LABEL_API_KEY (or OPENAI_LIKE_API_KEY / OPENAI_API_KEY) in .env."
                 )
@@ -515,12 +524,12 @@ class TopicModeler:
             entries.append((tid, keywords, docs_str))
 
         if not entries:
-            print("  No non-outlier topics to label.")
+            log_via(reporter, "  No non-outlier topics to label.")
             return {}
 
         if max_topics and max_topics < len(entries):
             entries = entries[:max_topics]  # topic_info is sorted by size
-            print(f"  Preview mode: labeling only the {max_topics} largest topics.")
+            log_via(reporter, f"  Preview mode: labeling only the {max_topics} largest topics.")
 
         system_msg = (
             "You annotate discussion topics from social media.\n"
@@ -577,16 +586,17 @@ class TopicModeler:
                         _load_openai_like_model(model, api_base, generator_kwargs.get("api_key", ""))
                         continue
                     if attempt == 1:
-                        print(f"  Batch {i // batch_size + 1}/{n_chunks}: {type(e).__name__}, retrying once.")
+                        log_via(reporter, f"  Batch {i // batch_size + 1}/{n_chunks}: {type(e).__name__}, retrying once.")
                         continue
                     failed += 1
-                    print(
+                    log_via(
+                        reporter,
                         f"  Batch {i // batch_size + 1}/{n_chunks} failed (non-fatal): "
                         f"{type(e).__name__}: {str(e)[:120]} — those topics keep keyword labels."
                     )
 
         if not llm_labels:
-            print("  No labels generated — topics keep their c-TF-IDF keyword labels.")
+            log_via(reporter, "  No labels generated — topics keep their c-TF-IDF keyword labels.")
             return {}
 
         # expose labels as an aspect column named LLM_label in get_topic_info()
@@ -609,7 +619,7 @@ class TopicModeler:
         info.to_csv(self.topic_path)
         ok = len(llm_labels)
         suffix = f", {failed} batch(es) failed" if failed else ""
-        print(f"  Labeled {ok} topics with '{model}' in {n_chunks} batch(es){suffix}.")
+        log_via(reporter, f"  Labeled {ok} topics with '{model}' in {n_chunks} batch(es){suffix}.")
         return labels
 
 

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/examples/template_csv.yml

@@ -41,4 +41,9 @@ network:
                                  # interaction_type=repost rows to build edges
   project_ttn: true
 
-output_dir: ./out/my_dataset
+output_dir: ./out/my_dataset     # research outputs only: networks/, run_manifest.json,
+                                 # run_status.json, stage_timings.json
+
+# cache_dir: ./cache             # optional — where reusable artifacts live (processed
+                                 # data, embeddings, topic model). Defaults to a
+                                 # 'cache/' folder next to the input data file.

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/examples/template_twitter.yml

@@ -79,6 +79,12 @@ network:
   project_ttn: true              # ttn only: also export the projected monopartite
                                  # version
 
-output_dir: ./out/cop22          # everything lands here: networks/, cache/
-                                 # (embeddings, model, labeled df), run_manifest.json,
-                                 # run_status.json
+output_dir: ./out/cop22          # research outputs only: networks/, run_manifest.json,
+                                 # run_status.json, stage_timings.json
+
+# cache_dir: ./cache              # optional — where reusable artifacts live (processed
+                                  # data, embeddings, topic model). Defaults to a
+                                  # 'cache/' folder next to the input data file, so it
+                                  # persists regardless of output_dir and is shared
+                                  # between runs; safe to share across datasets (files
+                                  # are prefixed with `name`)

{topiclayers-0.2.1 → topiclayers-0.3.0}/src/topiclayers/status.py

@@ -1,9 +1,19 @@
-"""Run status: live stage tracking and ETA for long-running pipeline runs."""
+"""Run status: live stage tracking and ETA for long-running pipeline runs.
+
+Writes `run_status.json` (machine-readable, consumed by `topiclayers status`)
+and, when attached to a terminal, renders live console output: colored stage
+banners, an in-place progress line with ETA, and a per-stage summary at the
+end. A heartbeat keeps the elapsed time ticking (and the status file fresh)
+during opaque stages that never call update(), so a silent stage is
+distinguishable from a stuck one.
+"""
 
 from __future__ import annotations
 
 import json
 import os
+import shutil
+import sys
 import threading
 import time
 from datetime import datetime, timezone
@@ -14,15 +24,32 @@ STAGES = ["load", "data", "embed", "topic_model", "label", "network", "manifest"
 _STATUS_FILE = "run_status.json"
 _TIMINGS_FILE = "stage_timings.json"
 _MIN_UPDATE_INTERVAL = 1.0
+_MIN_RENDER_INTERVAL = 0.2
 _HEARTBEAT_INTERVAL = 5.0
+_SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"]
+
+
+def _supports_ansi() -> bool:
+    return sys.stdout.isatty() and os.environ.get("NO_COLOR") is None
+
+
+def _c(code: str, text: str) -> str:
+    if not _supports_ansi():
+        return text
+    return f"\033[{code}m{text}\033[0m"
+
+
+def _fmt_count(n: int) -> str:
+    return f"{n:,}"
 
 
 class RunStatus:
-    def __init__(self, output_dir: Path, name: str):
+    def __init__(self, output_dir: Path, name: str, console: bool = True):
         self.output_dir = Path(output_dir)
         self.name = name
+        self.console = console
         self.path = self.output_dir / _STATUS_FILE
-        self.timings_path = self.output_dir / "cache" / _TIMINGS_FILE
+        self.timings_path = self.output_dir / _TIMINGS_FILE
         self.current_stage: Optional[str] = None
         self.stage_index: Optional[int] = None
         self.stage_total: Optional[int] = None
@@ -30,20 +57,111 @@ class RunStatus:
         self.started_at = datetime.now(timezone.utc)
         self.stage_started_at: Optional[datetime] = None
         self._last_write = 0.0
+        self._last_render = 0.0
+        self._line_active = False
+        self._spin_idx = 0
+        self._console_lock = threading.Lock()
+        self._stage_durations: dict[str, float] = {}
         self._stop_heartbeat = threading.Event()
         self._heartbeat_thread: Optional[threading.Thread] = None
 
+    # ------------------------------------------------------------------ console
+
+    def log(self, message: str) -> None:
+        """Print a message without clobbering the in-place progress line."""
+        with self._console_lock:
+            self._clear_line_locked()
+            print(message, flush=True)
+
+    def _clear_line_locked(self) -> None:
+        if self._line_active:
+            sys.stdout.write("\r\033[K")
+            sys.stdout.flush()
+            self._line_active = False
+
+    def _render(self, force: bool = False) -> None:
+        # in-place progress line; only meaningful on a real terminal
+        if not self.console or not sys.stdout.isatty():
+            return
+        now = time.monotonic()
+        if not force and now - self._last_render < _MIN_RENDER_INTERVAL:
+            return
+        self._last_render = now
+
+        with self._console_lock:
+            self._spin_idx += 1
+            spin = _c("2", _SPINNER[self._spin_idx % len(_SPINNER)])
+            stage = _c("36", self.current_stage) if self.current_stage else ""
+            bits = []
+            if (
+                self.processed is not None
+                and self.stage_total
+                and self.stage_total > 0
+            ):
+                pct = 100 * self.processed / self.stage_total
+                bits.append(
+                    f"{_fmt_count(self.processed)}/{_fmt_count(self.stage_total)}"
+                    f" ({pct:.0f}%)"
+                )
+                eta = self._eta_seconds()
+                if eta is not None:
+                    bits.append(f"eta ~{_c('33', format_seconds(eta))}")
+            else:
+                bits.append(_c("2", "running…"))
+            if self.stage_started_at is not None:
+                elapsed = (datetime.now(timezone.utc) - self.stage_started_at).total_seconds()
+                bits.append(f"elapsed {format_seconds(elapsed)}")
+
+            line = f"{spin} {stage}  " + _c("2", "·") + " " + f"  {_c('2', '·')}  ".join(bits)
+            width = shutil.get_terminal_size((80, 20)).columns
+            if len(line) + 8 > width:
+                line = line[: width - 8]
+            sys.stdout.write("\r\033[K" + line)
+            sys.stdout.flush()
+            self._line_active = True
+
+    def _banner(self, name: str, total: Optional[int], hist: Optional[float]) -> None:
+        with self._console_lock:
+            self._clear_line_locked()
+            if not self.console:
+                return
+            idx = STAGES.index(name) + 1 if name in STAGES else None
+            head = f"[{idx}/{len(STAGES)}] {name}" if idx else name
+            parts = [_c("2", "──"), _c("36;1", head)]
+            if total is not None:
+                parts.append(_c("2", f"── {_fmt_count(total)} items"))
+            if hist is not None:
+                parts.append(_c("2", f"── past runs ~{format_seconds(hist)}"))
+            print(" ".join(parts), flush=True)
+
+    def _summary(self, status: str) -> None:
+        with self._console_lock:
+            self._clear_line_locked()
+            if not self.console:
+                return
+            total = (datetime.now(timezone.utc) - self.started_at).total_seconds()
+            mark = _c("32", "✓") if status == "success" else _c("31", "✗")
+            print(f"\n{mark} run {status} in {format_seconds(total)}")
+            if self._stage_durations:
+                print(_c("2", "  per-stage:"))
+                for stage, duration in self._stage_durations.items():
+                    print(_c("2", f"    {stage:<14} {format_seconds(duration):>10}"))
+
+    # ------------------------------------------------------------------- status
+
     def start(self) -> None:
         self._start_heartbeat()
         self._write(status="running")
 
     def _start_heartbeat(self) -> None:
         # keeps stage_elapsed_seconds fresh during opaque stages (e.g. UMAP +
-        # HDBSCAN) that never call update()
+        # HDBSCAN) that never call update(); also re-renders the console line
+        # with an advancing spinner so silence is visibly not a hang
         def beat():
             while not self._stop_heartbeat.wait(_HEARTBEAT_INTERVAL):
                 if self.current_stage is not None:
                     self._write(status="running", force=True)
+                    self._render(force=True)
 
         self._heartbeat_thread = threading.Thread(target=beat, daemon=True)
         self._heartbeat_thread.start()
@@ -59,15 +177,20 @@ class RunStatus:
         self.processed = 0
         self.stage_started_at = now
         self._write(status="running", force=True)
+        if self.console:
+            self._banner(name, total, historical_eta(self.timings_path, name))
+            self._last_render = time.monotonic()
 
     def update(self, processed: int, total: Optional[int] = None) -> None:
         self.processed = processed
         if total is not None:
             self.stage_total = total
         self._write(status="running", throttle=True)
+        self._render()
 
     def flush(self) -> None:
         self._write(status="running", force=True)
+        self._render(force=True)
 
     def finish(self, status: str = "success") -> None:
         self._stop_heartbeat.set()
@@ -75,6 +198,10 @@ class RunStatus:
             duration = (datetime.now(timezone.utc) - self.stage_started_at).total_seconds()
             self._record_timing(self.current_stage, duration)
         self._write(status=status, force=True)
+        if self.console:
+            self._summary(status)
+
+    # ------------------------------------------------------------------ helpers
 
     def _eta_seconds(self) -> Optional[float]:
         if (
@@ -120,6 +247,7 @@ class RunStatus:
         os.replace(tmp, self.path)
 
     def _record_timing(self, stage: str, duration: float) -> None:
+        self._stage_durations[stage] = duration
         self.timings_path.parent.mkdir(parents=True, exist_ok=True)
         records: list[dict] = []
         if self.timings_path.exists():
@@ -139,6 +267,15 @@ class RunStatus:
         os.replace(tmp, self.timings_path)
 
 
+def log_via(reporter, message: str) -> None:
+    """Print a message through the reporter's console line (plain print fallback)."""
+    log_fn = getattr(reporter, "log", None)
+    if log_fn is not None:
+        log_fn(message)
+    else:
+        print(message)
+
+
 def load_status(path: Path) -> dict:
     with open(path) as f:
         return json.load(f)
@@ -202,7 +339,7 @@ def print_status(output_dir: Path) -> int:
         if eta is not None:
             print(f"Stage ETA: ~{format_seconds(eta)}")
         else:
-            hist = historical_eta(output_dir / "cache" / _TIMINGS_FILE, stage)
+            hist = historical_eta(output_dir / _TIMINGS_FILE, stage)
             if hist is not None:
                 print(f"No live ETA for this stage; past runs took ~{format_seconds(hist)} (median).")
             else: