topiclayers 0.2.0__tar.gz → 0.3.0__tar.gz

{topiclayers-0.2.0 → topiclayers-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: topiclayers
-Version: 0.2.0
+Version: 0.3.0
 Summary: From any collection of posts to topical multilayer networks — hardened and social-scientist-friendly
 License: MIT
 License-File: LICENSE
@@ -43,6 +43,10 @@ Description-Content-Type: text/markdown
 ```bash
 # Requires Python 3.12
 pip install topiclayers
+
+# copy runnable examples (configs + toy data) into ./examples/
+topiclayers init
+
 topiclayers run examples/generic.yml
 ```
 
@@ -99,7 +103,7 @@ Stage ETA: ~35m 20s
 Total elapsed: 21m 05s
 ```
 
-Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/cache/stage_timings.json`.
+Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/stage_timings.json`. While running, the terminal itself shows a live progress line (spinner, counter, ETA, elapsed time) that keeps ticking even in stages without a counter, plus a per-stage duration summary when the run finishes.
 
 ## Topic labeling with an LLM (optional)
 
@@ -161,7 +165,8 @@ CSV optional columns: `created_at`, `lang`, `interaction_type` (`original`/`repo
 | Symptom | Likely cause | Fix |
 |---------|-------------|-----|
 | "All topics are -1" / "no topics found" | `min_topic_size` too high or dataset too small | Halve `min_topic_size` in config, or use a larger dataset (>100 posts) |
-| "Module not found" | topiclayers not installed | `pip install -e .` |
+| "Module not found" | topiclayers not installed | `pip install topiclayers` |
+| `topiclayers: command not found` | installed in a venv that isn't active | Activate the venv (`source .venv/bin/activate`) or use `python -m topiclayers.cli`; on install with `--user` add `~/.local/bin` to PATH |
 | "OpenAI API key not set" | Using OpenAI embedder without key | Switch `embedder` to `all-MiniLM-L6-v2` (default, works offline) |
 | "Qdrant connection refused" | Qdrant vector DB not running | Ignore — Qdrant is optional. Set `QDRANT_URL` in `.env` to enable |
 | "Cannot create multilayer network" | All posts are outliers | Reduce `min_topic_size` or provide more data |
@@ -175,12 +180,7 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
 <output_dir>/
 ├── run_manifest.json              # Reproducibility metadata
 ├── run_status.json                # Live stage/ETA tracking (while running)
-├── cache/
-│   └── data/
-│       ├── tweets_<name>.pkl/.csv          # Full tweet table
-│       ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
-│       └── manifest_<name>.json            # Cache validity key
-│   └── stage_timings.json         # Per-stage durations for ETA prediction
+├── stage_timings.json             # Per-stage durations for ETA prediction
 └── networks/
     ├── <name>_retweet.gml                  # Single-layer retweet network
     ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
@@ -189,13 +189,28 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
         └── <name>__prj_<topic>.gml         # Per-topic projected networks
 ```
 
+Reusable artifacts (processed data, embeddings, the BERTopic model) go to the
+**cache**, not the output dir. By default the cache lives in a `cache/` folder
+**next to the input data file**, so it persists across runs regardless of
+`output_dir` and can be shared between configs (files are prefixed with the
+dataset `name`). Override the location with `cache_dir:` in the config:
+
+```
+<cache_dir>/
+├── data/
+│   ├── tweets_<name>.pkl/.csv          # Full tweet table
+│   ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
+│   └── manifest_<name>.json            # Cache validity key
+└── tm/<embedder_name>/
+    ├── embeddings_<name>.pkl           # Document embeddings
+    ├── topics_<name>.csv               # Topic table (with LLM labels)
+    └── model_<name>/                   # BERTopic safetensors dir
+```
+
 ## Requirements
 
 - Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
 - Optional: Ollama + `topiclayers[labeling]` extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)
 
-## Planned: JOSS software paper
 
-Once the API stabilises and the PLOS ONE core paper results are regenerated with this library,
-a short JOSS (Journal of Open Source Software) paper will be submitted with a Zenodo DOI.
 

{topiclayers-0.2.0 → topiclayers-0.3.0}/README.md

@@ -7,6 +7,10 @@
 ```bash
 # Requires Python 3.12
 pip install topiclayers
+
+# copy runnable examples (configs + toy data) into ./examples/
+topiclayers init
+
 topiclayers run examples/generic.yml
 ```
 
@@ -63,7 +67,7 @@ Stage ETA: ~35m 20s
 Total elapsed: 21m 05s
 ```
 
-Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/cache/stage_timings.json`.
+Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/stage_timings.json`. While running, the terminal itself shows a live progress line (spinner, counter, ETA, elapsed time) that keeps ticking even in stages without a counter, plus a per-stage duration summary when the run finishes.
 
 ## Topic labeling with an LLM (optional)
 
@@ -125,7 +129,8 @@ CSV optional columns: `created_at`, `lang`, `interaction_type` (`original`/`repo
 | Symptom | Likely cause | Fix |
 |---------|-------------|-----|
 | "All topics are -1" / "no topics found" | `min_topic_size` too high or dataset too small | Halve `min_topic_size` in config, or use a larger dataset (>100 posts) |
-| "Module not found" | topiclayers not installed | `pip install -e .` |
+| "Module not found" | topiclayers not installed | `pip install topiclayers` |
+| `topiclayers: command not found` | installed in a venv that isn't active | Activate the venv (`source .venv/bin/activate`) or use `python -m topiclayers.cli`; on install with `--user` add `~/.local/bin` to PATH |
 | "OpenAI API key not set" | Using OpenAI embedder without key | Switch `embedder` to `all-MiniLM-L6-v2` (default, works offline) |
 | "Qdrant connection refused" | Qdrant vector DB not running | Ignore — Qdrant is optional. Set `QDRANT_URL` in `.env` to enable |
 | "Cannot create multilayer network" | All posts are outliers | Reduce `min_topic_size` or provide more data |
@@ -139,12 +144,7 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
 <output_dir>/
 ├── run_manifest.json              # Reproducibility metadata
 ├── run_status.json                # Live stage/ETA tracking (while running)
-├── cache/
-│   └── data/
-│       ├── tweets_<name>.pkl/.csv          # Full tweet table
-│       ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
-│       └── manifest_<name>.json            # Cache validity key
-│   └── stage_timings.json         # Per-stage durations for ETA prediction
+├── stage_timings.json             # Per-stage durations for ETA prediction
 └── networks/
     ├── <name>_retweet.gml                  # Single-layer retweet network
     ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
@@ -153,12 +153,27 @@ For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`
         └── <name>__prj_<topic>.gml         # Per-topic projected networks
 ```
 
+Reusable artifacts (processed data, embeddings, the BERTopic model) go to the
+**cache**, not the output dir. By default the cache lives in a `cache/` folder
+**next to the input data file**, so it persists across runs regardless of
+`output_dir` and can be shared between configs (files are prefixed with the
+dataset `name`). Override the location with `cache_dir:` in the config:
+
+```
+<cache_dir>/
+├── data/
+│   ├── tweets_<name>.pkl/.csv          # Full tweet table
+│   ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
+│   └── manifest_<name>.json            # Cache validity key
+└── tm/<embedder_name>/
+    ├── embeddings_<name>.pkl           # Document embeddings
+    ├── topics_<name>.csv               # Topic table (with LLM labels)
+    └── model_<name>/                   # BERTopic safetensors dir
+```
+
 ## Requirements
 
 - Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
 - Optional: Ollama + `topiclayers[labeling]` extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)
 
-## Planned: JOSS software paper
 
-Once the API stabilises and the PLOS ONE core paper results are regenerated with this library,
-a short JOSS (Journal of Open Source Software) paper will be submitted with a Zenodo DOI.

{topiclayers-0.2.0 → topiclayers-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "topiclayers"
-version = "0.2.0"
+version = "0.3.0"
 description = "From any collection of posts to topical multilayer networks — hardened and social-scientist-friendly"
 authors = ["alessiogandelli <alessiogandelli99@gmail.com>"]
 readme = "README.md"

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/__init__.py

@@ -1,3 +1,3 @@
 """topiclayers — from posts to topical multilayer networks."""
 
-__version__ = "0.2.0"
+__version__ = "0.2.1"

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/cli.py

@@ -45,14 +45,16 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
 
     status.stage("load")
     fmt, loader = resolve_adapter(input_path)
-    print(f"Loading posts via '{fmt}' adapter from {input_path}")
+    status.log(f"Loading posts via '{fmt}' adapter from {input_path}")
     posts = loader(input_path)
-    print(f"  {len(posts)} posts loaded")
+    status.log(f"  {len(posts)} posts loaded")
 
     row_counts = {"raw_posts": len(posts)}
     tm = None
 
-    cache_dir = config.output_dir / "cache"
+    cache_dir = config.cache_dir or (input_path.parent / "cache")
+    cache_dir.mkdir(parents=True, exist_ok=True)
+    status.log(f"Cache: {cache_dir.resolve()}")
 
     status.stage("data", total=len(posts))
     ds = Dataset(posts, n_cop=config.name, file_user=config.input.users)
@@ -60,11 +62,11 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
     row_counts["tweets"] = len(ds.df_tweets)
     row_counts["original"] = len(ds.df_original)
     row_counts["retweets"] = len(ds.df_retweets)
-    print(f"  {len(ds.df_original)} original, {len(ds.df_retweets)} retweets, "
-          f"{len(ds.df_quotes)} quotes, {len(ds.df_reply)} replies")
+    status.log(f"  {len(ds.df_original)} original, {len(ds.df_retweets)} retweets, "
+               f"{len(ds.df_quotes)} quotes, {len(ds.df_reply)} replies")
 
     if len(ds.df_original) == 0:
-        print("  No original posts found — skipping topic modeling. All retweets will be assigned topic -1.")
+        status.log("  No original posts found — skipping topic modeling. All retweets will be assigned topic -1.")
         df_labeled = ds.df_original.copy()
         df_labeled["topic"] = -1
         df_labeled["topic_prob"] = None
@@ -88,15 +90,15 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
                 umap_init=config.topic_model.umap_init,
             )
         except RuntimeError as e:
-            print(f"  Topic modeling failed: {e}")
-            print("  Assigning topic -1 to all originals and continuing.")
+            status.log(f"  Topic modeling failed: {e}")
+            status.log("  Assigning topic -1 to all originals and continuing.")
             df_labeled = ds.df_original.copy()
             df_labeled["topic"] = -1
             df_labeled["topic_prob"] = None
         row_counts["labeled_originals"] = len(df_labeled)
         topic_counts = df_labeled["topic"].value_counts().to_dict()
         topic_dist = {str(k): int(v) for k, v in topic_counts.items()}
-        print(f"  {len(topic_dist)} topics found (including -1 outlier)")
+        status.log(f"  {len(topic_dist)} topics found (including -1 outlier)")
 
     status.stage("label")
     if config.topic_model.label_model and tm is not None and tm.model is not None:
@@ -104,8 +106,9 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
             model=config.topic_model.label_model,
             api_base=config.topic_model.label_api_base,
             label_context=config.topic_model.label_context,
+            reporter=status,
         )
-    df_labeled = ds.update_df(df_labeled)
+    df_labeled = ds.update_df(df_labeled, reporter=status)
     ds._save_labeled_dataframe(cache_dir)
     row_counts["retweets_labeled"] = len(df_labeled)
 
@@ -120,7 +123,7 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
         try:
             nw.create_retweet_ml(reporter=status)
         except ValueError as e:
-            print(f"  Skipping multilayer network: {e}")
+            status.log(f"  Skipping multilayer network: {e}")
     elif network_type == "retweet":
         nw.create_retweet_network(reporter=status)
     elif network_type == "ttn":
@@ -130,11 +133,11 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
         try:
             nw.create_retweet_ml(reporter=status)
         except ValueError as e:
-            print(f"  Skipping multilayer network: {e}")
+            status.log(f"  Skipping multilayer network: {e}")
         nw.create_ttnetwork(project=config.network.project_ttn)
     else:
-        print(f"Unknown network type: '{network_type}'. "
-              f"Use one of: multilayer_repost, retweet, ttn, all.", file=sys.stderr)
+        status.log(f"Unknown network type: '{network_type}'. "
+                    f"Use one of: multilayer_repost, retweet, ttn, all.")
         sys.exit(1)
 
     status.stage("manifest")
@@ -167,13 +170,37 @@ def _run_pipeline_stages(config: RunConfig, input_path: Path, status) -> None:
     print(f"\nDone. Output in {config.output_dir.resolve()}")
     print(f"  Run manifest: {config.output_dir / 'run_manifest.json'}")
 
+def _cmd_init(dest: Path) -> None:
+    import shutil
+    from importlib.resources import as_file, files
+
+    if dest.exists():
+        print(f"Directory '{dest}' already exists — remove it or pass another path.",
+              file=sys.stderr)
+        sys.exit(1)
+
+    with as_file(files("topiclayers") / "examples") as bundled:
+        shutil.copytree(bundled, dest)
+
+    print(f"Examples copied to {dest.resolve()}")
+    print("Try it:")
+    print(f"  topiclayers run {dest / 'generic.yml'}")
+
 
 def main() -> None:
-    if len(sys.argv) < 3 or sys.argv[1] not in ("run", "status"):
+    if len(sys.argv) < 2 or (
+        sys.argv[1] not in ("run", "status", "init")
+        or (sys.argv[1] != "init" and len(sys.argv) < 3)
+    ):
         print("Usage: topiclayers run <config.yaml>", file=sys.stderr)
         print("       topiclayers status <output_dir>", file=sys.stderr)
+        print("       topiclayers init [dest=./examples]", file=sys.stderr)
         sys.exit(1)
 
+    if sys.argv[1] == "init":
+        _cmd_init(Path(sys.argv[2]) if len(sys.argv) > 2 else Path("examples"))
+        return
+
     if sys.argv[1] == "status":
         from topiclayers.status import print_status
 

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/config.py

@@ -57,6 +57,12 @@ class RunConfig:
     topic_model: TopicModelConfig = field(default_factory=TopicModelConfig)
     network: NetworkConfig = field(default_factory=NetworkConfig)
     output_dir: Path = Path("./out")
+    # Where reusable artifacts (processed data, embeddings, models) live.
+    # Defaults to a 'cache/' folder next to the input data, so the cache
+    # survives output_dir changes and is shared across runs. Independent
+    # datasets can share one cache dir: all cache files are prefixed with
+    # the dataset `name`.
+    cache_dir: Optional[Path] = None
 
     @classmethod
     def from_yaml(cls, path: Path) -> "RunConfig":
@@ -75,10 +81,21 @@ class RunConfig:
                     p = (path.parent / p).resolve()
                 input_raw[key] = p
 
+        cache_dir = raw.get("cache_dir")
+        if cache_dir:
+            cache_dir = Path(cache_dir)
+            if not cache_dir.is_absolute():
+                cache_dir = (path.parent / cache_dir).resolve()
+        else:
+            # default: keep the cache next to the input data
+            posts = input_raw.get("posts")
+            cache_dir = posts.parent / "cache" if posts else Path("./cache")
+
         return cls(
             input=InputConfig(**input_raw),
             name=raw.get("name", "default"),
             topic_model=TopicModelConfig(**raw.get("topic_model", {})),
             network=NetworkConfig(**raw.get("network", {})),
             output_dir=Path(raw.get("output_dir", "./out")),
+            cache_dir=cache_dir,
         )

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/core/data.py

@@ -10,6 +10,7 @@ from typing import Optional
 import pandas as pd
 
 from topiclayers.support.cache import cache_key
+from topiclayers.status import log_via
 from topiclayers.schema import InteractionType, Post
 
 _INTERACTION_REVERSE_MAP = {
@@ -82,8 +83,8 @@ class Dataset:
                 self.df_users = pd.read_pickle(users_pkl)
         else:
             if tweets_pkl.exists():
-                print("Cache key mismatch — regenerating data cache.")
-            users_dict = self._load_user_dict() if self.file_user is not None else {}
+                log_via(reporter, "Cache key mismatch — regenerating data cache.")
+            users_dict = self._load_user_dict(reporter=reporter) if self.file_user is not None else {}
             tweets_dict = {}
             for i, post in enumerate(self.posts):
                 if reporter is not None:
@@ -135,13 +136,13 @@ class Dataset:
 
         self._create_dataframes()
 
-    def _load_user_dict(self) -> dict:
+    def _load_user_dict(self, reporter=None) -> dict:
         import json
 
         if self.file_user is None:
             return {}
         if not self.file_user.exists():
-            print(f"User file not found: {self.file_user} — skipping user metadata.")
+            log_via(reporter, f"User file not found: {self.file_user} — skipping user metadata.")
             return {}
 
         users = {}
@@ -167,7 +168,7 @@ class Dataset:
         self.df_quotes = df[df["referenced_type"] == "quoted"].copy()
         self.df_reply = df[df["referenced_type"] == "replied_to"].copy()
 
-    def update_df(self, df_labeled: pd.DataFrame) -> pd.DataFrame:
+    def update_df(self, df_labeled: pd.DataFrame, reporter=None) -> pd.DataFrame:
         if "id" in df_labeled.columns and df_labeled.index.name != "id":
             df_labeled = df_labeled.set_index("id")
 
@@ -197,9 +198,9 @@ class Dataset:
             resolved[idx] = val
 
         self.df_retweets_labeled["topic"] = self.df_retweets_labeled.index.map(resolved)
-        print(f"Unresolved references: {unresolved_count}")
+        log_via(reporter, f"Unresolved references: {unresolved_count}")
         if unresolved_count > 0:
-            print(f"  {unresolved_count} retweets could not be resolved to a topic. These will be excluded from the labeled output.")
+            log_via(reporter, f"  {unresolved_count} retweets could not be resolved to a topic. These will be excluded from the labeled output.")
 
         self.df_retweets_labeled = self.df_retweets_labeled[
             self.df_retweets_labeled["topic"].apply(lambda x: not isinstance(x, str))

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/core/network.py

@@ -5,6 +5,8 @@ from pathlib import Path
 import networkx as nx
 import pandas as pd
 
+from topiclayers.status import log_via
+
 
 class NetworkCreator:
     def __init__(self, df: pd.DataFrame, name: str, output_dir: Path):
@@ -25,7 +27,7 @@ class NetworkCreator:
         )
 
         if missing_refs:
-            print(f"  {missing_refs} retweets reference tweets not in dataset — skipped.")
+            log_via(reporter, f"  {missing_refs} retweets reference tweets not in dataset — skipped.")
 
         self._save_graph(G, "retweet")
         return G
@@ -101,7 +103,7 @@ class NetworkCreator:
             ml.add_nx_layer(ml_network, G, str(topic))
 
         if dropped_edges_total:
-            print(f"  {dropped_edges_total} edges dropped (referenced tweet missing in topic slice).")
+            log_via(reporter, f"  {dropped_edges_total} edges dropped (referenced tweet missing in topic slice).")
 
         self._save_multilayer(ml_network)
         self.ml_network = ml_network

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/core/topic.py

@@ -11,6 +11,7 @@ import numpy as np
 import pandas as pd
 
 from topiclayers.support.cache import cache_key
+from topiclayers.status import log_via
 
 
 class TopicModeler:
@@ -77,7 +78,7 @@ class TopicModeler:
                 except Exception:
                     pass
             if cache_valid:
-                print("using cached topics")
+                log_via(reporter, "using cached topics")
                 self.df = pd.read_pickle(self.df_labeled_path)
                 from bertopic import BERTopic
                 self.model = BERTopic.load(self.model_path)
@@ -85,7 +86,7 @@ class TopicModeler:
                     self.docs = self.df["new_text"].tolist()
                 return self.df
 
-        print("running topic modeling")
+        log_via(reporter, "running topic modeling")
         docs = self._preprocess()
 
         if self.embeddings_path.exists():
@@ -101,7 +102,7 @@ class TopicModeler:
                 except Exception:
                     pass
             if use_cached:
-                print("  loading embeddings from cache")
+                log_via(reporter, "  loading embeddings from cache")
                 with open(self.embeddings_path, "rb") as f:
                     self.embeddings = pickle.load(f)
             else:
@@ -214,6 +215,11 @@ class TopicModeler:
         repro_mode: str = "strict",
         umap_init: str = "pca",
     ) -> None:
+        # mark the stage before importing: torch/umap/bertopic imports take a
+        # while and would otherwise be billed to the previous stage
+        if reporter is not None:
+            reporter.stage("topic_model")
+
         import random
 
         from umap import UMAP
@@ -222,9 +228,6 @@ class TopicModeler:
         from bertopic import BERTopic
         from bertopic.vectorizers import ClassTfidfTransformer
 
-        if reporter is not None:
-            reporter.stage("topic_model")
-
         np.random.seed(random_state)
         random.seed(random_state)
         try:
@@ -293,7 +296,9 @@ class TopicModeler:
             # large corpora); the default computes the probability of the
             # assigned topic only — topics themselves are identical
             calculate_probabilities=exact_probabilities,
-            verbose=True,
+            # False: BERTopic's per-step INFO logging would clobber the live
+            # progress line; our reporter provides the progress feedback
+            verbose=False,
         )
 
         try:
@@ -350,7 +355,7 @@ class TopicModeler:
     def _save_qdrant(self, docs: list[str], reporter=None) -> None:
         qdrant_url = os.getenv("QDRANT_URL")
         if not qdrant_url:
-            print("Qdrant not configured — skipping. Set QDRANT_URL in .env to enable vector search.")
+            log_via(reporter, "Qdrant not configured — skipping. Set QDRANT_URL in .env to enable vector search.")
             return
 
         try:
@@ -386,8 +391,8 @@ class TopicModeler:
             if reporter is not None:
                 reporter.update(len(points), total=len(points))
         except Exception as e:
-            print(f"Qdrant upload failed (non-fatal): {type(e).__name__}: {e}")
-            print("The pipeline continues — networks and topic labels are unaffected.")
+            log_via(reporter, f"Qdrant upload failed (non-fatal): {type(e).__name__}: {e}")
+            log_via(reporter, "The pipeline continues — networks and topic labels are unaffected.")
 
     def _save_files(self) -> None:
         self.df_labeled_path.parent.mkdir(parents=True, exist_ok=True)
@@ -409,6 +414,7 @@ class TopicModeler:
         batch_size: int = 40,
         label_context: str = "",
         max_topics: int = 0,
+        reporter=None,
     ) -> dict[int, str]:
         """Label topics with an LLM, batching many topics per request.
 
@@ -443,18 +449,19 @@ class TopicModeler:
         api_base = api_base or os.getenv("LABEL_API_BASE", "")
 
         if self.model is None:
-            print("  No fitted topic model — skipping LLM labeling.")
+            log_via(reporter, "  No fitted topic model — skipping LLM labeling.")
             return {}
 
         topic_info = self.model.get_topic_info()
         if "LLM_label" in topic_info.columns and not overwrite:
-            print("  LLM labels already present — skipping (use overwrite=True).")
+            log_via(reporter, "  LLM labels already present — skipping (use overwrite=True).")
             return {}
 
         try:
             import litellm  # noqa: F401
         except ImportError:
-            print(
+            log_via(
+                reporter,
                 "  LLM labeling skipped: 'litellm' is not installed.\n"
                 "    Install it with: pip install 'topiclayers[labeling]'\n"
                 "    Topics keep their c-TF-IDF keyword labels."
@@ -470,7 +477,8 @@ class TopicModeler:
                 conn.getresponse().read()
                 conn.close()
             except (OSError, socket.timeout):
-                print(
+                log_via(
+                    reporter,
                     f"  LLM labeling skipped: Ollama not reachable at {host}:{port}.\n"
                     f"    Start it with: ollama serve   (then pull the model:\n"
                     f"    ollama pull {model.split('/', 1)[1]})\n"
@@ -486,7 +494,8 @@ class TopicModeler:
                 or os.getenv("OPENAI_API_KEY")
             )
             if not api_key:
-                print(
+                log_via(
+                    reporter,
                     "  LLM labeling skipped: custom api_base set but no API key found.\n"
                     "    Set LABEL_API_KEY (or OPENAI_LIKE_API_KEY / OPENAI_API_KEY) in .env."
                 )
@@ -515,12 +524,12 @@ class TopicModeler:
             entries.append((tid, keywords, docs_str))
 
         if not entries:
-            print("  No non-outlier topics to label.")
+            log_via(reporter, "  No non-outlier topics to label.")
             return {}
 
         if max_topics and max_topics < len(entries):
             entries = entries[:max_topics]  # topic_info is sorted by size
-            print(f"  Preview mode: labeling only the {max_topics} largest topics.")
+            log_via(reporter, f"  Preview mode: labeling only the {max_topics} largest topics.")
 
         system_msg = (
             "You annotate discussion topics from social media.\n"
@@ -577,16 +586,17 @@ class TopicModeler:
                         _load_openai_like_model(model, api_base, generator_kwargs.get("api_key", ""))
                         continue
                     if attempt == 1:
-                        print(f"  Batch {i // batch_size + 1}/{n_chunks}: {type(e).__name__}, retrying once.")
+                        log_via(reporter, f"  Batch {i // batch_size + 1}/{n_chunks}: {type(e).__name__}, retrying once.")
                         continue
                     failed += 1
-                    print(
+                    log_via(
+                        reporter,
                         f"  Batch {i // batch_size + 1}/{n_chunks} failed (non-fatal): "
                         f"{type(e).__name__}: {str(e)[:120]} — those topics keep keyword labels."
                     )
 
         if not llm_labels:
-            print("  No labels generated — topics keep their c-TF-IDF keyword labels.")
+            log_via(reporter, "  No labels generated — topics keep their c-TF-IDF keyword labels.")
             return {}
 
         # expose labels as an aspect column named LLM_label in get_topic_info()
@@ -609,7 +619,7 @@ class TopicModeler:
         info.to_csv(self.topic_path)
         ok = len(llm_labels)
         suffix = f", {failed} batch(es) failed" if failed else ""
-        print(f"  Labeled {ok} topics with '{model}' in {n_chunks} batch(es){suffix}.")
+        log_via(reporter, f"  Labeled {ok} topics with '{model}' in {n_chunks} batch(es){suffix}.")
         return labels
 
 

topiclayers-0.3.0/src/topiclayers/examples/generic/posts.csv

@@ -0,0 +1,8 @@
+post_id,user_id,text,created_at,lang,interaction_type,target_post_id
+t1,a,tweet1,2022-01-01,en,original,
+t2,b,tweet2,2022-01-02,en,repost,t1
+t3,c,tweet3,2022-01-03,en,repost,t2
+t4,c,tweet4,2022-01-04,en,original,
+t5,a,tweet5,2022-01-05,en,repost,t7
+t6,d,tweet6,2022-01-06,en,repost,t7
+t7,d,tweet7,2022-01-07,en,original,

topiclayers-0.3.0/src/topiclayers/examples/generic.yml

@@ -0,0 +1,15 @@
+input:
+  format: csv_posts
+  # paths are resolved relative to this config file's directory
+  posts: generic/posts.csv
+
+name: generic_toy
+
+topic_model:
+  embedder: all-MiniLM-L6-v2
+  min_topic_size: 2
+
+network:
+  type: multilayer_repost
+
+output_dir: ./out/generic_toy

topiclayers-0.3.0/src/topiclayers/examples/template_csv.yml

@@ -0,0 +1,49 @@
+# ============================================================
+# TEMPLATE — Generic CSV/Parquet posts (any platform)
+# Copy this file, rename it, adjust the values below.
+# Run:  topiclayers run examples/<your_config>.yml
+# Paths in `input` are resolved relative to THIS file's directory.
+# ============================================================
+
+input:
+  format: csv_posts              # or parquet_posts
+  posts: generic/posts.csv       # required columns: post_id, user_id, text
+                                 # optional columns: created_at, lang,
+                                 # interaction_type ("original"|"repost"|"quote"|"reply"),
+                                 # target_post_id, mentions (semicolon-separated),
+                                 # any extra_* column is carried through as metadata
+  # users: path/to/users.csv     # optional user metadata file
+
+name: my_dataset                 # prefixes outputs and cache keys; changing it starts
+                                 # a fresh cache
+
+topic_model:
+  embedder: all-MiniLM-L6-v2     # sentence-transformers model, or OpenAI
+                                 # "text-embedding-3-small"/"-large" (needs
+                                 # OPENAI_API_KEY in .env)
+  min_topic_size: 10             # small datasets need small values; start low and raise
+                                 # until topics look coherent
+  nr_topics: auto                # "auto", or an int to force reduction after fit
+  random_state: 42
+  umap_n_neighbors: 15
+  umap_n_components: 5
+  exact_probabilities: false     # true = full membership matrix (slow on big corpora)
+  repro_mode: fast               # "fast" = parallel, non-reproducible;
+                                 # "strict" = seeded single-threaded, bit-reproducible
+  umap_init: pca                 # default; ~20x faster than spectral, deterministic
+                                 # when seeded (docs/UMAP_INIT_BENCHMARK.md)
+  label_model: ollama/qwen2.5:7b # optional LLM topic labeling (LiteLLM string);
+                                 # "" disables. See template_twitter.yml for setup
+
+network:
+  type: retweet                  # for CSV data usually "retweet" (or "multilayer_repost"
+                                 # / "ttn" / "all"); requires target_post_id +
+                                 # interaction_type=repost rows to build edges
+  project_ttn: true
+
+output_dir: ./out/my_dataset     # research outputs only: networks/, run_manifest.json,
+                                 # run_status.json, stage_timings.json
+
+# cache_dir: ./cache             # optional — where reusable artifacts live (processed
+                                 # data, embeddings, topic model). Defaults to a
+                                 # 'cache/' folder next to the input data file.

topiclayers-0.3.0/src/topiclayers/examples/template_twitter.yml

@@ -0,0 +1,90 @@
+# ============================================================
+# TEMPLATE — Twitter/X JSONL input (Academic API dump)
+# Copy this file, rename it, adjust the values below.
+# Run:  topiclayers run examples/<your_config>.yml
+# Status from another terminal while running:
+#       topiclayers status <output_dir>
+# Paths in `input` are resolved relative to THIS file's directory.
+# ============================================================
+
+input:
+  format: twitter_jsonl          # one JSON tweet per line (Twitter API v2 fields:
+                                 # id, author_id, text, referenced_tweets, entities...)
+  posts: twitter/toy.json        # required — path to the .json/.jsonl tweet file
+  # users: path/to/users.dat     # optional — JSONL of user objects
+                                 # ({id, username, public_metrics}); used to map
+                                 # author ids -> usernames in outputs.
+                                 # Delete the line if you don't have it.
+
+name: cop22                      # dataset name: prefixes output files and cache keys.
+                                 # Changing it starts a fresh cache.
+
+topic_model:
+  embedder: all-MiniLM-L6-v2     # any sentence-transformers model, or an OpenAI
+                                 # embedding model ("text-embedding-3-small" /
+                                 # "text-embedding-3-large"; requires OPENAI_API_KEY
+                                 # in .env)
+  min_topic_size: 50             # smallest cluster HDBSCAN will accept (= BERTopic's
+                                 # granularity knob). Too high -> everything is outlier
+                                 # -1; too low -> thousands of micro-topics.
+                                 # Rule of thumb: ~1/2000 of originals for large sets.
+  nr_topics: auto                # "auto", or an int to force topic reduction after fit
+  random_state: 42               # seed; keep fixed for reproducibility
+  umap_n_neighbors: 15           # UMAP locality (kept for reference; pipeline uses
+                                 # adaptive values for small datasets)
+  umap_n_components: 5           # UMAP output dims (same note as above)
+  exact_probabilities: false     # true = full membership matrix (much slower on big
+                                 # corpora); topics are identical either way
+  repro_mode: fast               # "fast" = parallel UMAP+HDBSCAN (~20 min saved on
+                                 #   180k docs), results vary between runs;
+                                 # "strict" = seeded single-threaded UMAP,
+                                 #   bit-reproducible, slower
+  umap_init: pca                 # default; deterministic when seeded and ~20x faster
+                                 # than spectral (see docs/UMAP_INIT_BENCHMARK.md).
+                                 # Pass "spectral" only if you specifically need it.
+  label_model: ollama/qwen2.5:7b # LLM that writes a short human-readable label per
+                                 # topic (any LiteLLM model string). Local & free
+                                 # via Ollama (https://ollama.com):
+                                 #   ollama serve && ollama pull qwen2.5:7b
+                                 # Cloud examples: "gpt-4o-mini",
+                                 # "anthropic/claude-3-haiku-20240307".
+                                 # Custom OpenAI-compatible server (Unsloth,
+                                 # llama.cpp, vLLM...): prefix with "openai/" and
+                                 # set label_api_base, e.g.
+                                 #   label_model: openai/unsloth/Qwen3.8-27B-GGUF
+                                 #   label_api_base: http://localhost:8888/v1
+                                 # Requires the labeling extra:
+                                 #   pip install 'topiclayers[labeling]'
+                                 # API key from LABEL_API_KEY / OPENAI_LIKE_API_KEY
+                                 # / OPENAI_API_KEY in .env. Set label_model to ""
+                                 # to disable; non-fatal if unreachable.
+  # label_context: ""            # optional context sentence ("all documents are ...");
+                                 # by default derived from `name` for cop* datasets.
+                                 # Helps the LLM label the specific SUBTOPIC instead
+                                 # of the general domain. Topics are labeled in
+                                 # batches (~40 per call); use a strong model for
+                                 # publishable labels (e.g. gpt-4o-mini, <1 min and
+                                 # cents for a full dataset)
+  label_api_base: ""             # base URL of a custom OpenAI-compatible endpoint;
+                                 # leave "" for Ollama/OpenAI. Can also be set via
+                                 # LABEL_API_BASE in .env
+
+network:
+  type: multilayer_repost        # one of:
+                                 #   multilayer_repost -> retweet graph + multilayer
+                                 #                        network by topic (uunet)
+                                 #   retweet           -> retweet graph only (.gml)
+                                 #   ttn               -> topic-targeted network
+                                 #   all               -> all of the above
+  project_ttn: true              # ttn only: also export the projected monopartite
+                                 # version
+
+output_dir: ./out/cop22          # research outputs only: networks/, run_manifest.json,
+                                 # run_status.json, stage_timings.json
+
+# cache_dir: ./cache              # optional — where reusable artifacts live (processed
+                                  # data, embeddings, topic model). Defaults to a
+                                  # 'cache/' folder next to the input data file, so it
+                                  # persists regardless of output_dir and is shared
+                                  # between runs; safe to share across datasets (files
+                                  # are prefixed with `name`)

topiclayers-0.3.0/src/topiclayers/examples/twitter/toy.json

@@ -0,0 +1 @@
+{"lang": "en", "possibly_sensitive": false, "public_metrics": {"retweet_count": 8, "reply_count": 0, "like_count": 0, "quote_count": 0, "impression_count": 0}, "conversation_id": "815345170468208642", "entities": {"mentions": [{"start": 3, "end": 14, "username": "pablorodas", "id": "52119056"}], "hashtags": [{"start": 16, "end": 30, "tag": "CLIMATEchange"}, {"start": 31, "end": 34, "tag": "p2"}, {"start": 112, "end": 118, "tag": "COP22"}, {"start": 119, "end": 124, "tag": "tcot"}, {"start": 125, "end": 128, "tag": "2A"}], "urls": [{"start": 88, "end": 111, "url": "https://t.co/f5lY9m4sE2", "expanded_url": "http://dlvr.it/N0sp69", "display_url": "dlvr.it/N0sp69"}], "annotations": [{"start": 113, "end": 117, "probability": 0.775, "type": "Other", "normalized_text": "COP22"}]}, "referenced_tweets": [{"type": "retweeted", "id": "815344316495237120"}], "text": "RT @pablorodas: #CLIMATEchange #p2 RT Our best environment stories from this past\u00a0year. https://t.co/f5lY9m4sE2 #COP22 #tcot #2A https://t.\u2026", "id": "815345170468208642", "created_at": "2016-12-31T23:53:31.000Z", "author_id": "17227307", "context_annotations": [{"domain": {"id": "25", "name": "Events", "description": ""}, "entity": {"id": "10040144401", "name": "Extreme Weather + Climate Change"}}, {"domain": {"id": "25", "name": "Events", "description": ""}, "entity": {"id": "10040529339", "name": "Discovery VCN - Earth"}}], "edit_history_tweet_ids": ["815345170468208642"]}

{topiclayers-0.2.0 → topiclayers-0.3.0}/src/topiclayers/status.py

@@ -1,9 +1,19 @@
-"""Run status: live stage tracking and ETA for long-running pipeline runs."""
+"""Run status: live stage tracking and ETA for long-running pipeline runs.
+
+Writes `run_status.json` (machine-readable, consumed by `topiclayers status`)
+and, when attached to a terminal, renders live console output: colored stage
+banners, an in-place progress line with ETA, and a per-stage summary at the
+end. A heartbeat keeps the elapsed time ticking (and the status file fresh)
+during opaque stages that never call update(), so a silent stage is
+distinguishable from a stuck one.
+"""
 
 from __future__ import annotations
 
 import json
 import os
+import shutil
+import sys
 import threading
 import time
 from datetime import datetime, timezone
@@ -14,15 +24,32 @@ STAGES = ["load", "data", "embed", "topic_model", "label", "network", "manifest"
 _STATUS_FILE = "run_status.json"
 _TIMINGS_FILE = "stage_timings.json"
 _MIN_UPDATE_INTERVAL = 1.0
+_MIN_RENDER_INTERVAL = 0.2
 _HEARTBEAT_INTERVAL = 5.0
+_SPINNER = ["⠋", "⠙", "⠹", "⠸", "⠼", "⠴", "⠦", "⠧", "⠇", "⠏"]
+
+
+def _supports_ansi() -> bool:
+    return sys.stdout.isatty() and os.environ.get("NO_COLOR") is None
+
+
+def _c(code: str, text: str) -> str:
+    if not _supports_ansi():
+        return text
+    return f"\033[{code}m{text}\033[0m"
+
+
+def _fmt_count(n: int) -> str:
+    return f"{n:,}"
 
 
 class RunStatus:
-    def __init__(self, output_dir: Path, name: str):
+    def __init__(self, output_dir: Path, name: str, console: bool = True):
         self.output_dir = Path(output_dir)
         self.name = name
+        self.console = console
         self.path = self.output_dir / _STATUS_FILE
-        self.timings_path = self.output_dir / "cache" / _TIMINGS_FILE
+        self.timings_path = self.output_dir / _TIMINGS_FILE
         self.current_stage: Optional[str] = None
         self.stage_index: Optional[int] = None
         self.stage_total: Optional[int] = None
@@ -30,20 +57,111 @@ class RunStatus:
         self.started_at = datetime.now(timezone.utc)
         self.stage_started_at: Optional[datetime] = None
         self._last_write = 0.0
+        self._last_render = 0.0
+        self._line_active = False
+        self._spin_idx = 0
+        self._console_lock = threading.Lock()
+        self._stage_durations: dict[str, float] = {}
         self._stop_heartbeat = threading.Event()
         self._heartbeat_thread: Optional[threading.Thread] = None
 
+    # ------------------------------------------------------------------ console
+
+    def log(self, message: str) -> None:
+        """Print a message without clobbering the in-place progress line."""
+        with self._console_lock:
+            self._clear_line_locked()
+            print(message, flush=True)
+
+    def _clear_line_locked(self) -> None:
+        if self._line_active:
+            sys.stdout.write("\r\033[K")
+            sys.stdout.flush()
+            self._line_active = False
+
+    def _render(self, force: bool = False) -> None:
+        # in-place progress line; only meaningful on a real terminal
+        if not self.console or not sys.stdout.isatty():
+            return
+        now = time.monotonic()
+        if not force and now - self._last_render < _MIN_RENDER_INTERVAL:
+            return
+        self._last_render = now
+
+        with self._console_lock:
+            self._spin_idx += 1
+            spin = _c("2", _SPINNER[self._spin_idx % len(_SPINNER)])
+            stage = _c("36", self.current_stage) if self.current_stage else ""
+            bits = []
+            if (
+                self.processed is not None
+                and self.stage_total
+                and self.stage_total > 0
+            ):
+                pct = 100 * self.processed / self.stage_total
+                bits.append(
+                    f"{_fmt_count(self.processed)}/{_fmt_count(self.stage_total)}"
+                    f" ({pct:.0f}%)"
+                )
+                eta = self._eta_seconds()
+                if eta is not None:
+                    bits.append(f"eta ~{_c('33', format_seconds(eta))}")
+            else:
+                bits.append(_c("2", "running…"))
+            if self.stage_started_at is not None:
+                elapsed = (datetime.now(timezone.utc) - self.stage_started_at).total_seconds()
+                bits.append(f"elapsed {format_seconds(elapsed)}")
+
+            line = f"{spin} {stage}  " + _c("2", "·") + " " + f"  {_c('2', '·')}  ".join(bits)
+            width = shutil.get_terminal_size((80, 20)).columns
+            if len(line) + 8 > width:
+                line = line[: width - 8]
+            sys.stdout.write("\r\033[K" + line)
+            sys.stdout.flush()
+            self._line_active = True
+
+    def _banner(self, name: str, total: Optional[int], hist: Optional[float]) -> None:
+        with self._console_lock:
+            self._clear_line_locked()
+            if not self.console:
+                return
+            idx = STAGES.index(name) + 1 if name in STAGES else None
+            head = f"[{idx}/{len(STAGES)}] {name}" if idx else name
+            parts = [_c("2", "──"), _c("36;1", head)]
+            if total is not None:
+                parts.append(_c("2", f"── {_fmt_count(total)} items"))
+            if hist is not None:
+                parts.append(_c("2", f"── past runs ~{format_seconds(hist)}"))
+            print(" ".join(parts), flush=True)
+
+    def _summary(self, status: str) -> None:
+        with self._console_lock:
+            self._clear_line_locked()
+            if not self.console:
+                return
+            total = (datetime.now(timezone.utc) - self.started_at).total_seconds()
+            mark = _c("32", "✓") if status == "success" else _c("31", "✗")
+            print(f"\n{mark} run {status} in {format_seconds(total)}")
+            if self._stage_durations:
+                print(_c("2", "  per-stage:"))
+                for stage, duration in self._stage_durations.items():
+                    print(_c("2", f"    {stage:<14} {format_seconds(duration):>10}"))
+
+    # ------------------------------------------------------------------- status
+
     def start(self) -> None:
         self._start_heartbeat()
         self._write(status="running")
 
     def _start_heartbeat(self) -> None:
         # keeps stage_elapsed_seconds fresh during opaque stages (e.g. UMAP +
-        # HDBSCAN) that never call update()
+        # HDBSCAN) that never call update(); also re-renders the console line
+        # with an advancing spinner so silence is visibly not a hang
         def beat():
             while not self._stop_heartbeat.wait(_HEARTBEAT_INTERVAL):
                 if self.current_stage is not None:
                     self._write(status="running", force=True)
+                    self._render(force=True)
 
         self._heartbeat_thread = threading.Thread(target=beat, daemon=True)
         self._heartbeat_thread.start()
@@ -59,15 +177,20 @@ class RunStatus:
         self.processed = 0
         self.stage_started_at = now
         self._write(status="running", force=True)
+        if self.console:
+            self._banner(name, total, historical_eta(self.timings_path, name))
+            self._last_render = time.monotonic()
 
     def update(self, processed: int, total: Optional[int] = None) -> None:
         self.processed = processed
         if total is not None:
             self.stage_total = total
         self._write(status="running", throttle=True)
+        self._render()
 
     def flush(self) -> None:
         self._write(status="running", force=True)
+        self._render(force=True)
 
     def finish(self, status: str = "success") -> None:
         self._stop_heartbeat.set()
@@ -75,6 +198,10 @@ class RunStatus:
             duration = (datetime.now(timezone.utc) - self.stage_started_at).total_seconds()
             self._record_timing(self.current_stage, duration)
         self._write(status=status, force=True)
+        if self.console:
+            self._summary(status)
+
+    # ------------------------------------------------------------------ helpers
 
     def _eta_seconds(self) -> Optional[float]:
         if (
@@ -120,6 +247,7 @@ class RunStatus:
         os.replace(tmp, self.path)
 
     def _record_timing(self, stage: str, duration: float) -> None:
+        self._stage_durations[stage] = duration
         self.timings_path.parent.mkdir(parents=True, exist_ok=True)
         records: list[dict] = []
         if self.timings_path.exists():
@@ -139,6 +267,15 @@ class RunStatus:
         os.replace(tmp, self.timings_path)
 
 
+def log_via(reporter, message: str) -> None:
+    """Print a message through the reporter's console line (plain print fallback)."""
+    log_fn = getattr(reporter, "log", None)
+    if log_fn is not None:
+        log_fn(message)
+    else:
+        print(message)
+
+
 def load_status(path: Path) -> dict:
     with open(path) as f:
         return json.load(f)
@@ -202,7 +339,7 @@ def print_status(output_dir: Path) -> int:
         if eta is not None:
             print(f"Stage ETA: ~{format_seconds(eta)}")
         else:
-            hist = historical_eta(output_dir / "cache" / _TIMINGS_FILE, stage)
+            hist = historical_eta(output_dir / _TIMINGS_FILE, stage)
             if hist is not None:
                 print(f"No live ETA for this stage; past runs took ~{format_seconds(hist)} (median).")
             else: