topiclayers 0.2.0__tar.gz

topiclayers-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2023 Alessio Gandelli
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

topiclayers-0.2.0/PKG-INFO

@@ -0,0 +1,201 @@
+Metadata-Version: 2.4
+Name: topiclayers
+Version: 0.2.0
+Summary: From any collection of posts to topical multilayer networks — hardened and social-scientist-friendly
+License: MIT
+License-File: LICENSE
+Keywords: topic-modeling,network-analysis,social-science,polarisation
+Author: alessiogandelli
+Author-email: alessiogandelli99@gmail.com
+Requires-Python: >=3.12,<3.13
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.12
+Provides-Extra: labeling
+Requires-Dist: bertopic (>=0.17.4,<0.18.0)
+Requires-Dist: igraph (>=0.11.4)
+Requires-Dist: jsonlines (>=4.0.0,<5.0.0)
+Requires-Dist: litellm (>=1.40) ; extra == "labeling"
+Requires-Dist: networkx (>=3.6,<4.0)
+Requires-Dist: numpy (>=1.26,<3)
+Requires-Dist: openai (>=3.0.0,<4.0.0)
+Requires-Dist: pandas (>=2.2,<4)
+Requires-Dist: python-dotenv (>=1.0.1,<2.0.0)
+Requires-Dist: pyyaml (>=6.0,<7.0)
+Requires-Dist: qdrant-client (>=1.19.0,<2.0.0)
+Requires-Dist: scikit-learn (>=1.5)
+Requires-Dist: sentence-transformers (>=6.0.0,<7.0.0)
+Requires-Dist: umap-learn (>=0.5.12,<0.6.0)
+Requires-Dist: uunet (>=2.1.1,<3.0.0)
+Project-URL: Changelog, https://github.com/alessiogandelli/topiclayers/blob/main/CHANGELOG.md
+Project-URL: Issues, https://github.com/alessiogandelli/topiclayers/issues
+Project-URL: Repository, https://github.com/alessiogandelli/topiclayers
+Description-Content-Type: text/markdown
+
+# topiclayers
+
+> From posts to topical multilayer networks — hardened and social-scientist-friendly.
+
+## Quickstart
+
+```bash
+# Requires Python 3.12
+pip install topiclayers
+topiclayers run examples/generic.yml
+```
+
+That's it. You get a GML network file in `./out/generic_toy/networks/`.
+
+## What does it do?
+
+1. **Loads** your posts (Twitter JSONL, CSV, Parquet).
+2. **Cleans** text (removes URLs, @mentions, newlines).
+3. **Models topics** using BERTopic with SentenceTransformer embeddings.
+4. **Labels topics** with an LLM (local via Ollama by default) and propagates topic labels through retweet chains.
+5. **Builds networks**: single-layer retweet, multilayer per-topic, and bipartite temporal-text networks.
+
+## Minimal config
+
+Save this as `my_config.yml`:
+
+```yaml
+input:
+  format: csv_posts
+  posts: my_data.csv
+
+name: my_dataset
+
+topic_model:
+  embedder: all-MiniLM-L6-v2
+  min_topic_size: 50
+
+network:
+  type: multilayer_repost
+
+output_dir: ./out/my_dataset
+```
+
+Run it: `topiclayers run my_config.yml`
+
+## Checking progress
+
+Long runs write a live status file to `<output_dir>/run_status.json`. Check the current stage, elapsed time, and ETA from another terminal without touching the running job:
+
+```bash
+topiclayers status out/my_dataset
+```
+
+Example output:
+
+```
+Stage 3/7: embed
+Status: running
+Run: my_dataset
+Progress: 412000/1200000 (34%)
+Stage elapsed: 18m 12s
+Stage ETA: ~35m 20s
+Total elapsed: 21m 05s
+```
+
+Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/cache/stage_timings.json`.
+
+## Topic labeling with an LLM (optional)
+
+After clustering, the pipeline can ask a local LLM to write one short, readable
+label per topic (instead of raw keyword lists like `-1_proclamation_plante_trending`).
+Default is **Ollama** — free, local, nothing leaves your machine:
+
+```bash
+# one-time setup
+pip install 'topiclayers[labeling]'
+ollama pull qwen2.5:7b
+
+# start the server in another terminal
+ollama serve
+```
+
+Then just run the pipeline as usual (`label_model` is on by default). Any
+[LiteLLM](https://docs.litellm.ai/docs/providers) model string works:
+
+```yaml
+topic_model:
+  label_model: ollama/qwen2.5:7b      # default
+  # label_model: gpt-4o-mini          # OpenAI (needs OPENAI_API_KEY)
+  # label_model: anthropic/claude-3-haiku-20240307
+  # label_model: ""                   # disable labeling
+```
+
+### Custom / self-hosted OpenAI-compatible endpoints
+
+Any server speaking the OpenAI protocol (Unsloth, llama.cpp, vLLM, LM Studio...)
+works via `label_api_base`:
+
+```yaml
+topic_model:
+  label_model: openai/unsloth/Qwen3.8-27B-GGUF   # note the openai/ prefix
+  label_api_base: http://localhost:8888/v1
+```
+
+with the token in `.env` (key resolution order: `LABEL_API_KEY` →
+`OPENAI_LIKE_API_KEY` → `OPENAI_API_KEY`). You can also set `LABEL_API_BASE`
+in `.env` instead of the YAML. Find the exact model name your server exposes
+with `curl http://localhost:8888/v1/models`.
+
+Labeling is non-fatal: if the model server is unreachable, topics keep their
+keyword labels and everything else proceeds normally.
+
+## Input formats
+
+| Format | Extension | Description |
+|--------|-----------|-------------|
+| `twitter_jsonl` | `.json`, `.jsonl` | Twitter/X API v2 JSONL |
+| `csv_posts` | `.csv` | Generic CSV with `post_id`, `user_id`, `text` columns |
+| `parquet_posts` | `.parquet` | Same schema as CSV, Parquet format |
+
+CSV optional columns: `created_at`, `lang`, `interaction_type` (`original`/`repost`/`quote`/`reply`), `target_post_id`, `mentions` (semicolon-separated), `extra_*` passthrough columns.
+
+## What if something fails?
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| "All topics are -1" / "no topics found" | `min_topic_size` too high or dataset too small | Halve `min_topic_size` in config, or use a larger dataset (>100 posts) |
+| "Module not found" | topiclayers not installed | `pip install -e .` |
+| "OpenAI API key not set" | Using OpenAI embedder without key | Switch `embedder` to `all-MiniLM-L6-v2` (default, works offline) |
+| "Qdrant connection refused" | Qdrant vector DB not running | Ignore — Qdrant is optional. Set `QDRANT_URL` in `.env` to enable |
+| "Cannot create multilayer network" | All posts are outliers | Reduce `min_topic_size` or provide more data |
+| "UMAP spectral layout failed" | Dataset too small (<10 posts) | Topic modeling needs more data; consider using topic labels from elsewhere |
+
+## Output files
+
+For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`:
+
+```
+<output_dir>/
+├── run_manifest.json              # Reproducibility metadata
+├── run_status.json                # Live stage/ETA tracking (while running)
+├── cache/
+│   └── data/
+│       ├── tweets_<name>.pkl/.csv          # Full tweet table
+│       ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
+│       └── manifest_<name>.json            # Cache validity key
+│   └── stage_timings.json         # Per-stage durations for ETA prediction
+└── networks/
+    ├── <name>_retweet.gml                  # Single-layer retweet network
+    ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
+    ├── <name>_ttt.gml                      # Temporal-text bipartite network
+    └── projected/
+        └── <name>__prj_<topic>.gml         # Per-topic projected networks
+```
+
+## Requirements
+
+- Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
+- Optional: Ollama + `topiclayers[labeling]` extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)
+
+## Planned: JOSS software paper
+
+Once the API stabilises and the PLOS ONE core paper results are regenerated with this library,
+a short JOSS (Journal of Open Source Software) paper will be submitted with a Zenodo DOI.
+

topiclayers-0.2.0/README.md

@@ -0,0 +1,164 @@
+# topiclayers
+
+> From posts to topical multilayer networks — hardened and social-scientist-friendly.
+
+## Quickstart
+
+```bash
+# Requires Python 3.12
+pip install topiclayers
+topiclayers run examples/generic.yml
+```
+
+That's it. You get a GML network file in `./out/generic_toy/networks/`.
+
+## What does it do?
+
+1. **Loads** your posts (Twitter JSONL, CSV, Parquet).
+2. **Cleans** text (removes URLs, @mentions, newlines).
+3. **Models topics** using BERTopic with SentenceTransformer embeddings.
+4. **Labels topics** with an LLM (local via Ollama by default) and propagates topic labels through retweet chains.
+5. **Builds networks**: single-layer retweet, multilayer per-topic, and bipartite temporal-text networks.
+
+## Minimal config
+
+Save this as `my_config.yml`:
+
+```yaml
+input:
+  format: csv_posts
+  posts: my_data.csv
+
+name: my_dataset
+
+topic_model:
+  embedder: all-MiniLM-L6-v2
+  min_topic_size: 50
+
+network:
+  type: multilayer_repost
+
+output_dir: ./out/my_dataset
+```
+
+Run it: `topiclayers run my_config.yml`
+
+## Checking progress
+
+Long runs write a live status file to `<output_dir>/run_status.json`. Check the current stage, elapsed time, and ETA from another terminal without touching the running job:
+
+```bash
+topiclayers status out/my_dataset
+```
+
+Example output:
+
+```
+Stage 3/7: embed
+Status: running
+Run: my_dataset
+Progress: 412000/1200000 (34%)
+Stage elapsed: 18m 12s
+Stage ETA: ~35m 20s
+Total elapsed: 21m 05s
+```
+
+Stages are `load → data → embed → topic_model → label → network → manifest`. Stages with a per-item counter (data, embed, network) report a live ETA. Opaque stages (BERTopic's UMAP+HDBSCAN) fall back to the median duration of past runs, stored in `<output_dir>/cache/stage_timings.json`.
+
+## Topic labeling with an LLM (optional)
+
+After clustering, the pipeline can ask a local LLM to write one short, readable
+label per topic (instead of raw keyword lists like `-1_proclamation_plante_trending`).
+Default is **Ollama** — free, local, nothing leaves your machine:
+
+```bash
+# one-time setup
+pip install 'topiclayers[labeling]'
+ollama pull qwen2.5:7b
+
+# start the server in another terminal
+ollama serve
+```
+
+Then just run the pipeline as usual (`label_model` is on by default). Any
+[LiteLLM](https://docs.litellm.ai/docs/providers) model string works:
+
+```yaml
+topic_model:
+  label_model: ollama/qwen2.5:7b      # default
+  # label_model: gpt-4o-mini          # OpenAI (needs OPENAI_API_KEY)
+  # label_model: anthropic/claude-3-haiku-20240307
+  # label_model: ""                   # disable labeling
+```
+
+### Custom / self-hosted OpenAI-compatible endpoints
+
+Any server speaking the OpenAI protocol (Unsloth, llama.cpp, vLLM, LM Studio...)
+works via `label_api_base`:
+
+```yaml
+topic_model:
+  label_model: openai/unsloth/Qwen3.8-27B-GGUF   # note the openai/ prefix
+  label_api_base: http://localhost:8888/v1
+```
+
+with the token in `.env` (key resolution order: `LABEL_API_KEY` →
+`OPENAI_LIKE_API_KEY` → `OPENAI_API_KEY`). You can also set `LABEL_API_BASE`
+in `.env` instead of the YAML. Find the exact model name your server exposes
+with `curl http://localhost:8888/v1/models`.
+
+Labeling is non-fatal: if the model server is unreachable, topics keep their
+keyword labels and everything else proceeds normally.
+
+## Input formats
+
+| Format | Extension | Description |
+|--------|-----------|-------------|
+| `twitter_jsonl` | `.json`, `.jsonl` | Twitter/X API v2 JSONL |
+| `csv_posts` | `.csv` | Generic CSV with `post_id`, `user_id`, `text` columns |
+| `parquet_posts` | `.parquet` | Same schema as CSV, Parquet format |
+
+CSV optional columns: `created_at`, `lang`, `interaction_type` (`original`/`repost`/`quote`/`reply`), `target_post_id`, `mentions` (semicolon-separated), `extra_*` passthrough columns.
+
+## What if something fails?
+
+| Symptom | Likely cause | Fix |
+|---------|-------------|-----|
+| "All topics are -1" / "no topics found" | `min_topic_size` too high or dataset too small | Halve `min_topic_size` in config, or use a larger dataset (>100 posts) |
+| "Module not found" | topiclayers not installed | `pip install -e .` |
+| "OpenAI API key not set" | Using OpenAI embedder without key | Switch `embedder` to `all-MiniLM-L6-v2` (default, works offline) |
+| "Qdrant connection refused" | Qdrant vector DB not running | Ignore — Qdrant is optional. Set `QDRANT_URL` in `.env` to enable |
+| "Cannot create multilayer network" | All posts are outliers | Reduce `min_topic_size` or provide more data |
+| "UMAP spectral layout failed" | Dataset too small (<10 posts) | Topic modeling needs more data; consider using topic labels from elsewhere |
+
+## Output files
+
+For a dataset named `<name>` (e.g. `my_dataset`), output goes to `<output_dir>/`:
+
+```
+<output_dir>/
+├── run_manifest.json              # Reproducibility metadata
+├── run_status.json                # Live stage/ETA tracking (while running)
+├── cache/
+│   └── data/
+│       ├── tweets_<name>.pkl/.csv          # Full tweet table
+│       ├── retweet_labeled_<name>.pkl/.csv # Retweets with topic labels
+│       └── manifest_<name>.json            # Cache validity key
+│   └── stage_timings.json         # Per-stage durations for ETA prediction
+└── networks/
+    ├── <name>_retweet.gml                  # Single-layer retweet network
+    ├── <name>_retweet_network_ml.gml       # Multilayer (uunet format)
+    ├── <name>_ttt.gml                      # Temporal-text bipartite network
+    └── projected/
+        └── <name>__prj_<topic>.gml         # Per-topic projected networks
+```
+
+## Requirements
+
+- Python 3.12 (uunet, used for multilayer networks, does not ship wheels for 3.13+ yet)
+- Optional: Ollama + `topiclayers[labeling]` extra (LLM topic labeling), Docker (for Qdrant vector search), OpenAI API key (for OpenAI embeddings)
+
+## Planned: JOSS software paper
+
+Once the API stabilises and the PLOS ONE core paper results are regenerated with this library,
+a short JOSS (Journal of Open Source Software) paper will be submitted with a Zenodo DOI.

topiclayers-0.2.0/pyproject.toml

@@ -0,0 +1,52 @@
+[tool.poetry]
+name = "topiclayers"
+version = "0.2.0"
+description = "From any collection of posts to topical multilayer networks — hardened and social-scientist-friendly"
+authors = ["alessiogandelli <alessiogandelli99@gmail.com>"]
+readme = "README.md"
+packages = [{include = "topiclayers", from = "src"}]
+license = "MIT"
+keywords = ["topic-modeling", "network-analysis", "social-science", "polarisation"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Science/Research",
+    "Programming Language :: Python :: 3.12",
+]
+
+[tool.poetry.dependencies]
+python = ">=3.12,<3.13"
+jsonlines = "^4.0.0"
+pandas = ">=2.2,<4"
+numpy = ">=1.26,<3"
+sentence-transformers = "^6.0.0"
+bertopic = "^0.17.4"
+scikit-learn = ">=1.5"
+umap-learn = "^0.5.12"
+networkx = "^3.6"
+igraph = ">=0.11.4"
+uunet = "^2.1.1"
+qdrant-client = "^1.19.0"
+openai = "^3.0.0"
+python-dotenv = "^1.0.1"
+pyyaml = "^6.0"
+litellm = {version = ">=1.40", optional = true}
+
+[tool.poetry.extras]
+labeling = ["litellm"]
+
+[tool.poetry.group.dev.dependencies]
+pytest = ">=8.0"
+pytest-cov = ">=6.0"
+jupyter = "^1.0.0"
+
+[tool.poetry.urls]
+Repository = "https://github.com/alessiogandelli/topiclayers"
+Issues = "https://github.com/alessiogandelli/topiclayers/issues"
+Changelog = "https://github.com/alessiogandelli/topiclayers/blob/main/CHANGELOG.md"
+
+[tool.poetry.scripts]
+topiclayers = "topiclayers.cli:main"
+
+[build-system]
+requires = ["poetry-core"]
+build-backend = "poetry.core.masonry.api"

topiclayers-0.2.0/src/topiclayers/__init__.py

@@ -0,0 +1,3 @@
+"""topiclayers — from posts to topical multilayer networks."""
+
+__version__ = "0.2.0"

topiclayers-0.2.0/src/topiclayers/adapters/__init__.py

@@ -0,0 +1,7 @@
+"""Input adapters: convert various formats into a list of Post dataclasses."""
+
+from __future__ import annotations
+
+from .registry import AdapterRegistry, resolve_adapter
+
+__all__ = ["AdapterRegistry", "resolve_adapter"]

topiclayers-0.2.0/src/topiclayers/adapters/csv.py

@@ -0,0 +1,91 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import List
+
+import pandas as pd
+
+from topiclayers.adapters.registry import register
+from topiclayers.schema import InteractionType, Post
+
+_INTERACTION_MAP = {
+    "original": None,
+    "repost": InteractionType.REPOST,
+    "quote": InteractionType.QUOTE,
+    "reply": InteractionType.REPLY,
+}
+
+
+def _parse_interaction(value) -> InteractionType | None:
+    if value is None:
+        return None
+    if isinstance(value, float) and pd.isna(value):
+        return None
+    s = str(value).strip().lower()
+    if not s or s in ("nan", "none", ""):
+        return None
+    return _INTERACTION_MAP.get(s, None)
+
+
+def _parse_mentions(value) -> list[str]:
+    if value is None:
+        return []
+    if isinstance(value, float) and pd.isna(value):
+        return []
+    s = str(value).strip()
+    if not s or s.lower() in ("nan", "none"):
+        return []
+    return [m.strip() for m in s.split(";") if m.strip()]
+
+
+def _row_to_post(row, extra_cols: list[str]) -> Post:
+    extra = {}
+    for col in extra_cols:
+        val = row.get(col)
+        if isinstance(val, float) and pd.isna(val):
+            val = None
+        key = col.removeprefix("extra_")
+        extra[key] = val
+
+    return Post(
+        post_id=str(row["post_id"]),
+        user_id=str(row["user_id"]),
+        text=str(row.get("text", "")),
+        created_at=row.get("created_at") if not (isinstance(row.get("created_at"), float) and pd.isna(row.get("created_at"))) else None,
+        lang=row.get("lang") if not (isinstance(row.get("lang"), float) and pd.isna(row.get("lang"))) else None,
+        interaction_type=_parse_interaction(row.get("interaction_type")),
+        target_post_id=str(row["target_post_id"]) if row.get("target_post_id") and not (isinstance(row.get("target_post_id"), float) and pd.isna(row.get("target_post_id"))) else None,
+        mentions=_parse_mentions(row.get("mentions")),
+        extra=extra,
+    )
+
+
+def load_csv(path: Path) -> List[Post]:
+    df = pd.read_csv(path)
+    required = {"post_id", "user_id", "text"}
+    missing = required - set(df.columns)
+    if missing:
+        raise ValueError(
+            f"CSV file at {path} is missing required columns: {missing}. "
+            f"Required columns are: post_id, user_id, text. "
+            f"Optional: created_at, lang, interaction_type, target_post_id, target_user_id, mentions, and extra_* columns."
+        )
+    extra_cols = [c for c in df.columns if c.startswith("extra_")]
+    return [_row_to_post(row, extra_cols) for _, row in df.iterrows()]
+
+
+def load_parquet(path: Path) -> List[Post]:
+    df = pd.read_parquet(path)
+    required = {"post_id", "user_id", "text"}
+    missing = required - set(df.columns)
+    if missing:
+        raise ValueError(
+            f"Parquet file at {path} is missing required columns: {missing}. "
+            f"Required columns are: post_id, user_id, text."
+        )
+    extra_cols = [c for c in df.columns if c.startswith("extra_")]
+    return [_row_to_post(row, extra_cols) for _, row in df.iterrows()]
+
+
+register("csv_posts", load_csv)
+register("parquet_posts", load_parquet)

topiclayers-0.2.0/src/topiclayers/adapters/registry.py

@@ -0,0 +1,47 @@
+"""Input adapter registry — maps format strings to loader functions."""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Callable, List
+
+from topiclayers.schema import Post
+
+
+AdapterFn = Callable[[Path], List[Post]]
+
+_registry: dict[str, AdapterFn] = {}
+
+
+def register(format_name: str, fn: AdapterFn) -> None:
+    _registry[format_name] = fn
+
+
+def resolve_adapter(path: Path) -> tuple[str, AdapterFn]:
+    """Auto-detect format from file extension and return (format_name, loader)."""
+    ext = path.suffix.lower()
+    if ext in (".jsonl", ".json", ".dat"):
+        fmt = "twitter_jsonl"
+    elif ext == ".csv":
+        fmt = "csv_posts"
+    elif ext == ".parquet":
+        fmt = "parquet_posts"
+    else:
+        raise ValueError(
+            f"Cannot auto-detect format for extension '{ext}'. "
+            f"Supported: .jsonl, .json, .dat, .csv, .parquet. "
+            f"Use explicit `format:` in config for other formats."
+        )
+    if fmt not in _registry:
+        raise KeyError(
+            f"No adapter registered for format '{fmt}'. "
+            f"Available adapters: {list(_registry.keys())}. "
+            f"Did you import topiclayers.adapters.{fmt.split('_')[0]}?"
+        )
+    return fmt, _registry[fmt]
+
+
+class AdapterRegistry:
+    """Explicit registry for manual adapter lookups."""
+    pass

topiclayers-0.2.0/src/topiclayers/adapters/twitter.py

@@ -0,0 +1,73 @@
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import List
+
+from topiclayers.adapters.registry import register
+from topiclayers.schema import InteractionType, Post
+
+
+def _map_interaction_type(ref_type: str | None) -> InteractionType | None:
+    if ref_type == "retweeted":
+        return InteractionType.REPOST
+    elif ref_type == "quoted":
+        return InteractionType.QUOTE
+    elif ref_type == "replied_to":
+        return InteractionType.REPLY
+    return None
+
+
+def load(path: Path) -> List[Post]:
+    posts = []
+    with open(path) as f:
+        for line in f:
+            line = line.strip()
+            if not line:
+                continue
+            obj = json.loads(line)
+
+            post_id = str(obj.get("id", ""))
+            user_id = str(obj.get("author_id", ""))
+            text = obj.get("text", "")
+            created_at = obj.get("created_at")
+            lang = obj.get("lang")
+
+            ref_tweets = obj.get("referenced_tweets")
+            if ref_tweets and isinstance(ref_tweets, list) and len(ref_tweets) > 0:
+                ref = ref_tweets[0]
+                interaction_type = _map_interaction_type(ref.get("type"))
+                target_post_id = str(ref.get("id")) if ref.get("id") is not None else None
+            else:
+                interaction_type = None
+                target_post_id = None
+
+            entities = obj.get("entities", {})
+            mentions_raw = entities.get("mentions", []) if isinstance(entities, dict) else []
+            mentions = [m.get("username", "") for m in mentions_raw if isinstance(m, dict)]
+
+            extra = {
+                "conversation_id": str(obj.get("conversation_id")) if obj.get("conversation_id") is not None else None,
+                "context_annotations": obj.get("context_annotations", []),
+                "public_metrics": obj.get("public_metrics", {}),
+                "possibly_sensitive": obj.get("possibly_sensitive"),
+                "edit_history_tweet_ids": obj.get("edit_history_tweet_ids", []),
+            }
+
+            post = Post(
+                post_id=post_id,
+                user_id=user_id,
+                text=text,
+                created_at=created_at,
+                lang=lang,
+                interaction_type=interaction_type,
+                target_post_id=target_post_id,
+                mentions=mentions,
+                extra=extra,
+            )
+            posts.append(post)
+
+    return posts
+
+
+register("twitter_jsonl", load)