citation-topics 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- citation_topics-0.1.0/.gitignore +17 -0
- citation_topics-0.1.0/CHANGELOG.md +18 -0
- citation_topics-0.1.0/LICENSE +21 -0
- citation_topics-0.1.0/PKG-INFO +156 -0
- citation_topics-0.1.0/README.md +124 -0
- citation_topics-0.1.0/pyproject.toml +67 -0
- citation_topics-0.1.0/src/citation_topics/__init__.py +11 -0
- citation_topics-0.1.0/src/citation_topics/__main__.py +3 -0
- citation_topics-0.1.0/src/citation_topics/claude/agents/topic-namer.md +19 -0
- citation_topics-0.1.0/src/citation_topics/claude/skills/topic-namer/SKILL.md +63 -0
- citation_topics-0.1.0/src/citation_topics/claude/skills/topics-compare/SKILL.md +41 -0
- citation_topics-0.1.0/src/citation_topics/claude/skills/topics-run/SKILL.md +90 -0
- citation_topics-0.1.0/src/citation_topics/cli.py +173 -0
- citation_topics-0.1.0/src/citation_topics/cluster.py +154 -0
- citation_topics-0.1.0/src/citation_topics/compare.py +203 -0
- citation_topics-0.1.0/src/citation_topics/describe.py +356 -0
- citation_topics-0.1.0/src/citation_topics/export.py +117 -0
- citation_topics-0.1.0/src/citation_topics/io.py +102 -0
- citation_topics-0.1.0/src/citation_topics/names.py +58 -0
- citation_topics-0.1.0/src/citation_topics/network.py +112 -0
- citation_topics-0.1.0/src/citation_topics/py.typed +0 -0
- citation_topics-0.1.0/src/citation_topics/run.py +171 -0
- citation_topics-0.1.0/src/citation_topics/secondary.py +60 -0
- citation_topics-0.1.0/src/citation_topics/skills.py +39 -0
- citation_topics-0.1.0/tests/conftest.py +54 -0
- citation_topics-0.1.0/tests/test_cli.py +58 -0
- citation_topics-0.1.0/tests/test_pipeline.py +190 -0
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
__pycache__/
|
|
2
|
+
*.py[cod]
|
|
3
|
+
*.egg-info/
|
|
4
|
+
build/
|
|
5
|
+
dist/
|
|
6
|
+
.venv/
|
|
7
|
+
venv/
|
|
8
|
+
.pytest_cache/
|
|
9
|
+
.ruff_cache/
|
|
10
|
+
.coverage
|
|
11
|
+
coverage.xml
|
|
12
|
+
htmlcov/
|
|
13
|
+
# Add here any generated outputs or data files that hold personal data (e.g. downloaded pages)
|
|
14
|
+
# Never commit credentials
|
|
15
|
+
.env
|
|
16
|
+
.pypirc
|
|
17
|
+
*apikey*
|
|
@@ -0,0 +1,18 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
The format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/) and versions follow [Semantic Versioning](https://semver.org/).
|
|
4
|
+
|
|
5
|
+
## [Unreleased]
|
|
6
|
+
|
|
7
|
+
## [0.1.0] - 2026-11-15
|
|
8
|
+
|
|
9
|
+
### Added
|
|
10
|
+
- `run`: direct-citation network of a JSON or JSON Lines list of works, Leiden topics (igraph) and areas
|
|
11
|
+
(leidenalg, best of several seeds), second and third topics of every work; `works.jsonl.gz`, `topics.json`,
|
|
12
|
+
`areas.json`, `meta.json`.
|
|
13
|
+
- `describe`: dossier of every topic and area from the optional fields.
|
|
14
|
+
- `names`, `compare` (fates and names carried over), `agreement` (NMI/ARI, medoid), `export` (VOSviewer, CSV).
|
|
15
|
+
- Claude skills `topics-run`, `topic-namer`, `topics-compare` and the agent `topic-namer` (`skills install`).
|
|
16
|
+
|
|
17
|
+
[Unreleased]: https://github.com/restrepo/citation-topics/compare/v0.1.0...HEAD
|
|
18
|
+
[0.1.0]: https://github.com/restrepo/citation-topics/releases/tag/v0.1.0
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Diego Restrepo
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,156 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: citation-topics
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Research topics and areas from the citation links of any list of works, with Leiden clustering as CWTS does for OpenAlex.
|
|
5
|
+
Project-URL: Homepage, https://github.com/restrepo/citation-topics
|
|
6
|
+
Project-URL: Repository, https://github.com/restrepo/citation-topics
|
|
7
|
+
Project-URL: Issues, https://github.com/restrepo/citation-topics/issues
|
|
8
|
+
Project-URL: Changelog, https://github.com/restrepo/citation-topics/blob/main/CHANGELOG.md
|
|
9
|
+
Author-email: Diego Restrepo <restrepo@udea.edu.co>
|
|
10
|
+
License-Expression: MIT
|
|
11
|
+
License-File: LICENSE
|
|
12
|
+
Keywords: bibliometrics,citation network,community detection,leiden,scientometrics,topics
|
|
13
|
+
Classifier: Development Status :: 3 - Alpha
|
|
14
|
+
Classifier: Environment :: Console
|
|
15
|
+
Classifier: Intended Audience :: Science/Research
|
|
16
|
+
Classifier: Operating System :: OS Independent
|
|
17
|
+
Classifier: Programming Language :: Python :: 3
|
|
18
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
19
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
20
|
+
Classifier: Typing :: Typed
|
|
21
|
+
Requires-Python: >=3.10
|
|
22
|
+
Requires-Dist: igraph>=0.11
|
|
23
|
+
Requires-Dist: leidenalg>=0.10
|
|
24
|
+
Requires-Dist: numpy>=1.24
|
|
25
|
+
Provides-Extra: dev
|
|
26
|
+
Requires-Dist: build>=1.2; extra == 'dev'
|
|
27
|
+
Requires-Dist: pytest-cov>=5; extra == 'dev'
|
|
28
|
+
Requires-Dist: pytest>=8; extra == 'dev'
|
|
29
|
+
Requires-Dist: ruff==0.17.3; extra == 'dev'
|
|
30
|
+
Requires-Dist: twine>=5; extra == 'dev'
|
|
31
|
+
Description-Content-Type: text/markdown
|
|
32
|
+
|
|
33
|
+
# citation-topics
|
|
34
|
+
|
|
35
|
+
[](https://github.com/restrepo/citation-topics/actions/workflows/ci.yml)
|
|
36
|
+
[](https://pypi.org/project/citation-topics/)
|
|
37
|
+
[](https://pypi.org/project/citation-topics/)
|
|
38
|
+
|
|
39
|
+
Research topics and broad areas for any list of works (papers, theses, projects, patents) from their citation
|
|
40
|
+
links, with Leiden clustering of the direct-citation network as CWTS does for the OpenAlex topics. It also gives
|
|
41
|
+
each work a second and a third topic, describes every topic from whatever the input carries (titles, abstracts,
|
|
42
|
+
keywords, journals, categories, countries), and ships Claude skills and an agent that name the topics.
|
|
43
|
+
|
|
44
|
+
It was extracted from [restrepo/inspireauth10](https://github.com/restrepo/inspireauth10), where it groups 1.9
|
|
45
|
+
million INSPIRE-HEP papers into 304 topics and 6 areas
|
|
46
|
+
([how the topics are made](https://restrepo.github.io/inspireauth10/leiden.html)).
|
|
47
|
+
|
|
48
|
+
## Install
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
pip install citation-topics
|
|
52
|
+
citation-topics --help
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## Input
|
|
56
|
+
|
|
57
|
+
A JSON array or JSON Lines file (optionally gzipped), one object per work:
|
|
58
|
+
|
|
59
|
+
```json
|
|
60
|
+
{"id": "W1", "references": ["W7", "W9"], "year": 2019, "title": "...", "abstract": "...",
|
|
61
|
+
"keywords": ["..."], "journal": "...", "categories": ["..."], "countries": ["..."],
|
|
62
|
+
"authors": ["..."], "citations": 41, "url": "https://..."}
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
- `id` (unique; strings or integers, compared as strings) and `references` (ids of other works) are required.
|
|
66
|
+
References to ids outside the list and self-citations are ignored and counted.
|
|
67
|
+
- Every other field is optional. Other names are read with `--id-field`, `--refs-field` and
|
|
68
|
+
`--map NAME=FIELD` (for example `--map year=publication_year`).
|
|
69
|
+
|
|
70
|
+
| Optional field | Adds |
|
|
71
|
+
|---|---|
|
|
72
|
+
| `year` | years of each topic, time windows (`--from-year`, `--to-year`), comparisons between epochs |
|
|
73
|
+
| `title`, `abstract`, `keywords`, `journal`, `categories` | the description of each topic (`describe`), hence its name |
|
|
74
|
+
| `citations` | most cited works by that count (otherwise by citations inside the list) |
|
|
75
|
+
| `countries` | countries of each topic in the description |
|
|
76
|
+
| `authors`, `url` | counted in `meta.json`; `url` is copied to the most cited works of the description |
|
|
77
|
+
|
|
78
|
+
## Commands
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
citation-topics run works.jsonl.gz out/ # network, topics, areas, second and third topics
|
|
82
|
+
citation-topics describe works.jsonl.gz out/ # out/dossier.json: what each topic is about
|
|
83
|
+
citation-topics names out/ names.json # put names into topics.json and areas.json
|
|
84
|
+
citation-topics export out/ # VOSviewer JSON and map/network files, CSVs
|
|
85
|
+
citation-topics agreement out_s1/ out_s2/ ... # NMI/ARI between seeds, and the medoid to keep
|
|
86
|
+
citation-topics compare old/ new/ cmp/ # where earlier topics went; names carried over
|
|
87
|
+
citation-topics skills install # Claude skills and the topic-namer agent into ./.claude
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
`run` options and defaults (those of the INSPIRE run): `--resolution 16` (higher = more, smaller topics),
|
|
91
|
+
`--min-size 100` (smaller topics are merged into the topic they share the most links with), `--passes 20`
|
|
92
|
+
(stops early when the quality stops rising), `--seed 1`, `--area-resolution 0.3`, `--area-seeds 50` (the best
|
|
93
|
+
of that many runs). Tune them for other lists; the skill `topics-run` explains how.
|
|
94
|
+
|
|
95
|
+
## Method
|
|
96
|
+
|
|
97
|
+
1. **Network**: one node per work, an edge for each reference to another work of the list; undirected, each
|
|
98
|
+
pair of works once.
|
|
99
|
+
2. **Topics**: Leiden maximizing modularity with a resolution parameter (the RB configuration model), with
|
|
100
|
+
igraph's `community_leiden`, pass after pass until the quality stops rising. Works with no link stay
|
|
101
|
+
unassigned; topics under `--min-size` are merged into their most linked neighbour; topics are numbered by
|
|
102
|
+
size.
|
|
103
|
+
3. **Areas**: Leiden (leidenalg) on the graph of topics weighted by the links between them, with the links
|
|
104
|
+
inside each topic as self-loops, the best of `--area-seeds` runs. Every topic is in one area.
|
|
105
|
+
4. **Second and third topics**: a work's links are counted by the topic at the other end; the second and third
|
|
106
|
+
topics are the most linked other topics, scored as links to them / links to the main topic, capped at 1.
|
|
107
|
+
Works the clustering left out take as main topic the one they link to most (`inferred`).
|
|
108
|
+
5. **Description** (`describe`): per topic, its distinctive terms (shares in the topic against the rest of the
|
|
109
|
+
list), journals, categories, keywords, countries, years, linked topics and most cited works.
|
|
110
|
+
6. **Names**: written by Claude from the description (skill `topic-namer`), never by the package. `compare`
|
|
111
|
+
carries names over from an earlier run: a topic sharing 60% of its works both ways with an earlier one keeps
|
|
112
|
+
its name.
|
|
113
|
+
|
|
114
|
+
The same input, parameters and versions of igraph and leidenalg give the same topics; `meta.json` records them
|
|
115
|
+
with the MD5 of the input.
|
|
116
|
+
|
|
117
|
+
**Checked against inspireauth10.** On its 1,872,221 INSPIRE papers (44 million citation links) with the
|
|
118
|
+
defaults, `run` gives the same network, the same 304 topics and 6 areas, the same Leiden quality and the same
|
|
119
|
+
second and third topics as inspireauth10's own scripts, array for array (igraph 1.0.0, leidenalg 0.12.0, numpy
|
|
120
|
+
2.5.4, Python 3.13). It took 45 minutes and more than 4 GB of memory; `describe` with titles, keywords,
|
|
121
|
+
journals, categories and countries took 10 minutes and 4.4 GB.
|
|
122
|
+
|
|
123
|
+
## Output
|
|
124
|
+
|
|
125
|
+
| File | Content |
|
|
126
|
+
|---|---|
|
|
127
|
+
| `works.jsonl.gz` | per work: `id`, `topic`, `area` (null if none), `topics` (three, with `score` and `links`), `n_links`, `inferred` |
|
|
128
|
+
| `topics.json` | per topic: area, name, works, links inside and out, `share_inside`, years, most cited works |
|
|
129
|
+
| `areas.json` | per area: name, works, topics, links, `share_inside` |
|
|
130
|
+
| `meta.json` | parameters, versions, input MD5, network counts, optional fields present, quality |
|
|
131
|
+
| `dossier.json` | description of each topic and area (`describe`) |
|
|
132
|
+
| `topics_vosviewer.json`, `topics_map.txt`, `topics_network.txt` | graph of topics for VOSviewer (`export`) |
|
|
133
|
+
| `topics.csv`, `works_topics.csv.gz` | the same as tables (`export`) |
|
|
134
|
+
| `network.npz`, `clusters.npz` | arrays reused by the other commands |
|
|
135
|
+
|
|
136
|
+
## Claude skills
|
|
137
|
+
|
|
138
|
+
`citation-topics skills install` copies into `./.claude` (or `~/.claude` with `--user`):
|
|
139
|
+
|
|
140
|
+
- `topics-run`: checks the input, chooses the parameters, runs, describes and exports, and explains the result;
|
|
141
|
+
- `topic-namer`: names topics and areas from `dossier.json` and applies them, after carrying over earlier names;
|
|
142
|
+
- `topics-compare`: seeds (which run to keep) and earlier runs or time windows (fates, flows, names);
|
|
143
|
+
- the agent `topic-namer`, which names a batch of topics when the skill hands it one.
|
|
144
|
+
|
|
145
|
+
In Claude Desktop or Claude Code, ask for topics of a list of works and the skills take it from there.
|
|
146
|
+
|
|
147
|
+
## Development
|
|
148
|
+
|
|
149
|
+
```bash
|
|
150
|
+
python -m pip install -e ".[dev]"
|
|
151
|
+
python -m pytest
|
|
152
|
+
ruff check . && ruff format --check .
|
|
153
|
+
```
|
|
154
|
+
|
|
155
|
+
Releases are published to PyPI from GitHub Actions when a `v*` tag is pushed; see [PUBLISHING.md](PUBLISHING.md)
|
|
156
|
+
and [CHANGELOG.md](CHANGELOG.md).
|
|
@@ -0,0 +1,124 @@
|
|
|
1
|
+
# citation-topics
|
|
2
|
+
|
|
3
|
+
[](https://github.com/restrepo/citation-topics/actions/workflows/ci.yml)
|
|
4
|
+
[](https://pypi.org/project/citation-topics/)
|
|
5
|
+
[](https://pypi.org/project/citation-topics/)
|
|
6
|
+
|
|
7
|
+
Research topics and broad areas for any list of works (papers, theses, projects, patents) from their citation
|
|
8
|
+
links, with Leiden clustering of the direct-citation network as CWTS does for the OpenAlex topics. It also gives
|
|
9
|
+
each work a second and a third topic, describes every topic from whatever the input carries (titles, abstracts,
|
|
10
|
+
keywords, journals, categories, countries), and ships Claude skills and an agent that name the topics.
|
|
11
|
+
|
|
12
|
+
It was extracted from [restrepo/inspireauth10](https://github.com/restrepo/inspireauth10), where it groups 1.9
|
|
13
|
+
million INSPIRE-HEP papers into 304 topics and 6 areas
|
|
14
|
+
([how the topics are made](https://restrepo.github.io/inspireauth10/leiden.html)).
|
|
15
|
+
|
|
16
|
+
## Install
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
pip install citation-topics
|
|
20
|
+
citation-topics --help
|
|
21
|
+
```
|
|
22
|
+
|
|
23
|
+
## Input
|
|
24
|
+
|
|
25
|
+
A JSON array or JSON Lines file (optionally gzipped), one object per work:
|
|
26
|
+
|
|
27
|
+
```json
|
|
28
|
+
{"id": "W1", "references": ["W7", "W9"], "year": 2019, "title": "...", "abstract": "...",
|
|
29
|
+
"keywords": ["..."], "journal": "...", "categories": ["..."], "countries": ["..."],
|
|
30
|
+
"authors": ["..."], "citations": 41, "url": "https://..."}
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
- `id` (unique; strings or integers, compared as strings) and `references` (ids of other works) are required.
|
|
34
|
+
References to ids outside the list and self-citations are ignored and counted.
|
|
35
|
+
- Every other field is optional. Other names are read with `--id-field`, `--refs-field` and
|
|
36
|
+
`--map NAME=FIELD` (for example `--map year=publication_year`).
|
|
37
|
+
|
|
38
|
+
| Optional field | Adds |
|
|
39
|
+
|---|---|
|
|
40
|
+
| `year` | years of each topic, time windows (`--from-year`, `--to-year`), comparisons between epochs |
|
|
41
|
+
| `title`, `abstract`, `keywords`, `journal`, `categories` | the description of each topic (`describe`), hence its name |
|
|
42
|
+
| `citations` | most cited works by that count (otherwise by citations inside the list) |
|
|
43
|
+
| `countries` | countries of each topic in the description |
|
|
44
|
+
| `authors`, `url` | counted in `meta.json`; `url` is copied to the most cited works of the description |
|
|
45
|
+
|
|
46
|
+
## Commands
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
citation-topics run works.jsonl.gz out/ # network, topics, areas, second and third topics
|
|
50
|
+
citation-topics describe works.jsonl.gz out/ # out/dossier.json: what each topic is about
|
|
51
|
+
citation-topics names out/ names.json # put names into topics.json and areas.json
|
|
52
|
+
citation-topics export out/ # VOSviewer JSON and map/network files, CSVs
|
|
53
|
+
citation-topics agreement out_s1/ out_s2/ ... # NMI/ARI between seeds, and the medoid to keep
|
|
54
|
+
citation-topics compare old/ new/ cmp/ # where earlier topics went; names carried over
|
|
55
|
+
citation-topics skills install # Claude skills and the topic-namer agent into ./.claude
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
`run` options and defaults (those of the INSPIRE run): `--resolution 16` (higher = more, smaller topics),
|
|
59
|
+
`--min-size 100` (smaller topics are merged into the topic they share the most links with), `--passes 20`
|
|
60
|
+
(stops early when the quality stops rising), `--seed 1`, `--area-resolution 0.3`, `--area-seeds 50` (the best
|
|
61
|
+
of that many runs). Tune them for other lists; the skill `topics-run` explains how.
|
|
62
|
+
|
|
63
|
+
## Method
|
|
64
|
+
|
|
65
|
+
1. **Network**: one node per work, an edge for each reference to another work of the list; undirected, each
|
|
66
|
+
pair of works once.
|
|
67
|
+
2. **Topics**: Leiden maximizing modularity with a resolution parameter (the RB configuration model), with
|
|
68
|
+
igraph's `community_leiden`, pass after pass until the quality stops rising. Works with no link stay
|
|
69
|
+
unassigned; topics under `--min-size` are merged into their most linked neighbour; topics are numbered by
|
|
70
|
+
size.
|
|
71
|
+
3. **Areas**: Leiden (leidenalg) on the graph of topics weighted by the links between them, with the links
|
|
72
|
+
inside each topic as self-loops, the best of `--area-seeds` runs. Every topic is in one area.
|
|
73
|
+
4. **Second and third topics**: a work's links are counted by the topic at the other end; the second and third
|
|
74
|
+
topics are the most linked other topics, scored as links to them / links to the main topic, capped at 1.
|
|
75
|
+
Works the clustering left out take as main topic the one they link to most (`inferred`).
|
|
76
|
+
5. **Description** (`describe`): per topic, its distinctive terms (shares in the topic against the rest of the
|
|
77
|
+
list), journals, categories, keywords, countries, years, linked topics and most cited works.
|
|
78
|
+
6. **Names**: written by Claude from the description (skill `topic-namer`), never by the package. `compare`
|
|
79
|
+
carries names over from an earlier run: a topic sharing 60% of its works both ways with an earlier one keeps
|
|
80
|
+
its name.
|
|
81
|
+
|
|
82
|
+
The same input, parameters and versions of igraph and leidenalg give the same topics; `meta.json` records them
|
|
83
|
+
with the MD5 of the input.
|
|
84
|
+
|
|
85
|
+
**Checked against inspireauth10.** On its 1,872,221 INSPIRE papers (44 million citation links) with the
|
|
86
|
+
defaults, `run` gives the same network, the same 304 topics and 6 areas, the same Leiden quality and the same
|
|
87
|
+
second and third topics as inspireauth10's own scripts, array for array (igraph 1.0.0, leidenalg 0.12.0, numpy
|
|
88
|
+
2.5.4, Python 3.13). It took 45 minutes and more than 4 GB of memory; `describe` with titles, keywords,
|
|
89
|
+
journals, categories and countries took 10 minutes and 4.4 GB.
|
|
90
|
+
|
|
91
|
+
## Output
|
|
92
|
+
|
|
93
|
+
| File | Content |
|
|
94
|
+
|---|---|
|
|
95
|
+
| `works.jsonl.gz` | per work: `id`, `topic`, `area` (null if none), `topics` (three, with `score` and `links`), `n_links`, `inferred` |
|
|
96
|
+
| `topics.json` | per topic: area, name, works, links inside and out, `share_inside`, years, most cited works |
|
|
97
|
+
| `areas.json` | per area: name, works, topics, links, `share_inside` |
|
|
98
|
+
| `meta.json` | parameters, versions, input MD5, network counts, optional fields present, quality |
|
|
99
|
+
| `dossier.json` | description of each topic and area (`describe`) |
|
|
100
|
+
| `topics_vosviewer.json`, `topics_map.txt`, `topics_network.txt` | graph of topics for VOSviewer (`export`) |
|
|
101
|
+
| `topics.csv`, `works_topics.csv.gz` | the same as tables (`export`) |
|
|
102
|
+
| `network.npz`, `clusters.npz` | arrays reused by the other commands |
|
|
103
|
+
|
|
104
|
+
## Claude skills
|
|
105
|
+
|
|
106
|
+
`citation-topics skills install` copies into `./.claude` (or `~/.claude` with `--user`):
|
|
107
|
+
|
|
108
|
+
- `topics-run`: checks the input, chooses the parameters, runs, describes and exports, and explains the result;
|
|
109
|
+
- `topic-namer`: names topics and areas from `dossier.json` and applies them, after carrying over earlier names;
|
|
110
|
+
- `topics-compare`: seeds (which run to keep) and earlier runs or time windows (fates, flows, names);
|
|
111
|
+
- the agent `topic-namer`, which names a batch of topics when the skill hands it one.
|
|
112
|
+
|
|
113
|
+
In Claude Desktop or Claude Code, ask for topics of a list of works and the skills take it from there.
|
|
114
|
+
|
|
115
|
+
## Development
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
python -m pip install -e ".[dev]"
|
|
119
|
+
python -m pytest
|
|
120
|
+
ruff check . && ruff format --check .
|
|
121
|
+
```
|
|
122
|
+
|
|
123
|
+
Releases are published to PyPI from GitHub Actions when a `v*` tag is pushed; see [PUBLISHING.md](PUBLISHING.md)
|
|
124
|
+
and [CHANGELOG.md](CHANGELOG.md).
|
|
@@ -0,0 +1,67 @@
|
|
|
1
|
+
[build-system]
|
|
2
|
+
requires = ["hatchling>=1.24", "hatch-vcs>=0.4"]
|
|
3
|
+
build-backend = "hatchling.build"
|
|
4
|
+
|
|
5
|
+
[project]
|
|
6
|
+
name = "citation-topics"
|
|
7
|
+
dynamic = ["version"]
|
|
8
|
+
description = "Research topics and areas from the citation links of any list of works, with Leiden clustering as CWTS does for OpenAlex."
|
|
9
|
+
readme = "README.md"
|
|
10
|
+
license = "MIT"
|
|
11
|
+
license-files = ["LICENSE"]
|
|
12
|
+
requires-python = ">=3.10"
|
|
13
|
+
authors = [{ name = "Diego Restrepo", email = "restrepo@udea.edu.co" }]
|
|
14
|
+
keywords = ["bibliometrics", "citation network", "leiden", "community detection", "topics", "scientometrics"]
|
|
15
|
+
classifiers = [
|
|
16
|
+
"Development Status :: 3 - Alpha",
|
|
17
|
+
"Environment :: Console",
|
|
18
|
+
"Operating System :: OS Independent",
|
|
19
|
+
"Programming Language :: Python :: 3",
|
|
20
|
+
"Programming Language :: Python :: 3 :: Only",
|
|
21
|
+
"Intended Audience :: Science/Research",
|
|
22
|
+
"Topic :: Scientific/Engineering :: Information Analysis",
|
|
23
|
+
"Typing :: Typed",
|
|
24
|
+
]
|
|
25
|
+
dependencies = ["numpy>=1.24", "igraph>=0.11", "leidenalg>=0.10"]
|
|
26
|
+
|
|
27
|
+
[project.optional-dependencies]
|
|
28
|
+
dev = ["pytest>=8", "pytest-cov>=5", "ruff==0.17.3", "build>=1.2", "twine>=5"]
|
|
29
|
+
|
|
30
|
+
[project.scripts]
|
|
31
|
+
citation-topics = "citation_topics.cli:main"
|
|
32
|
+
|
|
33
|
+
[project.urls]
|
|
34
|
+
Homepage = "https://github.com/restrepo/citation-topics"
|
|
35
|
+
Repository = "https://github.com/restrepo/citation-topics"
|
|
36
|
+
Issues = "https://github.com/restrepo/citation-topics/issues"
|
|
37
|
+
Changelog = "https://github.com/restrepo/citation-topics/blob/main/CHANGELOG.md"
|
|
38
|
+
|
|
39
|
+
# Version from the git tag (v0.1.0 -> 0.1.0); commits after a tag get 0.1.1.devN
|
|
40
|
+
[tool.hatch.version]
|
|
41
|
+
source = "vcs"
|
|
42
|
+
|
|
43
|
+
[tool.hatch.version.raw-options]
|
|
44
|
+
local_scheme = "no-local-version" # PyPI and TestPyPI reject local versions (+g1234abc)
|
|
45
|
+
|
|
46
|
+
[tool.hatch.build.targets.sdist]
|
|
47
|
+
include = ["src/", "tests/", "README.md", "CHANGELOG.md", "LICENSE"]
|
|
48
|
+
|
|
49
|
+
[tool.hatch.build.targets.wheel]
|
|
50
|
+
packages = ["src/citation_topics"]
|
|
51
|
+
|
|
52
|
+
[tool.pytest.ini_options]
|
|
53
|
+
testpaths = ["tests"]
|
|
54
|
+
addopts = "-ra --strict-markers"
|
|
55
|
+
|
|
56
|
+
[tool.coverage.run]
|
|
57
|
+
source = ["citation_topics"]
|
|
58
|
+
branch = true
|
|
59
|
+
|
|
60
|
+
[tool.ruff]
|
|
61
|
+
line-length = 120
|
|
62
|
+
target-version = "py310"
|
|
63
|
+
src = ["src", "tests"]
|
|
64
|
+
|
|
65
|
+
[tool.ruff.lint]
|
|
66
|
+
select = ["E", "F", "W", "I", "B", "UP", "SIM", "RUF"]
|
|
67
|
+
ignore = ["B905"] # zip() over arrays of equal length by construction
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
"""Research topics and areas from the citation links of any list of works, with Leiden clustering as CWTS does
|
|
2
|
+
for the OpenAlex topics."""
|
|
3
|
+
|
|
4
|
+
from importlib.metadata import PackageNotFoundError, version
|
|
5
|
+
|
|
6
|
+
try:
|
|
7
|
+
__version__ = version("citation-topics")
|
|
8
|
+
except PackageNotFoundError: # running from a source tree that was never installed
|
|
9
|
+
__version__ = "0.0.0"
|
|
10
|
+
|
|
11
|
+
__all__ = ["__version__"]
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: topic-namer
|
|
3
|
+
description: Names a batch of citation-topics topics or areas from OUT/dossier.json and writes them to a names JSON file. Give it the dossier path, the topic (or area) numbers, the output file, and the names already used elsewhere so it avoids duplicates.
|
|
4
|
+
tools: Read, Write, Bash
|
|
5
|
+
---
|
|
6
|
+
|
|
7
|
+
You name research topics from their dossier. Read the entries of the requested topics (or areas) in the given
|
|
8
|
+
`dossier.json` (with `python3 -c` or `jq` if the file is large; do not load more than you need). For each:
|
|
9
|
+
|
|
10
|
+
- Write a name of 2 to 7 words (areas: 3 to 10) that says what sets it apart, grounded in its `terms`,
|
|
11
|
+
`most_cited` titles and abstracts, and checked against `journals`, `categories` and `keywords`.
|
|
12
|
+
- No generic names, no lists of terms, no duplicates among the batch or of the names you were told are taken;
|
|
13
|
+
use `linked_topics` to tell neighbours apart.
|
|
14
|
+
- Add a one-line `reason` citing the evidence.
|
|
15
|
+
|
|
16
|
+
Write the requested file as
|
|
17
|
+
`{"topics": {"N": {"name": "...", "reason": "..."}}, "meta": {"by": "topic-namer", "model": "<your model>", "date": "YYYY-MM-DD"}}`
|
|
18
|
+
(use `"areas"` instead of `"topics"` for areas) and reply with a short table of number and name. Do not run
|
|
19
|
+
`citation-topics names`; the caller applies the files.
|
|
@@ -0,0 +1,63 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: topic-namer
|
|
3
|
+
description: Name the topics and areas found by citation-topics from their dossier (dossier.json) - short, specific names written from each topic's distinctive terms and most cited titles, checked against its journals, categories and keywords - and apply them with `citation-topics names`; carries names over from an earlier run first so only new topics are named. Use whenever a citation-topics output folder has unnamed topics, the user asks to name, rename or check topic names, or after `citation-topics run` and `describe`.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Naming topics and areas
|
|
7
|
+
|
|
8
|
+
Names are the one step that needs reading; everything else is computed. They are written by Claude, so they
|
|
9
|
+
are not exactly repeatable: record who named them and when.
|
|
10
|
+
|
|
11
|
+
## 1. Prepare
|
|
12
|
+
|
|
13
|
+
- `OUT/dossier.json` must exist (`citation-topics describe INPUT OUT/`). Check its `missing` list: without
|
|
14
|
+
titles, keywords, journals or categories there is nothing to name from; tell the user and stop.
|
|
15
|
+
- If an earlier run of the same or an overlapping list was named, carry its names over first:
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
citation-topics compare OLD_OUT/ OUT/ CMP/ # CMP/names_carried.json: names + "to_name"
|
|
19
|
+
citation-topics names OUT/ CMP/names_carried.json
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Then name only the topics in `to_name`.
|
|
23
|
+
|
|
24
|
+
## 2. Name
|
|
25
|
+
|
|
26
|
+
For each topic, read in `dossier.json`: `terms` (with the share of its works using them), `most_cited` (titles,
|
|
27
|
+
years, and the start of the abstract of the first three), `journals`, `categories`, `keywords`, `years`,
|
|
28
|
+
`linked_topics`. Write:
|
|
29
|
+
|
|
30
|
+
- **2 to 7 words**, specific to what sets the topic apart: "Neutrino oscillations", "Lattice QCD at finite
|
|
31
|
+
temperature", "Dark matter direct detection". Not generic ("Physics", "Theory"), not a list of every term.
|
|
32
|
+
- Grounded in the dossier: the most cited titles and the top terms should agree; check the name against the
|
|
33
|
+
journals and categories. If they disagree, prefer what most of its works share (terms, categories) over a
|
|
34
|
+
single famous work.
|
|
35
|
+
- Distinct: two topics never get the same name; when two are close, name what differs (method, object,
|
|
36
|
+
energy range, period). `linked_topics` shows the neighbours to tell apart.
|
|
37
|
+
- A one-line `reason` citing the evidence (terms, titles, categories).
|
|
38
|
+
|
|
39
|
+
Areas: broader names (3 to 10 words) covering their topics (`areas[].topics` lists them with their names).
|
|
40
|
+
|
|
41
|
+
With many topics, work in batches of about 30. If the `topic-namer` agent is installed (`.claude/agents/`),
|
|
42
|
+
give each batch to it: the topic numbers, the path of `dossier.json` and the file to write; check its names
|
|
43
|
+
for duplicates afterwards.
|
|
44
|
+
|
|
45
|
+
## 3. Apply
|
|
46
|
+
|
|
47
|
+
Write `names.json`:
|
|
48
|
+
|
|
49
|
+
```json
|
|
50
|
+
{"topics": {"0": {"name": "...", "reason": "..."}},
|
|
51
|
+
"areas": {"0": {"name": "...", "reason": "..."}},
|
|
52
|
+
"meta": {"by": "topic-namer", "model": "<the model you are>", "date": "YYYY-MM-DD"}}
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
and merge it (it can be applied in parts; later files add to earlier ones):
|
|
56
|
+
|
|
57
|
+
```bash
|
|
58
|
+
citation-topics names OUT/ names.json
|
|
59
|
+
citation-topics export OUT/ # VOSviewer and CSVs with the names
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
The command lists any topic still unnamed. Show the user a table of number, works and name, and say which
|
|
63
|
+
names were carried over and which are new.
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: topics-compare
|
|
3
|
+
description: Compare runs of citation-topics - agreement between seeds (NMI, ARI, medoid to keep) and where the topics of an earlier run or time window went in a later one (same, absorbed, split, scattered, with flows between topics and names carried over). Use whenever the user asks how stable the topics are, which seed to keep, how topics defined up to some year changed, how a new version of a list changes its topics, or wants to keep names from a previous run.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Comparing runs of citation-topics
|
|
7
|
+
|
|
8
|
+
## Seeds of one list
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
for s in 1 2 3 4; do citation-topics run INPUT OUT_s$s/ --seed $s; done # same other parameters
|
|
12
|
+
citation-topics agreement OUT_s1/ OUT_s2/ OUT_s3/ OUT_s4/
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
It prints NMI and ARI for every pair, for topics and areas, on the works all runs assign, and the medoid (the
|
|
16
|
+
run with the highest mean topic NMI to the others): keep that one. Topic NMI around 0.85 between seeds is
|
|
17
|
+
usual for large citation networks; much lower means the resolution is too high for the list or the list is
|
|
18
|
+
small.
|
|
19
|
+
|
|
20
|
+
## An earlier run against a later one
|
|
21
|
+
|
|
22
|
+
The earlier run can be a time window of the same list (`run INPUT OLD/ --to-year 2010`) or a run of an older
|
|
23
|
+
version of the list. Works are joined by id.
|
|
24
|
+
|
|
25
|
+
```bash
|
|
26
|
+
citation-topics compare OLD/ NEW/ CMP/
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
`CMP/flows.json`:
|
|
30
|
+
|
|
31
|
+
- `old`: each earlier topic with its best new topic, `share` (of its works there), `back` (of that topic's
|
|
32
|
+
works that were in the earlier run) and `fate`: **same** (60% both ways), **absorbed** (60% one way),
|
|
33
|
+
**split** (two or more new topics with 15% each), **scattered**.
|
|
34
|
+
- `new`: each new topic with its works not in the earlier run and the earlier topic it is the same as.
|
|
35
|
+
- `flows`: `[old, new, works]` for every pair; `meta`: counts of fates, NMI and ARI.
|
|
36
|
+
|
|
37
|
+
`CMP/names_carried.json` holds the names of the earlier topics that are the same as a new one, ready for
|
|
38
|
+
`citation-topics names NEW/ CMP/names_carried.json`, and the topics left to name (skill `topic-namer`).
|
|
39
|
+
|
|
40
|
+
Report the counts of each fate, NMI and ARI, the largest splits and merges by name, and how many new topics
|
|
41
|
+
have most of their works after the earlier run.
|
|
@@ -0,0 +1,90 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: topics-run
|
|
3
|
+
description: Build research topics and broad areas for any list of works (papers, theses, projects, patents) from their citation links with the citation-topics package - Leiden clustering of the direct-citation network as CWTS does for the OpenAlex topics, plus second and third topics of every work, a description of each topic, names, and VOSviewer/CSV exports. Use whenever the user has a JSON or JSON Lines list of works with ids and references and wants topics, fields, clusters or a classification of them, wants to rerun or tune the clustering, or asks what the output files mean.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Topics of a list of works with citation-topics
|
|
7
|
+
|
|
8
|
+
The package does the computation; this skill decides the parameters and explains the result. Every step
|
|
9
|
+
writes into one output folder, `OUT/`.
|
|
10
|
+
|
|
11
|
+
## 1. Check the input
|
|
12
|
+
|
|
13
|
+
One JSON object per work, in a JSON array or JSON Lines file (`.gz` allowed):
|
|
14
|
+
|
|
15
|
+
```json
|
|
16
|
+
{"id": "W1", "references": ["W7", "W9"], "year": 2019, "title": "...", "abstract": "...",
|
|
17
|
+
"keywords": ["..."], "journal": "...", "categories": ["..."], "countries": ["..."], "authors": ["..."],
|
|
18
|
+
"citations": 41, "url": "https://..."}
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
Only `id` (unique) and `references` (ids of other works of the list) are required. Other names are read with
|
|
22
|
+
`--id-field`, `--refs-field` and `--map NAME=FIELD` (for example `--map year=publication_year`). References to
|
|
23
|
+
ids outside the list are ignored and counted. What each optional field adds:
|
|
24
|
+
|
|
25
|
+
| Field | Adds |
|
|
26
|
+
|---|---|
|
|
27
|
+
| `year` | years of each topic, time windows (`--from-year`, `--to-year`), comparisons between epochs |
|
|
28
|
+
| `title`, `abstract`, `keywords`, `journal`, `categories` | the description of each topic, hence its name |
|
|
29
|
+
| `citations` | most cited works by that count (otherwise by citations inside the list) |
|
|
30
|
+
| `countries`, `authors`, `url` | countries per topic in the description; kept for the boards; links |
|
|
31
|
+
|
|
32
|
+
Before a long run, look at a few lines of the input and tell the user which optional fields it has and what
|
|
33
|
+
will be missing.
|
|
34
|
+
|
|
35
|
+
## 2. Run
|
|
36
|
+
|
|
37
|
+
```bash
|
|
38
|
+
citation-topics run INPUT OUT/ [--resolution 16] [--min-size 100] [--passes 20] [--seed 1] \
|
|
39
|
+
[--area-resolution 0.3] [--area-seeds 50]
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
The defaults are those of the HEP run this package was made from (1.9 million works, 304 topics, 6 areas).
|
|
43
|
+
For another list, tune them:
|
|
44
|
+
|
|
45
|
+
- `--min-size`: the smallest topic. Aim for topics of a few hundred to a few thousand works; for a list of
|
|
46
|
+
N works, start near N / 5,000 (at least 20).
|
|
47
|
+
- `--resolution`: higher gives more, smaller topics. Run two or three values (for example 4, 8, 16) and compare
|
|
48
|
+
the number of topics, the share of links inside topics (`share_inside` in `topics.json`, higher is more
|
|
49
|
+
solid) and the unassigned works (`meta.json`).
|
|
50
|
+
- `--area-resolution`: 0.3 gave the steadiest areas in the HEP run; try 0.2-0.5 if the areas look too few or
|
|
51
|
+
too many.
|
|
52
|
+
- Seeds: Leiden depends on its random seed. For a result to keep, run 3-4 seeds (`--seed 1..4`, into different
|
|
53
|
+
folders) and keep the medoid with `citation-topics agreement OUT1 OUT2 OUT3 OUT4`.
|
|
54
|
+
|
|
55
|
+
Run large lists in the background. Measured on 1.9 million works and 44 million links: `run` 45 minutes with
|
|
56
|
+
20 passes (about 85 s per pass) and over 4 GB of memory; `describe` with titles 10 minutes and 4.4 GB. The same
|
|
57
|
+
input, parameters and versions (all in `meta.json`) give the same topics.
|
|
58
|
+
|
|
59
|
+
## 3. Describe and name
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
citation-topics describe INPUT OUT/ [--abstracts] # OUT/dossier.json
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Then follow the skill `topic-namer`: it names topics and areas from `dossier.json` and applies the names with
|
|
66
|
+
`citation-topics names OUT/ names.json`. Without titles, keywords, journals or categories the dossier has only
|
|
67
|
+
sizes, years and links; say so, and name only if the user can supply what a topic is about.
|
|
68
|
+
|
|
69
|
+
## 4. Export
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
citation-topics export OUT/ # topics_vosviewer.json, topics_map.txt, topics_network.txt, CSVs
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
`topics_vosviewer.json` opens in VOSviewer Online: `https://app.vosviewer.com/?json=URL` with a public URL of
|
|
76
|
+
the file.
|
|
77
|
+
|
|
78
|
+
## Output files
|
|
79
|
+
|
|
80
|
+
| File | Content |
|
|
81
|
+
|---|---|
|
|
82
|
+
| `works.jsonl.gz` | per work: `topic` and `area` from the clustering (null if none), `topics` (main, second, third with `score` = links to it / links to the main topic, capped at 1, and `links`), `n_links`, `inferred` (main topic from its links when the clustering left it out) |
|
|
83
|
+
| `topics.json` | per topic: area, name, works, links inside and out, `share_inside`, years, most cited works |
|
|
84
|
+
| `areas.json` | per area: name, works, topics, links, `share_inside` |
|
|
85
|
+
| `meta.json` | parameters, versions, input MD5, network counts, which optional fields were present, quality |
|
|
86
|
+
| `dossier.json` | description of each topic and area (after `describe`) |
|
|
87
|
+
| `network.npz`, `clusters.npz` | arrays used by the other commands |
|
|
88
|
+
|
|
89
|
+
Report the numbers that matter (topics, areas, works with a topic, unassigned, share of links inside) and
|
|
90
|
+
which steps the missing fields ruled out.
|