scopusflow 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- scopusflow-0.1.0/.gitattributes +4 -0
- scopusflow-0.1.0/.github/CODE_OF_CONDUCT.md +34 -0
- scopusflow-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md +31 -0
- scopusflow-0.1.0/.github/ISSUE_TEMPLATE/config.yml +5 -0
- scopusflow-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md +17 -0
- scopusflow-0.1.0/.github/PULL_REQUEST_TEMPLATE.md +11 -0
- scopusflow-0.1.0/.github/SECURITY.md +16 -0
- scopusflow-0.1.0/.github/dependabot.yml +21 -0
- scopusflow-0.1.0/.github/workflows/docs.yml +27 -0
- scopusflow-0.1.0/.github/workflows/publish.yml +52 -0
- scopusflow-0.1.0/.github/workflows/tests.yaml +36 -0
- scopusflow-0.1.0/.gitignore +27 -0
- scopusflow-0.1.0/CHANGELOG.md +81 -0
- scopusflow-0.1.0/CITATION.cff +27 -0
- scopusflow-0.1.0/CONTRIBUTING.md +69 -0
- scopusflow-0.1.0/LICENSE +21 -0
- scopusflow-0.1.0/PKG-INFO +146 -0
- scopusflow-0.1.0/README.md +108 -0
- scopusflow-0.1.0/docs/about.md +60 -0
- scopusflow-0.1.0/docs/assets/favicon.png +0 -0
- scopusflow-0.1.0/docs/assets/favicon.svg +1 -0
- scopusflow-0.1.0/docs/assets/logo.png +0 -0
- scopusflow-0.1.0/docs/changelog.md +1 -0
- scopusflow-0.1.0/docs/guides/analysing-a-literature.md +111 -0
- scopusflow-0.1.0/docs/guides/building-a-reference-set.md +120 -0
- scopusflow-0.1.0/docs/guides/comparing-topics.md +219 -0
- scopusflow-0.1.0/docs/guides/designing-queries.md +139 -0
- scopusflow-0.1.0/docs/guides/getting-started.md +95 -0
- scopusflow-0.1.0/docs/guides/keywords-and-references.md +111 -0
- scopusflow-0.1.0/docs/guides/plans-and-quota.md +136 -0
- scopusflow-0.1.0/docs/guides/the-app.md +138 -0
- scopusflow-0.1.0/docs/guides/tracking-literature-change.md +138 -0
- scopusflow-0.1.0/docs/index.md +65 -0
- scopusflow-0.1.0/docs/licence.md +15 -0
- scopusflow-0.1.0/docs/reference/analyse.md +18 -0
- scopusflow-0.1.0/docs/reference/dois.md +7 -0
- scopusflow-0.1.0/docs/reference/plan-and-query.md +13 -0
- scopusflow-0.1.0/docs/reference/records.md +14 -0
- scopusflow-0.1.0/docs/reference/retrieve.md +14 -0
- scopusflow-0.1.0/docs/stylesheets/brand.css +338 -0
- scopusflow-0.1.0/examples/quickstart.py +84 -0
- scopusflow-0.1.0/mkdocs.yml +111 -0
- scopusflow-0.1.0/overrides/partials/header.html +84 -0
- scopusflow-0.1.0/pyproject.toml +51 -0
- scopusflow-0.1.0/src/scopusflow/__init__.py +51 -0
- scopusflow-0.1.0/src/scopusflow/abstract.py +315 -0
- scopusflow-0.1.0/src/scopusflow/app.py +561 -0
- scopusflow-0.1.0/src/scopusflow/app_helpers.py +115 -0
- scopusflow-0.1.0/src/scopusflow/compare.py +123 -0
- scopusflow-0.1.0/src/scopusflow/corpus.py +92 -0
- scopusflow-0.1.0/src/scopusflow/count.py +39 -0
- scopusflow-0.1.0/src/scopusflow/diff.py +58 -0
- scopusflow-0.1.0/src/scopusflow/exceptions.py +18 -0
- scopusflow-0.1.0/src/scopusflow/export.py +138 -0
- scopusflow-0.1.0/src/scopusflow/fetch.py +169 -0
- scopusflow-0.1.0/src/scopusflow/plan.py +63 -0
- scopusflow-0.1.0/src/scopusflow/plots.py +344 -0
- scopusflow-0.1.0/src/scopusflow/py.typed +0 -0
- scopusflow-0.1.0/src/scopusflow/query.py +53 -0
- scopusflow-0.1.0/src/scopusflow/records.py +85 -0
- scopusflow-0.1.0/src/scopusflow/trend.py +55 -0
- scopusflow-0.1.0/tests/test_abstract.py +319 -0
- scopusflow-0.1.0/tests/test_app.py +104 -0
- scopusflow-0.1.0/tests/test_compare.py +207 -0
- scopusflow-0.1.0/tests/test_core.py +68 -0
- scopusflow-0.1.0/tests/test_corpus.py +112 -0
- scopusflow-0.1.0/tests/test_export.py +111 -0
- scopusflow-0.1.0/tests/test_fetch.py +188 -0
- scopusflow-0.1.0/tests/test_plots.py +60 -0
- scopusflow-0.1.0/tests/test_records.py +52 -0
- scopusflow-0.1.0/tests/test_trend.py +36 -0
|
@@ -0,0 +1,34 @@
|
|
|
1
|
+
# Code of Conduct
|
|
2
|
+
|
|
3
|
+
## Our pledge
|
|
4
|
+
|
|
5
|
+
We as members, contributors and maintainers pledge to make participation in the
|
|
6
|
+
scopusflow project a welcoming and harassment-free experience for everyone,
|
|
7
|
+
regardless of age, body size, visible or invisible disability, ethnicity, sex
|
|
8
|
+
characteristics, gender identity and expression, level of experience, education,
|
|
9
|
+
socio-economic status, nationality, personal appearance, race, religion or
|
|
10
|
+
sexual identity and orientation.
|
|
11
|
+
|
|
12
|
+
## Our standards
|
|
13
|
+
|
|
14
|
+
Examples of behaviour that helps create a positive environment include showing
|
|
15
|
+
empathy and kindness, respecting differing opinions and experiences, giving and
|
|
16
|
+
gracefully accepting constructive feedback, and focusing on what is best for the
|
|
17
|
+
community. Unacceptable behaviour includes harassment, insulting or derogatory
|
|
18
|
+
comments, personal or political attacks, publishing others' private information
|
|
19
|
+
without permission, and any conduct that would reasonably be considered
|
|
20
|
+
inappropriate in a professional setting.
|
|
21
|
+
|
|
22
|
+
## Enforcement
|
|
23
|
+
|
|
24
|
+
Instances of unacceptable behaviour may be reported to the maintainer at
|
|
25
|
+
pcbernabeu@gmail.com. All complaints will be reviewed and investigated promptly
|
|
26
|
+
and fairly, and the maintainer is obligated to respect the privacy and security
|
|
27
|
+
of the reporter of any incident.
|
|
28
|
+
|
|
29
|
+
## Attribution
|
|
30
|
+
|
|
31
|
+
This Code of Conduct is adapted from the
|
|
32
|
+
[Contributor Covenant](https://www.contributor-covenant.org), version 2.1,
|
|
33
|
+
available at
|
|
34
|
+
<https://www.contributor-covenant.org/version/2/1/code_of_conduct.html>.
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Bug report
|
|
3
|
+
about: Report something that is not working as expected
|
|
4
|
+
title: ''
|
|
5
|
+
labels: bug
|
|
6
|
+
assignees: ''
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
**Describe the bug**
|
|
10
|
+
A clear description of what went wrong and what you expected instead.
|
|
11
|
+
|
|
12
|
+
**Reproducible example**
|
|
13
|
+
A small, self-contained example helps a great deal. Please never paste your
|
|
14
|
+
Scopus API key into an issue; replace it with a placeholder.
|
|
15
|
+
|
|
16
|
+
```python
|
|
17
|
+
# your example here
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
**Environment**
|
|
21
|
+
<details>
|
|
22
|
+
|
|
23
|
+
```
|
|
24
|
+
# Python version, scopusflow version (scopusflow.__version__),
|
|
25
|
+
# operating system, and pybliometrics version if the problem involves retrieval.
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
</details>
|
|
29
|
+
|
|
30
|
+
**Anything else**
|
|
31
|
+
Any other context that might help.
|
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: Feature request
|
|
3
|
+
about: Suggest an idea or improvement
|
|
4
|
+
title: ''
|
|
5
|
+
labels: enhancement
|
|
6
|
+
assignees: ''
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
**What would you like to be able to do?**
|
|
10
|
+
Describe the workflow or capability you have in mind.
|
|
11
|
+
|
|
12
|
+
**Why is it useful?**
|
|
13
|
+
What problem does it solve, and who benefits?
|
|
14
|
+
|
|
15
|
+
**Possible approach**
|
|
16
|
+
If you have a sense of how it might work, sketch it here. An API or function
|
|
17
|
+
shape is welcome but not required.
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
<!-- Thank you for contributing to scopusflow. -->
|
|
2
|
+
|
|
3
|
+
**What does this change do?**
|
|
4
|
+
A short description, and a link to the issue it addresses if there is one.
|
|
5
|
+
|
|
6
|
+
**Checklist**
|
|
7
|
+
|
|
8
|
+
- [ ] `pytest` passes, and new behaviour has tests.
|
|
9
|
+
- [ ] `ruff check` is clean.
|
|
10
|
+
- [ ] The docs build (`mkdocs build --strict`) and `CHANGELOG.md` is updated where relevant.
|
|
11
|
+
- [ ] No API key or other secret appears anywhere in the diff.
|
|
@@ -0,0 +1,16 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## Reporting a vulnerability
|
|
4
|
+
|
|
5
|
+
If you find a security problem in scopusflow, please report it privately by
|
|
6
|
+
email to pcbernabeu@gmail.com rather than opening a public issue. A short
|
|
7
|
+
description of the problem and, where possible, a way to reproduce it is enough
|
|
8
|
+
to get started. You can expect an acknowledgement within a few days, and we will
|
|
9
|
+
keep you informed as the issue is investigated and resolved.
|
|
10
|
+
|
|
11
|
+
## A note on API keys
|
|
12
|
+
|
|
13
|
+
scopusflow never stores your Scopus API key. Retrieval runs through
|
|
14
|
+
pybliometrics, which reads the key from its own configuration file
|
|
15
|
+
(`~/.config/pybliometrics.cfg`). Keep the key there rather than in scripts or
|
|
16
|
+
notebooks, and never paste it into an issue, a pull request or a discussion.
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
version: 2
|
|
2
|
+
updates:
|
|
3
|
+
# Keep the GitHub Actions used by the workflows up to date.
|
|
4
|
+
- package-ecosystem: "github-actions"
|
|
5
|
+
directory: "/"
|
|
6
|
+
schedule:
|
|
7
|
+
interval: "weekly"
|
|
8
|
+
commit-message:
|
|
9
|
+
prefix: "ci"
|
|
10
|
+
labels:
|
|
11
|
+
- "dependencies"
|
|
12
|
+
|
|
13
|
+
# Track the Python dependencies declared in pyproject.toml.
|
|
14
|
+
- package-ecosystem: "pip"
|
|
15
|
+
directory: "/"
|
|
16
|
+
schedule:
|
|
17
|
+
interval: "weekly"
|
|
18
|
+
commit-message:
|
|
19
|
+
prefix: "deps"
|
|
20
|
+
labels:
|
|
21
|
+
- "dependencies"
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
name: docs
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main, master]
|
|
6
|
+
workflow_dispatch:
|
|
7
|
+
|
|
8
|
+
permissions:
|
|
9
|
+
contents: write
|
|
10
|
+
|
|
11
|
+
jobs:
|
|
12
|
+
deploy:
|
|
13
|
+
runs-on: ubuntu-latest
|
|
14
|
+
steps:
|
|
15
|
+
- uses: actions/checkout@v7
|
|
16
|
+
|
|
17
|
+
- uses: actions/setup-python@v6
|
|
18
|
+
with:
|
|
19
|
+
python-version: "3.12"
|
|
20
|
+
|
|
21
|
+
- name: Install docs dependencies
|
|
22
|
+
run: |
|
|
23
|
+
python -m pip install --upgrade pip
|
|
24
|
+
pip install -e ".[docs]"
|
|
25
|
+
|
|
26
|
+
- name: Build and deploy to GitHub Pages
|
|
27
|
+
run: mkdocs gh-deploy --force --no-history
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Build and publish a release to PyPI through trusted publishing (OIDC), so no
|
|
2
|
+
# API token is ever stored in the repository.
|
|
3
|
+
#
|
|
4
|
+
# One-time setup on PyPI: in the project's Publishing settings, add this
|
|
5
|
+
# repository as a trusted publisher with workflow file "publish.yml" and
|
|
6
|
+
# environment "pypi". The project does not exist on PyPI yet, so add it as a
|
|
7
|
+
# pending publisher (PyPI creates the project on the first upload). This workflow
|
|
8
|
+
# then publishes every release, starting with 0.1.0. It runs when a GitHub
|
|
9
|
+
# release is published.
|
|
10
|
+
name: Publish to PyPI
|
|
11
|
+
|
|
12
|
+
on:
|
|
13
|
+
release:
|
|
14
|
+
types: [published]
|
|
15
|
+
|
|
16
|
+
jobs:
|
|
17
|
+
build:
|
|
18
|
+
runs-on: ubuntu-latest
|
|
19
|
+
steps:
|
|
20
|
+
- uses: actions/checkout@v7
|
|
21
|
+
- uses: actions/setup-python@v6
|
|
22
|
+
with:
|
|
23
|
+
python-version: "3.12"
|
|
24
|
+
- name: Build the sdist and wheel
|
|
25
|
+
run: |
|
|
26
|
+
python -m pip install --upgrade build
|
|
27
|
+
python -m build
|
|
28
|
+
- name: Check the distribution
|
|
29
|
+
run: |
|
|
30
|
+
python -m pip install --upgrade twine
|
|
31
|
+
python -m twine check dist/*
|
|
32
|
+
- uses: actions/upload-artifact@v7
|
|
33
|
+
with:
|
|
34
|
+
name: dist
|
|
35
|
+
path: dist/
|
|
36
|
+
|
|
37
|
+
publish:
|
|
38
|
+
needs: build
|
|
39
|
+
runs-on: ubuntu-latest
|
|
40
|
+
environment:
|
|
41
|
+
name: pypi
|
|
42
|
+
url: https://pypi.org/project/scopusflow/
|
|
43
|
+
permissions:
|
|
44
|
+
id-token: write
|
|
45
|
+
steps:
|
|
46
|
+
- uses: actions/download-artifact@v8
|
|
47
|
+
with:
|
|
48
|
+
name: dist
|
|
49
|
+
path: dist/
|
|
50
|
+
- uses: pypa/gh-action-pypi-publish@release/v1
|
|
51
|
+
with:
|
|
52
|
+
skip-existing: true
|
|
@@ -0,0 +1,36 @@
|
|
|
1
|
+
name: tests
|
|
2
|
+
|
|
3
|
+
on:
|
|
4
|
+
push:
|
|
5
|
+
branches: [main, master]
|
|
6
|
+
pull_request:
|
|
7
|
+
workflow_dispatch:
|
|
8
|
+
|
|
9
|
+
permissions:
|
|
10
|
+
contents: read
|
|
11
|
+
|
|
12
|
+
jobs:
|
|
13
|
+
pytest:
|
|
14
|
+
runs-on: ubuntu-latest
|
|
15
|
+
strategy:
|
|
16
|
+
fail-fast: false
|
|
17
|
+
matrix:
|
|
18
|
+
python-version: ["3.10", "3.11", "3.12", "3.13"]
|
|
19
|
+
|
|
20
|
+
steps:
|
|
21
|
+
- uses: actions/checkout@v7
|
|
22
|
+
|
|
23
|
+
- uses: actions/setup-python@v6
|
|
24
|
+
with:
|
|
25
|
+
python-version: ${{ matrix.python-version }}
|
|
26
|
+
|
|
27
|
+
- name: Install with dev and plot extras
|
|
28
|
+
run: |
|
|
29
|
+
python -m pip install --upgrade pip
|
|
30
|
+
pip install -e ".[dev,plot]"
|
|
31
|
+
|
|
32
|
+
- name: Run the offline test suite
|
|
33
|
+
run: pytest -q
|
|
34
|
+
|
|
35
|
+
- name: Lint
|
|
36
|
+
run: ruff check .
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
# Python build / cache artefacts
|
|
2
|
+
__pycache__/
|
|
3
|
+
*.py[cod]
|
|
4
|
+
*.egg-info/
|
|
5
|
+
build/
|
|
6
|
+
dist/
|
|
7
|
+
.eggs/
|
|
8
|
+
|
|
9
|
+
# Test / tooling caches
|
|
10
|
+
.pytest_cache/
|
|
11
|
+
.ruff_cache/
|
|
12
|
+
.mypy_cache/
|
|
13
|
+
.coverage
|
|
14
|
+
htmlcov/
|
|
15
|
+
|
|
16
|
+
# Virtual environments
|
|
17
|
+
.venv/
|
|
18
|
+
venv/
|
|
19
|
+
env/
|
|
20
|
+
|
|
21
|
+
# MkDocs build output
|
|
22
|
+
site/
|
|
23
|
+
|
|
24
|
+
# Local harvest output / Scopus keys
|
|
25
|
+
harvest*/
|
|
26
|
+
*.parquet
|
|
27
|
+
pybliometrics.cfg
|
|
@@ -0,0 +1,81 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented here.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.1.0] - 2026-07-10
|
|
11
|
+
|
|
12
|
+
First release: a reproducible workflow layer over
|
|
13
|
+
[pybliometrics](https://pybliometrics.readthedocs.io) for Scopus searches, and
|
|
14
|
+
the Python twin of the R package
|
|
15
|
+
[scopusflow](https://pablobernabeu.github.io/scopusflow/).
|
|
16
|
+
|
|
17
|
+
### Added
|
|
18
|
+
|
|
19
|
+
- Search planning and queries: `SearchPlan` describes a search before running
|
|
20
|
+
it and can be partitioned by year, a field-tagged boolean query builder
|
|
21
|
+
(`scopus_query`, `wrap_field`) with a reference table of Scopus field tags,
|
|
22
|
+
and cheap result-size lookups (`scopus_count`) for quota-aware pre-flight
|
|
23
|
+
sizing.
|
|
24
|
+
- Harvesting: resumable, checkpointed retrieval of a plan (`fetch_plan`), with
|
|
25
|
+
per-cell progress logging on the `scopusflow` logger and co-operative
|
|
26
|
+
cancellation (`should_stop`). Checkpoints are keyed by the whole plan, and on
|
|
27
|
+
resume the recorded query is compared against the cell's, so a cache
|
|
28
|
+
directory written by a different plan is refetched with a warning rather
|
|
29
|
+
than silently reused.
|
|
30
|
+
- Records: one stable, tidy record schema across query types (`to_records`,
|
|
31
|
+
`RECORD_COLUMNS`), with an `authkeywords` column added when
|
|
32
|
+
`SearchPlan(view="COMPLETE")` is used, at no request cost beyond that view's
|
|
33
|
+
own smaller page size.
|
|
34
|
+
- Abstract retrieval: batch, per-identifier-resilient `scopus_abstract`
|
|
35
|
+
(default `view="META_ABS"`, so a plain call returns the abstract text), with
|
|
36
|
+
`include=("references", "keywords")` for a document's own reference list and
|
|
37
|
+
author keywords, per-identifier caching keyed by view and `include`
|
|
38
|
+
selection (`cache_dir`/`resume`), `n_requests`/`quota` accounting, a warning
|
|
39
|
+
when the references returned fall short of the document's reported
|
|
40
|
+
`refcount`, and a clear `ScopusFlowForbiddenError` on an entitlement 403
|
|
41
|
+
that stops the batch, suggesting the other of `"FULL"`/`"REF"` only when the
|
|
42
|
+
refused view was one of those two.
|
|
43
|
+
- Corpus assembly: `corpus` combines a search result with abstract retrieval
|
|
44
|
+
into a minimal `id`/`title`/`year`/`keywords`/`references` shape for
|
|
45
|
+
downstream tools such as keyword co-occurrence or citation-network analysis,
|
|
46
|
+
covered by the *Author keywords and references* guide.
|
|
47
|
+
- Trends and comparison: annual publication trends (`scopus_trend`,
|
|
48
|
+
`year_counts`) computed from cheap result-size lookups, and topic-trend
|
|
49
|
+
comparison (`compare_topics`) with per-step progress logging. The
|
|
50
|
+
comparison's documented count-request cost includes the reference topic:
|
|
51
|
+
(terms + 1) x years.
|
|
52
|
+
- Summaries and change tracking: source and author top tallies (`top`), DOI
|
|
53
|
+
extraction and change tracking between runs (`extract_dois`, `diff_dois`).
|
|
54
|
+
- Plots: brand-themed matplotlib figures for trends (`plot_trend`), top
|
|
55
|
+
tallies (`plot_top`, labelling each bar with its count and reserving x-axis
|
|
56
|
+
headroom so the longest label stays inside the axes) and topic comparison
|
|
57
|
+
(`plot_comparison`, mirroring the R `plot_scopus_comparison`): illustrative
|
|
58
|
+
Wilson stability bands with a Source/Wilson caption, a percent-formatted
|
|
59
|
+
y-axis, record counts in the labels by default (`counts_in_legend`), the
|
|
60
|
+
reference topic named in a subtitle, colour-matched end-of-line labels
|
|
61
|
+
spread apart at draw time in the same order as the line ends, an optional
|
|
62
|
+
`legend_inside` placing the legend in the emptiest corner, and full schema
|
|
63
|
+
validation with a friendly `ValueError`.
|
|
64
|
+
- Reference-manager export to BibTeX and RIS (`to_bibtex`, `to_ris`), so a
|
|
65
|
+
search carries into Zotero, EndNote, Mendeley or a LaTeX bibliography.
|
|
66
|
+
- A local, code-free NiceGUI app (`scopusflow-gui`, the `app` extra) that
|
|
67
|
+
drives the whole workflow through a browser tab: a background harvest with a
|
|
68
|
+
live progress terminal, every choice (including the chosen `view` and the
|
|
69
|
+
comparison) mirrored back as a runnable Python script, results shown as a
|
|
70
|
+
paginated table and plots with export, a Compare topics card with highlight,
|
|
71
|
+
stability-band and counts-in-label controls plus a count-request cost
|
|
72
|
+
estimate and CSV export, and a demo mode that needs no key. Harvest
|
|
73
|
+
checkpoints live under the temp directory and are removed when the tab
|
|
74
|
+
closes, so search terms do not linger on disk. Duplicate comparison terms
|
|
75
|
+
are dropped, a zero-record harvest is reported as a warning, and the
|
|
76
|
+
controls carry visible labels.
|
|
77
|
+
- Documentation guides that execute their examples, so tables and plots appear
|
|
78
|
+
inline as on a knitted vignette.
|
|
79
|
+
|
|
80
|
+
[Unreleased]: https://github.com/pablobernabeu/scopusflow-py/compare/v0.1.0...HEAD
|
|
81
|
+
[0.1.0]: https://github.com/pablobernabeu/scopusflow-py/releases/tag/v0.1.0
|
|
@@ -0,0 +1,27 @@
|
|
|
1
|
+
cff-version: 1.2.0
|
|
2
|
+
message: "If you use this software, please cite it as below."
|
|
3
|
+
title: "scopusflow: A reproducible workflow layer over pybliometrics for Scopus searches"
|
|
4
|
+
abstract: >-
|
|
5
|
+
A reproducible workflow layer over pybliometrics for Scopus searches. It builds
|
|
6
|
+
reproducible search plans, retrieves records with rate-limit handling and
|
|
7
|
+
resumable caching, normalises results to a stable tidy schema, tracks changes in
|
|
8
|
+
DOIs between runs, compares publication trends across topics and exports to
|
|
9
|
+
formats used by downstream bibliometric tools. It is the Python twin of the R
|
|
10
|
+
package scopusflow.
|
|
11
|
+
type: software
|
|
12
|
+
authors:
|
|
13
|
+
- family-names: Bernabeu
|
|
14
|
+
given-names: Pablo
|
|
15
|
+
orcid: "https://orcid.org/0000-0003-1083-2460"
|
|
16
|
+
version: 0.1.0
|
|
17
|
+
date-released: "2026-07-10"
|
|
18
|
+
license: MIT
|
|
19
|
+
doi: "10.5281/zenodo.21252666"
|
|
20
|
+
repository-code: "https://github.com/pablobernabeu/scopusflow-py"
|
|
21
|
+
url: "https://pablobernabeu.github.io/scopusflow-py/"
|
|
22
|
+
keywords:
|
|
23
|
+
- scopus
|
|
24
|
+
- bibliometrics
|
|
25
|
+
- pybliometrics
|
|
26
|
+
- literature
|
|
27
|
+
- workflow
|
|
@@ -0,0 +1,69 @@
|
|
|
1
|
+
# Contributing
|
|
2
|
+
|
|
3
|
+
Thanks for your interest in improving scopusflow. Contributions of all sizes are
|
|
4
|
+
welcome, from typo fixes to new workflow helpers.
|
|
5
|
+
|
|
6
|
+
## Setup
|
|
7
|
+
|
|
8
|
+
Clone the repository and install the package in editable mode with the
|
|
9
|
+
development and plotting extras:
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
pip install -e ".[dev,plot]"
|
|
13
|
+
```
|
|
14
|
+
|
|
15
|
+
The pure-logic helpers need no Scopus API key. Everything that contacts the API
|
|
16
|
+
calls pybliometrics, which expects a key in its standard
|
|
17
|
+
`~/.config/pybliometrics.cfg`.
|
|
18
|
+
|
|
19
|
+
## Run the tests
|
|
20
|
+
|
|
21
|
+
The test suite is offline by design, so it runs without a key or a network
|
|
22
|
+
connection. Point Python at `src` and run pytest:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
PYTHONPATH=src pytest
|
|
26
|
+
```
|
|
27
|
+
|
|
28
|
+
## Lint
|
|
29
|
+
|
|
30
|
+
Code is linted with [ruff](https://docs.astral.sh/ruff/) at a line length of
|
|
31
|
+
100. Please make sure your changes are clean before opening a pull request:
|
|
32
|
+
|
|
33
|
+
```bash
|
|
34
|
+
ruff check .
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## Releasing
|
|
38
|
+
|
|
39
|
+
A release runs through these steps:
|
|
40
|
+
|
|
41
|
+
1. Bump `__version__` in [`src/scopusflow/__init__.py`](https://github.com/pablobernabeu/scopusflow-py/blob/main/src/scopusflow/__init__.py) (the version in
|
|
42
|
+
`pyproject.toml` is dynamic and reads it from there), and date the new
|
|
43
|
+
section in `CHANGELOG.md`.
|
|
44
|
+
2. Verify with `PYTHONPATH=src pytest`, `ruff check .`, `python -m build` and
|
|
45
|
+
`python -m twine check dist/*`.
|
|
46
|
+
3. Commit, then tag with `git tag -a vX.Y.Z -m "scopusflow X.Y.Z"` and push the
|
|
47
|
+
commit and the tag.
|
|
48
|
+
4. Create a GitHub release for the tag. Publishing it runs `publish.yml`, which
|
|
49
|
+
builds the distribution and uploads it to PyPI through trusted publishing, so
|
|
50
|
+
no token is stored in the repository.
|
|
51
|
+
|
|
52
|
+
Trusted publishing needs a one-time setup on PyPI. In the project's Publishing
|
|
53
|
+
settings, add this repository as a trusted publisher with workflow file
|
|
54
|
+
`publish.yml` and environment `pypi`. The workflow handles every PyPI release
|
|
55
|
+
from 0.1.0 onwards.
|
|
56
|
+
|
|
57
|
+
## Relationship to other projects
|
|
58
|
+
|
|
59
|
+
scopusflow is the Python twin of the R package
|
|
60
|
+
[scopusflow](https://pablobernabeu.github.io/scopusflow/); the two aim to mirror
|
|
61
|
+
each other's behaviour and naming where it makes sense, so a fix in one is often
|
|
62
|
+
worth porting to the other.
|
|
63
|
+
|
|
64
|
+
It is built deliberately *on top of*
|
|
65
|
+
[pybliometrics](https://pybliometrics.readthedocs.io), which already handles the
|
|
66
|
+
Scopus HTTP, cursor pagination, quota rotation and per-query caching. scopusflow
|
|
67
|
+
adds the reproducible workflow around it rather than re-implementing that
|
|
68
|
+
plumbing, so changes that belong in the API layer should go upstream to
|
|
69
|
+
pybliometrics.
|
scopusflow-0.1.0/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Pablo Bernabeu
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,146 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: scopusflow
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: A reproducible workflow layer over pybliometrics for Scopus searches
|
|
5
|
+
Project-URL: Homepage, https://github.com/pablobernabeu/scopusflow-py
|
|
6
|
+
Project-URL: Documentation, https://pablobernabeu.github.io/scopusflow-py/
|
|
7
|
+
Project-URL: R twin, https://pablobernabeu.github.io/scopusflow/
|
|
8
|
+
Author-email: Pablo Bernabeu <pcbernabeu@gmail.com>
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: bibliometrics,literature,pybliometrics,scopus,workflow
|
|
12
|
+
Classifier: Development Status :: 3 - Alpha
|
|
13
|
+
Classifier: Intended Audience :: Science/Research
|
|
14
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
15
|
+
Classifier: Programming Language :: Python :: 3
|
|
16
|
+
Classifier: Programming Language :: Python :: 3.10
|
|
17
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
20
|
+
Classifier: Topic :: Scientific/Engineering :: Information Analysis
|
|
21
|
+
Requires-Python: >=3.10
|
|
22
|
+
Requires-Dist: pandas>=1.5
|
|
23
|
+
Requires-Dist: pybliometrics>=4.0
|
|
24
|
+
Provides-Extra: app
|
|
25
|
+
Requires-Dist: matplotlib>=3.5; extra == 'app'
|
|
26
|
+
Requires-Dist: nicegui>=2.0; extra == 'app'
|
|
27
|
+
Provides-Extra: dev
|
|
28
|
+
Requires-Dist: pytest>=7; extra == 'dev'
|
|
29
|
+
Requires-Dist: ruff>=0.4; extra == 'dev'
|
|
30
|
+
Provides-Extra: docs
|
|
31
|
+
Requires-Dist: markdown-exec>=1.8; extra == 'docs'
|
|
32
|
+
Requires-Dist: matplotlib>=3.5; extra == 'docs'
|
|
33
|
+
Requires-Dist: mkdocs-material>=9.5; extra == 'docs'
|
|
34
|
+
Requires-Dist: mkdocstrings[python]>=0.24; extra == 'docs'
|
|
35
|
+
Provides-Extra: plot
|
|
36
|
+
Requires-Dist: matplotlib>=3.5; extra == 'plot'
|
|
37
|
+
Description-Content-Type: text/markdown
|
|
38
|
+
|
|
39
|
+
# scopusflow (Python)
|
|
40
|
+
|
|
41
|
+
[](https://doi.org/10.5281/zenodo.21252666)
|
|
42
|
+
|
|
43
|
+
scopusflow is a reproducible workflow layer over [pybliometrics](https://pybliometrics.readthedocs.io) for Scopus searches. It is the Python twin of the R package [scopusflow](https://pablobernabeu.github.io/scopusflow/) and follows the same design, so a search written in one language reads much the same in the other.
|
|
44
|
+
|
|
45
|
+
This is an early release, covered by an offline test suite. The retrieval path is a thin driver over pybliometrics, so before you lean on it for a large live harvest it is worth a short trial run against your installed version.
|
|
46
|
+
|
|
47
|
+
## Why this exists
|
|
48
|
+
|
|
49
|
+
pybliometrics is the mature way to reach the Scopus API from Python. It wraps around ten endpoints and handles the HTTP, cursor pagination, weekly-quota rotation and per-query caching. What it does not provide is a workflow on top of that plumbing. There is no declarative search plan, no single record schema that stays the same across query types, no resumable harvest with checkpoints, no DOI change-tracking between runs, and nothing ready-made for trends, topic comparisons, abstracts or plots. Researchers tend to hand-roll those around pybliometrics, and that is the work scopusflow takes on. It depends on pybliometrics rather than re-implementing the plumbing that already works well.
|
|
50
|
+
|
|
51
|
+
| | pybliometrics | scopusflow |
|
|
52
|
+
|---|---|---|
|
|
53
|
+
| Reach the API (search, retrieval, quota, cursor, cache) | yes | delegates to pybliometrics |
|
|
54
|
+
| Declarative, reproducible search plan | no | yes |
|
|
55
|
+
| One stable record schema across query types | no | yes |
|
|
56
|
+
| Resumable, checkpointed harvest of a plan | no | yes |
|
|
57
|
+
| DOI extraction and change-tracking between runs | no | yes |
|
|
58
|
+
| Annual publication trends without downloading records | no | yes |
|
|
59
|
+
| Topic-trend comparison with stability bands | no | yes |
|
|
60
|
+
| Batch abstract retrieval, resilient per id | no | yes |
|
|
61
|
+
| Trend and top-source/author plots | no | yes |
|
|
62
|
+
| Export to reference managers (BibTeX, RIS) | no | yes |
|
|
63
|
+
| Minimal, uniform keyword/reference corpus export | no | yes |
|
|
64
|
+
|
|
65
|
+
The other Python options are not live alternatives. elsapy was archived as read-only in January 2025, and pyscopus last saw a release in January 2019.
|
|
66
|
+
|
|
67
|
+
## Install
|
|
68
|
+
|
|
69
|
+
You need a Scopus API key configured for pybliometrics, in its standard `~/.config/pybliometrics.cfg`.
|
|
70
|
+
|
|
71
|
+
```bash
|
|
72
|
+
pip install scopusflow
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
The optional extras add the figures and the app. Install `scopusflow[plot]` for the matplotlib plots and `scopusflow[app]` for the code-free app. To work on the package itself, install from a clone with `pip install -e ".[dev,plot]"`.
|
|
76
|
+
|
|
77
|
+
## A first search
|
|
78
|
+
|
|
79
|
+
The example below builds a query, plans a harvest partitioned by year, retrieves it with caching, and then draws on the stable record schema for everything that follows.
|
|
80
|
+
|
|
81
|
+
```python
|
|
82
|
+
import scopusflow as sf
|
|
83
|
+
|
|
84
|
+
q = sf.scopus_query("graphene", "supercapacitor", field="TITLE-ABS-KEY")
|
|
85
|
+
plan = sf.SearchPlan(q, years=range(2010, 2023), partition="year")
|
|
86
|
+
|
|
87
|
+
records = sf.fetch_plan(plan, cache_dir="harvest", resume=True)
|
|
88
|
+
|
|
89
|
+
sf.top(records, by="source")
|
|
90
|
+
dois = sf.extract_dois(records)
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
Re-running the same plan later and comparing the two retrievals shows which records have appeared or disappeared in the meantime.
|
|
94
|
+
|
|
95
|
+
```python
|
|
96
|
+
later = sf.fetch_plan(plan, cache_dir="harvest2")
|
|
97
|
+
sf.diff_dois(old=records, new=later)
|
|
98
|
+
```
|
|
99
|
+
|
|
100
|
+
The publication trend can be tallied from a harvest you already hold, or fetched directly as cheap per-year result-size lookups that never download the records themselves.
|
|
101
|
+
|
|
102
|
+
```python
|
|
103
|
+
trend = sf.year_counts(records)
|
|
104
|
+
trend = sf.scopus_trend(q, years=range(2010, 2023))
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
A topic comparison shows how sub-topics grow within the reference literature over time. The abstract and export helpers carry the work onward into reading and into a reference manager.
|
|
108
|
+
|
|
109
|
+
```python
|
|
110
|
+
cmp = sf.compare_topics(q, ["lithium-ion", "sodium-ion"], years=range(2015, 2023))
|
|
111
|
+
sf.plot_comparison(cmp)
|
|
112
|
+
|
|
113
|
+
abstracts = sf.scopus_abstract(dois[:10], by="doi")
|
|
114
|
+
|
|
115
|
+
with open("scopus-records.bib", "w", encoding="utf-8") as fh:
|
|
116
|
+
fh.write(sf.to_bibtex(records))
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
The pure-logic helpers, from query building to DOI tracking, need no API key and are exercised by the offline tests. Everything that contacts the API needs a key, and the plots need the optional `plot` extra.
|
|
120
|
+
|
|
121
|
+
## A code-free app
|
|
122
|
+
|
|
123
|
+
A local [NiceGUI](https://nicegui.io) app drives the whole workflow without writing code, and mirrors every choice back as a runnable Python script, so it works as an on-ramp to the package rather than a replacement. It runs on your own machine, so your API key never leaves it, and a demo mode lets you try the flow with synthetic data and no key at all.
|
|
124
|
+
|
|
125
|
+
```bash
|
|
126
|
+
pip install "scopusflow[app]"
|
|
127
|
+
scopusflow-gui
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
The retrieval runs in the background with a live progress terminal. Results appear as a paginated table and a pair of plots with one-click export, and a Compare topics card draws the same comparison figure the library produces. It is the Python twin of the R package's `scopusflow::run_app()`.
|
|
131
|
+
|
|
132
|
+
## Citation
|
|
133
|
+
|
|
134
|
+
If scopusflow contributes to published work, please cite it. GitHub builds a ready-made citation from [`CITATION.cff`](CITATION.cff) through the *Cite this repository* button.
|
|
135
|
+
|
|
136
|
+
> Bernabeu, P. (2026). scopusflow: A reproducible workflow layer over pybliometrics for Scopus searches. Python package version 0.1.0. https://doi.org/10.5281/zenodo.21252666
|
|
137
|
+
|
|
138
|
+
The [About page](https://pablobernabeu.github.io/scopusflow-py/about/) carries the same citation with a BibTeX entry.
|
|
139
|
+
|
|
140
|
+
## Developer
|
|
141
|
+
|
|
142
|
+
scopusflow is written by [Pablo Bernabeu](https://pablobernabeu.github.io/), a researcher in the Department of Education at the University of Oxford, with hands-on experience of behavioural experiments, EEG, corpus analysis, computational modelling and statistics. He develops open, reproducible research software in R and Python, and is a Fellow of the Software Sustainability Institute. His [ORCID record](https://orcid.org/0000-0003-1083-2460) lists his other work.
|
|
143
|
+
|
|
144
|
+
## Licence
|
|
145
|
+
|
|
146
|
+
MIT. Scopus is a trademark of Elsevier. This is an independent client and is not affiliated with or endorsed by Elsevier.
|