citations-collector 0.2.1__tar.gz

citations_collector-0.2.1/.gitignore

@@ -0,0 +1,4 @@
+.npm/
+__pycache__/
+*.lock
+src/citations_collector/_version.py

citations_collector-0.2.1/PKG-INFO

@@ -0,0 +1,354 @@
+Metadata-Version: 2.4
+Name: citations-collector
+Version: 0.2.1
+Summary: Discover and curate scholarly citations of datasets and software
+Project-URL: Homepage, https://github.com/dandi/citations-collector
+Project-URL: Repository, https://github.com/dandi/citations-collector
+Project-URL: Issues, https://github.com/dandi/citations-collector/issues
+Author-email: DANDI <team@dandiarchive.org>
+License: MIT
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Requires-Python: >=3.10
+Requires-Dist: click>=8.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: pyzotero>=1.5
+Requires-Dist: rapidfuzz>=3.0
+Requires-Dist: requests>=2.28
+Provides-Extra: ci
+Requires-Dist: mypy>=1.0; extra == 'ci'
+Requires-Dist: pre-commit>=3.0; extra == 'ci'
+Requires-Dist: pytest-cov>=4.0; extra == 'ci'
+Requires-Dist: pytest-timeout>=2.0; extra == 'ci'
+Requires-Dist: pytest>=7.0; extra == 'ci'
+Requires-Dist: responses>=0.23; extra == 'ci'
+Requires-Dist: ruff>=0.1.0; extra == 'ci'
+Requires-Dist: tox-gh-actions>=3.0; extra == 'ci'
+Requires-Dist: tox-uv>=1.0; extra == 'ci'
+Requires-Dist: tox>=4.0; extra == 'ci'
+Requires-Dist: types-pyyaml; extra == 'ci'
+Requires-Dist: types-requests; extra == 'ci'
+Provides-Extra: dandi
+Requires-Dist: dandi>=0.60.0; extra == 'dandi'
+Provides-Extra: devel
+Requires-Dist: mypy>=1.0; extra == 'devel'
+Requires-Dist: pre-commit>=3.0; extra == 'devel'
+Requires-Dist: pytest-cov>=4.0; extra == 'devel'
+Requires-Dist: pytest-timeout>=2.0; extra == 'devel'
+Requires-Dist: pytest>=7.0; extra == 'devel'
+Requires-Dist: responses>=0.23; extra == 'devel'
+Requires-Dist: ruff>=0.1.0; extra == 'devel'
+Requires-Dist: tox-uv>=1.0; extra == 'devel'
+Requires-Dist: tox>=4.0; extra == 'devel'
+Requires-Dist: types-pyyaml; extra == 'devel'
+Requires-Dist: types-requests; extra == 'devel'
+Provides-Extra: linkml
+Requires-Dist: linkml-runtime>=1.7.0; extra == 'linkml'
+Requires-Dist: linkml>=1.7.0; extra == 'linkml'
+Provides-Extra: test
+Requires-Dist: pytest-cov>=4.0; extra == 'test'
+Requires-Dist: pytest-timeout>=2.0; extra == 'test'
+Requires-Dist: pytest>=7.0; extra == 'test'
+Requires-Dist: responses>=0.23; extra == 'test'
+Description-Content-Type: text/markdown
+
+# citations-collector
+
+Discover and curate scholarly citations of datasets and software.
+
+## Features
+
+- **Citation Discovery**: Query CrossRef, OpenCitations, DataCite for citing papers
+- **Hierarchical Collections**: Organize citations by project/version (e.g., DANDI dandisets)
+- **Git-Friendly**: YAML collections + TSV citation records for version control
+- **Curation Workflow**: Mark citations as ignored, merge preprints with published versions
+- **PDF Acquisition**: Automatically download open-access PDFs via Unpaywall with optional git-annex tracking
+- **Merge Detection**: Auto-detect preprints with published versions using CrossRef relationships
+- **Zotero Integration**: Sync citations to hierarchical Zotero collections with automatic merged item relocation
+- **Incremental Updates**: Efficiently discover only new citations since last run
+
+## Installation
+
+```bash
+# Using uv (recommended)
+uv venv
+source .venv/bin/activate
+uv pip install citations-collector
+
+# Or using pip
+pip install citations-collector
+```
+
+## Quick Start
+
+### 1. Create a Collection
+
+Create `collection.yaml`:
+
+```yaml
+name: My Research Tools
+description: Software tools used in our lab
+items:
+  - item_id: my-tool
+    name: "My Analysis Tool"
+    flavors:
+      - flavor_id: "1.0.0"
+        refs:
+          - ref_type: doi
+            ref_value: "10.5281/zenodo.1234567"
+```
+
+### 2. Discover Citations
+
+```bash
+# Discover citations for all items in collection
+citations-collector discover collection.yaml --output citations.tsv
+
+# Use CrossRef polite pool (better rate limits)
+citations-collector discover collection.yaml --email your@email.org
+```
+
+### 3. View Results
+
+Citations are saved to `citations.tsv` - a tab-separated file you can open in Excel or edit manually for curation.
+
+## Advanced Workflows
+
+### PDF Acquisition
+
+Automatically download open-access PDFs using Unpaywall:
+
+```bash
+# Fetch PDFs for discovered citations
+citations-collector fetch-pdfs --config collection.yaml
+
+# Use git-annex for provenance tracking
+citations-collector fetch-pdfs --config collection.yaml --git-annex
+
+# Dry run to see what would be downloaded
+citations-collector fetch-pdfs --config collection.yaml --dry-run
+```
+
+PDFs are stored at `pdfs/{doi}/article.pdf` with accompanying `article.bib` BibTeX files.
+
+### Merge Detection
+
+Detect preprints that have published versions:
+
+```bash
+# Detect merges via CrossRef relationships
+citations-collector detect-merges --config collection.yaml
+
+# Also run fuzzy title matching (use with caution)
+citations-collector detect-merges --config collection.yaml --fuzzy-match
+
+# Preview without updating
+citations-collector detect-merges --config collection.yaml --dry-run
+```
+
+Detected preprints are marked with `citation_status=merged` and `citation_merged_into={published_doi}`.
+
+### Zotero Sync
+
+Sync citations to Zotero for collaborative browsing:
+
+```bash
+# Sync to Zotero (requires API key in config or env)
+citations-collector sync-zotero --config collection.yaml
+
+# Dry run to preview structure
+citations-collector sync-zotero --config collection.yaml --dry-run
+```
+
+Zotero hierarchy:
+```
+Top Collection/
+  ├── {item_id}/
+  │   ├── {flavor}/
+  │   │   ├── <active citations>
+  │   │   └── Merged/
+  │   │       └── <preprints and old versions>
+```
+
+### Unified Configuration
+
+Create a unified `collection.yaml` with all settings:
+
+```yaml
+name: My Research Collection
+description: Tools and datasets from our lab
+
+# Source items to track
+source:
+  items:
+    - item_id: dandi-000055
+      name: "AJILE12: Long-term naturalistic human intracranial neural recordings"
+      flavors:
+        - flavor_id: "0.220113.0400"
+          refs:
+            - ref_type: doi
+              ref_value: "10.48324/dandi.000055/0.220113.0400"
+
+# Citation discovery settings
+discover:
+  sources:
+    - crossref
+    - opencitations
+  email: your@email.org  # For CrossRef polite pool
+  incremental: true
+
+# PDF acquisition settings (optional)
+pdfs:
+  output_dir: pdfs/
+  unpaywall_email: your@email.org
+  git_annex: false
+
+# Zotero sync settings (optional)
+zotero:
+  library_type: group
+  library_id: "12345"
+  api_key: "YOUR_API_KEY"  # Or set ZOTERO_API_KEY env var
+  top_collection_key: "ABCD1234"
+```
+
+Then run the full workflow:
+
+```bash
+# 1. Discover citations
+citations-collector discover collection.yaml
+
+# 2. Fetch open-access PDFs
+citations-collector fetch-pdfs --config collection.yaml
+
+# 3. Detect merged preprints
+citations-collector detect-merges --config collection.yaml
+
+# 4. Sync to Zotero
+citations-collector sync-zotero --config collection.yaml
+```
+
+## Library Usage
+
+```python
+from citations_collector import CitationCollector
+
+# Load collection
+collector = CitationCollector.from_yaml("collection.yaml")
+
+# Discover citations (incremental by default)
+collector.discover_all(incremental=True, email="your@email.org")
+
+# Save results
+collector.save("collection.yaml", "citations.tsv")
+```
+
+## Examples
+
+See the `examples/` directory for:
+- **dandi-collection.yaml**: DANDI Archive dandisets with versioned DOIs
+- **repronim-tools.yaml**: ReproNim neuroimaging tools with RRIDs
+- **simple-resources.yaml**: Basic collection without versioning
+- **citations-example.tsv**: Example citation records with curation
+
+## Development
+
+### Setup
+
+```bash
+# Clone repository
+git clone https://github.com/dandi/citations-collector.git
+cd citations-collector
+
+# Setup development environment
+uv venv
+source .venv/bin/activate
+uv pip install -e ".[devel]"
+```
+
+### Running Tests
+
+```bash
+# Run all tests, linting, and type checking
+tox
+
+# Run specific environment
+tox -e py312      # Tests on Python 3.12
+tox -e lint       # Ruff linting
+tox -e type       # Mypy type checking
+tox -e cov        # Coverage report
+```
+
+### Regenerating LinkML Models
+
+When `schema/citations.yaml` changes:
+
+```bash
+# Install linkml tools
+uv pip install -e ".[linkml]"
+
+# Regenerate Pydantic models
+gen-pydantic schema/citations.yaml > src/citations_collector/models/generated.py
+
+# Regenerate JSON Schema
+gen-json-schema schema/citations.yaml > schema/citations.schema.json
+
+# Commit generated files
+git add src/citations_collector/models/generated.py schema/citations.schema.json
+git commit -m "Regenerate LinkML models"
+```
+
+## Architecture
+
+- **Library-First Design**: All functionality accessible programmatically
+- **LinkML Schema**: Validated data models from `schema/citations.yaml`
+- **Modular Structure**:
+  - `discovery/`: Citation API clients (CrossRef, OpenCitations, DataCite)
+  - `persistence/`: YAML/TSV I/O
+  - `importers/`: DANDI API, Zenodo, GitHub integrations
+  - `unpaywall.py`: Unpaywall API client for OA PDF URLs
+  - `pdf.py`: PDF acquisition with git-annex support
+  - `merge_detection.py`: Preprint/published version detection
+  - `zotero_sync.py`: Zotero hierarchical sync with merged item handling
+  - `core.py`: Main orchestration API
+  - `cli.py`: Click-based CLI (thin wrapper)
+
+## Citation Sources
+
+- **CrossRef**: Most comprehensive, best for DOI citations
+- **OpenCitations**: Open index, may lag behind CrossRef
+- **DataCite**: Good for dataset citations
+- **Europe PMC**: PubMed-indexed papers (future)
+- **Semantic Scholar**: AI-powered citation discovery (future)
+
+## License
+
+MIT License - see LICENSE file for details.
+
+## Contributing
+
+See CONSTITUTION.md for:
+- Code standards (Ruff, mypy, type hints)
+- Testing requirements (pytest, 100 lines max, mock HTTP)
+- Architecture principles (library-first, reliability, simplicity)
+
+Pull requests welcome!
+
+## Citation
+
+If you use citations-collector in your research, please cite:
+
+```bibtex
+@software{citations_collector,
+  title = {citations-collector: Discover and curate scholarly citations},
+  author = {{DANDI Team}},
+  url = {https://github.com/dandi/citations-collector},
+  license = {MIT}
+}
+```

citations_collector-0.2.1/README.md

@@ -0,0 +1,293 @@
+# citations-collector
+
+Discover and curate scholarly citations of datasets and software.
+
+## Features
+
+- **Citation Discovery**: Query CrossRef, OpenCitations, DataCite for citing papers
+- **Hierarchical Collections**: Organize citations by project/version (e.g., DANDI dandisets)
+- **Git-Friendly**: YAML collections + TSV citation records for version control
+- **Curation Workflow**: Mark citations as ignored, merge preprints with published versions
+- **PDF Acquisition**: Automatically download open-access PDFs via Unpaywall with optional git-annex tracking
+- **Merge Detection**: Auto-detect preprints with published versions using CrossRef relationships
+- **Zotero Integration**: Sync citations to hierarchical Zotero collections with automatic merged item relocation
+- **Incremental Updates**: Efficiently discover only new citations since last run
+
+## Installation
+
+```bash
+# Using uv (recommended)
+uv venv
+source .venv/bin/activate
+uv pip install citations-collector
+
+# Or using pip
+pip install citations-collector
+```
+
+## Quick Start
+
+### 1. Create a Collection
+
+Create `collection.yaml`:
+
+```yaml
+name: My Research Tools
+description: Software tools used in our lab
+items:
+  - item_id: my-tool
+    name: "My Analysis Tool"
+    flavors:
+      - flavor_id: "1.0.0"
+        refs:
+          - ref_type: doi
+            ref_value: "10.5281/zenodo.1234567"
+```
+
+### 2. Discover Citations
+
+```bash
+# Discover citations for all items in collection
+citations-collector discover collection.yaml --output citations.tsv
+
+# Use CrossRef polite pool (better rate limits)
+citations-collector discover collection.yaml --email your@email.org
+```
+
+### 3. View Results
+
+Citations are saved to `citations.tsv` - a tab-separated file you can open in Excel or edit manually for curation.
+
+## Advanced Workflows
+
+### PDF Acquisition
+
+Automatically download open-access PDFs using Unpaywall:
+
+```bash
+# Fetch PDFs for discovered citations
+citations-collector fetch-pdfs --config collection.yaml
+
+# Use git-annex for provenance tracking
+citations-collector fetch-pdfs --config collection.yaml --git-annex
+
+# Dry run to see what would be downloaded
+citations-collector fetch-pdfs --config collection.yaml --dry-run
+```
+
+PDFs are stored at `pdfs/{doi}/article.pdf` with accompanying `article.bib` BibTeX files.
+
+### Merge Detection
+
+Detect preprints that have published versions:
+
+```bash
+# Detect merges via CrossRef relationships
+citations-collector detect-merges --config collection.yaml
+
+# Also run fuzzy title matching (use with caution)
+citations-collector detect-merges --config collection.yaml --fuzzy-match
+
+# Preview without updating
+citations-collector detect-merges --config collection.yaml --dry-run
+```
+
+Detected preprints are marked with `citation_status=merged` and `citation_merged_into={published_doi}`.
+
+### Zotero Sync
+
+Sync citations to Zotero for collaborative browsing:
+
+```bash
+# Sync to Zotero (requires API key in config or env)
+citations-collector sync-zotero --config collection.yaml
+
+# Dry run to preview structure
+citations-collector sync-zotero --config collection.yaml --dry-run
+```
+
+Zotero hierarchy:
+```
+Top Collection/
+  ├── {item_id}/
+  │   ├── {flavor}/
+  │   │   ├── <active citations>
+  │   │   └── Merged/
+  │   │       └── <preprints and old versions>
+```
+
+### Unified Configuration
+
+Create a unified `collection.yaml` with all settings:
+
+```yaml
+name: My Research Collection
+description: Tools and datasets from our lab
+
+# Source items to track
+source:
+  items:
+    - item_id: dandi-000055
+      name: "AJILE12: Long-term naturalistic human intracranial neural recordings"
+      flavors:
+        - flavor_id: "0.220113.0400"
+          refs:
+            - ref_type: doi
+              ref_value: "10.48324/dandi.000055/0.220113.0400"
+
+# Citation discovery settings
+discover:
+  sources:
+    - crossref
+    - opencitations
+  email: your@email.org  # For CrossRef polite pool
+  incremental: true
+
+# PDF acquisition settings (optional)
+pdfs:
+  output_dir: pdfs/
+  unpaywall_email: your@email.org
+  git_annex: false
+
+# Zotero sync settings (optional)
+zotero:
+  library_type: group
+  library_id: "12345"
+  api_key: "YOUR_API_KEY"  # Or set ZOTERO_API_KEY env var
+  top_collection_key: "ABCD1234"
+```
+
+Then run the full workflow:
+
+```bash
+# 1. Discover citations
+citations-collector discover collection.yaml
+
+# 2. Fetch open-access PDFs
+citations-collector fetch-pdfs --config collection.yaml
+
+# 3. Detect merged preprints
+citations-collector detect-merges --config collection.yaml
+
+# 4. Sync to Zotero
+citations-collector sync-zotero --config collection.yaml
+```
+
+## Library Usage
+
+```python
+from citations_collector import CitationCollector
+
+# Load collection
+collector = CitationCollector.from_yaml("collection.yaml")
+
+# Discover citations (incremental by default)
+collector.discover_all(incremental=True, email="your@email.org")
+
+# Save results
+collector.save("collection.yaml", "citations.tsv")
+```
+
+## Examples
+
+See the `examples/` directory for:
+- **dandi-collection.yaml**: DANDI Archive dandisets with versioned DOIs
+- **repronim-tools.yaml**: ReproNim neuroimaging tools with RRIDs
+- **simple-resources.yaml**: Basic collection without versioning
+- **citations-example.tsv**: Example citation records with curation
+
+## Development
+
+### Setup
+
+```bash
+# Clone repository
+git clone https://github.com/dandi/citations-collector.git
+cd citations-collector
+
+# Setup development environment
+uv venv
+source .venv/bin/activate
+uv pip install -e ".[devel]"
+```
+
+### Running Tests
+
+```bash
+# Run all tests, linting, and type checking
+tox
+
+# Run specific environment
+tox -e py312      # Tests on Python 3.12
+tox -e lint       # Ruff linting
+tox -e type       # Mypy type checking
+tox -e cov        # Coverage report
+```
+
+### Regenerating LinkML Models
+
+When `schema/citations.yaml` changes:
+
+```bash
+# Install linkml tools
+uv pip install -e ".[linkml]"
+
+# Regenerate Pydantic models
+gen-pydantic schema/citations.yaml > src/citations_collector/models/generated.py
+
+# Regenerate JSON Schema
+gen-json-schema schema/citations.yaml > schema/citations.schema.json
+
+# Commit generated files
+git add src/citations_collector/models/generated.py schema/citations.schema.json
+git commit -m "Regenerate LinkML models"
+```
+
+## Architecture
+
+- **Library-First Design**: All functionality accessible programmatically
+- **LinkML Schema**: Validated data models from `schema/citations.yaml`
+- **Modular Structure**:
+  - `discovery/`: Citation API clients (CrossRef, OpenCitations, DataCite)
+  - `persistence/`: YAML/TSV I/O
+  - `importers/`: DANDI API, Zenodo, GitHub integrations
+  - `unpaywall.py`: Unpaywall API client for OA PDF URLs
+  - `pdf.py`: PDF acquisition with git-annex support
+  - `merge_detection.py`: Preprint/published version detection
+  - `zotero_sync.py`: Zotero hierarchical sync with merged item handling
+  - `core.py`: Main orchestration API
+  - `cli.py`: Click-based CLI (thin wrapper)
+
+## Citation Sources
+
+- **CrossRef**: Most comprehensive, best for DOI citations
+- **OpenCitations**: Open index, may lag behind CrossRef
+- **DataCite**: Good for dataset citations
+- **Europe PMC**: PubMed-indexed papers (future)
+- **Semantic Scholar**: AI-powered citation discovery (future)
+
+## License
+
+MIT License - see LICENSE file for details.
+
+## Contributing
+
+See CONSTITUTION.md for:
+- Code standards (Ruff, mypy, type hints)
+- Testing requirements (pytest, 100 lines max, mock HTTP)
+- Architecture principles (library-first, reliability, simplicity)
+
+Pull requests welcome!
+
+## Citation
+
+If you use citations-collector in your research, please cite:
+
+```bibtex
+@software{citations_collector,
+  title = {citations-collector: Discover and curate scholarly citations},
+  author = {{DANDI Team}},
+  url = {https://github.com/dandi/citations-collector},
+  license = {MIT}
+}
+```

citations_collector-0.2.1/examples/citations-example.tsv

@@ -0,0 +1,7 @@
+item_id	item_flavor	item_ref_type	item_ref_value	item_name	citation_doi	citation_title	citation_authors	citation_year	citation_journal	citation_relationship	citation_type	citation_source	discovered_date	citation_status	citation_merged_into	citation_comment	curated_by	curated_date
+dandi:000003	0.210812.1448	doi	10.48324/dandi.000003/0.210812.1448	Hippocampal Granule Cells	10.1016/j.neuron.2022.01.001	Analysis of hippocampal circuits	Smith J; Jones A; Wang L	2022	Neuron	Cites	Publication	crossref	2024-01-15	active
+dandi:000003	0.210812.1448	doi	10.48324/dandi.000003/0.210812.1448	Hippocampal Granule Cells	10.1101/2021.12.15.472789	Preprint version of the analysis	Smith J; Jones A	2021	bioRxiv	Cites	Preprint	crossref	2024-01-15	merged	10.1016/j.neuron.2022.01.001	Merged with published version	curator1	2024-01-20
+dandi:000003	0.220126.1853	doi	10.48324/dandi.000003/0.220126.1853	Hippocampal Granule Cells	10.1523/JNEUROSCI.1234-22.2022	Follow-up analysis using newer version	Brown K	2023	J Neuroscience	Uses	Publication	opencitations	2024-02-01	active
+fmriprep	latest	rrid	SCR_016216	fMRIPrep	10.1038/s41596-022-00710-2	Standard preprocessing for fMRI	Esteban O; et al	2022	Nature Protocols	Describes	Publication	scicrunch	2024-01-10	active
+fmriprep	latest	doi	10.5281/zenodo.852659	fMRIPrep	10.1016/j.neuroimage.2023.120123	Large-scale fMRI meta-analysis	Chen M; et al	2023	NeuroImage	Uses	Publication	datacite	2024-01-12	active
+fmriprep	23.1.0	doi	10.5281/zenodo.8076123	fMRIPrep	10.1101/2024.01.05.574321	Methods comparison study	Lee S	2024	bioRxiv	Cites	Preprint	crossref	2024-02-15	pending			Needs review - unclear citation