roagg 2025.0.8__tar.gz

roagg-2025.0.8/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Swedish National Data Service
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

roagg-2025.0.8/PKG-INFO

@@ -0,0 +1,6 @@
+Metadata-Version: 2.4
+Name: roagg
+Version: 2025.0.8
+Requires-Python: >=3.10
+License-File: LICENSE
+Dynamic: license-file

roagg-2025.0.8/README.md

@@ -0,0 +1,102 @@
+# Research output aggregator 
+> [!NOTE]
+> This script is under development
+
+The goal of this project is to create a script to get a summarization for a research organization about the research output.  
+First target is to query and process information from DataCite.  
+
+The goal for this script is to create a list over research output where an organization is mentioned as:
+* publisher
+* creator with affiliation to the organization
+* contributor with affiliation to the organization
+
+input: ROR-id and list of variants on the organization name.
+
+Properties to collect for each research output:
+|Field                                 |Type   |Comment                                                                                |
+|--------------------------------------|-------|---------------------------------------------------------------------------------------|
+|publicationYear                       |integer|The year of publication, can be empty in some cases                                    |
+|resourceType                          |string |The resource type (free text string)                                                   |
+|title                                 |string |Title of the resource (first one if multiple)                                          |
+|publisher                             |string |Publisher (free text)                                                                  |
+|createdAt                             |string |Created date if availible                                                              |
+|updatedAt                             |string |Updatade date if availible                                                             |
+|isPublisher                           |bool   |True if the the publisher match the requested organisation                             |
+|isFunder                              |bool   |True if the the funder match the requested organisation                                |
+|haveCreatorAffiliation                |bool   |True if the the any creator match the requested organisation                           |
+|haveContributorAffiliation            |bool   |True if the the any contributor match the requested organisation                       |
+|isLatestVersion                       |bool   |True if the DataCite metadata indicates this beeing the latest version                 |
+|isConceptDoi                          |bool   |True if the DataCite metadata indicates this beeing a concept DOI                      |
+|matchPublisherRor                     |bool   |True if the ROR id for publisher match the ROR in the provided argument                |
+|matchCreatorAffiliationRor            |bool   |True if the ROR id for a creator affiliation match the ROR in the provided argument    |
+|matchContributorAffiliationRor        |bool   |True if the ROR id for a contributor affiliation match the ROR in the provided argument|
+|matchFunderRor                        |bool   |True if the ROR id for funder match the ROR in the provided argument                   |
+|matchPublisherName                    |bool   |True if any of the names supplied matches the publisher name in the resource           |
+|matchCreatorName                      |bool   |True if any of the names supplied matches the creator name in the resource             |
+|matchContributorName                  |bool   |True if any of the names supplied matches the contributor name in the resource         |
+|matchFunderName                       |bool   |True if any of the names supplied matches the funder name in the resource              |
+|inDataCite                            |bool   |True if the DOI was matched in the DataCite                                            |
+|inOpenAire                            |bool   |True if the DOI was matched in OpenAire                                                |
+|inOpenAlex                            |bool   |True if the DOI was matched in OpenAlex                                                |
+|inCrossRef                            |bool   |True if the DOI was matched in CrossRef                                                |
+|dataCiteClientId                      |string |The client id for the organisation minting the DOI                                     |
+|dataCiteClientName                    |string |The human readable name of the minting organisation                                    |
+|dataCiteCitationCount                 |integer|Citation count for the resource provided by the DataCite API                           |
+|dataCiteReferenceCount                |integer|Reference count for the resource provided by the DataCite API                          |
+|dataCiteViewCount                     |integer|View count for the resource provided by the DataCite API                               |
+|dataCiteDownloadCount                 |integer|Download count for the resource provided by the DataCite API                           |
+|openAireBestAccessRight               |string |Access Rights for the resource indicated indicated by the OpenAire API                 |
+|openAireIndicatorsUsageCountsDownloads|integer|Download count for the resource indicated by the OpenAire API                          |
+|openAireIndicatorsUsageCountsViews    |integer|View count for the resource provided by the OpenAire API                               |
+|openAireId                            |string |Id for the resource in OpenAire                                                        |
+|openAlexId                            |string |Id for the resource in OpenAlex                                                        |
+|openAlexCitedByCount                  |integer|Citation count for the resource provided by the OpenAlex API                           |
+|openAlexReferencedWorksCount          |integer|Reference count for the resource provided by the OpenAlex API                          |
+|titleWordCount                        |integer|Number of words in the title (useful for sorting in some cases)                        |
+|referencedByDoi                       |string |                                                                                       |
+
+
+## Install
+`pip install .`
+
+## Install dev
+`pip install -e .`
+
+## Development stuff to do
+- [x] ROR get name variants from ROR
+- [x] CLI add options to get name list from txt
+- [x] DataCite API build query for matching publisher and affiliation
+- [ ] Crossref API build query for matching publisher and affiliation
+- [ ] Publish as cmd tool on PyPI
+
+## Run
+List arguments:  
+`roagg --help`  
+
+## Tests
+Some tests are available, to run them:  
+`python -m pytest`
+
+### Some example arguments
+Chalmers with ror and name list:  
+```bash
+roagg --ror https://ror.org/040wg7k59 --name-txt tests/name-lists/chalmers.txt --output chalmers.csv
+```
+
+GU with ror, name list and extra name not in the text file:  
+```bash
+roagg --name "Department of Nephrology Gothenburg" --ror https://ror.org/01tm6cn81 --name-txt tests/name-lists/gu.txt --output data/gu.csv
+```
+
+KTH with ror and name list:  
+```bash
+roagg --ror https://ror.org/026vcq606 --name-txt tests/name-lists/kth.txt --output data/kth.csv
+```
+
+KAU with ror:  
+```bash
+roagg --ror https://ror.org/05s754026 --output kau.csv
+```
+
+## License
+[MIT License](LICENSE)

roagg-2025.0.8/pyproject.toml

@@ -0,0 +1,48 @@
+[project]
+name = "roagg"
+version = "2025.0.8"
+requires-python = ">=3.10"
+dependencies = []
+
+[project.scripts]
+roagg = "roagg.cli:main"
+
+[project.entry-points."pipx.run"]
+roagg = "roagg.cli:main"
+
+[build-system]
+requires = ["setuptools>=42", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+package-dir = {"" = "src"}
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+# Specify test directory
+testpaths = ["tests"]
+
+# Minimum version of pytest
+minversion = "6.0"
+
+# Add command line options that are always used
+addopts = [
+    "-ra",              # Show summary of all test outcomes
+    "--strict-markers", # Raise error on unknown markers
+    "--strict-config",  # Raise error on invalid config
+    "--showlocals",     # Show local variables in tracebacks
+]
+
+# Configure test discovery patterns
+python_files = ["test_*.py", "*_test.py"]
+python_classes = ["Test*"]
+python_functions = ["test_*"]
+
+# Define custom markers (optional, for organizing tests)
+markers = [
+    "slow: marks tests as slow (deselect with '-m \"not slow\"')",
+    "integration: marks tests as integration tests",
+    "unit: marks tests as unit tests",
+]

roagg-2025.0.8/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

roagg-2025.0.8/src/roagg/__main__.py

@@ -0,0 +1,3 @@
+if __name__ == "__main__":
+    from roagg.cli import main
+    main()

roagg-2025.0.8/src/roagg/aggregator.py

@@ -0,0 +1,69 @@
+import sys
+from typing import List
+from roagg.helpers.ror import get_names_from_ror
+from roagg.providers.datacite import DataCiteAPI
+from roagg.providers.openaire import OpenAireAPI
+from roagg.providers.openalex import OpenAlexAPI
+import logging
+from roagg.models.research_output_item import ResearchOutputItem
+import json
+import csv
+from dataclasses import fields
+
+def aggregate(name: List[str] = [], ror: str = "", output: str = "output.csv") -> None:
+    if ror:
+        ror_name = get_names_from_ror(ror)
+        name.extend(ror_name)
+    
+    # remove duplicates
+    name = list(set(name))
+
+    datacite = DataCiteAPI(name=name, ror=ror)
+    url = datacite.api_request_url()
+    # debug print of the query string
+    logging.info("DataCite url:")
+    logging.info(url)
+
+    records = datacite.all()
+    research_output_items = []
+    logging.info(f"Checking {len(records)} records...")
+    for record in records:
+        research_output_items.append(datacite.get_record(record))
+
+    openaire = OpenAireAPI(ror=ror, results=research_output_items)
+    openaire_id = openaire.get_openaire_id_from_ror()
+    logging.info(f"OpenAire ID from ROR {ror} : {openaire_id}")
+    openaire.get_records()
+
+    openalex = OpenAlexAPI(ror=ror, results=research_output_items)
+    openalex_id = openalex.get_openalex_id_from_ror()
+    logging.info(f"OpenAlex ID from ROR {ror} : {openalex_id}")
+    openalex.get_records()
+
+    logging.info(f"Writing: {output}")
+    
+    write_csv(research_output_items, output)
+    logging.info(f"Writing output to csv: {output} - Done")
+
+def write_csv(records: List[ResearchOutputItem], output: str) -> None:
+    # Get field names from the dataclass
+    dataclass_fields = fields(ResearchOutputItem)
+    header = [field.name for field in dataclass_fields]
+    
+    def format_value(value):
+        """Format values for CSV output"""
+        if value is None:
+            return ""
+        elif isinstance(value, bool):
+            return 1 if value else 0
+        else:
+            return value
+
+    with open(output, 'w', newline='', encoding='utf-8') as file:
+        writer = csv.writer(file)
+        writer.writerow(header)
+        
+        writer.writerows([
+            [format_value(getattr(record, field.name)) for field in dataclass_fields]
+            for record in records
+        ])

roagg-2025.0.8/src/roagg/cli.py

@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+from typing import List, Optional
+import argparse
+import logging
+import sys
+from pathlib import Path
+from roagg.helpers.utils import get_roagg_version
+from roagg.aggregator import aggregate
+
+def validate_ror_id(ror_id: str) -> str:
+    """validate ROR ID format (should start with https://ror.org/)."""
+    if not ror_id.startswith('https://ror.org/'):
+        raise argparse.ArgumentTypeError("ROR ID must start with 'https://ror.org/'")
+    return ror_id
+
+def read_names_from_file(filepath: Path) -> List[str]:
+    """Read organization names from a file, one per line."""
+    try:
+        return [line.strip() for line in filepath.read_text().splitlines() if line.strip()]
+    except IOError as e:
+        logging.error(f"Failed to read names file: {e}")
+        sys.exit(1)
+
+def main() -> None:
+    """create a summary CSV file for all research output for an organization."""
+    logging.basicConfig(level=logging.INFO, format='%(levelname)s: %(message)s')
+
+    parser = argparse.ArgumentParser(
+        description="aggregate research outputs for an organization into a CSV file",
+        formatter_class=argparse.RawDescriptionHelpFormatter
+    )
+    
+    parser.add_argument(
+        "--version",
+        action="version",
+        version=get_roagg_version()
+    )
+
+    parser.add_argument(
+        "--name",
+        type=str,
+        action='append',
+        help="name variant of the organization (can be used multiple times)"
+    )
+    parser.add_argument(
+        "--name-txt",
+        type=Path,
+        help="path to text file containing organization name variants (one per line)"
+    )
+
+    parser.add_argument(
+        "--ror",
+        type=validate_ror_id,
+        help="ROR ID of the organization (must start with https://ror.org/)"
+    )
+
+    parser.add_argument(
+        "--source",
+        default="api",
+        choices=["api"],
+        help="source for resource aggregation (only api is supported right now)"
+    )
+
+    parser.add_argument(
+        "--output",
+        default="data/output.csv",
+        help="name of the output file (default: data/output.csv)"
+    )
+
+    args = parser.parse_args()
+
+    # print parser.print_help() if no argument for name, name-txt or ror is provided
+    if not any([args.name, args.name_txt, args.ror]):
+        parser.print_help()
+        sys.exit(1)
+
+    names: List[str] = []
+    if args.name:
+        names = args.name
+
+    if args.name_txt:
+        names.extend(read_names_from_file(args.name_txt))
+
+    try:
+        aggregate(names, args.ror, args.output)
+    except Exception as e:
+        logging.error(f"Aggregation failed: {e}")
+        sys.exit(1)
+
+if __name__ == "__main__":
+    main()

roagg-2025.0.8/src/roagg/helpers/ror.py

@@ -0,0 +1,14 @@
+import urllib.request
+import json
+from typing import List
+
+def get_ror_info(ror: str):
+    ror_id = ror.split('/')[-1]
+    url = f"https://api.ror.org/v2/organizations/{ror_id}"
+    with urllib.request.urlopen(url) as response:
+        return json.loads(response.read())
+
+def get_names_from_ror(ror: str) -> List[str]:
+    names = get_ror_info(ror)['names']
+    valid_types = {'alias', 'ror_display', 'label'}
+    return [n['value'] for n in names if valid_types.intersection(n['types'])]

roagg-2025.0.8/src/roagg/helpers/utils.py

@@ -0,0 +1,65 @@
+import importlib.metadata
+import re
+
+doi_pattern = re.compile(r'^10\.\d{4,9}/[-._;()/:A-Z0-9]+$', re.IGNORECASE)
+
+def is_valid_doi(s: str) -> bool:
+    return bool(doi_pattern.match(s))
+
+def find_doi_in_text(text: str) -> str | None:
+    return re.findall(r'\b10\.\d{4,9}/[-.;()/:\w]+', text)
+
+def remove_resolver_prefix_from_doi(doi: str) -> str:
+    if doi is None:
+        return None
+    prefixes = [
+        "https://doi.org/",
+        "http://doi.org/",
+        "doi.org/",
+        "https://dx.doi.org/",
+        "http://dx.doi.org/",
+        "dx.doi.org/"
+    ]
+    for prefix in prefixes:
+        if doi.lower().startswith(prefix):
+            return doi[len(prefix):]
+    return doi
+
+def match_patterns(string, patterns):
+    if string is None:
+        return False
+
+    for pattern in patterns:
+        # if pattern does not contain * or ? check if string contains pattern
+        if '*' not in pattern and '?' not in pattern:
+            if pattern.lower() is string.lower() or pattern.lower() in string.lower():
+                return True
+    
+        
+        if re.match(pattern_to_regexp(pattern), string, re.IGNORECASE):
+            return True
+    return False
+
+def pattern_to_regexp(pattern: str) -> str:
+    regex = ""
+    for char in pattern:
+        if char == '*':
+            regex += '.*'
+        elif char == '?':
+            regex += '.'
+        else:
+            regex += re.escape(char)
+    return '^' + regex + '$'
+
+def get_roagg_version() -> str:
+    """Get package version from metadata."""
+    try:
+        return importlib.metadata.version("roagg")
+    except importlib.metadata.PackageNotFoundError:
+        return "unknown"
+
+def string_word_count(string: str) -> int:
+    """Count words in a string after trimming whitespace."""
+    if not string:
+        return 0
+    return len(string.strip().split())

roagg-2025.0.8/src/roagg/models/research_output_item.py

@@ -0,0 +1,51 @@
+from dataclasses import dataclass
+
+@dataclass
+class ResearchOutputItem:
+    doi: str
+    publicationYear: int = None
+    resourceType: str = None
+    title: str = None
+    publisher: str = None
+    createdAt: str = ""
+    updatedAt: str = "" 
+    isPublisher: bool = False
+    isFunder: bool = None
+    haveCreatorAffiliation: bool = False
+    haveContributorAffiliation: bool = False
+    isLatestVersion: bool = True
+    isConceptDoi: bool = False
+    #match on ROR
+    matchPublisherRor: bool = False
+    matchCreatorAffiliationRor: bool = False
+    matchContributorAffiliationRor: bool = False
+    matchFunderRor: bool = False
+    #match on free text in name
+    matchPublisherName: bool = False
+    matchCreatorName: bool = False
+    matchContributorName: bool = False
+    matchFunderName: bool = False
+    #where was the match found
+    inDataCite: bool = None
+    inOpenAire: bool = None
+    inOpenAlex: bool = None
+    inCrossRef: bool = None
+    #datacite specific
+    dataCiteClientId: str = None
+    dataCiteClientName: str = None
+    dataCiteCitationCount: int = None
+    dataCiteReferenceCount: int = None
+    dataCiteViewCount: int = None
+    dataCiteDownloadCount: int = None
+    #openaire specific
+    openAireBestAccessRight: str = None
+    openAireIndicatorsUsageCountsDownloads: int = None
+    openAireIndicatorsUsageCountsViews: int = None
+    openAireId: str = None
+    #openalex specific
+    openAlexId: str = None
+    openAlexCitedByCount: int = None
+    openAlexReferencedWorksCount: int = None
+    #extra fields
+    titleWordCount: int = None
+    referencedByDoi: str = None

roagg-2025.0.8/src/roagg/providers/datacite.py

@@ -0,0 +1,178 @@
+from typing import List
+import urllib.request
+import logging
+import json
+from roagg.helpers.utils import get_roagg_version
+from roagg.models.research_output_item import ResearchOutputItem
+from roagg.helpers.utils import match_patterns, string_word_count
+
+class DataCiteAPI:
+    def __init__(self, page_size: int = 500, name: List[str] = [], ror: str = ""):
+        self.page_size = page_size
+        self.name = name
+        self.ror = ror
+
+    def get_query_string(self) -> str:
+        if not self.name and not self.ror:
+            return ""
+            
+        query_parts = []
+        
+        if self.name:
+            # Separate wildcard and exact matches, handle spaces in wildcard queries appropriately
+            wildcard = ' OR '.join(n.replace(" ", "\\ ") for n in self.name if '*' in n)
+            exact = ' OR '.join(f'"{n}"' for n in self.name if '*' not in n)
+            name_fields = [
+                "creators.affiliation.name",
+                "contributors.affiliation.name", 
+                "publisher.name"
+            ]
+
+            if wildcard and exact:
+                name_conditions = f'{wildcard} OR {exact}'
+            else:
+                name_conditions = wildcard or exact
+            
+            query_parts.extend([f"{field}:({name_conditions})" for field in name_fields])
+        
+        if self.ror:
+            ror_fields = [
+                "publisher.publisherIdentifier",
+                "creators.affiliation.affiliationIdentifier", 
+                "contributors.affiliation.affiliationIdentifier",
+                "creators.nameIdentifiers.nameIdentifier",
+                "contributors.nameIdentifiers.nameIdentifier",
+                "fundingReferences.funderIdentifier"
+            ]
+            query_parts.extend([f'{field}:"{self.ror}"' for field in ror_fields])
+            # nameIdentifiers are formated without https://ror.org/ prefix from some sources, so we need to check both
+            query_parts.extend([f'{field}:"{self.ror.split("https://ror.org/")[1]}"' for field in ror_fields])
+        
+        return " OR ".join(query_parts)
+
+    def api_request_url(self, page_size: int = None) -> str:
+        if page_size is None:
+            page_size = self.page_size
+        params = urllib.parse.urlencode({
+            'page[size]': page_size,
+            'page[cursor]': '1',
+            'affiliation': 'true',
+            'publisher': 'true',
+            'detail': 'true',
+            'disable-facets': 'false',
+            'query': self.get_query_string()
+        })
+        return f"https://api.datacite.org/dois?{params}"
+
+    @staticmethod
+    def get_api_result(url: str) -> dict:
+        request = urllib.request.Request(url)
+        version = get_roagg_version()
+        request.add_header('User-Agent', f'ResearchOutputAggregator/{version} (https://github.com/snd-sweden/research-output-aggregator; mailto:team-it@snd.se)')
+        try:
+            with urllib.request.urlopen(request) as response:
+                return json.loads(response.read())
+        except (urllib.error.URLError, json.JSONDecodeError, KeyError) as e:
+            raise RuntimeError(f"Failed run DataCite query: {e}")
+    
+    def get_record(self, item: dict) -> ResearchOutputItem:
+        attributes = item.get("attributes", {})
+        publisher_attr = attributes.get("publisher", {})
+        versionCount = 0 if attributes.get("versionCount", {}) is None else int(attributes.get("versionCount", {}))
+        versionOfCount = 0 if attributes.get("versionOfCount", {}) is None else int(attributes.get("versionOfCount", {}))
+
+        record = ResearchOutputItem(
+            doi=attributes.get("doi"),
+            dataCiteClientId=item["relationships"]["client"]["data"]["id"],
+            resourceType=attributes.get("types", None).get("resourceTypeGeneral"),
+            publisher=publisher_attr.get("name"),
+            publicationYear=attributes.get("publicationYear"),
+            title=item["attributes"]["titles"][0]["title"],
+            inDataCite=True,
+            dataCiteCitationCount=attributes.get("citationCount", None),
+            dataCiteReferenceCount=attributes.get("referenceCount", None),
+            dataCiteViewCount=attributes.get("viewCount", None),
+            dataCiteDownloadCount=attributes.get("downloadCount", None),
+            titleWordCount=string_word_count(item["attributes"]["titles"][0]["title"])
+        )
+
+        if record.resourceType is None or record.resourceType == "":
+            record.resourceType = attributes.get("types", {}).get("citeproc")
+        if record.resourceType is None or record.resourceType == "":
+            record.resourceType = attributes.get("types", {}).get("bibtex")
+            
+
+        record.isPublisher = (
+            publisher_attr.get("publisherIdentifier") == self.ror or 
+            match_patterns(publisher_attr.get("name"), self.name)
+        )
+
+        related = [
+            r for r in item["attributes"].get("relatedIdentifiers", [])
+            if (r.get("relationType") == "IsReferencedBy" or r.get("relationType") == "IsSupplementTo" or r.get("relationType") == "IsSourceOf") and r.get("relatedIdentifierType") == "DOI"
+        ]
+        if related and len(related) > 0:
+            record.referencedByDoi = related[0].get("relatedIdentifier")
+        else:
+            record.referencedByDoi = None
+
+        record.createdAt = str(attributes.get("created", "") or "")
+
+        record.updatedAt = max([
+            str(attributes.get("updated", "") or ""),
+            str(attributes.get("created", "") or ""),
+            str(attributes.get("registered", "") or "")
+        ])
+
+        for relation in attributes.get("relatedIdentifiers", []):
+            if relation.get("relationType") == "IsPreviousVersionOf":
+                record.isLatestVersion = False
+            if relation.get("relationType") == "HasVersion":
+                record.isLatestVersion = False
+
+        record.isConceptDoi = (
+            versionCount > 0 and
+            versionOfCount == 0   
+        )
+
+        record.haveCreatorAffiliation = self.check_agent_list_match(attributes.get("creators", []))
+        record.haveContributorAffiliation = self.check_agent_list_match(attributes.get("contributors", []))
+        return record
+
+    def check_agent_list_match(self, items: list) -> bool:
+        partial_ror = self.ror.split("https://ror.org/")[1] if self.ror else ""
+        for agent in items:
+            # Check if any nameIdentifier matches the ror
+            if any(identifier.get("nameIdentifier") == self.ror for identifier in agent.get("nameIdentifiers", [])):
+                return True
+            # Check if any nameIdentifier matches the partial ror
+            if any(identifier.get("nameIdentifier") == partial_ror for identifier in agent.get("nameIdentifiers", [])):
+                return True
+            # Check if the agent name matches any pattern
+            if match_patterns(agent.get("name"), self.name):
+                return True
+            # Check each affiliation
+            for affiliation in agent.get("affiliation", []):
+                if (affiliation.get("affiliationIdentifier") == self.ror or 
+                        match_patterns(affiliation.get("name"), self.name)):
+                    return True
+        return False
+
+    def all(self) -> list:
+        result = []
+        url = self.api_request_url()
+        while True:
+            response = self.get_api_result(url)
+            result.extend(response["data"])
+            logging.info(f"Retrieved DataCite {len(result)} of {response['meta']['total']}")
+            if response['links'].get('next'):
+                url = response['links']['next']
+            else:
+                break
+        return result
+
+    def count(self) -> int:
+        if not self.get_query_string():
+            return 0
+        url = self.api_request_url(page_size=0)
+        return self.get_api_result(url)["meta"]["total"]

roagg-2025.0.8/src/roagg/providers/openaire.py

@@ -0,0 +1,155 @@
+from typing import List
+import urllib.request
+import logging
+import json
+from roagg.models.research_output_item import ResearchOutputItem
+from roagg.helpers.utils import find_doi_in_text, is_valid_doi, string_word_count
+
+class OpenAireAPI:
+    openaire_base_url = "https://api.openaire.eu/graph/v1/"
+
+    def __init__(self, page_size: int = 100, ror: str = "", results: List[ResearchOutputItem] = []):
+        self.page_size = page_size
+        self.ror = ror
+        self.results = results
+    
+    def get_openaire_id_from_ror(self) -> str:
+        url = f"{self.openaire_base_url}organizations?pid={self.ror}"
+        with urllib.request.urlopen(url) as response:
+            json_response = json.loads(response.read())
+            
+            if 'results' in json_response and len(json_response['results']) > 0:
+                return json_response['results'][0]['id']
+            else:
+                return ""
+
+    def get_records(self) -> List[ResearchOutputItem]:
+        if not self.ror:
+            return []
+        openaire_results = []
+        openaire_id = self.get_openaire_id_from_ror()
+        
+        if not openaire_id:
+            logging.info(f"No OpenAire ID found for ROR {self.ror}")
+            return []
+
+        params = {
+            'pageSize': self.page_size,
+            'cursor': '*',
+            'type': 'dataset', # limit to only datasets for now
+            'relOrganizationId': openaire_id
+        }
+        retrieve_count = 0
+        while True:
+            query_string = urllib.parse.urlencode(params)
+            url = f"{self.openaire_base_url}researchProducts?{query_string}"
+            with urllib.request.urlopen(url) as response:
+                json_response = json.loads(response.read())
+                if 'results' in json_response:
+                    openaire_results.extend(json_response['results'])
+
+                retrieve_count = len(openaire_results)
+                logging.info(f"Retrieved OpenAire {retrieve_count} of {json_response['header']['numFound']}")
+
+                if 'nextCursor' in json_response['header'] and json_response['header']['nextCursor']:
+                    params['cursor'] = json_response['header']['nextCursor']
+                else:
+                    break
+
+        # Create a dictionary for O(1) lookups
+        doi_to_item = {item.doi.lower(): item for item in self.results if item.doi}
+        
+        for r in openaire_results:
+            openAireBestAccessRight = None
+            if 'bestAccessRight' in r and r['bestAccessRight'] and 'label' in r['bestAccessRight']:
+                openAireBestAccessRight = r['bestAccessRight']['label']
+            
+            openAireIndicatorsUsageCountsDownloads = None
+            if 'indicators' in r and r['indicators'] and 'usageCounts' in r['indicators']:
+                if 'downloads' in r['indicators']['usageCounts']:
+                    openAireIndicatorsUsageCountsDownloads = r['indicators']['usageCounts']['downloads']
+
+            openAireIndicatorsUsageCountsViews = None
+            if 'indicators' in r and r['indicators'] and 'usageCounts' in r['indicators']:
+                if 'views' in r['indicators']['usageCounts']:
+                    openAireIndicatorsUsageCountsViews = r['indicators']['usageCounts']['views']
+            
+            dois = self.get_doi_list_from_resource(r)
+            recordMatch = False
+            for doi in dois:
+                item = doi_to_item.get(doi.lower())
+                if item:
+                    recordMatch = True
+                    item.openAireBestAccessRight = openAireBestAccessRight
+                    item.openAireIndicatorsUsageCountsDownloads = openAireIndicatorsUsageCountsDownloads
+                    item.openAireIndicatorsUsageCountsViews = openAireIndicatorsUsageCountsViews
+                    item.inOpenAire = True
+            if not recordMatch and len(dois) > 0:
+                publication_date = r.get('publicationDate', None)
+                publication_year = None
+                if publication_date:
+                    publication_year = publication_date[:4] if len(publication_date) >= 4 else None
+                item = ResearchOutputItem(
+                    doi=dois[0],
+                    isPublisher=None,
+                    resourceType=r.get('type', None),
+                    title=r.get('mainTitle', None),
+                    publisher=r.get('publisher', None),
+                    publicationYear=publication_year,
+                    haveContributorAffiliation=None,
+                    haveCreatorAffiliation=None,
+                    isLatestVersion=None,
+                    isConceptDoi=None,
+                    inOpenAire=True,
+                    openAireBestAccessRight=openAireBestAccessRight,
+                    openAireIndicatorsUsageCountsDownloads=openAireIndicatorsUsageCountsDownloads,
+                    openAireIndicatorsUsageCountsViews=openAireIndicatorsUsageCountsViews,
+                    openAireId=r.get('id', None),
+                    titleWordCount=string_word_count(r.get('mainTitle', None))
+                )
+                self.results.append(item)
+                doi_to_item[item.doi.lower()] = item  # Add to lookup dictionary
+
+        return openaire_results
+
+    def get_doi_list_from_resource(self, resource: dict) -> List[str]:
+        doi_list = []
+        
+        for instance in resource['instances']:
+            logging.debug(f"Instance: {instance}")
+
+            if 'pids' in instance and len(instance['pids']) > 0:
+                for pid in instance['pids']:
+                    if pid['scheme'].lower() == 'doi':
+                        doi_list.append(pid['value'])
+
+            if 'alternateIdentifiers' in instance and len(instance['alternateIdentifiers']) > 0:
+                for alternateIdentifier in instance['alternateIdentifiers']:
+                    if alternateIdentifier['scheme'].lower() == 'doi':
+                        doi_list.append(alternateIdentifier['value'])
+
+            if len(doi_list) == 0:
+                # Normalize URLs to standard DOI format
+                url_replacements = [
+                    ("https://doi.pangaea.de/", "https://doi.org/"),
+                    ("https://zenodo.org/doi/", "https://doi.org/"),
+                    ("https://zenodo.org/records/", "https://doi.org/10.5281/zenodo.")
+                ]
+                
+                for url in instance['urls']:
+                    normalized_url = url
+                    for old_pattern, new_pattern in url_replacements:
+                        normalized_url = normalized_url.replace(old_pattern, new_pattern)
+
+                    for doi in find_doi_in_text(normalized_url):
+                        if is_valid_doi(doi):
+                            doi_list.append(doi)
+
+        # if doi_list is empty print json for instances
+        if len(doi_list) == 0:
+            logging.warning(f"No DOI found in resource: {json.dumps(resource['instances'], indent=2)}")
+
+        return list(set(doi_list))
+
+
+

roagg-2025.0.8/src/roagg/providers/openalex.py

@@ -0,0 +1,119 @@
+from typing import List
+import urllib.request
+import logging
+import json
+from roagg.models.research_output_item import ResearchOutputItem
+from roagg.helpers.utils import string_word_count, remove_resolver_prefix_from_doi
+
+class OpenAlexAPI:
+    openalex_base_url = "https://api.openalex.org/"
+
+    def __init__(self, page_size: int = 200, ror: str = "", results: List[ResearchOutputItem] = []):
+        self.page_size = page_size
+        self.ror = ror
+        self.results = results
+
+    def get_openalex_id_from_ror(self) -> str:
+        url = f"{self.openalex_base_url}institutions/ror:{self.ror}"
+        with urllib.request.urlopen(url) as response:
+            json_response = json.loads(response.read())
+            
+            if 'id' in json_response:
+                return json_response['id']
+            else:
+                return ""
+            
+    def get_records(self) -> List[ResearchOutputItem]:
+        if not self.ror:
+            return []
+        openalex_results = []
+        openalex_id = self.get_openalex_id_from_ror()
+        
+        if not openalex_id:
+            logging.info(f"No OpenAlex ID found for ROR {self.ror}")
+            return []
+
+        params = {
+            'per-page': self.page_size,
+            'cursor': '*',
+            'filter': f'institutions.id:{openalex_id},type:dataset' # limit to only datasets for now
+        }
+        retrieve_count = 0
+
+        while True:
+            query_string = urllib.parse.urlencode(params)
+            url = f"{self.openalex_base_url}works?{query_string}"
+            with urllib.request.urlopen(url) as response:
+                json_response = json.loads(response.read())
+                if 'results' in json_response:
+                    openalex_results.extend(json_response['results'])
+                retrieve_count = len(openalex_results)
+                logging.info(f"Retrieved OpenAlex {retrieve_count} of {json_response['meta']['count']}")
+
+                if 'next_cursor' in json_response['meta'] and json_response['meta']['next_cursor']:
+                    params['cursor'] = json_response['meta']['next_cursor']
+                else:
+                    break
+
+        # Create a dictionary for O(1) lookups
+        doi_to_item = {item.doi.lower(): item for item in self.results if item.doi}
+        
+        for r in openalex_results:
+            openAlexCitedByCount = None
+            if 'cited_by_count' in r:
+                openAlexCitedByCount = r['cited_by_count']
+
+            openAlexReferencedWorksCount = None
+            if 'referenced_works_count' in r:
+                openAlexReferencedWorksCount = r['referenced_works_count']
+
+            haveCreatorAffiliation = False
+ 
+            for authorship in r.get('institutions', []):
+                for affiliation in authorship.get('institutions', []):
+                    if affiliation.get('ror') == self.ror:
+                        haveCreatorAffiliation = True
+                        break
+
+            doi = remove_resolver_prefix_from_doi(r.get('doi', None))
+            if doi is None:
+                continue
+
+            recordMatch = False
+
+            item = doi_to_item.get(doi.lower())
+            if item:
+                recordMatch = True
+                item.openAlexCitedByCount = openAlexCitedByCount
+                item.openAlexReferencedWorksCount = openAlexReferencedWorksCount
+                item.inOpenAlex = True
+                item.openAlexId = r.get('id', None)
+                item.haveCreatorAffiliation = haveCreatorAffiliation
+            if not recordMatch:
+                publication_date = r.get('publication_date', None)
+                publication_year = r.get('publication_year', None)
+                if publication_date:
+                    publication_year = publication_date[:4] if len(publication_date) >= 4 else None
+                item = ResearchOutputItem(
+                    doi=doi,
+                    isPublisher=None,
+                    resourceType=r.get('type', None),
+                    title=r.get('title', None),
+                    publisher=None,
+                    publicationYear=publication_year,
+                    createdAt=r.get('created_date', None),
+                    updatedAt=r.get('updated_date', None),
+                    haveContributorAffiliation=None,
+                    haveCreatorAffiliation=haveCreatorAffiliation,
+                    isLatestVersion=None,   
+                    isConceptDoi=None,
+                    inOpenAlex=True,
+                    openAlexCitedByCount=openAlexCitedByCount,
+                    openAlexReferencedWorksCount=openAlexReferencedWorksCount,
+                    openAlexId=r.get('id', None),
+                    titleWordCount=string_word_count(r.get('title', None))
+                )
+                self.results.append(item)
+                doi_to_item[item.doi.lower()] = item  # Add to lookup dictionary
+
+        return openalex_results

roagg-2025.0.8/src/roagg.egg-info/PKG-INFO

@@ -0,0 +1,6 @@
+Metadata-Version: 2.4
+Name: roagg
+Version: 2025.0.8
+Requires-Python: >=3.10
+License-File: LICENSE
+Dynamic: license-file

roagg-2025.0.8/src/roagg.egg-info/SOURCES.txt

@@ -0,0 +1,19 @@
+LICENSE
+README.md
+pyproject.toml
+src/roagg/__init__.py
+src/roagg/__main__.py
+src/roagg/aggregator.py
+src/roagg/cli.py
+src/roagg.egg-info/PKG-INFO
+src/roagg.egg-info/SOURCES.txt
+src/roagg.egg-info/dependency_links.txt
+src/roagg.egg-info/entry_points.txt
+src/roagg.egg-info/top_level.txt
+src/roagg/helpers/ror.py
+src/roagg/helpers/utils.py
+src/roagg/models/research_output_item.py
+src/roagg/providers/datacite.py
+src/roagg/providers/openaire.py
+src/roagg/providers/openalex.py
+tests/test_utils.py

roagg-2025.0.8/src/roagg.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

roagg-2025.0.8/src/roagg.egg-info/entry_points.txt

@@ -0,0 +1,5 @@
+[console_scripts]
+roagg = roagg.cli:main
+
+[pipx.run]
+roagg = roagg.cli:main

roagg-2025.0.8/src/roagg.egg-info/top_level.txt

@@ -0,0 +1 @@
+roagg

roagg-2025.0.8/tests/test_utils.py

@@ -0,0 +1,255 @@
+import pytest
+from roagg.helpers.utils import is_valid_doi, find_doi_in_text, string_word_count
+
+
+class TestIsValidDoi:
+    """Test cases for the is_valid_doi function."""
+
+    def test_valid_doi_basic(self):
+        """Test basic valid DOI format."""
+        assert is_valid_doi("10.1234/example")
+
+    def test_valid_doi_with_longer_prefix(self):
+        """Test valid DOI with longer prefix number."""
+        assert is_valid_doi("10.123456789/test")
+
+    def test_valid_doi_with_special_characters(self):
+        """Test valid DOI with allowed special characters."""
+        assert is_valid_doi("10.1000/test-example_123")
+        assert is_valid_doi("10.1000/test.example")
+        assert is_valid_doi("10.1000/test(123)")
+        assert is_valid_doi("10.1000/test;123")
+        assert is_valid_doi("10.1000/test:123")
+        assert is_valid_doi("10.1000/test/subpath")
+
+    def test_valid_doi_with_uppercase(self):
+        """Test valid DOI with uppercase letters."""
+        assert is_valid_doi("10.1234/ABC-DEF")
+        assert is_valid_doi("10.1234/Test-Example")
+
+    def test_valid_doi_with_numbers(self):
+        """Test valid DOI with various numbers in suffix."""
+        assert is_valid_doi("10.1234/123456789")
+
+    def test_valid_doi_complex(self):
+        """Test complex real-world DOI examples."""
+        assert is_valid_doi("10.1000/182")
+        assert is_valid_doi("10.1038/nphys1170")
+        assert is_valid_doi("10.1016/j.cell.2009.01.002")
+
+    def test_invalid_doi_missing_prefix(self):
+        """Test invalid DOI without '10.' prefix."""
+        assert not is_valid_doi("11.1234/example")
+        assert not is_valid_doi("1.1234/example")
+        assert not is_valid_doi("1234/example")
+
+    def test_invalid_doi_short_prefix_number(self):
+        """Test invalid DOI with too short prefix number."""
+        assert not is_valid_doi("10.123/example")  # Only 3 digits
+        assert not is_valid_doi("10.12/example")   # Only 2 digits
+        assert not is_valid_doi("10.1/example")    # Only 1 digit
+
+    def test_invalid_doi_missing_slash(self):
+        """Test invalid DOI without slash separator."""
+        assert not is_valid_doi("10.1234-example")
+        assert not is_valid_doi("10.1234.example")
+
+    def test_invalid_doi_missing_suffix(self):
+        """Test invalid DOI without suffix."""
+        assert not is_valid_doi("10.1234/")
+        assert not is_valid_doi("10.1234")
+
+    def test_invalid_doi_empty_string(self):
+        """Test invalid DOI with empty string."""
+        assert not is_valid_doi("")
+
+    def test_invalid_doi_whitespace(self):
+        """Test invalid DOI with whitespace."""
+        assert not is_valid_doi("10.1234/ example")
+        assert not is_valid_doi("10.1234 /example")
+        assert not is_valid_doi(" 10.1234/example")
+        assert not is_valid_doi("10.1234/example ")
+
+    def test_invalid_doi_only_prefix(self):
+        """Test invalid DOI with only the prefix."""
+        assert not is_valid_doi("10.")
+
+    def test_invalid_doi_non_numeric_prefix(self):
+        """Test invalid DOI with non-numeric prefix."""
+        assert not is_valid_doi("10.abcd/example")
+
+    def test_edge_case_minimum_valid_length(self):
+        """Test DOI with minimum valid prefix length (4 digits)."""
+        assert is_valid_doi("10.1000/a")
+
+    def test_edge_case_maximum_valid_length(self):
+        """Test DOI with maximum valid prefix length (9 digits)."""
+        assert is_valid_doi("10.123456789/a")
+
+    def test_edge_case_too_long_prefix(self):
+        """Test DOI with prefix longer than 9 digits."""
+        assert not is_valid_doi("10.1234567890/example")
+
+
+class TestFindDoiInText:
+    """Test cases for the find_doi_in_text function."""
+
+    def test_pangea_doi_url(self):
+        """PANGAEA DOI URL."""
+        text = "https://doi.pangaea.de/10.1234/example"
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/example"]
+
+    def test_zenodo_doi_url(self):
+        """ZENODO DOI URL."""
+        text = "https://zenodo.org/doi/10.1234/example"
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/example"]
+
+    def test_single_doi_in_text(self):
+        """Test finding a single DOI in text."""
+        text = "This is a reference to 10.1234/example in the middle of text."
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/example"]
+
+    def test_multiple_dois_in_text(self):
+        """Test finding multiple DOIs in text."""
+        text = "See 10.1234/first and also 10.5678/second for more info."
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/first", "10.5678/second"]
+
+    def test_doi_at_start_of_text(self):
+        """Test finding DOI at the beginning of text."""
+        text = "10.1234/example is the DOI for this article."
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/example"]
+
+    def test_doi_at_end_of_text(self):
+        """Test finding DOI at the end of text."""
+        text = "The DOI for this article is 10.1234/example"
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/example"]
+
+    def test_doi_with_special_characters(self):
+        """Test finding DOIs with various special characters."""
+        text = "DOI: 10.1038/nphys1170 and 10.1016/j.cell.2009.01.002"
+        result = find_doi_in_text(text)
+        assert len(result) == 2
+        assert "10.1038/nphys1170" in result
+        assert "10.1016/j.cell.2009.01.002" in result
+
+    def test_doi_with_parentheses(self):
+        """Test finding DOI with parentheses."""
+        text = "See DOI 10.1234/test(2024) for details."
+        result = find_doi_in_text(text)
+        assert "10.1234/test(2024)" in result
+
+    def test_no_doi_in_text(self):
+        """Test text without any DOI."""
+        text = "This text has no DOI at all."
+        result = find_doi_in_text(text)
+        assert result == []
+
+    def test_empty_text(self):
+        """Test with empty string."""
+        result = find_doi_in_text("")
+        assert result == []
+
+    def test_doi_with_url(self):
+        """Test finding DOI within a URL."""
+        text = "Visit https://doi.org/10.1234/example for the article."
+        result = find_doi_in_text(text)
+        assert "10.1234/example" in result
+
+    def test_doi_minimum_prefix_length(self):
+        """Test DOI with minimum prefix length (4 digits)."""
+        text = "The DOI is 10.1000/test"
+        result = find_doi_in_text(text)
+        assert result == ["10.1000/test"]
+
+    def test_doi_maximum_prefix_length(self):
+        """Test DOI with maximum prefix length (9 digits)."""
+        text = "The DOI is 10.123456789/test"
+        result = find_doi_in_text(text)
+        assert result == ["10.123456789/test"]
+
+    def test_invalid_doi_too_short_prefix(self):
+        """Test that DOIs with too short prefix are not found."""
+        text = "This is not a valid DOI: 10.123/test"
+        result = find_doi_in_text(text)
+        assert result == []
+
+    def test_doi_with_underscores(self):
+        """Test finding DOI with underscores."""
+        text = "DOI: 10.1234/test_example_123"
+        result = find_doi_in_text(text)
+        assert "10.1234/test_example_123" in result
+
+    def test_doi_in_multiline_text(self):
+        """Test finding DOI in multiline text."""
+        text = """This is a paper.
+        The DOI is 10.1234/example
+        It was published in 2024."""
+        result = find_doi_in_text(text)
+        assert result == ["10.1234/example"]
+
+    def test_doi_with_mixed_case(self):
+        """Test finding DOI with mixed case letters."""
+        text = "DOI: 10.1234/AbCdEf123"
+        result = find_doi_in_text(text)
+        assert "10.1234/AbCdEf123" in result
+
+    def test_real_world_dois(self):
+        """Test finding real-world DOI examples."""
+        text = "See 10.1038/nature12373 and 10.1126/science.1259855 for more information."
+        result = find_doi_in_text(text)
+        assert len(result) == 2
+        assert "10.1038/nature12373" in result
+        assert "10.1126/science.1259855" in result
+
+class TestStringWordCount:
+    def test_single_word(self):
+        """Test with a single word."""
+        assert string_word_count("file.csv") == 1
+        assert string_word_count("another_file[2].jpg") == 1
+
+    def test_multiple_words(self):
+        """Test with multiple words."""
+        assert string_word_count("Hello World from Roagg") == 4
+
+    def test_leading_trailing_spaces(self):
+        """Test with leading and trailing spaces."""
+        # "Leading and trailing spaces" = 4 words (not 5)
+        assert string_word_count("   Leading and trailing spaces   ") == 4
+
+    def test_multiple_spaces_between_words(self):
+        """Test with multiple spaces between words."""
+        # "Multiple spaces between words" = 4 words (not 5)
+        assert string_word_count("Multiple   spaces   between   words") == 4
+
+    def test_empty_string(self):
+        """Test with an empty string."""
+        assert string_word_count("") == 0
+
+    def test_string_with_only_spaces(self):
+        """Test with a string containing only spaces."""
+        assert string_word_count("     ") == 0
+
+    def test_string_with_newlines_and_tabs(self):
+        """Test with newlines and tabs."""
+        assert string_word_count("Hello\nWorld\tfrom Roagg") == 4
+
+    def test_string_with_punctuation(self):
+        """Test with punctuation - split() doesn't separate on punctuation."""
+        # "Hello, world! This is Roagg." = 5 words (punctuation stays attached)
+        assert string_word_count("Hello, world! This is Roagg.") == 5
+
+    def test_string_with_special_characters(self):
+        """Test with special characters - split() doesn't separate on special chars."""
+        # "datafile.csv is ready!" = 3 words
+        assert string_word_count("datafile.csv is ready!  ") == 3
+
+    def test_string_with_numeric_characters(self):
+        """Test with numeric characters - split() doesn't separate on dots."""
+        # "Version 2.0 of the software" = 5 words (2.0 is one word)
+        assert string_word_count("Version 2.0 of the software    ") == 5