feed-ursus 1.0.0__tar.gz

feed_ursus-1.0.0/LICENSE

@@ -0,0 +1,29 @@
+BSD 3-Clause License
+
+Copyright (c) 2019, UCLA Library
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this
+   list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice,
+   this list of conditions and the following disclaimer in the documentation
+   and/or other materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its
+   contributors may be used to endorse or promote products derived from
+   this software without specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

feed_ursus-1.0.0/PKG-INFO

@@ -0,0 +1,134 @@
+Metadata-Version: 2.1
+Name: feed-ursus
+Version: 1.0.0
+Summary: Command line tool to feed Solr index for the UCLA Digital Library's frontend, Ursus (https://digital.library.ucla.edu/)
+Author: Your Name
+Author-email: you@example.com
+Requires-Python: >=3.10,<4.0
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Dist: click (>=8.1.3,<9.0.0)
+Requires-Dist: pysolr (>=3.8,<4.0)
+Requires-Dist: python-dateutil (>=2.8.2,<3.0.0)
+Requires-Dist: pyyaml (>=6.0.1,<7.0.0)
+Requires-Dist: requests (>=2.32.2,<3.0.0)
+Requires-Dist: rich (>=13.4.1,<14.0.0)
+Requires-Dist: setuptools (>=74.0.0,<75.0.0)
+Description-Content-Type: text/markdown
+
+# feed_ursus
+Script to process CSVs into an Ursus solr index.
+
+## Using feed_ursus
+
+For basic use, you can install feed_ursus as a systemwide command directly from github, without having to first clone the repository.
+
+### Installation
+
+We recommend installing with [pipx](https://pipx.pypa.io/). On MacOS, you can install pipx (and python!) with [homebrew](https://brew.sh):
+
+```
+brew install pipx pyenv
+pipx ensurepath
+```
+
+Then:
+
+```
+pipx install git+https://github.com/uclalibrary/feed_ursus.git
+```
+
+Pipx will install feed_ursus in its own virtualenv, but make the command accessible from anywhere so you don't need to active the virtualenv yourself.
+
+### Use
+
+Convert a csv into a json document that follows the data model of an Ursus solr index:
+
+```
+feed_ursus [path/to/your.csv]
+```
+
+This repo includes a docker-compose.yml file that will run local instances of solr and ursus for use in testing this script. To use them, first install [docker](https://docs.docker.com/install/) and [docker compose](https://docs.docker.com/compose/install/). Then run:
+
+```
+docker-compose up --detach
+docker-compose run web bundle exec rails db:setup
+```
+
+It might take a minute or so for solr to get up and running, at which point you should be able to see your new site at http://localhost:3000. Ursus will be empty, because you haven't loaded any data yet.
+
+To load data from a csv:
+
+```
+feed_ursus --solr_url=http://localhost:8983/solr/californica --mapping=dlp load [path/to/your.csv] 
+```
+
+### Mappers
+
+Different metadata mappings are included for general Digital Library use (`--mapping=dlp`) and for the Sinai Manuscripts Digital Library (`--mapping=sinai`). Because this script was originally used for the Sinai Manuscripts project, the default value is `sinai` for backwards compatibility.
+
+## Developing feed_ursus
+
+### Installing
+
+For development, clone the repository and use poetry to set up the virtualenv:
+
+```
+git clone git@github.com:UCLALibrary/feed_ursus.git
+cd feed_ursus
+pipx install poetry
+poetry install
+```
+
+Then, to activate the virtualenv:
+
+```
+poetry shell
+```
+
+The following will assume the virtualenv is active. You could also run e.g. `poetry run feed_ursus [path/to/your.csv]`
+
+### Using the development version
+
+```
+feed_ursus --solr_url http://localhost:8983/solr/californica load [path/to/your.csv]
+```
+
+### Running the tests
+
+Tests are written for [pytest](https://docs.pytest.org/en/latest/):
+
+```
+pytest
+```
+
+### Running the formatter and linters:
+
+black (formatter) will run in check mode in ci, so make sure you run it before committing:
+```
+black .
+```
+
+flake8 (linter) isn't currently running in ci, but should be put back in soon:
+```
+flake8
+```
+
+pylint (linter) isn't currently running in ci, but should be put back in soon:
+```
+pylint
+```
+
+mypy (static type checker) isn't currently running in ci, but should be put back in soon:
+```
+mypy
+```
+
+# Caveats
+
+## IIIF Manifests
+
+When importing a work, the script will always assume that a IIIF manifest exists at https://iiif.library.ucla.edu/[ark]/manifest, where [ark] is the URL-encoded Archival Resource Key of the work. This link should work, as long as a manifest has been pushed to that location by importing the work into [Fester](https://github.com/UCLALibrary/fester) or [Californica](https://github.com/UCLALibrary/californica). If you haven't done one of those, obviously, the link will fail and the image won't be visible, but metadata will import and be visible. A manifest can then be created and pushed to the expected location without re-running feed_ursus.py.
+

feed_ursus-1.0.0/README.md

@@ -0,0 +1,113 @@
+# feed_ursus
+Script to process CSVs into an Ursus solr index.
+
+## Using feed_ursus
+
+For basic use, you can install feed_ursus as a systemwide command directly from github, without having to first clone the repository.
+
+### Installation
+
+We recommend installing with [pipx](https://pipx.pypa.io/). On MacOS, you can install pipx (and python!) with [homebrew](https://brew.sh):
+
+```
+brew install pipx pyenv
+pipx ensurepath
+```
+
+Then:
+
+```
+pipx install git+https://github.com/uclalibrary/feed_ursus.git
+```
+
+Pipx will install feed_ursus in its own virtualenv, but make the command accessible from anywhere so you don't need to active the virtualenv yourself.
+
+### Use
+
+Convert a csv into a json document that follows the data model of an Ursus solr index:
+
+```
+feed_ursus [path/to/your.csv]
+```
+
+This repo includes a docker-compose.yml file that will run local instances of solr and ursus for use in testing this script. To use them, first install [docker](https://docs.docker.com/install/) and [docker compose](https://docs.docker.com/compose/install/). Then run:
+
+```
+docker-compose up --detach
+docker-compose run web bundle exec rails db:setup
+```
+
+It might take a minute or so for solr to get up and running, at which point you should be able to see your new site at http://localhost:3000. Ursus will be empty, because you haven't loaded any data yet.
+
+To load data from a csv:
+
+```
+feed_ursus --solr_url=http://localhost:8983/solr/californica --mapping=dlp load [path/to/your.csv] 
+```
+
+### Mappers
+
+Different metadata mappings are included for general Digital Library use (`--mapping=dlp`) and for the Sinai Manuscripts Digital Library (`--mapping=sinai`). Because this script was originally used for the Sinai Manuscripts project, the default value is `sinai` for backwards compatibility.
+
+## Developing feed_ursus
+
+### Installing
+
+For development, clone the repository and use poetry to set up the virtualenv:
+
+```
+git clone git@github.com:UCLALibrary/feed_ursus.git
+cd feed_ursus
+pipx install poetry
+poetry install
+```
+
+Then, to activate the virtualenv:
+
+```
+poetry shell
+```
+
+The following will assume the virtualenv is active. You could also run e.g. `poetry run feed_ursus [path/to/your.csv]`
+
+### Using the development version
+
+```
+feed_ursus --solr_url http://localhost:8983/solr/californica load [path/to/your.csv]
+```
+
+### Running the tests
+
+Tests are written for [pytest](https://docs.pytest.org/en/latest/):
+
+```
+pytest
+```
+
+### Running the formatter and linters:
+
+black (formatter) will run in check mode in ci, so make sure you run it before committing:
+```
+black .
+```
+
+flake8 (linter) isn't currently running in ci, but should be put back in soon:
+```
+flake8
+```
+
+pylint (linter) isn't currently running in ci, but should be put back in soon:
+```
+pylint
+```
+
+mypy (static type checker) isn't currently running in ci, but should be put back in soon:
+```
+mypy
+```
+
+# Caveats
+
+## IIIF Manifests
+
+When importing a work, the script will always assume that a IIIF manifest exists at https://iiif.library.ucla.edu/[ark]/manifest, where [ark] is the URL-encoded Archival Resource Key of the work. This link should work, as long as a manifest has been pushed to that location by importing the work into [Fester](https://github.com/UCLALibrary/fester) or [Californica](https://github.com/UCLALibrary/californica). If you haven't done one of those, obviously, the link will fail and the image won't be visible, but metadata will import and be visible. A manifest can then be created and pushed to the expected location without re-running feed_ursus.py.

feed_ursus-1.0.0/feed_ursus/date_parser.py

@@ -0,0 +1,60 @@
+"""
+Creates a multi-valued 'year_isim' field by parsing input strings.
+"""
+
+import re
+import typing
+from datetime import datetime
+from dateutil import parser
+
+
+NORMALIZED_RANGE = re.compile(r"(.*)/(.*)")
+
+
+def get_dates(normalized_dates: typing.Any):
+    """Maps a list of 'normalized_date' strings to a sorted list of datetime.
+
+    Args:
+        dates: A list of strings containing dates in the 'normalized_date' format.
+
+    Returns:
+        A list of years extracted from "dates".
+
+    """
+    if not isinstance(normalized_dates, typing.Iterable):
+        return []
+    solr_dts = set()
+    for normalized_date in normalized_dates:
+        if not isinstance(normalized_date, str):
+            continue
+        match = NORMALIZED_RANGE.search(normalized_date)
+        if match:
+            start_str, end_str = match.groups()
+            start = get_date(start_str)
+            end = get_date(end_str)
+            if start and end:
+                solr_dts.update({start, end})
+        else:
+            solr_date = get_date(normalized_date)
+            if solr_date:
+                solr_dts.add(solr_date)
+    return sorted(solr_dts)
+
+
+def get_date(date: str):
+    """Extracts the single 4-digit year found in the input date string.
+
+    Args:
+        date: a string containing a date in 'normalized_date' format.
+
+    Returns:
+        A single date.
+
+    """
+    try:
+        parsed_date = parser.parse(date, default=datetime(1978, 1, 1))
+        return parsed_date
+    except ValueError as err:
+        print(err)
+        return None
+    return None