folio-data-import 0.2.4__tar.gz → 0.2.6__tar.gz

folio_data_import-0.2.6/PKG-INFO

@@ -0,0 +1,121 @@
+Metadata-Version: 2.1
+Name: folio_data_import
+Version: 0.2.6
+Summary: A python module to interact with the data importing capabilities of the open-source FOLIO ILS
+License: MIT
+Author: Brooks Travis
+Author-email: brooks.travis@gmail.com
+Requires-Python: >=3.9,<4.0
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: flake8-bandit (>=4.1.1,<5.0.0)
+Requires-Dist: flake8-black (>=0.3.6,<0.4.0)
+Requires-Dist: flake8-bugbear (>=24.8.19,<25.0.0)
+Requires-Dist: flake8-docstrings (>=1.7.0,<2.0.0)
+Requires-Dist: flake8-isort (>=6.1.1,<7.0.0)
+Requires-Dist: folioclient (>=0.61.0,<0.62.0)
+Requires-Dist: httpx (>=0.27.2,<0.28.0)
+Requires-Dist: inquirer (>=3.4.0,<4.0.0)
+Requires-Dist: pyhumps (>=3.8.0,<4.0.0)
+Requires-Dist: pymarc (>=5.2.2,<6.0.0)
+Requires-Dist: tabulate (>=0.9.0,<0.10.0)
+Requires-Dist: tqdm (>=4.66.5,<5.0.0)
+Description-Content-Type: text/markdown
+
+# folio_data_import
+
+## Description
+
+This project is designed to import data into the FOLIO LSP. It provides a simple and efficient way to import data from various sources using FOLIO's REST APIs.
+
+## Features
+
+- Import MARC records using FOLIO's Data Import system
+- Import User records using FOLIO's User APIs
+
+## Installation
+
+## Installation
+
+To install the project using Poetry, follow these steps:
+
+1. Clone the repository.
+2. Navigate to the project directory: `$ cd /path/to/folio_data_import`.
+3. Install Poetry if you haven't already: `$ pip install poetry`.
+4. Install the project and its dependencies: `$ poetry install`.
+6. Run the application using Poetry: `$ poetry run python -m folio_data_import --help`.
+
+Make sure to activate the virtual environment created by Poetry before running the application.
+
+## Usage
+
+1. Prepare the data to be imported in the specified format.
+2. Run the application and follow the prompts to import the data.
+3. Monitor the import progress and handle any errors or conflicts that may arise.
+
+### folio-user-import
+When this package is installed via PyPI or using `poetry install` from this repository, it installs a convenience script in your `$PATH` called `folio-user-import`. To view all command line options for this script, run `folio-user-import -h`. In addition to supporting `mod-user-import`-style JSON objects, this script also allows you to manage service point assignments for users by specifying a `servicePointsUser` object in the JSON object, using service point codes in place of UUIDs in the `defaultServicePointId` and `servicePointIds` fields:
+```
+{
+    "username": "checkin-all",
+    "barcode": "1728439497039848103",
+    "active": true,
+    "type": "patron",
+    "patronGroup": "staff",
+    "departments": [],
+    "personal": {
+        "lastName": "Admin",
+        "firstName": "checkin-all",
+        "addresses": [
+          {
+            "countryId": "HU",
+            "addressLine1": "Andrássy Street 1.",
+            "addressLine2": "",
+            "city": "Budapest",
+            "region": "Pest",
+            "postalCode": "1061",
+            "addressTypeId": "Home",
+            "primaryAddress": true
+          }
+        ],
+        "preferredContactTypeId": "email"
+    },
+    "requestPreference": {
+        "holdShelf": true,
+        "delivery": false,
+        "fulfillment": "Hold Shelf"
+    }
+    "servicePointsUser": {
+        "defaultServicePointId": "cd1",
+        "servicePointsIds": [
+            "cd1",
+            "Online",
+            "000",
+            "cd2"
+        ]
+    }
+}
+```
+One thing to note here is that this importer does not require `externalSystemId` as the match point for your objects. While it is the default, if the user objects have `id` present, that will be used, falling back to `externalSystemId`. However, you can also specify `username` or `barcode` as the match point if desired, using the `--default_preferred_contact_type` argument.
+
+Another point of departure from the behavior of `mod-user-import` is the handling of `preferredContactTypeId`. This importer will accept either the `"001", "002", "003"...` values stored by the FOLIO, or the human-friendly strings used by `mod-user-import` (`"mail", "email", "text", "phone", "mobile"`). It will also __*set a customizable default for all users that do not otherwise have a valid value specified*__, unless a value is already present in the user record being updated.
+
+How to use:
+1. Generate a JSON lines (one JSON object per line) file of FOLIO user objects in the style of [mod-user-import](https://github.com/folio-org/mod-user-import)
+2. Run the script and specify the required arguments (and any desired optional arguments), including the path to your file of user objects
+
+
+## Contributing
+
+Contributions are welcome! If you have any ideas, suggestions, or bug reports, please open an issue or submit a pull request.
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).
+

folio_data_import-0.2.6/README.md

@@ -0,0 +1,90 @@
+# folio_data_import
+
+## Description
+
+This project is designed to import data into the FOLIO LSP. It provides a simple and efficient way to import data from various sources using FOLIO's REST APIs.
+
+## Features
+
+- Import MARC records using FOLIO's Data Import system
+- Import User records using FOLIO's User APIs
+
+## Installation
+
+## Installation
+
+To install the project using Poetry, follow these steps:
+
+1. Clone the repository.
+2. Navigate to the project directory: `$ cd /path/to/folio_data_import`.
+3. Install Poetry if you haven't already: `$ pip install poetry`.
+4. Install the project and its dependencies: `$ poetry install`.
+6. Run the application using Poetry: `$ poetry run python -m folio_data_import --help`.
+
+Make sure to activate the virtual environment created by Poetry before running the application.
+
+## Usage
+
+1. Prepare the data to be imported in the specified format.
+2. Run the application and follow the prompts to import the data.
+3. Monitor the import progress and handle any errors or conflicts that may arise.
+
+### folio-user-import
+When this package is installed via PyPI or using `poetry install` from this repository, it installs a convenience script in your `$PATH` called `folio-user-import`. To view all command line options for this script, run `folio-user-import -h`. In addition to supporting `mod-user-import`-style JSON objects, this script also allows you to manage service point assignments for users by specifying a `servicePointsUser` object in the JSON object, using service point codes in place of UUIDs in the `defaultServicePointId` and `servicePointIds` fields:
+```
+{
+    "username": "checkin-all",
+    "barcode": "1728439497039848103",
+    "active": true,
+    "type": "patron",
+    "patronGroup": "staff",
+    "departments": [],
+    "personal": {
+        "lastName": "Admin",
+        "firstName": "checkin-all",
+        "addresses": [
+          {
+            "countryId": "HU",
+            "addressLine1": "Andrássy Street 1.",
+            "addressLine2": "",
+            "city": "Budapest",
+            "region": "Pest",
+            "postalCode": "1061",
+            "addressTypeId": "Home",
+            "primaryAddress": true
+          }
+        ],
+        "preferredContactTypeId": "email"
+    },
+    "requestPreference": {
+        "holdShelf": true,
+        "delivery": false,
+        "fulfillment": "Hold Shelf"
+    }
+    "servicePointsUser": {
+        "defaultServicePointId": "cd1",
+        "servicePointsIds": [
+            "cd1",
+            "Online",
+            "000",
+            "cd2"
+        ]
+    }
+}
+```
+One thing to note here is that this importer does not require `externalSystemId` as the match point for your objects. While it is the default, if the user objects have `id` present, that will be used, falling back to `externalSystemId`. However, you can also specify `username` or `barcode` as the match point if desired, using the `--default_preferred_contact_type` argument.
+
+Another point of departure from the behavior of `mod-user-import` is the handling of `preferredContactTypeId`. This importer will accept either the `"001", "002", "003"...` values stored by the FOLIO, or the human-friendly strings used by `mod-user-import` (`"mail", "email", "text", "phone", "mobile"`). It will also __*set a customizable default for all users that do not otherwise have a valid value specified*__, unless a value is already present in the user record being updated.
+
+How to use:
+1. Generate a JSON lines (one JSON object per line) file of FOLIO user objects in the style of [mod-user-import](https://github.com/folio-org/mod-user-import)
+2. Run the script and specify the required arguments (and any desired optional arguments), including the path to your file of user objects
+
+
+## Contributing
+
+Contributions are welcome! If you have any ideas, suggestions, or bug reports, please open an issue or submit a pull request.
+
+## License
+
+This project is licensed under the [MIT License](LICENSE).

{folio_data_import-0.2.4 → folio_data_import-0.2.6}/pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "folio_data_import"
-version = "0.2.4"
+version = "0.2.6"
 description = "A python module to interact with the data importing capabilities of the open-source FOLIO ILS"
 authors = ["Brooks Travis <brooks.travis@gmail.com>"]
 license = "MIT"
@@ -14,8 +14,8 @@ folio-user-import = "folio_data_import.UserImport:sync_main"
 
 [tool.poetry.dependencies]
 python = "^3.9"
-folioclient = "^0.60.5"
-httpx = "^0.23.0"
+folioclient = "^0.61.0"
+httpx = "^0.27.2"
 pymarc = "^5.2.2"
 pyhumps = "^3.8.0"
 inquirer = "^3.4.0"

{folio_data_import-0.2.4 → folio_data_import-0.2.6}/src/folio_data_import/MARCDataImport.py

@@ -1,8 +1,10 @@
 import argparse
 import asyncio
 import glob
+import importlib
 import io
 import os
+import sys
 from typing import List
 import uuid
 from contextlib import ExitStack
@@ -30,6 +32,9 @@ except AttributeError:
 # The order in which the report summary should be displayed
 REPORT_SUMMARY_ORDERING = {"created": 0, "updated": 1, "discarded": 2, "error": 3}
 
+# Set default timeout and backoff values for HTTP requests when retrying job status and final summary checks
+RETRY_TIMEOUT_START = 1
+RETRY_TIMEOUT_RETRY_FACTOR = 2
 
 class MARCImportJob:
     """
@@ -69,6 +74,7 @@ class MARCImportJob:
         import_profile_name: str,
         batch_size=10,
         batch_delay=0,
+        marc_record_preprocessor=None,
         consolidate=False,
         no_progress=False,
     ) -> None:
@@ -79,6 +85,8 @@ class MARCImportJob:
         self.import_profile_name = import_profile_name
         self.batch_size = batch_size
         self.batch_delay = batch_delay
+        self.current_retry_timeout = None
+        self.marc_record_preprocessor = marc_record_preprocessor
 
     async def do_work(self) -> None:
         """
@@ -148,10 +156,23 @@ class MARCImportJob:
         Raises:
             IndexError: If the job execution with the specified ID is not found.
         """
-        job_status = self.folio_client.folio_get(
-            "/metadata-provider/jobExecutions?statusNot=DISCARDED&uiStatusAny"
-            "=PREPARING_FOR_PREVIEW&uiStatusAny=READY_FOR_PREVIEW&uiStatusAny=RUNNING&limit=50"
-        )
+        try:
+            self.current_retry_timeout = (
+                self.current_retry_timeout * RETRY_TIMEOUT_RETRY_FACTOR
+            ) if self.current_retry_timeout else RETRY_TIMEOUT_START
+            job_status = self.folio_client.folio_get(
+                "/metadata-provider/jobExecutions?statusNot=DISCARDED&uiStatusAny"
+                "=PREPARING_FOR_PREVIEW&uiStatusAny=READY_FOR_PREVIEW&uiStatusAny=RUNNING&limit=50"
+            )
+            self.current_retry_timeout = None
+        except httpx.ConnectTimeout:
+            sleep(.25)
+            with httpx.Client(
+                timeout=self.current_retry_timeout,
+                verify=self.folio_client.ssl_verify
+            ) as temp_client:
+                self.folio_client.httpx_client = temp_client
+                return await self.get_job_status()
         try:
             status = [
                 job for job in job_status["jobExecutions"] if job["id"] == self.job_id
@@ -316,6 +337,10 @@ class MARCImportJob:
                     await self.get_job_status()
                     sleep(0.25)
                 if record:
+                    if self.marc_record_preprocessor:
+                        record = await self.apply_marc_record_preprocessing(
+                            record, self.marc_record_preprocessor
+                        )
                     self.record_batch.append(record.as_marc())
                     counter += 1
                 else:
@@ -325,6 +350,39 @@ class MARCImportJob:
                     await self.create_batch_payload(counter, total_records, True),
                 )
 
+    @staticmethod
+    async def apply_marc_record_preprocessing(record: pymarc.Record, func_or_path) -> pymarc.Record:
+        """
+        Apply preprocessing to the MARC record before sending it to FOLIO.
+
+        Args:
+            record (pymarc.Record): The MARC record to preprocess.
+            func_or_path (Union[Callable, str]): The preprocessing function or its import path.
+
+        Returns:
+            pymarc.Record: The preprocessed MARC record.
+        """
+        if isinstance(func_or_path, str):
+            try:
+                path_parts = func_or_path.rsplit('.')
+                module_path, func_name = ".".join(path_parts[:-1]), path_parts[-1]
+                module = importlib.import_module(module_path)
+                func = getattr(module, func_name)
+            except (ImportError, AttributeError) as e:
+                print(f"Error importing preprocessing function {func_or_path}: {e}. Skipping preprocessing.")
+                return record
+        elif callable(func_or_path):
+            func = func_or_path
+        else:
+            print(f"Invalid preprocessing function: {func_or_path}. Skipping preprocessing.")
+            return record
+
+        try:
+            return func(record)
+        except Exception as e:
+            print(f"Error applying preprocessing function: {e}. Skipping preprocessing.")
+            return record
+
     async def create_batch_payload(self, counter, total_records, is_last) -> dict:
         """
         Create a batch payload for data import.
@@ -392,9 +450,7 @@ class MARCImportJob:
                     await self.get_job_status()
                 sleep(1)
             if self.finished:
-                job_summary = self.folio_client.folio_get(
-                    f"/metadata-provider/jobSummary/{self.job_id}"
-                )
+                job_summary = await self.get_job_summary()
                 job_summary.pop("jobExecutionId")
                 job_summary.pop("totalErrors")
                 columns = ["Summary"] + list(job_summary.keys())
@@ -425,6 +481,31 @@ class MARCImportJob:
             self.last_current = 0
             self.finished = False
 
+    async def get_job_summary(self) -> dict:
+        """
+        Retrieves the job summary for the current job execution.
+
+        Returns:
+            dict: The job summary for the current job execution.
+        """
+        try:
+            self.current_retry_timeout = (
+                self.current_retry_timeout * RETRY_TIMEOUT_RETRY_FACTOR
+            ) if self.current_retry_timeout else RETRY_TIMEOUT_START
+            job_summary = self.folio_client.folio_get(
+                f"/metadata-provider/jobSummary/{self.job_id}"
+            )
+            self.current_retry_timeout = None
+        except httpx.ReadTimeout:  #
+            sleep(.25)
+            with httpx.Client(
+                timeout=self.current_retry_timeout,
+                verify=self.folio_client.ssl_verify
+            ) as temp_client:
+                self.folio_client.httpx_client = temp_client
+                return await self.get_job_summary()
+        return job_summary
+
 
 async def main() -> None:
     """
@@ -467,6 +548,15 @@ async def main() -> None:
         help="The number of seconds to wait between record batches.",
         default=0.0,
     )
+    parser.add_argument(
+        "--preprocessor",
+        type=str,
+        help=(
+            "The path to a Python module containing a preprocessing function "
+            "to apply to each MARC record before sending to FOLIO."
+        ),
+        default=None,
+    )
     parser.add_argument(
         "--consolidate",
         action="store_true",
@@ -491,6 +581,17 @@ async def main() -> None:
     if args.member_tenant_id:
         folio_client.okapi_headers["x-okapi-tenant"] = args.member_tenant_id
 
+    if os.path.isabs(args.marc_file_path):
+        marc_files = [Path(x) for x in glob.glob(args.marc_file_path)]
+    else:
+        marc_files = list(Path("./").glob(args.marc_file_path))
+
+    if len(marc_files) == 0:
+        print(f"No files found matching {args.marc_file_path}. Exiting.")
+        sys.exit(1)
+    else:
+        print(marc_files)
+
     if not args.import_profile_name:
         import_profiles = folio_client.folio_get(
             "/data-import-profiles/jobProfiles",
@@ -511,8 +612,6 @@ async def main() -> None:
         ]
         answers = inquirer.prompt(questions)
         args.import_profile_name = answers["import_profile_name"]
-    marc_files = [Path(x) for x in glob.glob(args.marc_file_path, root_dir="./")]
-    print(marc_files)
     try:
         await MARCImportJob(
             folio_client,
@@ -520,6 +619,7 @@ async def main() -> None:
             args.import_profile_name,
             batch_size=args.batch_size,
             batch_delay=args.batch_delay,
+            marc_record_preprocessor=args.preprocessor,
             consolidate=bool(args.consolidate),
             no_progress=bool(args.no_progress),
         ).do_work()

{folio_data_import-0.2.4 → folio_data_import-0.2.6}/src/folio_data_import/UserImport.py

@@ -21,6 +21,14 @@ except AttributeError:
 
     utc = zoneinfo.ZoneInfo("UTC")
 
+# Mapping of preferred contact type IDs to their corresponding values
+PREFERRED_CONTACT_TYPES_MAP = {
+    "001": "mail",
+    "002": "email",
+    "003": "text",
+    "004": "phone",
+    "005": "mobile",
+}
 
 class UserImporter:  # noqa: R0902
     """
@@ -41,6 +49,7 @@ class UserImporter:  # noqa: R0902
         http_client: httpx.AsyncClient,
         user_match_key: str = "externalSystemId",
         only_update_present_fields: bool = False,
+        default_preferred_contact_type: str = "002",
     ) -> None:
         self.limit_simultaneous_requests = limit_simultaneous_requests
         self.batch_size = batch_size
@@ -56,10 +65,14 @@ class UserImporter:  # noqa: R0902
         self.department_map: dict = self.build_ref_data_id_map(
             self.folio_client, "/departments", "departments", "name"
         )
+        self.service_point_map: dict = self.build_ref_data_id_map(
+            self.folio_client, "/service-points", "servicepoints", "code"
+        )
         self.logfile: AsyncTextIOWrapper = logfile
         self.errorfile: AsyncTextIOWrapper = errorfile
         self.http_client: httpx.AsyncClient = http_client
         self.only_update_present_fields: bool = only_update_present_fields
+        self.default_preferred_contact_type: str = default_preferred_contact_type
         self.match_key = user_match_key
         self.lock: asyncio.Lock = asyncio.Lock()
         self.logs: dict = {"created": 0, "updated": 0, "failed": 0}
@@ -87,7 +100,8 @@ class UserImporter:  # noqa: R0902
 
         This method triggers the process of importing users by calling the `process_file` method.
         """
-        await self.process_file()
+        with open(self.user_file_path, "r", encoding="utf-8") as openfile:
+            await self.process_file(openfile)
 
     async def get_existing_user(self, user_obj) -> dict:
         """
@@ -255,13 +269,14 @@ class UserImporter:  # noqa: R0902
         if mapped_departments:
             user_obj["departments"] = mapped_departments
 
-    async def update_existing_user(self, user_obj, existing_user) -> Tuple[dict, dict]:
+    async def update_existing_user(self, user_obj, existing_user, protected_fields) -> Tuple[dict, dict]:
         """
         Updates an existing user with the provided user object.
 
         Args:
             user_obj (dict): The user object containing the updated user information.
             existing_user (dict): The existing user object to be updated.
+            protected_fields (dict): A dictionary containing the protected fields and their values.
 
         Returns:
             tuple: A tuple containing the updated existing user object and the API response.
@@ -270,6 +285,8 @@ class UserImporter:  # noqa: R0902
             None
 
         """
+        await self.set_preferred_contact_type(user_obj, existing_user)
+        preferred_contact_type = {"preferredContactTypeId": existing_user.get("personal", {}).pop("preferredContactTypeId")}
         if self.only_update_present_fields:
             new_personal = user_obj.pop("personal", {})
             existing_personal = existing_user.pop("personal", {})
@@ -290,6 +307,18 @@ class UserImporter:  # noqa: R0902
                 existing_user["personal"] = existing_personal
         else:
             existing_user.update(user_obj)
+        if "personal" in existing_user:
+            existing_user["personal"].update(preferred_contact_type)
+        else:
+            existing_user["personal"] = preferred_contact_type
+        for key, value in protected_fields.items():
+            if type(value) is dict:
+                try:
+                    existing_user[key].update(value)
+                except KeyError:
+                    existing_user[key] = value
+            else:
+                existing_user[key] = value
         create_update_user = await self.http_client.put(
             self.folio_client.okapi_url + f"/users/{existing_user['id']}",
             headers=self.folio_client.okapi_headers,
@@ -320,7 +349,41 @@ class UserImporter:  # noqa: R0902
             self.logs["created"] += 1
         return response.json()
 
-    async def create_or_update_user(self, user_obj, existing_user, line_number) -> dict:
+    async def set_preferred_contact_type(self, user_obj, existing_user) -> None:
+        """
+        Sets the preferred contact type for a user object. If the provided preferred contact type
+        is not valid, the default preferred contact type is used, unless the previously existing
+        user object has a valid preferred contact type set. In that case, the existing preferred
+        contact type is used.
+        """
+        if "personal" in user_obj and "preferredContactTypeId" in user_obj["personal"]:
+            current_pref_contact = user_obj["personal"].get(
+                "preferredContactTypeId", ""
+            )
+            if mapped_contact_type := dict([(v, k) for k, v in PREFERRED_CONTACT_TYPES_MAP.items()]).get(
+                current_pref_contact,
+                "",
+            ):
+                existing_user["personal"]["preferredContactTypeId"] = mapped_contact_type
+            else:
+                existing_user["personal"]["preferredContactTypeId"] = current_pref_contact if current_pref_contact in PREFERRED_CONTACT_TYPES_MAP else self.default_preferred_contact_type
+        else:
+            print(
+                f"Preferred contact type not provided or is not a valid option: {PREFERRED_CONTACT_TYPES_MAP}\n"
+                f"Setting preferred contact type to {self.default_preferred_contact_type} or using existing value"
+            )
+            await self.logfile.write(
+                f"Preferred contact type not provided or is not a valid option: {PREFERRED_CONTACT_TYPES_MAP}\n"
+                f"Setting preferred contact type to {self.default_preferred_contact_type} or using existing value\n"
+            )
+            mapped_contact_type = existing_user.get("personal", {}).get(
+                "preferredContactTypeId", ""
+            ) or self.default_preferred_contact_type
+            if "personal" not in existing_user:
+                existing_user["personal"] = {}
+            existing_user["personal"]["preferredContactTypeId"] = mapped_contact_type or self.default_preferred_contact_type
+
+    async def create_or_update_user(self, user_obj, existing_user, protected_fields, line_number) -> dict:
         """
         Creates or updates a user based on the given user object and existing user.
 
@@ -334,7 +397,7 @@ class UserImporter:  # noqa: R0902
         """
         if existing_user:
             existing_user, update_user = await self.update_existing_user(
-                user_obj, existing_user
+                user_obj, existing_user, protected_fields
             )
             try:
                 update_user.raise_for_status()
@@ -375,7 +438,7 @@ class UserImporter:  # noqa: R0902
 
     async def process_user_obj(self, user: str) -> dict:
         """
-        Process a user object.
+        Process a user object. If not type is found in the source object, type is set to "patron".
 
         Args:
             user (str): The user data to be processed, as a json string.
@@ -386,17 +449,34 @@ class UserImporter:  # noqa: R0902
         """
         user_obj = json.loads(user)
         user_obj["type"] = user_obj.get("type", "patron")
-        if "personal" in user_obj:
-            current_pref_contact = user_obj["personal"].get(
-                "preferredContactTypeId", ""
-            )
-            user_obj["personal"]["preferredContactTypeId"] = (
-                current_pref_contact
-                if current_pref_contact in ["001", "002", "003"]
-                else "002"
-            )
         return user_obj
 
+    async def get_protected_fields(self, existing_user) -> dict:
+        """
+        Retrieves the protected fields from the existing user object.
+
+        Args:
+            existing_user (dict): The existing user object.
+
+        Returns:
+            dict: A dictionary containing the protected fields and their values.
+        """
+        protected_fields = {}
+        protected_fields_list = existing_user.get("customFields", {}).get("protectedFields", "").split(",")
+        for field in protected_fields_list:
+            if len(field.split(".")) > 1:
+                field, subfield = field.split(".")
+                if field not in protected_fields:
+                    protected_fields[field] = {}
+                protected_fields[field][subfield] = existing_user.get(field, {}).pop(subfield, None)
+                if protected_fields[field][subfield] is None:
+                    protected_fields[field].pop(subfield)
+            else:
+                protected_fields[field] = existing_user.pop(field, None)
+                if protected_fields[field] is None:
+                    protected_fields.pop(field)
+        return protected_fields
+
     async def process_existing_user(self, user_obj) -> Tuple[dict, dict, dict, dict]:
         """
         Process an existing user.
@@ -410,14 +490,19 @@ class UserImporter:  # noqa: R0902
                    and the existing PU object (existing_pu).
         """
         rp_obj = user_obj.pop("requestPreference", {})
+        spu_obj = user_obj.pop("servicePointsUser")
         existing_user = await self.get_existing_user(user_obj)
         if existing_user:
             existing_rp = await self.get_existing_rp(user_obj, existing_user)
             existing_pu = await self.get_existing_pu(user_obj, existing_user)
+            existing_spu = await self.get_existing_spu(existing_user)
+            protected_fields = await self.get_protected_fields(existing_user)
         else:
             existing_rp = {}
             existing_pu = {}
-        return rp_obj, existing_user, existing_rp, existing_pu
+            existing_spu = {}
+            protected_fields = {}
+        return rp_obj, spu_obj, existing_user, protected_fields, existing_rp, existing_pu, existing_spu
 
     async def create_or_update_rp(self, rp_obj, existing_rp, new_user_obj):
         """
@@ -528,14 +613,14 @@ class UserImporter:  # noqa: R0902
         """
         async with self.limit_simultaneous_requests:
             user_obj = await self.process_user_obj(user)
-            rp_obj, existing_user, existing_rp, existing_pu = (
+            rp_obj, spu_obj, existing_user, protected_fields, existing_rp, existing_pu, existing_spu = (
                 await self.process_existing_user(user_obj)
             )
             await self.map_address_types(user_obj, line_number)
             await self.map_patron_groups(user_obj, line_number)
             await self.map_departments(user_obj, line_number)
             new_user_obj = await self.create_or_update_user(
-                user_obj, existing_user, line_number
+                user_obj, existing_user, protected_fields, line_number
             )
             if new_user_obj:
                 try:
@@ -572,42 +657,162 @@ class UserImporter:  # noqa: R0902
                         )
                         print(pu_error_message)
                         await self.logfile.write(pu_error_message + "\n")
+                await self.handle_service_points_user(spu_obj, existing_spu, new_user_obj)
+
+    async def map_service_points(self, spu_obj, existing_user):
+        """
+        Maps the service points of a user object using the provided service point map.
+
+        Args:
+            spu_obj (dict): The service-points-user object to update.
+            existing_user (dict): The existing user object associated with the spu_obj.
+
+        Returns:
+            None
+        """
+        if "servicePointsIds" in spu_obj:
+            mapped_service_points = []
+            for sp in spu_obj.pop("servicePointsIds", []):
+                try:
+                    mapped_service_points.append(self.service_point_map[sp])
+                except KeyError:
+                    print(
+                        f'Service point "{sp}" not found, excluding service point from user: '
+                        f'{self.service_point_map}'
+                    )
+            if mapped_service_points:
+                spu_obj["servicePointsIds"] = mapped_service_points
+        if "defaultServicePointId" in spu_obj:
+            sp_code = spu_obj.pop('defaultServicePointId', '')
+            try:
+                mapped_sp_id = self.service_point_map[sp_code]
+                if mapped_sp_id not in spu_obj.get('servicePointsIds', []):
+                    print(
+                        f'Default service point "{sp_code}" not found in assigned service points, '
+                        'excluding default service point from user'
+                    )
+                else:
+                    spu_obj['defaultServicePointId'] = mapped_sp_id
+            except KeyError:
+                print(
+                    f'Default service point "{sp_code}" not found, excluding default service '
+                    f'point from user: {existing_user["id"]}'
+                )
+
+    async def handle_service_points_user(self, spu_obj, existing_spu, existing_user):
+        """
+        Handles processing a service-points-user object for a user.
+
+        Args:
+            spu_obj (dict): The service-points-user object to process.
+            existing_spu (dict): The existing service-points-user object, if it exists.
+            existing_user (dict): The existing user object associated with the spu_obj.
+        """
+        if spu_obj is not None:
+            await self.map_service_points(spu_obj, existing_user)
+            if existing_spu:
+                await self.update_existing_spu(spu_obj, existing_spu)
+            else:
+                await self.create_new_spu(spu_obj, existing_user)
+
+    async def get_existing_spu(self, existing_user):
+        """
+        Retrieves the existing service-points-user object for a given user.
+
+        Args:
+            existing_user (dict): The existing user object.
+
+        Returns:
+            dict: The existing service-points-user object.
+        """
+        try:
+            existing_spu = await self.http_client.get(
+                self.folio_client.okapi_url + "/service-points-users",
+                headers=self.folio_client.okapi_headers,
+                params={"query": f"userId=={existing_user['id']}"},
+            )
+            existing_spu.raise_for_status()
+            existing_spu = existing_spu.json().get("servicePointsUsers", [])
+            existing_spu = existing_spu[0] if existing_spu else {}
+        except httpx.HTTPError:
+            existing_spu = {}
+        return existing_spu
+
+    async def create_new_spu(self, spu_obj, existing_user):
+        """
+        Creates a new service-points-user object for a given user.
+
+        Args:
+            spu_obj (dict): The service-points-user object to create.
+            existing_user (dict): The existing user object.
+
+        Returns:
+            None
+        """
+        spu_obj["userId"] = existing_user["id"]
+        response = await self.http_client.post(
+            self.folio_client.okapi_url + "/service-points-users",
+            headers=self.folio_client.okapi_headers,
+            json=spu_obj,
+        )
+        response.raise_for_status()
+
+    async def update_existing_spu(self, spu_obj, existing_spu):
+        """
+        Updates an existing service-points-user object with the provided service-points-user object.
+
+        Args:
+            spu_obj (dict): The service-points-user object containing the updated values.
+            existing_spu (dict): The existing service-points-user object to be updated.
+
+        Returns:
+            None
+        """
+        existing_spu.update(spu_obj)
+        response = await self.http_client.put(
+            self.folio_client.okapi_url + f"/service-points-users/{existing_spu['id']}",
+            headers=self.folio_client.okapi_headers,
+            json=existing_spu,
+        )
+        response.raise_for_status()
 
-    async def process_file(self) -> None:
+    async def process_file(self, openfile) -> None:
         """
         Process the user object file.
+
+        Args:
+            openfile: The file object to process.
         """
-        with open(self.user_file_path, "r", encoding="utf-8") as openfile:
-            tasks = []
-            for line_number, user in enumerate(openfile):
-                tasks.append(self.process_line(user, line_number))
-                if len(tasks) == self.batch_size:
-                    start = time.time()
-                    await asyncio.gather(*tasks)
-                    duration = time.time() - start
-                    async with self.lock:
-                        message = (
-                            f"{dt.now().isoformat(sep=' ', timespec='milliseconds')}: "
-                            f"Batch of {self.batch_size} users processed in {duration:.2f} "
-                            f"seconds. - Users created: {self.logs['created']} - Users updated: "
-                            f"{self.logs['updated']} - Users failed: {self.logs['failed']}"
-                        )
-                        print(message)
-                        await self.logfile.write(message + "\n")
-                    tasks = []
-            if tasks:
+        tasks = []
+        for line_number, user in enumerate(openfile):
+            tasks.append(self.process_line(user, line_number))
+            if len(tasks) == self.batch_size:
                 start = time.time()
                 await asyncio.gather(*tasks)
                 duration = time.time() - start
                 async with self.lock:
                     message = (
                         f"{dt.now().isoformat(sep=' ', timespec='milliseconds')}: "
-                        f"Batch of {self.batch_size} users processed in {duration:.2f} seconds. - "
-                        f"Users created: {self.logs['created']} - Users updated: "
+                        f"Batch of {self.batch_size} users processed in {duration:.2f} "
+                        f"seconds. - Users created: {self.logs['created']} - Users updated: "
                         f"{self.logs['updated']} - Users failed: {self.logs['failed']}"
                     )
                     print(message)
                     await self.logfile.write(message + "\n")
+                tasks = []
+        if tasks:
+            start = time.time()
+            await asyncio.gather(*tasks)
+            duration = time.time() - start
+            async with self.lock:
+                message = (
+                    f"{dt.now().isoformat(sep=' ', timespec='milliseconds')}: "
+                    f"Batch of {len(tasks)} users processed in {duration:.2f} seconds. - "
+                    f"Users created: {self.logs['created']} - Users updated: "
+                    f"{self.logs['updated']} - Users failed: {self.logs['failed']}"
+                )
+                print(message)
+                await self.logfile.write(message + "\n")
 
 
 async def main() -> None:
@@ -626,6 +831,10 @@ async def main() -> None:
         --batch_size (int): How many records to process before logging statistics. Default 250.
         --folio_password (str): The FOLIO password.
         --user_match_key (str): The key to use to match users. Default "externalSystemId".
+        --report_file_base_path (str): The base path for the log and error files. Default "./".
+        --update_only_present_fields (bool): Only update fields that are present in the new user object.
+        --default_preferred_contact_type (str): The default preferred contact type to use if the provided \
+            value is not valid or not present. Default "002".
 
     Raises:
         Exception: If an unknown error occurs during the import process.
@@ -663,11 +872,26 @@ async def main() -> None:
         choices=["externalSystemId", "barcode", "username"],
         default="externalSystemId",
     )
+    parser.add_argument(
+        "--report_file_base_path",
+        help="The base path for the log and error files",
+        default="./",
+    )
     parser.add_argument(
         "--update_only_present_fields",
         help="Only update fields that are present in the user object",
         action="store_true",
     )
+    parser.add_argument(
+        "--default_preferred_contact_type",
+        help=(
+            "The default preferred contact type to use if the provided value is not present or not valid. "
+            "Note: '002' is the default, and will be used if the provided value is not valid or not present, "
+            "unless the existing user object being updated has a valid preferred contact type set."
+        ),
+        choices=list(PREFERRED_CONTACT_TYPES_MAP.keys()) + list(PREFERRED_CONTACT_TYPES_MAP.values()),
+        default="002",
+    )
     args = parser.parse_args()
 
     library_name = args.library_name
@@ -692,13 +916,13 @@ async def main() -> None:
         folio_client.okapi_headers["x-okapi-tenant"] = args.member_tenant_id
 
     user_file_path = Path(args.user_file_path)
+    report_file_base_path = Path(args.report_file_base_path)
     log_file_path = (
-        user_file_path.parent.parent
-        / "reports"
+        report_file_base_path
         / f"log_user_import_{dt.now(utc).strftime('%Y%m%d_%H%M%S')}.log"
     )
     error_file_path = (
-        user_file_path.parent
+        report_file_base_path
         / f"failed_user_import_{dt.now(utc).strftime('%Y%m%d_%H%M%S')}.txt"
     )
     async with aiofiles.open(
@@ -719,6 +943,7 @@ async def main() -> None:
                 http_client,
                 args.user_match_key,
                 args.update_only_present_fields,
+                args.default_preferred_contact_type,
             )
             await importer.do_import()
         except Exception as ee:

folio_data_import-0.2.6/src/folio_data_import/marc_preprocessors/__init__.py

@@ -0,0 +1 @@
+from ._preprocessors import prepend_ppn_prefix_001, strip_999_ff_fields

folio_data_import-0.2.6/src/folio_data_import/marc_preprocessors/_preprocessors.py

@@ -0,0 +1,31 @@
+import pymarc
+
+def prepend_ppn_prefix_001(record: pymarc.Record) -> pymarc.Record:
+    """
+    Prepend the PPN prefix to the record's 001 field. Useful when
+    importing records from the ABES SUDOC catalog
+
+    Args:
+        record (pymarc.Record): The MARC record to preprocess.
+
+    Returns:
+        pymarc.Record: The preprocessed MARC record.
+    """
+    record['001'].data = '(PPN)' + record['001'].data
+    return record
+
+def strip_999_ff_fields(record: pymarc.Record) -> pymarc.Record:
+    """
+    Strip all 999 fields with ff indicators from the record.
+    Useful when importing records exported from another FOLIO system
+
+    Args:
+        record (pymarc.Record): The MARC record to preprocess.
+
+    Returns:
+        pymarc.Record: The preprocessed MARC record.
+    """
+    for field in record.get_fields('999'):
+        if field.indicators == pymarc.Indicators(*['f', 'f']):
+            record.remove_field(field)
+    return record

folio_data_import-0.2.4/PKG-INFO

@@ -1,68 +0,0 @@
-Metadata-Version: 2.1
-Name: folio_data_import
-Version: 0.2.4
-Summary: A python module to interact with the data importing capabilities of the open-source FOLIO ILS
-License: MIT
-Author: Brooks Travis
-Author-email: brooks.travis@gmail.com
-Requires-Python: >=3.9,<4.0
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Programming Language :: Python :: 3.11
-Classifier: Programming Language :: Python :: 3.12
-Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
-Requires-Dist: flake8-bandit (>=4.1.1,<5.0.0)
-Requires-Dist: flake8-black (>=0.3.6,<0.4.0)
-Requires-Dist: flake8-bugbear (>=24.8.19,<25.0.0)
-Requires-Dist: flake8-docstrings (>=1.7.0,<2.0.0)
-Requires-Dist: flake8-isort (>=6.1.1,<7.0.0)
-Requires-Dist: folioclient (>=0.60.5,<0.61.0)
-Requires-Dist: httpx (>=0.23.0,<0.24.0)
-Requires-Dist: inquirer (>=3.4.0,<4.0.0)
-Requires-Dist: pyhumps (>=3.8.0,<4.0.0)
-Requires-Dist: pymarc (>=5.2.2,<6.0.0)
-Requires-Dist: tabulate (>=0.9.0,<0.10.0)
-Requires-Dist: tqdm (>=4.66.5,<5.0.0)
-Description-Content-Type: text/markdown
-
-# folio_data_import
-
-## Description
-
-This project is designed to import data into the FOLIO LSP. It provides a simple and efficient way to import data from various sources using FOLIO's REST APIs.
-
-## Features
-
-- Import MARC records using FOLIO's Data Import system
-- Import User records using FOLIO's User APIs
-
-## Installation
-
-## Installation
-
-To install the project using Poetry, follow these steps:
-
-1. Clone the repository.
-2. Navigate to the project directory: `$ cd /path/to/folio_data_import`.
-3. Install Poetry if you haven't already: `$ pip install poetry`.
-4. Install the project dependencies: `$ poetry install`.
-6. Run the application using Poetry: `$ poetry run python -m folio_data_import --help`.
-
-Make sure to activate the virtual environment created by Poetry before running the application.
-
-## Usage
-
-1. Prepare the data to be imported in the specified format.
-2. Run the application and follow the prompts to import the data.
-3. Monitor the import progress and handle any errors or conflicts that may arise.
-
-## Contributing
-
-Contributions are welcome! If you have any ideas, suggestions, or bug reports, please open an issue or submit a pull request.
-
-## License
-
-This project is licensed under the [MIT License](LICENSE).
-

folio_data_import-0.2.4/README.md

@@ -1,38 +0,0 @@
-# folio_data_import
-
-## Description
-
-This project is designed to import data into the FOLIO LSP. It provides a simple and efficient way to import data from various sources using FOLIO's REST APIs.
-
-## Features
-
-- Import MARC records using FOLIO's Data Import system
-- Import User records using FOLIO's User APIs
-
-## Installation
-
-## Installation
-
-To install the project using Poetry, follow these steps:
-
-1. Clone the repository.
-2. Navigate to the project directory: `$ cd /path/to/folio_data_import`.
-3. Install Poetry if you haven't already: `$ pip install poetry`.
-4. Install the project dependencies: `$ poetry install`.
-6. Run the application using Poetry: `$ poetry run python -m folio_data_import --help`.
-
-Make sure to activate the virtual environment created by Poetry before running the application.
-
-## Usage
-
-1. Prepare the data to be imported in the specified format.
-2. Run the application and follow the prompts to import the data.
-3. Monitor the import progress and handle any errors or conflicts that may arise.
-
-## Contributing
-
-Contributions are welcome! If you have any ideas, suggestions, or bug reports, please open an issue or submit a pull request.
-
-## License
-
-This project is licensed under the [MIT License](LICENSE).