psdi-data-conversion 0.1.6__tar.gz → 0.2.0__tar.gz

psdi_data_conversion-0.2.0/CHANGELOG.md

@@ -0,0 +1,70 @@
+# Changelog for PSDI Data Conversion
+
+## v0.2.0
+
+### New and Changed Functionality
+
+- Changed the keyword arguments `upload_dir` and `download_dir` to `input_dir` and `output_dir` respectively
+- Formats can now be specified case-insensitively
+- When requesting details on a format through the command-line interface, details will be provided on which molecular properties it supports (e.g. whether or not it supports connections information)
+- Added function `database.get_conversion_pathway` which can be used to get possible conversion routes between formats a direct conversion isn't possible with any converter
+- When requesting details on two formats through the command-line interface and a direct conversion between them is not possible, a possible chain conversion will now be recommended
+
+### Bugfixes
+
+- Fixed bug where the `input_dir` keyword argument for `run_converter` was being ignored
+- Fixed bug where the local-mode-only text was incorrectly appearing on the report page in service mode
+
+### Testing Changes
+
+- Excluded GUI modules from the calculating unit test coverage which can't be measured by the tool
+- Added automated test that the production deployment is working on a schedule and after deploying to it
+
+### Documentation Changes
+
+- The Documentation page of the GUI now shows the mode that's being run, the most recent tag, and the SHA of the most recent commit (if this isn't the latest tagged commit)
+- Updated release procedure and checklist in `CONTRIBUTING.md` to reflect current procedure
+
+### Formatting and Refactoring Changes
+
+- Changed Documentation and Accessibility pages of the GUI to work as Flask templates
+- Cleaned up Flask files to not be all in one module
+- Changed the database functionality to store possible conversions as a graph instead of a table
+- Dockerfile now builds from `pyproject.toml`, with the now-unused `requirements.txt` removed
+
+### Stylistic Changes
+
+- Reformatted pages of the GUI/web app to use a two panel display, with instructions for components in boxes alongside them
+
+## v0.1.7
+
+### New and Changed Functionality
+
+- Version, SHA, and service/prod modes now always shown in new About section on the Documentation page
+
+### Documentation Changes
+
+- Added information about deployment to CONTRIBUTING.md
+
+### Bugfixes
+
+- Environmental variable indicating dev or production mode should now be properly set for the deployed service
+
+## v0.1.6
+
+### New and Changed Functionality
+
+- SHA banner at the bottom of home page now preferentially shows the version, only showing the SHA if the current version doesn't match the last tag
+
+### Bugfixes
+
+- Fixed bug which was blocking deployment to production
+
+## v0.1.0
+
+Initial public release. Features included:
+
+- Online server functionality
+- Locally-hosted server
+- Command-line interface
+- Python library

{psdi_data_conversion-0.1.6 → psdi_data_conversion-0.2.0}/CONTRIBUTING.md

@@ -15,13 +15,13 @@ This project uses a version of [GitLab Flow](https://about.gitlab.com/topics/ver
 
 The following tasks should be completed before merging a release candidate branch to `release`:
 
-- Detemine the target version based on the changes made:
+- Determine the target version based on the changes made:
 
   - If any breaking changes have been made (after version 1.0.0), the version will advance to the next major version - `X.Y.Z` to `(X+1).0.0`
   - Otherwise, if any features are added, or any breaking changes are made before version 1.0.0, the version will advance to the next minor version - `X.Y.Z` to `X.(Y+1).0`.
   - Otherwise, the version will advance to the next bugfix version - `X.Y.Z` to `X.Y.(Z+1)`.
 
-- Create a release candidate branch with the name `rc-<target-version>` (e.g. `rc-1.2.3`). This should trigger an automated workflow to create a Pull Request from this branch to`release`. You may wish to edit the PR's name and/or description.
+- Create a release candidate branch with the name `rc-<target-version>` (e.g. `rc-1.2.3`), branched off of `main`. This should trigger an automated workflow to create a Pull Request from this branch to `release`. You may wish to edit the PR's name and/or description.
 
 - Tagging of the release is handled by an automated workflow which determines the new version based on the previous version and the commit history, looking for any commits which indicate a feature addition or breaking change using [Angular convention](https://github.com/conventional-changelog/conventional-changelog/tree/master/packages/conventional-changelog-angular#commit-message-format). Since we don't practice this regularly, you'll need to make a commit with this style to indicate any feature additions or breaking changes (this can be done when updating the version in the next step):
 
@@ -53,7 +53,7 @@ The following tasks should be completed before merging a release candidate branc
 
 - Ensure that all automated tests and checks pass - these should be run automatically on the PR opened above
 
-- Manually test the local web interface
+- Manually test the web interface. At this stage, it should be deployed to dev at https://data-conversion-dev.psdi.ac.uk/ (requires VPN to access), and it can be run locally as well
 
   - If there have been any changes to the Python backend, run a test that a file can be converted successfully and produces a proper log
   - If there have been any changes to the web frontend, check the appearance of the site to ensure that it looks as desired. Test the Accessibility page to ensure that changes there work properly, are saved when requested and apply to other pages
@@ -65,7 +65,10 @@ If any of these tasks fail and require changes, make the needed changes and then
 Then, follow the following steps to make the release:
 
 1. Merge the pull request to `release`. The release candidate branch can be safely deleted. This should trigger an automated pipeline to tag, publish, and deploy and the new code.
-2. Merge `release` into `main` via PR (obviously don't delete `release` - if it even gives you the option to, something has gone wrong in the project rulesets, so report this).
+2. After the above pipeline finishes, confirm that the changes are shown live on the staging site at https://data-conversion-staging.psdi.ac.uk/ by checking the version shown at the bottom of the Documentation page. If necessary, double-check that nothing has broken due to the slight changes in appearance between the dev and staging sites
+3. Manually trigger the `CI - Deploy to production cluster` workflow on the `release` branch to deploy the site from the staging to release environment, which will make the changes visible to users
+4. After completion of the workflow, confirm that the changes are live on the production site at https://data-conversion.psdi.ac.uk/ by checking the version shown at the bottom of the Documentation page
+5. Merge `release` into `main` via PR (obviously don't delete `release` - if it even gives you the option to, something has gone wrong in the project rulesets, so report this).
 
 ## Changelog
 
@@ -193,13 +196,13 @@ class MyFileConverter(FileConverter):
 converter = MyFileConverter
 ```
 
-That's all you need to do! The `psdi_data_conversion.converter` module parses all modules in the `converters` package to find converters, so if you've done everything correctly, it will find the new converter and register it for you. You can test that it is properly registered by using the CLA to run:
+That's all you need to do! The `psdi_data_conversion.converter` module parses all modules in the `converters` package to find converters, so if you've done everything correctly, it will find the new converter and register it for you. You can test that it is properly registered by using the CLI to run:
 
 ```bash
 psdi-data-convert -l
 ```
 
-Your new converter should appear, or else you will probably see an error message which will detail an exception raised when trying to register it. Note that until the converter's information is added to the database (the file `psdi_data_conversion/static/data/data.json`), the CLA will show that it is unable to perform any conversions, and it will fail on any conversion (believing it to be impossible) unless you provide the `--nc/--no-check` command-line flag.
+Your new converter should appear, or else you will probably see an error message which will detail an exception raised when trying to register it. Note that until the converter's information is added to the database (the file `psdi_data_conversion/static/data/data.json`), the CLI will show that it is unable to perform any conversions, and it will fail on any conversion (believing it to be impossible) unless you provide the `--nc/--no-check` command-line flag.
 
 For file converters which can be run with a call to a script, this can be streamlined even further by taking advantage of the `ScriptFileConverter` subclass. With this, the converter's subclass can be defined even more succinctly:
 
@@ -273,7 +276,7 @@ pip install --editable .[test]
 pytest
 ```
 
-This installs the project in a virtual environment in "editable" mode (which means the source files will be used from where they are rather than being copied, so any changes to them will be directly reflected in tests and uses of the CLA) and then calls `pytest` to run the unit tests in the project. `pytest` will automatically pick up any extra tests you add and run them as well.
+This installs the project in a virtual environment in "editable" mode (which means the source files will be used from where they are rather than being copied, so any changes to them will be directly reflected in tests and uses of the CLI) and then calls `pytest` to run the unit tests in the project. `pytest` will automatically pick up any extra tests you add and run them as well.
 
 #### Web App Integration
 
@@ -320,13 +323,13 @@ This project uses various GitHub workflows to perform Continuous Integration tas
 - Testing and scanning (triggered by pushes to `main`, `release`, `feature*`, and `rc*` branches)
 - Periodic checks for updates to common assets
 - Automatic creation of pull requests for `feature*` and `rc*` branches
-- Automatic tagging, publishing, and deployment of the `release` branch
+- Automatic tagging, publishing, and deployment of the `main` and `release` branches to the development, staging and production environments
 
-See the commons within the files for further details.
+See the comments within the files for further details. See also the [section on deployment](#deployment) for details specific to deployment tasks.
 
 ## Publishing
 
-The Python library, CLA, and local GUI are published as a Python package via PyPI. This section describes how the package is set up and how it's published.
+The Python library, CLI, and local GUI are published as a Python package via PyPI. This section describes how the package is set up and how it's published.
 
 ### Package Setup
 
@@ -338,7 +341,7 @@ The version of the package is set to be determined from the version control syst
 
 ### Initial Publication
 
-This section details the proceduce for the initial publication of this package - now that this is complete, this section is left in for reference in case of future need.
+This section details the procedure for the initial publication of this package - now that this is complete, this section is left in for reference in case of future need.
 
 First, it's necessary to install a couple required packages in order to build a Python package: `build` to build it and `twine` to upload it. These can be installed with pip via:
 
@@ -376,8 +379,67 @@ The management page can also be used to add or remove collaborators through the
 
 ## Deployment
 
-Deployment is handled by the `job-deploy-k8s.yml` reusable workflow, which is triggered from `ci-main.yml` to deploy to dev on each push to `main` and `ci-release.yml` to deploy to staging on each push to `release`. When a push to production is desired, it must be triggered manually by calling the `ci-deploy-production.yml` workflow on the `release` branch (it's set up to fail if run on any other branch).
-
-TODO - Describe details of setup
+The `ci-main.yml`, `ci-release.yml` and `ci-deploy-production.yml` files in the `.github/workflows` directory house workflows which deploy
+the data conversion service to [Kubernetes](https://kubernetes.io/) clusters hosted in STFC. There are three clusters, each of which correspond
+to a different deployment _environment_ for the data conversion service. The three environments are `development`, `staging` and `production`.
+Deployment to `development`, `staging` and `production` is done from either the `main` or `release` branch. The table below indicates which
+branch deploys to which environment. The table also shows, for each environment:
+
+- the URL on which the service is exposed once it is successfully deployed
+- the accessibility of the service. Depending on the environment the service is either accessible to the _public_ at the specified URL,
+  or accessible only to IP addresses within the _STFC and University of Southampton subnets_
+- the trigger used to invoke the workflow which deploys the service from the source branch. Deployment is either _automatic_
+  upon a commit to the source branch which passes the unit-tests job; or results from a _manual_ invocation of a workflow by a
+  developer.
+
+| Environment   | URL                                        | Accessibility                              | Source branch | Deployment trigger |
+| ------------- | ------------------------------------------ | ------------------------------------------ | ------------- | ------------------ |
+| `development` | https://data-conversion-dev.psdi.ac.uk     | STFC and University of Southampton subnets | `main`        | Automatic          |
+| `staging`     | https://data-conversion-staging.psdi.ac.uk | STFC and University of Southampton subnets | `release`     | Automatic          |
+| `production`  | https://data-conversion.psdi.ac.uk         | public                                     | `release`     | Manual             |
+
+Thus the `main` is automatically deployed to the `development` environment, and the `release` branch is automatically deployed to the `staging`
+environment. However deployment from the `release` branch to the `production` environment is a manual process. This is to allow developers to
+manually check that the `release` version works correctly in the `staging` environment before deploying it to the `production` environment.
+The checks to `staging` before deployment to `production` should echo those described in the above
+section [Release Checklist and Procedure](#release-checklist-and-procedure).
+
+### How to deploy to the `production` environment
+
+To trigger the workflow which deploys the service to the `production` environment, from the [main page of the repo](https://github.com/PSDI-UK/psdi-data-conversion)
+navigate to [Actions](https://github.com/PSDI-UK/psdi-data-conversion/actions). Here you should see on the right a list of recent workflow
+runs, including whether or not they are successful (as indicated by a green tick); and on the left you should see a list of all workflows.
+
+The workflow which deploys the `release` branch to the `staging` environment is named `CI - Release`. As mentioned above, you should verify
+that this workflow successfully deployed the `release` version to `staging` before considering deploying to `production`. If you click on the
+latest workflow run of `CI - Release` then you can see a breakdown of the workflow into its constituent workflows. Note the `deploy-stfc-staging-k8s`
+job. If this job is successful then the `release` version has been successfully deployed to `staging`.
+
+Assuming this is the case, navigating back to [Actions](https://github.com/PSDI-UK/psdi-data-conversion/actions), note that there is a workflow
+listed on the left named `CI - Deploy to production cluster`. This is the workflow which must be invoked manually to deploy the `release`
+version to the `production` environment. Clicking on the link to this workflow gives
+[a list of recent invocations](https://github.com/PSDI-UK/psdi-data-conversion/actions/workflows/ci-deploy-production.yml) of the workflow.
+Moreover, a light blue banner appears which says `This workflow has a workflow_dispatch event trigger` on the left and has a `Run workflow` button
+on the right. To invoke the workflow, press this button, _select the `release` branch_ as the option for `Use workflow from` dropdown menu, and
+then finally click the green `Run workflow` button. Once the workflow has been invoked you should be able to see its progress in real time on the
+same page.
+
+### Further technical details
+
+The `ci-main.yml`, `ci-release.yml` and `ci-deploy-production.yml` workflows leverage the `job-deploy-k8s.yml` callable workflow to do the heavy
+lifting with regards to deployment to STFC Kubernetes environments. The `job-deploy-k8s.yml` workflow is coded to be invoked on PSDI's GitHub
+runners. These runners are hosted within STFC's network, providing a means to access the three STFC Kubernetes clusters corresponding to the
+`development`, `staging` and `production` environments. The environment to be targeted for deployment is passed to the `job-deploy-k8s.yml`
+workflow as an input parameter. Given a target environment `<env>`, the `job-deploy-k8s.yml` workflow will deploy the service to the
+the `<env>` environment using the Kubernetes manifests stored in the `deploy`/<env>` directory of this repository.
+
+The workflow relies on several repository secrets, namely `KUBECONF`, `IMAGEPULLSECRET`, `CERTIFICATE_PRIVATE_KEY` and `CERTIFICATE_PEM` for
+various purposes.
+
+- `KUBECONF` provides Kubernetes-specific information pertaining to the the Kubernetes cluster for the target environment. Note that
+  `KUBECONF` is an _environment_-dependent secret, taking different values for each of the environments.
+- `IMAGEPULLSECRET` enables the container image housing the data conversion service to be pulled from this GitHub repo to the PSDI runner
+- `CERTIFICATE_PRIVATE_KEY` and `CERTIFICATE_PEM` pertain to the TLS certificates for the service
+  For further information see the `job-deploy-k8s.yml` file and aforementioned Kubernetes manifests.
 
 The server can be configured by editing the environmental variables set in `Dockerfile`.

{psdi_data_conversion-0.1.6 → psdi_data_conversion-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: psdi_data_conversion
-Version: 0.1.6
+Version: 0.2.0
 Summary: Chemistry file format conversion service, provided by PSDI
 Project-URL: Homepage, https://data-conversion.psdi.ac.uk/
 Project-URL: Documentation, https://psdi-uk.github.io/psdi-data-conversion/
@@ -222,8 +222,13 @@ Classifier: Programming Language :: Python :: 3
 Classifier: Topic :: File Formats
 Classifier: Topic :: Scientific/Engineering :: Chemistry
 Requires-Python: >=3.12
+Requires-Dist: igraph
 Requires-Dist: openbabel-wheel
 Requires-Dist: py
+Provides-Extra: deploy
+Requires-Dist: flask; extra == 'deploy'
+Requires-Dist: gunicorn; extra == 'deploy'
+Requires-Dist: requests; extra == 'deploy'
 Provides-Extra: doc
 Requires-Dist: pdoc3; extra == 'doc'
 Provides-Extra: gui
@@ -243,6 +248,9 @@ Description-Content-Type: text/markdown
 
 # PSDI Data Conversion
 
+[![License Badge](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+![Coverage Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/brgillis/dbd938192dc4de9b7779978e515c0e79/raw/covbadge.json)
+
 Release date: 2024-04-29
 
 This is the repository for the PSDI PF2 Chemistry File Format Conversion project. The goal of this project is to provide utilities to assist in converting files between the many different file formats used in chemistry, providing information on what converters are available for a given conversion and the expected quality of it, and providing multiple interfaces to perform these conversions. These interfaces are:
@@ -327,7 +335,6 @@ This is the repository for the PSDI PF2 Chemistry File Format Conversion project
 - `LICENSE` (Apache License version 2.0)
 - `pyproject.toml` (Python project metadata and settings)
 - `README.md` (This file)
-- `requirements.txt` (Requirements for the web app deployment of this project)
 
 ## Requirements
 

{psdi_data_conversion-0.1.6 → psdi_data_conversion-0.2.0}/README.md

@@ -1,5 +1,8 @@
 # PSDI Data Conversion
 
+[![License Badge](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0)
+![Coverage Badge](https://img.shields.io/endpoint?url=https://gist.githubusercontent.com/brgillis/dbd938192dc4de9b7779978e515c0e79/raw/covbadge.json)
+
 Release date: 2024-04-29
 
 This is the repository for the PSDI PF2 Chemistry File Format Conversion project. The goal of this project is to provide utilities to assist in converting files between the many different file formats used in chemistry, providing information on what converters are available for a given conversion and the expected quality of it, and providing multiple interfaces to perform these conversions. These interfaces are:
@@ -84,7 +87,6 @@ This is the repository for the PSDI PF2 Chemistry File Format Conversion project
 - `LICENSE` (Apache License version 2.0)
 - `pyproject.toml` (Python project metadata and settings)
 - `README.md` (This file)
-- `requirements.txt` (Requirements for the web app deployment of this project)
 
 ## Requirements
 

psdi_data_conversion-0.2.0/psdi_data_conversion/app.py

@@ -0,0 +1,77 @@
+"""app.py
+
+Version 1.0, 8th November 2024
+
+This script acts as a server for the PSDI Data Conversion Service website.
+"""
+
+from argparse import ArgumentParser
+
+from psdi_data_conversion import constants as const
+from psdi_data_conversion.gui.env import update_env
+from psdi_data_conversion.gui.setup import get_app, limit_upload_size, start_app
+from psdi_data_conversion.main import print_wrap
+
+app = get_app()
+
+
+def main():
+    """Standard entry-point function for this script.
+    """
+
+    parser = ArgumentParser()
+
+    parser.add_argument("--use-env-vars", action="store_true",
+                        help="If set, all other arguments and defaults for this script are ignored, and environmental "
+                        "variables and their defaults will instead control execution. These defaults will result in "
+                        "the app running in production server mode.")
+
+    parser.add_argument("--max-file-size", type=float, default=const.DEFAULT_MAX_FILE_SIZE/const.MEGABYTE,
+                        help="The maximum allowed filesize in MB - 0 (default) indicates no maximum")
+
+    parser.add_argument("--max-file-size-ob", type=float, default=const.DEFAULT_MAX_FILE_SIZE_OB/const.MEGABYTE,
+                        help="The maximum allowed filesize in MB for the Open Babel converter, taking precendence over "
+                        "the general maximum file size when Open Babel is used - 0 indicates no maximum. Default 1 MB.")
+
+    parser.add_argument("--service-mode", action="store_true",
+                        help="If set, will run as if deploying a service rather than the local GUI")
+
+    parser.add_argument("--dev-mode", action="store_true",
+                        help="If set, will expose development elements, such as the SHA of the latest commit")
+
+    parser.add_argument("--debug", action="store_true",
+                        help="If set, will run the Flask server in debug mode, which will cause it to automatically "
+                        "reload if code changes and show an interactive debugger in the case of errors")
+
+    parser.add_argument("--log-mode", type=str, default=const.LOG_FULL,
+                        help="How logs should be stored. Allowed values are: \n"
+                        "- 'full' - Multi-file logging, not recommended for the CLI, but allowed for a compatible "
+                        "interface with the public web app"
+                        "- 'simple' - Logs saved to one file"
+                        "- 'stdout' - Output logs and errors only to stdout"
+                        "- 'none' - Output only errors to stdout")
+
+    parser.add_argument("--log-level", type=str, default=None,
+                        help="The desired level to log at. Allowed values are: 'DEBUG', 'INFO', 'WARNING', 'ERROR, "
+                             "'CRITICAL'. Default: 'INFO' for logging to file, 'WARNING' for logging to stdout")
+
+    # Set global variables for settings based on parsed arguments, unless it's set to use env vars
+    args = parser.parse_args()
+
+    if not args.use_env_vars:
+        # Overwrite the values from environmental variables with the values from the command-line arguments
+        update_env(args)
+
+    # Set the upload limit based on provided arguments now
+    limit_upload_size()
+
+    print_wrap("Starting the PSDI Data Conversion GUI. This GUI is run as a webpage, which you can open by "
+               "right-clicking the link below to open it in your default browser, or by copy-and-pasting it into your "
+               "browser of choice.")
+
+    start_app()
+
+
+if __name__ == "__main__":
+
+    main()

{psdi_data_conversion-0.1.6 → psdi_data_conversion-0.2.0}/psdi_data_conversion/constants.py

@@ -36,6 +36,9 @@ import shutil
 # The name of the command-line script
 CL_SCRIPT_NAME = "psdi-data-convert"
 
+# The name of the Flask app
+APP_NAME = "psdi_data_conversion"
+
 # Environmental variables
 LOG_MODE_EV = "LOG_MODE"
 LOG_LEVEL_EV = "LOG_LEVEL"
@@ -52,8 +55,8 @@ MEGABYTE = 1024*1024
 DEFAULT_MAX_FILE_SIZE = 0 * MEGABYTE
 DEFAULT_MAX_FILE_SIZE_OB = 1 * MEGABYTE
 
-DEFAULT_UPLOAD_DIR = './psdi_data_conversion/static/uploads'
-DEFAULT_DOWNLOAD_DIR = './psdi_data_conversion/static/downloads'
+DEFAULT_INPUT_DIR = './psdi_data_conversion/static/uploads'
+DEFAULT_OUTPUT_DIR = './psdi_data_conversion/static/downloads'
 
 # Filename of the database, relative to the base of the python package
 DATABASE_FILENAME = "static/data/data.json"
@@ -150,11 +153,6 @@ QUAL_2D_LABEL = "2D atomic coordinates are"
 QUAL_3D_KEY = "three_dim"
 QUAL_3D_LABEL = "2D atomic coordinates are"
 
-D_QUAL_LABELS = {QUAL_COMP_KEY: QUAL_COMP_LABEL,
-                 QUAL_CONN_KEY: QUAL_CONN_LABEL,
-                 QUAL_2D_KEY: QUAL_2D_LABEL,
-                 QUAL_3D_KEY: QUAL_3D_LABEL}
-
 # Notes for conversion quality
 QUAL_NOTE_IN_UNKNOWN = ("Potential data extrapolation: {} represented in the output format but its representation in "
                         "the input format is unknown")
@@ -187,3 +185,9 @@ ERR_WRONG_EXTENSIONS = "Input file '{file}' does not have expected extension '{e
 ERR_EMPTY_ARCHIVE = "No files to convert were contained in archive"
 ERR_CONVERSION_FAILED = ("File conversion failed for one or more files. Lines from the output log "
                          "{} which indicate possible sources of error: ")
+
+# Misc
+# ----
+
+# Year in seconds
+YEAR = 365*24*60*60

{psdi_data_conversion-0.1.6 → psdi_data_conversion-0.2.0}/psdi_data_conversion/converter.py

@@ -5,22 +5,20 @@ Created 2024-12-10 by Bryan Gillis.
 Class and functions to perform file conversion
 """
 
-from dataclasses import dataclass, field
-import json
 import glob
 import importlib
+import json
 import os
 import sys
 import traceback
-from typing import Any, Callable, NamedTuple
-from multiprocessing import Lock
-from psdi_data_conversion import log_utility
 from collections.abc import Callable
 from dataclasses import dataclass, field
+from multiprocessing import Lock
 from tempfile import TemporaryDirectory
 from typing import Any, NamedTuple
 
 from psdi_data_conversion import constants as const
+from psdi_data_conversion import log_utility
 from psdi_data_conversion.converters import base
 from psdi_data_conversion.converters.openbabel import CONVERTER_OB
 from psdi_data_conversion.file_io import (is_archive, is_supported_archive, pack_zip_or_tar, split_archive_ext,
@@ -184,9 +182,9 @@ def get_converter(*args, name=const.CONVERTER_DEFAULT, **converter_kwargs) -> ba
     use_envvars : bool
         If set to True, environment variables will be checked for any that set options for this class and used,
         default False
-    upload_dir : str
+    input_dir : str
         The location of input files relative to the current directory
-    download_dir : str
+    output_dir : str
         The location of output files relative to the current directory
     max_file_size : float
         The maximum allowed file size for input/output files, in MB, default 1 MB for Open Babel, unlimited for other
@@ -315,11 +313,18 @@ def check_from_format(filename: str,
     return False
 
 
+def _run_single_file_conversion(*args, **kwargs):
+    """Run a conversion on a single file, after all arguments have been checked
+    """
+    return get_converter(*args, **kwargs).run()
+
+
 def run_converter(filename: str,
                   to_format: str,
                   *args,
                   from_format: str | None = None,
-                  download_dir=const.DEFAULT_DOWNLOAD_DIR,
+                  input_dir=const.DEFAULT_INPUT_DIR,
+                  output_dir=const.DEFAULT_OUTPUT_DIR,
                   max_file_size=None,
                   log_file: str | None = None,
                   log_mode=const.LOG_SIMPLE,
@@ -351,9 +356,9 @@ def run_converter(filename: str,
     use_envvars : bool
         If set to True, environment variables will be checked for any that set options for this class and used,
         default False
-    upload_dir : str
+    input_dir : str
         The location of input files relative to the current directory
-    download_dir : str
+    output_dir : str
         The location of output files relative to the current directory
     strict : bool
         If True and `from_format` is not None, will fail if any input file has the wrong extension (including files
@@ -410,7 +415,12 @@ def run_converter(filename: str,
     # converter class, so it needs to be set up here to match what will be set up there
     if log_file is None:
         base_filename = os.path.basename(split_archive_ext(filename)[0])
-        log_file = os.path.join(download_dir, base_filename + const.OUTPUT_LOG_EXT)
+        log_file = os.path.join(output_dir, base_filename + const.OUTPUT_LOG_EXT)
+
+    if os.path.exists(filename):
+        qualified_filename = filename
+    else:
+        qualified_filename = os.path.join(input_dir, filename)
 
     # Check if the filename is for an archive file, and handle appropriately
 
@@ -425,15 +435,16 @@ def run_converter(filename: str,
         # Not an archive, so just get and run the converter straightforwardly
         if from_format is not None:
             check_from_format(filename, from_format, strict=strict)
-        l_run_output.append(get_converter(filename,
+        l_run_output.append(_run_single_file_conversion(filename,
                             to_format,
                             *args,
                             from_format=from_format,
-                            download_dir=download_dir,
+                            input_dir=input_dir,
+                            output_dir=output_dir,
                             max_file_size=max_file_size,
                             log_file=log_file,
                             log_mode=log_mode,
-                            **converter_kwargs).run())
+                            **converter_kwargs))
 
     elif not is_supported_archive(filename):
         raise base.FileConverterInputException(f"{filename} is an unsupported archive type. Supported types are: "
@@ -443,7 +454,7 @@ def run_converter(filename: str,
         # The filename is of a supported archive type. Make a temporary directory to extract its contents
         # to, then run the converter on each file extracted
         with TemporaryDirectory() as extract_dir:
-            l_filenames = unpack_zip_or_tar(filename, extract_dir=extract_dir)
+            l_filenames = unpack_zip_or_tar(qualified_filename, extract_dir=extract_dir)
 
             # Check for no files in archive
             if len(l_filenames) == 0:
@@ -468,15 +479,15 @@ def run_converter(filename: str,
                 individual_log_mode = log_mode if log_mode != const.LOG_FULL else const.LOG_FULL_FORCE
 
                 try:
-                    individual_run_output = get_converter(extracted_filename,
-                                                          to_format,
-                                                          *args,
-                                                          from_format=from_format,
-                                                          download_dir=download_dir,
-                                                          log_file=individual_log_file,
-                                                          log_mode=individual_log_mode,
-                                                          max_file_size=remaining_file_size,
-                                                          **converter_kwargs).run()
+                    individual_run_output = _run_single_file_conversion(extracted_filename,
+                                                                        to_format,
+                                                                        *args,
+                                                                        from_format=from_format,
+                                                                        output_dir=output_dir,
+                                                                        log_file=individual_log_file,
+                                                                        log_mode=individual_log_mode,
+                                                                        max_file_size=remaining_file_size,
+                                                                        **converter_kwargs)
                 except base.FileConverterAbortException as e:
                     # If the run fails, create a run output object to indicate that
                     individual_run_output = base.FileConversionResult(log_filename=individual_log_file,
@@ -573,12 +584,13 @@ def run_converter(filename: str,
                 "input_filename": in_filename,
                 "output_filename": run_output.output_filename[l_index:r_index],
                 "input_size": input_size,
-                "output_size": output_size }
+                "output_size": output_size}
 
-            for key in [ "converter", "coordinates", "coordOption", "from_flags",
-                "to_flags", "from_arg_flags", "to_arg_flags" ]:
+            for key in ["converter", "coordinates", "coordOption", "from_flags",
+                        "to_flags", "from_arg_flags", "to_arg_flags"]:
                 if key in converter_kwargs['data'] and converter_kwargs['data'][key] != "" and not \
-                    ((key == "coordinates" or key == "coordOption") and converter_kwargs['data']['coordinates'] == "neither") :
+                        ((key == "coordinates" or key == "coordOption") and
+                         converter_kwargs['data']['coordinates'] == "neither"):
                     entry[key] = converter_kwargs['data'][key]
 
             entry["outcome"] = outcome
@@ -595,6 +607,7 @@ def run_converter(filename: str,
 
     return run_output
 
+
 def set_size_units(size):
     if size >= 1024:
         return str('%.3f' % (size / 1024)) + ' kB'

{psdi_data_conversion-0.1.6 → psdi_data_conversion-0.2.0}/psdi_data_conversion/converters/base.py

@@ -206,8 +206,8 @@ class FileConverter:
                  data: dict[str, Any] | None = None,
                  abort_callback: Callable[[int], None] = abort_raise,
                  use_envvars=False,
-                 upload_dir=const.DEFAULT_UPLOAD_DIR,
-                 download_dir=const.DEFAULT_DOWNLOAD_DIR,
+                 input_dir=const.DEFAULT_INPUT_DIR,
+                 output_dir=const.DEFAULT_OUTPUT_DIR,
                  max_file_size=None,
                  no_check=False,
                  log_file: str | None = None,
@@ -235,9 +235,9 @@ class FileConverter:
         use_envvars : bool
             If set to True, environment variables will be checked for any that set options for this class and used,
             default False
-        upload_dir : str
+        input_dir : str
             The location of input files relative to the current directory
-        download_dir : str
+        output_dir : str
             The location of output files relative to the current directory
         max_file_size : float
             The maximum allowed file size for input/output files, in MB. If 0, will be unlimited. Default 0 (unlimited)
@@ -296,8 +296,8 @@ class FileConverter:
             # Set member variables directly from input
             self.in_filename = filename
             self.to_format = to_format
-            self.upload_dir = upload_dir
-            self.download_dir = download_dir
+            self.input_dir = input_dir
+            self.output_dir = output_dir
             self.log_file = log_file
             self.log_mode = log_mode
             self.log_level = log_level
@@ -328,17 +328,22 @@ class FileConverter:
             self.err: str | None = None
             self.quality: str | None = None
 
-            # Create directory 'uploads' if not extant.
-            if not os.path.exists(self.upload_dir):
-                os.makedirs(self.upload_dir, exist_ok=True)
+            # Determine if the filename is fully-qualified, and if not, find it in the upload dir
+            if not os.path.exists(self.in_filename):
+                qualified_in_filename = os.path.join(self.input_dir, self.in_filename)
+                if os.path.exists(qualified_in_filename):
+                    self.in_filename = qualified_in_filename
+                else:
+                    FileConverterInputException(f"Input file {self.in_filename} not found, either absolute or relative "
+                                                f"to {self.input_dir}")
 
             # Create directory 'downloads' if not extant.
-            if not os.path.exists(self.download_dir):
-                os.makedirs(self.download_dir, exist_ok=True)
+            if not os.path.exists(self.output_dir):
+                os.makedirs(self.output_dir, exist_ok=True)
 
             self.local_filename = os.path.split(self.in_filename)[1]
             self.filename_base = os.path.splitext(self.local_filename)[0]
-            self.out_filename = f"{self.download_dir}/{self.filename_base}.{self.to_format_info.name}"
+            self.out_filename = f"{self.output_dir}/{self.filename_base}.{self.to_format_info.name}"
 
             # Set up files to log to
             self._setup_loggers()
@@ -441,7 +446,7 @@ class FileConverter:
         if self.log_mode == const.LOG_FULL_FORCE:
             self.output_log = self.log_file
         else:
-            self.output_log = os.path.join(self.download_dir, f"{self.filename_base}{const.OUTPUT_LOG_EXT}")
+            self.output_log = os.path.join(self.output_dir, f"{self.filename_base}{const.OUTPUT_LOG_EXT}")
 
         # If any previous log exists, delete it
         if os.path.exists(self.output_log):