psdi-data-conversion 0.0.36__tar.gz → 0.0.38__tar.gz

{psdi_data_conversion-0.0.36 → psdi_data_conversion-0.0.38}/CONTRIBUTING.md

@@ -297,6 +297,22 @@ For debugging python issues, it's recommended to install the package in editable
 pip install --editable .'[gui-test]'
 ```
 
+### Running Unit Tests
+
+The GUI unit tests require a web driver to be installed, and will install one if one isn't. To avoid repeated unnecessary API calls for the installation, you can set the environmental variable `DRIVER` when running tests to the location of your driver. This can be set to be automatically picked up by many IDEs by setting this in a `.env` file in the root of your project, e.g.:
+
+```
+DRIVER=~/.wdm/drivers/geckodriver/linux64/v0.36.0/geckodriver
+```
+
+To get the address of your driver, you can run the following in a Python terminal from within this projects virtual environment (after installing the `[gui-test]` optional dependencies):
+
+```python
+from webdriver_manager.firefox import GeckoDriverManager
+driver_path = GeckoDriverManager().install()
+print(f"Gecko driver installed to: {driver_path}")
+```
+
 ## Continuous Integration
 
 This project uses various GitHub workflows to perform Continuous Integration tasks. These can be found in the `.github/workflows` folder. The files which start with "ci-" are the directly-triggered workflows, and the files which start with "job-" are reusable workflows called by the former. These workflows handle the following tasks:
@@ -310,7 +326,7 @@ See the commons within the files for further details.
 
 ## Publishing
 
-The Python library, CLA, and local GUI are (planned to be) published as a Python package via PyPI. This section describes how the package is set up and how it's published.
+The Python library, CLA, and local GUI are published as a Python package via PyPI. This section describes how the package is set up and how it's published.
 
 ### Package Setup
 
@@ -318,11 +334,11 @@ The package's setup is defined in the `pyproject.toml` file. This defines the pr
 
 The package uses [Hatch](https://hatch.pypa.io/latest/) for its build backend, as it is simpler to configure than the classic [Setuptools](https://setuptools.pypa.io/en/latest/userguide/) and provides some useful extensibility.
 
-The version of the package is set to be determined from the version control system, meaning on the release branch, the version will always match the latest tag.
+The version of the package is set to be determined from the version control system, meaning on the release branch, the version will always match the latest tag. This alleviates us of having to manually maintain the version for the package to keep it correct, but does result in some quirks. It's a bit fussier to set up (though that's done now), and it makes the user take an extra step if they want to install from source but haven't cloned the repository - this is noted in the installation instructions in the README.
 
 ### Initial Publication
 
-This section details the plan for the initial publication of this package - after this is complete, this section will be left in for reference in case of future need.
+This section details the proceduce for the initial publication of this package - now that this is complete, this section is left in for reference in case of future need.
 
 First, it's necessary to install a couple required packages in order to build a Python package: `build` to build it and `twine` to upload it. These can be installed with pip via:
 
@@ -350,6 +366,14 @@ To upload, follow [this tutorial](https://packaging.python.org/en/latest/tutoria
 
 The `ci-release.yml` workflow is planned to publish any new releases to PyPI after the initial publication. This can be set up by uncommenting the relevant lines in this file, possibly updating the `job-publish-pypi.yml` to enable this (testing will be necessary), and setting up Trusted Publishing for the project on PyPI (see [guide](https://docs.github.com/en/actions/security-for-github-actions/security-hardening-your-deployments/configuring-openid-connect-in-pypi))
 
+### Project Management
+
+This project is published on PyPI at https://pypi.org/project/psdi-data-conversion/ and on TestPyPI at https://test.pypi.org/project/psdi-data-conversion/. Maintainers can manage the project through the "Manage" link on that page or from their own projects page.
+
+The most important setting to be aware of here is Publishing -> Trusted Publisher Management. This is the system used to allow automatic publishing of releases from GitHub. It's set up so that the current project, organisation, environment, and workflow for publishing are approved. If any of these change, this will need to be updated by adding a new trusted publisher with the new settings (on both PyPI and TestPyPI) and removing the old one.
+
+The management page can also be used to add or remove collaborators through the Collaborators tab. Generally the project on these sites doesn't require much maintenance, but at least a few active collaborators should be on it at all times to avoid getting locked out if someone is suddenly unavailable.
+
 ## Deployment
 
 Deployment is handled by the `job-deploy-k8s.yml` reusable workflow, which is triggered from `ci-main.yml` to deploy to dev on each push to `main` and `ci-release.yml` to deploy to staging on each push to `release`. When a push to production is desired, it must be triggered manually by calling the `ci-deploy-production.yml` workflow on the `release` branch (it's set up to fail if run on any other branch).

{psdi_data_conversion-0.0.36 → psdi_data_conversion-0.0.38}/PKG-INFO

@@ -1,13 +1,11 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: psdi_data_conversion
-Version: 0.0.36
+Version: 0.0.38
 Summary: Chemistry file format conversion service, provided by PSDI
 Project-URL: Homepage, https://data-conversion.psdi.ac.uk/
 Project-URL: Documentation, https://psdi-uk.github.io/psdi-data-conversion/
-Project-URL: Repository, https://github.com/PSDI-UK/psdi-data-conversion
 Project-URL: Issues, https://github.com/PSDI-UK/psdi-data-conversion/issues
 Project-URL: Changelog, https://github.com/PSDI-UK/psdi-data-conversion/blob/main/CHANGELOG.md
-Project-URL: Download, https://github.com/PSDI-UK/psdi-data-conversion/releases/latest
 Author: Ray Whorley, Don Cruickshank, Tom Underwood
 Author-email: Samantha Pearman-Kanza <s.pearman-kanza@soton.ac.uk>, Bryan Gillis <7204836+brgillis@users.noreply.github.com>
 License:                                  Apache License
@@ -245,7 +243,7 @@ Description-Content-Type: text/markdown
 
 # PSDI Data Conversion
 
-Version: Pre-release 2024-02-27
+Version: Pre-release 2024-04-14
 
 This is the repository for the PSDI PF2 Chemistry File Format Conversion project. The goal of this project is to provide utilities to assist in converting files between the many different file formats used in chemistry, providing information on what converters are available for a given conversion and the expected quality of it, and providing multiple interfaces to perform these conversions. These interfaces are:
 
@@ -278,7 +276,14 @@ This is the repository for the PSDI PF2 Chemistry File Format Conversion project
   - [Installation and Setup](#installation-and-setup)
   - [Running the App](#running-the-app)
 - [Testing](#testing)
-- [Licencing](#licencing)
+- [Troubleshooting](#troubleshooting)
+  - [OSError: [Errno 24] Too many open files](#oserror-errno-24-too-many-open-files)
+  - [Errors running c2x or Atomsk converters](#errors-running-c2x-or-atomsk-converters)
+  - [A supported conversion fails](#a-supported-conversion-fails)
+    - [Input file is malformatted or corrupt](#input-file-is-malformatted-or-corrupt)
+    - [Input file's format is misidentified](#input-files-format-is-misidentified)
+    - [Other known issues](#other-known-issues)
+- [Licensing](#licensing)
 - [Contributors](#contributors)
 - [Funding](#funding)
 
@@ -312,15 +317,17 @@ This is the repository for the PSDI PF2 Chemistry File Format Conversion project
 - `test_data`
   - (Files used for testing the project)
 - `tests`
-  - (Unit tests for the project)
+  - `gui`
+    - (Unit tests for the GUI, aka the local version of the web app)
+  - `python`
+    - (Unit tests for the Python library and command-line application)
 - `CHANGELOG.md` (Updates since initial public release)
 - `CONTRIBUTING.md` (Guidelines and information for contributors to the project)
 - `DOCKERFILE` (Dockerfile for image containerising PSDI's data conversion service)
-- `LICENSE` (Apache Licence version 2.0)
+- `LICENSE` (Apache License version 2.0)
 - `pyproject.toml` (Python project metadata and settings)
 - `README.md` (This file)
 - `requirements.txt` (Requirements for the web app deployment of this project)
-- `run_local.sh` (Helper script to run the web app locally)
 
 ## Requirements
 
@@ -468,6 +475,24 @@ options] [-s/--strict] [--nc/--no-check] [-q/--quiet] [-g/--log-file <log file n
 
 Call `psdi-data-convert -h` for details on each of these options.
 
+Note that some requested conversions may involve ambiguous formats which share the same extension. In this case, the application will print a warning and list possible matching formats, with a disambiguating name that can be used to specify which one. For instance, the `c2x` converter can convert into two variants of the `pdb` format, and if you ask it to convert to `pdb` without specifying which one, you'll see:
+
+```
+ERROR: Extension 'pdb' is ambiguous and must be defined by ID. Possible formats and their IDs are:
+9: pdb-0 (Protein Data Bank)
+259: pdb-1 (Protein Data Bank with atoms numbered)
+```
+
+This provides the IDs ("9" and "259") and disambiguating names ("pdb-0" and "pdb-1") for the matching formats. Either can be used in the call to the converter, e.g.:
+
+```bash
+psdi-data-conversion nacl.cif -t 9 -w c2x
+# Or equivalently:
+psdi-data-conversion nacl.cif -t pdb-0 -w c2x
+```
+
+The "<format>-0" pattern can be used with any format, even if it's unambiguous, and will be interpreted as the first instance of the format in the database with valid conversions. Note that as the database expands in future versions and more valid conversions are added, these disambiguated names may change, so it is recommended to use the format's ID in scripts and with the library to ensure consistency between versions of this package.
+
 #### Requesting Information on Possible Conversions
 
 The script can also be used to get information on possible conversions by providing the `-l/--list` argument:
@@ -542,6 +567,8 @@ Where `filename` is the name of the file to convert (either fully-qualified or r
 
 See the method's documentation via `help(run_converter)` after importing it for further details on usage.
 
+Note that as with running the application through the command-line, some extra care may be needed in the case that the input or output format is ambiguous - see the [Data Conversion](#data-conversion) section above for more details on this. As with running through the command-line, a format's ID or disambiguated name must be used in the case of ambiguity.
+
 #### `constants`
 
 This package defines most constants used in the package. It may be imported via:
@@ -596,54 +623,15 @@ pip install .'[gui]'
 
 If your system does not allow installation in this manner, it may be necessary to set up a virtual environment. See the instructions in the [command-line application installation](#installation) section above for how to do that, and then try to install again once you've set one up and activated it.
 
-If you've installed this repository from source, you can use the provided `run_local.sh` bash script to run the application. Otherwise (e.g. if you've installed from a wheel or PyPI), copy and paste the following into a script:
-
-```bash
-#!/bin/bash
-
-# The envvar MAX_FILESIZE can be used to set the maximum allowed filesize in MB - 0 indicates no maximum
-if [ -z $MAX_FILESIZE ]; then
-  export MAX_FILESIZE=0
-fi
-
-# The envvar MAX_FILESIZE_OB can be used to set the maximum allowed filesize in MB for the Open Babel converter - 0
-# indicates no maximum. This is currently set to 1 MB by default as the converter seems to hang above this limit (not
-# even allowing the process to be cancelled). This can be changed in the future if this is patched
-if [ -z $MAX_FILESIZE_OB ]; then
-  export MAX_FILESIZE_OB=1
-fi
-
-# Logging control - "full" sets server-style logging, which is necessary to produce the output logs with the expected
-# names. This should not be changed, or else errors will occur
-export LOG_MODE=full
-
-# The level to log at. Leave blank for defaults, which logs at INFO level for user output and ERROR level for the server
-# log and stdout. If set to a different value here (e.g. DEBUG), all these channels will be set to that level
-export LOG_LEVEL=
-
-# The envvar SERVICE_MODE can be set to "true" to make this behave as if it's running as the public web service -
-# uncomment the following line to enable that
-# export SERVICE_MODE=true
-
-# Uncomment the following line to enable debug mode
-# export FLASK_ENV=development
-
-# Execute a local run of the application from the proper path
-
-PACKAGE_PATH=`python -c "import psdi_data_conversion; print(psdi_data_conversion.__path__[0])"`
-cd $PACKAGE_PATH/..
-python -m flask --app psdi_data_conversion/app.py run
-```
-
-If desired, you can modify the environmental variables set in this script to modify the operation - see the comments on each for details.
-
 ### Running the App
 
-Run the `run_local.sh` script to start the server. You can then access the website by going to <http://127.0.0.1:5000> in a browser (this will also be printed in the terminal, and you can CTRL+click it there to open it in your default browser). Guidance for using the app is given on each page of it.
+Once installed, the command-line script `psdi-data-convert-gui` will be made available, which can be called to start the server. You can then access the website by going to <http://127.0.0.1:5000> in a browser (this will also be printed in the terminal, and you can CTRL+click it there to open it in your default browser). Guidance for using the app is given on each page of it. When you're finished with the app, key CTRL+C in the terminal where you called the script to shut down the server, or, if the process was backgrounded, kill the appropriate process.
 
 In case of problems when using Chrome, try opening Chrome from the command line:
 open -a "Google Chrome.app" --args --allow-file-access-from-files
 
+The local version has some customisable options for running it, which can can be seen by running `psdi-data-convert-gui --help`. Most of these are only useful for development, but one notable setting is `--max-file-size-ob`, which sets the maximum allowed filesize for conversions with the Open Babel converter in megabytes. This is set to 1 MB by default, since Open Babel has a known bug which causes it to hang indefinitely for some conversions over this size (such as from large `mmcif` files). This can be set to a higher value (or to 0 to disable the limit) if the user wishes to disable this safeguard.
+
 ## Extending Functionality
 
 The Python library and CLA are written to make it easy to extend the functionality of this package to use other file format converters. This can be done by downloading or cloning the project's source from it's GitHub Repository (https://github.com/PSDI-UK/psdi-data-conversion), editing the code to add your converter following the guidance in the "[Adding File Format Converters](https://github.com/PSDI-UK/psdi-data-conversion/blob/main/CONTRIBUTING.md#adding-file-format-converters)" section of CONTRIBUTING.md to integrate it with the Python code, and installing the modified package on your system via:
@@ -662,28 +650,140 @@ To test the CLA and Python library, install the optional testing requirements lo
 
 ```bash
 pip install .'[test]'
-pytest
+pytest tests/python
 ```
 
 To test the local version of the web app, install the GUI testing requirements locally (which also include the standard GUI requirements and standard testing requirements), start the server, and test by executing the GUI test script:
 
 ```bash
 pip install .'[gui-test]'
-./run_local.sh & # Start the server for the web app in the background
-cd tests/selenium
-./run.sh
-kill %1 # Stop the web server - it may have a different job ID. If you don't know the job ID, you can alternatively call "fg" to bring the job to the foreground, then type CTRL+c to stop it
+pytest tests/gui
+```
+
+Both of these sets of tests can also be run together if desired through:
+
+```bash
+pip install .'[gui-test]'
+pytest
+```
+
+## Troubleshooting
+
+This section presents solutions for commonly-encountered issues.
+
+### OSError: [Errno 24] Too many open files
+
+You may see the error:
+
+```
+OSError: [Errno 24] Too many open files
 ```
 
-## Licencing
+while running the command-line application, using the Python library, or running tests This error is caused by a program hitting the limit of the number of open filehandles allowed by the OS. This limit is typically set to 1024 on Linux systems and 256 on MacOS systems, and thus this issue occurs much more often on the latter. You can see what your current limit is by running the command:
+
+```bash
+ulimit -a | grep "open files"
+```
+
+This limit can be temporarily increased for the current terminal session by running the command:
+
+```bash
+ulimit -n 1024 # Or another, higher number
+```
+
+First, try increasing the limit and then redo the operation which caused this error to see if this resolves it. If this does, you can make this change permanent in a few ways, the easiest of which is to add this command to your `.bashrc` file so it will be set for every new terminal session.
+
+If you see this error when the filehandle limit is already set to a high value such as 1024, this may indicate the presence of a bug in the project which causes a leaking of filehandles, so please open an issue about it, pasting the error you get and the details of your system, particularly including your current filehandle limit.
+
+### Errors running c2x or Atomsk converters
+
+We provide support for the c2x and Atomsk converters by packing binaries which support common Linux and MacOS platforms with this project, but we cannot guarantee universal support for these binaries. In particular, they may rely on dynamically-linked libraries which aren't installed on your system.
+
+Look through the error message you received for messages such as "Library not loaded" or "no such file", and see if they point to the name of a library which you can try installing. For instance, if you see that it's searching for `libquadmath.0.dylib` but not finding it, you can try installing this library. In this case, this library can be installed through apt with:
+
+```bash
+sudo apt install libquadmath0
+```
+
+or through brew via:
+
+```bash
+brew install gcc
+```
+
+Alternatively, you can run your own versions of the `c2x` and `atomsk` binaries with this project. Compile them yourself however you wish - see the projects at https://github.com/codenrx/c2x and https://github.com/pierrehirel/atomsk and follow their instructions to build a binary on your system. Once you've done so, add the binary to your `$PATH`, and this project will pick that up and use it in preference to the prepackaged binary.
+
+On the other hand, it's possible that an error of this sort will occur if you have a non-working binary of one of these converters in your `$PATH`. If this might be the case, you can try removing it and see if the prepackaged binary works for you, or else recompile it to try to fix errors.
+
+### A supported conversion fails
+
+Here we'll go over some of the most common reasons that a supported conversion might fail, and what can be done to fix the issue.
+
+#### Input file is malformatted or corrupt
+
+Usually if there is a problem with the input file, the error message you receive should indicate some difficulty reading it. If the error message indicates this might be the issue, try the following:
+
+Check the validity of the input file, ideally using another tool which can read in a file of its type, and confirm that it can be read successfully. This doesn't guarantee that the file is valid, as some utilities are tolerant to some formatting errors, but if you get an error here, then you know the issue is with the file. If the file can be read by another utility, see if the conversion you're attempting is supported by another converter - it might be that the file has a negligible issue that another converter is able to ignore.
+
+If you've confirmed that the input file is malformatted or corrupt, see if it's possible to regenerate it or fix it manually. There may be a bug in the program which generated it - if this is under your control, check the format's documentation to help fix it. Otherwise, you can see if you can use the format's documentation as a guide to manually fix the file.
+
+#### Input file's format is misidentified
+
+If you've followed the steps in the previous section and confirmed that the input file is valid, but you're still having issues with it, one possibility is that this application is misidentifying the file's format. This can happen if you've given the file an extension which isn't expected of its format, or in rare cases where an extension is shared by multiple formats.
+
+To remedy this, try explicitly specifying the format, rather than letting the application guess it from the extension. You can see all supported formats by running `psdi-data-convert -l`, and then get details on one with `psdi-data-convert -l -f <format>` to confirm that it's the correct format. You can then call the conversion script with the argument `-f <format>`, or within Python make a call to the library with `run_converter(..., from_format=<format>)` to specify the format.
+
+`<format>` here can be the standard extension of the format (in the case of unambiguous extensions), its ID, or its disambiguated name. To give an example which explains what each of these are, let's say you have an MDL MOL file you wish to convert to XYZ, so you get information about it and possible converters with `psdi-data-convert -l -f mol -t xyz`:
+
+```base
+$ psdi-data-convert -l -f mol
+WARNING: Format 'mol' is ambiguous and could refer to multiple formats. It may be necessary to explicitly specify which
+you want to use when calling this script, e.g. with '-f mol-0' - see the disambiguated names in the list below:
+
+18: mol-0 (MDL MOL)
+216: mol-1 (MOLDY)
+
+20: xyz (XYZ cartesian coordinates)
+
+The following registered converters can convert from mol-0 to xyz:
+
+    Open Babel
+    c2x
+
+For details on input/output flags and options allowed by a converter for this conversion, call:
+psdi-data-convert -l <converter name> -f mol-0 -t xyz
+
+The following registered converters can convert from mol-1 to xyz:
+
+    Atomsk
+
+For details on input/output flags and options allowed by a converter for this conversion, call:
+psdi-data-convert -l <converter name> -f mol-1 -t xyz
+```
+
+This output indicates that the application is aware of two formats which share the `mol` extension: MDL MOL and MOLDY. It lists the ID, disambiguated name, and description of each: ID `18` and disambiguated name `mol-0` for MDL MOL, and ID `216` and disambiguated name `mol-1` for MOLDY. The XYZ format, on the other hand, is unambiguous, and only lists the standard extension for it as its disambiguated name (although `xyz-0` will be accepted without error as well).
+
+The program then lists converters which can handle the requested conversion, revealing a potential pitfall: The Open Babel and c2x converters can convert from MDL MOL to XYZ, which the Atomsk converter can convert from MOLDY to XYZ. If you don't specify which format you're converting from, the script might assume you meant to use the other one, if that's the only one compatible with the converter you've requested (or with the default converter, Open Babel, if you didn't explicitly request one). So to be careful here, it's best to specify this input format unambiguously.
+
+Since in this example you have an MDL MOL file, you would use `-f 18` or `-f mol-0` to explicitly specify it in the command-line, or similarly provide one of these to the `from_format` argument of `run_converter` within Python. The application will then properly handle it, including alerting you if you request a conversion that isn't supported by your requested converter (e.g. if you request a conversion of this MDL MOL file to XYZ with Atomsk).
+
+Important note: The disambiguated name is generated dynamically and isn't stored in the database, and in rare cases may change for some formats in future versions of this application which expand support to more formats and conversions. For uses which require forward-compatibility with future versions of this application, the ID should be used instead.
+
+#### Other known issues
+
+Through testing, we've identified some other conversion issues, which we list here:
+
+- Open Babel will indefinitely hang when attempting to convert large files (more than ~1 MB) of certain types (such as `mmcif`). This is an issue with the converter itself and not our application, which we hope will be fixed in a future version. If this occurs, the job will have to be forcibly terminated. CTRL+C will fail to terminate it, but it can be stopped with CTRL+Z, then terminated with `kill %N`, where N is the number listed beside the job when it is stopped (usually 1). The conversion should then be attempted with another supported converter.
+
+## Licensing
 
 This project is provided under the Apache License version 2.0, the terms of which can be found in the file `LICENSE`.
 
-This project redistributes compiled binaries for the Atomsk and c2x converters. These are both licenced under the
+This project redistributes compiled binaries for the Atomsk and c2x converters. These are both licensed under the
 GNU General Public License version 3 and are redistributed per its terms. Any further redistribution of these binaries,
-including redistribution of this project as a whole, including them, must also follow the terms of this licence.
+including redistribution of this project as a whole, including them, must also follow the terms of this license.
 
-This requires conspicuously displaying notice of this licence, providing the text of of the licence (provided here in
+This requires conspicuously displaying notice of this license, providing the text of of the license (provided here in
 the files `psdi_data_conversion/bin/LICENSE_C2X` and `psdi_data_conversion/bin/LICENSE_ATOMSK`), and appropriately
 conveying the source code for each of these. Their respective source code may be found at: