cci-os-worker 0.5.0__tar.gz

cci_os_worker-0.5.0/LICENSE

@@ -0,0 +1,31 @@
+BSD 3-Clause License
+
+Copyright (c) 2023-2024, Centre of Environmental Data Analysis Developers,
+Scientific and Technical Facilities Council (STFC),
+UK Research and Innovation (UKRI).
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without 
+modification, are permitted provided that the following conditions are met:
+
+1. Redistributions of source code must retain the above copyright notice, this 
+list of conditions and the following disclaimer.
+
+2. Redistributions in binary form must reproduce the above copyright notice, this 
+list of conditions and the following disclaimer in the documentation and/or other 
+materials provided with the distribution.
+
+3. Neither the name of the copyright holder nor the names of its contributors 
+may be used to endorse or promote products derived from this software without 
+specific prior written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS”
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE 
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE 
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE 
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL 
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR 
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER 
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE 
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

cci_os_worker-0.5.0/PKG-INFO

@@ -0,0 +1,122 @@
+Metadata-Version: 2.3
+Name: cci-os-worker
+Version: 0.5.0
+Summary: Worker repo to execute the steps for CCI Opensearch Ingestion, namely facet and tag scanning and updates to the CEDA FBI
+License: BSD 3 - See License file
+Author: Daniel Westwood
+Author-email: daniel.westwood@stfc.ac.uk
+Requires-Python: >=3.9,<4
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python
+Classifier: Programming Language :: Python :: 3.8
+Requires-Dist: aiofiles (>=24.1.0,<25.0.0)
+Requires-Dist: cci-facet-scanner (>=0.3.1,<0.4.0)
+Requires-Dist: elasticsearch (>=7,<8)
+Requires-Dist: ldap3 (>=2.9,<3.0)
+Requires-Dist: python-dateutil (>=2.9.0.post0,<3.0.0)
+Requires-Dist: python-magic (>=0.4.22,<0.5.0)
+Requires-Dist: pyyaml (>=6.0.2,<7.0.0)
+Description-Content-Type: text/markdown
+
+# CCI Opensearch Worker repository
+
+![Static Badge](https://img.shields.io/badge/cci%20tagger%20workflow-8AD6F6)
+![GitHub Release](https://img.shields.io/github/v/release/cedadev/cci-os-worker)
+[![PyPI version](https://badge.fury.io/py/cci-os-worker.svg)](https://pypi.python.org/pypi/cci-os-worker/)
+
+
+**See release notes for change history**
+
+This package serves as a wrapper for the CCI Opensearch Workflow, which involves several independent packages with multiple dependencies. Primarily the CCI Tagger (cci-tag-scanner) and Facet scanner (cci-facet-scanner) are combined, with elements from the CEDA FBS (ceda-fbs-cci) package to create the components for Opensearch records in Elasticsearch.
+
+**NOTE:** When publishing a new tagged release of this package, please make sure to rebuild the corresponding Docker image, in the CEDA gitlab repository `cci_opensearch_base`. This repository has a single build-image step that should be rerun (following the steps found there) to ensure any changes to this package are picked up by the OS worker deployment.
+
+![CCI Opensearch Workflow](https://github.com/cedadev/cci-os-worker/blob/main/images/CCI_Workflow.png)
+
+## 1. Installation
+
+This package can be cloned directly or used as a dependency in a pyproject file.
+
+Set up a python virtual environment:
+```
+ $ python -m venv .venv
+ $ source .venv/bin/activate
+ $ pip install cci-os-worker
+```
+
+NOTE: As of 22nd Jan 2025 the `cci-os-worker` repository has been upgraded for use with Poetry version 2. The temporary solution to use a `requirements_fix.txt` file has been removed as this package is now on Pypi.
+
+### 1.1. Use in other packages
+
+**Poetry 1.8.5 and older**
+For use in another package as a dependency, use the following in your pyproject `[tool.poetry.dependencies]`:
+```
+cci-os-worker = { git = "https://github.com/cedadev/cci-os-worker.git", tag="v0.3.1"}
+```
+
+**Poetry 2.0.1 and later**
+This package is now a pip-installable published package as of 11th April 2025! That means for packages using Poetry 2 or higher, the cci-os-worker can be added via Poetry at version 0.5.0 or higher.
+
+```
+poetry add cci-os-worker^0.5.0
+```
+
+## 2. Usage
+
+## 2.1 Find datasets
+
+Determining the set of files to operate over can be done in two ways using built-in scripts here, or indeed by any other means. If the intention is to submit to a rabbit queue however, this script is required with the additional `-R` parameter to submit to a queue, and the configuration for the queue given by a yaml file provided as `--conf`.
+
+```
+rescan_dir path/to/json/directory/ --extension nc -l 1 -o path/to/dataset/filelist.txt
+```
+
+**NOTE**: As of v0.5.0 this changed from `fbi_rescan_dir` to simply `rescan_dir`.
+
+In the above command:
+ - `r` represents a recursive look through identified directories.
+ - `l` means the scan level. Scan level 1 will involve finding all the JSON files and expanding each `datasets` path into a list.
+ - `o` is the output file to send the list of datasets.
+ - `--extension` applies to the files identified and added to the output file. `nc` is the default value so is redundant here.
+ - `--file-regex` alternative to supplying just the extension, if a valid regex pattern can be matched to identify specific files it can be submitted here.
+
+This command can also be run for a known directory to expand into a list of datasets:
+
+```
+rescan_dir my/datasets/path/ -l 2 -o path/to/dataset/filelist.txt
+```
+
+In this case we specify `l` as 2 since there are no JSON files involved. The extension/file_regex options can also be added here, but as the `nc` option is a default value we have omitted it here.
+
+## 2.2 Run the facet scan workflow
+
+The facet scanner workflow utilises both the facet and tag scanners to produce the set of facets under `project.opensearch` in the resulting opensearch records. This workflow can be run using the `facetscan` entrypoint script installed with this package.
+
+The environment variable `JSON_TAGGER_ROOT` should be set, which should be the path to the top-level directory under which all JSON files are placed. These JSON files provide defaults and mappings to values placed in the opensearch records - supplementary material to aid facet scanning or replace found values.
+
+As of v0.5.0 the two workflows (facet and FBI) have been combined into one singular workflow to generate all portions of the Opensearch records. This can be run with the following command:
+
+```
+ $ cci_os_update path/to/dataset/filelist.txt path/to/config/file.yaml
+```
+(Note: Verbose flag -v can be added to the above command.)
+
+Where the yaml file should look something like this:
+
+```
+elasticsearch:
+  # Fill in with key value
+  x-api-key: ""
+facet_files_index:
+  name: facet-index-staging
+facet_files_test_index:
+  name: facet-index-staging
+ldap_configuration:
+  hosts:
+    - ldap://homer.esc.rl.ac.uk
+    - ldap://marge.esc.rl.ac.uk
+```
+
+

cci_os_worker-0.5.0/README.md

@@ -0,0 +1,99 @@
+# CCI Opensearch Worker repository
+
+![Static Badge](https://img.shields.io/badge/cci%20tagger%20workflow-8AD6F6)
+![GitHub Release](https://img.shields.io/github/v/release/cedadev/cci-os-worker)
+[![PyPI version](https://badge.fury.io/py/cci-os-worker.svg)](https://pypi.python.org/pypi/cci-os-worker/)
+
+
+**See release notes for change history**
+
+This package serves as a wrapper for the CCI Opensearch Workflow, which involves several independent packages with multiple dependencies. Primarily the CCI Tagger (cci-tag-scanner) and Facet scanner (cci-facet-scanner) are combined, with elements from the CEDA FBS (ceda-fbs-cci) package to create the components for Opensearch records in Elasticsearch.
+
+**NOTE:** When publishing a new tagged release of this package, please make sure to rebuild the corresponding Docker image, in the CEDA gitlab repository `cci_opensearch_base`. This repository has a single build-image step that should be rerun (following the steps found there) to ensure any changes to this package are picked up by the OS worker deployment.
+
+![CCI Opensearch Workflow](https://github.com/cedadev/cci-os-worker/blob/main/images/CCI_Workflow.png)
+
+## 1. Installation
+
+This package can be cloned directly or used as a dependency in a pyproject file.
+
+Set up a python virtual environment:
+```
+ $ python -m venv .venv
+ $ source .venv/bin/activate
+ $ pip install cci-os-worker
+```
+
+NOTE: As of 22nd Jan 2025 the `cci-os-worker` repository has been upgraded for use with Poetry version 2. The temporary solution to use a `requirements_fix.txt` file has been removed as this package is now on Pypi.
+
+### 1.1. Use in other packages
+
+**Poetry 1.8.5 and older**
+For use in another package as a dependency, use the following in your pyproject `[tool.poetry.dependencies]`:
+```
+cci-os-worker = { git = "https://github.com/cedadev/cci-os-worker.git", tag="v0.3.1"}
+```
+
+**Poetry 2.0.1 and later**
+This package is now a pip-installable published package as of 11th April 2025! That means for packages using Poetry 2 or higher, the cci-os-worker can be added via Poetry at version 0.5.0 or higher.
+
+```
+poetry add cci-os-worker^0.5.0
+```
+
+## 2. Usage
+
+## 2.1 Find datasets
+
+Determining the set of files to operate over can be done in two ways using built-in scripts here, or indeed by any other means. If the intention is to submit to a rabbit queue however, this script is required with the additional `-R` parameter to submit to a queue, and the configuration for the queue given by a yaml file provided as `--conf`.
+
+```
+rescan_dir path/to/json/directory/ --extension nc -l 1 -o path/to/dataset/filelist.txt
+```
+
+**NOTE**: As of v0.5.0 this changed from `fbi_rescan_dir` to simply `rescan_dir`.
+
+In the above command:
+ - `r` represents a recursive look through identified directories.
+ - `l` means the scan level. Scan level 1 will involve finding all the JSON files and expanding each `datasets` path into a list.
+ - `o` is the output file to send the list of datasets.
+ - `--extension` applies to the files identified and added to the output file. `nc` is the default value so is redundant here.
+ - `--file-regex` alternative to supplying just the extension, if a valid regex pattern can be matched to identify specific files it can be submitted here.
+
+This command can also be run for a known directory to expand into a list of datasets:
+
+```
+rescan_dir my/datasets/path/ -l 2 -o path/to/dataset/filelist.txt
+```
+
+In this case we specify `l` as 2 since there are no JSON files involved. The extension/file_regex options can also be added here, but as the `nc` option is a default value we have omitted it here.
+
+## 2.2 Run the facet scan workflow
+
+The facet scanner workflow utilises both the facet and tag scanners to produce the set of facets under `project.opensearch` in the resulting opensearch records. This workflow can be run using the `facetscan` entrypoint script installed with this package.
+
+The environment variable `JSON_TAGGER_ROOT` should be set, which should be the path to the top-level directory under which all JSON files are placed. These JSON files provide defaults and mappings to values placed in the opensearch records - supplementary material to aid facet scanning or replace found values.
+
+As of v0.5.0 the two workflows (facet and FBI) have been combined into one singular workflow to generate all portions of the Opensearch records. This can be run with the following command:
+
+```
+ $ cci_os_update path/to/dataset/filelist.txt path/to/config/file.yaml
+```
+(Note: Verbose flag -v can be added to the above command.)
+
+Where the yaml file should look something like this:
+
+```
+elasticsearch:
+  # Fill in with key value
+  x-api-key: ""
+facet_files_index:
+  name: facet-index-staging
+facet_files_test_index:
+  name: facet-index-staging
+ldap_configuration:
+  hosts:
+    - ldap://homer.esc.rl.ac.uk
+    - ldap://marge.esc.rl.ac.uk
+```
+

cci_os_worker-0.5.0/cci_os_worker/README.md

@@ -0,0 +1,32 @@
+# CCI OS Worker Source
+
+This package has now combined elements from the following packages, which have been dropped as dependencies:
+ - CEDA FBS
+ - Nappy
+
+The source code provided here should accomplish all tasks related to these dependency packages. Contact [Daniel Westwood](daniel.westwood@stfc.ac.uk) for more details
+
+## Entrypoint Scripts
+It is recommended to use the entrypoint scripts to interface with this package once installed:
+ - facetscan
+ - fbi_update
+
+NOTE: In a future update, these two scripts will be combined into a single workflow which generates the entire record for one dataset as a single process, rather than generating half the record in each case.
+
+### Facetscan
+
+Use this entrypoint to perform a facet scan given a set of dataset files and the worker config. This is automatically configured to create records in the elasticsearch index given by `facet_files_index` in the worker config.
+ - `datafile_path`: A path to a `datasets.txt` list of files.
+ - `conf`: Path to the worker config file.
+ - [OPTIONAL] `output`: Outputs failed datasets to a text file.
+
+Use `facetscan --help` for more information.
+
+### FBI Update
+
+This entrypoint completes the second part of the Opensearch record for each dataset, with the same configuration options as above:
+ - `datafile_path`: A path to a `datasets.txt` list of files.
+ - `conf`: Path to the worker config file.
+ - [OPTIONAL] `output`: Outputs failed datasets to a text file.
+
+Use `fbi_update --help` for more information.

cci_os_worker-0.5.0/cci_os_worker/__init__.py

@@ -0,0 +1,8 @@
+# Logger setup
+import logging
+
+logging.basicConfig(level=logging.DEBUG)
+logstream = logging.StreamHandler()
+
+formatter = logging.Formatter('%(levelname)s [%(name)s]: %(message)s')
+logstream.setFormatter(formatter)

cci_os_worker-0.5.0/cci_os_worker/all_facets.py

@@ -0,0 +1,211 @@
+# encoding: utf-8
+__author__ = 'Daniel Westwood'
+__date__ = '05 Nov 2024'
+__copyright__ = 'Copyright 2024 United Kingdom Research and Innovation'
+__license__ = 'BSD - see LICENSE file in top-level package directory'
+__contact__ = 'daniel.westwood@stfc.ac.uk'
+
+# Opens the file provided in the datasets folder.
+# Iterate over all provided dataset paths.
+
+# facet_scanner.get_handler('filepath')
+# facets = handler.get_facets('filepath')
+
+import logging
+import hashlib
+import argparse
+import os
+
+import asyncio
+
+from cci_facet_scanner.core.facet_scanner import FacetScanner
+from elasticsearch import Elasticsearch
+
+from cci_os_worker.filehandlers.util import LDAPIdentifier
+from cci_os_worker.filehandlers import NetCdfFile, GenericFile
+from cci_os_worker import logstream
+
+from .path_tools import PathTools
+from .directory import check_timeout
+from .utils import load_config, UpdateHandler, set_verbose
+from .errors import HandlerError, DocMetadataError
+
+logger = logging.getLogger(__name__)
+logger.addHandler(logstream)
+logger.propagate = False
+
+class FacetUpdateHandler(UpdateHandler):
+
+    def __init__(self, conf: dict, dryrun: bool = False, test: bool = False):
+        """
+        Initialise this class with the correct connections to 
+        establish an elasticsearch client.
+        """
+        logger.info('Loading Facet Updater')
+
+        super().__init__(conf, dryrun=dryrun, test=test)
+
+        self.facet_scanner = FacetScanner()
+
+        ldap_hosts = self._conf['ldap_configuration']['hosts']
+        self.ldap_interface = LDAPIdentifier(server=ldap_hosts, auto_bind=True)
+
+        self._spot_file = conf.get('spot_file',None)
+        self.pt = PathTools(spot_file=self._spot_file)
+
+    def _get_project_info(self, path):
+        """
+        Get project info for a specific path
+        """
+
+        extension = os.path.splitext(path.split('/')[-1])[-1]
+        extension = extension.lower()
+
+        if extension == '.nc':
+            handler = NetCdfFile
+        else:
+            handler = GenericFile
+
+        calculate_md5 = self._conf.get('calculate_md5',False)
+
+        if handler is None:
+            raise HandlerError(filename=path)
+
+        handler_instance = handler(path, 3, calculate_md5=calculate_md5)
+
+        # FutureDetail: Remove manifest from 'doc' if unneeded (no indexing required.)
+        doc, phenomena, spatial = handler_instance.get_metadata()
+
+        if doc is None:
+            raise DocMetadataError(filename=path)
+        if len(doc) > 1:
+            doc = doc[0]
+
+        if phenomena:
+            doc['info']['phenomena'] = phenomena
+        if spatial:
+            doc['info']['spatial'] = spatial
+
+        spot = self.pt.spots.get_spot(path)
+
+        if spot is not None:
+            doc['info']['spot_name'] = spot
+
+        # Replace the UID and GID with name and group
+        uid = doc['info']['user']
+        gid = doc['info']['group']
+
+        doc['info']['user'] = self.ldap_interface.get_user(uid)
+        doc['info']['group'] = self.ldap_interface.get_group(gid)
+
+        return doc['info']
+
+    def _single_process_file(self, filepath: str, index: int = None, total: int = None):
+        """
+        Perform facet scanning for a specific filepath
+        """
+
+        logger.info('--------------------------------')
+        if index is None:
+            logger.info(f'Processing {filepath.split("/")[-1]}')
+        else:
+            logger.info(f'Processing {filepath.split("/")[-1]} ({index}/{total})')
+
+        # Get the handler for this filepath
+        handler = self.facet_scanner.get_handler(filepath, json_files=None)
+
+        # Extract the facets
+        facets = handler.get_facets(filepath)
+
+        # Build the project dictionary using the handlers project name attr
+        project = {
+            'info': self._get_project_info(filepath),
+            'projects': {
+                handler.project_name: facets
+            }
+        }
+
+        if self._test:
+            index = self._conf['facet_files_test_index']['name']
+        else:
+            index = self._conf['facet_files_index']['name']
+
+        id = hashlib.sha1(filepath.encode(errors="ignore")).hexdigest()
+
+        # Send facets to elasticsearch
+        if not self._dryrun:
+            self.es.update(
+                index=index,
+                id=id,
+                body={'doc': project, 'doc_as_upsert': True}
+            )
+        else:
+            logger.info(f'DRYRUN: Skipped updating for {filepath.split("/")[-1]}')
+
+            self._local_cache(
+                filename=f'cache/{filepath.split("/")[-1]}-cache.json',
+                contents=project,
+            )
+
+def _get_command_line_args():
+    """
+    Get the command line arguments for the facet scan
+    """
+    parser = argparse.ArgumentParser(description='Entrypoint for the CCI OS Worker on the CMD Line')
+    parser.add_argument('datafile_path', type=str, help='Path to the "datasets.txt" file')
+    parser.add_argument('conf', type=str, help='Path to Yaml config file for Elasticsearch')
+
+    parser.add_argument('-d','--dryrun', dest='dryrun', action='store_true', help='Perform in dryrun mode')
+    parser.add_argument('-t','--test', dest='test', action='store_true', help='Perform in test/staging mode')
+    parser.add_argument('-p','--prefix', dest='prefix', default='', help='Prefix to apply to all filenames')
+    parser.add_argument('-v','--verbose', action='count', default=2, help='Set level of verbosity for logs')
+    parser.add_argument('-f','--file-count', dest='file_count', type=int, help='Add limit to number of files to process.')
+    parser.add_argument('-o','--output', dest='output', default=None, help='Send fail list to an output file')
+
+    args = parser.parse_args()
+
+    return {
+        'datafile_path': args.datafile_path,
+        'conf': args.conf,
+        'dryrun': args.dryrun,
+        'test': args.test,
+        'prefix': args.prefix,
+        'verbose': args.verbose-1,
+        'file_count': args.file_count,
+        'output': args.output
+    }
+
+def main(args: dict = None):
+    if args is None:
+        args = _get_command_line_args()
+    if isinstance(args['conf'], str):
+        conf = load_config(args['conf'])
+
+    if conf is None:
+        logger.error('Config file could not be loaded')
+        return
+    if not os.path.isfile(args['datafile_path']):
+        logger.error(f'Inaccessible Datafile - {args["datafile_path"]}')
+        return
+    
+    if check_timeout():
+        logger.error('Check-timeout failed')
+        return
+    
+    file_limit = conf.get('file_limit', None) or args.get('file_limit', None)
+
+    set_verbose(args['verbose'])
+
+    fs = FacetUpdateHandler(conf, dryrun=args['dryrun'], test=args['test'])
+    fail_list = fs.process_deposits(args['datafile_path'], args['prefix'], file_limit=file_limit)
+
+    logger.info('Failed items:')
+    for f in fail_list:
+        logger.info(f)
+
+    if args['output'] is not None and fail_list != []:
+        with open(args['output'],'w') as f:
+            f.write('\n'.join(fail_list))
+
+if __name__ == '__main__':
+    main()

cci_os_worker-0.5.0/cci_os_worker/deleter.py

@@ -0,0 +1,115 @@
+# encoding: utf-8
+__author__ = 'Daniel Westwood'
+__date__ = '05 Nov 2024'
+__copyright__ = 'Copyright 2024 United Kingdom Research and Innovation'
+__license__ = 'BSD - see LICENSE file in top-level package directory'
+__contact__ = 'daniel.westwood@stfc.ac.uk'
+
+import logging
+import hashlib
+import argparse
+import os
+
+from typing import Union
+
+from cci_facet_scanner.core.facet_scanner import FacetScanner
+from ceda_elasticsearch_tools.elasticsearch import CEDAElasticsearchClient
+from fbi_directory_check.utils import check_timeout
+
+from .utils import load_config, UpdateHandler, set_verbose
+
+from cci_os_worker import logstream
+
+logger = logging.getLogger(__name__)
+logger.addHandler(logstream)
+logger.propagate = False
+
+class ElasticsearchDeleter(UpdateHandler):
+
+    def __init__(self, conf: dict, dryrun: bool = False, test: bool = False):
+        """
+        Initialise this class with the correct connections to 
+        establish an elasticsearch client.
+        """
+        logger.info('Loading ES Deleter')
+
+        super().__init__(conf, dryrun=dryrun, test=test)
+
+    def _remove_file(self, filepath):
+        """
+        Delete all records for a specific dataset
+        """
+
+        index = self._conf['facet_files_index']['name']
+
+        id = hashlib.sha1(filepath.encode(errors="ignore")).hexdigest()
+
+        if not self._dryrun:
+            try:
+                self.es.delete(
+                    index=index,
+                    id=id,
+                )
+                return 0
+            except:
+                return 1
+        else:
+            logger.info(f'DRYRUN: Skipped deleting {filepath.split("/")[-1]}')
+            return 0
+        
+def _get_command_line_args():
+    """
+    Get the command line arguments for the facet scan
+    """
+    parser = argparse.ArgumentParser(description='Entrypoint for the CCI OS Worker on the CMD Line')
+    parser.add_argument('datafile_path', type=str, help='Path to the "datasets.txt" file')
+    parser.add_argument('conf', type=str, help='Path to Yaml config file for Elasticsearch')
+
+    parser.add_argument('-d','--dryrun', dest='dryrun', action='store_true', help='Perform in dryrun mode')
+    parser.add_argument('-t','--test', dest='test', action='store_true', help='Perform in test/staging mode')
+    parser.add_argument('-v','--verbose', action='count', default=2, help='Set level of verbosity for logs')
+    parser.add_argument('-o','--output', dest='output', default=None, help='Send fail list to an output file')
+
+    args = parser.parse_args()
+
+    return {
+        'datafile_path': args.datafile_path,
+        'conf': args.conf,
+        'dryrun': args.dryrun,
+        'test': args.test,
+        'verbose': args.verbose-1,
+        'output': args.output
+    }
+
+def main(args: dict = None):
+    if args is None:
+        args = _get_command_line_args()
+    if isinstance(args['conf'], str):
+        conf = load_config(args['conf'])
+
+    if conf is None:
+        logger.error('Config file could not be loaded')
+        return
+    if not os.path.isfile(args['datafile_path']):
+        logger.error(f'Inaccessible Datafile - {args["datafile_path"]}')
+        return
+    
+    if check_timeout():
+        logger.error('Check-timeout failed')
+        return
+
+    set_verbose(args['verbose'])
+
+    deleter = ElasticsearchDeleter(conf, dryrun=args['dryrun'], test=args['test'])
+    fail_list = deleter.process_removals(args['datafile_path'])
+
+    logger.info('Failed items:')
+    for f in fail_list:
+        logger.info(f)
+
+    if args['output'] is not None and fail_list != []:
+        with open(args['output'],'w') as f:
+            f.write('\n'.join(fail_list))
+
+if __name__ == '__main__':
+    main()