mapepire-python 0.1.4__tar.gz → 0.1.5__tar.gz

{mapepire_python-0.1.4 → mapepire_python-0.1.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: mapepire-python
-Version: 0.1.4
+Version: 0.1.5
 Author-email: Allen Institute for Artificial Intelligence <contact@allenai.org>
 License:                                  Apache License
                                    Version 2.0, January 2004
@@ -217,12 +217,15 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: dataclasses-json>=0.6.4
 Requires-Dist: websocket-client>=1.2.1
+Requires-Dist: websockets
+Requires-Dist: pyee
 Provides-Extra: dev
 Requires-Dist: ruff; extra == "dev"
 Requires-Dist: mypy<1.5,>=1.0; extra == "dev"
 Requires-Dist: black<24.0,>=23.0; extra == "dev"
 Requires-Dist: isort<5.13,>=5.12; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-asyncio; extra == "dev"
 Requires-Dist: pytest-sphinx; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: twine>=1.11.0; extra == "dev"
@@ -252,24 +255,36 @@ Requires-Dist: pre-commit; extra == "dev"
 <br/>
 </div>
 
-- [Overview](#overview)
-- [Setup](#setup)
-  - [Install with `pip`](#install-with-pip)
-  - [Server Component Setup (Forthcoming)](#server-component-setup-forthcoming)
-- [Example usage](#example-usage)
+- [mapepire-python](#mapepire-python)
+  - [Overview](#overview)
+  - [Setup](#setup)
+    - [Install with `pip`](#install-with-pip)
+    - [Server Component Setup](#server-component-setup)
+  - [Example usage](#example-usage)
+    - [Query and run](#query-and-run)
+    - [Asynchronous Query Execution](#asynchronous-query-execution)
+  - [Pooling (beta)](#pooling-beta)
 - [Development Setup](#development-setup)
   - [Setup python virtual environment with pip and venv](#setup-python-virtual-environment-with-pip-and-venv)
     - [Create a new virtual environment](#create-a-new-virtual-environment)
       - [Unix/macOS](#unixmacos)
       - [Windows](#windows)
     - [Activate the virtual environment:](#activate-the-virtual-environment)
+      - [Unix.macOS](#unixmacos-1)
+      - [Windows](#windows-1)
+      - [Unix/macOS](#unixmacos-2)
+      - [Windows](#windows-2)
     - [Prepare pip](#prepare-pip)
+      - [Unix/macOS](#unixmacos-3)
+      - [Windows](#windows-3)
     - [Install Dependencies using `requirements-dev.txt`](#install-dependencies-using-requirements-devtxt)
+      - [Unix/macOS](#unixmacos-4)
+      - [Windows](#windows-4)
   - [Setup Python virtual environment with Conda](#setup-python-virtual-environment-with-conda)
     - [Create an environment from an environment-dev.yml file](#create-an-environment-from-an-environment-devyml-file)
       - [1. Activate the new environment:](#1-activate-the-new-environment)
       - [2. Verify the new environment was installed:](#2-verify-the-new-environment-was-installed)
-- [Run local test suite](#run-local-test-suite)
+  - [Run local test suite](#run-local-test-suite)
 
 
 
@@ -299,26 +314,38 @@ To use mapire-python, you will need to have the Mapepire Server Component runnin
 
 ## Example usage
 
-The following script sets up a `DaemonServer` object that will be used to connect with the Server Component. Then a single `SQLJob` is created to facilitate the connection from the client side. 
+Setup the server credentials used to connect to the server. One way to do this is to create a `mapepire.ini` file in the root of your project with the following content:
+
+```ini
+[mapepire]
+SERVER="SERVER"
+PORT="PORT"
+USER="USER"
+PASSWORD="PASSWORD"
+```
+
+The following script sets up a `DaemonServer` object that will be used to connect with the Server Component. Then a single `SQLJob` is created to facilitate the connection from the client side.
 
 ```python
+import configparser
 from mapepire_python.client.sql_job import SQLJob
 from mapepire_python.types import DaemonServer
 
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
 creds = DaemonServer(
-    host="localhost",
-    port=8085,
-    user="USER",
-    password="PASSWORD",
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
     ignoreUnauthorized=True
 )
 
-
-job = SQLJob()
-res = job.connect(creds)
-query = job.query("select * from sample.employee")
-result = query.run(rows_to_fetch=3)
-print(result)
+with SQLJob(creds) as sql_job:
+    with sql_job.query("select * from sample.employee") as query:
+        result = query.run(rows_to_fetch=1)
+        print(result)
 ```
 
 Here is the output from the script above:
@@ -441,6 +468,156 @@ Here is the output from the script above:
 }
 
 ```
+The results object is a JSON object that contains the metadata and data from the query. Here are the different fields returned:
+- `id` field contains the query ID
+- `has_results` field indicates whether the query returned any results
+- `update_count` field indicates the number of rows updated by the query (-1 if the query did not update any rows)
+- `metadata` field contains information about the columns returned by the query
+- `data` field contains the results of the query 
+- `is_done` field indicates whether the query has finished executing
+- `success` field indicates whether the query was successful. 
+
+In the ouput above, the query was successful and returned one row of data.
+
+### Query and run 
+
+To create and run a query in a single step, use the `query_and_run` method: 
+
+```python
+import configparser
+from mapepire_python.client.sql_job import SQLJob
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+with SQLJob(creds) as sql_job:
+    # query automatically closed after running
+    results = sql_job.query_and_run("select * from sample.employee", rows_to_fetch=1)
+    print(result)
+```
+
+### Asynchronous Query Execution
+
+The `PoolJob` object can be used to create and run queries asynchronously:
+
+```python
+import asyncio
+import configparser
+from mapepire_python.pool.pool_job import PoolJob
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+async def main():
+    async with PoolJob(creds=creds) as pool_job:
+        async with pool_job.query('select * from sample.employee') as query:
+          res = await query.run(rows_to_fetch=1)
+
+if __name__ == '__main__':
+    asyncio.run(main())
+
+```
+
+To run a create and run a query asynchronously in a single step, use the `query_and_run` method:
+
+```python
+import asyncio
+import configparser
+from mapepire_python.pool.pool_job import PoolJob
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+async def main():
+    async with PoolJob(creds=creds) as pool_job:
+        res = await pool_job.query_and_run(rows_to_fetch=1)
+        print(res)
+
+if __name__ == '__main__':
+    asyncio.run(main())
+
+```
+
+
+## Pooling (beta)
+
+The `Pool` object can be used to create a pool of `PoolJob` objects to run queries concurrently. 
+
+```python
+import asyncio
+import configparser
+from mapepire_python.pool.pool_client import Pool, PoolOptions
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+
+async def main():
+    async with Pool(
+        options=PoolOptions(
+            creds=creds,
+            opts=None,
+            max_size=5,
+            starting_size=3
+        )
+    ) as pool:
+      job_names = []
+      resultsA = await asyncio.gather(
+          pool.execute('values (job_name)'),
+          pool.execute('values (job_name)'),
+          pool.execute('values (job_name)')
+      )
+      job_names = [res['data'][0]['00001'] for res in resultsA]
+
+      print(job_names)
+
+
+if __name__ == '__main__':
+    asyncio.run(main())
+
+```
+This script will create a pool of 3 `PoolJob` objects and run the query `values (job_name)` concurrently. The results will be printed to the console.
+
+```bash
+['004460/QUSER/QZDASOINIT', '005096/QUSER/QZDASOINIT', '005319/QUSER/QZDASOINIT']
+```
+
 
 # Development Setup
 

{mapepire_python-0.1.4 → mapepire_python-0.1.5}/README.md

@@ -12,24 +12,36 @@
 <br/>
 </div>
 
-- [Overview](#overview)
-- [Setup](#setup)
-  - [Install with `pip`](#install-with-pip)
-  - [Server Component Setup (Forthcoming)](#server-component-setup-forthcoming)
-- [Example usage](#example-usage)
+- [mapepire-python](#mapepire-python)
+  - [Overview](#overview)
+  - [Setup](#setup)
+    - [Install with `pip`](#install-with-pip)
+    - [Server Component Setup](#server-component-setup)
+  - [Example usage](#example-usage)
+    - [Query and run](#query-and-run)
+    - [Asynchronous Query Execution](#asynchronous-query-execution)
+  - [Pooling (beta)](#pooling-beta)
 - [Development Setup](#development-setup)
   - [Setup python virtual environment with pip and venv](#setup-python-virtual-environment-with-pip-and-venv)
     - [Create a new virtual environment](#create-a-new-virtual-environment)
       - [Unix/macOS](#unixmacos)
       - [Windows](#windows)
     - [Activate the virtual environment:](#activate-the-virtual-environment)
+      - [Unix.macOS](#unixmacos-1)
+      - [Windows](#windows-1)
+      - [Unix/macOS](#unixmacos-2)
+      - [Windows](#windows-2)
     - [Prepare pip](#prepare-pip)
+      - [Unix/macOS](#unixmacos-3)
+      - [Windows](#windows-3)
     - [Install Dependencies using `requirements-dev.txt`](#install-dependencies-using-requirements-devtxt)
+      - [Unix/macOS](#unixmacos-4)
+      - [Windows](#windows-4)
   - [Setup Python virtual environment with Conda](#setup-python-virtual-environment-with-conda)
     - [Create an environment from an environment-dev.yml file](#create-an-environment-from-an-environment-devyml-file)
       - [1. Activate the new environment:](#1-activate-the-new-environment)
       - [2. Verify the new environment was installed:](#2-verify-the-new-environment-was-installed)
-- [Run local test suite](#run-local-test-suite)
+  - [Run local test suite](#run-local-test-suite)
 
 
 
@@ -59,26 +71,38 @@ To use mapire-python, you will need to have the Mapepire Server Component runnin
 
 ## Example usage
 
-The following script sets up a `DaemonServer` object that will be used to connect with the Server Component. Then a single `SQLJob` is created to facilitate the connection from the client side. 
+Setup the server credentials used to connect to the server. One way to do this is to create a `mapepire.ini` file in the root of your project with the following content:
+
+```ini
+[mapepire]
+SERVER="SERVER"
+PORT="PORT"
+USER="USER"
+PASSWORD="PASSWORD"
+```
+
+The following script sets up a `DaemonServer` object that will be used to connect with the Server Component. Then a single `SQLJob` is created to facilitate the connection from the client side.
 
 ```python
+import configparser
 from mapepire_python.client.sql_job import SQLJob
 from mapepire_python.types import DaemonServer
 
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
 creds = DaemonServer(
-    host="localhost",
-    port=8085,
-    user="USER",
-    password="PASSWORD",
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
     ignoreUnauthorized=True
 )
 
-
-job = SQLJob()
-res = job.connect(creds)
-query = job.query("select * from sample.employee")
-result = query.run(rows_to_fetch=3)
-print(result)
+with SQLJob(creds) as sql_job:
+    with sql_job.query("select * from sample.employee") as query:
+        result = query.run(rows_to_fetch=1)
+        print(result)
 ```
 
 Here is the output from the script above:
@@ -201,6 +225,156 @@ Here is the output from the script above:
 }
 
 ```
+The results object is a JSON object that contains the metadata and data from the query. Here are the different fields returned:
+- `id` field contains the query ID
+- `has_results` field indicates whether the query returned any results
+- `update_count` field indicates the number of rows updated by the query (-1 if the query did not update any rows)
+- `metadata` field contains information about the columns returned by the query
+- `data` field contains the results of the query 
+- `is_done` field indicates whether the query has finished executing
+- `success` field indicates whether the query was successful. 
+
+In the ouput above, the query was successful and returned one row of data.
+
+### Query and run 
+
+To create and run a query in a single step, use the `query_and_run` method: 
+
+```python
+import configparser
+from mapepire_python.client.sql_job import SQLJob
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+with SQLJob(creds) as sql_job:
+    # query automatically closed after running
+    results = sql_job.query_and_run("select * from sample.employee", rows_to_fetch=1)
+    print(result)
+```
+
+### Asynchronous Query Execution
+
+The `PoolJob` object can be used to create and run queries asynchronously:
+
+```python
+import asyncio
+import configparser
+from mapepire_python.pool.pool_job import PoolJob
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+async def main():
+    async with PoolJob(creds=creds) as pool_job:
+        async with pool_job.query('select * from sample.employee') as query:
+          res = await query.run(rows_to_fetch=1)
+
+if __name__ == '__main__':
+    asyncio.run(main())
+
+```
+
+To run a create and run a query asynchronously in a single step, use the `query_and_run` method:
+
+```python
+import asyncio
+import configparser
+from mapepire_python.pool.pool_job import PoolJob
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+async def main():
+    async with PoolJob(creds=creds) as pool_job:
+        res = await pool_job.query_and_run(rows_to_fetch=1)
+        print(res)
+
+if __name__ == '__main__':
+    asyncio.run(main())
+
+```
+
+
+## Pooling (beta)
+
+The `Pool` object can be used to create a pool of `PoolJob` objects to run queries concurrently. 
+
+```python
+import asyncio
+import configparser
+from mapepire_python.pool.pool_client import Pool, PoolOptions
+from mapepire_python.types import DaemonServer
+
+config = configparser.ConfigParser()
+config.read('mapepire.ini')
+
+creds = DaemonServer(
+    host=config['mapepire']['SERVER'],
+    port=config['mapepire']['PORT'],
+    user=config['mapepire']['USER'],
+    password=config['mapepire']['PASSWORD'],
+    ignoreUnauthorized=True
+)
+
+
+async def main():
+    async with Pool(
+        options=PoolOptions(
+            creds=creds,
+            opts=None,
+            max_size=5,
+            starting_size=3
+        )
+    ) as pool:
+      job_names = []
+      resultsA = await asyncio.gather(
+          pool.execute('values (job_name)'),
+          pool.execute('values (job_name)'),
+          pool.execute('values (job_name)')
+      )
+      job_names = [res['data'][0]['00001'] for res in resultsA]
+
+      print(job_names)
+
+
+if __name__ == '__main__':
+    asyncio.run(main())
+
+```
+This script will create a pool of 3 `PoolJob` objects and run the query `values (job_name)` concurrently. The results will be printed to the console.
+
+```bash
+['004460/QUSER/QZDASOINIT', '005096/QUSER/QZDASOINIT', '005319/QUSER/QZDASOINIT']
+```
+
 
 # Development Setup
 

mapepire_python-0.1.5/mapepire_python/base_job.py

@@ -0,0 +1,42 @@
+from typing import Any, Dict, Optional, Union
+
+from .data_types import DaemonServer, JobStatus, QueryOptions
+
+
+class BaseJob:
+    def __init__(self, creds: DaemonServer = None, options: Dict[Any, Any] = {}) -> None:
+        self.creds = creds
+        self.options = options
+
+    def connect(self, db2_server: Union[DaemonServer, Dict[str, Any]]) -> Dict[str, Any]:
+        raise NotImplementedError()
+
+    def close(self) -> None:
+        raise NotImplementedError()
+
+    def get_status(self) -> JobStatus:
+        raise NotImplementedError()
+
+    def query(
+        self,
+        sql: str,
+        opts: Optional[Union[Dict[str, Any], QueryOptions]] = None,
+    ):
+        raise NotImplementedError()
+
+    def query_and_run(
+        self, sql: str, opts: Optional[Dict[str, Any]] = None, **kwargs
+    ) -> Dict[str, Any]:
+        raise NotImplementedError()
+
+    def __enter__(self):
+        raise NotImplementedError()
+
+    def __exit__(self, *args, **kwargs):
+        raise NotImplementedError()
+
+    async def __aenter__(self):
+        raise NotImplementedError()
+
+    async def __aexit__(self, *args, **kwargs):
+        raise NotImplementedError()

{mapepire_python-0.1.4 → mapepire_python-0.1.5}/mapepire_python/client/query.py

@@ -1,8 +1,8 @@
 import json
 from enum import Enum
-from typing import Any, Dict, Generic, List, Optional, TypeVar, Union
+from typing import Any, Dict, Generic, List, Optional, TypeVar
 
-from ..types import QueryOptions
+from ..data_types import QueryOptions
 from .sql_job import SQLJob
 
 T = TypeVar("T")
@@ -15,23 +15,14 @@ class QueryState(Enum):
     ERROR = 4
 
 
-def get_query_options(opts: Optional[Union[Dict[str, Any], QueryOptions]] = None) -> QueryOptions:
-    if isinstance(opts, QueryOptions):
-        return opts
-    elif opts:
-        return QueryOptions(**opts)
-    else:
-        return QueryOptions(isClCommand=False, parameters=None, autoClose=False)
-
-
 class Query(Generic[T]):
     global_query_list: List["Query[Any]"] = []
 
     def __init__(self, job: SQLJob, query: str, opts: QueryOptions) -> None:
         self.job = job
+        self.sql: str = query
         self.is_prepared: bool = True if opts.parameters is not None else False
         self.parameters: Optional[List[str]] = opts.parameters
-        self.sql: str = query
         self.is_cl_command: Optional[bool] = opts.isClCommand
         self.should_auto_close: Optional[bool] = opts.autoClose
         self.is_terse_results: Optional[bool] = opts.isTerseResults
@@ -41,6 +32,17 @@ class Query(Generic[T]):
 
         Query.global_query_list.append(self)
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, exc_type, exc_value, traceback):
+        self.close()
+
+    def _execute_query(self, qeury_object: Dict[str, Any]) -> Dict[str, Any]:
+        self.job.send(json.dumps(qeury_object))
+        query_result: Dict[str, Any] = json.loads(self.job._socket.recv())
+        return query_result
+
     def run(self, rows_to_fetch: Optional[int] = None) -> Dict[str, Any]:
         if rows_to_fetch is None:
             rows_to_fetch = self._rows_to_fetch
@@ -71,8 +73,7 @@ class Query(Generic[T]):
                 "parameters": self.parameters,
             }
 
-        self.job.send(json.dumps(query_object))
-        query_result: Dict[str, Any] = json.loads(self.job._socket.recv())
+        query_result: Dict[str, Any] = self._execute_query(query_object)
 
         self.state = (
             QueryState.RUN_DONE
@@ -81,7 +82,6 @@ class Query(Generic[T]):
         )
 
         if not query_result.get("success", False) and not self.is_cl_command:
-            print(query_result)
             self.state = QueryState.ERROR
             error_keys = ["error", "sql_state", "sql_rc"]
             error_list = {
@@ -116,8 +116,7 @@ class Query(Generic[T]):
         }
 
         self._rows_to_fetch = rows_to_fetch
-        self.job.send(json.dumps(query_object))
-        query_result: Dict[str, Any] = json.loads(self.job._socket.recv())
+        query_result: Dict[str, Any] = self._execute_query(query_object)
 
         self.state = (
             QueryState.RUN_DONE
@@ -130,3 +129,18 @@ class Query(Generic[T]):
             raise Exception(query_result["error"] or "Failed to run Query (unknown error)")
 
         return query_result
+
+    def close(self):
+        if not self.job._socket.connected:
+            raise Exception("SQL Job not connected")
+        if self._correlation_id and self.state is not QueryState.RUN_DONE:
+            self.state = QueryState.RUN_DONE
+            query_object = {
+                "id": self.job._get_unique_id("sqlclose"),
+                "cont_id": self._correlation_id,
+                "type": "sqlclose",
+            }
+
+            return self._execute_query(query_object)
+        elif not self._correlation_id:
+            self.state = QueryState.RUN_DONE