dnaStreaming 2.0.5__tar.gz → 2.2.0__tar.gz

{dnaStreaming-2.0.5/dnaStreaming.egg-info → dnastreaming-2.2.0}/PKG-INFO

@@ -1,20 +1,27 @@
 Metadata-Version: 2.1
 Name: dnaStreaming
-Version: 2.0.5
+Version: 2.2.0
 Summary: Dow Jones DNA Streaming Project
 Home-page: https://github.com/dowjones/dj-dna-streams-python
-Download-URL: https://github.com/dowjones/dj-dna-streams-python/archive/release-2.0.5.tar.gz
+Download-URL: https://github.com/dowjones/dj-dna-streams-python/archive/release-2.2.0.tar.gz
 Author: Zachary Kagan
 Author-email: zachary.kagan@dowjones.com
 License: MIT
 Keywords: DOWJONES,FACTIVA,STREAMS
-Platform: UNKNOWN
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Description-Content-Type: text/x-rst
 License-File: LICENSE
+Requires-Dist: googleapis-common-protos>=1.56.4
+Requires-Dist: google-auth>=2.11.0
+Requires-Dist: google-cloud-pubsub==2.13.6
+Requires-Dist: google-cloud-core==2.3.2
+Requires-Dist: google-api-core==2.10.1
+Requires-Dist: mock>=3.0.5
+Requires-Dist: oauth2client>=4.1.3
+Requires-Dist: requests>=2.28.0
 
 How To Use
 ----------
@@ -40,14 +47,17 @@ There is currently one way to authenticate, which is by using **your user key**.
 Configuring
 ___________
 
-To run this code, you need to provide credentials from one of the authentication methods and your subscriptions. There are 3 ways to do this. You can either set environment variables or you can use a configuration file.
+To run this code, you need to provide credentials from one of the authentication methods and your subscriptions. There are 3 ways to do this: you can set environment variables, use a configuration file or pass an argument to a constructor.
 
 1. Set environment variables.
 ###################################################################
 
-To set your service account credentials, set:
+To set your service account credentials, set the `USER_KEY` environment variable:
+
+.. code-block::
+
+    export USER_KEY="1234567890-098765432"
 
-- An environment variable named 'USER_KEY'.
 
 To set your subscription ID, simply set an environment variable named 'SUBSCRIPTION_ID' like so
 
@@ -61,18 +71,33 @@ The code above is the command line expression for setting this environment varia
 2. Using the configuration file.
 ###################################################################
 
-In this codebase you will find a file named 'customer_config.json'. You are not required to use this file. If you prefer to use this configuration file, follow these directions: Open this file and add your service account credentials. Then add your subscription IDs. Remember that this is a JSON file so follow basic JSON formatting and syntax conventions.
+In this codebase you will find a file named 'customer_config.json'. You are not required to use this file. If you prefer to use this option, fill the JSON object within by adding your user key and your subscription ID. Remember that this is a JSON file so follow basic JSON formatting and syntax conventions.
+
+> The listener will search for the `customer_config.json` file inside your `$HOME` directory by default.
+
+If you prefer using an explicit path to your configuration file, pass the absolute path to the Listener constructor like so:
+
+.. code-block:: python
+
+    from dnaStreaming.listener import Listener
+    # Config. file authentication
+    listener = Listener(config_file=<ABSOLUTE PATH TO YOUR CONFIG. FILE>)
+
 
 3. Pass in variables as function arguments.
 ###################################################################
 
-You may pass your service account credentials to the Listener constructor like so:
+You may pass your user key to the Listener constructor and your subscription ID to the listen method like so:
 
 .. code-block:: python
 
     from dnaStreaming.listener import Listener
-    # User key authentication
+    # Use the user_key argument to provide your credentials
     listener = Listener(user_key=<YOUR USER KEY>)
+    # Use the subscription_id argument to provide your subscription id to the listener
+    listener.listen(callback, subscription_id=<YOUR SUBSCRIPTION ID>)
+    # same parameter for the async variation
+    listener.listen_async(callback, subscription_id=<YOUR SUBSCRIPTION ID>)
 
 
 Or you may use the environment variables.
@@ -218,13 +243,6 @@ Or
     python ./dnaStreaming/demo/show_stream_async.py -s
 
 
-If you are having `ImportError:  No module named ...` run this in your terminal before running the demo:
-
-.. code-block::
-
-    export PYTHONPATH='.'
-
-
 Running Docker Demo
 
 Execute the following at the project root:
@@ -235,5 +253,3 @@ Execute the following at the project root:
     -e USER_KEY=<your user KEY> \
     -e SUBSCRIPTION_ID=<your subscription ID> \
     dj-dna-streaming-python
-
-

{dnaStreaming-2.0.5 → dnastreaming-2.2.0}/README.rst

@@ -22,14 +22,17 @@ There is currently one way to authenticate, which is by using **your user key**.
 Configuring
 ___________
 
-To run this code, you need to provide credentials from one of the authentication methods and your subscriptions. There are 3 ways to do this. You can either set environment variables or you can use a configuration file.
+To run this code, you need to provide credentials from one of the authentication methods and your subscriptions. There are 3 ways to do this: you can set environment variables, use a configuration file or pass an argument to a constructor.
 
 1. Set environment variables.
 ###################################################################
 
-To set your service account credentials, set:
+To set your service account credentials, set the `USER_KEY` environment variable:
+
+.. code-block::
+
+    export USER_KEY="1234567890-098765432"
 
-- An environment variable named 'USER_KEY'.
 
 To set your subscription ID, simply set an environment variable named 'SUBSCRIPTION_ID' like so
 
@@ -43,18 +46,33 @@ The code above is the command line expression for setting this environment varia
 2. Using the configuration file.
 ###################################################################
 
-In this codebase you will find a file named 'customer_config.json'. You are not required to use this file. If you prefer to use this configuration file, follow these directions: Open this file and add your service account credentials. Then add your subscription IDs. Remember that this is a JSON file so follow basic JSON formatting and syntax conventions.
+In this codebase you will find a file named 'customer_config.json'. You are not required to use this file. If you prefer to use this option, fill the JSON object within by adding your user key and your subscription ID. Remember that this is a JSON file so follow basic JSON formatting and syntax conventions.
+
+> The listener will search for the `customer_config.json` file inside your `$HOME` directory by default.
+
+If you prefer using an explicit path to your configuration file, pass the absolute path to the Listener constructor like so:
+
+.. code-block:: python
+
+    from dnaStreaming.listener import Listener
+    # Config. file authentication
+    listener = Listener(config_file=<ABSOLUTE PATH TO YOUR CONFIG. FILE>)
+
 
 3. Pass in variables as function arguments.
 ###################################################################
 
-You may pass your service account credentials to the Listener constructor like so:
+You may pass your user key to the Listener constructor and your subscription ID to the listen method like so:
 
 .. code-block:: python
 
     from dnaStreaming.listener import Listener
-    # User key authentication
+    # Use the user_key argument to provide your credentials
     listener = Listener(user_key=<YOUR USER KEY>)
+    # Use the subscription_id argument to provide your subscription id to the listener
+    listener.listen(callback, subscription_id=<YOUR SUBSCRIPTION ID>)
+    # same parameter for the async variation
+    listener.listen_async(callback, subscription_id=<YOUR SUBSCRIPTION ID>)
 
 
 Or you may use the environment variables.
@@ -200,13 +218,6 @@ Or
     python ./dnaStreaming/demo/show_stream_async.py -s
 
 
-If you are having `ImportError:  No module named ...` run this in your terminal before running the demo:
-
-.. code-block::
-
-    export PYTHONPATH='.'
-
-
 Running Docker Demo
 
 Execute the following at the project root:

dnastreaming-2.2.0/dnaStreaming/config.py

@@ -0,0 +1,109 @@
+from __future__ import absolute_import, division
+
+import errno
+import json
+import os
+from pathlib import Path
+
+class Config(object):
+
+    DEFAULT_HOST = 'https://api.dowjones.com'
+
+    DEFAULT_CUST_CONFIG_PATH = str(Path.home()) + '/customer_config.json'
+    ENV_VAR_SUBSCRIPTION_ID = 'SUBSCRIPTION_ID'
+    ENV_VAR_USER_KEY = 'USER_KEY'
+    ENV_VAR_SERVICE_ACCOUNT_ID = 'SERVICE_ACCOUNT_ID'
+    ENV_VAR_API_HOST = 'API_HOST'
+
+    def __init__(self, service_account_id=None, user_key=None, config_file=None):
+        self.customer_config_path = self.DEFAULT_CUST_CONFIG_PATH if config_file is None else config_file
+        self.initialized = False
+        self.service_account_id = service_account_id
+        self.user_key = user_key
+
+        self.headers = None
+
+    def _initialize(self):
+        self._validate()
+
+        with open(self.customer_config_path, 'r') as f:
+            self.customer_config = json.load(f)
+
+        self.initialized = True
+        self.headers = None
+
+    def _validate(self):
+        if not os.path.isfile(self.customer_config_path):
+            raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), self.customer_config_path)
+
+        if not os.access(self.customer_config_path, os.R_OK):
+            raise Exception('Encountered permission problem reading file from path \'{}\'.'.format(self.customer_config_path))
+
+    def get_headers(self):
+        if self.headers:
+            return self.headers
+        else:
+            self.headers = self.get_authentication_headers()
+            return self.headers
+
+    def get_authentication_headers(self):
+
+        user_key = self.get_user_key()
+        if user_key:
+            return {
+                'user-key': user_key
+            }
+
+        else:
+            msg = """
+                Unable to find credentials. Specify your account credentials by choosing one of the following ways:
+                - set your user key to an env var under the name 'USER_KEY'
+                - place your customer_config.json file in your home directory (e.g. ~/, $HOME/)
+                - pass the absolute path of your customer_config.json file to the constructor of the Listener class
+                - pass the user key as a parameter to the constructor of the Listener class
+            """
+            raise Exception(msg)
+
+    def get_uri_context(self):
+        host = os.getenv(self.ENV_VAR_API_HOST, self.DEFAULT_HOST)
+        return host
+
+    # in the following two methods, note that we use "SERVICE_ACCOUNT_ID" as a legacy,
+    # alternate name for the "USER_KEY" parameter, from the customer's perspective
+    def get_user_key(self):
+        user_key = self.user_key if self.user_key else self.service_account_id
+
+        if user_key is None:
+            user_key = os.getenv(self.ENV_VAR_USER_KEY,
+                                 os.getenv(self.ENV_VAR_SERVICE_ACCOUNT_ID))
+
+            if user_key is None:
+                user_key = self._user_key_id_from_file()
+
+        return user_key
+
+    def _user_key_id_from_file(self):
+        if not self.initialized:
+            self._initialize()
+
+        return self.customer_config.get('user_key',
+                                        self.customer_config.get('service_account_id'))
+
+    def subscription(self):
+        if os.getenv(self.ENV_VAR_SUBSCRIPTION_ID) is not None:
+            subscription = os.getenv(self.ENV_VAR_SUBSCRIPTION_ID)
+        else:
+            subscription = self._subscription_id_from_file()
+
+        return subscription
+
+    def _set_customer_config_path(self, path):
+        self.customer_config_path = path
+        self._initialize()
+
+    def _subscription_id_from_file(self):
+        if not self.initialized:
+            self._initialize()
+
+        return self.customer_config['subscription_id']
+

dnastreaming-2.2.0/dnaStreaming/customer_config.json

@@ -0,0 +1,5 @@
+{
+
+  "user_key": "",
+  "subscription_id": ""
+}

dnastreaming-2.2.0/dnaStreaming/demo/show_ha_stream_async.py

@@ -0,0 +1,36 @@
+import os
+from time import sleep
+from dnaStreaming.listener import Listener
+
+listener = Listener()
+quiet_demo = os.getenv('QUIET_DEMO', "false") == "true"
+max_secs = 5
+print("\n[ACTIVITY] Receiving messages (ASYNC) for {} seconds...\n[0]".format(max_secs), end='')
+
+
+def callback(message, subscription_id):
+    callback.counter += 1
+    if not quiet_demo:
+        if message['action'] != 'del':
+            print('[INFO] [MSG] [{}]: AN: {}, TITLE: {}'.format(callback.counter, message['an'], message['title']))
+        else:
+            print('[INFO] [MSG] [{}]: AN: {}, *** DELETE ***'.format(callback.counter, message['an']))
+    else:
+        if callback.counter % 10 == 0:
+            print('[{}]'.format(callback.counter), end='')
+        else:
+            print('.', end='')
+    return True
+
+
+callback.counter = 0
+listener_controller = listener.listen_async_ha(callback)
+
+# Stop receiving messages after 5 seconds
+for count in range(0, max_secs):
+    sleep(1)
+
+if listener_controller.listener_is_running():
+    listener_controller.stop_listener()
+
+print("stop receiving messages")

{dnaStreaming-2.0.5 → dnastreaming-2.2.0}/dnaStreaming/listener.py

@@ -1,7 +1,7 @@
 import time
 import requests
 import json
-from threading import Thread
+from threading import Thread, Event
 
 from google.api_core.exceptions import GoogleAPICallError, NotFound
 from google import pubsub_v1
@@ -9,14 +9,25 @@ from google import pubsub_v1
 from dnaStreaming import logger
 from dnaStreaming.config import Config
 from dnaStreaming.services import pubsub_service, credentials_service
+from dnaStreaming.services.availability_service import MAIN_REGION, BACKUP_REGION, ha_listen
 
+class ListenerController(object):
+    def __init__(self, thread, stop_event):
+        self.thread = thread
+        self.stop_event = stop_event
+
+    def stop_listener(self):
+        self.stop_event.set()
+        self.thread.join()
+
+    def listener_is_running(self):
+        self.thread.is_alive()
 
 class Listener(object):
     DEFAULT_UNLIMITED_MESSAGES = None
 
-    def __init__(self, service_account_id=None, user_key=None, user_id=None, client_id=None, password=None):
-        config = Config(service_account_id, user_key,
-                        user_id, client_id, password)
+    def __init__(self, service_account_id=None, user_key=None, config_file=None):
+        config = Config(service_account_id, user_key, config_file)
         self._initialize(config)
         self.current_subscription_index = 0
 
@@ -113,3 +124,57 @@ class Listener(object):
         logger.info(
             'Listeners for subscriptions have been configured, set and await message arrival.')
         return subscription
+
+    def listen_async_ha(self, on_message_callback, subscription_id=""):
+        def ack_message_and_callback(message):
+            pubsub_msg = json.loads(message.data)
+            logger.info("Received news message with ID: {}".format(
+                pubsub_msg['data'][0]['id']))
+            news_msg = pubsub_msg['data'][0]['attributes']
+            on_message_callback(news_msg, subscription_id)
+            message.ack()
+
+        main_pubsub_client = pubsub_service.get_client(self.config, MAIN_REGION)
+        backup_pubsub_client = pubsub_service.get_client(self.config, BACKUP_REGION)
+
+        subscription_id = subscription_id or self.config.subscription()
+
+        if not subscription_id:
+            raise Exception(
+                'No subscription specified. You must specify the subscription ID either through an environment variable, a config file or '
+                'by passing the value to the method.')
+
+        streaming_credentials = credentials_service.fetch_credentials(
+            self.config)
+
+        api_host = self.config.get_uri_context()
+        user_key = self.config.get_user_key()
+
+        main_subscription_id = subscription_id if "-bak-" not in subscription_id else subscription_id.replace("-filtered-bak-", "-filtered-")
+        backup_subscription_id = subscription_id if "-bak-" in subscription_id else subscription_id.replace("-filtered-", "-filtered-bak-")
+
+        main_subscription_path = main_pubsub_client.subscription_path(
+            streaming_credentials['project_id'], main_subscription_id)
+        backup_subscription_path = backup_pubsub_client.subscription_path(
+            streaming_credentials['project_id'], backup_subscription_id)
+
+        stop_event = Event()
+        listener_thread = Thread(target=ha_listen, daemon=True, args=(
+            api_host,
+            user_key,
+            subscription_id,
+            stop_event,
+            main_subscription_path,
+            backup_subscription_path,
+            main_pubsub_client,
+            backup_pubsub_client,
+            ack_message_and_callback
+        ))
+        listener_thread.start()
+
+        listener_controller = ListenerController(listener_thread, stop_event)
+
+        logger.info(
+            'Listeners for subscriptions have been configured, set and await message arrival.')
+
+        return listener_controller

dnastreaming-2.2.0/dnaStreaming/services/availability_service.py

@@ -0,0 +1,81 @@
+import requests
+import time
+
+from dnaStreaming import logger
+
+MAIN_REGION = "us-central1"
+BACKUP_REGION = "us-east5"
+COOLDOWN_PERIOD = 5 * 60
+
+def get_active_region(api_host, subscription_id, user_key):
+
+    try:
+        ha_status_url = f"{api_host}/stream_health/get_active_region"
+
+        headers = {
+            "subscription-id": subscription_id,
+            "user-key": user_key
+        }
+
+        response = requests.get(url=ha_status_url, headers=headers)
+
+        assert response.status_code == 200, "Failed to fetch active region"
+
+        payload = response.json()
+
+        active_region = payload["data"]["active_region"]
+
+        return active_region
+
+    except:
+
+        return None
+
+
+def ha_listen(api_host, user_key, subscription_id, stop_event, main_subscription_path, backup_subscription_path, main_pubsub_client, backup_pubsub_client, callback):
+
+    # The listener starts reading messages from the main region by default.
+
+    current_region = MAIN_REGION
+
+    streaming_pull_future = main_pubsub_client.subscribe(
+            main_subscription_path, callback=callback)
+    
+    while not stop_event.is_set():
+
+        active_region = get_active_region(api_host, subscription_id, user_key)
+
+        if active_region is None or active_region not in (MAIN_REGION, BACKUP_REGION):
+            logger.warning(f"Got invalid region from API: {active_region}. Listener will keep reading from region {current_region}")
+            continue
+        
+        if current_region != active_region:
+
+            logger.warning(f"Switch event detected in, switching from region {current_region} to region {active_region}")
+            
+            logger.warning(f"Stopping listener in region {current_region}...")
+
+            # We stop the previous listening process
+            streaming_pull_future.cancel()
+            streaming_pull_future.result()
+
+            logger.warning(f"Stopped listener in region {current_region}")
+
+            logger.warning(f"Starting listener in region {active_region}...")
+
+            if active_region == MAIN_REGION:
+                streaming_pull_future = main_pubsub_client.subscribe(
+                    main_subscription_path, callback=callback)
+            else: # active_region == BACKUP_REGION
+                streaming_pull_future = backup_pubsub_client.subscribe(
+                    backup_subscription_path, callback=callback)
+                
+            logger.warning(f"Started listener in region {active_region}")
+
+            current_region = active_region
+        
+        # We wait 5 minutes for cooldown for next call to API
+        time.sleep(COOLDOWN_PERIOD)
+
+    streaming_pull_future.cancel()
+    streaming_pull_future.result()

{dnaStreaming-2.0.5 → dnastreaming-2.2.0}/dnaStreaming/services/credentials_service.py

@@ -5,7 +5,7 @@ import requests
 
 
 def fetch_credentials(config):
-    response = _get_requests().get(config.get_uri_context() + '/accounts/streaming-credentials',
+    response = _get_requests().get(config.get_uri_context() + '/sns-accounts/streaming-credentials',
                                    headers=config.get_headers())
 
     if response.status_code == 401:

dnastreaming-2.2.0/dnaStreaming/services/pubsub_service.py

@@ -0,0 +1,19 @@
+from __future__ import absolute_import, division, print_function
+
+from google.cloud.pubsub_v1 import SubscriberClient
+
+from dnaStreaming.services import authentication_service
+from dnaStreaming.services import credentials_service
+from dnaStreaming.services.availability_service import MAIN_REGION, BACKUP_REGION
+
+
+def get_client(config, region=None):
+    streaming_credentials = credentials_service.fetch_credentials(config)
+    credentials = authentication_service.get_authenticated_oauth_credentials(streaming_credentials)
+
+    if region in (MAIN_REGION, BACKUP_REGION):
+        client_options = {"api_endpoint": f"{region}-pubsub.googleapis.com:443"}
+    else:
+        client_options = {"api_endpoint": "pubsub.googleapis.com:443"}
+
+    return SubscriberClient(credentials=credentials, client_options=client_options)

{dnaStreaming-2.0.5 → dnastreaming-2.2.0}/dnaStreaming/test/test_config.py

@@ -13,9 +13,6 @@ class TestConfig(TestCase):
         self.ensure_remove_environment_variable(Config.ENV_VAR_USER_KEY)
         self.ensure_remove_environment_variable(Config.ENV_VAR_SUBSCRIPTION_ID)
         self.ensure_remove_environment_variable(Config.ENV_VAR_API_HOST)
-        self.ensure_remove_environment_variable(Config.ENV_VAR_USER_ID)
-        self.ensure_remove_environment_variable(Config.ENV_VAR_CLIENT_ID)
-        self.ensure_remove_environment_variable(Config.ENV_VAR_PASSWORD)
 
     def ensure_remove_environment_variable(self, key):
         if key in os.environ:
@@ -55,22 +52,15 @@ class TestConfig(TestCase):
         # Act
         user_key = config.get_user_key()
         subscription = config.subscription()
-        oauth2_credentials = config.oauth2_credentials()
 
         # Assert
         assert user_key
         assert subscription == 'bar'
-        assert oauth2_credentials.get('user_id')
-        assert oauth2_credentials.get('password')
-        assert oauth2_credentials.get('client_id')
 
     def test_environment_variables_success(self):
         # Arrange
         os.environ[Config.ENV_VAR_USER_KEY] = '123'
         os.environ[Config.ENV_VAR_SUBSCRIPTION_ID] = 'ABC'
-        os.environ[Config.ENV_VAR_USER_ID] = 'user'
-        os.environ[Config.ENV_VAR_CLIENT_ID] = 'client'
-        os.environ[Config.ENV_VAR_PASSWORD] = 'password'
 
         # Act
         config = Config()
@@ -82,9 +72,6 @@ class TestConfig(TestCase):
         assert os.environ[Config.ENV_VAR_USER_KEY] == config.get_user_key()
         subscription_id = config.subscription()
         assert subscription_id == 'ABC'
-        assert os.environ[Config.ENV_VAR_USER_ID] == config.oauth2_credentials().get('user_id')
-        assert os.environ[Config.ENV_VAR_CLIENT_ID] == config.oauth2_credentials().get('client_id')
-        assert os.environ[Config.ENV_VAR_PASSWORD] == config.oauth2_credentials().get('password')
 
     def test_environment_variable_service_account_id_success(self):
         # Arrange
@@ -128,42 +115,5 @@ class TestConfig(TestCase):
         # Assert
         assert config.get_user_key() == '123'
 
-    @patch.object(Config, '_fetch_jwt', return_value='test_jwt_value')
-    def test_get_headers_jwt(self, fetch_jwt_mock):
-        # Arrange
-        config = Config()
-
-        fileFolder = os.path.dirname(os.path.realpath(__file__))
-        config._set_customer_config_path(os.path.join(fileFolder, 'test_customer_config.json'))
-
-        headers_expected = {
-            'Authorization': 'test_jwt_value'
-        }
-
-        # Act
-        headers_actual = config.get_authentication_headers()
-
-        # Assert
-        assert headers_actual == headers_expected
-        fetch_jwt_mock.assert_called_once()
-
-    @patch.object(Config, '_fetch_jwt', return_value='test_jwt_value')
-    def test_get_headers_user_key(self, fetch_jwt_mock):
-        # Arrange
-        user_key = "just some user key"
-        config = Config(user_key)
-
-        headers_expected = {
-            'user-key': user_key
-        }
-
-        # Act
-        headers_actual = config.get_authentication_headers()
-
-        # Assert
-        assert headers_actual == headers_expected
-        fetch_jwt_mock.assert_not_called()
-
-
 if __name__ == '__main__' and __package__ is None:
     unittest.main()

{dnaStreaming-2.0.5 → dnastreaming-2.2.0/dnaStreaming.egg-info}/PKG-INFO

@@ -1,20 +1,27 @@
 Metadata-Version: 2.1
 Name: dnaStreaming
-Version: 2.0.5
+Version: 2.2.0
 Summary: Dow Jones DNA Streaming Project
 Home-page: https://github.com/dowjones/dj-dna-streams-python
-Download-URL: https://github.com/dowjones/dj-dna-streams-python/archive/release-2.0.5.tar.gz
+Download-URL: https://github.com/dowjones/dj-dna-streams-python/archive/release-2.2.0.tar.gz
 Author: Zachary Kagan
 Author-email: zachary.kagan@dowjones.com
 License: MIT
 Keywords: DOWJONES,FACTIVA,STREAMS
-Platform: UNKNOWN
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
 Description-Content-Type: text/x-rst
 License-File: LICENSE
+Requires-Dist: googleapis-common-protos>=1.56.4
+Requires-Dist: google-auth>=2.11.0
+Requires-Dist: google-cloud-pubsub==2.13.6
+Requires-Dist: google-cloud-core==2.3.2
+Requires-Dist: google-api-core==2.10.1
+Requires-Dist: mock>=3.0.5
+Requires-Dist: oauth2client>=4.1.3
+Requires-Dist: requests>=2.28.0
 
 How To Use
 ----------
@@ -40,14 +47,17 @@ There is currently one way to authenticate, which is by using **your user key**.
 Configuring
 ___________
 
-To run this code, you need to provide credentials from one of the authentication methods and your subscriptions. There are 3 ways to do this. You can either set environment variables or you can use a configuration file.
+To run this code, you need to provide credentials from one of the authentication methods and your subscriptions. There are 3 ways to do this: you can set environment variables, use a configuration file or pass an argument to a constructor.
 
 1. Set environment variables.
 ###################################################################
 
-To set your service account credentials, set:
+To set your service account credentials, set the `USER_KEY` environment variable:
+
+.. code-block::
+
+    export USER_KEY="1234567890-098765432"
 
-- An environment variable named 'USER_KEY'.
 
 To set your subscription ID, simply set an environment variable named 'SUBSCRIPTION_ID' like so
 
@@ -61,18 +71,33 @@ The code above is the command line expression for setting this environment varia
 2. Using the configuration file.
 ###################################################################
 
-In this codebase you will find a file named 'customer_config.json'. You are not required to use this file. If you prefer to use this configuration file, follow these directions: Open this file and add your service account credentials. Then add your subscription IDs. Remember that this is a JSON file so follow basic JSON formatting and syntax conventions.
+In this codebase you will find a file named 'customer_config.json'. You are not required to use this file. If you prefer to use this option, fill the JSON object within by adding your user key and your subscription ID. Remember that this is a JSON file so follow basic JSON formatting and syntax conventions.
+
+> The listener will search for the `customer_config.json` file inside your `$HOME` directory by default.
+
+If you prefer using an explicit path to your configuration file, pass the absolute path to the Listener constructor like so:
+
+.. code-block:: python
+
+    from dnaStreaming.listener import Listener
+    # Config. file authentication
+    listener = Listener(config_file=<ABSOLUTE PATH TO YOUR CONFIG. FILE>)
+
 
 3. Pass in variables as function arguments.
 ###################################################################
 
-You may pass your service account credentials to the Listener constructor like so:
+You may pass your user key to the Listener constructor and your subscription ID to the listen method like so:
 
 .. code-block:: python
 
     from dnaStreaming.listener import Listener
-    # User key authentication
+    # Use the user_key argument to provide your credentials
     listener = Listener(user_key=<YOUR USER KEY>)
+    # Use the subscription_id argument to provide your subscription id to the listener
+    listener.listen(callback, subscription_id=<YOUR SUBSCRIPTION ID>)
+    # same parameter for the async variation
+    listener.listen_async(callback, subscription_id=<YOUR SUBSCRIPTION ID>)
 
 
 Or you may use the environment variables.
@@ -218,13 +243,6 @@ Or
     python ./dnaStreaming/demo/show_stream_async.py -s
 
 
-If you are having `ImportError:  No module named ...` run this in your terminal before running the demo:
-
-.. code-block::
-
-    export PYTHONPATH='.'
-
-
 Running Docker Demo
 
 Execute the following at the project root:
@@ -235,5 +253,3 @@ Execute the following at the project root:
     -e USER_KEY=<your user KEY> \
     -e SUBSCRIPTION_ID=<your subscription ID> \
     dj-dna-streaming-python
-
-

{dnaStreaming-2.0.5 → dnastreaming-2.2.0}/dnaStreaming.egg-info/SOURCES.txt

@@ -14,10 +14,12 @@ dnaStreaming.egg-info/requires.txt
 dnaStreaming.egg-info/top_level.txt
 dnaStreaming/demo/__init__.py
 dnaStreaming/demo/pubsub_stream.py
+dnaStreaming/demo/show_ha_stream_async.py
 dnaStreaming/demo/show_stream.py
 dnaStreaming/demo/show_stream_async.py
 dnaStreaming/services/__init__.py
 dnaStreaming/services/authentication_service.py
+dnaStreaming/services/availability_service.py
 dnaStreaming/services/credentials_service.py
 dnaStreaming/services/pubsub_service.py
 dnaStreaming/test/__init__.py

{dnaStreaming-2.0.5 → dnastreaming-2.2.0}/setup.py

@@ -6,7 +6,7 @@ this_directory = path.abspath(path.dirname(__file__))
 with open(path.join(this_directory, 'README.rst'), encoding='utf-8') as f:
     long_description = f.read()
 
-VERSION = "2.0.5"
+VERSION = "2.2.0"
 RELEASE_TAG = f"release-{VERSION}"
 
 setup(

dnaStreaming-2.0.5/dnaStreaming/config.py

@@ -1,202 +0,0 @@
-from __future__ import absolute_import, division
-
-import errno
-import json
-import os
-import requests
-
-class Config(object):
-    OAUTH_URL = 'https://accounts.dowjones.com/oauth2/v1/token'
-    DEFAULT_HOST = 'https://api.dowjones.com'
-
-    DEFAULT_CUST_CONFIG_PATH = os.path.abspath(os.path.join(os.path.dirname(__file__), './customer_config.json'))
-    ENV_VAR_SUBSCRIPTION_ID = 'SUBSCRIPTION_ID'
-    ENV_VAR_USER_KEY = 'USER_KEY'
-    ENV_VAR_SERVICE_ACCOUNT_ID = 'SERVICE_ACCOUNT_ID'
-    ENV_VAR_USER_ID = 'USER_ID'
-    ENV_VAR_CLIENT_ID = 'CLIENT_ID'
-    ENV_VAR_PASSWORD = 'PASSWORD'
-    ENV_VAR_API_HOST = 'API_HOST'
-
-    def __init__(self, service_account_id=None, user_key=None, user_id=None, client_id=None, password=None):
-        self.customer_config_path = self.DEFAULT_CUST_CONFIG_PATH
-        self.initialized = False
-        self.service_account_id = service_account_id
-        self.user_key = user_key
-        self.user_id = user_id
-        self.client_id = client_id
-        self.password = password
-
-        self.headers = None
-
-    def _initialize(self):
-        self._validate()
-
-        with open(self.customer_config_path, 'r') as f:
-            self.customer_config = json.load(f)
-
-        self.initialized = True
-        self.headers = None
-
-    def _validate(self):
-        if not os.path.isfile(self.customer_config_path):
-            raise FileNotFoundError(errno.ENOENT, os.strerror(errno.ENOENT), self.customer_config_path)
-
-        if not os.access(self.customer_config_path, os.R_OK):
-            raise Exception('Encountered permission problem reading file from path \'{}\'.'.format(self.customer_config_path))
-
-    def get_headers(self):
-        if self.headers:
-            return self.headers
-        else:
-            self.headers = self.get_authentication_headers()
-            return self.headers
-
-    def get_authentication_headers(self):
-        if self.oauth2_credentials():
-            return {
-                'Authorization': self._fetch_jwt()
-            }
-
-        # missing oauth creds, authenticate the old way via user key
-        user_key = self.get_user_key()
-        if user_key:
-            return {
-                'user-key': user_key
-            }
-
-        else:
-            msg = '''Could not find determine credentials:
-                Must specify account credentials as user_id, client_id, and password, either through env vars, customer_config.json, or as args to Listener constructor
-                (see README.rst)'''
-            raise Exception(msg)
-
-    def _fetch_jwt(self):
-        oauth2_credentials = self.oauth2_credentials()
-        user_id = oauth2_credentials.get('user_id')
-        client_id = oauth2_credentials.get('client_id')
-        password = oauth2_credentials.get('password')
-
-        # two requests need to be made, to the same URL, with slightly different params, to finally obtain a JWT
-        # the second request contains params returned in the response of the first request
-        # I know this makes no sense but it is what it is
-        body = {
-            'username': user_id,
-            'client_id': client_id,
-            'password': password,
-            'connection': 'service-account',
-            'grant_type': 'password',
-            'scope': 'openid service_account_id'
-        }
-
-        try:
-
-            response = _get_requests().post(self.OAUTH_URL, data=body).json()
-            body['scope'] = 'openid pib'
-            body['grant_type'] = 'urn:ietf:params:oauth:grant-type:jwt-bearer'
-            body['access_token'] = response.get('access_token')
-            body['assertion'] = response.get('id_token')
-
-            response = _get_requests().post(self.OAUTH_URL, data=body).json()
-
-            return '{0} {1}'.format(response['token_type'], response['access_token'])
-        except (KeyError, ValueError):
-            msg = '''Unable to retrieve JWT with the given credentials:
-                User ID: {0}
-                Client ID: {1}
-                Password: {2}
-            '''.format(user_id, client_id, password)
-            raise Exception(msg)
-
-    def get_uri_context(self):
-        headers = self.get_headers()
-        host = os.getenv(self.ENV_VAR_API_HOST, self.DEFAULT_HOST)
-        if "Authorization" in headers:
-            return host + '/dna'
-        elif 'user-key' in headers:
-            return host + '/alpha'
-        else:
-            msg = '''Could not determine user credentials:
-                Must specify account credentials as user_id, client_id, and password, either through env vars, customer_config.json, or as args to Listener constructor
-                (see README.rst)'''
-            raise Exception(msg)
-
-    # return credentials (user_id, client_id, and password) for obtaining a JWT via OAuth2 if all these fields are defined in the constructor, env vars or config file
-    # otherwise return None (the client will have to authenticate API request with an user key, i.e. the original standard)
-    def oauth2_credentials(self):
-        creds = self._build_oauth2_credentials(
-            self.user_id,
-            self.client_id,
-            self.password
-        )
-        if not creds:
-            creds = self._build_oauth2_credentials(
-                os.getenv(self.ENV_VAR_USER_ID),
-                os.getenv(self.ENV_VAR_CLIENT_ID),
-                os.getenv(self.ENV_VAR_PASSWORD)
-            )
-        if not creds:
-            creds = self._oauth2_credentials_from_file()
-        return creds
-
-    def _oauth2_credentials_from_file(self):
-        if not self.initialized:
-            self._initialize()
-
-        return self._build_oauth2_credentials(
-            self.customer_config.get('user_id'),
-            self.customer_config.get('client_id'),
-            self.customer_config.get('password')
-        )
-
-    def _build_oauth2_credentials(self, user_id, client_id, password):
-        if user_id and client_id and password:
-            return {
-                'user_id': user_id,
-                'client_id': client_id,
-                'password': password
-            }
-        return None
-
-    # in the following two methods, note that we use "SERVICE_ACCOUNT_ID" as a legacy,
-    # alternate name for the "USER_KEY" parameter, from the customer's perspective
-    def get_user_key(self):
-        user_key = self.user_key if self.user_key else self.service_account_id
-
-        if user_key is None:
-            user_key = os.getenv(self.ENV_VAR_USER_KEY,
-                                 os.getenv(self.ENV_VAR_SERVICE_ACCOUNT_ID))
-
-            if user_key is None:
-                user_key = self._user_key_id_from_file()
-
-        return user_key
-
-    def _user_key_id_from_file(self):
-        if not self.initialized:
-            self._initialize()
-
-        return self.customer_config.get('user_key',
-                                        self.customer_config.get('service_account_id'))
-
-    def subscription(self):
-        if os.getenv(self.ENV_VAR_SUBSCRIPTION_ID) is not None:
-            subscription = os.getenv(self.ENV_VAR_SUBSCRIPTION_ID)
-        else:
-            subscription = self._subscription_id_from_file()
-
-        return subscription
-
-    def _set_customer_config_path(self, path):
-        self.customer_config_path = path
-        self._initialize()
-
-    def _subscription_id_from_file(self):
-        if not self.initialized:
-            self._initialize()
-
-        return self.customer_config['subscription_id']
-
-
-def _get_requests():
-    return requests

dnaStreaming-2.0.5/dnaStreaming/customer_config.json

@@ -1,8 +0,0 @@
-{
-
-  "user_key": "",
-  "user_id": "",
-  "client_id": "",
-  "password": "",
-  "subscription_id": ""
-}

dnaStreaming-2.0.5/dnaStreaming/services/pubsub_service.py

@@ -1,13 +0,0 @@
-from __future__ import absolute_import, division, print_function
-
-from google.cloud.pubsub_v1 import SubscriberClient
-
-from dnaStreaming.services import authentication_service
-from dnaStreaming.services import credentials_service
-
-
-def get_client(config):
-    streaming_credentials = credentials_service.fetch_credentials(config)
-    credentials = authentication_service.get_authenticated_oauth_credentials(streaming_credentials)
-
-    return SubscriberClient(credentials=credentials)