domaintools-api 2.5.2__tar.gz → 2.6.0__tar.gz

{domaintools_api-2.5.2/domaintools_api.egg-info → domaintools_api-2.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: domaintools_api
-Version: 2.5.2
+Version: 2.6.0
 Summary: DomainTools Official Python API
 Author-email: DomainTools <integrations@domaintools.com>
 License: The MIT License (MIT)
@@ -265,25 +265,25 @@ Please see the [supported versions](https://github.com/DomainTools/python_api/ra
 for the DomainTools Python support policy.
 
 
-Real-Time Threat Intelligence Feeds
+Real-Time Threat Feeds
 ===================
 
-Real-Time Threat Intelligence Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs.
+Real-Time Threat Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs.
 
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY, always_sign_api_key=False)
+    api = API(USERNAME, KEY)
     api.nod(endpoint="feed", **kwargs)
     ```
 - `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so this will be defaulted to `False` even without explicitly setting it.
     ```python
-    api = API(USERNAME, KEY, always_sign_api_key=False)
-    api.nod(header_authentication=False, **kwargs)
+    api = API(USERNAME, KEY, header_authentication=False)
+    api.nod(**kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY, always_sign_api_key=False)
+    api = API(USERNAME, KEY)
     api.nod(output_format="csv", **kwargs)
     ```
 
@@ -304,7 +304,7 @@ Since we may dealing with large feeds datasets, the python wrapper uses `generat
 ```python
 from domaintools import API
 
-api = API(USERNAME, KEY, always_sign_api_key=False)
+api = API(USERNAME, KEY)
 results = api.nod(sessionID="my-session-id", after=-60)
 
 for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
@@ -315,7 +315,7 @@ for result in results.response() # generator that holds NOD feeds data for the p
 ```python
 from domaintools import API
 
-api = API(USERNAME, KEY, always_sign_api_key=False)
+api = API(USERNAME, KEY)
 results = api.nod(sessionID="my-session-id", after=-7200)
 
 for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
@@ -354,6 +354,10 @@ To add more e2e tests, put these in the `../tests/e2e` folder.
         export TEST_USER=<user-key>
         export TEST_KEY=<api-key>
     ```
+- Run unit tests.
+    ```bash
+        tox -e
+    ```
 
 ## Run the end-to-end test script
 - Before running the test, be sure that docker is running.

{domaintools_api-2.5.2 → domaintools_api-2.6.0}/README.md

@@ -215,25 +215,25 @@ Please see the [supported versions](https://github.com/DomainTools/python_api/ra
 for the DomainTools Python support policy.
 
 
-Real-Time Threat Intelligence Feeds
+Real-Time Threat Feeds
 ===================
 
-Real-Time Threat Intelligence Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs.
+Real-Time Threat Feeds provide data on the different stages of the domain lifecycle: from first-observed in the wild, to newly re-activated after a period of quiet. Access current feed data in real-time or retrieve historical feed data through separate APIs.
 
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY, always_sign_api_key=False)
+    api = API(USERNAME, KEY)
     api.nod(endpoint="feed", **kwargs)
     ```
 - `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so this will be defaulted to `False` even without explicitly setting it.
     ```python
-    api = API(USERNAME, KEY, always_sign_api_key=False)
-    api.nod(header_authentication=False, **kwargs)
+    api = API(USERNAME, KEY, header_authentication=False)
+    api.nod(**kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY, always_sign_api_key=False)
+    api = API(USERNAME, KEY)
     api.nod(output_format="csv", **kwargs)
     ```
 
@@ -254,7 +254,7 @@ Since we may dealing with large feeds datasets, the python wrapper uses `generat
 ```python
 from domaintools import API
 
-api = API(USERNAME, KEY, always_sign_api_key=False)
+api = API(USERNAME, KEY)
 results = api.nod(sessionID="my-session-id", after=-60)
 
 for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
@@ -265,7 +265,7 @@ for result in results.response() # generator that holds NOD feeds data for the p
 ```python
 from domaintools import API
 
-api = API(USERNAME, KEY, always_sign_api_key=False)
+api = API(USERNAME, KEY)
 results = api.nod(sessionID="my-session-id", after=-7200)
 
 for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
@@ -304,6 +304,10 @@ To add more e2e tests, put these in the `../tests/e2e` folder.
         export TEST_USER=<user-key>
         export TEST_KEY=<api-key>
     ```
+- Run unit tests.
+    ```bash
+        tox -e
+    ```
 
 ## Run the end-to-end test script
 - Before running the test, be sure that docker is running.

domaintools_api-2.6.0/VERSION

@@ -0,0 +1 @@
+2.6.0

{domaintools_api-2.5.2 → domaintools_api-2.6.0}/domaintools/_version.py

@@ -20,4 +20,4 @@ OTHER DEALINGS IN THE SOFTWARE.
 
 """
 
-current = "2.5.2"
+current = "2.6.0"

{domaintools_api-2.5.2 → domaintools_api-2.6.0}/domaintools/api.py

@@ -6,7 +6,13 @@ from typing import Union
 import re
 import ssl
 
-from domaintools.constants import Endpoint, ENDPOINT_TO_SOURCE_MAP, FEEDS_PRODUCTS_LIST, OutputFormat
+from domaintools.constants import (
+    Endpoint,
+    OutputFormat,
+    ENDPOINT_TO_SOURCE_MAP,
+    RTTF_PRODUCTS_LIST,
+    RTTF_PRODUCTS_CMD_MAPPING,
+)
 from domaintools._version import current as version
 from domaintools.results import (
     GroupedIterable,
@@ -63,7 +69,8 @@ class API(object):
         verify_ssl=True,
         rate_limit=True,
         proxy_url=None,
-        always_sign_api_key=True,
+        always_sign_api_key=None,
+        header_authentication=None,
         key_sign_hash="sha256",
         app_name="python_wrapper",
         app_version=version,
@@ -83,6 +90,7 @@ class API(object):
         self.proxy_url = proxy_url
         self.extra_request_params = {}
         self.always_sign_api_key = always_sign_api_key
+        self.header_authentication = header_authentication
         self.key_sign_hash = key_sign_hash
         self.default_parameters["app_name"] = app_name
         self.default_parameters["app_version"] = app_version
@@ -90,7 +98,9 @@ class API(object):
         self._build_api_url(api_url, api_port)
 
         if not https:
-            raise Exception("The DomainTools API endpoints no longer support http traffic. Please make sure https=True.")
+            raise Exception(
+                "The DomainTools API endpoints no longer support http traffic. Please make sure https=True."
+            )
         if proxy_url and not isinstance(proxy_url, str):
             raise Exception("Proxy URL must be a string. For example: '127.0.0.1:8888'")
 
@@ -108,8 +118,12 @@ class API(object):
 
         self._rest_api_url = rest_api_url
 
-    def _rate_limit(self):
+    def _rate_limit(self, product):
         """Pulls in and enforces the latest rate limits for the specified user"""
+        if product in RTTF_PRODUCTS_LIST:
+            self.limits_set = False
+            return
+
         self.limits_set = True
         for product in self.account_information():
             limit_minutes = product["per_minute_limit"] or None
@@ -124,24 +138,44 @@ class API(object):
     def _results(self, product, path, cls=Results, **kwargs):
         """Returns _results for the specified API path with the specified **kwargs parameters"""
         if product != "account-information" and self.rate_limit and not self.limits_set and not self.limits:
-            self._rate_limit()
+            always_sign_api_key_previous_value = self.always_sign_api_key
+            header_authentication_previous_value = self.header_authentication
+            self._rate_limit(product)
+            # Reset always_sign_api_key and header_authentication to its original
+            # User-set values as these might be affected when self.account_information() was executed
+            self.always_sign_api_key = always_sign_api_key_previous_value
+            self.header_authentication = header_authentication_previous_value
 
         uri = "/".join((self._rest_api_url, path.lstrip("/")))
         parameters = self.default_parameters.copy()
         parameters["api_username"] = self.username
-        header_authentication = kwargs.pop("header_authentication", True)  # Used only by Real-Time Threat Intelligence Feeds endpoints for now
-        self.handle_api_key(product, path, parameters, header_authentication)
-        parameters.update({key: str(value).lower() if value in (True, False) else value for key, value in kwargs.items() if value is not None})
+        is_rttf_product = product in RTTF_PRODUCTS_LIST
+        self._handle_api_key_parameters(is_rttf_product)
+        self.handle_api_key(is_rttf_product, path, parameters)
+        parameters.update(
+            {
+                key: str(value).lower() if value in (True, False) else value
+                for key, value in kwargs.items()
+                if value is not None
+            }
+        )
 
         return cls(self, product, uri, **parameters)
 
-    def handle_api_key(self, product, path, parameters, header_authentication):
+    def _handle_api_key_parameters(self, is_rttf_product):
+        if self.always_sign_api_key is None:
+            self.always_sign_api_key = not is_rttf_product
+
+        if self.header_authentication is None:
+            self.header_authentication = is_rttf_product
+
+    def handle_api_key(self, is_rttf_product, path, parameters):
         if self.https and not self.always_sign_api_key:
-            if product in FEEDS_PRODUCTS_LIST and header_authentication:
-                parameters["X-Api-Key"] = self.key
-            else:
-                parameters["api_key"] = self.key
+            parameters["api_key"] = self.key
         else:
+            if is_rttf_product:
+                # As per requirement in IDEV-2272, raise this error when the user explicitly sets signing of API key for RTTF endpoints
+                raise ValueError("Real Time Threat Feeds do not support signed API keys.")
             if self.key_sign_hash and self.key_sign_hash in AVAILABLE_KEY_SIGN_HASHES:
                 signing_hash = eval(self.key_sign_hash)
             else:
@@ -174,8 +208,30 @@ class API(object):
                 string[1:],
             )
 
-        api_calls = tuple((api_call for api_call in dir(API) if not api_call.startswith("_") and callable(getattr(API, api_call, None))))
-        return sorted([snakecase(p["id"]) for p in self.account_information()["products"] if snakecase(p["id"]) in api_calls])
+        api_calls = tuple(
+            (
+                api_call
+                for api_call in dir(API)
+                if not api_call.startswith("_") and callable(getattr(API, api_call, None))
+            )
+        )
+
+        account_information = self.account_information()
+
+        available_calls = set()
+        for product in self.account_information():
+            product_id = product["id"]
+            # for RTUF endpoints as we use different func name in our wrapper
+            if product_id in RTTF_PRODUCTS_LIST:
+                if rttf_api_command := RTTF_PRODUCTS_CMD_MAPPING.get(product_id):
+                    available_calls.add(rttf_api_command)
+
+            # for IRIS endpoints
+            snakecase_pid = snakecase(product_id)
+            if snakecase_pid in api_calls:
+                available_calls.add(snakecase_pid)
+
+        return sorted(available_calls)
 
     def brand_monitor(self, query, exclude=None, domain_status=None, days_back=None, **kwargs):
         """Pass in one or more terms as a list or separated by the pipe character ( | )"""
@@ -430,7 +486,16 @@ class API(object):
         """Performs a search for the provided search terms ANDed together,
         returning the pivot engine row data for the resulting domains.
         """
-        if not domain and not ip and not email and not nameserver and not registrar and not registrant and not registrant_org and not kwargs:
+        if (
+            not domain
+            and not ip
+            and not email
+            and not nameserver
+            and not registrar
+            and not registrant
+            and not registrant_org
+            and not kwargs
+        ):
             raise ValueError("At least one search term must be specified")
 
         return self._results(
@@ -1054,7 +1119,10 @@ class API(object):
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint)
-        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+        if (
+            endpoint == Endpoint.DOWNLOAD.value
+            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
+        ):
             # headers param is allowed only in Feed API and CSV format
             kwargs.pop("headers", None)
 
@@ -1086,7 +1154,10 @@ class API(object):
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+        if (
+            endpoint == Endpoint.DOWNLOAD.value
+            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
+        ):
             # headers param is allowed only in Feed API and CSV format
             kwargs.pop("headers", None)
 
@@ -1147,7 +1218,10 @@ class API(object):
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+        if (
+            endpoint == Endpoint.DOWNLOAD.value
+            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
+        ):
             # headers param is allowed only in Feed API and CSV format
             kwargs.pop("headers", None)
 
@@ -1179,7 +1253,10 @@ class API(object):
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+        if (
+            endpoint == Endpoint.DOWNLOAD.value
+            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
+        ):
             # headers param is allowed only in Feed API and CSV format
             kwargs.pop("headers", None)
 
@@ -1210,12 +1287,15 @@ class API(object):
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+        if (
+            endpoint == Endpoint.DOWNLOAD.value
+            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
+        ):
             # headers param is allowed only in Feed API and CSV format
             kwargs.pop("headers", None)
 
         return self._results(
-            f"domain-risk-feed-({source})",
+            f"real-time-domain-risk-({source})",
             f"v1/{endpoint}/domainrisk/",
             response_path=(),
             cls=FeedsResults,
@@ -1241,12 +1321,15 @@ class API(object):
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
         source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
-        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+        if (
+            endpoint == Endpoint.DOWNLOAD.value
+            or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value
+        ):
             # headers param is allowed only in Feed API and CSV format
             kwargs.pop("headers", None)
 
         return self._results(
-            f"domain-hotlist-feed-({source})",
+            f"real-time-domain-hotlist-({source})",
             f"v1/{endpoint}/domainhotlist/",
             response_path=(),
             cls=FeedsResults,

{domaintools_api-2.5.2 → domaintools_api-2.6.0}/domaintools/base_results.py

@@ -9,7 +9,11 @@ from copy import deepcopy
 from datetime import datetime
 from httpx import Client
 
-from domaintools.constants import FEEDS_PRODUCTS_LIST, OutputFormat, HEADER_ACCEPT_KEY_CSV_FORMAT
+from domaintools.constants import (
+    RTTF_PRODUCTS_LIST,
+    OutputFormat,
+    HEADER_ACCEPT_KEY_CSV_FORMAT,
+)
 from domaintools.exceptions import (
     BadRequestException,
     InternalServerErrorException,
@@ -53,6 +57,7 @@ class Results(MutableMapping, MutableSequence):
         self._response = None
         self._items_list = None
         self._data = None
+        self._status = None
 
     def _wait_time(self):
         if not self.api.rate_limit or not self.product in self.api.limits:
@@ -75,26 +80,30 @@ class Results(MutableMapping, MutableSequence):
 
         return wait_for
 
-    def _get_session_params(self):
-        parameters = deepcopy(self.kwargs)
-        parameters.pop("output_format", None)
-        parameters.pop(
-            "format", None
-        )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
+    def _get_session_params_and_headers(self):
         headers = {}
-        if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
-            parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
-            headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
-
-        header_api_key = parameters.pop("X-Api-Key", None)
-        if header_api_key:
-            headers["X-Api-Key"] = header_api_key
-
-        return {"parameters": parameters, "headers": headers}
+        parameters = deepcopy(self.kwargs)
+        is_rttf_product = self.product in RTTF_PRODUCTS_LIST
+        if is_rttf_product:
+            parameters.pop("output_format", None)
+            parameters.pop(
+                "format", None
+            )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
+            if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
+                parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
+                headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
+
+        if self.api.header_authentication:
+            header_key_for_api_key = "X-Api-Key" if is_rttf_product else "X-API-Key"
+            headers[header_key_for_api_key] = self.api.key
+
+        session_param_and_headers = {"parameters": parameters, "headers": headers}
+        return session_param_and_headers
 
     def _make_request(self):
-
         with Client(verify=self.api.verify_ssl, proxy=self.api.proxy_url, timeout=None) as session:
+            session_params_and_headers = self._get_session_params_and_headers()
+            headers = session_params_and_headers.get("headers")
             if self.product in [
                 "iris-investigate",
                 "iris-enrich",
@@ -102,18 +111,19 @@ class Results(MutableMapping, MutableSequence):
             ]:
                 post_data = self.kwargs.copy()
                 post_data.update(self.api.extra_request_params)
-                return session.post(url=self.url, data=post_data)
+                return session.post(url=self.url, data=post_data, headers=headers)
             elif self.product in ["iris-detect-manage-watchlist-domains"]:
                 patch_data = self.kwargs.copy()
                 patch_data.update(self.api.extra_request_params)
-                return session.patch(url=self.url, json=patch_data)
-            elif self.product in FEEDS_PRODUCTS_LIST:
-                session_params = self._get_session_params()
-                parameters = session_params.get("parameters")
-                headers = session_params.get("headers")
-                return session.get(url=self.url, params=parameters, headers=headers, **self.api.extra_request_params)
+                return session.patch(url=self.url, json=patch_data, headers=headers)
             else:
-                return session.get(url=self.url, params=self.kwargs, **self.api.extra_request_params)
+                parameters = session_params_and_headers.get("parameters")
+                return session.get(
+                    url=self.url,
+                    params=parameters,
+                    headers=headers,
+                    **self.api.extra_request_params,
+                )
 
     def _get_results(self):
         wait_for = self._wait_time()
@@ -152,7 +162,9 @@ class Results(MutableMapping, MutableSequence):
     def check_limit_exceeded(self):
         limit_exceeded, reason = False, ""
         if isinstance(self._data, dict) and (
-            "response" in self._data and "limit_exceeded" in self._data["response"] and self._data["response"]["limit_exceeded"] is True
+            "response" in self._data
+            and "limit_exceeded" in self._data["response"]
+            and self._data["response"]["limit_exceeded"] is True
         ):
             limit_exceeded, reason = True, self._data["response"]["message"]
         elif "response" in self._data and "limit_exceeded" in self._data:
@@ -163,14 +175,14 @@ class Results(MutableMapping, MutableSequence):
 
     @property
     def status(self):
-        if not getattr(self, "_status", None):
+        if not getattr(self, "_status", None) and not self.product in RTTF_PRODUCTS_LIST:
             self._status = self._get_results().status_code
 
         return self._status
 
-    def setStatus(self, code, response=None):
+    def setStatus(self, code, response=None, reason_text=None):
         self._status = code
-        if code == 200 or (self.product in FEEDS_PRODUCTS_LIST and code == 206):
+        if code == 200 or (self.product in RTTF_PRODUCTS_LIST and code == 206):
             return
 
         reason = None
@@ -181,6 +193,9 @@ class Results(MutableMapping, MutableSequence):
                 reason = response.text
                 if callable(reason):
                     reason = reason()
+        else:  # optionally pass a customize reason of error for better traceback
+            if reason_text is not None:
+                reason = reason_text
 
         if code in (400, 422):
             raise BadRequestException(code, reason)
@@ -330,4 +345,8 @@ class Results(MutableMapping, MutableSequence):
         return "\n".join([json.dumps(item, indent=4, separators=(",", ": ")) for item in self._items()])
 
     def __str__(self):
-        return str(json.dumps(self.data(), indent=4, separators=(",", ": ")) if self.kwargs.get("format", "json") == "json" else self.data())
+        return str(
+            json.dumps(self.data(), indent=4, separators=(",", ": "))
+            if self.kwargs.get("format", "json") == "json"
+            else self.data()
+        )

{domaintools_api-2.5.2 → domaintools_api-2.6.0}/domaintools/cli/api.py

@@ -9,7 +9,7 @@ from typing import Optional, Dict, Tuple
 from rich.progress import Progress, SpinnerColumn, TextColumn
 
 from domaintools.api import API
-from domaintools.constants import Endpoint, FEEDS_PRODUCTS_LIST, OutputFormat
+from domaintools.constants import Endpoint, RTTF_PRODUCTS_LIST, OutputFormat
 from domaintools.cli.utils import get_file_extension
 from domaintools.exceptions import ServiceException
 from domaintools._version import current as version
@@ -58,7 +58,9 @@ class DTCLICommand:
             datetime.strptime(value, "%Y-%m-%dT%H:%M:%SZ")
             return value
         except:
-            raise typer.BadParameter(f"{value} is neither an integer or a valid ISO 8601 datetime string in UTC form")
+            raise typer.BadParameter(
+                f"{value} is neither an integer or a valid ISO 8601 datetime string in UTC form"
+            )
 
     @staticmethod
     def validate_source_file_extension(value: str):
@@ -78,7 +80,9 @@ class DTCLICommand:
         ext = get_file_extension(value)
 
         if ext.lower() not in VALID_EXTENSIONS:
-            raise typer.BadParameter(f"{value} is not in valid extensions. Valid file extensions: {VALID_EXTENSIONS}")
+            raise typer.BadParameter(
+                f"{value} is not in valid extensions. Valid file extensions: {VALID_EXTENSIONS}"
+            )
 
         return value
 
@@ -110,8 +114,8 @@ class DTCLICommand:
     def _get_formatted_output(cls, cmd_name: str, response, out_format: str = "json"):
         if cmd_name in ("available_api_calls",):
             return "\n".join(response)
-        if response.product in FEEDS_PRODUCTS_LIST:
-            return "\n".join([data for data in response.response()])
+        if response.product in RTTF_PRODUCTS_LIST:
+            pass  # do nothing
         return str(getattr(response, out_format) if out_format != "list" else response.as_list())
 
     @classmethod
@@ -180,6 +184,7 @@ class DTCLICommand:
             out_file = params.pop("out_file", sys.stdout)
             verify_ssl = params.pop("no_verify_ssl", False)
             always_sign_api_key = params.pop("no_sign_api_key", False)
+            header_authentication = params.pop("no_header_authentication", False)
             source = None
 
             if "src_file" in params:
@@ -202,7 +207,7 @@ class DTCLICommand:
                 transient=True,
             ) as progress:
 
-                progress.add_task(
+                task_id = progress.add_task(
                     description=f"Using api credentials with a username of: [cyan]{user}[/cyan]\nExecuting [green]{name}[/green] api call...",
                     total=None,
                 )
@@ -214,29 +219,39 @@ class DTCLICommand:
                     verify_ssl=verify_ssl,
                     rate_limit=rate_limit,
                     always_sign_api_key=always_sign_api_key,
+                    header_authentication=header_authentication,
                 )
                 dt_api_func = getattr(dt_api, name)
-
                 params = params | kwargs
 
                 response = dt_api_func(**params)
-                progress.add_task(
+                progress.update(
+                    task_id,
                     description=f"Preparing results with format of {response_format}...",
-                    total=None,
                 )
 
-                output = cls._get_formatted_output(cmd_name=name, response=response, out_format=response_format)
+                output = cls._get_formatted_output(
+                    cmd_name=name, response=response, out_format=response_format
+                )
 
                 if isinstance(out_file, _io.TextIOWrapper):
+                    progress.update(
+                        task_id,
+                        description=f"Printing the results with format of {response_format}...",
+                    )
                     # use rich `print` command to prettify the ouput in sys.stdout
-                    if response.product in FEEDS_PRODUCTS_LIST:
-                        print(output)
+                    if name not in ("available_api_calls",) and response.product in RTTF_PRODUCTS_LIST:
+                        for feeds in response.response():
+                            print(feeds)
                     else:
                         print(response)
                 else:
+                    progress.update(
+                        task_id,
+                        description=f"Writing results to {out_file}",
+                    )
                     # if it's a file then write
                     out_file.write(output if output.endswith("\n") else output + "\n")
-                time.sleep(0.25)
         except Exception as e:
             if isinstance(e, ServiceException):
                 code = typer.style(getattr(e, "code", 400), fg=typer.colors.BRIGHT_RED)