domaintools-api-test-upload 2.2.1__tar.gz → 2.2.2__tar.gz

{domaintools_api_test_upload-2.2.1/domaintools_api_test_upload.egg-info → domaintools_api_test_upload-2.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: domaintools_api_test_upload
-Version: 2.2.1
+Version: 2.2.2
 Summary: DomainTools Official Python API
 Author-email: DomainTools <integrations@domaintools.com>
 License: The MIT License (MIT)
@@ -272,16 +272,51 @@ Real-Time Threat Intelligence Feeds provide data on the different stages of the
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(endpoint="feed", **kwargs)
     ```
 - `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so you need to set this to `False` when calling `download` API endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(header_authentication=False, **kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(output_format="csv", **kwargs)
     ```
+
+The Feed API standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds. Specify the range of data you receive in one of two ways:
+
+1. With `sessionID`: Make a call and provide a new `sessionID` parameter of your choosing. The API will return the last hour of data by default.
+    - Each subsequent call to the API using your `sessionID` will return all data since the last.
+    - Any single request returns a maximum of 10M results. Requests that exceed 10M results will return a HTTP 206 response code; repeat the same request (with the same `sessionID`) to receive the next tranche of data until receiving a HTTP 200 response code.
+2. Or, specify the time range in one of two ways:
+    - Either an `after=-60` query parameter, where (in this example) -60 indicates the previous 60 seconds.
+    - Or `after` and `before` query parameters for a time range, with each parameter accepting an ISO-8601 UTC formatted timestamp (a UTC date and time of the format YYYY-MM-DDThh:mm:ssZ)
+
+## Handling iterative response from RTUF endpoints:
+
+Since we may dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
+
+### Single request because the requested data is within the maximum result:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-60)
+
+for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
+    # do things to result
+```
+
+## Multiple requests because the requested data is more than the maximum result per request:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-7200)
+
+for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
+    # do things to partial_result
+```

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/README.md

@@ -223,16 +223,51 @@ Real-Time Threat Intelligence Feeds provide data on the different stages of the
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(endpoint="feed", **kwargs)
     ```
 - `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so you need to set this to `False` when calling `download` API endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(header_authentication=False, **kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(output_format="csv", **kwargs)
     ```
+
+The Feed API standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds. Specify the range of data you receive in one of two ways:
+
+1. With `sessionID`: Make a call and provide a new `sessionID` parameter of your choosing. The API will return the last hour of data by default.
+    - Each subsequent call to the API using your `sessionID` will return all data since the last.
+    - Any single request returns a maximum of 10M results. Requests that exceed 10M results will return a HTTP 206 response code; repeat the same request (with the same `sessionID`) to receive the next tranche of data until receiving a HTTP 200 response code.
+2. Or, specify the time range in one of two ways:
+    - Either an `after=-60` query parameter, where (in this example) -60 indicates the previous 60 seconds.
+    - Or `after` and `before` query parameters for a time range, with each parameter accepting an ISO-8601 UTC formatted timestamp (a UTC date and time of the format YYYY-MM-DDThh:mm:ssZ)
+
+## Handling iterative response from RTUF endpoints:
+
+Since we may dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
+
+### Single request because the requested data is within the maximum result:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-60)
+
+for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
+    # do things to result
+```
+
+## Multiple requests because the requested data is more than the maximum result per request:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-7200)
+
+for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
+    # do things to partial_result
+```

domaintools_api_test_upload-2.2.2/VERSION

@@ -0,0 +1 @@
+2.2.2

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/_version.py

@@ -20,4 +20,4 @@ OTHER DEALINGS IN THE SOFTWARE.
 
 """
 
-current = "2.2.1"
+current = "2.2.2"

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/api.py

@@ -1,6 +1,7 @@
 from datetime import datetime, timedelta, timezone
 from hashlib import sha1, sha256, md5
 from hmac import new as hmac
+
 import re
 
 from domaintools.constants import Endpoint, ENDPOINT_TO_SOURCE_MAP, FEEDS_PRODUCTS_LIST, OutputFormat
@@ -11,6 +12,7 @@ from domaintools.results import (
     ParsedDomainRdap,
     Reputation,
     Results,
+    FeedsResults,
 )
 from domaintools.filters import (
     filter_by_riskscore,
@@ -1065,7 +1067,7 @@ class API(object):
             **kwargs,
         )
 
-    def nod(self, **kwargs):
+    def nod(self, **kwargs) -> FeedsResults:
         """Returns back list of the newly observed domains feed"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1078,10 +1080,11 @@ class API(object):
             f"newly-observed-domains-feed-({source.value})",
             f"v1/{endpoint}/nod/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )
 
-    def nad(self, **kwargs):
+    def nad(self, **kwargs) -> FeedsResults:
         """Returns back list of the newly active domains feed"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1094,10 +1097,11 @@ class API(object):
             f"newly-active-domains-feed-({source})",
             f"v1/{endpoint}/nad/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )
 
-    def domainrdap(self, **kwargs):
+    def domainrdap(self, **kwargs) -> FeedsResults:
         """Returns changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP)"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1107,10 +1111,11 @@ class API(object):
             f"domain-registration-data-access-protocol-feed-({source})",
             f"v1/{endpoint}/domainrdap/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )
 
-    def domaindiscovery(self, **kwargs):
+    def domaindiscovery(self, **kwargs) -> FeedsResults:
         """Returns new domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1123,5 +1128,6 @@ class API(object):
             f"real-time-domain-discovery-feed-({source})",
             f"v1/{endpoint}/domaindiscovery/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/base_results.py

@@ -53,8 +53,6 @@ class Results(MutableMapping, MutableSequence):
         self._response = None
         self._items_list = None
         self._data = None
-        self._limit_exceeded = None
-        self._limit_exceeded_message = None
 
     def _wait_time(self):
         if not self.api.rate_limit or not self.product in self.api.limits:
@@ -77,6 +75,23 @@ class Results(MutableMapping, MutableSequence):
 
         return wait_for
 
+    def _get_session_params(self):
+        parameters = deepcopy(self.kwargs)
+        parameters.pop("output_format", None)
+        parameters.pop(
+            "format", None
+        )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
+        headers = {}
+        if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
+            parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
+            headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
+
+        header_api_key = parameters.pop("X-Api-Key", None)
+        if header_api_key:
+            headers["X-Api-Key"] = header_api_key
+
+        return {"parameters": parameters, "headers": headers}
+
     def _make_request(self):
 
         with Client(verify=self.api.verify_ssl, proxy=self.api.proxy_url, timeout=None) as session:
@@ -93,20 +108,9 @@ class Results(MutableMapping, MutableSequence):
                 patch_data.update(self.api.extra_request_params)
                 return session.patch(url=self.url, json=patch_data)
             elif self.product in FEEDS_PRODUCTS_LIST:
-                parameters = deepcopy(self.kwargs)
-                parameters.pop("output_format", None)
-                parameters.pop(
-                    "format", None
-                )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
-                headers = {}
-                if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
-                    parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
-                    headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
-
-                header_api_key = parameters.pop("X-Api-Key", None)
-                if header_api_key:
-                    headers["X-Api-Key"] = header_api_key
-
+                session_params = self._get_session_params()
+                parameters = session_params.get("parameters")
+                headers = session_params.get("headers")
                 return session.get(url=self.url, params=parameters, headers=headers, **self.api.extra_request_params)
             else:
                 return session.get(url=self.url, params=self.kwargs, **self.api.extra_request_params)
@@ -136,32 +140,26 @@ class Results(MutableMapping, MutableSequence):
         if self._data is None:
             results = self._get_results()
             self.setStatus(results.status_code, results)
-            if (
-                self.kwargs.get("format", "json") == "json"
-                and self.product not in FEEDS_PRODUCTS_LIST  # Special handling of feeds products' data to preserve the result in jsonline format
-            ):
+            if self.kwargs.get("format", "json") == "json":
                 self._data = results.json()
             else:
                 self._data = results.text
-            limit_exceeded, message = self.check_limit_exceeded()
 
-            if limit_exceeded:
-                self._limit_exceeded = True
-                self._limit_exceeded_message = message
+        self.check_limit_exceeded()
 
-        if self._limit_exceeded is True:
-            raise ServiceException(503, "Limit Exceeded{}".format(self._limit_exceeded_message))
-        else:
-            return self._data
+        return self._data
 
     def check_limit_exceeded(self):
-        if self.kwargs.get("format", "json") == "json" and self.product not in FEEDS_PRODUCTS_LIST:
-            if "response" in self._data and "limit_exceeded" in self._data["response"] and self._data["response"]["limit_exceeded"] is True:
-                return True, self._data["response"]["message"]
-        # TODO: handle html, xml response errors better.
+        limit_exceeded, reason = False, ""
+        if isinstance(self._data, dict) and (
+            "response" in self._data and "limit_exceeded" in self._data["response"] and self._data["response"]["limit_exceeded"] is True
+        ):
+            limit_exceeded, reason = True, self._data["response"]["message"]
         elif "response" in self._data and "limit_exceeded" in self._data:
-            return True, "limit exceeded"
-        return False, ""
+            limit_exceeded = True
+
+        if limit_exceeded:
+            raise ServiceException(503, f"Limit Exceeded {reason}")
 
     @property
     def status(self):
@@ -172,7 +170,7 @@ class Results(MutableMapping, MutableSequence):
 
     def setStatus(self, code, response=None):
         self._status = code
-        if code == 200:
+        if code == 200 or (self.product in FEEDS_PRODUCTS_LIST and code == 206):
             return
 
         reason = None
@@ -184,9 +182,9 @@ class Results(MutableMapping, MutableSequence):
                 if callable(reason):
                     reason = reason()
 
-        if code == 400:
+        if code in (400, 422):
             raise BadRequestException(code, reason)
-        elif code == 403:
+        elif code in (401, 403):
             raise NotAuthorizedException(code, reason)
         elif code == 404:
             raise NotFoundException(code, reason)

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/cli/api.py

@@ -9,7 +9,7 @@ from typing import Optional, Dict, Tuple
 from rich.progress import Progress, SpinnerColumn, TextColumn
 
 from domaintools.api import API
-from domaintools.constants import Endpoint, OutputFormat
+from domaintools.constants import Endpoint, FEEDS_PRODUCTS_LIST, OutputFormat
 from domaintools.cli.utils import get_file_extension
 from domaintools.exceptions import ServiceException
 from domaintools._version import current as version
@@ -110,6 +110,8 @@ class DTCLICommand:
     def _get_formatted_output(cls, cmd_name: str, response, out_format: str = "json"):
         if cmd_name in ("available_api_calls",):
             return "\n".join(response)
+        if response.product in FEEDS_PRODUCTS_LIST:
+            return "\n".join([data for data in response.response()])
         return str(getattr(response, out_format) if out_format != "list" else response.as_list())
 
     @classmethod
@@ -227,7 +229,10 @@ class DTCLICommand:
 
                 if isinstance(out_file, _io.TextIOWrapper):
                     # use rich `print` command to prettify the ouput in sys.stdout
-                    print(response)
+                    if response.product in FEEDS_PRODUCTS_LIST:
+                        print(output)
+                    else:
+                        print(response)
                 else:
                     # if it's a file then write
                     out_file.write(output if output.endswith("\n") else output + "\n")

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/cli/utils.py

@@ -85,6 +85,7 @@ _command_help_mapper = {
     c.FEEDS_NAD: "Returns back newly active domains feed.",
     c.FEEDS_NOD: "Returns back newly observed domains feed.",
     c.FEEDS_DOMAINRDAP: "Returns changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP).",
+    c.FEEDS_DOMAINDISCOVERY: "Returns new domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties.",
 }
 
 

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/results.py

@@ -11,6 +11,8 @@ except ImportError:  # pragma: no cover
     from ordereddict import OrderedDict
 
 from itertools import zip_longest
+from typing import Generator
+
 from domaintools_async import AsyncResults as Results
 
 
@@ -141,3 +143,28 @@ class ParsedDomainRdap(Results):
                     flat[f"contact_{contact_key}_{i}"] = " | ".join(contact_value) if type(contact_value) in (list, tuple) else contact_value
 
         return flat
+
+
+class FeedsResults(Results):
+    """Returns the generator for feeds results"""
+
+    def response(self) -> Generator:
+        status_code = None
+        while status_code != 200:
+            resp_data = self.data()
+            status_code = self.status
+            yield resp_data
+
+            self._data = None  # clear the data here
+            if not self.kwargs.get("sessionID"):
+                # we'll only do iterative request for queries that has sessionID.
+                # Otherwise, we will have an infinite request if sessionID was not provided but the required data asked is more than the maximum (1 hour of data)
+                break
+
+    def data(self):
+        results = self._get_results()
+        self.setStatus(results.status_code, results)
+        self._data = results.text
+        self.check_limit_exceeded()
+
+        return self._data

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools/utils.py

@@ -177,8 +177,12 @@ def validate_feeds_parameters(params):
     after = params.get("after")
     before = params.get("before")
     if not (sessionID or after or before):
-        raise ValueError("sessionID or after or before must be defined")
+        raise ValueError("sessionID or after or before must be provided")
 
     format = params.get("output_format")
-    if params.get("endpoint") == Endpoint.DOWNLOAD.value and format == OutputFormat.CSV.value:
+    endpoint = params.get("endpoint")
+    if endpoint == Endpoint.DOWNLOAD.value and format == OutputFormat.CSV.value:
         raise ValueError(f"{format} format is not available in {Endpoint.DOWNLOAD.value} API.")
+
+    if endpoint == Endpoint.DOWNLOAD.value and params.get("header_authentication", True):
+        raise ValueError(f"{Endpoint.DOWNLOAD.value} API does not support header authentication. Provide api_key in the parameter")

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2/domaintools_api_test_upload.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: domaintools_api_test_upload
-Version: 2.2.1
+Version: 2.2.2
 Summary: DomainTools Official Python API
 Author-email: DomainTools <integrations@domaintools.com>
 License: The MIT License (MIT)
@@ -272,16 +272,51 @@ Real-Time Threat Intelligence Feeds provide data on the different stages of the
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(endpoint="feed", **kwargs)
     ```
 - `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so you need to set this to `False` when calling `download` API endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(header_authentication=False, **kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(output_format="csv", **kwargs)
     ```
+
+The Feed API standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds. Specify the range of data you receive in one of two ways:
+
+1. With `sessionID`: Make a call and provide a new `sessionID` parameter of your choosing. The API will return the last hour of data by default.
+    - Each subsequent call to the API using your `sessionID` will return all data since the last.
+    - Any single request returns a maximum of 10M results. Requests that exceed 10M results will return a HTTP 206 response code; repeat the same request (with the same `sessionID`) to receive the next tranche of data until receiving a HTTP 200 response code.
+2. Or, specify the time range in one of two ways:
+    - Either an `after=-60` query parameter, where (in this example) -60 indicates the previous 60 seconds.
+    - Or `after` and `before` query parameters for a time range, with each parameter accepting an ISO-8601 UTC formatted timestamp (a UTC date and time of the format YYYY-MM-DDThh:mm:ssZ)
+
+## Handling iterative response from RTUF endpoints:
+
+Since we may dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
+
+### Single request because the requested data is within the maximum result:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-60)
+
+for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
+    # do things to result
+```
+
+## Multiple requests because the requested data is more than the maximum result per request:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-7200)
+
+for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
+    # do things to partial_result
+```

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/domaintools_async/__init__.py

@@ -52,20 +52,9 @@ class AsyncResults(Results):
             patch_data.update(self.api.extra_request_params)
             results = await session.patch(url=self.url, json=patch_data)
         elif self.product in FEEDS_PRODUCTS_LIST:
-            parameters = deepcopy(self.kwargs)
-            parameters.pop("output_format", None)
-            parameters.pop(
-                "format", None
-            )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
-            headers = {}
-            if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
-                parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
-                headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
-
-            header_api_key = parameters.pop("X-Api-Key", None)
-            if header_api_key:
-                headers["X-Api-Key"] = header_api_key
-
+            session_params = self._get_session_params()
+            parameters = session_params.get("parameters")
+            headers = session_params.get("headers")
             results = await session.get(url=self.url, params=parameters, headers=headers, **self.api.extra_request_params)
         else:
             results = await session.get(url=self.url, params=self.kwargs, **self.api.extra_request_params)
@@ -75,11 +64,8 @@ class AsyncResults(Results):
                 self._data = results.json()
             else:
                 self._data = results.text()
-            limit_exceeded, message = self.check_limit_exceeded()
 
-            if limit_exceeded:
-                self._limit_exceeded = True
-                self._limit_exceeded_message = message
+            self.check_limit_exceeded()
 
     async def __awaitable__(self):
         if self._data is None:

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/setup.py

@@ -37,7 +37,8 @@ if CYTHON:
         return module_names
 
     ext_modules = [
-        Extension("domaintools." + ext, [path.join("domaintools", ext + ".py")]) for ext in list_modules(path.join(MYDIR, "domaintools"))
+        Extension("domaintools." + ext, [path.join("domaintools", ext + ".py")])
+        for ext in list_modules(path.join(MYDIR, "domaintools"))
     ]
     cmdclass["build_ext"] = build_ext
 

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/tests/test_api.py

@@ -5,8 +5,10 @@ from os import environ
 import json
 import pytest
 
+from inspect import isgenerator
+
 from domaintools import API, exceptions
-from tests.settings import api, vcr
+from tests.settings import api, feeds_api, vcr
 
 
 @vcr.use_cassette
@@ -529,87 +531,139 @@ def test_limit_exceeded():
 
 @vcr.use_cassette
 def test_newly_observed_domains_feed():
-    results = api.nod(after="-60")
-    response = results.response()
-    rows = response.strip().split("\n")
+    results = feeds_api.nod(after="-60", header_authentication=False)
+    for response in results.response():
+        assert results.status == 200
+
+        rows = response.strip().split("\n")
+        assert response is not None
+        assert len(rows) >= 1
+
+        for row in rows:
+            feed_result = json.loads(row)
+            assert "timestamp" in feed_result.keys()
+            assert "domain" in feed_result.keys()
+
+
+@vcr.use_cassette
+def test_newly_observed_domains_feed_pagination():
+    results = feeds_api.nod(sessionID="integrations-testing", after="2025-01-16T10:20:00Z")
+    page_count = 0
+    for response in results.response():
+        rows = response.strip().split("\n")
+        assert response is not None
+        assert len(rows) >= 1
+
+        page_count += 1
 
-    assert response is not None
-    assert results.status == 200
-    assert len(rows) >= 1
+        for row in rows:
+            feed_result = json.loads(row)
+            assert "timestamp" in feed_result.keys()
+            assert "domain" in feed_result.keys()
 
-    for row in rows:
-        feed_result = json.loads(row)
-        assert "timestamp" in feed_result.keys()
-        assert "domain" in feed_result.keys()
+    assert page_count > 1
 
 
 @vcr.use_cassette
 def test_newly_active_domains_feed():
-    results = api.nad(after="-60")
-    response = results.response()
-    rows = response.strip().split("\n")
+    results = feeds_api.nad(after="-60", header_authentication=False)
+    for response in results.response():
+        assert results.status == 200
 
-    assert response is not None
-    assert results.status == 200
-    assert len(rows) >= 1
+        rows = response.strip().split("\n")
+        assert response is not None
+        assert len(rows) >= 1
 
-    for row in rows:
-        feed_result = json.loads(row)
-        assert "timestamp" in feed_result.keys()
-        assert "domain" in feed_result.keys()
+        for row in rows:
+            feed_result = json.loads(row)
+            assert "timestamp" in feed_result.keys()
+            assert "domain" in feed_result.keys()
 
 
 @vcr.use_cassette
 def test_domainrdap_feed():
-    results = api.domainrdap(after="-60", sessiondID="integrations-testing", top=5)
-    response = results.response()
-    rows = response.strip().split("\n")
+    results = feeds_api.domainrdap(after="-60", top=2, header_authenticationn=False)
+    for response in results.response():
+        assert results.status == 200
 
-    assert response is not None
-    assert results.status == 200
-    assert len(rows) == 5
+        rows = response.strip().split("\n")
 
-    for row in rows:
-        feed_result = json.loads(row)
-        assert "timestamp" in feed_result.keys()
-        assert "domain" in feed_result.keys()
-        assert "parsed_record" in feed_result.keys()
-        assert "domain" in feed_result["parsed_record"]["parsed_fields"]
-        assert "emails" in feed_result["parsed_record"]["parsed_fields"]
-        assert "contacts" in feed_result["parsed_record"]["parsed_fields"]
+        assert response is not None
+        assert len(rows) == 2
+
+        for row in rows:
+            feed_result = json.loads(row)
+            assert "timestamp" in feed_result.keys()
+            assert "domain" in feed_result.keys()
+            assert "parsed_record" in feed_result.keys()
+            assert "domain" in feed_result["parsed_record"]["parsed_fields"]
+            assert "emails" in feed_result["parsed_record"]["parsed_fields"]
+            assert "contacts" in feed_result["parsed_record"]["parsed_fields"]
 
 
 @vcr.use_cassette
 def test_domain_discovery_feed():
-    results = api.domaindiscovery(after="-60")
-    response = results.response()
-    rows = response.strip().split("\n")
+    results = feeds_api.domaindiscovery(after="-60", header_authentication=False)
+    for response in results.response():
+        assert results.status == 200
 
-    assert response is not None
-    assert results.status == 200
-    assert len(rows) >= 1
+        rows = response.strip().split("\n")
+        assert response is not None
+        assert len(rows) >= 1
 
-    for row in rows:
-        feed_result = json.loads(row)
-        assert "timestamp" in feed_result.keys()
-        assert "domain" in feed_result.keys()
+        for row in rows:
+            feed_result = json.loads(row)
+            assert "timestamp" in feed_result.keys()
+            assert "domain" in feed_result.keys()
 
 
 @vcr.use_cassette
 def test_domainrdap_feed_not_api_header_auth():
-    results = api.domainrdap(after="-60", sessiondID="integrations-testing", top=5, header_authentication=False)
-    response = results.response()
-    rows = response.strip().split("\n")
-
-    assert response is not None
-    assert results.status == 200
-    assert len(rows) == 5
-
-    for row in rows:
-        feed_result = json.loads(row)
-        assert "timestamp" in feed_result.keys()
-        assert "domain" in feed_result.keys()
-        assert "parsed_record" in feed_result.keys()
-        assert "domain" in feed_result["parsed_record"]["parsed_fields"]
-        assert "emails" in feed_result["parsed_record"]["parsed_fields"]
-        assert "contacts" in feed_result["parsed_record"]["parsed_fields"]
+    results = feeds_api.domainrdap(after="-60", sessiondID="integrations-testing", top=5, header_authenticationn=False)
+    for response in results.response():
+        assert results.status == 200
+
+        rows = response.strip().split("\n")
+
+        assert response is not None
+        assert len(rows) == 5
+
+        for row in rows:
+            feed_result = json.loads(row)
+            assert "timestamp" in feed_result.keys()
+            assert "domain" in feed_result.keys()
+            assert "parsed_record" in feed_result.keys()
+            assert "domain" in feed_result["parsed_record"]["parsed_fields"]
+            assert "emails" in feed_result["parsed_record"]["parsed_fields"]
+            assert "contacts" in feed_result["parsed_record"]["parsed_fields"]
+
+
+@vcr.use_cassette
+def test_verify_response_is_a_generator():
+    results = feeds_api.domaindiscovery(after="-60", header_authenticationn=False)
+
+    assert isgenerator(results.response())
+
+
+@vcr.use_cassette
+def test_feeds_endpoint_should_raise_error_if_download_api_using_header_auth():
+    with pytest.raises(ValueError) as excinfo:
+        feeds_api.domaindiscovery(after="-60", endpoint="download")
+
+    assert str(excinfo.value) == "download API does not support header authentication. Provide api_key in the parameter"
+
+
+@vcr.use_cassette
+def test_feeds_endpoint_should_raise_error_if_no_required_params():
+    with pytest.raises(ValueError) as excinfo:
+        feeds_api.domaindiscovery()
+
+    assert str(excinfo.value) == "sessionID or after or before must be provided"
+
+
+@vcr.use_cassette
+def test_feeds_endpoint_should_raise_error_if_asked_csv_format_for_download_api():
+    with pytest.raises(ValueError) as excinfo:
+        feeds_api.domaindiscovery(after="-60", output_format="csv", endpoint="download")
+
+    assert str(excinfo.value) == "csv format is not available in download API."

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.2.2}/tests/test_utils.py

@@ -1,4 +1,6 @@
 import json
+import pytest
+
 from datetime import datetime, timedelta
 
 from tests.responses import iris_investigate_data
@@ -15,24 +17,18 @@ def test_get_domain_age():
 def test_get_threat_component():
     threat_components = [
         {"name": "threat_profile_malware", "risk_score": 31},
-        {"name": "threat_profile_spam", "risk_score": 73, "threats": ["spam"]}
+        {"name": "threat_profile_spam", "risk_score": 73, "threats": ["spam"]},
     ]
     result = utils.get_threat_component(threat_components, "threat_profile_malware")
     assert result.get("risk_score") == 31
 
 
 def test_investigate_average_risk_score():
-    domains = [
-        {"domain_risk": {"risk_score": 25}},
-        {"domain_risk": {"risk_score": 27}}
-    ]
+    domains = [{"domain_risk": {"risk_score": 25}}, {"domain_risk": {"risk_score": 27}}]
     result = utils.get_average_risk_score(domains)
     assert result == 26
 
-    domains = [
-        {"domain_risk": {"risk_score": 25}},
-        {}
-    ]
+    domains = [{"domain_risk": {"risk_score": 25}}, {}]
     result = utils.get_average_risk_score(domains)
     assert result == 25
 
@@ -42,17 +38,11 @@ def test_investigate_average_risk_score():
 
 
 def test_detect_average_risk_score():
-    domains = [
-        {"risk_score": 25},
-        {"risk_score": 27}
-    ]
+    domains = [{"risk_score": 25}, {"risk_score": 27}]
     result = utils.get_average_risk_score(domains)
     assert result == 26
 
-    domains = [
-        {"risk_score": 25},
-        {"risk_score": None}
-    ]
+    domains = [{"risk_score": 25}, {"risk_score": None}]
     result = utils.get_average_risk_score(domains)
     assert result == 25
 
@@ -65,24 +55,15 @@ def test_investigate_average_age():
     two_days_ago = (datetime.now() - timedelta(days=2)).strftime("%Y-%m-%d")
     five_days_ago = (datetime.now() - timedelta(days=5)).strftime("%Y-%m-%d")
 
-    domains = [
-        {"create_date": {"value": two_days_ago}},
-        {"create_date": {"value": five_days_ago}}
-    ]
+    domains = [{"create_date": {"value": two_days_ago}}, {"create_date": {"value": five_days_ago}}]
     result = utils.get_average_age(domains)
     assert result == 3
 
-    domains = [
-        {"create_date": {"value": two_days_ago}},
-        {}
-    ]
+    domains = [{"create_date": {"value": two_days_ago}}, {}]
     result = utils.get_average_age(domains)
     assert result == 2
 
-    domains = [
-        {"create_date": {"value": two_days_ago}},
-        {"create_date": {"value": ""}}
-    ]
+    domains = [{"create_date": {"value": two_days_ago}}, {"create_date": {"value": ""}}]
     result = utils.get_average_age(domains)
     assert result == 2
 
@@ -94,17 +75,11 @@ def test_investigate_average_age():
 def test_detect_average_age():
     two_days_ago = int((datetime.now() - timedelta(days=2)).strftime("%Y%m%d"))
     five_days_ago = int((datetime.now() - timedelta(days=5)).strftime("%Y%m%d"))
-    domains = [
-        {"create_date": two_days_ago},
-        {"create_date": five_days_ago}
-    ]
+    domains = [{"create_date": two_days_ago}, {"create_date": five_days_ago}]
     result = utils.get_average_age(domains)
     assert result == 3
 
-    domains = [
-        {"create_date": two_days_ago},
-        {"create_date": None}
-    ]
+    domains = [{"create_date": two_days_ago}, {"create_date": None}]
     result = utils.get_average_age(domains)
     assert result == 2
 
@@ -121,26 +96,51 @@ def test_data_prune():
 
 def test_find_emails():
     emails = utils.find_emails(json.dumps(iris_investigate_data.domaintools()))
-    assert emails == {'abuse@enom.com', 'hostmaster@nsone.net'}
+    assert emails == {"abuse@enom.com", "hostmaster@nsone.net"}
 
 
 def test_find_ips():
     ips = utils.find_ips(json.dumps(iris_investigate_data.domaintools()))
-    assert ips == {'142.250.115.26',
-                   '142.250.141.27',
-                   '198.51.44.4',
-                   '198.51.44.68',
-                   '198.51.45.4',
-                   '198.51.45.68',
-                   '199.30.228.112',
-                   '64.233.171.26',
-                   '74.125.142.26'}
+    assert ips == {
+        "142.250.115.26",
+        "142.250.141.27",
+        "198.51.44.4",
+        "198.51.44.68",
+        "198.51.45.4",
+        "198.51.45.68",
+        "199.30.228.112",
+        "64.233.171.26",
+        "74.125.142.26",
+    }
+
 
 def test_get_pivots():
     pivots = utils.get_pivots(iris_investigate_data.domaintools().get("results"), "")
-    assert pivots == [
-        ['IP ADDRESS', ('199.30.228.112', 4)],
-        ['IP ASN', (17318, 111)],
-        ['IP ISP', ('DomainTools LLC', 222)]
-    ]
+    assert pivots == [["IP ADDRESS", ("199.30.228.112", 4)], ["IP ASN", (17318, 111)], ["IP ISP", ("DomainTools LLC", 222)]]
+
+
+def test_validate_feeds_parameters_should_raise_error_if_download_api_using_header_auth(test_feeds_params):
+    test_feeds_params["output_format"] = "jsonl"
+
+    with pytest.raises(ValueError) as excinfo:
+        utils.validate_feeds_parameters(test_feeds_params)
+
+    assert str(excinfo.value) == "download API does not support header authentication. Provide api_key in the parameter"
+
+
+def test_validate_feeds_parameters_should_raise_error_if_no_required_params(test_feeds_params):
+    test_feeds_params.pop("sessionID", None)
+    test_feeds_params.pop("after", None)
+    test_feeds_params.pop("before", None)
+
+    with pytest.raises(ValueError) as excinfo:
+        utils.validate_feeds_parameters(test_feeds_params)
+
+    assert str(excinfo.value) == "sessionID or after or before must be provided"
+
+
+def test_validate_feeds_parameters_should_raise_error_if_asked_csv_format_for_download_api(test_feeds_params):
+    with pytest.raises(ValueError) as excinfo:
+        utils.validate_feeds_parameters(test_feeds_params)
 
+    assert str(excinfo.value) == "csv format is not available in download API."

domaintools_api_test_upload-2.2.1/VERSION

@@ -1 +0,0 @@
-2.2.1