domaintools-api-test-upload 2.2.1__tar.gz → 2.4.0__tar.gz

{domaintools_api_test_upload-2.2.1/domaintools_api_test_upload.egg-info → domaintools_api_test_upload-2.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: domaintools_api_test_upload
-Version: 2.2.1
+Version: 2.4.0
 Summary: DomainTools Official Python API
 Author-email: DomainTools <integrations@domaintools.com>
 License: The MIT License (MIT)
@@ -46,6 +46,7 @@ Requires-Dist: typer
 Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
 Requires-Dist: mock; extra == "test"
+Dynamic: license-file
 
 ![domaintools](https://github.com/DomainTools/python_api/raw/main/artwork/logo.png)
 ===================
@@ -272,16 +273,51 @@ Real-Time Threat Intelligence Feeds provide data on the different stages of the
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(endpoint="feed", **kwargs)
     ```
-- `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so you need to set this to `False` when calling `download` API endpoints.
+- `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so this will be defaulted to `False` even without explicitly setting it.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(header_authentication=False, **kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(output_format="csv", **kwargs)
     ```
+
+The Feed API standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds. Specify the range of data you receive in one of two ways:
+
+1. With `sessionID`: Make a call and provide a new `sessionID` parameter of your choosing. The API will return the last hour of data by default.
+    - Each subsequent call to the API using your `sessionID` will return all data since the last.
+    - Any single request returns a maximum of 10M results. Requests that exceed 10M results will return a HTTP 206 response code; repeat the same request (with the same `sessionID`) to receive the next tranche of data until receiving a HTTP 200 response code.
+2. Or, specify the time range in one of two ways:
+    - Either an `after=-60` query parameter, where (in this example) -60 indicates the previous 60 seconds.
+    - Or `after` and `before` query parameters for a time range, with each parameter accepting an ISO-8601 UTC formatted timestamp (a UTC date and time of the format YYYY-MM-DDThh:mm:ssZ)
+
+## Handling iterative response from RTUF endpoints:
+
+Since we may dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
+
+### Single request because the requested data is within the maximum result:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-60)
+
+for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
+    # do things to result
+```
+
+## Multiple requests because the requested data is more than the maximum result per request:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-7200)
+
+for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
+    # do things to partial_result
+```

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/README.md

@@ -223,16 +223,51 @@ Real-Time Threat Intelligence Feeds provide data on the different stages of the
 Custom parameters aside from the common `GET` Request parameters:
 - `endpoint` (choose either `download` or `feed` API endpoint - default is `feed`)
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(endpoint="feed", **kwargs)
     ```
-- `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so you need to set this to `False` when calling `download` API endpoints.
+- `header_authentication`: by default, we're using API Header Authentication. Set this False if you want to use API Key and Secret Authentication. Apparently, you can't use API Header Authentication for `download` endpoints so this will be defaulted to `False` even without explicitly setting it.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(header_authentication=False, **kwargs)
     ```
 - `output_format`: (choose either `csv` or `jsonl` - default is `jsonl`). Cannot be used in `domainrdap` feeds. Additionally, `csv` is not available for `download` endpoints.
     ```python
-    api = API(USERNAME, KEY)
+    api = API(USERNAME, KEY, always_sign_api_key=False)
     api.nod(output_format="csv", **kwargs)
     ```
+
+The Feed API standard access pattern is to periodically request the most recent feed data, as often as every 60 seconds. Specify the range of data you receive in one of two ways:
+
+1. With `sessionID`: Make a call and provide a new `sessionID` parameter of your choosing. The API will return the last hour of data by default.
+    - Each subsequent call to the API using your `sessionID` will return all data since the last.
+    - Any single request returns a maximum of 10M results. Requests that exceed 10M results will return a HTTP 206 response code; repeat the same request (with the same `sessionID`) to receive the next tranche of data until receiving a HTTP 200 response code.
+2. Or, specify the time range in one of two ways:
+    - Either an `after=-60` query parameter, where (in this example) -60 indicates the previous 60 seconds.
+    - Or `after` and `before` query parameters for a time range, with each parameter accepting an ISO-8601 UTC formatted timestamp (a UTC date and time of the format YYYY-MM-DDThh:mm:ssZ)
+
+## Handling iterative response from RTUF endpoints:
+
+Since we may dealing with large feeds datasets, the python wrapper uses `generator` for efficient memory handling. Therefore, we need to iterate through the `generator` if we're accessing the partial results of the feeds data.
+
+### Single request because the requested data is within the maximum result:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-60)
+
+for result in results.response() # generator that holds NOD feeds data for the past 60 seconds and is expected to request only once
+    # do things to result
+```
+
+## Multiple requests because the requested data is more than the maximum result per request:
+```python
+from domaintools import API
+
+api = API(USERNAME, KEY, always_sign_api_key=False)
+results = api.nod(sessionID="my-session-id", after=-7200)
+
+for partial_result in results.response() # generator that holds NOD feeds data for the past 2 hours and is expected to request multiple times
+    # do things to partial_result
+```

domaintools_api_test_upload-2.4.0/VERSION

@@ -0,0 +1 @@
+2.4.0

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/_version.py

@@ -20,4 +20,4 @@ OTHER DEALINGS IN THE SOFTWARE.
 
 """
 
-current = "2.2.1"
+current = "2.4.0"

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/api.py

@@ -1,6 +1,7 @@
 from datetime import datetime, timedelta, timezone
 from hashlib import sha1, sha256, md5
 from hmac import new as hmac
+
 import re
 
 from domaintools.constants import Endpoint, ENDPOINT_TO_SOURCE_MAP, FEEDS_PRODUCTS_LIST, OutputFormat
@@ -11,6 +12,7 @@ from domaintools.results import (
     ParsedDomainRdap,
     Reputation,
     Results,
+    FeedsResults,
 )
 from domaintools.filters import (
     filter_by_riskscore,
@@ -87,11 +89,8 @@ class API(object):
 
         if not https:
             raise Exception("The DomainTools API endpoints no longer support http traffic. Please make sure https=True.")
-        if proxy_url:
-            if isinstance(proxy_url, str):
-                self.proxy_url = {"http://": proxy_url, "https://": proxy_url}
-            else:
-                raise Exception("Proxy URL must be a string. For example: '127.0.0.1:8888'")
+        if proxy_url and not isinstance(proxy_url, str):
+            raise Exception("Proxy URL must be a string. For example: '127.0.0.1:8888'")
 
     def _build_api_url(self, api_url=None, api_port=None):
         """Build the API url based on the given url and port. Defaults to `https://api.domaintools.com`"""
@@ -412,38 +411,6 @@ class API(object):
             **kwargs,
         )
 
-    def phisheye(self, query, days_back=None, **kwargs):
-        """Returns domain results for the specified term for today or the specified number of days_back.
-        Terms must be setup for monitoring via the web interface: https://research.domaintools.com/phisheye.
-
-        NOTE: Properties of a domain are only provided if we have been able to obtain them.
-              Many domains will have incomplete data because that information isn't available in their Whois records,
-              or they don't have DNS results for a name server or IP address.
-        """
-        return self._results(
-            "phisheye",
-            "/v1/phisheye",
-            query=query,
-            days_back=days_back,
-            items_path=("domains",),
-            **kwargs,
-        )
-
-    def phisheye_term_list(self, include_inactive=False, **kwargs):
-        """Provides a list of terms that are set up for this account.
-        This call is not charged against your API usage limit.
-
-        NOTE: The terms must be configured in the PhishEye web interface: https://research.domaintools.com/phisheye.
-              There is no API call to set up the terms.
-        """
-        return self._results(
-            "phisheye_term_list",
-            "/v1/phisheye/term-list",
-            include_inactive=include_inactive,
-            items_path=("terms",),
-            **kwargs,
-        )
-
     def iris(
         self,
         domain=None,
@@ -1065,7 +1032,7 @@ class API(object):
             **kwargs,
         )
 
-    def nod(self, **kwargs):
+    def nod(self, **kwargs) -> FeedsResults:
         """Returns back list of the newly observed domains feed"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1078,10 +1045,11 @@ class API(object):
             f"newly-observed-domains-feed-({source.value})",
             f"v1/{endpoint}/nod/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )
 
-    def nad(self, **kwargs):
+    def nad(self, **kwargs) -> FeedsResults:
         """Returns back list of the newly active domains feed"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1094,10 +1062,11 @@ class API(object):
             f"newly-active-domains-feed-({source})",
             f"v1/{endpoint}/nad/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )
 
-    def domainrdap(self, **kwargs):
+    def domainrdap(self, **kwargs) -> FeedsResults:
         """Returns changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP)"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1107,10 +1076,11 @@ class API(object):
             f"domain-registration-data-access-protocol-feed-({source})",
             f"v1/{endpoint}/domainrdap/",
             response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )
 
-    def domaindiscovery(self, **kwargs):
+    def domaindiscovery(self, **kwargs) -> FeedsResults:
         """Returns new domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties"""
         validate_feeds_parameters(kwargs)
         endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
@@ -1123,5 +1093,23 @@ class API(object):
             f"real-time-domain-discovery-feed-({source})",
             f"v1/{endpoint}/domaindiscovery/",
             response_path=(),
+            cls=FeedsResults,
+            **kwargs,
+        )
+
+    def noh(self, **kwargs) -> FeedsResults:
+        """Returns back list of the newly observed hostnames feed"""
+        validate_feeds_parameters(kwargs)
+        endpoint = kwargs.pop("endpoint", Endpoint.FEED.value)
+        source = ENDPOINT_TO_SOURCE_MAP.get(endpoint).value
+        if endpoint == Endpoint.DOWNLOAD.value or kwargs.get("output_format", OutputFormat.JSONL.value) != OutputFormat.CSV.value:
+            # headers param is allowed only in Feed API and CSV format
+            kwargs.pop("headers", None)
+
+        return self._results(
+            f"newly-observed-hosts-feed-({source})",
+            f"v1/{endpoint}/noh/",
+            response_path=(),
+            cls=FeedsResults,
             **kwargs,
         )

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/base_results.py

@@ -53,8 +53,6 @@ class Results(MutableMapping, MutableSequence):
         self._response = None
         self._items_list = None
         self._data = None
-        self._limit_exceeded = None
-        self._limit_exceeded_message = None
 
     def _wait_time(self):
         if not self.api.rate_limit or not self.product in self.api.limits:
@@ -77,6 +75,23 @@ class Results(MutableMapping, MutableSequence):
 
         return wait_for
 
+    def _get_session_params(self):
+        parameters = deepcopy(self.kwargs)
+        parameters.pop("output_format", None)
+        parameters.pop(
+            "format", None
+        )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
+        headers = {}
+        if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
+            parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
+            headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
+
+        header_api_key = parameters.pop("X-Api-Key", None)
+        if header_api_key:
+            headers["X-Api-Key"] = header_api_key
+
+        return {"parameters": parameters, "headers": headers}
+
     def _make_request(self):
 
         with Client(verify=self.api.verify_ssl, proxy=self.api.proxy_url, timeout=None) as session:
@@ -93,20 +108,9 @@ class Results(MutableMapping, MutableSequence):
                 patch_data.update(self.api.extra_request_params)
                 return session.patch(url=self.url, json=patch_data)
             elif self.product in FEEDS_PRODUCTS_LIST:
-                parameters = deepcopy(self.kwargs)
-                parameters.pop("output_format", None)
-                parameters.pop(
-                    "format", None
-                )  # For some unknownn reasons, even if "format" is not included in the cli params for feeds endpoint, it is being populated thus we need to remove it. Happens only if using CLI.
-                headers = {}
-                if self.kwargs.get("output_format", OutputFormat.JSONL.value) == OutputFormat.CSV.value:
-                    parameters["headers"] = int(bool(self.kwargs.get("headers", False)))
-                    headers["accept"] = HEADER_ACCEPT_KEY_CSV_FORMAT
-
-                header_api_key = parameters.pop("X-Api-Key", None)
-                if header_api_key:
-                    headers["X-Api-Key"] = header_api_key
-
+                session_params = self._get_session_params()
+                parameters = session_params.get("parameters")
+                headers = session_params.get("headers")
                 return session.get(url=self.url, params=parameters, headers=headers, **self.api.extra_request_params)
             else:
                 return session.get(url=self.url, params=self.kwargs, **self.api.extra_request_params)
@@ -136,32 +140,26 @@ class Results(MutableMapping, MutableSequence):
         if self._data is None:
             results = self._get_results()
             self.setStatus(results.status_code, results)
-            if (
-                self.kwargs.get("format", "json") == "json"
-                and self.product not in FEEDS_PRODUCTS_LIST  # Special handling of feeds products' data to preserve the result in jsonline format
-            ):
+            if self.kwargs.get("format", "json") == "json":
                 self._data = results.json()
             else:
                 self._data = results.text
-            limit_exceeded, message = self.check_limit_exceeded()
 
-            if limit_exceeded:
-                self._limit_exceeded = True
-                self._limit_exceeded_message = message
+        self.check_limit_exceeded()
 
-        if self._limit_exceeded is True:
-            raise ServiceException(503, "Limit Exceeded{}".format(self._limit_exceeded_message))
-        else:
-            return self._data
+        return self._data
 
     def check_limit_exceeded(self):
-        if self.kwargs.get("format", "json") == "json" and self.product not in FEEDS_PRODUCTS_LIST:
-            if "response" in self._data and "limit_exceeded" in self._data["response"] and self._data["response"]["limit_exceeded"] is True:
-                return True, self._data["response"]["message"]
-        # TODO: handle html, xml response errors better.
+        limit_exceeded, reason = False, ""
+        if isinstance(self._data, dict) and (
+            "response" in self._data and "limit_exceeded" in self._data["response"] and self._data["response"]["limit_exceeded"] is True
+        ):
+            limit_exceeded, reason = True, self._data["response"]["message"]
         elif "response" in self._data and "limit_exceeded" in self._data:
-            return True, "limit exceeded"
-        return False, ""
+            limit_exceeded = True
+
+        if limit_exceeded:
+            raise ServiceException(503, f"Limit Exceeded {reason}")
 
     @property
     def status(self):
@@ -172,7 +170,7 @@ class Results(MutableMapping, MutableSequence):
 
     def setStatus(self, code, response=None):
         self._status = code
-        if code == 200:
+        if code == 200 or (self.product in FEEDS_PRODUCTS_LIST and code == 206):
             return
 
         reason = None
@@ -184,9 +182,9 @@ class Results(MutableMapping, MutableSequence):
                 if callable(reason):
                     reason = reason()
 
-        if code == 400:
+        if code in (400, 422):
             raise BadRequestException(code, reason)
-        elif code == 403:
+        elif code in (401, 403):
             raise NotAuthorizedException(code, reason)
         elif code == 404:
             raise NotFoundException(code, reason)

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/cli/api.py

@@ -9,7 +9,7 @@ from typing import Optional, Dict, Tuple
 from rich.progress import Progress, SpinnerColumn, TextColumn
 
 from domaintools.api import API
-from domaintools.constants import Endpoint, OutputFormat
+from domaintools.constants import Endpoint, FEEDS_PRODUCTS_LIST, OutputFormat
 from domaintools.cli.utils import get_file_extension
 from domaintools.exceptions import ServiceException
 from domaintools._version import current as version
@@ -110,6 +110,8 @@ class DTCLICommand:
     def _get_formatted_output(cls, cmd_name: str, response, out_format: str = "json"):
         if cmd_name in ("available_api_calls",):
             return "\n".join(response)
+        if response.product in FEEDS_PRODUCTS_LIST:
+            return "\n".join([data for data in response.response()])
         return str(getattr(response, out_format) if out_format != "list" else response.as_list())
 
     @classmethod
@@ -227,7 +229,10 @@ class DTCLICommand:
 
                 if isinstance(out_file, _io.TextIOWrapper):
                     # use rich `print` command to prettify the ouput in sys.stdout
-                    print(response)
+                    if response.product in FEEDS_PRODUCTS_LIST:
+                        print(output)
+                    else:
+                        print(response)
                 else:
                     # if it's a file then write
                     out_file.write(output if output.endswith("\n") else output + "\n")

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/cli/commands/__init__.py

@@ -2,6 +2,5 @@ from .accounts import *
 from .iris import *
 from .domains import *
 from .ips import *
-from .phisheye import *
 from .detects import *
 from .feeds import *

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/cli/commands/feeds.py

@@ -314,3 +314,83 @@ def feeds_domaindiscovery(
     ),
 ):
     DTCLICommand.run(name=c.FEEDS_DOMAINDISCOVERY, params=ctx.params)
+
+
+@dt_cli.command(
+    name=c.FEEDS_NOH,
+    help=get_cli_helptext_by_name(command_name=c.FEEDS_NOH),
+)
+def feeds_noh(
+    ctx: typer.Context,
+    user: str = typer.Option(None, "-u", "--user", help="Domaintools API Username."),
+    key: str = typer.Option(None, "-k", "--key", help="DomainTools API key"),
+    creds_file: str = typer.Option(
+        "~/.dtapi",
+        "-c",
+        "--credfile",
+        help="Optional file with API username and API key, one per line.",
+    ),
+    no_verify_ssl: bool = typer.Option(
+        False,
+        "--no-verify-ssl",
+        help="Skip verification of SSL certificate when making HTTPs API calls",
+    ),
+    no_sign_api_key: bool = typer.Option(
+        False,
+        "--no-sign-api-key",
+        help="Skip signing of api key",
+    ),
+    header_authentication: bool = typer.Option(
+        True,
+        "--no-header-auth",
+        help="Don't use header authentication",
+    ),
+    output_format: str = typer.Option(
+        "jsonl",
+        "-f",
+        "--format",
+        help=f"Output format in [{OutputFormat.JSONL.value}, {OutputFormat.CSV.value}]",
+        callback=DTCLICommand.validate_feeds_format_input,
+    ),
+    endpoint: str = typer.Option(
+        Endpoint.FEED.value,
+        "-e",
+        "--endpoint",
+        help=f"Valid endpoints: [{Endpoint.FEED.value}, {Endpoint.DOWNLOAD.value}]",
+        callback=DTCLICommand.validate_endpoint_input,
+    ),
+    sessionID: str = typer.Option(
+        None,
+        "--session-id",
+        help="Unique identifier for the session",
+    ),
+    after: str = typer.Option(
+        None,
+        "--after",
+        help="Start of the time window, relative to the current time in seconds, for which data will be provided",
+        callback=DTCLICommand.validate_after_or_before_input,
+    ),
+    before: str = typer.Option(
+        None,
+        "--before",
+        help="The end of the query window in seconds, relative to the current time, inclusive",
+        callback=DTCLICommand.validate_after_or_before_input,
+    ),
+    domain: str = typer.Option(
+        None,
+        "-d",
+        "--domain",
+        help="A string value used to filter feed results",
+    ),
+    headers: bool = typer.Option(
+        False,
+        "--headers",
+        help="Adds a header to the first line of response when text/csv is set in header parameters",
+    ),
+    top: str = typer.Option(
+        None,
+        "--top",
+        help="Number of results to return in the response payload. This is ignored in download endpoint",
+    ),
+):
+    DTCLICommand.run(name=c.FEEDS_NOH, params=ctx.params)

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/cli/constants.py

@@ -39,12 +39,9 @@ IRIS_DETECT_MANAGE_WATCHLIST_DOMAINS = "iris_detect_manage_watchlist_domains"
 IRIS_DETECT_ESCALATE_DOMAINS = "iris_detect_escalate_domains"
 IRIS_DETECT_IGNORED_DOMAINS = "iris_detect_ignored_domains"
 
-# phisheye
-PHISHEYE = "phisheye"
-PHISHEYE_TERM_LIST = "phisheye_term_list"
-
 # feeds
 FEEDS_NAD = "nad"
 FEEDS_NOD = "nod"
+FEEDS_NOH = "noh"
 FEEDS_DOMAINRDAP = "domainrdap"
 FEEDS_DOMAINDISCOVERY = "domaindiscovery"

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/cli/utils.py

@@ -29,25 +29,6 @@ def _iris_investigate_helptext():
     """
 
 
-def _phisheye_helptext():
-    return """
-        Returns domain results for the specified term for today or the specified number of days_back. Terms must be setup for monitoring via the web interface: https://research.domaintools.com/phisheye. \n
-
-        NOTE: Properties of a domain are only provided if we have been able to obtain them. \n
-        Many domains will have incomplete data because that information isn't available in their Whois records,
-        or they don't have DNS results for a name server or IP address.
-    """
-
-
-def _phisheye_termlist():
-    return """
-        Provides a list of terms that are set up for this account. This call is not charged against your API usage limit.
-
-        NOTE: The terms must be configured in the PhishEye web interface: https://research.domaintools.com/phisheye. \n
-        There is no API call to set up the terms.
-    """
-
-
 _command_help_mapper = {
     c.ACCOUNT_INFORMATION: "Provides a snapshot of your accounts current API usage.",
     c.AVAILABLE_API_CALLS: "Provides a list of api calls that you can use based on your account information.",
@@ -71,8 +52,6 @@ _command_help_mapper = {
     c.REVERSE_WHOIS: "Provides list of one or more terms to search for in the Whois record, as a Python list or separated with the pipe character ( | ).",
     c.WHOIS: "Pass in a domain name or an IP address to perform a whois lookup.",
     c.WHOIS_HISTORY: "Retrieve historical Whois records of a given domain name.",
-    c.PHISHEYE: _phisheye_helptext(),
-    c.PHISHEYE_TERM_LIST: _phisheye_termlist(),
     c.IRIS: "Performs a search for the provided search terms ANDed together, returning the pivot engine row data for the resulting domains.",
     c.RISK: "Returns back the risk score for a given domain.",
     c.RISK_EVIDENCE: "Returns back the detailed risk evidence associated with a given domain.",
@@ -84,7 +63,9 @@ _command_help_mapper = {
     c.IRIS_DETECT_IGNORED_DOMAINS: "Returns back a list of ignored domains in Iris Detect based on the provided filters.",
     c.FEEDS_NAD: "Returns back newly active domains feed.",
     c.FEEDS_NOD: "Returns back newly observed domains feed.",
+    c.FEEDS_NOH: "Returns back newly observed hosts feed.",
     c.FEEDS_DOMAINRDAP: "Returns changes to global domain registration information, populated by the Registration Data Access Protocol (RDAP).",
+    c.FEEDS_DOMAINDISCOVERY: "Returns new domains as they are either discovered in domain registration information, observed by our global sensor network, or reported by trusted third parties.",
 }
 
 

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/constants.py

@@ -28,6 +28,8 @@ FEEDS_PRODUCTS_LIST = [
     "newly-active-domains-feed-(s3)",
     "newly-observed-domains-feed-(api)",
     "newly-observed-domains-feed-(s3)",
+    "newly-observed-hosts-feed-(api)",
+    "newly-observed-hosts-feed-(s3)",
     "domain-registration-data-access-protocol-feed-(api)",
     "domain-registration-data-access-protocol-feed-(s3)",
     "real-time-domain-discovery-feed-(api)",

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/results.py

@@ -11,6 +11,8 @@ except ImportError:  # pragma: no cover
     from ordereddict import OrderedDict
 
 from itertools import zip_longest
+from typing import Generator
+
 from domaintools_async import AsyncResults as Results
 
 
@@ -141,3 +143,28 @@ class ParsedDomainRdap(Results):
                     flat[f"contact_{contact_key}_{i}"] = " | ".join(contact_value) if type(contact_value) in (list, tuple) else contact_value
 
         return flat
+
+
+class FeedsResults(Results):
+    """Returns the generator for feeds results"""
+
+    def response(self) -> Generator:
+        status_code = None
+        while status_code != 200:
+            resp_data = self.data()
+            status_code = self.status
+            yield resp_data
+
+            self._data = None  # clear the data here
+            if not self.kwargs.get("sessionID"):
+                # we'll only do iterative request for queries that has sessionID.
+                # Otherwise, we will have an infinite request if sessionID was not provided but the required data asked is more than the maximum (1 hour of data)
+                break
+
+    def data(self):
+        results = self._get_results()
+        self.setStatus(results.status_code, results)
+        self._data = results.text
+        self.check_limit_exceeded()
+
+        return self._data

{domaintools_api_test_upload-2.2.1 → domaintools_api_test_upload-2.4.0}/domaintools/utils.py

@@ -177,8 +177,13 @@ def validate_feeds_parameters(params):
     after = params.get("after")
     before = params.get("before")
     if not (sessionID or after or before):
-        raise ValueError("sessionID or after or before must be defined")
+        raise ValueError("sessionID or after or before must be provided")
 
     format = params.get("output_format")
-    if params.get("endpoint") == Endpoint.DOWNLOAD.value and format == OutputFormat.CSV.value:
+    endpoint = params.get("endpoint")
+    if endpoint == Endpoint.DOWNLOAD.value and format == OutputFormat.CSV.value:
         raise ValueError(f"{format} format is not available in {Endpoint.DOWNLOAD.value} API.")
+
+    if endpoint == Endpoint.DOWNLOAD.value and params.get("header_authentication", True):
+        # For download endpoint, header_authentication will be False by default
+        params["header_authentication"] = False