opendns-dnsdb 0.1.0

data/docs/index.rst

@@ -0,0 +1,101 @@
+=============================
+OpenDNS Security Graph client
+=============================
+
+.. _installation:
+
+Installation
+============
+
+.. code-block:: bash
+
+  $ bundle && rake install
+
+Example
+=======
+
+.. code-block:: ruby
+
+  # Setup
+  db = OpenDNS::DNSDB.new(sslcert: 'client.p12', sslcertpasswd: 'opendns')
+
+  # A short list of known spam domains using a fast-flux infrastructure
+  spam_names = ['com-garciniac.net', 'bbc-global.co.uk', 'com-october.net']
+
+  # Retrieve all the IP addresses these morons have been using
+  ips = db.distinct_ips_by_name(spam_names)
+
+  # Discover new domains mapping to the IP addresses we just found
+  all_spam_names = db.distinct_names_by_ip(ips)
+
+  # Find all the name servers used by these new domains
+  all_spam_names_ns = db.distinct_nameservers_ips_by_name(all_spam_names)
+
+  # Find all the domains served by these name servers
+  maybe_more_spam = db.distinct_names_by_nameserver_ip(all_spam_names_ns)
+
+  # Return the subset of names not flagged as malware by OpenDNS yet
+  not_blocked_yet = db.not_suspicious_names(maybe_more_spam)
+
+  # Does this list of domains include domains used by malware?
+  is_malware = db.include_suspicious?(['wh4u6igxiglekn.su', 'excue.ru'])
+
+  # Specifically, is excue.ru suspicious?
+  is_suspicious = db.is_suspicious?('excue.ru')
+
+  # Find all .ru names frequently observed with wh4u6igxiglekn.su and excue.ru:
+  rel_ru = db.distinct_related_names(['wh4u6igxiglekn.su', 'excue.ru'],
+                                      max_names: 500,
+                                      max_depth: 4) { |n| n.end_with? '.ru.' }
+
+  # Get the number of daily requests for the past 10 days, for
+  # github.com and github.io:
+  traffic = db.daily_traffic_by_name(['www.github.com', 'www.github.io'],
+                                     days_back: 10)
+
+  # Cut the noise from this traffic - Days with less than 10 queries
+  traffic = db.high_pass_filter(traffic, cutoff: 10)
+
+  # Check if the traffic for github.io is suspiciously spiky:
+  traffic_is_suspicious =
+    db.relative_standard_deviation(traffic['www.github.io']) > 90
+
+Parallel requests
+=================
+
+This client library transparently supports parallel requests.
+
+Most operations can be given either a single name or single IP, as well
+as a list of names or IPs. The library will transparently paralellize
+operations in order for bulk queries to complete as fast as possible.
+
+Bulk operations can be performed on arbitrary large sets of names or
+IP addresses.
+
+Setup
+=====
+
+.. code-block:: ruby
+
+  require 'opendns-dnsdb'
+  db = OpenDNS::DNSDB.new(sslcert: 'client.p12', sslcertpasswd: 'opendns')
+
+Supported options:
+
+* ``timeout``: timeout for each query, in seconds (default: 15 seconds)
+* ``sslcert``: path to the SSL certificate
+* ``sslcerttype``: SSL certificate type, defaults to ``p12``
+* ``sslcertpasswd``: SSL certificate password
+* ``maxconnects``: max number of parallel operations (default: 10)
+
+Operations
+==========
+
+.. toctree::
+  :maxdepth: 1
+
+  operations/by_name
+  operations/by_ip
+  operations/label
+  operations/related
+  operations/traffic  

data/docs/make.bat

@@ -0,0 +1,242 @@
+@ECHO OFF
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set BUILDDIR=_build
+set ALLSPHINXOPTS=-d %BUILDDIR%/doctrees %SPHINXOPTS% .
+set I18NSPHINXOPTS=%SPHINXOPTS% .
+if NOT "%PAPER%" == "" (
+	set ALLSPHINXOPTS=-D latex_paper_size=%PAPER% %ALLSPHINXOPTS%
+	set I18NSPHINXOPTS=-D latex_paper_size=%PAPER% %I18NSPHINXOPTS%
+)
+
+if "%1" == "" goto help
+
+if "%1" == "help" (
+	:help
+	echo.Please use `make ^<target^>` where ^<target^> is one of
+	echo.  html       to make standalone HTML files
+	echo.  dirhtml    to make HTML files named index.html in directories
+	echo.  singlehtml to make a single large HTML file
+	echo.  pickle     to make pickle files
+	echo.  json       to make JSON files
+	echo.  htmlhelp   to make HTML files and a HTML help project
+	echo.  qthelp     to make HTML files and a qthelp project
+	echo.  devhelp    to make HTML files and a Devhelp project
+	echo.  epub       to make an epub
+	echo.  latex      to make LaTeX files, you can set PAPER=a4 or PAPER=letter
+	echo.  text       to make text files
+	echo.  man        to make manual pages
+	echo.  texinfo    to make Texinfo files
+	echo.  gettext    to make PO message catalogs
+	echo.  changes    to make an overview over all changed/added/deprecated items
+	echo.  xml        to make Docutils-native XML files
+	echo.  pseudoxml  to make pseudoxml-XML files for display purposes
+	echo.  linkcheck  to check all external links for integrity
+	echo.  doctest    to run all doctests embedded in the documentation if enabled
+	goto end
+)
+
+if "%1" == "clean" (
+	for /d %%i in (%BUILDDIR%\*) do rmdir /q /s %%i
+	del /q /s %BUILDDIR%\*
+	goto end
+)
+
+
+%SPHINXBUILD% 2> nul
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "html" (
+	%SPHINXBUILD% -b html %ALLSPHINXOPTS% %BUILDDIR%/html
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The HTML pages are in %BUILDDIR%/html.
+	goto end
+)
+
+if "%1" == "dirhtml" (
+	%SPHINXBUILD% -b dirhtml %ALLSPHINXOPTS% %BUILDDIR%/dirhtml
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The HTML pages are in %BUILDDIR%/dirhtml.
+	goto end
+)
+
+if "%1" == "singlehtml" (
+	%SPHINXBUILD% -b singlehtml %ALLSPHINXOPTS% %BUILDDIR%/singlehtml
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The HTML pages are in %BUILDDIR%/singlehtml.
+	goto end
+)
+
+if "%1" == "pickle" (
+	%SPHINXBUILD% -b pickle %ALLSPHINXOPTS% %BUILDDIR%/pickle
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished; now you can process the pickle files.
+	goto end
+)
+
+if "%1" == "json" (
+	%SPHINXBUILD% -b json %ALLSPHINXOPTS% %BUILDDIR%/json
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished; now you can process the JSON files.
+	goto end
+)
+
+if "%1" == "htmlhelp" (
+	%SPHINXBUILD% -b htmlhelp %ALLSPHINXOPTS% %BUILDDIR%/htmlhelp
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished; now you can run HTML Help Workshop with the ^
+.hhp project file in %BUILDDIR%/htmlhelp.
+	goto end
+)
+
+if "%1" == "qthelp" (
+	%SPHINXBUILD% -b qthelp %ALLSPHINXOPTS% %BUILDDIR%/qthelp
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished; now you can run "qcollectiongenerator" with the ^
+.qhcp project file in %BUILDDIR%/qthelp, like this:
+	echo.^> qcollectiongenerator %BUILDDIR%\qthelp\OpenDNSDNSDatabaseClientLibrary.qhcp
+	echo.To view the help file:
+	echo.^> assistant -collectionFile %BUILDDIR%\qthelp\OpenDNSDNSDatabaseClientLibrary.ghc
+	goto end
+)
+
+if "%1" == "devhelp" (
+	%SPHINXBUILD% -b devhelp %ALLSPHINXOPTS% %BUILDDIR%/devhelp
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished.
+	goto end
+)
+
+if "%1" == "epub" (
+	%SPHINXBUILD% -b epub %ALLSPHINXOPTS% %BUILDDIR%/epub
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The epub file is in %BUILDDIR%/epub.
+	goto end
+)
+
+if "%1" == "latex" (
+	%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished; the LaTeX files are in %BUILDDIR%/latex.
+	goto end
+)
+
+if "%1" == "latexpdf" (
+	%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
+	cd %BUILDDIR%/latex
+	make all-pdf
+	cd %BUILDDIR%/..
+	echo.
+	echo.Build finished; the PDF files are in %BUILDDIR%/latex.
+	goto end
+)
+
+if "%1" == "latexpdfja" (
+	%SPHINXBUILD% -b latex %ALLSPHINXOPTS% %BUILDDIR%/latex
+	cd %BUILDDIR%/latex
+	make all-pdf-ja
+	cd %BUILDDIR%/..
+	echo.
+	echo.Build finished; the PDF files are in %BUILDDIR%/latex.
+	goto end
+)
+
+if "%1" == "text" (
+	%SPHINXBUILD% -b text %ALLSPHINXOPTS% %BUILDDIR%/text
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The text files are in %BUILDDIR%/text.
+	goto end
+)
+
+if "%1" == "man" (
+	%SPHINXBUILD% -b man %ALLSPHINXOPTS% %BUILDDIR%/man
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The manual pages are in %BUILDDIR%/man.
+	goto end
+)
+
+if "%1" == "texinfo" (
+	%SPHINXBUILD% -b texinfo %ALLSPHINXOPTS% %BUILDDIR%/texinfo
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The Texinfo files are in %BUILDDIR%/texinfo.
+	goto end
+)
+
+if "%1" == "gettext" (
+	%SPHINXBUILD% -b gettext %I18NSPHINXOPTS% %BUILDDIR%/locale
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The message catalogs are in %BUILDDIR%/locale.
+	goto end
+)
+
+if "%1" == "changes" (
+	%SPHINXBUILD% -b changes %ALLSPHINXOPTS% %BUILDDIR%/changes
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.The overview file is in %BUILDDIR%/changes.
+	goto end
+)
+
+if "%1" == "linkcheck" (
+	%SPHINXBUILD% -b linkcheck %ALLSPHINXOPTS% %BUILDDIR%/linkcheck
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Link check complete; look for any errors in the above output ^
+or in %BUILDDIR%/linkcheck/output.txt.
+	goto end
+)
+
+if "%1" == "doctest" (
+	%SPHINXBUILD% -b doctest %ALLSPHINXOPTS% %BUILDDIR%/doctest
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Testing of doctests in the sources finished, look at the ^
+results in %BUILDDIR%/doctest/output.txt.
+	goto end
+)
+
+if "%1" == "xml" (
+	%SPHINXBUILD% -b xml %ALLSPHINXOPTS% %BUILDDIR%/xml
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The XML files are in %BUILDDIR%/xml.
+	goto end
+)
+
+if "%1" == "pseudoxml" (
+	%SPHINXBUILD% -b pseudoxml %ALLSPHINXOPTS% %BUILDDIR%/pseudoxml
+	if errorlevel 1 exit /b 1
+	echo.
+	echo.Build finished. The pseudo-XML files are in %BUILDDIR%/pseudoxml.
+	goto end
+)
+
+:end

data/docs/operations/by_ip.rst

@@ -0,0 +1,229 @@
+Getting information out of an IP address
+========================================
+
+Getting the list of names served by a name server
+-------------------------------------------------
+
+| This returns the list of names that have been served by an
+| authoritative name server:
+
+.. code-block:: ruby
+
+    db.names_by_nameserver_ip('199.185.137.3')
+
+Returns a ``Response::Distinct``:
+
+::
+
+    [
+        [ 0] "openbsd.com.",
+        [ 1] "openssh.com.",
+        [ 2] "yycix.ca.",
+        [ 3] "caisnet.com.",
+        [ 4] "cdnpowerpac.com.",
+        [ 5] "miarch.com.",
+        [ 6] "openbsd.org.",
+        [ 7] "theos.com.",
+        [ 8] "enhanced-business.com.",
+        [ 9] "onpa.ca.",
+        [10] "openbsdfoundation.org.",
+        [11] "eton-west.com.",
+        [12] "barr-ryder.com.",
+        [13] "chemco-elec.com.",
+        [14] "rakeng.com.",
+        [15] "yycix.com.",
+        [16] "elementsustainable.com.",
+        [17] "hartwigarchitecture.com.",
+        [18] "pentagonstructures.com.",
+        [19] "freezemaxwell.com.",
+        [20] "workungarrick.com.",
+        [21] "alpineheating.com.",
+        [22] "caisnet.ca.",
+        [23] "watertech.ca.",
+        [24] "desco.cc.",
+        [25] "openbsd.net.",
+        [26] "krawford.com.",
+        [27] "protostatix.com.",
+        [28] "rms-group.ca.",
+        [29] "cmroofing.ca.",
+        [30] "hoeng.com.",
+        [31] "openssh.net.",
+        [32] "cuthbertsmith.com.",
+        [33] "alta-tech.ca.",
+        [34] "bockroofing.com."
+    ]
+
+Getting the list of names that a set of name servers have been serving
+----------------------------------------------------------------------
+
+| This returns the list of names that have been served by a set of name
+| servers:
+
+.. code-block:: ruby
+
+    db.names_by_nameserver_ip(['199.185.137.3', '65.19.167.109'])
+
+Returns a ``Response::HashByIP``:
+
+::
+
+    {
+        "199.185.137.3" => [
+            [ 0] "openbsd.com.",
+            [ 1] "openssh.com.",
+            [ 2] "yycix.ca.",
+            [ 3] "caisnet.com.",
+            [ 4] "cdnpowerpac.com.",
+            [ 5] "miarch.com.",
+            [ 6] "openbsd.org.",
+            [ 7] "theos.com.",
+            [ 8] "enhanced-business.com.",
+            [ 9] "onpa.ca.",
+            [10] "openbsdfoundation.org.",
+            [11] "eton-west.com.",
+            [12] "barr-ryder.com.",
+            [13] "chemco-elec.com.",
+            [14] "rakeng.com.",
+            [15] "yycix.com.",
+            [16] "elementsustainable.com.",
+            [17] "hartwigarchitecture.com.",
+            [18] "pentagonstructures.com.",
+            [19] "freezemaxwell.com.",
+            [20] "workungarrick.com.",
+            [21] "alpineheating.com.",
+            [22] "caisnet.ca.",
+            [23] "watertech.ca.",
+            [24] "desco.cc.",
+            [25] "openbsd.net.",
+            [26] "krawford.com.",
+            [27] "protostatix.com.",
+            [28] "rms-group.ca.",
+            [29] "cmroofing.ca.",
+            [30] "hoeng.com.",
+            [31] "openssh.net.",
+            [32] "cuthbertsmith.com.",
+            [33] "alta-tech.ca.",
+            [34] "bockroofing.com."
+        ],
+        "65.19.167.109" => [
+            [0] "backplane.com.",
+            [1] "dragonflybsd.org."
+        ]
+    }
+
+Getting the list of unique names served by a set of name servers
+----------------------------------------------------------------
+
+This returns a ``Response::Distinct`` of unique names served by a set of name
+servers:
+
+.. code-block:: ruby
+
+    db.distinct_names_by_nameserver_ip(['199.185.137.3', '65.19.167.109'])
+
+Returns a ``Response::Distinct``:
+
+::
+
+    [
+        [ 0] "openbsd.com.",
+        [ 1] "openssh.com.",
+        [ 2] "yycix.ca.",
+        [ 3] "caisnet.com.",
+        [ 4] "cdnpowerpac.com.",
+        [ 5] "miarch.com.",
+        [ 6] "openbsd.org.",
+        [ 7] "theos.com.",
+        [ 8] "enhanced-business.com.",
+        [ 9] "onpa.ca.",
+        [10] "openbsdfoundation.org.",
+        [11] "eton-west.com.",
+        [12] "barr-ryder.com.",
+        [13] "chemco-elec.com.",
+        [14] "rakeng.com.",
+        [15] "yycix.com.",
+        [16] "elementsustainable.com.",
+        [17] "hartwigarchitecture.com.",
+        [18] "pentagonstructures.com.",
+        [19] "freezemaxwell.com.",
+        [20] "workungarrick.com.",
+        [21] "alpineheating.com.",
+        [22] "caisnet.ca.",
+        [23] "watertech.ca.",
+        [24] "desco.cc.",
+        [25] "openbsd.net.",
+        [26] "krawford.com.",
+        [27] "protostatix.com.",
+        [28] "rms-group.ca.",
+        [29] "cmroofing.ca.",
+        [30] "hoeng.com.",
+        [31] "openssh.net.",
+        [32] "cuthbertsmith.com.",
+        [33] "alta-tech.ca.",
+        [34] "bockroofing.com.",
+        [35] "backplane.com.",
+        [36] "dragonflybsd.org."
+    ]
+
+Getting the list of all names that resolved to an IP
+----------------------------------------------------
+
+| This returns all the names that have been seen for an IP over the past
+| 3 months:
+
+.. code-block:: ruby
+
+    db.names_by_ip('192.30.252.131')
+
+Returns a ``Response::Distinct``:
+
+::
+
+    [
+        [0] "github.com.",
+        [1] "ip1d-lb3-prd.iad.github.com."
+    ]
+
+Getting the list of all names that resolved to a set of IPs
+-----------------------------------------------------------
+
+| A bulk operation to retrieve the list of names having mapped to a set
+| of IPs:
+
+.. code-block:: ruby
+
+    db.names_by_ip(['192.30.252.131', '199.233.90.68'])
+
+Returns a ``Response::HashByIP``:
+
+::
+
+    {
+        "192.30.252.131" => [
+            [0] "github.com.",
+            [1] "ip1d-lb3-prd.iad.github.com."
+        ],
+         "199.233.90.68" => [
+            [0] "leaf.dragonflybsd.org."
+        ]
+    }
+
+Getting the list of unique names for a set of IPs
+-------------------------------------------------
+
+| This method returns a list of distinct names seen for a set of IP
+| addresses:
+
+.. code-block:: ruby
+
+    db.distinct_names_by_ip(['192.30.252.131', '199.233.90.68'])
+
+Returns a ``Response::Distinct``:
+
+::
+
+    [
+        [0] "github.com.",
+        [1] "ip1d-lb3-prd.iad.github.com.",
+        [2] "leaf.dragonflybsd.org."
+    ]