datasketches 0.3.1 → 0.4.0

data/vendor/datasketches-cpp/python/jupyter/FrequentItemsSketch.ipynb

@@ -1,354 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## Frequent Items Sketch Examples"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Basic Sketch Usage"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "More so than other sketches in the library, the Frequent Items sketch can take some practice to use since it identifies exceptionally heavy hitters rather than returning a \"top N\" list. We assume readers have already familiarized themselves with the [sketch documentation](https://datasketches.github.io/docs/Frequency/FrequentItemsOverview.html) and are aware of the key concepts around use of this sketch."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from datasketches import frequent_strings_sketch, frequent_items_error_type"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We'll use a very small sketch in this case so that we can easily fill it, otherwise the difference between error types is more difficult to demonstrate."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "k = 3\n",
-    "fi = frequent_strings_sketch(k)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "A brief digression into implementation details to help explain what we're doing here. The Frequent Items sketch maintains a list of items, but purges the least frequent items when the list fills. For this example, we'll keep inserting items until after a purge takes place.\n",
-    "\n",
-    "We'll insert items with exponentially decreasing weights, which in this case gives us a more interesting set of results when we later query things."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Update 1: 1 items\n",
-      "Update 2: 2 items\n",
-      "Update 3: 3 items\n",
-      "Update 4: 4 items\n",
-      "Update 5: 5 items\n",
-      "Update 6: 6 items\n",
-      "Update 7: 3 items\n",
-      "Update 8: 4 items\n"
-     ]
-    }
-   ],
-   "source": [
-    "n = 8\n",
-    "for i in range(0,n):\n",
-    "    fi.update(str(i), 2 ** (n-i))\n",
-    "    i += 1\n",
-    "    print('Update ' + str(i) + ': ' + str(fi.get_num_active_items()) + ' items')"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can see where the purge happened, and in this case we inserted a low-weight item after the purge. We can now compare querying items to exclude either false positives or false negatives.\n",
-    "  - `NO_FALSE_POSITIVES` returns all items with a _lower_ bound above the a posteriori error\n",
-    "  - `NO_FALSE_NEGATIVES` returns all items with an _upper_ bound above the a posteriori error\n",
-    "\n",
-    "The latter option will always include any results from the first set and may include others. Items are returned as (id, estimate, lower_bound, upper_bound) and are sorted by decreasing weight."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[('0', 256, 224, 256), ('1', 128, 96, 128)]"
-      ]
-     },
-     "execution_count": 5,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "fi.get_frequent_items(frequent_items_error_type.NO_FALSE_POSITIVES)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "[('0', 256, 224, 256),\n",
-       " ('1', 128, 96, 128),\n",
-       " ('2', 64, 32, 64),\n",
-       " ('7', 34, 2, 34)]"
-      ]
-     },
-     "execution_count": 6,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "fi.get_frequent_items(frequent_items_error_type.NO_FALSE_NEGATIVES)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The sketch also allows us to query for individual items directly."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "256\n",
-      "64\n",
-      "2\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(fi.get_estimate(\"0\"))\n",
-    "print(fi.get_upper_bound(\"2\"))\n",
-    "print(fi.get_lower_bound(\"7\"))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can also query for items not in the the list, whether the item has never been seen or if it has been evicted from the active set."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "0"
-      ]
-     },
-     "execution_count": 8,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "fi.get_estimate(\"5\")"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "The sketch may also be serialized for archiving, and reconstructed."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "84"
-      ]
-     },
-     "execution_count": 9,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "sk_bytes = fi.serialize()\n",
-    "len(sk_bytes)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "### Frequent items sketch summary:\n",
-      "   lg cur map size  : 3\n",
-      "   lg max map size  : 3\n",
-      "   num active items : 4\n",
-      "   total weight     : 510\n",
-      "   max error        : 32\n",
-      "### End sketch summary\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "fi2 = frequent_strings_sketch.deserialize(sk_bytes)\n",
-    "print(fi2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Merging Example"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Frequent Items sketches support `merge()` to combine sketches. Keep in mind that the combined sketches may not have any meaningfully frequent items, even if there were frequent items in one of the input sketches.\n",
-    "\n",
-    "We'll start by creating a sketch with lots of equally-weighted very light items, but with a combined weight several times greater than that of the first sketch, and then merge that into the first sketch."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "fi2 = frequent_strings_sketch(k)\n",
-    "wt = fi.get_total_weight()\n",
-    "for i in range(0,4*wt):\n",
-    "    fi2.update(str(i))\n",
-    "fi.merge(fi2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Even though all these new items have weight 1, there are so many of them that we have nothing if we ask for no fasle positives."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "0"
-      ]
-     },
-     "execution_count": 13,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "len(fi.get_frequent_items(frequent_items_error_type.NO_FALSE_POSITIVES))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We do, however, see a few potentially heavy items if we request no false negatives."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 14,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "3"
-      ]
-     },
-     "execution_count": 14,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "len(fi.get_frequent_items(frequent_items_error_type.NO_FALSE_NEGATIVES))"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.0"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

data/vendor/datasketches-cpp/python/jupyter/HLLSketch.ipynb

@@ -1,346 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "## HLL Sketch Examples"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Basic Sketch Usage"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 1,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "from datasketches import hll_sketch, hll_union, tgt_hll_type"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We'll create a sketch with log2(k) = 12"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 2,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "sk = hll_sketch(12)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Insert ~2 million points. Values are hashed, so using sequential integers is fine for demonstration purposes."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 3,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "### HLL SKETCH SUMMARY: \n",
-      "  Log Config K   : 12\n",
-      "  Hll Target     : HLL_4\n",
-      "  Current Mode   : HLL\n",
-      "  LB             : 2.06958e+06\n",
-      "  Estimate       : 2.09635e+06\n",
-      "  UB             : 2.12379e+06\n",
-      "  OutOfOrder flag: 0\n",
-      "  CurMin       : 7\n",
-      "  NumAtCurMin  : 72\n",
-      "  HipAccum     : 2.09635e+06\n",
-      "  KxQ0         : 5.80703\n",
-      "  KxQ1         : 0\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "n = 1 << 21\n",
-    "for i in range(0, n):\n",
-    "    sk.update(i)\n",
-    "print(sk)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Since we know the exact value of n we can look at the estimate and upper/lower bounds as a % of the true value. We'll look at the bounds at 1 standard deviation. In this case, the true value does lie within the bounds, but since these are probabilistic bounds the true value will sometimes be outside them (especially at 1 standard deviation)."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 4,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Upper bound (1 std. dev) as % of true value:  101.2703\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(\"Upper bound (1 std. dev) as % of true value: \", round(100*sk.get_upper_bound(1) / n, 4))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 5,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Estimate as % of true value:  99.9618\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(\"Estimate as % of true value: \", round(100*sk.get_estimate() / n, 4))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 6,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Lower bound (1 std. dev) as % of true value:  98.6852\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(\"Lower bound (1 std. dev) as % of true value: \", round(100*sk.get_lower_bound(1) / n, 4))"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Finally, we can serialize and deserialize the sketch, which will give us back the same structure."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 7,
-   "metadata": {},
-   "outputs": [
-    {
-     "data": {
-      "text/plain": [
-       "2096"
-      ]
-     },
-     "execution_count": 7,
-     "metadata": {},
-     "output_type": "execute_result"
-    }
-   ],
-   "source": [
-    "sk_bytes = sk.serialize_compact()\n",
-    "len(sk_bytes)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 8,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "### HLL SKETCH SUMMARY: \n",
-      "  Log Config K   : 12\n",
-      "  Hll Target     : HLL_4\n",
-      "  Current Mode   : HLL\n",
-      "  LB             : 2.06958e+06\n",
-      "  Estimate       : 2.09635e+06\n",
-      "  UB             : 2.12379e+06\n",
-      "  OutOfOrder flag: 0\n",
-      "  CurMin       : 7\n",
-      "  NumAtCurMin  : 72\n",
-      "  HipAccum     : 2.09635e+06\n",
-      "  KxQ0         : 5.80703\n",
-      "  KxQ1         : 0\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "sk2 = hll_sketch.deserialize(sk_bytes)\n",
-    "print(sk2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "### Sketch Union Usage"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Here, we'll create two sketches with partial overlap in values. For good measure, we'll let k be larger in one sketch. For most applications we'd generally create all new data using the same size sketch, allowing differences to creep in when combining new and historica data."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 9,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "k = 12\n",
-    "n = 1 << 20\n",
-    "offset = int(3 * n / 4)"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 10,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "sk1 = hll_sketch(k)\n",
-    "sk2 = hll_sketch(k + 1)\n",
-    "for i in range(0, n):\n",
-    "    sk1.update(i)\n",
-    "    sk2.update(i + offset)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Create a union object and add the sketches to that. To demonstrate smoothly handling multiple sketch sizes, we'll use a size of k+1 here."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 11,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "union = hll_union(k+1)\n",
-    "union.update(sk1)\n",
-    "union.update(sk2)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "Note how log config k has automatically adopted the value of the smaller input sketch."
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 12,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "### HLL SKETCH SUMMARY: \n",
-      "  Log Config K   : 12\n",
-      "  Hll Target     : HLL_4\n",
-      "  Current Mode   : HLL\n",
-      "  LB             : 1.80197e+06\n",
-      "  Estimate       : 1.83108e+06\n",
-      "  UB             : 1.86121e+06\n",
-      "  OutOfOrder flag: 1\n",
-      "  CurMin       : 6\n",
-      "  NumAtCurMin  : 2\n",
-      "  HipAccum     : 1.76932e+06\n",
-      "  KxQ0         : 6.60752\n",
-      "  KxQ1         : 0\n",
-      "\n"
-     ]
-    }
-   ],
-   "source": [
-    "result = union.get_result()\n",
-    "print(result)"
-   ]
-  },
-  {
-   "cell_type": "markdown",
-   "metadata": {},
-   "source": [
-    "We can again compare against the exact result, in this case 1.75*n"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": 13,
-   "metadata": {},
-   "outputs": [
-    {
-     "name": "stdout",
-     "output_type": "stream",
-     "text": [
-      "Estimate as % of true value:  99.7859\n"
-     ]
-    }
-   ],
-   "source": [
-    "print(\"Estimate as % of true value: \", round(100*result.get_estimate() / (7*n/4), 4))"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": []
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": "Python 3",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.7.0"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}