j1-template 2022.5.0.rc1 → 2022.5.0.rc4

data/lib/starter_web/pages/public/_jupyter/notebooks/nbi-docs/nbi_docs_examples_sampling_from_a_population.ipynb

@@ -0,0 +1,421 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# HIDDEN\n",
+    "\n",
+    "from datascience import *\n",
+    "import matplotlib\n",
+    "# matplotlib.use('Agg', warn=False)\n",
+    "matplotlib.use('Agg')\n",
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "plt.style.use('fivethirtyeight')\n",
+    "import numpy as np\n",
+    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
+    "import ipywidgets as widgets\n",
+    "import nbinteract as nbi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Sampling from a Population",
+    "\n",
+    "The law of averages also holds when the random sample is drawn from individuals in a large population.\n",
+    "\n",
+    "As an example, we will study a population of flight delay times. The table `united` contains data for United Airlines domestic flights departing from San Francisco in the summer of 2015. The data are made publicly available by the [Bureau of Transportation Statistics](http://www.transtats.bts.gov/Fields.asp?Table_ID=293) in the United States Department of Transportation.\n",
+    "\n",
+    "There are 13,825 rows, each corresponding to a flight. The columns are the date of the flight, the flight number, the destination airport code, and the departure delay time in minutes. Some delay times are negative; those flights left early."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "    <thead>\n",
+       "        <tr>\n",
+       "            <th>Date</th> <th>Flight Number</th> <th>Destination</th> <th>Delay</th>\n",
+       "        </tr>\n",
+       "    </thead>\n",
+       "    <tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>73           </td> <td>HNL        </td> <td>257  </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>217          </td> <td>EWR        </td> <td>28   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>237          </td> <td>STL        </td> <td>-3   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>250          </td> <td>SAN        </td> <td>0    </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>267          </td> <td>PHL        </td> <td>64   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>273          </td> <td>SEA        </td> <td>-6   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>278          </td> <td>SEA        </td> <td>-8   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>292          </td> <td>EWR        </td> <td>12   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>300          </td> <td>HNL        </td> <td>20   </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "        <tr>\n",
+       "            <td>6/1/15</td> <td>317          </td> <td>IND        </td> <td>-10  </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "</table>\n",
+       "<p>... (13815 rows omitted)</p>"
+      ],
+      "text/plain": [
+       "Date   | Flight Number | Destination | Delay\n",
+       "6/1/15 | 73            | HNL         | 257\n",
+       "6/1/15 | 217           | EWR         | 28\n",
+       "6/1/15 | 237           | STL         | -3\n",
+       "6/1/15 | 250           | SAN         | 0\n",
+       "6/1/15 | 267           | PHL         | 64\n",
+       "6/1/15 | 273           | SEA         | -6\n",
+       "6/1/15 | 278           | SEA         | -8\n",
+       "6/1/15 | 292           | EWR         | 12\n",
+       "6/1/15 | 300           | HNL         | 20\n",
+       "6/1/15 | 317           | IND         | -10\n",
+       "... (13815 rows omitted)"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "united = Table.read_table('https://raw.githubusercontent.com/data-8/materials-fa17/master/lec/united_summer2015.csv')\n",
+    "united"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "One flight departed 16 minutes early, and one was 580 minutes late. The other delay times were almost all between -10 minutes and 200 minutes, as the histogram below shows."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "-16"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "united.column('Delay').min()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "580"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "united.column('Delay').max()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "fe4718ae0fd64a3aaac7898cc69a5a28",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "A Jupyter Widget"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "a5d3a77049b14dcd908032b1dea0cd51",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "A Jupyter Widget"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "delay_opts = {\n",
+    "    'xlabel': 'Delay (minute)',\n",
+    "    'ylabel': 'Percent per minute',\n",
+    "    'xlim': (-20, 600),\n",
+    "    'ylim': (0, 0.045),\n",
+    "    'bins': 62,\n",
+    "}\n",
+    "\n",
+    "nbi.hist(united.column('Delay'), options=delay_opts)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "For the purposes of this section, it is enough to zoom in on the bulk of the data and ignore the 0.8% of flights that had delays of more than 200 minutes. This restriction is just for visual convenience; the table still retains all the data."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.008390596745027125"
+      ]
+     },
+     "execution_count": 6,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "united.where('Delay', are.above(200)).num_rows/united.num_rows"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "002b4b0b82f44802b6c0e156110608ed",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "A Jupyter Widget"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "e01c449aaa2543f296bf3ce6faa9e140",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "A Jupyter Widget"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "delay_opts = {\n",
+    "    'xlabel': 'Delay (minute)',\n",
+    "    'ylabel': 'Percent per minute',\n",
+    "    'xlim': (-20, 200),\n",
+    "    'ylim': (0, 0.045),\n",
+    "    'bins': 22,\n",
+    "}\n",
+    "\n",
+    "nbi.hist(united.column('Delay'), options=delay_opts)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The height of the [0, 10) bar is just under 3% per minute, which means that just under 30% of the flights had delays between 0 and 10 minutes. That is confirmed by counting rows: "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.2935985533453888"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "united.where('Delay', are.between(0, 10)).num_rows/united.num_rows"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Empirical Distribution of the Sample ###\n",
+    "\n",
+    "Let us now think of the 13,825 flights as a population, and draw random samples from it with replacement. It is helpful to package our analysis code into a function. The function `empirical_delays` takes the sample size as its argument and returns the array of sampled flight delays."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def empirical_hist_delay(sample_size):\n",
+    "    return united.sample(sample_size).column('Delay')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "As we saw with the dice, as the sample size increases, the empirical histogram of the sample more closely resembles the histogram of the population. Compare these histograms to the population histogram above."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8ff6b6131dfa45d9b9a78eca5fe19baa",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "A Jupyter Widget"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "811e890bacaa4297897a95bb77721dbb",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "A Jupyter Widget"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "nbi.hist(empirical_hist_delay,\n",
+    "         options=delay_opts,\n",
+    "         sample_size=widgets.ToggleButtons(options=[10, 100, 1000, 10000], description='Sample Size: '))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The most consistently visible discrepancies are among the values that are rare in the population. In our example, those values are in the the right hand tail of the distribution. But as the sample size increases, even those values begin to appear in the sample in roughly the correct proportions."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Convergence of the Empirical Histogram of the Sample ###\n",
+    "What we have observed in this section can be summarized as follows:\n",
+    "\n",
+    "For a large random sample, the empirical histogram of the sample resembles the histogram of the population, with high probability.\n",
+    "\n",
+    "This justifies the use of large random samples in statistical inference. The idea is that since a large random sample is likely to resemble the population from which it is drawn, quantities computed from the values in the sample are likely to be close to the corresponding quantities in the population."
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [default]",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.5.2"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}