j1-template 2022.5.0.rc1 → 2022.5.0.rc4

data/lib/starter_web/pages/public/_jupyter/notebooks/martin_renou_bqplot/data_files/state_codes.csv

@@ -0,0 +1,52 @@
+Name,FIPS,USPS
+Alabama,1,AL
+Alaska,2,AK
+Arizona,4,AZ
+Arkansas,5,AR
+California,6,CA
+Colorado,8,CO
+Connecticut,9,CT
+Delaware,10,DE
+District of Columbia,11,DC
+Florida,12,FL
+Georgia,13,GA
+Hawaii,15,HI
+Idaho,16,ID
+Illinois,17,IL
+Indiana,18,IN
+Iowa,19,IA
+Kansas,20,KS
+Kentucky,21,KY
+Louisiana,22,LA
+Maine,23,ME
+Maryland,24,MD
+Massachusetts,25,MA
+Michigan,26,MI
+Minnesota,27,MN
+Mississippi,28,MS
+Missouri,29,MO
+Montana,30,MT
+Nebraska,31,NE
+Nevada,32,NV
+New Hampshire,33,NH
+New Jersey,34,NJ
+New Mexico,35,NM
+New York,36,NY
+North Carolina,37,NC
+North Dakota,38,ND
+Ohio,39,OH
+Oklahoma,40,OK
+Oregon,41,OR
+Pennsylvania,42,PA
+Rhode Island,44,RI
+South Carolina,45,SC
+South Dakota,46,SD
+Tennessee,47,TN
+Texas,48,TX
+Utah,49,UT
+Vermont,50,VT
+Virginia,51,VA
+Washington,53,WA
+West Virginia,54,WV
+Wisconsin,55,WI
+Wyoming,56,WY

data/lib/starter_web/pages/public/_jupyter/notebooks/nbi-docs/.ipynb_checkpoints/nbi_docs_examples_central_limit_theorem-checkpoint.ipynb

@@ -0,0 +1,247 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# HIDDEN\n",
+    "from datascience import *\n",
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plots\n",
+    "plots.style.use('fivethirtyeight')\n",
+    "import math\n",
+    "import numpy as np\n",
+    "from scipy import stats\n",
+    "import ipywidgets as widgets\n",
+    "import nbinteract as nbi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### The Central Limit Theorem\n",
+    "Very few of the data histograms that we have seen in this course have been bell shaped.\n",
+    "When we have come across a bell shaped distribution, it has almost invariably been an\n",
+    "empirical histogram of a statistic based on a random sample.\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "**The Central Limit Theorem says that the probability distribution of the sum or average of a large random sample drawn with replacement will be roughly normal, *regardless of the distribution of the population from which the sample is drawn*.**\n",
+    "\n",
+    "As we noted when we were studying Chebychev's bounds, results that can be applied to random samples *regardless of the distribution of the population* are very powerful, because in data science we rarely know the distribution of the population.\n",
+    "\n",
+    "The Central Limit Theorem makes it possible to make inferences with very little knowledge about the population, provided we have a large random sample. That is why it is central to the field of statistical inference."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Proportion of Purple Flowers\n",
+    "\n",
+    "Recall Mendel's probability model for the colors of the flowers of a species of pea plant. The model says that the flower colors of the plants are like draws made at random with replacement from {Purple, Purple, Purple, White}.\n",
+    "\n",
+    "In a large sample of plants, about what proportion will have purple flowers? We would expect the answer to be about 0.75, the proportion purple in the model. And, because proportions are means, the Central Limit Theorem says that the distribution of the sample proportion of purple plants is roughly normal.\n",
+    "\n",
+    "We can confirm this by simulation. Let's simulate the proportion of purple-flowered plants in a sample of 200 plants."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/html": [
+       "<table border=\"1\" class=\"dataframe\">\n",
+       "    <thead>\n",
+       "        <tr>\n",
+       "            <th>Color</th>\n",
+       "        </tr>\n",
+       "    </thead>\n",
+       "    <tbody>\n",
+       "        <tr>\n",
+       "            <td>Purple</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>Purple</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>Purple</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>White </td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "</table>"
+      ],
+      "text/plain": [
+       "Color\n",
+       "Purple\n",
+       "Purple\n",
+       "Purple\n",
+       "White"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "colors = make_array('Purple', 'Purple', 'Purple', 'White')\n",
+    "\n",
+    "model = Table().with_column('Color', colors)\n",
+    "\n",
+    "model"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([0.765, 0.695, 0.69 , 0.76 , 0.78 ])"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "props = make_array()\n",
+    "\n",
+    "num_plants = 200\n",
+    "repetitions = 1000\n",
+    "\n",
+    "for i in np.arange(repetitions):\n",
+    "    sample = model.sample(num_plants)\n",
+    "    new_prop = np.count_nonzero(sample.column('Color') == 'Purple')/num_plants\n",
+    "    props = np.append(props, new_prop)\n",
+    "props[:5]"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "67c0c138c5c846efba2c9e32338801d0",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(interactive(children=(Output(),), _dom_classes=('widget-interact',)), Figure(axes=[Axis(label='…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "opts = {\n",
+    "    'title': 'Distribution of sample proportions',\n",
+    "    'xlabel': 'Sample Proportion',\n",
+    "    'ylabel': 'Percent per unit',\n",
+    "    'xlim': (0.64, 0.84),\n",
+    "    'ylim': (0, 25),\n",
+    "    'bins': 20,\n",
+    "}\n",
+    "nbi.hist(props, options=opts)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "There's that normal curve again, as predicted by the Central Limit Theorem, centered at around 0.75 just as you would expect.\n",
+    "\n",
+    "How would this distribution change if we increased the sample size? We can copy our sampling code into a function and then use interaction to see how the distribution changes as the sample size increases.\n",
+    "\n",
+    "We will keep the number of `repetitions` the same as before so that the two columns have the same length."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "def empirical_props(num_plants):\n",
+    "    props = make_array()\n",
+    "    for i in np.arange(repetitions):\n",
+    "        sample = model.sample(num_plants)\n",
+    "        new_prop = np.count_nonzero(sample.column('Color') == 'Purple')/num_plants\n",
+    "        props = np.append(props, new_prop)\n",
+    "    return props"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "539198ed74234f20ac5d449a617305f8",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(interactive(children=(ToggleButtons(description='num_plants', options=(100, 200, 400, 800), val…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "nbi.hist(empirical_props, options=opts,\n",
+    "         num_plants=widgets.ToggleButtons(options=[100, 200, 400, 800]))"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "All of the above distributions are approximately normal but become more narrow as the sample size increases. For example, the proportions based on a sample size of 800 are more tightly clustered around 0.75 than those from a sample size of 200. Increasing the sample size has decreased the variability in the sample proportion."
+   ]
+  }
+ ],
+ "metadata": {
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}

data/lib/starter_web/pages/public/_jupyter/notebooks/nbi-docs/.ipynb_checkpoints/nbi_docs_examples_variability_of_the_sample_mean-checkpoint.ipynb

@@ -0,0 +1,323 @@
+{
+ "cells": [
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "# HIDDEN\n",
+    "from datascience import *\n",
+    "import numpy as np\n",
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plots\n",
+    "plots.style.use('fivethirtyeight')\n",
+    "from ipywidgets import interact, interactive, fixed, interact_manual\n",
+    "import ipywidgets as widgets\n",
+    "import nbinteract as nbi"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### The Variability of the Sample Mean\n",
+    "By the Central Limit Theorem, the probability distribution of the mean of a large random sample is roughly normal. The bell curve is centered at the population mean. Some of the sample means are higher, and some lower, but the deviations from the population mean are roughly symmetric on either side, as we have seen repeatedly. Formally, probability theory shows that the sample mean is an *unbiased* estimate of the population mean.\n",
+    "\n",
+    "In our simulations, we also noticed that the means of larger samples tend to be more tightly clustered around the population mean than means of smaller samples. In this section, we will quantify the variability of the sample mean and develop a relation between the variability and the sample size."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let's start with our table of flight delays. The mean delay is about 16.7 minutes, and the distribution of delays is skewed to the right."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "united = Table.read_table('https://raw.githubusercontent.com/data-8/materials-fa17/master/lec/united_summer2015.csv')\n",
+    "delay = united.select('Delay')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "16.658155515370705"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "pop_mean = np.mean(delay.column('Delay'))\n",
+    "pop_mean"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "ecd1c015c6c447359952808947947b1a",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(interactive(children=(Output(),), _dom_classes=('widget-interact',)), Figure(axes=[Axis(label='…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "delay_opts = {\n",
+    "    'xlabel': 'Delay (minute)',\n",
+    "    'ylabel': 'Percent per minute',\n",
+    "    'xlim': (-20, 200),\n",
+    "    'ylim': (0, 0.037),\n",
+    "    'bins': 22,\n",
+    "}\n",
+    "\n",
+    "nbi.hist(united.column('Delay'), options=delay_opts)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Now let's take random samples and look at the probability distribution of the sample mean. As usual, we will use simulation to get an empirical approximation to this distribution.\n",
+    "\n",
+    "We will define a function `simulate_sample_mean` to do this, because we are going to vary the sample size later. The arguments are the name of the table, the label of the column containing the variable, the sample size, and the number of simulations."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {},
+   "outputs": [],
+   "source": [
+    "\"\"\"Empirical distribution of random sample means\"\"\"\n",
+    "\n",
+    "def simulate_sample_mean(table, label, sample_size, repetitions=1000):\n",
+    "    means = make_array()\n",
+    "\n",
+    "    for i in range(repetitions):\n",
+    "        new_sample = table.sample(sample_size)\n",
+    "        new_sample_mean = np.mean(new_sample.column(label))\n",
+    "        means = np.append(means, new_sample_mean)\n",
+    "    \n",
+    "    \n",
+    "    # Print all relevant quantities\n",
+    "    print(\"Sample size: \", sample_size)\n",
+    "    print(\"Population mean:\", np.mean(table.column(label)))\n",
+    "    print(\"Average of sample means: \", np.mean(means))\n",
+    "    print(\"Population SD:\", np.std(table.column(label)))\n",
+    "    print(\"SD of sample means:\", np.std(means))\n",
+    "    return means"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Let us simulate the mean of a random sample of 100 delays, then of 400 delays, and finally of 625 delays. We will perform 1000 repetitions of each of these process."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can interact with the buttons below to show the distribution of sample means for different sample sizes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "application/vnd.jupyter.widget-view+json": {
+       "model_id": "8a5e930638df4713b4adab229f07c657",
+       "version_major": 2,
+       "version_minor": 0
+      },
+      "text/plain": [
+       "VBox(children=(interactive(children=(ToggleButtons(description='sample_size', options=(100, 400, 625), value=1…"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "means_opts = {\n",
+    "    'xlabel': 'Sample Means',\n",
+    "    'ylabel': 'Percent per unit',\n",
+    "    'xlim': (5, 35),\n",
+    "    'ylim': (0, 0.25),\n",
+    "    'bins': 30,\n",
+    "}\n",
+    "\n",
+    "nbi.hist(simulate_sample_mean, table=fixed(delay), label=fixed('Delay'),\n",
+    "         sample_size=widgets.ToggleButtons(options=[100, 400, 625]),\n",
+    "         options=means_opts)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "You can see the Central Limit Theorem in action – the histograms of the sample means are roughly normal, even though the histogram of the delays themselves is far from normal.\n",
+    "\n",
+    "You can also see that each of the three histograms of the sample means is centered very close to the population mean. In each case, the \"average of sample means\" is very close to 16.66 minutes, the population mean. Both values are provided in the printout above each histogram. As expected, the sample mean is an unbiased estimate of the population mean."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### The SD of All the Sample Means\n",
+    "\n",
+    "You can also see that the histograms get narrower, and hence taller, as the sample size increases. We have seen that before, but now we will pay closer attention to the measure of spread.\n",
+    "\n",
+    "The SD of the population of all delays is about 40 minutes."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {},
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "39.480199851609314"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "pop_sd = np.std(delay.column('Delay'))\n",
+    "pop_sd"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Take a look at the SDs in the sample mean histograms above. In all three of them, the SD of the population of delays is about 40 minutes, because all the samples were taken from the same population.\n",
+    "\n",
+    "Now look at the SD of all 10,000 sample means, when the sample size is 100. That SD is about one-tenth of the population SD. When the sample size is 400, the SD of all the sample means is about one-twentieth of the population SD. When the sample size is 625, the SD of the sample means is about one-twentyfifth of the population SD.\n",
+    "\n",
+    "It seems like a good idea to compare the SD of the empirical distribution of the sample means to the quantity \"population SD divided by the square root of the sample size.\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "Here are the numerical values. For each sample size in the first column, 10,000 random samples of that size were drawn, and the 10,000 sample means were calculated. The second column contains the SD of those 10,000 sample means. The third column contains the result of the calculation \"population SD divided by the square root of the sample size.\""
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "The values in the second and third columns are very close. If we plot each of those columns with the sample size on the horizontal axis, the two graphs are essentially indistinguishable."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "There really are two curves there. But they are so close to each other that it looks as though there is just one.\n",
+    "\n",
+    "What we are seeing is an instance of a general result. Remember that the graph above is based on 10,000 replications for each sample size. But there are many more than 10,000 samples of each size. The probability distribution of the sample mean is based on the means of *all possible samples* of a fixed size.\n",
+    "\n",
+    "**Fix a sample size.** If the samples are drawn at random with replacement from the population, then\n",
+    "$$\n",
+    "{\\mbox{SD of all possible sample means}} ~=~\n",
+    "\\frac{\\mbox{Population SD}}{\\sqrt{\\mbox{sample size}}}\n",
+    "$$\n",
+    "\n",
+    "This is the standard deviation of the averages of all the possible samples that could be drawn. **It measures roughly how far off the sample means are from the population mean.**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### The Central Limit Theorem for the Sample Mean\n",
+    "If you draw a large random sample with replacement from a population, then, regardless of the distribution of the population, the probability distribution of the sample mean is roughly normal, centered at the population mean, with an SD equal to the population SD divided by the square root of the sample size."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### The Accuracy of the Sample Mean\n",
+    "The SD of all possible sample means measures how variable the sample mean can be. As such, it is taken as a measure of the accuracy of the sample mean as an estimate of the population mean. The smaller the SD, the more accurate the estimate.\n",
+    "\n",
+    "The formula shows that:\n",
+    "- The population size doesn't affect the accuracy of the sample mean. The population size doesn't appear anywhere in the formula.\n",
+    "- The population SD is a constant; it's the same for every sample drawn from the population. The sample size can be varied. Because the sample size appears in the denominator, the variability of the sample mean *decreases* as the sample size increases, and hence the accuracy increases."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### The Square Root Law\n",
+    "From the table of SD comparisons, you can see that the SD of the means of random samples of 25 flight delays is about 8 minutes. If you multiply the sample size by 4, you'll get samples of size 100. The SD of the means of all of those samples is about 4 minutes. That's smaller than 8 minutes, but it's not 4 times as small; it's only 2 times as small. That's because the sample size in the denominator has a square root over it. The sample size increased by a factor of 4, but the SD went down by a factor of $2 = \\sqrt{4}$. In other words, the accuracy went up by a factor of $2 = \\sqrt{4}$.\n",
+    "\n",
+    "In general, when you multiply the sample size by a factor, the accuracy of the sample mean goes up by the square root of that factor.\n",
+    "\n",
+    "So to increase accuracy by a factor of 10, you have to multiply sample size by a factor of 100. Accuracy doesn't come cheap!"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python 3",
+   "language": "python",
+   "name": "python3"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 3
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython3",
+   "version": "3.7.9"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 1
+}