my-markdown-library 0.1.0

data/F24Lec_MD/lec11/11.1-slr.md

@@ -0,0 +1,305 @@
+---
+title: "11.1-slr"
+type: lecture-notebook
+week: 11
+source_path: "/Users/ericvandusen/Documents/Data88E-ForTraining/F24Lec_NBs/lec11/11.1-slr.ipynb"
+---
+
+<table style="width: 100%;" id="nb-header">
+    <tr style="background-color: transparent;"><td>
+        <img src="https://data-88e.github.io/assets/images/blue_text.png" width="250px" style="margin-left: 0;" />
+    </td><td>
+        <p style="text-align: right; font-size: 10pt;"><strong>Economic Models</strong>, Fall 24<br>
+            Dr. Eric Van Dusen <br>
+            Vaidehi Bulusu <br>
+        <br>
+</table>
+
+# Lecture Notebook 11.1: Simple Linear Regression
+
+```python
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from matplotlib import patches
+import statsmodels.api as sm
+import warnings
+warnings.simplefilter("ignore")
+from datascience import *
+%matplotlib inline
+from sklearn.metrics import mean_squared_error
+import ipywidgets as widgets
+from ipywidgets import interact
+```
+
+# Simple Linear Regression
+
+Let's learn about simple linear regression using a dataset you're already familiar with from week 2: the dataset on the price and quantity demanded of avocados. Recall that in week 2, we used `np.polyfit()` to find the slope and intercept. Here, we'll use regression to calculate the slope and intercept from scratch and get the same results!
+
+Go over this lecture notebook along with the slides!
+
+```python
+av_df = Table.read_table("avocados.csv")
+av_df.show(5)
+```
+
+## Scatter Plot
+
+To investigate the relationship between price and quantity demanded, we can start by creating a scatter plot:
+
+```python
+av_df.scatter("Average Price", "Total Volume")
+plt.ylabel("Quantity Demanded")
+plt.xlabel("Price")
+plt.title("Demand Curve")
+plt.show()
+```
+
+By looking at the scatter plot, we get a sense that there's a somewhat strong negative relationship between price and quantity demanded. Notice that the points don't all lie on a straight line as it's not a perfect association. This is because there is some random variation in the data plus there are factors other than price that affect the quantity demanded (we'll talk more about this later).
+
+## Correlation Coefficient
+
+To quantify the association and give it a numerical value, we can calculate the correlation coefficient between price and quantity demanded. Recall from DATA 8 that the correlation coefficient only measures **linear** association so for the purpose of this class, we are assuming that the relationship between the price and quantity demanded of avocados is linear (in reality, it may be nonlinear).
+
+Let's start by defining a function to calculate the correlation coefficient.
+
+```python
+# Assumes that array_x and array_y are in standard units
+def correlation(array_x, array_y):
+    return np.mean(array_x * array_y)
+```
+
+The first step in calculating the correlation coefficient is converting the data to standard units. When you express a value in standard units, you are basically standardizing the variable by expressing it in terms of the number of standard deviations it is above or below the mean (you can think of this as the "distance" from the mean). For example, the first value of the price shown below is around 1.2 standard deviations below the mean price.
+
+Expressing the data in standard units removes the measured units (e.g. dollars) from the data. So, the correlation coefficient is not affected by a change in the units/scale (e.g. if you convert the prices from USD to CAD).
+
+```python
+# Expressing price in standard units
+price_su = (av_df["Average Price"] - np.mean(av_df["Average Price"]))/np.std(av_df["Average Price"])
+price_su[:5]
+```
+
+```python
+# Expressing quantity in standard units
+quantity_su = (av_df["Total Volume"] - np.mean(av_df["Total Volume"]))/np.std(av_df["Total Volume"])
+quantity_su[:5]
+```
+
+Now we can use the function we defined to calculate the correlation coefficient:
+
+```python
+r = correlation(price_su, quantity_su)
+r
+```
+
+As we expected, there is a pretty strong negative correlation between price and demand! It aligns with what we saw in the scatter plot.
+
+## Regression Equation
+
+Let's use the correlation coefficient to build the regression equation. The first step is to calculate the slope and intercept:
+
+```python
+slope = r * np.std(av_df["Total Volume"])/np.std(av_df["Average Price"])
+slope
+```
+
+```python
+intercept = np.mean(av_df["Total Volume"]) - (slope * np.mean(av_df["Average Price"]))
+intercept
+```
+
+We can now write the regression equation for the relationship between price and quantity demanded.
+
+$$\hat{quantity} = 1446951.641 -476412.719\hat{price}$$
+
+This equation allows you to do 2 things:
+
+- Quantify the association between price and quantity demanded: as price increases by $1, the quantity demanded decreases by around 476,000 units (avocados)
+
+
+- Predict values of quantity demanded: this is especially useful for values of price that are not already in the dataset
+
+Note that the regression line is the line of best fit for your data (more on this later). We can plot this line of best fit on the scatter plot from before:
+
+```python
+av_df.scatter("Average Price", "Total Volume")
+plt.plot(av_df["Average Price"], slope * av_df["Average Price"] + intercept, color='tab:orange')
+plt.show()
+```
+
+Does this graph look familiar? You've already done a similar analysis in week 2 when you used `np.polyfit` to find the slope and intercept of the line of best fit!
+
+```python
+params = np.polyfit(av_df["Average Price"], av_df["Total Volume"], 1)
+params
+```
+
+The slope and intercept you get from `np.polyfit` should be very close to the slope and intercept from before (there might be some tiny variation). This is because under the hood, np.polyfit is using simple linear regression to calculate the parameters.
+
+```python
+slope_param = params[0]
+np.round(slope_param, 5) == np.round(slope, 5)
+```
+
+```python
+intercept_param = params[1]
+np.round(intercept_param, 5) == np.round(intercept, 5)
+```
+
+## RMSE
+
+Let's use the slope and intercept to generate predictions for the quantity demanded:
+
+```python
+predicted_quantity = intercept + (slope * av_df["Average Price"])
+predicted_quantity[:5]
+```
+
+We want our predictions to be as accurate as possible. One metric we can use to measure the accuracy of our predictions is the root mean squared error (RMSE) which gives us a sense of how far our predictions are from the actual values, on average.
+
+Let's define a function for calculating the RMSE of our data (note that this function is specific to the avocados dataset).
+
+```python
+def rmse_p_q(slope, intercept):
+    predicted_y = intercept + (slope * av_df["Average Price"])
+    return (np.mean((av_df["Total Volume"] - predicted_y)**2))**0.5
+```
+
+```python
+rmse_1 = rmse_p_q(slope, intercept)
+rmse_1
+```
+
+On average, our predictions are off by about $132,000. This is a pretty high RMSE! This shows that even though price and quantity demanded have a pretty high correlation (about 0.72), it doesn't necessarily mean that price is a good predictor for the quantity demanded.
+
+We said earlier that the regression line is the line of best fit. The reason why is that the regression line minimizes the RMSE of your data. In other words, among all the lines you can use to model your data, the regression line is the one that is the most accurate as it minimizes RMSE – this is why this technique is called (ordinary) least squares regression.
+
+We can test this using the minimize function. Recall that minimize takes a function as the input as gives you values of the arguments of that function that minimize it. Below, we would expect the output of the minimize function to be similar to the slope and intercept we calculated:
+
+```python
+minimum = minimize(rmse_p_q)
+minimum
+```
+
+The slope and intercept from the minimize function is very similar to the slope and intercept we calculated, as expected!
+
+When you perform regression in Python (e.g. when you use `np.polyfit`), it tries to find the values of the slope and intercept that minimize the RMSE. You can build your intuition for how it does this by playing around with this slider from [last semester's notebook](https://datahub.berkeley.edu/hub/user-redirect/git-pull?repo=https%3A%2F%2Fgithub.com%2Fdata-88e%2Fecon-sp21&urlpath=tree%2Fecon-sp21%2Flec%2Flec12%2Flec12.ipynb&branch=main) (you can also find the link in on last semester's [website](https://data-88e.github.io/sp21/), in econometrics week). See if you can find values for the slope and intercept that minimize the squared error area.
+
+```python
+d = Table().with_columns(
+    'x', make_array(0,  1,  2,  3,  4),
+    'y', make_array(1, .5, -1,  2, -3))
+
+def rmse(target, pred):
+    return np.sqrt(mean_squared_error(target, pred))
+
+def plot_line_and_errors(slope, intercept):
+    print("RMSE:", rmse(slope * d.column('x') + intercept, d.column('y')))
+    plt.figure(figsize=(5,5))
+    points = make_array(-2, 7)
+    p = plt.plot(points, slope*points + intercept, color='orange', label='Proposed line')
+    ax = p[0].axes
+    
+    predicted_ys = slope*d.column('x') + intercept
+    diffs = predicted_ys - d.column('y')
+    for i in np.arange(d.num_rows):
+        x = d.column('x').item(i)
+        y = d.column('y').item(i)
+        diff = diffs.item(i)
+        
+        if diff > 0:
+            bottom_left_x = x
+            bottom_left_y = y
+        else:
+            bottom_left_x = x + diff
+            bottom_left_y = y + diff
+        
+        ax.add_patch(patches.Rectangle(make_array(bottom_left_x, bottom_left_y), abs(diff), abs(diff), color='red', alpha=.3, label=('Squared error' if i == 0 else None)))
+        plt.plot(make_array(x, x), make_array(y, y + diff), color='red', alpha=.6, label=('Error' if i == 0 else None))
+    
+    plt.scatter(d.column('x'), d.column('y'), color='blue', label='Points')
+    
+    plt.xlim(-4, 8)
+    plt.ylim(-6, 6)
+    plt.gca().set_aspect('equal', adjustable='box')
+    
+    plt.legend(bbox_to_anchor=(1.8, .8))
+    plt.show()
+
+interact(plot_line_and_errors, slope=widgets.FloatSlider(min=-4, max=4, step=.1), intercept=widgets.FloatSlider(min=-4, max=4, step=.1));
+```
+
+## Regression in Python
+
+So far, we've looked at the DATA 8 version of regression in which we calculated the correlated coefficient and used it to find the slope and intercept of our regression line.
+
+Now we can look at how regression is actually performed in Python using a library called `statsmodels`.
+
+```python
+# import statsmodels.api as sm – see first cell in the notebook
+x_1 = av_df.column("Average Price") # Selecting the independent variable(s)
+y_1 = av_df.column("Total Volume") # Selecting the dependent variable
+model_1 = sm.OLS(y_1, sm.add_constant(x_1)) # Performing OLS regression – make sure to include intercept using sm.add_constant
+result_1 = model_1.fit() # Fitting the model
+result_1.summary() # Showing a summary of the results of the regression
+```
+
+Let's look at the parameters (slope and intercept) of the regression and see if they are close to the parameters we calculated before:
+
+```python
+type(result_1.params)
+```
+
+```python
+np.round(result_1.params[0], 5) == np.round(intercept, 5)
+```
+
+```python
+np.round(result_1.params[1], 5) == np.round(slope, 5)
+```
+
+They are similar, as we expected!
+
+## Hypothesis Testing
+
+Since we are interested in whether or not there is a substantial association between x and y (in this case, price and quantity demanded), we can frame this as a hypothesis testing question:
+
+- The null hypothesis would be that there isn't a substantial association between x and y ($\beta = 0$)
+
+- The alternative hypothesis would be that there is a substantial association between x and y ($\beta ≠ 0$)
+
+Go back to the regression output from before: notice that it shows a 95% confidence interval for the slope. Let's try to visualize this confidence interval by creating a distribution of the regression slopes:
+
+```python
+slopes_array = make_array()
+for i in np.arange(5000):
+    bootstrapped_sample = av_df.select("Average Price", "Total Volume").sample()
+    x = bootstrapped_sample.column("Average Price")
+    y = bootstrapped_sample.column("Total Volume")
+    results = sm.OLS(y, sm.add_constant(x)).fit()
+    slopes_array = np.append(results.params[1], slopes_array)
+Table().with_column("Slopes", slopes_array).hist()
+print("Center of the distribution (mean):", np.mean(slopes_array))
+print("Standard deviation:", np.std(slopes_array))
+```
+
+```python
+ci_lower_bound = percentile(2.5, slopes_array)
+ci_lower_bound
+```
+
+```python
+ci_upper_bound = percentile(97.5, slopes_array)
+ci_upper_bound
+```
+
+Notice that the center of the distribution (average of the slopes) is similar to the slope from the regression output. The lower and upper bounds of the confidence interval are also similar.
+
+Recall that if the confidence interval contains 0, it means that there is evidence for the null hypothesis at the 5% significance level. If it doesn't contain 0, it means that there is evidence against the null hypothesis at the 5% level.
+
+The confidence interval in the regression output above doesn't contain 0 so there's evidence that price and quantity demanded have a significant association. This makes sense as they have a relatively high correlation coefficient.
+
+```python
+
+```
+

data/F24Lec_MD/lec11/11.2-mlr.md

@@ -0,0 +1,171 @@
+---
+title: "11.2-mlr"
+type: lecture-notebook
+week: 11
+source_path: "/Users/ericvandusen/Documents/Data88E-ForTraining/F24Lec_NBs/lec11/11.2-mlr.ipynb"
+---
+
+<table style="width: 100%;" id="nb-header">
+    <tr style="background-color: transparent;"><td>
+        <img src="https://data-88e.github.io/assets/images/blue_text.png" width="250px" style="margin-left: 0;" />
+    </td><td>
+        <p style="text-align: right; font-size: 10pt;"><strong>Economic Models</strong>, Fall 24 <br>
+            Dr. Eric Van Dusen <br>
+            Vaidehi Bulusu <br>
+        <br>
+</table>
+
+# Lecture Notebook 11.2: Multiple Linear Regression
+
+```python
+import numpy as np
+import pandas as pd
+import matplotlib.pyplot as plt
+from matplotlib import patches
+import statsmodels.api as sm
+import warnings
+warnings.simplefilter("ignore")
+from datascience import *
+%matplotlib inline
+from sklearn.metrics import mean_squared_error
+import ipywidgets as widgets
+from ipywidgets import interact
+```
+
+# Multiple Linear Regression
+
+In simple linear regression, we're only considering one independent variable. We're essentially assuming that this is the only variable that affects our dependent variable. Going back to the price and quantity demanded example, while price does have a huge effect on demand, there are other factors that also affect demand including income taxes, sales taxes, advertising, prices of related goods, etc. In the context of regression, because we are not including these variables in our model, they are called **omitted variables**.
+
+Omitted variables cause 2 main problems:
+
+- The regression parameters end up being inaccurate (biased) because of something called omitted variable bias. Your regression estimates for the slope and intercept are higher or lower than the actual values because of omitted variables (we are generally only concerned about the slope).
+
+
+- They prevent you from inferring a causal association between the independent and dependent variables. In other words, you can't say that it's *because* of an increase in price that your quantity demanded decreased as there are so many other factors – like the change in the price of a related good – that could be causing the decrease in demand.
+
+To try to eliminate omitted variable bias from our model, we take simple linear regression a step further: to multiple linear regression. In multiple linear regression, we are including more indepedent variables – variables we think are confounding variable that we've omitted – to reduce omitted variable bias.
+
+Let's look at multiple linear regression in Python using a new dataset on earnings and various other factors (check out the [data description](https://www.princeton.edu/~mwatson/Stock-Watson_3u/Students/EE_Datasets/CPS12_Description.pdf)) [Stock and Watson datasets](https://www.princeton.edu/~mwatson/Stock-Watson_3u/Students/Stock-Watson-EmpiricalExercises-DataSets.htm).
+
+```python
+cps = Table.read_table("CPS.csv")
+cps.show(5)
+```
+
+Say we want to look at the relationship between age and earnings (the `ahe` column which is the average hourly earnings). We would expect whether or not a person has a bachelor's degree to be a confounding variable as those with a bachelor's degree typically earn more than those with only a high school degree. This is how we would do multiple linear regression:
+
+```python
+x_2 = cps.select("bachelor", "age").values # This is how we include multiple independent variables in our model
+y_2 = cps.column("ahe")
+model_2 = sm.OLS(y_2, sm.add_constant(x_2))
+result_2 = model_2.fit()
+result_2.summary()
+```
+
+## Dummy Variables
+
+One type of variable commonly used in econometrics are dummy variables. These are variables that take a value of either 0 or 1 to indicate the presence of absence of a category. For example, take col: it takes the value of 1 to indicate that a person went to college and 0 to indicate that a person didn't go to college.
+
+Let's do a regression of `ahe` on only `bachelor` (the dummy variable).
+
+```python
+x_3 = cps.select("bachelor").values
+y_3 = cps.column("ahe")
+model_3 = sm.OLS(y_3, sm.add_constant(x_3))
+result_3 = model_3.fit()
+result_3.summary()
+```
+
+The coefficient (or slope) on `bachelor` is this:
+
+```python
+result_3.params[1]
+```
+
+This means that the people in our sample with a bachelor's degree earn around $6.583 more per hour than those with only a high school degree.
+
+An interesting fact about dummy variables is that we can calculate this coefficient another way:
+
+```python
+# Filter for bachelor = 1 and find mean earnings
+b_1_mean = np.mean(cps.where("bachelor", 1).column("ahe"))
+
+# Filter for bachelor = 0 and find mean earnings
+b_0_mean = np.mean(cps.where("bachelor", 0).column("ahe"))
+
+# Take the difference in the mean earnings
+diff = b_1_mean - b_0_mean
+diff
+```
+
+```python
+np.round(result_3.params[1], 5) == np.round(diff, 5)
+```
+
+These two values are pretty much the same! This is because the coefficient on a dummy x-variable is just equal to the difference of the mean of the y-variable when x = 1 and x = 0.
+
+### `pd.get_dummies()`
+
+We can convert categorical variables in our dataset to dummy variables using the pd.get_dummies() function. This function gives you a table showing the presence or absence of dummy variables for that category for each observation in the dataset. Here's an example of converting the year values to dummies. 1 indicates that the person was surveyed in that year (e.g. the first 4 people in the dataset were surveyed in 1992 according to the table below).
+
+```python
+year = cps.column("year")
+np.unique(year)
+```
+
+```python
+year_dummies = pd.get_dummies(year)
+year_dummies.head()
+```
+
+Let's do the same for the age variable. Take the first row as an example: since 1 is under 29, it means that that person is 29 years old.
+
+```python
+age = cps.column("age")
+np.unique(age)
+```
+
+```python
+age_dummies = pd.get_dummies(age)
+age_dummies.head()
+```
+
+### Dummy Variable Trap
+
+One problem you may run into when using dummy variables is called the dummy variable trap. This happens when you include variables for all the values of the dummy variable: for example, you include col, which takes on a value of 1 if the person went to college, as well as notcol, which takes on a value of 1 if the person didn't go to college.
+
+This causes redundancy as you can express one independent variable as a linear combination of another independent variable. In this case:
+
+$$ notcol = 1 - col$$
+
+This means that there is a perfect correlation between these two independent variables which reuslts in perfect multicollinearity. In general, multicollinearity occurs any time there is a high correlation between the independent variables in your model. It causes your regression estimates to be highly inaccurate.
+
+Let's see what happens in Python when multicollinearity happens:
+
+```python
+year_dummies["ahe"] = np.array(cps.column("ahe"))
+year_dummies.head()
+```
+
+```python
+x_3 = year_dummies[[1992, 2008]] # We are including dummy variables for each value of year
+y_3 = year_dummies["ahe"]
+model_3 = sm.OLS(y_3, sm.add_constant(x_3))
+result_3 = model_3.fit()
+result_3.summary()
+```
+
+Python detected the multicollinearity and gave you a warning. A solution is to just drop one of the dummy variables.
+
+```python
+x_3 = year_dummies[[1992, 2008]] # We are including dummy variables for each value of year
+y_3 = year_dummies["ahe"]
+model_3 = sm.OLS(y_3, x_3)
+result_3 = model_3.fit()
+result_3.summary()
+```
+
+```python
+
+```
+