scikit-doc 0.1.0__py3-none-any.whl
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- scikit_doc/__init__.py +328 -0
- scikit_doc/data/NOTEBOOK_EXAMPLES.md +3355 -0
- scikit_doc/data/SKLEARN_ROOT_HELP_1_6_1.txt +412 -0
- scikit_doc/data/USER_GUIDE.md +1088 -0
- scikit_doc/data/WORKED_EXAMPLES.md +4038 -0
- scikit_doc/data/examples.json +1864 -0
- scikit_doc/examples/customer_preprocessing.py +34 -0
- scikit_doc/examples/data_inspection.py +45 -0
- scikit_doc/examples/feature_selection.py +37 -0
- scikit_doc/examples/flight_features.py +49 -0
- scikit_doc/examples/house_cleaning.py +64 -0
- scikit_doc/examples/model_search.py +60 -0
- scikit_doc/examples/pca_ridgecv.py +22 -0
- scikit_doc/examples/regression_models.py +32 -0
- scikit_doc-0.1.0.dist-info/METADATA +21 -0
- scikit_doc-0.1.0.dist-info/RECORD +19 -0
- scikit_doc-0.1.0.dist-info/WHEEL +5 -0
- scikit_doc-0.1.0.dist-info/licenses/LICENSE +21 -0
- scikit_doc-0.1.0.dist-info/top_level.txt +1 -0
|
@@ -0,0 +1,4038 @@
|
|
|
1
|
+
# Worked examples
|
|
2
|
+
|
|
3
|
+
This collection reorganizes every technical exercise from the five supplied notebooks and the solved housing reference into topic-oriented examples. Dataset selectors and grading labels are omitted; each task appears as a standalone documentation-style example.
|
|
4
|
+
|
|
5
|
+
The notebook code fragments assume the source notebook has already loaded the relevant DataFrame and completed prerequisite transformations. The companion examples state their required variables explicitly. Recorded outputs are included only when present in the supplied source.
|
|
6
|
+
|
|
7
|
+
**Total worked examples: 127**
|
|
8
|
+
|
|
9
|
+
## Customer value records
|
|
10
|
+
|
|
11
|
+
### How many records (rows) are there in the dataset
|
|
12
|
+
|
|
13
|
+
**Task**
|
|
14
|
+
|
|
15
|
+
How many records (rows) are there in the dataset?
|
|
16
|
+
|
|
17
|
+
**Step-by-step approach**
|
|
18
|
+
|
|
19
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
20
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
21
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
22
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
23
|
+
|
|
24
|
+
**Solution**
|
|
25
|
+
|
|
26
|
+
```python
|
|
27
|
+
# Solution
|
|
28
|
+
print(df.shape[0])
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
**Recorded result**
|
|
32
|
+
|
|
33
|
+
```text
|
|
34
|
+
6257
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
---
|
|
38
|
+
|
|
39
|
+
### What are the unique values of the feature `qualification` in the dataset
|
|
40
|
+
|
|
41
|
+
**Task**
|
|
42
|
+
|
|
43
|
+
What are the unique values of the feature `qualification` in the dataset?
|
|
44
|
+
|
|
45
|
+
**Step-by-step approach**
|
|
46
|
+
|
|
47
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
48
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
49
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
50
|
+
|
|
51
|
+
**Solution**
|
|
52
|
+
|
|
53
|
+
```python
|
|
54
|
+
# Solution
|
|
55
|
+
print(df['qualification'].unique())
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
**Recorded result**
|
|
59
|
+
|
|
60
|
+
```text
|
|
61
|
+
['High School' 'Bachelor' 'Others']
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
### Which of the following columns have numerical (int/float) data
|
|
67
|
+
|
|
68
|
+
**Task**
|
|
69
|
+
|
|
70
|
+
Which of the following columns have numerical (int/float) data?
|
|
71
|
+
|
|
72
|
+
**Step-by-step approach**
|
|
73
|
+
|
|
74
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
75
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
76
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
77
|
+
|
|
78
|
+
**Solution**
|
|
79
|
+
|
|
80
|
+
```python
|
|
81
|
+
# Solution
|
|
82
|
+
print(df.dtypes)
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
**Recorded result**
|
|
86
|
+
|
|
87
|
+
```text
|
|
88
|
+
id int64
|
|
89
|
+
gender object
|
|
90
|
+
area object
|
|
91
|
+
qualification object
|
|
92
|
+
income float64
|
|
93
|
+
marital_status int64
|
|
94
|
+
vintage int64
|
|
95
|
+
claim_amount float64
|
|
96
|
+
num_policies object
|
|
97
|
+
policy object
|
|
98
|
+
type_of_policy object
|
|
99
|
+
cltv int64
|
|
100
|
+
dtype: object
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
---
|
|
104
|
+
|
|
105
|
+
### Plot the `heatmap` of the correlation matrix of the numerical columns and mark the pair which has the **lowest (most negative)**…
|
|
106
|
+
|
|
107
|
+
**Task**
|
|
108
|
+
|
|
109
|
+
Plot the `heatmap` of the correlation matrix of the numerical columns and mark the pair which has the **lowest (most negative)** correlation value.
|
|
110
|
+
|
|
111
|
+
**Step-by-step approach**
|
|
112
|
+
|
|
113
|
+
1. Select only the numeric columns before calculating correlations; strings and categories should not be treated as continuous measurements.
|
|
114
|
+
2. Calculate the requested correlation matrix and ignore its diagonal, because every feature is perfectly correlated with itself.
|
|
115
|
+
3. For the most negative pair, sort the off-diagonal values in ascending order; draw the heatmap only when a visual is requested.
|
|
116
|
+
|
|
117
|
+
**Solution**
|
|
118
|
+
|
|
119
|
+
```python
|
|
120
|
+
# Solution
|
|
121
|
+
corr = df.select_dtypes(include='number').corr()
|
|
122
|
+
plt.figure(figsize=(8, 6))
|
|
123
|
+
sns.heatmap(corr, annot=True, fmt='.3f', cmap='coolwarm')
|
|
124
|
+
plt.show()
|
|
125
|
+
# Locate the most negative off-diagonal pair
|
|
126
|
+
pairs = corr.where(~np.eye(len(corr), dtype=bool)).stack()
|
|
127
|
+
print(pairs.sort_values().head(4))
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
---
|
|
131
|
+
|
|
132
|
+
### Which of the following features has the **highest number** of `missing` values
|
|
133
|
+
|
|
134
|
+
**Task**
|
|
135
|
+
|
|
136
|
+
Which of the following features has the **highest number** of `missing` values?
|
|
137
|
+
|
|
138
|
+
**Step-by-step approach**
|
|
139
|
+
|
|
140
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
141
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
142
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
143
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
144
|
+
|
|
145
|
+
**Solution**
|
|
146
|
+
|
|
147
|
+
```python
|
|
148
|
+
# Solution
|
|
149
|
+
print(df.isna().sum().sort_values(ascending=False))
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
**Recorded result**
|
|
153
|
+
|
|
154
|
+
```text
|
|
155
|
+
income 401
|
|
156
|
+
area 384
|
|
157
|
+
policy 372
|
|
158
|
+
claim_amount 360
|
|
159
|
+
qualification 0
|
|
160
|
+
gender 0
|
|
161
|
+
id 0
|
|
162
|
+
marital_status 0
|
|
163
|
+
vintage 0
|
|
164
|
+
num_policies 0
|
|
165
|
+
type_of_policy 0
|
|
166
|
+
cltv 0
|
|
167
|
+
dtype: int64
|
|
168
|
+
```
|
|
169
|
+
|
|
170
|
+
---
|
|
171
|
+
|
|
172
|
+
### Break the dataset into features(`X`) and label (`y`), where the column `cltv` goes to `y` and the rest of the columns go to `X`
|
|
173
|
+
|
|
174
|
+
**Task**
|
|
175
|
+
|
|
176
|
+
Break the dataset into features(`X`) and label (`y`), where the column `cltv` goes to `y` and the rest of the columns go to `X`. Enter the **median** value of `cltv` column?
|
|
177
|
+
|
|
178
|
+
**Step-by-step approach**
|
|
179
|
+
|
|
180
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
181
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
182
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
183
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
184
|
+
|
|
185
|
+
**Solution**
|
|
186
|
+
|
|
187
|
+
```python
|
|
188
|
+
# Solution
|
|
189
|
+
X = df.drop(columns=['cltv'])
|
|
190
|
+
y = df['cltv']
|
|
191
|
+
print(y.median())
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
**Recorded result**
|
|
195
|
+
|
|
196
|
+
```text
|
|
197
|
+
66288.0
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
---
|
|
201
|
+
|
|
202
|
+
### Split the dataset into training and test dataset using `train_test_split` into `75:25` ratio while keeping random_state = 42
|
|
203
|
+
|
|
204
|
+
**Task**
|
|
205
|
+
|
|
206
|
+
Split the dataset into training and test dataset using `train_test_split` into `75:25` ratio while keeping random_state = 42. What is the shape of the test set (X_test)?
|
|
207
|
+
|
|
208
|
+
**Step-by-step approach**
|
|
209
|
+
|
|
210
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
211
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
212
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
213
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
214
|
+
|
|
215
|
+
**Solution**
|
|
216
|
+
|
|
217
|
+
```python
|
|
218
|
+
# Solution
|
|
219
|
+
from sklearn.model_selection import train_test_split
|
|
220
|
+
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
|
|
221
|
+
print(X_test.shape)
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
**Recorded result**
|
|
225
|
+
|
|
226
|
+
```text
|
|
227
|
+
(1565, 11)
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
---
|
|
231
|
+
|
|
232
|
+
### Drop(remove) `id` column from train and test data because it is not useful in model training
|
|
233
|
+
|
|
234
|
+
**Task**
|
|
235
|
+
|
|
236
|
+
Drop(remove) `id` column from train and test data because it is not useful in model training. Now how many of the remaining feature columns in the training dataset are of **object (string)** type?
|
|
237
|
+
|
|
238
|
+
**Step-by-step approach**
|
|
239
|
+
|
|
240
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
241
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
242
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
243
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
244
|
+
|
|
245
|
+
**Solution**
|
|
246
|
+
|
|
247
|
+
```python
|
|
248
|
+
# Solution
|
|
249
|
+
X_train = X_train.drop(columns=['id'])
|
|
250
|
+
X_test = X_test.drop(columns=['id'])
|
|
251
|
+
obj_cols = X_train.select_dtypes(include='object').columns
|
|
252
|
+
print(list(obj_cols))
|
|
253
|
+
print(len(obj_cols))
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
**Recorded result**
|
|
257
|
+
|
|
258
|
+
```text
|
|
259
|
+
['gender', 'area', 'qualification', 'num_policies', 'policy', 'type_of_policy']
|
|
260
|
+
6
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
---
|
|
264
|
+
|
|
265
|
+
### Compute and write the **mean** of the `income` column of X_train while ignoring the missing values
|
|
266
|
+
|
|
267
|
+
**Task**
|
|
268
|
+
|
|
269
|
+
Compute and write the **mean** of the `income` column of X_train while ignoring the missing values. Replace all NaN values in the income column of X_train and X_test by the mean computed from the X_train (upto two decimal).
|
|
270
|
+
|
|
271
|
+
**Step-by-step approach**
|
|
272
|
+
|
|
273
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
274
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
275
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
276
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
277
|
+
|
|
278
|
+
**Solution**
|
|
279
|
+
|
|
280
|
+
```python
|
|
281
|
+
# Solution
|
|
282
|
+
income_mean = X_train['income'].mean()
|
|
283
|
+
print(round(income_mean, 2))
|
|
284
|
+
X_train['income'] = X_train['income'].fillna(income_mean)
|
|
285
|
+
X_test['income'] = X_test['income'].fillna(income_mean)
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
**Recorded result**
|
|
289
|
+
|
|
290
|
+
```text
|
|
291
|
+
13.73
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
---
|
|
295
|
+
|
|
296
|
+
### How many missing values are present in the `policy` column of **X_test**
|
|
297
|
+
|
|
298
|
+
**Task**
|
|
299
|
+
|
|
300
|
+
How many missing values are present in the `policy` column of **X_test**? After answering, replace all NaN values in the `policy` column of X_train and X_test by the most frequent value in X_train.
|
|
301
|
+
|
|
302
|
+
**Step-by-step approach**
|
|
303
|
+
|
|
304
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
305
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
306
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
307
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
308
|
+
|
|
309
|
+
**Solution**
|
|
310
|
+
|
|
311
|
+
```python
|
|
312
|
+
# Solution
|
|
313
|
+
print(X_test['policy'].isna().sum())
|
|
314
|
+
policy_mode = X_train['policy'].mode()[0]
|
|
315
|
+
print('Most frequent policy in X_train:', policy_mode)
|
|
316
|
+
X_train['policy'] = X_train['policy'].fillna(policy_mode)
|
|
317
|
+
X_test['policy'] = X_test['policy'].fillna(policy_mode)
|
|
318
|
+
```
|
|
319
|
+
|
|
320
|
+
**Recorded result**
|
|
321
|
+
|
|
322
|
+
```text
|
|
323
|
+
102
|
|
324
|
+
Most frequent policy in X_train: A
|
|
325
|
+
```
|
|
326
|
+
|
|
327
|
+
---
|
|
328
|
+
|
|
329
|
+
### Replace all NaN values in the `area` column of X_train and X_test by the most frequent value from X_train
|
|
330
|
+
|
|
331
|
+
**Task**
|
|
332
|
+
|
|
333
|
+
Replace all NaN values in the `area` column of X_train and X_test by the most frequent value from X_train. After replacing, how many records in **X_train** have `area` equal to that most frequent value?
|
|
334
|
+
|
|
335
|
+
**Step-by-step approach**
|
|
336
|
+
|
|
337
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
338
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
339
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
340
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
341
|
+
|
|
342
|
+
**Solution**
|
|
343
|
+
|
|
344
|
+
```python
|
|
345
|
+
# Solution
|
|
346
|
+
area_mode = X_train['area'].mode()[0]
|
|
347
|
+
print('Most frequent area in X_train:', area_mode)
|
|
348
|
+
X_train['area'] = X_train['area'].fillna(area_mode)
|
|
349
|
+
X_test['area'] = X_test['area'].fillna(area_mode)
|
|
350
|
+
print((X_train['area'] == area_mode).sum())
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
**Recorded result**
|
|
354
|
+
|
|
355
|
+
```text
|
|
356
|
+
Most frequent area in X_train: Urban
|
|
357
|
+
3407
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
---
|
|
361
|
+
|
|
362
|
+
### Replace all NaN values in the claim_amount column of X_train and X_test by the **median** of claim_amount computed from X_train
|
|
363
|
+
|
|
364
|
+
**Task**
|
|
365
|
+
|
|
366
|
+
Replace all NaN values in the claim_amount column of X_train and X_test by the **median** of claim_amount computed from X_train. After replacing, what is the **mean** of the claim_amount column in **X_test**? (correct upto two decimal places)
|
|
367
|
+
|
|
368
|
+
**Step-by-step approach**
|
|
369
|
+
|
|
370
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
371
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
372
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
373
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
374
|
+
|
|
375
|
+
**Solution**
|
|
376
|
+
|
|
377
|
+
```python
|
|
378
|
+
# Solution
|
|
379
|
+
claim_median = X_train['claim_amount'].median()
|
|
380
|
+
X_train['claim_amount'] = X_train['claim_amount'].fillna(claim_median)
|
|
381
|
+
X_test['claim_amount'] = X_test['claim_amount'].fillna(claim_median)
|
|
382
|
+
print(round(X_test['claim_amount'].mean(), 2))
|
|
383
|
+
```
|
|
384
|
+
|
|
385
|
+
**Recorded result**
|
|
386
|
+
|
|
387
|
+
```text
|
|
388
|
+
4260.88
|
|
389
|
+
```
|
|
390
|
+
|
|
391
|
+
---
|
|
392
|
+
|
|
393
|
+
### Apply `StandardScaler` on the `vintage` column of X_train (store the result in a separate variable, do **not** overwrite X_train)
|
|
394
|
+
|
|
395
|
+
**Task**
|
|
396
|
+
|
|
397
|
+
Apply `StandardScaler` on the `vintage` column of X_train (store the result in a separate variable, do **not** overwrite X_train). Compute and write the **maximum** value of the scaled `vintage` column. (correct upto 2 decimal)
|
|
398
|
+
|
|
399
|
+
**Step-by-step approach**
|
|
400
|
+
|
|
401
|
+
1. Separate numeric and categorical columns, and map each feature to its required operation.
|
|
402
|
+
2. Build a Pipeline for any feature needing several operations in sequence, such as imputation followed by scaling.
|
|
403
|
+
3. Pass transformer blocks to ColumnTransformer in the exact requested order; that order determines transformed column indices.
|
|
404
|
+
4. Fit the full transformer on X_train, then transform both train and test data and inspect shape, mean, minimum or maximum as requested.
|
|
405
|
+
|
|
406
|
+
**Solution**
|
|
407
|
+
|
|
408
|
+
```python
|
|
409
|
+
# Solution
|
|
410
|
+
from sklearn.preprocessing import StandardScaler
|
|
411
|
+
vintage_scaled = StandardScaler().fit_transform(X_train[['vintage']])
|
|
412
|
+
print(round(vintage_scaled.max(), 2))
|
|
413
|
+
```
|
|
414
|
+
|
|
415
|
+
**Recorded result**
|
|
416
|
+
|
|
417
|
+
```text
|
|
418
|
+
1.47
|
|
419
|
+
```
|
|
420
|
+
|
|
421
|
+
---
|
|
422
|
+
|
|
423
|
+
### How many columns are produced by the **One-Hot Encoding step alone** (i
|
|
424
|
+
|
|
425
|
+
**Task**
|
|
426
|
+
|
|
427
|
+
How many columns are produced by the **One-Hot Encoding step alone** (i.e. the number of columns in $X_1$)?
|
|
428
|
+
|
|
429
|
+
[Options]
|
|
430
|
+
|
|
431
|
+
**Step-by-step approach**
|
|
432
|
+
|
|
433
|
+
1. List categorical columns and inspect their unique values, including missing values.
|
|
434
|
+
2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
|
|
435
|
+
3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
|
|
436
|
+
4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
|
|
437
|
+
|
|
438
|
+
**Solution**
|
|
439
|
+
|
|
440
|
+
```python
|
|
441
|
+
# Solution
|
|
442
|
+
print(ct.named_transformers_['ohe'].get_feature_names_out().shape[0])
|
|
443
|
+
```
|
|
444
|
+
|
|
445
|
+
**Recorded result**
|
|
446
|
+
|
|
447
|
+
```text
|
|
448
|
+
17
|
|
449
|
+
```
|
|
450
|
+
|
|
451
|
+
---
|
|
452
|
+
|
|
453
|
+
### Apply `SequentialFeatureSelector` transformer with direction= 'forward' with `LinearRegression()` estimator and select **6**…
|
|
454
|
+
|
|
455
|
+
**Task**
|
|
456
|
+
|
|
457
|
+
Apply `SequentialFeatureSelector` transformer with direction= 'forward' with `LinearRegression()` estimator and select **6** features by fitting to the X_train and y_train.
|
|
458
|
+
|
|
459
|
+
`Use cv = KFold(n_splits=5,random_state=42,shuffle=True) in SequentialFeatureSelector.`
|
|
460
|
+
|
|
461
|
+
### Which of the following options represents the correct integer index of the selected features list?
|
|
462
|
+
|
|
463
|
+
**Step-by-step approach**
|
|
464
|
+
|
|
465
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
466
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
467
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
468
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
469
|
+
|
|
470
|
+
**Solution**
|
|
471
|
+
|
|
472
|
+
```python
|
|
473
|
+
# Solution
|
|
474
|
+
from sklearn.feature_selection import SequentialFeatureSelector
|
|
475
|
+
from sklearn.linear_model import LinearRegression
|
|
476
|
+
from sklearn.model_selection import KFold
|
|
477
|
+
cv = KFold(n_splits=5, random_state=42, shuffle=True)
|
|
478
|
+
sfs = SequentialFeatureSelector(LinearRegression(), n_features_to_select=6, direction='forward', cv=cv)
|
|
479
|
+
sfs.fit(X_train, y_train)
|
|
480
|
+
print(np.where(sfs.get_support())[0])
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
**Recorded result**
|
|
484
|
+
|
|
485
|
+
```text
|
|
486
|
+
[ 4 5 7 9 13 19]
|
|
487
|
+
```
|
|
488
|
+
|
|
489
|
+
---
|
|
490
|
+
|
|
491
|
+
### Apply `LinearRegression` on the training set(`X_train` and `y_train`)
|
|
492
|
+
|
|
493
|
+
**Task**
|
|
494
|
+
|
|
495
|
+
Apply `LinearRegression` on the training set(`X_train` and `y_train`). What is the **mean absolute error** on the test set(`X_test` and `y_test`)? (Upto 2 digits after decimal point)
|
|
496
|
+
|
|
497
|
+
**Step-by-step approach**
|
|
498
|
+
|
|
499
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
500
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
501
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
502
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
503
|
+
|
|
504
|
+
**Solution**
|
|
505
|
+
|
|
506
|
+
```python
|
|
507
|
+
# Solution
|
|
508
|
+
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
|
|
509
|
+
lr = LinearRegression().fit(X_train, y_train)
|
|
510
|
+
print(round(mean_absolute_error(y_test, lr.predict(X_test)), 2))
|
|
511
|
+
```
|
|
512
|
+
|
|
513
|
+
**Recorded result**
|
|
514
|
+
|
|
515
|
+
```text
|
|
516
|
+
52712.56
|
|
517
|
+
```
|
|
518
|
+
|
|
519
|
+
---
|
|
520
|
+
|
|
521
|
+
### Using the `LinearRegression` model, compute the `cross-validation scores` for `5 splits` on training data (X_train and y_train)…
|
|
522
|
+
|
|
523
|
+
**Task**
|
|
524
|
+
|
|
525
|
+
Using the `LinearRegression` model, compute the `cross-validation scores` for `5 splits` on training data (X_train and y_train) using `cross_val_score`. Enter the **mean** value of the $R^2$ scores obtained. (Upto 4 digits after decimal point)
|
|
526
|
+
|
|
527
|
+
`Use cv = KFold(n_splits=5,random_state=42,shuffle=True) in cross_val_score.`
|
|
528
|
+
|
|
529
|
+
(**Hint**: By default cross_val_score uses LinearRegression's scoring metric, which is $R^2$ score.)
|
|
530
|
+
|
|
531
|
+
**Step-by-step approach**
|
|
532
|
+
|
|
533
|
+
1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
|
|
534
|
+
2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
|
|
535
|
+
3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
|
|
536
|
+
4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
|
|
537
|
+
|
|
538
|
+
**Solution**
|
|
539
|
+
|
|
540
|
+
```python
|
|
541
|
+
# Solution
|
|
542
|
+
from sklearn.model_selection import cross_val_score
|
|
543
|
+
scores = cross_val_score(LinearRegression(), X_train, y_train, cv=KFold(n_splits=5, random_state=42, shuffle=True))
|
|
544
|
+
print(scores)
|
|
545
|
+
print(round(scores.mean(), 4))
|
|
546
|
+
```
|
|
547
|
+
|
|
548
|
+
**Recorded result**
|
|
549
|
+
|
|
550
|
+
```text
|
|
551
|
+
[0.15765982 0.14449678 0.14566491 0.13373985 0.15322638]
|
|
552
|
+
0.147
|
|
553
|
+
```
|
|
554
|
+
|
|
555
|
+
---
|
|
556
|
+
|
|
557
|
+
### Apply `Ridge` regression with **random_state=42** and **alpha=10** on training set(`X_train and y_train`) and calculate the $R^2$…
|
|
558
|
+
|
|
559
|
+
**Task**
|
|
560
|
+
|
|
561
|
+
Apply `Ridge` regression with **random_state=42** and **alpha=10** on training set(`X_train and y_train`) and calculate the $R^2$ score on test_set (`X_test and y_test`). What is the correct score (Upto 4 digits after decimal point)?
|
|
562
|
+
|
|
563
|
+
**Step-by-step approach**
|
|
564
|
+
|
|
565
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
566
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
567
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
568
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
569
|
+
|
|
570
|
+
**Solution**
|
|
571
|
+
|
|
572
|
+
```python
|
|
573
|
+
# Solution
|
|
574
|
+
from sklearn.linear_model import Ridge
|
|
575
|
+
ridge = Ridge(alpha=10, random_state=42).fit(X_train, y_train)
|
|
576
|
+
print(round(ridge.score(X_test, y_test), 4))
|
|
577
|
+
```
|
|
578
|
+
|
|
579
|
+
**Recorded result**
|
|
580
|
+
|
|
581
|
+
```text
|
|
582
|
+
0.1421
|
|
583
|
+
```
|
|
584
|
+
|
|
585
|
+
---
|
|
586
|
+
|
|
587
|
+
### Apply `Lasso` regression with **random_state=42** and **regularization rate (alpha)=100** on the training data(`X_train & y_train`)
|
|
588
|
+
|
|
589
|
+
**Task**
|
|
590
|
+
|
|
591
|
+
Apply `Lasso` regression with **random_state=42** and **regularization rate (alpha)=100** on the training data(`X_train & y_train`). Enter the value of the **intercept** correct upto 2 digits after decimal point.
|
|
592
|
+
|
|
593
|
+
**Step-by-step approach**
|
|
594
|
+
|
|
595
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
596
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
597
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
598
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
599
|
+
|
|
600
|
+
**Solution**
|
|
601
|
+
|
|
602
|
+
```python
|
|
603
|
+
# Solution
|
|
604
|
+
from sklearn.linear_model import Lasso
|
|
605
|
+
lasso = Lasso(alpha=100, random_state=42).fit(X_train, y_train)
|
|
606
|
+
print(round(lasso.intercept_, 2))
|
|
607
|
+
```
|
|
608
|
+
|
|
609
|
+
**Recorded result**
|
|
610
|
+
|
|
611
|
+
```text
|
|
612
|
+
116658.43
|
|
613
|
+
```
|
|
614
|
+
|
|
615
|
+
---
|
|
616
|
+
|
|
617
|
+
### Fit SGDRegressor(`random_state=42`) estimator on the training data(`X_train & y_train`) and predict labels for test_data(`X_test`),…
|
|
618
|
+
|
|
619
|
+
**Task**
|
|
620
|
+
|
|
621
|
+
Fit SGDRegressor(`random_state=42`) estimator on the training data(`X_train & y_train`) and predict labels for test_data(`X_test`), lets call it as y_test_predict. The parameters are initialized with default values. Calculate the **root mean squared error** between y_test and y_test_predict. (Correct upto two decimals)
|
|
622
|
+
|
|
623
|
+
**Step-by-step approach**
|
|
624
|
+
|
|
625
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
626
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
627
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
628
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
629
|
+
|
|
630
|
+
**Solution**
|
|
631
|
+
|
|
632
|
+
```python
|
|
633
|
+
# Solution
|
|
634
|
+
from sklearn.linear_model import SGDRegressor
|
|
635
|
+
sgd = SGDRegressor(random_state=42).fit(X_train, y_train)
|
|
636
|
+
y_test_predict = sgd.predict(X_test)
|
|
637
|
+
print(round(np.sqrt(mean_squared_error(y_test, y_test_predict)), 2))
|
|
638
|
+
```
|
|
639
|
+
|
|
640
|
+
**Recorded result**
|
|
641
|
+
|
|
642
|
+
```text
|
|
643
|
+
88459.51
|
|
644
|
+
```
|
|
645
|
+
|
|
646
|
+
---
|
|
647
|
+
|
|
648
|
+
### Using SGDRegressor(random_state=42) as an estimator for exactly **25** iterations (`max_iter=25`)
|
|
649
|
+
|
|
650
|
+
**Task**
|
|
651
|
+
|
|
652
|
+
Using SGDRegressor(random_state=42) as an estimator for exactly **25** iterations (`max_iter=25`). Write the correct $R^2$ score on test data (correct Upto 4 digits)
|
|
653
|
+
|
|
654
|
+
**Step-by-step approach**
|
|
655
|
+
|
|
656
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
657
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
658
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
659
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
660
|
+
|
|
661
|
+
**Solution**
|
|
662
|
+
|
|
663
|
+
```python
|
|
664
|
+
# Solution
|
|
665
|
+
sgd25 = SGDRegressor(random_state=42, max_iter=25).fit(X_train, y_train)
|
|
666
|
+
print(round(sgd25.score(X_test, y_test), 4))
|
|
667
|
+
```
|
|
668
|
+
|
|
669
|
+
**Recorded result**
|
|
670
|
+
|
|
671
|
+
```text
|
|
672
|
+
0.1397
|
|
673
|
+
```
|
|
674
|
+
|
|
675
|
+
---
|
|
676
|
+
|
|
677
|
+
### Mark the best `alpha` value you got using above instructions
|
|
678
|
+
|
|
679
|
+
**Task**
|
|
680
|
+
|
|
681
|
+
Mark the best `alpha` value you got using above instructions.
|
|
682
|
+
|
|
683
|
+
**Step-by-step approach**
|
|
684
|
+
|
|
685
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
686
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
687
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
688
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
689
|
+
|
|
690
|
+
**Solution**
|
|
691
|
+
|
|
692
|
+
```python
|
|
693
|
+
# Solution
|
|
694
|
+
print(grid.best_params_['lasso__alpha'])
|
|
695
|
+
```
|
|
696
|
+
|
|
697
|
+
**Recorded result**
|
|
698
|
+
|
|
699
|
+
```text
|
|
700
|
+
10.0
|
|
701
|
+
```
|
|
702
|
+
|
|
703
|
+
---
|
|
704
|
+
|
|
705
|
+
### Enter the best polynomial degree value you got using above instructions
|
|
706
|
+
|
|
707
|
+
**Task**
|
|
708
|
+
|
|
709
|
+
Enter the best polynomial degree value you got using above instructions.
|
|
710
|
+
|
|
711
|
+
**Step-by-step approach**
|
|
712
|
+
|
|
713
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
714
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
715
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
716
|
+
|
|
717
|
+
**Solution**
|
|
718
|
+
|
|
719
|
+
```python
|
|
720
|
+
# Solution
|
|
721
|
+
print(grid.best_params_['poly__degree'])
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
**Recorded result**
|
|
725
|
+
|
|
726
|
+
```text
|
|
727
|
+
2
|
|
728
|
+
```
|
|
729
|
+
|
|
730
|
+
---
|
|
731
|
+
|
|
732
|
+
### What is the sum of `explained_variance_ratio_`
|
|
733
|
+
|
|
734
|
+
**Task**
|
|
735
|
+
|
|
736
|
+
What is the sum of `explained_variance_ratio_` ? (upto 4 decimal)
|
|
737
|
+
|
|
738
|
+
**Step-by-step approach**
|
|
739
|
+
|
|
740
|
+
1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
|
|
741
|
+
2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
|
|
742
|
+
3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
|
|
743
|
+
4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
|
|
744
|
+
|
|
745
|
+
**Solution**
|
|
746
|
+
|
|
747
|
+
```python
|
|
748
|
+
# Solution
|
|
749
|
+
print(round(pca.explained_variance_ratio_.sum(), 4))
|
|
750
|
+
```
|
|
751
|
+
|
|
752
|
+
**Recorded result**
|
|
753
|
+
|
|
754
|
+
```text
|
|
755
|
+
0.7816
|
|
756
|
+
```
|
|
757
|
+
|
|
758
|
+
---
|
|
759
|
+
|
|
760
|
+
### Use PCA transformed training data from earlier question and y_train to fit the `RidgeCV` estimator model having `alpha values as [0
|
|
761
|
+
|
|
762
|
+
**Task**
|
|
763
|
+
|
|
764
|
+
Use PCA transformed training data from earlier question and y_train to fit the `RidgeCV` estimator model having `alpha values as [0.01, 0.1, 1, 10, 100]`. Calculate the $R^2$ score you got from the model for transformed test data(PCA transformed X_test). (upto 4 decimal)
|
|
765
|
+
|
|
766
|
+
**Step-by-step approach**
|
|
767
|
+
|
|
768
|
+
1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
|
|
769
|
+
2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
|
|
770
|
+
3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
|
|
771
|
+
4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
|
|
772
|
+
|
|
773
|
+
**Solution**
|
|
774
|
+
|
|
775
|
+
```python
|
|
776
|
+
# Solution
|
|
777
|
+
from sklearn.linear_model import RidgeCV
|
|
778
|
+
rcv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100]).fit(X_train_pca, y_train)
|
|
779
|
+
print(round(rcv.score(X_test_pca, y_test), 4))
|
|
780
|
+
```
|
|
781
|
+
|
|
782
|
+
**Recorded result**
|
|
783
|
+
|
|
784
|
+
```text
|
|
785
|
+
0.1308
|
|
786
|
+
```
|
|
787
|
+
|
|
788
|
+
---
|
|
789
|
+
|
|
790
|
+
## Residential price records
|
|
791
|
+
|
|
792
|
+
### How many columns (features + target) are there in the dataset
|
|
793
|
+
|
|
794
|
+
**Task**
|
|
795
|
+
|
|
796
|
+
How many columns (features + target) are there in the dataset?
|
|
797
|
+
|
|
798
|
+
**Step-by-step approach**
|
|
799
|
+
|
|
800
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
801
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
802
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
803
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
804
|
+
|
|
805
|
+
**Solution**
|
|
806
|
+
|
|
807
|
+
```python
|
|
808
|
+
# Solution
|
|
809
|
+
print(df.shape[1])
|
|
810
|
+
```
|
|
811
|
+
|
|
812
|
+
**Recorded result**
|
|
813
|
+
|
|
814
|
+
```text
|
|
815
|
+
14
|
|
816
|
+
```
|
|
817
|
+
|
|
818
|
+
---
|
|
819
|
+
|
|
820
|
+
### What is the **median** house price (in lacs)
|
|
821
|
+
|
|
822
|
+
**Task**
|
|
823
|
+
|
|
824
|
+
What is the **median** house price (in lacs)?
|
|
825
|
+
|
|
826
|
+
**Step-by-step approach**
|
|
827
|
+
|
|
828
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
829
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
830
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
831
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
832
|
+
|
|
833
|
+
**Solution**
|
|
834
|
+
|
|
835
|
+
```python
|
|
836
|
+
# Solution
|
|
837
|
+
print(df['PRICE'].median())
|
|
838
|
+
```
|
|
839
|
+
|
|
840
|
+
**Recorded result**
|
|
841
|
+
|
|
842
|
+
```text
|
|
843
|
+
22.634566447766908
|
|
844
|
+
```
|
|
845
|
+
|
|
846
|
+
---
|
|
847
|
+
|
|
848
|
+
### How many houses have **9 or more** rooms
|
|
849
|
+
|
|
850
|
+
**Task**
|
|
851
|
+
|
|
852
|
+
How many houses have **9 or more** rooms?
|
|
853
|
+
|
|
854
|
+
**Step-by-step approach**
|
|
855
|
+
|
|
856
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
857
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
858
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
859
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
860
|
+
|
|
861
|
+
**Solution**
|
|
862
|
+
|
|
863
|
+
```python
|
|
864
|
+
# Solution
|
|
865
|
+
print((df['RM'] >= 9).sum())
|
|
866
|
+
```
|
|
867
|
+
|
|
868
|
+
**Recorded result**
|
|
869
|
+
|
|
870
|
+
```text
|
|
871
|
+
318
|
|
872
|
+
```
|
|
873
|
+
|
|
874
|
+
---
|
|
875
|
+
|
|
876
|
+
### What is the average price of the **10 least expensive** houses (in lacs)
|
|
877
|
+
|
|
878
|
+
**Task**
|
|
879
|
+
|
|
880
|
+
What is the average price of the **10 least expensive** houses (in lacs)?
|
|
881
|
+
|
|
882
|
+
**Step-by-step approach**
|
|
883
|
+
|
|
884
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
885
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
886
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
887
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
888
|
+
|
|
889
|
+
**Solution**
|
|
890
|
+
|
|
891
|
+
```python
|
|
892
|
+
# Solution
|
|
893
|
+
print(df['PRICE'].nsmallest(10).mean())
|
|
894
|
+
```
|
|
895
|
+
|
|
896
|
+
**Recorded result**
|
|
897
|
+
|
|
898
|
+
```text
|
|
899
|
+
7.278891822010996
|
|
900
|
+
```
|
|
901
|
+
|
|
902
|
+
---
|
|
903
|
+
|
|
904
|
+
### What is the **total** number of missing or unknown values in the `RM`, `AGE` and `RIVERSIDE` features **combined**
|
|
905
|
+
|
|
906
|
+
**Task**
|
|
907
|
+
|
|
908
|
+
What is the **total** number of missing or unknown values in the `RM`, `AGE` and `RIVERSIDE` features **combined**?
|
|
909
|
+
|
|
910
|
+
**Step-by-step approach**
|
|
911
|
+
|
|
912
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
913
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
914
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
915
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
916
|
+
|
|
917
|
+
**Solution**
|
|
918
|
+
|
|
919
|
+
```python
|
|
920
|
+
# Solution
|
|
921
|
+
print(rm_bad.sum() + age_bad.sum() + riv_bad.sum())
|
|
922
|
+
```
|
|
923
|
+
|
|
924
|
+
**Recorded result**
|
|
925
|
+
|
|
926
|
+
```text
|
|
927
|
+
178
|
|
928
|
+
```
|
|
929
|
+
|
|
930
|
+
---
|
|
931
|
+
|
|
932
|
+
### What is the average age (in years) of the houses, **ignoring** the missing or unknown values of the `AGE` feature
|
|
933
|
+
|
|
934
|
+
**Task**
|
|
935
|
+
|
|
936
|
+
What is the average age (in years) of the houses, **ignoring** the missing or unknown values of the `AGE` feature?
|
|
937
|
+
|
|
938
|
+
**Step-by-step approach**
|
|
939
|
+
|
|
940
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
941
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
942
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
943
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
944
|
+
|
|
945
|
+
**Solution**
|
|
946
|
+
|
|
947
|
+
```python
|
|
948
|
+
# Solution
|
|
949
|
+
print(df.loc[~age_bad, 'AGE'].mean())
|
|
950
|
+
```
|
|
951
|
+
|
|
952
|
+
**Recorded result**
|
|
953
|
+
|
|
954
|
+
```text
|
|
955
|
+
66.86582278481012
|
|
956
|
+
```
|
|
957
|
+
|
|
958
|
+
---
|
|
959
|
+
|
|
960
|
+
### What **percentage** of the houses have an unknown value in the `RIVERSIDE` feature
|
|
961
|
+
|
|
962
|
+
**Task**
|
|
963
|
+
|
|
964
|
+
What **percentage** of the houses have an unknown value in the `RIVERSIDE` feature? (Enter the answer correct upto 2 decimal places)
|
|
965
|
+
|
|
966
|
+
**Step-by-step approach**
|
|
967
|
+
|
|
968
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
969
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
970
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
971
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
972
|
+
|
|
973
|
+
**Solution**
|
|
974
|
+
|
|
975
|
+
```python
|
|
976
|
+
# Solution
|
|
977
|
+
print(round(riv_bad.mean() * 100, 2))
|
|
978
|
+
```
|
|
979
|
+
|
|
980
|
+
**Recorded result**
|
|
981
|
+
|
|
982
|
+
```text
|
|
983
|
+
2.2
|
|
984
|
+
```
|
|
985
|
+
|
|
986
|
+
---
|
|
987
|
+
|
|
988
|
+
### How many houses are **not** on the riverside and are **older than 60 years** (i
|
|
989
|
+
|
|
990
|
+
**Task**
|
|
991
|
+
|
|
992
|
+
How many houses are **not** on the riverside and are **older than 60 years** (i.e. AGE > 60)? For this question, ignore the rows that have missing values in either riverside feature or age feature.
|
|
993
|
+
|
|
994
|
+
**Step-by-step approach**
|
|
995
|
+
|
|
996
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
997
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
998
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
999
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
1000
|
+
|
|
1001
|
+
**Solution**
|
|
1002
|
+
|
|
1003
|
+
```python
|
|
1004
|
+
# Solution
|
|
1005
|
+
valid = ~riv_bad & ~age_bad
|
|
1006
|
+
print(((df['RIVERSIDE'] == 'NO') & (df['AGE'] > 60) & valid).sum())
|
|
1007
|
+
```
|
|
1008
|
+
|
|
1009
|
+
**Recorded result**
|
|
1010
|
+
|
|
1011
|
+
```text
|
|
1012
|
+
2088
|
|
1013
|
+
```
|
|
1014
|
+
|
|
1015
|
+
---
|
|
1016
|
+
|
|
1017
|
+
### How many houses are near to exactly 3, 4 or 5 highways (all three inclusive)
|
|
1018
|
+
|
|
1019
|
+
**Task**
|
|
1020
|
+
|
|
1021
|
+
How many houses are near to exactly 3, 4 or 5 highways (all three inclusive)?
|
|
1022
|
+
|
|
1023
|
+
**Step-by-step approach**
|
|
1024
|
+
|
|
1025
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
1026
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
1027
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
1028
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
1029
|
+
|
|
1030
|
+
**Solution**
|
|
1031
|
+
|
|
1032
|
+
```python
|
|
1033
|
+
# Solution
|
|
1034
|
+
print(df['HIGHWAYCOUNT'].between(3, 5).sum())
|
|
1035
|
+
```
|
|
1036
|
+
|
|
1037
|
+
**Recorded result**
|
|
1038
|
+
|
|
1039
|
+
```text
|
|
1040
|
+
1742
|
|
1041
|
+
```
|
|
1042
|
+
|
|
1043
|
+
---
|
|
1044
|
+
|
|
1045
|
+
### Create a column 'CATEGORY' and divide the houses in categories as following:
|
|
1046
|
+
|
|
1047
|
+
**Task**
|
|
1048
|
+
|
|
1049
|
+
Create a column 'CATEGORY' and divide the houses in categories as following:
|
|
1050
|
+
|
|
1051
|
+
- Category 1: house price <15 lacs
|
|
1052
|
+
- Category 2: 15 lacs <= house price <25 lacs
|
|
1053
|
+
- Category 3: 25 lacs <= house price <35 lacs
|
|
1054
|
+
- Category 4: house price >=35 lacs
|
|
1055
|
+
|
|
1056
|
+
Which category has the **lowest** number of records?
|
|
1057
|
+
|
|
1058
|
+
**Step-by-step approach**
|
|
1059
|
+
|
|
1060
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
1061
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
1062
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
1063
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
1064
|
+
|
|
1065
|
+
**Solution**
|
|
1066
|
+
|
|
1067
|
+
```python
|
|
1068
|
+
# Solution
|
|
1069
|
+
df['CATEGORY'] = pd.cut(df['PRICE'], bins=[-np.inf, 15, 25, 35, np.inf], right=False, labels=[1, 2, 3, 4])
|
|
1070
|
+
print(df['CATEGORY'].value_counts().sort_index())
|
|
1071
|
+
```
|
|
1072
|
+
|
|
1073
|
+
**Recorded result**
|
|
1074
|
+
|
|
1075
|
+
```text
|
|
1076
|
+
CATEGORY
|
|
1077
|
+
1 364
|
|
1078
|
+
2 2318
|
|
1079
|
+
3 889
|
|
1080
|
+
4 429
|
|
1081
|
+
Name: count, dtype: int64
|
|
1082
|
+
```
|
|
1083
|
+
|
|
1084
|
+
---
|
|
1085
|
+
|
|
1086
|
+
### What is the number of samples in the **test** set
|
|
1087
|
+
|
|
1088
|
+
**Task**
|
|
1089
|
+
|
|
1090
|
+
What is the number of samples in the **test** set?
|
|
1091
|
+
|
|
1092
|
+
**Step-by-step approach**
|
|
1093
|
+
|
|
1094
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
1095
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
1096
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
1097
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
1098
|
+
|
|
1099
|
+
**Solution**
|
|
1100
|
+
|
|
1101
|
+
```python
|
|
1102
|
+
# Solution
|
|
1103
|
+
from sklearn.model_selection import train_test_split
|
|
1104
|
+
df.loc[rm_bad, 'RM'] = np.nan
|
|
1105
|
+
df.loc[age_bad, 'AGE'] = np.nan
|
|
1106
|
+
df.loc[riv_bad, 'RIVERSIDE'] = np.nan
|
|
1107
|
+
X = df.drop(columns=['PRICE'])
|
|
1108
|
+
y = df['PRICE']
|
|
1109
|
+
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
|
|
1110
|
+
print(X_test.shape[0])
|
|
1111
|
+
```
|
|
1112
|
+
|
|
1113
|
+
**Recorded result**
|
|
1114
|
+
|
|
1115
|
+
```text
|
|
1116
|
+
1000
|
|
1117
|
+
```
|
|
1118
|
+
|
|
1119
|
+
---
|
|
1120
|
+
|
|
1121
|
+
### How many features are there after performing above transformation
|
|
1122
|
+
|
|
1123
|
+
**Task**
|
|
1124
|
+
|
|
1125
|
+
How many features are there after performing above transformation?
|
|
1126
|
+
|
|
1127
|
+
**Step-by-step approach**
|
|
1128
|
+
|
|
1129
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
1130
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
1131
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
1132
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
1133
|
+
|
|
1134
|
+
**Solution**
|
|
1135
|
+
|
|
1136
|
+
```python
|
|
1137
|
+
# Solution
|
|
1138
|
+
print(X_train_t.shape[1])
|
|
1139
|
+
```
|
|
1140
|
+
|
|
1141
|
+
**Recorded result**
|
|
1142
|
+
|
|
1143
|
+
```text
|
|
1144
|
+
13
|
|
1145
|
+
```
|
|
1146
|
+
|
|
1147
|
+
---
|
|
1148
|
+
|
|
1149
|
+
### What is the mean of the transformed **training** data (features only)
|
|
1150
|
+
|
|
1151
|
+
**Task**
|
|
1152
|
+
|
|
1153
|
+
What is the mean of the transformed **training** data (features only)?
|
|
1154
|
+
|
|
1155
|
+
Note : Compute the mean of the whole feature matrix i.e. mean of all values in the transformed training feature matrix
|
|
1156
|
+
|
|
1157
|
+
**Step-by-step approach**
|
|
1158
|
+
|
|
1159
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
1160
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
1161
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
1162
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
1163
|
+
|
|
1164
|
+
**Solution**
|
|
1165
|
+
|
|
1166
|
+
```python
|
|
1167
|
+
# Solution
|
|
1168
|
+
print(X_train_t.mean())
|
|
1169
|
+
```
|
|
1170
|
+
|
|
1171
|
+
**Recorded result**
|
|
1172
|
+
|
|
1173
|
+
```text
|
|
1174
|
+
0.28842989630040944
|
|
1175
|
+
```
|
|
1176
|
+
|
|
1177
|
+
---
|
|
1178
|
+
|
|
1179
|
+
### Enter the value of $R^2$ score on the test dataset
|
|
1180
|
+
|
|
1181
|
+
**Task**
|
|
1182
|
+
|
|
1183
|
+
Enter the value of $R^2$ score on the test dataset.
|
|
1184
|
+
|
|
1185
|
+
**Step-by-step approach**
|
|
1186
|
+
|
|
1187
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1188
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1189
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1190
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1191
|
+
|
|
1192
|
+
**Solution**
|
|
1193
|
+
|
|
1194
|
+
```python
|
|
1195
|
+
# Solution
|
|
1196
|
+
print(ridge.score(X_test, y_test))
|
|
1197
|
+
```
|
|
1198
|
+
|
|
1199
|
+
**Recorded result**
|
|
1200
|
+
|
|
1201
|
+
```text
|
|
1202
|
+
0.6639511913668861
|
|
1203
|
+
```
|
|
1204
|
+
|
|
1205
|
+
---
|
|
1206
|
+
|
|
1207
|
+
### What is the index of most important feature (highest absolute coefficient)
|
|
1208
|
+
|
|
1209
|
+
**Task**
|
|
1210
|
+
|
|
1211
|
+
What is the index of most important feature (highest absolute coefficient)? Note the index starts from 0. Ignore the intercept for this question.
|
|
1212
|
+
|
|
1213
|
+
**Step-by-step approach**
|
|
1214
|
+
|
|
1215
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
1216
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
1217
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
1218
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
1219
|
+
|
|
1220
|
+
**Solution**
|
|
1221
|
+
|
|
1222
|
+
```python
|
|
1223
|
+
# Solution
|
|
1224
|
+
print(np.abs(ridge.coef_))
|
|
1225
|
+
print(np.argmax(np.abs(ridge.coef_)))
|
|
1226
|
+
```
|
|
1227
|
+
|
|
1228
|
+
**Recorded result**
|
|
1229
|
+
|
|
1230
|
+
```text
|
|
1231
|
+
[ 0.80599934 4.36935144 0.54731649 0.57136401 11.03253838 6.96010246
|
|
1232
|
+
6.21337559 9.67022437 4.75155118 25.56498977 11.69237356 0.82743376
|
|
1233
|
+
0.83498997 0.83498997]
|
|
1234
|
+
9
|
|
1235
|
+
```
|
|
1236
|
+
|
|
1237
|
+
---
|
|
1238
|
+
|
|
1239
|
+
### What is the index of least important feature (lowest absolute coefficient)
|
|
1240
|
+
|
|
1241
|
+
**Task**
|
|
1242
|
+
|
|
1243
|
+
What is the index of least important feature (lowest absolute coefficient)? Note the index starts from 0. Ignore the intercept for this question.
|
|
1244
|
+
|
|
1245
|
+
**Step-by-step approach**
|
|
1246
|
+
|
|
1247
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
1248
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
1249
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
1250
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
1251
|
+
|
|
1252
|
+
**Solution**
|
|
1253
|
+
|
|
1254
|
+
```python
|
|
1255
|
+
# Solution
|
|
1256
|
+
print(np.argmin(np.abs(ridge.coef_)))
|
|
1257
|
+
```
|
|
1258
|
+
|
|
1259
|
+
**Recorded result**
|
|
1260
|
+
|
|
1261
|
+
```text
|
|
1262
|
+
2
|
|
1263
|
+
```
|
|
1264
|
+
|
|
1265
|
+
---
|
|
1266
|
+
|
|
1267
|
+
### What is the best **alpha**
|
|
1268
|
+
|
|
1269
|
+
**Task**
|
|
1270
|
+
|
|
1271
|
+
What is the best **alpha**?
|
|
1272
|
+
|
|
1273
|
+
> A 0.0001\
|
|
1274
|
+
> B 0.001\
|
|
1275
|
+
> C 0.01\
|
|
1276
|
+
> D 0.1\
|
|
1277
|
+
> E 1
|
|
1278
|
+
|
|
1279
|
+
**Step-by-step approach**
|
|
1280
|
+
|
|
1281
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
1282
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
1283
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
1284
|
+
|
|
1285
|
+
**Solution**
|
|
1286
|
+
|
|
1287
|
+
```python
|
|
1288
|
+
# Solution
|
|
1289
|
+
print(sgd_grid.best_params_['alpha'])
|
|
1290
|
+
```
|
|
1291
|
+
|
|
1292
|
+
**Recorded result**
|
|
1293
|
+
|
|
1294
|
+
```text
|
|
1295
|
+
0.0001
|
|
1296
|
+
```
|
|
1297
|
+
|
|
1298
|
+
---
|
|
1299
|
+
|
|
1300
|
+
### What will be value of **root mean squared error** on the test dataset
|
|
1301
|
+
|
|
1302
|
+
**Task**
|
|
1303
|
+
|
|
1304
|
+
What will be value of **root mean squared error** on the test dataset?
|
|
1305
|
+
|
|
1306
|
+
**Step-by-step approach**
|
|
1307
|
+
|
|
1308
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1309
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1310
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1311
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1312
|
+
|
|
1313
|
+
**Solution**
|
|
1314
|
+
|
|
1315
|
+
```python
|
|
1316
|
+
# Solution
|
|
1317
|
+
print(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))))
|
|
1318
|
+
```
|
|
1319
|
+
|
|
1320
|
+
**Recorded result**
|
|
1321
|
+
|
|
1322
|
+
```text
|
|
1323
|
+
5.126302501903342
|
|
1324
|
+
```
|
|
1325
|
+
|
|
1326
|
+
---
|
|
1327
|
+
|
|
1328
|
+
### If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset
|
|
1329
|
+
|
|
1330
|
+
**Task**
|
|
1331
|
+
|
|
1332
|
+
If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset?
|
|
1333
|
+
|
|
1334
|
+
**Step-by-step approach**
|
|
1335
|
+
|
|
1336
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1337
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1338
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1339
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1340
|
+
|
|
1341
|
+
**Solution**
|
|
1342
|
+
|
|
1343
|
+
```python
|
|
1344
|
+
# Solution
|
|
1345
|
+
print(mean_absolute_error(y_test, pca_grid.predict(X_test)))
|
|
1346
|
+
```
|
|
1347
|
+
|
|
1348
|
+
**Recorded result**
|
|
1349
|
+
|
|
1350
|
+
```text
|
|
1351
|
+
3.865036507995
|
|
1352
|
+
```
|
|
1353
|
+
|
|
1354
|
+
---
|
|
1355
|
+
|
|
1356
|
+
### What **percentage** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model
|
|
1357
|
+
|
|
1358
|
+
**Task**
|
|
1359
|
+
|
|
1360
|
+
What **percentage** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model?
|
|
1361
|
+
|
|
1362
|
+
**Step-by-step approach**
|
|
1363
|
+
|
|
1364
|
+
1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
|
|
1365
|
+
2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
|
|
1366
|
+
3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
|
|
1367
|
+
4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
|
|
1368
|
+
|
|
1369
|
+
**Solution**
|
|
1370
|
+
|
|
1371
|
+
```python
|
|
1372
|
+
# Solution
|
|
1373
|
+
print(pca_grid.best_estimator_.named_steps['pca'].explained_variance_ratio_[0] * 100)
|
|
1374
|
+
```
|
|
1375
|
+
|
|
1376
|
+
**Recorded result**
|
|
1377
|
+
|
|
1378
|
+
```text
|
|
1379
|
+
69.9021494220279
|
|
1380
|
+
```
|
|
1381
|
+
|
|
1382
|
+
---
|
|
1383
|
+
|
|
1384
|
+
### Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
|
|
1385
|
+
|
|
1386
|
+
**Task**
|
|
1387
|
+
|
|
1388
|
+
Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
|
|
1389
|
+
- For PolynomialFeatures:
|
|
1390
|
+
- interaction_only = False
|
|
1391
|
+
- degree = 2
|
|
1392
|
+
- For Lasso:
|
|
1393
|
+
- alpha = 0.5
|
|
1394
|
+
- warm_start = True
|
|
1395
|
+
- random state as 0
|
|
1396
|
+
|
|
1397
|
+
Fit the pipeline on the training dataset and find the **mean absolute error** on the test dataset.
|
|
1398
|
+
|
|
1399
|
+
**Step-by-step approach**
|
|
1400
|
+
|
|
1401
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1402
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1403
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1404
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1405
|
+
|
|
1406
|
+
**Solution**
|
|
1407
|
+
|
|
1408
|
+
```python
|
|
1409
|
+
# Solution
|
|
1410
|
+
from sklearn.preprocessing import PolynomialFeatures
|
|
1411
|
+
poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=2, interaction_only=False)),
|
|
1412
|
+
('lasso', Lasso(alpha=0.5, warm_start=True, random_state=0))])
|
|
1413
|
+
poly_pipe.fit(X_train, y_train)
|
|
1414
|
+
print(mean_absolute_error(y_test, poly_pipe.predict(X_test)))
|
|
1415
|
+
```
|
|
1416
|
+
|
|
1417
|
+
**Recorded result**
|
|
1418
|
+
|
|
1419
|
+
```text
|
|
1420
|
+
4.55907406372981
|
|
1421
|
+
```
|
|
1422
|
+
|
|
1423
|
+
---
|
|
1424
|
+
|
|
1425
|
+
### If you eliminate **2** features with recursive feature elimination, which features will be eliminated
|
|
1426
|
+
|
|
1427
|
+
**Task**
|
|
1428
|
+
|
|
1429
|
+
If you eliminate **2** features with recursive feature elimination, which features will be eliminated?
|
|
1430
|
+
|
|
1431
|
+
Type the indices of the eliminated features (index starts from 0).
|
|
1432
|
+
|
|
1433
|
+
Use LinearRegression model with default parameters as an estimator.
|
|
1434
|
+
|
|
1435
|
+
Use processed training data.
|
|
1436
|
+
|
|
1437
|
+
**Step-by-step approach**
|
|
1438
|
+
|
|
1439
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
1440
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
1441
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
1442
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
1443
|
+
|
|
1444
|
+
**Solution**
|
|
1445
|
+
|
|
1446
|
+
```python
|
|
1447
|
+
# Solution
|
|
1448
|
+
from sklearn.feature_selection import RFE
|
|
1449
|
+
from sklearn.linear_model import LinearRegression
|
|
1450
|
+
rfe = RFE(LinearRegression(), n_features_to_select=X_train.shape[1] - 2)
|
|
1451
|
+
rfe.fit(X_train, y_train)
|
|
1452
|
+
print(np.where(~rfe.support_)[0])
|
|
1453
|
+
print(rfe.ranking_)
|
|
1454
|
+
```
|
|
1455
|
+
|
|
1456
|
+
**Recorded result**
|
|
1457
|
+
|
|
1458
|
+
```text
|
|
1459
|
+
[2 3]
|
|
1460
|
+
[1 1 3 2 1 1 1 1 1 1 1 1 1 1]
|
|
1461
|
+
```
|
|
1462
|
+
|
|
1463
|
+
---
|
|
1464
|
+
|
|
1465
|
+
## Property listing records
|
|
1466
|
+
|
|
1467
|
+
### What are the unique values present in the `Residential` feature of the dataset
|
|
1468
|
+
|
|
1469
|
+
**Task**
|
|
1470
|
+
|
|
1471
|
+
What are the unique values present in the `Residential` feature of the dataset?
|
|
1472
|
+
- [Single Family, Two Family, Three Family, Four Family]
|
|
1473
|
+
- [Detached House, Duplex, Triplex, Fourplex]
|
|
1474
|
+
- [Detached House, Duplex, Triplex]
|
|
1475
|
+
- [Bridgeport, Fairfield, Greenwich, Norwalk]
|
|
1476
|
+
|
|
1477
|
+
**Step-by-step approach**
|
|
1478
|
+
|
|
1479
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
1480
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
1481
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
1482
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
1483
|
+
|
|
1484
|
+
**Solution**
|
|
1485
|
+
|
|
1486
|
+
```python
|
|
1487
|
+
# Solution
|
|
1488
|
+
print(df['Residential'].unique())
|
|
1489
|
+
```
|
|
1490
|
+
|
|
1491
|
+
**Recorded result**
|
|
1492
|
+
|
|
1493
|
+
```text
|
|
1494
|
+
['Detached House' 'Duplex' 'Triplex' 'Fourplex']
|
|
1495
|
+
```
|
|
1496
|
+
|
|
1497
|
+
---
|
|
1498
|
+
|
|
1499
|
+
### Which of the following columns have categorical data
|
|
1500
|
+
|
|
1501
|
+
**Task**
|
|
1502
|
+
|
|
1503
|
+
Which of the following columns have categorical data?
|
|
1504
|
+
- Year
|
|
1505
|
+
- Residential
|
|
1506
|
+
- carpet_area
|
|
1507
|
+
- Face
|
|
1508
|
+
- property_tax_rate
|
|
1509
|
+
|
|
1510
|
+
**Step-by-step approach**
|
|
1511
|
+
|
|
1512
|
+
1. List categorical columns and inspect their unique values, including missing values.
|
|
1513
|
+
2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
|
|
1514
|
+
3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
|
|
1515
|
+
4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
|
|
1516
|
+
|
|
1517
|
+
**Solution**
|
|
1518
|
+
|
|
1519
|
+
```python
|
|
1520
|
+
# Solution
|
|
1521
|
+
print(df.dtypes)
|
|
1522
|
+
```
|
|
1523
|
+
|
|
1524
|
+
**Recorded result**
|
|
1525
|
+
|
|
1526
|
+
```text
|
|
1527
|
+
Date object
|
|
1528
|
+
Year int64
|
|
1529
|
+
Locality object
|
|
1530
|
+
Estimated Value float64
|
|
1531
|
+
Sale Price float64
|
|
1532
|
+
Property object
|
|
1533
|
+
Residential object
|
|
1534
|
+
num_rooms int64
|
|
1535
|
+
num_bathrooms int64
|
|
1536
|
+
carpet_area float64
|
|
1537
|
+
property_tax_rate float64
|
|
1538
|
+
Face object
|
|
1539
|
+
dtype: object
|
|
1540
|
+
```
|
|
1541
|
+
|
|
1542
|
+
---
|
|
1543
|
+
|
|
1544
|
+
### Which of the following features have `missing`(NaN) or unknown ("
|
|
1545
|
+
|
|
1546
|
+
**Task**
|
|
1547
|
+
|
|
1548
|
+
Which of the following features have `missing`(NaN) or unknown ("?") values present in the dataset?
|
|
1549
|
+
- Date
|
|
1550
|
+
- Locality
|
|
1551
|
+
- carpet_area
|
|
1552
|
+
- Property
|
|
1553
|
+
- num_rooms
|
|
1554
|
+
- Face
|
|
1555
|
+
|
|
1556
|
+
**Step-by-step approach**
|
|
1557
|
+
|
|
1558
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
1559
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
1560
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
1561
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
1562
|
+
|
|
1563
|
+
**Solution**
|
|
1564
|
+
|
|
1565
|
+
```python
|
|
1566
|
+
# Solution
|
|
1567
|
+
print(df.isna().sum())
|
|
1568
|
+
print((df == '?').sum())
|
|
1569
|
+
```
|
|
1570
|
+
|
|
1571
|
+
**Recorded result**
|
|
1572
|
+
|
|
1573
|
+
```text
|
|
1574
|
+
Date 0
|
|
1575
|
+
Year 0
|
|
1576
|
+
Locality 1285
|
|
1577
|
+
Estimated Value 1281
|
|
1578
|
+
Sale Price 0
|
|
1579
|
+
Property 0
|
|
1580
|
+
Residential 0
|
|
1581
|
+
num_rooms 0
|
|
1582
|
+
num_bathrooms 0
|
|
1583
|
+
carpet_area 1282
|
|
1584
|
+
property_tax_rate 0
|
|
1585
|
+
Face 0
|
|
1586
|
+
dtype: int64
|
|
1587
|
+
Date 0
|
|
1588
|
+
Year 0
|
|
1589
|
+
Locality 0
|
|
1590
|
+
Estimated Value 0
|
|
1591
|
+
Sale Price 0
|
|
1592
|
+
Property 1873
|
|
1593
|
+
Residential 0
|
|
1594
|
+
num_rooms 0
|
|
1595
|
+
num_bathrooms 0
|
|
1596
|
+
carpet_area 0
|
|
1597
|
+
property_tax_rate 0
|
|
1598
|
+
Face 0
|
|
1599
|
+
dtype: int64
|
|
1600
|
+
```
|
|
1601
|
+
|
|
1602
|
+
---
|
|
1603
|
+
|
|
1604
|
+
### In the Year **2021**, how many houses (rows) located in the **Stamford** Locality have **2 or more** num_bathrooms, and are facing…
|
|
1605
|
+
|
|
1606
|
+
**Task**
|
|
1607
|
+
|
|
1608
|
+
In the Year **2021**, how many houses (rows) located in the **Stamford** Locality have **2 or more** num_bathrooms, and are facing towards either the **South or West**?
|
|
1609
|
+
|
|
1610
|
+
**Step-by-step approach**
|
|
1611
|
+
|
|
1612
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
1613
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
1614
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
1615
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
1616
|
+
|
|
1617
|
+
**Solution**
|
|
1618
|
+
|
|
1619
|
+
```python
|
|
1620
|
+
# Solution
|
|
1621
|
+
mask = (df['Year'] == 2021) & (df['Locality'] == 'Stamford') & (df['num_bathrooms'] >= 2) & (df['Face'].isin(['South', 'West']))
|
|
1622
|
+
print(mask.sum())
|
|
1623
|
+
```
|
|
1624
|
+
|
|
1625
|
+
**Recorded result**
|
|
1626
|
+
|
|
1627
|
+
```text
|
|
1628
|
+
44
|
|
1629
|
+
```
|
|
1630
|
+
|
|
1631
|
+
---
|
|
1632
|
+
|
|
1633
|
+
### Split the dataset into train dataset and test dataset in the following manner
|
|
1634
|
+
|
|
1635
|
+
**Task**
|
|
1636
|
+
|
|
1637
|
+
Split the dataset into train dataset and test dataset in the following manner
|
|
1638
|
+
1. data(rows) collected before the year of **2020** [2009-2019] should be the train dataset
|
|
1639
|
+
2. data(rows) collected in the year of **2020, 2021 and 2022** (all inclusive) should be the test dataset
|
|
1640
|
+
3. columns except of the label vector (`Sale Price`) should be the feature matrix (X_train or X_test)
|
|
1641
|
+
4. make label vector (Y_train or y_test) containing values only from the target feature.
|
|
1642
|
+
|
|
1643
|
+
How many rows are in the feature matrix of the **train** dataset?
|
|
1644
|
+
|
|
1645
|
+
**Step-by-step approach**
|
|
1646
|
+
|
|
1647
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1648
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1649
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1650
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1651
|
+
|
|
1652
|
+
**Solution**
|
|
1653
|
+
|
|
1654
|
+
```python
|
|
1655
|
+
# Solution
|
|
1656
|
+
train = df[df['Year'] <= 2019]
|
|
1657
|
+
test = df[df['Year'] >= 2020]
|
|
1658
|
+
X_train, y_train = train.drop(columns=['Sale Price']).copy(), train['Sale Price']
|
|
1659
|
+
X_test, y_test = test.drop(columns=['Sale Price']).copy(), test['Sale Price']
|
|
1660
|
+
print(X_train.shape, X_test.shape)
|
|
1661
|
+
```
|
|
1662
|
+
|
|
1663
|
+
**Recorded result**
|
|
1664
|
+
|
|
1665
|
+
```text
|
|
1666
|
+
(7560, 11) (2440, 11)
|
|
1667
|
+
```
|
|
1668
|
+
|
|
1669
|
+
---
|
|
1670
|
+
|
|
1671
|
+
### Write the **MEDIAN** value of the `carpet_area` column of the train dataset you found to replace all the Missing Values (NaN)
|
|
1672
|
+
|
|
1673
|
+
**Task**
|
|
1674
|
+
|
|
1675
|
+
Write the **MEDIAN** value of the `carpet_area` column of the train dataset you found to replace all the Missing Values (NaN). (upto 2 digits after decimal points)
|
|
1676
|
+
|
|
1677
|
+
**Step-by-step approach**
|
|
1678
|
+
|
|
1679
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
1680
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
1681
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
1682
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
1683
|
+
|
|
1684
|
+
**Solution**
|
|
1685
|
+
|
|
1686
|
+
```python
|
|
1687
|
+
# Solution
|
|
1688
|
+
print(round(carpet_median, 2))
|
|
1689
|
+
```
|
|
1690
|
+
|
|
1691
|
+
**Recorded result**
|
|
1692
|
+
|
|
1693
|
+
```text
|
|
1694
|
+
1021.0
|
|
1695
|
+
```
|
|
1696
|
+
|
|
1697
|
+
---
|
|
1698
|
+
|
|
1699
|
+
### What value did you use to replace the missing values (NaN) in the `Locality` column of the train dataset
|
|
1700
|
+
|
|
1701
|
+
**Task**
|
|
1702
|
+
|
|
1703
|
+
What value did you use to replace the missing values (NaN) in the `Locality` column of the train dataset?
|
|
1704
|
+
- Bridgeport
|
|
1705
|
+
- Waterbury
|
|
1706
|
+
- Fairfield
|
|
1707
|
+
- Stamford
|
|
1708
|
+
|
|
1709
|
+
**Step-by-step approach**
|
|
1710
|
+
|
|
1711
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
1712
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
1713
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
1714
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
1715
|
+
|
|
1716
|
+
**Solution**
|
|
1717
|
+
|
|
1718
|
+
```python
|
|
1719
|
+
# Solution
|
|
1720
|
+
print(loc_mode)
|
|
1721
|
+
```
|
|
1722
|
+
|
|
1723
|
+
**Recorded result**
|
|
1724
|
+
|
|
1725
|
+
```text
|
|
1726
|
+
Bridgeport
|
|
1727
|
+
```
|
|
1728
|
+
|
|
1729
|
+
---
|
|
1730
|
+
|
|
1731
|
+
### How many total features you got after one-hot encoding and scaling the training dataset
|
|
1732
|
+
|
|
1733
|
+
**Task**
|
|
1734
|
+
|
|
1735
|
+
How many total features you got after one-hot encoding and scaling the training dataset?
|
|
1736
|
+
- 10
|
|
1737
|
+
- 18
|
|
1738
|
+
- 25
|
|
1739
|
+
- 28
|
|
1740
|
+
|
|
1741
|
+
**Step-by-step approach**
|
|
1742
|
+
|
|
1743
|
+
1. List categorical columns and inspect their unique values, including missing values.
|
|
1744
|
+
2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
|
|
1745
|
+
3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
|
|
1746
|
+
4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
|
|
1747
|
+
|
|
1748
|
+
**Solution**
|
|
1749
|
+
|
|
1750
|
+
```python
|
|
1751
|
+
# Solution
|
|
1752
|
+
print(X_train_t.shape[1])
|
|
1753
|
+
```
|
|
1754
|
+
|
|
1755
|
+
**Recorded result**
|
|
1756
|
+
|
|
1757
|
+
```text
|
|
1758
|
+
25
|
|
1759
|
+
```
|
|
1760
|
+
|
|
1761
|
+
---
|
|
1762
|
+
|
|
1763
|
+
### What is the **maximum** value present in the transformed **test** feature matrix
|
|
1764
|
+
|
|
1765
|
+
**Task**
|
|
1766
|
+
|
|
1767
|
+
What is the **maximum** value present in the transformed **test** feature matrix? (upto 2 digits after decimal points)
|
|
1768
|
+
|
|
1769
|
+
**Step-by-step approach**
|
|
1770
|
+
|
|
1771
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
1772
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
1773
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
1774
|
+
|
|
1775
|
+
**Solution**
|
|
1776
|
+
|
|
1777
|
+
```python
|
|
1778
|
+
# Solution
|
|
1779
|
+
print(round(X_test_t.max(), 2))
|
|
1780
|
+
```
|
|
1781
|
+
|
|
1782
|
+
**Recorded result**
|
|
1783
|
+
|
|
1784
|
+
```text
|
|
1785
|
+
13.16
|
|
1786
|
+
```
|
|
1787
|
+
|
|
1788
|
+
---
|
|
1789
|
+
|
|
1790
|
+
### Apply LinearRegression on the train dataset(X_train and y_train)
|
|
1791
|
+
|
|
1792
|
+
**Task**
|
|
1793
|
+
|
|
1794
|
+
Apply LinearRegression on the train dataset(X_train and y_train). What is the **mean absolute error** on the test dataset(X_test and y_test)? (Upto 2 digits after decimal points)
|
|
1795
|
+
|
|
1796
|
+
**Step-by-step approach**
|
|
1797
|
+
|
|
1798
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1799
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1800
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1801
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1802
|
+
|
|
1803
|
+
**Solution**
|
|
1804
|
+
|
|
1805
|
+
```python
|
|
1806
|
+
# Solution
|
|
1807
|
+
from sklearn.linear_model import LinearRegression
|
|
1808
|
+
from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
|
|
1809
|
+
lr = LinearRegression().fit(X_train, y_train)
|
|
1810
|
+
print(round(mean_absolute_error(y_test, lr.predict(X_test)), 2))
|
|
1811
|
+
```
|
|
1812
|
+
|
|
1813
|
+
**Recorded result**
|
|
1814
|
+
|
|
1815
|
+
```text
|
|
1816
|
+
179131.9
|
|
1817
|
+
```
|
|
1818
|
+
|
|
1819
|
+
---
|
|
1820
|
+
|
|
1821
|
+
### Enter the **minimum** value of $R^2$ score you got using the LinearRegression model when computed with 5 folds from the training…
|
|
1822
|
+
|
|
1823
|
+
**Task**
|
|
1824
|
+
|
|
1825
|
+
Enter the **minimum** value of $R^2$ score you got using the LinearRegression model when computed with 5 folds from the training dataset (X_train and y_train) using cross_val_score. (Upto 4 digits after decimal points)
|
|
1826
|
+
|
|
1827
|
+
**Step-by-step approach**
|
|
1828
|
+
|
|
1829
|
+
1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
|
|
1830
|
+
2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
|
|
1831
|
+
3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
|
|
1832
|
+
4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
|
|
1833
|
+
|
|
1834
|
+
**Solution**
|
|
1835
|
+
|
|
1836
|
+
```python
|
|
1837
|
+
# Solution
|
|
1838
|
+
from sklearn.model_selection import cross_val_score
|
|
1839
|
+
scores = cross_val_score(LinearRegression(), X_train, y_train, cv=5)
|
|
1840
|
+
print(scores)
|
|
1841
|
+
print(round(scores.min(), 4))
|
|
1842
|
+
```
|
|
1843
|
+
|
|
1844
|
+
**Recorded result**
|
|
1845
|
+
|
|
1846
|
+
```text
|
|
1847
|
+
[0.61661573 0.79290189 0.72820221 0.82626052 0.64972083]
|
|
1848
|
+
0.6166
|
|
1849
|
+
```
|
|
1850
|
+
|
|
1851
|
+
---
|
|
1852
|
+
|
|
1853
|
+
### Train Ridge and Lasso with random_state=27 and keep other parameters as default using train dataset
|
|
1854
|
+
|
|
1855
|
+
**Task**
|
|
1856
|
+
|
|
1857
|
+
Train Ridge and Lasso with random_state=27 and keep other parameters as default using train dataset. Which one has the least **"mean absolute error"** for the test dataset?
|
|
1858
|
+
|
|
1859
|
+
- Ridge
|
|
1860
|
+
- Lasso
|
|
1861
|
+
- Both are Equal
|
|
1862
|
+
|
|
1863
|
+
**Step-by-step approach**
|
|
1864
|
+
|
|
1865
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1866
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1867
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1868
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1869
|
+
|
|
1870
|
+
**Solution**
|
|
1871
|
+
|
|
1872
|
+
```python
|
|
1873
|
+
# Solution
|
|
1874
|
+
from sklearn.linear_model import Ridge, Lasso
|
|
1875
|
+
ridge = Ridge(random_state=27).fit(X_train, y_train)
|
|
1876
|
+
lasso = Lasso(random_state=27).fit(X_train, y_train)
|
|
1877
|
+
print('Ridge MAE:', mean_absolute_error(y_test, ridge.predict(X_test)))
|
|
1878
|
+
print('Lasso MAE:', mean_absolute_error(y_test, lasso.predict(X_test)))
|
|
1879
|
+
```
|
|
1880
|
+
|
|
1881
|
+
**Recorded result**
|
|
1882
|
+
|
|
1883
|
+
```text
|
|
1884
|
+
Ridge MAE: 179994.97462090437
|
|
1885
|
+
Lasso MAE: 179123.65380001903
|
|
1886
|
+
```
|
|
1887
|
+
|
|
1888
|
+
---
|
|
1889
|
+
|
|
1890
|
+
### Train the `SGDRegressor(random_state=27, warm_start=True)` with maximum passes over the train dataset as **50**
|
|
1891
|
+
|
|
1892
|
+
**Task**
|
|
1893
|
+
|
|
1894
|
+
Train the `SGDRegressor(random_state=27, warm_start=True)` with maximum passes over the train dataset as **50**. Write the correct $R^2$ score for this estimator on test dataset (correct Upto 4 digits)
|
|
1895
|
+
|
|
1896
|
+
**Step-by-step approach**
|
|
1897
|
+
|
|
1898
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
1899
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
1900
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
1901
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
1902
|
+
|
|
1903
|
+
**Solution**
|
|
1904
|
+
|
|
1905
|
+
```python
|
|
1906
|
+
# Solution
|
|
1907
|
+
from sklearn.linear_model import SGDRegressor
|
|
1908
|
+
sgd = SGDRegressor(random_state=27, warm_start=True, max_iter=50).fit(X_train, y_train)
|
|
1909
|
+
print(round(sgd.score(X_test, y_test), 4))
|
|
1910
|
+
```
|
|
1911
|
+
|
|
1912
|
+
**Recorded result**
|
|
1913
|
+
|
|
1914
|
+
```text
|
|
1915
|
+
0.4676
|
|
1916
|
+
```
|
|
1917
|
+
|
|
1918
|
+
---
|
|
1919
|
+
|
|
1920
|
+
### Mark the best alpha value you got using above instructions
|
|
1921
|
+
|
|
1922
|
+
**Task**
|
|
1923
|
+
|
|
1924
|
+
Mark the best alpha value you got using above instructions.
|
|
1925
|
+
- 1
|
|
1926
|
+
- 10
|
|
1927
|
+
- 100
|
|
1928
|
+
- 1000
|
|
1929
|
+
|
|
1930
|
+
**Step-by-step approach**
|
|
1931
|
+
|
|
1932
|
+
1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
|
|
1933
|
+
2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
|
|
1934
|
+
3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
|
|
1935
|
+
4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
|
|
1936
|
+
|
|
1937
|
+
**Solution**
|
|
1938
|
+
|
|
1939
|
+
```python
|
|
1940
|
+
# Solution
|
|
1941
|
+
print(grid.best_params_['lasso__alpha'])
|
|
1942
|
+
```
|
|
1943
|
+
|
|
1944
|
+
**Recorded result**
|
|
1945
|
+
|
|
1946
|
+
```text
|
|
1947
|
+
1000
|
|
1948
|
+
```
|
|
1949
|
+
|
|
1950
|
+
---
|
|
1951
|
+
|
|
1952
|
+
### Enter the best polynomial degree value you got using above instructions
|
|
1953
|
+
|
|
1954
|
+
**Task**
|
|
1955
|
+
|
|
1956
|
+
Enter the best polynomial degree value you got using above instructions.
|
|
1957
|
+
|
|
1958
|
+
**Step-by-step approach**
|
|
1959
|
+
|
|
1960
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
1961
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
1962
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
1963
|
+
|
|
1964
|
+
**Solution**
|
|
1965
|
+
|
|
1966
|
+
```python
|
|
1967
|
+
# Solution
|
|
1968
|
+
print(grid.best_params_['poly__degree'])
|
|
1969
|
+
```
|
|
1970
|
+
|
|
1971
|
+
**Recorded result**
|
|
1972
|
+
|
|
1973
|
+
```text
|
|
1974
|
+
1
|
|
1975
|
+
```
|
|
1976
|
+
|
|
1977
|
+
---
|
|
1978
|
+
|
|
1979
|
+
### What is the sum of explained_variance_ratio_
|
|
1980
|
+
|
|
1981
|
+
**Task**
|
|
1982
|
+
|
|
1983
|
+
What is the sum of explained_variance_ratio_ ? (upto two digits after the decimal)
|
|
1984
|
+
|
|
1985
|
+
**Step-by-step approach**
|
|
1986
|
+
|
|
1987
|
+
1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
|
|
1988
|
+
2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
|
|
1989
|
+
3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
|
|
1990
|
+
4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
|
|
1991
|
+
|
|
1992
|
+
**Solution**
|
|
1993
|
+
|
|
1994
|
+
```python
|
|
1995
|
+
# Solution
|
|
1996
|
+
print(round(pca.explained_variance_ratio_.sum(), 2))
|
|
1997
|
+
```
|
|
1998
|
+
|
|
1999
|
+
**Recorded result**
|
|
2000
|
+
|
|
2001
|
+
```text
|
|
2002
|
+
0.86
|
|
2003
|
+
```
|
|
2004
|
+
|
|
2005
|
+
---
|
|
2006
|
+
|
|
2007
|
+
### Use PCA transformed training feature matrix from earlier question and training labels to fit the RidgeCV estimator with cv=5 and…
|
|
2008
|
+
|
|
2009
|
+
**Task**
|
|
2010
|
+
|
|
2011
|
+
Use PCA transformed training feature matrix from earlier question and training labels to fit the RidgeCV estimator with cv=5 and having alpha values as **[0.01, 0.1, 1, 10, 100, 1000]**. Calculate the $R^2$ score you got from the model for PCA transformed test dataset. (upto 4 digits after the decimal)
|
|
2012
|
+
|
|
2013
|
+
**Step-by-step approach**
|
|
2014
|
+
|
|
2015
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2016
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2017
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2018
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2019
|
+
|
|
2020
|
+
**Solution**
|
|
2021
|
+
|
|
2022
|
+
```python
|
|
2023
|
+
# Solution
|
|
2024
|
+
from sklearn.linear_model import RidgeCV
|
|
2025
|
+
rcv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100, 1000], cv=5).fit(X_train_pca, y_train)
|
|
2026
|
+
print(round(rcv.score(X_test_pca, y_test), 4))
|
|
2027
|
+
```
|
|
2028
|
+
|
|
2029
|
+
**Recorded result**
|
|
2030
|
+
|
|
2031
|
+
```text
|
|
2032
|
+
0.1789
|
|
2033
|
+
```
|
|
2034
|
+
|
|
2035
|
+
---
|
|
2036
|
+
|
|
2037
|
+
### How many features of the train dataset have a P-VALUE **less than 0
|
|
2038
|
+
|
|
2039
|
+
**Task**
|
|
2040
|
+
|
|
2041
|
+
How many features of the train dataset have a P-VALUE **less than 0.05**?
|
|
2042
|
+
|
|
2043
|
+
**Step-by-step approach**
|
|
2044
|
+
|
|
2045
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
2046
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
2047
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
2048
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
2049
|
+
|
|
2050
|
+
**Solution**
|
|
2051
|
+
|
|
2052
|
+
```python
|
|
2053
|
+
# Solution
|
|
2054
|
+
print(np.round(kb.pvalues_, 4))
|
|
2055
|
+
print((kb.pvalues_ < 0.05).sum())
|
|
2056
|
+
```
|
|
2057
|
+
|
|
2058
|
+
**Recorded result**
|
|
2059
|
+
|
|
2060
|
+
```text
|
|
2061
|
+
[0.000e+00 5.000e-04 0.000e+00 5.288e-01 9.124e-01 0.000e+00 0.000e+00
|
|
2062
|
+
9.500e-03 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 4.000e-03
|
|
2063
|
+
0.000e+00 6.493e-01 4.734e-01 5.820e-01 4.196e-01 1.600e-03 0.000e+00
|
|
2064
|
+
0.000e+00 0.000e+00 0.000e+00 2.442e-01]
|
|
2065
|
+
18
|
|
2066
|
+
```
|
|
2067
|
+
|
|
2068
|
+
---
|
|
2069
|
+
|
|
2070
|
+
### What is the **minimum** F_STATISTICS score you got for any feature of the train dataset
|
|
2071
|
+
|
|
2072
|
+
**Task**
|
|
2073
|
+
|
|
2074
|
+
What is the **minimum** F_STATISTICS score you got for any feature of the train dataset? (upto 2 digits after the decimal)
|
|
2075
|
+
|
|
2076
|
+
**Step-by-step approach**
|
|
2077
|
+
|
|
2078
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
2079
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
2080
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
2081
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
2082
|
+
|
|
2083
|
+
**Solution**
|
|
2084
|
+
|
|
2085
|
+
```python
|
|
2086
|
+
# Solution
|
|
2087
|
+
print(round(np.nanmin(kb.scores_), 2))
|
|
2088
|
+
```
|
|
2089
|
+
|
|
2090
|
+
**Recorded result**
|
|
2091
|
+
|
|
2092
|
+
```text
|
|
2093
|
+
0.01
|
|
2094
|
+
```
|
|
2095
|
+
|
|
2096
|
+
---
|
|
2097
|
+
|
|
2098
|
+
### What is the **mean absolute error** you got for this model using the test dataset
|
|
2099
|
+
|
|
2100
|
+
**Task**
|
|
2101
|
+
|
|
2102
|
+
What is the **mean absolute error** you got for this model using the test dataset? (upto 2 digits after the decimal)
|
|
2103
|
+
|
|
2104
|
+
**Step-by-step approach**
|
|
2105
|
+
|
|
2106
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2107
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2108
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2109
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2110
|
+
|
|
2111
|
+
**Solution**
|
|
2112
|
+
|
|
2113
|
+
```python
|
|
2114
|
+
# Solution
|
|
2115
|
+
print(round(mean_absolute_error(y_test, kbest_pipe.predict(X_test)), 2))
|
|
2116
|
+
```
|
|
2117
|
+
|
|
2118
|
+
**Recorded result**
|
|
2119
|
+
|
|
2120
|
+
```text
|
|
2121
|
+
170445.36
|
|
2122
|
+
```
|
|
2123
|
+
|
|
2124
|
+
---
|
|
2125
|
+
|
|
2126
|
+
## Flight fare records
|
|
2127
|
+
|
|
2128
|
+
### : What is the **median** of the flight ticket price
|
|
2129
|
+
|
|
2130
|
+
**Task**
|
|
2131
|
+
|
|
2132
|
+
: What is the **median** of the flight ticket price? Write your answer correct to two decimal places.
|
|
2133
|
+
|
|
2134
|
+
**Step-by-step approach**
|
|
2135
|
+
|
|
2136
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2137
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2138
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2139
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2140
|
+
|
|
2141
|
+
**Solution**
|
|
2142
|
+
|
|
2143
|
+
```python
|
|
2144
|
+
# Solution
|
|
2145
|
+
print(round(df['Price'].median(), 2))
|
|
2146
|
+
```
|
|
2147
|
+
|
|
2148
|
+
**Recorded result**
|
|
2149
|
+
|
|
2150
|
+
```text
|
|
2151
|
+
8366.0
|
|
2152
|
+
```
|
|
2153
|
+
|
|
2154
|
+
---
|
|
2155
|
+
|
|
2156
|
+
### : During which month did the **lowest** number of flights occur
|
|
2157
|
+
|
|
2158
|
+
**Task**
|
|
2159
|
+
|
|
2160
|
+
: During which month did the **lowest** number of flights occur? Months are represented by numerical codes, with January corresponding to 1, February to 2, and so forth.
|
|
2161
|
+
|
|
2162
|
+
**Step-by-step approach**
|
|
2163
|
+
|
|
2164
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2165
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2166
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2167
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2168
|
+
|
|
2169
|
+
**Solution**
|
|
2170
|
+
|
|
2171
|
+
```python
|
|
2172
|
+
# Solution
|
|
2173
|
+
print(df['Month'].value_counts())
|
|
2174
|
+
```
|
|
2175
|
+
|
|
2176
|
+
**Recorded result**
|
|
2177
|
+
|
|
2178
|
+
```text
|
|
2179
|
+
Month
|
|
2180
|
+
5 3092
|
|
2181
|
+
6 3044
|
|
2182
|
+
3 2388
|
|
2183
|
+
4 926
|
|
2184
|
+
Name: count, dtype: int64
|
|
2185
|
+
```
|
|
2186
|
+
|
|
2187
|
+
---
|
|
2188
|
+
|
|
2189
|
+
### : Which `Source` city has the **highest average** flight ticket price
|
|
2190
|
+
|
|
2191
|
+
**Task**
|
|
2192
|
+
|
|
2193
|
+
: Which `Source` city has the **highest average** flight ticket price?
|
|
2194
|
+
|
|
2195
|
+
**Step-by-step approach**
|
|
2196
|
+
|
|
2197
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2198
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2199
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2200
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2201
|
+
|
|
2202
|
+
**Solution**
|
|
2203
|
+
|
|
2204
|
+
```python
|
|
2205
|
+
# Solution
|
|
2206
|
+
print(df.groupby('Source')['Price'].mean().sort_values(ascending=False))
|
|
2207
|
+
```
|
|
2208
|
+
|
|
2209
|
+
**Recorded result**
|
|
2210
|
+
|
|
2211
|
+
```text
|
|
2212
|
+
Source
|
|
2213
|
+
Delhi 10637.617736
|
|
2214
|
+
Kolkata 9087.168003
|
|
2215
|
+
Banglore 7729.939275
|
|
2216
|
+
Mumbai 5148.631841
|
|
2217
|
+
Chennai 4695.082153
|
|
2218
|
+
Name: Price, dtype: float64
|
|
2219
|
+
```
|
|
2220
|
+
|
|
2221
|
+
---
|
|
2222
|
+
|
|
2223
|
+
### : Two of the entries in the `'Additional_Info'` column are `'No info'` and `'No Info'`
|
|
2224
|
+
|
|
2225
|
+
**Task**
|
|
2226
|
+
|
|
2227
|
+
: Two of the entries in the `'Additional_Info'` column are `'No info'` and `'No Info'`. Replace all occurrences of `'No Info'` with `'No info'`. How many flights fall under airline **`'Jet Airways'`** and have `'No info'` as additional information?
|
|
2228
|
+
|
|
2229
|
+
#### **Note:** Please ensure that you make the changes directly within the dataset and continue to use that modified dataset for subsequent questions.
|
|
2230
|
+
|
|
2231
|
+
**Step-by-step approach**
|
|
2232
|
+
|
|
2233
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2234
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2235
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2236
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2237
|
+
|
|
2238
|
+
**Solution**
|
|
2239
|
+
|
|
2240
|
+
```python
|
|
2241
|
+
# Solution
|
|
2242
|
+
df['Additional_Info'] = df['Additional_Info'].replace('No Info', 'No info')
|
|
2243
|
+
print(((df['Airline'] == 'Jet Airways') & (df['Additional_Info'] == 'No info')).sum())
|
|
2244
|
+
```
|
|
2245
|
+
|
|
2246
|
+
**Recorded result**
|
|
2247
|
+
|
|
2248
|
+
```text
|
|
2249
|
+
2119
|
|
2250
|
+
```
|
|
2251
|
+
|
|
2252
|
+
---
|
|
2253
|
+
|
|
2254
|
+
### : Convert the values of `'Duration'` into **minutes**
|
|
2255
|
+
|
|
2256
|
+
**Task**
|
|
2257
|
+
|
|
2258
|
+
: Convert the values of `'Duration'` into **minutes**. Enter the **median** duration (in minutes) of a flight.
|
|
2259
|
+
|
|
2260
|
+
#### **Note:** Please ensure that you make the changes directly within the dataset and continue to use that modified dataset for subsequent questions.
|
|
2261
|
+
|
|
2262
|
+
**Step-by-step approach**
|
|
2263
|
+
|
|
2264
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2265
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2266
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2267
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2268
|
+
|
|
2269
|
+
**Solution**
|
|
2270
|
+
|
|
2271
|
+
```python
|
|
2272
|
+
# Solution
|
|
2273
|
+
def to_minutes(s):
|
|
2274
|
+
h = m = 0
|
|
2275
|
+
for part in s.split():
|
|
2276
|
+
if part.endswith('h'):
|
|
2277
|
+
h = int(part[:-1])
|
|
2278
|
+
elif part.endswith('m'):
|
|
2279
|
+
m = int(part[:-1])
|
|
2280
|
+
return h * 60 + m
|
|
2281
|
+
df['Duration'] = df['Duration'].apply(to_minutes)
|
|
2282
|
+
print(df['Duration'].median())
|
|
2283
|
+
```
|
|
2284
|
+
|
|
2285
|
+
**Recorded result**
|
|
2286
|
+
|
|
2287
|
+
```text
|
|
2288
|
+
520.0
|
|
2289
|
+
```
|
|
2290
|
+
|
|
2291
|
+
---
|
|
2292
|
+
|
|
2293
|
+
### : How many flights started in the **Evening** and arrived at the destination at **Night**
|
|
2294
|
+
|
|
2295
|
+
**Task**
|
|
2296
|
+
|
|
2297
|
+
: How many flights started in the **Evening** and arrived at the destination at **Night**?
|
|
2298
|
+
|
|
2299
|
+
**Step-by-step approach**
|
|
2300
|
+
|
|
2301
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2302
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2303
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2304
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2305
|
+
|
|
2306
|
+
**Solution**
|
|
2307
|
+
|
|
2308
|
+
```python
|
|
2309
|
+
# Solution
|
|
2310
|
+
print(((df['Dep_Time'] == 'Evening') & (df['Arrival_Time'] == 'Night')).sum())
|
|
2311
|
+
```
|
|
2312
|
+
|
|
2313
|
+
**Recorded result**
|
|
2314
|
+
|
|
2315
|
+
```text
|
|
2316
|
+
732
|
|
2317
|
+
```
|
|
2318
|
+
|
|
2319
|
+
---
|
|
2320
|
+
|
|
2321
|
+
### : Encode the values of column `'WeekDay'` as follows:
|
|
2322
|
+
|
|
2323
|
+
**Task**
|
|
2324
|
+
|
|
2325
|
+
: Encode the values of column `'WeekDay'` as follows:
|
|
2326
|
+
# Weekends (Sunday, Saturday) = 1
|
|
2327
|
+
# all remaining five days = 0
|
|
2328
|
+
|
|
2329
|
+
# How many flights have WeekDay encoded as **1** (Weekends)?
|
|
2330
|
+
|
|
2331
|
+
**Step-by-step approach**
|
|
2332
|
+
|
|
2333
|
+
1. Normalize the raw text/category values first, including differences in capitalization and spacing.
|
|
2334
|
+
2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
|
|
2335
|
+
3. Create day-part or weekend features from the original time/date columns before encoding.
|
|
2336
|
+
4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
|
|
2337
|
+
|
|
2338
|
+
**Solution**
|
|
2339
|
+
|
|
2340
|
+
```python
|
|
2341
|
+
# Solution
|
|
2342
|
+
df['WeekDay'] = df['WeekDay'].isin(['Saturday', 'Sunday']).astype(int)
|
|
2343
|
+
print(df['WeekDay'].sum())
|
|
2344
|
+
```
|
|
2345
|
+
|
|
2346
|
+
**Recorded result**
|
|
2347
|
+
|
|
2348
|
+
```text
|
|
2349
|
+
2799
|
|
2350
|
+
```
|
|
2351
|
+
|
|
2352
|
+
---
|
|
2353
|
+
|
|
2354
|
+
### : Split the dataset into train dataset and test dataset in the following manner
|
|
2355
|
+
|
|
2356
|
+
**Task**
|
|
2357
|
+
|
|
2358
|
+
: Split the dataset into train dataset and test dataset in the following manner
|
|
2359
|
+
|
|
2360
|
+
- Use `train_test_split` to split the dataset into train and test dataset with test size equal to $0.25(25\%)$ and random_state equal to 42.
|
|
2361
|
+
- columns except of the label vector (`Price`) should be the feature matrix (X_train or X_test)
|
|
2362
|
+
- make label vector (Y_train or y_test) containing values only from the target feature (`Price`).
|
|
2363
|
+
|
|
2364
|
+
# How many rows are in the feature matrix of the **train** dataset?
|
|
2365
|
+
|
|
2366
|
+
**Step-by-step approach**
|
|
2367
|
+
|
|
2368
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2369
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2370
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2371
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2372
|
+
|
|
2373
|
+
**Solution**
|
|
2374
|
+
|
|
2375
|
+
```python
|
|
2376
|
+
# Solution
|
|
2377
|
+
from sklearn.model_selection import train_test_split
|
|
2378
|
+
X = df.drop(columns=['Price'])
|
|
2379
|
+
y = df['Price']
|
|
2380
|
+
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
|
|
2381
|
+
print(X_train.shape, X_test.shape)
|
|
2382
|
+
```
|
|
2383
|
+
|
|
2384
|
+
**Recorded result**
|
|
2385
|
+
|
|
2386
|
+
```text
|
|
2387
|
+
(7087, 12) (2363, 12)
|
|
2388
|
+
```
|
|
2389
|
+
|
|
2390
|
+
---
|
|
2391
|
+
|
|
2392
|
+
### : Were any unseen values encountered in the test dataset during the transformation process
|
|
2393
|
+
|
|
2394
|
+
**Task**
|
|
2395
|
+
|
|
2396
|
+
: Were any unseen values encountered in the test dataset during the transformation process?
|
|
2397
|
+
|
|
2398
|
+
**Step-by-step approach**
|
|
2399
|
+
|
|
2400
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2401
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2402
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2403
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2404
|
+
|
|
2405
|
+
**Solution**
|
|
2406
|
+
|
|
2407
|
+
```python
|
|
2408
|
+
# Solution
|
|
2409
|
+
print((X_test_t == -1).sum())
|
|
2410
|
+
```
|
|
2411
|
+
|
|
2412
|
+
**Recorded result**
|
|
2413
|
+
|
|
2414
|
+
```text
|
|
2415
|
+
1
|
|
2416
|
+
```
|
|
2417
|
+
|
|
2418
|
+
---
|
|
2419
|
+
|
|
2420
|
+
### : After transforming the test data, all of its entries become numeric values
|
|
2421
|
+
|
|
2422
|
+
**Task**
|
|
2423
|
+
|
|
2424
|
+
: After transforming the test data, all of its entries become numeric values. What is the **highest** value present in the transformed **test** feature matrix?
|
|
2425
|
+
|
|
2426
|
+
**Step-by-step approach**
|
|
2427
|
+
|
|
2428
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
2429
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
2430
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
2431
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
2432
|
+
|
|
2433
|
+
**Solution**
|
|
2434
|
+
|
|
2435
|
+
```python
|
|
2436
|
+
# Solution
|
|
2437
|
+
print(X_test_t.max())
|
|
2438
|
+
```
|
|
2439
|
+
|
|
2440
|
+
**Recorded result**
|
|
2441
|
+
|
|
2442
|
+
```text
|
|
2443
|
+
27.0
|
|
2444
|
+
```
|
|
2445
|
+
|
|
2446
|
+
---
|
|
2447
|
+
|
|
2448
|
+
### : After transforming the training data, all of its entries become numeric values
|
|
2449
|
+
|
|
2450
|
+
**Task**
|
|
2451
|
+
|
|
2452
|
+
: After transforming the training data, all of its entries become numeric values. What is the mean of the transformed **training** data matrix? Enter your answer correct to two decimal places.
|
|
2453
|
+
|
|
2454
|
+
**Step-by-step approach**
|
|
2455
|
+
|
|
2456
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
2457
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
2458
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
2459
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
2460
|
+
|
|
2461
|
+
**Solution**
|
|
2462
|
+
|
|
2463
|
+
```python
|
|
2464
|
+
# Solution
|
|
2465
|
+
print(round(X_train_t.mean(), 2))
|
|
2466
|
+
```
|
|
2467
|
+
|
|
2468
|
+
**Recorded result**
|
|
2469
|
+
|
|
2470
|
+
```text
|
|
2471
|
+
3.19
|
|
2472
|
+
```
|
|
2473
|
+
|
|
2474
|
+
---
|
|
2475
|
+
|
|
2476
|
+
### : What is the **highest** value of the scaled 'Duration' column in the transformed **test** data
|
|
2477
|
+
|
|
2478
|
+
**Task**
|
|
2479
|
+
|
|
2480
|
+
: What is the **highest** value of the scaled 'Duration' column in the transformed **test** data? Enter your answer correct to two decimal places.
|
|
2481
|
+
|
|
2482
|
+
**Step-by-step approach**
|
|
2483
|
+
|
|
2484
|
+
1. Separate numeric and categorical columns, and map each feature to its required operation.
|
|
2485
|
+
2. Build a Pipeline for any feature needing several operations in sequence, such as imputation followed by scaling.
|
|
2486
|
+
3. Pass transformer blocks to ColumnTransformer in the exact requested order; that order determines transformed column indices.
|
|
2487
|
+
4. Fit the full transformer on X_train, then transform both train and test data and inspect shape, mean, minimum or maximum as requested.
|
|
2488
|
+
|
|
2489
|
+
**Solution**
|
|
2490
|
+
|
|
2491
|
+
```python
|
|
2492
|
+
# Solution
|
|
2493
|
+
dur_idx = 7 # position of Duration in the ColumnTransformer output
|
|
2494
|
+
print(round(X_test_t[:, dur_idx].max(), 2))
|
|
2495
|
+
```
|
|
2496
|
+
|
|
2497
|
+
**Recorded result**
|
|
2498
|
+
|
|
2499
|
+
```text
|
|
2500
|
+
0.82
|
|
2501
|
+
```
|
|
2502
|
+
|
|
2503
|
+
---
|
|
2504
|
+
|
|
2505
|
+
### Train the ridge model on the training data with the following parameters:
|
|
2506
|
+
|
|
2507
|
+
**Task**
|
|
2508
|
+
|
|
2509
|
+
Train the ridge model on the training data with the following parameters:
|
|
2510
|
+
# `alpha = 1`
|
|
2511
|
+
# `solver = 'saga'`
|
|
2512
|
+
# `tol = 1e-3`
|
|
2513
|
+
# `random_state = 42`
|
|
2514
|
+
|
|
2515
|
+
# Enter the value of $R^2$ score on the test dataset.
|
|
2516
|
+
|
|
2517
|
+
**Step-by-step approach**
|
|
2518
|
+
|
|
2519
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2520
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2521
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2522
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2523
|
+
|
|
2524
|
+
**Solution**
|
|
2525
|
+
|
|
2526
|
+
```python
|
|
2527
|
+
# Solution
|
|
2528
|
+
from sklearn.linear_model import Ridge
|
|
2529
|
+
from sklearn.metrics import mean_absolute_error, mean_squared_error
|
|
2530
|
+
ridge = Ridge(alpha=1, solver='saga', tol=1e-3, random_state=42).fit(X_train, y_train)
|
|
2531
|
+
print(ridge.score(X_test, y_test))
|
|
2532
|
+
```
|
|
2533
|
+
|
|
2534
|
+
**Recorded result**
|
|
2535
|
+
|
|
2536
|
+
```text
|
|
2537
|
+
0.4134746439199958
|
|
2538
|
+
```
|
|
2539
|
+
|
|
2540
|
+
---
|
|
2541
|
+
|
|
2542
|
+
### : What is the best **alpha**
|
|
2543
|
+
|
|
2544
|
+
**Task**
|
|
2545
|
+
|
|
2546
|
+
: What is the best **alpha**?
|
|
2547
|
+
|
|
2548
|
+
**Step-by-step approach**
|
|
2549
|
+
|
|
2550
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
2551
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
2552
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
2553
|
+
|
|
2554
|
+
**Solution**
|
|
2555
|
+
|
|
2556
|
+
```python
|
|
2557
|
+
# Solution
|
|
2558
|
+
print(sgd_grid.best_params_['alpha'])
|
|
2559
|
+
```
|
|
2560
|
+
|
|
2561
|
+
**Recorded result**
|
|
2562
|
+
|
|
2563
|
+
```text
|
|
2564
|
+
1
|
|
2565
|
+
```
|
|
2566
|
+
|
|
2567
|
+
---
|
|
2568
|
+
|
|
2569
|
+
### : What will be value of **root mean squared error** on the test dataset
|
|
2570
|
+
|
|
2571
|
+
**Task**
|
|
2572
|
+
|
|
2573
|
+
: What will be value of **root mean squared error** on the test dataset? Enter your answer correct to two decimal places.
|
|
2574
|
+
|
|
2575
|
+
**Step-by-step approach**
|
|
2576
|
+
|
|
2577
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2578
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2579
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2580
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2581
|
+
|
|
2582
|
+
**Solution**
|
|
2583
|
+
|
|
2584
|
+
```python
|
|
2585
|
+
# Solution
|
|
2586
|
+
print(round(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))), 2))
|
|
2587
|
+
```
|
|
2588
|
+
|
|
2589
|
+
**Recorded result**
|
|
2590
|
+
|
|
2591
|
+
```text
|
|
2592
|
+
3580.16
|
|
2593
|
+
```
|
|
2594
|
+
|
|
2595
|
+
---
|
|
2596
|
+
|
|
2597
|
+
### : If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset
|
|
2598
|
+
|
|
2599
|
+
**Task**
|
|
2600
|
+
|
|
2601
|
+
: If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset? Enter your answer correct to two decimal places.
|
|
2602
|
+
|
|
2603
|
+
**Step-by-step approach**
|
|
2604
|
+
|
|
2605
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2606
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2607
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2608
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2609
|
+
|
|
2610
|
+
**Solution**
|
|
2611
|
+
|
|
2612
|
+
```python
|
|
2613
|
+
# Solution
|
|
2614
|
+
print(round(mean_absolute_error(y_test, pca_grid.predict(X_test)), 2))
|
|
2615
|
+
```
|
|
2616
|
+
|
|
2617
|
+
**Recorded result**
|
|
2618
|
+
|
|
2619
|
+
```text
|
|
2620
|
+
3630.4
|
|
2621
|
+
```
|
|
2622
|
+
|
|
2623
|
+
---
|
|
2624
|
+
|
|
2625
|
+
### : What **fraction** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model
|
|
2626
|
+
|
|
2627
|
+
**Task**
|
|
2628
|
+
|
|
2629
|
+
: What **fraction** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model? Enter your answer correct to two decimal places.
|
|
2630
|
+
|
|
2631
|
+
**Step-by-step approach**
|
|
2632
|
+
|
|
2633
|
+
1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
|
|
2634
|
+
2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
|
|
2635
|
+
3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
|
|
2636
|
+
4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
|
|
2637
|
+
|
|
2638
|
+
**Solution**
|
|
2639
|
+
|
|
2640
|
+
```python
|
|
2641
|
+
# Solution
|
|
2642
|
+
print(round(pca_grid.best_estimator_.named_steps['pca'].explained_variance_ratio_[0], 2))
|
|
2643
|
+
```
|
|
2644
|
+
|
|
2645
|
+
**Recorded result**
|
|
2646
|
+
|
|
2647
|
+
```text
|
|
2648
|
+
0.83
|
|
2649
|
+
```
|
|
2650
|
+
|
|
2651
|
+
---
|
|
2652
|
+
|
|
2653
|
+
### : create a pipeline of the PolynomialFeatures() as transformer and Lasso as an estimator with the following parameters:
|
|
2654
|
+
|
|
2655
|
+
**Task**
|
|
2656
|
+
|
|
2657
|
+
: create a pipeline of the PolynomialFeatures() as transformer and Lasso as an estimator with the following parameters:
|
|
2658
|
+
- For PolynomialFeatures()
|
|
2659
|
+
- interaction_only = False
|
|
2660
|
+
- degree = 3
|
|
2661
|
+
- For lasso()
|
|
2662
|
+
- alpha = 50
|
|
2663
|
+
- warm_start = True
|
|
2664
|
+
|
|
2665
|
+
Fit the pipeline on the training dataset and find the $R^2$ score on the test dataset.
|
|
2666
|
+
|
|
2667
|
+
**Step-by-step approach**
|
|
2668
|
+
|
|
2669
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2670
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2671
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2672
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2673
|
+
|
|
2674
|
+
**Solution**
|
|
2675
|
+
|
|
2676
|
+
```python
|
|
2677
|
+
# Solution
|
|
2678
|
+
from sklearn.preprocessing import PolynomialFeatures
|
|
2679
|
+
poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=3, interaction_only=False)),
|
|
2680
|
+
('lasso', Lasso(alpha=50, warm_start=True))])
|
|
2681
|
+
poly_pipe.fit(X_train, y_train)
|
|
2682
|
+
print(poly_pipe.score(X_test, y_test))
|
|
2683
|
+
```
|
|
2684
|
+
|
|
2685
|
+
**Recorded result**
|
|
2686
|
+
|
|
2687
|
+
```text
|
|
2688
|
+
0.5260776843902365
|
|
2689
|
+
```
|
|
2690
|
+
|
|
2691
|
+
---
|
|
2692
|
+
|
|
2693
|
+
## Insurance premium records
|
|
2694
|
+
|
|
2695
|
+
### Which of the following columns have **numerical** (int/float) datatype
|
|
2696
|
+
|
|
2697
|
+
**Task**
|
|
2698
|
+
|
|
2699
|
+
Which of the following columns have **numerical** (int/float) datatype?
|
|
2700
|
+
|
|
2701
|
+
**Step-by-step approach**
|
|
2702
|
+
|
|
2703
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
2704
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
2705
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
2706
|
+
|
|
2707
|
+
**Solution**
|
|
2708
|
+
|
|
2709
|
+
```python
|
|
2710
|
+
# Solution
|
|
2711
|
+
print(df.dtypes)
|
|
2712
|
+
```
|
|
2713
|
+
|
|
2714
|
+
**Recorded result**
|
|
2715
|
+
|
|
2716
|
+
```text
|
|
2717
|
+
Age float64
|
|
2718
|
+
Gender object
|
|
2719
|
+
Annual Income float64
|
|
2720
|
+
Number of Dependents float64
|
|
2721
|
+
Education Level object
|
|
2722
|
+
Health Score float64
|
|
2723
|
+
Location object
|
|
2724
|
+
Policy Type object
|
|
2725
|
+
Active Policy days int64
|
|
2726
|
+
Previous Claims float64
|
|
2727
|
+
Credit Score float64
|
|
2728
|
+
Insurance Duration float64
|
|
2729
|
+
Customer Feedback object
|
|
2730
|
+
Smoking Status object
|
|
2731
|
+
Exercise Frequency object
|
|
2732
|
+
Property Type object
|
|
2733
|
+
Premium Amount float64
|
|
2734
|
+
dtype: object
|
|
2735
|
+
```
|
|
2736
|
+
|
|
2737
|
+
---
|
|
2738
|
+
|
|
2739
|
+
### In this dataset, what is the most common **Policy Type** among customers **aged** 35 or below, living in **Suburban** locations and…
|
|
2740
|
+
|
|
2741
|
+
**Task**
|
|
2742
|
+
|
|
2743
|
+
In this dataset, what is the most common **Policy Type** among customers **aged** 35 or below, living in **Suburban** locations and having a **Credit Score** of 600 or above?
|
|
2744
|
+
|
|
2745
|
+
(A) Basic\
|
|
2746
|
+
(B) Comprehensive\
|
|
2747
|
+
(C) Premium\
|
|
2748
|
+
(D) Standard
|
|
2749
|
+
|
|
2750
|
+
V1: B\
|
|
2751
|
+
V2: A\
|
|
2752
|
+
V3: A
|
|
2753
|
+
|
|
2754
|
+
**Step-by-step approach**
|
|
2755
|
+
|
|
2756
|
+
1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
|
|
2757
|
+
2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
|
|
2758
|
+
3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
|
|
2759
|
+
4. Convert a scalar count/result to int when a clean numeric answer is needed.
|
|
2760
|
+
|
|
2761
|
+
**Solution**
|
|
2762
|
+
|
|
2763
|
+
```python
|
|
2764
|
+
# Solution
|
|
2765
|
+
sub = df[(df['Age'] <= 35) & (df['Location'] == 'Suburban') & (df['Credit Score'] >= 600)]
|
|
2766
|
+
print(sub['Policy Type'].value_counts())
|
|
2767
|
+
```
|
|
2768
|
+
|
|
2769
|
+
**Recorded result**
|
|
2770
|
+
|
|
2771
|
+
```text
|
|
2772
|
+
Policy Type
|
|
2773
|
+
Comprehensive 203
|
|
2774
|
+
Basic 200
|
|
2775
|
+
Premium 192
|
|
2776
|
+
Name: count, dtype: int64
|
|
2777
|
+
```
|
|
2778
|
+
|
|
2779
|
+
---
|
|
2780
|
+
|
|
2781
|
+
### Which of the following options represent all the unique categories present in the **Exercise Frequency** feature
|
|
2782
|
+
|
|
2783
|
+
**Task**
|
|
2784
|
+
|
|
2785
|
+
Which of the following options represent all the unique categories present in the **Exercise Frequency** feature?
|
|
2786
|
+
|
|
2787
|
+
(A) ['Weekly', 'Rarely', 'Monthly', 'Daily', 'Never']
|
|
2788
|
+
|
|
2789
|
+
(B) ['Weekly', 'Monthly', 'Daily']
|
|
2790
|
+
|
|
2791
|
+
(C) ['Weekly', 'Rarely', 'Monthly', 'Daily']
|
|
2792
|
+
|
|
2793
|
+
(D) ['Regularly', 'Rarely', 'Monthly', 'Weekly']
|
|
2794
|
+
|
|
2795
|
+
(E) None of these
|
|
2796
|
+
|
|
2797
|
+
All Versions: C
|
|
2798
|
+
|
|
2799
|
+
**Step-by-step approach**
|
|
2800
|
+
|
|
2801
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
2802
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
2803
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
2804
|
+
|
|
2805
|
+
**Solution**
|
|
2806
|
+
|
|
2807
|
+
```python
|
|
2808
|
+
# Solution
|
|
2809
|
+
print(df['Exercise Frequency'].unique())
|
|
2810
|
+
```
|
|
2811
|
+
|
|
2812
|
+
**Recorded result**
|
|
2813
|
+
|
|
2814
|
+
```text
|
|
2815
|
+
['Rarely' 'Weekly' 'Daily' 'Monthly']
|
|
2816
|
+
```
|
|
2817
|
+
|
|
2818
|
+
---
|
|
2819
|
+
|
|
2820
|
+
### Create feature matrix(X) and label vector(y) using following instructions:
|
|
2821
|
+
|
|
2822
|
+
**Task**
|
|
2823
|
+
|
|
2824
|
+
Create feature matrix(X) and label vector(y) using following instructions:
|
|
2825
|
+
* Premium Amount is the target column (y).
|
|
2826
|
+
* All the columns except the target column are part of the feature matrix (X).
|
|
2827
|
+
|
|
2828
|
+
Compare the correlation values for the following feature pairs of the feature matrix (X) and among the options select the one with the **strongest positive** correlation.
|
|
2829
|
+
|
|
2830
|
+
(A) `Annual Income` vs `Age`\
|
|
2831
|
+
(B) `Active Policy days` vs `Insurance Duration`\
|
|
2832
|
+
(C) `Previous Claims` vs `Credit Score`\
|
|
2833
|
+
(D) `Annual Income` vs `Credit Score`\
|
|
2834
|
+
(E) `Health Score` vs `Number of Dependents`
|
|
2835
|
+
|
|
2836
|
+
All Versions: C
|
|
2837
|
+
|
|
2838
|
+
**Step-by-step approach**
|
|
2839
|
+
|
|
2840
|
+
1. Select only the numeric columns before calculating correlations; strings and categories should not be treated as continuous measurements.
|
|
2841
|
+
2. Calculate the requested correlation matrix and ignore its diagonal, because every feature is perfectly correlated with itself.
|
|
2842
|
+
3. For the most negative pair, sort the off-diagonal values in ascending order; draw the heatmap only when a visual is requested.
|
|
2843
|
+
|
|
2844
|
+
**Solution**
|
|
2845
|
+
|
|
2846
|
+
```python
|
|
2847
|
+
# Solution
|
|
2848
|
+
X = df.drop(columns=['Premium Amount'])
|
|
2849
|
+
y = df['Premium Amount']
|
|
2850
|
+
corr = X.select_dtypes(include='number').corr()
|
|
2851
|
+
pairs = [('Annual Income', 'Age'), ('Active Policy days', 'Insurance Duration'),
|
|
2852
|
+
('Previous Claims', 'Credit Score'), ('Annual Income', 'Credit Score'),
|
|
2853
|
+
('Health Score', 'Number of Dependents')]
|
|
2854
|
+
for a, b in pairs:
|
|
2855
|
+
print(f'{a} vs {b}: {corr.loc[a, b]:.4f}')
|
|
2856
|
+
```
|
|
2857
|
+
|
|
2858
|
+
**Recorded result**
|
|
2859
|
+
|
|
2860
|
+
```text
|
|
2861
|
+
Annual Income vs Age: 0.0250
|
|
2862
|
+
Active Policy days vs Insurance Duration: -0.0285
|
|
2863
|
+
Previous Claims vs Credit Score: 0.0433
|
|
2864
|
+
Annual Income vs Credit Score: -0.2016
|
|
2865
|
+
Health Score vs Number of Dependents: 0.0046
|
|
2866
|
+
```
|
|
2867
|
+
|
|
2868
|
+
---
|
|
2869
|
+
|
|
2870
|
+
### Which of the following columns have missing values
|
|
2871
|
+
|
|
2872
|
+
**Task**
|
|
2873
|
+
|
|
2874
|
+
Which of the following columns have missing values?
|
|
2875
|
+
|
|
2876
|
+
**Step-by-step approach**
|
|
2877
|
+
|
|
2878
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
2879
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
2880
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
2881
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
2882
|
+
|
|
2883
|
+
**Solution**
|
|
2884
|
+
|
|
2885
|
+
```python
|
|
2886
|
+
# Solution
|
|
2887
|
+
print(df.isna().sum())
|
|
2888
|
+
```
|
|
2889
|
+
|
|
2890
|
+
**Recorded result**
|
|
2891
|
+
|
|
2892
|
+
```text
|
|
2893
|
+
Age 161
|
|
2894
|
+
Gender 0
|
|
2895
|
+
Annual Income 344
|
|
2896
|
+
Number of Dependents 884
|
|
2897
|
+
Education Level 0
|
|
2898
|
+
Health Score 616
|
|
2899
|
+
Location 0
|
|
2900
|
+
Policy Type 0
|
|
2901
|
+
Active Policy days 0
|
|
2902
|
+
Previous Claims 0
|
|
2903
|
+
Credit Score 1137
|
|
2904
|
+
Insurance Duration 0
|
|
2905
|
+
Customer Feedback 644
|
|
2906
|
+
Smoking Status 0
|
|
2907
|
+
Exercise Frequency 0
|
|
2908
|
+
Property Type 0
|
|
2909
|
+
Premium Amount 0
|
|
2910
|
+
dtype: int64
|
|
2911
|
+
```
|
|
2912
|
+
|
|
2913
|
+
---
|
|
2914
|
+
|
|
2915
|
+
### Split the dataset into train dataset and test dataset in the following manner
|
|
2916
|
+
|
|
2917
|
+
**Task**
|
|
2918
|
+
|
|
2919
|
+
Split the dataset into train dataset and test dataset in the following manner.
|
|
2920
|
+
|
|
2921
|
+
- Use sklearn train_test_split function to split the data.
|
|
2922
|
+
- Use only **25%** data as test_set and keep random_state = 42
|
|
2923
|
+
|
|
2924
|
+
How many **test** labels (**y_test**) are considered outliers? Outliers are defined as values that exceed the upper bound (Third Quartile + 1.5 x Interquartile Range) or fall below the lower bound (First Quartile - 1.5 x Interquartile Range). Compute the quartiles on **y_test** itself.
|
|
2925
|
+
|
|
2926
|
+
V1: 117\
|
|
2927
|
+
V2: 113\
|
|
2928
|
+
V3: 126
|
|
2929
|
+
|
|
2930
|
+
**Step-by-step approach**
|
|
2931
|
+
|
|
2932
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
2933
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
2934
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
2935
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
2936
|
+
|
|
2937
|
+
**Solution**
|
|
2938
|
+
|
|
2939
|
+
```python
|
|
2940
|
+
# Solution
|
|
2941
|
+
from sklearn.model_selection import train_test_split
|
|
2942
|
+
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
|
|
2943
|
+
q1, q3 = y_test.quantile(0.25), y_test.quantile(0.75)
|
|
2944
|
+
iqr = q3 - q1
|
|
2945
|
+
print(((y_test < q1 - 1.5 * iqr) | (y_test > q3 + 1.5 * iqr)).sum())
|
|
2946
|
+
```
|
|
2947
|
+
|
|
2948
|
+
**Recorded result**
|
|
2949
|
+
|
|
2950
|
+
```text
|
|
2951
|
+
117
|
|
2952
|
+
```
|
|
2953
|
+
|
|
2954
|
+
---
|
|
2955
|
+
|
|
2956
|
+
### Write the **MEAN** value of the **Age** column of the train dataset you found to replace all the Missing Values (NaN)
|
|
2957
|
+
|
|
2958
|
+
**Task**
|
|
2959
|
+
|
|
2960
|
+
Write the **MEAN** value of the **Age** column of the train dataset you found to replace all the Missing Values (NaN). (upto 1 digit after decimal point)
|
|
2961
|
+
|
|
2962
|
+
**Step-by-step approach**
|
|
2963
|
+
|
|
2964
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
2965
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
2966
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
2967
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
2968
|
+
|
|
2969
|
+
**Solution**
|
|
2970
|
+
|
|
2971
|
+
```python
|
|
2972
|
+
# Solution
|
|
2973
|
+
print(round(fill_values['Age'], 1))
|
|
2974
|
+
```
|
|
2975
|
+
|
|
2976
|
+
**Recorded result**
|
|
2977
|
+
|
|
2978
|
+
```text
|
|
2979
|
+
41.3
|
|
2980
|
+
```
|
|
2981
|
+
|
|
2982
|
+
---
|
|
2983
|
+
|
|
2984
|
+
### Write the value you used to replace the missing values in the **Number of Dependents** feature of the train dataset
|
|
2985
|
+
|
|
2986
|
+
**Task**
|
|
2987
|
+
|
|
2988
|
+
Write the value you used to replace the missing values in the **Number of Dependents** feature of the train dataset
|
|
2989
|
+
(A) 0\
|
|
2990
|
+
(B) 1\
|
|
2991
|
+
(C) 2\
|
|
2992
|
+
(D) 3\
|
|
2993
|
+
(E) 4\
|
|
2994
|
+
(F) None of these
|
|
2995
|
+
|
|
2996
|
+
**Step-by-step approach**
|
|
2997
|
+
|
|
2998
|
+
1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
|
|
2999
|
+
2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
|
|
3000
|
+
3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
|
|
3001
|
+
4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
|
|
3002
|
+
|
|
3003
|
+
**Solution**
|
|
3004
|
+
|
|
3005
|
+
```python
|
|
3006
|
+
# Solution
|
|
3007
|
+
print(fill_values['Number of Dependents'])
|
|
3008
|
+
```
|
|
3009
|
+
|
|
3010
|
+
**Recorded result**
|
|
3011
|
+
|
|
3012
|
+
```text
|
|
3013
|
+
3.0
|
|
3014
|
+
```
|
|
3015
|
+
|
|
3016
|
+
---
|
|
3017
|
+
|
|
3018
|
+
### Calculate the sum of all the values present in the **first 50 rows** of the transformed test feature matrix
|
|
3019
|
+
|
|
3020
|
+
**Task**
|
|
3021
|
+
|
|
3022
|
+
Calculate the sum of all the values present in the **first 50 rows** of the transformed test feature matrix ? (upto 2 digits after the decimal)
|
|
3023
|
+
|
|
3024
|
+
${\text{V1 : 405.34 (Range: 403, 408)}}$\
|
|
3025
|
+
${\text{V2 : 399.38 (Range: 397, 402)}}$\
|
|
3026
|
+
${\text{V3 : 419.33 (Range: 417, 422)}}$
|
|
3027
|
+
|
|
3028
|
+
**Step-by-step approach**
|
|
3029
|
+
|
|
3030
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
3031
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
3032
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
3033
|
+
|
|
3034
|
+
**Solution**
|
|
3035
|
+
|
|
3036
|
+
```python
|
|
3037
|
+
# Solution
|
|
3038
|
+
print(round(X_test_t[:50].sum(), 2))
|
|
3039
|
+
```
|
|
3040
|
+
|
|
3041
|
+
**Recorded result**
|
|
3042
|
+
|
|
3043
|
+
```text
|
|
3044
|
+
405.34
|
|
3045
|
+
```
|
|
3046
|
+
|
|
3047
|
+
---
|
|
3048
|
+
|
|
3049
|
+
### Enter the value of **Mean Absolute Error** on the test dataset
|
|
3050
|
+
|
|
3051
|
+
**Task**
|
|
3052
|
+
|
|
3053
|
+
Enter the value of **Mean Absolute Error** on the test dataset.
|
|
3054
|
+
|
|
3055
|
+
**Step-by-step approach**
|
|
3056
|
+
|
|
3057
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
3058
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
3059
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
3060
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
3061
|
+
|
|
3062
|
+
**Solution**
|
|
3063
|
+
|
|
3064
|
+
```python
|
|
3065
|
+
# Solution
|
|
3066
|
+
print(mean_absolute_error(y_test, ridge.predict(X_test)))
|
|
3067
|
+
```
|
|
3068
|
+
|
|
3069
|
+
**Recorded result**
|
|
3070
|
+
|
|
3071
|
+
```text
|
|
3072
|
+
666.013227845856
|
|
3073
|
+
```
|
|
3074
|
+
|
|
3075
|
+
---
|
|
3076
|
+
|
|
3077
|
+
### What is the **highest absolute coefficient value** of the trained Ridge model
|
|
3078
|
+
|
|
3079
|
+
**Task**
|
|
3080
|
+
|
|
3081
|
+
What is the **highest absolute coefficient value** of the trained Ridge model? (Ignore Intercept) (upto 2 digits after the decimal)
|
|
3082
|
+
|
|
3083
|
+
**Step-by-step approach**
|
|
3084
|
+
|
|
3085
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
3086
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
3087
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
3088
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
3089
|
+
|
|
3090
|
+
**Solution**
|
|
3091
|
+
|
|
3092
|
+
```python
|
|
3093
|
+
# Solution
|
|
3094
|
+
print(np.round(np.abs(ridge.coef_), 3))
|
|
3095
|
+
print(round(np.abs(ridge.coef_).max(), 2))
|
|
3096
|
+
```
|
|
3097
|
+
|
|
3098
|
+
**Recorded result**
|
|
3099
|
+
|
|
3100
|
+
```text
|
|
3101
|
+
[ 9.474 4.219 3.804 7.92 13.654 27.433 13.221 4.76 8.723 6.265
|
|
3102
|
+
6.321 0.711 1.628 1.783 18.109 11.759 0.085 13.013 36.555 9.772]
|
|
3103
|
+
36.55
|
|
3104
|
+
```
|
|
3105
|
+
|
|
3106
|
+
---
|
|
3107
|
+
|
|
3108
|
+
### Identify the index of the feature with the lowest absolute coefficient value
|
|
3109
|
+
|
|
3110
|
+
**Task**
|
|
3111
|
+
|
|
3112
|
+
Identify the index of the feature with the lowest absolute coefficient value. Note that index starts from 0. (Ignore Intercept)
|
|
3113
|
+
|
|
3114
|
+
**Step-by-step approach**
|
|
3115
|
+
|
|
3116
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
3117
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
3118
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
3119
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
3120
|
+
|
|
3121
|
+
**Solution**
|
|
3122
|
+
|
|
3123
|
+
```python
|
|
3124
|
+
# Solution
|
|
3125
|
+
print(np.argmin(np.abs(ridge.coef_)))
|
|
3126
|
+
```
|
|
3127
|
+
|
|
3128
|
+
**Recorded result**
|
|
3129
|
+
|
|
3130
|
+
```text
|
|
3131
|
+
16
|
|
3132
|
+
```
|
|
3133
|
+
|
|
3134
|
+
---
|
|
3135
|
+
|
|
3136
|
+
### What is the best **alpha**
|
|
3137
|
+
|
|
3138
|
+
**Task**
|
|
3139
|
+
|
|
3140
|
+
What is the best **alpha**?
|
|
3141
|
+
|
|
3142
|
+
**Step-by-step approach**
|
|
3143
|
+
|
|
3144
|
+
1. Identify the input columns, target (if any), and exact output requested.
|
|
3145
|
+
2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
|
|
3146
|
+
3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
|
|
3147
|
+
|
|
3148
|
+
**Solution**
|
|
3149
|
+
|
|
3150
|
+
```python
|
|
3151
|
+
# Solution
|
|
3152
|
+
print(sgd_grid.best_params_['alpha'])
|
|
3153
|
+
```
|
|
3154
|
+
|
|
3155
|
+
**Recorded result**
|
|
3156
|
+
|
|
3157
|
+
```text
|
|
3158
|
+
0.1
|
|
3159
|
+
```
|
|
3160
|
+
|
|
3161
|
+
---
|
|
3162
|
+
|
|
3163
|
+
### What will be value of **root mean squared error** on the test dataset
|
|
3164
|
+
|
|
3165
|
+
**Task**
|
|
3166
|
+
|
|
3167
|
+
What will be value of **root mean squared error** on the test dataset?
|
|
3168
|
+
|
|
3169
|
+
**Step-by-step approach**
|
|
3170
|
+
|
|
3171
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
3172
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
3173
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
3174
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
3175
|
+
|
|
3176
|
+
**Solution**
|
|
3177
|
+
|
|
3178
|
+
```python
|
|
3179
|
+
# Solution
|
|
3180
|
+
print(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))))
|
|
3181
|
+
```
|
|
3182
|
+
|
|
3183
|
+
**Recorded result**
|
|
3184
|
+
|
|
3185
|
+
```text
|
|
3186
|
+
872.3850595099008
|
|
3187
|
+
```
|
|
3188
|
+
|
|
3189
|
+
---
|
|
3190
|
+
|
|
3191
|
+
### If we fit the pipeline on the training dataset, what will be the **Mean Absolute Error** on the test dataset
|
|
3192
|
+
|
|
3193
|
+
**Task**
|
|
3194
|
+
|
|
3195
|
+
If we fit the pipeline on the training dataset, what will be the **Mean Absolute Error** on the test dataset?
|
|
3196
|
+
|
|
3197
|
+
**Step-by-step approach**
|
|
3198
|
+
|
|
3199
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
3200
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
3201
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
3202
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
3203
|
+
|
|
3204
|
+
**Solution**
|
|
3205
|
+
|
|
3206
|
+
```python
|
|
3207
|
+
# Solution
|
|
3208
|
+
print(mean_absolute_error(y_test, pca_grid.predict(X_test)))
|
|
3209
|
+
```
|
|
3210
|
+
|
|
3211
|
+
**Recorded result**
|
|
3212
|
+
|
|
3213
|
+
```text
|
|
3214
|
+
665.4809335203206
|
|
3215
|
+
```
|
|
3216
|
+
|
|
3217
|
+
---
|
|
3218
|
+
|
|
3219
|
+
### How many principal components are retained by the PCA step of the **best** pipeline found above
|
|
3220
|
+
|
|
3221
|
+
**Task**
|
|
3222
|
+
|
|
3223
|
+
How many principal components are retained by the PCA step of the **best** pipeline found above?
|
|
3224
|
+
|
|
3225
|
+
**Step-by-step approach**
|
|
3226
|
+
|
|
3227
|
+
1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
|
|
3228
|
+
2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
|
|
3229
|
+
3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
|
|
3230
|
+
4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
|
|
3231
|
+
|
|
3232
|
+
**Solution**
|
|
3233
|
+
|
|
3234
|
+
```python
|
|
3235
|
+
# Solution
|
|
3236
|
+
print(pca_grid.best_estimator_.named_steps['pca'].n_components_)
|
|
3237
|
+
```
|
|
3238
|
+
|
|
3239
|
+
**Recorded result**
|
|
3240
|
+
|
|
3241
|
+
```text
|
|
3242
|
+
15
|
|
3243
|
+
```
|
|
3244
|
+
|
|
3245
|
+
---
|
|
3246
|
+
|
|
3247
|
+
### Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
|
|
3248
|
+
|
|
3249
|
+
**Task**
|
|
3250
|
+
|
|
3251
|
+
Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
|
|
3252
|
+
- For PolynomialFeatures:
|
|
3253
|
+
- interaction_only = False
|
|
3254
|
+
- degree = 2
|
|
3255
|
+
- For Lasso:
|
|
3256
|
+
- alpha = 2
|
|
3257
|
+
- warm_start = True
|
|
3258
|
+
- random state as 0
|
|
3259
|
+
|
|
3260
|
+
Fit the pipeline on the training dataset and find the **Root Mean Square Error** on the test dataset.
|
|
3261
|
+
|
|
3262
|
+
**Step-by-step approach**
|
|
3263
|
+
|
|
3264
|
+
1. Identify the target column and remove it from the feature matrix before splitting.
|
|
3265
|
+
2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
|
|
3266
|
+
3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
|
|
3267
|
+
4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
|
|
3268
|
+
|
|
3269
|
+
**Solution**
|
|
3270
|
+
|
|
3271
|
+
```python
|
|
3272
|
+
# Solution
|
|
3273
|
+
from sklearn.preprocessing import PolynomialFeatures
|
|
3274
|
+
poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=2, interaction_only=False)),
|
|
3275
|
+
('lasso', Lasso(alpha=2, warm_start=True, random_state=0))])
|
|
3276
|
+
poly_pipe.fit(X_train, y_train)
|
|
3277
|
+
print(np.sqrt(mean_squared_error(y_test, poly_pipe.predict(X_test))))
|
|
3278
|
+
```
|
|
3279
|
+
|
|
3280
|
+
**Recorded result**
|
|
3281
|
+
|
|
3282
|
+
```text
|
|
3283
|
+
872.5174524584622
|
|
3284
|
+
```
|
|
3285
|
+
|
|
3286
|
+
---
|
|
3287
|
+
|
|
3288
|
+
### Create a pipeline for feature selection using f_regression as transformer and SGDRegressor as an estimator with the following…
|
|
3289
|
+
|
|
3290
|
+
**Task**
|
|
3291
|
+
|
|
3292
|
+
Create a pipeline for feature selection using f_regression as transformer and SGDRegressor as an estimator with the following parameters:
|
|
3293
|
+
- For feature selection:
|
|
3294
|
+
- select best **4** features based on f_regression score
|
|
3295
|
+
- For SGDRegressor:
|
|
3296
|
+
- random state as 42
|
|
3297
|
+
|
|
3298
|
+
Fit the pipeline on the training dataset and select the option that correctly displays the selected feature names in descending order based on their f_regression scores.
|
|
3299
|
+
|
|
3300
|
+
(A) ['Previous Claims', 'Location_Urban', 'Annual Income', 'Active Policy days']\
|
|
3301
|
+
(B) ['Previous Claims', 'Annual Income', 'Property Type_House', 'Customer Feedback_Good']\
|
|
3302
|
+
(C) ['Previous Claims', 'Policy Type_Comprehensive', 'Annual Income', 'Health Score'] \
|
|
3303
|
+
(D) ['Previous Claims', 'Annual Income', 'Location_Urban', 'Education Level'] \
|
|
3304
|
+
(E) None of the Above
|
|
3305
|
+
|
|
3306
|
+
**Step-by-step approach**
|
|
3307
|
+
|
|
3308
|
+
1. Fit the requested estimator/selector on the training feature matrix only.
|
|
3309
|
+
2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
|
|
3310
|
+
3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
|
|
3311
|
+
4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
|
|
3312
|
+
|
|
3313
|
+
**Solution**
|
|
3314
|
+
|
|
3315
|
+
```python
|
|
3316
|
+
# Solution
|
|
3317
|
+
from sklearn.feature_selection import SelectKBest, f_regression
|
|
3318
|
+
fs_pipe = Pipeline([('kbest', SelectKBest(f_regression, k=4)), ('sgd', SGDRegressor(random_state=42))])
|
|
3319
|
+
fs_pipe.fit(X_train, y_train)
|
|
3320
|
+
kb = fs_pipe.named_steps['kbest']
|
|
3321
|
+
top = np.argsort(kb.scores_)[::-1][:4]
|
|
3322
|
+
print(list(X_train.columns[top]))
|
|
3323
|
+
```
|
|
3324
|
+
|
|
3325
|
+
**Recorded result**
|
|
3326
|
+
|
|
3327
|
+
```text
|
|
3328
|
+
['Previous Claims', 'Location_Urban', 'Annual Income', 'Active Policy days']
|
|
3329
|
+
```
|
|
3330
|
+
|
|
3331
|
+
---
|
|
3332
|
+
|
|
3333
|
+
### For SGDRegressor, what is the actual number of iterations completed during model training
|
|
3334
|
+
|
|
3335
|
+
**Task**
|
|
3336
|
+
|
|
3337
|
+
For SGDRegressor, what is the actual number of iterations completed during model training?
|
|
3338
|
+
|
|
3339
|
+
**Step-by-step approach**
|
|
3340
|
+
|
|
3341
|
+
1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
|
|
3342
|
+
2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
|
|
3343
|
+
3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
|
|
3344
|
+
4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
|
|
3345
|
+
|
|
3346
|
+
**Solution**
|
|
3347
|
+
|
|
3348
|
+
```python
|
|
3349
|
+
# Solution
|
|
3350
|
+
print(fs_pipe.named_steps['sgd'].n_iter_)
|
|
3351
|
+
```
|
|
3352
|
+
|
|
3353
|
+
**Recorded result**
|
|
3354
|
+
|
|
3355
|
+
```text
|
|
3356
|
+
16
|
|
3357
|
+
```
|
|
3358
|
+
|
|
3359
|
+
---
|
|
3360
|
+
|
|
3361
|
+
## Residential valuation reference
|
|
3362
|
+
|
|
3363
|
+
### Load a CSV with a final target column
|
|
3364
|
+
|
|
3365
|
+
**Task**
|
|
3366
|
+
|
|
3367
|
+
Load a CSV whose first row contains column names, with no index column. The final column is the target and all preceding columns are features.
|
|
3368
|
+
|
|
3369
|
+
**Step-by-step approach**
|
|
3370
|
+
|
|
3371
|
+
1. Read the file with pd.read_csv and do not pass index_col=0 when the file has no index column.
|
|
3372
|
+
2. Inspect df.head(), df.shape and df.columns to confirm the header was read as column names.
|
|
3373
|
+
3. Use iloc to split by position: every column except the last becomes X and the last becomes y.
|
|
3374
|
+
|
|
3375
|
+
**Solution**
|
|
3376
|
+
|
|
3377
|
+
```python
|
|
3378
|
+
df_model = pd.read_csv("model_data.csv")
|
|
3379
|
+
print(df_model.head())
|
|
3380
|
+
X_model = df_model.iloc[:, :-1]
|
|
3381
|
+
y_model = df_model.iloc[:, -1]
|
|
3382
|
+
print(X_model.shape, y_model.shape)
|
|
3383
|
+
```
|
|
3384
|
+
|
|
3385
|
+
---
|
|
3386
|
+
|
|
3387
|
+
### Count dataset samples
|
|
3388
|
+
|
|
3389
|
+
**Task**
|
|
3390
|
+
|
|
3391
|
+
How many samples are there in the dataset?
|
|
3392
|
+
|
|
3393
|
+
**Step-by-step approach**
|
|
3394
|
+
|
|
3395
|
+
1. Read the row dimension, not the number of columns.
|
|
3396
|
+
2. Use shape[0]; shape[1] would count features.
|
|
3397
|
+
|
|
3398
|
+
**Solution**
|
|
3399
|
+
|
|
3400
|
+
```python
|
|
3401
|
+
print(df.shape[0])
|
|
3402
|
+
```
|
|
3403
|
+
|
|
3404
|
+
**Recorded result**
|
|
3405
|
+
|
|
3406
|
+
```text
|
|
3407
|
+
4000
|
|
3408
|
+
```
|
|
3409
|
+
|
|
3410
|
+
---
|
|
3411
|
+
|
|
3412
|
+
### Compute the mean target value
|
|
3413
|
+
|
|
3414
|
+
**Task**
|
|
3415
|
+
|
|
3416
|
+
What is the average house price (in lacs)?
|
|
3417
|
+
|
|
3418
|
+
**Step-by-step approach**
|
|
3419
|
+
|
|
3420
|
+
1. Select the target series.
|
|
3421
|
+
2. Use mean(); pandas skips genuine NaN values by default.
|
|
3422
|
+
|
|
3423
|
+
**Solution**
|
|
3424
|
+
|
|
3425
|
+
```python
|
|
3426
|
+
print(df["PRICE"].mean())
|
|
3427
|
+
```
|
|
3428
|
+
|
|
3429
|
+
**Recorded result**
|
|
3430
|
+
|
|
3431
|
+
```text
|
|
3432
|
+
24.355923220694248
|
|
3433
|
+
```
|
|
3434
|
+
|
|
3435
|
+
---
|
|
3436
|
+
|
|
3437
|
+
### Count records above a room threshold
|
|
3438
|
+
|
|
3439
|
+
**Task**
|
|
3440
|
+
|
|
3441
|
+
How many houses have 5 or more rooms?
|
|
3442
|
+
|
|
3443
|
+
**Step-by-step approach**
|
|
3444
|
+
|
|
3445
|
+
1. Compare the room-count series with the inclusive threshold.
|
|
3446
|
+
2. Sum the boolean mask to count True records.
|
|
3447
|
+
|
|
3448
|
+
**Solution**
|
|
3449
|
+
|
|
3450
|
+
```python
|
|
3451
|
+
print(int(df["RM"].ge(5).sum()))
|
|
3452
|
+
```
|
|
3453
|
+
|
|
3454
|
+
**Recorded result**
|
|
3455
|
+
|
|
3456
|
+
```text
|
|
3457
|
+
3953
|
|
3458
|
+
```
|
|
3459
|
+
|
|
3460
|
+
---
|
|
3461
|
+
|
|
3462
|
+
### Average of the ten highest target values
|
|
3463
|
+
|
|
3464
|
+
**Task**
|
|
3465
|
+
|
|
3466
|
+
What is the average price of the top 10 most expensive houses (in lacs)?
|
|
3467
|
+
|
|
3468
|
+
**Step-by-step approach**
|
|
3469
|
+
|
|
3470
|
+
1. Select exactly the ten largest target values with nlargest(10).
|
|
3471
|
+
2. Calculate the mean of that selected series.
|
|
3472
|
+
|
|
3473
|
+
**Solution**
|
|
3474
|
+
|
|
3475
|
+
```python
|
|
3476
|
+
print(df["PRICE"].nlargest(10).mean())
|
|
3477
|
+
```
|
|
3478
|
+
|
|
3479
|
+
**Recorded result**
|
|
3480
|
+
|
|
3481
|
+
```text
|
|
3482
|
+
52.36590175716407
|
|
3483
|
+
```
|
|
3484
|
+
|
|
3485
|
+
---
|
|
3486
|
+
|
|
3487
|
+
### Detect invalid room counts
|
|
3488
|
+
|
|
3489
|
+
**Task**
|
|
3490
|
+
|
|
3491
|
+
What is the total number of missing or unknown values in the number of rooms feature?
|
|
3492
|
+
|
|
3493
|
+
**Step-by-step approach**
|
|
3494
|
+
|
|
3495
|
+
1. Inspect unique values before declaring an invalid code.
|
|
3496
|
+
2. A room count below one is implausible in this dataset; the recorded sentinel is -1.
|
|
3497
|
+
3. Count the mask before replacing it with NaN.
|
|
3498
|
+
|
|
3499
|
+
**Solution**
|
|
3500
|
+
|
|
3501
|
+
```python
|
|
3502
|
+
bad_rooms = df["RM"].lt(1)
|
|
3503
|
+
print(int(bad_rooms.sum()))
|
|
3504
|
+
```
|
|
3505
|
+
|
|
3506
|
+
**Recorded result**
|
|
3507
|
+
|
|
3508
|
+
```text
|
|
3509
|
+
40
|
|
3510
|
+
```
|
|
3511
|
+
|
|
3512
|
+
---
|
|
3513
|
+
|
|
3514
|
+
### Detect invalid property ages
|
|
3515
|
+
|
|
3516
|
+
**Task**
|
|
3517
|
+
|
|
3518
|
+
What is the total number of missing or unknown values in the age feature?
|
|
3519
|
+
|
|
3520
|
+
**Step-by-step approach**
|
|
3521
|
+
|
|
3522
|
+
1. Inspect the minimum and unique age values.
|
|
3523
|
+
2. Negative ages are invalid; create a mask with lt(0).
|
|
3524
|
+
3. Count the mask before conversion to NaN.
|
|
3525
|
+
|
|
3526
|
+
**Solution**
|
|
3527
|
+
|
|
3528
|
+
```python
|
|
3529
|
+
bad_age = df["AGE"].lt(0)
|
|
3530
|
+
print(int(bad_age.sum()))
|
|
3531
|
+
```
|
|
3532
|
+
|
|
3533
|
+
**Recorded result**
|
|
3534
|
+
|
|
3535
|
+
```text
|
|
3536
|
+
50
|
|
3537
|
+
```
|
|
3538
|
+
|
|
3539
|
+
---
|
|
3540
|
+
|
|
3541
|
+
### Detect unknown riverside values
|
|
3542
|
+
|
|
3543
|
+
**Task**
|
|
3544
|
+
|
|
3545
|
+
What is the total number of missing or unknown values in the RIVERSIDE feature?
|
|
3546
|
+
|
|
3547
|
+
**Step-by-step approach**
|
|
3548
|
+
|
|
3549
|
+
1. Inspect value_counts(dropna=False) to discover the non-binary label.
|
|
3550
|
+
2. The recorded unknown label is UNKNOWN.
|
|
3551
|
+
3. Count it, then replace it with NaN before imputation.
|
|
3552
|
+
|
|
3553
|
+
**Solution**
|
|
3554
|
+
|
|
3555
|
+
```python
|
|
3556
|
+
bad_riverside = df["RIVERSIDE"].eq("UNKNOWN")
|
|
3557
|
+
print(int(bad_riverside.sum()))
|
|
3558
|
+
```
|
|
3559
|
+
|
|
3560
|
+
**Recorded result**
|
|
3561
|
+
|
|
3562
|
+
```text
|
|
3563
|
+
88
|
|
3564
|
+
```
|
|
3565
|
+
|
|
3566
|
+
---
|
|
3567
|
+
|
|
3568
|
+
### Count recent riverside houses
|
|
3569
|
+
|
|
3570
|
+
**Task**
|
|
3571
|
+
|
|
3572
|
+
How many houses are on riverside and were built within the last 50 years? Ignore rows missing either riverside status or age.
|
|
3573
|
+
|
|
3574
|
+
**Step-by-step approach**
|
|
3575
|
+
|
|
3576
|
+
1. Create a validity mask requiring both columns to be present.
|
|
3577
|
+
2. Combine the conditions: RIVERSIDE is YES and AGE is at most 50.
|
|
3578
|
+
3. Sum the combined boolean mask.
|
|
3579
|
+
|
|
3580
|
+
**Solution**
|
|
3581
|
+
|
|
3582
|
+
```python
|
|
3583
|
+
valid = df["RIVERSIDE"].notna() & df["AGE"].notna()
|
|
3584
|
+
recent_riverside = valid & df["RIVERSIDE"].eq("YES") & df["AGE"].le(50)
|
|
3585
|
+
print(int(recent_riverside.sum()))
|
|
3586
|
+
```
|
|
3587
|
+
|
|
3588
|
+
**Recorded result**
|
|
3589
|
+
|
|
3590
|
+
```text
|
|
3591
|
+
44
|
|
3592
|
+
```
|
|
3593
|
+
|
|
3594
|
+
---
|
|
3595
|
+
|
|
3596
|
+
### Count a highway-count interval
|
|
3597
|
+
|
|
3598
|
+
**Task**
|
|
3599
|
+
|
|
3600
|
+
How many houses are near to exactly 6, 7 or 8 highways?
|
|
3601
|
+
|
|
3602
|
+
**Step-by-step approach**
|
|
3603
|
+
|
|
3604
|
+
1. The interval includes both 6 and 8.
|
|
3605
|
+
2. Use between(6, 8, inclusive="both") and sum the mask.
|
|
3606
|
+
|
|
3607
|
+
**Solution**
|
|
3608
|
+
|
|
3609
|
+
```python
|
|
3610
|
+
print(int(df["HIGHWAYCOUNT"].between(6, 8, inclusive="both").sum()))
|
|
3611
|
+
```
|
|
3612
|
+
|
|
3613
|
+
**Recorded result**
|
|
3614
|
+
|
|
3615
|
+
```text
|
|
3616
|
+
1211
|
|
3617
|
+
```
|
|
3618
|
+
|
|
3619
|
+
---
|
|
3620
|
+
|
|
3621
|
+
### Create price categories
|
|
3622
|
+
|
|
3623
|
+
**Task**
|
|
3624
|
+
|
|
3625
|
+
Create CATEGORY using five price bands and find the category with the most records.
|
|
3626
|
+
|
|
3627
|
+
**Step-by-step approach**
|
|
3628
|
+
|
|
3629
|
+
1. Define bin edges so each range is left-closed and right-open.
|
|
3630
|
+
2. Use pd.cut with right=False to implement the stated boundaries.
|
|
3631
|
+
3. Count categories and choose the category with the largest count.
|
|
3632
|
+
|
|
3633
|
+
**Solution**
|
|
3634
|
+
|
|
3635
|
+
```python
|
|
3636
|
+
bins = [-np.inf, 10, 20, 30, 40, np.inf]
|
|
3637
|
+
labels = [1, 2, 3, 4, 5]
|
|
3638
|
+
df["CATEGORY"] = pd.cut(df["PRICE"], bins=bins, labels=labels, right=False)
|
|
3639
|
+
counts = df["CATEGORY"].value_counts(sort=False)
|
|
3640
|
+
print(counts)
|
|
3641
|
+
print("Most frequent category:", counts.idxmax())
|
|
3642
|
+
```
|
|
3643
|
+
|
|
3644
|
+
**Recorded result**
|
|
3645
|
+
|
|
3646
|
+
```text
|
|
3647
|
+
3
|
|
3648
|
+
```
|
|
3649
|
+
|
|
3650
|
+
---
|
|
3651
|
+
|
|
3652
|
+
### Clean sentinels and split housing data
|
|
3653
|
+
|
|
3654
|
+
**Task**
|
|
3655
|
+
|
|
3656
|
+
Replace invalid RM, AGE and RIVERSIDE values with NaN, use PRICE as target, and create a 70:30 split with random_state=0. Find the training sample count.
|
|
3657
|
+
|
|
3658
|
+
**Step-by-step approach**
|
|
3659
|
+
|
|
3660
|
+
1. Create masks from the observed sentinel rules: RM below 1, AGE below 0, and RIVERSIDE equal to UNKNOWN.
|
|
3661
|
+
2. Replace those entries with np.nan on a copy of the data.
|
|
3662
|
+
3. Separate PRICE from the feature matrix, then split with test_size=0.30 and random_state=0.
|
|
3663
|
+
4. The training set retains 70% of 4000 rows.
|
|
3664
|
+
|
|
3665
|
+
**Solution**
|
|
3666
|
+
|
|
3667
|
+
```python
|
|
3668
|
+
data = df.copy()
|
|
3669
|
+
rm_bad = data["RM"].lt(1)
|
|
3670
|
+
age_bad = data["AGE"].lt(0)
|
|
3671
|
+
riv_bad = data["RIVERSIDE"].eq("UNKNOWN")
|
|
3672
|
+
data.loc[rm_bad, "RM"] = np.nan
|
|
3673
|
+
data.loc[age_bad, "AGE"] = np.nan
|
|
3674
|
+
data.loc[riv_bad, "RIVERSIDE"] = np.nan
|
|
3675
|
+
X = data.drop(columns=["PRICE", "CATEGORY"], errors="ignore")
|
|
3676
|
+
y = data["PRICE"]
|
|
3677
|
+
X_train, X_test, y_train, y_test = train_test_split(
|
|
3678
|
+
X, y, test_size=0.30, random_state=0
|
|
3679
|
+
)
|
|
3680
|
+
print(X_train.shape[0])
|
|
3681
|
+
```
|
|
3682
|
+
|
|
3683
|
+
**Recorded result**
|
|
3684
|
+
|
|
3685
|
+
```text
|
|
3686
|
+
2800
|
|
3687
|
+
```
|
|
3688
|
+
|
|
3689
|
+
---
|
|
3690
|
+
|
|
3691
|
+
### Build an ordered heterogeneous preprocessor
|
|
3692
|
+
|
|
3693
|
+
**Task**
|
|
3694
|
+
|
|
3695
|
+
Apply the specified min-max scaling, standard scaling, train-fitted imputation and one-hot encoding in the stated feature order; find the transformed feature count.
|
|
3696
|
+
|
|
3697
|
+
**Step-by-step approach**
|
|
3698
|
+
|
|
3699
|
+
1. Drop the derived CATEGORY column before model preprocessing.
|
|
3700
|
+
2. Create one transformer block per ordered feature group; a Pipeline performs imputation before scaling.
|
|
3701
|
+
3. Fit ColumnTransformer on X_train and only transform X_test.
|
|
3702
|
+
4. OneHotEncoder creates two output columns for the two known riverside categories; verify the final matrix width.
|
|
3703
|
+
|
|
3704
|
+
**Solution**
|
|
3705
|
+
|
|
3706
|
+
```python
|
|
3707
|
+
X_train = X_train.drop(columns=["CATEGORY"], errors="ignore")
|
|
3708
|
+
X_test = X_test.drop(columns=["CATEGORY"], errors="ignore")
|
|
3709
|
+
preprocessor = ColumnTransformer([
|
|
3710
|
+
("crim", MinMaxScaler(), ["CRIM"]),
|
|
3711
|
+
("zn", MinMaxScaler(), ["ZN"]),
|
|
3712
|
+
("indus", StandardScaler(), ["INDUS"]),
|
|
3713
|
+
("polindex", MinMaxScaler(), ["POLINDEX"]),
|
|
3714
|
+
("dis", MinMaxScaler(), ["DIS"]),
|
|
3715
|
+
("highways", MinMaxScaler(), ["HIGHWAYCOUNT"]),
|
|
3716
|
+
("tax", MinMaxScaler(), ["TAX"]),
|
|
3717
|
+
("ptratio", MinMaxScaler(), ["PTRATIO"]),
|
|
3718
|
+
("imm", MinMaxScaler(), ["IMM"]),
|
|
3719
|
+
("bpl", MinMaxScaler(), ["BPL"]),
|
|
3720
|
+
("rooms", Pipeline([
|
|
3721
|
+
("imputer", SimpleImputer(strategy="median")),
|
|
3722
|
+
("scaler", MinMaxScaler()),
|
|
3723
|
+
]), ["RM"]),
|
|
3724
|
+
("age", Pipeline([
|
|
3725
|
+
("imputer", SimpleImputer(strategy="mean")),
|
|
3726
|
+
("scaler", MinMaxScaler()),
|
|
3727
|
+
]), ["AGE"]),
|
|
3728
|
+
("riverside", Pipeline([
|
|
3729
|
+
("imputer", SimpleImputer(strategy="most_frequent")),
|
|
3730
|
+
("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
|
|
3731
|
+
]), ["RIVERSIDE"]),
|
|
3732
|
+
], remainder="drop")
|
|
3733
|
+
X_train_t = preprocessor.fit_transform(X_train)
|
|
3734
|
+
X_test_t = preprocessor.transform(X_test)
|
|
3735
|
+
print(X_train_t.shape[1])
|
|
3736
|
+
```
|
|
3737
|
+
|
|
3738
|
+
**Recorded result**
|
|
3739
|
+
|
|
3740
|
+
```text
|
|
3741
|
+
14
|
|
3742
|
+
```
|
|
3743
|
+
|
|
3744
|
+
---
|
|
3745
|
+
|
|
3746
|
+
### Compute the global mean of transformed test data
|
|
3747
|
+
|
|
3748
|
+
**Task**
|
|
3749
|
+
|
|
3750
|
+
What is the mean of the transformed test data, using all matrix entries?
|
|
3751
|
+
|
|
3752
|
+
**Step-by-step approach**
|
|
3753
|
+
|
|
3754
|
+
1. Use the test matrix after applying the fitted training preprocessor.
|
|
3755
|
+
2. Calling mean() on the NumPy matrix with no axis computes one scalar across all rows and columns.
|
|
3756
|
+
|
|
3757
|
+
**Solution**
|
|
3758
|
+
|
|
3759
|
+
```python
|
|
3760
|
+
print(X_test_t.mean())
|
|
3761
|
+
```
|
|
3762
|
+
|
|
3763
|
+
**Recorded result**
|
|
3764
|
+
|
|
3765
|
+
```text
|
|
3766
|
+
0.38617456550975626
|
|
3767
|
+
```
|
|
3768
|
+
|
|
3769
|
+
---
|
|
3770
|
+
|
|
3771
|
+
### Split model data and fit Ridge
|
|
3772
|
+
|
|
3773
|
+
**Task**
|
|
3774
|
+
|
|
3775
|
+
Use the last column as y, split with test_size=0.30 and random_state=42, then fit Ridge(alpha=10, solver="saga", tol=1e-4, random_state=42). Find test R².
|
|
3776
|
+
|
|
3777
|
+
**Step-by-step approach**
|
|
3778
|
+
|
|
3779
|
+
1. Treat every column except the last as X and the last as y.
|
|
3780
|
+
2. Use the exact random split settings.
|
|
3781
|
+
3. Fit Ridge with the fixed hyperparameters, then call score on the test data.
|
|
3782
|
+
|
|
3783
|
+
**Solution**
|
|
3784
|
+
|
|
3785
|
+
```python
|
|
3786
|
+
X_model = df_model.iloc[:, :-1]
|
|
3787
|
+
y_model = df_model.iloc[:, -1]
|
|
3788
|
+
X_train, X_test, y_train, y_test = train_test_split(
|
|
3789
|
+
X_model, y_model, test_size=0.30, random_state=42
|
|
3790
|
+
)
|
|
3791
|
+
ridge = Ridge(alpha=10, solver="saga", tol=1e-4, random_state=42)
|
|
3792
|
+
ridge.fit(X_train, y_train)
|
|
3793
|
+
print(ridge.score(X_test, y_test))
|
|
3794
|
+
```
|
|
3795
|
+
|
|
3796
|
+
**Recorded result**
|
|
3797
|
+
|
|
3798
|
+
```text
|
|
3799
|
+
0.6613547575262211
|
|
3800
|
+
```
|
|
3801
|
+
|
|
3802
|
+
---
|
|
3803
|
+
|
|
3804
|
+
### Find the most important Ridge coefficient
|
|
3805
|
+
|
|
3806
|
+
**Task**
|
|
3807
|
+
|
|
3808
|
+
What is the zero-based index of the feature with the highest absolute coefficient, ignoring the intercept?
|
|
3809
|
+
|
|
3810
|
+
**Step-by-step approach**
|
|
3811
|
+
|
|
3812
|
+
1. Read coef_; intercept_ is separate and must not be included.
|
|
3813
|
+
2. Take absolute values because importance is based on magnitude, not sign.
|
|
3814
|
+
3. Use argmax to obtain the zero-based position.
|
|
3815
|
+
|
|
3816
|
+
**Solution**
|
|
3817
|
+
|
|
3818
|
+
```python
|
|
3819
|
+
most_important_index = int(np.argmax(np.abs(ridge.coef_)))
|
|
3820
|
+
print(most_important_index)
|
|
3821
|
+
```
|
|
3822
|
+
|
|
3823
|
+
**Recorded result**
|
|
3824
|
+
|
|
3825
|
+
```text
|
|
3826
|
+
9
|
|
3827
|
+
```
|
|
3828
|
+
|
|
3829
|
+
---
|
|
3830
|
+
|
|
3831
|
+
### Find the least important Ridge coefficient
|
|
3832
|
+
|
|
3833
|
+
**Task**
|
|
3834
|
+
|
|
3835
|
+
What is the zero-based index of the feature with the lowest absolute coefficient, ignoring the intercept?
|
|
3836
|
+
|
|
3837
|
+
**Step-by-step approach**
|
|
3838
|
+
|
|
3839
|
+
1. Read the fitted coefficient vector only.
|
|
3840
|
+
2. Take absolute values and use argmin to find the smallest magnitude.
|
|
3841
|
+
|
|
3842
|
+
**Solution**
|
|
3843
|
+
|
|
3844
|
+
```python
|
|
3845
|
+
least_important_index = int(np.argmin(np.abs(ridge.coef_)))
|
|
3846
|
+
print(least_important_index)
|
|
3847
|
+
```
|
|
3848
|
+
|
|
3849
|
+
**Recorded result**
|
|
3850
|
+
|
|
3851
|
+
```text
|
|
3852
|
+
0
|
|
3853
|
+
```
|
|
3854
|
+
|
|
3855
|
+
---
|
|
3856
|
+
|
|
3857
|
+
### Tune SGDRegressor with GridSearchCV
|
|
3858
|
+
|
|
3859
|
+
**Task**
|
|
3860
|
+
|
|
3861
|
+
Tune penalty, alpha and tol for SGDRegressor(random_state=42), using five folds and negative mean absolute error. Find the best penalty.
|
|
3862
|
+
|
|
3863
|
+
**Step-by-step approach**
|
|
3864
|
+
|
|
3865
|
+
1. Use a parameter grid containing every specified candidate.
|
|
3866
|
+
2. Set cv=5 and scoring="neg_mean_absolute_error" exactly.
|
|
3867
|
+
3. Fit the grid on training data and inspect best_params_.
|
|
3868
|
+
|
|
3869
|
+
**Solution**
|
|
3870
|
+
|
|
3871
|
+
```python
|
|
3872
|
+
sgd_grid = GridSearchCV(
|
|
3873
|
+
SGDRegressor(random_state=42),
|
|
3874
|
+
{
|
|
3875
|
+
"penalty": ["l1", "l2"],
|
|
3876
|
+
"alpha": [1e-5, 1e-4, 1e-3, 1e-2, 1e-1],
|
|
3877
|
+
"tol": [1e-4, 1e-3, 1e-2, 1e-1],
|
|
3878
|
+
},
|
|
3879
|
+
cv=5, scoring="neg_mean_absolute_error", n_jobs=-1,
|
|
3880
|
+
)
|
|
3881
|
+
sgd_grid.fit(X_train, y_train)
|
|
3882
|
+
print(sgd_grid.best_params_["penalty"])
|
|
3883
|
+
```
|
|
3884
|
+
|
|
3885
|
+
**Recorded result**
|
|
3886
|
+
|
|
3887
|
+
```text
|
|
3888
|
+
l2
|
|
3889
|
+
```
|
|
3890
|
+
|
|
3891
|
+
---
|
|
3892
|
+
|
|
3893
|
+
### Evaluate the tuned SGDRegressor
|
|
3894
|
+
|
|
3895
|
+
**Task**
|
|
3896
|
+
|
|
3897
|
+
What is the mean absolute error on the held-out test dataset using the best SGD model?
|
|
3898
|
+
|
|
3899
|
+
**Step-by-step approach**
|
|
3900
|
+
|
|
3901
|
+
1. GridSearchCV refits the best estimator on all training rows by default.
|
|
3902
|
+
2. Predict the test target and calculate positive MAE.
|
|
3903
|
+
|
|
3904
|
+
**Solution**
|
|
3905
|
+
|
|
3906
|
+
```python
|
|
3907
|
+
y_pred = sgd_grid.predict(X_test)
|
|
3908
|
+
print(mean_absolute_error(y_test, y_pred))
|
|
3909
|
+
```
|
|
3910
|
+
|
|
3911
|
+
**Recorded result**
|
|
3912
|
+
|
|
3913
|
+
```text
|
|
3914
|
+
3.8131121797994014
|
|
3915
|
+
```
|
|
3916
|
+
|
|
3917
|
+
---
|
|
3918
|
+
|
|
3919
|
+
### Tune a PCA and Lasso pipeline
|
|
3920
|
+
|
|
3921
|
+
**Task**
|
|
3922
|
+
|
|
3923
|
+
Build PCA followed by Lasso and tune n_components and alpha with five-fold negative-MAE GridSearchCV. Evaluate the best pipeline with test R².
|
|
3924
|
+
|
|
3925
|
+
**Step-by-step approach**
|
|
3926
|
+
|
|
3927
|
+
1. Put PCA and Lasso in a Pipeline so PCA is refitted inside each CV fold.
|
|
3928
|
+
2. Use the step-prefixed keys pca__n_components and lasso__alpha.
|
|
3929
|
+
3. Fit GridSearchCV on training data, then score the best refitted pipeline on X_test.
|
|
3930
|
+
|
|
3931
|
+
**Solution**
|
|
3932
|
+
|
|
3933
|
+
```python
|
|
3934
|
+
pca_lasso = Pipeline([("pca", PCA()), ("lasso", Lasso())])
|
|
3935
|
+
pca_grid = GridSearchCV(
|
|
3936
|
+
pca_lasso,
|
|
3937
|
+
{"pca__n_components": [0.9, 0.95], "lasso__alpha": [10, 1, 0.01, 0.001]},
|
|
3938
|
+
cv=5, scoring="neg_mean_absolute_error", n_jobs=-1,
|
|
3939
|
+
)
|
|
3940
|
+
pca_grid.fit(X_train, y_train)
|
|
3941
|
+
print(pca_grid.best_estimator_.score(X_test, y_test))
|
|
3942
|
+
```
|
|
3943
|
+
|
|
3944
|
+
**Recorded result**
|
|
3945
|
+
|
|
3946
|
+
```text
|
|
3947
|
+
0.6288625430197575
|
|
3948
|
+
```
|
|
3949
|
+
|
|
3950
|
+
---
|
|
3951
|
+
|
|
3952
|
+
### Read the first PCA eigenvalue
|
|
3953
|
+
|
|
3954
|
+
**Task**
|
|
3955
|
+
|
|
3956
|
+
How much variance (eigenvalue) is explained by the first principal component of the best PCA pipeline?
|
|
3957
|
+
|
|
3958
|
+
**Step-by-step approach**
|
|
3959
|
+
|
|
3960
|
+
1. Access the fitted PCA object from best_estimator_.named_steps.
|
|
3961
|
+
2. Use explained_variance_[0] for the eigenvalue. Do not substitute explained_variance_ratio_[0], which is a fraction.
|
|
3962
|
+
|
|
3963
|
+
**Solution**
|
|
3964
|
+
|
|
3965
|
+
```python
|
|
3966
|
+
best_pca = pca_grid.best_estimator_.named_steps["pca"]
|
|
3967
|
+
print(best_pca.explained_variance_[0])
|
|
3968
|
+
```
|
|
3969
|
+
|
|
3970
|
+
**Recorded result**
|
|
3971
|
+
|
|
3972
|
+
```text
|
|
3973
|
+
1.1635075742239045
|
|
3974
|
+
```
|
|
3975
|
+
|
|
3976
|
+
---
|
|
3977
|
+
|
|
3978
|
+
### Fit polynomial features with Lasso
|
|
3979
|
+
|
|
3980
|
+
**Task**
|
|
3981
|
+
|
|
3982
|
+
Build PolynomialFeatures(degree=2, interaction_only=False) followed by Lasso(alpha=1, warm_start=True, random_state=0). Find test R².
|
|
3983
|
+
|
|
3984
|
+
**Step-by-step approach**
|
|
3985
|
+
|
|
3986
|
+
1. Create the transformer and estimator with the exact stated parameters.
|
|
3987
|
+
2. Place them in a Pipeline so prediction applies the same polynomial expansion.
|
|
3988
|
+
3. Fit the pipeline on training data and evaluate with score on test data.
|
|
3989
|
+
|
|
3990
|
+
**Solution**
|
|
3991
|
+
|
|
3992
|
+
```python
|
|
3993
|
+
poly_lasso = Pipeline([
|
|
3994
|
+
("poly", PolynomialFeatures(degree=2, interaction_only=False)),
|
|
3995
|
+
("lasso", Lasso(alpha=1, warm_start=True, random_state=0)),
|
|
3996
|
+
])
|
|
3997
|
+
poly_lasso.fit(X_train, y_train)
|
|
3998
|
+
print(poly_lasso.score(X_test, y_test))
|
|
3999
|
+
```
|
|
4000
|
+
|
|
4001
|
+
**Recorded result**
|
|
4002
|
+
|
|
4003
|
+
```text
|
|
4004
|
+
0.157678032410551
|
|
4005
|
+
```
|
|
4006
|
+
|
|
4007
|
+
---
|
|
4008
|
+
|
|
4009
|
+
### Remove one feature with recursive elimination
|
|
4010
|
+
|
|
4011
|
+
**Task**
|
|
4012
|
+
|
|
4013
|
+
Use RFE with default LinearRegression to eliminate one feature from the processed training matrix. Find the eliminated zero-based index.
|
|
4014
|
+
|
|
4015
|
+
**Step-by-step approach**
|
|
4016
|
+
|
|
4017
|
+
1. Set n_features_to_select to the input feature count minus one.
|
|
4018
|
+
2. Fit RFE on the training features and labels.
|
|
4019
|
+
3. support_ is True for retained features; its logical inverse identifies eliminated positions.
|
|
4020
|
+
|
|
4021
|
+
**Solution**
|
|
4022
|
+
|
|
4023
|
+
```python
|
|
4024
|
+
rfe = RFE(
|
|
4025
|
+
estimator=LinearRegression(),
|
|
4026
|
+
n_features_to_select=X_train.shape[1] - 1,
|
|
4027
|
+
)
|
|
4028
|
+
rfe.fit(X_train, y_train)
|
|
4029
|
+
print(np.flatnonzero(~rfe.support_))
|
|
4030
|
+
```
|
|
4031
|
+
|
|
4032
|
+
**Recorded result**
|
|
4033
|
+
|
|
4034
|
+
```text
|
|
4035
|
+
2
|
|
4036
|
+
```
|
|
4037
|
+
|
|
4038
|
+
---
|