scikit-doc 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,4038 @@
1
+ # Worked examples
2
+
3
+ This collection reorganizes every technical exercise from the five supplied notebooks and the solved housing reference into topic-oriented examples. Dataset selectors and grading labels are omitted; each task appears as a standalone documentation-style example.
4
+
5
+ The notebook code fragments assume the source notebook has already loaded the relevant DataFrame and completed prerequisite transformations. The companion examples state their required variables explicitly. Recorded outputs are included only when present in the supplied source.
6
+
7
+ **Total worked examples: 127**
8
+
9
+ ## Customer value records
10
+
11
+ ### How many records (rows) are there in the dataset
12
+
13
+ **Task**
14
+
15
+ How many records (rows) are there in the dataset?
16
+
17
+ **Step-by-step approach**
18
+
19
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
20
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
21
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
22
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
23
+
24
+ **Solution**
25
+
26
+ ```python
27
+ # Solution
28
+ print(df.shape[0])
29
+ ```
30
+
31
+ **Recorded result**
32
+
33
+ ```text
34
+ 6257
35
+ ```
36
+
37
+ ---
38
+
39
+ ### What are the unique values of the feature `qualification` in the dataset
40
+
41
+ **Task**
42
+
43
+ What are the unique values of the feature `qualification` in the dataset?
44
+
45
+ **Step-by-step approach**
46
+
47
+ 1. Identify the input columns, target (if any), and exact output requested.
48
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
49
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
50
+
51
+ **Solution**
52
+
53
+ ```python
54
+ # Solution
55
+ print(df['qualification'].unique())
56
+ ```
57
+
58
+ **Recorded result**
59
+
60
+ ```text
61
+ ['High School' 'Bachelor' 'Others']
62
+ ```
63
+
64
+ ---
65
+
66
+ ### Which of the following columns have numerical (int/float) data
67
+
68
+ **Task**
69
+
70
+ Which of the following columns have numerical (int/float) data?
71
+
72
+ **Step-by-step approach**
73
+
74
+ 1. Identify the input columns, target (if any), and exact output requested.
75
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
76
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
77
+
78
+ **Solution**
79
+
80
+ ```python
81
+ # Solution
82
+ print(df.dtypes)
83
+ ```
84
+
85
+ **Recorded result**
86
+
87
+ ```text
88
+ id int64
89
+ gender object
90
+ area object
91
+ qualification object
92
+ income float64
93
+ marital_status int64
94
+ vintage int64
95
+ claim_amount float64
96
+ num_policies object
97
+ policy object
98
+ type_of_policy object
99
+ cltv int64
100
+ dtype: object
101
+ ```
102
+
103
+ ---
104
+
105
+ ### Plot the `heatmap` of the correlation matrix of the numerical columns and mark the pair which has the **lowest (most negative)**…
106
+
107
+ **Task**
108
+
109
+ Plot the `heatmap` of the correlation matrix of the numerical columns and mark the pair which has the **lowest (most negative)** correlation value.
110
+
111
+ **Step-by-step approach**
112
+
113
+ 1. Select only the numeric columns before calculating correlations; strings and categories should not be treated as continuous measurements.
114
+ 2. Calculate the requested correlation matrix and ignore its diagonal, because every feature is perfectly correlated with itself.
115
+ 3. For the most negative pair, sort the off-diagonal values in ascending order; draw the heatmap only when a visual is requested.
116
+
117
+ **Solution**
118
+
119
+ ```python
120
+ # Solution
121
+ corr = df.select_dtypes(include='number').corr()
122
+ plt.figure(figsize=(8, 6))
123
+ sns.heatmap(corr, annot=True, fmt='.3f', cmap='coolwarm')
124
+ plt.show()
125
+ # Locate the most negative off-diagonal pair
126
+ pairs = corr.where(~np.eye(len(corr), dtype=bool)).stack()
127
+ print(pairs.sort_values().head(4))
128
+ ```
129
+
130
+ ---
131
+
132
+ ### Which of the following features has the **highest number** of `missing` values
133
+
134
+ **Task**
135
+
136
+ Which of the following features has the **highest number** of `missing` values?
137
+
138
+ **Step-by-step approach**
139
+
140
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
141
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
142
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
143
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
144
+
145
+ **Solution**
146
+
147
+ ```python
148
+ # Solution
149
+ print(df.isna().sum().sort_values(ascending=False))
150
+ ```
151
+
152
+ **Recorded result**
153
+
154
+ ```text
155
+ income 401
156
+ area 384
157
+ policy 372
158
+ claim_amount 360
159
+ qualification 0
160
+ gender 0
161
+ id 0
162
+ marital_status 0
163
+ vintage 0
164
+ num_policies 0
165
+ type_of_policy 0
166
+ cltv 0
167
+ dtype: int64
168
+ ```
169
+
170
+ ---
171
+
172
+ ### Break the dataset into features(`X`) and label (`y`), where the column `cltv` goes to `y` and the rest of the columns go to `X`
173
+
174
+ **Task**
175
+
176
+ Break the dataset into features(`X`) and label (`y`), where the column `cltv` goes to `y` and the rest of the columns go to `X`. Enter the **median** value of `cltv` column?
177
+
178
+ **Step-by-step approach**
179
+
180
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
181
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
182
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
183
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
184
+
185
+ **Solution**
186
+
187
+ ```python
188
+ # Solution
189
+ X = df.drop(columns=['cltv'])
190
+ y = df['cltv']
191
+ print(y.median())
192
+ ```
193
+
194
+ **Recorded result**
195
+
196
+ ```text
197
+ 66288.0
198
+ ```
199
+
200
+ ---
201
+
202
+ ### Split the dataset into training and test dataset using `train_test_split` into `75:25` ratio while keeping random_state = 42
203
+
204
+ **Task**
205
+
206
+ Split the dataset into training and test dataset using `train_test_split` into `75:25` ratio while keeping random_state = 42. What is the shape of the test set (X_test)?
207
+
208
+ **Step-by-step approach**
209
+
210
+ 1. Identify the target column and remove it from the feature matrix before splitting.
211
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
212
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
213
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
214
+
215
+ **Solution**
216
+
217
+ ```python
218
+ # Solution
219
+ from sklearn.model_selection import train_test_split
220
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
221
+ print(X_test.shape)
222
+ ```
223
+
224
+ **Recorded result**
225
+
226
+ ```text
227
+ (1565, 11)
228
+ ```
229
+
230
+ ---
231
+
232
+ ### Drop(remove) `id` column from train and test data because it is not useful in model training
233
+
234
+ **Task**
235
+
236
+ Drop(remove) `id` column from train and test data because it is not useful in model training. Now how many of the remaining feature columns in the training dataset are of **object (string)** type?
237
+
238
+ **Step-by-step approach**
239
+
240
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
241
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
242
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
243
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
244
+
245
+ **Solution**
246
+
247
+ ```python
248
+ # Solution
249
+ X_train = X_train.drop(columns=['id'])
250
+ X_test = X_test.drop(columns=['id'])
251
+ obj_cols = X_train.select_dtypes(include='object').columns
252
+ print(list(obj_cols))
253
+ print(len(obj_cols))
254
+ ```
255
+
256
+ **Recorded result**
257
+
258
+ ```text
259
+ ['gender', 'area', 'qualification', 'num_policies', 'policy', 'type_of_policy']
260
+ 6
261
+ ```
262
+
263
+ ---
264
+
265
+ ### Compute and write the **mean** of the `income` column of X_train while ignoring the missing values
266
+
267
+ **Task**
268
+
269
+ Compute and write the **mean** of the `income` column of X_train while ignoring the missing values. Replace all NaN values in the income column of X_train and X_test by the mean computed from the X_train (upto two decimal).
270
+
271
+ **Step-by-step approach**
272
+
273
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
274
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
275
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
276
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
277
+
278
+ **Solution**
279
+
280
+ ```python
281
+ # Solution
282
+ income_mean = X_train['income'].mean()
283
+ print(round(income_mean, 2))
284
+ X_train['income'] = X_train['income'].fillna(income_mean)
285
+ X_test['income'] = X_test['income'].fillna(income_mean)
286
+ ```
287
+
288
+ **Recorded result**
289
+
290
+ ```text
291
+ 13.73
292
+ ```
293
+
294
+ ---
295
+
296
+ ### How many missing values are present in the `policy` column of **X_test**
297
+
298
+ **Task**
299
+
300
+ How many missing values are present in the `policy` column of **X_test**? After answering, replace all NaN values in the `policy` column of X_train and X_test by the most frequent value in X_train.
301
+
302
+ **Step-by-step approach**
303
+
304
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
305
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
306
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
307
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
308
+
309
+ **Solution**
310
+
311
+ ```python
312
+ # Solution
313
+ print(X_test['policy'].isna().sum())
314
+ policy_mode = X_train['policy'].mode()[0]
315
+ print('Most frequent policy in X_train:', policy_mode)
316
+ X_train['policy'] = X_train['policy'].fillna(policy_mode)
317
+ X_test['policy'] = X_test['policy'].fillna(policy_mode)
318
+ ```
319
+
320
+ **Recorded result**
321
+
322
+ ```text
323
+ 102
324
+ Most frequent policy in X_train: A
325
+ ```
326
+
327
+ ---
328
+
329
+ ### Replace all NaN values in the `area` column of X_train and X_test by the most frequent value from X_train
330
+
331
+ **Task**
332
+
333
+ Replace all NaN values in the `area` column of X_train and X_test by the most frequent value from X_train. After replacing, how many records in **X_train** have `area` equal to that most frequent value?
334
+
335
+ **Step-by-step approach**
336
+
337
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
338
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
339
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
340
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
341
+
342
+ **Solution**
343
+
344
+ ```python
345
+ # Solution
346
+ area_mode = X_train['area'].mode()[0]
347
+ print('Most frequent area in X_train:', area_mode)
348
+ X_train['area'] = X_train['area'].fillna(area_mode)
349
+ X_test['area'] = X_test['area'].fillna(area_mode)
350
+ print((X_train['area'] == area_mode).sum())
351
+ ```
352
+
353
+ **Recorded result**
354
+
355
+ ```text
356
+ Most frequent area in X_train: Urban
357
+ 3407
358
+ ```
359
+
360
+ ---
361
+
362
+ ### Replace all NaN values in the claim_amount column of X_train and X_test by the **median** of claim_amount computed from X_train
363
+
364
+ **Task**
365
+
366
+ Replace all NaN values in the claim_amount column of X_train and X_test by the **median** of claim_amount computed from X_train. After replacing, what is the **mean** of the claim_amount column in **X_test**? (correct upto two decimal places)
367
+
368
+ **Step-by-step approach**
369
+
370
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
371
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
372
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
373
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
374
+
375
+ **Solution**
376
+
377
+ ```python
378
+ # Solution
379
+ claim_median = X_train['claim_amount'].median()
380
+ X_train['claim_amount'] = X_train['claim_amount'].fillna(claim_median)
381
+ X_test['claim_amount'] = X_test['claim_amount'].fillna(claim_median)
382
+ print(round(X_test['claim_amount'].mean(), 2))
383
+ ```
384
+
385
+ **Recorded result**
386
+
387
+ ```text
388
+ 4260.88
389
+ ```
390
+
391
+ ---
392
+
393
+ ### Apply `StandardScaler` on the `vintage` column of X_train (store the result in a separate variable, do **not** overwrite X_train)
394
+
395
+ **Task**
396
+
397
+ Apply `StandardScaler` on the `vintage` column of X_train (store the result in a separate variable, do **not** overwrite X_train). Compute and write the **maximum** value of the scaled `vintage` column. (correct upto 2 decimal)
398
+
399
+ **Step-by-step approach**
400
+
401
+ 1. Separate numeric and categorical columns, and map each feature to its required operation.
402
+ 2. Build a Pipeline for any feature needing several operations in sequence, such as imputation followed by scaling.
403
+ 3. Pass transformer blocks to ColumnTransformer in the exact requested order; that order determines transformed column indices.
404
+ 4. Fit the full transformer on X_train, then transform both train and test data and inspect shape, mean, minimum or maximum as requested.
405
+
406
+ **Solution**
407
+
408
+ ```python
409
+ # Solution
410
+ from sklearn.preprocessing import StandardScaler
411
+ vintage_scaled = StandardScaler().fit_transform(X_train[['vintage']])
412
+ print(round(vintage_scaled.max(), 2))
413
+ ```
414
+
415
+ **Recorded result**
416
+
417
+ ```text
418
+ 1.47
419
+ ```
420
+
421
+ ---
422
+
423
+ ### How many columns are produced by the **One-Hot Encoding step alone** (i
424
+
425
+ **Task**
426
+
427
+ How many columns are produced by the **One-Hot Encoding step alone** (i.e. the number of columns in $X_1$)?
428
+
429
+ [Options]
430
+
431
+ **Step-by-step approach**
432
+
433
+ 1. List categorical columns and inspect their unique values, including missing values.
434
+ 2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
435
+ 3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
436
+ 4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
437
+
438
+ **Solution**
439
+
440
+ ```python
441
+ # Solution
442
+ print(ct.named_transformers_['ohe'].get_feature_names_out().shape[0])
443
+ ```
444
+
445
+ **Recorded result**
446
+
447
+ ```text
448
+ 17
449
+ ```
450
+
451
+ ---
452
+
453
+ ### Apply `SequentialFeatureSelector` transformer with direction= 'forward' with `LinearRegression()` estimator and select **6**…
454
+
455
+ **Task**
456
+
457
+ Apply `SequentialFeatureSelector` transformer with direction= 'forward' with `LinearRegression()` estimator and select **6** features by fitting to the X_train and y_train.
458
+
459
+ `Use cv = KFold(n_splits=5,random_state=42,shuffle=True) in SequentialFeatureSelector.`
460
+
461
+ ### Which of the following options represents the correct integer index of the selected features list?
462
+
463
+ **Step-by-step approach**
464
+
465
+ 1. Fit the requested estimator/selector on the training feature matrix only.
466
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
467
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
468
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
469
+
470
+ **Solution**
471
+
472
+ ```python
473
+ # Solution
474
+ from sklearn.feature_selection import SequentialFeatureSelector
475
+ from sklearn.linear_model import LinearRegression
476
+ from sklearn.model_selection import KFold
477
+ cv = KFold(n_splits=5, random_state=42, shuffle=True)
478
+ sfs = SequentialFeatureSelector(LinearRegression(), n_features_to_select=6, direction='forward', cv=cv)
479
+ sfs.fit(X_train, y_train)
480
+ print(np.where(sfs.get_support())[0])
481
+ ```
482
+
483
+ **Recorded result**
484
+
485
+ ```text
486
+ [ 4 5 7 9 13 19]
487
+ ```
488
+
489
+ ---
490
+
491
+ ### Apply `LinearRegression` on the training set(`X_train` and `y_train`)
492
+
493
+ **Task**
494
+
495
+ Apply `LinearRegression` on the training set(`X_train` and `y_train`). What is the **mean absolute error** on the test set(`X_test` and `y_test`)? (Upto 2 digits after decimal point)
496
+
497
+ **Step-by-step approach**
498
+
499
+ 1. Identify the target column and remove it from the feature matrix before splitting.
500
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
501
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
502
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
503
+
504
+ **Solution**
505
+
506
+ ```python
507
+ # Solution
508
+ from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
509
+ lr = LinearRegression().fit(X_train, y_train)
510
+ print(round(mean_absolute_error(y_test, lr.predict(X_test)), 2))
511
+ ```
512
+
513
+ **Recorded result**
514
+
515
+ ```text
516
+ 52712.56
517
+ ```
518
+
519
+ ---
520
+
521
+ ### Using the `LinearRegression` model, compute the `cross-validation scores` for `5 splits` on training data (X_train and y_train)…
522
+
523
+ **Task**
524
+
525
+ Using the `LinearRegression` model, compute the `cross-validation scores` for `5 splits` on training data (X_train and y_train) using `cross_val_score`. Enter the **mean** value of the $R^2$ scores obtained. (Upto 4 digits after decimal point)
526
+
527
+ `Use cv = KFold(n_splits=5,random_state=42,shuffle=True) in cross_val_score.`
528
+
529
+ (**Hint**: By default cross_val_score uses LinearRegression's scoring metric, which is $R^2$ score.)
530
+
531
+ **Step-by-step approach**
532
+
533
+ 1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
534
+ 2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
535
+ 3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
536
+ 4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
537
+
538
+ **Solution**
539
+
540
+ ```python
541
+ # Solution
542
+ from sklearn.model_selection import cross_val_score
543
+ scores = cross_val_score(LinearRegression(), X_train, y_train, cv=KFold(n_splits=5, random_state=42, shuffle=True))
544
+ print(scores)
545
+ print(round(scores.mean(), 4))
546
+ ```
547
+
548
+ **Recorded result**
549
+
550
+ ```text
551
+ [0.15765982 0.14449678 0.14566491 0.13373985 0.15322638]
552
+ 0.147
553
+ ```
554
+
555
+ ---
556
+
557
+ ### Apply `Ridge` regression with **random_state=42** and **alpha=10** on training set(`X_train and y_train`) and calculate the $R^2$…
558
+
559
+ **Task**
560
+
561
+ Apply `Ridge` regression with **random_state=42** and **alpha=10** on training set(`X_train and y_train`) and calculate the $R^2$ score on test_set (`X_test and y_test`). What is the correct score (Upto 4 digits after decimal point)?
562
+
563
+ **Step-by-step approach**
564
+
565
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
566
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
567
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
568
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
569
+
570
+ **Solution**
571
+
572
+ ```python
573
+ # Solution
574
+ from sklearn.linear_model import Ridge
575
+ ridge = Ridge(alpha=10, random_state=42).fit(X_train, y_train)
576
+ print(round(ridge.score(X_test, y_test), 4))
577
+ ```
578
+
579
+ **Recorded result**
580
+
581
+ ```text
582
+ 0.1421
583
+ ```
584
+
585
+ ---
586
+
587
+ ### Apply `Lasso` regression with **random_state=42** and **regularization rate (alpha)=100** on the training data(`X_train & y_train`)
588
+
589
+ **Task**
590
+
591
+ Apply `Lasso` regression with **random_state=42** and **regularization rate (alpha)=100** on the training data(`X_train & y_train`). Enter the value of the **intercept** correct upto 2 digits after decimal point.
592
+
593
+ **Step-by-step approach**
594
+
595
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
596
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
597
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
598
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
599
+
600
+ **Solution**
601
+
602
+ ```python
603
+ # Solution
604
+ from sklearn.linear_model import Lasso
605
+ lasso = Lasso(alpha=100, random_state=42).fit(X_train, y_train)
606
+ print(round(lasso.intercept_, 2))
607
+ ```
608
+
609
+ **Recorded result**
610
+
611
+ ```text
612
+ 116658.43
613
+ ```
614
+
615
+ ---
616
+
617
+ ### Fit SGDRegressor(`random_state=42`) estimator on the training data(`X_train & y_train`) and predict labels for test_data(`X_test`),…
618
+
619
+ **Task**
620
+
621
+ Fit SGDRegressor(`random_state=42`) estimator on the training data(`X_train & y_train`) and predict labels for test_data(`X_test`), lets call it as y_test_predict. The parameters are initialized with default values. Calculate the **root mean squared error** between y_test and y_test_predict. (Correct upto two decimals)
622
+
623
+ **Step-by-step approach**
624
+
625
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
626
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
627
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
628
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
629
+
630
+ **Solution**
631
+
632
+ ```python
633
+ # Solution
634
+ from sklearn.linear_model import SGDRegressor
635
+ sgd = SGDRegressor(random_state=42).fit(X_train, y_train)
636
+ y_test_predict = sgd.predict(X_test)
637
+ print(round(np.sqrt(mean_squared_error(y_test, y_test_predict)), 2))
638
+ ```
639
+
640
+ **Recorded result**
641
+
642
+ ```text
643
+ 88459.51
644
+ ```
645
+
646
+ ---
647
+
648
+ ### Using SGDRegressor(random_state=42) as an estimator for exactly **25** iterations (`max_iter=25`)
649
+
650
+ **Task**
651
+
652
+ Using SGDRegressor(random_state=42) as an estimator for exactly **25** iterations (`max_iter=25`). Write the correct $R^2$ score on test data (correct Upto 4 digits)
653
+
654
+ **Step-by-step approach**
655
+
656
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
657
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
658
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
659
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
660
+
661
+ **Solution**
662
+
663
+ ```python
664
+ # Solution
665
+ sgd25 = SGDRegressor(random_state=42, max_iter=25).fit(X_train, y_train)
666
+ print(round(sgd25.score(X_test, y_test), 4))
667
+ ```
668
+
669
+ **Recorded result**
670
+
671
+ ```text
672
+ 0.1397
673
+ ```
674
+
675
+ ---
676
+
677
+ ### Mark the best `alpha` value you got using above instructions
678
+
679
+ **Task**
680
+
681
+ Mark the best `alpha` value you got using above instructions.
682
+
683
+ **Step-by-step approach**
684
+
685
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
686
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
687
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
688
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
689
+
690
+ **Solution**
691
+
692
+ ```python
693
+ # Solution
694
+ print(grid.best_params_['lasso__alpha'])
695
+ ```
696
+
697
+ **Recorded result**
698
+
699
+ ```text
700
+ 10.0
701
+ ```
702
+
703
+ ---
704
+
705
+ ### Enter the best polynomial degree value you got using above instructions
706
+
707
+ **Task**
708
+
709
+ Enter the best polynomial degree value you got using above instructions.
710
+
711
+ **Step-by-step approach**
712
+
713
+ 1. Identify the input columns, target (if any), and exact output requested.
714
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
715
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
716
+
717
+ **Solution**
718
+
719
+ ```python
720
+ # Solution
721
+ print(grid.best_params_['poly__degree'])
722
+ ```
723
+
724
+ **Recorded result**
725
+
726
+ ```text
727
+ 2
728
+ ```
729
+
730
+ ---
731
+
732
+ ### What is the sum of `explained_variance_ratio_`
733
+
734
+ **Task**
735
+
736
+ What is the sum of `explained_variance_ratio_` ? (upto 4 decimal)
737
+
738
+ **Step-by-step approach**
739
+
740
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
741
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
742
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
743
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
744
+
745
+ **Solution**
746
+
747
+ ```python
748
+ # Solution
749
+ print(round(pca.explained_variance_ratio_.sum(), 4))
750
+ ```
751
+
752
+ **Recorded result**
753
+
754
+ ```text
755
+ 0.7816
756
+ ```
757
+
758
+ ---
759
+
760
+ ### Use PCA transformed training data from earlier question and y_train to fit the `RidgeCV` estimator model having `alpha values as [0
761
+
762
+ **Task**
763
+
764
+ Use PCA transformed training data from earlier question and y_train to fit the `RidgeCV` estimator model having `alpha values as [0.01, 0.1, 1, 10, 100]`. Calculate the $R^2$ score you got from the model for transformed test data(PCA transformed X_test). (upto 4 decimal)
765
+
766
+ **Step-by-step approach**
767
+
768
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
769
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
770
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
771
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
772
+
773
+ **Solution**
774
+
775
+ ```python
776
+ # Solution
777
+ from sklearn.linear_model import RidgeCV
778
+ rcv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100]).fit(X_train_pca, y_train)
779
+ print(round(rcv.score(X_test_pca, y_test), 4))
780
+ ```
781
+
782
+ **Recorded result**
783
+
784
+ ```text
785
+ 0.1308
786
+ ```
787
+
788
+ ---
789
+
790
+ ## Residential price records
791
+
792
+ ### How many columns (features + target) are there in the dataset
793
+
794
+ **Task**
795
+
796
+ How many columns (features + target) are there in the dataset?
797
+
798
+ **Step-by-step approach**
799
+
800
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
801
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
802
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
803
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
804
+
805
+ **Solution**
806
+
807
+ ```python
808
+ # Solution
809
+ print(df.shape[1])
810
+ ```
811
+
812
+ **Recorded result**
813
+
814
+ ```text
815
+ 14
816
+ ```
817
+
818
+ ---
819
+
820
+ ### What is the **median** house price (in lacs)
821
+
822
+ **Task**
823
+
824
+ What is the **median** house price (in lacs)?
825
+
826
+ **Step-by-step approach**
827
+
828
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
829
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
830
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
831
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
832
+
833
+ **Solution**
834
+
835
+ ```python
836
+ # Solution
837
+ print(df['PRICE'].median())
838
+ ```
839
+
840
+ **Recorded result**
841
+
842
+ ```text
843
+ 22.634566447766908
844
+ ```
845
+
846
+ ---
847
+
848
+ ### How many houses have **9 or more** rooms
849
+
850
+ **Task**
851
+
852
+ How many houses have **9 or more** rooms?
853
+
854
+ **Step-by-step approach**
855
+
856
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
857
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
858
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
859
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
860
+
861
+ **Solution**
862
+
863
+ ```python
864
+ # Solution
865
+ print((df['RM'] >= 9).sum())
866
+ ```
867
+
868
+ **Recorded result**
869
+
870
+ ```text
871
+ 318
872
+ ```
873
+
874
+ ---
875
+
876
+ ### What is the average price of the **10 least expensive** houses (in lacs)
877
+
878
+ **Task**
879
+
880
+ What is the average price of the **10 least expensive** houses (in lacs)?
881
+
882
+ **Step-by-step approach**
883
+
884
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
885
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
886
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
887
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
888
+
889
+ **Solution**
890
+
891
+ ```python
892
+ # Solution
893
+ print(df['PRICE'].nsmallest(10).mean())
894
+ ```
895
+
896
+ **Recorded result**
897
+
898
+ ```text
899
+ 7.278891822010996
900
+ ```
901
+
902
+ ---
903
+
904
+ ### What is the **total** number of missing or unknown values in the `RM`, `AGE` and `RIVERSIDE` features **combined**
905
+
906
+ **Task**
907
+
908
+ What is the **total** number of missing or unknown values in the `RM`, `AGE` and `RIVERSIDE` features **combined**?
909
+
910
+ **Step-by-step approach**
911
+
912
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
913
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
914
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
915
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
916
+
917
+ **Solution**
918
+
919
+ ```python
920
+ # Solution
921
+ print(rm_bad.sum() + age_bad.sum() + riv_bad.sum())
922
+ ```
923
+
924
+ **Recorded result**
925
+
926
+ ```text
927
+ 178
928
+ ```
929
+
930
+ ---
931
+
932
+ ### What is the average age (in years) of the houses, **ignoring** the missing or unknown values of the `AGE` feature
933
+
934
+ **Task**
935
+
936
+ What is the average age (in years) of the houses, **ignoring** the missing or unknown values of the `AGE` feature?
937
+
938
+ **Step-by-step approach**
939
+
940
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
941
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
942
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
943
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
944
+
945
+ **Solution**
946
+
947
+ ```python
948
+ # Solution
949
+ print(df.loc[~age_bad, 'AGE'].mean())
950
+ ```
951
+
952
+ **Recorded result**
953
+
954
+ ```text
955
+ 66.86582278481012
956
+ ```
957
+
958
+ ---
959
+
960
+ ### What **percentage** of the houses have an unknown value in the `RIVERSIDE` feature
961
+
962
+ **Task**
963
+
964
+ What **percentage** of the houses have an unknown value in the `RIVERSIDE` feature? (Enter the answer correct upto 2 decimal places)
965
+
966
+ **Step-by-step approach**
967
+
968
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
969
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
970
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
971
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
972
+
973
+ **Solution**
974
+
975
+ ```python
976
+ # Solution
977
+ print(round(riv_bad.mean() * 100, 2))
978
+ ```
979
+
980
+ **Recorded result**
981
+
982
+ ```text
983
+ 2.2
984
+ ```
985
+
986
+ ---
987
+
988
+ ### How many houses are **not** on the riverside and are **older than 60 years** (i
989
+
990
+ **Task**
991
+
992
+ How many houses are **not** on the riverside and are **older than 60 years** (i.e. AGE > 60)? For this question, ignore the rows that have missing values in either riverside feature or age feature.
993
+
994
+ **Step-by-step approach**
995
+
996
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
997
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
998
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
999
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1000
+
1001
+ **Solution**
1002
+
1003
+ ```python
1004
+ # Solution
1005
+ valid = ~riv_bad & ~age_bad
1006
+ print(((df['RIVERSIDE'] == 'NO') & (df['AGE'] > 60) & valid).sum())
1007
+ ```
1008
+
1009
+ **Recorded result**
1010
+
1011
+ ```text
1012
+ 2088
1013
+ ```
1014
+
1015
+ ---
1016
+
1017
+ ### How many houses are near to exactly 3, 4 or 5 highways (all three inclusive)
1018
+
1019
+ **Task**
1020
+
1021
+ How many houses are near to exactly 3, 4 or 5 highways (all three inclusive)?
1022
+
1023
+ **Step-by-step approach**
1024
+
1025
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1026
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1027
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1028
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1029
+
1030
+ **Solution**
1031
+
1032
+ ```python
1033
+ # Solution
1034
+ print(df['HIGHWAYCOUNT'].between(3, 5).sum())
1035
+ ```
1036
+
1037
+ **Recorded result**
1038
+
1039
+ ```text
1040
+ 1742
1041
+ ```
1042
+
1043
+ ---
1044
+
1045
+ ### Create a column 'CATEGORY' and divide the houses in categories as following:
1046
+
1047
+ **Task**
1048
+
1049
+ Create a column 'CATEGORY' and divide the houses in categories as following:
1050
+
1051
+ - Category 1: house price <15 lacs
1052
+ - Category 2: 15 lacs <= house price <25 lacs
1053
+ - Category 3: 25 lacs <= house price <35 lacs
1054
+ - Category 4: house price >=35 lacs
1055
+
1056
+ Which category has the **lowest** number of records?
1057
+
1058
+ **Step-by-step approach**
1059
+
1060
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1061
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1062
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1063
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1064
+
1065
+ **Solution**
1066
+
1067
+ ```python
1068
+ # Solution
1069
+ df['CATEGORY'] = pd.cut(df['PRICE'], bins=[-np.inf, 15, 25, 35, np.inf], right=False, labels=[1, 2, 3, 4])
1070
+ print(df['CATEGORY'].value_counts().sort_index())
1071
+ ```
1072
+
1073
+ **Recorded result**
1074
+
1075
+ ```text
1076
+ CATEGORY
1077
+ 1 364
1078
+ 2 2318
1079
+ 3 889
1080
+ 4 429
1081
+ Name: count, dtype: int64
1082
+ ```
1083
+
1084
+ ---
1085
+
1086
+ ### What is the number of samples in the **test** set
1087
+
1088
+ **Task**
1089
+
1090
+ What is the number of samples in the **test** set?
1091
+
1092
+ **Step-by-step approach**
1093
+
1094
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1095
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1096
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1097
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1098
+
1099
+ **Solution**
1100
+
1101
+ ```python
1102
+ # Solution
1103
+ from sklearn.model_selection import train_test_split
1104
+ df.loc[rm_bad, 'RM'] = np.nan
1105
+ df.loc[age_bad, 'AGE'] = np.nan
1106
+ df.loc[riv_bad, 'RIVERSIDE'] = np.nan
1107
+ X = df.drop(columns=['PRICE'])
1108
+ y = df['PRICE']
1109
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
1110
+ print(X_test.shape[0])
1111
+ ```
1112
+
1113
+ **Recorded result**
1114
+
1115
+ ```text
1116
+ 1000
1117
+ ```
1118
+
1119
+ ---
1120
+
1121
+ ### How many features are there after performing above transformation
1122
+
1123
+ **Task**
1124
+
1125
+ How many features are there after performing above transformation?
1126
+
1127
+ **Step-by-step approach**
1128
+
1129
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1130
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1131
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1132
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1133
+
1134
+ **Solution**
1135
+
1136
+ ```python
1137
+ # Solution
1138
+ print(X_train_t.shape[1])
1139
+ ```
1140
+
1141
+ **Recorded result**
1142
+
1143
+ ```text
1144
+ 13
1145
+ ```
1146
+
1147
+ ---
1148
+
1149
+ ### What is the mean of the transformed **training** data (features only)
1150
+
1151
+ **Task**
1152
+
1153
+ What is the mean of the transformed **training** data (features only)?
1154
+
1155
+ Note : Compute the mean of the whole feature matrix i.e. mean of all values in the transformed training feature matrix
1156
+
1157
+ **Step-by-step approach**
1158
+
1159
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1160
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1161
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1162
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1163
+
1164
+ **Solution**
1165
+
1166
+ ```python
1167
+ # Solution
1168
+ print(X_train_t.mean())
1169
+ ```
1170
+
1171
+ **Recorded result**
1172
+
1173
+ ```text
1174
+ 0.28842989630040944
1175
+ ```
1176
+
1177
+ ---
1178
+
1179
+ ### Enter the value of $R^2$ score on the test dataset
1180
+
1181
+ **Task**
1182
+
1183
+ Enter the value of $R^2$ score on the test dataset.
1184
+
1185
+ **Step-by-step approach**
1186
+
1187
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1188
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1189
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1190
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1191
+
1192
+ **Solution**
1193
+
1194
+ ```python
1195
+ # Solution
1196
+ print(ridge.score(X_test, y_test))
1197
+ ```
1198
+
1199
+ **Recorded result**
1200
+
1201
+ ```text
1202
+ 0.6639511913668861
1203
+ ```
1204
+
1205
+ ---
1206
+
1207
+ ### What is the index of most important feature (highest absolute coefficient)
1208
+
1209
+ **Task**
1210
+
1211
+ What is the index of most important feature (highest absolute coefficient)? Note the index starts from 0. Ignore the intercept for this question.
1212
+
1213
+ **Step-by-step approach**
1214
+
1215
+ 1. Fit the requested estimator/selector on the training feature matrix only.
1216
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
1217
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
1218
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
1219
+
1220
+ **Solution**
1221
+
1222
+ ```python
1223
+ # Solution
1224
+ print(np.abs(ridge.coef_))
1225
+ print(np.argmax(np.abs(ridge.coef_)))
1226
+ ```
1227
+
1228
+ **Recorded result**
1229
+
1230
+ ```text
1231
+ [ 0.80599934 4.36935144 0.54731649 0.57136401 11.03253838 6.96010246
1232
+ 6.21337559 9.67022437 4.75155118 25.56498977 11.69237356 0.82743376
1233
+ 0.83498997 0.83498997]
1234
+ 9
1235
+ ```
1236
+
1237
+ ---
1238
+
1239
+ ### What is the index of least important feature (lowest absolute coefficient)
1240
+
1241
+ **Task**
1242
+
1243
+ What is the index of least important feature (lowest absolute coefficient)? Note the index starts from 0. Ignore the intercept for this question.
1244
+
1245
+ **Step-by-step approach**
1246
+
1247
+ 1. Fit the requested estimator/selector on the training feature matrix only.
1248
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
1249
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
1250
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
1251
+
1252
+ **Solution**
1253
+
1254
+ ```python
1255
+ # Solution
1256
+ print(np.argmin(np.abs(ridge.coef_)))
1257
+ ```
1258
+
1259
+ **Recorded result**
1260
+
1261
+ ```text
1262
+ 2
1263
+ ```
1264
+
1265
+ ---
1266
+
1267
+ ### What is the best **alpha**
1268
+
1269
+ **Task**
1270
+
1271
+ What is the best **alpha**?
1272
+
1273
+ > A 0.0001\
1274
+ > B 0.001\
1275
+ > C 0.01\
1276
+ > D 0.1\
1277
+ > E 1
1278
+
1279
+ **Step-by-step approach**
1280
+
1281
+ 1. Identify the input columns, target (if any), and exact output requested.
1282
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
1283
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
1284
+
1285
+ **Solution**
1286
+
1287
+ ```python
1288
+ # Solution
1289
+ print(sgd_grid.best_params_['alpha'])
1290
+ ```
1291
+
1292
+ **Recorded result**
1293
+
1294
+ ```text
1295
+ 0.0001
1296
+ ```
1297
+
1298
+ ---
1299
+
1300
+ ### What will be value of **root mean squared error** on the test dataset
1301
+
1302
+ **Task**
1303
+
1304
+ What will be value of **root mean squared error** on the test dataset?
1305
+
1306
+ **Step-by-step approach**
1307
+
1308
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1309
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1310
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1311
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1312
+
1313
+ **Solution**
1314
+
1315
+ ```python
1316
+ # Solution
1317
+ print(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))))
1318
+ ```
1319
+
1320
+ **Recorded result**
1321
+
1322
+ ```text
1323
+ 5.126302501903342
1324
+ ```
1325
+
1326
+ ---
1327
+
1328
+ ### If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset
1329
+
1330
+ **Task**
1331
+
1332
+ If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset?
1333
+
1334
+ **Step-by-step approach**
1335
+
1336
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1337
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1338
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1339
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1340
+
1341
+ **Solution**
1342
+
1343
+ ```python
1344
+ # Solution
1345
+ print(mean_absolute_error(y_test, pca_grid.predict(X_test)))
1346
+ ```
1347
+
1348
+ **Recorded result**
1349
+
1350
+ ```text
1351
+ 3.865036507995
1352
+ ```
1353
+
1354
+ ---
1355
+
1356
+ ### What **percentage** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model
1357
+
1358
+ **Task**
1359
+
1360
+ What **percentage** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model?
1361
+
1362
+ **Step-by-step approach**
1363
+
1364
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
1365
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
1366
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
1367
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
1368
+
1369
+ **Solution**
1370
+
1371
+ ```python
1372
+ # Solution
1373
+ print(pca_grid.best_estimator_.named_steps['pca'].explained_variance_ratio_[0] * 100)
1374
+ ```
1375
+
1376
+ **Recorded result**
1377
+
1378
+ ```text
1379
+ 69.9021494220279
1380
+ ```
1381
+
1382
+ ---
1383
+
1384
+ ### Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
1385
+
1386
+ **Task**
1387
+
1388
+ Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
1389
+ - For PolynomialFeatures:
1390
+ - interaction_only = False
1391
+ - degree = 2
1392
+ - For Lasso:
1393
+ - alpha = 0.5
1394
+ - warm_start = True
1395
+ - random state as 0
1396
+
1397
+ Fit the pipeline on the training dataset and find the **mean absolute error** on the test dataset.
1398
+
1399
+ **Step-by-step approach**
1400
+
1401
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1402
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1403
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1404
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1405
+
1406
+ **Solution**
1407
+
1408
+ ```python
1409
+ # Solution
1410
+ from sklearn.preprocessing import PolynomialFeatures
1411
+ poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=2, interaction_only=False)),
1412
+ ('lasso', Lasso(alpha=0.5, warm_start=True, random_state=0))])
1413
+ poly_pipe.fit(X_train, y_train)
1414
+ print(mean_absolute_error(y_test, poly_pipe.predict(X_test)))
1415
+ ```
1416
+
1417
+ **Recorded result**
1418
+
1419
+ ```text
1420
+ 4.55907406372981
1421
+ ```
1422
+
1423
+ ---
1424
+
1425
+ ### If you eliminate **2** features with recursive feature elimination, which features will be eliminated
1426
+
1427
+ **Task**
1428
+
1429
+ If you eliminate **2** features with recursive feature elimination, which features will be eliminated?
1430
+
1431
+ Type the indices of the eliminated features (index starts from 0).
1432
+
1433
+ Use LinearRegression model with default parameters as an estimator.
1434
+
1435
+ Use processed training data.
1436
+
1437
+ **Step-by-step approach**
1438
+
1439
+ 1. Fit the requested estimator/selector on the training feature matrix only.
1440
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
1441
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
1442
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
1443
+
1444
+ **Solution**
1445
+
1446
+ ```python
1447
+ # Solution
1448
+ from sklearn.feature_selection import RFE
1449
+ from sklearn.linear_model import LinearRegression
1450
+ rfe = RFE(LinearRegression(), n_features_to_select=X_train.shape[1] - 2)
1451
+ rfe.fit(X_train, y_train)
1452
+ print(np.where(~rfe.support_)[0])
1453
+ print(rfe.ranking_)
1454
+ ```
1455
+
1456
+ **Recorded result**
1457
+
1458
+ ```text
1459
+ [2 3]
1460
+ [1 1 3 2 1 1 1 1 1 1 1 1 1 1]
1461
+ ```
1462
+
1463
+ ---
1464
+
1465
+ ## Property listing records
1466
+
1467
+ ### What are the unique values present in the `Residential` feature of the dataset
1468
+
1469
+ **Task**
1470
+
1471
+ What are the unique values present in the `Residential` feature of the dataset?
1472
+ - [Single Family, Two Family, Three Family, Four Family]
1473
+ - [Detached House, Duplex, Triplex, Fourplex]
1474
+ - [Detached House, Duplex, Triplex]
1475
+ - [Bridgeport, Fairfield, Greenwich, Norwalk]
1476
+
1477
+ **Step-by-step approach**
1478
+
1479
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
1480
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
1481
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
1482
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
1483
+
1484
+ **Solution**
1485
+
1486
+ ```python
1487
+ # Solution
1488
+ print(df['Residential'].unique())
1489
+ ```
1490
+
1491
+ **Recorded result**
1492
+
1493
+ ```text
1494
+ ['Detached House' 'Duplex' 'Triplex' 'Fourplex']
1495
+ ```
1496
+
1497
+ ---
1498
+
1499
+ ### Which of the following columns have categorical data
1500
+
1501
+ **Task**
1502
+
1503
+ Which of the following columns have categorical data?
1504
+ - Year
1505
+ - Residential
1506
+ - carpet_area
1507
+ - Face
1508
+ - property_tax_rate
1509
+
1510
+ **Step-by-step approach**
1511
+
1512
+ 1. List categorical columns and inspect their unique values, including missing values.
1513
+ 2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
1514
+ 3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
1515
+ 4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
1516
+
1517
+ **Solution**
1518
+
1519
+ ```python
1520
+ # Solution
1521
+ print(df.dtypes)
1522
+ ```
1523
+
1524
+ **Recorded result**
1525
+
1526
+ ```text
1527
+ Date object
1528
+ Year int64
1529
+ Locality object
1530
+ Estimated Value float64
1531
+ Sale Price float64
1532
+ Property object
1533
+ Residential object
1534
+ num_rooms int64
1535
+ num_bathrooms int64
1536
+ carpet_area float64
1537
+ property_tax_rate float64
1538
+ Face object
1539
+ dtype: object
1540
+ ```
1541
+
1542
+ ---
1543
+
1544
+ ### Which of the following features have `missing`(NaN) or unknown ("
1545
+
1546
+ **Task**
1547
+
1548
+ Which of the following features have `missing`(NaN) or unknown ("?") values present in the dataset?
1549
+ - Date
1550
+ - Locality
1551
+ - carpet_area
1552
+ - Property
1553
+ - num_rooms
1554
+ - Face
1555
+
1556
+ **Step-by-step approach**
1557
+
1558
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1559
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1560
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1561
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1562
+
1563
+ **Solution**
1564
+
1565
+ ```python
1566
+ # Solution
1567
+ print(df.isna().sum())
1568
+ print((df == '?').sum())
1569
+ ```
1570
+
1571
+ **Recorded result**
1572
+
1573
+ ```text
1574
+ Date 0
1575
+ Year 0
1576
+ Locality 1285
1577
+ Estimated Value 1281
1578
+ Sale Price 0
1579
+ Property 0
1580
+ Residential 0
1581
+ num_rooms 0
1582
+ num_bathrooms 0
1583
+ carpet_area 1282
1584
+ property_tax_rate 0
1585
+ Face 0
1586
+ dtype: int64
1587
+ Date 0
1588
+ Year 0
1589
+ Locality 0
1590
+ Estimated Value 0
1591
+ Sale Price 0
1592
+ Property 1873
1593
+ Residential 0
1594
+ num_rooms 0
1595
+ num_bathrooms 0
1596
+ carpet_area 0
1597
+ property_tax_rate 0
1598
+ Face 0
1599
+ dtype: int64
1600
+ ```
1601
+
1602
+ ---
1603
+
1604
+ ### In the Year **2021**, how many houses (rows) located in the **Stamford** Locality have **2 or more** num_bathrooms, and are facing…
1605
+
1606
+ **Task**
1607
+
1608
+ In the Year **2021**, how many houses (rows) located in the **Stamford** Locality have **2 or more** num_bathrooms, and are facing towards either the **South or West**?
1609
+
1610
+ **Step-by-step approach**
1611
+
1612
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1613
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1614
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1615
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1616
+
1617
+ **Solution**
1618
+
1619
+ ```python
1620
+ # Solution
1621
+ mask = (df['Year'] == 2021) & (df['Locality'] == 'Stamford') & (df['num_bathrooms'] >= 2) & (df['Face'].isin(['South', 'West']))
1622
+ print(mask.sum())
1623
+ ```
1624
+
1625
+ **Recorded result**
1626
+
1627
+ ```text
1628
+ 44
1629
+ ```
1630
+
1631
+ ---
1632
+
1633
+ ### Split the dataset into train dataset and test dataset in the following manner
1634
+
1635
+ **Task**
1636
+
1637
+ Split the dataset into train dataset and test dataset in the following manner
1638
+ 1. data(rows) collected before the year of **2020** [2009-2019] should be the train dataset
1639
+ 2. data(rows) collected in the year of **2020, 2021 and 2022** (all inclusive) should be the test dataset
1640
+ 3. columns except of the label vector (`Sale Price`) should be the feature matrix (X_train or X_test)
1641
+ 4. make label vector (Y_train or y_test) containing values only from the target feature.
1642
+
1643
+ How many rows are in the feature matrix of the **train** dataset?
1644
+
1645
+ **Step-by-step approach**
1646
+
1647
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1648
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1649
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1650
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1651
+
1652
+ **Solution**
1653
+
1654
+ ```python
1655
+ # Solution
1656
+ train = df[df['Year'] <= 2019]
1657
+ test = df[df['Year'] >= 2020]
1658
+ X_train, y_train = train.drop(columns=['Sale Price']).copy(), train['Sale Price']
1659
+ X_test, y_test = test.drop(columns=['Sale Price']).copy(), test['Sale Price']
1660
+ print(X_train.shape, X_test.shape)
1661
+ ```
1662
+
1663
+ **Recorded result**
1664
+
1665
+ ```text
1666
+ (7560, 11) (2440, 11)
1667
+ ```
1668
+
1669
+ ---
1670
+
1671
+ ### Write the **MEDIAN** value of the `carpet_area` column of the train dataset you found to replace all the Missing Values (NaN)
1672
+
1673
+ **Task**
1674
+
1675
+ Write the **MEDIAN** value of the `carpet_area` column of the train dataset you found to replace all the Missing Values (NaN). (upto 2 digits after decimal points)
1676
+
1677
+ **Step-by-step approach**
1678
+
1679
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1680
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1681
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1682
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1683
+
1684
+ **Solution**
1685
+
1686
+ ```python
1687
+ # Solution
1688
+ print(round(carpet_median, 2))
1689
+ ```
1690
+
1691
+ **Recorded result**
1692
+
1693
+ ```text
1694
+ 1021.0
1695
+ ```
1696
+
1697
+ ---
1698
+
1699
+ ### What value did you use to replace the missing values (NaN) in the `Locality` column of the train dataset
1700
+
1701
+ **Task**
1702
+
1703
+ What value did you use to replace the missing values (NaN) in the `Locality` column of the train dataset?
1704
+ - Bridgeport
1705
+ - Waterbury
1706
+ - Fairfield
1707
+ - Stamford
1708
+
1709
+ **Step-by-step approach**
1710
+
1711
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1712
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1713
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1714
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1715
+
1716
+ **Solution**
1717
+
1718
+ ```python
1719
+ # Solution
1720
+ print(loc_mode)
1721
+ ```
1722
+
1723
+ **Recorded result**
1724
+
1725
+ ```text
1726
+ Bridgeport
1727
+ ```
1728
+
1729
+ ---
1730
+
1731
+ ### How many total features you got after one-hot encoding and scaling the training dataset
1732
+
1733
+ **Task**
1734
+
1735
+ How many total features you got after one-hot encoding and scaling the training dataset?
1736
+ - 10
1737
+ - 18
1738
+ - 25
1739
+ - 28
1740
+
1741
+ **Step-by-step approach**
1742
+
1743
+ 1. List categorical columns and inspect their unique values, including missing values.
1744
+ 2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
1745
+ 3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
1746
+ 4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
1747
+
1748
+ **Solution**
1749
+
1750
+ ```python
1751
+ # Solution
1752
+ print(X_train_t.shape[1])
1753
+ ```
1754
+
1755
+ **Recorded result**
1756
+
1757
+ ```text
1758
+ 25
1759
+ ```
1760
+
1761
+ ---
1762
+
1763
+ ### What is the **maximum** value present in the transformed **test** feature matrix
1764
+
1765
+ **Task**
1766
+
1767
+ What is the **maximum** value present in the transformed **test** feature matrix? (upto 2 digits after decimal points)
1768
+
1769
+ **Step-by-step approach**
1770
+
1771
+ 1. Identify the input columns, target (if any), and exact output requested.
1772
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
1773
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
1774
+
1775
+ **Solution**
1776
+
1777
+ ```python
1778
+ # Solution
1779
+ print(round(X_test_t.max(), 2))
1780
+ ```
1781
+
1782
+ **Recorded result**
1783
+
1784
+ ```text
1785
+ 13.16
1786
+ ```
1787
+
1788
+ ---
1789
+
1790
+ ### Apply LinearRegression on the train dataset(X_train and y_train)
1791
+
1792
+ **Task**
1793
+
1794
+ Apply LinearRegression on the train dataset(X_train and y_train). What is the **mean absolute error** on the test dataset(X_test and y_test)? (Upto 2 digits after decimal points)
1795
+
1796
+ **Step-by-step approach**
1797
+
1798
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1799
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1800
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1801
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1802
+
1803
+ **Solution**
1804
+
1805
+ ```python
1806
+ # Solution
1807
+ from sklearn.linear_model import LinearRegression
1808
+ from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
1809
+ lr = LinearRegression().fit(X_train, y_train)
1810
+ print(round(mean_absolute_error(y_test, lr.predict(X_test)), 2))
1811
+ ```
1812
+
1813
+ **Recorded result**
1814
+
1815
+ ```text
1816
+ 179131.9
1817
+ ```
1818
+
1819
+ ---
1820
+
1821
+ ### Enter the **minimum** value of $R^2$ score you got using the LinearRegression model when computed with 5 folds from the training…
1822
+
1823
+ **Task**
1824
+
1825
+ Enter the **minimum** value of $R^2$ score you got using the LinearRegression model when computed with 5 folds from the training dataset (X_train and y_train) using cross_val_score. (Upto 4 digits after decimal points)
1826
+
1827
+ **Step-by-step approach**
1828
+
1829
+ 1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
1830
+ 2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
1831
+ 3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
1832
+ 4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
1833
+
1834
+ **Solution**
1835
+
1836
+ ```python
1837
+ # Solution
1838
+ from sklearn.model_selection import cross_val_score
1839
+ scores = cross_val_score(LinearRegression(), X_train, y_train, cv=5)
1840
+ print(scores)
1841
+ print(round(scores.min(), 4))
1842
+ ```
1843
+
1844
+ **Recorded result**
1845
+
1846
+ ```text
1847
+ [0.61661573 0.79290189 0.72820221 0.82626052 0.64972083]
1848
+ 0.6166
1849
+ ```
1850
+
1851
+ ---
1852
+
1853
+ ### Train Ridge and Lasso with random_state=27 and keep other parameters as default using train dataset
1854
+
1855
+ **Task**
1856
+
1857
+ Train Ridge and Lasso with random_state=27 and keep other parameters as default using train dataset. Which one has the least **"mean absolute error"** for the test dataset?
1858
+
1859
+ - Ridge
1860
+ - Lasso
1861
+ - Both are Equal
1862
+
1863
+ **Step-by-step approach**
1864
+
1865
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1866
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1867
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1868
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1869
+
1870
+ **Solution**
1871
+
1872
+ ```python
1873
+ # Solution
1874
+ from sklearn.linear_model import Ridge, Lasso
1875
+ ridge = Ridge(random_state=27).fit(X_train, y_train)
1876
+ lasso = Lasso(random_state=27).fit(X_train, y_train)
1877
+ print('Ridge MAE:', mean_absolute_error(y_test, ridge.predict(X_test)))
1878
+ print('Lasso MAE:', mean_absolute_error(y_test, lasso.predict(X_test)))
1879
+ ```
1880
+
1881
+ **Recorded result**
1882
+
1883
+ ```text
1884
+ Ridge MAE: 179994.97462090437
1885
+ Lasso MAE: 179123.65380001903
1886
+ ```
1887
+
1888
+ ---
1889
+
1890
+ ### Train the `SGDRegressor(random_state=27, warm_start=True)` with maximum passes over the train dataset as **50**
1891
+
1892
+ **Task**
1893
+
1894
+ Train the `SGDRegressor(random_state=27, warm_start=True)` with maximum passes over the train dataset as **50**. Write the correct $R^2$ score for this estimator on test dataset (correct Upto 4 digits)
1895
+
1896
+ **Step-by-step approach**
1897
+
1898
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1899
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1900
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1901
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1902
+
1903
+ **Solution**
1904
+
1905
+ ```python
1906
+ # Solution
1907
+ from sklearn.linear_model import SGDRegressor
1908
+ sgd = SGDRegressor(random_state=27, warm_start=True, max_iter=50).fit(X_train, y_train)
1909
+ print(round(sgd.score(X_test, y_test), 4))
1910
+ ```
1911
+
1912
+ **Recorded result**
1913
+
1914
+ ```text
1915
+ 0.4676
1916
+ ```
1917
+
1918
+ ---
1919
+
1920
+ ### Mark the best alpha value you got using above instructions
1921
+
1922
+ **Task**
1923
+
1924
+ Mark the best alpha value you got using above instructions.
1925
+ - 1
1926
+ - 10
1927
+ - 100
1928
+ - 1000
1929
+
1930
+ **Step-by-step approach**
1931
+
1932
+ 1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
1933
+ 2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
1934
+ 3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
1935
+ 4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
1936
+
1937
+ **Solution**
1938
+
1939
+ ```python
1940
+ # Solution
1941
+ print(grid.best_params_['lasso__alpha'])
1942
+ ```
1943
+
1944
+ **Recorded result**
1945
+
1946
+ ```text
1947
+ 1000
1948
+ ```
1949
+
1950
+ ---
1951
+
1952
+ ### Enter the best polynomial degree value you got using above instructions
1953
+
1954
+ **Task**
1955
+
1956
+ Enter the best polynomial degree value you got using above instructions.
1957
+
1958
+ **Step-by-step approach**
1959
+
1960
+ 1. Identify the input columns, target (if any), and exact output requested.
1961
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
1962
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
1963
+
1964
+ **Solution**
1965
+
1966
+ ```python
1967
+ # Solution
1968
+ print(grid.best_params_['poly__degree'])
1969
+ ```
1970
+
1971
+ **Recorded result**
1972
+
1973
+ ```text
1974
+ 1
1975
+ ```
1976
+
1977
+ ---
1978
+
1979
+ ### What is the sum of explained_variance_ratio_
1980
+
1981
+ **Task**
1982
+
1983
+ What is the sum of explained_variance_ratio_ ? (upto two digits after the decimal)
1984
+
1985
+ **Step-by-step approach**
1986
+
1987
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
1988
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
1989
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
1990
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
1991
+
1992
+ **Solution**
1993
+
1994
+ ```python
1995
+ # Solution
1996
+ print(round(pca.explained_variance_ratio_.sum(), 2))
1997
+ ```
1998
+
1999
+ **Recorded result**
2000
+
2001
+ ```text
2002
+ 0.86
2003
+ ```
2004
+
2005
+ ---
2006
+
2007
+ ### Use PCA transformed training feature matrix from earlier question and training labels to fit the RidgeCV estimator with cv=5 and…
2008
+
2009
+ **Task**
2010
+
2011
+ Use PCA transformed training feature matrix from earlier question and training labels to fit the RidgeCV estimator with cv=5 and having alpha values as **[0.01, 0.1, 1, 10, 100, 1000]**. Calculate the $R^2$ score you got from the model for PCA transformed test dataset. (upto 4 digits after the decimal)
2012
+
2013
+ **Step-by-step approach**
2014
+
2015
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2016
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2017
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2018
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2019
+
2020
+ **Solution**
2021
+
2022
+ ```python
2023
+ # Solution
2024
+ from sklearn.linear_model import RidgeCV
2025
+ rcv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100, 1000], cv=5).fit(X_train_pca, y_train)
2026
+ print(round(rcv.score(X_test_pca, y_test), 4))
2027
+ ```
2028
+
2029
+ **Recorded result**
2030
+
2031
+ ```text
2032
+ 0.1789
2033
+ ```
2034
+
2035
+ ---
2036
+
2037
+ ### How many features of the train dataset have a P-VALUE **less than 0
2038
+
2039
+ **Task**
2040
+
2041
+ How many features of the train dataset have a P-VALUE **less than 0.05**?
2042
+
2043
+ **Step-by-step approach**
2044
+
2045
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2046
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2047
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2048
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2049
+
2050
+ **Solution**
2051
+
2052
+ ```python
2053
+ # Solution
2054
+ print(np.round(kb.pvalues_, 4))
2055
+ print((kb.pvalues_ < 0.05).sum())
2056
+ ```
2057
+
2058
+ **Recorded result**
2059
+
2060
+ ```text
2061
+ [0.000e+00 5.000e-04 0.000e+00 5.288e-01 9.124e-01 0.000e+00 0.000e+00
2062
+ 9.500e-03 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 4.000e-03
2063
+ 0.000e+00 6.493e-01 4.734e-01 5.820e-01 4.196e-01 1.600e-03 0.000e+00
2064
+ 0.000e+00 0.000e+00 0.000e+00 2.442e-01]
2065
+ 18
2066
+ ```
2067
+
2068
+ ---
2069
+
2070
+ ### What is the **minimum** F_STATISTICS score you got for any feature of the train dataset
2071
+
2072
+ **Task**
2073
+
2074
+ What is the **minimum** F_STATISTICS score you got for any feature of the train dataset? (upto 2 digits after the decimal)
2075
+
2076
+ **Step-by-step approach**
2077
+
2078
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2079
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2080
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2081
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2082
+
2083
+ **Solution**
2084
+
2085
+ ```python
2086
+ # Solution
2087
+ print(round(np.nanmin(kb.scores_), 2))
2088
+ ```
2089
+
2090
+ **Recorded result**
2091
+
2092
+ ```text
2093
+ 0.01
2094
+ ```
2095
+
2096
+ ---
2097
+
2098
+ ### What is the **mean absolute error** you got for this model using the test dataset
2099
+
2100
+ **Task**
2101
+
2102
+ What is the **mean absolute error** you got for this model using the test dataset? (upto 2 digits after the decimal)
2103
+
2104
+ **Step-by-step approach**
2105
+
2106
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2107
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2108
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2109
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2110
+
2111
+ **Solution**
2112
+
2113
+ ```python
2114
+ # Solution
2115
+ print(round(mean_absolute_error(y_test, kbest_pipe.predict(X_test)), 2))
2116
+ ```
2117
+
2118
+ **Recorded result**
2119
+
2120
+ ```text
2121
+ 170445.36
2122
+ ```
2123
+
2124
+ ---
2125
+
2126
+ ## Flight fare records
2127
+
2128
+ ### : What is the **median** of the flight ticket price
2129
+
2130
+ **Task**
2131
+
2132
+ : What is the **median** of the flight ticket price? Write your answer correct to two decimal places.
2133
+
2134
+ **Step-by-step approach**
2135
+
2136
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2137
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2138
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2139
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2140
+
2141
+ **Solution**
2142
+
2143
+ ```python
2144
+ # Solution
2145
+ print(round(df['Price'].median(), 2))
2146
+ ```
2147
+
2148
+ **Recorded result**
2149
+
2150
+ ```text
2151
+ 8366.0
2152
+ ```
2153
+
2154
+ ---
2155
+
2156
+ ### : During which month did the **lowest** number of flights occur
2157
+
2158
+ **Task**
2159
+
2160
+ : During which month did the **lowest** number of flights occur? Months are represented by numerical codes, with January corresponding to 1, February to 2, and so forth.
2161
+
2162
+ **Step-by-step approach**
2163
+
2164
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2165
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2166
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2167
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2168
+
2169
+ **Solution**
2170
+
2171
+ ```python
2172
+ # Solution
2173
+ print(df['Month'].value_counts())
2174
+ ```
2175
+
2176
+ **Recorded result**
2177
+
2178
+ ```text
2179
+ Month
2180
+ 5 3092
2181
+ 6 3044
2182
+ 3 2388
2183
+ 4 926
2184
+ Name: count, dtype: int64
2185
+ ```
2186
+
2187
+ ---
2188
+
2189
+ ### : Which `Source` city has the **highest average** flight ticket price
2190
+
2191
+ **Task**
2192
+
2193
+ : Which `Source` city has the **highest average** flight ticket price?
2194
+
2195
+ **Step-by-step approach**
2196
+
2197
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2198
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2199
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2200
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2201
+
2202
+ **Solution**
2203
+
2204
+ ```python
2205
+ # Solution
2206
+ print(df.groupby('Source')['Price'].mean().sort_values(ascending=False))
2207
+ ```
2208
+
2209
+ **Recorded result**
2210
+
2211
+ ```text
2212
+ Source
2213
+ Delhi 10637.617736
2214
+ Kolkata 9087.168003
2215
+ Banglore 7729.939275
2216
+ Mumbai 5148.631841
2217
+ Chennai 4695.082153
2218
+ Name: Price, dtype: float64
2219
+ ```
2220
+
2221
+ ---
2222
+
2223
+ ### : Two of the entries in the `'Additional_Info'` column are `'No info'` and `'No Info'`
2224
+
2225
+ **Task**
2226
+
2227
+ : Two of the entries in the `'Additional_Info'` column are `'No info'` and `'No Info'`. Replace all occurrences of `'No Info'` with `'No info'`. How many flights fall under airline **`'Jet Airways'`** and have `'No info'` as additional information?
2228
+
2229
+ #### **Note:** Please ensure that you make the changes directly within the dataset and continue to use that modified dataset for subsequent questions.
2230
+
2231
+ **Step-by-step approach**
2232
+
2233
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2234
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2235
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2236
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2237
+
2238
+ **Solution**
2239
+
2240
+ ```python
2241
+ # Solution
2242
+ df['Additional_Info'] = df['Additional_Info'].replace('No Info', 'No info')
2243
+ print(((df['Airline'] == 'Jet Airways') & (df['Additional_Info'] == 'No info')).sum())
2244
+ ```
2245
+
2246
+ **Recorded result**
2247
+
2248
+ ```text
2249
+ 2119
2250
+ ```
2251
+
2252
+ ---
2253
+
2254
+ ### : Convert the values of `'Duration'` into **minutes**
2255
+
2256
+ **Task**
2257
+
2258
+ : Convert the values of `'Duration'` into **minutes**. Enter the **median** duration (in minutes) of a flight.
2259
+
2260
+ #### **Note:** Please ensure that you make the changes directly within the dataset and continue to use that modified dataset for subsequent questions.
2261
+
2262
+ **Step-by-step approach**
2263
+
2264
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2265
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2266
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2267
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2268
+
2269
+ **Solution**
2270
+
2271
+ ```python
2272
+ # Solution
2273
+ def to_minutes(s):
2274
+ h = m = 0
2275
+ for part in s.split():
2276
+ if part.endswith('h'):
2277
+ h = int(part[:-1])
2278
+ elif part.endswith('m'):
2279
+ m = int(part[:-1])
2280
+ return h * 60 + m
2281
+ df['Duration'] = df['Duration'].apply(to_minutes)
2282
+ print(df['Duration'].median())
2283
+ ```
2284
+
2285
+ **Recorded result**
2286
+
2287
+ ```text
2288
+ 520.0
2289
+ ```
2290
+
2291
+ ---
2292
+
2293
+ ### : How many flights started in the **Evening** and arrived at the destination at **Night**
2294
+
2295
+ **Task**
2296
+
2297
+ : How many flights started in the **Evening** and arrived at the destination at **Night**?
2298
+
2299
+ **Step-by-step approach**
2300
+
2301
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2302
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2303
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2304
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2305
+
2306
+ **Solution**
2307
+
2308
+ ```python
2309
+ # Solution
2310
+ print(((df['Dep_Time'] == 'Evening') & (df['Arrival_Time'] == 'Night')).sum())
2311
+ ```
2312
+
2313
+ **Recorded result**
2314
+
2315
+ ```text
2316
+ 732
2317
+ ```
2318
+
2319
+ ---
2320
+
2321
+ ### : Encode the values of column `'WeekDay'` as follows:
2322
+
2323
+ **Task**
2324
+
2325
+ : Encode the values of column `'WeekDay'` as follows:
2326
+ # Weekends (Sunday, Saturday) = 1
2327
+ # all remaining five days = 0
2328
+
2329
+ # How many flights have WeekDay encoded as **1** (Weekends)?
2330
+
2331
+ **Step-by-step approach**
2332
+
2333
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2334
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2335
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2336
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2337
+
2338
+ **Solution**
2339
+
2340
+ ```python
2341
+ # Solution
2342
+ df['WeekDay'] = df['WeekDay'].isin(['Saturday', 'Sunday']).astype(int)
2343
+ print(df['WeekDay'].sum())
2344
+ ```
2345
+
2346
+ **Recorded result**
2347
+
2348
+ ```text
2349
+ 2799
2350
+ ```
2351
+
2352
+ ---
2353
+
2354
+ ### : Split the dataset into train dataset and test dataset in the following manner
2355
+
2356
+ **Task**
2357
+
2358
+ : Split the dataset into train dataset and test dataset in the following manner
2359
+
2360
+ - Use `train_test_split` to split the dataset into train and test dataset with test size equal to $0.25(25\%)$ and random_state equal to 42.
2361
+ - columns except of the label vector (`Price`) should be the feature matrix (X_train or X_test)
2362
+ - make label vector (Y_train or y_test) containing values only from the target feature (`Price`).
2363
+
2364
+ # How many rows are in the feature matrix of the **train** dataset?
2365
+
2366
+ **Step-by-step approach**
2367
+
2368
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2369
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2370
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2371
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2372
+
2373
+ **Solution**
2374
+
2375
+ ```python
2376
+ # Solution
2377
+ from sklearn.model_selection import train_test_split
2378
+ X = df.drop(columns=['Price'])
2379
+ y = df['Price']
2380
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
2381
+ print(X_train.shape, X_test.shape)
2382
+ ```
2383
+
2384
+ **Recorded result**
2385
+
2386
+ ```text
2387
+ (7087, 12) (2363, 12)
2388
+ ```
2389
+
2390
+ ---
2391
+
2392
+ ### : Were any unseen values encountered in the test dataset during the transformation process
2393
+
2394
+ **Task**
2395
+
2396
+ : Were any unseen values encountered in the test dataset during the transformation process?
2397
+
2398
+ **Step-by-step approach**
2399
+
2400
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2401
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2402
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2403
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2404
+
2405
+ **Solution**
2406
+
2407
+ ```python
2408
+ # Solution
2409
+ print((X_test_t == -1).sum())
2410
+ ```
2411
+
2412
+ **Recorded result**
2413
+
2414
+ ```text
2415
+ 1
2416
+ ```
2417
+
2418
+ ---
2419
+
2420
+ ### : After transforming the test data, all of its entries become numeric values
2421
+
2422
+ **Task**
2423
+
2424
+ : After transforming the test data, all of its entries become numeric values. What is the **highest** value present in the transformed **test** feature matrix?
2425
+
2426
+ **Step-by-step approach**
2427
+
2428
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2429
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2430
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2431
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2432
+
2433
+ **Solution**
2434
+
2435
+ ```python
2436
+ # Solution
2437
+ print(X_test_t.max())
2438
+ ```
2439
+
2440
+ **Recorded result**
2441
+
2442
+ ```text
2443
+ 27.0
2444
+ ```
2445
+
2446
+ ---
2447
+
2448
+ ### : After transforming the training data, all of its entries become numeric values
2449
+
2450
+ **Task**
2451
+
2452
+ : After transforming the training data, all of its entries become numeric values. What is the mean of the transformed **training** data matrix? Enter your answer correct to two decimal places.
2453
+
2454
+ **Step-by-step approach**
2455
+
2456
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2457
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2458
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2459
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2460
+
2461
+ **Solution**
2462
+
2463
+ ```python
2464
+ # Solution
2465
+ print(round(X_train_t.mean(), 2))
2466
+ ```
2467
+
2468
+ **Recorded result**
2469
+
2470
+ ```text
2471
+ 3.19
2472
+ ```
2473
+
2474
+ ---
2475
+
2476
+ ### : What is the **highest** value of the scaled 'Duration' column in the transformed **test** data
2477
+
2478
+ **Task**
2479
+
2480
+ : What is the **highest** value of the scaled 'Duration' column in the transformed **test** data? Enter your answer correct to two decimal places.
2481
+
2482
+ **Step-by-step approach**
2483
+
2484
+ 1. Separate numeric and categorical columns, and map each feature to its required operation.
2485
+ 2. Build a Pipeline for any feature needing several operations in sequence, such as imputation followed by scaling.
2486
+ 3. Pass transformer blocks to ColumnTransformer in the exact requested order; that order determines transformed column indices.
2487
+ 4. Fit the full transformer on X_train, then transform both train and test data and inspect shape, mean, minimum or maximum as requested.
2488
+
2489
+ **Solution**
2490
+
2491
+ ```python
2492
+ # Solution
2493
+ dur_idx = 7 # position of Duration in the ColumnTransformer output
2494
+ print(round(X_test_t[:, dur_idx].max(), 2))
2495
+ ```
2496
+
2497
+ **Recorded result**
2498
+
2499
+ ```text
2500
+ 0.82
2501
+ ```
2502
+
2503
+ ---
2504
+
2505
+ ### Train the ridge model on the training data with the following parameters:
2506
+
2507
+ **Task**
2508
+
2509
+ Train the ridge model on the training data with the following parameters:
2510
+ # `alpha = 1`
2511
+ # `solver = 'saga'`
2512
+ # `tol = 1e-3`
2513
+ # `random_state = 42`
2514
+
2515
+ # Enter the value of $R^2$ score on the test dataset.
2516
+
2517
+ **Step-by-step approach**
2518
+
2519
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2520
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2521
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2522
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2523
+
2524
+ **Solution**
2525
+
2526
+ ```python
2527
+ # Solution
2528
+ from sklearn.linear_model import Ridge
2529
+ from sklearn.metrics import mean_absolute_error, mean_squared_error
2530
+ ridge = Ridge(alpha=1, solver='saga', tol=1e-3, random_state=42).fit(X_train, y_train)
2531
+ print(ridge.score(X_test, y_test))
2532
+ ```
2533
+
2534
+ **Recorded result**
2535
+
2536
+ ```text
2537
+ 0.4134746439199958
2538
+ ```
2539
+
2540
+ ---
2541
+
2542
+ ### : What is the best **alpha**
2543
+
2544
+ **Task**
2545
+
2546
+ : What is the best **alpha**?
2547
+
2548
+ **Step-by-step approach**
2549
+
2550
+ 1. Identify the input columns, target (if any), and exact output requested.
2551
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
2552
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
2553
+
2554
+ **Solution**
2555
+
2556
+ ```python
2557
+ # Solution
2558
+ print(sgd_grid.best_params_['alpha'])
2559
+ ```
2560
+
2561
+ **Recorded result**
2562
+
2563
+ ```text
2564
+ 1
2565
+ ```
2566
+
2567
+ ---
2568
+
2569
+ ### : What will be value of **root mean squared error** on the test dataset
2570
+
2571
+ **Task**
2572
+
2573
+ : What will be value of **root mean squared error** on the test dataset? Enter your answer correct to two decimal places.
2574
+
2575
+ **Step-by-step approach**
2576
+
2577
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2578
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2579
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2580
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2581
+
2582
+ **Solution**
2583
+
2584
+ ```python
2585
+ # Solution
2586
+ print(round(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))), 2))
2587
+ ```
2588
+
2589
+ **Recorded result**
2590
+
2591
+ ```text
2592
+ 3580.16
2593
+ ```
2594
+
2595
+ ---
2596
+
2597
+ ### : If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset
2598
+
2599
+ **Task**
2600
+
2601
+ : If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset? Enter your answer correct to two decimal places.
2602
+
2603
+ **Step-by-step approach**
2604
+
2605
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2606
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2607
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2608
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2609
+
2610
+ **Solution**
2611
+
2612
+ ```python
2613
+ # Solution
2614
+ print(round(mean_absolute_error(y_test, pca_grid.predict(X_test)), 2))
2615
+ ```
2616
+
2617
+ **Recorded result**
2618
+
2619
+ ```text
2620
+ 3630.4
2621
+ ```
2622
+
2623
+ ---
2624
+
2625
+ ### : What **fraction** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model
2626
+
2627
+ **Task**
2628
+
2629
+ : What **fraction** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model? Enter your answer correct to two decimal places.
2630
+
2631
+ **Step-by-step approach**
2632
+
2633
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
2634
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
2635
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
2636
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
2637
+
2638
+ **Solution**
2639
+
2640
+ ```python
2641
+ # Solution
2642
+ print(round(pca_grid.best_estimator_.named_steps['pca'].explained_variance_ratio_[0], 2))
2643
+ ```
2644
+
2645
+ **Recorded result**
2646
+
2647
+ ```text
2648
+ 0.83
2649
+ ```
2650
+
2651
+ ---
2652
+
2653
+ ### : create a pipeline of the PolynomialFeatures() as transformer and Lasso as an estimator with the following parameters:
2654
+
2655
+ **Task**
2656
+
2657
+ : create a pipeline of the PolynomialFeatures() as transformer and Lasso as an estimator with the following parameters:
2658
+ - For PolynomialFeatures()
2659
+ - interaction_only = False
2660
+ - degree = 3
2661
+ - For lasso()
2662
+ - alpha = 50
2663
+ - warm_start = True
2664
+
2665
+ Fit the pipeline on the training dataset and find the $R^2$ score on the test dataset.
2666
+
2667
+ **Step-by-step approach**
2668
+
2669
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2670
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2671
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2672
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2673
+
2674
+ **Solution**
2675
+
2676
+ ```python
2677
+ # Solution
2678
+ from sklearn.preprocessing import PolynomialFeatures
2679
+ poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=3, interaction_only=False)),
2680
+ ('lasso', Lasso(alpha=50, warm_start=True))])
2681
+ poly_pipe.fit(X_train, y_train)
2682
+ print(poly_pipe.score(X_test, y_test))
2683
+ ```
2684
+
2685
+ **Recorded result**
2686
+
2687
+ ```text
2688
+ 0.5260776843902365
2689
+ ```
2690
+
2691
+ ---
2692
+
2693
+ ## Insurance premium records
2694
+
2695
+ ### Which of the following columns have **numerical** (int/float) datatype
2696
+
2697
+ **Task**
2698
+
2699
+ Which of the following columns have **numerical** (int/float) datatype?
2700
+
2701
+ **Step-by-step approach**
2702
+
2703
+ 1. Identify the input columns, target (if any), and exact output requested.
2704
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
2705
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
2706
+
2707
+ **Solution**
2708
+
2709
+ ```python
2710
+ # Solution
2711
+ print(df.dtypes)
2712
+ ```
2713
+
2714
+ **Recorded result**
2715
+
2716
+ ```text
2717
+ Age float64
2718
+ Gender object
2719
+ Annual Income float64
2720
+ Number of Dependents float64
2721
+ Education Level object
2722
+ Health Score float64
2723
+ Location object
2724
+ Policy Type object
2725
+ Active Policy days int64
2726
+ Previous Claims float64
2727
+ Credit Score float64
2728
+ Insurance Duration float64
2729
+ Customer Feedback object
2730
+ Smoking Status object
2731
+ Exercise Frequency object
2732
+ Property Type object
2733
+ Premium Amount float64
2734
+ dtype: object
2735
+ ```
2736
+
2737
+ ---
2738
+
2739
+ ### In this dataset, what is the most common **Policy Type** among customers **aged** 35 or below, living in **Suburban** locations and…
2740
+
2741
+ **Task**
2742
+
2743
+ In this dataset, what is the most common **Policy Type** among customers **aged** 35 or below, living in **Suburban** locations and having a **Credit Score** of 600 or above?
2744
+
2745
+ (A) Basic\
2746
+ (B) Comprehensive\
2747
+ (C) Premium\
2748
+ (D) Standard
2749
+
2750
+ V1: B\
2751
+ V2: A\
2752
+ V3: A
2753
+
2754
+ **Step-by-step approach**
2755
+
2756
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2757
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2758
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2759
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2760
+
2761
+ **Solution**
2762
+
2763
+ ```python
2764
+ # Solution
2765
+ sub = df[(df['Age'] <= 35) & (df['Location'] == 'Suburban') & (df['Credit Score'] >= 600)]
2766
+ print(sub['Policy Type'].value_counts())
2767
+ ```
2768
+
2769
+ **Recorded result**
2770
+
2771
+ ```text
2772
+ Policy Type
2773
+ Comprehensive 203
2774
+ Basic 200
2775
+ Premium 192
2776
+ Name: count, dtype: int64
2777
+ ```
2778
+
2779
+ ---
2780
+
2781
+ ### Which of the following options represent all the unique categories present in the **Exercise Frequency** feature
2782
+
2783
+ **Task**
2784
+
2785
+ Which of the following options represent all the unique categories present in the **Exercise Frequency** feature?
2786
+
2787
+ (A) ['Weekly', 'Rarely', 'Monthly', 'Daily', 'Never']
2788
+
2789
+ (B) ['Weekly', 'Monthly', 'Daily']
2790
+
2791
+ (C) ['Weekly', 'Rarely', 'Monthly', 'Daily']
2792
+
2793
+ (D) ['Regularly', 'Rarely', 'Monthly', 'Weekly']
2794
+
2795
+ (E) None of these
2796
+
2797
+ All Versions: C
2798
+
2799
+ **Step-by-step approach**
2800
+
2801
+ 1. Identify the input columns, target (if any), and exact output requested.
2802
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
2803
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
2804
+
2805
+ **Solution**
2806
+
2807
+ ```python
2808
+ # Solution
2809
+ print(df['Exercise Frequency'].unique())
2810
+ ```
2811
+
2812
+ **Recorded result**
2813
+
2814
+ ```text
2815
+ ['Rarely' 'Weekly' 'Daily' 'Monthly']
2816
+ ```
2817
+
2818
+ ---
2819
+
2820
+ ### Create feature matrix(X) and label vector(y) using following instructions:
2821
+
2822
+ **Task**
2823
+
2824
+ Create feature matrix(X) and label vector(y) using following instructions:
2825
+ * Premium Amount is the target column (y).
2826
+ * All the columns except the target column are part of the feature matrix (X).
2827
+
2828
+ Compare the correlation values for the following feature pairs of the feature matrix (X) and among the options select the one with the **strongest positive** correlation.
2829
+
2830
+ (A) `Annual Income` vs `Age`\
2831
+ (B) `Active Policy days` vs `Insurance Duration`\
2832
+ (C) `Previous Claims` vs `Credit Score`\
2833
+ (D) `Annual Income` vs `Credit Score`\
2834
+ (E) `Health Score` vs `Number of Dependents`
2835
+
2836
+ All Versions: C
2837
+
2838
+ **Step-by-step approach**
2839
+
2840
+ 1. Select only the numeric columns before calculating correlations; strings and categories should not be treated as continuous measurements.
2841
+ 2. Calculate the requested correlation matrix and ignore its diagonal, because every feature is perfectly correlated with itself.
2842
+ 3. For the most negative pair, sort the off-diagonal values in ascending order; draw the heatmap only when a visual is requested.
2843
+
2844
+ **Solution**
2845
+
2846
+ ```python
2847
+ # Solution
2848
+ X = df.drop(columns=['Premium Amount'])
2849
+ y = df['Premium Amount']
2850
+ corr = X.select_dtypes(include='number').corr()
2851
+ pairs = [('Annual Income', 'Age'), ('Active Policy days', 'Insurance Duration'),
2852
+ ('Previous Claims', 'Credit Score'), ('Annual Income', 'Credit Score'),
2853
+ ('Health Score', 'Number of Dependents')]
2854
+ for a, b in pairs:
2855
+ print(f'{a} vs {b}: {corr.loc[a, b]:.4f}')
2856
+ ```
2857
+
2858
+ **Recorded result**
2859
+
2860
+ ```text
2861
+ Annual Income vs Age: 0.0250
2862
+ Active Policy days vs Insurance Duration: -0.0285
2863
+ Previous Claims vs Credit Score: 0.0433
2864
+ Annual Income vs Credit Score: -0.2016
2865
+ Health Score vs Number of Dependents: 0.0046
2866
+ ```
2867
+
2868
+ ---
2869
+
2870
+ ### Which of the following columns have missing values
2871
+
2872
+ **Task**
2873
+
2874
+ Which of the following columns have missing values?
2875
+
2876
+ **Step-by-step approach**
2877
+
2878
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2879
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2880
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2881
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2882
+
2883
+ **Solution**
2884
+
2885
+ ```python
2886
+ # Solution
2887
+ print(df.isna().sum())
2888
+ ```
2889
+
2890
+ **Recorded result**
2891
+
2892
+ ```text
2893
+ Age 161
2894
+ Gender 0
2895
+ Annual Income 344
2896
+ Number of Dependents 884
2897
+ Education Level 0
2898
+ Health Score 616
2899
+ Location 0
2900
+ Policy Type 0
2901
+ Active Policy days 0
2902
+ Previous Claims 0
2903
+ Credit Score 1137
2904
+ Insurance Duration 0
2905
+ Customer Feedback 644
2906
+ Smoking Status 0
2907
+ Exercise Frequency 0
2908
+ Property Type 0
2909
+ Premium Amount 0
2910
+ dtype: int64
2911
+ ```
2912
+
2913
+ ---
2914
+
2915
+ ### Split the dataset into train dataset and test dataset in the following manner
2916
+
2917
+ **Task**
2918
+
2919
+ Split the dataset into train dataset and test dataset in the following manner.
2920
+
2921
+ - Use sklearn train_test_split function to split the data.
2922
+ - Use only **25%** data as test_set and keep random_state = 42
2923
+
2924
+ How many **test** labels (**y_test**) are considered outliers? Outliers are defined as values that exceed the upper bound (Third Quartile + 1.5 x Interquartile Range) or fall below the lower bound (First Quartile - 1.5 x Interquartile Range). Compute the quartiles on **y_test** itself.
2925
+
2926
+ V1: 117\
2927
+ V2: 113\
2928
+ V3: 126
2929
+
2930
+ **Step-by-step approach**
2931
+
2932
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2933
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2934
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2935
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2936
+
2937
+ **Solution**
2938
+
2939
+ ```python
2940
+ # Solution
2941
+ from sklearn.model_selection import train_test_split
2942
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
2943
+ q1, q3 = y_test.quantile(0.25), y_test.quantile(0.75)
2944
+ iqr = q3 - q1
2945
+ print(((y_test < q1 - 1.5 * iqr) | (y_test > q3 + 1.5 * iqr)).sum())
2946
+ ```
2947
+
2948
+ **Recorded result**
2949
+
2950
+ ```text
2951
+ 117
2952
+ ```
2953
+
2954
+ ---
2955
+
2956
+ ### Write the **MEAN** value of the **Age** column of the train dataset you found to replace all the Missing Values (NaN)
2957
+
2958
+ **Task**
2959
+
2960
+ Write the **MEAN** value of the **Age** column of the train dataset you found to replace all the Missing Values (NaN). (upto 1 digit after decimal point)
2961
+
2962
+ **Step-by-step approach**
2963
+
2964
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2965
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2966
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2967
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2968
+
2969
+ **Solution**
2970
+
2971
+ ```python
2972
+ # Solution
2973
+ print(round(fill_values['Age'], 1))
2974
+ ```
2975
+
2976
+ **Recorded result**
2977
+
2978
+ ```text
2979
+ 41.3
2980
+ ```
2981
+
2982
+ ---
2983
+
2984
+ ### Write the value you used to replace the missing values in the **Number of Dependents** feature of the train dataset
2985
+
2986
+ **Task**
2987
+
2988
+ Write the value you used to replace the missing values in the **Number of Dependents** feature of the train dataset
2989
+ (A) 0\
2990
+ (B) 1\
2991
+ (C) 2\
2992
+ (D) 3\
2993
+ (E) 4\
2994
+ (F) None of these
2995
+
2996
+ **Step-by-step approach**
2997
+
2998
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2999
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
3000
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
3001
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
3002
+
3003
+ **Solution**
3004
+
3005
+ ```python
3006
+ # Solution
3007
+ print(fill_values['Number of Dependents'])
3008
+ ```
3009
+
3010
+ **Recorded result**
3011
+
3012
+ ```text
3013
+ 3.0
3014
+ ```
3015
+
3016
+ ---
3017
+
3018
+ ### Calculate the sum of all the values present in the **first 50 rows** of the transformed test feature matrix
3019
+
3020
+ **Task**
3021
+
3022
+ Calculate the sum of all the values present in the **first 50 rows** of the transformed test feature matrix ? (upto 2 digits after the decimal)
3023
+
3024
+ ${\text{V1 : 405.34 (Range: 403, 408)}}$\
3025
+ ${\text{V2 : 399.38 (Range: 397, 402)}}$\
3026
+ ${\text{V3 : 419.33 (Range: 417, 422)}}$
3027
+
3028
+ **Step-by-step approach**
3029
+
3030
+ 1. Identify the input columns, target (if any), and exact output requested.
3031
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
3032
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
3033
+
3034
+ **Solution**
3035
+
3036
+ ```python
3037
+ # Solution
3038
+ print(round(X_test_t[:50].sum(), 2))
3039
+ ```
3040
+
3041
+ **Recorded result**
3042
+
3043
+ ```text
3044
+ 405.34
3045
+ ```
3046
+
3047
+ ---
3048
+
3049
+ ### Enter the value of **Mean Absolute Error** on the test dataset
3050
+
3051
+ **Task**
3052
+
3053
+ Enter the value of **Mean Absolute Error** on the test dataset.
3054
+
3055
+ **Step-by-step approach**
3056
+
3057
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3058
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3059
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3060
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3061
+
3062
+ **Solution**
3063
+
3064
+ ```python
3065
+ # Solution
3066
+ print(mean_absolute_error(y_test, ridge.predict(X_test)))
3067
+ ```
3068
+
3069
+ **Recorded result**
3070
+
3071
+ ```text
3072
+ 666.013227845856
3073
+ ```
3074
+
3075
+ ---
3076
+
3077
+ ### What is the **highest absolute coefficient value** of the trained Ridge model
3078
+
3079
+ **Task**
3080
+
3081
+ What is the **highest absolute coefficient value** of the trained Ridge model? (Ignore Intercept) (upto 2 digits after the decimal)
3082
+
3083
+ **Step-by-step approach**
3084
+
3085
+ 1. Fit the requested estimator/selector on the training feature matrix only.
3086
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
3087
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
3088
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
3089
+
3090
+ **Solution**
3091
+
3092
+ ```python
3093
+ # Solution
3094
+ print(np.round(np.abs(ridge.coef_), 3))
3095
+ print(round(np.abs(ridge.coef_).max(), 2))
3096
+ ```
3097
+
3098
+ **Recorded result**
3099
+
3100
+ ```text
3101
+ [ 9.474 4.219 3.804 7.92 13.654 27.433 13.221 4.76 8.723 6.265
3102
+ 6.321 0.711 1.628 1.783 18.109 11.759 0.085 13.013 36.555 9.772]
3103
+ 36.55
3104
+ ```
3105
+
3106
+ ---
3107
+
3108
+ ### Identify the index of the feature with the lowest absolute coefficient value
3109
+
3110
+ **Task**
3111
+
3112
+ Identify the index of the feature with the lowest absolute coefficient value. Note that index starts from 0. (Ignore Intercept)
3113
+
3114
+ **Step-by-step approach**
3115
+
3116
+ 1. Fit the requested estimator/selector on the training feature matrix only.
3117
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
3118
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
3119
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
3120
+
3121
+ **Solution**
3122
+
3123
+ ```python
3124
+ # Solution
3125
+ print(np.argmin(np.abs(ridge.coef_)))
3126
+ ```
3127
+
3128
+ **Recorded result**
3129
+
3130
+ ```text
3131
+ 16
3132
+ ```
3133
+
3134
+ ---
3135
+
3136
+ ### What is the best **alpha**
3137
+
3138
+ **Task**
3139
+
3140
+ What is the best **alpha**?
3141
+
3142
+ **Step-by-step approach**
3143
+
3144
+ 1. Identify the input columns, target (if any), and exact output requested.
3145
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
3146
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
3147
+
3148
+ **Solution**
3149
+
3150
+ ```python
3151
+ # Solution
3152
+ print(sgd_grid.best_params_['alpha'])
3153
+ ```
3154
+
3155
+ **Recorded result**
3156
+
3157
+ ```text
3158
+ 0.1
3159
+ ```
3160
+
3161
+ ---
3162
+
3163
+ ### What will be value of **root mean squared error** on the test dataset
3164
+
3165
+ **Task**
3166
+
3167
+ What will be value of **root mean squared error** on the test dataset?
3168
+
3169
+ **Step-by-step approach**
3170
+
3171
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3172
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3173
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3174
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3175
+
3176
+ **Solution**
3177
+
3178
+ ```python
3179
+ # Solution
3180
+ print(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))))
3181
+ ```
3182
+
3183
+ **Recorded result**
3184
+
3185
+ ```text
3186
+ 872.3850595099008
3187
+ ```
3188
+
3189
+ ---
3190
+
3191
+ ### If we fit the pipeline on the training dataset, what will be the **Mean Absolute Error** on the test dataset
3192
+
3193
+ **Task**
3194
+
3195
+ If we fit the pipeline on the training dataset, what will be the **Mean Absolute Error** on the test dataset?
3196
+
3197
+ **Step-by-step approach**
3198
+
3199
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3200
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3201
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3202
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3203
+
3204
+ **Solution**
3205
+
3206
+ ```python
3207
+ # Solution
3208
+ print(mean_absolute_error(y_test, pca_grid.predict(X_test)))
3209
+ ```
3210
+
3211
+ **Recorded result**
3212
+
3213
+ ```text
3214
+ 665.4809335203206
3215
+ ```
3216
+
3217
+ ---
3218
+
3219
+ ### How many principal components are retained by the PCA step of the **best** pipeline found above
3220
+
3221
+ **Task**
3222
+
3223
+ How many principal components are retained by the PCA step of the **best** pipeline found above?
3224
+
3225
+ **Step-by-step approach**
3226
+
3227
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
3228
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
3229
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
3230
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
3231
+
3232
+ **Solution**
3233
+
3234
+ ```python
3235
+ # Solution
3236
+ print(pca_grid.best_estimator_.named_steps['pca'].n_components_)
3237
+ ```
3238
+
3239
+ **Recorded result**
3240
+
3241
+ ```text
3242
+ 15
3243
+ ```
3244
+
3245
+ ---
3246
+
3247
+ ### Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
3248
+
3249
+ **Task**
3250
+
3251
+ Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
3252
+ - For PolynomialFeatures:
3253
+ - interaction_only = False
3254
+ - degree = 2
3255
+ - For Lasso:
3256
+ - alpha = 2
3257
+ - warm_start = True
3258
+ - random state as 0
3259
+
3260
+ Fit the pipeline on the training dataset and find the **Root Mean Square Error** on the test dataset.
3261
+
3262
+ **Step-by-step approach**
3263
+
3264
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3265
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3266
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3267
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3268
+
3269
+ **Solution**
3270
+
3271
+ ```python
3272
+ # Solution
3273
+ from sklearn.preprocessing import PolynomialFeatures
3274
+ poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=2, interaction_only=False)),
3275
+ ('lasso', Lasso(alpha=2, warm_start=True, random_state=0))])
3276
+ poly_pipe.fit(X_train, y_train)
3277
+ print(np.sqrt(mean_squared_error(y_test, poly_pipe.predict(X_test))))
3278
+ ```
3279
+
3280
+ **Recorded result**
3281
+
3282
+ ```text
3283
+ 872.5174524584622
3284
+ ```
3285
+
3286
+ ---
3287
+
3288
+ ### Create a pipeline for feature selection using f_regression as transformer and SGDRegressor as an estimator with the following…
3289
+
3290
+ **Task**
3291
+
3292
+ Create a pipeline for feature selection using f_regression as transformer and SGDRegressor as an estimator with the following parameters:
3293
+ - For feature selection:
3294
+ - select best **4** features based on f_regression score
3295
+ - For SGDRegressor:
3296
+ - random state as 42
3297
+
3298
+ Fit the pipeline on the training dataset and select the option that correctly displays the selected feature names in descending order based on their f_regression scores.
3299
+
3300
+ (A) ['Previous Claims', 'Location_Urban', 'Annual Income', 'Active Policy days']\
3301
+ (B) ['Previous Claims', 'Annual Income', 'Property Type_House', 'Customer Feedback_Good']\
3302
+ (C) ['Previous Claims', 'Policy Type_Comprehensive', 'Annual Income', 'Health Score'] \
3303
+ (D) ['Previous Claims', 'Annual Income', 'Location_Urban', 'Education Level'] \
3304
+ (E) None of the Above
3305
+
3306
+ **Step-by-step approach**
3307
+
3308
+ 1. Fit the requested estimator/selector on the training feature matrix only.
3309
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
3310
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
3311
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
3312
+
3313
+ **Solution**
3314
+
3315
+ ```python
3316
+ # Solution
3317
+ from sklearn.feature_selection import SelectKBest, f_regression
3318
+ fs_pipe = Pipeline([('kbest', SelectKBest(f_regression, k=4)), ('sgd', SGDRegressor(random_state=42))])
3319
+ fs_pipe.fit(X_train, y_train)
3320
+ kb = fs_pipe.named_steps['kbest']
3321
+ top = np.argsort(kb.scores_)[::-1][:4]
3322
+ print(list(X_train.columns[top]))
3323
+ ```
3324
+
3325
+ **Recorded result**
3326
+
3327
+ ```text
3328
+ ['Previous Claims', 'Location_Urban', 'Annual Income', 'Active Policy days']
3329
+ ```
3330
+
3331
+ ---
3332
+
3333
+ ### For SGDRegressor, what is the actual number of iterations completed during model training
3334
+
3335
+ **Task**
3336
+
3337
+ For SGDRegressor, what is the actual number of iterations completed during model training?
3338
+
3339
+ **Step-by-step approach**
3340
+
3341
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
3342
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
3343
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
3344
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
3345
+
3346
+ **Solution**
3347
+
3348
+ ```python
3349
+ # Solution
3350
+ print(fs_pipe.named_steps['sgd'].n_iter_)
3351
+ ```
3352
+
3353
+ **Recorded result**
3354
+
3355
+ ```text
3356
+ 16
3357
+ ```
3358
+
3359
+ ---
3360
+
3361
+ ## Residential valuation reference
3362
+
3363
+ ### Load a CSV with a final target column
3364
+
3365
+ **Task**
3366
+
3367
+ Load a CSV whose first row contains column names, with no index column. The final column is the target and all preceding columns are features.
3368
+
3369
+ **Step-by-step approach**
3370
+
3371
+ 1. Read the file with pd.read_csv and do not pass index_col=0 when the file has no index column.
3372
+ 2. Inspect df.head(), df.shape and df.columns to confirm the header was read as column names.
3373
+ 3. Use iloc to split by position: every column except the last becomes X and the last becomes y.
3374
+
3375
+ **Solution**
3376
+
3377
+ ```python
3378
+ df_model = pd.read_csv("model_data.csv")
3379
+ print(df_model.head())
3380
+ X_model = df_model.iloc[:, :-1]
3381
+ y_model = df_model.iloc[:, -1]
3382
+ print(X_model.shape, y_model.shape)
3383
+ ```
3384
+
3385
+ ---
3386
+
3387
+ ### Count dataset samples
3388
+
3389
+ **Task**
3390
+
3391
+ How many samples are there in the dataset?
3392
+
3393
+ **Step-by-step approach**
3394
+
3395
+ 1. Read the row dimension, not the number of columns.
3396
+ 2. Use shape[0]; shape[1] would count features.
3397
+
3398
+ **Solution**
3399
+
3400
+ ```python
3401
+ print(df.shape[0])
3402
+ ```
3403
+
3404
+ **Recorded result**
3405
+
3406
+ ```text
3407
+ 4000
3408
+ ```
3409
+
3410
+ ---
3411
+
3412
+ ### Compute the mean target value
3413
+
3414
+ **Task**
3415
+
3416
+ What is the average house price (in lacs)?
3417
+
3418
+ **Step-by-step approach**
3419
+
3420
+ 1. Select the target series.
3421
+ 2. Use mean(); pandas skips genuine NaN values by default.
3422
+
3423
+ **Solution**
3424
+
3425
+ ```python
3426
+ print(df["PRICE"].mean())
3427
+ ```
3428
+
3429
+ **Recorded result**
3430
+
3431
+ ```text
3432
+ 24.355923220694248
3433
+ ```
3434
+
3435
+ ---
3436
+
3437
+ ### Count records above a room threshold
3438
+
3439
+ **Task**
3440
+
3441
+ How many houses have 5 or more rooms?
3442
+
3443
+ **Step-by-step approach**
3444
+
3445
+ 1. Compare the room-count series with the inclusive threshold.
3446
+ 2. Sum the boolean mask to count True records.
3447
+
3448
+ **Solution**
3449
+
3450
+ ```python
3451
+ print(int(df["RM"].ge(5).sum()))
3452
+ ```
3453
+
3454
+ **Recorded result**
3455
+
3456
+ ```text
3457
+ 3953
3458
+ ```
3459
+
3460
+ ---
3461
+
3462
+ ### Average of the ten highest target values
3463
+
3464
+ **Task**
3465
+
3466
+ What is the average price of the top 10 most expensive houses (in lacs)?
3467
+
3468
+ **Step-by-step approach**
3469
+
3470
+ 1. Select exactly the ten largest target values with nlargest(10).
3471
+ 2. Calculate the mean of that selected series.
3472
+
3473
+ **Solution**
3474
+
3475
+ ```python
3476
+ print(df["PRICE"].nlargest(10).mean())
3477
+ ```
3478
+
3479
+ **Recorded result**
3480
+
3481
+ ```text
3482
+ 52.36590175716407
3483
+ ```
3484
+
3485
+ ---
3486
+
3487
+ ### Detect invalid room counts
3488
+
3489
+ **Task**
3490
+
3491
+ What is the total number of missing or unknown values in the number of rooms feature?
3492
+
3493
+ **Step-by-step approach**
3494
+
3495
+ 1. Inspect unique values before declaring an invalid code.
3496
+ 2. A room count below one is implausible in this dataset; the recorded sentinel is -1.
3497
+ 3. Count the mask before replacing it with NaN.
3498
+
3499
+ **Solution**
3500
+
3501
+ ```python
3502
+ bad_rooms = df["RM"].lt(1)
3503
+ print(int(bad_rooms.sum()))
3504
+ ```
3505
+
3506
+ **Recorded result**
3507
+
3508
+ ```text
3509
+ 40
3510
+ ```
3511
+
3512
+ ---
3513
+
3514
+ ### Detect invalid property ages
3515
+
3516
+ **Task**
3517
+
3518
+ What is the total number of missing or unknown values in the age feature?
3519
+
3520
+ **Step-by-step approach**
3521
+
3522
+ 1. Inspect the minimum and unique age values.
3523
+ 2. Negative ages are invalid; create a mask with lt(0).
3524
+ 3. Count the mask before conversion to NaN.
3525
+
3526
+ **Solution**
3527
+
3528
+ ```python
3529
+ bad_age = df["AGE"].lt(0)
3530
+ print(int(bad_age.sum()))
3531
+ ```
3532
+
3533
+ **Recorded result**
3534
+
3535
+ ```text
3536
+ 50
3537
+ ```
3538
+
3539
+ ---
3540
+
3541
+ ### Detect unknown riverside values
3542
+
3543
+ **Task**
3544
+
3545
+ What is the total number of missing or unknown values in the RIVERSIDE feature?
3546
+
3547
+ **Step-by-step approach**
3548
+
3549
+ 1. Inspect value_counts(dropna=False) to discover the non-binary label.
3550
+ 2. The recorded unknown label is UNKNOWN.
3551
+ 3. Count it, then replace it with NaN before imputation.
3552
+
3553
+ **Solution**
3554
+
3555
+ ```python
3556
+ bad_riverside = df["RIVERSIDE"].eq("UNKNOWN")
3557
+ print(int(bad_riverside.sum()))
3558
+ ```
3559
+
3560
+ **Recorded result**
3561
+
3562
+ ```text
3563
+ 88
3564
+ ```
3565
+
3566
+ ---
3567
+
3568
+ ### Count recent riverside houses
3569
+
3570
+ **Task**
3571
+
3572
+ How many houses are on riverside and were built within the last 50 years? Ignore rows missing either riverside status or age.
3573
+
3574
+ **Step-by-step approach**
3575
+
3576
+ 1. Create a validity mask requiring both columns to be present.
3577
+ 2. Combine the conditions: RIVERSIDE is YES and AGE is at most 50.
3578
+ 3. Sum the combined boolean mask.
3579
+
3580
+ **Solution**
3581
+
3582
+ ```python
3583
+ valid = df["RIVERSIDE"].notna() & df["AGE"].notna()
3584
+ recent_riverside = valid & df["RIVERSIDE"].eq("YES") & df["AGE"].le(50)
3585
+ print(int(recent_riverside.sum()))
3586
+ ```
3587
+
3588
+ **Recorded result**
3589
+
3590
+ ```text
3591
+ 44
3592
+ ```
3593
+
3594
+ ---
3595
+
3596
+ ### Count a highway-count interval
3597
+
3598
+ **Task**
3599
+
3600
+ How many houses are near to exactly 6, 7 or 8 highways?
3601
+
3602
+ **Step-by-step approach**
3603
+
3604
+ 1. The interval includes both 6 and 8.
3605
+ 2. Use between(6, 8, inclusive="both") and sum the mask.
3606
+
3607
+ **Solution**
3608
+
3609
+ ```python
3610
+ print(int(df["HIGHWAYCOUNT"].between(6, 8, inclusive="both").sum()))
3611
+ ```
3612
+
3613
+ **Recorded result**
3614
+
3615
+ ```text
3616
+ 1211
3617
+ ```
3618
+
3619
+ ---
3620
+
3621
+ ### Create price categories
3622
+
3623
+ **Task**
3624
+
3625
+ Create CATEGORY using five price bands and find the category with the most records.
3626
+
3627
+ **Step-by-step approach**
3628
+
3629
+ 1. Define bin edges so each range is left-closed and right-open.
3630
+ 2. Use pd.cut with right=False to implement the stated boundaries.
3631
+ 3. Count categories and choose the category with the largest count.
3632
+
3633
+ **Solution**
3634
+
3635
+ ```python
3636
+ bins = [-np.inf, 10, 20, 30, 40, np.inf]
3637
+ labels = [1, 2, 3, 4, 5]
3638
+ df["CATEGORY"] = pd.cut(df["PRICE"], bins=bins, labels=labels, right=False)
3639
+ counts = df["CATEGORY"].value_counts(sort=False)
3640
+ print(counts)
3641
+ print("Most frequent category:", counts.idxmax())
3642
+ ```
3643
+
3644
+ **Recorded result**
3645
+
3646
+ ```text
3647
+ 3
3648
+ ```
3649
+
3650
+ ---
3651
+
3652
+ ### Clean sentinels and split housing data
3653
+
3654
+ **Task**
3655
+
3656
+ Replace invalid RM, AGE and RIVERSIDE values with NaN, use PRICE as target, and create a 70:30 split with random_state=0. Find the training sample count.
3657
+
3658
+ **Step-by-step approach**
3659
+
3660
+ 1. Create masks from the observed sentinel rules: RM below 1, AGE below 0, and RIVERSIDE equal to UNKNOWN.
3661
+ 2. Replace those entries with np.nan on a copy of the data.
3662
+ 3. Separate PRICE from the feature matrix, then split with test_size=0.30 and random_state=0.
3663
+ 4. The training set retains 70% of 4000 rows.
3664
+
3665
+ **Solution**
3666
+
3667
+ ```python
3668
+ data = df.copy()
3669
+ rm_bad = data["RM"].lt(1)
3670
+ age_bad = data["AGE"].lt(0)
3671
+ riv_bad = data["RIVERSIDE"].eq("UNKNOWN")
3672
+ data.loc[rm_bad, "RM"] = np.nan
3673
+ data.loc[age_bad, "AGE"] = np.nan
3674
+ data.loc[riv_bad, "RIVERSIDE"] = np.nan
3675
+ X = data.drop(columns=["PRICE", "CATEGORY"], errors="ignore")
3676
+ y = data["PRICE"]
3677
+ X_train, X_test, y_train, y_test = train_test_split(
3678
+ X, y, test_size=0.30, random_state=0
3679
+ )
3680
+ print(X_train.shape[0])
3681
+ ```
3682
+
3683
+ **Recorded result**
3684
+
3685
+ ```text
3686
+ 2800
3687
+ ```
3688
+
3689
+ ---
3690
+
3691
+ ### Build an ordered heterogeneous preprocessor
3692
+
3693
+ **Task**
3694
+
3695
+ Apply the specified min-max scaling, standard scaling, train-fitted imputation and one-hot encoding in the stated feature order; find the transformed feature count.
3696
+
3697
+ **Step-by-step approach**
3698
+
3699
+ 1. Drop the derived CATEGORY column before model preprocessing.
3700
+ 2. Create one transformer block per ordered feature group; a Pipeline performs imputation before scaling.
3701
+ 3. Fit ColumnTransformer on X_train and only transform X_test.
3702
+ 4. OneHotEncoder creates two output columns for the two known riverside categories; verify the final matrix width.
3703
+
3704
+ **Solution**
3705
+
3706
+ ```python
3707
+ X_train = X_train.drop(columns=["CATEGORY"], errors="ignore")
3708
+ X_test = X_test.drop(columns=["CATEGORY"], errors="ignore")
3709
+ preprocessor = ColumnTransformer([
3710
+ ("crim", MinMaxScaler(), ["CRIM"]),
3711
+ ("zn", MinMaxScaler(), ["ZN"]),
3712
+ ("indus", StandardScaler(), ["INDUS"]),
3713
+ ("polindex", MinMaxScaler(), ["POLINDEX"]),
3714
+ ("dis", MinMaxScaler(), ["DIS"]),
3715
+ ("highways", MinMaxScaler(), ["HIGHWAYCOUNT"]),
3716
+ ("tax", MinMaxScaler(), ["TAX"]),
3717
+ ("ptratio", MinMaxScaler(), ["PTRATIO"]),
3718
+ ("imm", MinMaxScaler(), ["IMM"]),
3719
+ ("bpl", MinMaxScaler(), ["BPL"]),
3720
+ ("rooms", Pipeline([
3721
+ ("imputer", SimpleImputer(strategy="median")),
3722
+ ("scaler", MinMaxScaler()),
3723
+ ]), ["RM"]),
3724
+ ("age", Pipeline([
3725
+ ("imputer", SimpleImputer(strategy="mean")),
3726
+ ("scaler", MinMaxScaler()),
3727
+ ]), ["AGE"]),
3728
+ ("riverside", Pipeline([
3729
+ ("imputer", SimpleImputer(strategy="most_frequent")),
3730
+ ("encoder", OneHotEncoder(handle_unknown="ignore", sparse_output=False)),
3731
+ ]), ["RIVERSIDE"]),
3732
+ ], remainder="drop")
3733
+ X_train_t = preprocessor.fit_transform(X_train)
3734
+ X_test_t = preprocessor.transform(X_test)
3735
+ print(X_train_t.shape[1])
3736
+ ```
3737
+
3738
+ **Recorded result**
3739
+
3740
+ ```text
3741
+ 14
3742
+ ```
3743
+
3744
+ ---
3745
+
3746
+ ### Compute the global mean of transformed test data
3747
+
3748
+ **Task**
3749
+
3750
+ What is the mean of the transformed test data, using all matrix entries?
3751
+
3752
+ **Step-by-step approach**
3753
+
3754
+ 1. Use the test matrix after applying the fitted training preprocessor.
3755
+ 2. Calling mean() on the NumPy matrix with no axis computes one scalar across all rows and columns.
3756
+
3757
+ **Solution**
3758
+
3759
+ ```python
3760
+ print(X_test_t.mean())
3761
+ ```
3762
+
3763
+ **Recorded result**
3764
+
3765
+ ```text
3766
+ 0.38617456550975626
3767
+ ```
3768
+
3769
+ ---
3770
+
3771
+ ### Split model data and fit Ridge
3772
+
3773
+ **Task**
3774
+
3775
+ Use the last column as y, split with test_size=0.30 and random_state=42, then fit Ridge(alpha=10, solver="saga", tol=1e-4, random_state=42). Find test R².
3776
+
3777
+ **Step-by-step approach**
3778
+
3779
+ 1. Treat every column except the last as X and the last as y.
3780
+ 2. Use the exact random split settings.
3781
+ 3. Fit Ridge with the fixed hyperparameters, then call score on the test data.
3782
+
3783
+ **Solution**
3784
+
3785
+ ```python
3786
+ X_model = df_model.iloc[:, :-1]
3787
+ y_model = df_model.iloc[:, -1]
3788
+ X_train, X_test, y_train, y_test = train_test_split(
3789
+ X_model, y_model, test_size=0.30, random_state=42
3790
+ )
3791
+ ridge = Ridge(alpha=10, solver="saga", tol=1e-4, random_state=42)
3792
+ ridge.fit(X_train, y_train)
3793
+ print(ridge.score(X_test, y_test))
3794
+ ```
3795
+
3796
+ **Recorded result**
3797
+
3798
+ ```text
3799
+ 0.6613547575262211
3800
+ ```
3801
+
3802
+ ---
3803
+
3804
+ ### Find the most important Ridge coefficient
3805
+
3806
+ **Task**
3807
+
3808
+ What is the zero-based index of the feature with the highest absolute coefficient, ignoring the intercept?
3809
+
3810
+ **Step-by-step approach**
3811
+
3812
+ 1. Read coef_; intercept_ is separate and must not be included.
3813
+ 2. Take absolute values because importance is based on magnitude, not sign.
3814
+ 3. Use argmax to obtain the zero-based position.
3815
+
3816
+ **Solution**
3817
+
3818
+ ```python
3819
+ most_important_index = int(np.argmax(np.abs(ridge.coef_)))
3820
+ print(most_important_index)
3821
+ ```
3822
+
3823
+ **Recorded result**
3824
+
3825
+ ```text
3826
+ 9
3827
+ ```
3828
+
3829
+ ---
3830
+
3831
+ ### Find the least important Ridge coefficient
3832
+
3833
+ **Task**
3834
+
3835
+ What is the zero-based index of the feature with the lowest absolute coefficient, ignoring the intercept?
3836
+
3837
+ **Step-by-step approach**
3838
+
3839
+ 1. Read the fitted coefficient vector only.
3840
+ 2. Take absolute values and use argmin to find the smallest magnitude.
3841
+
3842
+ **Solution**
3843
+
3844
+ ```python
3845
+ least_important_index = int(np.argmin(np.abs(ridge.coef_)))
3846
+ print(least_important_index)
3847
+ ```
3848
+
3849
+ **Recorded result**
3850
+
3851
+ ```text
3852
+ 0
3853
+ ```
3854
+
3855
+ ---
3856
+
3857
+ ### Tune SGDRegressor with GridSearchCV
3858
+
3859
+ **Task**
3860
+
3861
+ Tune penalty, alpha and tol for SGDRegressor(random_state=42), using five folds and negative mean absolute error. Find the best penalty.
3862
+
3863
+ **Step-by-step approach**
3864
+
3865
+ 1. Use a parameter grid containing every specified candidate.
3866
+ 2. Set cv=5 and scoring="neg_mean_absolute_error" exactly.
3867
+ 3. Fit the grid on training data and inspect best_params_.
3868
+
3869
+ **Solution**
3870
+
3871
+ ```python
3872
+ sgd_grid = GridSearchCV(
3873
+ SGDRegressor(random_state=42),
3874
+ {
3875
+ "penalty": ["l1", "l2"],
3876
+ "alpha": [1e-5, 1e-4, 1e-3, 1e-2, 1e-1],
3877
+ "tol": [1e-4, 1e-3, 1e-2, 1e-1],
3878
+ },
3879
+ cv=5, scoring="neg_mean_absolute_error", n_jobs=-1,
3880
+ )
3881
+ sgd_grid.fit(X_train, y_train)
3882
+ print(sgd_grid.best_params_["penalty"])
3883
+ ```
3884
+
3885
+ **Recorded result**
3886
+
3887
+ ```text
3888
+ l2
3889
+ ```
3890
+
3891
+ ---
3892
+
3893
+ ### Evaluate the tuned SGDRegressor
3894
+
3895
+ **Task**
3896
+
3897
+ What is the mean absolute error on the held-out test dataset using the best SGD model?
3898
+
3899
+ **Step-by-step approach**
3900
+
3901
+ 1. GridSearchCV refits the best estimator on all training rows by default.
3902
+ 2. Predict the test target and calculate positive MAE.
3903
+
3904
+ **Solution**
3905
+
3906
+ ```python
3907
+ y_pred = sgd_grid.predict(X_test)
3908
+ print(mean_absolute_error(y_test, y_pred))
3909
+ ```
3910
+
3911
+ **Recorded result**
3912
+
3913
+ ```text
3914
+ 3.8131121797994014
3915
+ ```
3916
+
3917
+ ---
3918
+
3919
+ ### Tune a PCA and Lasso pipeline
3920
+
3921
+ **Task**
3922
+
3923
+ Build PCA followed by Lasso and tune n_components and alpha with five-fold negative-MAE GridSearchCV. Evaluate the best pipeline with test R².
3924
+
3925
+ **Step-by-step approach**
3926
+
3927
+ 1. Put PCA and Lasso in a Pipeline so PCA is refitted inside each CV fold.
3928
+ 2. Use the step-prefixed keys pca__n_components and lasso__alpha.
3929
+ 3. Fit GridSearchCV on training data, then score the best refitted pipeline on X_test.
3930
+
3931
+ **Solution**
3932
+
3933
+ ```python
3934
+ pca_lasso = Pipeline([("pca", PCA()), ("lasso", Lasso())])
3935
+ pca_grid = GridSearchCV(
3936
+ pca_lasso,
3937
+ {"pca__n_components": [0.9, 0.95], "lasso__alpha": [10, 1, 0.01, 0.001]},
3938
+ cv=5, scoring="neg_mean_absolute_error", n_jobs=-1,
3939
+ )
3940
+ pca_grid.fit(X_train, y_train)
3941
+ print(pca_grid.best_estimator_.score(X_test, y_test))
3942
+ ```
3943
+
3944
+ **Recorded result**
3945
+
3946
+ ```text
3947
+ 0.6288625430197575
3948
+ ```
3949
+
3950
+ ---
3951
+
3952
+ ### Read the first PCA eigenvalue
3953
+
3954
+ **Task**
3955
+
3956
+ How much variance (eigenvalue) is explained by the first principal component of the best PCA pipeline?
3957
+
3958
+ **Step-by-step approach**
3959
+
3960
+ 1. Access the fitted PCA object from best_estimator_.named_steps.
3961
+ 2. Use explained_variance_[0] for the eigenvalue. Do not substitute explained_variance_ratio_[0], which is a fraction.
3962
+
3963
+ **Solution**
3964
+
3965
+ ```python
3966
+ best_pca = pca_grid.best_estimator_.named_steps["pca"]
3967
+ print(best_pca.explained_variance_[0])
3968
+ ```
3969
+
3970
+ **Recorded result**
3971
+
3972
+ ```text
3973
+ 1.1635075742239045
3974
+ ```
3975
+
3976
+ ---
3977
+
3978
+ ### Fit polynomial features with Lasso
3979
+
3980
+ **Task**
3981
+
3982
+ Build PolynomialFeatures(degree=2, interaction_only=False) followed by Lasso(alpha=1, warm_start=True, random_state=0). Find test R².
3983
+
3984
+ **Step-by-step approach**
3985
+
3986
+ 1. Create the transformer and estimator with the exact stated parameters.
3987
+ 2. Place them in a Pipeline so prediction applies the same polynomial expansion.
3988
+ 3. Fit the pipeline on training data and evaluate with score on test data.
3989
+
3990
+ **Solution**
3991
+
3992
+ ```python
3993
+ poly_lasso = Pipeline([
3994
+ ("poly", PolynomialFeatures(degree=2, interaction_only=False)),
3995
+ ("lasso", Lasso(alpha=1, warm_start=True, random_state=0)),
3996
+ ])
3997
+ poly_lasso.fit(X_train, y_train)
3998
+ print(poly_lasso.score(X_test, y_test))
3999
+ ```
4000
+
4001
+ **Recorded result**
4002
+
4003
+ ```text
4004
+ 0.157678032410551
4005
+ ```
4006
+
4007
+ ---
4008
+
4009
+ ### Remove one feature with recursive elimination
4010
+
4011
+ **Task**
4012
+
4013
+ Use RFE with default LinearRegression to eliminate one feature from the processed training matrix. Find the eliminated zero-based index.
4014
+
4015
+ **Step-by-step approach**
4016
+
4017
+ 1. Set n_features_to_select to the input feature count minus one.
4018
+ 2. Fit RFE on the training features and labels.
4019
+ 3. support_ is True for retained features; its logical inverse identifies eliminated positions.
4020
+
4021
+ **Solution**
4022
+
4023
+ ```python
4024
+ rfe = RFE(
4025
+ estimator=LinearRegression(),
4026
+ n_features_to_select=X_train.shape[1] - 1,
4027
+ )
4028
+ rfe.fit(X_train, y_train)
4029
+ print(np.flatnonzero(~rfe.support_))
4030
+ ```
4031
+
4032
+ **Recorded result**
4033
+
4034
+ ```text
4035
+ 2
4036
+ ```
4037
+
4038
+ ---