scikit-doc 0.1.0__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,3355 @@
1
+ # Notebook-derived examples
2
+
3
+ **Total: 104 examples**
4
+
5
+ ## Customer value records
6
+
7
+ ### How many records (rows) are there in the dataset
8
+
9
+ **Task**
10
+
11
+ How many records (rows) are there in the dataset?
12
+
13
+ **Step-by-step approach**
14
+
15
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
16
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
17
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
18
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
19
+
20
+ **Solution**
21
+
22
+ ```python
23
+ # Solution
24
+ print(df.shape[0])
25
+ ```
26
+
27
+ **Recorded result**
28
+
29
+ ```text
30
+ 6257
31
+ ```
32
+
33
+ ---
34
+
35
+ ### What are the unique values of the feature `qualification` in the dataset
36
+
37
+ **Task**
38
+
39
+ What are the unique values of the feature `qualification` in the dataset?
40
+
41
+ **Step-by-step approach**
42
+
43
+ 1. Identify the input columns, target (if any), and exact output requested.
44
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
45
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
46
+
47
+ **Solution**
48
+
49
+ ```python
50
+ # Solution
51
+ print(df['qualification'].unique())
52
+ ```
53
+
54
+ **Recorded result**
55
+
56
+ ```text
57
+ ['High School' 'Bachelor' 'Others']
58
+ ```
59
+
60
+ ---
61
+
62
+ ### Which of the following columns have numerical (int/float) data
63
+
64
+ **Task**
65
+
66
+ Which of the following columns have numerical (int/float) data?
67
+
68
+ **Step-by-step approach**
69
+
70
+ 1. Identify the input columns, target (if any), and exact output requested.
71
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
72
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
73
+
74
+ **Solution**
75
+
76
+ ```python
77
+ # Solution
78
+ print(df.dtypes)
79
+ ```
80
+
81
+ **Recorded result**
82
+
83
+ ```text
84
+ id int64
85
+ gender object
86
+ area object
87
+ qualification object
88
+ income float64
89
+ marital_status int64
90
+ vintage int64
91
+ claim_amount float64
92
+ num_policies object
93
+ policy object
94
+ type_of_policy object
95
+ cltv int64
96
+ dtype: object
97
+ ```
98
+
99
+ ---
100
+
101
+ ### Plot the `heatmap` of the correlation matrix of the numerical columns and mark the pair which has the **lowest (most negative)**…
102
+
103
+ **Task**
104
+
105
+ Plot the `heatmap` of the correlation matrix of the numerical columns and mark the pair which has the **lowest (most negative)** correlation value.
106
+
107
+ **Step-by-step approach**
108
+
109
+ 1. Select only the numeric columns before calculating correlations; strings and categories should not be treated as continuous measurements.
110
+ 2. Calculate the requested correlation matrix and ignore its diagonal, because every feature is perfectly correlated with itself.
111
+ 3. For the most negative pair, sort the off-diagonal values in ascending order; draw the heatmap only when a visual is requested.
112
+
113
+ **Solution**
114
+
115
+ ```python
116
+ # Solution
117
+ corr = df.select_dtypes(include='number').corr()
118
+ plt.figure(figsize=(8, 6))
119
+ sns.heatmap(corr, annot=True, fmt='.3f', cmap='coolwarm')
120
+ plt.show()
121
+ # Locate the most negative off-diagonal pair
122
+ pairs = corr.where(~np.eye(len(corr), dtype=bool)).stack()
123
+ print(pairs.sort_values().head(4))
124
+ ```
125
+
126
+ ---
127
+
128
+ ### Which of the following features has the **highest number** of `missing` values
129
+
130
+ **Task**
131
+
132
+ Which of the following features has the **highest number** of `missing` values?
133
+
134
+ **Step-by-step approach**
135
+
136
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
137
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
138
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
139
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
140
+
141
+ **Solution**
142
+
143
+ ```python
144
+ # Solution
145
+ print(df.isna().sum().sort_values(ascending=False))
146
+ ```
147
+
148
+ **Recorded result**
149
+
150
+ ```text
151
+ income 401
152
+ area 384
153
+ policy 372
154
+ claim_amount 360
155
+ qualification 0
156
+ gender 0
157
+ id 0
158
+ marital_status 0
159
+ vintage 0
160
+ num_policies 0
161
+ type_of_policy 0
162
+ cltv 0
163
+ dtype: int64
164
+ ```
165
+
166
+ ---
167
+
168
+ ### Break the dataset into features(`X`) and label (`y`), where the column `cltv` goes to `y` and the rest of the columns go to `X`
169
+
170
+ **Task**
171
+
172
+ Break the dataset into features(`X`) and label (`y`), where the column `cltv` goes to `y` and the rest of the columns go to `X`. Enter the **median** value of `cltv` column?
173
+
174
+ **Step-by-step approach**
175
+
176
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
177
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
178
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
179
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
180
+
181
+ **Solution**
182
+
183
+ ```python
184
+ # Solution
185
+ X = df.drop(columns=['cltv'])
186
+ y = df['cltv']
187
+ print(y.median())
188
+ ```
189
+
190
+ **Recorded result**
191
+
192
+ ```text
193
+ 66288.0
194
+ ```
195
+
196
+ ---
197
+
198
+ ### Split the dataset into training and test dataset using `train_test_split` into `75:25` ratio while keeping random_state = 42
199
+
200
+ **Task**
201
+
202
+ Split the dataset into training and test dataset using `train_test_split` into `75:25` ratio while keeping random_state = 42. What is the shape of the test set (X_test)?
203
+
204
+ **Step-by-step approach**
205
+
206
+ 1. Identify the target column and remove it from the feature matrix before splitting.
207
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
208
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
209
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
210
+
211
+ **Solution**
212
+
213
+ ```python
214
+ # Solution
215
+ from sklearn.model_selection import train_test_split
216
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
217
+ print(X_test.shape)
218
+ ```
219
+
220
+ **Recorded result**
221
+
222
+ ```text
223
+ (1565, 11)
224
+ ```
225
+
226
+ ---
227
+
228
+ ### Drop(remove) `id` column from train and test data because it is not useful in model training
229
+
230
+ **Task**
231
+
232
+ Drop(remove) `id` column from train and test data because it is not useful in model training. Now how many of the remaining feature columns in the training dataset are of **object (string)** type?
233
+
234
+ **Step-by-step approach**
235
+
236
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
237
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
238
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
239
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
240
+
241
+ **Solution**
242
+
243
+ ```python
244
+ # Solution
245
+ X_train = X_train.drop(columns=['id'])
246
+ X_test = X_test.drop(columns=['id'])
247
+ obj_cols = X_train.select_dtypes(include='object').columns
248
+ print(list(obj_cols))
249
+ print(len(obj_cols))
250
+ ```
251
+
252
+ **Recorded result**
253
+
254
+ ```text
255
+ ['gender', 'area', 'qualification', 'num_policies', 'policy', 'type_of_policy']
256
+ 6
257
+ ```
258
+
259
+ ---
260
+
261
+ ### Compute and write the **mean** of the `income` column of X_train while ignoring the missing values
262
+
263
+ **Task**
264
+
265
+ Compute and write the **mean** of the `income` column of X_train while ignoring the missing values. Replace all NaN values in the income column of X_train and X_test by the mean computed from the X_train (upto two decimal).
266
+
267
+ **Step-by-step approach**
268
+
269
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
270
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
271
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
272
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
273
+
274
+ **Solution**
275
+
276
+ ```python
277
+ # Solution
278
+ income_mean = X_train['income'].mean()
279
+ print(round(income_mean, 2))
280
+ X_train['income'] = X_train['income'].fillna(income_mean)
281
+ X_test['income'] = X_test['income'].fillna(income_mean)
282
+ ```
283
+
284
+ **Recorded result**
285
+
286
+ ```text
287
+ 13.73
288
+ ```
289
+
290
+ ---
291
+
292
+ ### How many missing values are present in the `policy` column of **X_test**
293
+
294
+ **Task**
295
+
296
+ How many missing values are present in the `policy` column of **X_test**? After answering, replace all NaN values in the `policy` column of X_train and X_test by the most frequent value in X_train.
297
+
298
+ **Step-by-step approach**
299
+
300
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
301
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
302
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
303
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
304
+
305
+ **Solution**
306
+
307
+ ```python
308
+ # Solution
309
+ print(X_test['policy'].isna().sum())
310
+ policy_mode = X_train['policy'].mode()[0]
311
+ print('Most frequent policy in X_train:', policy_mode)
312
+ X_train['policy'] = X_train['policy'].fillna(policy_mode)
313
+ X_test['policy'] = X_test['policy'].fillna(policy_mode)
314
+ ```
315
+
316
+ **Recorded result**
317
+
318
+ ```text
319
+ 102
320
+ Most frequent policy in X_train: A
321
+ ```
322
+
323
+ ---
324
+
325
+ ### Replace all NaN values in the `area` column of X_train and X_test by the most frequent value from X_train
326
+
327
+ **Task**
328
+
329
+ Replace all NaN values in the `area` column of X_train and X_test by the most frequent value from X_train. After replacing, how many records in **X_train** have `area` equal to that most frequent value?
330
+
331
+ **Step-by-step approach**
332
+
333
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
334
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
335
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
336
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
337
+
338
+ **Solution**
339
+
340
+ ```python
341
+ # Solution
342
+ area_mode = X_train['area'].mode()[0]
343
+ print('Most frequent area in X_train:', area_mode)
344
+ X_train['area'] = X_train['area'].fillna(area_mode)
345
+ X_test['area'] = X_test['area'].fillna(area_mode)
346
+ print((X_train['area'] == area_mode).sum())
347
+ ```
348
+
349
+ **Recorded result**
350
+
351
+ ```text
352
+ Most frequent area in X_train: Urban
353
+ 3407
354
+ ```
355
+
356
+ ---
357
+
358
+ ### Replace all NaN values in the claim_amount column of X_train and X_test by the **median** of claim_amount computed from X_train
359
+
360
+ **Task**
361
+
362
+ Replace all NaN values in the claim_amount column of X_train and X_test by the **median** of claim_amount computed from X_train. After replacing, what is the **mean** of the claim_amount column in **X_test**? (correct upto two decimal places)
363
+
364
+ **Step-by-step approach**
365
+
366
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
367
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
368
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
369
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
370
+
371
+ **Solution**
372
+
373
+ ```python
374
+ # Solution
375
+ claim_median = X_train['claim_amount'].median()
376
+ X_train['claim_amount'] = X_train['claim_amount'].fillna(claim_median)
377
+ X_test['claim_amount'] = X_test['claim_amount'].fillna(claim_median)
378
+ print(round(X_test['claim_amount'].mean(), 2))
379
+ ```
380
+
381
+ **Recorded result**
382
+
383
+ ```text
384
+ 4260.88
385
+ ```
386
+
387
+ ---
388
+
389
+ ### Apply `StandardScaler` on the `vintage` column of X_train (store the result in a separate variable, do **not** overwrite X_train)
390
+
391
+ **Task**
392
+
393
+ Apply `StandardScaler` on the `vintage` column of X_train (store the result in a separate variable, do **not** overwrite X_train). Compute and write the **maximum** value of the scaled `vintage` column. (correct upto 2 decimal)
394
+
395
+ **Step-by-step approach**
396
+
397
+ 1. Separate numeric and categorical columns, and map each feature to its required operation.
398
+ 2. Build a Pipeline for any feature needing several operations in sequence, such as imputation followed by scaling.
399
+ 3. Pass transformer blocks to ColumnTransformer in the exact requested order; that order determines transformed column indices.
400
+ 4. Fit the full transformer on X_train, then transform both train and test data and inspect shape, mean, minimum or maximum as requested.
401
+
402
+ **Solution**
403
+
404
+ ```python
405
+ # Solution
406
+ from sklearn.preprocessing import StandardScaler
407
+ vintage_scaled = StandardScaler().fit_transform(X_train[['vintage']])
408
+ print(round(vintage_scaled.max(), 2))
409
+ ```
410
+
411
+ **Recorded result**
412
+
413
+ ```text
414
+ 1.47
415
+ ```
416
+
417
+ ---
418
+
419
+ ### How many columns are produced by the **One-Hot Encoding step alone** (i
420
+
421
+ **Task**
422
+
423
+ How many columns are produced by the **One-Hot Encoding step alone** (i.e. the number of columns in $X_1$)?
424
+
425
+ [Options]
426
+
427
+ **Step-by-step approach**
428
+
429
+ 1. List categorical columns and inspect their unique values, including missing values.
430
+ 2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
431
+ 3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
432
+ 4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
433
+
434
+ **Solution**
435
+
436
+ ```python
437
+ # Solution
438
+ print(ct.named_transformers_['ohe'].get_feature_names_out().shape[0])
439
+ ```
440
+
441
+ **Recorded result**
442
+
443
+ ```text
444
+ 17
445
+ ```
446
+
447
+ ---
448
+
449
+ ### Apply `SequentialFeatureSelector` transformer with direction= 'forward' with `LinearRegression()` estimator and select **6**…
450
+
451
+ **Task**
452
+
453
+ Apply `SequentialFeatureSelector` transformer with direction= 'forward' with `LinearRegression()` estimator and select **6** features by fitting to the X_train and y_train.
454
+
455
+ `Use cv = KFold(n_splits=5,random_state=42,shuffle=True) in SequentialFeatureSelector.`
456
+
457
+ ### Which of the following options represents the correct integer index of the selected features list?
458
+
459
+ **Step-by-step approach**
460
+
461
+ 1. Fit the requested estimator/selector on the training feature matrix only.
462
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
463
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
464
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
465
+
466
+ **Solution**
467
+
468
+ ```python
469
+ # Solution
470
+ from sklearn.feature_selection import SequentialFeatureSelector
471
+ from sklearn.linear_model import LinearRegression
472
+ from sklearn.model_selection import KFold
473
+ cv = KFold(n_splits=5, random_state=42, shuffle=True)
474
+ sfs = SequentialFeatureSelector(LinearRegression(), n_features_to_select=6, direction='forward', cv=cv)
475
+ sfs.fit(X_train, y_train)
476
+ print(np.where(sfs.get_support())[0])
477
+ ```
478
+
479
+ **Recorded result**
480
+
481
+ ```text
482
+ [ 4 5 7 9 13 19]
483
+ ```
484
+
485
+ ---
486
+
487
+ ### Apply `LinearRegression` on the training set(`X_train` and `y_train`)
488
+
489
+ **Task**
490
+
491
+ Apply `LinearRegression` on the training set(`X_train` and `y_train`). What is the **mean absolute error** on the test set(`X_test` and `y_test`)? (Upto 2 digits after decimal point)
492
+
493
+ **Step-by-step approach**
494
+
495
+ 1. Identify the target column and remove it from the feature matrix before splitting.
496
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
497
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
498
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
499
+
500
+ **Solution**
501
+
502
+ ```python
503
+ # Solution
504
+ from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
505
+ lr = LinearRegression().fit(X_train, y_train)
506
+ print(round(mean_absolute_error(y_test, lr.predict(X_test)), 2))
507
+ ```
508
+
509
+ **Recorded result**
510
+
511
+ ```text
512
+ 52712.56
513
+ ```
514
+
515
+ ---
516
+
517
+ ### Using the `LinearRegression` model, compute the `cross-validation scores` for `5 splits` on training data (X_train and y_train)…
518
+
519
+ **Task**
520
+
521
+ Using the `LinearRegression` model, compute the `cross-validation scores` for `5 splits` on training data (X_train and y_train) using `cross_val_score`. Enter the **mean** value of the $R^2$ scores obtained. (Upto 4 digits after decimal point)
522
+
523
+ `Use cv = KFold(n_splits=5,random_state=42,shuffle=True) in cross_val_score.`
524
+
525
+ (**Hint**: By default cross_val_score uses LinearRegression's scoring metric, which is $R^2$ score.)
526
+
527
+ **Step-by-step approach**
528
+
529
+ 1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
530
+ 2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
531
+ 3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
532
+ 4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
533
+
534
+ **Solution**
535
+
536
+ ```python
537
+ # Solution
538
+ from sklearn.model_selection import cross_val_score
539
+ scores = cross_val_score(LinearRegression(), X_train, y_train, cv=KFold(n_splits=5, random_state=42, shuffle=True))
540
+ print(scores)
541
+ print(round(scores.mean(), 4))
542
+ ```
543
+
544
+ **Recorded result**
545
+
546
+ ```text
547
+ [0.15765982 0.14449678 0.14566491 0.13373985 0.15322638]
548
+ 0.147
549
+ ```
550
+
551
+ ---
552
+
553
+ ### Apply `Ridge` regression with **random_state=42** and **alpha=10** on training set(`X_train and y_train`) and calculate the $R^2$…
554
+
555
+ **Task**
556
+
557
+ Apply `Ridge` regression with **random_state=42** and **alpha=10** on training set(`X_train and y_train`) and calculate the $R^2$ score on test_set (`X_test and y_test`). What is the correct score (Upto 4 digits after decimal point)?
558
+
559
+ **Step-by-step approach**
560
+
561
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
562
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
563
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
564
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
565
+
566
+ **Solution**
567
+
568
+ ```python
569
+ # Solution
570
+ from sklearn.linear_model import Ridge
571
+ ridge = Ridge(alpha=10, random_state=42).fit(X_train, y_train)
572
+ print(round(ridge.score(X_test, y_test), 4))
573
+ ```
574
+
575
+ **Recorded result**
576
+
577
+ ```text
578
+ 0.1421
579
+ ```
580
+
581
+ ---
582
+
583
+ ### Apply `Lasso` regression with **random_state=42** and **regularization rate (alpha)=100** on the training data(`X_train & y_train`)
584
+
585
+ **Task**
586
+
587
+ Apply `Lasso` regression with **random_state=42** and **regularization rate (alpha)=100** on the training data(`X_train & y_train`). Enter the value of the **intercept** correct upto 2 digits after decimal point.
588
+
589
+ **Step-by-step approach**
590
+
591
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
592
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
593
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
594
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
595
+
596
+ **Solution**
597
+
598
+ ```python
599
+ # Solution
600
+ from sklearn.linear_model import Lasso
601
+ lasso = Lasso(alpha=100, random_state=42).fit(X_train, y_train)
602
+ print(round(lasso.intercept_, 2))
603
+ ```
604
+
605
+ **Recorded result**
606
+
607
+ ```text
608
+ 116658.43
609
+ ```
610
+
611
+ ---
612
+
613
+ ### Fit SGDRegressor(`random_state=42`) estimator on the training data(`X_train & y_train`) and predict labels for test_data(`X_test`),…
614
+
615
+ **Task**
616
+
617
+ Fit SGDRegressor(`random_state=42`) estimator on the training data(`X_train & y_train`) and predict labels for test_data(`X_test`), lets call it as y_test_predict. The parameters are initialized with default values. Calculate the **root mean squared error** between y_test and y_test_predict. (Correct upto two decimals)
618
+
619
+ **Step-by-step approach**
620
+
621
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
622
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
623
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
624
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
625
+
626
+ **Solution**
627
+
628
+ ```python
629
+ # Solution
630
+ from sklearn.linear_model import SGDRegressor
631
+ sgd = SGDRegressor(random_state=42).fit(X_train, y_train)
632
+ y_test_predict = sgd.predict(X_test)
633
+ print(round(np.sqrt(mean_squared_error(y_test, y_test_predict)), 2))
634
+ ```
635
+
636
+ **Recorded result**
637
+
638
+ ```text
639
+ 88459.51
640
+ ```
641
+
642
+ ---
643
+
644
+ ### Using SGDRegressor(random_state=42) as an estimator for exactly **25** iterations (`max_iter=25`)
645
+
646
+ **Task**
647
+
648
+ Using SGDRegressor(random_state=42) as an estimator for exactly **25** iterations (`max_iter=25`). Write the correct $R^2$ score on test data (correct Upto 4 digits)
649
+
650
+ **Step-by-step approach**
651
+
652
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
653
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
654
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
655
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
656
+
657
+ **Solution**
658
+
659
+ ```python
660
+ # Solution
661
+ sgd25 = SGDRegressor(random_state=42, max_iter=25).fit(X_train, y_train)
662
+ print(round(sgd25.score(X_test, y_test), 4))
663
+ ```
664
+
665
+ **Recorded result**
666
+
667
+ ```text
668
+ 0.1397
669
+ ```
670
+
671
+ ---
672
+
673
+ ### Mark the best `alpha` value you got using above instructions
674
+
675
+ **Task**
676
+
677
+ Mark the best `alpha` value you got using above instructions.
678
+
679
+ **Step-by-step approach**
680
+
681
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
682
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
683
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
684
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
685
+
686
+ **Solution**
687
+
688
+ ```python
689
+ # Solution
690
+ print(grid.best_params_['lasso__alpha'])
691
+ ```
692
+
693
+ **Recorded result**
694
+
695
+ ```text
696
+ 10.0
697
+ ```
698
+
699
+ ---
700
+
701
+ ### Enter the best polynomial degree value you got using above instructions
702
+
703
+ **Task**
704
+
705
+ Enter the best polynomial degree value you got using above instructions.
706
+
707
+ **Step-by-step approach**
708
+
709
+ 1. Identify the input columns, target (if any), and exact output requested.
710
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
711
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
712
+
713
+ **Solution**
714
+
715
+ ```python
716
+ # Solution
717
+ print(grid.best_params_['poly__degree'])
718
+ ```
719
+
720
+ **Recorded result**
721
+
722
+ ```text
723
+ 2
724
+ ```
725
+
726
+ ---
727
+
728
+ ### What is the sum of `explained_variance_ratio_`
729
+
730
+ **Task**
731
+
732
+ What is the sum of `explained_variance_ratio_` ? (upto 4 decimal)
733
+
734
+ **Step-by-step approach**
735
+
736
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
737
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
738
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
739
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
740
+
741
+ **Solution**
742
+
743
+ ```python
744
+ # Solution
745
+ print(round(pca.explained_variance_ratio_.sum(), 4))
746
+ ```
747
+
748
+ **Recorded result**
749
+
750
+ ```text
751
+ 0.7816
752
+ ```
753
+
754
+ ---
755
+
756
+ ### Use PCA transformed training data from earlier question and y_train to fit the `RidgeCV` estimator model having `alpha values as [0
757
+
758
+ **Task**
759
+
760
+ Use PCA transformed training data from earlier question and y_train to fit the `RidgeCV` estimator model having `alpha values as [0.01, 0.1, 1, 10, 100]`. Calculate the $R^2$ score you got from the model for transformed test data(PCA transformed X_test). (upto 4 decimal)
761
+
762
+ **Step-by-step approach**
763
+
764
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
765
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
766
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
767
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
768
+
769
+ **Solution**
770
+
771
+ ```python
772
+ # Solution
773
+ from sklearn.linear_model import RidgeCV
774
+ rcv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100]).fit(X_train_pca, y_train)
775
+ print(round(rcv.score(X_test_pca, y_test), 4))
776
+ ```
777
+
778
+ **Recorded result**
779
+
780
+ ```text
781
+ 0.1308
782
+ ```
783
+
784
+ ---
785
+
786
+ ## Residential price records
787
+
788
+ ### How many columns (features + target) are there in the dataset
789
+
790
+ **Task**
791
+
792
+ How many columns (features + target) are there in the dataset?
793
+
794
+ **Step-by-step approach**
795
+
796
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
797
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
798
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
799
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
800
+
801
+ **Solution**
802
+
803
+ ```python
804
+ # Solution
805
+ print(df.shape[1])
806
+ ```
807
+
808
+ **Recorded result**
809
+
810
+ ```text
811
+ 14
812
+ ```
813
+
814
+ ---
815
+
816
+ ### What is the **median** house price (in lacs)
817
+
818
+ **Task**
819
+
820
+ What is the **median** house price (in lacs)?
821
+
822
+ **Step-by-step approach**
823
+
824
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
825
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
826
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
827
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
828
+
829
+ **Solution**
830
+
831
+ ```python
832
+ # Solution
833
+ print(df['PRICE'].median())
834
+ ```
835
+
836
+ **Recorded result**
837
+
838
+ ```text
839
+ 22.634566447766908
840
+ ```
841
+
842
+ ---
843
+
844
+ ### How many houses have **9 or more** rooms
845
+
846
+ **Task**
847
+
848
+ How many houses have **9 or more** rooms?
849
+
850
+ **Step-by-step approach**
851
+
852
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
853
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
854
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
855
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
856
+
857
+ **Solution**
858
+
859
+ ```python
860
+ # Solution
861
+ print((df['RM'] >= 9).sum())
862
+ ```
863
+
864
+ **Recorded result**
865
+
866
+ ```text
867
+ 318
868
+ ```
869
+
870
+ ---
871
+
872
+ ### What is the average price of the **10 least expensive** houses (in lacs)
873
+
874
+ **Task**
875
+
876
+ What is the average price of the **10 least expensive** houses (in lacs)?
877
+
878
+ **Step-by-step approach**
879
+
880
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
881
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
882
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
883
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
884
+
885
+ **Solution**
886
+
887
+ ```python
888
+ # Solution
889
+ print(df['PRICE'].nsmallest(10).mean())
890
+ ```
891
+
892
+ **Recorded result**
893
+
894
+ ```text
895
+ 7.278891822010996
896
+ ```
897
+
898
+ ---
899
+
900
+ ### What is the **total** number of missing or unknown values in the `RM`, `AGE` and `RIVERSIDE` features **combined**
901
+
902
+ **Task**
903
+
904
+ What is the **total** number of missing or unknown values in the `RM`, `AGE` and `RIVERSIDE` features **combined**?
905
+
906
+ **Step-by-step approach**
907
+
908
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
909
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
910
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
911
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
912
+
913
+ **Solution**
914
+
915
+ ```python
916
+ # Solution
917
+ print(rm_bad.sum() + age_bad.sum() + riv_bad.sum())
918
+ ```
919
+
920
+ **Recorded result**
921
+
922
+ ```text
923
+ 178
924
+ ```
925
+
926
+ ---
927
+
928
+ ### What is the average age (in years) of the houses, **ignoring** the missing or unknown values of the `AGE` feature
929
+
930
+ **Task**
931
+
932
+ What is the average age (in years) of the houses, **ignoring** the missing or unknown values of the `AGE` feature?
933
+
934
+ **Step-by-step approach**
935
+
936
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
937
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
938
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
939
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
940
+
941
+ **Solution**
942
+
943
+ ```python
944
+ # Solution
945
+ print(df.loc[~age_bad, 'AGE'].mean())
946
+ ```
947
+
948
+ **Recorded result**
949
+
950
+ ```text
951
+ 66.86582278481012
952
+ ```
953
+
954
+ ---
955
+
956
+ ### What **percentage** of the houses have an unknown value in the `RIVERSIDE` feature
957
+
958
+ **Task**
959
+
960
+ What **percentage** of the houses have an unknown value in the `RIVERSIDE` feature? (Enter the answer correct upto 2 decimal places)
961
+
962
+ **Step-by-step approach**
963
+
964
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
965
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
966
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
967
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
968
+
969
+ **Solution**
970
+
971
+ ```python
972
+ # Solution
973
+ print(round(riv_bad.mean() * 100, 2))
974
+ ```
975
+
976
+ **Recorded result**
977
+
978
+ ```text
979
+ 2.2
980
+ ```
981
+
982
+ ---
983
+
984
+ ### How many houses are **not** on the riverside and are **older than 60 years** (i
985
+
986
+ **Task**
987
+
988
+ How many houses are **not** on the riverside and are **older than 60 years** (i.e. AGE > 60)? For this question, ignore the rows that have missing values in either riverside feature or age feature.
989
+
990
+ **Step-by-step approach**
991
+
992
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
993
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
994
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
995
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
996
+
997
+ **Solution**
998
+
999
+ ```python
1000
+ # Solution
1001
+ valid = ~riv_bad & ~age_bad
1002
+ print(((df['RIVERSIDE'] == 'NO') & (df['AGE'] > 60) & valid).sum())
1003
+ ```
1004
+
1005
+ **Recorded result**
1006
+
1007
+ ```text
1008
+ 2088
1009
+ ```
1010
+
1011
+ ---
1012
+
1013
+ ### How many houses are near to exactly 3, 4 or 5 highways (all three inclusive)
1014
+
1015
+ **Task**
1016
+
1017
+ How many houses are near to exactly 3, 4 or 5 highways (all three inclusive)?
1018
+
1019
+ **Step-by-step approach**
1020
+
1021
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1022
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1023
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1024
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1025
+
1026
+ **Solution**
1027
+
1028
+ ```python
1029
+ # Solution
1030
+ print(df['HIGHWAYCOUNT'].between(3, 5).sum())
1031
+ ```
1032
+
1033
+ **Recorded result**
1034
+
1035
+ ```text
1036
+ 1742
1037
+ ```
1038
+
1039
+ ---
1040
+
1041
+ ### Create a column 'CATEGORY' and divide the houses in categories as following:
1042
+
1043
+ **Task**
1044
+
1045
+ Create a column 'CATEGORY' and divide the houses in categories as following:
1046
+
1047
+ - Category 1: house price <15 lacs
1048
+ - Category 2: 15 lacs <= house price <25 lacs
1049
+ - Category 3: 25 lacs <= house price <35 lacs
1050
+ - Category 4: house price >=35 lacs
1051
+
1052
+ Which category has the **lowest** number of records?
1053
+
1054
+ **Step-by-step approach**
1055
+
1056
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1057
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1058
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1059
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1060
+
1061
+ **Solution**
1062
+
1063
+ ```python
1064
+ # Solution
1065
+ df['CATEGORY'] = pd.cut(df['PRICE'], bins=[-np.inf, 15, 25, 35, np.inf], right=False, labels=[1, 2, 3, 4])
1066
+ print(df['CATEGORY'].value_counts().sort_index())
1067
+ ```
1068
+
1069
+ **Recorded result**
1070
+
1071
+ ```text
1072
+ CATEGORY
1073
+ 1 364
1074
+ 2 2318
1075
+ 3 889
1076
+ 4 429
1077
+ Name: count, dtype: int64
1078
+ ```
1079
+
1080
+ ---
1081
+
1082
+ ### What is the number of samples in the **test** set
1083
+
1084
+ **Task**
1085
+
1086
+ What is the number of samples in the **test** set?
1087
+
1088
+ **Step-by-step approach**
1089
+
1090
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1091
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1092
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1093
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1094
+
1095
+ **Solution**
1096
+
1097
+ ```python
1098
+ # Solution
1099
+ from sklearn.model_selection import train_test_split
1100
+ df.loc[rm_bad, 'RM'] = np.nan
1101
+ df.loc[age_bad, 'AGE'] = np.nan
1102
+ df.loc[riv_bad, 'RIVERSIDE'] = np.nan
1103
+ X = df.drop(columns=['PRICE'])
1104
+ y = df['PRICE']
1105
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=0)
1106
+ print(X_test.shape[0])
1107
+ ```
1108
+
1109
+ **Recorded result**
1110
+
1111
+ ```text
1112
+ 1000
1113
+ ```
1114
+
1115
+ ---
1116
+
1117
+ ### How many features are there after performing above transformation
1118
+
1119
+ **Task**
1120
+
1121
+ How many features are there after performing above transformation?
1122
+
1123
+ **Step-by-step approach**
1124
+
1125
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1126
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1127
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1128
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1129
+
1130
+ **Solution**
1131
+
1132
+ ```python
1133
+ # Solution
1134
+ print(X_train_t.shape[1])
1135
+ ```
1136
+
1137
+ **Recorded result**
1138
+
1139
+ ```text
1140
+ 13
1141
+ ```
1142
+
1143
+ ---
1144
+
1145
+ ### What is the mean of the transformed **training** data (features only)
1146
+
1147
+ **Task**
1148
+
1149
+ What is the mean of the transformed **training** data (features only)?
1150
+
1151
+ Note : Compute the mean of the whole feature matrix i.e. mean of all values in the transformed training feature matrix
1152
+
1153
+ **Step-by-step approach**
1154
+
1155
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1156
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1157
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1158
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1159
+
1160
+ **Solution**
1161
+
1162
+ ```python
1163
+ # Solution
1164
+ print(X_train_t.mean())
1165
+ ```
1166
+
1167
+ **Recorded result**
1168
+
1169
+ ```text
1170
+ 0.28842989630040944
1171
+ ```
1172
+
1173
+ ---
1174
+
1175
+ ### Enter the value of $R^2$ score on the test dataset
1176
+
1177
+ **Task**
1178
+
1179
+ Enter the value of $R^2$ score on the test dataset.
1180
+
1181
+ **Step-by-step approach**
1182
+
1183
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1184
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1185
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1186
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1187
+
1188
+ **Solution**
1189
+
1190
+ ```python
1191
+ # Solution
1192
+ print(ridge.score(X_test, y_test))
1193
+ ```
1194
+
1195
+ **Recorded result**
1196
+
1197
+ ```text
1198
+ 0.6639511913668861
1199
+ ```
1200
+
1201
+ ---
1202
+
1203
+ ### What is the index of most important feature (highest absolute coefficient)
1204
+
1205
+ **Task**
1206
+
1207
+ What is the index of most important feature (highest absolute coefficient)? Note the index starts from 0. Ignore the intercept for this question.
1208
+
1209
+ **Step-by-step approach**
1210
+
1211
+ 1. Fit the requested estimator/selector on the training feature matrix only.
1212
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
1213
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
1214
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
1215
+
1216
+ **Solution**
1217
+
1218
+ ```python
1219
+ # Solution
1220
+ print(np.abs(ridge.coef_))
1221
+ print(np.argmax(np.abs(ridge.coef_)))
1222
+ ```
1223
+
1224
+ **Recorded result**
1225
+
1226
+ ```text
1227
+ [ 0.80599934 4.36935144 0.54731649 0.57136401 11.03253838 6.96010246
1228
+ 6.21337559 9.67022437 4.75155118 25.56498977 11.69237356 0.82743376
1229
+ 0.83498997 0.83498997]
1230
+ 9
1231
+ ```
1232
+
1233
+ ---
1234
+
1235
+ ### What is the index of least important feature (lowest absolute coefficient)
1236
+
1237
+ **Task**
1238
+
1239
+ What is the index of least important feature (lowest absolute coefficient)? Note the index starts from 0. Ignore the intercept for this question.
1240
+
1241
+ **Step-by-step approach**
1242
+
1243
+ 1. Fit the requested estimator/selector on the training feature matrix only.
1244
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
1245
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
1246
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
1247
+
1248
+ **Solution**
1249
+
1250
+ ```python
1251
+ # Solution
1252
+ print(np.argmin(np.abs(ridge.coef_)))
1253
+ ```
1254
+
1255
+ **Recorded result**
1256
+
1257
+ ```text
1258
+ 2
1259
+ ```
1260
+
1261
+ ---
1262
+
1263
+ ### What is the best **alpha**
1264
+
1265
+ **Task**
1266
+
1267
+ What is the best **alpha**?
1268
+
1269
+ > A 0.0001\
1270
+ > B 0.001\
1271
+ > C 0.01\
1272
+ > D 0.1\
1273
+ > E 1
1274
+
1275
+ **Step-by-step approach**
1276
+
1277
+ 1. Identify the input columns, target (if any), and exact output requested.
1278
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
1279
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
1280
+
1281
+ **Solution**
1282
+
1283
+ ```python
1284
+ # Solution
1285
+ print(sgd_grid.best_params_['alpha'])
1286
+ ```
1287
+
1288
+ **Recorded result**
1289
+
1290
+ ```text
1291
+ 0.0001
1292
+ ```
1293
+
1294
+ ---
1295
+
1296
+ ### What will be value of **root mean squared error** on the test dataset
1297
+
1298
+ **Task**
1299
+
1300
+ What will be value of **root mean squared error** on the test dataset?
1301
+
1302
+ **Step-by-step approach**
1303
+
1304
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1305
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1306
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1307
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1308
+
1309
+ **Solution**
1310
+
1311
+ ```python
1312
+ # Solution
1313
+ print(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))))
1314
+ ```
1315
+
1316
+ **Recorded result**
1317
+
1318
+ ```text
1319
+ 5.126302501903342
1320
+ ```
1321
+
1322
+ ---
1323
+
1324
+ ### If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset
1325
+
1326
+ **Task**
1327
+
1328
+ If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset?
1329
+
1330
+ **Step-by-step approach**
1331
+
1332
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1333
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1334
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1335
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1336
+
1337
+ **Solution**
1338
+
1339
+ ```python
1340
+ # Solution
1341
+ print(mean_absolute_error(y_test, pca_grid.predict(X_test)))
1342
+ ```
1343
+
1344
+ **Recorded result**
1345
+
1346
+ ```text
1347
+ 3.865036507995
1348
+ ```
1349
+
1350
+ ---
1351
+
1352
+ ### What **percentage** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model
1353
+
1354
+ **Task**
1355
+
1356
+ What **percentage** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model?
1357
+
1358
+ **Step-by-step approach**
1359
+
1360
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
1361
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
1362
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
1363
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
1364
+
1365
+ **Solution**
1366
+
1367
+ ```python
1368
+ # Solution
1369
+ print(pca_grid.best_estimator_.named_steps['pca'].explained_variance_ratio_[0] * 100)
1370
+ ```
1371
+
1372
+ **Recorded result**
1373
+
1374
+ ```text
1375
+ 69.9021494220279
1376
+ ```
1377
+
1378
+ ---
1379
+
1380
+ ### Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
1381
+
1382
+ **Task**
1383
+
1384
+ Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
1385
+ - For PolynomialFeatures:
1386
+ - interaction_only = False
1387
+ - degree = 2
1388
+ - For Lasso:
1389
+ - alpha = 0.5
1390
+ - warm_start = True
1391
+ - random state as 0
1392
+
1393
+ Fit the pipeline on the training dataset and find the **mean absolute error** on the test dataset.
1394
+
1395
+ **Step-by-step approach**
1396
+
1397
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1398
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1399
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1400
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1401
+
1402
+ **Solution**
1403
+
1404
+ ```python
1405
+ # Solution
1406
+ from sklearn.preprocessing import PolynomialFeatures
1407
+ poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=2, interaction_only=False)),
1408
+ ('lasso', Lasso(alpha=0.5, warm_start=True, random_state=0))])
1409
+ poly_pipe.fit(X_train, y_train)
1410
+ print(mean_absolute_error(y_test, poly_pipe.predict(X_test)))
1411
+ ```
1412
+
1413
+ **Recorded result**
1414
+
1415
+ ```text
1416
+ 4.55907406372981
1417
+ ```
1418
+
1419
+ ---
1420
+
1421
+ ### If you eliminate **2** features with recursive feature elimination, which features will be eliminated
1422
+
1423
+ **Task**
1424
+
1425
+ If you eliminate **2** features with recursive feature elimination, which features will be eliminated?
1426
+
1427
+ Type the indices of the eliminated features (index starts from 0).
1428
+
1429
+ Use LinearRegression model with default parameters as an estimator.
1430
+
1431
+ Use processed training data.
1432
+
1433
+ **Step-by-step approach**
1434
+
1435
+ 1. Fit the requested estimator/selector on the training feature matrix only.
1436
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
1437
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
1438
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
1439
+
1440
+ **Solution**
1441
+
1442
+ ```python
1443
+ # Solution
1444
+ from sklearn.feature_selection import RFE
1445
+ from sklearn.linear_model import LinearRegression
1446
+ rfe = RFE(LinearRegression(), n_features_to_select=X_train.shape[1] - 2)
1447
+ rfe.fit(X_train, y_train)
1448
+ print(np.where(~rfe.support_)[0])
1449
+ print(rfe.ranking_)
1450
+ ```
1451
+
1452
+ **Recorded result**
1453
+
1454
+ ```text
1455
+ [2 3]
1456
+ [1 1 3 2 1 1 1 1 1 1 1 1 1 1]
1457
+ ```
1458
+
1459
+ ---
1460
+
1461
+ ## Property listing records
1462
+
1463
+ ### What are the unique values present in the `Residential` feature of the dataset
1464
+
1465
+ **Task**
1466
+
1467
+ What are the unique values present in the `Residential` feature of the dataset?
1468
+ - [Single Family, Two Family, Three Family, Four Family]
1469
+ - [Detached House, Duplex, Triplex, Fourplex]
1470
+ - [Detached House, Duplex, Triplex]
1471
+ - [Bridgeport, Fairfield, Greenwich, Norwalk]
1472
+
1473
+ **Step-by-step approach**
1474
+
1475
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
1476
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
1477
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
1478
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
1479
+
1480
+ **Solution**
1481
+
1482
+ ```python
1483
+ # Solution
1484
+ print(df['Residential'].unique())
1485
+ ```
1486
+
1487
+ **Recorded result**
1488
+
1489
+ ```text
1490
+ ['Detached House' 'Duplex' 'Triplex' 'Fourplex']
1491
+ ```
1492
+
1493
+ ---
1494
+
1495
+ ### Which of the following columns have categorical data
1496
+
1497
+ **Task**
1498
+
1499
+ Which of the following columns have categorical data?
1500
+ - Year
1501
+ - Residential
1502
+ - carpet_area
1503
+ - Face
1504
+ - property_tax_rate
1505
+
1506
+ **Step-by-step approach**
1507
+
1508
+ 1. List categorical columns and inspect their unique values, including missing values.
1509
+ 2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
1510
+ 3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
1511
+ 4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
1512
+
1513
+ **Solution**
1514
+
1515
+ ```python
1516
+ # Solution
1517
+ print(df.dtypes)
1518
+ ```
1519
+
1520
+ **Recorded result**
1521
+
1522
+ ```text
1523
+ Date object
1524
+ Year int64
1525
+ Locality object
1526
+ Estimated Value float64
1527
+ Sale Price float64
1528
+ Property object
1529
+ Residential object
1530
+ num_rooms int64
1531
+ num_bathrooms int64
1532
+ carpet_area float64
1533
+ property_tax_rate float64
1534
+ Face object
1535
+ dtype: object
1536
+ ```
1537
+
1538
+ ---
1539
+
1540
+ ### Which of the following features have `missing`(NaN) or unknown ("
1541
+
1542
+ **Task**
1543
+
1544
+ Which of the following features have `missing`(NaN) or unknown ("?") values present in the dataset?
1545
+ - Date
1546
+ - Locality
1547
+ - carpet_area
1548
+ - Property
1549
+ - num_rooms
1550
+ - Face
1551
+
1552
+ **Step-by-step approach**
1553
+
1554
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1555
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1556
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1557
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1558
+
1559
+ **Solution**
1560
+
1561
+ ```python
1562
+ # Solution
1563
+ print(df.isna().sum())
1564
+ print((df == '?').sum())
1565
+ ```
1566
+
1567
+ **Recorded result**
1568
+
1569
+ ```text
1570
+ Date 0
1571
+ Year 0
1572
+ Locality 1285
1573
+ Estimated Value 1281
1574
+ Sale Price 0
1575
+ Property 0
1576
+ Residential 0
1577
+ num_rooms 0
1578
+ num_bathrooms 0
1579
+ carpet_area 1282
1580
+ property_tax_rate 0
1581
+ Face 0
1582
+ dtype: int64
1583
+ Date 0
1584
+ Year 0
1585
+ Locality 0
1586
+ Estimated Value 0
1587
+ Sale Price 0
1588
+ Property 1873
1589
+ Residential 0
1590
+ num_rooms 0
1591
+ num_bathrooms 0
1592
+ carpet_area 0
1593
+ property_tax_rate 0
1594
+ Face 0
1595
+ dtype: int64
1596
+ ```
1597
+
1598
+ ---
1599
+
1600
+ ### In the Year **2021**, how many houses (rows) located in the **Stamford** Locality have **2 or more** num_bathrooms, and are facing…
1601
+
1602
+ **Task**
1603
+
1604
+ In the Year **2021**, how many houses (rows) located in the **Stamford** Locality have **2 or more** num_bathrooms, and are facing towards either the **South or West**?
1605
+
1606
+ **Step-by-step approach**
1607
+
1608
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
1609
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
1610
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
1611
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
1612
+
1613
+ **Solution**
1614
+
1615
+ ```python
1616
+ # Solution
1617
+ mask = (df['Year'] == 2021) & (df['Locality'] == 'Stamford') & (df['num_bathrooms'] >= 2) & (df['Face'].isin(['South', 'West']))
1618
+ print(mask.sum())
1619
+ ```
1620
+
1621
+ **Recorded result**
1622
+
1623
+ ```text
1624
+ 44
1625
+ ```
1626
+
1627
+ ---
1628
+
1629
+ ### Split the dataset into train dataset and test dataset in the following manner
1630
+
1631
+ **Task**
1632
+
1633
+ Split the dataset into train dataset and test dataset in the following manner
1634
+ 1. data(rows) collected before the year of **2020** [2009-2019] should be the train dataset
1635
+ 2. data(rows) collected in the year of **2020, 2021 and 2022** (all inclusive) should be the test dataset
1636
+ 3. columns except of the label vector (`Sale Price`) should be the feature matrix (X_train or X_test)
1637
+ 4. make label vector (Y_train or y_test) containing values only from the target feature.
1638
+
1639
+ How many rows are in the feature matrix of the **train** dataset?
1640
+
1641
+ **Step-by-step approach**
1642
+
1643
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1644
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1645
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1646
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1647
+
1648
+ **Solution**
1649
+
1650
+ ```python
1651
+ # Solution
1652
+ train = df[df['Year'] <= 2019]
1653
+ test = df[df['Year'] >= 2020]
1654
+ X_train, y_train = train.drop(columns=['Sale Price']).copy(), train['Sale Price']
1655
+ X_test, y_test = test.drop(columns=['Sale Price']).copy(), test['Sale Price']
1656
+ print(X_train.shape, X_test.shape)
1657
+ ```
1658
+
1659
+ **Recorded result**
1660
+
1661
+ ```text
1662
+ (7560, 11) (2440, 11)
1663
+ ```
1664
+
1665
+ ---
1666
+
1667
+ ### Write the **MEDIAN** value of the `carpet_area` column of the train dataset you found to replace all the Missing Values (NaN)
1668
+
1669
+ **Task**
1670
+
1671
+ Write the **MEDIAN** value of the `carpet_area` column of the train dataset you found to replace all the Missing Values (NaN). (upto 2 digits after decimal points)
1672
+
1673
+ **Step-by-step approach**
1674
+
1675
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1676
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1677
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1678
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1679
+
1680
+ **Solution**
1681
+
1682
+ ```python
1683
+ # Solution
1684
+ print(round(carpet_median, 2))
1685
+ ```
1686
+
1687
+ **Recorded result**
1688
+
1689
+ ```text
1690
+ 1021.0
1691
+ ```
1692
+
1693
+ ---
1694
+
1695
+ ### What value did you use to replace the missing values (NaN) in the `Locality` column of the train dataset
1696
+
1697
+ **Task**
1698
+
1699
+ What value did you use to replace the missing values (NaN) in the `Locality` column of the train dataset?
1700
+ - Bridgeport
1701
+ - Waterbury
1702
+ - Fairfield
1703
+ - Stamford
1704
+
1705
+ **Step-by-step approach**
1706
+
1707
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
1708
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
1709
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
1710
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
1711
+
1712
+ **Solution**
1713
+
1714
+ ```python
1715
+ # Solution
1716
+ print(loc_mode)
1717
+ ```
1718
+
1719
+ **Recorded result**
1720
+
1721
+ ```text
1722
+ Bridgeport
1723
+ ```
1724
+
1725
+ ---
1726
+
1727
+ ### How many total features you got after one-hot encoding and scaling the training dataset
1728
+
1729
+ **Task**
1730
+
1731
+ How many total features you got after one-hot encoding and scaling the training dataset?
1732
+ - 10
1733
+ - 18
1734
+ - 25
1735
+ - 28
1736
+
1737
+ **Step-by-step approach**
1738
+
1739
+ 1. List categorical columns and inspect their unique values, including missing values.
1740
+ 2. Choose OneHotEncoder for unordered categories or OrdinalEncoder only when a meaningful order is explicitly specified.
1741
+ 3. Fit the encoder on training data, configure unseen-category handling where appropriate, and transform test data with the same fitted encoder.
1742
+ 4. Use get_feature_names_out() or the transformed matrix shape to verify output order and dimensionality.
1743
+
1744
+ **Solution**
1745
+
1746
+ ```python
1747
+ # Solution
1748
+ print(X_train_t.shape[1])
1749
+ ```
1750
+
1751
+ **Recorded result**
1752
+
1753
+ ```text
1754
+ 25
1755
+ ```
1756
+
1757
+ ---
1758
+
1759
+ ### What is the **maximum** value present in the transformed **test** feature matrix
1760
+
1761
+ **Task**
1762
+
1763
+ What is the **maximum** value present in the transformed **test** feature matrix? (upto 2 digits after decimal points)
1764
+
1765
+ **Step-by-step approach**
1766
+
1767
+ 1. Identify the input columns, target (if any), and exact output requested.
1768
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
1769
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
1770
+
1771
+ **Solution**
1772
+
1773
+ ```python
1774
+ # Solution
1775
+ print(round(X_test_t.max(), 2))
1776
+ ```
1777
+
1778
+ **Recorded result**
1779
+
1780
+ ```text
1781
+ 13.16
1782
+ ```
1783
+
1784
+ ---
1785
+
1786
+ ### Apply LinearRegression on the train dataset(X_train and y_train)
1787
+
1788
+ **Task**
1789
+
1790
+ Apply LinearRegression on the train dataset(X_train and y_train). What is the **mean absolute error** on the test dataset(X_test and y_test)? (Upto 2 digits after decimal points)
1791
+
1792
+ **Step-by-step approach**
1793
+
1794
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1795
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1796
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1797
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1798
+
1799
+ **Solution**
1800
+
1801
+ ```python
1802
+ # Solution
1803
+ from sklearn.linear_model import LinearRegression
1804
+ from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score
1805
+ lr = LinearRegression().fit(X_train, y_train)
1806
+ print(round(mean_absolute_error(y_test, lr.predict(X_test)), 2))
1807
+ ```
1808
+
1809
+ **Recorded result**
1810
+
1811
+ ```text
1812
+ 179131.9
1813
+ ```
1814
+
1815
+ ---
1816
+
1817
+ ### Enter the **minimum** value of $R^2$ score you got using the LinearRegression model when computed with 5 folds from the training…
1818
+
1819
+ **Task**
1820
+
1821
+ Enter the **minimum** value of $R^2$ score you got using the LinearRegression model when computed with 5 folds from the training dataset (X_train and y_train) using cross_val_score. (Upto 4 digits after decimal points)
1822
+
1823
+ **Step-by-step approach**
1824
+
1825
+ 1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
1826
+ 2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
1827
+ 3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
1828
+ 4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
1829
+
1830
+ **Solution**
1831
+
1832
+ ```python
1833
+ # Solution
1834
+ from sklearn.model_selection import cross_val_score
1835
+ scores = cross_val_score(LinearRegression(), X_train, y_train, cv=5)
1836
+ print(scores)
1837
+ print(round(scores.min(), 4))
1838
+ ```
1839
+
1840
+ **Recorded result**
1841
+
1842
+ ```text
1843
+ [0.61661573 0.79290189 0.72820221 0.82626052 0.64972083]
1844
+ 0.6166
1845
+ ```
1846
+
1847
+ ---
1848
+
1849
+ ### Train Ridge and Lasso with random_state=27 and keep other parameters as default using train dataset
1850
+
1851
+ **Task**
1852
+
1853
+ Train Ridge and Lasso with random_state=27 and keep other parameters as default using train dataset. Which one has the least **"mean absolute error"** for the test dataset?
1854
+
1855
+ - Ridge
1856
+ - Lasso
1857
+ - Both are Equal
1858
+
1859
+ **Step-by-step approach**
1860
+
1861
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1862
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1863
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1864
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1865
+
1866
+ **Solution**
1867
+
1868
+ ```python
1869
+ # Solution
1870
+ from sklearn.linear_model import Ridge, Lasso
1871
+ ridge = Ridge(random_state=27).fit(X_train, y_train)
1872
+ lasso = Lasso(random_state=27).fit(X_train, y_train)
1873
+ print('Ridge MAE:', mean_absolute_error(y_test, ridge.predict(X_test)))
1874
+ print('Lasso MAE:', mean_absolute_error(y_test, lasso.predict(X_test)))
1875
+ ```
1876
+
1877
+ **Recorded result**
1878
+
1879
+ ```text
1880
+ Ridge MAE: 179994.97462090437
1881
+ Lasso MAE: 179123.65380001903
1882
+ ```
1883
+
1884
+ ---
1885
+
1886
+ ### Train the `SGDRegressor(random_state=27, warm_start=True)` with maximum passes over the train dataset as **50**
1887
+
1888
+ **Task**
1889
+
1890
+ Train the `SGDRegressor(random_state=27, warm_start=True)` with maximum passes over the train dataset as **50**. Write the correct $R^2$ score for this estimator on test dataset (correct Upto 4 digits)
1891
+
1892
+ **Step-by-step approach**
1893
+
1894
+ 1. Identify the target column and remove it from the feature matrix before splitting.
1895
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
1896
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
1897
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
1898
+
1899
+ **Solution**
1900
+
1901
+ ```python
1902
+ # Solution
1903
+ from sklearn.linear_model import SGDRegressor
1904
+ sgd = SGDRegressor(random_state=27, warm_start=True, max_iter=50).fit(X_train, y_train)
1905
+ print(round(sgd.score(X_test, y_test), 4))
1906
+ ```
1907
+
1908
+ **Recorded result**
1909
+
1910
+ ```text
1911
+ 0.4676
1912
+ ```
1913
+
1914
+ ---
1915
+
1916
+ ### Mark the best alpha value you got using above instructions
1917
+
1918
+ **Task**
1919
+
1920
+ Mark the best alpha value you got using above instructions.
1921
+ - 1
1922
+ - 10
1923
+ - 100
1924
+ - 1000
1925
+
1926
+ **Step-by-step approach**
1927
+
1928
+ 1. Create the exact estimator and parameter grid. Pipeline parameters must use the step prefix, for example model__alpha.
1929
+ 2. Set the required cross-validation count and scoring string. Error scorers use a negative sign so that larger scores remain better.
1930
+ 3. Fit GridSearchCV only on training data. Read best_params_ and best_estimator_ for selected settings.
1931
+ 4. Evaluate the refitted best estimator on the held-out test data using the requested metric.
1932
+
1933
+ **Solution**
1934
+
1935
+ ```python
1936
+ # Solution
1937
+ print(grid.best_params_['lasso__alpha'])
1938
+ ```
1939
+
1940
+ **Recorded result**
1941
+
1942
+ ```text
1943
+ 1000
1944
+ ```
1945
+
1946
+ ---
1947
+
1948
+ ### Enter the best polynomial degree value you got using above instructions
1949
+
1950
+ **Task**
1951
+
1952
+ Enter the best polynomial degree value you got using above instructions.
1953
+
1954
+ **Step-by-step approach**
1955
+
1956
+ 1. Identify the input columns, target (if any), and exact output requested.
1957
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
1958
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
1959
+
1960
+ **Solution**
1961
+
1962
+ ```python
1963
+ # Solution
1964
+ print(grid.best_params_['poly__degree'])
1965
+ ```
1966
+
1967
+ **Recorded result**
1968
+
1969
+ ```text
1970
+ 1
1971
+ ```
1972
+
1973
+ ---
1974
+
1975
+ ### What is the sum of explained_variance_ratio_
1976
+
1977
+ **Task**
1978
+
1979
+ What is the sum of explained_variance_ratio_ ? (upto two digits after the decimal)
1980
+
1981
+ **Step-by-step approach**
1982
+
1983
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
1984
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
1985
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
1986
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
1987
+
1988
+ **Solution**
1989
+
1990
+ ```python
1991
+ # Solution
1992
+ print(round(pca.explained_variance_ratio_.sum(), 2))
1993
+ ```
1994
+
1995
+ **Recorded result**
1996
+
1997
+ ```text
1998
+ 0.86
1999
+ ```
2000
+
2001
+ ---
2002
+
2003
+ ### Use PCA transformed training feature matrix from earlier question and training labels to fit the RidgeCV estimator with cv=5 and…
2004
+
2005
+ **Task**
2006
+
2007
+ Use PCA transformed training feature matrix from earlier question and training labels to fit the RidgeCV estimator with cv=5 and having alpha values as **[0.01, 0.1, 1, 10, 100, 1000]**. Calculate the $R^2$ score you got from the model for PCA transformed test dataset. (upto 4 digits after the decimal)
2008
+
2009
+ **Step-by-step approach**
2010
+
2011
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2012
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2013
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2014
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2015
+
2016
+ **Solution**
2017
+
2018
+ ```python
2019
+ # Solution
2020
+ from sklearn.linear_model import RidgeCV
2021
+ rcv = RidgeCV(alphas=[0.01, 0.1, 1, 10, 100, 1000], cv=5).fit(X_train_pca, y_train)
2022
+ print(round(rcv.score(X_test_pca, y_test), 4))
2023
+ ```
2024
+
2025
+ **Recorded result**
2026
+
2027
+ ```text
2028
+ 0.1789
2029
+ ```
2030
+
2031
+ ---
2032
+
2033
+ ### How many features of the train dataset have a P-VALUE **less than 0
2034
+
2035
+ **Task**
2036
+
2037
+ How many features of the train dataset have a P-VALUE **less than 0.05**?
2038
+
2039
+ **Step-by-step approach**
2040
+
2041
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2042
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2043
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2044
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2045
+
2046
+ **Solution**
2047
+
2048
+ ```python
2049
+ # Solution
2050
+ print(np.round(kb.pvalues_, 4))
2051
+ print((kb.pvalues_ < 0.05).sum())
2052
+ ```
2053
+
2054
+ **Recorded result**
2055
+
2056
+ ```text
2057
+ [0.000e+00 5.000e-04 0.000e+00 5.288e-01 9.124e-01 0.000e+00 0.000e+00
2058
+ 9.500e-03 0.000e+00 0.000e+00 0.000e+00 0.000e+00 0.000e+00 4.000e-03
2059
+ 0.000e+00 6.493e-01 4.734e-01 5.820e-01 4.196e-01 1.600e-03 0.000e+00
2060
+ 0.000e+00 0.000e+00 0.000e+00 2.442e-01]
2061
+ 18
2062
+ ```
2063
+
2064
+ ---
2065
+
2066
+ ### What is the **minimum** F_STATISTICS score you got for any feature of the train dataset
2067
+
2068
+ **Task**
2069
+
2070
+ What is the **minimum** F_STATISTICS score you got for any feature of the train dataset? (upto 2 digits after the decimal)
2071
+
2072
+ **Step-by-step approach**
2073
+
2074
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2075
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2076
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2077
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2078
+
2079
+ **Solution**
2080
+
2081
+ ```python
2082
+ # Solution
2083
+ print(round(np.nanmin(kb.scores_), 2))
2084
+ ```
2085
+
2086
+ **Recorded result**
2087
+
2088
+ ```text
2089
+ 0.01
2090
+ ```
2091
+
2092
+ ---
2093
+
2094
+ ### What is the **mean absolute error** you got for this model using the test dataset
2095
+
2096
+ **Task**
2097
+
2098
+ What is the **mean absolute error** you got for this model using the test dataset? (upto 2 digits after the decimal)
2099
+
2100
+ **Step-by-step approach**
2101
+
2102
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2103
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2104
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2105
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2106
+
2107
+ **Solution**
2108
+
2109
+ ```python
2110
+ # Solution
2111
+ print(round(mean_absolute_error(y_test, kbest_pipe.predict(X_test)), 2))
2112
+ ```
2113
+
2114
+ **Recorded result**
2115
+
2116
+ ```text
2117
+ 170445.36
2118
+ ```
2119
+
2120
+ ---
2121
+
2122
+ ## Flight fare records
2123
+
2124
+ ### : What is the **median** of the flight ticket price
2125
+
2126
+ **Task**
2127
+
2128
+ : What is the **median** of the flight ticket price? Write your answer correct to two decimal places.
2129
+
2130
+ **Step-by-step approach**
2131
+
2132
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2133
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2134
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2135
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2136
+
2137
+ **Solution**
2138
+
2139
+ ```python
2140
+ # Solution
2141
+ print(round(df['Price'].median(), 2))
2142
+ ```
2143
+
2144
+ **Recorded result**
2145
+
2146
+ ```text
2147
+ 8366.0
2148
+ ```
2149
+
2150
+ ---
2151
+
2152
+ ### : During which month did the **lowest** number of flights occur
2153
+
2154
+ **Task**
2155
+
2156
+ : During which month did the **lowest** number of flights occur? Months are represented by numerical codes, with January corresponding to 1, February to 2, and so forth.
2157
+
2158
+ **Step-by-step approach**
2159
+
2160
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2161
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2162
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2163
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2164
+
2165
+ **Solution**
2166
+
2167
+ ```python
2168
+ # Solution
2169
+ print(df['Month'].value_counts())
2170
+ ```
2171
+
2172
+ **Recorded result**
2173
+
2174
+ ```text
2175
+ Month
2176
+ 5 3092
2177
+ 6 3044
2178
+ 3 2388
2179
+ 4 926
2180
+ Name: count, dtype: int64
2181
+ ```
2182
+
2183
+ ---
2184
+
2185
+ ### : Which `Source` city has the **highest average** flight ticket price
2186
+
2187
+ **Task**
2188
+
2189
+ : Which `Source` city has the **highest average** flight ticket price?
2190
+
2191
+ **Step-by-step approach**
2192
+
2193
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2194
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2195
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2196
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2197
+
2198
+ **Solution**
2199
+
2200
+ ```python
2201
+ # Solution
2202
+ print(df.groupby('Source')['Price'].mean().sort_values(ascending=False))
2203
+ ```
2204
+
2205
+ **Recorded result**
2206
+
2207
+ ```text
2208
+ Source
2209
+ Delhi 10637.617736
2210
+ Kolkata 9087.168003
2211
+ Banglore 7729.939275
2212
+ Mumbai 5148.631841
2213
+ Chennai 4695.082153
2214
+ Name: Price, dtype: float64
2215
+ ```
2216
+
2217
+ ---
2218
+
2219
+ ### : Two of the entries in the `'Additional_Info'` column are `'No info'` and `'No Info'`
2220
+
2221
+ **Task**
2222
+
2223
+ : Two of the entries in the `'Additional_Info'` column are `'No info'` and `'No Info'`. Replace all occurrences of `'No Info'` with `'No info'`. How many flights fall under airline **`'Jet Airways'`** and have `'No info'` as additional information?
2224
+
2225
+ #### **Note:** Please ensure that you make the changes directly within the dataset and continue to use that modified dataset for subsequent questions.
2226
+
2227
+ **Step-by-step approach**
2228
+
2229
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2230
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2231
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2232
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2233
+
2234
+ **Solution**
2235
+
2236
+ ```python
2237
+ # Solution
2238
+ df['Additional_Info'] = df['Additional_Info'].replace('No Info', 'No info')
2239
+ print(((df['Airline'] == 'Jet Airways') & (df['Additional_Info'] == 'No info')).sum())
2240
+ ```
2241
+
2242
+ **Recorded result**
2243
+
2244
+ ```text
2245
+ 2119
2246
+ ```
2247
+
2248
+ ---
2249
+
2250
+ ### : Convert the values of `'Duration'` into **minutes**
2251
+
2252
+ **Task**
2253
+
2254
+ : Convert the values of `'Duration'` into **minutes**. Enter the **median** duration (in minutes) of a flight.
2255
+
2256
+ #### **Note:** Please ensure that you make the changes directly within the dataset and continue to use that modified dataset for subsequent questions.
2257
+
2258
+ **Step-by-step approach**
2259
+
2260
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2261
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2262
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2263
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2264
+
2265
+ **Solution**
2266
+
2267
+ ```python
2268
+ # Solution
2269
+ def to_minutes(s):
2270
+ h = m = 0
2271
+ for part in s.split():
2272
+ if part.endswith('h'):
2273
+ h = int(part[:-1])
2274
+ elif part.endswith('m'):
2275
+ m = int(part[:-1])
2276
+ return h * 60 + m
2277
+ df['Duration'] = df['Duration'].apply(to_minutes)
2278
+ print(df['Duration'].median())
2279
+ ```
2280
+
2281
+ **Recorded result**
2282
+
2283
+ ```text
2284
+ 520.0
2285
+ ```
2286
+
2287
+ ---
2288
+
2289
+ ### : How many flights started in the **Evening** and arrived at the destination at **Night**
2290
+
2291
+ **Task**
2292
+
2293
+ : How many flights started in the **Evening** and arrived at the destination at **Night**?
2294
+
2295
+ **Step-by-step approach**
2296
+
2297
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2298
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2299
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2300
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2301
+
2302
+ **Solution**
2303
+
2304
+ ```python
2305
+ # Solution
2306
+ print(((df['Dep_Time'] == 'Evening') & (df['Arrival_Time'] == 'Night')).sum())
2307
+ ```
2308
+
2309
+ **Recorded result**
2310
+
2311
+ ```text
2312
+ 732
2313
+ ```
2314
+
2315
+ ---
2316
+
2317
+ ### : Encode the values of column `'WeekDay'` as follows:
2318
+
2319
+ **Task**
2320
+
2321
+ : Encode the values of column `'WeekDay'` as follows:
2322
+ # Weekends (Sunday, Saturday) = 1
2323
+ # all remaining five days = 0
2324
+
2325
+ # How many flights have WeekDay encoded as **1** (Weekends)?
2326
+
2327
+ **Step-by-step approach**
2328
+
2329
+ 1. Normalize the raw text/category values first, including differences in capitalization and spacing.
2330
+ 2. Convert duration strings into numeric minutes by parsing hours and minutes separately; preserve missing values as NaN.
2331
+ 3. Create day-part or weekend features from the original time/date columns before encoding.
2332
+ 4. Fit ordered or one-hot encoders on training data and explicitly handle categories that appear only in test data.
2333
+
2334
+ **Solution**
2335
+
2336
+ ```python
2337
+ # Solution
2338
+ df['WeekDay'] = df['WeekDay'].isin(['Saturday', 'Sunday']).astype(int)
2339
+ print(df['WeekDay'].sum())
2340
+ ```
2341
+
2342
+ **Recorded result**
2343
+
2344
+ ```text
2345
+ 2799
2346
+ ```
2347
+
2348
+ ---
2349
+
2350
+ ### : Split the dataset into train dataset and test dataset in the following manner
2351
+
2352
+ **Task**
2353
+
2354
+ : Split the dataset into train dataset and test dataset in the following manner
2355
+
2356
+ - Use `train_test_split` to split the dataset into train and test dataset with test size equal to $0.25(25\%)$ and random_state equal to 42.
2357
+ - columns except of the label vector (`Price`) should be the feature matrix (X_train or X_test)
2358
+ - make label vector (Y_train or y_test) containing values only from the target feature (`Price`).
2359
+
2360
+ # How many rows are in the feature matrix of the **train** dataset?
2361
+
2362
+ **Step-by-step approach**
2363
+
2364
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2365
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2366
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2367
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2368
+
2369
+ **Solution**
2370
+
2371
+ ```python
2372
+ # Solution
2373
+ from sklearn.model_selection import train_test_split
2374
+ X = df.drop(columns=['Price'])
2375
+ y = df['Price']
2376
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
2377
+ print(X_train.shape, X_test.shape)
2378
+ ```
2379
+
2380
+ **Recorded result**
2381
+
2382
+ ```text
2383
+ (7087, 12) (2363, 12)
2384
+ ```
2385
+
2386
+ ---
2387
+
2388
+ ### : Were any unseen values encountered in the test dataset during the transformation process
2389
+
2390
+ **Task**
2391
+
2392
+ : Were any unseen values encountered in the test dataset during the transformation process?
2393
+
2394
+ **Step-by-step approach**
2395
+
2396
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2397
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2398
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2399
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2400
+
2401
+ **Solution**
2402
+
2403
+ ```python
2404
+ # Solution
2405
+ print((X_test_t == -1).sum())
2406
+ ```
2407
+
2408
+ **Recorded result**
2409
+
2410
+ ```text
2411
+ 1
2412
+ ```
2413
+
2414
+ ---
2415
+
2416
+ ### : After transforming the test data, all of its entries become numeric values
2417
+
2418
+ **Task**
2419
+
2420
+ : After transforming the test data, all of its entries become numeric values. What is the **highest** value present in the transformed **test** feature matrix?
2421
+
2422
+ **Step-by-step approach**
2423
+
2424
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2425
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2426
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2427
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2428
+
2429
+ **Solution**
2430
+
2431
+ ```python
2432
+ # Solution
2433
+ print(X_test_t.max())
2434
+ ```
2435
+
2436
+ **Recorded result**
2437
+
2438
+ ```text
2439
+ 27.0
2440
+ ```
2441
+
2442
+ ---
2443
+
2444
+ ### : After transforming the training data, all of its entries become numeric values
2445
+
2446
+ **Task**
2447
+
2448
+ : After transforming the training data, all of its entries become numeric values. What is the mean of the transformed **training** data matrix? Enter your answer correct to two decimal places.
2449
+
2450
+ **Step-by-step approach**
2451
+
2452
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2453
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2454
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2455
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2456
+
2457
+ **Solution**
2458
+
2459
+ ```python
2460
+ # Solution
2461
+ print(round(X_train_t.mean(), 2))
2462
+ ```
2463
+
2464
+ **Recorded result**
2465
+
2466
+ ```text
2467
+ 3.19
2468
+ ```
2469
+
2470
+ ---
2471
+
2472
+ ### : What is the **highest** value of the scaled 'Duration' column in the transformed **test** data
2473
+
2474
+ **Task**
2475
+
2476
+ : What is the **highest** value of the scaled 'Duration' column in the transformed **test** data? Enter your answer correct to two decimal places.
2477
+
2478
+ **Step-by-step approach**
2479
+
2480
+ 1. Separate numeric and categorical columns, and map each feature to its required operation.
2481
+ 2. Build a Pipeline for any feature needing several operations in sequence, such as imputation followed by scaling.
2482
+ 3. Pass transformer blocks to ColumnTransformer in the exact requested order; that order determines transformed column indices.
2483
+ 4. Fit the full transformer on X_train, then transform both train and test data and inspect shape, mean, minimum or maximum as requested.
2484
+
2485
+ **Solution**
2486
+
2487
+ ```python
2488
+ # Solution
2489
+ dur_idx = 7 # position of Duration in the ColumnTransformer output
2490
+ print(round(X_test_t[:, dur_idx].max(), 2))
2491
+ ```
2492
+
2493
+ **Recorded result**
2494
+
2495
+ ```text
2496
+ 0.82
2497
+ ```
2498
+
2499
+ ---
2500
+
2501
+ ### Train the ridge model on the training data with the following parameters:
2502
+
2503
+ **Task**
2504
+
2505
+ Train the ridge model on the training data with the following parameters:
2506
+ # `alpha = 1`
2507
+ # `solver = 'saga'`
2508
+ # `tol = 1e-3`
2509
+ # `random_state = 42`
2510
+
2511
+ # Enter the value of $R^2$ score on the test dataset.
2512
+
2513
+ **Step-by-step approach**
2514
+
2515
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2516
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2517
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2518
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2519
+
2520
+ **Solution**
2521
+
2522
+ ```python
2523
+ # Solution
2524
+ from sklearn.linear_model import Ridge
2525
+ from sklearn.metrics import mean_absolute_error, mean_squared_error
2526
+ ridge = Ridge(alpha=1, solver='saga', tol=1e-3, random_state=42).fit(X_train, y_train)
2527
+ print(ridge.score(X_test, y_test))
2528
+ ```
2529
+
2530
+ **Recorded result**
2531
+
2532
+ ```text
2533
+ 0.4134746439199958
2534
+ ```
2535
+
2536
+ ---
2537
+
2538
+ ### : What is the best **alpha**
2539
+
2540
+ **Task**
2541
+
2542
+ : What is the best **alpha**?
2543
+
2544
+ **Step-by-step approach**
2545
+
2546
+ 1. Identify the input columns, target (if any), and exact output requested.
2547
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
2548
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
2549
+
2550
+ **Solution**
2551
+
2552
+ ```python
2553
+ # Solution
2554
+ print(sgd_grid.best_params_['alpha'])
2555
+ ```
2556
+
2557
+ **Recorded result**
2558
+
2559
+ ```text
2560
+ 1
2561
+ ```
2562
+
2563
+ ---
2564
+
2565
+ ### : What will be value of **root mean squared error** on the test dataset
2566
+
2567
+ **Task**
2568
+
2569
+ : What will be value of **root mean squared error** on the test dataset? Enter your answer correct to two decimal places.
2570
+
2571
+ **Step-by-step approach**
2572
+
2573
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2574
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2575
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2576
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2577
+
2578
+ **Solution**
2579
+
2580
+ ```python
2581
+ # Solution
2582
+ print(round(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))), 2))
2583
+ ```
2584
+
2585
+ **Recorded result**
2586
+
2587
+ ```text
2588
+ 3580.16
2589
+ ```
2590
+
2591
+ ---
2592
+
2593
+ ### : If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset
2594
+
2595
+ **Task**
2596
+
2597
+ : If we fit the pipeline on the training dataset, what will be the **mean absolute error** on the test dataset? Enter your answer correct to two decimal places.
2598
+
2599
+ **Step-by-step approach**
2600
+
2601
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2602
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2603
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2604
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2605
+
2606
+ **Solution**
2607
+
2608
+ ```python
2609
+ # Solution
2610
+ print(round(mean_absolute_error(y_test, pca_grid.predict(X_test)), 2))
2611
+ ```
2612
+
2613
+ **Recorded result**
2614
+
2615
+ ```text
2616
+ 3630.4
2617
+ ```
2618
+
2619
+ ---
2620
+
2621
+ ### : What **fraction** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model
2622
+
2623
+ **Task**
2624
+
2625
+ : What **fraction** of the variance is explained by the first principal component (`explained_variance_ratio_`) of the best model? Enter your answer correct to two decimal places.
2626
+
2627
+ **Step-by-step approach**
2628
+
2629
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
2630
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
2631
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
2632
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
2633
+
2634
+ **Solution**
2635
+
2636
+ ```python
2637
+ # Solution
2638
+ print(round(pca_grid.best_estimator_.named_steps['pca'].explained_variance_ratio_[0], 2))
2639
+ ```
2640
+
2641
+ **Recorded result**
2642
+
2643
+ ```text
2644
+ 0.83
2645
+ ```
2646
+
2647
+ ---
2648
+
2649
+ ### : create a pipeline of the PolynomialFeatures() as transformer and Lasso as an estimator with the following parameters:
2650
+
2651
+ **Task**
2652
+
2653
+ : create a pipeline of the PolynomialFeatures() as transformer and Lasso as an estimator with the following parameters:
2654
+ - For PolynomialFeatures()
2655
+ - interaction_only = False
2656
+ - degree = 3
2657
+ - For lasso()
2658
+ - alpha = 50
2659
+ - warm_start = True
2660
+
2661
+ Fit the pipeline on the training dataset and find the $R^2$ score on the test dataset.
2662
+
2663
+ **Step-by-step approach**
2664
+
2665
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2666
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2667
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2668
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2669
+
2670
+ **Solution**
2671
+
2672
+ ```python
2673
+ # Solution
2674
+ from sklearn.preprocessing import PolynomialFeatures
2675
+ poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=3, interaction_only=False)),
2676
+ ('lasso', Lasso(alpha=50, warm_start=True))])
2677
+ poly_pipe.fit(X_train, y_train)
2678
+ print(poly_pipe.score(X_test, y_test))
2679
+ ```
2680
+
2681
+ **Recorded result**
2682
+
2683
+ ```text
2684
+ 0.5260776843902365
2685
+ ```
2686
+
2687
+ ---
2688
+
2689
+ ## Insurance premium records
2690
+
2691
+ ### Which of the following columns have **numerical** (int/float) datatype
2692
+
2693
+ **Task**
2694
+
2695
+ Which of the following columns have **numerical** (int/float) datatype?
2696
+
2697
+ **Step-by-step approach**
2698
+
2699
+ 1. Identify the input columns, target (if any), and exact output requested.
2700
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
2701
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
2702
+
2703
+ **Solution**
2704
+
2705
+ ```python
2706
+ # Solution
2707
+ print(df.dtypes)
2708
+ ```
2709
+
2710
+ **Recorded result**
2711
+
2712
+ ```text
2713
+ Age float64
2714
+ Gender object
2715
+ Annual Income float64
2716
+ Number of Dependents float64
2717
+ Education Level object
2718
+ Health Score float64
2719
+ Location object
2720
+ Policy Type object
2721
+ Active Policy days int64
2722
+ Previous Claims float64
2723
+ Credit Score float64
2724
+ Insurance Duration float64
2725
+ Customer Feedback object
2726
+ Smoking Status object
2727
+ Exercise Frequency object
2728
+ Property Type object
2729
+ Premium Amount float64
2730
+ dtype: object
2731
+ ```
2732
+
2733
+ ---
2734
+
2735
+ ### In this dataset, what is the most common **Policy Type** among customers **aged** 35 or below, living in **Suburban** locations and…
2736
+
2737
+ **Task**
2738
+
2739
+ In this dataset, what is the most common **Policy Type** among customers **aged** 35 or below, living in **Suburban** locations and having a **Credit Score** of 600 or above?
2740
+
2741
+ (A) Basic\
2742
+ (B) Comprehensive\
2743
+ (C) Premium\
2744
+ (D) Standard
2745
+
2746
+ V1: B\
2747
+ V2: A\
2748
+ V3: A
2749
+
2750
+ **Step-by-step approach**
2751
+
2752
+ 1. Translate the wording into a pandas operation: filter first when conditions are present, then aggregate or count.
2753
+ 2. Use boolean masks with parentheses, .between() for inclusive ranges, groupby() for group summaries, and nlargest()/nsmallest() for ranked rows.
2754
+ 3. Check whether missing rows should be excluded and whether interval endpoints are inclusive.
2755
+ 4. Convert a scalar count/result to int when a clean numeric answer is needed.
2756
+
2757
+ **Solution**
2758
+
2759
+ ```python
2760
+ # Solution
2761
+ sub = df[(df['Age'] <= 35) & (df['Location'] == 'Suburban') & (df['Credit Score'] >= 600)]
2762
+ print(sub['Policy Type'].value_counts())
2763
+ ```
2764
+
2765
+ **Recorded result**
2766
+
2767
+ ```text
2768
+ Policy Type
2769
+ Comprehensive 203
2770
+ Basic 200
2771
+ Premium 192
2772
+ Name: count, dtype: int64
2773
+ ```
2774
+
2775
+ ---
2776
+
2777
+ ### Which of the following options represent all the unique categories present in the **Exercise Frequency** feature
2778
+
2779
+ **Task**
2780
+
2781
+ Which of the following options represent all the unique categories present in the **Exercise Frequency** feature?
2782
+
2783
+ (A) ['Weekly', 'Rarely', 'Monthly', 'Daily', 'Never']
2784
+
2785
+ (B) ['Weekly', 'Monthly', 'Daily']
2786
+
2787
+ (C) ['Weekly', 'Rarely', 'Monthly', 'Daily']
2788
+
2789
+ (D) ['Regularly', 'Rarely', 'Monthly', 'Weekly']
2790
+
2791
+ (E) None of these
2792
+
2793
+ All Versions: C
2794
+
2795
+ **Step-by-step approach**
2796
+
2797
+ 1. Identify the input columns, target (if any), and exact output requested.
2798
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
2799
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
2800
+
2801
+ **Solution**
2802
+
2803
+ ```python
2804
+ # Solution
2805
+ print(df['Exercise Frequency'].unique())
2806
+ ```
2807
+
2808
+ **Recorded result**
2809
+
2810
+ ```text
2811
+ ['Rarely' 'Weekly' 'Daily' 'Monthly']
2812
+ ```
2813
+
2814
+ ---
2815
+
2816
+ ### Create feature matrix(X) and label vector(y) using following instructions:
2817
+
2818
+ **Task**
2819
+
2820
+ Create feature matrix(X) and label vector(y) using following instructions:
2821
+ * Premium Amount is the target column (y).
2822
+ * All the columns except the target column are part of the feature matrix (X).
2823
+
2824
+ Compare the correlation values for the following feature pairs of the feature matrix (X) and among the options select the one with the **strongest positive** correlation.
2825
+
2826
+ (A) `Annual Income` vs `Age`\
2827
+ (B) `Active Policy days` vs `Insurance Duration`\
2828
+ (C) `Previous Claims` vs `Credit Score`\
2829
+ (D) `Annual Income` vs `Credit Score`\
2830
+ (E) `Health Score` vs `Number of Dependents`
2831
+
2832
+ All Versions: C
2833
+
2834
+ **Step-by-step approach**
2835
+
2836
+ 1. Select only the numeric columns before calculating correlations; strings and categories should not be treated as continuous measurements.
2837
+ 2. Calculate the requested correlation matrix and ignore its diagonal, because every feature is perfectly correlated with itself.
2838
+ 3. For the most negative pair, sort the off-diagonal values in ascending order; draw the heatmap only when a visual is requested.
2839
+
2840
+ **Solution**
2841
+
2842
+ ```python
2843
+ # Solution
2844
+ X = df.drop(columns=['Premium Amount'])
2845
+ y = df['Premium Amount']
2846
+ corr = X.select_dtypes(include='number').corr()
2847
+ pairs = [('Annual Income', 'Age'), ('Active Policy days', 'Insurance Duration'),
2848
+ ('Previous Claims', 'Credit Score'), ('Annual Income', 'Credit Score'),
2849
+ ('Health Score', 'Number of Dependents')]
2850
+ for a, b in pairs:
2851
+ print(f'{a} vs {b}: {corr.loc[a, b]:.4f}')
2852
+ ```
2853
+
2854
+ **Recorded result**
2855
+
2856
+ ```text
2857
+ Annual Income vs Age: 0.0250
2858
+ Active Policy days vs Insurance Duration: -0.0285
2859
+ Previous Claims vs Credit Score: 0.0433
2860
+ Annual Income vs Credit Score: -0.2016
2861
+ Health Score vs Number of Dependents: 0.0046
2862
+ ```
2863
+
2864
+ ---
2865
+
2866
+ ### Which of the following columns have missing values
2867
+
2868
+ **Task**
2869
+
2870
+ Which of the following columns have missing values?
2871
+
2872
+ **Step-by-step approach**
2873
+
2874
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2875
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2876
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2877
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2878
+
2879
+ **Solution**
2880
+
2881
+ ```python
2882
+ # Solution
2883
+ print(df.isna().sum())
2884
+ ```
2885
+
2886
+ **Recorded result**
2887
+
2888
+ ```text
2889
+ Age 161
2890
+ Gender 0
2891
+ Annual Income 344
2892
+ Number of Dependents 884
2893
+ Education Level 0
2894
+ Health Score 616
2895
+ Location 0
2896
+ Policy Type 0
2897
+ Active Policy days 0
2898
+ Previous Claims 0
2899
+ Credit Score 1137
2900
+ Insurance Duration 0
2901
+ Customer Feedback 644
2902
+ Smoking Status 0
2903
+ Exercise Frequency 0
2904
+ Property Type 0
2905
+ Premium Amount 0
2906
+ dtype: int64
2907
+ ```
2908
+
2909
+ ---
2910
+
2911
+ ### Split the dataset into train dataset and test dataset in the following manner
2912
+
2913
+ **Task**
2914
+
2915
+ Split the dataset into train dataset and test dataset in the following manner.
2916
+
2917
+ - Use sklearn train_test_split function to split the data.
2918
+ - Use only **25%** data as test_set and keep random_state = 42
2919
+
2920
+ How many **test** labels (**y_test**) are considered outliers? Outliers are defined as values that exceed the upper bound (Third Quartile + 1.5 x Interquartile Range) or fall below the lower bound (First Quartile - 1.5 x Interquartile Range). Compute the quartiles on **y_test** itself.
2921
+
2922
+ V1: 117\
2923
+ V2: 113\
2924
+ V3: 126
2925
+
2926
+ **Step-by-step approach**
2927
+
2928
+ 1. Identify the target column and remove it from the feature matrix before splitting.
2929
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
2930
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
2931
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
2932
+
2933
+ **Solution**
2934
+
2935
+ ```python
2936
+ # Solution
2937
+ from sklearn.model_selection import train_test_split
2938
+ X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
2939
+ q1, q3 = y_test.quantile(0.25), y_test.quantile(0.75)
2940
+ iqr = q3 - q1
2941
+ print(((y_test < q1 - 1.5 * iqr) | (y_test > q3 + 1.5 * iqr)).sum())
2942
+ ```
2943
+
2944
+ **Recorded result**
2945
+
2946
+ ```text
2947
+ 117
2948
+ ```
2949
+
2950
+ ---
2951
+
2952
+ ### Write the **MEAN** value of the **Age** column of the train dataset you found to replace all the Missing Values (NaN)
2953
+
2954
+ **Task**
2955
+
2956
+ Write the **MEAN** value of the **Age** column of the train dataset you found to replace all the Missing Values (NaN). (upto 1 digit after decimal point)
2957
+
2958
+ **Step-by-step approach**
2959
+
2960
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2961
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2962
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2963
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2964
+
2965
+ **Solution**
2966
+
2967
+ ```python
2968
+ # Solution
2969
+ print(round(fill_values['Age'], 1))
2970
+ ```
2971
+
2972
+ **Recorded result**
2973
+
2974
+ ```text
2975
+ 41.3
2976
+ ```
2977
+
2978
+ ---
2979
+
2980
+ ### Write the value you used to replace the missing values in the **Number of Dependents** feature of the train dataset
2981
+
2982
+ **Task**
2983
+
2984
+ Write the value you used to replace the missing values in the **Number of Dependents** feature of the train dataset
2985
+ (A) 0\
2986
+ (B) 1\
2987
+ (C) 2\
2988
+ (D) 3\
2989
+ (E) 4\
2990
+ (F) None of these
2991
+
2992
+ **Step-by-step approach**
2993
+
2994
+ 1. Inspect both true nulls and the distinct values in the feature. A numeric sentinel such as -1 or a string such as UNKNOWN is not counted by isna() until it is converted.
2995
+ 2. Create one boolean mask per invalid-value rule and count each mask before changing the data.
2996
+ 3. Replace confirmed sentinels with numpy.nan, then calculate the imputation statistic from the training partition only when a split is involved.
2997
+ 4. Apply the same fitted statistic or transformer to the test partition; never learn it from test data.
2998
+
2999
+ **Solution**
3000
+
3001
+ ```python
3002
+ # Solution
3003
+ print(fill_values['Number of Dependents'])
3004
+ ```
3005
+
3006
+ **Recorded result**
3007
+
3008
+ ```text
3009
+ 3.0
3010
+ ```
3011
+
3012
+ ---
3013
+
3014
+ ### Calculate the sum of all the values present in the **first 50 rows** of the transformed test feature matrix
3015
+
3016
+ **Task**
3017
+
3018
+ Calculate the sum of all the values present in the **first 50 rows** of the transformed test feature matrix ? (upto 2 digits after the decimal)
3019
+
3020
+ ${\text{V1 : 405.34 (Range: 403, 408)}}$\
3021
+ ${\text{V2 : 399.38 (Range: 397, 402)}}$\
3022
+ ${\text{V3 : 419.33 (Range: 417, 422)}}$
3023
+
3024
+ **Step-by-step approach**
3025
+
3026
+ 1. Identify the input columns, target (if any), and exact output requested.
3027
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
3028
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
3029
+
3030
+ **Solution**
3031
+
3032
+ ```python
3033
+ # Solution
3034
+ print(round(X_test_t[:50].sum(), 2))
3035
+ ```
3036
+
3037
+ **Recorded result**
3038
+
3039
+ ```text
3040
+ 405.34
3041
+ ```
3042
+
3043
+ ---
3044
+
3045
+ ### Enter the value of **Mean Absolute Error** on the test dataset
3046
+
3047
+ **Task**
3048
+
3049
+ Enter the value of **Mean Absolute Error** on the test dataset.
3050
+
3051
+ **Step-by-step approach**
3052
+
3053
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3054
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3055
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3056
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3057
+
3058
+ **Solution**
3059
+
3060
+ ```python
3061
+ # Solution
3062
+ print(mean_absolute_error(y_test, ridge.predict(X_test)))
3063
+ ```
3064
+
3065
+ **Recorded result**
3066
+
3067
+ ```text
3068
+ 666.013227845856
3069
+ ```
3070
+
3071
+ ---
3072
+
3073
+ ### What is the **highest absolute coefficient value** of the trained Ridge model
3074
+
3075
+ **Task**
3076
+
3077
+ What is the **highest absolute coefficient value** of the trained Ridge model? (Ignore Intercept) (upto 2 digits after the decimal)
3078
+
3079
+ **Step-by-step approach**
3080
+
3081
+ 1. Fit the requested estimator/selector on the training feature matrix only.
3082
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
3083
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
3084
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
3085
+
3086
+ **Solution**
3087
+
3088
+ ```python
3089
+ # Solution
3090
+ print(np.round(np.abs(ridge.coef_), 3))
3091
+ print(round(np.abs(ridge.coef_).max(), 2))
3092
+ ```
3093
+
3094
+ **Recorded result**
3095
+
3096
+ ```text
3097
+ [ 9.474 4.219 3.804 7.92 13.654 27.433 13.221 4.76 8.723 6.265
3098
+ 6.321 0.711 1.628 1.783 18.109 11.759 0.085 13.013 36.555 9.772]
3099
+ 36.55
3100
+ ```
3101
+
3102
+ ---
3103
+
3104
+ ### Identify the index of the feature with the lowest absolute coefficient value
3105
+
3106
+ **Task**
3107
+
3108
+ Identify the index of the feature with the lowest absolute coefficient value. Note that index starts from 0. (Ignore Intercept)
3109
+
3110
+ **Step-by-step approach**
3111
+
3112
+ 1. Fit the requested estimator/selector on the training feature matrix only.
3113
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
3114
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
3115
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
3116
+
3117
+ **Solution**
3118
+
3119
+ ```python
3120
+ # Solution
3121
+ print(np.argmin(np.abs(ridge.coef_)))
3122
+ ```
3123
+
3124
+ **Recorded result**
3125
+
3126
+ ```text
3127
+ 16
3128
+ ```
3129
+
3130
+ ---
3131
+
3132
+ ### What is the best **alpha**
3133
+
3134
+ **Task**
3135
+
3136
+ What is the best **alpha**?
3137
+
3138
+ **Step-by-step approach**
3139
+
3140
+ 1. Identify the input columns, target (if any), and exact output requested.
3141
+ 2. Inspect the relevant values and data types before choosing a pandas or scikit-learn operation.
3142
+ 3. Execute the solution, inspect the shape or fitted attributes, and report only the requested result.
3143
+
3144
+ **Solution**
3145
+
3146
+ ```python
3147
+ # Solution
3148
+ print(sgd_grid.best_params_['alpha'])
3149
+ ```
3150
+
3151
+ **Recorded result**
3152
+
3153
+ ```text
3154
+ 0.1
3155
+ ```
3156
+
3157
+ ---
3158
+
3159
+ ### What will be value of **root mean squared error** on the test dataset
3160
+
3161
+ **Task**
3162
+
3163
+ What will be value of **root mean squared error** on the test dataset?
3164
+
3165
+ **Step-by-step approach**
3166
+
3167
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3168
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3169
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3170
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3171
+
3172
+ **Solution**
3173
+
3174
+ ```python
3175
+ # Solution
3176
+ print(np.sqrt(mean_squared_error(y_test, sgd_grid.predict(X_test))))
3177
+ ```
3178
+
3179
+ **Recorded result**
3180
+
3181
+ ```text
3182
+ 872.3850595099008
3183
+ ```
3184
+
3185
+ ---
3186
+
3187
+ ### If we fit the pipeline on the training dataset, what will be the **Mean Absolute Error** on the test dataset
3188
+
3189
+ **Task**
3190
+
3191
+ If we fit the pipeline on the training dataset, what will be the **Mean Absolute Error** on the test dataset?
3192
+
3193
+ **Step-by-step approach**
3194
+
3195
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3196
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3197
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3198
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3199
+
3200
+ **Solution**
3201
+
3202
+ ```python
3203
+ # Solution
3204
+ print(mean_absolute_error(y_test, pca_grid.predict(X_test)))
3205
+ ```
3206
+
3207
+ **Recorded result**
3208
+
3209
+ ```text
3210
+ 665.4809335203206
3211
+ ```
3212
+
3213
+ ---
3214
+
3215
+ ### How many principal components are retained by the PCA step of the **best** pipeline found above
3216
+
3217
+ **Task**
3218
+
3219
+ How many principal components are retained by the PCA step of the **best** pipeline found above?
3220
+
3221
+ **Step-by-step approach**
3222
+
3223
+ 1. Fit PCA only on the training feature matrix (or inside a Pipeline when using cross-validation).
3224
+ 2. For a fractional n_components value, inspect the fitted n_components_ to see how many components were retained.
3225
+ 3. Use explained_variance_ratio_ for variance fractions and explained_variance_ for component eigenvalues; they are different quantities.
3226
+ 4. Transform test data with the same fitted PCA object and evaluate the downstream estimator on that transformed matrix.
3227
+
3228
+ **Solution**
3229
+
3230
+ ```python
3231
+ # Solution
3232
+ print(pca_grid.best_estimator_.named_steps['pca'].n_components_)
3233
+ ```
3234
+
3235
+ **Recorded result**
3236
+
3237
+ ```text
3238
+ 15
3239
+ ```
3240
+
3241
+ ---
3242
+
3243
+ ### Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
3244
+
3245
+ **Task**
3246
+
3247
+ Create a pipeline of the PolynomialFeatures as transformer and Lasso as an estimator with the following parameters:
3248
+ - For PolynomialFeatures:
3249
+ - interaction_only = False
3250
+ - degree = 2
3251
+ - For Lasso:
3252
+ - alpha = 2
3253
+ - warm_start = True
3254
+ - random state as 0
3255
+
3256
+ Fit the pipeline on the training dataset and find the **Root Mean Square Error** on the test dataset.
3257
+
3258
+ **Step-by-step approach**
3259
+
3260
+ 1. Identify the target column and remove it from the feature matrix before splitting.
3261
+ 2. Copy the exact test_size, random_state, and any shuffle/stratify requirements from the task.
3262
+ 3. Keep X and y paired in the same train_test_split call, and check the resulting shapes.
3263
+ 4. Do all learned preprocessing on X_train; call transform (not fit_transform) on X_test.
3264
+
3265
+ **Solution**
3266
+
3267
+ ```python
3268
+ # Solution
3269
+ from sklearn.preprocessing import PolynomialFeatures
3270
+ poly_pipe = Pipeline([('poly', PolynomialFeatures(degree=2, interaction_only=False)),
3271
+ ('lasso', Lasso(alpha=2, warm_start=True, random_state=0))])
3272
+ poly_pipe.fit(X_train, y_train)
3273
+ print(np.sqrt(mean_squared_error(y_test, poly_pipe.predict(X_test))))
3274
+ ```
3275
+
3276
+ **Recorded result**
3277
+
3278
+ ```text
3279
+ 872.5174524584622
3280
+ ```
3281
+
3282
+ ---
3283
+
3284
+ ### Create a pipeline for feature selection using f_regression as transformer and SGDRegressor as an estimator with the following…
3285
+
3286
+ **Task**
3287
+
3288
+ Create a pipeline for feature selection using f_regression as transformer and SGDRegressor as an estimator with the following parameters:
3289
+ - For feature selection:
3290
+ - select best **4** features based on f_regression score
3291
+ - For SGDRegressor:
3292
+ - random state as 42
3293
+
3294
+ Fit the pipeline on the training dataset and select the option that correctly displays the selected feature names in descending order based on their f_regression scores.
3295
+
3296
+ (A) ['Previous Claims', 'Location_Urban', 'Annual Income', 'Active Policy days']\
3297
+ (B) ['Previous Claims', 'Annual Income', 'Property Type_House', 'Customer Feedback_Good']\
3298
+ (C) ['Previous Claims', 'Policy Type_Comprehensive', 'Annual Income', 'Health Score'] \
3299
+ (D) ['Previous Claims', 'Annual Income', 'Location_Urban', 'Education Level'] \
3300
+ (E) None of the Above
3301
+
3302
+ **Step-by-step approach**
3303
+
3304
+ 1. Fit the requested estimator/selector on the training feature matrix only.
3305
+ 2. For coefficients, exclude the intercept and compare absolute coefficient magnitudes; argmax/argmin return zero-based feature positions.
3306
+ 3. For selectors, use get_support(), scores_, pvalues_ or ranking_ to recover selected features and their order.
3307
+ 4. Map positions back to the exact input or transformed feature names, preserving ColumnTransformer output order.
3308
+
3309
+ **Solution**
3310
+
3311
+ ```python
3312
+ # Solution
3313
+ from sklearn.feature_selection import SelectKBest, f_regression
3314
+ fs_pipe = Pipeline([('kbest', SelectKBest(f_regression, k=4)), ('sgd', SGDRegressor(random_state=42))])
3315
+ fs_pipe.fit(X_train, y_train)
3316
+ kb = fs_pipe.named_steps['kbest']
3317
+ top = np.argsort(kb.scores_)[::-1][:4]
3318
+ print(list(X_train.columns[top]))
3319
+ ```
3320
+
3321
+ **Recorded result**
3322
+
3323
+ ```text
3324
+ ['Previous Claims', 'Location_Urban', 'Annual Income', 'Active Policy days']
3325
+ ```
3326
+
3327
+ ---
3328
+
3329
+ ### For SGDRegressor, what is the actual number of iterations completed during model training
3330
+
3331
+ **Task**
3332
+
3333
+ For SGDRegressor, what is the actual number of iterations completed during model training?
3334
+
3335
+ **Step-by-step approach**
3336
+
3337
+ 1. Instantiate the specified estimator and preserve every stated parameter; leave unspecified parameters at their defaults.
3338
+ 2. Fit the estimator on X_train and y_train, then call predict(X_test) once to obtain held-out predictions.
3339
+ 3. Calculate the requested metric with the matching sklearn.metrics function or the estimator score method.
3340
+ 4. Use fitted attributes such as coef_, intercept_, n_iter_ and best_params_ when the requested result describes the fitted model.
3341
+
3342
+ **Solution**
3343
+
3344
+ ```python
3345
+ # Solution
3346
+ print(fs_pipe.named_steps['sgd'].n_iter_)
3347
+ ```
3348
+
3349
+ **Recorded result**
3350
+
3351
+ ```text
3352
+ 16
3353
+ ```
3354
+
3355
+ ---