Evaluating regression performance

So far, we've covered in depth four popular regression algorithms and implemented them from scratch and by using several prominent libraries. Instead of judging how well a model works on testing sets by printing out the prediction, we need to evaluate its performance by the following metrics which give us better insight:

  • The MSE, as we mentioned, measures the squared loss corresponding to the expected value. Sometimes the square root is taken on top of the MSE in order to convert the value back into the original scale of the target variable being estimated. This yields the root mean squared error (RMSE).
  • The mean absolute error (MAE) on the other hand measures the absolute loss. It uses the same scale as the target variable and gives an idea of how close predictions are to the actual values.
For both the MSE and MAE, the smaller value, the better regression model.
  • R2 (pronounced as r squared) indicates the goodness of the fit of a regression model. It ranges from 0 to 1, meaning from no fit to perfect prediction. 

Let's compute these three measurements on a linear regression model using corresponding functions from scikit-learn:

  1. We re-work on the diabetes dataset and fine-tune the parameters of linear regression model using the grid search technique:
>>> diabetes = datasets.load_diabetes()
>>> num_test = 30 # the last 30 samples as testing set
>>> X_train = diabetes.data[:-num_test, :]
>>> y_train = diabetes.target[:-num_test]
>>> X_test = diabetes.data[-num_test:, :]
>>> y_test = diabetes.target[-num_test:]
>>> param_grid = {
... "alpha": [1e-07, 1e-06, 1e-05],
... "penalty": [None, "l2"],
... "eta0": [0.001, 0.005, 0.01],
... "n_iter": [300, 1000, 3000]
... }
>>> from sklearn.model_selection import GridSearchCV
>>> regressor = SGDRegressor(loss='squared_loss',
learning_rate='constant')
>>> grid_search = GridSearchCV(regressor, param_grid, cv=3)
  1. We obtain the optimal set of parameters:
>>> grid_search.fit(X_train, y_train)
>>> print(grid_search.best_params_)
{'penalty': None, 'alpha': 1e-05, 'eta0': 0.01, 'n_iter': 300}
>>> regressor_best = grid_search.best_estimator_
  1. We predict the testing set with the optimal model:
>>> predictions = regressor_best.predict(X_test)
  1. We evaluate the performance on testing sets based on the MSE, MAE, and R2 metrics:
>>> from sklearn.metrics import mean_squared_error, 
mean_absolute_error, r2_score
>>> mean_squared_error(y_test, predictions)
1862.0518552093429
>>> mean_absolute_error(y_test, predictions)
34.605923224169558
>>> r2_score(y_test, predictions)
0.63859162277753756
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
3.138.105.215