Views Prediction - A Quora Challenge - Part II (Regression)

Now prepare the scaler. By the way, since we are not composing estimators, we don’t need to use a pipeline.

# RobustScaler
from sklearn.preprocessing import RobustScaler
scaler = RobustScaler(quantile_range=(5,95)).fit(x_train)

Then apply the scaler to both the train and test datasets, excluding the target feature; notice that the target feature is stored in dataframe y.

X_train = scaler.transform(x_train)
X_test = scaler.transform(x_test)

Regression

At this point, we are ready to use regression algorithms.

Decision Tree Regressor

Using the decision tree regressor, our model is created by predicting the value of the target feature by learning simple decision rules inferred from the data set we fit it on. In simpler terms, the model is created by performing a large scale pivoting.

First we will do it on our training set; that is, fitting the regressor with X_train and y_train:

from sklearn.tree import DecisionTreeRegressor

tree_reg = DecisionTreeRegressor()
tree_reg.fit(X_train, y_train)

Then make a prediction of the target feature from X_train.

y_pred_from_train = tree_reg.predict(X_train)

Finally, check the accuracy by calculating Root Mean Squared Error, that is, check how far off the predicted y is from the actual y_train.

from sklearn.metrics import mean_squared_error
import numpy as np
tree_mse = mean_squared_error(y_pred_from_train, y_train)
tree_rmse = np.sqrt(tree_mse)

2.9395110047712159

This sounds too good to be true! Now, do this on the test dataset. Calculate rmse after predicting y using X_test. Recall that the regressor is trained on X_train and has not seen X_test. So, this is the real test of our model.

y_pred_from_test = tree_reg.predict(X_test) #make the prediction
tree_mse_test = mean_squared_error(y_pred_from_test, y_test) #calculate rmse
tree_rmse_test = np.sqrt(tree_mse_test)

40.278871508455829

Okay, the error on the test set is 14 times that of the training set!! This doesn’t look good. But it’s not surprising as Decision Tree Regressor is prone to overfitting. So, it overfit the training set and thus performed badly on an unknown set, i.e., our test set.

Overfitting can be abated using sklearn’s Cross-Validation feature. I will demonstrate how we can perform K-fold cross-validation with K=10.

from sklearn.model_selection import cross_val_score
tree_scores = cross_val_score(tree_reg, X_train, y_train,
                        scoring = "neg_mean_squared_error", cv=10
                        )
tree_rmse_scores = np.sqrt(-tree_scores)

# taken from Aurelien Geron's Hands on ML book
def display_scores(scores):
    print("scores: ", scores)
    print("mean:", scores.mean())
    print("standard deviation:", scores.std())

scores:  [ 29.75540989  56.09513377  44.12899068  73.75383811  39.68194397
      38.37672842  53.38223776  61.47702739  47.48127876  37.58361917]
    mean: 48.1716207909
    standard deviation: 12.4800354025

Linear Regressor

A linear regression model works by computing the target feature (__ans__ here) as a weighted sum of the predictor features (columns of X) plus an intercept term (which the model determines looking at the relationship between the predictors and the target features). Decision Tree, on the other hand, finds complex non-linear relationships in our data that the linear regressor misses.

Now let’s see how well the Linear Regression model does.

from sklearn.linear_model import LinearRegression

lin_reg = LinearRegression()
lin_reg.fit(X_train, y_train)

y_pred_train_lin = lin_reg.predict(X_train)

lin_rmse_train = np.sqrt(mean_squared_error(y_pred_train_lin, y_train))

26.527626721102603

How well does it do on the test set?

y_pred_test_lin = lin_reg.predict(X_test)
lin_rmse_test = np.sqrt(mean_squared_error(y_pred_test_lin, y_test))

36.563500562427755

That’s quite impressive. On the test set, it performed better than the tree regressor. This shows that it’s realiable. Okay, but how badly is it overfitting?

lin_scores = cross_val_score(lin_reg, X_train, y_train,
                        scoring = "neg_mean_squared_error", cv=10
                        )
lin_rmse_scores = np.sqrt(-lin_scores)

scores:  [ 30.05448244  33.26047826  39.80200849  42.4842238   21.47302677
      31.36803597  18.7646719   63.04366609  44.5649017   30.41392853]
    mean: 35.5229423942
    standard deviation: 12.0935723986

Indeed, the Linear Model is not overfitting as bad as the Decision Tree model, and in fact, it also has a lower generalization error.

Random Forest Regressor

Random Forest model is prepared by performing decision tree regression on random subsets of the training data (X_train) and averaging out the predictions (essentially reducing variance). It is a type of ensemble learning algorithm.

from sklearn.ensemble import RandomForestRegressor
forest_reg = RandomForestRegressor()

First fit the regressor with X_train and y_train:

RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
           max_features='auto', max_leaf_nodes=None,
           min_impurity_split=1e-07, min_samples_leaf=1,
           min_samples_split=2, min_weight_fraction_leaf=0.0,
           n_estimators=10, n_jobs=1, oob_score=False, random_state=None,
           verbose=0, warm_start=False)

Then make a prediction of the target feature from X_train.

y_pred_train_forest = forest_reg.predict(X_train)

Finally, check the accuracy by calculating Root Mean Squared Error, that is, check how far off the predicted y is from the actual y_train.

# root mean square error
from sklearn.metrics import mean_squared_error
import numpy as np
forest_mse_train = mean_squared_error(y_pred_from_train, y_train)
forest_rmse_train = np.sqrt(forest_mse)

17.194583863664128

Now, do this on the test dataset. Calculate rmse after predicting y using X_test. Recall that the regressor is trained on X_train and has not seen X_test. So, this is the real test of our model.

y_pred_test_forest = forest_reg.predict(X_test) #make the prediction
forest_mse_test = mean_squared_error(y_pred_from_test, y_test) #calculate rmse
forest_rmse_test = np.sqrt(forest_mse_test)

35.263863915914143

The generalization error is only twice the training error. As expected, since the Random Forest model averages out it’s prediction on subsets of the training set, it is less prone to overfitting than the Decision Tree model which had a generalization error that was 14 times the training error.

Looking Ahead

Regularization

To abate overfitting of our models, constraints known as hyperparameters need to be introduced on the learning algorithms; thus, effectively simplifying our model. One such hyperparameter is regularization.

A hyperparameter can be too constraining, if it’s set to too large a value for instance. In that case, overfitting will certainly be avoided, but the model will be less likely to find a good solution. So, fine tuning the hyperparameters of the Decision Tree Algorithm, for example, is a natural next step.

Richer Features

Another action to improve our models is to, for instance, use the Linear Regressor but with a richer set of features. That is engineering features from the predictor features, including the text features we dropped for this post. This also involves getting rid of irrelevant features (maybe the anonymous series?)

Another Look at the Outliers?

Yet another option is to fix the quality of our data. That is, discarding the outliers or manually fixing them.

One concern here is that the outliers in the data, and the ones we detected above are not actually a result of a measurement or data collection error. Indeed, at the time of collection, any quora question has a certain value of number of questions, number of followers, etc. So, just discarding the anomalies as noise does not feel right. Need to think about this more.

Side Note

That the Linear Regression model performed as good (and even better) as the more complex decision tree model points the same direction as what Michele Banko and Eric Brill showed in their paper The Unreasonable Effectiveness of Data. They found out that fairly simple algorithms were able to perform identically well on a complex ml problem once both models were fed enough data. But don’t be mistaken, our dataset is not really large enough for us to actually reach the same conclusions as they did!

References

1. Scikit-Learn Documentation.
2. Hands-On Machine Learning with Scikit-Learn and TensorFlow by Aurélien Géron.