Building a Regression Model to Predict Real Estate Sales Price

September 1, 2016
Predict the real estate sales price of a house based upon various quantitative features about the house and sale.
#Follow Along A fully illustrated version of this tutorial can be found [here. Follow along there]( [![Foo]( )]( #Objective Predict the real estate sales price of a house based upon various features about the house and the sales transaction. #Data [Ames housing dataset]( includes 81 features and 1460 observations. Each observation represents the sale of a home and each feature is an attribute describing the house or the circumstance of the sale. [Click here for the full list of feature descriptions]( #Initial Feature Selection Some low quality features were removed to improve the model's performance. Low quality includes lack of representative categories, too many missing values, or noisy features. #Categorical Casting [Nominal categorical features]( were identified and cast to categorical data types using the meta data editor to ensure proper mathematical treatment by the machine learning algorithm. #Cleaning Missing Data For simplicity, all categorical missing values were cleaned with the mode and all numeric features were cleaned using the median. To further improve a model's performance, custom cleaning functions should be tried and implemented on each individual feature rather than a blanket transformation of all columns. #Statistical Feature Selection Not every feature in its current form is expected to contain predictive value to the model, and may mislead or add noise to the model. To filter these out we will perform a [Pearson correlation]( to test all features against the response class (sales price) as a quick measure of their predictive strength, only picking the top X strongest features from this method, the remaining features will be left behind. This number can be tuned for further model performance increases. #Algorithm A regularized gradient descent variant of the linear regression model will be used to reduce over-fitting of the model. * To ensure a stable convergence of weight and biases, all features except the response class must be normalized to be placed into the same range. #Evaluation The method of cross validation will be used to evaluate the predictive performance of the model as well as that performance's stability in regard to new data. Cross validation will build ten different models on the same algorithm but with different and nonrepeating subsets of the same dataset. The evaluation metrics on each of the ten models will be averaged and a standard deviation will infer to the stability of the average performance. #Parameter Tuning This experiment will build a model which minimizes mean RMSE of the cross validation results with the lowest variance possible (but also consider bias-variance trade-offs). 1. The first regression model was built using default parameters and produced a very inaccurate model ($124,942 mean RMSE) and was very unstable (11,699 standard deviation). 2. The high bias and high variance of the previous model suggest the model is over-fitting to the outliers and is under-fitting the general population. The L2 regularization weight will be decreased to lower the penalty of higher coefficients. After lowering the L2 regularization weight, the model is more accurate with an average cross validation RMSE of $42,366. 3. The previous model is still quite unstable with a standard deviation of $8,121. Since this is a dataset with a small number of observations (1460), it may be better to increase the number of training epochs so that the algorithm has more passes to reach convergence. This will increase training times but also increase stability.The third linear model had the number of training epochs increased and saw a better mean cross validation RMSE of $36,684 and a much more stable standard deviation of $3,849. 4. The final model had a slight increase in the learning rate which improved both mean cross validation RMSE and the standard deviation. #Deployment The algorithm parameters that yeilded the best results will be the one that is shipped. The best algorithm (the last one) will be retrained using 100% of the data since cross validation leaves 10% out each time for validation. #Further Improve this Model Feature engineering was entirely left out of this experiment. Try engineering more features from the existing dataset to see if the model will improve. Some columns that were originally dropped may become useful when combined with other features. For example try bucketing the years in which the house was built by decade. Clustering the data may also yeild some hidden insights. # Related 1. [Detailed Tutorial: Building and deploying a classification model in Azure Machine Learning Studio]( 2. [Demo: Interact with the user interface of a model deployed as service]( 3. [Tutorial: Creating a random forest regression model in R and using it for scoring]( 4. [Tutorial: Obtaining feature importance using variable importance plots]( [1]: