Predictive Maintenance using PySpark

April 28, 2017

Report Abuse
A collection of three Jupyter notebooks that demonstrate end-to-end data science workflow for predictive maintenance using PySpark.
Predictive maintenance is one of the most common machine learning use cases and with the latest advancements in information technology, the volume of stored data is growing faster in this domain than ever before which makes it necessary to leverage big data analytic capabilities to efficiently transform large amounts of data into business intelligence. Microsoft has published a series of learning materials including blogs, solution templates, modeling guides and sample tutorials in the domain of predictive maintenance. In this tutorial, we extended those materials by providing a detailed step-by-step process of using Spark Python API [PySpark]( to demonstrate how to approach predictive maintenance for big data scenarios. The tutorial covers typical data science steps such as data ingestion, cleansing, feature engineering and model development. <br> **Business Scenario and Data** The input data is simulated to reflect features that are generic for most of the predictive maintenance scenarios. To enable the tutorial to be completed very quickly, the data was simulated to be around 1.3 GB but the same PySpark framework can be easily applied to a much larger data set. The data is hosted on a publicly accessible Azure Blob Storage container and can be downloaded from [here]( In this tutorial, we import the data directly from the blob storage. The data set has around 2 million records with 172 columns simulated for 1900 machines collected over 4 years. Each machine includes a device which stores data such as warnings, problems and errors generated by the machine over time. Each record has a Device ID and time stamp for each day and aggregated features for that day such as total number of a certain type of warning received in a day. Four categorical columns were also included to demonstrate generic handling of categorical variables. The goal is to predict if a machine will fail in the next 7 days. The last column of the data set indicates if a failure occurred and reported on that day. <br> **Jupyter Notebooks** There are three Jupyter Notebooks on the GitHub repository. To visit the repository, click the green "View Tutorial" button at right. - Notebook_1_DataCleansing_FeatureEngineering - Notebook_2_FeatureEngineering_RollingCompute - Notebook_3_Labeling_FeatureSelection_Modeling We formatted this tutorial as Jupyter notebooks because it is easy to show the step-by-step process this way. You can also easily compile the executable PySpark script(s) using your favorite IDE. <br> **Hardware Specifications** The hardware used in this tutorial is a [Linux Data Science Virtual Machine]( with 32 cores and 448 GB memory. For more detailed information of the Data Science Virtue Machine, please visit the link. For the size of the data used in this tutorial (1.3 GB), a machine with less cores and memory would also be adequate. However, in real life scenarios, one should choose the hardware configuration that is appropriate for the specific big data use case. Jupyter Notebooks included in this tutorial can also be downloaded and run on any machine that has PySpark enabled. <br> **Spark Configuration** The Spark version installed on the Linux Data Science Virtual Machine for this tutorial is **2.0.2** with Python version **2.7.5**. Here are some configurations that needs to be performed before running this tutorial on a Linux machine. <br> 1. For standalone Spark, driver is the executor. The default memory for executor is 5g. This needs to be manually changed in "spark-defaults.conf" by using the following commands from the Linux Terminal: <pre><code>cd /dsvm/tools/spark/current/conf sudo cp spark-defaults.conf.template spark-defaults.conf sudo vi spark-defaults.conf</code></pre> <br> 2. If using a machine with many cores (e.g.32), you will usually encounter the error "spark job failed with too many open files". This is because the default soft limit is 1024. It is recommended to increase ulimit to 64K. You can configure ulimit using the following commands: <pre><code>sudo vi /etc/security/limits.conf</code></pre> Add following lines into the configuration file, logout to make the change effective, then login again: <pre><code>* soft nofile 65536 * hard nofile 65536</code></pre> <br> 3. Sometimes you might encounter the error of job failed with no space left on device. It is because by default Spark uses /tmp directory to store intermediate data. To solve this problem, add the following line to the spark-defaults.conf <pre><code>spark.local.dir SOME/DIR/WHERE/YOU/HAVE/SPACE</code></pre> <br> **Prerequisites** 1. The user should already know some basics of PySpark. This is not meant to be a PySpark 101 tutorial. 2. Have PySpark (Spark 2.0., Python 2.7) already configured. Please note if you are using Python 3 on your machine, a few functions in this tutorial require some very minor tweaks because some Python 2 functions deprecated in Python 3. <br> **References** 1. 2. 3. 4. 5. 6. <br> **Acknowledgement** Special thanks to Said Bleik, Yiyu Chen and Ke Huang for learning PySpark together. Thank Fidan Boylu Uz for proof reading and modifying the tutorial materials.