Analyzing data streams in real-time is a problem that a lot of businesses can relate to. In this fast paced digital era, enterprises depend on making quick and intelligent decisions. This requires them to analyze data streams from various sources in real time. For example, they might want to monitor weather data from a sensor, financial data from the stock market or social media reaction to a newly launched product or campaign. We show you how to address this using a Cortana Intelligence Solution, which sets up an end-to-end pipeline to ingest Twitter feeds based on a user defined set of keywords, analyze their sentiment using an Azure Machine Learning based webservice and visualize the predicted sentiment in a Power BI dashboard.
> **Note:** A newer version is available [**here**](https://gallery.cortanaintelligence.com/Tutorial/Stream-Analysis-with-Azure-ML-3). > **Note:** If you have already deployed this solution, click [here](https://start.cortanaintelligence.com/Deployments?type=streamanalysiswithml) to view your deployment. ### Estimated Provisioning Time: 10 Minutes [![Solution Diagram](https://caqsres.blob.core.windows.net/streamanalysiswithml/StreamAnalysisWithMLDiagram.JPG)](https://caqsres.blob.core.windows.net/streamanalysiswithml/StreamAnalysisWithMLDiagram.JPG) This solution sets up the infrastructure in the diagram above. The various steps are as follows: * Setting up an Azure WebJob to collect Twitter data based on user specified keywords. * Pumping ingested tweets into Azure Event Hub which can accept millions of events per second * Processing incoming tweets with an Azure Stream Analytics job that stores the raw data in Azure Blob Storage. In addition, the Stream Analytics job calls an Azure Machine Learning web service to determine the sentiment of each tweet. * Visualizing real-time metrics about inferred sentiment using Power BI. ## Prerequisites This solution sets up an [Azure Web Job](https://azure.microsoft.com/en-us/documentation/articles/websites-webjobs-resources/) called Twitter Client to collect tweets. To run the TwitterClient web job, you will need: 1. A [Twitter account](https://twitter.com/login) 2. A [Twitter application](https://apps.twitter.com) 3. Twitter's Streaming API OAuth credentials - On the Twitter application page, click on the *Keys and Access Tokens* tab - *Consumer Key (API Key)* and *Consumer Secret (API Secret)* can be found under **Application Settings** section - Under **Your Access Token** section, click on *Create my access token* to obtain both *Access Token* and *Access Token Secret* More details on Twitter's Streaming API OAuth access token can be found [here](https://dev.twitter.com/oauth/overview/application-owner-access-tokens).