Azure Databricks Spark Streaming

By for May 18, 2018

Report Abuse
Analyzing data streams in real-time using Azure Databricks Spark Streaming, Azure Event Hubs, Cognitive Services Text Analytics, and Time Series Insights.
> **Important:** This solution will be removed from this gallery on **September 9th 2018**. Please contact us at [**cisolutions@microsoft.com**](mailto:cisolutions@microsoft.com) and reference this solution if you have any questions. > **Note:** If you have already deployed this solution, click [here](https://quickstart.azure.ai/track/Deployments?type=databricks) to view your deployment. ### Estimated Provisioning Time: 15 Minutes [![Solution Diagram](https://raw.githubusercontent.com/Azure/data-ai-iot/master/databricks/assets/databricksTwitter.JPG)](https://raw.githubusercontent.com/Azure/data-ai-iot/master/databricks/assets/databricksTwitter.JPG) This solution sets up the infrastructure in the diagram above. The various steps are as follows: * Setting up an Azure WebJob to collect Twitter data based on user specified keywords. * Ingesting tweets into Azure Event Hub. * Visualizing real-time metrics using Time Series Insights for Operational Technology real-time monitoring. * Stream data with Azure Databricks using Spark Streaming * Text Analytics Cognitive Service to provide sentiment regarding the tweets * Using a second Azure Event Hub to send the Spark Streaming output to Time Series Insights ## Thank you to the following people that inspired, and helped unblock me along the way: * **Andrew Ivanov** for all the work on this Twitter streaming solution template since the "Project Hudson" days * **Garren Staubli** (Solution Architect at [Blueprint](https://bpcs.com/)) for all the help with Spark Streaming and Scala * **Sabee Grewal** for the help with the [Azure Event Hubs Connector for Apache Spark](https://github.com/Azure/azure-event-hubs-spark) * **Lena Hall** for the scenario code from [Azure.com](https://docs.microsoft.com/en-us/azure/azure-databricks/databricks-sentiment-analysis-cognitive-services) * **Giuliano Rapoz** for figuring out why the sentiment was not returning from Cognitive Services * **Jay Desai** for helping me get this Solution Template published ## Prerequisites To run the TwitterClient web job, you will need: 1. A [Twitter account](https://twitter.com/login) 2. A [Twitter application](https://apps.twitter.com) 3. Twitter's Streaming API OAuth credentials - On the Twitter application page, click on the *Keys and Access Tokens* tab - *Consumer Key (API Key)* and *Consumer Secret (API Secret)* can be found under **Application Settings** section - Under **Your Access Token** section, click on *Create my access token* to obtain both *Access Token* and *Access Token Secret* More details on Twitter's Streaming API OAuth access token can be found [here](https://dev.twitter.com/oauth/overview/application-owner-access-tokens). ## Disclaimer ©2017 Microsoft Corporation. All rights reserved. This information is provided "as-is" and may change without notice. Microsoft makes no warranties, express or implied, with respect to the information provided here. Third party data was used to generate the Solution. You are responsible for respecting the rights of others, including procuring and complying with relevant licenses in order to create similar datasets. ![ ](https://quickstart.azure.ai/track?solutionid=databricks)