Data analytics solutions often involve ingesting and processing continuous streams, known as stream processing. Stream processing differs from batch processing in that streams cannot be interrupted at regular intervals but must be treated continuously instead of periodically.
Azure Stream Analytics is a cloud service that lets you build queries to fundamental process streaming source type like Azure Event Hub or IoT Hub in real time, sending results straight into Microsoft Power BI for real-time data visualization.
In this exercise, Azure Stream Analytics allows you to process an ongoing stream of sales order data from an online retail app and transmit it into Azure Event Hubs, where Azure Stream Analytics tasks will read it and summarize it before visualizing it into Power BI reports for visualization.
What Is Azure Stream Analytics?
Azure Stream Analytics is a fully managed stream processing engine capable of processing large volumes of streaming data within submillisecond latencies, such as from applications, devices, sensors, clickstreams, or social media feeds. Data can come from anywhere - applications, devices, sensors, clickstreams, social media feeds. Analyzing patterns or linkages found can trigger alerts, create reports, feed information into reporting tools, or store transformed data for later use. Stream Analytics is even available as part of Azure IoT Edge runtime to process IoT device data directly.
Key Capabilities And Benefits
Ease Of Use
Azure Stream Analytics makes getting started simple, with connections between sources. It sinks being easily established with just a few clicks, creating an end-to-end pipeline of Azure Stream Analytics, IoT Hub, Blob storage, or SQL Database available as streaming data sources and static or slow-changing reference data used as input for job execution on rushing data streams. Stream Analytics allows routing access to many storage solutions such as Azure Blob storage, SQL Database, Data Lake Store, and Cosmos DB.
Azure Synapse Analytics or HDInsight provides batch analytics on stream outputs. If desired, work can also be transferred directly into another service like Event Hubs for consumption or Power BI for real time visualization - please see Understand Outputs from Azure Stream Analytics for Realtime Data in Power BI, a comprehensive listing of outputs from this solution.
The Azure Stream Analytics no-code editor provides an experience that simplifies Stream Analytics task creation without writing code. See No-code stream processing in Azure Stream Analytics for additional details on this tool.
Programmer Productivity
Azure Stream Analytics uses an enhanced SQL query language with temporal constraints for its data analytics capabilities. Developer tools, including Azure PowerShell, CLI, and Visual Studio Tools (both with extensions available for Visual Studio Code and Resource Manager templates), may also be utilized to generate jobs; such tools offer offline transformation queries, which are then submitted via Azure via their CI/CD pipelines.
The Stream Analytics query language facilitates Complex Event Processing by offering a versatile set of functions for evaluating streaming data. Simple manipulation, data aggregation/analysis operations, geospatial capabilities, and pattern matching/anomaly detection are all supported in this query language. You can test queries against sample streams using our portal or development tools before moving them live into production.
Enhance the query language's capabilities by creating and executing new functions. When taking advantage of Azure Machine Learning solutions, function calls may also be made from Azure Machine Learning into user-defined functions (UDFs or UDAs) written in JavaScript or C# to perform complex calculations as part of a Stream Analytics query.
Fully Managed
Azure Stream Analytics is a fully managed (PaaS) Azure product, eliminating any need to provision or manage infrastructure or update operating systems or applications. Azure Stream Analytics fully oversees every task required so you can focus on growing your business rather than maintaining infrastructure.
Integration Of Power BI With Azure: Unleashing The Full Potential Of Data Analytics
Power BI, Microsoft's premiere business intelligence tool, and Azure - its cloud computing platform - make an unbeatable combination for data analytics. Their seamless connection enables an array of advanced processing, storage, and visual analytics options; in this technical blog post, we explore their interaction and how their combined strength enables organizations to maximize the full potential of data analytics while making sound data-driven decisions.
- Azure Data Sources and Power BI Integration: Azure provides access to an amazing set of data services, including SQL Database and Cosmos DB. This article discusses how Power BI may connect directly to the sources in order to build intelligent reports with real-time Azure database access.
- Data Flows with Azure Data Factory: Azure Data Factory offers businesses a powerful ETL (Extract, Transform, and Load) solution for automating data transfers and transformation.
- Data Streaming with Azure Stream Analytics: Azure Stream Analytics can ingest and process high-velocity data streams for RealTime Data Analytics, providing live streaming capability and valuable insights into dynamic data sets. Here, we demonstrate its use alongside Power BI to enable real-time insights.
- Azure Cognitive Services with Power BI: Azure Cognitive Services, an array of AI-powered APIs, can be combined with Power BI to offer advanced analytics capabilities. We will cover how these can be utilized within Power BI reports to perform natural language processing, sentiment analysis, and image recognition tasks.
- Azure Data Lake Gen2 and Power BI Dataflows: Power BI Dataflows offer self-service data preparation and cleansing. In this discussion, we'll look at using Power BI Dataflows to store and manage data in Azure Data Lake Gen2, increasing collaboration across Power BI datasets while increasing the reusability of information.
- Azure Web Apps and Power BI Embedded: Azure Web Apps offer an ideal hosting environment to embed Power BI reports within custom apps, creating a cohesive user experience for all. We will show how embedding reports within Azure Web Apps makes this seamless integration for an improved user journey.
- Combine Power BI Premium with Azure Analysis Services: Power BI Premium provides dedicated capacity that improves both performance and scalability, and we will discuss how Azure Analysis Services may complement this offering to provide comprehensive modeling and querying capabilities suitable for enterprise-grade solutions.
- Monitoring and Governance with Azure Monitor: Azure Monitor provides powerful monitoring and logging tools to keep Power BI and Azure environments healthy, including how to track workspace resources, datasets, and resources hosted within them. We will explore various techniques for using this monitoring solution effectively in this session.
Also Read: Real-Time Data Streaming: The Key to Maximum Software Solutions Impact?
Provision Azure Resources
This exercise requires an Azure Synapse Analytics workspace with access to data lake storage and an allocated SQL pool for optimal results. Furthermore, an Event Hubs namespace must also exist to transmit streaming source order data without delay. These resources must then be provisioned using a PowerShell script and an ARM template.
- Sign in to Azure Portal at portal.azure.com/
- Use the [>_] button on the right of the search bar at the top of this page to create a Cloud Shell in Azure storage Portal, selecting PowerShell environment and storage if prompted; your Cloud Shell provides a command-line interface in a pane at the bottom of the Azure analysis service portal as shown here:
- Note that you can resize the cloud shell by dragging its separator bar to the top right or using its -, +, and X icons at its right top corner to minimize, maximize, or close its pane as appropriate. For more information about using Azure Cloud Shell, please consult its documentation.
- Enter these commands into PowerShell to clone this exercise repository:
- Once the repository has been cloned, issue these commands to navigate directly into its directory for this exercise and execute its setup.ps1 script:
- When asked, select which Azure subscription to use (this only occurs if multiple Azure data lake storage gen2 accounts exist).
- While waiting for your script to run smoothly, move on to another task.
Create A Power BI Workspace
Workspaces in Power BI are used for organizing datasets, reports, and other resources within its service. Each Power BI user receives their default workspace, "My Workspace," which you may utilize during this exercise, but generally speaking, it would be best practice to establish separate workspaces for every reporting solution you intend on managing.
- Use your Power BI service login details to log into the Power BI app.
- On the left-side menu bar, choose Workspaces.
- Create a workspace with an appropriate name (for example, mslearn-streaming), selecting Pro licensing mode.
- Noting the workspace's globally unique identifier (GUID), found within its page URL.
Use Azure Stream Analytics To Process Streaming Data
Streaming with Power BI and Azure Stream Analytics jobs represent continuous queries that operate on streaming data from one or more inputs and send the results out as outputs.
Create A Stream Analytics Job
- You return to the Azure portal browser tab. Upon script completion, you noted where the cisin203-a resource group was provisioned.
- On the Home page of the Azure portal, select + Create a resource and search for "Stream Analytics job." Once found, create one with these properties:
- Subscription: Your Azure subscription
- Resource group: Select the existing Cis203-xxxxxxx resource group.
- Name: stream-orders
- Region: Select the area where your Synapse Analytics workspace is provisioned.
- Hosting environment: Cloud
- Streaming units: 1
- Wait until deployment has been completed before accessing the deployed Stream Analytics job resource.
Create an event data stream input
- Select the Inputs page from the stream-orders overview page. Add Event Hub input with the following properties using the Add input menu:
- Input alias: orders
- Select Event Hub from your subscriptions: Selected.
- Subscription: Your Azure subscription
- Event Hub namespace: Select the events Event Hubs namespace
- Event Hub name: Select the existing event hub.
- Consumers of Event Hub: Choose the pre-existing $Default consumer group.
- Authentication mode: Create a system-assigned managed identity
- Partition key: Leave blank
- Event serialization format: JSON
- Encoding: UTF-8
- Save the input, wait while it is created, and monitor notifications until a Successful connection test notification appears.
Create An Output For The Power Bi Workspace
- View the Outputs page of your stream-orders Stream Analytics job and use the Add output menu to add a Power BI output with these properties:
- Output alias: power bi-dataset
- Select Power BI settings manually: Selected.
- Group workspace: The GUID for your workspace
- Authentication mode: Select the User token and then use the Authorize button at the bottom to sign into your Power BI account
- Dataset name: realtime-data
- Table name: orders
- Save the output and wait while it is generated until notifications of its completion appear. Look out for an announcement stating: Successful connection test notification.
Make A Query To Summarize The Event Stream
- View the stream-orders Stream Analytics job's query page.
- Change the default query as follows.
- SELECT
- DateAdd(second,-5,System.TimeStamp) AS StartTime,
- System.TimeStamp AS EndTime,
- ProductID,
- SUM(Quantity) AS Orders
- INTO
- [powerbi-dataset]
- FROM
- [orders] TIMESTAMP BY EventEnqueuedUtcTime
- GROUP BY ProductID, TumblingWindow(second, 5)
- HAVING COUNT(*) > 1
- The System. The timestamp field (based on the EventEnqueuedUtcTime field) establishes start and end timestamps of 5-second non-overlapping sequential windows in which total quantities for every product ID can be calculated.
- Save the query.
Run The Streaming Job To Process Order Data
- Consider accessing the Overview page of a stream-orders Stream Analytics job and, on its Properties tab, reviewing its Inputs, Queries, Outputs, and Functions. If they total zero, use the Refresh button on that same Overview page to display order inputs and power bi-dataset outputs.
- Select the Start button now to initiate streaming immediately, then wait until you receive a notification that it has started successfully.
- To submit 100 orders, reopen the Cloud Shell pane and run this command. Refresh the Power BI app page until the real time data dataset created by the Azure Stream Analytics job appears in your workspace.
Conclusion
Power BI with Azure integration offers organizations a dynamic ecosystem to unleash the true potential of data analytics. By combining Microsoft Azure development services, AI capabilities, cloud computing resources, and Power BI's straightforward data visualization and reporting tools, they can transform raw data into meaningful insights that transform into actionable intelligence for decision-makers and strategic partnerships.
Power BI with Azure Analysis Services database provides an end-to-end solution for modern data-driven companies, from data intake and preparation to advanced analytics and real time reporting. Experience its power for yourself as you progress toward data analytics proficiency.
