Azure Integration Options and Overview

Today I’d like to talk about Azure integration services that you can leverage within the platform. Integration services allow data to interact with other applications or businesses and to move data around within the platform. I’d like to discuss what is currently available, where it might fit, as well as give you an overview of the capabilities of Azure integration.

Visual Integration Tools in Azure

Let’s start with the “business” or user friendly integration tools.

Flow

This workflow engine comes with Office 365. It’s a lightweight data integration engine that allows business users the ability to integrate between different types of applications. For example, you can use flow to integrate between SharePoint and drop data into email or use an email and drop data into Azure Storage. It’s set up to be simple to use and no custom coding is really allowed. Power users that are used to moving data around or have worked with SharePoint workflows will find this easy to transition to.

Logic Apps

Logic Apps is available in Azure as opposed to Office 365. It does allow you to go after the code and create within the context of Visual Studio. Its graphical interface is almost identical to flow, with almost the same capabilities, but we can expand on those capabilities. It also gives you more around security and things that make it more IT friendly.

Azure Data Week is coming soon – October 2018

Developer Integration Tools in Azure

The other integration services get a bit more sophisticated and require development, but they give you better options around source control and other areas:

Service Bus

This is about moving messages around; it’s a message broker service. It handles data from all kinds of sources, on and off premises, cloud based, mobile, etc. It’s a highly scalable, high volume service and one of the most mature services in the Azure platform.

Azure Data Factory

This integration tool does a lot of the same types of workloads as SQL Server Integration Services (SSIS) or Informatica. With Azure Data Factory V2, you have the ability to take SSIS and put it in as an integration runtime and run SSIS packages in the context of Azure Data Factory, thus making its capability even greater.

This awesome V2 version has a lot of capabilities that are not available in V1 and it has much more control around it. It enhances our ability to integrate data and ETL type of functionality and workflow.

Event Grid

This tool manages events coming off different devices and applications and simplifies the whole process of anything event driven. So, if you’re tracking events off an app or an IoT and trying to figure out what you’re going to do with that data or how you’re going to manage it, the event grid simplifies that infrastructure for you.

Web Job and Functions

Web jobs are part of the app service, functions are not, they are serverless. The idea behind these is they allow you to basically distribute code and have the ability to do other interactions.

So, you have all these options available for integration. The key is to understand what makes the most sense for you and your business. Each gives the opportunity to leverage it in a certain space, some have more expandability than others and some require more coding.

What you need to do is find the right tool for the job. In some cases, you may need more of a toolbox scenario, where you put pieces together, so you can get the best of all of them. You need to determine the best way to integrate the apps you have, from everywhere your apps and users exist, across your enterprise.

Intro to Data Factory–Training on the T’s Follow Up Post

PragmaticWorks-LogoThis is a follow up blog post based on the Intro to Data Factory session I gave on the Training on the T’s with Pragmatic Works. Find more free training from the past and upcoming here. I did my session on January 13, 2015.

 Intro To Data Factory

In this session, I gave a simple introduction to new Azure Data Factory using a CopyActivity pipeline between Azure Blob Storage and Azure SQL Database. Below is a diagram illustrating the factory that is created in the demo.

image

I have published my presentation materials here. This includes the sample JSON files, the Movies.csv, and PowerShell scripts.

Q & A

Here are a few questions that were answered during the session.

1. Does Availability refer to when data that has been transferred will be available? Or when the data source is actually available for query?

Availability refers to when the datasets will make a slice available. This is the when the dataset can be consumed as an input or be targeted as an output. This means you can consume data hourly but choose to push it to its final destination on a different cadence to prevent issues on the receiving end.

2. What pre-requisites are must haves?…e.g.(Azure account, HDInsight, Blob Storage Accounts, etc.)

    • An Azure Account is the only real must have. You could use two on premise SQL Server instances.
    • HDInsight if you want to use the HDInsight activitities
    • An Azure Storage account to use blob or table storage

3. How do you decide to use a Factory or Warehouse?

The factory is more of a data movement tool. A warehouse could be a source or target of a factory pipeline.

4. Is this similar to SSIS in SQL Server?

Yes and no. SSIS is definitely more mature and has more tooling available such as data sources and transformations. SSIS also have a good workflow constructor. The focus of the Data Factory initially was to load HDInsight tables from a variety of sources with more flexibility. The other note here is that Data Factory is being built from the ground up to support the scale of the cloud or Azure.

5. Can this be used for Big Data?

Absolutely. I would say that it is one of the primary reasons for the tool. In reference to the previous question, it will likely be the tool of choice for big data operations because it will be able to scale with Azure.

Links to Additional Resources on Data Factory or tools that were used in the presentation:

Azure Data Factory on Azure’s Website

Azure Data Factory Documentation

Azure Data Factory Pricing

Azure Storage Explorer

Azure PowerShell Documentation

Thanks for joining me for this presentation. We look forward to seeing you at the next Free Training on the T’s.