If you are looking for how to get started with Azure Data Factory (ADF), I will definitely recommend you to go through Cathrine Wilhelmsen’sBeginner’s Guide to Azure Data Factory. It helped me to understand the basics for ADF before starting work on a project I recently had a chance to participate at my workplace.

The compute infrastructure that is required by ADF provides the following data integration capabilities across different network environment:

  1. Data Flow Execution
  2. Data Movement Execution
  3. Dispatch of Activities
  4. SSIS Package Execution

Data Factory offers three types of Integration Runtime (IR), and you should choose the type that best serve the data integration capabilities and network environment needs you’re looking for. These three types are:

  1. Azure
  2. Self-hosted
  3. Azure-SSIS

This blog will help you setup the Self-Hosted Integration Runtime (SHIR) in Windows Container and show you the steps to run it on a Windows machine as well as on Azure Container Instance.

Prerequisites

  1. Windows Container

    The Windows container feature is available on Windows Server (Semi-Annual Channel), Windows Server 2019, Windows Server 2016, and Windows 10 Professional and Enterprise Editions (version 1607 and later).

  2. Docker Version 2.3 or later.

  3. Self-Hosted Integration Runtime Version 4.11.7512.1 and later. Download the latest version available.

Build the Docker image

To build the docker image we will be using Windows 10 Professional with Docker Desktop 3.1.0 in Windows Container mode.

Download the the latest source code for the SHIR windows container support here.

Download the latest version of SHIR into SHIR folder [5.2.7681.6 in this case]

SHIR_Location

On the shell navigate to the folder where you downloaded the project from github to build the docker image. Execute the command below:

1
2
cd "pathToTheGithubCheckout"
docker build -t sujaypillai/adfir:5.2.7681.6 .

SHIR_BuildDockerImage

You can find the same image on DockerHub as sujaypillai/adfir:5.2.7681.6

To register your SHIR Windows container you would need a valid Authentication Key which is passed as an environment variable AUTH_KEY in the command below:

1
2
3
4
dockerrun-d-eNODE_NAME="SHIRWindowsContainer"\
              -eAUTH_KEY="GET_THE_KEY_FROM_IR_SETUP"\
              -eENABLE_HA=true\
              -e HA_PORT=8060 sujaypillai/adfir:5.2.7681.6

To get the AUTH_KEY you will have to create a SHIR and get the value from the settings tab as shown below:

SHIR_Setup

SHIR_Setup

1
2
3
4
dockerrun-d-eNODE_NAME="SHIRWindowsContainer01"\
              -eAUTH_KEY="IR@18bbe05f-4f23-4dc4-85bc-310ec29fc549@adf-sp-lab@ServiceEndpoint=adf-sp-lab.southeastasia.datafactory.azure.net@RUW7AAC7Voog0gJcFZ7tL9le/yE9PxTXNQj8WItroZk="\
              -eENABLE_HA=true\
              -e HA_PORT=8060 sujaypillai/adfir:5.2.7681.6

SHIR_Setup

SHIR_Setup

You can get more details about your SHIR by clicking on the Monitor icon:

SHIR_Setup

Now that we have the image pushed on DockerHub how about running it on ACI ?

Set the environment variables required by the SHIR docker image sujaypillai/adfir:5.2.7681.6

SHIR_Setup

Lets deploy the container instance using Powershell as below:

SHIR_Setup

This being a windows container it takes sometime for the image to be pulled down and for the ACI instance to be up and running.

SHIR_Setup

Once the container is running you should see itself registering it as a SHIR.

SHIR_Setup

Thus we were able to run the SHIR Docker container on a virtual machine as well as an Azure Container Instance.