Azure Event Hubs - MigratoryData

This guide describes how to deploy a MigratoryData cluster with Azure Kubernetes Service (AKS) and using Azure Event Hub as a MigratoryData backplane.

Before reading this document, it is recommended to read Apache Kafka as a MigratoryData backplane to enable IoT applications with millions of devices.

Prerequisites

A Microsoft Azure account
Azure CLI
Kubectl tool

Variables

Let’s use the following variables for this tutorial:

export RESOURCE_GROUP=rg-migratorydata
export AKS_CLUSTER=aks-migratorydata
export EVENTHUBS_NAMESPACE=evhns-migratorydata
export EVENTHUBS_TOPIC=vehicles

Create an AKS cluster

Let’s create a Kubernetes cluster with at least three and at most five nodes.

# login to AKS
az login

# create the resource group
az group create --name $RESOURCE_GROUP --location eastus

# create the cluster and enable the cluster autoscaling
az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --node-count 3 \
  --vm-set-type VirtualMachineScaleSets \
  --generate-ssh-keys \
  --load-balancer-sku standard \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 5

Connect to the AKS cluster and check if the AKS nodes are ready

# connect to the AKS cluster
az aks get-credentials \
--resource-group $RESOURCE_GROUP \
--name $AKS_CLUSTER

# check if the nodes of the cluster are up
kubectl get nodes

Create an Azure Event Hubs topic

To create a Kafka topic on Azure Event Hubs, run the following commands:

# create a namespace into the Event Hubs
az eventhubs namespace create --name $EVENTHUBS_NAMESPACE \
--resource-group $RESOURCE_GROUP -l eastus

# create the Kafka topic
az eventhubs eventhub create --name $EVENTHUBS_TOPIC \
--resource-group $RESOURCE_GROUP \
--namespace-name $EVENTHUBS_NAMESPACE

Authenticate to Azure Event Hubs with SASL using JAAS

# fetch the Event Hubs rule/policy and pick the value of the "name" 
# attribute from the JSON response of the following command
az eventhubs namespace authorization-rule list \
--resource-group $RESOURCE_GROUP \
--namespace-name $EVENTHUBS_NAMESPACE

# suppose the policy picked above is RootManageSharedAccessKey, 
# then pick the value of the attribute "primaryConnectionString" 
# from the JSON response of the following command
az eventhubs namespace authorization-rule keys list \
--resource-group $RESOURCE_GROUP \
--namespace-name $EVENTHUBS_NAMESPACE \
--name RootManageSharedAccessKey

The value of the attribute primaryConnectionString picked from the response of the last command should look as follows:

Endpoint=sb://evhns-migratorydata.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxxxxxxxxxxxxx

Therefore, the JAAS config to authenticate to Azure Event Hubs with SASL should look as follows:

KafkaClient {
        org.apache.kafka.common.security.plain.PlainLoginModule required
        username="$ConnectionString"
        password="Endpoint=sb://evhns-migratorydata.servicebus.windows.net/;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=xxxxxxxxxxxxxxxxx";
};

Copy the JAAS config to a file, say jaas.config. We will need this configuration later to connect a Kafka consumer and producer to the Azure Event Hubs with SASL.

Create secret from the JAAS config file

Because the JAAS config file obtained in the previous step must be included in the pod configuration, we should create a secret from jaas.config which will be mounted as a volume in Kubernetes.

kubectl create secret generic migratory-secret --from-file=jaas.config

Deploy MigratoryData cluster

Copy the following Kubernetes configuration to a file, say cluster.yaml and edit it by replacing the variables EVENTHUBS_NAMESPACE and EVENTHUBS_TOPIC with the values defined or obtained above:

apiVersion: v1
kind: Service
metadata:
  name: migratorydata-cs
  labels:
    app: migratorydata
spec:
  type: LoadBalancer
  ports:
    - name: client-port
      port: 80
      protocol: TCP
      targetPort: 8800
  selector:
    app: migratorydata
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: migratorydata
spec:
  selector:
    matchLabels:
      app: migratorydata
  replicas: 3
  template:
    metadata:
      labels:
        app: migratorydata
    spec:
      containers:
      - name: migratorydata-cluster
        imagePullPolicy: Always
        image: migratorydata/server:latest
        volumeMounts:
        - name: migratory-secret
          mountPath: "/migratorydata/secrets/jaas.config"
          subPath: jaas.config
          readOnly: true
        env:
          - name: MIGRATORYDATA_EXTRA_OPTS
            value: "-DMemory=128MB \
              -DLogLevel=INFO \
              -DX.ConnectionOffload=true \
              -DClusterEngine=kafka"
          - name: MIGRATORYDATA_KAFKA_EXTRA_OPTS
            value: "-Dbootstrap.servers=$EVENTHUBS_NAMESPACE.servicebus.windows.net:9093 \
              -Dtopics=$EVENTHUBS_TOPIC \
              -Dsecurity.protocol=SASL_SSL \
              -Dsasl.mechanism=PLAIN
              -Djava.security.auth.login.config=/migratorydata/secrets/jaas.config"
          - name: MIGRATORYDATA_JAVA_EXTRA_OPTS
            value: "-XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap"
        resources:
          requests:
            memory: "256Mi"
            cpu: "0.5"
        ports:
          - name: client-port
            containerPort: 8800
        readinessProbe:
          tcpSocket:
            port: 8800
          initialDelaySeconds: 10
          periodSeconds: 5
        livenessProbe:
          tcpSocket:
            port: 8800
          initialDelaySeconds: 10
          periodSeconds: 5
      volumes:
      - name: migratory-secret
        secret:
          secretName: migratory-secret

The manifest above contains a Service and a Deployment. The Service is used to handle the clients of the cluster over the port 80.

In order to deploy the cluster of MigratoryData on the AKS cluster, run the following command:

$ kubectl apply -f cluster.yaml

Check if the pods are up and running

By running the following command, you can check that the three pods of the example configuration are up and running:

kubectl get pods

NAME                                READY   STATUS    RESTARTS   AGE
migratorydata-57848575bd-4tnbz   1/1     Running   0          4m32s
migratorydata-57848575bd-gjmld   1/1     Running   0          4m32s
migratorydata-57848575bd-tcbtf   1/1     Running   0          4m32s

and you can check the logs of each cluster member by running a command as follows:

kubectl logs migratorydata-57848575bd-4tnbz

Also, by running the following command, you can check that the service is up and running:

kubectl get svc

NAME                  TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)        AGE
kubernetes            ClusterIP      10.0.0.1      <none>        443/TCP        37m
migratorydata-cs      LoadBalancer   10.0.90.187   YourExternalIP    80:30546/TCP   5m27s

You should now be able to connect to http://YourExternalIP and run the demo app bundled with the MigratoryData broker.

Scaling MigratoryData on AKS

The stateless nature of the MigratoryData broker using the Azure Event Hubs backplane, where each cluster member is independent from the others, highly simplifies the horizontal scaling on AKS.

Manual Scaling

For example, if the load of your system increases substantially, and supposing your nodes have enough resources available, you can add two new members to the cluster by modifying the replicas field as follows:

kubectl scale deployment migratorydata --replicas=5

If the load of your system decreases significantly, then you might remove three members from the cluster by modifying the replicas field as follows:

kubectl scale deployment migratorydata --replicas=2

Autoscaling

Manual scaling is practical if the load of your system changes gradually. Otherwise, you can use the autoscaling feature of Kubernetes.

Kubernetes can monitor the load of your system, typically expressed in CPU usage, and scale your MigratoryData cluster up and down by automatically modifying the replicas field.

In the example above, to add one or more new members up to a maximum of 5 cluster members if the CPU usage of the existing members becomes higher than 50%, or remove one or more of the existing members if the CPU usage of the existing members becomes lower than 50%, use the following command:

kubectl autoscale deployment migratorydata --cpu-percent=50 --min=2 --max=5

Now, you can display information about the autoscaler object above using the following command:

kubectl get hpa

and display CPU usage of cluster members with:

kubectl top pods

While testing cluster autoscaling, it is important to be aware that the Kubenetes autoscaler gets the CPU usage periodically from the cluster members, so autoscaling might appear to be not immediate. However, this is normal Kubernetes behavior.

Node Autoscaling

Cloud infrastructure like Azure Kubernetes Service (AKS) can be configured to automatically spin up additional nodes if the resources of the existing nodes are insufficient to create new cluster members. In the example above, we have created the AKS cluster with a minimum of 3 nodes and a maximum of 5 nodes and enabled node autoscaling with the command that we recall here:

# create the cluster and enable the cluster autoscaling
az aks create \
  --resource-group $RESOURCE_GROUP \
  --name $AKS_CLUSTER \
  --node-count 3 \
  --vm-set-type VirtualMachineScaleSets \
  --generate-ssh-keys \
  --load-balancer-sku standard \
  --enable-cluster-autoscaler \
  --min-count 3 \
  --max-count 5

While testing node autoscaling, it is important to be aware that adding or disposing nodes from AKS might take some time. For example, to evict an unused node back to AKS might take up to several minutes. Here are more details about time granularity on AKS cluster node autoscaling:

https://docs.microsoft.com/en-us/azure/aks/cluster-autoscaler#using-the-autoscaler-profile

Node Failure Testing

MigratoryData clustering tolerates node failures provided that at least one node remain up and running. In order to test an AKS node failure, use:

kubectl drain <node-name> --force --delete-local-data --ignore-daemonsets

Then, to start an AKS node, use:

kubectl uncordon <node-name>

Maintenance

If something gets wrong with the commands above, you can remove the MigratoryData cluster with the command below and try again:

kubectl delete -f cluster.yaml

Finally when you don’t need anymore the AKS cluster and the Event Hubs topic for testing, remove the resources allocated with:

az group delete --name $RESOURCE_GROUP --yes --no-wait