Deploy on Docker, with Kafka support

This tutorial shows how to deploy MigratoryData — with Kafka support, in conjunction with Apache Kafka, using Docker.

Prerequisites

Before installing MigratoryData, ensure that you have the Docker runtime installed.

Pull images

MigratoryData

Pull the image of the latest version of the MigratoryData server from Docker Hub:

docker pull migratorydata/server:latest

Kafka

Pull the image of the latest version of Kafka, provided by Bitnami, from Docker Hub:

docker pull bitnami/kafka:latest

Create Network

Creates a Docker network with the name migNetwork:

docker network create migNetwork

Check that the network has been created:

docker network ls

This network will be used to allow communication between the MigratoryData and Apache Kafka containers.

Install

Kafka

Install the Kafka server as a Docker container named kafka on the Docker network migNetwork created earlier:

docker run -d --name kafka --network migNetwork \
-e KAFKA_CFG_NODE_ID=0 \
-e KAFKA_CFG_PROCESS_ROLES=broker,controller \
-e KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER \
-e KAFKA_CFG_LISTENERS=PLAINTEXT://:9092,CONTROLLER://:9093,EXTERNAL://:9094 \
-e KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT \
-e KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://kafka:9092,EXTERNAL://localhost:9094 \
-e KAFKA_CFG_CONTROLLER_QUORUM_VOTERS=0@kafka:9093 \
-e ALLOW_PLAINTEXT_LISTENER=yes \
-p 9094:9094 bitnami/kafka:latest

Several environment variables, such as KAFKA_KFG_*, are used to customize the Kafka container for the purposes of this tutorial. For further details on these variables, you can consult the Bitnami package documentation.

MigratoryData

Install the MigratoryData server as a Docker container named migratorydata, accepting clients on port 8800, on the Docker network migNetwork created earlier:

docker run -d --name migratorydata --network migNetwork \
-e MIGRATORYDATA_EXTRA_OPTS='-DClusterEngine=kafka' \
-e MIGRATORYDATA_KAFKA_EXTRA_OPTS='-Dbootstrap.servers=kafka:9092 -Dtopics=server' \
-p 8800:8800 migratorydata/server:latest

To customize the MigratoryData server within a Docker container, the environment variable MIGRATORYDATA_EXTRA_OPTS offers the flexibility to define specific parameters or adjust the default value of any parameter listed in the Configuration Guide. In the command above, we’ve used this environment variable to modify the default value of the parameter ClusterEngine, to enable MigratoryData’s Kafka native add-on.

To customize the MigratoryData’s native add-on for Kafka, the environment variable MIGRATORYDATA_KAFKA_EXTRA_OPTS offers the flexibility to define specific parameters or adjust the default value of any parameter of the Kafka native add-on. In the command above, we’ve used this environment variable to modify the default values of the parameters bootstrap.servers and topics to connect to the kafka container and consume the Kafka topic server.

Verify installation

Check the running containers to ensure that both kafka and migratorydata containers are up:

docker ps

Test installation

If you have installed MigratoryData locally, access the following URL in your web browser:

http://localhost:8800

Otherwise, if you have installed MigratoryData on a remote machine, and assuming the host name for that machine is push.example.com (and the firewall does not block the port 8800), access the following URL in your web browser:

http://push.example.com:8800

In either scenario, you should see a welcome page that features a demo application under the Debug Console menu for publishing real-time messages to and consuming real-time messages from your MigratoryData installation via your Kafka installation.

Uninstall

Kafka

To stop and remove the kafka container use:

docker stop kafka
docker rm kafka

MigratoryData

To stop and remove the migratorydata container use:

docker stop migratorydata
docker rm migratorydata

Network

To remove the Docker network migNetwork created earlier use:

docker network rm migNetwork

Build realtime apps

First, please read the documentation of the Kafka native add-on to understand the automatic mapping between MigratoryData subjects and Kafka topics.

Utilize MigratoryData’s client APIs to create real-time applications that communicate with your MigratoryData installation via your Kafka installation.

Also, employ Kafka’s APIs or tools to generate real-time messages destined for Kafka, which are subsequently delivered to MigratoryData’s clients. Similarly, consume real-time messages from Kafka that originate from MigratoryData’s clients.