Getting Started

In this section we demonstrate how to achieve web messaging into Kafka.

Install MigratoryData

Download the tarball package of the MigratoryData server from the downloads page, unzip the tarball to any folder, change to that folder, and run on Linux/Unix/MacOS:

$ ./start-migratorydata.sh

By default, the MigratoryData server will accept client connections on the address localhost:8800. Therefore, open in a browser the following url:

http://localhost:8800

A welcome page should be loaded. Click on the DEMO button and a demo web app should be loaded. Click on the buttons Connect, then Subscribe and Publish to verify that you installation is correct.

If you encounter any issue with the installation, please check out the Installation Guide.

Install Kafka

Download the tarball binary package of Kafka from the Kafka downloads page, unzip the tarball to any folder, change to that folder, and run on Linux/Unix/MacOS:

$ ./bin/zookeeper-server-start.sh config/zookeeper.properties
$ ./bin/kafka-server-start.sh config/server.properties

Install Kafka Connect

Download the MigratoryData Connector for Kafka from the MigratoryData Connectors section of the downloads page, unzip it to any folder, say /tmp/kafka/connectors. Change to the folder where you installed Kafka at the previous step and edit the configuration file config/connect-distributed.properties as follows:

plugin.path = /tmp/kafka/connectors/migratorydata-connector-kafka

Finally, start the Kafka Connect distributed service as follows:

$ ./bin/connect-distributed.sh config/connect-distributed.properties

Deploy the Connector

In this example, we show how to deploy MigratoryData Source Connector as two Kafka Connect tasks which consumes the MigratoryData subjects /server/status1 and /server/status2, and maps them to the Kafka topics as follows:

To load the source connector, run the following command:

$ curl --header "Content-Type: application/json" \
  --request PUT \
  --data '{
      "connector.class":"com.migratorydata.kafka.source.MigratoryDataSourceConnector",
      "key.converter":"org.apache.kafka.connect.storage.StringConverter",
      "value.converter":"org.apache.kafka.connect.storage.StringConverter",
      "tasks.max": "2",
      "migratorydata.servers":"127.0.0.1:8800",
      "migratorydata.entitlement_token":"some-token",
      "migratorydata.subject.server.status1":"topic1",
      "migratorydata.subject.server.status2":"topic2",
      "migratorydata.key.server.status2":"mykey"
}' \
http://127.0.0.1:8083/connectors/migratory_data_source_00/config

To check that the connector is up and running, run the following command:

curl -s localhost:8083/connectors/migratory_data_source_00/status | jq .

Test the Connector

Open two Kafka consumers for the topics topic1 and topic2 as follows:

$ ./bin/kafka-console-consumer.sh --topic topic1 --from-beginning --bootstrap-server localhost:9092
$ ./bin/kafka-console-consumer.sh --topic topic2 --from-beginning --bootstrap-server localhost:9092

Open the demo web app at http://localhost:8800 as detailed above, connect and publish on the subjects:

• /server/status1
• /server/status2

You should be able to see in your Kafka consumers in real-time the web messages published by the web app.

Delete the Connector

To delete the MigratoryData Source Connector installed above, run the following command:

$ curl --header "Content-Type: application/json" \
  --request DELETE \
  http://127.0.0.1:8083/connectors/migratory_data_source_00