MigratoryData Source Connector for Kafka consists of Kafka-defined parameters, MigratoryData-specific parameters, as well as subjects-to-topics mapping parameters.

Kafka-defined parameters

Entry-point Class

The Java class of the connector is defined as follows:

"connector.class":"com.migratorydata.kafka.source.MigratoryDataSourceConnector"

Converters

In MigratoryData both the content and the subject of the messages are UTF-8 strings. Therefore, the connector should use the following convertors:

"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"org.apache.kafka.connect.storage.StringConverter"

Tasks

MigratoryData Source Connector can be split into multiple tasks across the Workers of the Kafka Connect distributed service, where each task will consume messages from a distinct subset of MigratoryData subjects. The Kafka Connect service is in charge with:

  • the distribution of tasks across the Kafka Connect workers
  • the distribution of MigratoryData subjects across the tasks
  • the restart of a task after a failure or a load re-balance

For example, to split the MigratoryData Source Connector into at most five tasks, configure the connector as follows:

"tasks.max": "5",

MigratoryData-specific parameters

Define the MigratoryData cluster where the connector should connect to. For example, supposing the MigratoryData cluster consists of two instances which accept encrypted client connections at the addresses push1.example.com:443, push2.example.com:443, then configure this connector as follows:

"migratorydata.servers": "push1.example.com:443,push2.example.com:443",
"migratorydata.encryption": "true"

In order to be able to consume messages from a MigratoryData cluster, a client must provide an entitlement token such that message consumption is allowed only on the entitled subjects. To allow the connector to subscribe to messages, configure the entitlement token as in the following example:

"migratorydata.entitlement_token": "some-token"

Subjects-to-topics mapping parameters

The list of MigratoryData subject and their mapping to Kafka topics is defined as follows.

A MigratoryData subject, say /mysubject/1 can be mapped into a Kafka topic, say mytopic1 using the following configuration:

"migratorydata.subject.mysubject.1": "mytopic1"

Additionally, a MigratoryData message can be published to Kafka with a key. For example, the subject /mysubject/2 can be mapped into the topic mytopic2 with the key mykey2 as follows:

"migratorydata.subject.mysubject.2": "mytopic2"
"migratorydata.key.mysubject.2": "mykey2"