Configuration

MigratoryData Sink Connector for Kafka consists of Kafka-defined parameters, MigratoryData-specific parameters, as well as topics-to-subjects mapping parameters.

Kafka-defined parameters

Entry-point Class

The Java class of the connector is defined as follows:

"connector.class":"com.migratorydata.kafka.sink.MigratoryDataSinkConnector"

Converters

In MigratoryData both the content and the subject of the messages are UTF-8 strings. Therefore, the connector should use the following convertors:

"key.converter":"org.apache.kafka.connect.storage.StringConverter",
"value.converter":"org.apache.kafka.connect.storage.StringConverter"

Tasks

MigratoryData Sink Connector can be split into multiple tasks across the Workers of the Kafka Connect distributed service, where each task will consume messages from a distinct subset of topic partitions. The Kafka Connect service is in charge with:

• the distribution of tasks across the Kafka Connect workers
• the distribution of the Kafka topic partitions for each task
• the restart of a task after a failure or a load re-balance

For example, to split the MigratoryData Sink Connector into at most five tasks, configure the connector as follows:

"tasks.max": "5",

MigratoryData-specific parameters

Define the MigratoryData cluster where the connector should connect to. For example, supposing the MigratoryData cluster consists of two instances which accept encrypted client connections at the addresses push1.example.com:443, push2.example.com:443, then configure this connector as follows:

"migratorydata.servers": "push1.example.com:443,push2.example.com:443",
"migratorydata.encryption": "true"

In order to be able to publish messages to a MigratoryData cluster, a client must provide an entitlement token such that message publication is allowed only on the entitled subjects. To allow the connector to publish messages, configure the entitlement token as in the following example:

"migratorydata.entitlement_token": "some-token"

Topics-to-subjects mapping parameters

The list of Kafka topics to be consumed by the adapter can be specified:

• either as a list of Kafka topics
• or a a regexp pattern to match certain Kafka topics

A Kafka message received on a Kafka topic which is in the list or matches the regexp will be published by the connector on a MigratoryData subject as defined by a mapping you can defined.

Indeed, each Kafka topic can be mapped into one or more MigratoryData subjects as detailed below.

Topics as list

Assuming the connector consumes the Kafka topics t_1 and t_2, and maps Kafka topics to MigratoryData subjects as follows:

where <KEY> is the key of the Kafka message received on the Kafka topic t_2. Then, configure the connector as follows:

"topics":"t_1, t_2",
"kafka.topics.t_1":"/sbj/a",
"kafka.topics.t_2":"/sbj/x, /sbj/y/${topic}, /sbj/z/${topic}/${key}"

Topics as regexp

Assuming the connector consumes the Kafka topics matching the regexp pattern t_(.)*, and maps each matched topic to the following MigratoryData subjects /sbj/a, /sbj/b/<TOPIC>, and /sbj/c/<TOPIC>/<KEY> where <TOPIC> is the matched topic of the received Kafka message, and <KEY> is the key of the received Kafka message. Then, configure the connector as follows:

"topics.regex":"t_(.*)",
"kafka.topics.regex":"/sbj/a, /sbj/b/${topic}, /sbj/c/${topic}/${key}"