MigratoryData Source Connector for Kafka consists of Kafka-defined parameters, MigratoryData-specific parameters, as well as subjects-to-topics mapping parameters.
The Java class of the connector is defined as follows:
In MigratoryData both the content and the subject of the messages are UTF-8 strings. Therefore, the connector should use the following convertors:
MigratoryData Source Connector can be split into multiple tasks across the Workers of the Kafka Connect distributed service, where each task will consume messages from a distinct subset of MigratoryData subjects. The Kafka Connect service is in charge with:
- the distribution of tasks across the Kafka Connect workers
- the distribution of MigratoryData subjects across the tasks
- the restart of a task after a failure or a load re-balance
For example, to split the MigratoryData Source Connector into at most five tasks, configure the connector as follows:
Define the MigratoryData cluster where the connector should connect to. For example,
supposing the MigratoryData cluster consists of two instances which accept
encrypted client connections at the addresses
push2.example.com:443, then configure this connector as follows:
"migratorydata.servers": "push1.example.com:443,push2.example.com:443", "migratorydata.encryption": "true"
In order to be able to consume messages from a MigratoryData cluster, a client must provide an entitlement token such that message consumption is allowed only on the entitled subjects. To allow the connector to subscribe to messages, configure the entitlement token as in the following example:
Subjects-to-topics mapping parameters
The list of MigratoryData subject and their mapping to Kafka topics is defined as follows.
A MigratoryData subject, say
/mysubject/1 can be mapped into a Kafka topic, say
mytopic1 using the following configuration:
Additionally, a MigratoryData message can be published to Kafka with a key. For example, the
/mysubject/2 can be mapped into the topic
mytopic2 with the key
mykey2 as follows:
"migratorydata.subject.mysubject.2": "mytopic2" "migratorydata.key.mysubject.2": "mykey2"