Enabling the add-on

Kafka Native Add-on is preinstalled in your MigratoryData server. In order to enable it, edit the main configuration file of the MigratoryData server and configure the parameter ClusterEngine as follows:

ClusterEngine = kafka

With Config Files

MigratoryData comes with the following configuration files related to Kafka Native Add-on, located either into the folder /etc/migratorydata/ when installed using the deb/rpm installers, or into the root folder of the tarball installer:

Configuration File Name Description
integrations/kafka/consumer.properties Configuration file for built-in Kafka consumers
integrations/kafka/producer.properties Configuration file of built-in Kafka producers

The configuration files of MigratoryData have comments and optional parameters besides required parameters. The optional parameters have default values. An optional parameter that is not present in the configuration file will be used with its default value.

Kafka Native Add-on implements a logic of Kafka consumer group and Kafka producer group. So, there are two types of parameters:

  • Kafka-defined parameters
  • MigratoryData-specific parameters

Kafka Consumers

The parameters of this section should be defined in the configuration file for built-in Kafka consumers, i.e. integrations/kafka/consumer.properties.

Kafka-defined Parameters

You can use any parameter made available by Kafka API for consumers. Please refer to the Kafka documentation to learn about each of these parameters. The following Kafka-defined parameters are notable for MigratoryData:

Parameter Description
bootstrap.servers A comma-separated list of Kafka node addresses where MigratoryData will connect for Kafka cluster discovery
group.id The name of the built-in Kafka consumers group

MigratoryData-specific Parameters

The following MigratoryData-specific for Kafka consuming are available. Note that the parameters topics or topics.regex are mutually exclusive, specify either one or the other.

either the parameter topics or topics.regex should be specified.

topics  
Description A comma-separated list of Kafka topics to consume
Default value none
Required parameter Required
A MigratoryData subject can be dynamically mapped to a Kafka topic for subscription purposes only if that Kafka topic is either present in the list of the topics defined by this parameter or it matches the regular expression defined by the parameter topics.regex below.
topics.regex  
Description A Java-like regular expression giving topics to consume
Default value none
Required parameter Required - note that only the parameter topics or topics.regex should be specified.
A MigratoryData subject can be dynamically mapped to a Kafka topic for subscription purposes only if that Kafka topic is either present in the list of the topics matching the regular expression defined by this parameter or is present in the topics defined by the parameter topics above.
consumers.size  
Description Specify the number of consumers in the built-in Kafka consumers group
Default value 1
Required parameter Optional

In order to increase the message consumption capacity, multiple Kafka consumers can be configured using this parameter. All consumers belong to the Kafka consumer group defined by the Kafka parameter group.id.

recovery.on.start  
Description Specify whether or not to recover historical messages at start
Default value no
Required parameter Optional

If this parameter is set on yes, at the start time, MigratoryData will try to recover from Kafka all messages for the Kafka topics defined by the Kafka parameter topics occurred in the last number of seconds defined by the main parameter of MigratoryData CacheExpireTime which defaults to 180 seconds.

If this parameter is not defined, or it is set on no, at the start time, MigratoryData will not get any historical messages from Kafka, but starts from the latest offsets found in Kafka for the topics defined by the parameter topics.

Kafka Producers

The parameters of this section should be defined in the configuration file for built-in Kafka producers, i.e. integrations/kafka/producer.properties.

Kafka-defined Parameters

You can use any parameter made available by Kafka API for producers. Please refer to the Kafka documentation to learn about each of these parameters. The following Kafka-defined parameters are notable for MigratoryData:

Parameters Description
bootstrap.servers A list of Kafka node addresses where MigratoryData will connect for Kafka cluster discovery
partitioner.class Partitioner class to distribute messages across topic partitions
All Kafka messages with a non-null key are delivered by default to the clients of the MigratoryData server in-order and without message loss, i.e. using guaranteed delivery. However, Kafka messages without a key (i.e. where the key is null) are delivered by default to the clients of the MigratoryData server unordered, i.e. using standard delivery. To send all Kafka messages either with or without key in-order and with guaranteed delivery, configure the parameter above partitioner.class as com.migratorydata.kafka.agent.KeyPartitioner. In this way, the messages without key will be always written by the producers to the partition 0, and therefore the order will be preserved.

MigratoryData-specific Parameters

The following MigratoryData-specific for Kafka producing are available.

producers.size  
Description Specify the number of producers in the built-in Kafka producers group
Default value 1
Required parameter Optional

In order to increase the message production capacity, multiple Kafka producers can be configured using this parameter.

With System Variables

You might use the environment variable defined below to customize various aspects of your Kafka Native Add-on.

System configuration file Platform
/etc/default/migratorydata For deb-based Linux (Debian, Ubuntu)
/etc/sysconfig/migratorydata For rpm-based Linux (RHEL, CentOS)

The following environment variable should be defined in one of the configuration files above:

MIGRATORYDATA_KAFKA_EXTRA_OPTS  
Description Specifies various options for Kafka consumer and producer
Default value ""
Required parameter Optional

Use this environment variable to define the Kafka consumer and producer options or override the value of one or more of these options. Every of these options defined with this environment variable must have the following syntax:

-Dparameter=value

where the value of the parameter should be defined without spaces and quotes.

For example, to configure (or override) the values of the parameters bootstrap.servers and topics of the built-in Kafka consumers with the values kafka.example.com:9092 and respectively vehicles use:

MIGRATORYDATA_KAFKA_EXTRA_OPTS = \
'-Dbootstrap.servers=kafka.example.com:9092 -Dtopics=vehicles'