MigratoryData KE implements a logic of Kafka consumer group and Kafka producer group. So, there are two types of parameters related to Kafka:

  • Kafka-defined parameters
  • MigratoryData-specific parameters

Kafka Consumers

Kafka-defined Parameters

You can use any parameter made available by Kafka for consumers. Please refer to the Kafka documentation to learn about each of these parameters at:

https://kafka.apache.org/documentation/#consumerconfigs

However, the following Kafka-defined parameters are notable for MigratoryData KE:

Kafka-defined parameters Description
bootstrap.servers A list of Kafka node addresses where
MigratoryData KE will connect to
for Kafka cluster discovery
group.id The name of the Kafka consumers group

MigratoryData-specific Parameters

topics

   
Description A list of Kafka topics to consume
Default value none
Required parameter Required - note that only the parameter topics or topics.regex should be specified.
A MigratoryData subject can be dynamically mapped to a Kafka topic for subscription purposes only if that Kafka topic is either present in the list of the topics defined by the parameter topics above or it matches the regular expression defined by the parameter topics.regex above.

topics.regex

   
Description A Java-like regular expression giving topics to consume
Default value none
Required parameter Required - note that only the parameter topics or topics.regex should be specified.
A MigratoryData subject can be dynamically mapped to a Kafka topic for subscription purposes only if that Kafka topic is either present in the list of the topics defined by the parameter topics above or it matches the regular expression defined by the parameter topics.regex above.

consumers.size

   
Description Specify the number of consumers in the Kafka consumers group
Default value 1
Required parameter Optional

In order to increase the message consumption capacity, multiple Kafka consumers can be configured using this parameter. All consumers belong to the Kafka consumer group defined by the Kafka parameter group.id.

recovery.on.start

   
Description Specify whether to recover historical messages at start
Default value no
Required parameter Optional

If this parameter is set on yes, at the start time, MigratoryData KE will try to recover from Kafka all messages for the Kafka topics defined by the Kafka parameter topics occurred in the last number of seconds defined by the main parameter CacheExpireTime which defaults to 180 seconds.

If this parameter is not defined, or it is set on no, at the start time, MigratoryData KE will not attempt to get any historical messages from Kafka, but starts from the latest offsets found in Kafka for the topics defined by the parameter topics.

Kafka Producers

Kafka-defined Parameters

You can use any parameter made available by Kafka for producers. Please refer to the Kafka documentation to learn about each of these parameters at:

https://kafka.apache.org/documentation/#producerconfigs

However, the following Kafka-defined parameters are notable for MigratoryData KE:

Kafka-defined parameters Description
bootstrap.servers A list of Kafka node addresses where
MigratoryData KE will connect to
for Kafka cluster discovery
partitioner.class Partitioner class to distribute messages across topic partitions

All Kafka messages with non-null key are delivered by default to the clients of MigratoryData KE in-order and without message loss, i.e. using Guaranteed quality of service (QoS).

However, Kafka messages without key (i.e. where the key is null) are delivered by default to the clients of MigratoryData KE unordered, i.e. using Standard QoS.

To send all Kafka messages either with or without key in-order and with Guaranteed QoS, set on com.migratorydata.kafka.agent.KeyPartitioner the parameter above partitioner.class. In this way, the messages without key will be always written by the producers to the partition 0, and therefore the order will be preserved.

MigratoryData-specific Parameters

producers.size

   
Description Specify the number of producers in the Kafka producers group
Default value 1
Required parameter Optional

In order to increase the message production capacity, multiple Kafka producers can be configured using this parameter.