Bidirectional, Native Communication with Kafka

Unlike the MigratoryData Sink/Source Connector for Kafka which uses an intermediary Kafka Connect service, the Kafka Native Add-on, built with the Kafka native API, integrates MigratoryData with Kafka directly, without using an intermediary layer.

Separately Licensed

While the MigratoryData Sink/Source Connector for Kafka is part of the MigratoryData product, the Kafka Native Add-on is separately licensed. You may use it for development and testing purposes using the default evaluation license key which is:

LicenseKey = zczuvikp41d2jb2o7j6n

To activate the Kafka Native Add-on for production use, a license key must be obtained from MigratoryData.

Stateless Active/Active Clustering

MigratoryData with Kafka Native Add-on enabled can be deployed as a stateless cluster of multiple independent nodes where Kafka plays the role of communication engine between the nodes. Not sharing any user state across the cluster, MigratoryData with Kafka Native Add-on enabled scales horizontally in a linear fashion both in terms of subscribers and publishers.

Also, the stateless nature of the MigratoryData cluster when using Kafka Native Add-on highly simplifies the cluster management in the cloud, using the elasticity function of the cloud technologies like Kubernetes.

Enabling the add-on

Kafka Native Add-on is preinstalled in your MigratoryData server version 6.0.6 or later. In order to enable it, use a valid license key (see licensing section above) and edit the main configuration file of the MigratoryData server and configure the parameter ClusterEngine as follows:

ClusterEngine = kafka

Dynamic Mapping between MigratoryData Subjects and Kafka Topics

Thanks to the compatibility between MigratoryData and Kafka, the mapping between the MigratoryData subjects and the Kafka topics is automatic using a simple convention, all without the need to define the mapping statically in a configuration file like in the case of the MigratoryData Connector for Kafka.

A MigratoryData subject is a string of UTF-8 characters that respects a syntax similar to the Unix absolute paths. It consists of an initial slash (/) character followed by one or more strings of characters separated by the slash (/) character, called segments. Within a segment, the slash (/) character is reserved. For example, the following string /Stocks/NYSE/IBM, composed by the segments Stocks, NYSE, and IBM is a valid MigratoryData subject.

The dynamic mapping between a MigratoryData subject and a Kafka topic is defined by the following convention. The first segment of the MigratoryData subject is mapped into a Kafka topic, and the remaining segments of the MigratoryDta subject are mapped into the key of that Kafka topic. Therefore, the first segment must respect the Kafka syntax for topics, i.e. it must be non-empty, and use at most 249 of the following characters [a-zA-Z0-9._-]. The remaining segments can use any UTF-8 characters because there is no syntax restriction for keys in Kafka. If a MigratoryData subject consists of a single segment, then the key of the Kafka topic given by the first segment is null.

Here are some examples of mappings between MigratoryData subjects and Kafka topics:

MigratoryData Subject Kafka Topic and Key
/vehicles/1 The Kafka topic is vehicles and the key of the topic is 1
/vehicles/1/speed The Kafka topic is vehicles and the key of the topic is 1/speed
/vehicles The Kafka topic is vehicles and the key is null

Configuration

The configuration section provides information on how to configure Kafka Native Add-on.