MigratoryData - Kafka Edition

MigratoryData - Kafka Edition

MigratoryData - Kafka Edition (MigratoryData KE) is a specialized product, separately licensed, which integrates natively with Kafka. Leveraging the MigratoryData product, it inherits most of its advantages, including its high scalability and reliability, all client API libraries, and the plugin API libraries for authorization and audit. However, MigratoryData KE implements a new clustering architecture built on Apache Kafka which provides two substantial advantages when compared to the MigratoryData product: linear horizontal scalability for publishers besides linear horizontal scalability for subscribers, and stateless clustering rather than stateful clustering.

Therefore, the very high vertical scalability and the linear horizontal scalability both for publishers and subscribers make MigratoryData KE as a cost-efficient solution for any realtime scenario with lots of users or devices. Unlike the MigratoryData product which excels in subscribers-dominant scenarios, MigratoryData KE excels both in subscribers-dominant scenarios and in publishers-dominant scenarios, such as the IoT scenarios, where lots of publishers (sensors, connected cars, etc) send realtime messages to the backend for realtime processing.

Also, the stateless clustering of MigratoryData KE highly simplifies the deployment because the nodes in the cluster are independent. They don't need to be configured to know each other as it is the case with the MigratoryData product. Therefore, clustering elasticity is straightforward for MigratoryData KE with cloud deployment solutions like Kubernetes.

To learn more about the Kafka integration and how the MigratoryData KE specialized product differs from the MigratoryData product bundled with Kafka add-ons, please visit the Kafka solution page.

Key Features

Bidirectional, Native Communication with Kafka

MigratoryData KE uses the Kafka native protocol for bi-directional messaging to and from your Kafka cluster, all without the need of additional services such as Kafka Connect.

Dynamic Mapping between Subjects and Topics

Thanks to the compatibility between MigratoryData KE and Kafka, the mapping between the MigratoryData subjects and the Kafka topics is automatic using a simple convention, without the need to define such a mapping statically in a configuration file.

Massive Vertical Scalability

MigratoryData KE has been designed to scale up to the entire capacity of the allocated hardware such as CPU and network. One instance of MigratoryData KE running on a single commodity server has been benchmarked to handle 10 million concurrent users. So, MigratoryData KE achieves 1000x more scalability than the well-known C10K problem (a scalability challenge in the realm of web servers).

Linear Horizontal Scalability

Besides its unparalleled vertical scalability, MigratoryData KE scales horizontally in a linear manner both for subscribers and for publishers. For example, supposing a cluster of one MigratoryData KE instance can handle 100,000 concurrent subscribers and 100,000 concurrent publishers in a given scenario, then, by adding another MigratoryData KE instance to the cluster, the capacity of the cluster will double, it will be able to handle 200,000 concurrent subscribers and 200,000 concurrent publishers.

Universal Client API using WebSockets

MigratoryData KE exposes a simple publish/subscribe client API with libraries for the most popular programming languages and platforms.

Stateless Active/Active Clustering

MigratoryData KE can be deployed as a cluster of multiple independent nodes for achieving linear horizontal scalability, elasticity, fault tolerance, no single point of failure, and guaranteed delivery. The stateless nature of the clustering highly simplifies the deployment of MigratoryData KE in the cloud using technologies like Kubernetes.

Guaranteed Delivery & In-Order Messaging

MigratoryData KE implements guaranteed delivery and in-order messaging even in the event of unexpected events (e.g. hardware failures, network disconnections, cluster node restart) using concepts like epoch and sequence numbers for messages, message replication across the cluster, and in-memory message caching.

Advanced Monitoring

Encrypted and password-protected monitoring is made available though the industry standards like JMX and HTTP which can be integrated with most enterprise management systems. A modern monitoring option using Prometheus and Grafana is available as well.

Extensible with Plugins

MigratoryData KE provides certain Java-based Plugin SDKs for building plugins for authorization and audit.

Enterprise-Grade Security

MigratoryData KE uses the industry standards like TLS/SSL for data encryption and data authorization.

Get Started for Free

Download MigratoryData KE for free and use it for evaluation for 45 days. It is fully functional, up to 100 concurrent connections, with no feature restriction.