MigratoryData Kafka Edition

Extending Kafka Messaging Across the Web

Why MigratoryData and Kafka?

Why MigratoryData and Kafka?

Apache Kafka is a popular realtime messaging technology with a rich ecosystem. More than 100 connectors are available for technologies like stream processing, big data, machine learning, databases, and others to get data in and out of Kafka. Therefore, Kafka excels at connecting backend technologies, usually across a trusted and reliable network like Intranet.

Connecting realtime web and mobile apps to Kafka over an untrusted and unreliable network like Internet while preserving the key features of Kafka like guaranteed delivery, message ordering, scalability to handle large volumes of messages, and high availability is far from being simple task.

There are some attempts to access Kafka through the Websocket protocol, which is a good start. But simply using WebSockets doesn’t offer much help as this does not scale. Indeed, Kafka has been designed for connecting a relatively few backend systems. However, web and mobile apps usually have many users. Not being optimized to handle lots of connections and topics, Kafka should offload the web messaging task to another realtime messaging layer.

MigratoryData, being able to handle 10 million concurrent WebSocket connections and 10 million concurrent topics on a single commodity server, is best suited to offload and accelerate Kafka messaging across the web. Moreover, its enterprise messaging features, proven in production for over a decade, are perfectly compatible with Kafka. So, MigratoryData naturally extends to the Internet the Kafka’s key features like ordering and delivery guarantees and high availability, all without putting any pressure on Kafka.

Compatibility & Complementarity

Apache Kafka MigratoryData
Designed for Intranet Designed for Internet
Distributed pub/sub technology
  • clustering
  • high availability
  • no single point of failure
  • guaranteed delivery
  • message ordering
Distributed pub/sub technology
  • clustering
  • high availability
  • no single point of failure
  • guaranteed delivery
  • message ordering
Designed for backend systems
  • # of backend systems - few
Designed for web and mobile apps
  • # of app users - many
Pull-based realtime messaging
  • long polling
Push-based realtime messaging
  • websockets
Persistent message cache
  • TTL - days, months
In-memory message cache
  • TTL - minutes, hours

MigratoryData Kafka Edition = MigratoryData + Kafka Extension

MigratoryData provides an Apache Kafka Extension built using the Kafka native API for extending Kafka messaging to web, mobile, and IoT apps.

  • Preinstalled in MigratoryData, requires a separate license to be enabled
  • Kafka-native solution, with no additional services such as Kafka Connect required
  • Dynamic mapping between Kafka topics and MigratoryData subjects
  • Stateless active/active clustering (built on Kafka)
  • Guaranteed message delivery
  • Client API for building realtime web, mobile, and IoT apps
  • Extension API for Audit and Authorization
  • Huge vertical scalability (millions of WebSocket connections)
  • Linear horizontal scalability for subscribers - ideal for fan-out use cases
  • Linear horizontal scalability for publishers - ideal for IoT use cases