Extending Kafka to Web and Mobile Apps

Why MigratoryData and Kafka?

Why MigratoryData and Kafka?

Apache Kafka is a very popular real-time messaging system with a rich ecosystem. More than 100 connectors are available for technologies like stream processing, big data, machine learning, databases, and others to get data in and out of Kafka. Therefore, Kafka excels at connecting backend technologies, usually across a trusted and reliable network like Intranet.

Connecting real-time web and mobile apps to Kafka over an untrusted and unreliable network like Internet while preserving the key features which made Kafka so popular - such as guaranteed delivery, message ordering, scalability, high availability - is far from being simple task.

There are some attempts to access Kafka through the Websocket standard, which is a good start. But simply using WebSockets doesn’t offer much help as this does not scale. Indeed, Kafka has been designed for connecting a relatively few backend systems. However, web and mobile apps usually have many users. Not being optimized to handle lots of connections and topics, Kafka should offload the web messaging task to another messaging layer.

MigratoryData, the industry’s most scalable web messaging system, the only messaging system which solved the C10M problem - being able to handle 10 million concurrent WebSocket connections and 10 million concurrent topics from a single commodity machine - is best suited to offload and accelerate Kafka messaging across the web.

Not only MigratoryData is a very scalable messaging system proven for over a decade in real world web and mobile apps to deliver real-time data to millions of users, but its enterprise messaging features are perfectly compatible with Kafka. So, MigratoryData naturally extends to the Internet the Kafka’s key features like guaranteed delivery, message ordering, high availability, all without putting any pressure on Kafka.

Last but not least, MigratoryData’s enterprise-grade security provides a security barrier for Kafka, protecting it against DDoS attacks, keeping closed all inbound ports on your internal firewall (it’s Kafka which connects to MigratoryData), providing data access (control which app users receive which messages).

Compatibility & Complementarity

Kafka MigratoryData
Designed for Intranet Designed for Internet
Distributed pub/sub system
  • clustering
  • high availability
  • no single point of failure
  • guaranteed delivery
  • message ordering
Distributed pub/sub system
  • clustering
  • high availability
  • no single point of failure
  • guaranteed delivery
  • message ordering
Designed for backend systems
  • # of backend systems - few
Designed for web and mobile apps
  • # of app users - many
Pull-based real-time data
  • long polling
Push-based real-time data
  • websockets
Persistent message cache
  • TTL - days, months
In-memory message cache
  • TTL - minutes, hours

Ready to Get Started?

We have everything you need to extend Kafka to real-time web and mobile apps. Start by reading the documentation of MigratoryData Connector for Kafka (both source and sink), then download the connector and try it.