Blog

Meet Kafkorama - a solution built around the MigratoryData server and its native integration with Apache Kafka.

This article presents a new benchmark designed for a smart factory use case. It demonstrates MigratoryData's effectiveness in handling large numbers of users and devices, while delivering high volumes of real-time messages with minimal latency.

This post shows how to deploy an ultra scalable WebSockets service on Azure. This service, based on MigratoryaData's WebSockets server, enables bidirectional real-time messaging between your existing enterprise applications and millions of web, mobile, or IoT users. It also provides the option to integrate natively with Azure Event Hubs, allowing real-time communication with various Azure services. The deployment is orchestrated using Terraform and Azure Kubernetes Service (AKS).

This post shows how to deploy an ultra scalable WebSockets service on AWS. This service, based on MigratoryaData's WebSockets server, enables bidirectional real-time messaging between your existing enterprise applications and millions of web, mobile, or IoT users. It also provides the option to integrate natively with Amazon Managed Streaming for Apache Kafka (MSK), allowing real-time communication with various AWS services. The deployment is orchestrated using Terraform and Amazon Elastic Kubernetes Service (EKS).

In this post, we present a fresh benchmark for real-time gaming, showcasing how a single instance of MigratoryData can extend real-time Kafka messaging over WebSockets to one million concurrent gamers. Furthermore, we emphasize that by clustering N instances of MigratoryData, this scalability can be magnified by a factor of N, enabling cost-effective management of any volume of gamers.

MigratoryData Hub introduces a fresh and thrilling feature to the MigratoryData product. Embracing the concepts of API management, which deals with request/response REST APIs, and extending them into the realm of publish/subscribe real-time streaming APIs, MigratoryData Hub enables developers to publish, discover, monitor, and securely connect to real-time streaming APIs. Similar to API management solutions, MigratoryData Hub empowers organizations to build, and possibly monetize, collaborative hubs for internal, partner, or external developers through published streaming APIs, offering similar business benefits but tailored specifically for real-time use cases.

MQTT is a popular protocol at the heart of many enterprise-ready IoT messaging solutions. The most serious problem with MQTT in terms of scalability is a side-effect of its main advantage: «the compactness». For compactness reasons, MQTT designed guaranteed message delivery through simple acknowledgements. Therefore, a message sent to one million devices using MQTT will generate one million acknowledgement messages to the server, which may face «the feedback implosion problem», or else have a negative impact on vertical scalability. On the other hand, MigratoryData, which uses a publish/subscribe messaging protocol similar to MQTT, designed guaranteed message delivery through «negative feedbacks». The compactness of the protocol is slightly increased with sequence and epoch numbers and a failure detection mechanism. Hence, if a message is sent to one million devices using MigratoryData, there will be zero or few feedbacks — only from the clients which failed to get the message, due to a network issue for example. In this case, those clients automatically reconnect to the MigratoryData cluster and get the messages from the last received sequence and epoch numbers from the cache of any cluster member, as messages are replicated across the cluster. This and other design ideas have led to a significant advancement in messaging scalability, as MigratoryData demonstrated reliable messaging to 10 million concurrent users from a single server. MigratoryData can therefore reduce the expenses incurred in your IoT messaging by handling more IoT devices with a smaller number of machines. In this blog post, you will learn how to use MigratoryData and its MicroPython library on Raspberry Pi Pico W IoT devices.

MigratoryData has just released official support for interactive publishing to MigratoryData as an off-the-shelf add-on to the MigratoryData server. It allows you to build interactive publisher applications.

MigratoryData has just released official support for authorization using the JSON Web Tokens (JWT) standard as an off-the-shelf JWT Authorization add-on to the MigratoryData server.

Flutter and Kafka are two popular technologies. Flutter is a framework for building multi-platform native apps, including web apps and mobile apps both for Android and iOS, from a single codebase written in the Dart language. Kafka is a distributed event streaming platform for interconnecting the backend systems of a company, typically across a data center network. Kafka is not designed to address the challenges of connecting lots of devices across the public Internet. Therefore, building a realtime web or mobile app with lots of users to communicate with Kafka over the Internet needs a complementary technology. This blog post takes a look at the technologies available to extend Kafka across the Internet, and discusses the challenges faced to scale to millions of users. Finally, we show how to build a realtime sports scores app that scales using Flutter and Kafka.

Today MigratoryData unveiled MigratoryData Cloud, a cloud-native edition of its ultra-scalable WebSockets pub/sub message broker that has been powering realtime web and mobile apps with millions of users for over a decade.

Integrating Apache Kafka with millions of Internet facing devices is challenging. Typically used to interconnect backend systems, Kafka needs to delegate its messaging over the Internet to another service. WebSockets is the modern standard for implementing realtime messaging over the Internet. In this article, we discuss how to scale a WebSockets service to extend Kafka messaging to and from millions of connected devices. We also show how an Application Delivery Controller (ADC), such as F5 BIG-IP or Citrix ADC, can be used to accelerate Kafka messaging across the Internet.

In this blog post, we introduce an alternative approach to the MigratoryData built-in clustering—using Apache Kafka as a messaging backplane and a Kafka native add-on just released with MigratoryData 6.0.6—and discuss its application for an IoT use case.

In this post we show how to create an interactive gamification feature that scales. We demonstrate that it is feasible and cost effective to build such kind of interactive features using MigratoryData to engage in realtime with millions of users. To this end, we perform a benchmark study that demonstrates that MigratoryData scales vertically up to one million concurrent users on a single instance in the given scenario, and scales horizontally in a linear fashion, with a shared-nothing clustering architecture, by simply adding more instances to the cluster.

In this article we provide a step-by-step guide to getting started with MigratoryData with its Kafka Native Add-on enabled, an ultra-scalable Websocket publish/subscribe broker which integrates natively with Apache Kafka. Thanks to its huge vertical scalability (1000x more scalable than the C10K problem) and linear horizontal scalability both for publishers and subscribers, MigratoryData can enable bidirectional messaging communication between Apache Kafka and millions of Web and Mobile users or IoT devices - and do so cost-effectively.

We've built a live demo of MigratoryData simulating a realtime stock trading platform. In this article, we provide a step-by-step guide on how to monitor MigratoryData with Prometheus and Grafana, and present the MigratoryData metrics of the live demo as Grafana dashboard charts.

MigratoryData just released the integration of MigratoryData with Apache Kafka. This is a significant step forward both for Kafka users and MigratoryData users. On one hand, MigratoryData is going to provide to Kafka a scalable extension for Kafka data pipelines across the Web while preserving its key feature like real-time messaging, high availability, guaranteed delivery, message ordering. On the other hand, Kafka is going to provide to MigratoryData a reliable layer for message storage and integration with Kafka data pipelines like stream processing, databases, microservices. In this blog post, we outline the compatibility and complementarity of MigratoryData and Apache Kafka.

In this paper we present a fault-tolerant mechanism for sending notifications to offline users, to complement MigratoryData’s realtime messaging to online users. Hence, MigratoryData’s unified client API can be used to achieve both realtime messaging to online users and notifications – such as Push Notifications, Email, SMS, and more – to offline users. We exemplify the new mechanism with a chat mobile app for Android and iOS.

MigratoryData 6 adds support for the MQTT protocol - the de facto messaging standard for the Internet of Things.

In this blog post, we show how to use popular open-source big data platform Elastic Stack for searching, analyzing, and visualizing data produced by MigratoryData.

Using the RESTful HTTP request-response approach can become very inefficient for websites requiring real-time communication. We propose a new approach and exemplify it with a well-known feature that requires real-time communication, and which is included by most websites: search box autocomplete.

In our previous world-record-setting C10M benchmark, we showed that MigratoryData Server can achieve extreme high-scalability by delivering real-time messaging to 10 million concurrent users on a single 1U machine. Here, we show that MigratoryData Server can achieve simultaneously both extreme high-scalability and consistent low latency.

The C10M problem relates to the classic C10K Internet scalability problem, which originally occurred in the context of Web servers. The C10K problem consists of successfully supporting 10,000 concurrent connections on a single machine. While the C10K problem is currently solved by certain Web servers, the C10M problem, which consists in supporting 10 million concurrent connections on a single machine, remains a challenging problem not only for Web servers, but for any Internet server in general. In this article, we demonstrate that MigratoryData Server is able to handle 10 million concurrent connections on a single commodity machine. Moreover, it is able to push almost 1 Gbps live data to these 10 million users (each user receiving a 512-byte message per minute) with an average latency of under 100 milliseconds.

Massive scalability is the biggest challenge we undertake at MigratoryData, a provider of an enterprise publish-subscribe messaging system for building very scalable real-time web and mobile applications. We recently published a blog post demonstrating 12 million concurrent connections with MigratoryData WebSocket Server running on a single 1U server. We are going to share some lessons learned while pushing the boundaries of scalability with MigratoryData WebSocket Server.

We have recently completed a new performance benchmark which demonstrates that MigratoryData is able to handle 12 million concurrent users from a single server Dell PowerEdge R610 while pushing a substantial amount of live data (1.015 gigabit per second).

This benchmark shows that MigratoryData achieves 8X higher scalability than the record obtained by the competition in the same benchmark category; reaffirming it is the most scalable WebSocket server. This benchmark result also demonstrates that, using MigratoryData WebSocket Server, it is feasible and affordable to build real-time web applications delivering high volumes of real-time information to a high number of concurrent users.