In this paper we present a fault-tolerant mechanism for sending notifications to offline users, to complement MigratoryData’s realtime messaging to online users. Hence MigratoryData’s unified client API can be used to achieve both realtime messaging to online users and notifications – such as Push Notifications, Email, SMS, and more – to offline users. We exemplify the new mechanism with a chat mobile app for Android and iOS.

Overview

MigratoryData is a highly scalable pub/sub middleware for reliable messaging to web and mobile users. Typical deployments consist of small active/active clusters of MigratoryData servers and involve millions of users.

Users connect to the MigratoryData cluster through web or mobile apps built upon MigratoryData’s unified client API. The client API library uses WebSocket or HTTP persistent connections for connecting users to the cluster. Messages flow in realtime between each user and the cluster as long as the user remains connected to the cluster.

When a connection between a user and a cluster member is broken, due, for example, to a network condition, the client API library will automatically reconnect the user to another cluster member. Moreover, any message published during the failover time for the subjects subscribed by the user, will be recovered by the user at reconnection time. To this end, MigratoryData maintains a cache for each subject and adopts replication and other techniques for keeping the cache of each subject in sync among all cluster members to achieve guaranteed message delivery and fault tolerance with no single point of failure.

However, if the mobile app of the user is stopped or the user’s mobile device enters into sleep mode, the connection breaks and the client API library cannot reconnect the user to another cluster member as the mobile app does not run anymore. For such situations, we have introduced a Presence Extension API which can be used to develop a plugin-based mechanism for sending notifications to offline users across notification services like Push Notifications, Email, SMS, and more.

The Problem: Using MigratoryData for both Realtime and Anytime

A message received by the MigratoryData cluster from a publisher on a subject should be sent to all users who are online and who subscribed for that subject. In addition, it should be sent as a notification to all users who are offline and who subscribed for that subject, as depicted in the diagram below. Users should use MigratoryData’s unified client API to both subscribe and publish. In this way, realtime messaging and anytime messagings (i.e. notifications) should be done transparently, without any coding in the publisher and subscriber applications.

Solution

The newly introduced Presence Extension API can be used to build a plugin for the MigratoryData server to keep track of offline users as well as of their subscribed subjects. When a message is received from a publisher on a certain subject, the MigratoryData server will publish that message to all online users subscribed to that subject. Moreover, it will pass the message to the presence plugin. The presence plugin can send the message to the offline users subscribed to the message’s subject as a notification via one or more notification services as depicted in the diagram below.

Client API

MigratoryData provides a common, unified client API with libraries for the most popular languages and technologies such as Java, C++, Objective-C, C#, JavaScript, .NET etc. The API provides methods for connecting and disconnecting from the MigratoryData cluster, subscribing and unsubscribe from subjects.

In order to integrate with various notification services, the client API has been extended with the following method:

public void setExternalToken(String externalToken);

This method should be used to assign to a user a token which identifies the user in the notification service such as a FCM token, an email address, or an SMS-capable phone number. This token is passed to the presence plugin via the Presence Extension API. So, presence plugin can use it to send notifications to the user when the user goes offline.

Presence Extension API

The MigratoryData server is extensible. It exposes a number of server-side Extension APIs for building plugins such as custom entitlement plugins. Presence Extension API is such a server-side extension API which exposes the following two methods:

public void onUserPresence(User user);
public void onClusterMessage(Message message);

The callback onUserPresence is called whenever a user connects or disconnects from a cluster member. This method is also called whenever a connected user changes its list of subscribed subjects (by subscribing to new subjects or unsubscribing from existing subjects). MigratoryData replicates the user presence information across the cluster and calls this method for each cluster member, not only for the cluster member to which the user connected to or disconnected from. The argument of the method provides the details about the user as follows:

interface User {
public String getExternalToken();
public long getSessionId();
public List<String> getSubjects();
public boolean isOffline();
public Map<String, Object> getAdditionalInfo();
}

The callback onClusterMessage is called whenever a MigratoryData cluster accepts a message received from a MigratoryData client. The message is always accepted by the cluster member which is the coordinator of the subject of the message. So, this method is called once, only by the cluster member which is the coordinator of the subject of the accepted message. A client may publish a message to any cluster member, however, the MigratoryData cluster will forward the message to the cluster member which is the coordinator of the subject of the message in order to accept the message as further detailed in the next section Coordinators, Ordering, and Fault Tolerance. The argument of the method provides the details about the accepted message as follows:

interface Message {
public String getSubject();
public byte[] getContent();
public Map<String, Object> getAdditionalInfo();
}

Coordinators and Ordering

In order to keep unique the publication order, the MigratoryData cluster elects a cluster member as a coordinator for each subject. Therefore, each cluster member coordinates a distinct subset of subjects. So, at any given time, a subject is coordinated by at most one cluster member, always the same until that cluster member fails or is stopped. When a cluster member fails or is stopped, its subjects are automatically redistributed to the remaining cluster members such that there are no two cluster members which coordinate simultaneously any given subject.

An incoming message on a subject is forwarded to the coordinator of that subject which assigns a sequence number, notify the presence callback onClusterMessage, then broadcast the message to the entire cluster, such that each cluster member will send that message to all online users who subscribed to the subject of that message as further detailed in the diagram below.

Note that onClusterMessage – which is used to send notifications to offline users – is always called by exactly one cluster member, the coordinator of the subject of the message which is accepted by the cluster. If the coordinator of a subject fails, then another coordinator is automatically elected, and the method onClusterMessage is called by the newly elected coordinator. Therefore, the notification mechanism is fault tolerant.

Workflow Example

The diagram below presents a workflow example for both realtime and anytime messaging.

A chat example

A chat system consisting of a chat mobile app for Android and iOS as well as a plugin built with MigratoryData Presence Extension API are available from our github repository at:

https://github.com/migratorydata

This chat system uses MigratoryData’s realtime messaging for online users and Push Notifications via Firebase Cloud Messaging (FCM) for offline users.