Upgrade from 5.x to 6

The section describes the steps required to upgrade MigratoryData 5.x to MigratoryData 6.

MigratoryData Server

List of Changes

This is the list of important changes in MigratoryData 6:

#1: Enhanced the recovery time for clients after a cluster member restart

If a server of a MigratoryData cluster, configured in Guaranteed Message Delivery mode, restarts due to a failure or maintenance process, then MigratoryData 6 will synchronize its in-memory cache with the rest of the cluster such that the clients which will fail over on the restarted server will immediately recover their data from the synchronized cache. In MigratoryData 5.x, the cache of the server which restarts is not synchronized, which means that during a certain amount of time (until the cache of the restarted server is populated again with new messages) the clients which fail over on the restarted server will be redirected to another server of the cluster to recover their data, thus increasing the recovery time for clients.

#2: Enhanced message recovery at epoch change

In order to implement Guaranteed Message Delivery, MigratoryData uses sequence numbers and epochs for messages. Each server of the cluster coordinates a distinct subset of subjects. So, at each moment, each subject is coordinated by at most one server of the cluster, called the coordinator of that subject. When the cluster elects a coordinator for a subject, it will assign an epoch number to that subject, which will remain unchanged until the cluster will elect another coordinator for that subject (when the previous coordinator is unreachable due to a failure or a maintenance restart).

In MigratoryData 5.x, it is guaranteed that a client which subscribes to a topic will recover all messages for the current epoch of that subject, using consecutive sequence numbers. However, when an epoch change occurs for a topic, its sequence number is reset to zero. A client subscribed to that subject and which fails over immediately after an epoch change might not get some messages from the previous epoch. In this case, the client reconnects to the cluster as a fresh client and a notification NOTIFY_DATA_RESYNC is triggered for that client to inform it about this situation.

In MigratoryData 6, it is guaranteed that the client will recover all messages, including from the current and from previous epoch, if an epoch change occurs. Therefore, the recovery will be possible for most clients, and the number of notifications NOTIFY_DATA_RESYNC should be substantially reduced for clients.

#3: Enhanced inter-cluster communication

MigratoryData 6 uses now non-blocking IO (NIO) for the communication between the servers of the cluster. This was a configurable option in MigratoryData 5.x.

Note

The parameters ClusterCommunication.NIO* have been removed in the MigratoryData server 6.

#4: Removed audit logging, now available as audit plugin

The new MigratoryData Extension SDK for Audit has been introduced for building audit plugins for the MigratoryData server to handle audit events, including access events, message events, cache events, and stats events. A basic audit plugin built on log4j is distributed with the server package. It is open source and available from github at:

You can use it as an example for building your own audit plugin.

Note

A new parameter Extension.Audit.Stats has been introduced. The parameters AccessLog, CacheLog, and MessageLog have been renamed into Extension.Audit.Access, Extension.Audit.Cache, and respectively Extension.Audit.Message. Also, the parameters AccessLog.*, CacheLog.*, MessageLog.*, and PublishLog* have been removed.

#5: Removed conflation and message fields

If necessary, the content of the MigratoryData messages can be structured to include a list of fields. Therefore, MigratoryData 6 does not attach anymore fields information to messages. In this way, the conflation feature, which is strongly related to the message fields, is no longer supported in MigratoryData 6. For applications requiring aggregation of data via conflation, this should be done upstream, at the publisher level.

Note

The method MigratoryDataClient.subscribeWithConflation() of the client API has been removed.

#6: Removed push monitoring along meta-subjects, now available via the audit plugin

In MigratoryData 6 we maintain only the JMX and HTTP monitoring, and removed the push monitoring along meta-subjects. The real-time information provided by the push monitoring is now provided as stats events by the newly introduced MigratoryData Extension SDK for Audit.

Note

The parameters MonitorPUSH.* have been removed.

#7: Removed the round-trip latency sampling

In MigratoryData 6, the round-trip latency sampling is no longer available. This could be easily implemented at the application level.

Note

The parameters PublishLog* have been removed.

#8: MigratoryData Server 6 is backward compatible with the client API libraries 5.x

The applications built with the client libraries 5.x are compatible with the MigratoryData server 6, as long as they do not use the features removed by MigratoryData 6: conflation or message fields.

#9: Introduced a new, fully-featured client API for Python

The previous publish-only REST-based Python client API library, has been replaced with a new fully-featured publish/subscribe client API library.

#10: Introduced a new, modern WebSocket-only client SDK for JavaScript (available from NPM)

Because now all modern browsers have support for WebSockets, we have introduced a new, modern WebSocket-only client SDK for JavaScript which is available from NPM.

Note

For applications requiring publish/subscribe real-time web messaging support for modern and old browsers, with and without WebSocket support, you can continue to use the MigratoryData Client SDK for JavaScript available from the MigratoryData downloads portal, and which will be called the legacy version.

#11: Enhanced MigratoryData client SDK for Node.js available from NPM

The client SDK for Node.js is available from NPM with TypeScript support.

#12: Removed the experimental support for the MQTT protocol

The experimental support for the MQTT protocol, introduced in the beta version of MigratoryData 6, has been removed.

Configuration

The following table details the parameters of the MigratoryData server impacted by the upgrade:

MigratoryData 5.x

MigratoryData 6

Description

SocketBufferLimit

MaxMessageSize

Renamed

MonitorPUSH.*

-

Removed, see change #6

PublishSnapshotMessage

-

Removed, see API change #a1

ClusterCommunication.NIO.*

-

Removed, see change #3

AccessLog

Extension.Audit.Access

Renamed, see change #4

MessageLog

Extension.Audit.Message

Renamed, see change #4

CacheLog

Extension.Audit.Cache

Renamed, see change #4

-

Extension.Audit.Stats

New, see change #4

AccessLog.*

-

Removed, see change #4

MessageLog.*

-

Removed, see change #4

CacheLog.*

-

Removed, see change #4

PublishLog*

-

Removed, see change #4

MigratoryData Client SDKs

List of Changes

#1: New structure for MigartoryData messages

  • Removed the snapshot property. This information is now provided by the new property type.

  • Removed the fields property. Fields have been removed, see server change #5.

  • Renamed the replyToSubject property to replySubject.

  • Added a new retained property. It indicates whether or not the message should be/was retained by the cluster as a snapshot message.

  • Added a new qos property which indicates the quality of the service (qos) of the message. The possible values are:

    • STANDARD

    • GUARANTEED (default)

  • Added a new type property which indicates the type of the message. The possible values are:

    • SNAPSHOT

    • UPDATE (default)

    • HISTORICAL

    • RECOVERED

#2: Add a new API method ‘connect()’

MigratoryDataClient.connect()

Historically, before WebSockets exist, MigratoryData had to implement the connect operation during the first subscribe operation to optimize two network operations into a single one. Now, this optimization is no longer necessary.

Use this method to connect the client to one of the MigratoryData servers you specified with MigratoryDataClient.setServers(), and subscribe to the subjects you specified with MigratoryDataClient.subscribe(), if any.

#3: Add a new API method ‘setTransport()’

As WebSockets are now a standard, we are migrating from the HTTP streaming to WebSockets for all client libraries. Also, we maintain the old HTTP streaming transport. MigratoryData 6 introduces a new API method to select one of these two transport types as follows:

MigratoryDataClient.setTransport(type)

The possible transport type values are:

  • WEBSOCKET (default)

  • HTTP

Note

Currently the client libraries for JavaScript, Java, and Node.js implements both the WebSockets and HTTP transports. The legacy client library for JavaScript implements fallback mechanisms from WebSockets to HTTP streaming, so setting the transport type for the legacy JavaScript client library is not necessary.

#4: Removed the API method ‘subscribeWithConflation()’

The conflation feature has been removed, see server change #5.

#5: Externalized the logging

A new log interface like MigratoryDataLogListener is now available:

public interface MigratoryDataLogListener {
      void onLog(String, MigratoryDataLogLevel);
}

Therefore logs are provided via the onLog() callback. To attach a log listener to your MigratoryDataClient object, a new method has been introduced:

MigratoryDataClient.setLogListener(MigratoryDataLogListener, MigratoryDataLogLevel)

and the old logging implementation available via methods like the following has been removed:

MigratoryDataClient.setLogging(MigratoryDataLogLevel, ...)

#6: Removed the interactive listener

The interactive listener was available only in the Java client API of MigratoryData 5.x along with a coupling plugin. Both have been removed in MigratoryData 6. This functionality can be now implemented at the application level, using the new Extension SDKs made available by MigratoryData 6.

#7: New renamed API method notifyAfterFailedConnectionAttempts()

The methods like setServersDownBeforeNotify(int) and notifyAfterReconnectRetries (int) have been renamed into a new API method as follows:

MigratoryDataClient.notifyAfterFailedConnectionAttempts(int)

MigratoryData Plugin SDKs

Entitlement

Renamed the package name

The package name has been renamed from com.migratorydata.extension to com.migratorydata.extensions.authorization.

Renamed the jar name

The entitlement plugin has been renamed from extension.jar into authorization.jar.

API changes

The name of the entitlement interface changed from MigratoryDataExtensionListener to MigratoryDataEntitlementListener, and its callbacks changed as follows:

  • the subscribe entitlement callback:

boolean onEntitlementSubscribeCheck (String subject, String token)

changed into:

void onSubscribe(MigratoryDataSubscribeRequest);
  • the publish entitlement callback:

boolean onEntitlementPublishCheck(String subject, String token);

changed into:

void onPublish(MigratoryDataPublishRequest);

where MigratoryDataSubscribeRequest and MigratoryDataPublishRequest provide not only the subject to subscribe to or publish on and the entitlement token of the user to be authorized, but also other details about the user such as the address of the user.

Also, to perform an authorization, the return value of the callback is no longer used, but this is done using the request argument. For example:

void onPublish(MigratoryDataPublishRequest req) {
      if (req.getSubject().equals("/some/allowed/subject")) {
            req.setAllowed(true);
      }
      req.sendResponse();
  }

For more details, please refer to the documentation of the MigratoryData Extension SDK for Entitlement.

Presence

Renamed the package name

The package name has been renamed from com.migratorydata.extension to com.migratorydata.extensions.presence.

Renamed the jar name

The name of the presence plugin has been changed from presence-extension.jar into presence.jar

API changes

There are no changes for the presence API. Nevertheless, a new attribute monotonicId is available as part of the map of attributes MigratoryDataPresenceListener.User.getAdditionalInfo() which can be assigned on the client-side with the following API method:

void setExternalToken(String, MonotonicId)

The monotonically incremented id generated on the client-side should be used to preserve the order between connect and disconnect presence events. If the monotonicId of a presence event is smaller than the current monotonicId of the user (maintained by the presence plugin), then it means that the received presence event is an old, delayed presence event, so it can be ignored by the presence plugin.

For more details, please refer to the documentation of the MigratoryData Extension SDK for Presence.

Audit

This is a new extension SDK. Please refer to the documentation of the MigratoryData Extension SDK for Audit.