The section describes the steps required to upgrade MigratoryData 5 to MigratoryData 6.

Changes for the MigratoryData server

This is the list of important changes in MigratoryData 6:

1. Enhanced recovery time for clients after a cluster member

If a server of a MigratoryData cluster, configured in Guaranteed Message Delivery mode, restarts due to a failure or maintenance process, then MigratoryData 6 will synchronize its in-memory cache with the rest of the cluster such that the clients which will fail over on the restarted server will immediately recover their data from the synchronized cache. In MigratoryData 5, the cache of the server which restarts is not synchronized, which means that during a certain amount of time (until the cache of the restarted server is populated again with new messages) the clients which fail over on the restarted server will be redirected to another server of the cluster to recover their data, thus increasing the recovery time for clients.

2. Enhanced message recovery at epoch change

In order to implement Guaranteed Message Delivery, MigratoryData uses sequence numbers and epochs for messages. Each server of the cluster coordinates a distinct subset of subjects. So, at each moment, each subject is coordinated by at most one server of the cluster, called the coordinator of that subject. When the cluster elects a coordinator for a subject, it will assign an epoch number to that subject, which will remain unchanged until the cluster will elect another coordinator for that subject (when the previous coordinator is unreachable due to a failure or a maintenance restart).

In MigratoryData 5, it is guaranteed that a client which subscribes to a topic will recover all messages for the current epoch of that subject, using consecutive sequence numbers. However, when an epoch change occurs for a topic, its sequence number is reset to zero. A client subscribed to that subject and fails over immediately after an epoch change might not get some messages from the previous epoch. In this case, the client reconnects to the cluster as a fresh client and a notification NOTIFY_DATA_RESYNC is triggered for that client to inform it about this situation.

In MigratoryData 6, it is guaranteed that the client will recover all messages, including from the current and previous epoch, if an epoch change occurs. Therefore, the recovery will be possible for most clients, and the number of notifications NOTIFY_DATA_RESYNC should be substantially reduced for clients.

3. Enhanced inter-cluster communication

MigratoryData 6 uses now non-blocking IO (NIO) for the communication between the servers of the cluster. This was a configurable option in MigratoryData 5.

4. Removed audit logging, now available as audit plugin

The new Server Extensions API for Audit has been introduced for building audit plugins for the MigratoryData server to handle audit events, including access events, message events, cache events, and stats events. A basic audit plugin built on log4j is distributed with the server package. It is open source and available from github at:

https://github.com/migratorydata/migratorydata-plugin-audit-log4j

You can use it as an example for building your own audit plugin.

5. Removed conflation and message fields

If necessary, the content of the MigratoryData messages can be structured to include a list of fields. Therefore, MigratoryData 6 does not attach anymore fields information to messages. In this way, the conflation feature, which is strongly related to the message fields, is no longer supported in MigratoryData 6. For applications requiring aggregation of data via conflation, this should be done upstream, at the publisher level.

6. Removed push monitoring along meta-subjects, now available via the audit plugin

In MigratoryData 6 we maintain only the JMX and HTTP monitoring, and removed the push monitoring along meta-subjects. The real-time information provided by the push monitoring is now provided as stats events by the newly introduced Server Extensions API for Audit.

7. Removed the round-trip latency sampling

In MigratoryData 6, the round-trip latency sampling is no longer available. This could be easily implemented at the application level.

8. MigratoryData Server 6 is backward compatible with the client API libraries 5

The applications built with the client libraries 5 are compatible with the MigratoryData server 6, as long as they do not use the features removed by MigratoryData 6: conflation or message fields.

9. Introduced a new, fully-featured bidirectional client API for Python

The previous publish-only REST-based Python client API library, has been replaced with a new fully-featured publish/subscribe client API library.

10. Introduced a new, fully-featured bidirectional client API for PHP

The previous publish-only REST-based Python client API library is maintained. The new publish/subscribe API for PHP depends on ReactPHP.

11. Introduced a new modern Client API for JavaScript (available from NPM)

Because now all modern browsers have support for WebSockets, we have introduced a new, WebSocket-only Client API for JavaScript which is available from NPM.

12. Enhanced MigratoryData Client API for Node.js available from NPM

The Client API for Node.js is available from NPM with TypeScript support.

13. Add message compression support in all client APIs

The compression is available at the message level. Moreover, if the compressed size of a message (for which compression has been enabled) is larger than its uncompressed size, the message is sent uncompressed, to save some CPU cycles on subscriber side.

14. Removed the experimental support for the MQTT protocol

The experimental support for the MQTT protocol, introduced in the beta version of MigratoryData 6, has been removed.

Configuration

The following table details the parameters of the MigratoryData server impacted by the upgrade:

Changes for the MigratoryData Client APIs

1. New structure for MigartoryData messages

• Removed the snapshot property. This information is now provided by the new property type.

• Removed the fields property. Fields have been removed, see server change 5 above.

• Renamed the replyToSubject property to replySubject.

• Added a new retained property. It indicates whether or not the message should be/was retained by the cluster as a snapshot message.

• Added a new compressed property. It indicates whether or not the message should be/was compressed.

• Added a new qos property which indicates the quality of the service (qos) of the message. The possible values are:
  - STANDARD
  - GUARANTEED (default)

• Added a new type property which indicates the type of the message. The possible values are:
  - SNAPSHOT
  - UPDATE (default)
  - HISTORICAL
  - RECOVERED

2. Add a new API method 'connect()'

MigratoryDataClient.connect()

Historically, before WebSockets exist, MigratoryData had to implement the connect operation during the first subscribe operation to optimize two network operations into a single one. Now, this optimization is no longer necessary.

Use this method to connect the client to one of the MigratoryData servers you specified with MigratoryDataClient.setServers(), and subscribe to the subjects you specified with MigratoryDataClient.subscribe(), if any.

3. Add a new API method 'setTransport()'

As WebSockets are now a standard, we have migrated from the HTTP streaming to WebSockets for all client libraries. Also, we maintain the old HTTP streaming transport. MigratoryData 6 introduces a new API method to select one of these two transport types as follows:

MigratoryDataClient.setTransport(type)

The possible transport type values are:
- WEBSOCKET (default)
- HTTP

4. Removed the API method 'subscribeWithConflation()'

The conflation feature has been removed, see server change 5 above.

5. Externalized the logging

A new log interface like MigratoryDataLogListener is now available:

public interface MigratoryDataLogListener {
    void onLog(String, MigratoryDataLogLevel);
}

Therefore logs are provided via the onLog() callback. To attach a log listener to your MigratoryDataClient object, a new method has been introduced:

MigratoryDataClient.setLogListener(
    MigratoryDataLogListener, 
    MigratoryDataLogLevel)

and the old logging implementation available via methods like the following has been removed:

MigratoryDataClient.setLogging(MigratoryDataLogLevel, ...)

6. Removed the interactive listener

The interactive listener was available only in the Java client API of MigratoryData 5 along with a coupling plugin. Both have been removed in MigratoryData 6. This functionality can be now implemented at the application level, using the new Extension SDKs made available by MigratoryData 6.

7. New renamed API method notifyAfterFailedConnectionAttempts()

The methods like setServersDownBeforeNotify(int) and notifyAfterReconnectRetries (int) have been renamed into a new API method as follows:

MigratoryDataClient.notifyAfterFailedConnectionAttempts(int)

Changes for the Server Extensions API

Entitlement

Renamed the package name

The package name has been renamed from com.migratorydata.extension to com.migratorydata.extensions.authorization.

Renamed the jar name

The entitlement plugin has been renamed from extension.jar into authorization.jar.

API changes

The name of the entitlement interface changed from MigratoryDataExtensionListener to MigratoryDataEntitlementListener, and its callbacks changed as follows:

• the subscribe entitlement callback:

boolean onEntitlementSubscribeCheck (String subject, String token)

changed into:

void onSubscribe(MigratoryDataSubscribeRequest);

• the publish entitlement callback:

boolean onEntitlementPublishCheck(String subject, String token);

changed into:

void onPublish(MigratoryDataPublishRequest);

where MigratoryDataSubscribeRequest and MigratoryDataPublishRequest provide not only the subject to subscribe to or publish on and the entitlement token of the user to be authorized, but also other details about the user such as the address of the user.

Also, to perform an authorization, the return value of the callback is no longer used, but this is done using the request argument. For example:

void onPublish(MigratoryDataPublishRequest req) {
      if (req.getSubject().equals("/some/allowed/subject")) {
            req.setAllowed(true);
      }
      req.sendResponse();
}

For more details, please refer to the documentation of the MigratoryData Extension SDK for Entitlement.

Presence

Renamed the package name

The package name has been renamed from com.migratorydata.extension to com.migratorydata.extensions.presence.

Renamed the jar name

The name of the presence plugin has been changed from presence-extension.jar into presence.jar

API changes

There are no changes for the presence API. Nevertheless, a new attribute monotonicId is available as part of the map of attributes MigratoryDataPresenceListener.User.getAdditionalInfo() which can be assigned on the client-side with the following API method:

void setExternalToken(String, MonotonicId) 

The monotonically incremented id generated on the client-side should be used to preserve the order between connect and disconnect presence events. If the monotonicId of a presence event is smaller than the current monotonicId of the user (maintained by the presence plugin), then it means that the received presence event is an old, delayed presence event, so it can be ignored by the presence plugin.

For more details, please refer to the documentation of the MigratoryData Extension SDK for Presence.

Audit

This is a new extension SDK. Please refer to the documentation of the Server Extensions API for Audit.