This chapter describes the core concepts of MigratoryData KE such as:

  • Publishers and Subscribers
  • Subjects and Subjects-Topics Mapping
  • Messages
  • Snapshot Message

Publishers and Subscribers

MigratoryData KE implements a subject-based publish/subscribe model. Consumers of realtime data are subscribers to particular subjects, and producers of realtime data are publishers to these subjects.

Subjects and Subjects-Topics Mapping

A subject is a string of characters that respects a syntax similar to the Unix absolute paths. It consists of an initial slash (/) character followed by one or more strings of characters separated by the slash (/) character, called segments. Within a segment, the slash (/) character is reserved.

For example, the following string /Stocks/NYSE/IBM, composed by the segments Stocks, NYSE, and IBM is a valid subject for MigratoryData KE .

Moreover, because the first segment of a subject is mapped to a Kafka topic, and the remaining segments are mapped to the key of that Kafka topic, the first segment must respect the Kafka syntax for topics, i.e. it must be non-empty, and use at most 249 of the following characters [a-zA-Z0-9._-]. The remaining segments can use any UTF-8 characters because there is no syntax restriction for keys in Kafka. If a subject consists of a single segment, then the key of the Kafka topic given by the first segment is considered as null.

Here are some examples of invalid subjects:

Invalid Subject Reason
/Stocks//IBM/BID The slash (/) is not allowed in a segment or the 2nd segment is empty
Stocks/IBM/BID The subject does not start with a slash (/) character
/Stocks/ The second segment is empty
/Stocks/IBM/BID/ The last segment is empty
/ The first segment is empty
/Stocks! The first segment contains ! which is not in [a-zA-Z0-9._-]

Here are some examples of mappings between MigratoryData subjects and Kafka topics:

Subject Topic and Key
/vehicles/1 The Kafka topic is vehicles and the key of the topic is 1
/vehicles/1/speed The Kafka topic is vehicles and the key of the topic is 1/speed
/vehicles The Kafka topic is vehicles and the key is null

Messages

A message consists of several pieces of information as follows:

Property Description
subject the subject of the message
content the content of the message
closure optional metadata (e.g. the ID of the message)
retained indicate whether or not the message should be/was retained
compressed indicate whether or not the message should be/was compressed
qos indicate the quality of the service of the message, which can be:
- QoS_STANDARD (at-most-once delivery semantics)
- QoS_GUARANTEED (at-least-once delivery semantics)
type the type of the message, which can be:
- snapshot
- update
- historical
- recovered
replySubject a subject to reply with a response to a request message

The Client SDKs provide methods to create messages from application-specific data, publish messages, and retrieve the application-specific data from the messages when received by the subscriber.

Snapshot Message

For each subject, MigratoryData KE maintains a snapshot message which is the most recent message received by MigratoryData KE having the property retained on true.

The following tables shows the snapshot message of the subject /Stocks/NYSE/IBM as new messages are received by MigratoryData KE.

Time Received Message Snapshot Message
10:12
(first message)
subject=/car/speed
content=110
retained=false
No snapshot available
for /car/speed
10:13 subject=/car/speed
content=70
retained=false
No snapshot available
for /car/speed
at this time
10:15 subject=/car/speed
content=121
retained=true
subject=/car/speed
content=121
retained=true
10:25 subject=/car/speed
content=144
retained=false
subject=/car/speed
content=121
retained=true
10:40 subject=/car/speed
content=102
retained=true
subject=/car/speed
content=102
retained=true

When a client subscribes to a subject, MigratoryData KE will firstly send to that subscriber the snapshot message of that subject (if available), then it will send the subsequent real-time messages for that subject as they are received by MigratoryData KE from a publisher.