Monday, March 16, 2015

WSO2 Message Broker 3.0.0 Slot Based Architecture

The major problem we had in MB 2.2.0 is that message copying from Global queues to Node queues and vice versa.  This consumes lots of time since each message goes through several database readings and writings before delivering.
As a solution to this problem slot based message delivery system was introduced to WSO2 MB 3.0.0. Slot is basically a chunk of messages in a message store which can only be owned by one node at a time. A queue is mapped to a row in message store and queue can be divided into several slots. Below diagram shows how slots are divided for a queue name foo. All the messages for the queue foo is stored in one row in message store.


In new message delivery model slot manager acts as the coordinator of slot distribution among subscribers. Above use case diagram shows the services provided by the slot manager. At publisher’s side publisher returns its last message ID to slot manager after every 1000 messages (this value is configurable) or after a timeout. Slot manager keeps these IDs in a hazelcast distributed map and use it to generate slots when a subscriber asks.

At subscriber’s side, subscriber talks to slot manager at several times.

  1. Get a slot
When a subscriber arrives, subscriber asks for a slot from slot manager. If there are free slots which are not allocated to any node, the slot manager returns a slot to subscriber node and update the slot assignment map . Slot assignment map is kept to trace currently assigned non-empty slots.

  1. Delete slot
When all the messages read from slot are sent and all the acks are received, member node asks the slot manager to delete the slot. At slot manager’s side, when a slot delete call is triggered, slot manager removes the entry from slot assignment map.

  1. Re-assign slot when last subscriber leaves
This method is called when the last subscriber of a particular node leaves the cluster. When this method is triggered, slot manager reassign the non-empty slots which were belonged to callee node, to free slots pool.

Other than these functions, when a member node leaves the cluster, slot manager reassign the non-empty slots which were belonged to the leaving node, to free slots pool.

Message Publishing

When publishing messages, a row in message store is dedicated to each queue. Row key is equal to queue name as shown in the image. All the publishers for that queue will store messages in that row.

Message Delivering

Message delivering is done by a thread known as Slot Delivery Worker(SDW). There can be more than one Slot Delivery Workers and each slot delivery worker is assigned with set of queues. Each slot delivery worker ask for a slot from slot manager when a subscriber arrives. If the slot delivery worker gets a non-empty slot, it reads all the messages in the slot in one row and deliver it to any subscriber in round robin manner. If a message delivery is failed or rejected those messages will be buffered in a queue at Message Flusher which is responsible of delivering messages subscribers. These messages are buffered queue wise in Message
Flusher. Slot Delivery Worker read the messages from the slot and pass it to the Message Flusher. Message Flusher passes these messages to its queue wise separated message buffers. Message slot state diagram is shown in the below diagram.

Message Flow state diagram

Saturday, June 15, 2013

Send Messages to a Given Sequence by Message Injector Task in WSO2 ESB

The third project I was given in ESB was an improvement to Message Injector task in ESB. A task can be scheduled on the ESB to execute periodically. The Message Injector can inject the message to ESB environment. At the time of assigning this project ESB Message Injector could only inject the message to the main sequence and that message should be routed from the main sequence. Therefore it was hard to run lots of tasks in the same time, because there was no way to directly inject a message to a sequence other than the main sequence.

My task was to improve Message Injector task in a way that it can directly send messages to a given sequence. Message Injector task reads some properties from its configuration when it is defined. Therefore I added a new property called “sequence” to task configuration. This sequence property is an optional requirement. If sequence property is not given in the configuration messages will be injected to the main sequence. That means the default sequence of the Message Injector task is the main sequence. 
Message Injector Architecture
I added a new attribute called “injectSequence” to MessageContext. Getters and setters of the injectSequence attribute are included in Axis2MessageContext class which is a child class of MessageContext class. Before Message injector class calls the injectMessage method in Axis2SyanapseEnvironment class, injectSequence attribute is set. If the injectSequence attribute is no null, message will be injected to the sequence specified by injectSequence attribute at the injectMessage method. Above figure shows what I described above. Below figure shows the User interface of the Message Injector task with sequence property. 

User Interface of Message Injector Task