How to handle Mina Exception Logs in WSO2 APIM Gateway

 When I work in WSO2 I have seen some customers are facing the below error in their APIM Gateway nodes intermittently. 

"ERROR {org.wso2.andes.transport.network.mina.MinaNetworkHandler} - Exception caught by Mina java.io.IOException: Connection timed out"

This error could occur when there is a firewall between the Gateway node and Traffic Manager(TM) node and connections getting into an idle state because of the network inactivity. In such Idle states, the firewall breaks the connection, hence this error can occur.

In order to prevent this issue, we need to keep the connection in an active state. Therefore, we can achieve this by establishing a heartbeat between two servers. Once the heartbeat is configured in the client, it periodically sends small frames to the server in order to verify the connection status. Since this is a small frame, it won't generate considerable network traffic. Please follow the below steps to configure the heartbeat value as 60 seconds.

• Update the <APIM_HOME>/repository/conf/deployment.toml file as below of the Traffic Manager nodes.

[qpid.heartbeat] 

delay = 60

timeout_factor = 2.0

• Restart all the APIM-TM nodes and gateway nodes. 

Ref: https://apim.docs.wso2.com/en/latest/reference/config-catalog/#qpid-broker-heartbeat-configurations

Configuring XA transactions with mysql database in ESB

In this example scenario I'm showing you how to configure XA-Transaction with ESB using mysql database. First you need to login to the mysql database and create a database call "test". Inside that database execute the following command.

CREATE TABLE Persons
(
PersonID int,
Name varchar(255)
);

Use case:

1. create new transaction
2. insert into database
3. Call a rest endpoint. 
4. When response is received commit the  transaction.

When configuring XA transactions the best practice is to use datasource class name instead of using the driver class name. Please follow the below instructions to configure master-datasources.xml with datasource class name.

• Open <ESB_HOME>/repository/conf/datasources/master-datasources.xml.
• Add the below configuration under <datasources> tag. (You can change the datasource name to any name you like.)

<datasource>
    <name>MYSQLDS</name>
    <jndiConfig>
        <name>jdbc/MYSQLDS</name>
    </jndiConfig>
    <definition type="RDBMS">
        <configuration>
            <dataSourceClassName>com.mysql.jdbc.jdbc2.optional.MysqlXADataSource</dataSourceClassName>
            <defaultAutoCommit>false</defaultAutoCommit>
            <maxActive>150</maxActive>
            <InitialSize>100</InitialSize>
            <maxWait>60000</maxWait>
            <validationQuery>select 1</validationQuery>
            <validationInterval>30000</validationInterval>
            <testWhileIdle>true</testWhileIdle>
            <maxIdle>25</maxIdle>
            <minIdle>20</minIdle>
            <removeAbandonedTimeout>60</removeAbandonedTimeout>
            <removeAbandoned>true</removeAbandoned>
            <logAbandoned>true</logAbandoned>
            <testOnBorrow>true</testOnBorrow>
            <accessToUnderlyingConnectionAllowed>true</accessToUnderlyingConnectionAllowed>
            <dataSourceProps>
                <property name="serverName">localhost</property>
                <property name="portNumber">3306</property>
                <property name="databaseName">test</property>
                <property name="user">root</property>
                <property name="password">root</property>
            </dataSourceProps>
        </configuration>
    </definition>
</datasource>

• Copy mysql driver library to <ESB_HOME>/repository/components/lib folder. 
• Re-start the ESB server. 

Below is the ESB API configuration for this use case.

 <api context="/bpmn" name="xa_transaction_example">

        <resource faultSequence="faultSequenceREST" methods="POST" uri-template="/test">

            <inSequence>

                <log level="custom">

                    <property name="text" value="** In Sequencce started**"/>

                </log>

                <transaction action="new"/>

                <dbreport useTransaction="true">

                    <connection>

                        <pool>

                            <dsName>jdbc/MYSQLDS</dsName>

                        </pool>

                    </connection>

                    <statement>

                        <sql><![CDATA[INSERT INTO Persons (PersonID,Name) VALUES ('1234','Tom')]]></sql>

                    </statement>

                </dbreport>

                <call blocking="true">

                    <endpoint>

                        <address uri="http://www.mocky.io/v2/577f4bc2100000152df26247"/>

                    </endpoint>

                </call>

                <log level="full"/>

                <transaction action="commit"/>

                <log level="custom">

                    <property name="text" value="**Transaction commited**"/>

                </log>

                <respond/>

            </inSequence>

        </resource>

    </api>

Get Redirection URL from ESB

In this article I will show you how to get redirection URL from ESB. Assume that you call a back-end service from ESB proxy service and that back-end service re-direct to another service. In that case you will get a response from the server something like below.

HTTP/1.1 308 Redirection (308)[\r][\n]"
Server: Cowboy[\r][\n]
Connection: close[\r][\n]
Location: http://www.mocky.io/v2/57889ef20f0000701dbd16e3DDDD[\r][\n]
Content-Type: application/xml; charset=utf-8[\r][\n]
Date: Mon, 18 Jul 2016 05:40:21 GMT[\r][\n]
Via: 1.1 vegur[\r][\n]
[\r][\n]
<hi></hi>

If you want to get the re-direction url which is with the "Location" header,  you can use the the property mediator as below. 

 <property xmlns:ns="http://org.apache.synapse/xsd"
                   name="Location"
                   expression="$axis2:PRE_LOCATION_HEADER"/>

Enable Mutual SSL with WSO2 ESB and IBM MQ

This article will explain how to connect to an SSL enabled channel in MQ using WSO2 ESB. Here I’m using ESB version 4.8.1 and MQ 8.0.0.2 versions. Java version java 1.7.0-60 or above.

• First we have to import the MQ certificate to WSO2 ESB’s client trust store. You can import the certificate using the following command in keytool.
• ./keytool -import -file PATH_TO_MQ_CERTIFICATE/USU08QM2.cer -alias ALIAS_NAME_OF_MQ_CERTIFICATE -keystore PATH_TO_CLIENT_TRUST_STORE/client-truststore.jks -storepass password

• Then you have to export the certificate from wso2carbon.jks and import that to MQ keystore in MQ server. To export the certificate from wso2carbon.jks use the following command in keytool.

•  ./keytool -export -keystore PATH_TO_KEY_STORE/wso2carbon.jks -storepass KEY_SOTRE_PASSWORD -alias ALIAS_NAME_OF_WSO2_KEY_STORE -file PATH_TO_STORE_THE_CERTIFICATE/wso2esb.cer

•  Then this certificate should be should be imported to MQ JKS using the command listed in step 1.

• Then you have to specify the matching cipher suite for ESB. In IBM MQ when SSL is enabled, they specify a cipher spec to the channel. You have to find the matching cipher suite for that cipher spec and specify it in ESB as an environment variable. Cipher spec to cipher suite mapping can be found in here. Environment variable can be set in ESB by setting the following propery in wso2server.sh file.
•  -DCMQC.SSL_CIPHER_SUITE_PROPERTY="SSL_RSA_WITH_3DES_EDE_CBC_SHA" \

• Note:Here I have specified the cipher suite as SSL_RSA_WITH_3DES_EDE_CBC_SHA. The matching cipher spec for this cipher suite is TLS_RSA_WITH_3DES_EDE_CBC_SHA.

• Next you have to copy MQ jta.jar to ESB_HOME/repository/components/lib folder. (Remove the old jar files if there are any existing ones). Make sure you copy MQ 8.0.0.2 jar files. The client jars in [1] is not working with SSL. Therefore you have to create a new OSGI bundle with new MQ 8.0.0.2 client jars. To do that you can clone the wmq-client-8.0.0.2 project from here and copy following jar files to lib folder in wmq-client-8.0.0.2/lib. Then build the project.
• com.ibm.mq.allclient.jar
• fscontext.jar
• jms.jar
• providerutil.jar

• Note: Copy the wmq-client-8.0.0.2.jar file in target directory to ESB_HOME/repository/components/dropins folder. Remove if there are any old jar files.

• Remove following line from <ESB_4.8.1_Home>\repository\conf\etc\launch.ini
• javax.jms,\

• Regenerate .bindings file with following property.
• Provider Version : 8

Then restart the server and then you are ready to go with SSL. !!!


[1] http://nandikajayawardana.blogspot.com/2015/03/configuring-ibm-mq-with-wso2-esb.html

WSO2 Message Broker 3.0.0 Slot Based Architecture

The major problem we had in MB 2.2.0 is that message copying from Global queues to Node queues and vice versa.  This consumes lots of time since each message goes through several database readings and writings before delivering.

As a solution to this problem slot based message delivery system was introduced to WSO2 MB 3.0.0. Slot is basically a chunk of messages in a message store which can only be owned by one node at a time. A queue is mapped to a row in message store and queue can be divided into several slots. Below diagram shows how slots are divided for a queue name foo. All the messages for the queue foo is stored in one row in message store.

Architecture

In new message delivery model slot manager acts as the coordinator of slot distribution among subscribers. Above use case diagram shows the services provided by the slot manager. At publisher’s side publisher returns its last message ID to slot manager after every 1000 messages (this value is configurable) or after a timeout. Slot manager keeps these IDs in a hazelcast distributed map and use it to generate slots when a subscriber asks.

At subscriber’s side, subscriber talks to slot manager at several times.

1. Get a slot

When a subscriber arrives, subscriber asks for a slot from slot manager. If there are free slots which are not allocated to any node, the slot manager returns a slot to subscriber node and update the slot assignment map . Slot assignment map is kept to trace currently assigned non-empty slots.

2. Delete slot

When all the messages read from slot are sent and all the acks are received, member node asks the slot manager to delete the slot. At slot manager’s side, when a slot delete call is triggered, slot manager removes the entry from slot assignment map.

3. Re-assign slot when last subscriber leaves

This method is called when the last subscriber of a particular node leaves the cluster. When this method is triggered, slot manager reassign the non-empty slots which were belonged to callee node, to free slots pool.

Other than these functions, when a member node leaves the cluster, slot manager reassign the non-empty slots which were belonged to the leaving node, to free slots pool.

Message Publishing

When publishing messages, a row in message store is dedicated to each queue. Row key is equal to queue name as shown in the image. All the publishers for that queue will store messages in that row.

Message Delivering

Message delivering is done by a thread known as Slot Delivery Worker(SDW). There can be more than one Slot Delivery Workers and each slot delivery worker is assigned with set of queues. Each slot delivery worker ask for a slot from slot manager when a subscriber arrives. If the slot delivery worker gets a non-empty slot, it reads all the messages in the slot in one row and deliver it to any subscriber in round robin manner. If a message delivery is failed or rejected those messages will be buffered in a queue at Message Flusher which is responsible of delivering messages subscribers. These messages are buffered queue wise in Message Flusher. Slot Delivery Worker read the messages from the slot and pass it to the Message Flusher. Message Flusher passes these messages to its queue wise separated message buffers. Message slot state diagram is shown in the below diagram.

Message Flow state diagram

Send Messages to a Given Sequence by Message Injector Task in WSO2 ESB

The third project I was given in ESB was an improvement to Message Injector task in ESB. A task can be scheduled on the ESB to execute periodically. The Message Injector can inject the message to ESB environment. At the time of assigning this project ESB Message Injector could only inject the message to the main sequence and that message should be routed from the main sequence. Therefore it was hard to run lots of tasks in the same time, because there was no way to directly inject a message to a sequence other than the main sequence.

My task was to improve Message Injector task in a way that it can directly send messages to a given sequence. Message Injector task reads some properties from its configuration when it is defined. Therefore I added a new property called “sequence” to task configuration. This sequence property is an optional requirement. If sequence property is not given in the configuration messages will be injected to the main sequence. That means the default sequence of the Message Injector task is the main sequence. 

Message Injector Architecture

I added a new attribute called “injectSequence” to MessageContext. Getters and setters of the injectSequence attribute are included in Axis2MessageContext class which is a child class of MessageContext class. Before Message injector class calls the injectMessage method in Axis2SyanapseEnvironment class, injectSequence attribute is set. If the injectSequence attribute is no null, message will be injected to the sequence specified by injectSequence attribute at the injectMessage method. Above figure shows what I described above. Below figure shows the User interface of the Message Injector task with sequence property. 

User Interface of Message Injector Task