Wednesday, December 5, 2012

How to create a Governance Registry cluster with WSO2 ELB.


G-Reg clustering is different from the usual clustering from other WSO2 products like ESB, AS,..etc. Therefore, this blog post is to provide the basic guide for this. Following details are based on WSO2 Governance Registry 4.5.3 and WSO2 ELB 2.0.3.

wso2 Governance Registry Clustering.


1.We are going to use MySQL as the registry database, but MySQL cluster is  most suitable for the production deployment.

Open a terminal and log-in to the MySQL server.

create database governancedb;

mysql -u root -p governancedb   < wso2greg-4.5.3/dbscripts/mysql.sql

2. Create JDBC mounts(Config and Governance registry spaces) from all the "Read Write" G-Reg nodes to registry database("governancedb") of the cluster.

i) Open the master-datasources.xml file under wso2greg-4.5.3/repository/conf/datasources. Add the following <datasource> config under <datasources> configs.

<datasource>
            <name>WSO2_CARBON_DB_mount</name>
            <description>The datasource used for registry and user manager</description>
            <jndiConfig>
                <name>jdbc/WSO2CarbonDB_mount</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/governancedb</url>
                    <username>root</username>
                    <password>root123</password>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
</datasource>

ii) Add following configs to registry.xml file under wso2greg-4.5.3/repository/conf.

<dbConfig name="wso2registry_mount">
        <dataSource>jdbc/WSO2CarbonDB_mount</dataSource>
 </dbConfig>


<remoteInstance url="https://localhost:9443/registry">
        <id>instanceid</id>
        <dbConfig>wso2registry_mount</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
</remoteInstance>


<mount path="/_system/config" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/config</targetPath>
    </mount>

<mount path="/_system/governance" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/governance</targetPath>
</mount>

iii) Open the axis2.xml file under wso2greg-4.5.3/repository/conf/axis2 and enable clustering.

 (G-Reg use local instance cache and cache invalidation messages to update the cache across the cluster nodes. Axis2 cluster messages are used to send those invalidation messages)

<clustering class="org.apache.axis2.clustering.tribes.TribesClusteringAgent" enable="true">

iv) If you are going to run the multiple G-Reg servers in same machine, you should change the port <Offset> value in carbon.xml file to avoid the port conflicts.

3. All the G-reg nodes in cluster  are used embedded H2 as the local registry.
(No need additional configurations, because default database settings are configured to embedded H2 database.)

4. Copy the mysql jdbc driver to wso2greg-4.5.3/repository/components/lib.

5. Configure the WSO2 Load balance server in front of the G-Reg nodes.

i) Download the WSO2 Elastic Load Balancer(WSO2 ELB).

ii) Open the loadbalancer.conf file under wso2elb-2.0.3/repository/conf.

iii) Add the governance service under <services> config.

governance {
                domains {

                        wso2.governance.domain {
                                hosts           governance.cluster.wso2.com;
                                tenant_range    *;
                        }
                }
        }

iv) Open the axis2.xml file and add configure the  localMemberHost.(This is to be exposed to members of the cluster.)
<parameter name="localMemberHost">127.0.0.1</parameter>
v) Update the host file.
127.0.0.1   governance.cluster.wso2.com
vi) Start the WSO2 ELB instance.
 Go to the wso2elb-2.0.3/bin and execute the wso2server.sh or wso2server.bat.

6. Configure Governance Registry nodes.

i) Open axis2.xml of G-Reg node1 and configure the "membershipScheme" as "wka".
<parameter name="membershipScheme">wka</parameter>
ii) Configure the "domain" ,"localMemberHost", and "localMemberPort" as follow in the axis2.xml.
<parameter name="domain">wso2.governance.domain</parameter>
<parameter name="localMemberHost">governance.cluster.wso2.com</parameter>
<parameter name="localMemberPort">4250</parameter>
iii) Open the axis2.xml file of node1 and add the following details of LB.
<members>
   <member>
      <hostName>127.0.0.1</hostName> <!-- LB host name-->
      <port>4000</port> <!--LB  local member port -->
   </member>
</members>
iv) Open the catalina-server.xml file of node1 and add the proxyPort attribute in
HTTP and HTTPS connectors as shown below.

  In HTTP Connector : proxyPort="8280"
  In HTTPS Connector : proxyPort="8243"

 v) Update the  "HostName" and  "MgtHostName" element in carbon.xml as shown below.
<HostName>governance.cluster.wso2.com</HostName>
<MgtHostName>governance.cluster.wso2.com</MgtHostName>
vi)Open the user-mgt.xml file of node1 and update the "datasource" property as follow. Now user manager tables also will create in "governancedb".
<Property name="dataSource">jdbc/WSO2CarbonDB_mount</Property>

vi) Repeat the same steps for node2 and node3 (You only need to change the value of the "localMemberPort" to 4251 and 4252 in axis2.xml).

vii) Start  the G-Reg node1 ,node2 & node3.

viii) After started the node1 and node2, you should see the following type of logs in LB console.
2013-02-19 01:06:55,647]  INFO - RpcMembershipRequestHandler Received JOIN message from 127.0.0.1:4250(wso2.governance.domain)
[2013-02-19 01:06:55,648]  INFO - MembershipManager Application member 127.0.0.1:4250(wso2.governance.domain) joined group wso2.governance.domain
[2013-02-19 01:07:06,652]  INFO - DefaultGroupManagementAgent Application member Host:127.0.0.1, Port: 4250, HTTP:9764, HTTPS:9444, Domain: wso2.governance.domain, Sub-domain:null, Active:true joined application cluster


[2013-02-19 01:11:07,423]  INFO - RpcMembershipRequestHandler Received JOIN message from 127.0.0.1:4251(wso2.governance.domain)
[2013-02-19 01:11:07,423]  INFO - MembershipManager Application member 127.0.0.1:4251(wso2.governance.domain) joined group wso2.governance.domain
[2013-02-19 01:11:18,425]  INFO - DefaultGroupManagementAgent Application member Host:127.0.0.1, Port: 4251, HTTP:9765, HTTPS:9445, Domain: wso2.governance.domain, Sub-domain:null, Active:true joined application cluster

ix) Point your browser to https://governance.cluster.wso2.com:8243/carbon .

Note: You can set up  WSO2 ELB cluster to achieve the high availability of the LB layer.

(**) 
If you are going to use any other load balancer , then you only need the above (1),(2),(3) & (4) steps.  after that you can set up your own load balancer in front of the G-Reg nodes to make the accessibility through the single URL to handle the high load. 

(**)
If you need to achieve the high availability, then you can create the virtual IP and server pool to include the G-Reg servers.

Download the sample configs : http://sdrv.ms/VsgfVl