Sunday, August 16, 2015

[WSO2 AM] API Manager distributed setup with Apache proxy

In this post I'm going to explain about how to setup API Manager distributed setup with Apache proxy. (The configuration details include for  both Windows and Linux environments)


  • The proxy1 used to accept the API request and distribute among the gaeway nodes.
  • The proxy2 used to distribute the key validation and authentication request among the manager nodes.
  • The proxy3 used to publish API to gateway nodes.
  • SVN/Rsync used to synchronize the API artifacts xml across the gateway nodes.
  • The registry(config and governance), user management and API manager database shared across all the nodes. 
Required applications:

1. Java 1.7.X
2 Apache2
3. Open SSL
4. MySQL

 1.0 Install and configure the Apache2 on Windows

i) You can download and install XAMP (https://www.apachefriends.org/index.html), that provides a control panel to manage the Apache server on Windows.

 Open the httpd.conf file (C:\xampp\apache\conf) and enable the following modules.
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule ssl_module modules/mod_ssl.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
LoadModule lbmethod_byrequests_module modules/mod_lbmethod_byrequests.so
LoadModule slotmem_shm_module modules/mod_slotmem_shm.so
 ii) In Linux environment.

Install Apache2 using the following command.
$sudo apt-get install apache2
To enable the modules.
$sudo a2enmod proxy_http
$sudo a2enmod ssl
$sudo a2enmod proxy_balancer
$sudo a2enmod lbmethod_byrequests
$sudo a2enmod slotmem_shm
$sudo a2enmod proxy_wstunnel 
2.0 Create virtual host configuration for proxy-1.

i) In  windows,  virtual host file (eg: gw.wso2am.conf) should be saved in C:\xampp\apache\conf\extra directory.  (in Linux /etc/apache2/site-available).
<VirtualHost gw.wso2am:443>

 ServerAdmin abc@wso2.com
 ServerName  gw.wso2am
 ServerAlias gw.wso2am
 ProxyRequests Off
 SSLEngine On
 SSLProxyEngine On
 #SSLProxyVerify none
 SSLProxyCheckPeerCN off
 SSLProxyCheckPeerName off
 #SSLProxyCheckPeerExpire off

 SSLCertificateFile /home/ajith/ca.crt
 SSLCertificateKeyFile /home/ajith/ca.key

<Proxy balancer://gw.wso2am>
        BalancerMember https://gw1.wso2am:8243 route=gwNode1 loadfactor=1
        BalancerMember https://gw2.wso2am:8244 route=gwNode2 loadfactor=1
        ProxySet lbmethod=byrequests
        ProxySet stickysession=JSESSIONID|jsessionid
</Proxy>

ProxyPass /  balancer://gw.wso2am/
ProxyPassReverse / balancer://gw.wso2am/

</VirtualHost>
ii) In windows,  include this virtual host configuration in httpd.conf file. (Find the # Virtual hosts configuration section)
Include conf/extra/gw.wso2am.conf
iii) In linux you can use the following command.
sudo a2ensite gw.wso2am.conf
3.0 Create certificate file and key for the proxy-1.

i) You can download the open ssl for windows (https://code.google.com/p/openssl-for-windows/) . Exact the zip file and set the path variable to bin directory.

Generate key:
openssl genrsa -out ca.key 1024 (windows)
sudo openssl genrsa -out ca.key 1024 (linux)
Generate CSR (Certificate Signing Request ).
  • Here you need to enter CSR details and make sure to enter the gw.wso2am as the Common Name (CN).
openssl req -new -key ca.key -out ca.csr (windows)
sudo openssl req -new -key ca.key -out ca.csr (linux)
Generate self sign certificate.
openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt (Windows)
sudo openssl x509 -req -days 365 -in ca.csr -signkey ca.key -out ca.crt (linux)


  • The ca.crt and ca.key has configured as SSLCertificateFile  and SSLCertificateKeyFile in above virtual host configuration.
  •  If the CN (Common Name) of the back end server certificate is different from the virtual host name, then you need to disable the SSLProxyCheckPeerCN and SSLProxyCheckPeerName in the virtual host configuration.
4.0 Create new key store for gateway nodes.

i) Generate new key store.
  •  Make sure to enter the gw.wso2am as the first and last name , that value will be set as common name (CN).
  •  Enter the key store password as wso2carbon , then you don't need to change the default key store configurations in API Manager.
keytool -genkey -keyalg RSA -alias wso2carbon -keystore wso2carbon.jks -storepass wso2carbon -validity 360 -keysize 2048

ii) Export the public  key.
keytool -export -alias wso2carbon -keystore wso2carbon.jks -storepass wso2carbon -file wso2carbon.pem



iii) Copy this new  wso2carbon.jks and wso2carbon.pem files to repository\resources\security directory of the gateway nodes.

iv) Go to the repository\resources\security directory from command line and execute the following command to remove the default wso2carbon certificate from the client-truststore.jks.
keytool -delete -alias wso2carbon -keystore client-truststore.jks
v) Import new certificate.
keytool -import -alias wso2carbon -file wso2carbon.pem -keystore client-truststore.jks -storepass wso2carbon
5.0 Configure the gateway nodes.

i) Open the carbon.xml file (repository\conf) and define the HostName and MgtHostName.
<HostName>gw1.wso2am</HostName>
<MgtHostName>mgt1.gw.wso2am</MgtHostName> 
ii) Open the catalina-server.xml file (repository\conf\tomcat) and add the following two properties in http connector configuration.
proxyPort="80"
proxyName="gw.wso2am"

iii) Open the catalina-server.xml file (repository\conf\tomcat) add add the following two properties in https connector configuration.
proxyPort="443"
proxyName="gw.wso2am"

iv) Open the master-datasources.xml and configure new data source for mount registry.
<datasource>       
    <name>WSO2_CARBON_DB_cluster_db</name>
    <description>The datasource used for mount registry </description>
    <jndiConfig>
        <name>jdbc/WSO2CarbonDB_cluster_db</name>
    </jndiConfig>
    <definition type="RDBMS">
    <configuration>
        <url>jdbc:mysql://localhost:3306/cluster_db</url>
        <username>root</username>
        <password></password>
        <driverClassName>com.mysql.jdbc.Driver</driverClassName>
        <maxActive>50</maxActive>
        <maxWait>60000</maxWait>
        <testOnBorrow>true</testOnBorrow>
        <validationQuery>SELECT 1</validationQuery>
        <validationInterval>30000</validationInterval>
    </configuration>
</definition>
</datasource>
v) Open the master-datasources.xml and update the WSO2AM_DB  data source.
<datasource>
            <name>WSO2AM_DB</name>
            <description>The datasource used for API Manager database</description>
            <jndiConfig>
                <name>jdbc/WSO2AM_DB</name>
            </jndiConfig>
            <definition type="RDBMS">
                <configuration>
                    <url>jdbc:mysql://localhost:3306/cluster_db</url>
                    <username>root</username>
                    <password></password>
                    <defaultAutoCommit>false</defaultAutoCommit>
                    <driverClassName>com.mysql.jdbc.Driver</driverClassName>
                    <maxActive>50</maxActive>
                    <maxWait>60000</maxWait>
                    <testOnBorrow>true</testOnBorrow>
                    <validationQuery>SELECT 1</validationQuery>
                    <validationInterval>30000</validationInterval>
                </configuration>
            </definition>
        </datasource>
vi) Open the registry.xml and add the following configuration for registry mounting.
<dbConfig name="wso2registry_mount">
        <dataSource>jdbc/WSO2CarbonDB_cluster_db</dataSource>
    </dbConfig>
    <remoteInstance url="https://localhost:9443/registry">
        <id>instanceid</id>
        <dbConfig>wso2registry_mount</dbConfig>
        <readOnly>false</readOnly>
        <enableCache>true</enableCache>
        <registryRoot>/</registryRoot>
        <cacheId>root@jdbc:mysql://localhost:3306/cluster_db</cacheId>
    </remoteInstance>

    <mount path="/_system/config" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/nodes</targetPath>
    </mount>
       <mount path="/_system/governance" overwrite="true">
        <instanceId>instanceid</instanceId>
        <targetPath>/_system/governance</targetPath>
    </mount> 
vii) Open the user-mgt.xml and  update the dataSource property as bellow.
<Property name="dataSource">jdbc/WSO2CarbonDB_cluster_db</Property>
6.0 Add the virtual host configuration for proxy-2.

i) In windows,  this file (mgt.wso2am.conf)should be save in C:\xampp\apache\conf\extra directory (in Linux /etc/apache2/site-available)
<VirtualHost mgt.wso2am:443>


 ServerAdmin abc@wso2.com
 ServerName  mgt.wso2am
 ServerAlias mgt.wso2am

 ProxyRequests Off

 SSLEngine On
 SSLProxyEngine On

 #SSLProxyVerify none
 SSLProxyCheckPeerName off
 SSLProxyCheckPeerCN off
 #SSLProxyCheckPeerExpire off


 SSLCertificateFile /home/ajith/ca.crt
 SSLCertificateKeyFile /home/ajith/ca.key

<Proxy balancer://mgt.wso2am>
        BalancerMember https://mgt1.wso2am:9445 route=amNode1 loadfactor=1
        BalancerMember https://mgt2.wso2am:9446 route=amNode2 loadfactor=1
        ProxySet lbmethod=byrequests
        ProxySet stickysession=JSESSIONID|jsessionid
</Proxy>

ProxyPass /  balancer://mgt.wso2am/
ProxyPassReverse / balancer://mgt.wso2am/

</VirtualHost>

ii) In windows, include this virtual host configuration in httpd.conf file. (Find the # Virtual hosts configuration section and include )
Include conf/extra/mgt.wso2am.conf
iii) In linux you can use the following command.
sudo a2ensite mgt.wso2am.conf
iv) Create new key and crt for the proxy-2 like in steps 3.0.
  • Make sure to enter the mgt.wso2am as the Common Name (CN). 
  • The ca.crt and ca.key has configured as SSLCertificateFile  and SSLCertificateKeyFile in above virtual host configuration.
v) Generate new key store for manager nodes like in steps 4.0.
  • Make sure to enter the mgt.am.wso2.com as the first and last name , that value will be set as common name (CN).
  •  Enter the key store password as wso2carbon , then you don't need to change the default key store configurations.
 7.0 Configure the manager nodes.

i) Open the carbon.xml file (repository\conf) and define the HostName and MgtHostName in manager-1 and manager-2.
<HostName>am1.wso2am</HostName>
<MgtHostName>mgt1.wso2am</MgtHostName>
<HostName>am2.wso2am</HostName>
<MgtHostName>mgt2.wso2am</MgtHostName> 
ii) Open the catalina-server.xml file (repository\conf\tomcat) add add the following two properties in http connector configuration.
proxyPort="80"
proxyName="mgt.wso2am"
iii) Open the catalina-server.xml file (repository\conf\tomcat) add add the following two properties in https connector configuration.
proxyPort="443"
proxyName="mgt.wso2am"
 iv) Repeat the  same configuration steps mentioned  in 5. iv) , 5. v) , 5. vi) , 5. vii) for manager nodes

8.0 Add virtual host configuration for gateway management requests(publish API). (proxy-3).

<VirtualHost mgt.gw.wso2am:443>


 ServerAdmin abc@wso2.com
 ServerName  mgt.gw.wso2am
 ServerAlias mgt.gw.wso2am

 ProxyRequests Off

 SSLEngine On
 SSLProxyEngine On

 #SSLProxyVerify none
 SSLProxyCheckPeerName off
 SSLProxyCheckPeerCN off
 #SSLProxyCheckPeerExpire off


 SSLCertificateFile /home/ajith/ca.crt
 SSLCertificateKeyFile /home/ajith/ca.key

<Proxy balancer://mgt.gw.wso2am>
        BalancerMember https://mgt1.gw.wso2am:9443 route=gwNode1 loadfactor=1
        BalancerMember https://mgt2.gw.wso2am:9444 route=gwNode2 loadfactor=1
        ProxySet lbmethod=byrequests
        ProxySet stickysession=JSESSIONID|jsessionid
</Proxy>

ProxyPass /  balancer://mgt.gw.wso2am/
ProxyPassReverse / balancer://mgt.gw.wso2am/

</VirtualHost>
i) In windows,  include this virtual host configuration (mgt.gw.wso2am.conf) in httpd.conf file. (Find the # Virtual hosts configuration section and include )
Include conf/extra/mgt.gw.wso2am.conf
ii) In linux you can use the following command.
sudo a2ensite mgt.gw.wso2am.conf
9. Create database .
  • Here I'm going to create one database for shared registry, user manager and API manager database. In production setup that should configure as separate databases.
i) Login to the MySQL server and execute the following command. (I'm using MySQL 5.6.26)
CREATE DATABASE cluster_db CHARACTER SET latin1;  
ii) Download the  connector jar ("mysql-connector-java-5.1.29.jar") and copy to the repository\components\lib directory of all the gateway and manager nodes.

10. Configure api-manager.xml

i) Open the api-manager.xml file of the gateway nodes and configure the <AuthManager> configuration section. (ServerURL should point to proxy-2).
<AuthManager>
        <!--
            Server URL of the Authentication service
        -->
        <ServerURL>https://mgt.wso2am/services/</ServerURL>
        <!--
            Admin username for the Authentication manager.
        -->
        <Username>${admin.username}</Username>
        <!--
            Admin password for the Authentication manager.
        -->
        <Password>${admin.password}</Password>
    </AuthManager>
ii) Open the api-manager.xml file of the gateway node and configure the <APIKeyValidator> section. (the ServerURL should point to proxy2).
        <ServerURL>https://mgt.wso2am/services/</ServerURL>

        <!--
            Admin username for API key manager.
        -->
        <Username>${admin.username}</Username>

        <!--
            Admin password for API key manager.
        -->
        <Password>${admin.password}</Password>
iii) Open the api-manager.xml file of the manager nodes and configure the <APIGateway> configuration.(ServerURL should point to proxy-3)
<APIGateway>
    <!-- The environments to which an API will be published -->
    <Environments>
        <!-- Environments can be of different types. Allowed values are 'hybrid', 'production' and 'sandbox'.
             An API deployed on a 'production' type gateway will only support production keys
             An API deployed on a 'sandbox' type gateway will only support sandbox keys
             An API deployed on a 'hybrid' type gateway will support both production and sandbox keys -->
                <Environment type="hybrid" api-console="true">
                        <Name>Production and Sandbox</Name>
                    <Description> Description of environment</Description>
            <!--
                        Server URL of the API gateway.
                -->
                        <ServerURL>https://mgt.gw.wso2am/services/</ServerURL>
            <!--
                        Admin username for the API gateway.
                -->
                        <Username>${admin.username}</Username>
            <!--
                        Admin password for the API gateway.
                -->
                        <Password>${admin.password}</Password>
            <!--
                        Endpoint URLs for the APIs hosted in this API gateway.
                -->
                        <GatewayEndpoint>http://gw.wso2am,https://gw.wso2am</GatewayEndpoint>
                </Environment>
        </Environments>
</APIGateway>

11.  Insall the proxy certificates to the client-truststore.jks file in gateway and manager nodes.

12. Configure gateway cluster.

i) Open the axis2.xml (repository\conf\axis2) file and enable clsutering.
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
ii) Change membershipScheme as wka.
<parameter name="membershipScheme">wka</parameter>
iii) Add a name for cluster domain. (eg: wso2.gw.domain)
<parameter name="domain">wso2.gw.domain</parameter>
iv) Add localMemberHost name as server IP (eg for gateway-1: *.*.*.1 and for gateway-2: *.*.*.2 ).
<parameter name="localMemberHost">*.*.*.1</parameter>
v) Add localMemberPort.   (eg: use 4100 for gateway-1 and 4200 for gateway-2).
<parameter name="localMemberPort">4100</parameter>
vi ) Add members.
for gateway-1:
 <members>
          <member>
             <hostName>*.*.*.2</hostName>
             <port>4200</port>
          </member>
  </members>
for gateway-2:
 <members>
        <member>
             <hostName>*.*.*.1</hostName>
             <port>4100</port>
        </member>
  </members>

13. Start servers.

i) Go to the bin directory and execute the startup script (wso2server.sh for Linux and wso2server.bat for windows) as bellow. (If you have alreay started the server ,then delete the repository\database directory before start the serevr - This is to create the mount configuration in local registry).
Eg: for Linux
Gateway-1/bin >  sh wso2server.sh -Dsetup -DjvmRoute=gwNode1 -Dprofile=gateway-worker
Gateway-2/bin >  sh wso2server.sh -Dsetup -DjvmRoute=gwNode2 -Dprofile=gateway-worker
Manager-1/bin >  sh wso2server.sh -Dsetup -DjvmRoute=amNode1
Manager-2/bin >  sh wso2server.sh -Dsetup -DjvmRoute=amNode1
  • The "jvmRoute" Identifier which must be used in load balancing scenarios to enable session affinity.
  • The value for the "jvmRoute" is configured in virtual host configuration.
  • The -Dsetup parameter is to populate the tables for registry, user manager and api manager in cluster_db.
  • The -Dprofile=gateway-worker is to start the gateway nodes from gateway worker profile.
14. Add the hostnames to the etc host file. (in Windows C:\Windows\System32\drivers\etc\hosts)

15. Login to the publisher (https://mgt.wso2am/publisher) and deploy the default sample.

16. Login to the store (https://mgt.wso2am/store) , then subscribe to API and generate token.

17. You can use the SoapUI REST project for https://gw.wso2am//weatherapi/1.0.0?q=colombo&mode=xml

i) Add Authorization header and invoke the API.