Thursday, April 16, 2015

Audit log publsiher for WSO2 Identity Server + WSO2 Business Activity Monitor(BAM)

WSO2 Identity server create an audit logs for  user account activities (Add users/roles, Delete users/roles, assign users to roles, ..etc) . We can publish those logs to BAM and analyze .

This is a custom log appender which is written to publish audit logs to BAM.

How to run.
-------------------
1.  Download the source from here .

2. Open the build.xml file and change the value of product.home property and execute the ant command to build the jar.

3. Copy the org.wso2.carbon.auditlog.publisher-1.0.0.jar file to <IS_HOME>/repository/components/lib

4.  Open the log4j.properties file and add the following configuration. (Change BAM url, username,password according to BAM configurations.)
log4j.appender.AUDIT_LOGFILE1=org.wso2.carbon.logging.appender.AuditLogEventAppender
log4j.appender.AUDIT_LOGFILE1.File=${carbon.home}/repository/logs/audit.log
log4j.appender.AUDIT_LOGFILE1.Append=true
log4j.appender.AUDIT_LOGFILE1.layout=org.wso2.carbon.utils.logging.TenantAwarePatternLayout
log4j.appender.AUDIT_LOGFILE1.layout.ConversionPattern=[%d] %P%5p - %x %m %n
log4j.appender.AUDIT_LOGFILE1.layout.TenantPattern=%U%@%D [%T] [%S]
log4j.appender.AUDIT_LOGFILE1.threshold=INFO
log4j.appender.AUDIT_LOGFILE1.url=tcp://localhost:7611
log4j.appender.AUDIT_LOGFILE1.columnList=%T,%S,%A,%d,%H,%c,%p,%m,%I,%Stacktrace
log4j.appender.AUDIT_LOGFILE1.userName=admin
log4j.appender.AUDIT_LOGFILE1.password=admin
log4j.appender.AUDIT_LOGFILE1.processingLimit=1000
log4j.appender.AUDIT_LOGFILE1.maxTolerableConsecutiveFailure=20
log4j.appender.AUDIT_LOGFILE1.trustStorePassword=wso2carbon
log4j.appender.AUDIT_LOGFILE1.truststorePath=/repository/resources/security/wso2carbon.jks 
5. Add the AUDIT_LOGFILE1 name.
log4j.logger.AUDIT_LOG=INFO, AUDIT_LOGFILE, AUDIT_LOGFILE1
6. Start the BAM server first.

7. Start the IS server.

8. Create  users/roles.

9. Logged in to the BAM and browse the Cassandra using "Explore Cluster" feature. Now you should see the "audit_log_IS" under the "EVENT_KS"


10. Browse the rows to view the attributes of the published events.


11. Run the following hive query to summarize the audit logs.

CREATE EXTERNAL TABLE IF NOT EXISTS ACCOUNT_ACTIVITY
(key STRING, initiator STRING, action STRING, target STRING, result STRING, uuid STRING, logTime BIGINT) STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES (
"wso2.carbon.datasource.name" = "WSO2BAM_CASSANDRA_DATASOURCE",
"cassandra.cf.name" = "audit_log_IS" ,
"cassandra.columns.mapping" =
":key,payload_initiator, payload_action, payload_target, payload_result,payload_uuid, payload_logTime" );


CREATE EXTERNAL TABLE IF NOT EXISTS ACCOUNT_ACTIVITY_SUMMARY1(initiator STRING, action STRING, target STRING, result STRING,uuid STRING, logTime STRING) STORED BY
'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES (
'wso2.carbon.datasource.name'='WSO2BAM_DATASOURCE',
'hive.jdbc.update.on.duplicate' = 'false' ,
'hive.jdbc.primary.key.fields' = 'uuid' ,
'hive.jdbc.table.create.query' = 'CREATE TABLE ACCOUNT_ACTIVITY_SUMMARY1_TBL (initiator VARCHAR(100), action VARCHAR(100), target VARCHAR(100),result VARCHAR(100),uuid VARCHAR(100), logTime VARCHAR(100))' );

insert overwrite table ACCOUNT_ACTIVITY_SUMMARY1 select initiator,action,target,result,uuid, from_unixtime(cast(logTime/1000 as BIGINT), 'yyyy-MM-dd HH:mm:ss') as logTime  from ACCOUNT_ACTIVITY;

 CREATE EXTERNAL TABLE IF NOT EXISTS ACCOUNT_ACTIVITY_SUMMARY2(initiator STRING, action STRING, result STRING,totalcount INT) STORED BY
 'org.wso2.carbon.hadoop.hive.jdbc.storage.JDBCStorageHandler' TBLPROPERTIES ( 
 'wso2.carbon.datasource.name'='WSO2BAM_DATASOURCE',
 'hive.jdbc.update.on.duplicate' = 'false' , 
 'hive.jdbc.primary.key.fields' = 'uuid' , 
 'hive.jdbc.table.create.query' = 'CREATE TABLE ACCOUNT_ACTIVITY_SUMMARY2_TBL (initiator VARCHAR(100), action VARCHAR(100),result VARCHAR(100),totalcount INT)' );
 
insert overwrite table ACCOUNT_ACTIVITY_SUMMARY2 select initiator,action,result, count(DISTINCT key) from ACCOUNT_ACTIVITY group by initiator,action,result;


The summarized tables (ACCOUNT_ACTIVITY_SUMMARY1_TBL and ACCOUNT_ACTIVITY_SUMMARY2_TBL) will be created in BAM_STATS_DB  which is configured in bam-datasources.xml under the <BAM_HOME>/repository/conf/datasources.

12. Generate a gadget based on the summarized data.

JDBC URL*: jdbc:h2:repository/database/samples/BAM_STATS_DB;AUTO_SERVER=TRUE
Driver Class Name : org.h2.Driver
User Name* : wso2carbon
Password*  : wso2carbon

 




 

13. Generate gadget , copy the url.



 14. Go to the dashboard and add new gadget.