It is almost 'THE END'

Now this is the summery of What I have done

 It is agreed to measure the database space usage by each tenant. Here we will not limit the tenant(in terms of database access) on its DB usage but will keep track on excess DB space use by each tenant.

Component level view of the process.

Changes to each component:

Rss-manager: This component will be used to gather usage data from the RSS. And this will add those data to a queue which in turn will be retrieved by usage agent component. This Usage data collection will be handle through couple of newly added classes. And this is scheduled to be run daily. And it is configurable to run starting from a given time and repeated with given time gap(currently decided to run it in 24h intervals). Here we will only interested in tenants with exceeded usage. So it is needed to know the usage plan of a interested tenant, in order to get its limits. We thought of only publishing information about those tenants who exceeds the space limits, due to two reasons.

1. To reduce the data transfer between components and to the BAM server.
2. Exceeded DB size is all we need for billing calculations.

Usage-agent: This component will retrieve usage data from the queue(above mentioned) in the rss-manager. This is handled by newly added class, DatabaseUsageDataRetrievalTask. This is also scheduled to be run daily. And it is configurable to run starting from a given time and repeated with given time gap(currently decided to run it in 24h intervals).

Stratos-commons: This is where usage plan details are manipulated. Here plan details are read from 'multitenancy-packages.xml' and made available for use through a service. Here I have changed the xml file, xml reading class, data storing bean, to contain DB usage related data.

Dependencies: this depends on the yet to develop component (to get the tenant usage plan given the tenant domain/id) and that component is required for the RSS-Manager component changed to work perfectly.

Back to the Frozen project

With the end of the automation hackathon we are back on our original projects which were frozen over a month. I have forgotten almost everything related to my project. Luckily I have my own blog to read :D. Several thing to do before starting it all over again. First I need to move in to the trunk again (as this will not be there in the next release). I have to take a svn up in the trunk and build it (as we were working in the branch for the last month).

I got a svn up and started building it. Soon as I enter the component in platform I got a error and looks like it is not fixed yet and the fix seems to be long. At the same time I got the feeling that this specific component might not be wanted for the product I am working in(DSS). As I don't have a clear and complete idea about the components that are used in the DSS, hoping that is not used in the DSS and to avoid unwanted problems, I started the build with the option -P product-<product name> in this case it is mvn install -P product-dss. Soon I got to know that component is there in dss too :(. Anyway it is good to know that maven command to build only one product.

Building only one product

This is something new I learned when I was in the automation hackathon. There we worked in a product called greg and didn't want to build anything other than what is needed to continue our work with greg testing. Building the whole thing is too expensive. So for anyone who needs to build only the components, stubs, etc. that is need to a one product, can use the below method.

1. go to the platform directory
2. enter command for building(that you normally use) with following added to the end -P product-<product name>, replacing the <product name> with the product name that you are targeting.

Using OSGi console to debug things

This OSGi console was very helpful to me in finding out problems relevant to OSGi bundles. You can have lot of information about this in the Internet. In here what I am going to write is about how I used it in the debugging.

First start the server with following option

-DosgiConsole. This will start the server with OSGi console. This is very helpful in situations like, checking whether all bundles that you put in to droppings have activated. When the server starts. Put ss in console. This will give a list of all bundles.

Use it with 'like' modifier to get the relevant bundles only. Ex ss like student

Now you can see state of all bundles. With the installation of a bundle in the OSGi runtime this bundle is persisted in a local bundle cache. The OSGi runtime then tries to resolve all dependencies of the bundle.If all required dependencies are resolved the bundle is in the status RESOLVED otherwise it is in the status INSTALLED.

If several bundles exist which would satisfy the dependency, then the bundle with the highest version is used. If the versions are the same, then the bundle with the lowest ID will be used. If the bundle is started, its status is STARTING. Afterwards it gets the ACTIVE status. - (http://www.vogella.com/articles/OSGi/article.html#osgiarch_bundles good reference)

You have to have all your bundles acticvated. If not check what is wrong with them using 'bundle <bundle number>' command. If there is no error, try starting it manually using 'start <bundle number>'

Publishing to BAM

I created this just to publish data I collect to BAM, so this do not adhere to good programming practices.

package org.wso2.carbon.usage.agent.util;

import java.net.MalformedURLException;
import java.net.SocketException;
import java.net.UnknownHostException;

import javax.security.sasl.AuthenticationException;

import org.wso2.carbon.eventbridge.agent.thrift.Agent;
import org.wso2.carbon.eventbridge.agent.thrift.DataPublisher;
import org.wso2.carbon.eventbridge.agent.thrift.conf.AgentConfiguration;
import org.wso2.carbon.eventbridge.agent.thrift.exception.AgentException;
import org.wso2.carbon.eventbridge.commons.Event;
import org.wso2.carbon.eventbridge.commons.exception.DifferentStreamDefinitionAlreadyDefinedException;
import org.wso2.carbon.eventbridge.commons.exception.MalformedStreamDefinitionException;
import org.wso2.carbon.eventbridge.commons.exception.NoStreamDefinitionExistException;
import org.wso2.carbon.eventbridge.commons.exception.StreamDefinitionException;
import org.wso2.carbon.eventbridge.commons.exception.TransportException;

public class PublishUtil2 {
    public static final String STREAM_NAME1 = "org.wso2.db6.kpiii";
    public static final String VERSION1 = "1.0.6";
    private static String streamId1;
    private static DataPublisher dataPublisher = null;
    
    

    public static void publish(long exceededBytes, long databasesize, String tenentdomain) throws AgentException, MalformedStreamDefinitionException,
    StreamDefinitionException, DifferentStreamDefinitionAlreadyDefinedException, MalformedURLException,
    AuthenticationException, NoStreamDefinitionExistException,
    org.wso2.carbon.eventbridge.commons.exception.AuthenticationException, TransportException, SocketException, UnknownHostException{
        
     System.out.println("Starting BAM KPI Agent");
        AgentConfiguration agentConfiguration = new AgentConfiguration();
        String currentDir = System.getProperty("user.dir");
        System.setProperty("javax.net.ssl.trustStore", currentDir + "/repository/resources/security/client-truststore.jks");
        System.setProperty("javax.net.ssl.trustStorePassword", "wso2carbon");
        Agent agent = new Agent(agentConfiguration);
        
     dataPublisher = null;
        try {
        dataPublisher = new DataPublisher("tcp://10.100.3.80:7613", "admin", "admin", agent);
        } catch (Throwable e){
         e.printStackTrace();
        }

        streamId1 = null;


        try {
            streamId1 = dataPublisher.findEventStream(STREAM_NAME1, VERSION1);
            System.out.println("Stream already defined");

        } catch (NoStreamDefinitionExistException e) {
            streamId1 = dataPublisher.defineEventStream("{" +
                    "  'name':'" + STREAM_NAME1 + "'," +
                    "  'version':'" + VERSION1 + "'," +
                    "  'nickName': 'DSSUsage'," +
                    "  'description': 'Exceeded DB Use'," +
                    "  'metaData':[" +
                    "          {'name':'clientType','type':'STRING'}" +
                    "  ]," +
                    "  'payloadData':[" +
                    "          {'name':'exceededBytes','type':'LONG'}," +
                    "          {'name':'databasesize','type':'LONG'}," +
                    "          {'name':'tenentdomain','type':'STRING'}" +
                    "  ]" +
                    "}");
        }
        //Publish event for a valid stream
        if (!streamId1.isEmpty()) {
            System.out.println("Stream ID: " + streamId1);
            publishEvents(tenentdomain, exceededBytes, databasesize);

//            for (int i = 0; i < 1; i++) {
//                publishEvents("malinga");
//                System.out.println("Events published : " + (i + 1) * 2);
//            }
//            try {
//                Thread.sleep(3000);
//            } catch (InterruptedException e) {
//            }

            dataPublisher.stop();
        }
    }
    
    public static void publishEvents(String name, long exceededBytes, long databasesize) throws AgentException {
     System.out.println(name);
     publishEvents(dataPublisher, streamId1, name, exceededBytes, databasesize);

    }

    
    public static void publishEvents(DataPublisher dataPublisher, String streamId, String name, long exceededBytes, long databasesize) throws AgentException {
        Event eventOne = new Event(streamId, System.currentTimeMillis(), new Object[]{"external"}, null,
                new Object[]{exceededBytes, databasesize, 3600.0, name});
        dataPublisher.publish(eventOne);

    }

}

Problems I had to face.

I tried to change the streamId1, but it was not possible. It gave a error in BAM side. Then I got to know that schema is saved under the STREAM_NAME1, and if I want to change it, I have to do that with a change in STREAM_NAME1 too.

There was no way to check the the published data as it would show some rubbish, in the data viewer in BAM. I got a nice client for one of my mentors that can get the data from Cassandra cluster. This is written by 'Shariq Muhammed', he is a software engineer at WSO2. This is also written just to read the data and he haven't thought much about adhering to good programming practices. You can have it from below link

https://svn.wso2.org/repos/wso2/people/shariq/BAMStatClient/

First I couldn't sent long, It took me a while to figure that out. It was because the number I send was taken as a int. I had to add the 'L' to end of it to get it working.

OSGi Services

Full system(carbon core + products) is built as a combination of OSGi bundles. Some of theses bundles expose services at OSGi level. I have to use them to get the tenant usage plan.

There is a service in manager which gives the usage plan of the tenant. To use it first I have to catch them, unluckily those manager component that is needed for the service it not there in the DSS, so we have to add them to DSS temporary as a feature. core and mgt are placed in DSS as a feature. First we thought is putting those to into droppings. But it didn't work as mgt package tried to start before core which is not possible as it requires core. Because of that we have to make it as a feature.

After putting them we have catch that service. Those are normally catches at serviceComponents(one class with 'activate' method). Below is how I did it. There are some methods that is used to bind and unbind the service objects we have to have them where we catch the service

I added the osgi comments as below  

/** 
 * @scr.component name="org.wso2.carbon.rssmanager" immediate="true" 
 * @scr.reference name="default.tenant.billing.service" 
 *                interface="org.wso2.carbon.stratos.common.TenantBillingService" 
 *                cardinality="1..1" policy="dynamic" 
 *                bind="setTenantBillingService" 
 *                unbind="unsetTenantBillingService" 
 * @scr.reference name="user.realmservice.default" 
 *                  interface="org.wso2.carbon.user.core.service.RealmService" 
 *                  cardinality="1..1" policy="dynamic" 
 *                  bind="setRealmService" 
 *                  unbind="unsetRealmService" 
 */ 

then I added following methods

/** 
     * osgi bind method for RealmService 
     * @param tenantBillingService 
     */ 
    protected void setRealmService(RealmService realmService) { 
        this.realmService = realmService; 
    } 

    /** 
     * osgi unbind method for RealmService 
     * @param tenantBillingService 
     */ 
    protected void unsetRealmService(RealmService realmService) { 
        this.realmService =null; 
    } 
   
    /** 
     * returns a RealmService objects that is used to get the tenant list. 
     * @param tenantBillingService 
     */ 
    private RealmService getRealmService() { 
        return realmService; 
    }

After catching it I can use it as below,

if(realmService!=null){ 
        TenantManager tenantManager = getRealmService().getTenantManager(); 
        Tenant[] tenants; 
        try { 
            tenants = (Tenant[]) tenantManager.getAllTenants(); 
            for (int i = 0; i < tenants.length; i++) { 
               // check for each tenant 
            } 
        } catch (UserStoreException e1) { 
            // TODO Auto-generated catch block 
            e1.printStackTrace(); 
        } 
        }else{ 
            System.out.println("realmService NULL"); 
        }

Problems came across Service was not available in DSS First I tried to catch this in my own java class (not in RSSManagerServiceComponent) which was OK but was not the standard. Standard is to catch it in RSSManagerServiceComponent. Anyway it should work, on matter how we do it. In my case it didn't work for tenant.billing.service . Problem was same service (tenant.billing.service not realmservice) was caught in RSSManagerServiceComponent before coming in to my class. This is where I figured out the importance of that standard. Had to include lot of other jar files in droppings.