It is almost 'THE END'

Now this is the summery of What I have done

 It is agreed to measure the database space usage by each tenant. Here we will not limit the tenant(in terms of database access) on its DB usage but will keep track on excess DB space use by each tenant.

Component level view of the process.

Changes to each component:

Rss-manager: This component will be used to gather usage data from the RSS. And this will add those data to a queue which in turn will be retrieved by usage agent component. This Usage data collection will be handle through couple of newly added classes. And this is scheduled to be run daily. And it is configurable to run starting from a given time and repeated with given time gap(currently decided to run it in 24h intervals). Here we will only interested in tenants with exceeded usage. So it is needed to know the usage plan of a interested tenant, in order to get its limits. We thought of only publishing information about those tenants who exceeds the space limits, due to two reasons.

1. To reduce the data transfer between components and to the BAM server.
2. Exceeded DB size is all we need for billing calculations.

Usage-agent: This component will retrieve usage data from the queue(above mentioned) in the rss-manager. This is handled by newly added class, DatabaseUsageDataRetrievalTask. This is also scheduled to be run daily. And it is configurable to run starting from a given time and repeated with given time gap(currently decided to run it in 24h intervals).

Stratos-commons: This is where usage plan details are manipulated. Here plan details are read from 'multitenancy-packages.xml' and made available for use through a service. Here I have changed the xml file, xml reading class, data storing bean, to contain DB usage related data.

Dependencies: this depends on the yet to develop component (to get the tenant usage plan given the tenant domain/id) and that component is required for the RSS-Manager component changed to work perfectly.

Replacing for ntask(quartz-scheduler), using timerTask or scheduler

I tried using ntask(a scheduler that uses quartz-scheduler inside) but for some reason it is not working, I found no one to ask about that, ones that have used it don't have a clear idea of it. Therefor they can't figure out what went wrong. Unluckily non of my mentors are familiar with it. So I decided to find a alternative. So I thought of java in built TimerTask

According to quartz-scheduler web site they mention several things on “Why not just use java.util.Timer ?”

“There are many reasons! Here are a few:

1. Timers have no persistence mechanism.
2. Timers have inflexible scheduling (only able to set start-time & repeat interval, nothing based on dates, time of day, etc.)
3. Timers don't utilize a thread-pool (one thread per timer)
4. Timers have no real management schemes - you'd have to write your own mechanism for being able to remember, organize and retrieve your tasks by name, etc.
   

...of course to some simple applications these features may not be important, in which case it may then be the right decision not to use Quartz.”



Seems like I can live with TimerTask. I don't need any “persistence mechanism” as I am starting everything all over again with server startup. And with a server restart all the threads will be gone and I just need one thread, and don't need a thread-pool. And this is so simple and I don't need to manage my tasks after I create them. And On flexibility, As I know we can give a stat date(Util.date), and that is what I am using.

Timer timer = new Timer(true);
Calendar firstRun = Calendar.getInstance();

firstRun.set(Calendar.HOUR_OF_DAY,HOUR_OF_THE_DAY_TO_RUN_DB_USAGE_RETRIEVAL);
firstRun.set(Calendar.MINUTE, 0);
firstRun.set(Calendar.SECOND, 0);
timer.schedule(checker,firstRun.getTime(), 20000);

So I created the wanted date, by getting a date and setting the time part and we are good to go.

Difference between two time ignoring the date - Using Java

When you want to have the amount of time to 12.00 mid night, It doesn't really matter what is the date. If you want to measure time between two instants it is simple take time in milliseconds in both occation and then subtract one from another. But here we want to take difference between two times where one is a future time on the same date. In my application I wanted to schedule a script at 12.00 midnight by giving a delay. I can simply do it using a cron-job, but in my case it didn't work. So I wanted to know the delay between two times.

Calendar future = Calendar.getInstance();
future.set(Calendar.HOUR_OF_DAY, HOUR_TO_RUN_DB_USAGE_RETRIEVAL);
future.set(Calendar.MINUTE, 0);
future.set(Calendar.SECOND, 0);
Calendar now = Calendar.getInstance();
long requiredDelay = future.getTimeInMillis() - now.getTimeInMillis();

Here I take today this time as future and only change the time part(HH:mm:ss) so the date remains the same. Then get another time calendar instance for now and by subtracting getTimeInMillis() parts of those two you get the desired result.  

Using Hive Scripts to Analyze and Summarize BAM data

As I have completed up to publishing usage data, now I need to analyze and summarize those data. This can be simply done by a hive script and scheduling it within BAM. In the main menu of BAM you will find a manage menu. In manage menu, there is a menu item analyze. Under analyze menu item you get two more sub menus, one to list existing scripts and one to add new scripts.

Now go to 'add' sub menu there(Main>Manage>Analytics>Add). Here you get the chance to write your script and schedule it.

Bellow is a simple script written by Shariq Muhammed, SE @ WSO2. I used this script to summarize data in one of my tables created while pumping data in to BAM. I have removed some parts init as It won't be relevant to you.

CREATE EXTERNAL TABLE IF NOT EXISTS UsageStatsTable (id STRING,
        payload_ServerName STRING,
        payload_TenantID STRING,
        payload_Data STRING,
        payload_Value BIGINT,
        timestamp BIGINT) 
        STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( 
        //sort Properties
);

CREATE EXTERNAL TABLE IF NOT EXISTS UsageStatsHourFact (id String, 
        hour_fact STRING,
        payload_ServerName STRING, 
        payload_TenantID STRING,        
        payload_Data STRING,
        payload_Value BIGINT) 
        STORED BY 'org.apache.hadoop.hive.cassandra.CassandraStorageHandler' WITH SERDEPROPERTIES ( 
        //sort properties
);

select #some columns from my table#

insert into table #some other table#

like above you can select and group the pumped data and insert summery data into a new table. If you don't know hive syntax, it is similar to SQL and you can have a great tutorial @ the following link, https://cwiki.apache.org/Hive/tutorial.html.

Strange things do happen

This a about a basic error that did. I got this error saying “class not found” in runtime. Let me explain the situation. I was calling a method from another osgi bundle staying in another osgi bundle. If I go in to more specific level I was working in usage agent component and I needed to call a method (a static method) that was in the rss-manager component.

In rss manager,

public static Queue <DBStatisticsEntry> getStats() {
return stats;
}

In Usage agent

dbStats = DBSizeChecker.getStats();

For a Java user it don't have any errors. But remember that this did give the error, “class not found” so before getting to know that method, it even don't know the class. And IDE don't give any errors, means that it is not a syntax error. And it don't give any warnings, no best practice violations.

My guesses of what has went wrong..

1. rss-manager jar inside the server do not have the DBSizeChecker class in it. As this class was added lately by me, I thought for a second what it the jar inside the server is not the current one that I am using? But opening the jar showed that class is there.
2. Problem with IDE, there might be a syntax issue but IDE won't show it due to some problem within it. And my server don't through any exceptions, only IDE gives me that exception while debugging.
3. Is this the right way to use static methods? I was confident that it was the way, but as I didn't have clue on what went wrong, I even did a search on “how to call a static method from another class in java”
4. rss manager might not have started, I checked it using osgi console and it was active and running.

And what was the real problem.

DBSizeChecker class was in the 'internal' folder in the OSGI bundle. If you don't know what that is, in OSGI we do not expose internal classes to the outside. Thats is why we didn't get a error from IDE but from runtime. In runtime OSGI comes in to action. This is basics in OSGI, but those matters.

This problem showed me how important is it to have someone to help when you get errors. This error is due to something simple but still I didn't get it, but when I showed it to a senior (kasun) he nailed the error in a minute.

The Inevitable Change

Change is there in everything you see, that is why 'change' is known as the only 'not changing thing'. When I moved in to my old(main) project, I felt that I have to start from the scratch again. Code-lines I wrote didn't work and almost all the lines had errors in them. Those errors due to missing classes, missing methods, changed signatures, etc. It was a hard job bring it back to the earlier state. By now it is collecting and publishing usage data as intended. I worked over a week on this, but added nothing extra, only took it back to where it was.

Project Progress

What I have done

Completed collecting database usage data.

Completed publishing them to BAM

What more to do

By now I don't have a way to get the usage plan of each tenant, so for the sake of testing I have hard corded it.

Need to analyze and summarize usage data that was sent to BAM, this is done using hive scripts.

Need to cleanup and reformat the code according to the best practices where I have missed them.

WSO2 Storage Server

This is a new product by WSO2. This contain some of the data related functionalists the was in WSO2 DSS (Data Service Server). These removed functionalists do not go with the word "data services" but they were useful for data services. So I think It is a good Idea to have those as a separate product.

This new product (WSO2 Storage Server) will include(most probably) following as basics,

RSS (Relational Storage Service/Server)

CSS (Column Storage Service/Server)

HDFSSS (HDFS Storage Service/Server)

This is not there to be downloaded yet. But will be soon. I only know few details about the storage server. When I get to know more, I'll add those details here as well.

Back to the Frozen project

With the end of the automation hackathon we are back on our original projects which were frozen over a month. I have forgotten almost everything related to my project. Luckily I have my own blog to read :D. Several thing to do before starting it all over again. First I need to move in to the trunk again (as this will not be there in the next release). I have to take a svn up in the trunk and build it (as we were working in the branch for the last month).

I got a svn up and started building it. Soon as I enter the component in platform I got a error and looks like it is not fixed yet and the fix seems to be long. At the same time I got the feeling that this specific component might not be wanted for the product I am working in(DSS). As I don't have a clear and complete idea about the components that are used in the DSS, hoping that is not used in the DSS and to avoid unwanted problems, I started the build with the option -P product-<product name> in this case it is mvn install -P product-dss. Soon I got to know that component is there in dss too :(. Anyway it is good to know that maven command to build only one product.

Building only one product

This is something new I learned when I was in the automation hackathon. There we worked in a product called greg and didn't want to build anything other than what is needed to continue our work with greg testing. Building the whole thing is too expensive. So for anyone who needs to build only the components, stubs, etc. that is need to a one product, can use the below method.

1. go to the platform directory
2. enter command for building(that you normally use) with following added to the end -P product-<product name>, replacing the <product name> with the product name that you are targeting.

Ending The Automation Hackathon

We ended the automation hackathon on 31-08-2012(last Friday). There was three main parts in our automation hackathon. In the first part we wrote tests for the QA passed test cases. And in the second part we did the patch automation. In the last part we ported the old tests to the new framework. I completed all three parts two days before the end of test automation.

Problem Occurred just as we put all the test together. Lot of test cases failed due to uncleaned artifacts deployed by the other tests. So we have to clean it to the best we can. We are asked not to fully clean the whole registry. It would be really easy to clean the whole thing up, but correct thing to do would be cleaning things that are deployed by ourselves. According to our team-lead server should come to the initial state(before running the tests) after running our tests.

Even at the end of the hackathon, some errors was there in migrated tests. We might have to work with the team-lead for few more days to get them corrected. I did the governance API testing. Which include lot of test classes. I got some help from two of my friends to complete the test migration, as there was around 45 test classes with over 300 test cases. Hardest part of porting those tests was, cleaning up the resources. In those old tests they use very low level commands to clean the whole registry. But as we are asked to clean only things that we deployed, it was not that easy.

With only new tests(without any ported tests) (1100+ test cases) we achieved over 45 line coverage and over 60 class coverage. After all testing is not about getting it 100% covered. No matter what is the %, our goal should be to deliver a product will least number of failures. If the product is failing, % will not help. I think we did a great job towards our goal.