Database Space usage monitoring for storage server

For our new product “Storage Server” we need monitoring functionalities. It was possible to change my main project to cater that need. In my main project I collect usage data for all the tenants and publish data of the usage exceeded tenants to the BAM server. I just needed to publish all the data to the BAM so it can work as a monitoring feature for Storage Server.
I removed some parts from my main project to suit the current need. I didn't need tenantBillingService”, which hold me back in the main project. And do not need to change “stratos common” (which is used to get the package details which was needed in the calculations) anymore, as we are publishing all the details that we are collecting. Component level architecture of the project will be

It is almost 'THE END'

Now this is the summery of What I have done

 It is agreed to measure the database space usage by each tenant. Here we will not limit the tenant(in terms of database access) on its DB usage but will keep track on excess DB space use by each tenant.

Component level view of the process.

Changes to each component:

Rss-manager: This component will be used to gather usage data from the RSS. And this will add those data to a queue which in turn will be retrieved by usage agent component. This Usage data collection will be handle through couple of newly added classes. And this is scheduled to be run daily. And it is configurable to run starting from a given time and repeated with given time gap(currently decided to run it in 24h intervals). Here we will only interested in tenants with exceeded usage. So it is needed to know the usage plan of a interested tenant, in order to get its limits. We thought of only publishing information about those tenants who exceeds the space limits, due to two reasons.

1. To reduce the data transfer between components and to the BAM server.
2. Exceeded DB size is all we need for billing calculations.

Usage-agent: This component will retrieve usage data from the queue(above mentioned) in the rss-manager. This is handled by newly added class, DatabaseUsageDataRetrievalTask. This is also scheduled to be run daily. And it is configurable to run starting from a given time and repeated with given time gap(currently decided to run it in 24h intervals).

Stratos-commons: This is where usage plan details are manipulated. Here plan details are read from 'multitenancy-packages.xml' and made available for use through a service. Here I have changed the xml file, xml reading class, data storing bean, to contain DB usage related data.

Dependencies: this depends on the yet to develop component (to get the tenant usage plan given the tenant domain/id) and that component is required for the RSS-Manager component changed to work perfectly.

The Inevitable Change

Change is there in everything you see, that is why 'change' is known as the only 'not changing thing'. When I moved in to my old(main) project, I felt that I have to start from the scratch again. Code-lines I wrote didn't work and almost all the lines had errors in them. Those errors due to missing classes, missing methods, changed signatures, etc. It was a hard job bring it back to the earlier state. By now it is collecting and publishing usage data as intended. I worked over a week on this, but added nothing extra, only took it back to where it was.

Project Progress

What I have done

Completed collecting database usage data.

Completed publishing them to BAM

What more to do

By now I don't have a way to get the usage plan of each tenant, so for the sake of testing I have hard corded it.

Need to analyze and summarize usage data that was sent to BAM, this is done using hive scripts.

Need to cleanup and reformat the code according to the best practices where I have missed them.

WSO2 Storage Server

This is a new product by WSO2. This contain some of the data related functionalists the was in WSO2 DSS (Data Service Server). These removed functionalists do not go with the word "data services" but they were useful for data services. So I think It is a good Idea to have those as a separate product.

This new product (WSO2 Storage Server) will include(most probably) following as basics,

RSS (Relational Storage Service/Server)

CSS (Column Storage Service/Server)

HDFSSS (HDFS Storage Service/Server)

This is not there to be downloaded yet. But will be soon. I only know few details about the storage server. When I get to know more, I'll add those details here as well.