Showing posts with label database. Show all posts
Showing posts with label database. Show all posts
Friday, November 30, 2012
Database Space usage monitoring for storage server
For our new product “Storage Server” we need monitoring functionalities. It was possible to change my main project to cater that need. In my main project I collect usage data for all the tenants and publish data of the usage exceeded tenants to the BAM server. I just needed to publish all the data to the BAM so it can work as a monitoring feature for Storage Server.
I removed some parts from my main project to suit the current need. I didn't need tenantBillingService”, which hold me back in the main project. And do not need to change “stratos common” (which is used to get the package details which was needed in the calculations) anymore, as we are publishing all the details that we are collecting. Component level architecture of the project will be
Labels:
database,
monitoring,
Space,
usage,
WSO2 Storage Server
Measuring/Billing database usage in StratosLive - Summery
This Article collects all the posts under the Measuring/Billing database usage in StratosLive.
My Job
WSO2 Data Services Server User Guide
Need to find I/O rates, bandwidth used by each Database user
Limiting The Resource Use
I continued
Suggestions and replies
Collecting and summarizing the captured data
Followed the BAM samples.
Do you need data to play with?
Prototype version 1
Prototype version 1 has to be verified.
1st Verification
OSGi Services
Publishing to BAM
Using OSGi console to debug things
[Break for Test Automation]
Back to the Frozen project
WSO2 Storage Server
The Inevitable Change
Strange things do happen
Using Hive Scripts to Analyze and Summarize BAM data
Difference between two time ignoring the date
Replacing for ntask(quartz-scheduler), using timer task
It is almost 'THE END'
My Job
WSO2 Data Services Server User Guide
Need to find I/O rates, bandwidth used by each Database user
Limiting The Resource Use
I continued
Suggestions and replies
Collecting and summarizing the captured data
Followed the BAM samples.
Do you need data to play with?
Prototype version 1
Prototype version 1 has to be verified.
1st Verification
OSGi Services
Publishing to BAM
Using OSGi console to debug things
[Break for Test Automation]
Back to the Frozen project
WSO2 Storage Server
The Inevitable Change
Strange things do happen
Using Hive Scripts to Analyze and Summarize BAM data
Difference between two time ignoring the date
Replacing for ntask(quartz-scheduler), using timer task
It is almost 'THE END'
Labels:
bam,
BAM2,
billing,
database,
database size,
mysql,
remote debugging,
StratosLive RSS,
usage agent,
WSO2 Data Services Server
Saturday, June 30, 2012
1st Verification – Verification of “index length + data length = actual disk space used”
I think I found where DBs are stored.
As I am working in XAMPP there were stored in /opt/lampp/var/mysql
there where folders for each database. When tried to open them it
gives me a error saying “you don't have permission” Next thing I
tried was trying to open it in the terminal using sudo. Unlucky me it
gave me a error saying “sudo: cd:
command not found”. Have to find a way to open such folders.
I'll write a post on this, If I find a way (Or any one who know a way
can help me by commenting below). Till then I used “sudo
nautilus path/”
There were 3 files for each table in
each db. A .frm file, .myd file, .myi file. All three are
contributing to the space used
- FRM files contain table structure information.
- MYD contains the actual table data.
- MYI files contain the indexes.
-s, --summarize (display only a total for each argument)
-h, --human-readable ( print sizes in human readable format (e.g., 1K 234M 2G))
so It showed that figures given in the information schema is similar to the results given by the above command. But size of the FRM file was not take in to account by me. But looks like all the .frm files are having the same length (12K) so I can sum them up If I know the number of tables. However I have to check why others ignore this file when they are calculating.
Some databases only have .frm file
Going through the database folders, I saw that some folders only have a .frm file in it. Searching for that I found out that there is two major engines within MySQL: MyISAM and InnoDB. If the table belongs to InnoDB, it only includes .frm file. Data of that kind of DBs are stored in a single or multiple (you can configure that) .idb files.
Labels:
database,
database size,
mysql,
wso2,
WSO2 Data Services Server
Prototype version 1 has to be verified.
This prototype is tested on my own
machine with several MySQL users created by me. So this is not
guaranteed to work in the real scenario. This should work with the
actual billing architecture, to be any use for my work. This is the
second and biggest verification that I have to do.
Before that there is a simple and basic
verification that I have to consider. In prototype code I calculate a
table size as follows (here information from
information_schema.TABLES is used)
Table Size = Data Length + Index Length
Both Data Length and Index Length can
be found in information_schema.TABLES. I have to verify that figure
given by the above calculation is the correct table size.
Prototype version 1
As I was asked to build a proof of
concept(POC), as I mentioned in before posts, I started working on
it. By now I have a working prototype that logs the DB size
information. Here I will create two tables one contain user details
like user name, user DB size limit and binary information column
'exceeded' that will indicate whether subject user has exceeded his
limits (this column is not used feather in my currant implementation,
but I thought it would be helpful to have such a column in future).
Second table is a logging table, this
logs the disk space used by each user, time to time. This again has 3
columns. A time stamp, user name and exceeded number of bytes (used
number of bytes-user limit in bytes) are the information that is
included in that table.
Below is the simple code that do that
job
include code here.
import java.sql.*; public class sizeCheck { public static void main(String args[]) { Connection con = null; Statement stat, stat2; boolean debug = true; try { Class.forName("com.mysql.jdbc.Driver").newInstance(); con = DriverManager.getConnection("jdbc:mysql://localhost/", "root", "root"); if (debug && !con.isClosed()) System.out.println("Successfully connected to " + "MySQL server using TCP/IP..."); stat = con.createStatement(); stat2 = con.createStatement(); // query to select all of the data from a table String selectQuery = "SELECT user,sizelimit FROM `Quota`.`Quota`"; // get the results ResultSet results = stat.executeQuery(selectQuery); // output the results while (results.next()) { String getDBsizes = "SELECT SUM(DATA_LENGTH) as sumData, SUM(INDEX_LENGTH) as sumIndex FROM `information_schema`.`TABLES` WHERE TABLE_SCHEMA LIKE '" + results.getString("user") + "%'"; ResultSet sizeResult = stat2.executeQuery(getDBsizes); sizeResult.next(); if (debug) { System.out.println(sizeResult.getInt("sumData")); System.out.println(sizeResult.getInt("sumIndex")); System.out.println(sizeResult.getInt("sumData") + sizeResult.getInt("sumIndex")); } if (sizeResult.getInt("sumData") + sizeResult.getInt("sumIndex") > results.getInt("sizelimit")) { int exccededGB = sizeResult.getInt("sumData") + sizeResult.getInt("sumIndex") - results.getInt("sizelimit"); String setExceeded = "UPDATE `Quota`.`Quota` SET `exceeded` = '1' WHERE `Quota`.`user` = '" + results.getString("user") + "';"; stat2.executeUpdate(setExceeded); String logExceeded = "INSERT INTO `Quota`.`exceed` (`datetime` ,`user` ,`exceedbytes` ,`other`) VALUES (CURRENT_TIMESTAMP , '" + results.getString("user") + "', '" + exccededGB + "', '');"; stat2.execute(logExceeded); // String revoke = // "REVOKE INSERT, UPDATE, CREATE, ALTER ON `"+ // results.getString("user") +"\\_%` . * FROM '"+ // results.getString("user") +"'@'localhost';"; // if(debug) // System.out.println(revoke); // stat2.execute(revoke); } else { String setExceeded = "UPDATE `Quota`.`Quota` SET `exceeded` = '0' WHERE `Quota`.`user` = '" + results.getString("user") + "';"; stat2.executeUpdate(setExceeded); // String grant = "GRANT ALL PRIVILEGES ON `"+ // results.getString("user") +"\\_%` . * TO '"+ // results.getString("user") +"'@'localhost';"; // if(debug) // System.out.println(grant); // stat2.execute(grant); } } } catch (Exception e) { System.err.println("Exception: " + e.getMessage()); } finally { try { if (con != null) con.close(); } catch (SQLException e) { } } } }
you can download it form below
Subscribe to:
Posts (Atom)