add multiple engines support

add multiple tables support
update README

README.md

@@ -3,6 +3,9 @@ nodejs-statsd-mysql-backend
 
 MySQL backend for Statsd
 
+## Contributors
+This statsd backend is developped by Nicolas FRADIN and Damien PACAUD.
+
 ## Install
 Go into Statsd parent directory and execute :
 ```bash
@@ -42,8 +45,8 @@ Required parameters :
 
 Optional parameters :
 
-* `tables`: List of tables names used (ex: ["stats", "users"]).
-* `engines`: List of MySQL Backend engines (see 'MySQL Bakend Engines' chapter for more details).
+* `tables`: List of tables names used (see 'Customize MySQL Bakend Database' section for more details).
+* `engines`: List of MySQL Backend engines (see 'Customize MySQL Bakend Engines' section for more details).
 
 
 ## Introduction
@@ -51,11 +54,113 @@ This is node.js backend for statsd. It is written in JavaScript, does not requir
 
 It save statsd received values to a MySQL database.
 
-## Data Structure for statsd counters
-By default, values are stored into a 'counters_statistics' table. This table has a very simple structure with 3 columns :
+## Data Structure for Counters
+By default, counters values are stored into a 'counters_statistics' table. This table has a very simple structure with 3 columns :
 * `timestamp`: The timestamp sent by statsd flush event.
 * `name`: The counter name.
 * `value`: The counter value.
 
-The primary key of this table is composed by fields: timestamp and name. It means when a new value arrives for a counter, this value is added to the previous one and stored in database. With this mechanism, we can keep a log of counters values.
+The primary key of this table is composed by fields: `timestamp` and `name`. It means when a new value arrives for a counter, this value is added to the previous one and stored in database. With this mechanism, we can keep a log of counters values.
+
+
+## Customize MySQL Backend Database
+
+If you want to change where statsd datas are stored just follow the guide :)
+
+By default database tables are defined like that :
+````js
+{
+	counters: ["counters_statistics"],
+	gauges: ["gauges_statistics"],
+	timers: ["timers_statistics"],
+	sets: ["sets_statistics"]
+}
+```
+
+If we want to duplicate statsd counters datas into a new table called 'duplicate_counters_stats', we have to add new table name to counters tables list.
+
+Open stats config file and add tables configuration :
+```
+mysql: { 
+   host: "localhost", 
+   port: 3306, 
+   user: "root", 
+   password: "root", 
+   database: "statsd_db"
+
+   // Tables configuration
+   tables: {
+   		counters: ["counters_statistics", "duplicate_counters_stats"]
+   }
+}
+```
+
+Then place new table creation script into "nodejs-statsd-mysql-backend/tables" directory.
+The file should be nammed "[table_name].sql", so create a file named 'duplicate_counters_stats.sql'.
+
+Example of creation script 'duplicate_counters_stats.sql' :
+```sql
+CREATE  TABLE `statsd_db`.`duplicate_counters_stats` (
+    `timestamp` BIGINT NOT NULL ,
+    `name` VARCHAR(255) NOT NULL ,
+    `value` VARCHAR(45) NOT NULL ,
+PRIMARY KEY (`timestamp`, `name`) );
+```
+
+The last step is the modification of the Counters Query Engine. We can also create a new Query Engine but we will see how to do that in the next section.
+
+Open the file "nodejs-statsd-mysql-backend/engines/countersEngine.js".
+
+We will focus on a specific line of this file :
+```js
+querries.push("insert into `counters_statistics` (`timestamp`,`name`,`value`) values(" + time_stamp + ",'" + userCounterName +"'," + counterValue + ") on duplicate key update value = value + " + counterValue + ", timestamp = " + time_stamp);
+```
+
+Just duplicate this line and change the table name :
+```js
+querries.push("insert into `counters_statistics` (`timestamp`,`name`,`value`) values(" + time_stamp + ",'" + userCounterName +"'," + counterValue + ") on duplicate key update value = value + " + counterValue + ", timestamp = " + time_stamp);
+
+querries.push("insert into `duplicate_counters_stats` (`timestamp`,`name`,`value`) values(" + time_stamp + ",'" + userCounterName +"'," + counterValue + ") on duplicate key update value = value + " + counterValue + ", timestamp = " + time_stamp);
+```
+
+Values will be inserted in two tables: 'counters_statistics' and 'duplicate_counters_stats'.
+
+In this example, colums are the same in two table so, we just have to change the table name.
+
+But you can customize this...
+
+
+## Customize MySQL Backend Query Engines
+
+If you want to add customized querry engines to MySQL Backend, it's very simple.
+
+First, create a new engine in "nodejs-statsd-mysql-backend/engines" directory.
+For example, copy the existing "countersEngine.js" and rename it into "customizedCountersEngine.js".
+
+Make some modifications inside it...
+
+Then, declare the new engine in MySQL Backend configuration.
+Open statsd config file and add engines configuration:
+
+```js
+mysql: { 
+   host: "localhost", 
+   port: 3306, 
+   user: "root", 
+   password: "root", 
+   database: "statsd_db"
+
+   // Tables configuration
+   tables: {
+   		counters: ["counters_statistics", "duplicate_counters_stats"]
+   }
+
+   // Query Engines configuration
+   engines: {
+   		counters: ["engines/countersEngine.js", "engines/customizedCountersEngine.js"]
+   }
+}
+
+```
 
+Your querry engine will be triggered for each new Counter data.

countersEngine.js

@@ -20,7 +20,14 @@ MySQLBackendCountersEngine.prototype.buildQuerries = function(userCounters, time
       if(counterValue === 0) {
         continue;
       } else {
-        querries.push("insert into `counters_statistics` (`timestamp`,`name`,`value`) values(" + time_stamp + ",'" + escape(userCounterName) +"'," + counterValue + ") on duplicate key update value = value + " + counterValue + ", timestamp = " + time_stamp);
+        /**********************************************************************
+         * Edit following line to custumize where statsd datas are inserted
+         *
+         * Parameters :
+         *    - userCounterName: Counter name
+         *    - counterValue: Counter value
+         */
+        querries.push("insert into `counters_statistics` (`timestamp`,`name`,`value`) values(" + time_stamp + ",'" + userCounterName +"'," + counterValue + ") on duplicate key update value = value + " + counterValue + ", timestamp = " + time_stamp);
       }
     }
 

mysql-backend.js

@@ -36,7 +36,12 @@ var STATSD_BAD_LINES = "statsd.bad_lines_seen";
 function StatdMySQLBackend(startupTime, config, emitter) {
   var self = this;
   self.config = config.mysql || {};
-  self.engines = {};
+  self.engines = {
+    counters: [],
+    gauges: [],
+    timers: [],
+    sets: []
+  };
 
   // Verifying that the config file contains enough information for this backend to work  
   if(!this.config.host || !this.config.database || !this.config.user) {
@@ -59,13 +64,17 @@ function StatdMySQLBackend(startupTime, config, emitter) {
 
   //Default tables
   if(!this.config.tables) {
-    this.config.tables = ["counters_statistics"];
+    this.config.tables = {counters: ["counters_statistics"]};
   }
 
   // Default engines
   if(!self.config.engines) {
-    self.config.engines = {};
-    self.config.engines.countersEngine = "./countersEngine.js";
+    self.config.engines = {
+      counters: ["engines/countersEngine.js"],
+      gauges: [],
+      timers: [],
+      sets: []
+    };
   }
   
 
@@ -82,17 +91,30 @@ function StatdMySQLBackend(startupTime, config, emitter) {
 
 
 /**
- *
+ * Load MySQL Backend Query Engines
  *
  */
 StatdMySQLBackend.prototype.loadEngines = function() {
 	var self = this;
 
-  // Load counters engine
-  self.engines.countersEngine = require(self.config.engines.countersEngine).init();
-  if(self.engines.countersEngine === undefined) {
-    console.log("Unable to load counter engine ! Please check...");
-    process.exit(-1);
+  // Iterate on each engine type defined in configuration
+  for(var engineType in self.config.engines) {
+    var typeEngines = self.config.engines[engineType];
+
+    // Load engines for current type
+    for(var engineIndex in typeEngines) {
+      // Get current engine path
+      var enginePath = typeEngines[engineIndex];
+
+      // Load current engine
+      var currentEngine = require(self.config.backendPath + enginePath).init();
+      if(currentEngine === undefined) {
+        console.log("Unable to load engine '" + enginePath + "' ! Please check...");
+        process.exit(-1);
+      }
+      // Add engine to MySQL Backend engines
+      self.engines.counters.push(currentEngine);
+    }
   }
 
 }
@@ -137,10 +159,10 @@ StatdMySQLBackend.prototype.closeMySqlConnection = function() {
 
 
 /**
- *
+ * Check if required tables are created. If not create them.
  *
  */
-StatdMySQLBackend.prototype.checkDatabase = function(callback) {
+StatdMySQLBackend.prototype.checkDatabase = function() {
   var self = this;
 
   var isConnected = self.openMySqlConnection();
@@ -149,50 +171,52 @@ StatdMySQLBackend.prototype.checkDatabase = function(callback) {
     process.exit(-1);
   }
 
-  // Check if tables exists
-  for(var table_index in self.config.tables) {
-    var table_name = self.config.tables[table_index];
-    console.log("Check if table exists : '" + table_name + "'");
-    self.sqlConnection.query('show tables like "'+table_name+'";', function(err, results, fields) {
-      if(err) {
-        console.log("Unbale to execute query !");
-        process.exit(-1);
-      }
-
-      // If table doesn't exists
-      if(results.length > 0) {
-        console.log("Table '" + table_name + "' was found !");
-        if(table_index == self.config.tables.length-1) {
-          console.log("-- MySQL Backend is ready !");
+  // Iterate on each stat type (counters, gauges, ...)
+  var tables = self.config.tables
+  for(var statType in tables) {
+
+    // Get tables for current stat type
+    var typeTables = tables[statType];
+
+    // Iterate on each table for current stat type
+    for(var table_index in typeTables) {
+      var table_name = typeTables[table_index];
+      console.log("Check if table exists : '" + table_name + "'");
+      self.sqlConnection.query('show tables like "'+table_name+'";', function(err, results, fields) {
+        if(err) {
+          console.log("Unbale to execute query !");
+          process.exit(-1);
         }
-      } else {
-        console.log("Table '" + table_name + "' was not found !");
-
-        // Try to read SQL file for this table
-        var sqlFilePath = self.config.backendPath + 'tables/' + table_name + '.sql';
-        fs.readFile(sqlFilePath, 'utf8', function (err,data) {
-          if (err) {
-            console.log("Unable to read file: '" + sqlFilePath + "' ! Exit...");
-            process.exit(-1);
-          }
-          
-          self.sqlConnection.query(data, function(err, results, fields) {
-            if(err) {
-              console.log("Unable to create table: '" + table_name +"' ! Exit...");
-              process.exit(-1);
-            } 
-
-            console.log("Table '" + table_name +"' was created with success.");
 
-            if(table_index == self.config.tables.length-1) {
-              console.log("-- MySQL Backend is ready !");
+        // If table doesn't exists
+        if(results.length > 0) {
+          console.log("Table '" + table_name + "' was found !");
+        } else {
+          console.log("Table '" + table_name + "' was not found !");
+
+          // Try to read SQL file for this table
+          var sqlFilePath = self.config.backendPath + 'tables/' + table_name + '.sql';
+          fs.readFile(sqlFilePath, 'utf8', function (err,data) {
+            if (err) {
+              console.log("Unable to read file: '" + sqlFilePath + "' ! Exit...");
+              process.exit(-1);
             }
+            
+            self.sqlConnection.query(data, function(err, results, fields) {
+              if(err) {
+                console.log("Unable to create table: '" + table_name +"' ! Exit...");
+                process.exit(-1);
+              } 
+
+              console.log("Table '" + table_name +"' was created with success.");
+            });
           });
-        });
-      }
+        }
+      });
+    }
 
-    });
   }
+
 }
 
 
@@ -212,6 +236,7 @@ StatdMySQLBackend.prototype.onFlush = function(time_stamp, metrics) {
   var sets = metrics['sets'];
   var pctThreshold = metrics['pctThreshold'];
 
+  // Handle statsd counters
   self.handleCounters(counters,time_stamp);
   
 }
@@ -238,13 +263,23 @@ StatdMySQLBackend.prototype.handleCounters = function(_counters, time_stamp) {
 
     console.log("Preaparing querries...");
 
-    // Call countersEngine's buildQuerries method
-    querries = self.engines.countersEngine.buildQuerries(userCounters, time_stamp);
-    var querriesCount = querries.length;
+    //////////////////////////////////////////////////////////////////////
+    // Call buildQuerries method on each counterEngine
+    for(var countersEngineIndex in self.engines.counters) {
+      console.log("countersEngineIndex = " + countersEngineIndex);
+      var countersEngine = self.engines.counters[countersEngineIndex];
 
+      // Add current engine querries to querries list
+      var engineQuerries = countersEngine.buildQuerries(userCounters, time_stamp);
+      querries = querries.concat(engineQuerries);
+    }
+
+    // Display querries count
+    var querriesCount = querries.length;
     console.log("Querries count : " + querriesCount );
 
-    // If at least one querry can be executed
+    //////////////////////////////////////////////////////////////////////
+    // If at least one querry can be executed, execute pending querries
     if(querriesCount > 0) {
 
       // Open MySQL connection
@@ -262,8 +297,6 @@ StatdMySQLBackend.prototype.handleCounters = function(_counters, time_stamp) {
 
     }
 
-  } else {
-    console.log("No user packets received.");
   }
 
   return;