Non Functional Stats Service

The Non Functional Stats Service (NFSS) is composed of a few different parts:

  • A python3 / pyramid app that exposes a RESTful interface to a postgres data store.
  • A client-side javascript-based UI that talks to the above.

While the system is generally very simple, there are a few things that sysadmins ought to be aware of:

Generating Client Keys

Test data is submitted to the data store via the RESTful api, and this API call is secured with oauth. Client access keys need to be generated for every external client that wants to be able to post data into the database. This is achieved by changing to the nf-stats-service directory and running:

$ python3 -m nfss keys-add

This is an interactive script that will ask for client details, and finally will write a python script that can be used by the external client to insert data into the data store.

A list of client keys can be generated in a similar fashion:

$ python3 -m nfss keys-list

A specific client id can be revoked by specifying it’s client access key like so:

$ python3 -m nfss keys-del cj2DriLAGxmxinDyJzvDVQVltRSLNI

(obviously the client key will change, this is just an example).

Cleaning the database

Part of the oauth authentication scheme involves storing nonce values in the database. In order to prevent this table from filling up, we install a daily cron job that cleans the database. This can be achieved manually by running:

$ python3 -m nfss database-clean

Although this should never need to be done manually, since the restish charm installs a cron daily job to run this command.

Defining your Graph

Outline

A graph definition consists of two files; a .js and a .html.

The javascript file defines a controller that handlers receiving the ‘newdata’ event and prepares the raw data for display.

The html file presents the massaged data to the user (for instance creating a table using ng-repeat directives).

The front end also needs to know about the graph definition and needs to be added to the ‘definitions’ datastructure found in graphing.module.js.

Develop Locally With Ease

Under normal conditions, the Same Origin Policy prevents Javascript-enabled websites such as this one from making AJAX requests to sites on other domains. This is an important security measure on the web, but unfortunately prevents us from being able to develop the graph definition files locally and test them with live data from the production server.

Fortunately, Chromium offers a way to disable the Same Origin Policy for development purposes, allowing you to develop your graph definition files locally while still making AJAX requests to the live production server, so you have live production data to display in your experimental graph.

All you have to do is launch Chromium with this command:

chromium-browser --disable-web-security /path/to/web_static/index.html

Obviously it is extremely important not to surf the wider internet while Chromium is in this mode, but it sure makes it easy to iterate rapidly on the graph definition files. Don’t forget to close Chromium when you’re done!

Naming your Graph Definition File

Graph definition files are stored under /web_static/graphs and when looking to load them, the web page will first attempt to load your-project-name_your-test-name.html, and failing to find that, it will then search for your-test-name.html and finally your-project-name.html before giving up.

When deciding which name to use for your graph definition, you need to consider the data structure output by your tests.

  • If you have one project that has a number of different tests which all output the same data structure, you’ll want to define your-project-name.html to render that graph data.
  • If you have many projects all running one test which all output the same data structure, you’ll want to define your-test-name.html
  • If for some reason you have a combination of projects and tests which do not output a consistent data structure, then you have the ability to define graphs that are unique to each project+test combination, however this is discouraged because it will likely result in a high degree of code duplication between graph definitions. In this case use your-project-name_your-test-name.html

Data Processing

Alright, so you’ve got your data in the db and now you want to display a pretty graph of it? Great! First, you should read the files in web_static/graphs/ to get some examples of what graph definition files look like. Graph definition files can contain any arbitrary HTML, and can also contain any arbitrary AngularJS directives, such as ng-repeat.

The only hard requirement is that a controller defined in the javascript listens out for the ‘newdata’ event and handles the raw data in some way (otherwise it will never receive any data).

For instance the simplest controller + event handler is shown below (taken from the default graph definition code).

Note. these two examples are the complete code for the default graph definitions (default.html & default.js).

angular.module('NonFunctional.graphing')
.controllerProvider.register("defaultGraphData", ['$scope', '$rootScope',
    function($scope, $rootScope) {
        $scope.graphData = [];
        // Expose a couple of helper methods to the template.
        $scope.dateFormatter = dates.dateFormatter;
        $scope.dateParser = dates.dateParser;

        $rootScope.$on("newdata", function(event, data) {
            $scope.graphData = [{ values: data.data }];
        });
    }
]);

With this data handling in place we can display a simple list of the data like so:

<div ng-controller="defaultGraphData">
  <ol>
    <li ng-repeat="item in graphData[0].values">
      <b>{{dateFormatter()(dateParser(item.date_entered))}}:</b>
      <br />
      {{item.data}}
    </li>
  </ol>
</div>

This will result in the following output in the browser:

For a more thorough real-world example for handling data I’ll use the bootspeed data. The JSON blob that you submit to the db looks like this:

{
    "image_release": "utopic",
    "image_build_number": "1:20140428:20140411.3",
    "kernel": 4.42,
    "xorg": 4.77,
    "kernel_init": 0,
    "desktop": 15.15,
    "plumbing": 8.43,
    "image_arch": "mako",
    "image_md5": "n/a",
    "ran_at": "2014-04-28 14:18:17.76909-04",
    "image_variant": "touch",
    "machine_mac": "",
    "machine_name": "mako",
    "boot": 32.77
}

Then the db will return that data wrapped with a little bit of metadata like this:

{
    "data": {
        "image_release": "utopic",
        "image_build_number": "1:20140428:20140411.3",
        "kernel": 4.42,
        "xorg": 4.77,
        "kernel_init": 0,
        "desktop": 15.15,
        "plumbing": 8.43,
        "image_arch": "mako",
        "image_md5": "n/a",
        "ran_at": "2014-04-28 14:18:17.76909-04",
        "image_variant": "touch",
        "machine_mac": "",
        "machine_name": "mako",
        "boot": 32.77
    },
    "date_entered": "2014-04-28 18:18:17.76909+00",
    "id": 21272
}

But, and here’s the big trick, the d3 graphing library we’re using doesn’t like this data structure at all. As you can see, all the numbers we want to graph (kernel, xorg, desktop, plumbing) are all keys in the same object. D3 doesn’t have any provision for having multiple data points within the same object, and instead requires a data structure that looks like this:

[
    {
        key: 'Kernel',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    },
    {
        key: 'Plumbing',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    },
    {
        key: 'XOrg',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    },
    {
        key: 'Desktop',
        values: [ { date_entered: "...", value: 1 }, { date_entered: "...", value: 2 }, ... ]
    }
]

So, in order to convert from the original data structure to the expected data structure you must write a function to massage the data into a usable state. Obviously, the function that converts your data into d3’s graphable data structure is going to depend heavily upon the structure of your data, so there aren’t any hard and fast rules I can give here. The example from bootspeed.html is a good starting point, which I will attempt to explain:

// setup event handling any new data and refreshing the graph.
$rootScope.$on("newdata", function(event, data) {
    // Massage the raw data into something usable by the tables/charts
    // (declared in the html).
    massageGraphData(data);
    // Refresh the graph once the new data is ready.
    $scope.api.refresh();
});

// Helper function used in massaging the data.
function isolator(key) {
    return function(item) {
        return {
            // using the dates service here that provides date manipulation
            // functions.
            date_entered: dates.dateParser(item.date_entered),
            timespan: Math.max(item.data[key], 0) }
    }
}

$scope.massageGraphData = function(blob) {
    $scope.rawData = blob;
    $scope.graphData = [
    {
        key: 'Kernel',
        values: blob.data.map(isolator('kernel'))
    },
    {
        key: 'Plumbing',
        values: blob.data.map(isolator('plumbing'))
    },
    {
        key: 'XOrg',
        values: blob.data.map(isolator('xorg'))
    },
    {
        key: 'Desktop',
        values: blob.data.map(isolator('desktop'))
    }
    ];
}

In this example, there are two functions, isolator and massageGraphData. The isolator is a meta-function (that is, a function that returns another function). The purpose of this, is that you can call it with the json key you want to extract from the larger blob, and it returns a function that is programmed to take a blob, return just that one key, along with the date_entered key, and nothing else (it strips non-essential data from the json blob). So the inner function inside isolator will return an object that only contains keys date_entered (which we graph on the X axis) and timespan (which we graph on the Y axis).

Note that there’s nothing special about the name timespan. You can use whatever name makes sense for your data. The only important thing is that later on when you define your data accessors, you need to use the same name so that d3 can find your data within the structure.

We setup an event handler that will trigger whenever any new data is provided. The data supplied in this events corresponds to the complete json data blob returned by the REST API endpoint /api/v1/:project_id/:test_id.

Note. By default this is all of your data points from within the last 30 days, but that can be controlled by the start_date and end_date URL query parameters.

Next, there’s the massageGraphData function. This function will take the raw blob data from the event handler and produce a usable subset of the data for graphing.

In particular, blob.data will be a list of objects that look like the second JSON example listed earlier in this document. As you can see I’m calling blob.data.map several times in massageGraphData. The map function iterates over every item in the list (eg, every data point), calls the function returned by isolator, and returns a new list with the successive values returned by the function returned by isolator. The end result of this is that scope.graphData[0].values is a list of objects which contain only date_entered and timespan keys, such that timespan refers to the kernel value from the original data blob. scope.graphData[1].values will be similar, but with the plumbing key instead of the kernel key, and so on.

Note: Assigning your massageGraphData function to the $scope like this allows you to write unit tests against your controller so that you can prove that it works as expected.

Choosing your Chart Type

The next part of the graph definition is technically free-form HTML, although most likely you’ll want to define some sort of chart or graph. Technically speaking, you can do absolutely anything you want with AngularJS directives. If you wanted to go totally crazy, you could create a scatter plot by defining your own SVG tag, and then using AngularJS’ ng-repeat in order to create an arbitrary number of circles with your x, y, and radius values dropped in, however I don’t generally recommend fiddling with SVG data directly because then you don’t get nice things like labelled axes.

In general, you’re going to want to use one of the pre-defined nvd3 charts, which you can read more about here:

http://krispo.github.io/angular-nvd3

I’ll continue using the bootspeed graph as my example, which uses a stacked-area-chart, but you can also refer to app_startup_benchmark.html which defines a (non-stacked) line chart.

The setup for the graph in the html is pretty simple. Note the surrounding div that has a ‘ng-controller’, attribute. This tells angular that the bootspeedCtrl controller (defined in the js file) is the contoller backend for this div and will supply the ‘options’ and ‘data’ datastructures.

<div ng-controller="bootspeedCtrl">
    <nvd3 options="options" data="graphData" api="api"></nvd3>
</div>

The actual options for the chart are defined in the controller and is a dictionary containing the key/values for the chart.

$scope.options = {
    chart: {
        type: "stackedAreaChart",
        height: 400,
        margin: {left:100, top:10, bottom:40, right:100},
        x: modifiers.accessor('date_entered'),
        y: modifiers.accessor('timespan'),
        useInteractiveGuideLine: true,
        xAxis: {
            tickFormat: dates.dateFormatter(),
            staggerLabels: true,
        },
        yAxis: {
            tickFormat: modifiers.numberFormatter(',.2f', 's'),
            tickPadding: 10,
        },
    },
};
  • type This is the most important option, without it nothing will show. In this example we are using a “stackedAreaChart”. Please see this link for more options: http://krispo.github.io/angular-nvd3/.
  • height and margin can be adjusted to your liking. Don’t define a width because it’s defined to be 100% in the CSS, which makes the most efficient use of the screen space.
  • x and y tell d3 how to find the x and y values in the data structure you created. There’s nothing special about the values timespan or date_entered, they just need to match what you defined in isolator example.
  • xAxisTickFormat and yAxisTickFormat can be adjusted to your liking. If you don’t like the default date format, you can pass in a printf-style date format string to the dateFormatter() function, or you can change it to the numberFormatter() if your X axis isn’t time for whatever reason.
  • modifiers.numberFormatter() (available in support.module.js) takes two arguments, the format string, and the units. In this case the numbers we’re graphing are seconds, so ‘s’ is passed in. This can be any arbitrary string and is simply concatenated onto the end of the formatted numbers for display purposes only. You can read more about the format string mini-language here:

So for example I’m using ,.2f here, which means “round the number to two decimal places, and use a comma as the thousands-separator”.

Raw Data Table

If you would like to display a raw data table beneath your graph, you can pretty well copy & paste this exact snippet into your code:

<table>
    <tr ng-repeat="series in graphData | reverse">
        <td><b>{{series.key}}:</b></td>
        <td ng-repeat="item in series.values">{{numberFormatter(',.2f', 's')(item.timespan)}}</td>
    </tr>
    <tr>
        <td><b>Date Entered:</b></td>
        <td ng-repeat="item in graphData[0].values">{{dateFormatter(ISO_ISH)(item.date_entered)}}</td>
    </tr>
</table>

// You need to expose the helper methods to the scope (within the
// controller):
$scope.ISO_ISH = dates.ISO_ISH;
$scope.dateFormatter = dates.dateFormatter;
$scope.numberFormatter = modifiers.numberFormatter;

This example is using Angular directives to fill out data values into a literal HTML table, and does so in a way that doesn’t hard-code any knowledge about the number of data series (aka lines) on the graph or their names. Notice how similar this snippet is between bootspeed.html and app_startup_benchmark.html (basically the only difference is item.timespan vs item.delta).

And again I’d like to emphasize the true arbitrariness of this HTML here. If you don’t want the data table, don’t include it. If you don’t want the graph, don’t include it! If you want to add some paragraphs explaining how to interpret the data, by all means, throw some p tags in there. Honey badger don’t care.

If the default.js data handler does what you need for the data but you want to display it differently then you can re-use the default.js file, define your own html with whatever custom tables etc. you need. The trick to do this is when you add the graph definition to the definitions dictionary:

this.definitions = {
    // Existing default definition.
    'default': {
        'templateUrl': 'graphs/default.html',
        'deps': 'graphs/default.js'
    },
    // Defining your graph def here.
    'my_test_name': {
        'templateUrl': 'graphs/my-custom-test-display.html',
        'deps': 'graphs/default.js'  // <-- Note the use of the default controller here.
    },
};