Scaling Graphite

Graphite is a great tool to collect metrics on anything. Graphite has great performances by itself. We collect over 300K metrics per second on a single virtual server. But for the hardware we have, that’s close to the limit, we start to loose metrics from time to time. So, weekend exercise : how to scale Graphite to multiple servers.

The goal is :

  • increase the number of servers to allow more scalability
  • keep a single interface to consult metrics
  • keep a single interface to publish metrics
  • keep things as simple as possible (but no more)
  • have everything deployed with Puppet

The result looks more or less like this (based on PlantUML):
graphite-components

There is already good documentation, I will not repeat it. But I was not able to find a complete code example. My implementation is expressed as a few Vagrant VMs and is available on GitHub.

Still to be done

  • Adding more servers to the Graphite cluster means moving existing data around, some scripts need to be integrated for that
  • scripts to clean up corrupted data need to be added to the puppet-carbon module

Helpful sources

Automation of Rundeck

Rundeck is usually thought of as a tool to automate your infrastructure. As we know, it is turtles all the way down, so the obvious question is how do we automate Rundeck itself.

Why would we want to automate Rundeck ? What exactly do we want to automate ? I’ll present my use case, let me know if it applies for you as well:

Scalability
We manage a bit less than a hundred different applications. We manage deployment and some administrative tasks with Rundeck. Those tasks are similar between applications. We don’t want to configure 80 different rundeck jobs for each task, instead we want a generic definition of those jobs and instanciate it for each application, taking care of the specificities if needed.
Reproducibility
Everything deployed on production environment should be tested somewhere else, validate, tested, … To ensure that what you deploy on production is the same as what you tested on test environment, you need a way to version your jobs and have a promotion workflow from test to production.
Coherence
With multiple projects, if the jobs for each application are managed separately, they will start to diverge. We need to ensure that the procedures implemented as rundeck jobs are coherent and that common code is properly factorized.

What I want, is to manage my Rundeck configuration in the same way I manage my applications and my Puppet modules. With the same build, release, versioning and promotion mechanism. For that, I need simple Puppet resources that will allow me to create projects and jobs. Then, I will be able to control them from a mix of role classes and hiera.

What is missing?

Creating projects is done with the command line rd-project tool. Configuring the project is done through the Web interface, or by editing the project.properties file. Project definitions are reloaded mostly automagically, so everything is fine here. I have a basic implementation of a Puppet resource to manage Rundeck project on GitHub.

Managing jobs is a bit harder. You can create jobs through the rd-jobs, but you don’t get direct access to the job definition once it is created. You can reload a job with the same name and it will override the job definition. This makes it harder to manage correctly modifications to jobs, or job removal. I have a buggy half working implementation of jobs, also on GitHub.

What I want in the future?

I’ll keep working on my Puppet module to better handle the quirks of Rundeck. It already works for the basic use cases, but need some love to manage more complex workflows.

Modification to Rundeck itself could make the job much easier, for example by storing all project and job definition directly in files. While the Rundeck code base is of reasonable size and quite readable, I probably won’t have time to dig into it right now. Any help is welcomed on that side …

My griefs with hiera-gpg

Hiera-gpg seems to be the standard way to to store encrypted data in hiera. Craig Dunn has a pretty good article on how to deploy and use it. Storing sensitive data encrypted is a pretty good idea, using asymmetric crypto is even better. But still I am frustrated.

First, let’s have a look at the problem space. In our case we have the following requirements:

  1. all configuration should be stored in a version control system – VCS (SNV / Git / …)
  2. sensitive data should be stored and transmitted securely
  3. only our puppet master should able to decrypt data
  4. anybody should be able to encrypt data
  5. authorization should be handled only by the VCS

hiera-gpg fills the first three requirements, but fails just short on 5 and 6.

To encrypt data, no problem, you create a new yaml file, encrypt it, publish it to your VCS. Anyone can access thewe puppet master’s public key, so anyone can encrypt data. As long as you have access to the VCS, you can publish this encrypted file.

Let’s face it, most of the time we don’t publish a brand new configuration, we modify an existing one. To modify it, we must have the plaintext version. We could either store the plaintext somewhere, but that defeats the whole purpose of encrypting it. We could encrypt data with our own public key to be able to retrieve the plaintext, but that defeats requirement #3. Further more, we work in teams. We could share our private key with our team, but key repudiation becomes an issue when people leave the company. And even more, we work with multiple teams, with divers responsibilities. No team should have the full knowledge of all our private information.

How can we do better …

Instead of encrypting the whole file, we can encrypt individual properties. Each property can than be modified by anyone, but read by no one. When you modify a property, you know its new value, but not its old value. We could even mix encrypted and non encrypted properties. The format would look like this:

---
- database: 
    username: dbuser
    password: ENC(XXXXXXXX)

Anyone can modify this password, but nobody can decrypt it.

Now I just need time to learn ruby and write a hiera extension.

The cost of HTTP redirects

We all know HTTP redirects are costly. But how costly ?

I had this conversation at work. To serve different images for different platforms (desktop, mobile, tablet, …) we have to introduce a level of indirection between the image URL and the actual file. For example, the URL http://mysite.net/images/123/mobile will resolve to a file called 456.png. Pretty standard, should be easy. Unfortunately, we use a framework that will make us write pretty ugly code to achieve this server side. The question was asked: “Could we not use HTTP redirect? This is a standard web technique…”.

I dont think we should ever expose crap to our clients, even if it allow us to write cleaner code. In this world appearence is key, not inner beauty. Still the question was asked: “How much would those redirects cost?”. Should be simple to test …

An evening of coding in front of the TV later, I have a very simple test case, which is probably wrong on so many levels that Christopher Nolan might have written it himslef, but still, I have numbers:

Downloading 10 images to my desktop with an HTTP redirect each time takes on the order of 450ms, while downloading the same images without the redirects takes about 250ms. On my mobile phone on a 3G connection, the number climb to 1200ms and 750ms respectively. Not the end of the world, but still, we could do a lot of better things in those 100s of milliseconds.

The test is available on CloudBees, go see for yourself (your numbers may vary). The source is available on GitHub. The implementation is very simple :

  • a servlet redirecting all requests
  • a servlet serving always the same image (different URL, so no caching)
  • an HTML page loading images with Javascript and publishing the time it took

Graphing performance of your datasource

One of the project on which I am working is extremely dependent on the database performance. This project executes an extremely high number of small SQL queries. Each query usually takes less than one millisecond. If the average query time goes from 1 [ms] to 1.5 [ms], after a complex calculation, you can see that we degrade performance by 50%.

Performance from the database side are still considered as good. We need a way to measure those performance in a very precise way to see if performance degradation actually come from the database or if we need to look for problem somewhere else. And we need a way to graph those data to identify problems quickly.

We already use Graphite and Statsd, and we already use Tomcat with its tomcat-jdbc pool. The tomcat-jdbc pool provides an interceptor mechanism that can be used in our case.

The implementation of our interceptor is minimal. First step, let’s get the timings of all operations on the connections. For that, we just need to override the invoke() method :

@Override
public final Object invoke(final Object proxy, final Method method,
        final Object[] args) throws Throwable {
    String methodName = method.getName();
    long start = System.nanoTime();

    Object result = super.invoke(proxy, method, args);

    increment(methodName + ".count");
    timing(methodName + ".timing", System.nanoTime() - start);
    return result;
}

The increment() and timing() methods are taken almost directly from the Statsd Java example. The actual implementation is slightly more complex than what is shown here to ensure that we also proxy the statements created from the connection and get their performances as well.

We need to give the interceptor some information, the name and port of our Statsd server, the sampling rate to make sure we dont hurt performances to badly and a prefix to organize our data.

Activating the interceptor is done by modifying your Tocmat datasource :

<Resource name="jdbc/TestDB"
          type="javax.sql.DataSource"
          factory="org.apache.tomcat.jdbc.pool.DataSourceFactory"
          jdbcInterceptors="ch.ledcom.tomcat.interceptors.StatsdInterceptor(hostname=localhost,port=8125,sampleRate=0.1,prefix=myapplication.jdbc)"
          username="myusername"
          password="password"
          [...]
          driverClassName="com.mysql.jdbc.Driver"
          url="jdbc:mysql://localhost:3306/mysql"/>

Conclusion: JDBC interceptors are a great and easy way to get interesting information from your datasource !

The full code can be found on GitHub, binaries can be downloaded from Maven repo1 repository.

Gamestorming

I have a soft spot for group dynamic. Finding ways to help groups work better, make people think in new ways … that’s a lot of fun. I have worked quite a bit with teenagers and young adults in different settings, trying to get them to think about common problems in new ways. I have tried to bring that knowledge and that energy to my workplace, but it’s been harder than I thought.

When I found “Gamestorming” by Dave Gray, Sunni Brown, James Macanufo (O’Reilly Media) I had great hopes. It starts with the basic theory of animation games, introduces some basic tools, some basic games and provides a large range of games that can be used in an enterprise. It made me think, it made me review the basics, it motivated me to try new things. Goal achieved.

Despite that, it could be better. It could go further. It could be bolder, be further out of the proverbial box. Maybe the enterprise is just not ready for that. Just an example:

A brainstorming is much more about the storm than about the brain. It’s a technique that creates chaos to shut down our intellect and get our creativity flowing. Without the chaos, a brainstorming just another session where we try to generate new ideas and fail. The node generation section (chap. 2, section 4, page 19) goes just the other way, by explaining how to try to put order right from the start, proposing people to write notes silently on post-it. Of course, it’s closer to what we are used to, easier to bring to a group not used to working outside of its confidence zone. Yes, this technique can be useful in certain situations, but no, it is not the canonical example of a node generation exercise.

That being said, I think that my criticism is more toward the enterprise world than toward the book. My experience of the kind of games presented in “Gamestorming” is by doing them in associations, where people much more ready to try new things, to push boundaries further. I’ll keep thinking on how I can bring that to my workplace …

SiteSpeed.io

I just found out about Sitespeed.io. Basically it is an integration between a site scraper, a standalone, headless webkit implementation (PhantomJS) and YSlow. Sounds like a simple idea, but the execution is really good ! There are plenty of browser plugins, or online solutions to measure the client-side performance of your website, but they are not easy to automate. Sitespeed.io allows me to add a simple step to my integration tests and get metrics on the quality of my app each time a developer makes a commit. This should be implemented in my apps next week.

What could be improved ?

No, we don’t live in a perfect world, so yes, there is always something that can be improved.

  1. Find a way to collect metrics over time,
  2. integrate with Maven,
  3. support Java 6.

Metrics over time:

I have strong suspicions towards any kind of metrics on their own. I don’t really care if my test coverage is 60% or 80% or 110%. What I care about is that my metrics tell me where I can improve my application and that my application is getting better overtime, not worse. In my workplace, we use Sonar for that. That way, we can make sure that test coverage improve overtime, or we can easily identify which classes are too complex and need to be refactored. I’d love to have the same kind of metrics for client side code quality as well.

Maven integration:

Maven integration would allow to integrate the collection of metrics in the standard build process. Integrating SiteSpeed.io in a Continuous Integration tool is pretty easy, so this is definitely a minor point. A better integration with our build tool would make it even easier for developers to collect metrics. Even if I am not personally a big fan of this approach, it would allow us to fail the build if a minimum quality is not reached.

Java 6 support:

Yes, I know, Java 7 is out and is the third best thing in the world (just after sliced bread and portioned coffee). But for us stuck in the corporate world, Java 6 is still the norm (when it’s not Java 5). It should be pretty easy to compile dependent Jars for Java 6. I’ll have a go at that as soon as I can …

Overall, great project ! I’ll tell you more as soon as I have a bit more experience with it.