On the importance of not being smart

I just started a new job, which brings interesting reflections. My new colleagues are incredibly smart (possibly even smarter than than the bunch I just left, but that’s still open for discussion). Smart people tend to be attracted to smart problems. For example, one of my first task was to add configuration so that our Elasticsearch cluster sends its logs to Logstash. Logstash itself is backed by another Elasticsearch cluster, so of course, you need to make sure that this ES cluster does not stat sending logs to itself.

All this is managed by Puppet. A very much simplified interface is:

class elasticsearch (
    $enable_log_shipping = false,
    $logstash_server = undef,
) {

Now if you want to be clever, you could simplify this interface and assume that if no logstash server is defined, we do not want to ship logs:

class elasticsearch (
    $logstash_server = undef,
) {
    $enable_log_shipping = $logstash_server {
        undef   => false,
        default => true,

I would argue that this second version has less expressiveness, but that’s besides the point. The interesting issue we now have is “how do we configure this in hiera“.

A simplified hiera hierarchy could be:

  - "roles/%{::role}"
  - "common"

And of course, we’d like that by default log shipping is enabled, but disabled for role == 'logstash'. This raises a few questions:

  • How do I represent an undef value in hiera?
  • Does hiera stops resolution when it finds an undef value? In other term, are a non existing value and a value explicitly defined as undef the same thing or not?

If you can answer both of the questions above without help from Google, congratulation, you are part of the “knowledgeable” category (disclaimer: I’m not). If the questions above pick your curiosity and you know you will not be able to sleep this evening without the answer, congratulation, you are part of the “smart” category.

Let’s highlight a few more difficulties:

YAML has a concept of language independant null type. It explicitly states that “a mapping entry with some key and a null value is valid and different from not having that key in the mapping”, which is different than how Puppet treats undef (“The undef value is usually useful for testing whether a variable has been set. It can also be used as the value of a resource attribute, which can let you un-set any value inherited from a resource default and cause the attribute to be unmanaged”).

Now, if I set the following hiera configuration:

elasticsearch::logstash_server: !!null
elasticsearch::logstash_server: logstash.example.net

What will happen? Can I assume that logs will be disabled the nodes having role == 'logstash'? What happen if I change logstash.yaml to

elasticsearch::logstash_server: undef

Now let’s compare this to the first proposed interface. The configuration would look like:

elasticsearch::enable_log_shipping: false
elasticsearch::enable_log_shipping: true
elasticsearch::logstash_server: logstash.example.net

A bit more obvious, don’t you think?

Now the actual point of this post:

Smart people are fascinated by smart and / or tricky questions. As soon as this question was asked with my coworkers, they wanted to have the answer (you noticed I have not told you yet?). As smart people, we are attracted in doing the smart thing. It is fun. It is how we learn. It is how we get even smarter. And this is how we screw up 6 month from now when we still have the smart, but we lost the knowledgeable.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s