Static Analysis—what's it good for?

Let’s face it, writing software is hard. And frankly we humans suck at it. We need all the help we can get. Our industry has developed many tools and techniques over the years to provide “safety rails”, from the invention of the macro assembler through to sophisticated integration and automated testing frameworks. But somewhere along the way the idea of static analysis went out of favour.

I’m here to convince you that static analysis tools still have a place in modern software engineering.

Note that I am avoiding the word “metrics” here. That is how a lot of people think of these tools, and the whole idea of “measuring” a developer’s work rightly has a terrible reputation (“what does that number even mean?“). Static analysis is simply about providing information that a human can use to learn more about a code base.

What is static analysis?

Simply, it’s a tool that analyses your code “at rest”. That is, it inspects the source code (or in some cases object code) statically, not in a running environment. (Dynamic analysis of running systems, such as with memory profilers like yourkit and valgrind, is a whole other topic.)

Why is it so unpopular?

One of our developers recently made the following comment in an internal chat channel:

After seeing some refactoring that people have done to satisfy static code quality analysis tools, I question their value.

This is a common response to the use of such tools, and perfectly reasonable. But it misses the point. Of course static analysis can be misused, but that doesn’t mean it has to be.

Another common complaint is “but these tools can’t replace the eye of an experienced developer!“. No, they can’t. But they can help focus that experienced eye where it is most needed.

So what is it good for?

  • Early warning of problems

    By scanning the daily report for issues in recently introduced code, tech leads and senior developers can talk with the developers involved. Together they can work out better approaches before the problematic code becomes ossified in the code base and inevitably replicated by copy/paste.

  • Identifying “hot spots” in the code that warrant further attention

  • Overall sense of the “health” of a codebase

  • See trends over time

    Time series graphs can give you a good overview of how your code base is growing and changing. Steadily increasing LoC in a mature system might be a sign that it’s time to factor out a submodule. Or maybe increasing complexity indicates too much pressure to rush out features without enough consideration for design.

The static analysis tool itself is not going to tell you any of these things, but it might suggest places to look for potential trouble.

What is it NOT good for?

  • Gating check-ins/failing builds

    This usually just leads to “gaming the system” or poor refactoring to “get around” the rules, which helps no one. Static analyses are information that needs to be intrepeted by people, not an automatic way to prevent bad code being committed.

  • Measuring developers’ performance

    Hopefully I don’t need to explain why this is a terrible idea. The output of these tools is the start of a conversation, and should certainly never be used against people or teams.

How should I use my analysis tools?

  • Daily report to tech lead - new issues

    Tech leads can review a daily report as a starting point for conversations with developers. E.g. “I see you commited this method with a lot of nested ifs.. have you considered doing it this way instead?”

  • High level graphs over time - complexity, etc.

    Dashboard of health. Are we getting worse? Should we be putting more effort into refactoring and clean up?

  • Predicting “cost of change”

    If a code base has a high complexity–relative to others in your org, or its own past state–the cost of change is likely to be higher. This can be useful information when estimating/predicting future effort.

  • Enforcing style guides

    This is a bit more controversial, and not really the kind of analysis I am talking about… but there is an argument to be made for using tools like checkstyle and rubocop to enforce your local style conventions. If nothing else, it makes arguments about brace position, white space, etc. moot.

Quality is a people problem

“It is impossible to do a true control test in software development, but I feel the success that we have had with code analysis has been clear enough that I will say plainly it is irresponsible to not use it.” – John Carmack, In-Depth: Static Code Analysis

No tool is going to be a silver bullet. Software quality is and always has been primarily a “people problem”. Tools can help, but they cannot automatically fix all your problems and enforce all your “rules”. They simply provide information that can help people focus on the areas most needing attention, and highlight potential problems that might otherwised have been missed.

Static analysis tools (aka “quality metrics”) can be a useful way to gain more insight into your code and identify areas that need more attention.

(This article was originally published on the REA Tech Blog.)

A Clojure library to authenticate with LDAP

My employer has released a small Clojure library I wrote that allows you to easily authenticate users against an LDAP server:

It uses the UnboundID LDAP SDK for Java to look up a user name in an LDAP server and attempt to bind with specified credentials.

The simplest usage looks like:

(require '[clj-ldap-auth.ldap :as ldap])

(if (ldap/bind? username password)
  (do something-great)

That works, but isn’t very helpful when authentication fails. So you can also pass a function that will be called with a diagnostic message in the event that authentication fails:

(let [reason (atom nil)]
  (if (ldap/bind? username password #(reset! reason %1))
    (do something-great)
    (unauthorised @reason)))

The provided function should take a single argument, which will be a string.

Configuration of the library (i.e. the ldap server to connect to, etc.) is via system properties. See the README for details.


The library first establishes a connection to the server, optionally using SSL. If a bind-dn is configured (i.e. credentials with which to connect to the LDAP server), it is used to bind to the server. If that’s successful, we then look up the provided username (in the attribute uid). If found, the entry’s distinguished name (DN) is extracted and this DN and the provided password are used to bind a new connection.

If any of these steps fail (e.g. the binddn is unauthorised, the username can’t be found, or the looked up DN and password can’t bind) the function returns false (and calls the provided sink function to say why). If everything works and the connection can be bound with the target DN and password, it returns true (and the sink function is not called).


It would probably be useful to be able to specify what attribute(s) to use for looking up the username, but for now it is hard coded to uid. Also, current test coverage (using midje) is minimal. UnboundID provide an in-memory LDAP server implementation, which could probably be used to build some fast-running integration tests.

Do you have anything to declare, sir?

One of the cornerstones of modern software engineering is dependancy management systems. Think Bundler, Leiningen, or (forgive me) Maven. We stand on the shoulders of giants when we write our apps, and we need a way of specifying which giants. Modern systems like RubyGems are pretty good at this. But not perfect.

I have a dream

I have a simple dream. All I want is this: I want to be able to checkout your project on any computer, install the appropriate language runtime and dependency manager and type make (or rake, or lein, or ./, or …) and have a running system.

None of this is new. Joel said it, Twelve Factor App says it. But surprisingly few people seem to actually do it.

Undeclared dependencies are the root of all evil

It’s very easy as a developer to introduce dependencies into your project without even realising. Our workstations get all sorts of cruft installed on them over time, and the chances are something lying around fulfills an undeclared transitive dependency for the library you just installed. But the next developer may not be so lucky.

I don’t want to have to find out by trial and error what native libraries your code depends on. I don’t want to belatedly discover some assumptions you made about what else would be running on my computer. I just want to type make.

So what’s the point?

Just this:

Your project’s build system has the responsibilty to install any library that is required to support your code.

Hopefully most of these will be taken care of by your dependency management system. But for those that aren’t (e.g. native libraries that are required by Ruby Gems) your build system needs to make sure they are installed.

A simple polling function in Clojure

One of my projects at work is to build an internal web service around AWS to support our internal tooling. (This led to the development my clj-aws-ec2 library.)

The web service needs “integration” tests that exercise its RESTful API to manipulate AWS resources (i.e. create instances, add tags, etc.). This sort of testing is fraught for many reasons and should be kept to a minimum, but it does provide a bit of an assurance that the service will actually respond to its published interface when deployed.

One of the reasons this sort of testing is fraught is that it depends on an external service that is beyond our control (i.e. AWS). Many things can go wrong when talking to AWS, and everything takes time. So my test needs to invoke the service to perform an action, then wait until the expected state is achieved (or a timer elapses causing the test to fail). What I’d like to be able to write is something like:

(deftest ^:integration instance-lifecycle
  (testing "create instance"
    (def result (POST "/instances" (with-principal {:name "rea-ec2-tests/int-test-micro", :instance-type "t1.micro"})))
    (has-status result 200)
    (let [id (first (:body result))]
      (prn (str "Created instance " id))
      (testing "get instance"
        (has-status (GET (str "/instances/" id)) 200)
        (is (wait-for-instance-state id "running")))
      (testing "stop instance"
        (has-status (PUT (str "/instances/" id "/stop")) 200)
        (is (wait-for-instance-state id "stopped")))
      (testing "start instance"
        (has-status (PUT (str "/instances/" id "/start")) 200)
        (is (wait-for-instance-state id "running")))
      (testing "delete instance"
        (has-status (DELETE (str "/instances/" id)) 200)
        (is (wait-for-instance-state id "terminated"))))))

But how do you write a polling loop in Clojure? A bit of clicking around on Google led me to a function written by Chas Emerick for his bandalore library:

(defn polling-receive
  [client queue-url & {:keys [period max-wait]
                       :or {period 500
                            max-wait 5000}
                       :as receive-opts}]
  (let [waiting (atom 0)
        receive-opts (mapcat identity receive-opts)
        message-seq (fn message-seq []
                        (if-let [msgs (seq (apply receive client queue-url receive-opts))]
                            (reset! waiting 0)
                            (concat msgs (message-seq)))
                            (when (<= (swap! waiting + period) max-wait)
                              (Thread/sleep period)

That seems pretty close! I generalised it a bit to remove dependencies on Chas’s messaging routines and just take a predicate function:

Finally, a couple of helper functions to tie it all together and enable the tests to be written as above:

(defn get-instance-state [id] (:state (:body (GET (str "/instances/" id)))))
(defn wait-for-instance-state [id state] (wait-for #(= (get-instance-state id) state)))

There’s a couple of improvements that could be made to wait-for, the most obvious being to use a “wall clock” for the timeout. The current implementation will actually wait for timeout + (time-to-evaluate-predicate * number-of-invocations) which is probably not what you want, especially when the predicate could take a non-trivial amount of time to evaluate because it is invoking an external service.

Comments and improvements welcome!

UPDATE: My colleague Eric Entzel pointed out that there is no need to use an atom to store and update the “waiting” counter, its state can just be passed around with function invocations (and recursion). The above gist has been simplified to reflect this observation.

UPDATE: Even better, when I went to implement the “wall clock” timeout, I realised there is no need to maintain any state at all, since the absolute timeout time can be calculated up front and compared to the system clock on each evaluation. (I also flipped the timeout test and the sleep, to more accurately relfect the intent of a timeout.) Gist updated again.

UPDATE: And finally, Adam Fitzpatrick noticed that there’s no longer any need to let bind the poller function to a symbol, we can just put its contents in the main function body. Gist updated again.

New release of clj-aws-ec2

I have released version 0.2.0 of clj-aws-ec2. This version contains no changes from 0.1.11. I’m just trying to adhere more closely to semantic versioning, having been fairly slack about it so far.

This version does however contain many changes since I last [mentioned it here](“/2012/11/introducing-clj-aws-ec2.html”). It can now describe, create and delete tags on resources, and create and deregister images (AMIs).

I consider this more or less “feature complete” for my current purposes. Of course, it only covers a very small fraction of the available EC2 SDK but hopefully it is on the right side of the 8020 rule. :-) I am open to feature requests—or even better pull requests—for further elements of the API that you would like to see supported.

Introducing clj-aws-ec2

We use Amazon’s AWS quite heavily at work, and part of my job involves building internal tools that wrap the public AWS API to provide customised internal services.

I am building some of these tools in Clojure, and I needed a way to call the Amazon API. Amazon provide a Java SDK so it’s a fairly simple matter to wrap this in Clojure. In fact James Reeves had already done so for the S3 API. So I took his good work and adapted it to work with the EC2 components of the API:

The library tries to stay true to Amazon’s official Java SDK, but with an idiomatic Clojure flavour. In particular, it accepts and returns pure Clojure data structures (seqs of maps mostly). For example:

user=> (require '[aws.sdk.ec2 :as ec2])
user=> (def cred {:access-key "..." :secret-key "..."})
user=> (ec2/describe-instances cred (ec2/instance-id-filter "i-b3385c89"))

    ({:id "i-b3385c89",
      :state {:name "running",
              :code 272},
      :type "t1.micro",
      :placement {:availability-zone "ap-southeast-2a",
                  :group-name "",
                  :tenancy "default"}, 
      :tags {:node-name "tockle",
             :name "mrowe/tockle",
             :environment "mrowe"},
      :image "ami-df8611e5",
      :launch-time #<Date Tue Nov 13 08:23:09 EST 2012>}),
  :group-names (),
  :groups ({:id "sg-338f1909", :name "quicklaunch-1"})})

This is still a work in progress. So far, you can describe instances and images, and stop and start EBS-backed instances. I plan to work on adding create/terminate instances next.

UPDATE: I just released v0.1.6 which includes run_instance and terminate_instance support.

Find files in a git project redux

In my previous post I described an update to an Emacs Anything source to “Find files in a git project”. This works great if you are inside an git-managed project, but fails horribly if you are not.

Here is a version that fixes that:

(defvar anything-c-source-git-project-files-cache nil "(path signature cached-buffer)")
(defvar anything-c-source-git-project-files
  '((name . "Files from Current GIT Project")
    (init . (lambda ()
              (let* ((git-top-dir (magit-get-top-dir (if (buffer-file-name)
                                                         (file-name-directory (buffer-file-name))
                     (top-dir (if git-top-dir
                                  (file-truename git-top-dir)
                     (default-directory top-dir)
                     (signature (magit-rev-parse "HEAD")))

                (unless (and anything-c-source-git-project-files-cache
                             (third anything-c-source-git-project-files-cache)
                             (equal (first anything-c-source-git-project-files-cache) top-dir)
                             (equal (second anything-c-source-git-project-files-cache) signature))
                  (if (third anything-c-source-git-project-files-cache)
                      (kill-buffer (third anything-c-source-git-project-files-cache)))
                  (setq anything-c-source-git-project-files-cache
                        (list top-dir
                              (anything-candidate-buffer 'global)))
                  (with-current-buffer (third anything-c-source-git-project-files-cache)
                    (dolist (filename (mapcar (lambda (file) (concat default-directory file))
                                              (magit-git-lines "ls-files")))
                      (insert filename)
                (anything-candidate-buffer (third anything-c-source-git-project-files-cache)))))

    (type . file)

As a diff from the previous version:

@@ -2,9 +2,12 @@
 (defvar anything-c-source-git-project-files
   '((name . "Files from Current GIT Project")
     (init . (lambda ()
-              (let* ((top-dir (file-truename (magit-get-top-dir (if (buffer-file-name)
-                                                                    (file-name-directory (buffer-file-name))
-                                                                  default-directory))))
+              (let* ((git-top-dir (magit-get-top-dir (if (buffer-file-name)
+                                                         (file-name-directory (buffer-file-name))
+                                                       default-directory)))
+                     (top-dir (if git-top-dir
+                                  (file-truename git-top-dir)
+                                default-directory))
                      (default-directory top-dir)
                      (signature (magit-rev-parse "HEAD")))

Emacs Anything - Find files in a git project

If you use Emacs you really should take a look at Anything. When you do, you’ll probably want to use it to replicate TextMate’s fabled “Go to file…”. Ken Wu wrote a nice little anything-source that uses git to derive a file list for a project, but he was obviously using an old version of magit. Here’s a tweaked version of his code that works with Magit v1.1.1:

(defvar anything-c-source-git-project-files-cache nil "(path signature cached-buffer)")
(defvar anything-c-source-git-project-files
  '((name . "Files from Current GIT Project")
    (init . (lambda ()
              (let* ((top-dir (file-truename (magit-get-top-dir (if (buffer-file-name)
                                                                    (file-name-directory (buffer-file-name))
                     (default-directory top-dir)
                     (signature (magit-rev-parse "HEAD")))

                (unless (and anything-c-source-git-project-files-cache
                             (third anything-c-source-git-project-files-cache)
                             (equal (first anything-c-source-git-project-files-cache) top-dir)
                             (equal (second anything-c-source-git-project-files-cache) signature))
                  (if (third anything-c-source-git-project-files-cache)
                      (kill-buffer (third anything-c-source-git-project-files-cache)))
                  (setq anything-c-source-git-project-files-cache
                        (list top-dir
                              (anything-candidate-buffer 'global)))
                  (with-current-buffer (third anything-c-source-git-project-files-cache)
                    (dolist (filename (mapcar (lambda (file) (concat default-directory file))
                                              (magit-git-lines "ls-files")))
                      (insert filename)
                (anything-candidate-buffer (third anything-c-source-git-project-files-cache)))))

    (type . file)

I tried to update the Emacs Wiki page to include this fix but couldn’t. Not sure what I was doing wrong… The changes I made to Ken’s code:

@@ -6,7 +6,7 @@
                                                                     (file-name-directory (buffer-file-name))
                      (default-directory top-dir)
-                     (signature (magit-shell (magit-format-git-command "rev-parse --verify HEAD" nil))))
+                     (signature (magit-rev-parse "HEAD")))
                 (unless (and anything-c-source-git-project-files-cache
                              (third anything-c-source-git-project-files-cache)
@@ -20,10 +20,14 @@
                               (anything-candidate-buffer 'global)))
                   (with-current-buffer (third anything-c-source-git-project-files-cache)
                     (dolist (filename (mapcar (lambda (file) (concat default-directory file))
-                                              (magit-shell-lines (magit-format-git-command "ls-files" nil))))
+                                              (magit-git-lines "ls-files")))
                       (insert filename)
                 (anything-candidate-buffer (third anything-c-source-git-project-files-cache)))))

Chef doesn't lock node data when updating

We came across an exciting Chef bug today.

Chef tracks metadata about nodes in its database. This includes operational facts about the node (uptime, memory, etc.), and chef-related things like when the node last checked in. It also includes intentional data such as what run list should be applied to the node.

Periodically, a node polls its server for updates. What happens is:

  • node checks in with server

  • node gets current metadata from server, including its run list of recipes and roles

  • node performs actions as per the run list

  • node saves its metadata back to the server, including the run list it just applied

All well and good, except that step three can potentially be long running. There’s plenty of time for an administrator to change the node’s desired run list (or other intentional metadata) using the knife tool or the web interface. But now, when the node’s run completes, it saves its old state back to the server, over-writing whatever updates an administrator applied while it was running. And you won’t know unless you look.

This is unfortunate.

There’s a bug that more or less describes this in the project’s tracker. It was raised quite recently, so hopefully someone from the Chef team will take a look at it soon. There’s also a thread on the Chef mailing list.

Jekyll archives grouped by date

One thing Jekyll doesn’t provide out of the box (as fas I can tell) is any sort of archive functionality. (Aside: I really like what Tumblr does for archives.)

I would have liked something a bit more flexible, but for now this site’s archive displays a list of all entries grouped by year. Here’s the template code I’m using:

  {{'{'}}% for post in site.posts >}}

    {{'{'}}% unless >}}
      <h3>{{'{'}}{ | date: '%Y' }}</h3>
    {{'{'}}% else >}}
      {{'{'}}% capture year >}}{{'{'}}{{'{'}} | date: '%Y' }}{{'{'}}% endcapture >}}
      {{'{'}}% capture nyear >}}{{'{'}}{{'{'}} | date: '%Y' }}{{'{'}}% endcapture >}}
      {{'{'}}% if year != nyear >}}
        <h3>{{'{'}}{{'{'}} | date: '%Y' }}</h3>
      {{'{'}}% endif >}}
    {{'{'}}% endunless >}}

    <li>{{'{'}}{{'{'}} | date:"%b" }} <a href="{{'{'}}{{'{'}} post.url }}">{{'{'}}{{'{'}} post.title }}</a></li>
  {{'{'}}% endfor >}}

which was shamelessly ripped off from