Chef doesn't lock node data when updating

We came across an exciting Chef bug today.

Chef tracks metadata about nodes in its database. This includes operational facts about the node (uptime, memory, etc.), and chef-related things like when the node last checked in. It also includes intentional data such as what run list should be applied to the node.

Periodically, a node polls its server for updates. What happens is:

• node checks in with server

• node gets current metadata from server, including its run list of recipes and roles

• node performs actions as per the run list

• node saves its metadata back to the server, including the run list it just applied

All well and good, except that step three can potentially be long running. There’s plenty of time for an administrator to change the node’s desired run list (or other intentional metadata) using the knife tool or the web interface. But now, when the node’s run completes, it saves its old state back to the server, over-writing whatever updates an administrator applied while it was running. And you won’t know unless you look.

This is unfortunate.

There’s a bug that more or less describes this in the project’s tracker. It was raised quite recently, so hopefully someone from the Chef team will take a look at it soon. There’s also a thread on the Chef mailing list.