Something's Rotten in the State of Jenkins
Automated, repeatable testing is a fairly widely-accepted cornerstone of mature software development. Jenkins (and its predecessor, Hudson) has the unique privilege of being both an early player in the niche and free-as-in-beer. The blog space is littered with interesting articles about continuous builds, automated testing, and continuous deployment, all of which conclude on “how do we make Jenkins do it?”
This is unfortunate, because Jenkins has some serious problems, and I want it to stop informing the discussion.
There's A Plugin For That
Almost everything in the following can be addressed using one or more plugins from Jenkins' extensive plugin repository. That's good - a build system you can't extend is kind of screwed - but it also means that the Jenkins team haven't felt a lot of pressure to address key problems in Jenkins proper.
(Plus, the plugin ecosystem is its own kind of screwed. More on that later.)
To be clear: being able to fix it with plugins does not make Jenkins itself good. Plugins are a non-response to fundamental problems with Jenkins.
Jenkins builds are atomic: they either pass en suite, or fail en suite. Jenkins has no built-in support for recording that basic compilation succeeded, unit tests failed, but linting also succeeded.
You can fix this by running more builds, but then you run into problems with ...
... the inability to wait for multiple upstream jobs before continuing a downstream job in a job chain. If your notional build pipeline is
- Compile, then
- Lint and unit test, then
- Publish binaries for testers/users
then you need to combine the lint and unit test steps into a single build, or tolerate occasionally publishing between zero and two copies of the same original source tree.
The above are actually symptomatic of a more fundamental design problem in Jenkins: there's no build pipeline. Jenkins is a task runner: triggers cause tasks to run, which can cause further triggers. (Without plugins, Jenkins can't even ensure that chains of jobs all build the same revisioins from source control.)
I haven't met many projects whose build process was so simple you could treat it as a single, pass-fail task, whose results are only interesting if the whole thing succeeds.
Plugin the Gap
To build a functional, non-trivial build process on top of Jenkins, you will inevitably need plugins: plugins for source control, plugins for notification, plugins for managing build steps, plugins for managing various language runtimes, you name it.
The plugin ecosystem is run on an entirely volunteer basis, and anyone can get a new plugin into the official plugin registry. This is good, in as much as the barrier to entry should be low and people should be encouraged to scratch itches, but it also means that the plugin registry is a swamp of sporadically-maintained one-offs with inconsistent interfaces.
(Worse, even some core plugins have serious maintenance deficits: have a look at how long JENKINS-20767 was open. How many Jenkins users use Git?)
The Plugin API
The plugin API also, critically, locks Jenkins into some internal design
problems. The sheer number of plugins, and the sheer number of maintainers,
effectively prevents any major refactoring of Jenkins from making progress.
Breaking poorly-maintained plugins inevitably pisses off the users who were,
quite happily, using whatever they'd cooked up, but with the maintainership
of plugins so spread out and so sporadic, there's no easy way for the Jenkins
team to, for example, break up the 4,000-line
What Is To Be Done
Jenkins is great and I'm glad it exists. Jenkins moved the state of the art for build servers forward very effectively, and successfully out-competed more carefully-designed offerings that were not, in fact, better: Continuum is more or less abandoned, and when was the last time you saw a CruiseControl (caution: SourceForge) install?
It's interesting to compare the state of usability in, eg., Jenkins, to the state of usability in some paid-product build systems (Bamboo and TeamCity for example) on the above points, as well as looking at the growing number of hosted build systems (TravisCI, MagnumCI) for ideas. A number of folks have also written insightful musings on what they want to see in the next CI tool: Susan Potter's Carson includes an interesting motivating metaphor (if you're going to use butlers, why not use the whole butler mileu?) and some good observations on how Jenkins lets us all down, for example.
I think it's time to put Jenkins to bed and write its successor.