PathTo for Python gets a JSON-capable HTTP client

February 18th, 2010

…and it works!

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')
>>> app.login.post(credentials, expected_status=302)
>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}
>>> product = app.products['Foo'].get(expected_status=200).parsed
>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)
>>> app.products['Foo'].get(expected_status=200).parsed['description']
u'Updated!'

OK, so it wasn’t really “example.com”, and the updated product wasn’t called “Foo”, but the rest is for real.

In stages:

0) The Pylons-based server has the JSON-capable @validate (un)decorator of my previous post and DescribedRoutesMiddleware from DescribedRoutes installed in its wsgi stack.

1) Create a client-side proxy to the app. Following link headers published by the app, it finds the ResourceTemplates description, which it retrieves in JSON format.

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')

The format='json' is a small red herring here, it’s simply remembered for later.

2) Log in by posting credentials, expecting a 302 status (a failed attempt would return a 200 and the validation errors in the body). The client handles cookies automatically behind the scenes.

>>> app.login.post(credentials, expected_status=302)

3) View a friendly representation of the metadata (or rather the part that relates to products):

>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}

4) Get a product identified by ‘Foo’ and return the JSON payload parsed into a Python dict.:

>>> product = app.products['Foo'].get(expected_status=200).parsed

Behind the scenes it has expanded the “http://example.com/products/{product}{.format}” template seen previously into “http://example.com/products/Foo.json”, using the remembered format parameter and the supplied ‘Foo’ key which is assumed to correspond to the required product parameter.

5) Update the local product representation and send it back (it gets converted back to JSON along the way):

>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)

6) Finally, demonstrate that it was successful!

>>> app.products['Foo'].get(expected_status=200).parsed['description']
u'Updated!'

Experimental: Pylons validation (un)decorator with JSON support

February 13th, 2010

In this gist (it’s not even a patch, just some stuff I keep in my app’s lib/base.by) is a refactored @validate.

On the surface it’s just the same as the standard pylons.decorators.validate:

    @validate(form='edit')
    def update(self):
        ...

but it has been broken up into a number of controller methods, each of which is usable directly. So if you find the decorator inflexible, you can write your actions in the undecorated form

    def update(self):
        try:
            self._parse(request, schema=MyForm())
        except Invalid as e:
            return self._render_invalid(e, form='edit')
        ...

It’s more than just a stylistic change; in addition to the usual HTML form data it will also accept POST data sent in JSON and send back any validation errors in JSON too if the client so desires. The presence of JSON is recognized by an sent_json() helper that looks to see if request’s route had a format parameter set to ‘json’ (see previous post) or if the Content-Type header is ‘application/json’; responses work similarly via an accepts_json() helper, but checking the Accept header.

Throw in a render_json() helper and here are idioms for actions that will quite happily accept and produce JSON or HTML. First the traditional, decorated look:

    @validate(form='edit')
    def update(self):
        ...
        if accepts_json():
            return render_json(a_json_serialisable_thing)
        else:
            return render('template')

And undecorated:

    def update(self):
        try:
            self._parse(request, schema=MyForm())
            ...
        except Invalid as e:
            return self._render_invalid(e, form='edit')
        ...
        if accepts_json():
            return render_json(a_json_serialisable_thing)
        else:
            return render('template')

The undecorated form is slightly longer — potentially addressed by sharing exception handling controller-wide — but it offers the intriguing opportunity to move some validation to the model &emdash; any Invalid exceptions raised in the try block will be rendered appropriately, regardless of where they came from.

Experimental {.format} in Routes

February 10th, 2010

A while back I raised on the URI Template mailing list the idea of a {.var} syntax that generates optional path extensions, “.html”, “.json”, etc.

For example, given a path template of /releases/{id}/notes{.format} and an id parameter of “1″, paths of /releases/1/notes or /releases/1/notes.pdf will be generated depending on whether the supplied format parameters are undefined or “pdf” respectively.

It’s not a completely original idea.  Rails has a (.:format) syntax that DescribedRoutes maps to the flexible but somewhat clunky {-prefix|.|format} syntax of the old draft URI Template spec. The proposed syntax seems more in keeping with current draft (very much a work in progress) though.

Today my Routes fork gained support for the syntax and tweaks to the new SubMapper helpers so that duplicate “formatted routes” aren’t generated (something that happened to Rails a long time ago).

Example config (<code>config/routing.py</code> in your Pylons app):

with mapper.collection(
                'releases',
                'release',
                requirements={'id': 'd+'}) as c:
    c.member.link(rel='notes', name='release_notes')

And in the paster shell:

>>> print mapper
Route name     Methods Path
releases       GET     /releases{.format}
create_release POST    /releases{.format}
new_release    GET     /releases/new{.format}
release        GET     /releases/{id}{.format}
update_release PUT     /releases/{id}{.format}
delete_release DELETE  /releases/{id}{.format}
edit_release   GET     /releases/{id}/edit{.format}
release_notes  GET     /releases/{id}/notes{.format}
>>> url('release_notes', id=1, format='pdf')
'/releases/1/notes.pdf'
>>> url('release_notes', id=1)
'/releases/1/notes'
>>> mapper.match('/releases/1/notes')
{'action': u'notes', 'controller': u'release', 'id': u'1', 'format': None}
>>> mapper.match('/releases/1/notes.pdf')
{'action': u'notes', 'controller': u'release', 'id': u'1', 'format': u'pdf'}

Just add formatted=False to your collection() call if you don’t want this new behaviour.

I haven’t sent the Routes guys a pull request yet. I’d like some feedback first, and meanwhile I will try to come up with a nice helper that takes the format extension to override content negotiation. My app has some basic header-based conneg already but I would like the convenience of being able to override it in the URL.

Small steps along my personal Python roadmap

January 20th, 2010

Three new things tried in recent days:

  1. Packaging a Python module
  2. Using doctest in anger (tweaking doctests in Routes doesn’t really count)
  3. Publishing a package to PyPi

For #1, I used setuptools, kick-started by Ian Bicking’s excellent presentation (the official docs are good too).  Completely painless, and it pretty much covers #3 too.  All in all I would say that it was easier than building and publishing a Rubygem.

Re #2, doctest isn’t going to replace regular unit tests for me, but for describing a public API I think it’s excellent.  In the case of LinkHeader, exercising the public API doesn’t leave much else - it’s a very simple package.  Here’s a small excerpt:

>>> parse('<http://example.com/foo>; rel="foo bar", <http://example.com>; rel=up; type=text/html')
LinkHeader([Link('http://example.com/foo', rel='foo bar'), Link('http://example.com', rel='up', type='text/html')])

which demonstrates how to parse a link header into something a client can easily interrogate.

LinkHeader will be used by DescribedRoutes to add discoverability to web apps and by PathTo clients in order to discover and interact with application resources.  All of this works in Ruby already of course; my current personal project is to replicate it in Python with support for the Pylons web framework.  The Link Header standard is progressing meanwhile, with a new version published only this week.

Fun with parameter-collecting proxies

January 1st, 2010

Further inspired by the Routes submapper idea (see DRY up your routes) here’s a generic proxy object that collects arguments repeated across method calls:

class Params(object):
    def __init__(self, obj, *args, **kwargs):
        self.__obj = obj
        self.__args = args
        self.__kwargs = kwargs

    def __getattr__(self, name):
        try:
            attr = getattr(self.__obj, name)
            if callable(attr):
                # Target attr is callable; return another that
                # will sneak in our remembered args
                def wrapped_method(*args, **kwargs):
                    return attr(*(self.__args + args),
                                **dict(self.__kwargs, **kwargs))
                return wrapped_method
            else:
                # Plain old attribute
                return attr
        except AttributeError:
            # self.foo(bar, *args, **kwargs)
            # -> scope(self, foo=bar, *args, **kwargs)
            def wrapped_scope(*args, **kwargs):
                return Params(self, *args[1:],
                              **dict({name: args[0]}, **kwargs))
            return wrapped_scope

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        pass

params = Params

You can use it submapper-style (with or without the ‘with‘ syntax), collecting arbitrary args and kwargs as follows:

with params(mapper, controller='entries') as entries:
    entries.connect(...)
    entries.connect(...)
    ...

Here controller='entries' gets included in each method call to the underlying mapper.

Equivalently, it supports a “fluent” or DSL-ish style where named parameters are transfomed into (chainable) methods:

m = params(mapper)
with m.controller('entries') as entries:
    entries.connect(...)
    entries.connect(...)
    ...

Of course the Routes API (in the dev version at least) already has submappers and param makes a poor substititute, but as tool for reducing duplication elsewhere it may yet prove useful.

DRY up your routes - UPDATES

December 28th, 2009

Two updates to the previous post:

Update 1: Routes fork on bitbucket

See http://bitbucket.org/asplake/routes/. I’ve sent a pull request with my changes (the prettyprinter included), but no response yet and I’m delighted to say that Ben has now incorporated them in the base Routes repo so expect to see them soon in Routes 1.12.

Update 2: Tidier collection() usage

The end-state of the last post

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

can be written as

    with mapper.collection(
                    'releases',
                    'release',
                    requirements={'id': 'd+'}) as c:
        c.member.link(rel='notes', name='release_notes')

Losing the with syntax, a vanilla collection resource with integer id’s looks like this:

    mapper.collection(
                'releases',
                'release',
                requirements={'id': 'd+'})

This applies requirements to the collection-level routes as well as the member-level routes but the overhead (unquantified) must be very small and I prefer this tidier form.  It would be easy enough though to provide a member_requirements argument if the performance difference proves to be significant.

DRY up your routes - a Pylons routing refactoring

December 21st, 2009

[See UPDATES]

This post is in several acts, each one a refactoring. Taking the stage will be some familiar-looking application code; behind the scenes lurks some enhanced framework code that makes the refactorings possible.

Our story starts with an old-school (pre-REST) routing configuration. It’s in Python, for Pylons and other Routes-based web frameworks, but let me reassure the Ruby audience that Routes borrows very heavily from Rails and their routing configurations look quite similar.

We will finish with collection(), an experimental (but I hope worthy) alternative to resource(), the routing helper that rose to prominence when Rails first made its big push to REST.

The starting point: textbook routing.py

from routes import Mapper

def make_map():

    ...

    # Releases - Collection
    mapper.connect('releases', '/releases',
              controller='release', action='index', conditions=dict(method='GET'))
    mapper.connect('create_release', '/releases',
              controller='release', action='create', conditions=dict(method='POST'))
    mapper.connect('new_release', '/releases/new',
              controller='release', action='new', conditions=dict(method='GET'))

    # Releases - Members
    mapper.connect('release', '/releases/{id}',
              controller='release', action='show',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('update_release', '/releases/{id}',
              controller='release', action='update',
              requirements=dict(id='d+'), conditions=dict(method='PUT'))
    mapper.connect('delete_release', '/releases/{id}',
              controller='release', action='delete',
              requirements=dict(id='d+'), conditions=dict(method='DELETE'))
    mapper.connect('edit_release', '/releases/{id}/edit',
              controller='release', action='edit',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('release_notes', '/releases/{id}/notes',
              controller='release', action='release_notes',
              requirements=dict(id='d+'), conditions=dict(method='GET'))

Full of duplication, verbose, even ugly. Just because it sits in a config directory, does it have to look that bad?

Refactoring 1: Introducing submapper()

New in Routes 1.11 (so really quite new), submapper()provides the means to pull out parameters previously shared across multiple connect() calls.

One particularly nice feature is that the SubMapper objects returned by submapper() support the Python managed object protocol so if you’re on Python 2.5 or above you can use them with the with syntax like this:

from release_tool.lib.mapper import Mapper

def make_map():

    ...

    # Releases - Collection
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))

    # Releases - Members
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases/{id}',
                    requirements=dict(id='d+')) as m:
        m.connect('release', '', action='show',
                conditions=dict(method='GET'))
        m.connect('update_release', '', action='update',
                conditions=dict(method='PUT'))
        m.connect('delete_release', '', action='delete',
                conditions=dict(method='DELETE'))
        m.connect('edit_release', '/edit', action='edit',
                conditions=dict(method='GET'))
        m.connect('release_notes', '/notes', action='release_notes',
                conditions=dict(method='GET'))

That’s a definite improvement (and credit where credit is due - submappers are great innovation, though a small bug requires our local extensions to be imported here), but notice that there’s still some duplication between the two submappers blocks, the first one corresponding to the collection resource, the second to that same collection’s members. Wouldn’t it be cool if we could nest them?

Refactoring 2: Submapper nesting

On the surface, the Routes 1.11 API doesn’t appear to support submapper nesting, but the SubMapper objects do indeed nest. Adding a submapper() method to SubMapper is a trivial change, and it take only minor internal tweaks for deeper nestings to function correctly.

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.connect('release', '', action='show',
                    conditions=dict(method='GET'))
            m.connect('update_release', '', action='update',
                    conditions=dict(method='PUT'))
            m.connect('delete_release', '', action='delete',
                    conditions=dict(method='DELETE'))
            m.connect('edit_release', '/edit', action='edit',
                    conditions=dict(method='GET'))
            m.connect('release_notes', '/notes', action='release_notes',
                    conditions=dict(method='GET'))

We seem to be on to something here, but what about the repetition (in one guise or another) of the resource name “release”? And what’s with those strange empty string ('') parameters?

Refactoring 3: Links and actions

To fix the repetition issue it’s clear that we need helpers that will generate those connect() calls for us more intelligently. But before we do that, let’s recognise that there are really two kinds of routes being generated here:

  1. Links to singleton subresources that support only one HTTP method, the most ubiquitous examples being the new and edit representations for which GET is the applicable HTTP method, but special subresources that are the targets for POST actions are common too;
  2. Actions that correspond directly to HTTP methods: index and create for GET and POST[1] on collection resources, show, update, delete for GET, PUT, DELETE on member resources.

It’s that second type that give rise to those strange empty strings as there is no further navigation to be done relative to the target resource.

So let’s add link() and action() helpers:

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.action(action='index', name='releases')
        c.action(action='create', method='POST')
        c.link('new')
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.action(action='show', name='release')
            m.action(action='update', method='PUT')
            m.action(action='delete', method='DELETE')
            m.link(rel='edit')
            m.link(rel='notes', name='release_notes')

Things are still moving in the right direction, but given that most of those links and actions follow a very well-worn convention, how about specific helpers for those?

Refactoring 4: More helpers

I’ll be honest - this is just an intermediate stage. You can see where we’re headed now though:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.index()
        c.create()
        c.new()
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.show()
            m.update()
            m.delete()
            m.edit()
            m.link(rel='notes', name='release_notes')

Refactoring 5: Parameter-driven generation

We’ll be using those same names a lot across our resources, so let’s just identify the ones we want in a list:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases',
                    actions=['index', 'create', 'new']) as c:
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+'),
                    actions=['show', 'update', 'delete', 'edit']) as m:
            m.link(rel='notes', name='release_notes')

But even this pattern will be repeated across resources, so let’s define a collection() helper that does it all:

Refactoring 6: Using the collection() helper

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

And we’re done.

Let’s compare this to the equivalent resource() call:

    # Releases
    mapper.resource(
                'release',
                'releases',
                controller='release',
                # no requirements!
                member = {'notes':'GET'})

Superficially similar, but I’m unable to specify requirements on member paths and I don’t get to name my custom route either - the generated name “notes_release” just isn’t right! The bottom line is that resource() is approaching a dead end; its only way forward is to make its parameters ever more complex.

So… how about we put resource() on notice and bring in something more flexible and - dare I say - Pythonic? And wouldn’t it look just as nice in Ruby too? Answers on a postcard please…

Prettyprinter for Pylons routes

December 15th, 2009

No, not described_routes, just something basic to plug a gap. For those suffering Rails-envy there’s no “paster routes” command, but it’s easy enough to use in the Paster shell:

bash-3.2$ paster shell test.ini
Pylons Interactive Shell
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)]

  All objects from encore.lib.base are available
  Additional Objects:
  mapper     -  Routes mapper object
  wsgiapp    -  This project's WSGI App instance
  app        -  paste.fixture wrapped around wsgiapp

>>> print mapper
Route name   Methods Path
                     /error/{action}
                     /error/{action}/{id}
home         GET     /
things       GET     /things
create_thing POST    /things
new_thing    GET     /things/new
thing        GET     /thing/{id}
...

It works by subclassing routes.Mapper (I have other reasons to do this which I’ll explain soon) and you’ll need to use this new Mapper in your config/routing.py.

That should be all you need to know. The code for the new Mapper class is below. Note: That second from last join() isn’t coming out very well - it has a backslash and an ‘n’ in it - honest!

Enjoy!

import routes.mapper

class Mapper(routes.Mapper):
    #
    # Pretty string representation - returns a string formatted like this:
    #
    #    Route name   Methods Path
    #                         /error/{action}
    #                         /error/{action}/{id}
    #    home         GET     /
    #    things       GET     /things
    #    create_thing POST    /things
    #    new_thing    GET     /things/new
    #    thing                  GET     /thing/{id}
    #
    #    etc
    #
    # Enter 'print mapper' in the paster shell to use it, or (TBD) run it via a
    # configured paster command.
    #
    def __str__(self):
        def format_methods(r):
            if r.conditions:
                method = r.conditions.get('method', '')
                return method if type(method) is str else ', '.join(method)
            else:
                return ''

        table = [('Route name', 'Methods', 'Path')] + [
            (
                r.name or '',
                format_methods(r),
                r.routepath or''
            )
            for r in self.matchlist]

        widths = [
            max(len(row[col]) for row in table)
            for col in range(len(table[0]))]

        return '
'.join(
            ' '.join(
                    row[col].ljust(widths[col])
                    for col in range(len(widths))
                ).rstrip()
            for row in table)

Nested forms in Pylons

November 5th, 2009

Easy when you know how, but finding that out was struggle - at least it was for me! The crucial piece of information I was lacking was the correspondence between variabledecode.variable_encode() which flattens nested values into an array and variabledecode.NestedVariables which does the reverse.

I have factored these lines below out to my lib/base.py. Now all my form schemas (nested or not) inherit from BaseSchema and I use fill_render() whenever I want to render html with values.

import formencode
from formencode import htmlfill
from formencode import variabledecode

def fill_render(template_name, values):
    """
    Render a template filled with values.  Values may be nested.
    """
    return htmlfill.render(
                    render(template_name),
                    variabledecode.variable_encode(values))

class BaseSchema(formencode.Schema):
    """
    Base form schema.  Any nested forms will be converted to nested values automatically.
    """
    allow_extra_fields = True
    filter_extra_fields = True
    pre_validators = [variabledecode.NestedVariables()]

It leads to DRYer controller code, reduces the number of imports required and hides some complexity. Can’t understand why it isn’t the default already in Pylons, but hey, it works.

To see nested forms in action, go to the interesting but (at the time of writing) not quite complete Solving the Repeating Fields Problem (Pylons Book v1.1 documentation).

Credit to Ian Wilson for pointing me in the right direction in this thread on pylons-discuss.

Principles of Product Development Flow - review & retrospective

October 6th, 2009

Introduction

I have been encouraged by Eric Willeke and Alan Shalloway via the kanbandev group to share my thoughts on Donald Reinertsen’s new book Principles of Product Development Flow and I promised a review that would also serve as something of a personal retrospective. As book reviews go the result is probably a mess, but you can read clean & tidy book reviews elsewhere!

Why the retrospective? As past readers of my blog will know, I’ve recently undergone some significant change of late. I left behind a 12 year career in the City, took a very welcome 3 month break from work, and at the end of that time made something of a lifestyle change by moving well away from London to live in a cottage in the picturesque Derbyshire Dales. At the time we committed ourselves to the move I didn’t know exactly would come next but there was sure to be a Lean flavour to it, and during my time out I quickly fell in with the Kanban crowd. When I landed my new role it was clear that Kanban would feature prominently from day 1; two months have since passed…

Back to the book: it’s not about Kanban (though it is mentioned). Neither is it a book about software development (it’s much more general than that, though the specifics of software development occur frequently throughout the book). It is however a book that has shone light on both subjects for me – not as another methodology book, but by strengthening significantly my understanding of how development processes work in general, helping me to really understand why I’ve seen what I’ve seen in past projects (good and bad, mine and others’), and by helping me to crystallise how I can deal in an economically justifiable way with things that appear on the surface to be completely intangible.

To be more specific with that last thought, what excited me most about the book was the way it ties the management of queues – covered very well in their own right – to economics. This happens early in the book, changes (or at least changed for me) the way the rest of the book is read, and hugely increases its value. It may sound academic, but I found it motivating (if occasionally uncomfortable) that I could link so directly the way I run projects past, present and future to business performance. And making that link was straightforward; it’s a book that is free of hand-waving and its examples ring true to my own experience.

Diving in

Let’s get down to details with some quotes:

The issue is not queues, it is the economics of queues (p17)

We can define waste as a failure to optimize [globally] our economics (p33)

Optimum queue size is an economic trade-off (p68)

Focus on the wrong queue can actually create push rather than pull. Why (speaking in retrospect with embarrassment at my own stupidity, not to criticise the jargon of certain branded methods) emphasise the development backlog when you haven’t achieved flow downstream? Are all of your queues even visible (always, to all, on the wall)? Do you have any idea – even roughly – of your cost of delay (“COD”) and consciously optimise (portfolio-wide) to minimise it? I’m beginning to get a handle on my COD, and the numbers are staggering, especially when I’m tempted to apply a P/E multiple to the answer!

But other queues may not matter so much. On my Kanban wall I have a “Proposed” column preceding “Prioritised” and it’s my job to move items between the two. I don’t lie awake feeling overwhelmed by the large number of proposed items – in fact quite the reverse: it feels good to deny attention to work (even whole projects) that would only add delay to work actually in progress. Is the movement between the two columns push or pull in nature? Probably a bit of both, but it’s one part of the process that won’t get reduced to a system of rules – the sequencing and regulation of work going into the process is far too important for that, and I have the book to thank for giving me a fresh perspective on my personal responsibilities toward both the process and the end result.

Use CFDs to monitor queues (p71)

Cumulative Flow Diagrams (CFDs) can be done independently of Kanban but they make most sense done together - just log each day the number of work items held in each column and chart cumulatively over time - easy! What does my chart reveal? I work a pattern of alternate weeks, one week with the team in Budapest and one at home and this is reflected very clearly in the chart. Most re-planning happens when I’m in Budapest (near both team and customer), leading quite naturally to a 2-week cadence for the whole team. The CFD does hint at the cost of being apart and we work hard collectively to mitigate it, but my data collection gets a little patchy when I’m away so I haven’t properly quantified it yet. I check the CFD regularly to measure lead times, to look for any signs of growing bottlenecks that aren’t already glaing at me from the wall, and I plan to use it as a presentation tool in future management meetings.

It’s nice to discover that one can monitor lead times so easily and I’m prepared even to read some predictive power into the chart without ever requiring development estimates. In fact estimates have become yet another thing to be added the ever-growing list of supposedly indispensible things that I’m simply not missing. And let’s face it: so often they’re rubbish anyway!

Create [decentralized] systems to harvest the early cheap opportunities (p40)

Understanding decentralisation and delegation is especially important when running distributed development. Here’s a technical example (though the point isn’t the technique): I don’t (can’t!) review every line of code that gets committed but it is easily decentralized. Not only can developers review each other’s code, but my loathing of duplication in code has been communicated strongly, along with a counterbalancing worry about breaking working code without economic justification. As a result, the kinds of refactorings that lead to a smaller codebase and consequently to a smaller maintenance headache happen regularly without my direct involvement, and (thanks also to investments in unit tests and a CI system) seemingly without too much pain. But is this really a “cheap opportunity”? Yes, one has to be concerned about overdoing the up-front investment, but in our case it has already made for a more comfortable and productive environment and I’m happy that the pay-back period is short enough to make this a no-brainer.

Onwards and upwards

All of this has been drawn from or prompted by just the first three chapters, there being nine in total. Reinertsen goes on to discuss variability, batch sizes, WIP constraints, controllability and feedback before taking an interesting turn with a chapter on decentralisation, drawing heavily from military theory. All of it is excellent, some of it very quotable, some of it very valuable reference material (and no less readable for that). I make no apologies though for emphasising the core concept Don surely intends to keep at the fore as we continue with the rest of the book.

Nits to pick? Only one springs to mind, and it’s not a problem, just a minor missed opportunity. It follows from the book’s analysis of the economics of the timing of decisions that an un-made decision has real economic value, but this could have been made explicit. A link to Real Options Theory would have been very nice here, and it might have informed some of the later parts of the book. I have a vague recollection that Real Options do get mentioned, but I couldn’t find it in the index.

I have two concluding remarks to make. The first is about Kanban, and it is to restate a comment I made on one of the mailing lists: implementing Kanban was undoubtedly one of my better professional decisions in a 20-something year career as developer, project and program manager and now the senior IT guy in my new organisation. The second is about the book: quite simply it’s brilliant; buy it!