Small steps along my personal Python roadmap

January 20th, 2010

Three new things tried in recent days:

  1. Packaging a Python module
  2. Using doctest in anger (tweaking doctests in Routes doesn’t really count)
  3. Publishing a package to PyPi

For #1, I used setuptools, kick-started by Ian Bicking’s excellent presentation (the official docs are good too).  Completely painless, and it pretty much covers #3 too.  All in all I would say that it was easier than building and publishing a Rubygem.

Re #2, doctest isn’t going to replace regular unit tests for me, but for describing a public API I think it’s excellent.  In the case of LinkHeader, exercising the public API doesn’t leave much else - it’s a very simple package.  Here’s a small excerpt:

>>> parse('<http://example.com/foo>; rel="foo bar", <http://example.com>; rel=up; type=text/html')
LinkHeader([Link('http://example.com/foo', rel='foo bar'), Link('http://example.com', rel='up', type='text/html')])

which demonstrates how to parse a link header into something a client can easily interrogate.

LinkHeader will be used by DescribedRoutes to add discoverability to web apps and by PathTo clients in order to discover and interact with application resources.  All of this works in Ruby already of course; my current personal project is to replicate it in Python with support for the Pylons web framework.  The Link Header standard is progressing meanwhile, with a new version published only this week.

Fun with parameter-collecting proxies

January 1st, 2010

Further inspired by the Routes submapper idea (see DRY up your routes) here’s a generic proxy object that collects arguments repeated across method calls:

class Params(object):
    def __init__(self, obj, *args, **kwargs):
        self.__obj = obj
        self.__args = args
        self.__kwargs = kwargs

    def __getattr__(self, name):
        try:
            attr = getattr(self.__obj, name)
            if callable(attr):
                # Target attr is callable; return another that
                # will sneak in our remembered args
                def wrapped_method(*args, **kwargs):
                    return attr(*(self.__args + args),
                                **dict(self.__kwargs, **kwargs))
                return wrapped_method
            else:
                # Plain old attribute
                return attr
        except AttributeError:
            # self.foo(bar, *args, **kwargs)
            # -> scope(self, foo=bar, *args, **kwargs)
            def wrapped_scope(*args, **kwargs):
                return Params(self, *args[1:],
                              **dict({name: args[0]}, **kwargs))
            return wrapped_scope

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        pass

params = Params

You can use it submapper-style (with or without the ‘with‘ syntax), collecting arbitrary args and kwargs as follows:

with params(mapper, controller='entries') as entries:
    entries.connect(...)
    entries.connect(...)
    ...

Here controller='entries' gets included in each method call to the underlying mapper.

Equivalently, it supports a “fluent” or DSL-ish style where named parameters are transfomed into (chainable) methods:

m = params(mapper)
with m.controller('entries') as entries:
    entries.connect(...)
    entries.connect(...)
    ...

Of course the Routes API (in the dev version at least) already has submappers and param makes a poor substititute, but as tool for reducing duplication elsewhere it may yet prove useful.

DRY up your routes - UPDATES

December 28th, 2009

Two updates to the previous post:

Update 1: Routes fork on bitbucket

See http://bitbucket.org/asplake/routes/. I’ve sent a pull request with my changes (the prettyprinter included), but no response yet and I’m delighted to say that Ben has now incorporated them in the base Routes repo so expect to see them soon in Routes 1.12.

Update 2: Tidier collection() usage

The end-state of the last post

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

can be written as

    with mapper.collection(
                    'releases',
                    'release',
                    requirements={'id': 'd+'}) as c:
        c.member.link(rel='notes', name='release_notes')

Losing the with syntax, a vanilla collection resource with integer id’s looks like this:

    mapper.collection(
                'releases',
                'release',
                requirements={'id': 'd+'})

This applies requirements to the collection-level routes as well as the member-level routes but the overhead (unquantified) must be very small and I prefer this tidier form.  It would be easy enough though to provide a member_requirements argument if the performance difference proves to be significant.

DRY up your routes - a Pylons routing refactoring

December 21st, 2009

[See UPDATES]

This post is in several acts, each one a refactoring. Taking the stage will be some familiar-looking application code; behind the scenes lurks some enhanced framework code that makes the refactorings possible.

Our story starts with an old-school (pre-REST) routing configuration. It’s in Python, for Pylons and other Routes-based web frameworks, but let me reassure the Ruby audience that Routes borrows very heavily from Rails and their routing configurations look quite similar.

We will finish with collection(), an experimental (but I hope worthy) alternative to resource(), the routing helper that rose to prominence when Rails first made its big push to REST.

The starting point: textbook routing.py

from routes import Mapper

def make_map():

    ...

    # Releases - Collection
    mapper.connect('releases', '/releases',
              controller='release', action='index', conditions=dict(method='GET'))
    mapper.connect('create_release', '/releases',
              controller='release', action='create', conditions=dict(method='POST'))
    mapper.connect('new_release', '/releases/new',
              controller='release', action='new', conditions=dict(method='GET'))

    # Releases - Members
    mapper.connect('release', '/releases/{id}',
              controller='release', action='show',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('update_release', '/releases/{id}',
              controller='release', action='update',
              requirements=dict(id='d+'), conditions=dict(method='PUT'))
    mapper.connect('delete_release', '/releases/{id}',
              controller='release', action='delete',
              requirements=dict(id='d+'), conditions=dict(method='DELETE'))
    mapper.connect('edit_release', '/releases/{id}/edit',
              controller='release', action='edit',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('release_notes', '/releases/{id}/notes',
              controller='release', action='release_notes',
              requirements=dict(id='d+'), conditions=dict(method='GET'))

Full of duplication, verbose, even ugly. Just because it sits in a config directory, does it have to look that bad?

Refactoring 1: Introducing submapper()

New in Routes 1.11 (so really quite new), submapper()provides the means to pull out parameters previously shared across multiple connect() calls.

One particularly nice feature is that the SubMapper objects returned by submapper() support the Python managed object protocol so if you’re on Python 2.5 or above you can use them with the with syntax like this:

from release_tool.lib.mapper import Mapper

def make_map():

    ...

    # Releases - Collection
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))

    # Releases - Members
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases/{id}',
                    requirements=dict(id='d+')) as m:
        m.connect('release', '', action='show',
                conditions=dict(method='GET'))
        m.connect('update_release', '', action='update',
                conditions=dict(method='PUT'))
        m.connect('delete_release', '', action='delete',
                conditions=dict(method='DELETE'))
        m.connect('edit_release', '/edit', action='edit',
                conditions=dict(method='GET'))
        m.connect('release_notes', '/notes', action='release_notes',
                conditions=dict(method='GET'))

That’s a definite improvement (and credit where credit is due - submappers are great innovation, though a small bug requires our local extensions to be imported here), but notice that there’s still some duplication between the two submappers blocks, the first one corresponding to the collection resource, the second to that same collection’s members. Wouldn’t it be cool if we could nest them?

Refactoring 2: Submapper nesting

On the surface, the Routes 1.11 API doesn’t appear to support submapper nesting, but the SubMapper objects do indeed nest. Adding a submapper() method to SubMapper is a trivial change, and it take only minor internal tweaks for deeper nestings to function correctly.

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.connect('release', '', action='show',
                    conditions=dict(method='GET'))
            m.connect('update_release', '', action='update',
                    conditions=dict(method='PUT'))
            m.connect('delete_release', '', action='delete',
                    conditions=dict(method='DELETE'))
            m.connect('edit_release', '/edit', action='edit',
                    conditions=dict(method='GET'))
            m.connect('release_notes', '/notes', action='release_notes',
                    conditions=dict(method='GET'))

We seem to be on to something here, but what about the repetition (in one guise or another) of the resource name “release”? And what’s with those strange empty string ('') parameters?

Refactoring 3: Links and actions

To fix the repetition issue it’s clear that we need helpers that will generate those connect() calls for us more intelligently. But before we do that, let’s recognise that there are really two kinds of routes being generated here:

  1. Links to singleton subresources that support only one HTTP method, the most ubiquitous examples being the new and edit representations for which GET is the applicable HTTP method, but special subresources that are the targets for POST actions are common too;
  2. Actions that correspond directly to HTTP methods: index and create for GET and POST[1] on collection resources, show, update, delete for GET, PUT, DELETE on member resources.

It’s that second type that give rise to those strange empty strings as there is no further navigation to be done relative to the target resource.

So let’s add link() and action() helpers:

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.action(action='index', name='releases')
        c.action(action='create', method='POST')
        c.link('new')
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.action(action='show', name='release')
            m.action(action='update', method='PUT')
            m.action(action='delete', method='DELETE')
            m.link(rel='edit')
            m.link(rel='notes', name='release_notes')

Things are still moving in the right direction, but given that most of those links and actions follow a very well-worn convention, how about specific helpers for those?

Refactoring 4: More helpers

I’ll be honest - this is just an intermediate stage. You can see where we’re headed now though:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.index()
        c.create()
        c.new()
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.show()
            m.update()
            m.delete()
            m.edit()
            m.link(rel='notes', name='release_notes')

Refactoring 5: Parameter-driven generation

We’ll be using those same names a lot across our resources, so let’s just identify the ones we want in a list:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases',
                    actions=['index', 'create', 'new']) as c:
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+'),
                    actions=['show', 'update', 'delete', 'edit']) as m:
            m.link(rel='notes', name='release_notes')

But even this pattern will be repeated across resources, so let’s define a collection() helper that does it all:

Refactoring 6: Using the collection() helper

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

And we’re done.

Let’s compare this to the equivalent resource() call:

    # Releases
    mapper.resource(
                'release',
                'releases',
                controller='release',
                # no requirements!
                member = {'notes':'GET'})

Superficially similar, but I’m unable to specify requirements on member paths and I don’t get to name my custom route either - the generated name “notes_release” just isn’t right! The bottom line is that resource() is approaching a dead end; its only way forward is to make its parameters ever more complex.

So… how about we put resource() on notice and bring in something more flexible and - dare I say - Pythonic? And wouldn’t it look just as nice in Ruby too? Answers on a postcard please…

Prettyprinter for Pylons routes

December 15th, 2009

No, not described_routes, just something basic to plug a gap. For those suffering Rails-envy there’s no “paster routes” command, but it’s easy enough to use in the Paster shell:

bash-3.2$ paster shell test.ini
Pylons Interactive Shell
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)]

  All objects from encore.lib.base are available
  Additional Objects:
  mapper     -  Routes mapper object
  wsgiapp    -  This project's WSGI App instance
  app        -  paste.fixture wrapped around wsgiapp

>>> print mapper
Route name   Methods Path
                     /error/{action}
                     /error/{action}/{id}
home         GET     /
things       GET     /things
create_thing POST    /things
new_thing    GET     /things/new
thing        GET     /thing/{id}
...

It works by subclassing routes.Mapper (I have other reasons to do this which I’ll explain soon) and you’ll need to use this new Mapper in your config/routing.py.

That should be all you need to know. The code for the new Mapper class is below. Note: That second from last join() isn’t coming out very well - it has a backslash and an ‘n’ in it - honest!

Enjoy!

import routes.mapper

class Mapper(routes.Mapper):
    #
    # Pretty string representation - returns a string formatted like this:
    #
    #    Route name   Methods Path
    #                         /error/{action}
    #                         /error/{action}/{id}
    #    home         GET     /
    #    things       GET     /things
    #    create_thing POST    /things
    #    new_thing    GET     /things/new
    #    thing                  GET     /thing/{id}
    #
    #    etc
    #
    # Enter 'print mapper' in the paster shell to use it, or (TBD) run it via a
    # configured paster command.
    #
    def __str__(self):
        def format_methods(r):
            if r.conditions:
                method = r.conditions.get('method', '')
                return method if type(method) is str else ', '.join(method)
            else:
                return ''

        table = [('Route name', 'Methods', 'Path')] + [
            (
                r.name or '',
                format_methods(r),
                r.routepath or''
            )
            for r in self.matchlist]

        widths = [
            max(len(row[col]) for row in table)
            for col in range(len(table[0]))]

        return '
'.join(
            ' '.join(
                    row[col].ljust(widths[col])
                    for col in range(len(widths))
                ).rstrip()
            for row in table)

Nested forms in Pylons

November 5th, 2009

Easy when you know how, but finding that out was struggle - at least it was for me! The crucial piece of information I was lacking was the correspondence between variabledecode.variable_encode() which flattens nested values into an array and variabledecode.NestedVariables which does the reverse.

I have factored these lines below out to my lib/base.py. Now all my form schemas (nested or not) inherit from BaseSchema and I use fill_render() whenever I want to render html with values.

import formencode
from formencode import htmlfill
from formencode import variabledecode

def fill_render(template_name, values):
    """
    Render a template filled with values.  Values may be nested.
    """
    return htmlfill.render(
                    render(template_name),
                    variabledecode.variable_encode(values))

class BaseSchema(formencode.Schema):
    """
    Base form schema.  Any nested forms will be converted to nested values automatically.
    """
    allow_extra_fields = True
    filter_extra_fields = True
    pre_validators = [variabledecode.NestedVariables()]

It leads to DRYer controller code, reduces the number of imports required and hides some complexity. Can’t understand why it isn’t the default already in Pylons, but hey, it works.

To see nested forms in action, go to the interesting but (at the time of writing) not quite complete Solving the Repeating Fields Problem (Pylons Book v1.1 documentation).

Credit to Ian Wilson for pointing me in the right direction in this thread on pylons-discuss.

Principles of Product Development Flow - review & retrospective

October 6th, 2009

Introduction

I have been encouraged by Eric Willeke and Alan Shalloway via the kanbandev group to share my thoughts on Donald Reinertsen’s new book Principles of Product Development Flow and I promised a review that would also serve as something of a personal retrospective. As book reviews go the result is probably a mess, but you can read clean & tidy book reviews elsewhere!

Why the retrospective? As past readers of my blog will know, I’ve recently undergone some significant change of late. I left behind a 12 year career in the City, took a very welcome 3 month break from work, and at the end of that time made something of a lifestyle change by moving well away from London to live in a cottage in the picturesque Derbyshire Dales. At the time we committed ourselves to the move I didn’t know exactly would come next but there was sure to be a Lean flavour to it, and during my time out I quickly fell in with the Kanban crowd. When I landed my new role it was clear that Kanban would feature prominently from day 1; two months have since passed…

Back to the book: it’s not about Kanban (though it is mentioned). Neither is it a book about software development (it’s much more general than that, though the specifics of software development occur frequently throughout the book). It is however a book that has shone light on both subjects for me – not as another methodology book, but by strengthening significantly my understanding of how development processes work in general, helping me to really understand why I’ve seen what I’ve seen in past projects (good and bad, mine and others’), and by helping me to crystallise how I can deal in an economically justifiable way with things that appear on the surface to be completely intangible.

To be more specific with that last thought, what excited me most about the book was the way it ties the management of queues – covered very well in their own right – to economics. This happens early in the book, changes (or at least changed for me) the way the rest of the book is read, and hugely increases its value. It may sound academic, but I found it motivating (if occasionally uncomfortable) that I could link so directly the way I run projects past, present and future to business performance. And making that link was straightforward; it’s a book that is free of hand-waving and its examples ring true to my own experience.

Diving in

Let’s get down to details with some quotes:

The issue is not queues, it is the economics of queues (p17)

We can define waste as a failure to optimize [globally] our economics (p33)

Optimum queue size is an economic trade-off (p68)

Focus on the wrong queue can actually create push rather than pull. Why (speaking in retrospect with embarrassment at my own stupidity, not to criticise the jargon of certain branded methods) emphasise the development backlog when you haven’t achieved flow downstream? Are all of your queues even visible (always, to all, on the wall)? Do you have any idea – even roughly – of your cost of delay (“COD”) and consciously optimise (portfolio-wide) to minimise it? I’m beginning to get a handle on my COD, and the numbers are staggering, especially when I’m tempted to apply a P/E multiple to the answer!

But other queues may not matter so much. On my Kanban wall I have a “Proposed” column preceding “Prioritised” and it’s my job to move items between the two. I don’t lie awake feeling overwhelmed by the large number of proposed items – in fact quite the reverse: it feels good to deny attention to work (even whole projects) that would only add delay to work actually in progress. Is the movement between the two columns push or pull in nature? Probably a bit of both, but it’s one part of the process that won’t get reduced to a system of rules – the sequencing and regulation of work going into the process is far too important for that, and I have the book to thank for giving me a fresh perspective on my personal responsibilities toward both the process and the end result.

Use CFDs to monitor queues (p71)

Cumulative Flow Diagrams (CFDs) can be done independently of Kanban but they make most sense done together - just log each day the number of work items held in each column and chart cumulatively over time - easy! What does my chart reveal? I work a pattern of alternate weeks, one week with the team in Budapest and one at home and this is reflected very clearly in the chart. Most re-planning happens when I’m in Budapest (near both team and customer), leading quite naturally to a 2-week cadence for the whole team. The CFD does hint at the cost of being apart and we work hard collectively to mitigate it, but my data collection gets a little patchy when I’m away so I haven’t properly quantified it yet. I check the CFD regularly to measure lead times, to look for any signs of growing bottlenecks that aren’t already glaing at me from the wall, and I plan to use it as a presentation tool in future management meetings.

It’s nice to discover that one can monitor lead times so easily and I’m prepared even to read some predictive power into the chart without ever requiring development estimates. In fact estimates have become yet another thing to be added the ever-growing list of supposedly indispensible things that I’m simply not missing. And let’s face it: so often they’re rubbish anyway!

Create [decentralized] systems to harvest the early cheap opportunities (p40)

Understanding decentralisation and delegation is especially important when running distributed development. Here’s a technical example (though the point isn’t the technique): I don’t (can’t!) review every line of code that gets committed but it is easily decentralized. Not only can developers review each other’s code, but my loathing of duplication in code has been communicated strongly, along with a counterbalancing worry about breaking working code without economic justification. As a result, the kinds of refactorings that lead to a smaller codebase and consequently to a smaller maintenance headache happen regularly without my direct involvement, and (thanks also to investments in unit tests and a CI system) seemingly without too much pain. But is this really a “cheap opportunity”? Yes, one has to be concerned about overdoing the up-front investment, but in our case it has already made for a more comfortable and productive environment and I’m happy that the pay-back period is short enough to make this a no-brainer.

Onwards and upwards

All of this has been drawn from or prompted by just the first three chapters, there being nine in total. Reinertsen goes on to discuss variability, batch sizes, WIP constraints, controllability and feedback before taking an interesting turn with a chapter on decentralisation, drawing heavily from military theory. All of it is excellent, some of it very quotable, some of it very valuable reference material (and no less readable for that). I make no apologies though for emphasising the core concept Don surely intends to keep at the fore as we continue with the rest of the book.

Nits to pick? Only one springs to mind, and it’s not a problem, just a minor missed opportunity. It follows from the book’s analysis of the economics of the timing of decisions that an un-made decision has real economic value, but this could have been made explicit. A link to Real Options Theory would have been very nice here, and it might have informed some of the later parts of the book. I have a vague recollection that Real Options do get mentioned, but I couldn’t find it in the index.

I have two concluding remarks to make. The first is about Kanban, and it is to restate a comment I made on one of the mailing lists: implementing Kanban was undoubtedly one of my better professional decisions in a 20-something year career as developer, project and program manager and now the senior IT guy in my new organisation. The second is about the book: quite simply it’s brilliant; buy it!

described_routes is Rack middleware

August 4th, 2009

Last week I received this:

#described_routes could make beautiful middleware

Why didn’t I think of that?

So I took a quick look at Rack, found there was almost nothing to learn, and over the weekend made the change. And it was well worth it: integrating described_routes into your Rails application is now much easier. There’s no need to modify your routes (the middleware recognizes and serves requests to /described_routes automatically) or your controllers (the discovery protocol’s link headers are added automatically to other requests). In fact the old integration method looks so ugly by comparison that I’ve deprecated it - it’s *that* embarrassing!

Now, run-time integration needs only this modification to your environment.rb’s Rails::Initializer.run block:

require 'described_routes/middleware/rails'
Rails::Initializer.run do |config|
  ...
  config.middleware.use DescribedRoutes::Middleware::Rails
  ...
end

Revised instructions (compare with the old):

  1. Install the described_routes gem for the server
  2. Add build-time integration to the server (a one-liner to add some useful Rake tasks)
  3. Add run-time integration to the server (just the environment.rb modification above)
  4. Install and run path-to (for an “instant” client API)
  5. Profit!

Yes, it’s for Rack for Rails

The sharp-eyed reader will have noticed that despite the move to Rack, we’re still discussing a Rails integration. What about described_routes for other Rack-aware or Rack-based frameworks?

Most of the new middleware’s functionality exists in an abstract class DescribedRoutes::Middleware::Base but it needs two methods implemented for each framework:

  1. get_resource_templates - get named routes from the application and convert to ResourceTemplates
  2. get_resource_routing - map a request to a ResourceTemplate and its parameter list

In DescribedRoutes::Middleware::Rails:

  1. get_resource_templates hooks into described_routes code originally based on ‘rake routes’, and
  2. get_resource_routing extracts the controller name, action name and path parameters from a Rack environment member ‘action_controller.request.path_parameters’ populated by Rails as it processes the request - our stuff must happen afterwards.  The route name is reverse-mapped from the controller and action name.[1]

It should be clear now that both methods are necessarily framework-specific; indeed Rack does not provide routing itself.

The test of described_routes’s underlying framework-neutrality will be its integration with a framework other than Ruby on Rails. This should be easier to achieve now than previously; perhaps someone with more knowledge and need than me will beat me to it (please be my guest!). I’m tempted meanwhile to double the challenge and attempt it in Python for WSGI-based frameworks (wish me luck!).

[1]Actually it’s a shame that Rails doesn’t make the request’s route name or route object available anywhere - is there anyone else who would use it?

One link to rule them all

August 3rd, 2009

A quick demonstration of the described_routes discovery protocol is in order. It’s all done with link headers, one per resource.

Starting with some arbitrary resource in the application:

$ curl --silent --head http://localhost:3000/users/dojo | grep Link
Link: ; rel="describedby"; meta="ResourceTemplate"
$ curl --silent --head http://localhost:3000/described_routes/user?user_id=dojo | grep Link
Link: ; rel="index"; meta="ResourceTemplates"

So after the first HEAD request we have the location of the resource’s ResourceTemplate metadata; after the second we have the location of the site ResourceTemplates (plural) metadata.

A HEAD request to the site metadata location merely confirms (via the meta tag) what we already knew:

$ curl --silent --head http://localhost:3000/described_routes | grep Link
Link: ; rel="self"; meta="ResourceTemplates"

On the root resource, a single HEAD request links us to the site metadata location directly:

$ curl --silent --head http://localhost:3000 | grep Link
Link: ; rel="describedby"; meta="ResourceTemplates"

After that, it’s up to the client.

When you create a path-to client thus:

app = PathTo::DescribedRoutes::Application.discover("http://localhost:3000/users/dojo")

path-to discovers the site metadata and GETs it, specifying JSON as the acceptable format. The JSON response is converted to basic Ruby objects and used to initialize the Application object app. The client now knows the application’s resource structure:

app.users['dojo'].uri
=> "http://localhost:3000/users/dojo"

and we’re back where we started!

Putting it all together - ResourceTemplates, described_routes and path-to

July 27th, 2009

We’ve watched described_routes and path-to grow here over the past few weeks. And fun though it has been for me, it must be hard to get a good overview from blog posts triggered by design challenges! So here goes: an attempt at a one-post overview.

In a sentence?

Add described_routes to your Rails application to give it header-based site discovery with ResourceTemplate metadata that enables instant Ruby APIs on the client side with path-to.

And if I’m not using Rails or even Ruby?

There’s library support for ResourceTemplate metadata in Ruby (for the moment it’s included as part of described_routes) and Python (see described_routes-py). There’s a simpler version of path-to available in Python also, namely path-to-py. And there’s nothing Rails-specific or complicated about ResourceTemplates either - strip away the JSON, YAML or XML formatting and they’re not much more than named resources arranged hierarchically with their URI templates and supported HTTP methods as properties - and (so I’m told) can be useful even without path-to.

OK - I’m sold! What do I have to do?

Just follow these simple steps:

  1. Install described_routes for the server
  2. Add build-time integration to the server
  3. Add basic run-time integration to the server
  4. Add site discovery to the server
  5. Install and run path-to
  6. Profit!

The README files of the two gems tell you all you need to know in detail, but here in one place:

1. Install described_routes for the server

$ sudo gem install described_routes

2. Add build-time integration to the server

This is just a set of Rake tasks that lets you see immediately what the metadata looks like. In your Rakefile:

require 'tasks/described_routes'

Then try it out:

$ rake --tasks described_routes
rake described_routes:json        # Describe resource structure in JSON format
rake described_routes:xml         # Describe resource structure in XML format
rake described_routes:yaml        # Describe resource structure in YAML format
rake described_routes:text        # Describe resource structure in text (comparable to "rake routes")

Specify the base URI of your app with "BASE=http://..." to see full URIs in the output.

3. Add basic run-time integration to the server

Somewhere in your application include the controller, perhaps in an initializer:

require 'described_routes/rails_controller'

Add the following route in config/routes.rb:

map.resources :described_routes, :controller => "described_routes/rails"

You can now browse to /described_routes(.format) and /described_routes/{controller_name}(.format) and see the data generated at run time.

4. Add site discovery to the server

Site discovery (linking resources to their resource-specific and site-wide metadata) works via link headers (”Link:“) added to the responses served by one or more controllers. This has a double benefit:

i) Resources gain some type information derived from the Rails route name of the resource that may be of value to clients
ii) A path-to client (or any other client interested in the ResourceTemplate metadata) can be initialised from a regular resource URI; prior knowledge of metadata location is not needed.

According to your requirements, add the set_link_header filter to either the controller of your root resource (&/or or other specific controllers) or to ApplicationController in order to benefit all controllers:

require 'described_routes/helpers/described_routes_helper'

class MyController < ApplicationController
  include DescribedRoutes::DescribedRoutesHelper
  after_filter :set_link_header
end

Install and run path-to, …profit!

$ sudo gem install path-to

It is now a one-liner to bootstrap a client application, in this example a test blog with user and article resources:

require "path-to/described_routes"

# bootstrap a path-to client from the test_rails_app provided in described_routes
app = PathTo::DescribedRoutes::Application.discover("http://localhost:3000/")
#=> <PathTo::DescribedRoutes::Application>

# get user 'dojo'
puts app.users['dojo'].get
#=> '<html>...</html>

#get a JSON representation of the recent articles of user 'dojo'
puts app.users['dojo'].articles.recent['format' => 'json'].get
#=> [...]

Profit!

This bit is up to you, but metadata-enhanced web apps and instant client APIs achieved for so little work has to be worth something!