The A3 challenge

My previous post has been sitting there at the top of my front page for too long now, and it doesn’t reach my usual levels of positivity!  So let’s change that with a quick challenge:

  1. Can you relate the features in your development backlog (or at least the prioritised and in-progress portion thereof) to an identifiable business initiative?
  2. If – for real or in your imagination – you justified (compellingly), scoped and planned each of these business initiatives in just two sides of A4, how many of the features in your backlog would deserve a mention?

I take it for granted that you organise most of your development work by feature.  Large initiatives are allowed sub-initiatives, each also described (compellingly still)  in no more than two sides of A4 (real or imaginary).

The “A3″ of this post’s title refers to an A3 report, presented on a sheet of A3 paper (the size of two sheets of A4).  I wholeheartedly recommend John Shook’s Managing to Learn if this concept is new to you.

PathTo for Python gets a JSON-capable HTTP client

…and it works!

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')
>>> app.login.post(credentials, expected_status=302)
>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}
>>> product = app.products['Foo'].get(expected_status=200).parsed
>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)
>>> app.products['Foo'].get(expected_status=200).parsed['description']
u'Updated!'

OK, so it wasn’t really “example.com”, and the updated product wasn’t called “Foo”, but the rest is for real.

In stages:

0) The Pylons-based server has the JSON-capable @validate (un)decorator of my previous post and DescribedRoutesMiddleware from DescribedRoutes installed in its wsgi stack.

1) Create a client-side proxy to the app. Following link headers published by the app, it finds the ResourceTemplates description, which it retrieves in JSON format.

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')

The format='json' is a small red herring here, it’s simply remembered for later.

2) Log in by posting credentials, expecting a 302 status (a failed attempt would return a 200 and the validation errors in the body). The client handles cookies automatically behind the scenes.

>>> app.login.post(credentials, expected_status=302)

3) View a friendly representation of the metadata (or rather the part that relates to products):

>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}

4) Get a product identified by ‘Foo’ and return the JSON payload parsed into a Python dict.:

>>> product = app.products['Foo'].get(expected_status=200).parsed

Behind the scenes it has expanded the “http://example.com/products/{product}{.format}” template seen previously into “http://example.com/products/Foo.json”, using the remembered format parameter and the supplied ‘Foo’ key which is assumed to correspond to the required product parameter.

5) Update the local product representation and send it back (it gets converted back to JSON along the way):

>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)

6) Finally, demonstrate that it was successful!

>>> app.products['Foo'].get(expected_status=200).parsed['description']
u'Updated!'

Experimental {.format} in Routes

A while back I raised on the URI Template mailing list the idea of a {.var} syntax that generates optional path extensions, “.html”, “.json”, etc.

For example, given a path template of /releases/{id}/notes{.format} and an id parameter of “1″, paths of /releases/1/notes or /releases/1/notes.pdf will be generated depending on whether the supplied format parameters are undefined or “pdf” respectively.

It’s not a completely original idea.  Rails has a (.:format) syntax that DescribedRoutes maps to the flexible but somewhat clunky {-prefix|.|format} syntax of the old draft URI Template spec. The proposed syntax seems more in keeping with current draft (very much a work in progress) though.

Today my Routes fork gained support for the syntax and tweaks to the new SubMapper helpers so that duplicate “formatted routes” aren’t generated (something that happened to Rails a long time ago).

Example config (<code>config/routing.py</code> in your Pylons app):

with mapper.collection(
                'releases',
                'release',
                requirements={'id': 'd+'}) as c:
    c.member.link(rel='notes', name='release_notes')

And in the paster shell:

>>> print mapper
Route name     Methods Path
releases       GET     /releases{.format}
create_release POST    /releases{.format}
new_release    GET     /releases/new{.format}
release        GET     /releases/{id}{.format}
update_release PUT     /releases/{id}{.format}
delete_release DELETE  /releases/{id}{.format}
edit_release   GET     /releases/{id}/edit{.format}
release_notes  GET     /releases/{id}/notes{.format}
>>> url('release_notes', id=1, format='pdf')
'/releases/1/notes.pdf'
>>> url('release_notes', id=1)
'/releases/1/notes'
>>> mapper.match('/releases/1/notes')
{'action': u'notes', 'controller': u'release', 'id': u'1', 'format': None}
>>> mapper.match('/releases/1/notes.pdf')
{'action': u'notes', 'controller': u'release', 'id': u'1', 'format': u'pdf'}

Just add formatted=False to your collection() call if you don’t want this new behaviour.

I haven’t sent the Routes guys a pull request yet. I’d like some feedback first, and meanwhile I will try to come up with a nice helper that takes the format extension to override content negotiation. My app has some basic header-based conneg already but I would like the convenience of being able to override it in the URL.

Small steps along my personal Python roadmap

Three new things tried in recent days:

  1. Packaging a Python module
  2. Using doctest in anger (tweaking doctests in Routes doesn’t really count)
  3. Publishing a package to PyPi

For #1, I used setuptools, kick-started by Ian Bicking’s excellent presentation (the official docs are good too).  Completely painless, and it pretty much covers #3 too.  All in all I would say that it was easier than building and publishing a Rubygem.

Re #2, doctest isn’t going to replace regular unit tests for me, but for describing a public API I think it’s excellent.  In the case of LinkHeader, exercising the public API doesn’t leave much else – it’s a very simple package.  Here’s a small excerpt:

>>> parse('<http://example.com/foo>; rel="foo bar", <http://example.com>; rel=up; type=text/html')
LinkHeader([Link('http://example.com/foo', rel='foo bar'), Link('http://example.com', rel='up', type='text/html')])

which demonstrates how to parse a link header into something a client can easily interrogate.

LinkHeader will be used by DescribedRoutes to add discoverability to web apps and by PathTo clients in order to discover and interact with application resources.  All of this works in Ruby already of course; my current personal project is to replicate it in Python with support for the Pylons web framework.  The Link Header standard is progressing meanwhile, with a new version published only this week.

Prettyprinter for Pylons routes

No, not described_routes, just something basic to plug a gap. For those suffering Rails-envy there’s no “paster routes” command, but it’s easy enough to use in the Paster shell:

bash-3.2$ paster shell test.ini
Pylons Interactive Shell
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)]

  All objects from encore.lib.base are available
  Additional Objects:
  mapper     -  Routes mapper object
  wsgiapp    -  This project's WSGI App instance
  app        -  paste.fixture wrapped around wsgiapp

>>> print mapper
Route name   Methods Path
                     /error/{action}
                     /error/{action}/{id}
home         GET     /
things       GET     /things
create_thing POST    /things
new_thing    GET     /things/new
thing        GET     /thing/{id}
...

It works by subclassing routes.Mapper (I have other reasons to do this which I’ll explain soon) and you’ll need to use this new Mapper in your config/routing.py.

That should be all you need to know. The code for the new Mapper class is below. Note: That second from last join() isn’t coming out very well – it has a backslash and an ‘n’ in it – honest!

Enjoy!

import routes.mapper

class Mapper(routes.Mapper):
    #
    # Pretty string representation - returns a string formatted like this:
    #
    #    Route name   Methods Path
    #                         /error/{action}
    #                         /error/{action}/{id}
    #    home         GET     /
    #    things       GET     /things
    #    create_thing POST    /things
    #    new_thing    GET     /things/new
    #    thing                  GET     /thing/{id}
    #
    #    etc
    #
    # Enter 'print mapper' in the paster shell to use it, or (TBD) run it via a
    # configured paster command.
    #
    def __str__(self):
        def format_methods(r):
            if r.conditions:
                method = r.conditions.get('method', '')
                return method if type(method) is str else ', '.join(method)
            else:
                return ''

        table = [('Route name', 'Methods', 'Path')] + [
            (
                r.name or '',
                format_methods(r),
                r.routepath or''
            )
            for r in self.matchlist]

        widths = [
            max(len(row[col]) for row in table)
            for col in range(len(table[0]))]

        return '
'.join(
            ' '.join(
                    row[col].ljust(widths[col])
                    for col in range(len(widths))
                ).rstrip()
            for row in table)

described_routes is Rack middleware

Last week I received this:

#described_routes could make beautiful middleware

Why didn’t I think of that?

So I took a quick look at Rack, found there was almost nothing to learn, and over the weekend made the change. And it was well worth it: integrating described_routes into your Rails application is now much easier. There’s no need to modify your routes (the middleware recognizes and serves requests to /described_routes automatically) or your controllers (the discovery protocol’s link headers are added automatically to other requests). In fact the old integration method looks so ugly by comparison that I’ve deprecated it – it’s *that* embarrassing!

Now, run-time integration needs only this modification to your environment.rb‘s Rails::Initializer.run block:

require 'described_routes/middleware/rails'
Rails::Initializer.run do |config|
  ...
  config.middleware.use DescribedRoutes::Middleware::Rails
  ...
end

Revised instructions (compare with the old):

  1. Install the described_routes gem for the server
  2. Add build-time integration to the server (a one-liner to add some useful Rake tasks)
  3. Add run-time integration to the server (just the environment.rb modification above)
  4. Install and run path-to (for an “instant” client API)
  5. Profit!

Yes, it’s for Rack for Rails

The sharp-eyed reader will have noticed that despite the move to Rack, we’re still discussing a Rails integration. What about described_routes for other Rack-aware or Rack-based frameworks?

Most of the new middleware’s functionality exists in an abstract class DescribedRoutes::Middleware::Base but it needs two methods implemented for each framework:

  1. get_resource_templates – get named routes from the application and convert to ResourceTemplates
  2. get_resource_routing – map a request to a ResourceTemplate and its parameter list

In DescribedRoutes::Middleware::Rails:

  1. get_resource_templates hooks into described_routes code originally based on ‘rake routes’, and
  2. get_resource_routing extracts the controller name, action name and path parameters from a Rack environment member ‘action_controller.request.path_parameters’ populated by Rails as it processes the request – our stuff must happen afterwards.  The route name is reverse-mapped from the controller and action name.[1]

It should be clear now that both methods are necessarily framework-specific; indeed Rack does not provide routing itself.

The test of described_routes‘s underlying framework-neutrality will be its integration with a framework other than Ruby on Rails. This should be easier to achieve now than previously; perhaps someone with more knowledge and need than me will beat me to it (please be my guest!). I’m tempted meanwhile to double the challenge and attempt it in Python for WSGI-based frameworks (wish me luck!).

[1]Actually it’s a shame that Rails doesn’t make the request’s route name or route object available anywhere – is there anyone else who would use it?

One link to rule them all

A quick demonstration of the described_routes discovery protocol is in order. It’s all done with link headers, one per resource.

Starting with some arbitrary resource in the application:

$ curl --silent --head http://localhost:3000/users/dojo | grep Link
Link: ; rel="describedby"; meta="ResourceTemplate"
$ curl --silent --head http://localhost:3000/described_routes/user?user_id=dojo | grep Link
Link: ; rel="index"; meta="ResourceTemplates"

So after the first HEAD request we have the location of the resource’s ResourceTemplate metadata; after the second we have the location of the site ResourceTemplates (plural) metadata.

A HEAD request to the site metadata location merely confirms (via the meta tag) what we already knew:

$ curl --silent --head http://localhost:3000/described_routes | grep Link
Link: ; rel="self"; meta="ResourceTemplates"

On the root resource, a single HEAD request links us to the site metadata location directly:

$ curl --silent --head http://localhost:3000 | grep Link
Link: ; rel="describedby"; meta="ResourceTemplates"

After that, it’s up to the client.

When you create a path-to client thus:

app = PathTo::DescribedRoutes::Application.discover("http://localhost:3000/users/dojo")

path-to discovers the site metadata and GETs it, specifying JSON as the acceptable format. The JSON response is converted to basic Ruby objects and used to initialize the Application object app. The client now knows the application’s resource structure:

app.users['dojo'].uri
=> "http://localhost:3000/users/dojo"

and we’re back where we started!

Putting it all together – ResourceTemplates, described_routes and path-to

We’ve watched described_routes and path-to grow here over the past few weeks. And fun though it has been for me, it must be hard to get a good overview from blog posts triggered by design challenges! So here goes: an attempt at a one-post overview.

In a sentence?

Add described_routes to your Rails application to give it header-based site discovery with ResourceTemplate metadata that enables instant Ruby APIs on the client side with path-to.

And if I’m not using Rails or even Ruby?

There’s library support for ResourceTemplate metadata in Ruby (for the moment it’s included as part of described_routes) and Python (see described_routes-py). There’s a simpler version of path-to available in Python also, namely path-to-py. And there’s nothing Rails-specific or complicated about ResourceTemplates either – strip away the JSON, YAML or XML formatting and they’re not much more than named resources arranged hierarchically with their URI templates and supported HTTP methods as properties – and (so I’m told) can be useful even without path-to.

OK – I’m sold! What do I have to do?

Just follow these simple steps:

  1. Install described_routes for the server
  2. Add build-time integration to the server
  3. Add basic run-time integration to the server
  4. Add site discovery to the server
  5. Install and run path-to
  6. Profit!

The README files of the two gems tell you all you need to know in detail, but here in one place:

1. Install described_routes for the server

$ sudo gem install described_routes

2. Add build-time integration to the server

This is just a set of Rake tasks that lets you see immediately what the metadata looks like. In your Rakefile:

require 'tasks/described_routes'

Then try it out:

$ rake --tasks described_routes
rake described_routes:json        # Describe resource structure in JSON format
rake described_routes:xml         # Describe resource structure in XML format
rake described_routes:yaml        # Describe resource structure in YAML format
rake described_routes:text        # Describe resource structure in text (comparable to "rake routes")

Specify the base URI of your app with "BASE=http://..." to see full URIs in the output.

3. Add basic run-time integration to the server

Somewhere in your application include the controller, perhaps in an initializer:

require 'described_routes/rails_controller'

Add the following route in config/routes.rb:

map.resources :described_routes, :controller => "described_routes/rails"

You can now browse to /described_routes(.format) and /described_routes/{controller_name}(.format) and see the data generated at run time.

4. Add site discovery to the server

Site discovery (linking resources to their resource-specific and site-wide metadata) works via link headers (“Link:“) added to the responses served by one or more controllers. This has a double benefit:

i) Resources gain some type information derived from the Rails route name of the resource that may be of value to clients
ii) A path-to client (or any other client interested in the ResourceTemplate metadata) can be initialised from a regular resource URI; prior knowledge of metadata location is not needed.

According to your requirements, add the set_link_header filter to either the controller of your root resource (&/or or other specific controllers) or to ApplicationController in order to benefit all controllers:

require 'described_routes/helpers/described_routes_helper'

class MyController < ApplicationController
  include DescribedRoutes::DescribedRoutesHelper
  after_filter :set_link_header
end

Install and run path-to, …profit!

$ sudo gem install path-to

It is now a one-liner to bootstrap a client application, in this example a test blog with user and article resources:

require "path-to/described_routes"

# bootstrap a path-to client from the test_rails_app provided in described_routes
app = PathTo::DescribedRoutes::Application.discover("http://localhost:3000/")
#=> <PathTo::DescribedRoutes::Application>

# get user 'dojo'
puts app.users['dojo'].get
#=> '<html>...</html>

#get a JSON representation of the recent articles of user 'dojo'
puts app.users['dojo'].articles.recent['format' => 'json'].get
#=> [...]

Profit!

This bit is up to you, but metadata-enhanced web apps and instant client APIs achieved for so little work has to be worth something!

New link_header gem

My latest project on github and Rubyforge is link_header, a small rubygem for parsing and generating HTTP link headers as per the latest spec draft-nottingham-http-link-header-06.txt.

Usage

The usual install:

sudo gem install link_header

The library’s LinkHeader and LinkHeader::Link classes follow a pattern established in the ResourceTemplate and ResourceTemplate classes in that they offer easy conversions both to & from Ruby primitives, i.e. Arrays, Strings etc. This in turn makes them easy to prettyprint, convert to & from JSON and YAML, create from test fixtures, and so on. [Aside: @kevinrutherford and I discussed this idea on Twitter a few days ago in response to his blog post "factory method in ruby". It's worth a read.]

Link attribute names can appear more than once, so I have chosen a list of attribute/value pairs rather than a hash to represent link attributes. Link objects do however have an #attrs method that will lazily generate a hash if that’s convenient (it’s left to you to decide whether it’s safe!). There’s an example of this below.

So:

require "link_header"
require "pp"

#
# Create a LinkHeader with Link objects
#
link_header = LinkHeader.new([
  LinkHeader::Link.new("http://example.com/foo", [["rel", "self"]]),
  LinkHeader::Link.new("http://example.com/",    [["rel", "up"]])])

puts link_header.to_s
#=> <http://example.com/foo>; rel="self", <http://example.com/>; rel="up"

link_header.links.map do |link|
  puts "href #{link.href.inspect}, attr_pairs #{link.attr_pairs.inspect}, attrs #{link.attrs.inspect}"
end
#=> href "http://example.com/foo", attr_pairs [["rel", "self"]], attrs {"rel"=>"self"}
#   href "http://example.com/", attr_pairs [["rel", "up"]], attrs {"rel"=>"up"}

#
# Create a LinkHeader from raw (JSON-friendly) data
#
puts LinkHeader.new([
  ["http://example.com/foo", [["rel", "self"]]],
  ["http://example.com/",    [["rel", "up"]]]]).to_s
#=> <http://example.com/foo>; rel="self", <http://example.com/>; rel="up"

#
# Parse a link header into a LinkHeader object then produce its raw data representation
#
pp LinkHeader.parse('<http://example.com/foo>; rel="self", <http://example.com/>; rel = "up"').to_a
#=> [["http://example.com/foo", [["rel", "self"]]],
#    ["http://example.com/", [["rel", "up"]]]]

Later…

My next programming task will be some minor refactoring on described_routes and path-to take advantage of this new gem. The driver behind this all is an efficient discovery protocol and a significant reduction in the number of links reported by default – I realised that it was wasteful to produce multiple links on every request that are (let’s be honest) be of no interest at all to clients, when just one of those links points to metadata that carries that same information and more! Then for those server apps that generate the correct headers, a short one-liner will initialize a path-to client given the address of any resource served by the application, i.e. without special knowledge of the location of the app’s metadata resources.

Refactoring aside, the described_routes part of this is done (so servers support the protocol already); I just need to finish path-to part to take advantage of it on the client side.

I can’t make any promises about timelines at the moment (new job starts soon) but a Python version should be forthcoming soon(ish) also. Meanwhile, enjoy the Ruby version if you can!

Distracted

Yes, things are a bit slow on the blog front at the moment. Excuses: we move to a Georgian cottage in the Derbyshire Dales on Wednesday, I start an exciting new job on August 3rd (more on that nearer the time), and we have a new miniature schnauzer puppy named Klaus!

I haven’t forgotten described_routes and path-to though. There’s a serious link header tidy-up underway that delivers an efficient discovery protocol. Details to follow…

All in all, I’ve had a very enjoyable and quite productive 3 month break from full-time work. As well as the Ruby and Python programming I’ve been hanging out (as @asplake) with the #kanban guys on Twitter – a great bunch of people – and enjoyed @dpjoyce’s SkillsMatter presentation on kanban at the BBC in person a few weeks ago. It has also been good to clear the house of clutter (how Lean is that!), though to be honest, most of the credit for that goes to my very hardworking wife Sharon, who has been wonderful throughout this period of great change. Credit too to my teenage boys for agreeing to take this leap into the unknown!