New on github: described_routes-py

described_routes-py – a Python port of the core metadata classes ResourceTemplate and ResourceTemplates.  It’s been fun learning Python and I’m pleased with the result – it looks cleaner somehow and I’m seriously considering treating this as the reference version.

I need to add support for partial template expansion to Joe Gregorio’s URI Template parser (to be found here) and add some industrial-strength tests. I’m putting off XML support (perhaps indefinitely), and coming round to the conclusion that I needn’t add explicit JSON and YAML support as it’s trivially easy – using your favourite JSON or YAML module – to go to or from the hash and dict representations already supported.   So that’s the core of it done really.

That leaves client and server integration. Unless I’m deluding myself, I think my Python is up to a port of the path-to client, though it will need some different metaprogramming tricks to the Ruby version. On the server side, I have the challenge of learning Django without (still!) knowing whether its routing datastructures are open to the same treatment I gave to Rails. I guess it will be fun trying! Worst case would be some hand crafting, not too terrible.

Next up: Python!

I’ve recently read and enjoyed Mark Pilgrim’s Dive into Python and I’m now about half way (i.e. 500 pages!) through Wesley Chun’s Core Python Programming.

The next stage in my Python education will be to port ResourceTemplate, the metadata core of described_routes. It would be nice also to replicate for Django the Rails parts of described_routes, but I don’t know yet how feasible this will be. Thoughts, anyone?

ResourceTemplate refactorings

I’ve tidied up the ResourceTempate API quite a bit over the past few days.  The most important changes are:

  1. described_routes 0.5.0 introduces a new ResourceTemplates class (subclassed from Array) to which have been moved many of the class methods of ResourceTemplate, especially those that took an array of ResourceTemplate objects as an argument.
  2. In described_routes 0.5.1 the ResourceTemplate initializer takes a Hash instead of 8 (!) separate parameters.

The two changes together make it so easy to initialize whole hierarchies (or collections of hierarchies) from JSON or YAML that I’ve deleted the methods that previously did this explicitly.  Now it’s:

ResourceTemplate.new(JSON.parse(json))
ResourceTemplate::ResourceTemplates.new(JSON.parse(json))

or

ResourceTemplate.new(YAML.load(yaml))
ResourceTemplate::ResourceTemplates.new(YAML.load(yaml))

XML remains a bit of a second-class citizen -  I still haven’t got round to implementing XML parsing.  Perhaps I should drop it altogether?

I’ve updated path-to to take account of these changes.  One minor slip though: the latest path-to gem claims to be dependent on described_routes 0.5.0 rather than 0.5.1.  This will be fixed in the next path-to release; meanwhile simply make sure you have only the latest described_routes gem installed and you’ll be fine.

Latest gems, URI Template update

Bob Aman (sporkmonger) has released the latest version of the excellent addressable to RubyForge so that it’s no longer necessary to get it from github.  This incorporates amongst other things (my minor bug fixes included) Bob’s cool new partial URI Template expansion feature that is the foundation of the changes described in my previous post.  I’ve updated both described_routes and path-to so that installing either of these will pull in addressable automatically.

So:

$ sudo gem install described_routes path-to
Successfully installed addressable-2.1.0
Successfully installed described_routes-0.4.1
Successfully installed path-to-0.4.1
3 gems installed

and you’re done.

Meanwhile, the uri@w3.org list awoke this week in response to a request from Joe Gregorio for examples of real-world usage of URI Templates and feedback on whether simplification was warranted.  Bob and I were united in asking that the advance features not be withdrawn – they are genuinely useful!

Partial template expansion in described_routes

In between a trip to the local dump, another meeting with HR (role officially redundant now, though still on the payroll for a few weeks) and getting our signatures witnessed for our house sale, I added partial template expansion to described_routes today.  This is potentially a lot more important than it sounds, but before I go into that, an explanation of the functionality.

What it does

Any query parameters passed to the included Rails controller will be used to pre-populate the generated resource templates and their URI templates.  In this example, the <code>article_id</code> and <code>format</code> parameters have been replaced with actual values, leaving the <code>article_id</code> unchanged:

$ curl "http://localhost:3000/described_routes/user.text?format=json&user_id=dojo"
user                 user                 GET, PUT, DELETE       http://localhost:3000/users/dojo.json
  edit               edit_user            GET                    http://localhost:3000/users/dojo/edit.json
  articles           user_articles        GET, POST              http://localhost:3000/users/dojo/articles.json
    new_user_article new_user_article     GET                    http://localhost:3000/users/dojo/articles/new.json
    recent           recent_user_articles GET                    http://localhost:3000/users/dojo/articles/recent.json
    {article_id}     user_article         GET, PUT, DELETE       http://localhost:3000/users/dojo/articles/{article_id}.json
      edit           edit_user_article    GET                    http://localhost:3000/users/dojo/articles/{article_id}/edit.json
  profile            user_profile         GET, PUT, DELETE, POST http://localhost:3000/users/dojo/profile.json
    edit             edit_user_profile    GET                    http://localhost:3000/users/dojo/profile/edit.json
    new              new_user_profile     GET                    http://localhost:3000/users/dojo/profile/new.json

More typically, JSON, YAML or XML format would be requested.  Here’s a JSON example (after pretty printing – the original output was all on one line):

$ curl "http://localhost:3000/described_routes/user_articles.yaml?user_id=dojo&format=json"
{
   "name":"user_articles",
   "rel":"articles",
   "path_template":"/users/dojo/articles.json",
   "uri_template":"http://localhost:3000/users/dojo/articles.json",
   "options":["GET", "POST"],
   "resource_templates":[
      {
         "name":"new_user_article",
         "options":["GET"],
         "path_template":"/users/dojo/articles/new.json",
         "uri_template":"http://localhost:3000/users/dojo/articles/new.json",
         "rel":"new_user_article"
      },
      {
         "name":"recent_user_articles",
         "options":["GET"],
         "path_template":"/users/dojo/articles/recent.json",
         "uri_template":"http://localhost:3000/users/dojo/articles/recent.json",
         "rel":"recent"
      },
      {
         "name":"user_article",
         "resource_templates":[
            {
               "name":"edit_user_article",
               "options":["GET"],
               "path_template":"/users/dojo/articles/{article_id}/edit.json",
               "uri_template":"http://localhost:3000/users/dojo/articles/{article_id}/edit.json",
               "rel":"edit",
               "params":["article_id"]
            }
         ],
         "options":["GET", "PUT", "DELETE"],
         "path_template":"/users/dojo/articles/{article_id}.json",
         "uri_template":"http://localhost:3000/users/dojo/articles/{article_id}.json",
         "params":["article_id"]
      }
   ]
}

Putting it to use, and why it’s important

Up to now, we’ve focussed on describing whole applications, providing client applications with metadata that helps them navigate the application effectively, even driving client APIs such as those generated dynamically by path-to.  This is an efficient way of doing things and it maps well onto the way familiar web APIs work now, but can we unify it with the RESTful concepts around hypertext (HATEOAS and all that)?  I think we can.

Imagine putting in the <head> section of the HTML representation of John Doe’s article collection at http://localhost:3000/users/dojo/articles a <link> element that references its own specific metadata at http://localhost:3000/described_routes/user_articles.json?user_id=dojo&format=json. A client reading that document can follow that link and now has in its possession the recipe for accessing all of the resources that belong to that collection – whether through complete URIs (e.g. for recent articles) or through parameterised templates (e.g. for specific articles, identified by id).

We can improve on this by returning these resource-specific links in HTTP headers. This makes them independent of the representation format (<head> and <link> are HTML-specific) and will work even on requests that return no body at all, HEAD in particular.  At the cost of one header (easily suppressed for clients that don’t need it), every resource is uniquely enhanced, minimising the amount of state that needs to be retained in the client.  For very lightweight clients (e.g. JavaScript libraries embedded in the browser) or for applications lacking a stable overall structure, this could be very useful indeed.

A mini path-to app centered on Mr Doe’s article collection could look something like this:

require "path-to/described_routes"
articles = PathTo::DescribedRoutes.bootstrap(
               "http://localhost:3000/users/dojo/articles")
articles.recent.get.first
articles["hateoas-and-other-7-letter-acronyms"].get
articles.post("title" => "The power of reflection", ...)

Here the bootstrap method creates an articles object that has an API that follows the structure given in the metadata discovered via headers found at http://localhost:3000/users/dojo/articles. Easy!

Positional params for path-to/described_routes

I’ve checked off another roadmap item for path-to, namely positional parameters.  This makes its metadata-driven client APIs feel even more natural.

Now, when indexing collection resources such as those seen in the Rails example, expressions such as

app.users['user_id' => 'dojo']

can be written

app.users['dojo']

which is quite a lot neater!  This style can still be mixed with hash style parameters, such as

app.users['dojo', {'format' => 'json'}]

or

app.users['dojo']['format' => 'json']

In the Delicious API example,

delicious.posts['tag' => 'ruby']

can be written

delicious.posts('ruby')

The square brackets become round brackets in this case as the metadata doesn’t model posts as a collection – it’s just a resource that may take a number of optional parameters, the first of which is the tag name.  It took me a few moments to get used to this distinction, but I’m happy now that this gives a more obvious mapping to the underlying API.

The joy of finishing

It’s been three weeks since my last project management post.  Yes, the web integration stuff I’ve blogged about meanwhile has kept me busy, but the truth is that I’ve been procrastinating.  Ironically, the title “The joy of finishing” has been in my mind for a while!

So let me quote from this morning’s wake-up call: John Heintz on the Lean & Kanban 2009 Conference:

Here’s what I thought Kanban was before last week:

* A Big Visible Board
* A Prioritized Backlog
* Close communication, minimizing hand-offs
* Rules about cards on the wall

No Iteration/Sprint boundaries (I’m thinking more efficient but maybe losing something important…)

That’s all well and good and true enough. Easy to justify writing it all off with “I already know enough to help teams make a big difference”. In fact, Kanban can be boiled down to one single rule:

* Limit the number of things in work to a fixed number

(emphasis mine)

When I think of the number of times we’ve done great work but failed to release it on time thanks to overcommitment, I feel embarrassed. Quite right too. This is coming from someone who talks about minimising backlogs and work in progress. As I write, I wonder if sometimes I have confused the two?

Update: I really must include this tweet from Scott Berkun too:

berkun Stressed out means 1 of 2 things: 1) Your priorities are wrong 2) You’re not respecting the priorities you have.

This applies to teams and projects just as much as it does to people. Inexcusable when it’s comes from the project manager, often more a carrier of stress than a sufferrer…

Delicious API with path-to/described_routes

The latest path-to contains an example (based on HTTParty’s) that queries Delicious via a Ruby API that’s 100% metadata driven.  Stripped of comments (the original commented version is here), the code is tiny:

require 'path-to/described_routes'
require 'pp'

config = YAML::load(File.read(File.join(ENV['HOME'], '.delicious')))

delicious = PathTo::DescribedRoutes::Application.new(
              :yaml => File.read(File.join(
                                     File.dirname(__FILE__),
                                     'delicious.yaml')),
              :http_options => {
                  :basic_auth => {
                      :username => config['username'],
                      :password => config['password']}})

pp delicious.posts['tag' => 'ruby'].get
pp delicious.posts['tag' => 'ruby'].recent['count' => '5'].get
delicious.recent_posts.get['posts']['post'].each do |post|
  puts post['href']
end

A couple of things to highlight are:

  1. The methods posts, recent, recent_posts are available thanks to the metadata – they’re not explicitly coded anywhere.  Similarly, URIs aren’t coded here either.
  2. The resource hiercharchy.  These are all separately identifiable things:
    • posts
    • posts['tag' => 'ruby']
    • posts['tag' => 'ruby'].recent
    • posts['tag' => 'ruby'].recent['count' => '5']

The metadata takes the form of described_routes-style resource templates, and I did it by hand this time in YAML:

---
- name: posts
  options:
  - GET
  uri_template: https://api.del.icio.us/v1/posts/get?{-join|&|tag,dt,url}
  optional_parameters:
  - tag
  - dt
  - url
  resource_templates:
  - name: recent_posts
    options:
    - GET
    uri_template: https://api.del.icio.us/v1/posts/recent?{-join|&|tag,count}
    rel: recent
    optional_parameters:
    - tag
    - count

Manifesto and roadmaps for described_routes and path-to

Introduction

Clients of RESTful web applications typically use prior knowledge of the target application’s structure to generate URIs.  This approach is often very convenient, but much of this URI generation is hard-coded, and (worse) spread across client code. This introduces a high degree of coupling and makes clients unnecessarily vulnerable to server-side change.

Steps to improve this situation:

  1. In clients, centralise the generation of URIs and make the process driven by configuration data
  2. Have servers publish the required configuration data – i.e. application metadata – in a readily understood format

path-to provides the means for client applications to model web applications in terms of logical structure and URI mappings, and to interact with them through dynamically-generated application-specific APIs.  described_routes supports an application metadata structure (published in JSON, YAML and XML formats) that can be consumed by path-to, and (helpfully) generates it automatically online or offline from the routes configured for a Rails-based application.

The two libraries can be used separately or together – an JavaScript client is under independent development for example.  Moreover, the underlying metadata format is framework-neutral; we have been careful not to “leak” Rails concepts into it.

1. described_routes

“Framework-neutral metadata describing Rails routes in JSON, YAML, XML and plain text formats”

The project has three main elements:

  1. Rake tasks that produce representations of a Rails application’s entire structure offline
  2. A simple controller that supports online access to these representations, adding the ability to pick out structures rooted on specified named routes
  3. The ResourceTemplate class and the formats that it generates and parses.    This is completely independent of Rails, in terms of both code dependencies and data structure.

The ResourceTemplate format describes a hierarchical organisation of web resources together with their:

  • logical names (their route names in the Rails case, e.g. “user_articles”)
  • logical parent-child relationships (e.g. “edit”, for a resource’s edit form)
  • URI Templates and the names of the parameters required to populate them
  • supported HTTP options (GET, PUT, POST, DELETE)

These metadata representations are derived automatically from the routes description that is conventionally coded in config/routes.rb.  See the appendix at the end of this article for examples.

Current status:

  • Working, published (as a github repository and as an installable rubygem on RubyForge)
  • To my knowledge it has been tested only by myself (on some of the examples given in the boilerplate config/routes.rb)

Still outstanding:

  • Parsing of the XML representation (and tests of round trips with the JSON and YAML formats)
  • The offline rake tasks are unaware of the application’s base URI and therefore can only produce templates to paths.  Find a way to get the base URI for a Rails environment offline (I’m not even sure this is possible), or allow one to be provided as a parameter. UPDATE (May 6th): added parameter “BASE=http://…” to rake tasks

Possible future work:

  • Solicit real-world examples for testing, unusual ones especially!  All I would need are config/routes.rb files and willingness to review/discuss the resulting representations
  • Break out the ResourceTemplate class into its own gem
  • Generate ResourceTemplate representations for specific resource instances (where the URI templates of child resources are partially populated with the root instance’s URI parameters), facilitating a more dynamic (and arguably more RESTful) interaction style
  • Generate URI Template mappings for query parameters
  • Implementations for Ruby web frameworks other than Rails.  Where frameworks lack the routing metadata of Rails this will entail work on the part of the application developer however.
  • Non-Ruby implementations UPDATE (June 2nd) basic Python ports done
  • Investigate alternatives to URI Templates, make this a pluggable choice
  • Add customisation hooks (@siebertm): UPDATE (May 6th): done

2. path-to

“Model web apps easily and access them via nice app-specific Ruby APIs”

path-to predates described_routes, and with a modest level of coding effort it can be used without it (see this early example). From the beginning I had however hoped to find a suitable format already in existence but with no real success so far!  With described_routes, path-to provides a Ruby client API to a Rails application for almost no effort, as shown in this more recent example.

Current status:

Still outstanding:

  • Demonstrations of access to well known web APIs – realistic examples with authentication etc.  UPDATE (May 8th): added Delicious example.

Possible future work:

  • positional parameters, allowing (for example) users['dojo'] as well as users['user_id' => 'dojo'] UDPDATE (May 12th): done
  • Support other metadata representations as they are brought to my attention (are there any good ones out there?)
  • Port to Python; demonstrate easy Python/Rails interop UPDATE (June 2nd) basic Python ports done

Appendix – examples

Text format for described_routes

described_routes is all about about metadata (e.g. for use by path_to),  but why not a more human-readable format?

So further to these instructions, the latest described_routes now has a plain text format.  It’s somewhat comparable to doing a rake routes, but brings out the structure of your application’s resources rather more clearly I think:

  $ rake described_routes:text
  (in /Users/asplake/railsapps/testapp)
  root                   root                                          /
  admin_products         admin_products         GET, POST              /admin/products{-prefix|.|format}
    new_admin_product    new_admin_product      GET                    /admin/products/new{-prefix|.|format}
    {product_id}         admin_product          GET, PUT, DELETE       /admin/products/{product_id}{-prefix|.|format}
      edit               edit_admin_product     GET                    /admin/products/{product_id}/edit{-prefix|.|format}
  described_routes       described_routes       GET, POST              /described_routes{-prefix|.|format}
    new_described_route  new_described_route    GET                    /described_routes/new{-prefix|.|format}
    {route_name}         described_route        GET, PUT, DELETE       /described_routes/{route_name}{-prefix|.|format}
      edit               edit_described_route   GET                    /described_routes/{route_name}/edit{-prefix|.|format}
  pages                  pages                  GET, POST              /pages{-prefix|.|format}
    new_page             new_page               GET                    /pages/new{-prefix|.|format}
    {page_id}            page                   GET, PUT, DELETE       /pages/{page_id}{-prefix|.|format}
      edit               edit_page              GET                    /pages/{page_id}/edit{-prefix|.|format}
      summary            summary_page           GET                    /pages/{page_id}/summary{-prefix|.|format}
      toggle_visibility  toggle_visibility_page POST                   /pages/{page_id}/toggle_visibility{-prefix|.|format}
  users                  users                  GET, POST              /users{-prefix|.|format}
    new_user             new_user               GET                    /users/new{-prefix|.|format}
    {user_id}            user                   GET, PUT, DELETE       /users/{user_id}{-prefix|.|format}
      edit               edit_user              GET                    /users/{user_id}/edit{-prefix|.|format}
      articles           user_articles          GET, POST              /users/{user_id}/articles{-prefix|.|format}
        new_user_article new_user_article       GET                    /users/{user_id}/articles/new{-prefix|.|format}
        recent           recent_user_articles   GET                    /users/{user_id}/articles/recent{-prefix|.|format}
        {article_id}     user_article           GET, PUT, DELETE       /users/{user_id}/articles/{article_id}{-prefix|.|format}
          edit           edit_user_article      GET                    /users/{user_id}/articles/{article_id}/edit{-prefix|.|format}
      profile            user_profile           GET, PUT, DELETE, POST /users/{user_id}/profile{-prefix|.|format}
        edit             edit_user_profile      GET                    /users/{user_id}/profile/edit{-prefix|.|format}
        new              new_user_profile       GET                    /users/{user_id}/profile/new{-prefix|.|format}

If you install described_routes for runtime use, you get (1) full URI Templates (not just paths) and (2) the ability to pick out parts of the hierarchy:

  $ curl http://localhost:3000/described_routes/users.txt
  users                  users                GET, POST              http://localhost:3000/users{-prefix|.|format}
    new_user             new_user             GET                    http://localhost:3000/users/new{-prefix|.|format}
    {user_id}            user                 GET, PUT, DELETE       http://localhost:3000/users/{user_id}{-prefix|.|format}
      edit               edit_user            GET                    http://localhost:3000/users/{user_id}/edit{-prefix|.|format}
      articles           user_articles        GET, POST              http://localhost:3000/users/{user_id}/articles{-prefix|.|format}
        new_user_article new_user_article     GET                    http://localhost:3000/users/{user_id}/articles/new{-prefix|.|format}
        recent           recent_user_articles GET                    http://localhost:3000/users/{user_id}/articles/recent{-prefix|.|format}
        {article_id}     user_article         GET, PUT, DELETE       http://localhost:3000/users/{user_id}/articles/{article_id}{-prefix|.|format}
          edit           edit_user_article    GET                    http://localhost:3000/users/{user_id}/articles/{article_id}/edit{-prefix|.|format}
      profile            user_profile         GET, PUT, DELETE, POST http://localhost:3000/users/{user_id}/profile{-prefix|.|format}
        edit             edit_user_profile    GET                    http://localhost:3000/users/{user_id}/profile/edit{-prefix|.|format}
        new              new_user_profile     GET                    http://localhost:3000/users/{user_id}/profile/new{-prefix|.|format}

Enjoy!