Positive Incline Mike Burrows (@asplake) moving on up, positively

February 18, 2010

PathTo for Python gets a JSON-capable HTTP client

Filed under: Web Integration — Tags: , , , , , , , , — Mike @ 1:55 pm

…and it works!

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')
>>> app.login.post(credentials, expected_status=302)
>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}
>>> product = app.products['Foo'].get(expected_status=200).parsed
>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)
>>> app.products['Foo'].get(expected_status=200).parsed['description']

OK, so it wasn’t really “example.com”, and the updated product wasn’t called “Foo”, but the rest is for real.

In stages:

0) The Pylons-based server has the JSON-capable @validate (un)decorator of my previous post and DescribedRoutesMiddleware from DescribedRoutes installed in its wsgi stack.

1) Create a client-side proxy to the app. Following link headers published by the app, it finds the ResourceTemplates description, which it retrieves in JSON format.

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')

The format='json' is a small red herring here, it’s simply remembered for later.

2) Log in by posting credentials, expecting a 302 status (a failed attempt would return a 200 and the validation errors in the body). The client handles cookies automatically behind the scenes.

>>> app.login.post(credentials, expected_status=302)

3) View a friendly representation of the metadata (or rather the part that relates to products):

>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}

4) Get a product identified by ‘Foo’ and return the JSON payload parsed into a Python dict.:

>>> product = app.products['Foo'].get(expected_status=200).parsed

Behind the scenes it has expanded the “http://example.com/products/{product}{.format}” template seen previously into “http://example.com/products/Foo.json”, using the remembered format parameter and the supplied ‘Foo’ key which is assumed to correspond to the required product parameter.

5) Update the local product representation and send it back (it gets converted back to JSON along the way):

>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)

6) Finally, demonstrate that it was successful!

>>> app.products['Foo'].get(expected_status=200).parsed['description']

August 4, 2009

described_routes is Rack middleware

Filed under: Web Integration — Tags: , , , , , , , — Mike @ 12:01 pm

Last week I received this:

#described_routes could make beautiful middleware

Why didn’t I think of that?

So I took a quick look at Rack, found there was almost nothing to learn, and over the weekend made the change. And it was well worth it: integrating described_routes into your Rails application is now much easier. There’s no need to modify your routes (the middleware recognizes and serves requests to /described_routes automatically) or your controllers (the discovery protocol’s link headers are added automatically to other requests). In fact the old integration method looks so ugly by comparison that I’ve deprecated it – it’s *that* embarrassing!

Now, run-time integration needs only this modification to your environment.rb‘s Rails::Initializer.run block:

require 'described_routes/middleware/rails'
Rails::Initializer.run do |config|
  config.middleware.use DescribedRoutes::Middleware::Rails

Revised instructions (compare with the old):

  1. Install the described_routes gem for the server
  2. Add build-time integration to the server (a one-liner to add some useful Rake tasks)
  3. Add run-time integration to the server (just the environment.rb modification above)
  4. Install and run path-to (for an “instant” client API)
  5. Profit!

Yes, it’s for Rack for Rails

The sharp-eyed reader will have noticed that despite the move to Rack, we’re still discussing a Rails integration. What about described_routes for other Rack-aware or Rack-based frameworks?

Most of the new middleware’s functionality exists in an abstract class DescribedRoutes::Middleware::Base but it needs two methods implemented for each framework:

  1. get_resource_templates – get named routes from the application and convert to ResourceTemplates
  2. get_resource_routing – map a request to a ResourceTemplate and its parameter list

In DescribedRoutes::Middleware::Rails:

  1. get_resource_templates hooks into described_routes code originally based on ‘rake routes’, and
  2. get_resource_routing extracts the controller name, action name and path parameters from a Rack environment member ‘action_controller.request.path_parameters’ populated by Rails as it processes the request – our stuff must happen afterwards.  The route name is reverse-mapped from the controller and action name.[1]

It should be clear now that both methods are necessarily framework-specific; indeed Rack does not provide routing itself.

The test of described_routes‘s underlying framework-neutrality will be its integration with a framework other than Ruby on Rails. This should be easier to achieve now than previously; perhaps someone with more knowledge and need than me will beat me to it (please be my guest!). I’m tempted meanwhile to double the challenge and attempt it in Python for WSGI-based frameworks (wish me luck!).

[1]Actually it’s a shame that Rails doesn’t make the request’s route name or route object available anywhere – is there anyone else who would use it?

June 30, 2009

Bootstrapping REST

Filed under: Web Integration — Tags: , , , , , , — Mike @ 6:55 pm

In this article I’m approaching REST from the perspective of client/server interaction, most especially (and slightly unusually I think) on the needs of the client and you, the client’s developer. I hope you find it helpful.

A caveat (no apology): my interest is in structure and navigation. HTTP has mechanisms already for clients and servers to negotiate content types and I’ve made a conscious design decision to rely on their existence. In other words, I ignore it completely here. Orthogonality is a wonderful thing!

We start at the beginning:

The completely hand-crafted client

At the most extreme, you’re working against an undocumented, unsupported server API that lacks regular structure. Any complexity implies significant reverse-engineering effort, made worse if you’re tracking a moving target. If you’re lucky though, you’ll be working against a supported and stable API.

Relying on regularity

Servers built using frameworks such as Rails (and even many that aren’t) will tend to exhibit some predictable patterns. The details may be framework-specific, but a collection resource, say “users”, can be expected to support these (or similar) operations and subresources:

GET, POST         http://www.example.com/users
GET               http://www.example.com/users/new
GET, PUT, DELETE  http://www.example.com/users/{user_id}
GET               http://www.example.com/users/{user_id}/edit

and a nested collection resource (each user’s articles, say) can be expected to follow a similar structure, rooted on a specific user resource.

GET, POST         http://www.example.com/users/{user_id}/articles
GET               http://www.example.com/users/{user_id}/articles/new
GET, PUT, DELETE  http://www.example.com/users/{user_id}/articles/{article_id}
GET               http://www.example.com/users/{user_id}/articles/{article_id}/edit

Armed therefore with just the knowledge of the user and articles collections, your client’s internal object model might easily support expressions such as


You’ll pay a price when the application departs from its usual pattern, but most of the time you’ll be fine. In fact for many client applications the edit and new are irrelevant, so all you need is the basic hierarchical structure and knowledge of the representations you can GET, PUT, POST there, also how to DELETE things.

Model-driven development

One approach to speeding application development is to abstract out that object model and generate skeleton code for both client and server from some common, logical representation. Not only are you spared the task of building the object model, but (given the right toolset) you can expect to have the web client functionality baked in for you. Most importantly, it’s much cheaper to track any changes made on the server.

The downside? You are now tied to a toolset. That might be a price worth paying if you’re in control of both client and server (for an internal application, say), but for many this will be unpalatable.

The no-model option

[Stick with me – this apparent digression makes metadata-driven clients easier to introduce!]

Dynamic languages make it possible to support expression like


even if your app object doesn’t actually have a users member (method, attribute or property) and articles and new exist nowhere either. See [1] and [2] to read how (generally) this can be done in Ruby and Python, and [3] for a description of path-to [4], an actual client that can work this way.

Even without these dynamic features, a similar effect can be achieved with operator overloading. This, for example, is perfectly valid Java:

app / users / member(user_id) / articles / _new

The objects returned by such expressions can easily have a URI representation, request methods and so on.

But beware! Suppose a user resource defines an “up” link. Take for example


To which of these URIs (below) does this expression refer?


We’ll come back to this issue later.

Metadata-driven clients

You can take the no-model approach and validate each navigation against it, so that for a non-existent path foo,


raises an error just as the model-driven version would have done. To your client application code, this implied object model looks no different to the generated one, even though (in a way) it doesn’t really exist! I like to think of this as the model-driven approach turned inside-out – instead of running code generated from a model, we run code that interprets a model.

If that metadata comes from the server (so much better than a build-time configuration step), you’re getting closer to that RESTful goal of application interaction driven by links, or

Hypertext as the engine of application state (HATEOAS) [5]

But before claiming this goal genuinely you will need to address a couple of issues that we’ve glossed over so far:

  1. how to generate URIs for collection members, for example turning users["dojo"] into http://example.com/users/dojo
  2. how to generate URIs for navigations like up and first

The cleanest answer to the first issue is the URI Template (a parameterised URI with a standardised syntax), sourced (naturally) from the server. With Fielding’s quote in mind it’s clear that the resolution to the second issue must lie also in server-provided information. In fact we’ll see both issues come together as we understand that the problem really isn’t one of describing URI structure but instead one of describing resource relationships and their manifestations in links.

URI Templates and Resource Templates

A URI Template that maps users["dojo"] into http://example.com/users/dojo might look this this:


The syntax isn’t that important (the draft standard looks set to change significantly), and yes, we’ve seen them here already without introduction. They’re pretty self-documenting and (with a good library implementation) easy to use. If you stick to a standard there’s nothing unRESTful about them – having your application create a URI from template is really no different from a browser creating a URI as the result of GET-based HTML form.

In more complex cases – in particular where there is a mixture of mandatory and optional parameters – it’s advantageous to wrap the URI Template in some basic metadata. WADL [6] is a well-supported example of this idea (well almost – see the comments [7]) ; another is my own Resource Template format [8] (actually more a logical schema than a format since it’s available in JSON, YAML and XML).

WADL and Resource Templates typically describe an entire application, each element describing a class of resource, its relationships, its URI template (so that relationships can be turned into links), parameters, supported HTTP methods and so on.

Integrated with Rails, Resource Templates are generated by the server at run-time from the application’s routes, information already available. Although (most typically) they can be generated and consumed for the whole application in one go, they can be generated for a single resource already pre-populated with that specific resource’s known parameters.

At last: links!

From those pre-populated, resource-specific Resource Templates we can generate links – as HTML or XML elements or as HTTP headers – that our client can consume and navigate, resource by self-described resource. Every resource becomes a potential entry point to a virtual application that can easily span multiple servers. Finally, we got there: true hyperlink-based client/server interaction, and where the framework support exists, for very little effort too.


In this article I have described the journey I travelled myself over a period of weeks following my decision to expose a complex existing application to scripting via HTTP. For me, technology-neutrality was a determining principle, and selecting HTTP and REST was an easy decision, first to make and then to sell. A proof of concept and a good demo based on a hand-crafted object model came quickly; metadata and templates were the result of some serious refactoring. Resource Templates came after I left the project, as did Rails integration (not a project requirement) and the path-to client in both Ruby and Python.

At the time of writing, link generation is supported on the server side but not yet on the client. The purist in me likes the fact that it is theoretically possible and will strive to keep it that way; the engineer in me is for the moment content with slurping up a whole application’s worth of metadata in one go and thereafter restricting interactions and payloads to the application level. Even as a purist, I see elegance in a simple metadata format, avoidance of hand-coded generation of URIs, respect for the power of HTTP and the orthogonality of formats; and if you believe as I do that RESTfulness lies in how clients and servers interact and not just in how server resources support the HTTP methods, I’ve probably succeeded in taking it further than most.


  1. Dynamically extending object behaviour in Ruby and Python [positiveincline.com]
  2. Programmatically adding methods to classes and objects: more Ruby/Python comparisons [positiveincline.com]
  3. path-to is born: nice client-side APIs to web applications in Ruby, also articles tagged path-to [positiveincline.com]
  4. path-to (Ruby) and path-to-py (Python) [github.com]
  5. Representational State Transfer (REST) (Chapter 5 of Architectural Styles and the Design of Network-based Software Architectures) [www.ics.uci.edu]
  6. Web Application Description Language [wikipedia.com]
  7. Partial template expansion in described_routes (comments)” [positiveincline.com]
  8. described_routes: hierarchical, framework-neutral and machine-readable descriptions of Rails routes [positiveincline.com], described_routes (Ruby) and described_routes-py (Python) [github.com]

June 25, 2009


Filed under: Web Integration — Tags: , , , , — Mike @ 1:49 pm

In my previous post I was casting around for an alternative to type as the name of a link header attribute, since in the standard this refers to a content type, not the “logical” type of the resource pointed to by the link.

I have decided to go with role, borrowed from XLink. I know that we’re not dealing with XML here and XLink isn’t wildly popular, but it’s documented, accepted and understood, so why reinvent the wheel?

Consistent with both XLink and the existing rel attributes, I’ve made these role attributes contain URIs, as in this example:

Link: <http://example.com/users/dojo>; rel="self"; role="http://example.com/described_routes#user",
           <http://example.com/described_routes/user>; rel="describedby"; role="http://example.com/described_routes#described_route",
           <http://example.com/described_routes/user?user_id=dojo>; rel="describedby"; role="http://example.com/described_routes#ResourceTemplate",
           <http://example.com/users>; rel="up"; role="http://example.com/described_routes#users",
           <http://example.com/users/dojo/edit>; rel="edit"; rel="http://example.com/described_routes/user#edit"; role="http://example.com/described_routes#edit_user",
           <http://example.com/users/dojo/articles>; rel="http://example.com/described_routes/user#articles"; role="http://example.com/described_routes#user_articles"

You can think of this in terms of entities and relationships in the ER model:


identifies a target entity (“user_articles” in this example, named after its Rails route) within the overall schema, and


identifies a relationship (“articles”) within the description of the source entity (“user”).

These URIs are globally unique, and even if they are never dereferenced by the client (i.e. the client isn’t interested in the ResourceTemplate metadata there – shame on them!), these values can be used by clients to select links reliably.


One of my sources of both good practice and inspiration has been Tim Berners-Lee’s Design Issues. I hadn’t got to the Metadata Architecture article until after drafting the above, and it’s quite gratifying to read it now.  Tim makes the ER comparison too, but rather than rehash that I will quote some of his key principles:

Metadata about one document can occur within the document, or within a separate document, or it may be transferred accompanying the document.

Link headers and link elements pointing to ResourceTemplate documents exemplify all three possibilities.

The URL space is an appropriate space for the definition of attribute names

Not only appropriate, but a lot more powerful than arbitrary tokens.  My initial plan to use simple names for rel and type attributes were definitely misguided.

As much as possible of the syntax and semantics should be able to be acquired by reference from a metadata document.

Within the scope of defining navigations around a web application I think this is achieved successfully.  A lot can be done by the client even without the metadata document, though clients will still need help in constructing URIs for members of collections.  I think there would a be place for the ResourceTemplate format (or something very much like it) even if there was a standard for URI Templates in link headers, but that’s a whole new article, perhaps my next one.

June 23, 2009

Link header dilemmas

Filed under: Web Integration — Tags: , , , , — Mike @ 12:07 pm

I received a very reasonable answer to my question (thank you Phil), but as a result of trawling through 12 months of archive came to realise that I’d misunderstood the guidance (or rather the relative lack thereof) regarding the use of the type link attribute.  The spec cites RFC 4287 The Atom Syndication Format and it appears that I have abused it by populating it with resource template names (sourced from Rails route names).  My fault entirely, lesson learned.

I’ll need to choose a more appropriate name for it therefore – either that or remove it altogether!  Possibilities:

  • name?  I quite like this, but is it too generic?
  • resourcename? Is this any better than name?
  • routename? Accurate when driven from Rails, but otherwise too implementation-specific I think
  • linkname? I quite link this too, but why not call the linkname on a link just name?
  • target?  Already taken, or at least too reminiscent of the target frame/window attribute.  Perhaps I’m on a better track here though? targettype maybe?

Aarrgghh!  We know that naming is hard; naming a name is doubly so.  I’ll sit on this for a while and hope that either inspiration or feedback comes…

June 19, 2009

Link headers, link elements, and REST

Filed under: Web Integration — Tags: , , , , , — Mike @ 3:14 pm

The house move looms (still no date yet, frustratingly) so I’ve had less time for programming (or blogging for that matter!).  I do however have an experimental and unreleased enhancement to the Ruby version of described_routes that generates link elements (for the <head> section of your html) or the yet-to-be-standardized link headers completely automatically.  Integrating them into your Rails application requires just a one-liner per layout or controller (or you can do all controllers in one go).  Drop me a line if you would like to play with it.

I have one question outstanding on the new spec (see it here on the ietf-http-wg list archives) but I have to say that I’m pretty happy with it all.  It takes the ideas of Partial template expansion in described_routes and adds a standardised way to relate content and metadata, making it even easier clients to navigate web applications without necessarily slurping up the entire site’s schema in one go.  It allows interaction to be 100% RESTful (HATEOAS and everything), and even if you prefer not to go that way as a client you can at least be confident that the application will be transparent, self-documenting and well-behaved.

May 29, 2009

New on github: described_routes-py

Filed under: Web Integration — Tags: , , , , — Mike @ 6:51 pm

described_routes-py – a Python port of the core metadata classes ResourceTemplate and ResourceTemplates.  It’s been fun learning Python and I’m pleased with the result – it looks cleaner somehow and I’m seriously considering treating this as the reference version.

I need to add support for partial template expansion to Joe Gregorio’s URI Template parser (to be found here) and add some industrial-strength tests. I’m putting off XML support (perhaps indefinitely), and coming round to the conclusion that I needn’t add explicit JSON and YAML support as it’s trivially easy – using your favourite JSON or YAML module – to go to or from the hash and dict representations already supported.   So that’s the core of it done really.

That leaves client and server integration. Unless I’m deluding myself, I think my Python is up to a port of the path-to client, though it will need some different metaprogramming tricks to the Ruby version. On the server side, I have the challenge of learning Django without (still!) knowing whether its routing datastructures are open to the same treatment I gave to Rails. I guess it will be fun trying! Worst case would be some hand crafting, not too terrible.

May 25, 2009

Next up: Python!

Filed under: Web Integration — Tags: , , — Mike @ 5:15 pm

I’ve recently read and enjoyed Mark Pilgrim’s Dive into Python and I’m now about half way (i.e. 500 pages!) through Wesley Chun’s Core Python Programming.

The next stage in my Python education will be to port ResourceTemplate, the metadata core of described_routes. It would be nice also to replicate for Django the Rails parts of described_routes, but I don’t know yet how feasible this will be. Thoughts, anyone?

Powered by WordPress