Manifesto and roadmaps for described_routes and path-to

Introduction

Clients of RESTful web applications typically use prior knowledge of the target application’s structure to generate URIs.  This approach is often very convenient, but much of this URI generation is hard-coded, and (worse) spread across client code. This introduces a high degree of coupling and makes clients unnecessarily vulnerable to server-side change.

Steps to improve this situation:

  1. In clients, centralise the generation of URIs and make the process driven by configuration data
  2. Have servers publish the required configuration data – i.e. application metadata – in a readily understood format

path-to provides the means for client applications to model web applications in terms of logical structure and URI mappings, and to interact with them through dynamically-generated application-specific APIs.  described_routes supports an application metadata structure (published in JSON, YAML and XML formats) that can be consumed by path-to, and (helpfully) generates it automatically online or offline from the routes configured for a Rails-based application.

The two libraries can be used separately or together – an JavaScript client is under independent development for example.  Moreover, the underlying metadata format is framework-neutral; we have been careful not to “leak” Rails concepts into it.

1. described_routes

“Framework-neutral metadata describing Rails routes in JSON, YAML, XML and plain text formats”

The project has three main elements:

  1. Rake tasks that produce representations of a Rails application’s entire structure offline
  2. A simple controller that supports online access to these representations, adding the ability to pick out structures rooted on specified named routes
  3. The ResourceTemplate class and the formats that it generates and parses.    This is completely independent of Rails, in terms of both code dependencies and data structure.

The ResourceTemplate format describes a hierarchical organisation of web resources together with their:

  • logical names (their route names in the Rails case, e.g. “user_articles”)
  • logical parent-child relationships (e.g. “edit”, for a resource’s edit form)
  • URI Templates and the names of the parameters required to populate them
  • supported HTTP options (GET, PUT, POST, DELETE)

These metadata representations are derived automatically from the routes description that is conventionally coded in config/routes.rb.  See the appendix at the end of this article for examples.

Current status:

  • Working, published (as a github repository and as an installable rubygem on RubyForge)
  • To my knowledge it has been tested only by myself (on some of the examples given in the boilerplate config/routes.rb)

Still outstanding:

  • Parsing of the XML representation (and tests of round trips with the JSON and YAML formats)
  • The offline rake tasks are unaware of the application’s base URI and therefore can only produce templates to paths.  Find a way to get the base URI for a Rails environment offline (I’m not even sure this is possible), or allow one to be provided as a parameter. UPDATE (May 6th): added parameter “BASE=http://…” to rake tasks

Possible future work:

  • Solicit real-world examples for testing, unusual ones especially!  All I would need are config/routes.rb files and willingness to review/discuss the resulting representations
  • Break out the ResourceTemplate class into its own gem
  • Generate ResourceTemplate representations for specific resource instances (where the URI templates of child resources are partially populated with the root instance’s URI parameters), facilitating a more dynamic (and arguably more RESTful) interaction style
  • Generate URI Template mappings for query parameters
  • Implementations for Ruby web frameworks other than Rails.  Where frameworks lack the routing metadata of Rails this will entail work on the part of the application developer however.
  • Non-Ruby implementations UPDATE (June 2nd) basic Python ports done
  • Investigate alternatives to URI Templates, make this a pluggable choice
  • Add customisation hooks (@siebertm): UPDATE (May 6th): done

2. path-to

“Model web apps easily and access them via nice app-specific Ruby APIs”

path-to predates described_routes, and with a modest level of coding effort it can be used without it (see this early example). From the beginning I had however hoped to find a suitable format already in existence but with no real success so far!  With described_routes, path-to provides a Ruby client API to a Rails application for almost no effort, as shown in this more recent example.

Current status:

Still outstanding:

  • Demonstrations of access to well known web APIs – realistic examples with authentication etc.  UPDATE (May 8th): added Delicious example.

Possible future work:

  • positional parameters, allowing (for example) users['dojo'] as well as users['user_id' => 'dojo'] UDPDATE (May 12th): done
  • Support other metadata representations as they are brought to my attention (are there any good ones out there?)
  • Port to Python; demonstrate easy Python/Rails interop UPDATE (June 2nd) basic Python ports done

Appendix – examples

Nested path-to/described_routes and HTTParty

I’ve finished the support for nested described_routes in path-to.  Example:

  require 'path-to/described_routes'
  app = PathTo::DescribedRoutes::Application.new(
      :json => Net::HTTP.get(URI.parse("http://example.com/described_routes.json")))

  app.users["user_id" => "dojo"].articles.recent
  #=> http://example.com/users/dojo/articles/recent
  app.users["user_id" => "dojo"].articles.recent.get
  #=> "<html>...</html>"

Or if you still prefer the named route:

  app.recent_user_articles("user_id" => "dojo").get   #=> "<html>...</html>"

And that’s it!  Is that a client-side API for free or what?  This just has to be better than hard-coded URIs (ugh).

I have also enhanced the Rails controller in described_routes so that you can get the JSON, YAML or XML for a given named route.  In this example we see HTTParty (path-to’s default HTTP client) parsing a JSON response (which I’ve reformatted just a little bit):

  app.described_route("route_name" => "admin_product", "format" => "json").get
  #=> {"name"=>"admin_product", "options"=>["GET", "PUT", "DELETE"],
       "resource_templates"=>[
          {"name"=>"edit_admin_product", "options"=>["GET"],
           "uri_template"=>"http://localhost:3000/admin/products/{product_id}/edit{-prefix|.|format}",
           "path_template"=>"/admin/products/{product_id}/edit{-prefix|.|format}", "rel"=>"edit",
           "optional_params"=>["format"], "params"=>["product_id"]
          }],
       "uri_template"=>"http://localhost:3000/admin/products/{product_id}{-prefix|.|format}",
       "path_template"=>"/admin/products/{product_id}{-prefix|.|format}",
       "optional_params"=>["format"], "params"=>["product_id"]}

One last thing: as previously mentioned you’ll need grab the latest addressable from github – the published gem is a fix or two behind.

described_routes at run time

The first described_routes gem went up on RubyForge yesterday.  Usual install:

  sudo gem install described_routes

then follow these very simple instructions to have your routes described in JSON, YAML (long and short versions) or XML from the command line.

Today I added a Rails controller so that you can GET any of these formats from your server at runtime.  This takes just two 1-line changes:

1) Add to your project a file <railsapp>/initializers/described_routes.rb containing the following:

  require 'described_routes/rails_controller'

2) Add this new route to your config/routes.rb file:

  map.resources :described_routes, :controller => "described_routes/rails"

You can then browse to <yourapp>/described_routes.json (or .yaml or .xml).  For the shorter yaml description, navigate to <yourapp>/described_routes.yaml?short=true.

Without a format extension you’ll get an error unless you provide a suitable HTML template, perhaps a suitable place for some human-friendly API documentation?

As an added bonus, reports generated at run time include full URI templates (I haven’t yet found out how to get to the application’s root url from Rake)

Examples:

I’ll be getting back to the path-to / described_routes integration after this.  Getting nested routes working shouldn’t be a huge task (separately, both libraries support them already) but there’s some refactoring to be done.

path-to and described_routes playing nicely – but is it RESTful?

I started to get path-to and described_routes to play together today.  With them I can:

1) Load a JSON or YAML description of an application into a generic client app like this:

  app = PathTo::DescribedRoutes::Application.new(
      DescribedRoutes.parse_json(json),
      "http://localhost")

2) See what it has to offer:

  app.resources_by_name.keys.sort.inspect
  #=> ["admin_product", "admin_products", "edit_admin_product", "edit_page",
  #    "edit_user", "edit_user_article", "edit_user_profile", "new_admin_product",
  #    "new_page", "new_user", "new_user_article", "new_user_profile", "page",
  #    "pages", "recent_user_articles", "root", "summary_page",
  #    "toggle_visibility_page", "user", "user_article", "user_articles",
  #    "user_profile", "users"]

3) Generate some URIs:

  app.users.uri
  #=> "http://localhost/users"
  app.recent_user_articles[:user_id => "dojo"].uri
  #=> "http://localhost/users/dojo/articles/recent"

4) And access them:

  app.users.get
  app.recent_user_articles[:user_id => "dojo"].get

OK I lied about that last bit – my server app is no more than Rails boilerplate with described_routes and an example config/routes.rb, but (thanks to the HTTP support in path-to provided by HTTParty) it would have worked, I promise!

Next Steps

Next up is properly integrating the chaining stuff in path-to.  I want that last example to look more like

  app.users[:user_id => "dojo"].articles.recent.get

Is it REST?

stilkov: @christo4ferris There’d only be HATEOAS in that REST API if the URI templates were retrieved from the server dynamically

Indeed, and I make no great claims that this style of interaction is 100% RESTful, though when you look again at

  app.users[:user_id => "dojo"].articles.recent.get

can you be sure whether it is or it isn’t?  Who knows what’s going on behind the scenes?  The point really is that given some minimal discoverability in the server, the predictability of the REST and a good scripting language, tricks like this are easy to pull off.  And that’s way better than the highly coupled client-side URI generation splattered all over many of the so-called RESTful examples I see out there on the web.

If all of that sounds defensive, be assured that I’d be only too delighted to see this superseded by something of similar power that has more universal acceptance.  As far as I can tell though, right now that’s a little way off.

described_routes: hierarchical, framework-neutral and machine-readable descriptions of Rails routes

I’ve pushed to github (rubyforge approval is still pending) a new gem described_routes which produces machine-readable (JSON, YAML, XML and Ruby) representations of the resources supported by your Rails app.

It’s very easy to use: just drop this into your project’s Rakefile:

  require 'tasks/described_routes'

Then see:

  $ rake --tasks described_routes
  rake described_routes:json        # Describe resource structure in JSON format
  rake described_routes:ruby        # Describe resource structure as a Ruby literal
  rake described_routes:xml         # Describe resource structure in XML format
  rake described_routes:yaml        # Describe resource structure in YAML format
  rake described_routes:yaml_short  # Describe resource structure in YAML format (basic structure only)

To get a flavour, check out the short form below. See how (1) the resource hierarchy is represented, and (2) (with the “rel” attributes) it has picked out named relationships between parent and child resources:

  $ rake described_routes:yaml_short
  - name: users
    path_template: /users{-prefix|.|format}
    resources:
    - name: user
      path_template: /users/{user_id}{-prefix|.|format}
      resources:
      - name: user_articles
        rel: articles
        path_template: /users/{user_id}/articles{-prefix|.|format}
        resources:
        - name: user_article
          path_template: /users/{user_id}/articles/{article_id}{-prefix|.|format}
          resources:
          - name: edit_user_article
            path_template: /users/{user_id}/articles/{article_id}/edit{-prefix|.|format}
            rel: edit
        - name: new_user_article
          path_template: /users/{user_id}/articles/new{-prefix|.|format}
          rel: new
        - name: recent_user_articles
          path_template: /users/{user_id}/articles/recent{-prefix|.|format}
          rel: recent
      - name: edit_user
        path_template: /users/{user_id}/edit{-prefix|.|format}
        rel: edit

The longer forms include “options” (the HTTP methods applicable to each resource) and parameter lists.

The path templates are URI Templates but relative to the application’s base. If someone can show me how to derive that base URI at Rake time, please let me know!

Do please note that there’s nothing Rails-specific about these formats. Although they’re potentially useful as project documentation, they’re really for the benefit of client applications, and clients of course shouldn’t care how their respective servers were built!  My next step is to add client support for one or more of these formats in my path-to gem (announcement, github, rubyforge).

path-to is born: nice client-side APIs to web applications in Ruby

An initial cut of path-to is now on github.  There’s no gem on Rubyforge just yet but it’s not far away.

Path-to’s basic premise is that it should be easy to describe client interfaces to typical web applications (RESTful ones especially), and have created for you a Ruby API, even in the absence of a discoverable representation of the web API on the server side. My longer term aim is to identify convenient discovery protocols, have them implemented in the server apps I have influence over, and (naturally) include support for them in path-to.

A very simple example (expanded in the README) looks like this:

    require "path-to"

    app = Application.new(
            :users    => "http://example.com/users/{user}",
            :articles => "http://example.com/users/{user}/articles/{slug}")

    app.users                                                   #=> http://example.com/users/
    app.users(:user => "dojo")                                  #=> http://example.com/users/dojo
    app.users[:user => "dojo"]                                  #=> http://example.com/users/dojo
    app.articles(:user => "dojo", :slug => "my-article")        #=> http://example.com/users/dojo/articles/my-article
    app.users[:user => "dojo"].articles[:slug => "my-article"]  #=> http://example.com/users/dojo/articles/my-article

A key theme of path-to is chaining, so that if app.x represents a resource, then so (typically) do app.x.y and app.x[params]. But you’re not forced into this style: you could go straight for app.x(params) which involves one less object under the covers. Furthermore, you have control over the classes of the objects created, and are free to override #[] so that you can (for example) have calls like

    app.users["dojo"].articles["my-article"]                    #=> http://example.com/users/dojo/articles/my-article

Another important theme is that client applications shouldn’t be forced to follow my preferences and dependencies. For example, I’m testing it with HTTParty (so that app.x.get is delegated to HTTParty.get), but you don’t have to. Similarly, it’s easy to swap out the URI Template support I’m getting from Addressable – which is just as well, as I’m entertaining serious second thoughts about the whole URI Templates thing!

URI templates in Ruby, discovery concerns, looking at XRD

We’ve been adding RESTful interfaces to an existing distributed app to give it some much-needed scriptability.  I was able to cook up a client very easily in Ruby, using URI templates to describe the server API.  This approach got us up and running nice and quickly and I was happy enough with it, but I would have been happier still if it didn’t feel so, well, homebrewed.

Among my readings on REST and answering some of my concerns over our app’s discoverability, this very interesting post from Dare Obasanjo on REST API Design pointed me to the XRD initiative.  It’s covered in multiple posts on the excellent Hueniverse, of which this XRD Sneak-Peek is a good place to start.

XRD seems to have everything I’m looking for in a standardised web API description but for this one issue: given multiple templates that map to resources of the same type, how do I choose the the one most appropriate to the the set of parameters I have in my client?  For example, my blog might support two:

  http://positiveincline.com/articles/{slug}

http://positiveincline.com?p={id}

Having the server prioritise these (as provided for in XRD) makes no sense in this instance; the client needs to be able to choose.  How about then have some way of marking template variables as mandatory, like {+slug} perhaps?  Thoughts, anyone?

Back to Ruby meanwhile, may I recommend sporkmonger’s Addressable gem, an improved URI class with template support.  My first foray into open source contribution resulted with a minor fix to it being accepted today.  Thanks to the amazing github, this transaction was concluded in a matter of minutes!

All of this is one step along the way towards a new gem (tentatively “path-to”, registered on rubyforge) which will make it very easy to describe web applications on the client side in Ruby, and later to support discovery mechanisms such as XRD.  There are some corporate hoops to jump through before I can take this further, so watch this space.

P.S. XRD gives me hope that URI templates might still be “officially” alive, despite the expiry of the draft 3 spec.  Is there anyone out there making this so?  A lot of googling and trawling through w3c archives later and I’m still not sure.