Positive Incline Mike Burrows (@asplake) moving on up, positively

February 18, 2010

PathTo for Python gets a JSON-capable HTTP client

Filed under: Web Integration — Tags: , , , , , , , , — Mike @ 1:55 pm

…and it works!

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')
>>> app.login.post(credentials, expected_status=302)
>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}
>>> product = app.products['Foo'].get(expected_status=200).parsed
>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)
>>> app.products['Foo'].get(expected_status=200).parsed['description']
u'Updated!'

OK, so it wasn’t really “example.com”, and the updated product wasn’t called “Foo”, but the rest is for real.

In stages:

0) The Pylons-based server has the JSON-capable @validate (un)decorator of my previous post and DescribedRoutesMiddleware from DescribedRoutes installed in its wsgi stack.

1) Create a client-side proxy to the app. Following link headers published by the app, it finds the ResourceTemplates description, which it retrieves in JSON format.

>>> import path_to
>>> app = path_to.open_app('http://example.com', format='json')

The format='json' is a small red herring here, it’s simply remembered for later.

2) Log in by posting credentials, expecting a 302 status (a failed attempt would return a 200 and the validation errors in the body). The client handles cookies automatically behind the scenes.

>>> app.login.post(credentials, expected_status=302)

3) View a friendly representation of the metadata (or rather the part that relates to products):

>>> print app.products.resource_template
products      products     GET, POST http://example.com/products{.format}
  new_product new_product  GET       http://example.com/products/new{.format}
  {product}   product      GET, PUT  http://example.com/products/{product}{.format}
    edit      edit_product GET       http://example.com/products/{product}/edit{.format}

4) Get a product identified by ‘Foo’ and return the JSON payload parsed into a Python dict.:

>>> product = app.products['Foo'].get(expected_status=200).parsed

Behind the scenes it has expanded the “http://example.com/products/{product}{.format}” template seen previously into “http://example.com/products/Foo.json”, using the remembered format parameter and the supplied ‘Foo’ key which is assumed to correspond to the required product parameter.

5) Update the local product representation and send it back (it gets converted back to JSON along the way):

>>> product['description'] = 'Updated!'
>>> app.products['Foo'].put(product, expected_status=302)

6) Finally, demonstrate that it was successful!

>>> app.products['Foo'].get(expected_status=200).parsed['description']
u'Updated!'

February 13, 2010

Experimental: Pylons validation (un)decorator with JSON support

Filed under: Web Integration,Work — Tags: , — Mike @ 12:24 pm

In this gist (it’s not even a patch, just some stuff I keep in my app’s lib/base.by) is a refactored @validate.

On the surface it’s just the same as the standard pylons.decorators.validate:

    @validate(form='edit')
    def update(self):
        ...

but it has been broken up into a number of controller methods, each of which is usable directly. So if you find the decorator inflexible, you can write your actions in the undecorated form

    def update(self):
        try:
            self._parse(request, schema=MyForm())
        except Invalid as e:
            return self._render_invalid(e, form='edit')
        ...

It’s more than just a stylistic change; in addition to the usual HTML form data it will also accept POST data sent in JSON and send back any validation errors in JSON too if the client so desires. The presence of JSON is recognized by an sent_json() helper that looks to see if request’s route had a format parameter set to ‘json’ (see previous post) or if the Content-Type header is ‘application/json’; responses work similarly via an accepts_json() helper, but checking the Accept header.

Throw in a render_json() helper and here are idioms for actions that will quite happily accept and produce JSON or HTML. First the traditional, decorated look:

    @validate(form='edit')
    def update(self):
        ...
        if accepts_json():
            return render_json(a_json_serialisable_thing)
        else:
            return render('template')

And undecorated:

    def update(self):
        try:
            self._parse(request, schema=MyForm())
            ...
        except Invalid as e:
            return self._render_invalid(e, form='edit')
        ...
        if accepts_json():
            return render_json(a_json_serialisable_thing)
        else:
            return render('template')

The undecorated form is slightly longer — potentially addressed by sharing exception handling controller-wide — but it offers the intriguing opportunity to move some validation to the model &emdash; any Invalid exceptions raised in the try block will be rendered appropriately, regardless of where they came from.

February 10, 2010

Experimental {.format} in Routes

Filed under: Web Integration — Tags: , , , — Mike @ 7:34 pm

A while back I raised on the URI Template mailing list the idea of a {.var} syntax that generates optional path extensions, “.html”, “.json”, etc.

For example, given a path template of /releases/{id}/notes{.format} and an id parameter of “1”, paths of /releases/1/notes or /releases/1/notes.pdf will be generated depending on whether the supplied format parameters are undefined or “pdf” respectively.

It’s not a completely original idea.  Rails has a (.:format) syntax that DescribedRoutes maps to the flexible but somewhat clunky {-prefix|.|format} syntax of the old draft URI Template spec. The proposed syntax seems more in keeping with current draft (very much a work in progress) though.

Today my Routes fork gained support for the syntax and tweaks to the new SubMapper helpers so that duplicate “formatted routes” aren’t generated (something that happened to Rails a long time ago).

Example config (<code>config/routing.py</code> in your Pylons app):

with mapper.collection(
                'releases',
                'release',
                requirements={'id': 'd+'}) as c:
    c.member.link(rel='notes', name='release_notes')

And in the paster shell:

>>> print mapper
Route name     Methods Path
releases       GET     /releases{.format}
create_release POST    /releases{.format}
new_release    GET     /releases/new{.format}
release        GET     /releases/{id}{.format}
update_release PUT     /releases/{id}{.format}
delete_release DELETE  /releases/{id}{.format}
edit_release   GET     /releases/{id}/edit{.format}
release_notes  GET     /releases/{id}/notes{.format}
>>> url('release_notes', id=1, format='pdf')
'/releases/1/notes.pdf'
>>> url('release_notes', id=1)
'/releases/1/notes'
>>> mapper.match('/releases/1/notes')
{'action': u'notes', 'controller': u'release', 'id': u'1', 'format': None}
>>> mapper.match('/releases/1/notes.pdf')
{'action': u'notes', 'controller': u'release', 'id': u'1', 'format': u'pdf'}

Just add formatted=False to your collection() call if you don’t want this new behaviour.

I haven’t sent the Routes guys a pull request yet. I’d like some feedback first, and meanwhile I will try to come up with a nice helper that takes the format extension to override content negotiation. My app has some basic header-based conneg already but I would like the convenience of being able to override it in the URL.

December 28, 2009

DRY up your routes – UPDATES

Filed under: Web Integration — Tags: — Mike @ 11:09 am

Two updates to the previous post:

Update 1: Routes fork on bitbucket

See http://bitbucket.org/asplake/routes/. I’ve sent a pull request with my changes (the prettyprinter included), but no response yet and I’m delighted to say that Ben has now incorporated them in the base Routes repo so expect to see them soon in Routes 1.12.

Update 2: Tidier collection() usage

The end-state of the last post

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

can be written as

    with mapper.collection(
                    'releases',
                    'release',
                    requirements={'id': 'd+'}) as c:
        c.member.link(rel='notes', name='release_notes')

Losing the with syntax, a vanilla collection resource with integer id’s looks like this:

    mapper.collection(
                'releases',
                'release',
                requirements={'id': 'd+'})

This applies requirements to the collection-level routes as well as the member-level routes but the overhead (unquantified) must be very small and I prefer this tidier form.  It would be easy enough though to provide a member_requirements argument if the performance difference proves to be significant.

December 21, 2009

DRY up your routes – a Pylons routing refactoring

Filed under: Programming,Web Integration — Tags: , , , , , — Mike @ 11:43 am

[See UPDATES]

This post is in several acts, each one a refactoring. Taking the stage will be some familiar-looking application code; behind the scenes lurks some enhanced framework code that makes the refactorings possible.

Our story starts with an old-school (pre-REST) routing configuration. It’s in Python, for Pylons and other Routes-based web frameworks, but let me reassure the Ruby audience that Routes borrows very heavily from Rails and their routing configurations look quite similar.

We will finish with collection(), an experimental (but I hope worthy) alternative to resource(), the routing helper that rose to prominence when Rails first made its big push to REST.

The starting point: textbook routing.py

from routes import Mapper

def make_map():

    ...

    # Releases - Collection
    mapper.connect('releases', '/releases',
              controller='release', action='index', conditions=dict(method='GET'))
    mapper.connect('create_release', '/releases',
              controller='release', action='create', conditions=dict(method='POST'))
    mapper.connect('new_release', '/releases/new',
              controller='release', action='new', conditions=dict(method='GET'))

    # Releases - Members
    mapper.connect('release', '/releases/{id}',
              controller='release', action='show',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('update_release', '/releases/{id}',
              controller='release', action='update',
              requirements=dict(id='d+'), conditions=dict(method='PUT'))
    mapper.connect('delete_release', '/releases/{id}',
              controller='release', action='delete',
              requirements=dict(id='d+'), conditions=dict(method='DELETE'))
    mapper.connect('edit_release', '/releases/{id}/edit',
              controller='release', action='edit',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('release_notes', '/releases/{id}/notes',
              controller='release', action='release_notes',
              requirements=dict(id='d+'), conditions=dict(method='GET'))

Full of duplication, verbose, even ugly. Just because it sits in a config directory, does it have to look that bad?

Refactoring 1: Introducing submapper()

New in Routes 1.11 (so really quite new), submapper()provides the means to pull out parameters previously shared across multiple connect() calls.

One particularly nice feature is that the SubMapper objects returned by submapper() support the Python managed object protocol so if you’re on Python 2.5 or above you can use them with the with syntax like this:

from release_tool.lib.mapper import Mapper

def make_map():

    ...

    # Releases - Collection
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))

    # Releases - Members
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases/{id}',
                    requirements=dict(id='d+')) as m:
        m.connect('release', '', action='show',
                conditions=dict(method='GET'))
        m.connect('update_release', '', action='update',
                conditions=dict(method='PUT'))
        m.connect('delete_release', '', action='delete',
                conditions=dict(method='DELETE'))
        m.connect('edit_release', '/edit', action='edit',
                conditions=dict(method='GET'))
        m.connect('release_notes', '/notes', action='release_notes',
                conditions=dict(method='GET'))

That’s a definite improvement (and credit where credit is due – submappers are great innovation, though a small bug requires our local extensions to be imported here), but notice that there’s still some duplication between the two submappers blocks, the first one corresponding to the collection resource, the second to that same collection’s members. Wouldn’t it be cool if we could nest them?

Refactoring 2: Submapper nesting

On the surface, the Routes 1.11 API doesn’t appear to support submapper nesting, but the SubMapper objects do indeed nest. Adding a submapper() method to SubMapper is a trivial change, and it take only minor internal tweaks for deeper nestings to function correctly.

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.connect('release', '', action='show',
                    conditions=dict(method='GET'))
            m.connect('update_release', '', action='update',
                    conditions=dict(method='PUT'))
            m.connect('delete_release', '', action='delete',
                    conditions=dict(method='DELETE'))
            m.connect('edit_release', '/edit', action='edit',
                    conditions=dict(method='GET'))
            m.connect('release_notes', '/notes', action='release_notes',
                    conditions=dict(method='GET'))

We seem to be on to something here, but what about the repetition (in one guise or another) of the resource name “release”? And what’s with those strange empty string ('') parameters?

Refactoring 3: Links and actions

To fix the repetition issue it’s clear that we need helpers that will generate those connect() calls for us more intelligently. But before we do that, let’s recognise that there are really two kinds of routes being generated here:

  1. Links to singleton subresources that support only one HTTP method, the most ubiquitous examples being the new and edit representations for which GET is the applicable HTTP method, but special subresources that are the targets for POST actions are common too;
  2. Actions that correspond directly to HTTP methods: index and create for GET and POST[1] on collection resources, show, update, delete for GET, PUT, DELETE on member resources.

It’s that second type that give rise to those strange empty strings as there is no further navigation to be done relative to the target resource.

So let’s add link() and action() helpers:

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.action(action='index', name='releases')
        c.action(action='create', method='POST')
        c.link('new')
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.action(action='show', name='release')
            m.action(action='update', method='PUT')
            m.action(action='delete', method='DELETE')
            m.link(rel='edit')
            m.link(rel='notes', name='release_notes')

Things are still moving in the right direction, but given that most of those links and actions follow a very well-worn convention, how about specific helpers for those?

Refactoring 4: More helpers

I’ll be honest – this is just an intermediate stage. You can see where we’re headed now though:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.index()
        c.create()
        c.new()
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.show()
            m.update()
            m.delete()
            m.edit()
            m.link(rel='notes', name='release_notes')

Refactoring 5: Parameter-driven generation

We’ll be using those same names a lot across our resources, so let’s just identify the ones we want in a list:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases',
                    actions=['index', 'create', 'new']) as c:
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+'),
                    actions=['show', 'update', 'delete', 'edit']) as m:
            m.link(rel='notes', name='release_notes')

But even this pattern will be repeated across resources, so let’s define a collection() helper that does it all:

Refactoring 6: Using the collection() helper

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

And we’re done.

Let’s compare this to the equivalent resource() call:

    # Releases
    mapper.resource(
                'release',
                'releases',
                controller='release',
                # no requirements!
                member = {'notes':'GET'})

Superficially similar, but I’m unable to specify requirements on member paths and I don’t get to name my custom route either – the generated name “notes_release” just isn’t right! The bottom line is that resource() is approaching a dead end; its only way forward is to make its parameters ever more complex.

So… how about we put resource() on notice and bring in something more flexible and – dare I say – Pythonic? And wouldn’t it look just as nice in Ruby too? Answers on a postcard please…

December 15, 2009

Prettyprinter for Pylons routes

Filed under: Uncategorized,Web Integration — Tags: , , — Mike @ 7:27 pm

No, not described_routes, just something basic to plug a gap. For those suffering Rails-envy there’s no “paster routes” command, but it’s easy enough to use in the Paster shell:

bash-3.2$ paster shell test.ini
Pylons Interactive Shell
Python 2.6.2 (r262:71605, Apr 14 2009, 22:40:02) [MSC v.1500 32 bit (Intel)]

  All objects from encore.lib.base are available
  Additional Objects:
  mapper     -  Routes mapper object
  wsgiapp    -  This project's WSGI App instance
  app        -  paste.fixture wrapped around wsgiapp

>>> print mapper
Route name   Methods Path
                     /error/{action}
                     /error/{action}/{id}
home         GET     /
things       GET     /things
create_thing POST    /things
new_thing    GET     /things/new
thing        GET     /thing/{id}
...

It works by subclassing routes.Mapper (I have other reasons to do this which I’ll explain soon) and you’ll need to use this new Mapper in your config/routing.py.

That should be all you need to know. The code for the new Mapper class is below. Note: That second from last join() isn’t coming out very well – it has a backslash and an ‘n’ in it – honest!

Enjoy!

import routes.mapper

class Mapper(routes.Mapper):
    #
    # Pretty string representation - returns a string formatted like this:
    #
    #    Route name   Methods Path
    #                         /error/{action}
    #                         /error/{action}/{id}
    #    home         GET     /
    #    things       GET     /things
    #    create_thing POST    /things
    #    new_thing    GET     /things/new
    #    thing                  GET     /thing/{id}
    #
    #    etc
    #
    # Enter 'print mapper' in the paster shell to use it, or (TBD) run it via a
    # configured paster command.
    #
    def __str__(self):
        def format_methods(r):
            if r.conditions:
                method = r.conditions.get('method', '')
                return method if type(method) is str else ', '.join(method)
            else:
                return ''

        table = [('Route name', 'Methods', 'Path')] + [
            (
                r.name or '',
                format_methods(r),
                r.routepath or''
            )
            for r in self.matchlist]

        widths = [
            max(len(row[col]) for row in table)
            for col in range(len(table[0]))]

        return '
'.join(
            ' '.join(
                    row[col].ljust(widths[col])
                    for col in range(len(widths))
                ).rstrip()
            for row in table)

November 5, 2009

Nested forms in Pylons

Filed under: Web Integration — Tags: , — Mike @ 1:10 pm

Easy when you know how, but finding that out was struggle – at least it was for me! The crucial piece of information I was lacking was the correspondence between variabledecode.variable_encode() which flattens nested values into an array and variabledecode.NestedVariables which does the reverse.

I have factored these lines below out to my lib/base.py. Now all my form schemas (nested or not) inherit from BaseSchema and I use fill_render() whenever I want to render html with values.

import formencode
from formencode import htmlfill
from formencode import variabledecode

def fill_render(template_name, values):
    """
    Render a template filled with values.  Values may be nested.
    """
    return htmlfill.render(
                    render(template_name),
                    variabledecode.variable_encode(values))

class BaseSchema(formencode.Schema):
    """
    Base form schema.  Any nested forms will be converted to nested values automatically.
    """
    allow_extra_fields = True
    filter_extra_fields = True
    pre_validators = [variabledecode.NestedVariables()]

It leads to DRYer controller code, reduces the number of imports required and hides some complexity. Can’t understand why it isn’t the default already in Pylons, but hey, it works.

To see nested forms in action, go to the interesting but (at the time of writing) not quite complete Solving the Repeating Fields Problem (Pylons Book v1.1 documentation).

Credit to Ian Wilson for pointing me in the right direction in this thread on pylons-discuss.

Powered by WordPress