Positive Incline Mike Burrows (@asplake) moving on up, positively

January 1, 2010

Fun with parameter-collecting proxies

Filed under: Programming — Tags: — Mike @ 1:13 pm

Further inspired by the Routes submapper idea (see DRY up your routes) here’s a generic proxy object that collects arguments repeated across method calls:

class Params(object):
    def __init__(self, obj, *args, **kwargs):
        self.__obj = obj
        self.__args = args
        self.__kwargs = kwargs

    def __getattr__(self, name):
        try:
            attr = getattr(self.__obj, name)
            if callable(attr):
                # Target attr is callable; return another that
                # will sneak in our remembered args
                def wrapped_method(*args, **kwargs):
                    return attr(*(self.__args + args),
                                **dict(self.__kwargs, **kwargs))
                return wrapped_method
            else:
                # Plain old attribute
                return attr
        except AttributeError:
            # self.foo(bar, *args, **kwargs)
            # -> scope(self, foo=bar, *args, **kwargs)
            def wrapped_scope(*args, **kwargs):
                return Params(self, *args[1:],
                              **dict({name: args[0]}, **kwargs))
            return wrapped_scope

    def __enter__(self):
        return self

    def __exit__(self, type, value, tb):
        pass

params = Params

You can use it submapper-style (with or without the ‘with‘ syntax), collecting arbitrary args and kwargs as follows:

with params(mapper, controller='entries') as entries:
    entries.connect(...)
    entries.connect(...)
    ...

Here controller='entries' gets included in each method call to the underlying mapper.

Equivalently, it supports a “fluent” or DSL-ish style where named parameters are transfomed into (chainable) methods:

m = params(mapper)
with m.controller('entries') as entries:
    entries.connect(...)
    entries.connect(...)
    ...

Of course the Routes API (in the dev version at least) already has submappers and param makes a poor substititute, but as tool for reducing duplication elsewhere it may yet prove useful.

December 21, 2009

DRY up your routes – a Pylons routing refactoring

Filed under: Programming,Web Integration — Tags: , , , , , — Mike @ 11:43 am

[See UPDATES]

This post is in several acts, each one a refactoring. Taking the stage will be some familiar-looking application code; behind the scenes lurks some enhanced framework code that makes the refactorings possible.

Our story starts with an old-school (pre-REST) routing configuration. It’s in Python, for Pylons and other Routes-based web frameworks, but let me reassure the Ruby audience that Routes borrows very heavily from Rails and their routing configurations look quite similar.

We will finish with collection(), an experimental (but I hope worthy) alternative to resource(), the routing helper that rose to prominence when Rails first made its big push to REST.

The starting point: textbook routing.py

from routes import Mapper

def make_map():

    ...

    # Releases - Collection
    mapper.connect('releases', '/releases',
              controller='release', action='index', conditions=dict(method='GET'))
    mapper.connect('create_release', '/releases',
              controller='release', action='create', conditions=dict(method='POST'))
    mapper.connect('new_release', '/releases/new',
              controller='release', action='new', conditions=dict(method='GET'))

    # Releases - Members
    mapper.connect('release', '/releases/{id}',
              controller='release', action='show',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('update_release', '/releases/{id}',
              controller='release', action='update',
              requirements=dict(id='d+'), conditions=dict(method='PUT'))
    mapper.connect('delete_release', '/releases/{id}',
              controller='release', action='delete',
              requirements=dict(id='d+'), conditions=dict(method='DELETE'))
    mapper.connect('edit_release', '/releases/{id}/edit',
              controller='release', action='edit',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('release_notes', '/releases/{id}/notes',
              controller='release', action='release_notes',
              requirements=dict(id='d+'), conditions=dict(method='GET'))

Full of duplication, verbose, even ugly. Just because it sits in a config directory, does it have to look that bad?

Refactoring 1: Introducing submapper()

New in Routes 1.11 (so really quite new), submapper()provides the means to pull out parameters previously shared across multiple connect() calls.

One particularly nice feature is that the SubMapper objects returned by submapper() support the Python managed object protocol so if you’re on Python 2.5 or above you can use them with the with syntax like this:

from release_tool.lib.mapper import Mapper

def make_map():

    ...

    # Releases - Collection
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))

    # Releases - Members
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases/{id}',
                    requirements=dict(id='d+')) as m:
        m.connect('release', '', action='show',
                conditions=dict(method='GET'))
        m.connect('update_release', '', action='update',
                conditions=dict(method='PUT'))
        m.connect('delete_release', '', action='delete',
                conditions=dict(method='DELETE'))
        m.connect('edit_release', '/edit', action='edit',
                conditions=dict(method='GET'))
        m.connect('release_notes', '/notes', action='release_notes',
                conditions=dict(method='GET'))

That’s a definite improvement (and credit where credit is due – submappers are great innovation, though a small bug requires our local extensions to be imported here), but notice that there’s still some duplication between the two submappers blocks, the first one corresponding to the collection resource, the second to that same collection’s members. Wouldn’t it be cool if we could nest them?

Refactoring 2: Submapper nesting

On the surface, the Routes 1.11 API doesn’t appear to support submapper nesting, but the SubMapper objects do indeed nest. Adding a submapper() method to SubMapper is a trivial change, and it take only minor internal tweaks for deeper nestings to function correctly.

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.connect('release', '', action='show',
                    conditions=dict(method='GET'))
            m.connect('update_release', '', action='update',
                    conditions=dict(method='PUT'))
            m.connect('delete_release', '', action='delete',
                    conditions=dict(method='DELETE'))
            m.connect('edit_release', '/edit', action='edit',
                    conditions=dict(method='GET'))
            m.connect('release_notes', '/notes', action='release_notes',
                    conditions=dict(method='GET'))

We seem to be on to something here, but what about the repetition (in one guise or another) of the resource name “release”? And what’s with those strange empty string ('') parameters?

Refactoring 3: Links and actions

To fix the repetition issue it’s clear that we need helpers that will generate those connect() calls for us more intelligently. But before we do that, let’s recognise that there are really two kinds of routes being generated here:

  1. Links to singleton subresources that support only one HTTP method, the most ubiquitous examples being the new and edit representations for which GET is the applicable HTTP method, but special subresources that are the targets for POST actions are common too;
  2. Actions that correspond directly to HTTP methods: index and create for GET and POST[1] on collection resources, show, update, delete for GET, PUT, DELETE on member resources.

It’s that second type that give rise to those strange empty strings as there is no further navigation to be done relative to the target resource.

So let’s add link() and action() helpers:

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.action(action='index', name='releases')
        c.action(action='create', method='POST')
        c.link('new')
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.action(action='show', name='release')
            m.action(action='update', method='PUT')
            m.action(action='delete', method='DELETE')
            m.link(rel='edit')
            m.link(rel='notes', name='release_notes')

Things are still moving in the right direction, but given that most of those links and actions follow a very well-worn convention, how about specific helpers for those?

Refactoring 4: More helpers

I’ll be honest – this is just an intermediate stage. You can see where we’re headed now though:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.index()
        c.create()
        c.new()
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.show()
            m.update()
            m.delete()
            m.edit()
            m.link(rel='notes', name='release_notes')

Refactoring 5: Parameter-driven generation

We’ll be using those same names a lot across our resources, so let’s just identify the ones we want in a list:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases',
                    actions=['index', 'create', 'new']) as c:
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+'),
                    actions=['show', 'update', 'delete', 'edit']) as m:
            m.link(rel='notes', name='release_notes')

But even this pattern will be repeated across resources, so let’s define a collection() helper that does it all:

Refactoring 6: Using the collection() helper

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

And we’re done.

Let’s compare this to the equivalent resource() call:

    # Releases
    mapper.resource(
                'release',
                'releases',
                controller='release',
                # no requirements!
                member = {'notes':'GET'})

Superficially similar, but I’m unable to specify requirements on member paths and I don’t get to name my custom route either – the generated name “notes_release” just isn’t right! The bottom line is that resource() is approaching a dead end; its only way forward is to make its parameters ever more complex.

So… how about we put resource() on notice and bring in something more flexible and – dare I say – Pythonic? And wouldn’t it look just as nice in Ruby too? Answers on a postcard please…

June 4, 2009

Programmatically adding methods to classes and objects: more Ruby/Python comparisons

Filed under: Programming — Tags: , — Mike @ 6:53 pm

In Dynamically extending object behaviour in Ruby and Python we explored techniques for extending object behaviour dynamically by catching calls to undefined methods and having the target objects respond in some interesting way. In this sequel we achieve similar results by a very different (put perhaps more straightforward) approach, adding methods to existing classes and objects ahead of time programmatically.

I’ve avoided here approaches that simply involve generating program source as strings and then eval-ing it in some way. This  can be effective, but it’s uninteresting and perhaps a bit inelegant! Instead, we use closures, in the form of blocks, lambdas and nested functions.

Straight then to code. First comes Ruby, which I’ve done for 5 or so years on and off, then Python – for 3 weeks maybe? With that admission out there I’ll be as fair as I can!

Ruby

class Hello
  attr_reader :name

  def initialize(name)
    @name = name
  end

  # define a method named x that says hi to x
  def self.def_hi(x)
    define_method(x) { "Hi #{x} from #{name}" }
  end

  # handy helper - see http://whytheluckystiff.net/articles/seeingMetaclassesClearly.html
  def metaclass; class << self; self; end; end

  # uniquely for the object, define a method named x that says hi to x
  def def_obj_yo(x)
    metaclass.send(:define_method, x) { "Hi #{x} from #{name}" }
  end

  def_hi "everyone"
end

h1 = Hello.new("h1")
h2 = Hello.new("h2")

Hello.def_hi("foo")
h1.def_obj_yo("baz")

puts h1.everyone      #=> "Hi everyone from h1"
puts h2.everyone      #=> "Hi everyone from h2"
puts h1.foo           #=> "Hi foo from h1"
puts h2.foo           #=> "Hi foo from h2"
puts h1.foo           #=> "Hi bar from h1"
puts h2.foo           #=> "Hi bar from h2"
puts h1.baz           #=> "Hi Baz from h1"
puts h2.baz           #=> NoMethodError: undefined method ‘baz’ for #

Here, the everyone and foo methods are defined on the class as normal instance methods, and baz is defined as a method on h1‘s “metaclass” (thanks to _why for the helper), making it specific to the h1 instance.

In each case, define_method does the job of creating and installing the method on a class; the subtlety is in the way the blocks (the {...} bits) capture the x parameters for later use.

Newbie Python

class Hello(object):
    def __init__(self, name):
        self.name = name

    @classmethod
    def def_hi(cls, x):
        'define a method named x that says hi to x'
        setattr(cls, x, lambda self: "Hi %s from %s" % (x, self.name))

    @classmethod
    def def_hello(cls, x):
        'define a method x that says hi to x'
        def say_hello(self):
            return "Hello %s from %s" % (x, self.name)
        setattr(cls, x, say_hello)

    def def_obj_yo(self, x):
        'uniqely for the object, add a callable attribute named x that says hi to x'
        setattr(self, x, lambda: "Yo %s from %s" % (x, self.name))

h1 = Hello("h1")
h2 = Hello("h2")

Hello.def_hi("foo")
Hello.def_hello("bar")
h1.def_obj_yo("baz")

print h1.foo()           #=> "Hi foo from h1"
print h2.foo()           #=> "Hi foo from h2"
print h1.baz()           #=> "Yo baz from h1"
print h2.baz()           #=> AttributeError: 'Hello' object has no attribute 'baz'

To be honest, I’m not sure how idiomatic this Python code is, but it certainly works in Python 2.6.2. The reason I mention the version is that older documentation refers to a now-deprecated module “new”, that contains tools similar to Ruby’s define_method. My code is remarkably simple for what it achieves: it just adds lambda objects (in def_hi and def_obj_yo) or function objects (in def_hello) and sets them as attributes of the class or object. Again, we’re relying on those closures to capture the x parameters indefinitely.  The “hi” and “hello” versions aren’t very different (except in length);  sometimes a lambda is too restrictive and a nested function is more appropriate.

I’m 99.9% sure satisfied now (I was rather less sure as I posted the first drafts of this article; I’ve since read the relevant parts of the Language Reference) that the foo objects are proper methods in the strictest sense, though they definitely do behave like them. It’s probably fair to say though that though baz isn’t really a method; self isn’t set as a result of calling the method – it’s just another variable captured in the lambda. Sneaky! If we copied it onto another object, self would still refer to h1.

Comparison

These two lines of Ruby:

  attr_reader :name
  def_hi "everyone"

are executed as the class definition is created. They are very simple examples of how we can define and use what look like new keywords but are in reality just calls to class methods. More complicated examples accept blocks, providing nice API-defined extension points for subclasses to add custom behaviour. Ruby excels in its ability to define APIs not just for their functionality but for their sheer prettiness from the user’s (especially the extender’s) perspective.

Even disregarding cosmetic details such as the optionality of parentheses in Ruby, I was unable to execute similar code within the scope of the Python class definition. Please do correct if me if I’ve got this wrong – having commented this morning (perhaps prematurely) on Yehuda Katz’s interesting article The Importance of Executable Class Bodies it would be good to clear this up!

[UPDATE: yes you can execute arbitrary code within the scope of the class definition, but it’s not quite as simple as calling class methods – in fact I still haven’t found a way to access the class object as it’s being defined (perhaps it doesn’t exist yet!).  The gory details of something that looks pretty close are in 157768: Automating simple property creation (from 2002, perhaps superseded since)]

There is however still a certain elegance to the Python idea (explored more fully last time) of methods being just function-valued (or callable) attributes of objects. No special syntax needed (I don’t think I can ever learn to love Ruby’s class << self idiom) and I love the ease with which function objects can be obtained, installed and invoked, and how, nested within other functions, they can be closures.

On balance? Ruby does seem capable of supporting the nicest APIs (DSLs even), but there’s a simplicity to Python that perhaps makes for more understandable code internally. There’s an interesting balance there to be struck, though neither one wins on a knockout. We’re down now to the level of personal preference, though even without choosing a winner it has definitely been a beneficial experience to have taken on a new language in a serious way. It changes you!

Alternative URI Templates implementation in Python; fun with decorators

Filed under: Programming,Web Integration — Tags: , , — Mike @ 8:20 am

I’ve added uri_templates.py to the described_routes-py repository.  Feature-wise, it differs from Joe’s in that it supports a partial expansion mode (see this post for motivation).  Implementation-wise, it’s all done with regexps – no special parsers required.

There are a couple of missing features (I own up to them in the docstring) but they probably won’t get fixed. Not only do I not need them, but we’re expecting the spec to undergo some some significant change soon and it would probably be wasted effort.

Fun with decorators

For a bit of Python newbie fun, I used a decorator to populate the operator dispatch table:

def operator(name):
    def save_operator(func):
        operators[name] = func
        return func
    return save_operator

@operator('opt')
def op_opt(arg, variables, params, encoding, partial):
    etc

They then get called by:

    operators[operator](arg, variables, params, encoding, partial)

Easy! Perhaps this is overkill, but it seems nicer than techniques like calling functions via constructed function names and it might allow new operators to be added from outside the module.

This must be the simplest example of parameterised decorators you’re ever likely to find as there’s no function wrapping involved. The one thing to understand is that operator('opt') returns a new function that “remembers” the name ‘opt’.

June 1, 2009

Dynamically extending object behaviour in Ruby and Python – a quick warts-and-all comparison

Filed under: Programming,Web Integration — Tags: , , , , — Mike @ 1:25 pm

In path-to, the seemingly innocuous expressions

app.users['dojo'].articles['behind-the-scenes'].edit
app.user('dojo')

are relying on behaviour that is added dynamically. Here, users, articles, edit and user don’t actually exist as defined properties or methods – they are caught and simulated behind the scenes, driven by metadata (descriptions of web resources) loaded at runtime.

Because of fundamental differences in the way object member access works in Ruby and Python, the path-to implementations differ quite significantly under the covers. As promised in my previous post, I explore some of the issues here.

Object member access

Fundamentally different!

  • All Ruby object access is via methods, some of which may be parameterless accessors
  • All Python object access is via properties, some of which may be methods

So,

foo.bar

in Ruby invokes a method that must be either parameterless or whose parameters all have default values defined. In Python it retrieves a property, which may be a value or a function object. Importantly, that function object is not invoked – only foo.bar() would do that.

Adding behaviour dynamically with method_missing and __getattr__

Recall:

  • All Ruby object access is via methods, some of which may be parameterless accessors

In Ruby, when you invoke a non-existent method (e.g. app.users), Ruby calls the target object’s method_missing, which you can override to do something interesting and return a value (all Ruby functions return values, even if only nil).

  • All Python object access is via properties, some of which may be methods

In Python, an attempt to retrieve a non-existent property results in a call to __getattr__. Again, you can override this to do something interesting and return a value. But what if the thing returned needs to behave like a method?

Chaining with Python’s callables

Callables in Python are objects that can be invoked as functions. These include the obvious things like functions and methods (yes they’re objects too), but also includes lambdas, classes (invoking a class creates an instance) and potentially even regular user-defined objects.

In path-to-py, I wanted to emulate a style that you get for free in Ruby, and (thankfully) it turns out to be not so hard in Python either. The challenge is this: when following a resource relationship that doesn’t need parameters, e.g. from the users collection to its new input form:

app.users.new

I didn’t want to force this kind of style, e.g.

app.users().new()

even though any of these navigations may take optional parameters, such as

app.users.new(format="json")

The trick is to make each of these objects callable by defining a __call__ method. Then app.users.new returns one object, and the (format="json") returns another, slightly more specialised one. It’s a particular example of chaining, which is – as you may have gathered already – at the heart of both versions of path-to.

Lambdas

In the case that the callable returned by Python’s __getattr__ must take arguments, the answer is more straightforward: return a function or a function-like object. In path-to-py, we return a lambda that invokes the object’s child method, passing all arguments (positional and keyword) to it, thus:

return lambda *args, **kwargs: self.child(attr, *args, **kwargs)

Some have criticised Python’s lambda for being limited to single expressions. I’m fine with it myself; the simple case looks very neat, and the general case is more than adequately covered by nested functions. And Ruby isn’t without its warts either:

l = lambda {|foo| ... }
l(bar)      # doesn't invoke l
l.call(bar) # ugh
l[bar]      # what???

And nested functions in Ruby don’t work as closures either. Much as I love Ruby, Python deserves better press in this regard I think!

Indexing collections

This works similarly in both languages, though this time my niggles are with Python. If you want an object to behave like an array or dictionary/hash, simply define the [] operator in Ruby or the __getitem__ method in Python. In path-to, these always return new objects (yes, it’s that chaining pattern again).

Python’s warts: you can declare your method thus to allow arbitrary parameters:

def __getitem__(self, *args):

but its behaviour is inconsistent. Sending it one argument will give you the expected one-element args tuple; two or more arguments give you a one-element tuple containing a tuple! In path-to-py, there’s code to detect this and where necessary flatten args before passing it on. And don’t bother with keyword arguments (adding **kwargs to the declaration) – any attempt to use it, e.g.

app.users['dojo', format='json']

actually gives a syntax error. In path-to-py, you can choose between one of these forms (below) instead. The first two are entirely equivalent; the second looks uglier but removes a level of chaining:

app.users['dojo'](format='json')
app.users['dojo'].with_options(format='json')
app.users['dojo', {'format': 'json'}]

Conclusion

Niggles aside, adding behaviour dynamically in Ruby and Python isn’t all that scary really! Their underlying models are surprisingly different, yet there’s no lack of power in either one.

Powered by WordPress