Positive Incline Mike Burrows (@asplake) moving on up, positively

June 1, 2009

Dynamically extending object behaviour in Ruby and Python – a quick warts-and-all comparison

Filed under: Programming,Web Integration — Tags: , , , , — Mike @ 1:25 pm

In path-to, the seemingly innocuous expressions

app.users['dojo'].articles['behind-the-scenes'].edit
app.user('dojo')

are relying on behaviour that is added dynamically. Here, users, articles, edit and user don’t actually exist as defined properties or methods – they are caught and simulated behind the scenes, driven by metadata (descriptions of web resources) loaded at runtime.

Because of fundamental differences in the way object member access works in Ruby and Python, the path-to implementations differ quite significantly under the covers. As promised in my previous post, I explore some of the issues here.

Object member access

Fundamentally different!

  • All Ruby object access is via methods, some of which may be parameterless accessors
  • All Python object access is via properties, some of which may be methods

So,

foo.bar

in Ruby invokes a method that must be either parameterless or whose parameters all have default values defined. In Python it retrieves a property, which may be a value or a function object. Importantly, that function object is not invoked – only foo.bar() would do that.

Adding behaviour dynamically with method_missing and __getattr__

Recall:

  • All Ruby object access is via methods, some of which may be parameterless accessors

In Ruby, when you invoke a non-existent method (e.g. app.users), Ruby calls the target object’s method_missing, which you can override to do something interesting and return a value (all Ruby functions return values, even if only nil).

  • All Python object access is via properties, some of which may be methods

In Python, an attempt to retrieve a non-existent property results in a call to __getattr__. Again, you can override this to do something interesting and return a value. But what if the thing returned needs to behave like a method?

Chaining with Python’s callables

Callables in Python are objects that can be invoked as functions. These include the obvious things like functions and methods (yes they’re objects too), but also includes lambdas, classes (invoking a class creates an instance) and potentially even regular user-defined objects.

In path-to-py, I wanted to emulate a style that you get for free in Ruby, and (thankfully) it turns out to be not so hard in Python either. The challenge is this: when following a resource relationship that doesn’t need parameters, e.g. from the users collection to its new input form:

app.users.new

I didn’t want to force this kind of style, e.g.

app.users().new()

even though any of these navigations may take optional parameters, such as

app.users.new(format="json")

The trick is to make each of these objects callable by defining a __call__ method. Then app.users.new returns one object, and the (format="json") returns another, slightly more specialised one. It’s a particular example of chaining, which is – as you may have gathered already – at the heart of both versions of path-to.

Lambdas

In the case that the callable returned by Python’s __getattr__ must take arguments, the answer is more straightforward: return a function or a function-like object. In path-to-py, we return a lambda that invokes the object’s child method, passing all arguments (positional and keyword) to it, thus:

return lambda *args, **kwargs: self.child(attr, *args, **kwargs)

Some have criticised Python’s lambda for being limited to single expressions. I’m fine with it myself; the simple case looks very neat, and the general case is more than adequately covered by nested functions. And Ruby isn’t without its warts either:

l = lambda {|foo| ... }
l(bar)      # doesn't invoke l
l.call(bar) # ugh
l[bar]      # what???

And nested functions in Ruby don’t work as closures either. Much as I love Ruby, Python deserves better press in this regard I think!

Indexing collections

This works similarly in both languages, though this time my niggles are with Python. If you want an object to behave like an array or dictionary/hash, simply define the [] operator in Ruby or the __getitem__ method in Python. In path-to, these always return new objects (yes, it’s that chaining pattern again).

Python’s warts: you can declare your method thus to allow arbitrary parameters:

def __getitem__(self, *args):

but its behaviour is inconsistent. Sending it one argument will give you the expected one-element args tuple; two or more arguments give you a one-element tuple containing a tuple! In path-to-py, there’s code to detect this and where necessary flatten args before passing it on. And don’t bother with keyword arguments (adding **kwargs to the declaration) – any attempt to use it, e.g.

app.users['dojo', format='json']

actually gives a syntax error. In path-to-py, you can choose between one of these forms (below) instead. The first two are entirely equivalent; the second looks uglier but removes a level of chaining:

app.users['dojo'](format='json')
app.users['dojo'].with_options(format='json')
app.users['dojo', {'format': 'json'}]

Conclusion

Niggles aside, adding behaviour dynamically in Ruby and Python isn’t all that scary really! Their underlying models are surprisingly different, yet there’s no lack of power in either one.

16 Comments

  1. __getitem__ is actually acting consistently here – but you’re complicating it with the unneccesary *args syntax. __getitem__ always takes a single argument: the object to use as a key. When you use obj[x], that object is x. When you use obj[(x,y)], that object is the tuple “(x,y)”, which, since parentheses are optional for tuples can also be written obj[x,y]. ie. ‘dojo’, ‘json’ aren’t arguments, they’re a tuple.

    Comment by Brian — June 1, 2009 @ 3:21 pm

  2. Yeah, I kinda got that, but it seems like an unnecessary special case to me, like it was designed only for use by list and dict. Why can’t [] just be syntactic sugar for __getitem__? And without the *, I seemed to be able to use only one argument.

    Comment by Mike — June 1, 2009 @ 3:25 pm

  3. It helps to remember that __getitem__ has to handle slicing in a sane way, for example arr[1:5:2].

    Also, minor typo near the end of your article:
    app.users[‘dojo’, {‘format’: ‘json’})
    I think you meant:
    app.users[‘dojo’, {‘format’: ‘json’}]

    Good post, though.

    Comment by Kevin Gadd — June 1, 2009 @ 3:45 pm

  4. Oops – I’ve fixed the typo and thanks for the feedback.

    I got thinking about slices after responding to Brian. I could argue that I’m thinking slightly more generally about querying collections. I don’t see why I should have to create a range-type object with which to do it, though I can well imagine that some would accuse me of abusing the notation. Anyway, I’m very new to Python so I’m not going to push it!

    Comment by Mike — June 1, 2009 @ 3:54 pm

  5. Using *args shouldn’t make any difference. All it will do is wrap the single argument you always get into an enclosing list. obj[x] IS just syntax sugar for __getitem__(x), it’s just that __getitem__ only ever takes a single argument. Unfortunately this single argument does means it isn’t a direct equivalent to __call__, so can’t be used identically for your purposes, but this has advantages too – it avoids ambiguities between things like obj[(a,b,c)] vs obj[a,b,c], and prevents having to special case tuple syntax in that context.

    Comment by Brian — June 1, 2009 @ 4:18 pm

  6. Thanks again, it’s getting clearer!

    Comment by Mike — June 1, 2009 @ 4:23 pm

  7. Nice article! and free of any Ruby or Python zealotry (for a change!)

    I also dislike the my_lambda.call() syntax in Ruby but perhaps it’s a necessary evil given the optional parentheses for method invocation.

    Also, in my opinion, optional parentheses is a much bigger win than the loss of the somewhat ugly lambda syntax. (Ruby 1.9.1 slightly improves this syntax letting you go: my_lambda.() in place of my_lambda.call() )

    John

    Comment by John Mair — June 1, 2009 @ 4:42 pm

  8. Thanks!

    I’ve shown that with a bit of effort you can make parentheses optional in Python too. Perhaps when I get my head around descriptors and decorators I might work out a way of doing it via one-liners. Something to aim for anyway!

    Comment by Mike — June 1, 2009 @ 4:47 pm

  9. Just one further comment, you say: “…nested functions in Ruby don’t work as closures either…”.

    What do you mean by ‘nested functions’ ? Ruby doesn’t actually support nested functions πŸ˜‰

    John

    Comment by John Mair — June 1, 2009 @ 4:47 pm

  10. Nothing stopping you nesting one definition inside another (just verified that again with 1.8.6), but it doesn’t achieve anything useful.

    Comment by Mike — June 1, 2009 @ 4:54 pm

  11. well you can define a method within another method, sure, but it’s not ‘nested’ in the sense that it is in python. (i.e in Ruby the ‘nested’ method is accessible outside the method it’s defined in; but in Python, afaik, it isn’t.)

    Comment by John Mair — June 1, 2009 @ 5:03 pm

  12. Yep

    Comment by Mike — June 1, 2009 @ 5:09 pm

  13. […] Dynamically extending object behaviour in Ruby and Python – a quick warts-and-all comparison we explored techniques for extending object behaviour dynamically by catching calls to undefined […]

    Pingback by Programmatically adding methods to classes and objects: more Ruby/Python comparisons « Positive Incline — June 4, 2009 @ 6:53 pm

  14. […] Dynamically extending object behaviour in Ruby and Python [positiveincline.com] […]

    Pingback by Bootstrapping REST « Positive Incline — July 1, 2009 @ 8:04 am

  15. See, told you I was right. πŸ˜›

    Comment by John Mair — May 10, 2010 @ 7:54 am

  16. πŸ™‚

    Comment by Mike — May 10, 2010 @ 9:30 am

RSS feed for comments on this post.

Sorry, the comment form is closed at this time.

Powered by WordPress