Positive Incline Mike Burrows (@asplake) moving on up, positively

February 28, 2011

Positive Incline Limited

Filed under: Kanban,lean,Project Management,Work — Tags: , , , , — Mike @ 9:38 pm

Not just a blog, from March 1st Positive Incline is a multi-client management consultancy :–)

My specialties:

  • IT/product development process design, organisation, operation and improvement
  • Kanban, Lean, Agile development
  • The design and implementation of business process and quality improvement initiatives

My biases will be familiar to regular readers:

  • Speed, flow
  • Transparency, visualisation
  • Observation, evidence, feedback loops
  • Continuous learning & relentless improvement
  • Customer focus (getting right behind the sources of demand)
  • Sound project/process economics

I come from a development background (I still love programming); a former development manager, product manager and leader of change initiatives.

My job is to help you understand your organisation’s systems more deeply (not just conceptually, but how they really behave in practice) and to help find both improvements and ways to keep them improving for the longer term.  Look elsewhere if you want an off-the-shelf solution laden with jargon or lots of acronyms!

December 21, 2009

DRY up your routes – a Pylons routing refactoring

Filed under: Programming,Web Integration — Tags: , , , , , — Mike @ 11:43 am

[See UPDATES]

This post is in several acts, each one a refactoring. Taking the stage will be some familiar-looking application code; behind the scenes lurks some enhanced framework code that makes the refactorings possible.

Our story starts with an old-school (pre-REST) routing configuration. It’s in Python, for Pylons and other Routes-based web frameworks, but let me reassure the Ruby audience that Routes borrows very heavily from Rails and their routing configurations look quite similar.

We will finish with collection(), an experimental (but I hope worthy) alternative to resource(), the routing helper that rose to prominence when Rails first made its big push to REST.

The starting point: textbook routing.py

from routes import Mapper

def make_map():

    ...

    # Releases - Collection
    mapper.connect('releases', '/releases',
              controller='release', action='index', conditions=dict(method='GET'))
    mapper.connect('create_release', '/releases',
              controller='release', action='create', conditions=dict(method='POST'))
    mapper.connect('new_release', '/releases/new',
              controller='release', action='new', conditions=dict(method='GET'))

    # Releases - Members
    mapper.connect('release', '/releases/{id}',
              controller='release', action='show',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('update_release', '/releases/{id}',
              controller='release', action='update',
              requirements=dict(id='d+'), conditions=dict(method='PUT'))
    mapper.connect('delete_release', '/releases/{id}',
              controller='release', action='delete',
              requirements=dict(id='d+'), conditions=dict(method='DELETE'))
    mapper.connect('edit_release', '/releases/{id}/edit',
              controller='release', action='edit',
              requirements=dict(id='d+'), conditions=dict(method='GET'))
    mapper.connect('release_notes', '/releases/{id}/notes',
              controller='release', action='release_notes',
              requirements=dict(id='d+'), conditions=dict(method='GET'))

Full of duplication, verbose, even ugly. Just because it sits in a config directory, does it have to look that bad?

Refactoring 1: Introducing submapper()

New in Routes 1.11 (so really quite new), submapper()provides the means to pull out parameters previously shared across multiple connect() calls.

One particularly nice feature is that the SubMapper objects returned by submapper() support the Python managed object protocol so if you’re on Python 2.5 or above you can use them with the with syntax like this:

from release_tool.lib.mapper import Mapper

def make_map():

    ...

    # Releases - Collection
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))

    # Releases - Members
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases/{id}',
                    requirements=dict(id='d+')) as m:
        m.connect('release', '', action='show',
                conditions=dict(method='GET'))
        m.connect('update_release', '', action='update',
                conditions=dict(method='PUT'))
        m.connect('delete_release', '', action='delete',
                conditions=dict(method='DELETE'))
        m.connect('edit_release', '/edit', action='edit',
                conditions=dict(method='GET'))
        m.connect('release_notes', '/notes', action='release_notes',
                conditions=dict(method='GET'))

That’s a definite improvement (and credit where credit is due – submappers are great innovation, though a small bug requires our local extensions to be imported here), but notice that there’s still some duplication between the two submappers blocks, the first one corresponding to the collection resource, the second to that same collection’s members. Wouldn’t it be cool if we could nest them?

Refactoring 2: Submapper nesting

On the surface, the Routes 1.11 API doesn’t appear to support submapper nesting, but the SubMapper objects do indeed nest. Adding a submapper() method to SubMapper is a trivial change, and it take only minor internal tweaks for deeper nestings to function correctly.

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.connect('releases', '', action='index',
                conditions=dict(method='GET'))
        c.connect('create_release', '', action='create',
                conditions=dict(method='POST'))
        c.connect('new_release', '/new', action='new',
                conditions=dict(method='GET'))
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.connect('release', '', action='show',
                    conditions=dict(method='GET'))
            m.connect('update_release', '', action='update',
                    conditions=dict(method='PUT'))
            m.connect('delete_release', '', action='delete',
                    conditions=dict(method='DELETE'))
            m.connect('edit_release', '/edit', action='edit',
                    conditions=dict(method='GET'))
            m.connect('release_notes', '/notes', action='release_notes',
                    conditions=dict(method='GET'))

We seem to be on to something here, but what about the repetition (in one guise or another) of the resource name “release”? And what’s with those strange empty string ('') parameters?

Refactoring 3: Links and actions

To fix the repetition issue it’s clear that we need helpers that will generate those connect() calls for us more intelligently. But before we do that, let’s recognise that there are really two kinds of routes being generated here:

  1. Links to singleton subresources that support only one HTTP method, the most ubiquitous examples being the new and edit representations for which GET is the applicable HTTP method, but special subresources that are the targets for POST actions are common too;
  2. Actions that correspond directly to HTTP methods: index and create for GET and POST[1] on collection resources, show, update, delete for GET, PUT, DELETE on member resources.

It’s that second type that give rise to those strange empty strings as there is no further navigation to be done relative to the target resource.

So let’s add link() and action() helpers:

    # Releases
    with mapper.submapper(
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.action(action='index', name='releases')
        c.action(action='create', method='POST')
        c.link('new')
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.action(action='show', name='release')
            m.action(action='update', method='PUT')
            m.action(action='delete', method='DELETE')
            m.link(rel='edit')
            m.link(rel='notes', name='release_notes')

Things are still moving in the right direction, but given that most of those links and actions follow a very well-worn convention, how about specific helpers for those?

Refactoring 4: More helpers

I’ll be honest – this is just an intermediate stage. You can see where we’re headed now though:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases') as c:
        # Collection
        c.index()
        c.create()
        c.new()
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+')) as m:
            m.show()
            m.update()
            m.delete()
            m.edit()
            m.link(rel='notes', name='release_notes')

Refactoring 5: Parameter-driven generation

We’ll be using those same names a lot across our resources, so let’s just identify the ones we want in a list:

    # Releases
    with mapper.submapper(
                    collection_name='releases',
                    controller='release',
                    path_prefix='/releases',
                    actions=['index', 'create', 'new']) as c:
        # Members
        with c.submapper(
                    path_prefix='/{id}',
                    requirements=dict(id='d+'),
                    actions=['show', 'update', 'delete', 'edit']) as m:
            m.link(rel='notes', name='release_notes')

But even this pattern will be repeated across resources, so let’s define a collection() helper that does it all:

Refactoring 6: Using the collection() helper

    with mapper.collection(
                    'releases',
                    'release',
                    member_options={
                        'requirements': {'id': 'd+'}}) as c:
        c.member.link(rel='notes', name='release_notes')

And we’re done.

Let’s compare this to the equivalent resource() call:

    # Releases
    mapper.resource(
                'release',
                'releases',
                controller='release',
                # no requirements!
                member = {'notes':'GET'})

Superficially similar, but I’m unable to specify requirements on member paths and I don’t get to name my custom route either – the generated name “notes_release” just isn’t right! The bottom line is that resource() is approaching a dead end; its only way forward is to make its parameters ever more complex.

So… how about we put resource() on notice and bring in something more flexible and – dare I say – Pythonic? And wouldn’t it look just as nice in Ruby too? Answers on a postcard please…

July 23, 2009

New link_header gem

Filed under: Web Integration — Tags: , , , , , — Mike @ 9:26 pm

My latest project on github and Rubyforge is link_header, a small rubygem for parsing and generating HTTP link headers as per the latest spec draft-nottingham-http-link-header-06.txt.

Usage

The usual install:

sudo gem install link_header

The library’s LinkHeader and LinkHeader::Link classes follow a pattern established in the ResourceTemplate and ResourceTemplate classes in that they offer easy conversions both to & from Ruby primitives, i.e. Arrays, Strings etc. This in turn makes them easy to prettyprint, convert to & from JSON and YAML, create from test fixtures, and so on. [Aside: @kevinrutherford and I discussed this idea on Twitter a few days ago in response to his blog post “factory method in ruby“. It’s worth a read.]

Link attribute names can appear more than once, so I have chosen a list of attribute/value pairs rather than a hash to represent link attributes. Link objects do however have an #attrs method that will lazily generate a hash if that’s convenient (it’s left to you to decide whether it’s safe!). There’s an example of this below.

So:

require "link_header"
require "pp"

#
# Create a LinkHeader with Link objects
#
link_header = LinkHeader.new([
  LinkHeader::Link.new("http://example.com/foo", [["rel", "self"]]),
  LinkHeader::Link.new("http://example.com/",    [["rel", "up"]])])

puts link_header.to_s
#=> <http://example.com/foo>; rel="self", <http://example.com/>; rel="up"

link_header.links.map do |link|
  puts "href #{link.href.inspect}, attr_pairs #{link.attr_pairs.inspect}, attrs #{link.attrs.inspect}"
end
#=> href "http://example.com/foo", attr_pairs [["rel", "self"]], attrs {"rel"=>"self"}
#   href "http://example.com/", attr_pairs [["rel", "up"]], attrs {"rel"=>"up"}

#
# Create a LinkHeader from raw (JSON-friendly) data
#
puts LinkHeader.new([
  ["http://example.com/foo", [["rel", "self"]]],
  ["http://example.com/",    [["rel", "up"]]]]).to_s
#=> <http://example.com/foo>; rel="self", <http://example.com/>; rel="up"

#
# Parse a link header into a LinkHeader object then produce its raw data representation
#
pp LinkHeader.parse('<http://example.com/foo>; rel="self", <http://example.com/>; rel = "up"').to_a
#=> [["http://example.com/foo", [["rel", "self"]]],
#    ["http://example.com/", [["rel", "up"]]]]

Later…

My next programming task will be some minor refactoring on described_routes and path-to take advantage of this new gem. The driver behind this all is an efficient discovery protocol and a significant reduction in the number of links reported by default – I realised that it was wasteful to produce multiple links on every request that are (let’s be honest) be of no interest at all to clients, when just one of those links points to metadata that carries that same information and more! Then for those server apps that generate the correct headers, a short one-liner will initialize a path-to client given the address of any resource served by the application, i.e. without special knowledge of the location of the app’s metadata resources.

Refactoring aside, the described_routes part of this is done (so servers support the protocol already); I just need to finish path-to part to take advantage of it on the client side.

I can’t make any promises about timelines at the moment (new job starts soon) but a Python version should be forthcoming soon(ish) also. Meanwhile, enjoy the Ruby version if you can!

June 1, 2009

Dynamically extending object behaviour in Ruby and Python – a quick warts-and-all comparison

Filed under: Programming,Web Integration — Tags: , , , , — Mike @ 1:25 pm

In path-to, the seemingly innocuous expressions

app.users['dojo'].articles['behind-the-scenes'].edit
app.user('dojo')

are relying on behaviour that is added dynamically. Here, users, articles, edit and user don’t actually exist as defined properties or methods – they are caught and simulated behind the scenes, driven by metadata (descriptions of web resources) loaded at runtime.

Because of fundamental differences in the way object member access works in Ruby and Python, the path-to implementations differ quite significantly under the covers. As promised in my previous post, I explore some of the issues here.

Object member access

Fundamentally different!

  • All Ruby object access is via methods, some of which may be parameterless accessors
  • All Python object access is via properties, some of which may be methods

So,

foo.bar

in Ruby invokes a method that must be either parameterless or whose parameters all have default values defined. In Python it retrieves a property, which may be a value or a function object. Importantly, that function object is not invoked – only foo.bar() would do that.

Adding behaviour dynamically with method_missing and __getattr__

Recall:

  • All Ruby object access is via methods, some of which may be parameterless accessors

In Ruby, when you invoke a non-existent method (e.g. app.users), Ruby calls the target object’s method_missing, which you can override to do something interesting and return a value (all Ruby functions return values, even if only nil).

  • All Python object access is via properties, some of which may be methods

In Python, an attempt to retrieve a non-existent property results in a call to __getattr__. Again, you can override this to do something interesting and return a value. But what if the thing returned needs to behave like a method?

Chaining with Python’s callables

Callables in Python are objects that can be invoked as functions. These include the obvious things like functions and methods (yes they’re objects too), but also includes lambdas, classes (invoking a class creates an instance) and potentially even regular user-defined objects.

In path-to-py, I wanted to emulate a style that you get for free in Ruby, and (thankfully) it turns out to be not so hard in Python either. The challenge is this: when following a resource relationship that doesn’t need parameters, e.g. from the users collection to its new input form:

app.users.new

I didn’t want to force this kind of style, e.g.

app.users().new()

even though any of these navigations may take optional parameters, such as

app.users.new(format="json")

The trick is to make each of these objects callable by defining a __call__ method. Then app.users.new returns one object, and the (format="json") returns another, slightly more specialised one. It’s a particular example of chaining, which is – as you may have gathered already – at the heart of both versions of path-to.

Lambdas

In the case that the callable returned by Python’s __getattr__ must take arguments, the answer is more straightforward: return a function or a function-like object. In path-to-py, we return a lambda that invokes the object’s child method, passing all arguments (positional and keyword) to it, thus:

return lambda *args, **kwargs: self.child(attr, *args, **kwargs)

Some have criticised Python’s lambda for being limited to single expressions. I’m fine with it myself; the simple case looks very neat, and the general case is more than adequately covered by nested functions. And Ruby isn’t without its warts either:

l = lambda {|foo| ... }
l(bar)      # doesn't invoke l
l.call(bar) # ugh
l[bar]      # what???

And nested functions in Ruby don’t work as closures either. Much as I love Ruby, Python deserves better press in this regard I think!

Indexing collections

This works similarly in both languages, though this time my niggles are with Python. If you want an object to behave like an array or dictionary/hash, simply define the [] operator in Ruby or the __getitem__ method in Python. In path-to, these always return new objects (yes, it’s that chaining pattern again).

Python’s warts: you can declare your method thus to allow arbitrary parameters:

def __getitem__(self, *args):

but its behaviour is inconsistent. Sending it one argument will give you the expected one-element args tuple; two or more arguments give you a one-element tuple containing a tuple! In path-to-py, there’s code to detect this and where necessary flatten args before passing it on. And don’t bother with keyword arguments (adding **kwargs to the declaration) – any attempt to use it, e.g.

app.users['dojo', format='json']

actually gives a syntax error. In path-to-py, you can choose between one of these forms (below) instead. The first two are entirely equivalent; the second looks uglier but removes a level of chaining:

app.users['dojo'](format='json')
app.users['dojo'].with_options(format='json')
app.users['dojo', {'format': 'json'}]

Conclusion

Niggles aside, adding behaviour dynamically in Ruby and Python isn’t all that scary really! Their underlying models are surprisingly different, yet there’s no lack of power in either one.

Powered by WordPress