Customizable Site Templates with Stevedore

A Pluggable Python Application Case Study

Python makes it very easy to load code dynamically; handling the subtle bugs and sometimes bizarre failure modes that arise in home-grown module loading systems is another matter entirely. stevedore provides a framework for managing plugins based on the patterns and needs demonstrated by real applications and systems. If you want to implement a plugin architecture for your application, stevedore will accelerate your development by allowing you to focus on application abstractions instead of plugin loading mechanisms.

In this post, I'd like to

• demonstrate the basic ideas and mechanisms used in stevedore, and
• illustrate the concepts with a practical example drawn from a production system

Follow along by grabbing the case source, which includes the IPython notebook source of this post and an example web application demonstrating the techniques described here.

Why?¶

Building applications is hard. Many forces exert influence on a software project as it ages, and the physics of software imply that, over time, these forces will tend to increase the complexity of the software. Features creep in; "temporary" hacks endure, becoming larger and more entrenched; code begins to smell and then to rot. In the swirling chaos of time pressures and user demands, how do we apply sound design principles and create software suffused with simplicity and elegance?

Simplicity in software is the result of deliberate, focused, and continuous attention and cultivation; in a very real sense, it is "unnatural." Yet like any craft, building sound software is a practice that can be learned and mastered, and there is much wisdom and shared experience available to us with an ease and at a scale that is historically unprecedented. Principles like the Single Responsibility Principle encourage us to envision software composed of simple interacting components; if we cannot avoid complexity, ideas like abstraction and encapsulation suggest that we bury it, decoupling it and hiding it away from the rest of our application as far as possible. Taking these ideas at scale, a common way to build applications is to use a plugin architecture, applying the same principles used when designing clean functions at the coarser level of granularity of an entire application. Rather than being conceived as monolithic entities, pluggable applications define a core framework and rely on interchangeable, communicating components to define policy, implement business rules, and interact with the world.

What is stevedore?¶

stevedore is Doug Hellmann's system for managing dynamic plugins in Python. It was created after Doug conducted an extensive taxonomy of the varied (and varying quality) plugin mechanisms used by popular Python systems, which revealed several common interaction patterns used in these high profile applications. For example, many applications have the notion of a driver plugin, which allows an application to be written in terms of abstract representations of resources—say a database or a network interface—with plugins providing the implementations of those abstractions tailored to a specific resource type.

Rather than reinventing the plugin loading wheel (again), stevedore provides a set of higher-level abstractions over existing dynamic discovery and loading machinery. It utilizes setuptools entry points as the mechanism for defining plugins. For example, this entry_points definition in setup.py defines three entry points in the sweet_app.templates namespace: base, shared, and client_custom.

setup(
# ...
entry_points="""
    [console_scripts]
    serve_example = base_app.main:main
    enumerate_extensions = demo:enumerate_template_extensions

    [sweet_app.templates]
    base = base_app.templates
    shared = shared_lib.templates
    client_custom = client_custom.templates
"""
)

In this example, sweet_app.templates is the entry point namespace, inside of which I've declared three entry points: base, shared, and client_custom. The right-hand side defines the entry point target, which can be either a Python package or an attribute within a package.

The namespace serves as the collection point for all plugins of a certain kind within an application. As with all things software development, naming namespaces and selecting the right number of them is something of an art. One rule of thumb I am illustrating here is to prefix the namespaces for an application with an application identifier: templates on its own is too generic to make a good namespace; sweet_app.templates identifies these plugins as the templates belonging to this sweet app.

Just for fun, I've also declared two console_scripts that can be run from the command line once you've installed this post's sample code. We'll come back to serve_example, but enumerate_extensions is a bit of a stevedore "hello world" that demonstrates the basic mechanics by enumerating the extensions in the templates namespace:

In [1]:

%load demo/__init__.py

In [2]:

import logging

from stevedore import ExtensionManager


def enumerate_template_extensions():
    # logging.basicConfig(level=logging.DEBUG)
    namespace = 'sweet_app.templates'
    manager = ExtensionManager(namespace)

    print 'Extensions in namespace {}:'.format(namespace)

    for extension in manager:
        print '\t{} (module {})'.format(extension.name,
                                        extension.entry_point.module_name)

In [3]:

enumerate_template_extensions()

Extensions in namespace sweet_app.templates:
 shared (module shared_lib.templates)
 base (module base_app.templates)
 client_custom (module client_custom.templates)

Here's how it works: stevedore provides a set of manager classes to, well, manage the entry points defined by an application. In this example we use an ExtensionManager, the most generic manager, to show the plugins defined in a given namespace. First, create the manager, passing the desired namespace to its constructor (line 9 above). Manager objects are iterable, returning the list of discovered extensions in the supplied namespace. Each returned object is an Extension, a small bookkeeping wrapper around an entry point:

In [4]:

manager = ExtensionManager('sweet_app.templates')
extension = next(iter(manager))
type(extension)

Out[4]:

stevedore.extension.Extension

The extension exposes the entry point target (either a module or an attribute within a module, say a class or a function) as its plugin attribute:

In [5]:

extension.plugin

Out[5]:

<module 'shared_lib.templates' from 'shared_lib/templates/__init__.pyc'>

The rest of the example merely iterates over the plugins discovered in the templates namespace, printing out the name and module target of each.

Here is a quick summary of the main concepts used in stevedore:

• a plugin is simply a Python module or an attribute (function, class, etc.) within a module
• namespaces collect an application's plugins into logical groups
• plugins are defined using setuptools entry points as part of a normal Python distribution
• stevedore provides a set of manager classes that offer higher-level abstractions over the setuptools plugin machinery

Notice that stevedore does not provide any sort of application plugin framework, nor does it have any particular opinion about what plugins are or how they work, since these concepts are generally assumed to be part of the definition of the application. In other words, stevedore intends to provide higher-level tools that can be used to build an application's plugin system without having to reimplement the dynamic code loading mechanics. That said, stevedore's documentation offers a nice tutorial with tips for getting started and a worked out example of a small plugin architecture.

As a complement to stevedore's own documentation, let's have a look at another example: a pluggable template resolution system drawn from a production web application.

The Case: Customizable Web Templates¶

Imagine for a moment that you're stuck building applications with an ancient web stack (ahem) and that you have Django envy: one of Django's nicer features is its pluggable application mechanism, which allows developers to build systems by composing small, focused subcomponents (what Django calls "apps," not to be confused with those shiny doodads on your phone). Django apps are a powerful abstraction mechanism that can utilize the full gamut of features in the Django ecosystem: they can define new ORM models, create forms, hook in to the application's URL space, notify observers of interesting events, and even override templates defined in another app to extend or augment functionality.

Lets use stevedore to build an implementation of Django's template override mechanism for our hypothetical ancient web stack; our goal here will be to demonstrate stevedore in a realistic scenario by using it to build just the template discovery portion of Django's app system (the other bits left as an exercise for the reader ;). Here's the idea: I have a base web application that defines a common structure and style for the app, including a master template that groups that structure into blocks. The base application provides definitions for each block that can be overridden by more specialized versions defined in other packages:

In [6]:

from IPython.display import SVG
SVG('./site_outline.svg')

Out[6]:

Each block is defined in a separate HTML file that the template system stitches together to produce the final rendered HTML. This is pretty normal fare for template systems; the special sauce that we are going to add is a custom template resolver that knows how to look for templates contained in a configurable set of locations rather than a single template directory. The resolution process requires us to define the template roots, the packages containing the various collections of templates, and the resolution order, which determines the precedence that will be used when searching the various packages for templates. As an example, the setup.py file shown above defines three template roots in the imaginatively-named templates namespace that correspond to three packages in the sample code:

In [9]:

!tree -P "*.html" client_custom shared_lib base_app

client_custom
└── templates
    └── nav.html
shared_lib
└── templates
    └── jumbo.html
base_app
└── templates
    ├── admin
    │   └── index.html
    ├── index.html
    ├── jumbo.html
    └── nav.html

4 directories, 6 files

Note that the three packages share a common directory structure; overriding a template means placing a file with the same name at the same location in a template package with higher resolution precedence. For the sake of convenience the example houses all of these packages in the same distribution, but an actual application would likely organize things into separate, independently installable distributions.

To perform template resolution, we need to

1. Determine the namespace that houses the template plugins, the extension names to use, and the resolution order. In our production system, we use a fixed shared namespace (templates) and each application deployment determines the extension names and resolution order from configuration.

2. Create an ExtensionManager instance to load the template plugin packages. stevedore's NamedExtensionManager matches our use case perfectly, allowing an application to supply a named list of extensions to use within a given namespace. This means that we can organize our package code as best fits our application independent of the concern of plugin activation. The actual use of a given plugin will be set explicitly at runtime, usually through some form of application configuration passed to NamedExtensionManager's constructor.

3. At resolution time, search the list of extensions for the requested template name, returning the first one found. We use package relative naming: the template name admin.index means "search for the template named index.html in plugin_package.admin, where plugin_package is expanded to each of the plugins in turn." With a little tweaking, the same idea could be used with path-based naming if that is more your thing.

And again, in Python:

In [10]:

from resolver import TemplateResolver

# step 1: define the plugins we want to look for templates in and the resolution order
plugin_names = ['client_custom', 'shared', 'base']

# step 2: create the ExtensionManager (part of the constructor, hang tight)
resolve = TemplateResolver(plugin_names)

# step 3: resolve a template name to a file path
resolve('index')

Out[10]:

'base_app/templates/index.html'

Remember that blaring trumpet sound that Windows used to start with? That belongs here: Ta Da!

Here's a juicier example that shows off overriding, with nav being pulled from the client custom package and the jumbotron from a shared library of templates:

In [12]:

import pandas as pd

templates = [(template, resolve(template)) for template in ('nav', 'jumbo', 'admin.index')]

pd.DataFrame(templates, columns=['template_name', 'resolved_template'])

Out[12]:

	template_name	resolved_template
0	nav	client_custom/templates/nav.html
1	jumbo	shared_lib/templates/jumbo.html
2	admin.index	base_app/templates/admin/index.html

3 rows × 2 columns

and to continue beating things so they are good and dead, here's what would happen if you changed the list of plugins to use:

In [13]:

plugin_names = ['base']
resolve = TemplateResolver(plugin_names)

resolve('nav')

Out[13]:

'base_app/templates/nav.html'

Note that we haven't changed anything about the deployment or plugin set up at all, merely the list of extensions we want to select from the universe of available plugins in the namespace.

If you'd like to see these concepts in action, activate the environment in which you installed the sample code accompanying this post and run the command

serve_example

then browse to http://localhost:5000, which will fire up a sample application demonstrating this technique in a small web application. Pay particular attention to the tabbed navigation in the upper right and the large Jumbotron in the center of the application, both of which exhibit the overriding behavior detailed above.

But dan, how does it all work?¶

Here's the payoff: stevedore makes implementing the ideas I've described here remarkably simple. Check it:

In [14]:

%load -r 20-40 ./resolver/__init__.py

In [15]:

class TemplateResolver(object):
    def __init__(self, template_modules, namespace='templates'):
        self.template_modules = template_modules
        self.extension_manager = NamedExtensionManager(namespace,
                                                       template_modules,
                                                       name_order=True)

    def __call__(self, name):
        package_suffix, target_file, join = self._process_name(name)

        for extension in self.extension_manager.extensions:
            target_package = join((extension.entry_point.module_name,
                                   package_suffix))

            if resource_exists(target_package, target_file):
                return resource_filename(target_package, target_file)
        else:
            raise ValueError(
                'Could not locate template "{}" in any loaded template '
                'module: {}'.format(name, self.template_modules))

Yep. That's it: lines 4–6 initialize the extension manager (our step #2 above), and lines 11–16 are the heart of the resolver (step #3). The NamedExtensionManager provides two helpful features for this use case. First, it filters all of the available extensions within a namespace to an explicitly-declared list, which allows us to specify the desired template roots in configuration. It also allows you to impose an ordering for the extensions: setting name_order=True means that the extensions will be returned from the manager in the same order as the name list we supply to the manager's constructor (without this option we would get arbitrary ordering: entry point load order is effectively undefined from an application's perspective).

Resolution is straightforward: after processing the requested name to handle dotted package paths (line 9), the resolver loops through the enabled extensions and looks to see if the package referenced by the extension contains the requested file. As soon as there's a match, resolution ends by returning the file path of the requested template (resource_exists and resource_filename are functions for looking up files contained within Python packages; both are provided by pkg_resources).

If no package contains the requested template, an exception is raised; the exception's message contains the list of activated template packages, as misconfiguration here is the most common source of failure we run into:

In [16]:

resolve('notgonnawork')

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-6e96b4ec4eb9> in <module>()
----> 1 resolve('notgonnawork')

/home/dan/source/posts/stevedore_template_resolver/resolver/__init__.pyc in __call__(self, name)
     37             raise ValueError(
     38                 'Could not locate template "{}" in any loaded template '
---> 39                 'module: {}'.format(name, self.template_modules))
     40 
     41     def _process_name(self, name):

ValueError: Could not locate template "notgonnawork" in any loaded template module: ['base']

Takeaways¶

1. A plugin architecture can help you simplify, simply

A plugin architecture encourages you to think about applications in terms of abstractions and interfaces. Rather than relying on conditional logic to accommodate alternate workflows and implementations, a plugin architecture allows an application to work at a consistent level of abstraction, deferring implementation details to plugins behind a defined interface. While this is commonly cited as an advantage for allowing third parties to provide concrete implementations (e.g. drivers), it can also significantly improve the internal implementation of an application by encouraging a clean separation between high-level abstractions and implementation details.

But that does not mean that a plugin architecture has to be complicated! Plugins are simply functions, classes, or modules that are exposed by name via the entry points machinery; stevedore, as we saw above, makes retrieving plugins almost trivially easy…

2. stevedore understands entry points so you don't have to

stevedore is built for application developers, not packaging gurus: to use it, simply label a function, class, or module as an entry point and choose the appropriate manager. The managers it provides represent the vast majority of plugin loading patterns used by major applications in the Python ecosystem and allow your application to discover, load, and interact with plugins based on the desired usage pattern. Developers can spend their time designing clean application interfaces rather than reinventing dynamic code loading or digging through the documentation for entry points.

3. These ideas are widely applicable; stevedore makes it easy to get started

It is probably obvious that template resolution is not the first application that comes to mind when people talk about plugin architectures. Yet this example, while perhaps somewhat atypical, illustrates the flexibility of these ideas and the ease with which developers can get started using stevedore. With very little effort, we were able to significantly extend the capabilities of our existing application, offering flexibility and customization capacity to our customers while reducing our ongoing maintenance burden: where we once would have resorted to conditional logic, we can now offer client customizations by merely deploying the requested customization, which is automatically picked up by the resolution system.

Resources¶

• Using stevedore in Your Application and stevedore's documentation home
• Use stevedore's Launchpad page to request features or file bug reports

Special thanks to Doug Hellmann for reviewing an earlier draft of this post.