Deck the Halls with Raspberry Pi

I'm continually on the lookout for ways to connect with my children, especially my eldest whose natural interests seem to be rather different from mine. Going into parenthood I expected to engage my children easily over interesting science and technology subjects. It's certainly in their blood: their mother is a scientist and I am a software developer and general tinkerer. Reality has, as it so often does, proven less straightforward, more ambiguous. I'm sure there are many causes and influences, but one obvious theme concerns the sheer scope and pace of technological advancement since my own childhood. My first computer — a Commodore VIC-20 — seems hopelessly primitive next to the bewildering assortment of hypernetworked computing devices my children see their grandparents carry in a pocket.

While I struggle to identify the essential difference in our experiences, I suspect that part of what has changed is that my children aren't interested in computers as such the way I was. They beg to get tablet time to watch videos or play an online game, but everything they experience is professionally produced and logically ordered, with all the rough edges and sharp corners neatly rounded off into beautiful, pleasing curves.

My formative experiences were quite different. To them, using a VIC-20 would seem like a pathetically empty experience by contrast with the devices they're used to. No slick fade ins to carefully crafted notifications blending seamlessly with expensive, highly tuned interfaces to precisely-defined destinations. Even turning the VIC on felt like a primordial act, the massive, honest-to-goodness SPST coursing with power that flowed past your finger and off toward unknown and terrifying adventures. Yet even after that nothing much happened, nothing but a blinking blue expectancy, a half imagined whisper of potential. But of course that was the magic of it: you didn't turn on a VIC so much for what it could already do; you turned it on because of what you could make it do, for all of the wonderful creative possibilities its existence suggested. While the intervening decades have brought transformative advancements in the capabilities of our information technologies, the exploratory capacity, this frontier-like horizon that was the VIC's foremost asset, can become lost in the well-worn ruts of design.

This helps explain why, on a whim shortly before Christmas, I packed my boys into the truck and headed over to Micro Center to pick up a Raspberry Pi 2. The actual purchase jarred me a little by its contrast with my childhood memories, the agonized handwringing and haggling over the purchase of some part or accessory, which cost much more and did much less than the tiny little box the three of us carried out of the store. But there's something compelling about these little devices that manages to recapture some of the joy of possibility that hooked me on computing in the first place. Perhaps it's the physical board itself, so unlike a phone or tablet, with its rough circuit board edges crammed to bursting with connectors, sharp solder points and interface pins sticking weirdly out of every surface. Or maybe it's the surprising scale that delights, the wonder that something so small can do so much. It could be all those I/O pins practically begging to be hooked up to lights, sensors, and other gizmos. Whatever the reason, there is something about this device that manages to break through the conventional abstractions to reawaken wonder, the possibility of possibility.

Our first Pi project was a humble one: we hooked up a software-controlled LED, which is pretty much the equivalent of “hello, world!” for the Raspberry Pi.

As with most tech projects, even something as seemingly simple as a light on a switch requires daunting amounts of paraphernalia, patience, and time. For us, though, that was part of the fun: trucking around town on some fool idea of dad's, rummaging in obscure corners of the basement closet for resistors, LEDs, and bits of wire, the improbability of seeing the Pi driving an HDTV, fidgeting while the adults scratched their heads over some technical snag.

To get started on your own Pi adventure, you'll need a Raspberry Pi board [1], a micro SD card with the Raspbian OS installed [2], a micro USB power source (such as a phone charger), a screen with HDMI or composite video inputs and appropriate cabling, and a USB keyboard and mouse. (Whew!) Test booting the Pi up to make sure you have a working setup; there is nothing quite like the frustration of getting stuck for an hour and then discovering a bad cable.

For the LED mini project you'll need a few more parts: an LED, a current limiting resistor, some wire, and jumpers to attach to the Pi's GPIO pins. If you've got an old desktop computer lying around, you can scrounge the LED, wire, and jumpers from its indicator lights. And you should probably get yourself a breadboard and jumper wires for it, but for this simple circuit I was content to solder the components directly.

There are many excellent showcase and tutorial sites out there to spur project ideas and help you get going. We headed over to Gordon's Projects Single LED tutorial, which takes you through the steps of getting the LED connected; he does a nice job of breaking the narrative into digestible chunks and interspersing concept explanations to help orient beginners. (I still find the multiplicity of GPIO pin numbering and addressing schemes confusing, but I don't at all fault Gordon for that). Following his instructions we quickly had the LED wired up and connected to our Pi, which we then tested with the gpio utility, prepackaged with Raspbian Jessie, that can be used to communicate with the Pi's GPIO bus. From the console we ran these commands:

The pattern of these commands is gpio <subcommand> <pin> <value>, so the first command translates roughly to “set GPIO user pin 0 to output mode.” GPIO pins can be configured for input or output, depending on whether you are reading from a sensor or driving a motor or LED. Once the pin was configured, we used the second command to turn the LED on and the third to turn it off.

With that working, the obvious next step was to control the LED using Python. There are several Python libraries for interacting with GPIO; we used GPIO Zero, as documented in this tutorial by the Raspberry Pi Foundation. Here's the same sequence of commands as above, but this time using Python:

Instead of interacting directly with the bus, GPIO Zero provides a number of utility classes for dealing with commonly used devices; unsurprisingly the LED class is used to control an LED. The class encapsulates the fact that the LED is an output device and configures the pin to output mode for us when we instantiate it.

Why 17? That's the pin number according to the GPIO numbering scheme, which is equivalent to wiringPi user pin 0 (hence the 0 in the gpio commands above); both refer to physical pin 11 on the board. Ahem.

Pin numbers aside, our next challenge was to make the LED blink, which we accomplished with this simple Python script:

By this point I'd pretty much exhausted the boys' attention capacity, so we got out coloring supplies and drew some Christmas trees. Taking the LED we'd wired up and pushing it through the cardboard yielded a simple Raspberry Pi-powered Christmas display! Over the next few days I added three additional LEDs and some programming to expand the display:

So was this project successful in establishing a deeper interest in the workings of computing technology with my children? Time will tell, but one gain afforded by our little hack is that it has given us a platform to talk about how other things work. They have many small toys that use microcontrollers to control lights, drive motors, react to button presses, and play sounds and music; seeing the construction of this simple project has let me draw their attention to the basic workings of such devices via analogy to our Pi tree. At least in a small way I think they are beginning to understand that electronic devices are not magic and to see how we might build simple devices like the toys they're used to.

My next project? I'm working on building a Raspberry Pi-based geotagger for my Nikon DSLR; I'll be documenting the steps in a series of posts over the coming weeks, so stay tuned!

* * *

Resources

• source code for our Christmas tree display
• Raspberry Pi Learning Resources from the Raspberry Pi Foundation
• GPIO Zero library documentation
• Gordon's Projects GPIO Examples
• Gadgetoid has a nice interactive pinout guide

---

[1]	I recommend the Raspberry Pi 2 Model B, but the older models should work as well. Avoid the Raspberry Pi Zero when you're starting out: the Zero is incredibly cool and tiny, but it is less user-friendly than the other models as it lacks some of their connectivity conveniences.

[2]	There are serveral other OS options for the Pi, and you should obviously go play around with all of them. The examples in this post assume the presence of several supporting tools and libraries that are preloaded on Raspbian Jessie.

Of Dictionaries and Analysis

I greatly enjoyed reading Andrew Kuchling's Beautiful Code article “Python's Dictionary Implementation: Being All Things to All People,” [1] in which Kuchling explores some of the implementation details of what is arguably CPython's most important data structure: the dict. While these implementation details themselves are fascinating, the article offers a deft illumination of the thought process that led to the selection of a given implementation over its alternatives. The lessons here are not just for open source hackers working on the performance-critical portions of a popular language: they offer sound process guidance to anyone needing to make wise choices in the face of conflicting needs and trade-offs when building any kind of software.

Kuchling summarizes his main thesis about halfway through the article:

When trying to be all things to all people — a time- and memory-efficient data type for Python users, an internal data structure used as part of the interpreter's implementation, and a readable and maintainable code base for Python's developers — it's necessary to complicate a pure, theoretically elegant implementation with special-case code for particular cases… but not too much.

The brilliance of the article rests on how Kuchling shows the available trade-offs for a given problem explicitly, always couching them in reference to those user groups and their real-world needs. It's interesting as a postmortem, but also offers a valuable blueprint for structuring design processes such that compromises get debated on their specific, explicitly articulated merits and shortcomings rather than on hidden assumptions, preferences, and biases. Once the options are on the table, it becomes possible to consider their strengths and weaknesses in light of the needs of affected users rather than having to rely solely on theoretical or aesthetic considerations.

With that in mind, here are some specific lessons we can draw from the article about how to structure the process of analyzing potential software solutions.

Lesson 1: Maintainability Matters, Therefore Maintainers Matter

An important but subtle take away from Kuchling's thesis is that developers form one of the user groups with a stake in the outcome, and that therefore code readability and maintainability constitute legitimate, first-order user concerns that deserve weighted consideration alongside the needs of other user groups. Although Kuchling doesn't provide specific examples demonstrating this trade-off in action, this idea forms a theme woven throughout the entire discussion: unless you have a compelling, measured reason not to, prefer simplicity.

This can be challenging conversation to have, especially in commercial contexts with their strong emphasis on feature delivery and satisfying client user demands. A mechanism we use to allow compromise on this front, which we can call the “maintainability gradient,” suggests that changes that have broader reach or usage mandate a higher level of scrutiny with respect to complexity and maintainability. When considering a client request, for each solution under discussion the development team offers an assessment of its maintainability: the solution's inherent complexity, the ease or difficulty of implementing it within the framework and underlying assumptions of the existing platform, especially under time constraints, and the likelihood that we will be required to tinker with it after delivery. This becomes one of the inputs that helps us determine where in the layered architecture a given solution will be allowed:

Figure 1: Our layered architecture. The top of the stack is the most-specific, least-shared code, while the bottom layers are most shared.

In this context, our maintainability gradient provides guidance about where complication is acceptable and where it is not: while we always strive to create simple solutions, if an end user insists on something complicated or messy, it is shunted to client-specific code where its impact on the platform can be minimized and contained. Conversely, proposed additions to the core platform must meet a higher standard of maintainability: does this change fit within the existing context of the platform? Can it be implemented simply and readably?

Note that this is not an excuse to do slipshod work, even in obscure cul-de-sacs of custom code. Quite the opposite: this approach empowers developers to do great work everywhere within the context of a resource-constrained commercial enterprise. If a solution can't be maintainably generalized under current constraints, we can implement a smaller, specific version where the solution is easier to isolate. We don't always get the trade-off right, but this architecture gives us an axis of compromise that allows maintainability to have a voice in the conversation.

Lesson 2: Whom Do You Serve?

Another key design process lesson from Kuchling's article is that it is imperative to understand users and their needs, and to evaluate proposed solutions in light of those needs. Concretely, this means

1. listing each alternative explicitly
2. describing the characteristics of each solution relative to each user group
3. choosing the solution that is the best compromise among the various alternatives

The short section on handling hash collisions in the article illustrates this process nicely. The problem is this: when implementing a hash table, what do we do when two keys inevitably hash to the same slot?

One option is to use chaining: instead of single values, each slot holds a list of values with the same hash. CPython does not do this “because creating linked lists would require allocating memory for each list item, a relatively slow operation.” In other words, CPython does not use chaining because this solution fails to meet the needs of at least two of the important user groups identified earlier: end-users of Python needing an efficient mapping data structure, and the language implementors who need an efficient platform on which to build objects, modules, and functions.

The next option up for consideration is linear probing: if a needed hash slot $i$ is occupied, examine $i+1$, $i+2$, etc. This solution is quite simple to implement, but again CPython does not use it. Why not? “Many programs use consecutive integers as keys,” which, due to how CPython hashes integers, consume contiguous blocks of the hash table that are inefficient for linear probing. Having identified the performance characteristics of this solution explicitly, it is easy to see the trade-offs being made here and the user groups affected by those trade-offs: linear probing is simple to implement — a boon for the maintainers — but that simplicity yields a solution that does not serve the performance needs of the language implementors or end-users very well.

This analysis is predicated on both understanding who the users are and what their usage patterns look like while simultaneously understanding the spectrum of available solutions considered in light of the needs of those various user groups. In the end, CPython's hash collision resolution algorithm is a pragmatic compromise, guided by experimentation, measurement, and feedback, that concedes a little code complexity in exchange for good performance characteristics in the real world.

Lesson 3: … but not too much

My favorite optimization that Kuchling describes is a complexity/performance trade-off. The dilemma is this: how can the dict implementation serve a general audience — by handling non-string keys, rich comparison operators, and possible exceptions — while also offering good performance for the huge swath of dict usages with exclusively string keys, such as those that underlie the implementation of classes, modules, and functions?

One approach would be to implement separate classes: a specialized, string-only dict optimized for high-performance for this use case along with a more general purpose implementation for regular use. In fact this is exactly what Jython, the Python implementation that runs on the JVM, does: it utilizes Java's java.util.HashMap as the basis for the dict implementation for general purpose use but employs a separate, optimized class to handle the performance-sensitive language implementation bits.

CPython instead uses a single implementation but employs two different lookup functions. By default, dicts are assumed to contain only string keys, and since many of the complexities and failure modes that arise when comparing two arbitrary objects are impossible when comparing two strings, the default lookup function can be optimized to eliminate checks for cases or errors that simply cannot happen. The implementation is a classic use of indirection that relies on a simple function pointer: if an arbitrary, non-string object is searched for or inserted into a dict instance, the search function pointer for that instance is updated to reference the slower but more general purpose lookup implementation that can handle the added complexities concomitant with comparing arbitrary objects.

This example beautifully illuminates a pragmatic, “not too much” compromise that serves its various users well. For the language implementors, the solution provides an important performance boost for lookups in classes and modules and passing of keyword arguments to functions, all of which are exclusively string- keyed and are the most frequent operations in a running Python program. It is also fairly simple, easy to understand, and is likely less code than a two-class solution would be. This in turn helps minimize the potential for implementation drift and maintains the “fits in your head” quality valued by the Python community.

The compromise also serves end users very well: string-keyed dicts are very common in many applications, so end users reap a double benefit: attribute lookups and function calls are faster, and, as an added bonus, portions of their systems that rely on string-keyed dicts are also sped up.

* * *

[1]	I couldn't find the article online except at Safari (free trial available).

Clean and Green: Pragmatic Architecture Patterns

Thanks to everyone who came to my PyTennessee 2015 talk, Clean and Green: Pragmatic Architecture Patterns! 

• screencast of the presentation
• the slides include additional material we did not have time to cover

If you are interested in learning more about the Clean Architecture, the following resources provide a wealth of information about the theory and practice:

• Brandon Rhodes, The Clean Architecture in Python, (slides, video)
• Uncle Bob Martin, The Clean Architecture, (and reactions)
• Gary Bernhardt, Boundaries
• J B Rainsberger, Integrated Tests are a Scam

Customizable Site Templates with Stevedore

A Pluggable Python Application Case Study

Python makes it very easy to load code dynamically; handling the subtle bugs and sometimes bizarre failure modes that arise in home-grown module loading systems is another matter entirely. stevedore provides a framework for managing plugins based on the patterns and needs demonstrated by real applications and systems. If you want to implement a plugin architecture for your application, stevedore will accelerate your development by allowing you to focus on application abstractions instead of plugin loading mechanisms.

In this post, I'd like to

• demonstrate the basic ideas and mechanisms used in stevedore, and
• illustrate the concepts with a practical example drawn from a production system

Follow along by grabbing the case source, which includes the IPython notebook source of this post and an example web application demonstrating the techniques described here.

Why?¶

Building applications is hard. Many forces exert influence on a software project as it ages, and the physics of software imply that, over time, these forces will tend to increase the complexity of the software. Features creep in; "temporary" hacks endure, becoming larger and more entrenched; code begins to smell and then to rot. In the swirling chaos of time pressures and user demands, how do we apply sound design principles and create software suffused with simplicity and elegance?

Simplicity in software is the result of deliberate, focused, and continuous attention and cultivation; in a very real sense, it is "unnatural." Yet like any craft, building sound software is a practice that can be learned and mastered, and there is much wisdom and shared experience available to us with an ease and at a scale that is historically unprecedented. Principles like the Single Responsibility Principle encourage us to envision software composed of simple interacting components; if we cannot avoid complexity, ideas like abstraction and encapsulation suggest that we bury it, decoupling it and hiding it away from the rest of our application as far as possible. Taking these ideas at scale, a common way to build applications is to use a plugin architecture, applying the same principles used when designing clean functions at the coarser level of granularity of an entire application. Rather than being conceived as monolithic entities, pluggable applications define a core framework and rely on interchangeable, communicating components to define policy, implement business rules, and interact with the world.

What is stevedore?¶

stevedore is Doug Hellmann's system for managing dynamic plugins in Python. It was created after Doug conducted an extensive taxonomy of the varied (and varying quality) plugin mechanisms used by popular Python systems, which revealed several common interaction patterns used in these high profile applications. For example, many applications have the notion of a driver plugin, which allows an application to be written in terms of abstract representations of resources—say a database or a network interface—with plugins providing the implementations of those abstractions tailored to a specific resource type.

Rather than reinventing the plugin loading wheel (again), stevedore provides a set of higher-level abstractions over existing dynamic discovery and loading machinery. It utilizes setuptools entry points as the mechanism for defining plugins. For example, this entry_points definition in setup.py defines three entry points in the sweet_app.templates namespace: base, shared, and client_custom.

setup(
# ...
entry_points="""
    [console_scripts]
    serve_example = base_app.main:main
    enumerate_extensions = demo:enumerate_template_extensions

    [sweet_app.templates]
    base = base_app.templates
    shared = shared_lib.templates
    client_custom = client_custom.templates
"""
)

In this example, sweet_app.templates is the entry point namespace, inside of which I've declared three entry points: base, shared, and client_custom. The right-hand side defines the entry point target, which can be either a Python package or an attribute within a package.

The namespace serves as the collection point for all plugins of a certain kind within an application. As with all things software development, naming namespaces and selecting the right number of them is something of an art. One rule of thumb I am illustrating here is to prefix the namespaces for an application with an application identifier: templates on its own is too generic to make a good namespace; sweet_app.templates identifies these plugins as the templates belonging to this sweet app.

Just for fun, I've also declared two console_scripts that can be run from the command line once you've installed this post's sample code. We'll come back to serve_example, but enumerate_extensions is a bit of a stevedore "hello world" that demonstrates the basic mechanics by enumerating the extensions in the templates namespace:

In [1]:

%load demo/__init__.py

In [2]:

import logging

from stevedore import ExtensionManager


def enumerate_template_extensions():
    # logging.basicConfig(level=logging.DEBUG)
    namespace = 'sweet_app.templates'
    manager = ExtensionManager(namespace)

    print 'Extensions in namespace {}:'.format(namespace)

    for extension in manager:
        print '\t{} (module {})'.format(extension.name,
                                        extension.entry_point.module_name)

In [3]:

enumerate_template_extensions()

Extensions in namespace sweet_app.templates:
 shared (module shared_lib.templates)
 base (module base_app.templates)
 client_custom (module client_custom.templates)

Here's how it works: stevedore provides a set of manager classes to, well, manage the entry points defined by an application. In this example we use an ExtensionManager, the most generic manager, to show the plugins defined in a given namespace. First, create the manager, passing the desired namespace to its constructor (line 9 above). Manager objects are iterable, returning the list of discovered extensions in the supplied namespace. Each returned object is an Extension, a small bookkeeping wrapper around an entry point:

In [4]:

manager = ExtensionManager('sweet_app.templates')
extension = next(iter(manager))
type(extension)

Out[4]:

stevedore.extension.Extension

The extension exposes the entry point target (either a module or an attribute within a module, say a class or a function) as its plugin attribute:

In [5]:

extension.plugin

Out[5]:

<module 'shared_lib.templates' from 'shared_lib/templates/__init__.pyc'>

The rest of the example merely iterates over the plugins discovered in the templates namespace, printing out the name and module target of each.

Here is a quick summary of the main concepts used in stevedore:

• a plugin is simply a Python module or an attribute (function, class, etc.) within a module
• namespaces collect an application's plugins into logical groups
• plugins are defined using setuptools entry points as part of a normal Python distribution
• stevedore provides a set of manager classes that offer higher-level abstractions over the setuptools plugin machinery

Notice that stevedore does not provide any sort of application plugin framework, nor does it have any particular opinion about what plugins are or how they work, since these concepts are generally assumed to be part of the definition of the application. In other words, stevedore intends to provide higher-level tools that can be used to build an application's plugin system without having to reimplement the dynamic code loading mechanics. That said, stevedore's documentation offers a nice tutorial with tips for getting started and a worked out example of a small plugin architecture.

As a complement to stevedore's own documentation, let's have a look at another example: a pluggable template resolution system drawn from a production web application.

The Case: Customizable Web Templates¶

Imagine for a moment that you're stuck building applications with an ancient web stack (ahem) and that you have Django envy: one of Django's nicer features is its pluggable application mechanism, which allows developers to build systems by composing small, focused subcomponents (what Django calls "apps," not to be confused with those shiny doodads on your phone). Django apps are a powerful abstraction mechanism that can utilize the full gamut of features in the Django ecosystem: they can define new ORM models, create forms, hook in to the application's URL space, notify observers of interesting events, and even override templates defined in another app to extend or augment functionality.

Lets use stevedore to build an implementation of Django's template override mechanism for our hypothetical ancient web stack; our goal here will be to demonstrate stevedore in a realistic scenario by using it to build just the template discovery portion of Django's app system (the other bits left as an exercise for the reader ;). Here's the idea: I have a base web application that defines a common structure and style for the app, including a master template that groups that structure into blocks. The base application provides definitions for each block that can be overridden by more specialized versions defined in other packages:

In [6]:

from IPython.display import SVG
SVG('./site_outline.svg')

Out[6]:

Each block is defined in a separate HTML file that the template system stitches together to produce the final rendered HTML. This is pretty normal fare for template systems; the special sauce that we are going to add is a custom template resolver that knows how to look for templates contained in a configurable set of locations rather than a single template directory. The resolution process requires us to define the template roots, the packages containing the various collections of templates, and the resolution order, which determines the precedence that will be used when searching the various packages for templates. As an example, the setup.py file shown above defines three template roots in the imaginatively-named templates namespace that correspond to three packages in the sample code:

In [9]:

!tree -P "*.html" client_custom shared_lib base_app

client_custom
└── templates
    └── nav.html
shared_lib
└── templates
    └── jumbo.html
base_app
└── templates
    ├── admin
    │   └── index.html
    ├── index.html
    ├── jumbo.html
    └── nav.html

4 directories, 6 files

Note that the three packages share a common directory structure; overriding a template means placing a file with the same name at the same location in a template package with higher resolution precedence. For the sake of convenience the example houses all of these packages in the same distribution, but an actual application would likely organize things into separate, independently installable distributions.

To perform template resolution, we need to

1. Determine the namespace that houses the template plugins, the extension names to use, and the resolution order. In our production system, we use a fixed shared namespace (templates) and each application deployment determines the extension names and resolution order from configuration.

2. Create an ExtensionManager instance to load the template plugin packages. stevedore's NamedExtensionManager matches our use case perfectly, allowing an application to supply a named list of extensions to use within a given namespace. This means that we can organize our package code as best fits our application independent of the concern of plugin activation. The actual use of a given plugin will be set explicitly at runtime, usually through some form of application configuration passed to NamedExtensionManager's constructor.

3. At resolution time, search the list of extensions for the requested template name, returning the first one found. We use package relative naming: the template name admin.index means "search for the template named index.html in plugin_package.admin, where plugin_package is expanded to each of the plugins in turn." With a little tweaking, the same idea could be used with path-based naming if that is more your thing.

And again, in Python:

In [10]:

from resolver import TemplateResolver

# step 1: define the plugins we want to look for templates in and the resolution order
plugin_names = ['client_custom', 'shared', 'base']

# step 2: create the ExtensionManager (part of the constructor, hang tight)
resolve = TemplateResolver(plugin_names)

# step 3: resolve a template name to a file path
resolve('index')

Out[10]:

'base_app/templates/index.html'

Remember that blaring trumpet sound that Windows used to start with? That belongs here: Ta Da!

Here's a juicier example that shows off overriding, with nav being pulled from the client custom package and the jumbotron from a shared library of templates:

In [12]:

import pandas as pd

templates = [(template, resolve(template)) for template in ('nav', 'jumbo', 'admin.index')]

pd.DataFrame(templates, columns=['template_name', 'resolved_template'])

Out[12]:

	template_name	resolved_template
0	nav	client_custom/templates/nav.html
1	jumbo	shared_lib/templates/jumbo.html
2	admin.index	base_app/templates/admin/index.html

3 rows × 2 columns

and to continue beating things so they are good and dead, here's what would happen if you changed the list of plugins to use:

In [13]:

plugin_names = ['base']
resolve = TemplateResolver(plugin_names)

resolve('nav')

Out[13]:

'base_app/templates/nav.html'

Note that we haven't changed anything about the deployment or plugin set up at all, merely the list of extensions we want to select from the universe of available plugins in the namespace.

If you'd like to see these concepts in action, activate the environment in which you installed the sample code accompanying this post and run the command

serve_example

then browse to http://localhost:5000, which will fire up a sample application demonstrating this technique in a small web application. Pay particular attention to the tabbed navigation in the upper right and the large Jumbotron in the center of the application, both of which exhibit the overriding behavior detailed above.

But dan, how does it all work?¶

Here's the payoff: stevedore makes implementing the ideas I've described here remarkably simple. Check it:

In [14]:

%load -r 20-40 ./resolver/__init__.py

In [15]:

class TemplateResolver(object):
    def __init__(self, template_modules, namespace='templates'):
        self.template_modules = template_modules
        self.extension_manager = NamedExtensionManager(namespace,
                                                       template_modules,
                                                       name_order=True)

    def __call__(self, name):
        package_suffix, target_file, join = self._process_name(name)

        for extension in self.extension_manager.extensions:
            target_package = join((extension.entry_point.module_name,
                                   package_suffix))

            if resource_exists(target_package, target_file):
                return resource_filename(target_package, target_file)
        else:
            raise ValueError(
                'Could not locate template "{}" in any loaded template '
                'module: {}'.format(name, self.template_modules))

Yep. That's it: lines 4–6 initialize the extension manager (our step #2 above), and lines 11–16 are the heart of the resolver (step #3). The NamedExtensionManager provides two helpful features for this use case. First, it filters all of the available extensions within a namespace to an explicitly-declared list, which allows us to specify the desired template roots in configuration. It also allows you to impose an ordering for the extensions: setting name_order=True means that the extensions will be returned from the manager in the same order as the name list we supply to the manager's constructor (without this option we would get arbitrary ordering: entry point load order is effectively undefined from an application's perspective).

Resolution is straightforward: after processing the requested name to handle dotted package paths (line 9), the resolver loops through the enabled extensions and looks to see if the package referenced by the extension contains the requested file. As soon as there's a match, resolution ends by returning the file path of the requested template (resource_exists and resource_filename are functions for looking up files contained within Python packages; both are provided by pkg_resources).

If no package contains the requested template, an exception is raised; the exception's message contains the list of activated template packages, as misconfiguration here is the most common source of failure we run into:

In [16]:

resolve('notgonnawork')

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-16-6e96b4ec4eb9> in <module>()
----> 1 resolve('notgonnawork')

/home/dan/source/posts/stevedore_template_resolver/resolver/__init__.pyc in __call__(self, name)
     37             raise ValueError(
     38                 'Could not locate template "{}" in any loaded template '
---> 39                 'module: {}'.format(name, self.template_modules))
     40 
     41     def _process_name(self, name):

ValueError: Could not locate template "notgonnawork" in any loaded template module: ['base']

Takeaways¶

1. A plugin architecture can help you simplify, simply

A plugin architecture encourages you to think about applications in terms of abstractions and interfaces. Rather than relying on conditional logic to accommodate alternate workflows and implementations, a plugin architecture allows an application to work at a consistent level of abstraction, deferring implementation details to plugins behind a defined interface. While this is commonly cited as an advantage for allowing third parties to provide concrete implementations (e.g. drivers), it can also significantly improve the internal implementation of an application by encouraging a clean separation between high-level abstractions and implementation details.

But that does not mean that a plugin architecture has to be complicated! Plugins are simply functions, classes, or modules that are exposed by name via the entry points machinery; stevedore, as we saw above, makes retrieving plugins almost trivially easy…

2. stevedore understands entry points so you don't have to

stevedore is built for application developers, not packaging gurus: to use it, simply label a function, class, or module as an entry point and choose the appropriate manager. The managers it provides represent the vast majority of plugin loading patterns used by major applications in the Python ecosystem and allow your application to discover, load, and interact with plugins based on the desired usage pattern. Developers can spend their time designing clean application interfaces rather than reinventing dynamic code loading or digging through the documentation for entry points.

3. These ideas are widely applicable; stevedore makes it easy to get started

It is probably obvious that template resolution is not the first application that comes to mind when people talk about plugin architectures. Yet this example, while perhaps somewhat atypical, illustrates the flexibility of these ideas and the ease with which developers can get started using stevedore. With very little effort, we were able to significantly extend the capabilities of our existing application, offering flexibility and customization capacity to our customers while reducing our ongoing maintenance burden: where we once would have resorted to conditional logic, we can now offer client customizations by merely deploying the requested customization, which is automatically picked up by the resolution system.

Resources¶

• Using stevedore in Your Application and stevedore's documentation home
• Use stevedore's Launchpad page to request features or file bug reports

Special thanks to Doug Hellmann for reviewing an earlier draft of this post.

A Native Win32 Application in Python

I'm currently working on a small personal project, one component of which needs to use a native Windows edit control. Without getting too bogged down in the why for the moment, I have started exploring the possibility of creating a native implementation in Python using pywin32, rather than relying on a bridge layer such as IronPython/.NET or wx.

What is pywin32? pywin32 is a set of Python bindings over a large chunk of the Windows API. That API presents a formidable visage to a non-Windows developer like me, with its bewildering naming conventions, indistinguishable groupings and boundaries, and sheer age and scale. pywin32 lets you work with much of this API within the comfort of Python, handling many of the tricky details like type marshaling and function/pointer conversions for you. However, it does assume a high degree of familiarity with the Windows API, and the documentation, while a useful reference, doesn't do much to mitigate the difficulties of approaching that API for a beginner.

My goal in this post is to demonstrate how to create a native dialog-based window with Python and pywin32. Our little app will contain a RichEdit text editing control and demonstrate two mechanisms for receiving messages: the parent window uses a message map to route messages to Python methods, but we'll also see how to override the edit control's WndProc function, which offers absolute control over its behavior.

The core of the app is derived from the win32gui_dialog sample in pywin32. If you'd like to try it out yourself, the entire source is here. I'm running Python 2.7.5 32-bit with pywin32 build 218 on Windows XP SP3.

Let's get started!

import win32¶

As usual, we begin with some essential imports.

In [16]:

import struct

import win32gui_struct
import win32api
import win32con
import winerror
import winxpgui as win32gui

win32con is a sort of win32.h header file for Python, defining many of the constants and flags needed when making any meaningful Windows calls. win32api provides access to the core Windows API, including facilities for loading and accessing dynamic libraries.

But it is win32gui that provides all the critical calls needed to create a native Windows application. pywin32 supplies two versions: winxpgui is identical to win32gui except for being built with manifest settings for Windows XP, allowing it to take advantage of certain window system features like the updated theme support built into that version. As I was working, I found the C source for win32gui to be a helpful guide out of a few sticky spots.

win32con defines many, but not all, of the constants needed by our little app. I'm going to define a few of the missing ones I know we'll need now but defer the explanation of what they are for until we need them.

In [17]:

EM_GETEVENTMASK = win32con.WM_USER + 59
EM_SETEVENTMASK = win32con.WM_USER + 69
EM_SETTEXTEX = win32con.WM_USER + 97
EM_EXSETSEL = 1079

As usual when I'm doing any sort of exploratory work in Python, I rely on IPython's object querying facilities to interrogate module attributes and interfaces. For example, this search locates all the attributes in win32con that start with EM and contain the string TEXT; the results confirm the need for the definition of EM_SETTEXTEX above:

In [43]:

win32con.EM*TEXT*?

       win32con.EMR_EXTTEXTOUTA
       win32con.EMR_EXTTEXTOUTW
       win32con.EMR_POLYTEXTOUTA
       win32con.EMR_POLYTEXTOUTW
       win32con.EMR_SCALEVIEWPORTEXTEX
       win32con.EMR_SETTEXTALIGN
       win32con.EMR_SETTEXTCOLOR
       win32con.EMR_SETVIEWPORTEXTEX
       win32con.EM_GETLIMITTEXT
       win32con.EM_LIMITTEXT
       win32con.EM_SETLIMITTEXT

Having established the constants we'll need, we next define a local identifier for our edit control:

In []:

IDC_EDIT = 1024

The IDC prefix seems to be a Windows convention that I guess stands for IDentifier for Control; this constant can be used to retrieve the control's handle and will allow us to filter messages in the parent window later.

The last of the initialization steps get the Windows controls ready for use. In addition to initializing the standard controls, I'm also going to load the RichEdit 4.1 control since that's the version I'd like to use. Even my plain vanilla install of XP probably has at least 3 versions of RichEdit available, so you need to be specific if you want to take advantage of features in the later versions. Be warned: the versioning scheme for this control is completely inscrutable.

In [18]:

win32gui.InitCommonControls()
hInstance = win32gui.dllhandle

# Load the RichEdit 4.1 control
win32api.LoadLibrary('MSFTEDIT.dll')

Out[18]:

1262485504

Scaffolding¶

With the initialization out of the way, we can turn to the task of defining the Python class to encapsulate our window. A typical first chore in any Windows application is to register a window class (WNDCLASS) for your application's main window which defines the options the window system will use when drawing the window. The meat of this registration code is drawn directly from the win32gui_dialog sample; I have simplified it slightly and added the guard to ensure that the registration happens only once.

In [31]:

class DemoWindow(object):
    class_name = "Pywin32DialogDemo"
    class_atom = None

    @classmethod
    def _register_wnd_class(cls):
        if cls.class_atom:
            return

        message_map = {}
        wc = win32gui.WNDCLASS()
        wc.SetDialogProc()  # Make it a dialog class.
        wc.hInstance = hInstance
        wc.lpszClassName = cls.class_name
        wc.style = win32con.CS_VREDRAW | win32con.CS_HREDRAW
        wc.hCursor = win32gui.LoadCursor(0, win32con.IDC_ARROW)
        wc.hbrBackground = win32con.COLOR_WINDOW + 1
        wc.lpfnWndProc = message_map  # could also specify a wndproc.
        # C code: wc.cbWndExtra = DLGWINDOWEXTRA + sizeof(HBRUSH)
        #                         + (sizeof(COLORREF));
        wc.cbWndExtra = win32con.DLGWINDOWEXTRA + struct.calcsize("Pi")

        # load icon from python executable
        this_app = win32api.GetModuleHandle(None)
        wc.hIcon = win32gui.LoadIcon(this_app, 1)

        try:
            cls.class_atom = win32gui.RegisterClass(wc)
        except win32gui.error, err_info:
            if err_info.winerror != winerror.ERROR_CLASS_ALREADY_EXISTS:
                raise

Next we define the configuration for the specific window instance we want to create in the form of a dialog template. The Windows API provides some helper functions that allow dialogs to be created via a list of templates: the template for the window itself followed by any number of templates for items to go into the window. As with the window class registration, the options I'm using here are straight out of the sample; MSDN has more information about the available window style options and the dialog specific options.

In []:

    @classmethod
    def _get_dialog_template(cls):
        title = "pywin32 Dialog Demo"
        style = (win32con.WS_THICKFRAME | win32con.WS_POPUP |
                 win32con.WS_VISIBLE | win32con.WS_CAPTION |
                 win32con.WS_SYSMENU | win32con.DS_SETFONT |
                 win32con.WS_MINIMIZEBOX)

        # These are "dialog coordinates," not pixels. Sigh.
        bounds = 0, 0, 250, 210

        # Window frame and title, a PyDLGTEMPLATE object
        dialog = [(title, bounds, style, None, None, None, cls.class_name), ]

        return dialog

At last we get to the constructor, which registers the window class (if necessary), gets the dialog template, and asks the windowing system to create the dialog window:

In []:

    def __init__(self):
        message_map = {
            win32con.WM_SIZE: self.on_size,
            win32con.WM_COMMAND: self.on_command,
            win32con.WM_NOTIFY: self.on_notify,
            win32con.WM_INITDIALOG: self.on_init_dialog,
            win32con.WM_CLOSE: self.on_close,
            win32con.WM_DESTROY: self.on_destroy,
        }

        self._register_wnd_class()
        template = self._get_dialog_template()

        # Create the window via CreateDialogBoxIndirect - it can then
        # work as a "normal" window, once a message loop is established.
        win32gui.CreateDialogIndirect(hInstance, template, 0, message_map)

Up to this point, the code we've seen is, syntactic differences aside, more or less what you would find in a C or C++ program. Here we come to the first significant difference between a Python and C/C++ implementation: message dispatching. Traditional Windows applications use a "window procedure" (WndProc) to handle the messages that a control receives throughout its existence. These procedures are the brain of the control, defining how it is painted, how it responds to user input, and what control-specific capabilities it offers. They are typically implemented as a switch statement that dispatches messages received from the window system to appropriate handlers.

As we will see shortly, the same capability is available in Python, but pywin32 offers an alternative in the form of a message_map, a dict mapping message identifiers to Python callables. When defined this way, pywin32 builds a small wrapper procedure that uses the message_map to dispatch incoming messages to the appropriate handler in Python. If no handler is defined in the map, the default handler for the control is called. This mechanism makes it easy to subscribe to only messages of interest using whatever Python callable type happens to be convenient. Here we're using methods, but a function or other callable would work just as well.

The main window of our application is listening for two basic kinds of messages: notifications from the windowing system (WM_SIZE, WM_INITDIALOG, WM_CLOSE, and WM_DESTROY) and notifications from child controls (WM_COMMAND and WM_NOTIFY). What child controls? Erm… Right! Child controls, I guess we'll need some of those!

Control Your Edits¶

If you've made it this far and are working on your own application (rather than using notepad), chances are very high that you want to do something other than the default behavior for certain events. In my case, I am primarily interested in three kinds of changes: changes to the text, changes to the selection state, and keystrokes that did not alter the text.

We'll examine the last two cases later, but to detect text changes, I'm going to use a custom WndProc for our (yet to be created) RichEdit control:

In []:

    def edit_proc(self, hwnd, msg, wparam, lparam):
        print 'edit_proc', hwnd, msg, wparam, lparam

        if msg == EM_SETTEXTEX:
            print 'EM_SETTEXTEX'
        elif msg == win32con.WM_SETTEXT:
            print 'WM_SETTEXT'
        elif msg == win32con.EM_REPLACESEL:
            print 'EM_REPLACESEL'

        # I'm not sure why this is needed (i.e. not handled by the default
        # handler), but without it the process hangs on window close.
        elif msg == win32con.WM_DESTROY:
            win32gui.DestroyWindow(hwnd)
            return

        # "super" call to the overridden WndProc
        return win32gui.CallWindowProc(self.old_edit_proc, hwnd, msg, wparam,
                                       lparam)

This method redefines how our edit control will process messages it receives. The msg parameter contains the message identifier that indicates what the message is and determines the meaning of the wparam and lparam arguments. Many of these messages are control-specific, and RichEdit boasts a rich set of messages it understands.

I'm primarily interested in text changes, which are one type of message that can be sent to the control. For this example, when the control receives a message to change all or part of the text (EM_SETTEXTEX, WM_SETTEXT, or EM_REPLACESEL), we simply print out the receipt of the message and continue; an actual application could take any desired action here.

A WndProc can preserve the inherited behavior of the control by calling the default procedure for the control using CallWindowProc; of course nothing requires us to send the same message we received up the chain…

So where did this mysterious old_edit_proc come from? I'm glad you asked!

In []:

    def _setup_edit(self):
        class_name = 'RICHEDIT50W'
        initial_text = ''
        child_style = (win32con.WS_CHILD | win32con.WS_VISIBLE |
                       win32con.WS_HSCROLL | win32con.WS_VSCROLL |
                       win32con.WS_TABSTOP | win32con.WS_VSCROLL |
                       win32con.ES_MULTILINE | win32con.ES_WANTRETURN)
        parent = self.hwnd

Not there!

Following the lead of the win32gui_dialog example, I have deferred the creation of the edit control to the handler for WM_INITDIALOG, which calls _setup_edit to actually create the edit control and install it in our window. If there is a non-whimsical reason for doing it this way, as opposed to installing the control via the dialog template mechanism above, I don't know what it is. I'll make something up if you like.

Regardless, we first establish the basic parameters we want for the control, then create it:

In []:

        self.edit_hwnd = win32gui.CreateWindow(
            class_name, initial_text, child_style,
            0, 0, 0, 0,
            parent, IDC_EDIT, hInstance, None)

Surprise! Yup, you read that correctly: the command used to create a RichEdit control, CreateWindow, is the same command that is used in Windows to create… (drumroll)… windows! I guess it's windows all the way down…

A few notes about this call: the class_name indicates that we want a RichEdit 4.1 control. Yes it does. I warned you that the versioning is weird… We also set the parent of our edit control to be our dialog window and set the local identifier for the control to our fortuitously predefined constant IDC_EDIT. The zeros specify the bounds of the control; we have to pass something, but I want the control to fill the window, which we will handle later.

Still looking for old_edit_proc are you?

The next thing we do is to tell the edit control to notify its parent (our dialog window) of keystroke events, selection changes, and text updates. To do this, we send an EM_SETEVENTMASK message to the RichEdit control, indicating in the message payload which messages should be sent:

In []:

        message_mask = (win32con.ENM_KEYEVENTS | win32con.ENM_SELCHANGE |
                        win32con.ENM_CHANGE | win32con.ENM_UPDATE)
        win32gui.SendMessage(self.edit_hwnd, EM_SETEVENTMASK, 0, message_mask)

Finally, we install the custom edit WndProc we defined above, and, in a fit of nostalgia, save a reference to the old one. (Actually, we save it because we need it…)

In []:

        self.old_edit_proc = win32gui.SetWindowLong(self.edit_hwnd,
                                                    win32con.GWL_WNDPROC,
                                                    self.edit_proc)

There's that rascal old_edit_proc! When changing a control's WndProc, SetWindowLong returns a reference to the old procedure, which we can call by passing the reference to CallWindowProc as above.

So far, we have registered a window class for our main window, created an instance of that window class using a dialog template, added a RichEdit control as a child control of that window, and installed a custom WndProc for the edit control to customize the way the control processes messages. In the next section, we'll see how to configure the notification handlers to allow the main window to respond to notifications it receives from the window system and its contained controls.

Sizing things up¶

The next major section of the application defines the notification handlers referenced earlier in our dialog window's message_map. On initialization, we set up the edit control, then resize it to fill the window:

In []:

    def on_init_dialog(self, hwnd, msg, wparam, lparam):
        self.hwnd = hwnd
        self._setup_edit()
        l, t, r, b = win32gui.GetWindowRect(self.hwnd)
        self._do_size(r, b, 1)

    def _do_size(self, cx, cy, repaint=1):
        # expand the textbox to fill the window
        win32gui.MoveWindow(self.edit_hwnd, 0, 0, cx, cy, repaint)

The resize handler is straightforward: when the user resizes the window, the window system notifies the window with a WM_SIZE message, passing the new size as the lparam payload. Our handler simply extracts the new width and height of the window and passes them to _do_size:

In []:

    def on_size(self, hwnd, msg, wparam, lparam):
        x = win32api.LOWORD(lparam)
        y = win32api.HIWORD(lparam)
        self._do_size(x, y)
        return 1

What's happening?¶

Earlier, we asked the edit control to send certain messages to its parent window. Messages from child controls to their parent use one of two message envelopes, WM_COMMAND or WM_NOTIFY. Which envelope gets used is message-specific and largely a matter of historical accident; RichEdit, being an old but frequently updated control, uses both.

When listening for these messages, the handler will need to examine the envelope to determine the origin control and the type of message being sent. For WM_COMMAND encapsulated messages, lparam is the window handle of the origin control and the upper bits of wparam contain the message identifier:

In []:

    def on_command(self, hwnd, msg, wparam, lparam):
        print 'on_command', hwnd, msg, wparam, lparam

        msg_id = win32api.HIWORD(wparam)
        print msg_id

        if lparam != self.edit_hwnd:
            print 'origin not edit, skipping'
            return 1

        if msg_id == win32con.EN_CHANGE:
            print 'EN_CHANGE'
        elif msg_id == win32con.EN_UPDATE:
            print 'EN_UPDATE'

        return 1

WM_NOTIFY messages are essentially the same but use a different packaging structure: wparam contains the local identifier for the control (e.g. IDC_EDIT) and the message identifier is part of the NMITEMACTIVATE struct passed as lparam. Fortunately for us, win32gui_struct already knows how to parse these structures, so we get the convenience of working with Python objects:

In []:

    def on_notify(self, hwnd, msg, wparam, lparam):
        print 'on_notify', hwnd, msg, wparam, lparam

        info = win32gui_struct.UnpackNMITEMACTIVATE(lparam)

        if wparam != IDC_EDIT:
            print 'origin not edit, skipping'
            return 1

        if info.code == win32con.EN_MSGFILTER:
            print 'EN_MSGFILTER'
        elif info.code == win32con.EN_SELCHANGE:
            print 'EN_SELCHANGE'

        print info.code

        return 1

Wrapping up¶

The last two handlers define our application's shutdown sequence: on_close is triggered when the user closes the window, which gives the application a chance to not close if that is appropriate. For example, this is where a real application would prompt the user if the edit buffer was dirty.

In []:

    def on_close(self, hwnd, msg, wparam, lparam):
        win32gui.DestroyWindow(hwnd)

    def on_destroy(self, hwnd, msg, wparam, lparam):
        # Terminate the app
        win32gui.PostQuitMessage(0)

Our main function is very simple: we create an instance of our window and then start the Windows message pump, which will run until the PostQuitMessage call in on_destroy.

In []:

def main():
    DemoWindow()
    # PumpMessages runs until PostQuitMessage() is called by someone.
    win32gui.PumpMessages()


if __name__ == '__main__':
    main()

Conclusions¶

We have now built a complete, if small, Windows application, including the infrastructure needed to receive and act on messages in both the main window and the edit control. The application is composed of two stages: initialization and message processing. The initialization stage includes registering the window class, setting up configuration parameters for the controls, and wiring up message handling. Message processing involves listening for and reacting to the various forms of messages our application can receive. Our application demonstrates two forms of message processing: the dict-based message_map approach and a control with an overridden WndProc.

I hope that this little tour will help you better navigate the vast and varied terrain in the Windows API if you find yourself needing to write a native application. And what of my motivation, the coy why above? That, dear reader, is a story for another day.

Resources

• the complete source as a gist
• pywin32 download and source root
• Windows and Messages Home on MSDN
• A simpler application in C