News

Using virtualenv and zc.buildout together

cwebber, March 16th, 2010

Virtualenv and zc.buildout are both great ways to develop python packages and deploy collections of packages without needing to touch the system library. They are both fairly similar, but also fairly different.

The primary difference between them is that zc.buildout focuses on having a single package, and all relevant dependencies are installed automatically within that package’s directory via the buildout script (Nathan Yergler points out that you don’t have to use things this way, but that seems to me to be the way things happen in practice… anyway, I’m not a buildout expert). The buildout script is very automagical and does all the configuration and installation of dependencies for you.  Since this is a build system, you can also configure it to do a number of other neat things, such as compile all your gettext catalogs, or scp the latest cheesefile.txt from themoon.example.org… whatever you need to do to build a package.

Virtualenv is mostly the same creature, but it’s like you reached your hand inside and pulled it inside out. Instead of a bunch  of packages installed within a subdirectory of one package, there is a more generic directory layout that allows you to set up a number of packages within it. Installing a package and keeping it up to date is much more manual in general, but also a bit more flexible in the sense that you can switch paths around within the environment fairly easily and simultaneously developing multiple interwoven packages is not difficult.

I came to CC with a lot of experience with virtualenv and no experience with zc.buildout. Initially I could discern no differences of use case between them, but now I have a pretty good sense of when you’d want to use one over the other. An example use case, which has come up pretty often with me actually: say you have two packages, one of which is a dependency on the other. In this case, we’ll use both cc.license and cc.engine, where cc.engine has cc.license as a
dependency.

Now say I’m adding a feature to cc.engine, but this feature also requires that I add something in cc.license. At this point it is
easier for me to switch to using virtualenv; I can set up both development packages in the same virtualenv and use them together.  This is great because it means that I should have little to no difficulty switching back and forth between both of them. If I make a change in cc.license it is immediately available to me in cc.engine.  This also  prevents either having to set up a tedious to switch around configuration checking out cc.license into cc.engine and etc, or making a bunch of unnecessary releases just to make sure things work, etc. It’s easier to work on multiple packages at once in
virtualenv in my experience.

Now let’s assume that we got things in working order, cc.license has the new feature and cc.engine is able to use it properly, tests are passing, and et cetera. At this point is where I think returning to zc.buildout is a good idea. One of the things I like about zc.buildout is that it provides a certain type of integrity checking with the buildout command. If you forget to mark a dependency or even remove it from setup.py on accident or whatever, buildout will simply unlink it from your path the next time you run it. In this case, I think zc.buildout is especially useful because I might forget to make a cc.license release here or some such thing. There are some other reasons for using zc.buildout (as the name implies, buildout is a full build system, so there are a lot of neat things you can do with it), but for a forgetful person such as myself this is by far the most important to me (and the most relevant to this example).

So I’ve described use cases for both cc.engine and cc.license. How do we get them to work nicely together? Let’s assume we just want to check out these packages once. Let’s also assume that our virtualenv directory is ~/env/ccommons (because I’m clearly basing this off my own current setup currently, heh).

First, we’ll create our virtualenv environment, if we haven’t already:

$ virtualenv ~/env/ccommons

Next, we’ll check out cc.engine and cc.license into ~/devel/ and run
buildout on each:

$ cd ~/devel/
$ git clone git://code.creativecommons.org/cc.license.git
$ git clone git://code.creativecommons.org/cc.engine.git

Next, we’ll buildout the packages:

$ cd ~/devel/cc.license
$ wget http://svn.zope.org/*checkout*/zc.buildout/trunk/bootstrap/bootstrap.py
$ python bootstrap.py
$ ./bin/buildout
$ cd ~/devel/cc.engine
$ python bootstrap.py # the cc engine already has bootstrap.py checked in
$ ./bin/buildout

Buildout can take a while, so be prepared to go grab some cookies and coffee and/or tea. But once it’s done, getting these packages set up in virtualenv is super simple.

First activate the virtualenv environment:

$ source ~/env/ccommons/bin/activate
$ cd ~/devel/cc.license
$ python setup.py develop
$ cd ~/devel/cc.engine
$ python setup.py develop

That’s it! Now we can verify that these packages are set up in virtualenv. Open python and verify that you get the following output (adjusted to your own home directory and etc):

>>> import cc.engine
>>> cc.engine.__file__
'/home/cwebber/devel/cc.engine/cc/engine/__init__.pyc'
>>> import cc.license
>>> cc.license.__file__
'/home/cwebber/devel/cc.license-git/cc/license/__init__.pyc'

To leave virtualenv, you can simply type “deactivate”.

That’s it! Now you have a fully functional zc.buildout AND virtualenv setup, where switching back and forth is super simple.

6 Responses to “Using virtualenv and zc.buildout together”

  1. Wyatt says:

    > I can set up both development packages in the same virtualenv and use them together.

    You can do essentially the same thing with Buildout. Just set the `develop` property in your buildout.cfg. For example (assuming you have cc.engine and cc.license in a common projects directory):

    develop = ../cc.license
    /path/to/some/other/thing
    /and/so/on

  2. Kevin Teague says:

    The ‘develop’ directive in Buildout is for pulling in multiple packages under development. For example, in ‘cc.license’, the develop line is:

    develop = .

    Which means that the package in the cc.license directory will be picked by Buildout instead of Buildout trying to fetch and install a cc.license package. It’s very handy to twiddle with this line, I’ll often do something like:

    develop = . ../cc.engine

    And Buildout will find the source files for a package checked out at ../cc.engine.

    You won’t usually find too many examples of using Buildout like this, since most buildout.cfg files people use as references are checked into a project, and it doesn’t make sense for that file to have assumptions about the layout of stuff specific to one developer’s setup. But it is quite handy, and also convenient, as the develop directive is usually listed first in the config file. An advantage of twiddling the develop line is that it’s easy to switch between using a released version of a package and a development version, since you can have both packages installed at the same time, and Buildout is only modifying the generated script wrappers.

    If you’re doing this with more than a few packages though, then using mr.developer, a Buildout extension, is the way to go. This lets you do additional source code checkouts when buildout is run, and you can apply buildout and version control commands en-masse.

  3. Rob Miller says:

    You may or may not find this useful, but I’ve put together a zc.buildout recipe that will automatically turn the buildout root into a virtualenv, and ensure that bootstrap.py and buildout itself are always run within the context of that virtualenv. It’s available on pypi:

    http://pypi.python.org/pypi/rjm.recipe.venv

  4. You might want to take a look at mr.developer:
    http://pypi.python.org/pypi/mr.developer
    In addition to handle development versions in buildout with ease, it also integrates with various version control systems.

  5. You can work on multiple packages with buildout, easily.

    suppose you have done your checkout in ./src/cc.*. You have:

    ./src/cc.engine/
    ./src/cc.license/

    In your buildout.cfg, you just have to declare these paths in develop. It would look like this:

    [buildout]
    develop =
    ./src/cc.engine
    ./src/cc.license

    parts = eggs

    [eggs]
    recipe = zc.recipe.egg

    eggs =
    cc.engine
    cc.license

    (hint: develop accept any path, so it could start with ../…)

  6. cwebber says:

    Hey Wyat, Kevin, Bernard:

    Thanks! This information is good to know. Nathan Yergler did indicate that this was possible with a develop= line but didn’t really go into details, so it is good to read how to go about this.

    At any rate, I still think that it is valuable to know how to use both of these tools together. For example, I am developing interest in Silver Lining, and it might be good to be able to use some of the build system tools of buildout along with Silver Lining’s virtualenv deployment setup. Even so, that’s *not* what I’m doing now, so knowing how to do this with zc.buildout is very helpful… thanks!

    Rob & Bertrand:

    Thanks, I’ve taken note to look at those tools soon!