How Shrinkwrap Works

Shrinkwrap was designed to scratch a particular itch: automating the installation of software in a self-contained environment.

Nearly all package managers assume they are managing software installation for the entire system, and don’t allow isolated environments. ZeroInstall is a notable exception which installs software into a user’s home directory and can manage multiple versions of a single package. However, ZeroInstall is more aimed at providing specific applications to an end user, rather than providing a shell environment with a set of packages installed and ready for use.

Python’s virtualenv and easy_install/pip tools are a great example of a userspace packaging solution that creates self-contained, easy-to-use environments. Since we work in mixed environments where C, C++ and Python software needs to coexist, shrinkwrap is our attempt to bring all that software under one roof.

Shrinkwrap achieves this goal by modifying the virtualenv in three ways:

1. It provides an alternate implementation of the python setup.py install command that makes it easy to insert an arbitrary installer function. This function typically downloads source tarballs, unpacks them, compiles and installs to the virtualenv base directory.

2. It adds new optional keyword arguments to setup() to specify the installer function, and also to list packages that are build dependencies of this package. See Package Dependencies for details.

3. It creates a new $VIRTUAL_ENV/env.d directory and patches the $VIRTUAL_ENV/bin/activate script to source any files found in that directory. This allows shrinkwrap packages to install scripts that make changes to the shell environment. Many of the software packages we work with require environment variables to function properly.

With these changes, it becomes very easy to write small Python packages that download, compile, and install software into the virtualenv environment.

Installing shrinkwrap packages

Installing a shrinkwrapped package is identical to installing any package with pip. You can install the package file directly from the filesystem:

pip install curl-7.27.0.tar.gz

or from a URL:

pip install http://mtrr.org/packages/curl/curl-7.27.0.tar.gz

or set the URL of extra package repository which contains shrinkwrapped packages:

export PIP_EXTRA_INDEX_URL=http://mtrr.org/packages/
pip install curl

If the shrinkwrapped package contains the standard test for the shrinkwrap module at the top (see Built-in installers), then shrinkwrap will be automatically installed to your virtualenv.

Note that environment files in the $VIRTUAL_ENV/env.d/ directory are only sourced when the $VIRTUAL_ENV/bin/activate script is sourced. You will need to source that script again to refresh your environment if the shrinkwrapped package added new environment files.

As true Python packages, shrinkwrapped packages can be used in a requirements.txt files passed to pip. Just place the extra index URL argument at the top:

--extra-index-url http://mtrr.org/packages/
curl
fftw
nose

Disabling system software autodetection

Some shrinkwrap packages will auto-detect whether the software is already installed system-wide. If it is, installation is skipped and the package is marked as installed since the dependency is satisfied.

To force installation into the virtual environment, this auto-detection can be disabled with an environment variable:

SHRINKWRAP_NEVER_SKIP

Set this variable to ‘1’ to forego system package detection and always install the shrinkwrap package. Unset or change to ‘0’ to restore the default behavior.

Limitations

We are the first to admit that shrinkwrap is straining the intended purpose of the Python packaging tools. As a result, shrinkwrap+pip has some shortcomings compared to more sophisticated package managers:

• Shrinkwrap can only be used with a virtualenv environment. Shrinkwrap tests for the presence of $VIRTUAL_ENV and will throw an exception if it is not found.
• Shrinkwrapped packages have only been tested to work with pip as the installer. We do not support using easy_install, although it might work.
• pip uninstall does not remove files installed by the package because we do not yet have a way to track changes made by arbitrary installer functions to the filesystem.
• We do not provide any mechanism to have build options selected at installation time (such as variants in MacPorts or USE variables in Gentoo emerge). Repositories of shrinkwrapped packages are easy to make with the shrinkwrap command line tool, so we encourage you to have different repositories for different kinds of deployment environments. Then you can tailor the build options of your source packages for each one.
• Shrinkwrap is currently focused on deploying source packages rather than precompiled packages. There is no reason you can’t unpack a tarball of compiled code in a shrinkwrap installer() function, but we do not provide any mechanism for having packages compiled for different architectures in the same repository. Again, repositories are easy to make, so we would suggest one per architecture if this is your use case. (Selecting platform-specific build options in your installer() function, however, is no problem.)