Creating Packages ================= Writing wrapper packages ------------------------ The goal of shrinkwrap is to create Python packages for non-Python software and install them using Python package management. To this end, shrinkwrap "packages" are simply setuptools setup scripts which know how to download and install other software. A shrinkwrap package may either use a built-in installer function (for common installation methods) or define a custom one. .. _built_in_installers: Built-in installers ``````````````````` An installer for software using autoconf is available to simplify things. The autoconf shrinkwrap installer assumes the tarfile unpacks to a directory with the same name as the tarfile with .tar.{gz,bz2} removed, contains a configure script, and understands the --prefix option to control where the package is installed. For example, to package `curl`, one may write the following (as, e.g. `curl-7.27.0.py`):: try: from shrinkwrap.install import ShrinkwrapInstall except ImportError: import subprocess; subprocess.check_call('pip install -b $VIRTUAL_ENV/build/build-shrinkwrap shrinkwrap', shell=True) from shrinkwrap.install import ShrinkwrapInstall from setuptools import setup version = '7.27.0' setup( name='curl', version=version, author='Stan Seibert', author_email='stan@mtrr.org', shrinkwrap_installer='autoconf', shrinkwrap_source_url='http://curl.haxx.se/download/curl-%s.tar.bz2' % version, cmdclass={'install': ShrinkwrapInstall}, ) To install curl, one would simply run:: python curl-7.27.0.py install The boilerplate at the top of the script is required to ensure that shrinkwrap is installed before setuptools is imported. The ``cmdclass`` option must be set as shown above in order to use the ``ShrinkwrapInstall`` command rather than the default install command provided by setuptools. The filename of the python script is not used by shrinkwrap. Custom installers ````````````````` The following package for bzip2 illustrates a custom installer function:: try: from shrinkwrap.install import ShrinkwrapInstall except ImportError: import subprocess; subprocess.check_call('pip install shrinkwrap', shell=True) from shrinkwrap.install import ShrinkwrapInstall import os from setuptools import setup version = '1.0.6' source_url = 'http://www.bzip.org/%(version)s/bzip2-%(version)s.tar.gz' % {'version': version} def installer(inst): self.download_and_unpack_tarball(source_url) bzip2_dir = 'bzip2-' + version os.chdir(bzip2_dir) self.make(extra_opts=['install', 'PREFIX=%s' % self.virtualenv]) setup( name='bzip2', version=version, author='Andy Mastbaum', author_email='mastbaum@hep.upenn.edu', shrinkwrap_installer=installer, cmdclass={'install': ShrinkwrapInstall}, ) As before, installing bzip2 simply requires:: python bzip2-1.0.6.py install Here, the ``shrinkwrap_installer`` argument to ``setup()`` is set to our own installer function. The installer function is called with an instance of ShrinkwrapInstall as the argument, which provides several convenience functions. This installer uses two of these functions, ``download_and_unpack_tarball`` and ``make`` to download, untar, and compile the bzip2 library. See :ref:`shrinkwrap_install_api` for a complete list of available functions. By passing extra options to make, the software is installed into the root of the active virtualenv. .. note:: For several examples of custom installers, see `https://bitbucket.org/seibert/shrinkwrap_pkgs `_. Examples include getting code from version control, installing with cmake, and customizing install paths. .. _system_packages: Detecting System-provided Packages `````````````````````````````````` In some situations, you only want to install a package if the operating system does not already provide it. The ``shrinkwrap_skip`` argument to ``setup()`` allows you to specify a function to call to check if installation is performed. If the skip function returns True, then the ``installer()`` function is skipped, but the package is marked as installed. Here is an example of a package that installs curl only if version 7.26.0 or newer is not present:: try: from shrinkwrap.install import ShrinkwrapInstall except ImportError: import subprocess; subprocess.check_call('pip install -b $VIRTUAL_ENV/build/build-shrinkwrap shrinkwrap', shell=True) from shrinkwrap.install import ShrinkwrapInstall from setuptools import setup from distutils.version import LooseVersion version = '7.27.0' def skip(inst): try: output = inst.cmd_output('curl-config --version') name, version_str = output.split() system_version = LooseVersion(version_str) min_version = LooseVersion('7.26.0') if system_version > min_version: return True # Don't install else: return False # Version too old except: return False # install package if anything went wrong setup( name='curl-check', version=version, author='Stan Seibert', author_email='stan@mtrr.org', shrinkwrap_installer='autoconf', shrinkwrap_skip=skip, shrinkwrap_source_url='http://curl.haxx.se/download/curl-%s.tar.bz2' % version, cmdclass={'install': ShrinkwrapInstall}, ) .. _package_dependencies: Package Dependencies ```````````````````` The handling of package dependencies in pip (as of version 1.1, anyway) does not, unfortunately, meet the requirements for installing compiled software. Packages listed in the ``install_requires`` keyword argument to ``setup()`` will be discovered by pip and installed, but in an arbitrary order. Packages listed in the ``setup_requires`` keyword argument are not actually installed to the virtualenv, but rather made available in .egg files in the build directory, which also is not a solution for compiled code. As a result, shrinkwrap adds a new keyword argument, ``shrinkwrap_requires``, to ``setup()``. All dependencies listed here are guaranteed to be fully installed to the virtualenv before the ``installer()`` function is run. A separate pip process is spawned to install each one, so the dependency string can include package versions, just as ``setup_requires`` and ``install_requires`` allow. The shrinkwrap dependencies can be any kind of package, so if your library uses `SCons `_ as the build system, you can list it in the ``shrinkwrap_requires`` field:: setup( name='foo', version=version, author='Example Author' author_email='author@example.com' shrinkwrap_installer=installer, shrinkwrap_requires=['scons'], cmdclass={'install': ShrinkwrapInstall}, ) Environment Variables ````````````````````` If your package also needs to set some shell environment variables for operation, they can be placed in the ``$VIRTUAL_ENV/env.d`` directory, and they will be sourced automatically when the ``$VIRTUAL_ENV/bin/activate`` script is sourced. See `the package file for CERN's ROOT library `_ for an example of using the ``install_env()`` convenience function. Packaging for a package index ----------------------------- To share shrinkwrap packages via a package index like PyPI (remember: don't actually upload shrinkwrap packages to PyPI), you must create distribution tarballs. Shrinkwrap includes a tool to create these automatically from wrapper setup.py files:: shrinkwrap createpkg bzip2-1.0.6.py This will create a tarball in the current directory suitable for uploading. Wildcards are valid for generating many packages at once. .. note:: You can name the python file anything you want (not just setup.py). It will be copied to setup.py in a temp directory and the name of the resulting tarball will be derived from the package metadata you specified. .. _running_package_index: Running your own package index ------------------------------ As mentioned in the :ref:`limitations` section, shrinkwrap packages are small and easy to deploy. It is better to have different package repositories when you want to build a source package with different options in different situations, rather than have One Repository To Rule Them All [#onerepo]_ . If you wish to serve your own package index, use the ``-p`` option to place output tarballs into one properly-formatted directory:: shrinkwrap createpkg -p packages my_shrinkwrap_pkgs/* Simply serve the `packages` directory on the web, and pip clients can interact with it just like PyPI:: client$ export PIP_EXTRA_INDEX_URL=http://your-server.com/packages/ client$ pip install bzip2 Extra package indices can be specified in a requirements.txt file with an option at the top:: --extra-index-url http://mtrr.org/packages/ .. [#onerepo] In order to avoid creating One Repository To Rule Them All, we do not plan to ever offer a repository of "official" shrinkwrap packages. Feel free to copy other people's package files to your repository, however.