Creating Packages ================= Writing wrapper packages ------------------------ The goal of shrinkwrap is to create Python packages for non-Python software and install them using Python package management. To this end, shrinkwrap "packages" are simply setuptools setup scripts which know how to download and install other software. A shrinkwrap package may either use a built-in installer function (for common installation methods) or define a custom one. .. _built_in_installers: Built-in installers ``````````````````` An installer for software using autoconf is available to simplify things. The autoconf shrinkwrap installer assumes the tarfile unpacks to a directory with the same name as the tarfile with .tar.{gz,bz2} removed, contains a configure script, and understands the --prefix option to control where the package is installed. For example, to package `curl`, one may write the following (as, e.g. `curl-7.27.0.py`):: try: from shrinkwrap.install import ShrinkwrapInstall except ImportError: import subprocess; subprocess.check_call('pip install -b $VIRTUAL_ENV/build/build-shrinkwrap shrinkwrap', shell=True) from shrinkwrap.install import ShrinkwrapInstall from setuptools import setup version = '7.27.0' setup( name='curl', version=version, author='Stan Seibert', author_email='stan@mtrr.org', shrinkwrap_installer='autoconf', shrinkwrap_source_url='http://curl.haxx.se/download/curl-%s.tar.bz2' % version, cmdclass={'install': ShrinkwrapInstall}, ) To install curl, one would simply run:: python curl-7.27.0.py install The boilerplate at the top of the script is required to ensure that shrinkwrap is installed before setuptools is imported. The ``cmdclass`` option must be set as shown above in order to use the ``ShrinkwrapInstall`` command rather than the default install command provided by setuptools. The filename of the python script is not used by shrinkwrap. Custom installers ````````````````` The following package for bzip2 illustrates a custom installer function:: try: from shrinkwrap.install import ShrinkwrapInstall except ImportError: import subprocess; subprocess.check_call('pip install shrinkwrap', shell=True) from shrinkwrap.install import ShrinkwrapInstall import os from setuptools import setup version = '1.0.6' source_url = 'http://www.bzip.org/%(version)s/bzip2-%(version)s.tar.gz' % {'version': version} def installer(self): self.download_and_unpack_tarball(source_url) bzip2_dir = 'bzip2-' + version os.chdir(bzip2_dir) self.make(extra_opts=['install', 'PREFIX=%s' % self.virtualenv]) setup( name='bzip2', version=version, author='Andy Mastbaum', author_email='mastbaum@hep.upenn.edu', shrinkwrap_installer=installer, cmdclass={'install': ShrinkwrapInstall}, ) As before, installing bzip2 simply requires:: python bzip2-1.0.6.py install Here, the ``shrinkwrap_installer`` argument to ``setup()`` is set to our own installer function. The installer uses two shrinkwrap-provided convenience functions, ``download_and_unpack_tarball`` and ``make`` to download, untar, and compile the bzip2 library. See :ref:`shrinkwrap_install_api` for a complete list of convenice functions. By passing extra options to make, the software is installed into the root of the active virtualenv. .. note:: For several examples of custom installers, see `https://bitbucket.org/seibert/shrinkwrap_pkgs `_. Examples include getting code from version control, installing with cmake, and customizing install paths. .. _package_dependencies: Package Dependencies ```````````````````` The handling of package dependencies in pip (as of version 1.1, anyway) does not, unfortunately, meet the requirements for installing compiled software. Packages listed in the ``install_requires`` keyword argument to ``setup()`` will be discovered by pip and installed, but in an arbitrary order. Packages listed in the ``setup_requires`` keyword argument are not actually installed to the virtualenv, but rather made available in .egg files in the build directory, which also is not a solution for compiled code. As a result, shrinkwrap adds a new keyword argument, ``shrinkwrap_requires``, to ``setup()``. All dependencies listed here are guaranteed to be fully installed to the virtualenv before the ``installer()`` function is run. A separate pip process is spawned to install each one, so the dependency string can include package versions, just as ``setup_requires`` and ``install_requires`` allow. The shrinkwrap dependencies can be any kind of package, so if your library uses `SCons `_ as the build system, you can list it in the ``shrinkwrap_requires`` field:: setup( name='foo', version=version, author='Example Author' author_email='author@example.com' shrinkwrap_installer=installer, shrinkwrap_requires=['scons'], cmdclass={'install': ShrinkwrapInstall}, ) Environment Variables ````````````````````` If your package also needs to set some shell environment variables for operation, they can be placed in the ``$VIRTUAL_ENV/env.d`` directory, and they will be sourced automatically when the ``$VIRTUAL_ENV/bin/activate`` script is sourced. See `the package file for CERN's ROOT library `_ for an example of using the ``install_env()`` convenience function. Packaging for a package index ----------------------------- To share shrinkwrap packages via a package index like PyPI (remember: don't actually upload shrinkwrap packages to PyPI), you must create distribution tarballs. Shrinkwrap includes a tool to create these automatically from wrapper setup.py files:: shrinkwrap createpkg bzip2-1.0.6.py This will create a tarball in the current directory suitable for uploading. Wildcards are valid for generating many packages at once. .. note:: You can name the python file anything you want (not just setup.py). It will be copied to setup.py in a temp directory and the name of the resulting tarball will be derived from the package metadata you specified. .. _running_package_index: Running your own package index ------------------------------ As mentioned in the :ref:`limitations` section, shrinkwrap packages are small and easy to deploy. It is better to have different package repositories when you want to build a source package with different options in different situations, rather than have One Repository To Rule Them All [#onerepo]_ . If you wish to serve your own package index, use the ``-p`` option to place output tarballs into one properly-formatted directory:: shrinkwrap createpkg -p packages my_shrinkwrap_pkgs/* Simply serve the `packages` directory on the web, and pip clients can interact with it just like PyPI:: client$ export PIP_EXTRA_INDEX_URL=http://your-server.com/packages/ client$ pip install bzip2 Extra package indices can be specified in a requirements.txt file with an option at the top:: --extra-index-url http://mtrr.org/packages/ .. [#onerepo] In order to avoid creating One Repository To Rule Them All, we do not plan to ever offer a repository of "official" shrinkwrap packages. Feel free to copy other people's package files to your repository, however.