Python Packaging

We will briefly cover the steps you need to take to publish your package on the Python Package Index (PyPI). You can use these steps to help you publish plugins for the CrossCompute Analytics Automation Framework. These steps are up-to-date as of September 2021.

  1. Add pyproject.toml
  2. Add setup.cfg
  3. Test with pip
  4. Publish on PyPI with twine

Configure Package Build System

The pyproject.toml configuration file provides a hook so that you can specify a different package build system. For example, JupyterLab uses a custom build system.

The pyproject.toml configuration file can also contain configuration options for other development tools such as tox and coverage. However, we decided to put the these other configuration options into setup.cfg instead, as it is more widely supported.

Here is the default configuration that uses the standard setuptools:

# pyproject.toml

[build-system]
requires = [
    'setuptools >= 40.9.0',
    'wheel',
]
build-backend = 'setuptools.build_meta'

You might notice during pip install -e . that the build process is slower than setup.py. If you want to continue using setup.py with all the benefits of setup.cfg, you can opt for the following instead of pyproject.toml:

# setup.py

from setuptools import setup
setup()

Define Package Metadata

The Python Packaging User Guide now recommends replacing setup.py with setup.cfg . Here are some advantages:

  • Use attr: to include a value from a module.
  • Use file: to include content from a file.
  • Include configuration options from other development tools like flake8, coverage, tox and GitHub Actions.

Here are other points:

  • You can write long_description using Markdown.
  • Use classifiers to help people find your package.
  • Specify project_urls for related links to display next to your package on PyPI.
# setup.cfg

[metadata]
name = crosscompute
version = attr: crosscompute.__version__
description = Automate your Jupyter notebooks and scripts as web-based reports, tools, widgets, dashboards, wizards.
long_description = file: README.md
long_description_content_type = text/markdown
url = https://crosscompute.com
author = CrossCompute Inc.
author_email = support@crosscompute.com
license = MIT
license_file = LICENSE.md
platforms = any
classifiers =
    Development Status :: 3 - Alpha
    Framework :: CrossCompute
    Framework :: Pyramid
    Intended Audience :: Analysts
    Intended Audience :: Developers
    Intended Audience :: Information Technology
    Intended Audience :: Science/Research
    Intended Audience :: System Administrators
    License :: OSI Approved :: MIT License
    Operating System :: OS Independent
    Programming Language :: Python :: 3
    Topic :: Internet :: WWW/HTTP :: Dynamic Content
    Topic :: Internet :: WWW/HTTP :: WSGI
    Topic :: Internet :: WWW/HTTP :: WSGI :: Application
    Topic :: Software Development :: Libraries :: Application Frameworks
project_urls =
    Bug Tracker = https://github.com/crosscompute/crosscompute/issues
    Documentation = https://github.com/crosscompute/crosscompute-docs
    Source Code = https://github.com/crosscompute/crosscompute
[options]
packages = find:
python_requires = >=3.6
install_requires =
    pyramid
zip_safe = True
[options.entry_points]
console_scripts =
    crosscompute = crosscompute.scripts:launch
[options.extras_require]
test =
    # check-manifest
    hypothesis
    # pytest-check-links
    pytest-console-scripts
    pytest-cov
    pytest-mock
    pytest-xtest
    requests-cache

[flake8]
max-line-length = 79
select = B,C,E,F,W
ignore = W503

[coverage:run]
parallel = true
branch = true
source = crosscompute
[coverage:paths]
source = crosscompute

[tox]
envlist = lint,py36,py37,py38,py39,coverage-report
[testenv]
deps =
    .[test]
commands =
    coverage run -m pytest -vv tests -n auto
[testenv:coverage-report]
deps = coverage
skip_install = true
commands =
    coverage combine
    coverage report
[testenv:lint]
deps = flake8
skip_install = true
commands =
    flake8 crosscompute

[gh-actions]
python =
    3.6: py36
    3.7: py37
    3.8: py38
    3.9: py39

Test Packaging

Looking at setup.cfg for Flask, Pandas and JupyterLab, we found a few interesting packages for testing:

Here are other packages that we have been using already:

  • pytest-cov: Include line-by-line coverage analysis in your test results.
  • pytest-mock: Mock functions in your tests.
# Clone repository
git clone https://github.com/crosscompute/crosscompute

# Install with dependencies for tests
cd crosscompute
pip install -e .[test]

# Run tests
pytest \
    --cov=crosscompute \
    --cov-config=tox.ini \
    --cov-report term-missing:skip-covered \
    tests

# Build package for PyPI
pip install build
python -m build --sdist --wheel
  1. After building your source distribution, inspect its contents in the dist folder to make sure the package contains all the files it needs.
  2. Change to a different folder that does not contain your package, start Python and try importing your package to check for import errors.

Publish on PyPI

All of your tests have passed. You are now ready to publish your package! You will need to create accounts on both TestPyPI and PyPI.

  1. Publish on TestPyPI to make sure that everything works properly.
pip install twine --upgrade
python -m twine upload --repository testpypi dist/*
  1. Publish on PyPI.
pip install twine --upgrade
python -m twine upload dist/*

If you publish packages frequently, you might want to define the .pypi.rc configuration file in your home folder. You can specify your PyPI API token in this file.

# .pypi.rc

[testpypi]
username = __token__
password = Your TestPyPI API token

[pypi]
username = __token__
password = Your PyPI API token

For more information on Python packaging, please see the official Python Packaging User Guide.