We will briefly cover the steps you need to take to publish your package on the Python Package Index (PyPI). You can use these steps to help you publish plugins for the CrossCompute Analytics Automation Framework. These steps are up-to-date as of September 2021.
- Add
pyproject.toml - Add
setup.cfg - Test with
pip - Publish on PyPI with
twine
Configure Package Build System
The pyproject.toml configuration file provides a hook so that you can specify a different package build system. For example, JupyterLab uses a custom build system.
The pyproject.toml configuration file can also contain configuration options for other development tools such as tox and coverage. However, we decided to put the these other configuration options into setup.cfg instead, as it is more widely supported.
Here is the default configuration that uses the standard setuptools:
# pyproject.toml
[build-system]
requires = [
'setuptools >= 40.9.0',
'wheel',
]
build-backend = 'setuptools.build_meta'
You might notice during pip install -e . that the build process is slower than setup.py. If you want to continue using setup.py with all the benefits of setup.cfg, you can opt for the following instead of pyproject.toml:
# setup.py
from setuptools import setup
setup()
Define Package Metadata
The Python Packaging User Guide now recommends replacing setup.py with setup.cfg . Here are some advantages:
- Use
attr:to include a value from a module. - Use
file:to include content from a file. - Include configuration options from other development tools like
flake8,coverage,toxand GitHub Actions.
Here are other points:
- You can write
long_descriptionusing Markdown. - Use
classifiersto help people find your package. - Specify
project_urlsfor related links to display next to your package on PyPI.
# setup.cfg
[metadata]
name = crosscompute
version = attr: crosscompute.__version__
description = Automate your Jupyter notebooks and scripts as web-based reports, tools, widgets, dashboards, wizards.
long_description = file: README.md
long_description_content_type = text/markdown
url = https://crosscompute.com
author = CrossCompute Inc.
author_email = support@crosscompute.com
license = MIT
license_file = LICENSE.md
platforms = any
classifiers =
Development Status :: 3 - Alpha
Framework :: CrossCompute
Framework :: Pyramid
Intended Audience :: Analysts
Intended Audience :: Developers
Intended Audience :: Information Technology
Intended Audience :: Science/Research
Intended Audience :: System Administrators
License :: OSI Approved :: MIT License
Operating System :: OS Independent
Programming Language :: Python :: 3
Topic :: Internet :: WWW/HTTP :: Dynamic Content
Topic :: Internet :: WWW/HTTP :: WSGI
Topic :: Internet :: WWW/HTTP :: WSGI :: Application
Topic :: Software Development :: Libraries :: Application Frameworks
project_urls =
Bug Tracker = https://github.com/crosscompute/crosscompute/issues
Documentation = https://github.com/crosscompute/crosscompute-docs
Source Code = https://github.com/crosscompute/crosscompute
[options]
packages = find:
python_requires = >=3.6
install_requires =
pyramid
zip_safe = True
[options.entry_points]
console_scripts =
crosscompute = crosscompute.scripts:launch
[options.extras_require]
test =
# check-manifest
hypothesis
# pytest-check-links
pytest-console-scripts
pytest-cov
pytest-mock
pytest-xtest
requests-cache
[flake8]
max-line-length = 79
select = B,C,E,F,W
ignore = W503
[coverage:run]
parallel = true
branch = true
source = crosscompute
[coverage:paths]
source = crosscompute
[tox]
envlist = lint,py36,py37,py38,py39,coverage-report
[testenv]
deps =
.[test]
commands =
coverage run -m pytest -vv tests -n auto
[testenv:coverage-report]
deps = coverage
skip_install = true
commands =
coverage combine
coverage report
[testenv:lint]
deps = flake8
skip_install = true
commands =
flake8 crosscompute
[gh-actions]
python =
3.6: py36
3.7: py37
3.8: py38
3.9: py39
Test Packaging
Looking at setup.cfg for Flask, Pandas and JupyterLab, we found a few interesting packages for testing:
-
check-manifest: Check your
MANIFEST.infor errors and omissions. - hypothesis: Find edge cases through semi-automated testing.
- pytest-check-links: Check URLs in your Markdown files and Jupyter notebooks.
- pytest-console-scripts: Test console scripts more efficiently.
- pytest-xtest: Run tests in parallel if your machine has multiple cores.
- requests-cache: Cache outgoing HTTP requests.
Here are other packages that we have been using already:
- pytest-cov: Include line-by-line coverage analysis in your test results.
- pytest-mock: Mock functions in your tests.
# Clone repository
git clone https://github.com/crosscompute/crosscompute
# Install with dependencies for tests
cd crosscompute
pip install -e .[test]
# Run tests
pytest \
--cov=crosscompute \
--cov-config=tox.ini \
--cov-report term-missing:skip-covered \
tests
# Build package for PyPI
pip install build
python -m build --sdist --wheel
- After building your source distribution, inspect its contents in the
distfolder to make sure the package contains all the files it needs. - Change to a different folder that does not contain your package, start Python and try importing your package to check for import errors.
Publish on PyPI
All of your tests have passed. You are now ready to publish your package! You will need to create accounts on both TestPyPI and PyPI.
- Publish on TestPyPI to make sure that everything works properly.
pip install twine --upgrade
python -m twine upload --repository testpypi dist/*
- Publish on PyPI.
pip install twine --upgrade
python -m twine upload dist/*
If you publish packages frequently, you might want to define the .pypi.rc configuration file in your home folder. You can specify your PyPI API token in this file.
# .pypi.rc
[testpypi]
username = __token__
password = Your TestPyPI API token
[pypi]
username = __token__
password = Your PyPI API token
For more information on Python packaging, please see the official Python Packaging User Guide.