Python Development and Releases

This document describes Python-specific details about the Hubverse’s development and release process.

Hubverse Python packages that will be released to PyPI must go through a one-time setup process as described in the Creating PyPI and TestPyPI workflows section below.

Once that setup is complete, use the checklists below for updating and releasing the package.

Checklists

Development checklist

To update a Hubverse Python package:

• Create a branch from main in the format of <initials>/<feature>/<issue> (e.g., kj/add-bucket-versioning/111 )

• Solve the issue, update/add test if necessary, commit, push

• Open a pull request (PR).

  • Pull requests should be as small as possible and should focus on independent features.

  • Each PR should include a corresponding update to the [unreleased] section at the top of CHANGELOG.md (if applicable). Changelog contents and style should follow the guidelines outlined in keepachangelog.com.

  • In the PR description, include a link to the issue (e.g., resolves #111) as well as any context that will help code reviewers.

• If the PR introduces a breaking change, these changes should be tested and communicated with the community.

• Once the PR has been approved and all checks have passed, merge it.

Tip

Review often brings up potential non-blocking features/bug fixes that are orthogonal to the original PR. In these cases, instead of creating a PR to merge into the original PR, it’s best to create a new issue from the PR review and, after merging, create a new PR to fix that issue. This helps keep disparate bugfixes and features separate.

Release checklist

When it’s time to release the package to PyPI:

• Proofread CHANGELOG.md and change the [unreleased] heading at the top to the new release’s version number. Make sure to acknowledge any contributors outside of the core dev team by linking to their GitHub handles.

• Submit a PR, get a review, and merge the CHANGELOG.md updates.

• Create a new tag for the release as described in the Hubverse Release Process.

• If you created the release tag locally, push it to the package’s repository (for example, git push v0.2.4).

Hotfix checklist

A hotfix is a bug fix that is independent from in-development features and needs to be deployed within a day. Details on hotfixes can be found on the hotfix page.

To patch and release a hotfix:

• Create a new branch from the latest tag using pattern <initials>/hotfix/<issue>

  (main)$ git switch --detach 0.14.0 # checkout the tag
  ((0.14.0))$ git switch -c znk/hotfix/143 # create a new branch
  (znk/hotfix/143)$
  

• Write a test, fix the bug, and update CHANGELOG.md with the patch version and description. Push these changes upstream:

  (znk/hotfix/143)$ git commit -m 'hotfix for #143'
  (znk/hotfix/143)$ git push -u origin znk/hotfix/143 # push the hotfix
  

• Create a PR, get a review, and confirm that tests pass against the released version of the package.

• From the hotfix branch, create a tag for the release.

• Resolve conflicts in the PR and merge into main.

Creating a new Hubverse Python package

Unlike R, the Python ecosystem doesn’t have a single, agreed-upon best practice for package creation, structure, and development. In general, Hubverse Python packages:

• Use uv for managing Python versions, virtual environments, and dependencies

• Use a pyproject.toml file to describe the project (versus the older setup.py)

• Use the src layout (the pyOpenSci website has a good overview of this layout)

• Type hint function arguments and return values (at a minimum—other code may have type hints for clarity)

• Use numpy-style docstrings

• Use ruff for linting and code formatting (generally with the default settings)

• Use pytest for creating and running tests

• Use Sphinx for documentation (the page you’re reading now was created with Sphinx)

• Prefer logging to stdout over print statements

Creating a new Python package (empty)

Theuv init command can create an new, empty Python package structure using the src layout. The following command creates a directory called new-package in the current working directory:

uv init --package new-package

The resulting directory structure looks like this:

new-package
├── README.md
├── pyproject.toml
└── src
    └── new_package
        └── __init__.py

Creating a new Python package (with logging setup, test harness, CI, and docs)

The pyprefab package is a simple, prompt-driven tool for creating new Python packages that has boilerplate code for logging, testing, GitHub actions, Sphinx documentation, and a Python-based CONTRIBUTING.md file.

You don’t need to install pyprefab to use it. The tools feature of uv can invoke pyprefab directly:

uvx pyprefab

This command will then prompt for a package name, author name, and a few other pieces of information. The resulting package will have the following structure:

new-package
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
├── README.md
├── docs
│   └── source
│       ├── CHANGELOG.md
│       ├── CONTRIBUTING.md
│       ├── _static
│       │   └── custom.css
│       ├── conf.py
│       ├── index.rst
│       ├── readme.md
│       └── usage.md
├── pyproject.toml
├── src
│   └── new_package
│       ├── __init__.py
│       ├── __main__.py
│       ├── app.py
│       └── logging.py
└── test
    └── test_app.py

Releasing Python packages

Unlike CRAN, Python’s official package index, PyPI, does not require manual review. Thus, the Hubverse can release Python packages right to PyPI without an intermediary like R-Universe. In addition, TestPyPI allows us to test release processes and tools without publishing to the real index.

The documentation below assumes that Hubverse PyPI packages will be released to PyPI, allowing us to follow the “main = stable dev branch” release process as outlined in the Hubverse release process.

Overview

PyPI (and TestPyPI) allow trusted publishers as way to publish packages via GitHub actions without embedding secrets in the project repository.

Any Hubverse Python packages published to PyPI will use trusted publishing, which protects against supply chain attacks and credential leaks.

• merges to the stable dev branch (main) will be released on TestPyPI as an end-to-end test

• adding a release tag to the repo (i.e., vX.Y.Z) will trigger a release to PyPI

This article contains a clearly-written deep dive into how trusted publishing works and the advantages of using it.

TestPYPI setup

Once you complete the GitHub and TestPYPI setup as outlined below, the new Hubverse package will be pushed to TestPyPI automatically. You will only need to do this once.

Publishing to TestPyPI doesn’t release the software. Instead, it’s a way to test the end-to-end publication process each time a new update is merged to the package’s main branch.

GitHub

1. In the package’s GitHub repo, create an environment called pypi-test to use for the TestPyPI deployment.

   Note

   Because this environment is for test deployments, you don’t need to add protection rules or fill out any other information when configuring it.

2. Add the publish-pypy-test.yaml workflow the package’s Github repo:

   • Copy the publish-pypi-test.yaml workflow from hubverse-developer-actions.

   • Change <package> in the TestPyPI url to the name of your package (e.g., https://test.pypi.org/p/hubDataPy)

TestPyPI

1. Create a TestPyPI account if you don’t already have one.

2. Log in to TestPyPI.

3. Create a new Trusted Publisher for the Hubverse package.

   • PyPI Project Name: name in the [project] section of the package’s pyproject.toml file

   • Owner: GitHub organization name (hubverse-io)

   • Repository name: the package’s GitHub repository name

   • Workflow name: full file name of the GitHub workflow that publishes to TestPyPI (e.g. publish-pypi-test.yaml)

   • Environment name: name of the GitHub environment created above (pypi-test)

PyPI setup

Once you complete the GitHub and PyPI setup as outlined below, your package will be published to PyPI whenever a new release tag is pushed to the package’s repository.

You will only need to do this once.

GitHub

1. In the package’s GitHub repo, create an environment called pypi to use for the PyPI deployment.

   Important

   Because this environment will be used for publishing to production, check the Required reviewers option and add a list of Hubverse devs who are authorized to approve releases.

2. Add the publish-pypy.yaml workflow the package’s Github repo:

   • Copy the publish-pypi workflow from hubverse-developer-actions.

   • Change <package> in the PyPI url to the name of your package (e.g., https://pypi.org/p/hubDataPy)

PyPI

1. Create a PyPI account if you don’t already have one.

2. Log in to PyPI.

3. Create a new Trusted Publisher for the Hubverse package:

   • PyPI Project Name: name in the [project] section of the package’s pyproject.toml file

   • Owner: GitHub organization name (hubverse-io)

   • Repository name: the package’s GitHub repository name

   • Workflow name: full file name of the GitHub workflow that publishes to PyPI (e.g. publish-pypi.yaml)

   • Environment name: name of the GitHub environment created above (pypi)

Add package maintainers

To ensure continuity, it’s important that Hubverse packages on both PyPI and TestPYPI have multiple maintainers and collaborators. You can add other Hubverse devs to these roles from the project’s Collaborators page on PyPI/TestPyPI.