(Level 5) Share your code as a publicly installable package

Overview

Welcome! By the end of the tutorial, you will be able to share your code as a publicly installable package that can be installed using pip install <package-name> and conda install <package-name>.

Prerequisites

We assume that you have a basic understanding of starting a new project and have hosted at least one project on GitHub. If you are new to GitHub, we recommend you start from (Level 4) Share your code as a locally installable Python package where you will learn how to create a new project and host it on GitHub while using GitHub Actions to automatically format your code and run unit tests.

Make sure you have the latest version of scikit-package installed:

Install scikit-package with conda

1. Ensure that conda is installed on your local computer as instructed in (Required) Use conda environment to install packages and run Python code.

2. Ensure that you have added the conda-forge channel to your conda configuration.

   $ conda config --add channels conda-forge
   

3. Create a new conda environment skpkg-env and install scikit-package and pre-commit simultaneously by running the following command:

   $ conda create -n skpkg-env scikit-package pre-commit
   

   Note

   If the above command does not work, ensure you have added the conda-forge channel to your conda configuration in Step 2.

4. Activate the environment:

   $ conda activate skpkg-env
   

5. Confirm that you have the latest version of scikit-package available on GitHub:

   $ pip show scikit-package
   

Step 1. Create a new project with scikit-package

Create a new GitHub repository

1. Visit https://github.com/new.

2. Enter the Repository name. You may use my-package or any other name you like.

3. Choose Public.

4. We want to create an empty repository so set None under Add .gitignore.

5. Set None Under Choose a license.

6. Click the Create repository green button to create the repository.

7. Done!

Create a new project with scikit-package

1. Run the following command to create a new project folder with scikit-package using the Level 5 public template:

   $ package create public
   

2. Answer the following questions:

   Prompt	Description and example
   maintainer_name	The name of the project maintainer. This person will make the public releases. e.g., Simon Billinge
   maintainer_email	The maintainer’s email address. e.g., sbillinge@columbia.edu
   maintainer_github_username	The maintainer’s GitHub username. e.g., sbillinge
   contributors	Individuals or groups contributing to the project. e.g., Sangjoon Lee, Simon Billinge, Billinge Group members
   license_holders	The license holders listed in LICENSE.rst. e.g., The Trustees of Columbia University in the City of New York
   project_name	The name displayed in the README.rst and documentation. Use name-with-hyphens e.g., my-package. To support namespace imports, see FAQ
   github_username_or_orgname	The GitHub username or organization name. e.g., sbillinge or billingegroup
   github_repo_name	The GitHub repository name. Use name-with-hyphens e.g., my-package
   conda_pypi_package_dist_name	The name used for publishing to PyPI and conda-forge. Use name-with-hyphens e.g., my-package
   package_dir_name	The name of the package directory under src. This name will be used in when importing modules from this package. Use name_with_underscores e.g., my_package
   project_short_description	A brief description of the project, shown in pyproject.toml. e.g., A Python package standard for scientific code
   project_keywords	A list of keywords included in pyproject.toml. e.g., PDF, diffraction, neutron, x-ray
   min_python_version	The minimum supported Python version. e.g. 3.11. Current practice (SPEC 0 specification) is to drop support for Python versions that were released more than 72 months ago.
   max_python_version	The maximum supported Python version e.g. 3.13. Ideally, the current latest version.
   needs_c_code_compiled	Specifies whether C code compilation is required. For pure Python packages, the default value is No.
   has_gui_tests	Specifies whether GUI tests are included. For most packages, the default value is No but will be yes if your app has a gui and there are tests that run the GUI.

   Note

   You may press the “Enter” key to accept the default values for the questions.

3. Now type ls to check a new folder has been created.

Install your package locally

1. Create a new conda environment. Let’s call this environment my-package-env:

   $ conda create -n my-package-env python=3.13 \
       --file requirements/conda.txt \
       --file requirements/test.txt
   

2. Activate the conda environment:

   $ conda activate my-package-env
   

3. Build and install the package locally:

   $ pip install -e . --no-deps
   

   Note

   What is the -e flag?

   The -e flag indicates that you want to install the package in “editable” mode, which means that any changes you make to the source code will be reflected immediately without needing to reinstall the package. This is useful for development purposes.

   Note

   What is the --no-deps flag?

   The --no-deps flag tells pip not to install any dependencies listed in requirements/pip.txt. This is because we have already installed the dependencies in the conda environment using the command above.

   See also

   Why is it required to list dependencies both under pip.txt and conda.txt? Please refer to the FAQ section Dependency management.

4. Then, run the tests using the following command:

   $ pytest
   

5. Ensure tests all pass with green checkmarks. Notice that in tests/test_functions.py, we are importing the locally installed package.

6. Done!

Build documentation locally

/doc is the the Sphinx documentation folder. The documentation will be built locally first and then automatically built and hosted on GitHub Pages when a new release is created.

1. Install documentation related dependencies:

   $ conda install --file requirements/docs.txt
   

2. Enter into the doc project directory and render documentation:

   $ cd doc
   $ make html
   

3. Open the rendered documentation via web browser:

   $ open _build/html/index.html
   

4. Here is a shortcut if you want to use it from the root directory of the project:

   $ cd doc && make html && open _build/html/index.html && cd ..
   

   See also

   You can use a alias shortcut. Open ~/.bashrc in your text editor and add the following line:

   alias doc='cd doc && make html && open _build/html/index.html && cd ..'
   

   Apply the changes to your current terminal session:

   $ source ~/.bashrc
   

   Now, you can simply enter the doc command in your terminal to build and open the documentation:

   $ doc
   

(Optional for macOS/Linux only) Do you want to re-render documentation without running doc command every time? You can use sphinx-reload.

1. Install the dependencies including sphinx-reload sourced from PyPI:

   $ conda install --file requirements/docs.txt
   $ pip install sphinx-reload
   

2. Run the following command to start live-reloading:

   $ sphinx-reload doc
   

3. Now, each time you make changes to the documentation, it will be automatically reloaded in your web browser.

Upload README.md to your GitHub repository

At the moment, the GitHub repository is empty. Let’s create a local branch called main and upload this local branch to the remote GitHub repository.

1. Follow the series of steps to initialite Git. You only have to do this once.

   $ git init
   $ git remote add origin <your-github-repo-url>
   $ git branch -M main
   

2. Commit the README file to the Git database and push it to the remote GitHub repository:

   $ git add README.md   # If you are using Level 4
   $ git add README.rst  # If you are using Level 5
   $ git commit -m "docs: add README"
   $ git push -u origin main
   

   Note

   What’s origin?

   origin is the default name for the remote repository under your GitHub account. You can think of it as a nickname for the remote repository. You can also use any other name you like, but origin is the most common convention. For more, please read Do you have a general summary of each term used in the GitHub workflow?.

   Note

   What is -u next to git push?

   The -u flag tells Git to set the upstream (remote) branch for the local branch. This means that in the future, you can simply use git push without specifying the remote and branch name, and Git will know where to push your changes.

Step 2. Automate code linting and testing with GitHub Actions

We will do 3 things in order to automate testing, linting, infrastructure in your GitHub repository. Rest assured that these configuration steps are only done once!

• 1. Setup pre-commit CI in the remote repository in each pull request

• 2. Setup Codecov token for GitHub repository

• 3. Allow GitHub Actions to write comments in PRs

The above steps will take 5 to 10 minutes in total but save hours and days of time in the long run.

1. Setup pre-commit CI in the remote repository in each pull request

Now, we want to ensure pre-commit hooks are also executed when a pull request is made in the remote repository.

1. Visit https://github.com/apps/pre-commit-ci and click Configure.

2. Select the repository(s).

3. Done!

2. Setup Codecov token for GitHub repository

We also want to ensure we report that tests are written for the incoming code and report in the incoming PR as shown below:

Warning

Is this NOT your first time setting up Codecov? Setting up the Codecov report can be done just once for all projects under your account or a GitHub organization with the “global token.” Please check whether you already have a Codecov token. If it exists, you may still follow the steps, but you don’t have to create a new token. Instead, you can use the existing token.

1. Ensure your GitHub repository is public.

2. Visit https://app.codecov.io/account/gh/<your_github_username_or_orgname>/org-upload-token, replacing <your_github_username_or_orgname> with your actual GitHub username or organization name.

3. Under Select an authentication option, select Required.

4. Click Generate or Regenerate to create a new token.

5. Click on the clipboard symbol to copy CODECOV_TOKEN. Copy the one that starts with CODECOV_TOKEN=. Here is an example of what it looks like:

   CODECOV_TOKEN=abcd1234-5678-1234-5678-b862619523bd
   

6. In your GitHub repository, visit Settings ‣ Actions ‣ Secrets and Variables.

7. If the repository is under your personal account, click New repository secret.

8. If the repository is under an organization, click Manage organization secrets ‣ New organization secret.

9. Under the Name field, type CODECOV_TOKEN.

10. Under the Secret field, paste the CODECOV_TOKEN value you copied earlier without any modification.

11. Click Add secret to save the token.

12. Done. From now on, a new Codecov comment will be generated on each PR!

3. Allow GitHub Actions to write comments in PRs

We’d like to have GitHub Actions write comments in the incoming pull request to guide us. We need to provide permissions in the GitHub repository settings by following the steps below.

1. Visit the Settings page of the GitHub repository.

2. Click on Actions in the left sidebar.

3. Click on General in the left sidebar.

4. Scroll down to the Workflow permissions section.

5. Select Read and write permissions.

6. Done!

Step 3. Upload rest of files to GitHub repository with pull request and news item

Upload remaining files

Before we upload anything to the main branch, we want to check the incoming code formatted, tested, and reviewed. We will try to automate these tasks as much as we can while creating a pull request (PR) to the main branch.

1. Just in case, pull the latest code from the remote main branch to your local main branch:

   $ git checkout main
   $ git pull origin main
   

2. Setup pre-commit locally so that code is linted before a commit is made:

   $ pre-commit install
   

3. Checkout a new branch called skpkg-public from the main branch:

   $ git checkout -b skpkg-public
   $ git add .
   $ git commit -m "skpkg: start a new level 5 project with skpkg"
   $ git push -u origin skpkg-public
   

   Note

   Did you see any failed pre-commit hooks after you typed git commit -m? If so, no commit will be made. Confirm by typing git log. Then, simply re-run git add <file> on the files that have been modified by pre-commit and re-enter the same commit message again, such as git commit -m "skpkg: start a new project with skpkg template". If you are having trouble getting a commit to be accepted, please refer to the FAQ section How do I fix conflicted hook auto-fix errors?.

Add news items in the GitHub pull request

Before merging to main, we require that each PR includes a file documenting the changes under \news. This ensures that the changes are documented in the CHANGELOG.rst when you create a new release, as shown in https://scikit-package.github.io/scikit-package/release.html, for example.

Important

If no news file is created for this PR, the CI will not only fail but also write a comment to remind you to create a news file. Recall we granted GitHub Actions permission to write comments in the PR in the previous section.

Let’s create a news item for the changes made in this PR.

1. Check out the skpkg-public branch and sync with the remote branch.

   $ git pull origin skpkg-public
   

2. Make a copy of news/TEMPLATE.rst and rename to news/<branch-name>.rst.

3. (optional) If you are using a Linux shell, you can setup an alias to make the creation of the news file ready for editing much quicker and easier. Read How can I automate the process of creating a new file from the command line? to learn how to setup shortcuts.

4. Do not delete news/TEMPLATE.rst. Leave it as it is.

5. Do not modify other section headers in the rst file. Replace * <news item> with the following item:

   **Added:**
   
   * Support public releases with scikit-package by migrating the package from Level 4 to Level 5 in the scikit-package standard.
   

6. Push the change to the remote GitHub repository.

   $ git add news/skpkg-public.rst
   $ git commit -m "chore: Add news item for skpkg-public"
   $ git push origin skpkg-public
   

Create a pull request

1. In your GitHub repository, click Compare & pull request.

2. Set the PR title as skpkg: start a new project with Level 5 tutorial.

3. The base branch should be main and the compare branch should be skpkg-public.

4. Click Create pull request.

5. Wait all GitHub Action workflows to run including Test on PR.

6. Also wait for pre-commit CI to run and pass.

   Note

   Did pre-commit CI fail?

   If the pre-commit failed, you will need to first pull the new commit created by pre-commit CI before making any new edits locally. You can do this by running the following command:

   $ git pull origin skpkg-public
   

   If you have more problems, please read the FAQ section on How do I fix conflicted hook auto-fix errors?.

7. Click Files changed in the PR to to review the new files added to the repository.

8. Once reviewed, click Merge pull request.

9. Delete the remote branch after merging.

10. Visit your GitHub repository and confirm that the main branch is updated.

11. Congratulations! You are done with creating a Level 5 project with scikit-package.

Ready for public release?

Congratulations! Let’s release your package to PyPI and conda-forge. Visit Publish on GitHub and to PyPI to have your package available via conda install and pip install!

(Recommended) How to develop your code moving forward using pull requests

Assume that you have successfully followed the previous steps. Now, you want to add new code to your GitHub repository. Perhaps you are working with a group of people. Here is a high-level overview with step-by-step instructions on how to do that:

1. Pull the latest code from the remote main branch:

   $ git checkout main
   $ git pull origin main
   

   Note

   Recall that we used the name origin as the nickname for the remote GitHub repository.

2. Ensure that your local main branch is synced with the remote main branch by running:

   $ git log
   

3. Create a new local branch from the main branch. Let’s call this branch skpkg:

   $ git checkout -b <branch-name>
   

4. Modify any file that you want. Then, stage and commit the changes:

   $ git add <file-modified-added-deleted>
   $ git commit -m "feat: <your commit message>"
   

5. Push your code from <branch-name> to the remote <branch-name> branch:

   $ git push --set-upstream origin <branch-name>
   

6. Visit your GitHub repository.

7. Create a PR from origin/<branch-name> to origin/main.

8. Wait for the Tests on PR and pre-commit checks to pass.

9. Merge the PR, delete the branch.

10. Repeat the steps in this section.

11. Done!

(Optional) Useful features available in Level 5

Preview documentation in each pull request

Do you want to review documentation changes easily? Follow the instructions in How can I preview the documentation in each pull request? to view rendered documentation for each pull request with a public URL.

Build API reference documentation

Here is how to create an API reference section, like https://www.diffpy.org/diffpy.utils/api/diffpy.utils.html:

• If your package can be imported as import <package_name>, visit How do I build API .rst files automatically for a standard Python package?.

• If your package supports namespace import, like import <namespace_name>.<package_name>, visit How do I build API .rst files for a Python package with a namespace import?.

Set up default prompt values

When you create a new project in Level 3, 4 or 5, you will be prompted to enter the values for some variables, like maintainer_name, maintainer_email and maintainer_github_username. These variables have default values in the package. It can save your time and reduce errors if you can change the default values that are more relevant for your installation of scikit-package.

To set up your own default promprt values, please refer to How can I change the default values that appear in the prompt when creating projects in Level 3,4,5?.