(Level 4) Share your code as a locally installable Python package

Overview

In this tutorial, you will learn how to share code across any files in your computer. By the end of this tutorial, you will also have your package hosted on GitHub, where you will utilize GitHub Actions to run automatic linting and testing for the incoming code in your GitHub repository.

This tutorial should take about 10 to 15 minutes.

Table of contents

1. Step 1. Create a new package with scikit-package

2. Step 2. Automate code linting and testing with GitHub Actions

3. Step 3. Upload rest of files to GitHub repository with pull request

Prerequisites

For Level 4, we assume you have prior experience in developing scientific code in Python.

See also

Are you new to Git and GitHub?

Please read the GitHub workflow section in GitHub workflow to familiarize yourself with the common jargon and terms used. There are also many online resources for learning Git and GitHub.

Install scikit-package with conda

1. Ensure that conda is installed on your local computer as instructed in (Required) Use conda environment to install packages and run Python code.

2. Ensure that you have added the conda-forge channel to your conda configuration.

   $ conda config --add channels conda-forge
   

3. Create a new conda environment skpkg-env and install scikit-package and pre-commit simultaneously by running the following command:

   $ conda create -n skpkg-env scikit-package pre-commit
   

   Note

   If the above command does not work, ensure you have added the conda-forge channel to your conda configuration in Step 2.

4. Activate the environment:

   $ conda activate skpkg-env
   

5. Confirm that you have the latest version of scikit-package available on GitHub:

   $ pip show scikit-package
   

Step 1. Create a new package with scikit-package

Create a new GitHub repository

In this tutorial, we will start by creating a new GitHub repository. The GitHub repository is where we will host our code online (remote).

1. Visit https://github.com/new.

2. Enter the Repository name. You may use my-package or any other name you like.

3. Choose Public.

4. We want to create an empty repository so set None under Add .gitignore.

5. Set None Under Choose a license.

6. Click the Create repository green button to create the repository.

7. Done!

Let’s now create a new package in your computer (local) using scikit-package.

Create a new project with scikit-package

1. Run the following command to create a new project with scikit-package:

   $ package create system
   

2. Respond to the following prompts:

   Prompt	Description and example
   project_name	The project name. Will be used for the top level folder name. e.g., my-package
   github_username_or_orgname	Your GitHub username, or GitHub organization name, where it will be hosted on GitHub. e.g., sbillinge or billingegroup
   github_repo_name	The GitHub repository name. Usually the same as the project_name. Use name-with-hypens. e.g., my-package
   conda_pypi_package_dist_name	The name displayed on PyPI and conda-forge. Must not already exist there. Use name-with-hypens. e.g., my-package
   package_dir_name	The name of the package directory under src. Usually the same as the project name but use name_with_underscores. e.g., my_package
   contributors	The contributors involved in the project. e.g., Sangjoon Lee, Simon Billinge

   Note

   You may press the “Enter” key on your keyboard to accept the default value for the field provided in parentheses.

3. cd into the project directory created by the package create system command above. We will assume that the user has entered the project name as my-package.

   $ cd my-package
   

4. Confirm that you have the following folder structure shown below:

   my-package/
   ├── LICENSE.rst
   ├── README.md
   ├── pyproject.toml
   ├── requirements
   │   ├── conda.txt
   │   ├── pip.txt
   │   └── test.txt
   ├── src
   │   └── my_package
   │   ├── __init__.py
   │   └── functions.py
   └── tests
       └── test_functions.py
   

5. Done! Let’s now install your package in your local computer where the code can be used in any Python script or Jupyter notebook.

Install your package locally

1. Create a new conda environment. Let’s call this environment my-package-env:

   $ conda create -n my-package-env python=3.13 \
       --file requirements/conda.txt \
       --file requirements/test.txt
   

2. Activate the conda environment:

   $ conda activate my-package-env
   

3. Build and install the package locally:

   $ pip install -e . --no-deps
   

   Note

   What is the -e flag?

   The -e flag indicates that you want to install the package in “editable” mode, which means that any changes you make to the source code will be reflected immediately without needing to reinstall the package. This is useful for development purposes.

   Note

   What is the --no-deps flag?

   The --no-deps flag tells pip not to install any dependencies listed in requirements/pip.txt. This is because we have already installed the dependencies in the conda environment using the command above.

   See also

   Why is it required to list dependencies both under pip.txt and conda.txt? Please refer to the FAQ section Dependency management.

4. Then, run the tests using the following command:

   $ pytest
   

5. Ensure tests all pass with green checkmarks. Notice that in tests/test_functions.py, we are importing the locally installed package.

6. Done!

Upload README.md to your GitHub repository

At the moment, the GitHub repository is empty. Let’s create a local branch called main and upload this local branch to the remote GitHub repository.

1. Follow the series of steps to initialite Git. You only have to do this once.

   $ git init
   $ git remote add origin <your-github-repo-url>
   $ git branch -M main
   

2. Commit the README file to the Git database and push it to the remote GitHub repository:

   $ git add README.md   # If you are using Level 4
   $ git add README.rst  # If you are using Level 5
   $ git commit -m "docs: add README"
   $ git push -u origin main
   

   Note

   What’s origin?

   origin is the default name for the remote repository under your GitHub account. You can think of it as a nickname for the remote repository. You can also use any other name you like, but origin is the most common convention. For more, please read Do you have a general summary of each term used in the GitHub workflow?.

   Note

   What is -u next to git push?

   The -u flag tells Git to set the upstream (remote) branch for the local branch. This means that in the future, you can simply use git push without specifying the remote and branch name, and Git will know where to push your changes.

Step 2. Automate code linting and testing with GitHub Actions

Setup pre-commit to lint code before making a commit

First, let’s ensure that before we upload any of our code to the remote repository, we lint the code and ensure that it is formatted correctly. We will use a library called pre-commit to do this.

1. Configure pre-commit to run each time a new commit is made. You only have to do this once. We assume you have pre-commit installed in your environment by running conda install pre-commit:

   $ pre-commit install
   

2. Let’s now stage and commit the code:

   $ git add .
   $ git commit -m "skpkg: initial commit of a new project"
   

3. Ensure that all of the pre-commit hooks pass:

   black....................................................................Passed
   prettier.................................................................Passed
   docformatter.............................................................Passed
   

   Note

   black is a tool that automatically formats Python code to conform to the PEP 8 style guide. prettier is a tool that formats code in various languages, including .md, .rst, and .json files. docformatter is a tool that formats docstrings in Python code.

4. You will see the new commit in the git log:

   $ git log
   

   Note

   Did you see any failed pre-commit hooks? If so, no commit will be made. Simply re-run git add <file> on the files that have been modified by pre-commit and re-enter the same commit message again, such as git commit -m "skpkg: start a new project with skpkg template". If you are having trouble getting a commit to be accepted, please refer to the FAQ section here.

Setup pre-commit CI in the remote repository in each pull request

Now, we want to ensure pre-commit hooks are also executed when a pull request is made in the remote repository.

1. Visit https://github.com/apps/pre-commit-ci and click Configure.

2. Select the repository(s).

3. Done!

Step 3. Upload rest of files to GitHub repository with pull request

While we previously uploaded the README file to the remote GitHub main repository, this is not a recommended workflow. We want to ensure that before any code is pushed to the main branch, the incoming code formatted, tested, and reviewed. We will try to automate these tasks as much as we can while creating a pull request (PR) to the main branch.

1. Just in case, pull the latest code from the remote main branch to your local main branch:

   $ git checkout main
   $ git pull origin main
   

2. Checkout a new branch called skpkg-system from the main branch:

   $ git checkout -b skpkg-system
   $ git add .
   $ git commit -m "skpkg: start a new project with skpkg"
   $ git push -u origin skpkg-system
   

3. Visit your GitHub repository online.

4. Click on the new green button that says Compare & pull request.

5. The PR title can be skpkg: start a new project with skpkg template.

6. The base branch should be main and the compare branch should be skpkg-system.

7. Click on the Create pull request button.

8. Wait for Tests on PR to run and pass. It runs pytest on the incoming code in each pull request.

9. Also wait for pre-commit CI to run and pass.

   Note

   Did pre-commit CI fail?

   If the pre-commit failed, you will need to first pull the new commit created by pre-commit CI before making any new edits locally. You can do this by running the following command:

   $ git pull origin skpkg-system
   $ git add <file-modified-that-fixes-pre-commit-error>
   $ git commit -m "chore: <your commit message>"
   $ git push
   

10. Click Files changed in the PR to to review the new files added to the repository.

11. Once reviewed, click Merge pull request.

12. Delete the skpkg-system remote branch after merging.

13. Visit your GitHub repository and confirm that the main branch is updated.

14. Congratulations! You are done!

(Recommended) How to develop your code moving forward using pull requests

Assume that you have successfully followed the previous steps. Now, you want to add new code to your GitHub repository. Perhaps you are working with a group of people. Here is a high-level overview with step-by-step instructions on how to do that:

1. Pull the latest code from the remote main branch:

   $ git checkout main
   $ git pull origin main
   

   Note

   Recall that we used the name origin as the nickname for the remote GitHub repository.

2. Ensure that your local main branch is synced with the remote main branch by running:

   $ git log
   

3. Create a new local branch from the main branch. Let’s call this branch skpkg:

   $ git checkout -b <branch-name>
   

4. Modify any file that you want. Then, stage and commit the changes:

   $ git add <file-modified-added-deleted>
   $ git commit -m "feat: <your commit message>"
   

5. Push your code from <branch-name> to the remote <branch-name> branch:

   $ git push --set-upstream origin <branch-name>
   

6. Visit your GitHub repository.

7. Create a PR from origin/<branch-name> to origin/main.

8. Wait for the Tests on PR and pre-commit checks to pass.

9. Merge the PR, delete the branch.

10. Repeat the steps in this section.

11. Done!

What’s next?

Once you are ready to release your package to the wider world, let’s proceed to (Level 5) Share your code as a publicly installable package where you will learn to release your package to PyPI and conda-forge so that your package can be installed by anyone in the world.