Migrate your existing package with scikit-package

Prerequisites

This guide is for developers who have an existing Python package under Git control and want to migrate it to the Level 5 scikit-package standard.

We assume you have experience creating a package using scikit-package Level 4 and 5. We also assume you are familiar with the forking workflow on GitHub.

See also

If you are not familiar with contributing via forking, please read What is the general the workflow?.

Tips and how to receive support

We understand that migration can be challenging. We offer many ways to help you guide through the process:

1. You may cross-check with the Billinge group’s up-to-date package, diffpy.utils: https://github.com/diffpy/diffpy.utils.

2. We provide Frequently asked questions (FAQ) describing design decisions and how to override them in the scikit-package template.

3. After you’ve cross-checked and searched through the documentation (tutorials, examples, FAQs) please feel free to ask questions by creating an issue in the GitHub repository here.

Install scikit-package with conda

1. Ensure that conda is installed on your local computer as instructed in (Required) Use conda environment to install packages and run Python code.

2. Ensure that you have added the conda-forge channel to your conda configuration.

   $ conda config --add channels conda-forge
   

3. Create a new conda environment skpkg-env and install scikit-package and pre-commit simultaneously by running the following command:

   $ conda create -n skpkg-env scikit-package pre-commit
   

   Note

   If the above command does not work, ensure you have added the conda-forge channel to your conda configuration in Step 2.

4. Activate the environment:

   $ conda activate skpkg-env
   

5. Confirm that you have the latest version of scikit-package available on GitHub:

   $ pip show scikit-package
   

Step 1. Pre-commit workflow

1. Clone the repository and set upstream to the original repository.

   $ git clone <URL-of-the-forked-repo>
   $ cd <package-name>
   $ git remote add upstream <URL-of-the-original-repo>
   

   Note

   • Are you the creator of the repository? You can type origin instead of upstream in the rest of the tutorial, e.g., git pull origin main instead of git pull upstream main. Here, we assume a forking workflow.

   • Is your default branch called master? Run git pull upstream master throughout the guide instead. However, please note that main is the default branch name for GitHub.

Run black in your codebase

1. Create a new branch called black-edits. This branch will be used to apply black to all files in the old project directory:

   $ git checkout main
   $ git pull upstream main
   

2. Activate the conda environment and install black:

   $ conda activate skpkg-env
   $ conda install black
   

3. Create a new branch called black-edits and create a new file called pyproject.toml:

   $ git checkout -b black-edits
   $ touch pyproject.toml
   

4. Copy and paste the following content at the bottom of pyproject.toml:

   [tool.black]
   line-length = 79
   include = '\.pyi?$'
   exclude = '''
   /(
       \.git
   | \.hg
   | \.mypy_cache
   | \.tox
   | \.venv
   | \.rst
   | \.txt
   | _build
   | buck-out
   | build
   | dist
   | blib2to3
   | tests/data
   )/
   

5. Lint the code:

   $ black .
   

   See also

   To skip certain files, add them under the exclude section in the pyproject.toml.

6. Push the changes to the black-edits branch:

   $ git add .
   $ git commit -m "skpkg: apply black to all files in the project directory"
   $ git push origin black-edits
   

7. Create a PR from username/black-edits to upstream/main.

   The PR title can be skpkg: apply black line-length 79 to all files in the project directory.

8. Review and wait for the PR to be merged to upstream/main. If you are the project maintainer, you can merge the PR yourself.

9. Done!

Apply pre-commit auto-fixes without manual edits

1. Sync with the upstream/main branch:

   $ git checkout main && git pull upstream main
   

2. Create a new branch called pre-commit-auto:

   $ git checkout -b pre-commit-auto
   

3. Create a new package using scikit-package:

   $ package create public
   

4. Copy the pre-commit configuration files from the new to the old directory:

   $ cp <package-name>/.pre-commit-config.yaml .
   $ cp <package-name>/.isort.cfg .
   $ cp <package-name>/.flake8 .
   

5. Trigger hooks and auto-fixes without manual edits:

   $ pre-commit run --all-files
   

6. Add the changes to the pre-commit-auto branch:

   $ git add . && git commit -m "style: apply pre-commit hooks with no manual edits"``
   

7. Push the changes to the remote repository:

   $ git push origin pre-commit-auto
   

8. Create a PR from username/pre-commit-auto to upstream/migration. The PR title can be skpkg: apply pre-commit to project directory with no manual edits.

   Note

   The new upstream/migration branch can be created by the project maintainer or owner. On the main page of the upstream repository, click main ‣ Switch branches/tags ‣ Find or create a branch and type migration. This will create a new branch called migration.

9. Wait for the PR to be merged to upstream/migration branch.

Apply manual edits to pass pre-commit hooks

The package will most likely have failed pre-commit hooks. We will manually fix the errors raised by flake8, codespell, etc.

Note

Do you have no pre-commit errors after running pre-commit run --all-files? Great! You can skip this section and move on to the next section.

Here, instead of fixing all errors at once, we will address each type of error one at a time. For example, a branch called pre-commit-spelling may contain spelling fixes, while another branch, pre-commit-flake8-line fixes of line length errors raised by flake8.

1. Sync with the upstream/migration branch:

   $ git checkout migration
   $ git pull upstream migration
   

2. Create a new branch that will be used to fix the type of errors:

   $ git checkout -b pre-commit-<theme>
   

3. Run pre-commit run --all-files to see the errors:

   See also

   Do you want to ignore certain files/folders and also set the preferred line-length? Visit Pre-commit in the FAQ section to learn how to customize line-lenghts, ignore certain errors, and skip certain files and paths.

   If you are suppressing flake8 errors, you can create GitHub issues to resolve them after migration.

4. Fix the errors manually for the specific theme.

5. Once you are done with fixing the type of errors commit, push the changes to your branch

   $ git add <files-modified-to-fix-error>
   $ git commit -m "skpkg: fix <theme> errors"
   $ git push origin pre-commit-<theme>
   

6. Create a PR from username/pre-commit-<theme> to upstream/migration. The PR title can be skpkg: fix <theme> errors.

7. Wait for the PR to be merged to upstream/migration branch.

8. Repeat for other themes of errors, always pull the latest commits from upstream/migration before creating a new branch:

   $ git checkout migration
   $ git pull upstream migration
   $ git checkout -b pre-commit-<another-theme>
   

9. Are all the PRs merged and do all pre-commit hooks pass? If so, you are ready for the next section! Before that, let’s automatically trigger pre-commit hooks going forward:

   $ pre-commit install
   

Setup pre-commit CI

Now, we want to ensure pre-commit hooks are also executed when a pull request is made in the remote repository.

1. Visit https://github.com/apps/pre-commit-ci and click Configure.

2. Select the repository(s).

3. Done!

Step 2. Migration workflow

Let’s first migrate the essential files from the old project to the new project directory.

Attention

Please read the following carefully before proceeding:

• Please Do NOT delete/remove any files before confirming that it is absolutely unnecessary. If you are unsure, contact the project maintainer first.

• Please Do NOT delete project-specific content such as project descriptions in README, license information, authors, tutorials, examples.

Move essential files to run local tests

1. Pull the latest commits from upstream/migration

   $ git checkout migration && git pull upstream migration
   

2. Move into the new directory created by scikit-package:

   $ cd <package-name>
   

3. Move .git from the old (..) to the new directory (.):

   $ mv ../<package-name>/.git .
   

4. See a list of files that have been (1) untracked, (2) deleted, (3) modified:

   $ git status
   

   See also

   • untracked are new files created by scikit-package.

   • deleted are files in the old directory but the files that are not in the new directory. At the moment, most of the src and tests and doc files will be in this category. We will move them from the old to the new directory in the next few steps.

   • modified are files that that exist both in the old and the new directory, while the scikig-package has made changes to them.

5. Create a new branch called called setup-CI:

   git checkout -b setup-CI
   

6. Now you will move src and tests folders in the following steps.

7. Copy files

   $ cp -n -r ../src .
   $ cp -n -r ../tests .
   

   See also

   The -n option is used to prevent overwriting existing files in the destination directory. If you want to overwrite existing files, remove the -n option.

8. Confirm the src and tests files that showed as deleted are no longer in the deleted.

   $ git status
   

9. Commit the changes:

   $ git add src && git commit -m "skpkg: mirate src folder"
   $ git add tests && git commit -m "skpkg: migrate tests folder"
   

10. Manually list the dependencies under requirements/pip.txt, requirements/tests.txt, requirements/docs.txt, requirements/conda.txt.

11. Delete requirements/build.txt if your package only contains Python code.

    $ rm requirements/build.txt
    

12. Add and commit the changes in the requirements folder:

    $ git add requirements
    $ git commit -m "skpkg: list dependencies in requirements folder"
    

13. Test your package from a new conda environment:

    $ conda create -n <package-name>-env python=3.13 \
        --file requirements/conda.txt \
        --file requirements/test.txt
    $ conda activate <package-name>-env
    $ pip install -e . --no-deps
    $ pytest
    

    Note

    Include --file requirements/build.txt if it exists.

14. Congratulations! After the tests pass, let’s now setup GitHub Actions in the following section.

Setup repository

We want to run GitHub Actions, in order to do so

• 1. Setup pre-commit CI in the remote repository in each pull request

• 2. Setup Codecov token for GitHub repository

• 3. Allow GitHub Actions to write comments in PRs

Setup GitHub Actions

1. Add and commit the files in the .github folder:

   $ git add .github/workflows/tests-on-pr.yml .gitignore
   $ git commit -m "skpkg: add CI and issue/PR templates"
   

   Attention

   If your package does not support the latest Python version of 3.13, you will need to specify the Python version supported by your package. Follow the instructions here to set the Python version under .github/workflows in In Level 5, How do I set different Python versions for GitHub CI?.

2. Add and commit the requirements folder:

   git add requirements
   git commit -m "skpkg: list dependencies in requirements folder"
   

3. Add and commit pyproject.toml:

   git add pyproject.toml
   git commit -m "skpkg: add pyproject.toml"
   

4. Push the changes to the setup-CI branch:

   git push origin setup-CI
   

5. Create a PR from username/setup-CI to upstream/migration.

   The pull request title can be skpkg: setup CI after migrating tests, src, requirements, and .github folder.

6. Wait for review after CI passes.

7. Once the PR is merged, let’s upload other files.

Add configuration files

1. Pull the latest commits from upstream/migration and create a new branch called config:

   $ git checkout migration
   $ git pull upstream migration
   $ git checkout -b config
   

2. Add and commit configuration files:

   $ git add .pre-commit-config.yaml .codespell .flake8 .isort.cfg
   $ git commit -m "skpkg: add config files for pre-commit "
   $ git add .readthedocs.yaml .codecov.yml .github
   $ git commit -m "skpkg: add config files readthedocs, codecov, GitHub"
   

3. Create a PR from username/config to upstream/migration.

   The PR title can be skpkg: add configuration files for pre-commit, readthedocs, codecov.

4. Once the PR is merged, move to the next section.

Move documentation files

1. Pull the latest commits from upstream/migration and create a new branch called doc:

   $ git checkout migration
   $ git pull upstream migration
   $ git checkout -b doc
   

2. Copy documentation from the old to the new repository:

   $ cp -n -r ../doc/source/* ./doc/source.
   

   Note

   If files are moved to a different path like doc/manual/source (old) to doc/source (new), open the project in IDE and do a global search (ctrl + shift + f) for ../ or .. and modify all relative path instances.

3. Ensure the documentation can be built locally:

   $ conda install --file requirements/docs.txt
   $ cd doc && make html
   & open _build/html/index.html
   

4. Add and commit the changes:

   $ git add doc
   $ git commit -m "skpkg: migrate documentation"
   

5. By hand, migrate content over to README.rst.

   Note

   In Level 5, we provide a rich template for README.rst instead of using README.md. If you already had a rich README.md in Level 4, you can use a tool to convert .md to .rst. For example, you may use this free CloudConvert tool.

6. Add other public facing static files:

   $ git add AUTHORS.rst CHANGELOG.rst CODE_OF_CONDUCT.rst LICENSE.rst
   $ git commit -m "skpkg: add config files for authors, changelog, code of conduct, license"
   $ git add MANIFEST.in
   $ git commit -m "skpkg: add MANIFEST.in"
   $ git add README.rst
   $ git commit -m "skpkg: add README.rst"
   

7. Create a news file:

   $ cp news/TEMPLATE.rst news/doc.rst
   

8. In news/docs..rst, add the following content under Fixed::

   **Added:**
   
   * <news item>
   
   **Fixed:**
   
   * Support ``scikit-package`` Level 5 standard (https://scikit-package.github.io/scikit-package/).
   

9. Add the news files:

   $ git add news
   $ git commit -m "skpkg: add news files"
   

10. Create a PR from usernmae/doc to upstream/migration.

    The PR title can be skpkg: migrate documentation, README, and public static files.

11. Once the PR is merged to upstream/main, move to the final step!

Step 3. Final check

1. Review the following items:

   • All the badges on README.rst are passing.

   • LICENSE.rst is verified as correct.

   • Locally rendered documentation contains all appropriate pages, tutorials, and other human-written text is up-to-date with any changes in the code.

   • Installation instructions in the README.rst, documentation, and the website are updated.

   • Successfully run any tutorial examples or do functional testing with the latest Python version.

   • Grammar and writing quality are checked (no typos).

2. Ask the project maintainer to create a PR from upstream/migration to upstream/main

3. After the PR is merged to upstream/main, archive the old repository by naming it:

   $ mv <package-name> <package-name>-archive
   

4. Clone the latest version of the package from the remote:

   $ cd ~/dev
   $ git clone <URL-of-the-forked-repo>
   $ git remote add upstream <URL-of-the-original-repo>
   $ git pull upstream main
   

5. Now, you should be able to run the following to test your package!

   $ conda activate <package-name>-env
   $ pytest
   $ pre-commit run --all-files
   

6. Congratulations! You are done with migration!

Ready for release?

Are you ready to release your package to PyPI and conda-forge? Let’s start from Publish on GitHub and to PyPI!