My Python Setup

[…] because the language is easy to learn and put to use, many practicing Python programmers leverage only a fraction of its powerful features.
— Luciano Ramalho¹

Is the thought of not fully leveraging the Python environment daunting you?² We are not that different then. Thus my constant search for new tools and ways of improving my Python experience.

This is an opinionated summary-guide of my current setup for the future me. Feel free to follow along with me!

My current operating system is Windows. I am using WSL2 inside Windows Terminal. My IDE of choice is PyCharm.

Installing Python

The first step in setting up our dev environment is installing Python in our WSL. We use sudo apt-get update && sudo apt-get dist-upgrade to re-synchronize the package index files, upgrade, and handles dependencies conflicts.

If you are using multiple versions of Python, think about using pyenv. Otherwise, we install our preferred version of Python using the following commands:

sudo apt install software-properties-common
sudo add-apt-repository -y ppa:deadsnakes/ppa
sudo apt install python3.8

Then we check everything went as expected:

$ python3 --version
Python 3.8.5

I prefer having Python 3 as my default version – using python instead of python3. On Ubuntu 20.04+ you can do the following to achieve that:

sudo apt install python-is-python3

Then:

$ python --version
Python 3.8.5

Installing Poetry

Poetry is a tool for dependency management and packaging in Python.

Poetry is my chosen solution for dependencies management. I switched from Pyenv after some stability issues.

To install Poetry:

$ curl -sSL https://raw.githubusercontent.com/python-poetry/poetry/master/get-poetry.py | python -

A quick health check:

$ poetry --version
Poetry version 1.1.4

My playground is laboratory on my C drive. To create a new project:

$ cd /mnt/c/laboratory
$ poetry new python-boilerplate
Created package python_boilerplate in python-boilerplate

Then inside python-boilerplate

$ cd python-boilerplate
$ poetry install
Creating virtualenv python-boilerplate-zUD6aEZx-py3.8 in /home/ayoub/.cache/pypoetry/virtualenvs
Updating dependencies
Resolving dependencies... (1.7s)

Writing lock file

Package operations: 8 installs, 0 updates, 0 removals

  • Installing pyparsing (2.4.7)
  • Installing attrs (20.3.0)
  • Installing more-itertools (8.6.0)
  • Installing packaging (20.7)
  • Installing pluggy (0.13.1)
  • Installing py (1.9.0)
  • Installing wcwidth (0.2.5)
  • Installing pytest (5.4.3)

Installing the current project: python-boilerplate (0.1.0)

Notice that Pytest is installed by default. The highlighted line contains the path to your virtual environments; /home/ayoub/.cache/pypoetry/virtualenvs. We will use this path to set up PyCharm. Keep it!

Setting up PyCharm

We first open the project using your PyCharm. My project directory is /mnt/c/laboratory inside WSL or C:\laboratory\python-boilerplate in Windows.

To add a new Python interpreter to PyCharm: CTRL + ALT + S > Python Interpreter > > Add.

Then from the side menu: WSL > ... > /home/ayoub/.cache/pypoetry/virtualenvs/python-boilerplate-zUD6aEZx-py3.8/bin/python3.8. The path is to your virtual enviroment generated by Poetry. Use poetry env info to print it again.

With Python interpreter set up. Let’s change the Terminal to WSL instead of CMD: CTRL + ALT + S > Terminal > Shell path > wsl.exe

Installing development dependencies

Testing

Programmers! Cast out your guilt! Spend half your time in joyous testing and debugging! Stalk bugs with care, methodology, and reason. Build traps for them. Be more artful than those devious bugs and taste the joys of guiltless programming! ³
— Boris Beizer

Pytest, the framework of choice for testing in Python, is installed by default with Poetry. To keep an eye on coverage we use Coverage.py:

$ poetry add --dev coverage[toml] pytest-cov

The [toml] extension allows us to configure coverage through pyproject.toml file. While pytest-cov is the plugin to use Pytest to generate coverage reports like so:

pytest --cov=myproj tests/

-------------------- coverage: ... ---------------------
Name                 Stmts   Miss  Cover
----------------------------------------
myproj/__init__          2      0   100%
myproj/myproj          257     13    94%
myproj/feature4286      94      7    92%
----------------------------------------
TOTAL                  353     20    94%

If you feel optimistic add the following configuration to your pyproject.toml file:

# pyproject.toml
[tool.coverage.report]
fail_under = 100

For some “Mocking”, explore pytest-mock.

Code linting

Flake8 is a wrapper around these tools:

• PyFlakes checks Python source files for errors.
• Pycodestyle checks your Python code against some of the style conventions in PEP 8.
• Ned Batchelder’s McCabe script checks McCabe complexity

Flake8 runs all those tools by launching a single flake8 command. It displays the warnings in a per-file, merged output. Flake8 could be reinforced with additional extensions. We will use a few of those here.

$ poetry add --dev flake8

To configure your Flake8 create .flake8 in the root of your project and add these lines:

# .flake8
[flake8]
extend-ignore = E203, E266, E501
max-line-length = 88
max-complexity = 18

For Flake8 to play nice with Black we will be ignoring some rules:

• E203: Whitespace before ‘:’
• E266: Too many leading ‘#’ for block comment
• E501: Line longer than 79 characters

We ignored the E501 while setting the max line length to 88; the default in Black.

Code formatting

Speaking of which, Black is the uncompromising Python code formatter. Black just works. No need for configuration. We use Flake8’s extension flake8-black to run Black --check ... from within the Flake8 plugin ecosystem.

$ poetry add --dev black flake8-black

The basic configuration for Black:

# Borrowed from: https://github.com/psf/black/blob/master/pyproject.toml
# pyproject.toml
[tool.black]
include = '\.pyi?$'
exclude = '''
/(
    \.git
  | \.hg
  | \.mypy_cache
  | \.tox
  | \.venv
  | _build
  | buck-out
  | build
  | dist
)/
'''

We add isort a Python utility to sort imports alphabetically, and automatically separated into sections and by type.

$ poetry add --dev isort flake8-isort

Then again:

# pyproject.toml
[tool.isort]
multi_line_output = 3
include_trailing_comma = true
force_grid_wrap = 0
use_parentheses = true
ensure_newline_before_comments = true
line_length = 88

Defining tasks

Every development pipeline has tasks, such as test, lint or publish. With taskipy, you can define those tasks in one file and run them with a simple command.

$ poetry add --dev taskipy

# myproject.toml
[tool.taskipy.tasks]
format = "black . && isort ."

Pre-commit hooks

I think we should use Git for this project, even though python-boilerplate-v3 is a much more elegant solution than Git syntax.

To make the pipeline more robust we use pre-commit. A framework for managing and maintaining multi-language pre-commit hooks.

Git hook scripts are useful for identifying simple issues before submission to code review. We run our hooks on every commit to automatically point out issues in code such as missing semicolons, trailing whitespace, and debug statements.

$ poetry add --dev pre-commit

Place the configuration in the root directory inside .pre-commit-config.yaml:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/psf/black
    rev: 20.8b1
    hooks:
    -   id: black
  - repo: https://github.com/PyCQA/isort
    rev: '5.6.4'
    hooks:
    -   id: isort
  - repo: https://gitlab.com/pycqa/flake8
    rev: 3.8.4
    hooks:
      - id: flake8

We run with this configuration Black, Isort, and Flake8 before each commit.

Documentation

I always document my functions. To do the same, and once you are done writing your function in PyCharm, right after your function definition, start typing a multiline string starting with """. PyCharm will generate the boilerplate for your function docstring documentation automatically.

To check for these docstrings we will use flake8-docstrings.

$ poetry add --dev flake8-docstrings

We use Sphinx to extract these definitions into standalone documentation. And sphinx-autodoc-typehints brings type hints supports to Sphinx.

$ poetry add --dev sphinx sphinx-autodoc-typehints

Create a docs directory to store your documentation:

$ mkdir docs && cd docs
$ poetry run sphinx-quickstart

Then change the list of extensions inside of docs/cong.py

# docs/cong.py
extensions = [
    "sphinx.ext.autodoc",
    "sphinx_autodoc_typehints",
]

After that let’s create the following file:

.. docs/api.rst
API!
===============================================
.. automodule:: src.python_boilerplate.main
   :members:

And reference it inside the index.rst:

.. docs/index.rst
.. toctree::
   :maxdepth: 2
   :caption: Contents:

   api

$ poetry run make html

Check the generated HTML files for the docs!

Conclusion

The combinations of Python libraries are truly endless. And that’s the power of Python — laugh in Elm. We didn’t get into CI/CD pipelines and containerization, I opted to keep those for future explorations. And with that, I wish you a Happy new year!