Do not rely on system Python

Getting started with Python should be easy, right? After all, most OS ship with a version of Python. However, most OS ship with a specific version on Python which they rely on to run services and scripts. The installed version may change after upgrades. It is recommended to not install packages system-wide. System Python is best managed by the OS.

Package Managers are cool

How should you install the Python version needed for development? Most OS ship with a package manager. Linux variants have yum, apt, dnf etc. MacOS users can install brew . Package managers fetch info from a repository and install the right version based on the OS variant, version and system architecture. This is a good way to start but it gets complicated to manage multiple Python versions. You may need different version to test compatibility or try new language features.

The download section for Python 3.12 lists 9 files. 2 are compressed source code and rest are installers covering major OS and architectures:

Notice that there is no installer for Linux. Perhaps, creating a universal installer for Windows and MacOS is easier. Linux distributions maintain their Python packages which are built from the source releases. Package managers can only install the versions available in the repository. This is important so that users only install stable versions.

There must be a better way

To install a dev or beta version, you either compile it from source or use a non-official package repo. You would agree that a single way to install released/dev versions would be convenient. One way to do so is using pyenv .

Pyenv

Pyenv helps install multiple Python versions (including release candidates, dev) and different implementations like pypy, stackless, pyston, etc. Start with the installation instructions here which depends on OS.

Usage

Once installed, you can search the available versions like this:

# Using 3.12 as an example as it has the least options right now
pyenv install -l | grep 3.12
  3.12.0rc2
  3.12-dev
  pypy2.7-7.3.12-src
  pypy2.7-7.3.12
  pypy3.9-7.3.12-src
  pypy3.9-7.3.12
  pypy3.10-7.3.12-src
  pypy3.10-7.3.12

Other common commands are:

• pyenv versions which lists installed versions

• pyenv global lets you manage the global Python version

• pyenv update works on Linux. On MacOS, this can be done with brew update

Removal is as easy as rm -rf ~/.pyenv/versions/"X.Y.Z" (Use -rf with caution). Less complicated than using system package manager to find and remove older Python versions.

Troubleshooting

Pyenv relies on the sources and compiles them on your machine. You may get errors in compilation if the required OS packages are not found. This is opposite of the convenience promised. This is the reason we discussed package managers and the challenges associated at the start. If you run into an issue, refer to common build problems.

Dependency Management

Pip Problems

Once you have a Python version installed, you need to manage projects. There are two problem:

1. Install a project specific Python version.

2. Manage the package dependencies.

There are many ways to create virtual environments, the simplest being python -m venv <name> . Dependencies can be managed by pip using a requirements.txt file. These are part of the standard Python library.

Things get complicated as the size of project and its dependencies increases. Most packages depend on other packages which are installed with them. Say, if you install PackageA, and it installs PackageB, PackageB. pip freeze will list all the installed packages. There's no easy way to separate top-level packages and you end up managing the requirements.txt file manually. Similarly, when packages need to be updated, there may be conflicts. As with all things Python, there are multiple solutions to managing dependencies.

Poetic Solutions

Poetry, also, solves these issues and provides a way to quick start new projects.

poetry new demo
Created package demo in demo
➜  demo cd demo 
➜  demo ls -lh
total 8
-rw-r--r--  1 manas  staff     0B Dec  9 14:14 README.md
drwxr-xr-x  3 manas  staff    96B Dec  9 14:14 demo
-rw-r--r--  1 manas  staff   257B Dec  9 14:14 pyproject.toml
drwxr-xr-x  3 manas  staff    96B Dec  9 14:14 tests
➜  demo cat pyproject.toml 
[tool.poetry]
name = "demo"
version = "0.1.0"
description = ""
authors = ["Your Name <you@example.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"


[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

The pyproject.toml file is used to manage project and dependencies:

• poetry env manage virtualenvs.

• poetry add <package> installs packages

• poetry update will update dependencies

• You can organise packages in separate dependency groups like dev , docs, test. For example, IPython, ruff, rich, or black can be added to a dev group. While pytest can be in a test group. This helps you install only the required packages when creating containers or deploying code.

Read the docs, and explore all the features.

Note that both pyenv and poetry can be installed at system level while other things are at project level.

Code Quality

Lastly, if you code and care about code quality, use the following:

• Ruff is an extremely fast code linter and formatter, or Black

• Open source code analyser SonarQube Python for large projects

P.S. Get work done

Remember that the time spent in figuring out the best Linux distro, programming language (or paradigm), and package manager may not count as work.

Python has no compile step but you may continue to look for the best editor, typeface, and theme while the code is running.