Managing Python Versions and Packages
Opinionated Guide to Managing Python Dev Environments
Do not rely on system Python
Getting started with Python should be easy, right? After all, most OS ship with a version of Python. However, most OS ship with a specific version on Python which they rely on to run services and scripts. The installed version may change after upgrades. It is recommended to not install packages system-wide. System Python is best managed by the OS.
Package Managers are cool
How should you install the Python version needed for development? Most OS ship with a package manager. Linux variants have
dnf etc. MacOS users can install
brew . Package managers fetch info from a repository and install the right version based on the OS variant, version and system architecture. This is a good way to start but it gets complicated to manage multiple Python versions. You may need different version to test compatibility or try new language features.
The download section for Python 3.12 lists 9 files. 2 are compressed source code and rest are installers covering major OS and architectures:
|Gzipped source tarball
|XZ compressed source tarball
|macOS 64-bit universal2 installer
|Windows embeddable package (32-bit)
|Windows embeddable package (64-bit)
|Windows embeddable package (ARM64)
|Windows installer (32 -bit)
|Windows installer (64-bit)
|Windows installer (ARM64)
Notice that there is no installer for Linux. Perhaps, creating a universal installer for Windows and MacOS is easier. Linux distributions maintain their Python packages which are built from the source releases. Package managers can only install the versions available in the repository. This is important so that users only install stable versions.
There must be a better way
To install a dev or beta version, you either compile it from source or use a non-official package repo. You would agree that a single way to install released/dev versions would be convenient. One way to do so is using
Pyenv helps install multiple Python versions (including release candidates, dev) and different implementations like pypy, stackless, pyston, etc. Start with the installation instructions here which depends on OS.
Once installed, you can search the available versions like this:
# Using 3.12 as an example as it has the least options right now
pyenv install -l | grep 3.12
Other common commands are:
pyenv versionswhich lists installed versions
pyenv globallets you manage the global Python version
pyenv updateworks on Linux. On MacOS, this can be done with
Removal is as easy as
rm -rf ~/.pyenv/versions/"X.Y.Z" (Use
-rf with caution). Less complicated than using system package manager to find and remove older Python versions.
Pyenv relies on the sources and compiles them on your machine. You may get errors in compilation if the required OS packages are not found. This is opposite of the convenience promised. This is the reason we discussed package managers and the challenges associated at the start. If you run into an issue, refer to common build problems.
Once you have a Python version installed, you need to manage projects. There are two problem:
Install a project specific Python version.
Manage the package dependencies.
There are many ways to create virtual environments, the simplest being
python -m venv <name> . Dependencies can be managed by
pip using a
requirements.txt file. These are part of the standard Python library.
Things get complicated as the size of project and its dependencies increases. Most packages depend on other packages which are installed with them. Say, if you install PackageA, and it installs PackageB, PackageB.
pip freeze will list all the installed packages. There's no easy way to separate top-level packages and you end up managing the
requirements.txt file manually. Similarly, when packages need to be updated, there may be conflicts. As with all things Python, there are multiple solutions to managing dependencies.
Poetry, also, solves these issues and provides a way to quick start new projects.
poetry new demo
Created package demo in demo
➜ demo cd demo
➜ demo ls -lh
-rw-r--r-- 1 manas staff 0B Dec 9 14:14 README.md
drwxr-xr-x 3 manas staff 96B Dec 9 14:14 demo
-rw-r--r-- 1 manas staff 257B Dec 9 14:14 pyproject.toml
drwxr-xr-x 3 manas staff 96B Dec 9 14:14 tests
➜ demo cat pyproject.toml
name = "demo"
version = "0.1.0"
description = ""
authors = ["Your Name <email@example.com>"]
readme = "README.md"
python = "^3.12"
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
The pyproject.toml file is used to manage project and dependencies:
poetry envmanage virtualenvs.
poetry add <package>installs packages
poetry updatewill update dependencies
You can organise packages in separate dependency groups like
test. For example, IPython, ruff, rich, or black can be added to a
pytestcan be in a
testgroup. This helps you install only the required packages when creating containers or deploying code.
Read the docs, and explore all the features.
Note that both pyenv and poetry can be installed at system level while other things are at project level.
Lastly, if you code and care about code quality, use the following:
Open source code analyser SonarQube Python for large projects
P.S. Get work done
Remember that the time spent in figuring out the best Linux distro, programming language (or paradigm), and package manager may not count as work.
Python has no compile step but you may continue to look for the best editor, typeface, and theme while the code is running.