About modern packaging

requirements.txt vs. pyproject.toml

requirements.txt – The Traditional Approach

• requirements.txt has been the de facto standard for listing dependencies since early Python packaging days.
• It was primarily designed to list and freeze dependencies for reproducible environments.
• Supported by pip (pip install -r requirements.txt).

Pros of requirements.txt

• ✅ Simple and explicit – Just a plain text file listing dependencies.
• ✅ Compatible with all tools – Works with pip, virtualenv, Docker, etc.
• ✅ Pinning dependencies – You can specify exact versions for reproducibility.

Cons of requirements.txt

• ❌ Lacks metadata – No package name, versioning, authors, or descriptions.
• ❌ No native support for optional dependencies – You need multiple files (dev-requirements.txt, test-requirements.txt).
• ❌ Not declarative – Doesn’t integrate well with modern Python package management tools.

Use Case for requirements.txt

• Production deployments – When freezing exact dependencies (pip freeze > requirements.txt).
• Docker builds – COPY requirements.txt / && pip install -r requirements.txt.
• Virtual environments – Simple dependency installation.

pyproject.toml – The Modern Standard

• Introduced in PEP 518 (2018) as part of modernizing Python packaging.
• Designed to replace setup.py, requirements.txt, and setup.cfg for package metadata and dependencies.
• Supported by modern tools like Poetry, Hatch, PDM, uv.

Pros of pyproject.toml

• ✅ Standardized format – Defines dependencies, build system, and metadata in one place.
• ✅ Supports optional dependencies – dev, test, docs, etc.
• ✅ Compatible with modern tools – uv, poetry, hatch, pdm, etc.
• ✅ Cleaner and more structured – Uses TOML (more human-readable than text files).

Cons of pyproject.toml

• ❌ Not natively supported by pip – You need a tool like uv, poetry, or hatch to install from pyproject.toml.
• ❌ More complex syntax – Uses TOML instead of plain text.
• ❌ Adoption is still growing – Some projects still use requirements.txt due to legacy compatibility.

Use Case for pyproject.toml

• Modern Python package management – Especially when using uv, Poetry, or Hatch.
• Open-source Python libraries – Provides a standardized package structure.
• Full-fledged projects – Where multiple dependency groups (e.g., dev, test, docs) are needed.

Can requirements.txt and pyproject.toml Coexist? 🤝

Yes! Some projects use both:

1. Keep requirements.txt for production & Docker, but use pyproject.toml for development:

   pip install -r requirements.txt  # Production install
   uv pip install -e .[dev]         # Development install

2. Generate requirements.txt from pyproject.toml:

   uv pip compile -o requirements.txt

Comparing uv vs. Poetry vs. Pip

uv is ideal for fast, lightweight dependency management without lockfiles, whereas Poetry is better for projects that require strict version pinning.

uv

uv is a fast Python package manager that serves as an alternative to pip, pip-tools, and Poetry. It is designed for speed and simplicity while being compatible with pyproject.toml.

Create a New Package

uv init mypackage

This will generate a project structure like:

mypackage/
│── src/
│   ├── mypackage/
│   │   ├── __init__.py
│── tests/
│   ├── test_mypackage.py
│── pyproject.toml
│── README.md
│── .gitignore

Basic pyproject.toml

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[project]
name = "mypackage"
version = "0.1.0"
description = "A modern Python package using uv"
authors = [{ name = "Your Name", email = "your.email@example.com" }]
dependencies = [
    "requests",  # Example dependency
]

[project.optional-dependencies]
dev = [
    "pytest",
    "black",
    "ruff",
    "mypy",
]

[tool.hatch.metadata]
allow-direct-references = true

[tool.black]
line-length = 88

[tool.ruff]
line-length = 88

[tool.mypy]
strict = true

Install Dependencies

To install all dependencies defined in pyproject.toml:

uv pip install -e .[dev]

This is equivalent to pip install -e .[dev], but faster and more efficient.

If you need to sync dependencies (like pip-sync in pip-tools):

uv venv sync

To install a new package:

uv pip install numpy

To install a package and update pyproject.toml:

uv pip install numpy --editable

Writing Code

Inside src/mypackage/, create a module:

# src/mypackage/module.py
def hello():
    return "Hello, uv!"

And in src/mypackage/__init__.py:

from .module import hello

Running Tests

Inside tests/, create test_module.py:

import mypackage

def test_hello():
    assert mypackage.hello() == "Hello, uv!"

Run the tests with:

uv venv exec pytest

Or simply:

pytest

Formatting & Linting

To format code with black:

uv venv exec black .

To check for linting errors with ruff:

uv venv exec ruff .

To run mypy:

uv venv exec mypy src

Building and Publishing the Package

To build your package:

uv venv exec python -m build

To publish to PyPI:

uv venv exec twine upload dist/*

9. Using Your Package Locally

To install the package in development mode:

uv pip install -e .

To run Python with the package available:

uv venv exec python

Now, you can import your package:

import mypackage
print(mypackage.hello())

uv as a global python environment

Installing a Global Python Version with uv

To install a specific Python version globally, you can use the --preview flag, which installs the Python executable into your ~/.local/bin directory:

uv python install 3.12 --preview

This command installs Python 3.12 and places the executable in ~/.local/bin.

To make the newly installed Python version the default, you can add ~/.local/bin to your PATH environment variable:

export PATH="$HOME/.local/bin:$PATH"

Using .python-version Files for Directory-Specific Python Versions

For project-specific Python versions, uv respects .python-version files. To set a Python version for a specific project directory:

uv python pin 3.12

This command creates a .python-version file in the current directory, specifying that uv should use Python 3.12 when operating within this directory.

Advanced uv Workflows for Python Package Management 🚀

Unlike pip, uv optimizes dependency resolution and installation without a lockfile. However, you can still manage dependency updates efficiently.

Updating Dependencies Efficiently

uv pip sync

To upgrade a specific package and ensure pyproject.toml is updated:

uv pip install --editable requests@latest

To pin a package to a specific version:

uv pip install --editable numpy==1.24.0

Efficient Virtual Environment Management

uv handles virtual environments efficiently with built-in isolation.

Create and Activate a Virtual Environment

To explicitly create a new virtual environment:

uv venv create

To activate the environment manually:

uv venv exec bash  # macOS/Linux
uv venv exec cmd   # Windows

Alternatively, run commands inside the uv virtual environment without activation:

uv venv exec python -m mypackage
uv venv exec pytest

Rebuild the Virtual Environment

If your dependencies change, rebuild the virtual environment:

uv venv sync --rebuild

This ensures a clean slate.

Remove the Virtual Environment

To remove all installed dependencies:

uv venv destroy

Environment-Specific Dependencies

Sometimes, you need different dependencies for different environments (e.g., production vs. development).

Installing Only Production Dependencies

By default, uv pip install installs all dependencies, including optional ones. To install only production dependencies:

uv pip install --without dev

Using Multiple Dependency Groups

You can define separate dependency groups in pyproject.toml:

[project.optional-dependencies]
dev = ["pytest", "black", "ruff", "mypy"]
ci = ["tox", "pre-commit"]
docs = ["mkdocs", "mkdocs-material"]

Then install a specific group:

uv pip install --with dev
uv pip install --with docs

Or install multiple groups:

uv pip install --with dev --with docs

Running Scripts Inside the Virtual Environment

Instead of manually activating your virtual environment, use:

uv venv exec python script.py

For example, to run your main package entry point:

uv venv exec python -m mypackage

Running Tests Faster

Instead of manually invoking pytest, use:

uv venv exec pytest

This ensures pytest runs inside the managed virtual environment.

Running Type Checking

uv venv exec mypy src

Using uv for Continuous Integration (CI/CD)

uv works seamlessly with CI/CD pipelines. Here’s how to set it up for GitHub Actions:

GitHub Actions Workflow

Create .github/workflows/ci.yml:

name: CI

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Install uv
        run: pip install uv

      - name: Install Dependencies
        run: uv pip install --with dev

      - name: Run Tests
        run: uv venv exec pytest

Using uv in Docker

If you want to use uv inside a Docker container, you can create a Dockerfile:

FROM python:3.11

# Install uv
RUN pip install uv

# Set the working directory
WORKDIR /app

# Copy project files
COPY . .

# Install dependencies
RUN uv pip install --with dev

# Run the application
CMD ["uv", "venv", "exec", "python", "-m", "mypackage"]

Build and run the container:

docker build -t mypackage .
docker run --rm mypackage

Dependency-Free Builds

To package your project without installing dependencies, you can use:

uv venv exec python -m build

Or, if using hatch:

uv venv exec hatch build

Reproducible Installs with uv

Unlike pip, uv avoids dependency conflicts without needing a requirements.txt. However, if you still need a fully locked dependency list:

uv pip freeze > requirements.txt

To install from this file:

uv pip install -r requirements.txt

However, it's recommended to use uv's native approach:

uv pip install --without dev

uv Cheat Sheet

Poetry

Poetry is a modern dependency management and packaging tool that simplifies handling dependencies, building, and publishing Python packages.

Create a New Package

poetry new mypackage
cd mypackage

This automatically generates a structured project:

mypackage/
│── mypackage/
│   ├── __init__.py
│── tests/
│   ├── test_mypackage.py
│── pyproject.toml
│── README.rst
│── .gitignore

Example pyproject.toml

[tool.poetry]
name = "mypackage"
version = "0.1.0"
description = "A modern Python package using Poetry"
authors = ["Your Name <your.email@example.com>"]
license = "MIT"
readme = "README.md"
packages = [{ include = "mypackage", from = "src" }]

[tool.poetry.dependencies]
python = "^3.8"
requests = "^2.28.1"  # Example dependency

[tool.poetry.group.dev.dependencies]
pytest = "^7.2.0"
black = "^23.1.0"
ruff = "^0.0.287"
mypy = "^1.0.0"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

• [tool.poetry] → Defines package metadata.
• [tool.poetry.dependencies] → Lists runtime dependencies.
• [tool.poetry.group.dev.dependencies] → Lists development dependencies (formatting, linting, testing).
• [build-system] → Specifies how the package is built.

Install Dependencies

To install the dependencies defined in pyproject.toml:

poetry install

This also creates a virtual environment for your project.

To enter the environment:

poetry shell

Add Dependencies

To add a new runtime dependency:

poetry add numpy

For development dependencies:

poetry add --group dev pytest

Writing Code

Inside mypackage/, create a module:

# mypackage/module.py
def hello():
    return "Hello, Poetry!"

And in mypackage/__init__.py:

from .module import hello

Running Tests

Inside tests/, create test_module.py:

import mypackage

def test_hello():
    assert mypackage.hello() == "Hello, Poetry!"

Run the tests:

poetry run pytest

Formatting & Linting

To format code with black:

poetry run black .

To check for linting errors with ruff:

poetry run ruff .

Building and Publishing the Package

To build your package:

poetry build

This creates a dist/ folder with the package distribution files.

To publish to PyPI, first set up your credentials:

poetry config pypi-token.pypi YOUR_TOKEN

Then, publish:

poetry publish

Using Your Package Locally

To install the package in development mode:

poetry install

To run Python with the package available:

poetry run python

Now, you can import your package:

import mypackage
print(mypackage.hello())