Python notes

ch13.interface_protocols_abc

Ch13. Interfaces, Protocols, and ABCs 

graph LR;

A["Four Approaches to Typing"]

A --> AA["Duck Typing\ntraditional python\ntype defined by what method it provides\ninterface isn't enforced"]
A --> AB["Goose Typing\nPython 2.6 through ABC\nEnforce runtime check\nEnsure object conforms to an expected interface"]
A --> AC["Static Typing\nPython 3.5\ntyping module"]
A --> AD["Static Duck Typing\nPython 3.8\ntyping.Protocol"]

graph LR;
B["Protocol in Python's context?"] --> BA["Object Protocol\nA set of methods that\nan object must provide \nto fulfill a role"]
C["Protocol Type"]

C --> CA["Dynamic Protocols\ninformal, implicit interfaces\ndefined by convention and\ndescribed in documentation."]
C --> CB["Static Protocols\ndefined explicitly\nusing typing.Protocol subclasses."]

graph LR;
D["Monkey Patching\nDynamically modifying\na module, class, or function \nat runtime to add features or fix bugs\nwithout altering the original source code."]


E["EAFP\nEasier to Ask Forgiveness\nthan Permission"] --> EA["attempt the operation directly\nhandling any exceptions that arise."]

graph LR;
F["Virtual Subclasses\nusing the register method on the ABC"] --> FA["No inheritance needed"]
F --> FB["Pass issubclass and\nisinstance checks"]
F --> FC["Often used to register built-in or\nexternal classes as \nvirtual subclasses of ABCs."]

graph LR;
G["Nominal Typing\nInvolves explicitly linking\na class to an ABC through\ninheritance or registration"] <--> GA["Structural Typing\nDetermines the type\nbased on its structure\nspecifically the methods and\nattributes it implements\n\nnot through\ninheritance hierarchy"]
GA --> GAA["Achieved by static protocols\nIn Python 3.8\ntyping.Protocol"]
GA --> GAB["@runtime_checkable decorator\nmade runtime checkable"]

Programming to an Interface

  • Object-Oriented Programming (OOP) and Interfaces:
    • The core principle of OOP is to "program to an interface, not an implementation," as emphasized by Gamma, Helm, Johnson, and Vlissides.
    • In Python, understanding a type is rooted in knowing the methods it provides — its interface.

Four Approaches to Typing in Python:

  • Python supports multiple ways to define and use interfaces, especially since Python 3.8:
  1. Duck Typing:
    • The traditional Python approach where an object's suitability is determined by the methods it supports, rather than its class inheritance.
    • Example: "If it looks like a duck and quacks like a duck, it's a duck."
  2. Goose Typing:
    • Introduced in Python 2.6 through Abstract Base Classes (ABCs).
    • Provides a way to enforce type checking at runtime by comparing objects against ABCs.
    • Focuses on ensuring that an object conforms to an expected interface.
  3. Static Typing:
    • Traditional typing method seen in languages like C and Java, now supported in Python via the typing module (since Python 3.5).
    • Relies on external tools like MyPy to enforce type hints and ensure code correctness before runtime.
  4. Static Duck Typing:
    • Inspired by Go, this approach uses subclasses of typing.Protocol (introduced in Python 3.8) to define interfaces that can be checked statically.
    • Combines the flexibility of duck typing with the rigor of static type checking.

Two Kinds of Protocols in Python

  • Understanding Protocols:

    • The term "protocol" in computer science varies by context.
    • In Python, it refers to a set of methods that an object must provide to fulfill a role, commonly known as an "object protocol."

  • Dynamic Protocols (Informal Interfaces):

    • Definition:

      • These are informal, implicit interfaces defined by convention and described in documentation.
      • Python's dynamic protocols are a key part of its flexibility, allowing objects to behave in certain ways by implementing specific methods, even if only partially.

    • Example: The sequence protocol is a common example. By implementing __getitem__, a class can behave like a sequence, allowing indexing, iteration, and the in operator.

      class Vowels:
    def __getitem__(self, i):
        return 'AEIOU'[i]

v = Vowels()
print(v[0])  # Output: 'A'
print(v[-1])  # Output: 'U'
for c in v: print(c)  # Iterates over 'A', 'E', 'I', 'O', 'U'
print('E' in v)  # Output: True
print('Z' in v)  # Output: False

    • Partial Implementation:

      • A class can implement only part of a protocol (e.g., just __getitem__ without __len__), and still function in some contexts.

    • Sequence Protocol:

      • In the Python/C API, the PySequence_Check function returns 1 if an object has a __getitem__ method (unless it's a subclass of dict), confirming the sequence protocol's dynamic nature.

  • Static Protocols (Explicit Interfaces):

    • Definition:
      • Introduced in Python 3.8 with PEP 544, static protocols are defined explicitly using typing.Protocol subclasses.
      • Static protocols enforce the implementation of all methods specified in the protocol class, ensuring full compliance.
    • Example:
      • A static protocol defines a set of methods that a class must implement to satisfy static type checkers like MyPy.
      • Strict Compliance:
        • Unlike dynamic protocols, static protocols require every method in the protocol to be implemented, even if some methods are not used in the program.
      • Static Type Checking:
        • Static protocols are verified by static type checkers, providing a level of assurance not available with dynamic protocols.

  • Key Differences:

    • Dynamic Protocols:
      • Implicit and informal.
      • Partial implementation is often acceptable.
      • Cannot be verified by static type checkers.
    • Static Protocols:
      • Explicitly defined with typing.Protocol.
      • Requires full implementation of all methods in the protocol.
      • Can be verified by static type checkers.

Programming Ducks

  • Python and Sequences:
    • Python's data model is designed to cooperate with dynamic protocols, especially for sequence-like objects.
    • The interpreter makes every effort to work with even the most basic sequence implementations.

Python Digs Sequences

  • Formal Sequence Interface:

    • The Sequence interface is formalized as an Abstract Base Class (ABC) in the collections.abc module, but Python's interpreter and built-in sequence types (e.g., list, str) do not rely on this ABC.
    • The Sequence ABC is a guide to what a complete sequence type should support.

  • Core Methods:

    • A subclass of Sequence must implement:
      • __getitem__: Allows item access by index.
      • __len__ (inherited from Sized): Allows the use of len() on the sequence.
    • Other methods in Sequence are concrete and can be inherited or overridden.

  • Dynamic Sequence Behavior:

    • Fallback Mechanisms:

      • Python makes sequence-like objects work even with minimal implementations.
      • If a class has __getitem__ but lacks __iter__ and __contains__, Python:
        • Iterates over the object by calling __getitem__ with increasing integer indices starting from 0.
        • Uses a sequential scan to implement the in operator, checking for item membership.
      • Example with Vowels:
        • The Vowels class from the previous example only implements __getitem__, but Python still supports iteration and in operations on it.

    • Full Sequence Protocol: The FrenchDeck class implements both __getitem__ and __len__, making it a proper sequence without subclassing abc.Sequence

    import collections
Card = collections.namedtuple('Card', ['rank', 'suit'])

class FrenchDeck:
    ranks = [str(n) for n in range(2, 11)] + list('JQKA')
    suits = 'spades diamonds clubs hearts'.split()

    def __init__(self):
        self._cards = [Card(rank, suit) for suit in self.suits for rank in self.ranks]

    def __len__(self):
        return len(self._cards)

    def __getitem__(self, position):
        return self._cards[position]

  • Python's Iteration Mechanism:

    • Python's ability to iterate over objects without relying on the Sequence ABC demonstrates extreme duck typing.
    • The interpreter directly implements these behaviors in C, independent of the Sequence ABC.
    • Sequence ABC Methods:
      • The __iter__ and __contains__ methods in the Sequence ABC mimic the interpreter's built-in behavior but are not relied upon by the interpreter itself.

Monkey Patching: Implementing a Protocol at Runtime

  • Monkey Patching: Dynamically modifying a module, class, or function at runtime to add features or fix bugs without altering the original source code.

    • However, it tightly couples the patching code to the internals of the program, often involving private and undocumented attributes.

  • Use Case Example: Shuffling a FrenchDeck Instance:

    • The FrenchDeck class is designed to behave like a sequence but lacks the ability to be shuffled using Python's built-in random.shuffle() function.

  • Standard random.shuffle Method:

    • Purpose: Shuffles a sequence in place.

    • Example:

      from random import shuffle
l = list(range(10))
shuffle(l)
print(l)  # Output: [5, 2, 9, 7, 8, 3, 1, 4, 0, 6]  # Example shuffled list

  • Problem:

    • When attempting to shuffle a FrenchDeck instance, a TypeError occurs:

      from random import shuffle
from frenchdeck import FrenchDeck

deck = FrenchDeck()
shuffle(deck)
# Output: TypeError: 'FrenchDeck' object does not support item assignment

    • Reason: FrenchDeck implements the immutable sequence protocol (supports __getitem__), but random.shuffle requires the mutable sequence protocol (which includes __setitem__).

  • Monkey Patching to Add __setitem__:

    • Solution: Dynamically add a __setitem__ method to the FrenchDeck class, enabling it to support item assignment and thus be compatible with random.shuffle.

      1. Define a set_card Function to allow item assignment within the deck.
      def set_card(deck, position, card):
    deck._cards[position] = card
      1. Monkey Patch the FrenchDeck Class: Assign the set_card function to the __setitem__ attribute of the FrenchDeck class.
      FrenchDeck.__setitem__ = set_card
      1. Shuffle the Deck: With __setitem__ added, random.shuffle can now shuffle the FrenchDeck instance.
      shuffle(deck)
print(deck[:5])  # Output: [Card(rank='3', suit='hearts'), Card(rank='4', suit='diamonds'), ...]

  • Signature of __setitem__:

    • The __setitem__ method is defined with three parameters: self, key, and value. In the set_card function, these are represented by deck, position, and card.
    • Naming the first argument self is conventional, but any name can be used.

  • How It Works:

    • The set_card function interacts with the private _cards attribute of FrenchDeck, which must be a mutable sequence (e.g., a list). This allows the deck to be shuffled.

Defensive Programming and “Fail Fast” in Python

  • Defensive Programming:

    • Similar to defensive driving, it involves practices to make code more robust and resistant to errors, particularly when faced with unexpected inputs or actions by other developers.
    • Emphasizes writing code that anticipates and handles potential problems gracefully, often by checking conditions and validating inputs.

  • Fail Fast Principle:

    • In dynamically typed languages like Python, "fail fast" means raising errors as soon as something goes wrong, rather than allowing the program to continue and potentially fail in a more obscure way later.
    • The goal is to detect and address problems early, making them easier to identify and fix.

  • Example of Fail Fast:

    • When a function or method expects a sequence, don’t enforce that it must be a list. Instead, convert the input to a list immediately:

      def __init__(self, iterable):
    self._balls = list(iterable)

    • Benefits:

      • Flexibility: This approach accepts any iterable (e.g., tuples, sets, generators) that can be converted into a list.
      • Failing Fast: If the argument is not iterable, a TypeError is raised immediately, making the error easy to trace back to its source.

  • Considerations for Fail Fast:

    • Avoiding Unnecessary Copies:

      • If the data should not be copied (e.g., due to size or mutability requirements), converting to a list may not be appropriate. In such cases: Use isinstance(x, abc.MutableSequence) to check if the input is a mutable sequence.
      if not isinstance(x, abc.MutableSequence):
    raise TypeError("Expected a mutable sequence")

    • Handling Infinite Generators:

      • To prevent handling infinite generators, call len() on the argument first. This ensures the object supports the sequence protocol without consuming the generator:
      if len(x) < 0:
    raise ValueError("Expected a finite sequence")

    • Accepting Any Iterable:

      • If the function should accept any iterable, use iter(x) to obtain an iterator as soon as possible. If x is not iterable, this will raise a TypeError immediately:
      x = iter(x)  # Fails fast if x is not iterable

  • Defensive Duck Typing:

    • Duck typing in Python allows handling different types without relying on explicit type checks like isinstance() or hasattr().
    • Consider the field_names argument in collections.namedtuple, which can accept a single string of identifiers or a sequence of identifiers:
    try:
    field_names = field_names.replace(',', ' ').split()
except AttributeError:
    pass
field_names = tuple(field_names)
if not all(s.isidentifier() for s in field_names):
    raise ValueError('field_names must all be valid identifiers')
    • EAFP (Easier to Ask Forgiveness than Permission): Assume field_names is a string and try to replace commas with spaces and split it into a list. If this fails, it's not a string, so it’s assumed to be an iterable.
    • Fail Fast: Converts field_names into a tuple and checks if all items are valid identifiers. If not, it raises a ValueError immediately.

  • Comparison with Static Typing:

    • Limitations of Type Hints:

      • Type hints can catch some issues early, but they can't handle all dynamic behaviors. For example, there’s no type hint that can specify "a string of identifiers separated by spaces or commas."

      • Example of a type hint:

        from typing import Union, Iterable

def namedtuple(
    typename: str,
    field_names: Union[str, Iterable[str]],
    ...

      • While the type hint suggests that field_names can be a string or an iterable, it doesn’t catch all edge cases that duck typing can handle dynamically.

Goose Typing

  • Goose Typing: A method of runtime type checking in Python that utilizes Abstract Base Classes (ABCs) to define and enforce interfaces.

    • Unlike duck typing, which focuses on an object’s behavior regardless of its type, goose typing explicitly checks if an object is a subclass of a given ABC, making type expectations clear and enforceable.

  • Python doesn’t have an interface keyword like some other languages. Instead, it uses Abstract Base Classes (ABCs) to define interfaces for both runtime and static type checking.

  • ABCs complement duck typing by providing a formal way to define and enforce interfaces, especially when duck typing might be too ambiguous or error-prone.

  • Waterfowl Analogy:

    • Duck Typing: Focuses on observable traits (e.g., an object can be used like a duck if it behaves like a duck).
    • Goose Typing: Similar to the biological approach of cladistics, where the focus is on inherited traits from a common ancestor rather than just behavior. This approach in programming ensures that objects not only have the required methods but are also part of a recognized hierarchy (ABCs).

  • Example Scenario:

    • Consider a scenario with classes Artist, Gunslinger, and Lottery, all of which have a draw() method. Despite the same method name, these classes likely serve different purposes. Goose typing helps ensure that such methods align with the expected behavior by checking against an ABC.

  • Using ABCs:

    • Subclassing ABCs:

      • You can create classes that inherit from an ABC to explicitly declare that they implement a certain interface.
      from abc import ABC, abstractmethod

class Drawable(ABC):
    @abstractmethod
    def draw(self):
        pass

class Artist(Drawable):
    def draw(self):
        print("Drawing a picture")
      • Artist must implement the draw method, as required by the Drawable ABC.

    • Virtual Subclasses:

      • Classes can be registered as virtual subclasses of an ABC even if they don’t inherit from it. This allows them to be recognized by isinstance and issubclass checks as if they were actual subclasses.
      from collections.abc import Sequence

class FrenchDeck:
    def __len__(self):
        return 52

    def __getitem__(self, position):
        return Card(rank='A', suit='hearts')

Sequence.register(FrenchDeck)
      • FrenchDeck is now recognized as a Sequence by isinstance and issubclass checks.

    • Built-in ABCs:

      • Python’s standard library provides many useful ABCs in modules like collections.abc and numbers.

      • For example, abc.Sized checks if a class implements the __len__ method:

        from collections import abc

class Struggle:
    def __len__(self):
        return 23

print(isinstance(Struggle(), abc.Sized))  # Output: True

  • Advantages of Goose Typing:

    • Explicit Declaration: Subclassing from an ABC makes it explicit that a class is intended to implement a particular interface, making the developer’s intent clear.
    • Runtime Type Checking: Using isinstance and issubclass with ABCs provides a more flexible and safe way to enforce type constraints, as ABCs can be extended with virtual subclass registration.
    • Polymorphism: By using ABCs, type checks remain flexible and support polymorphism, a key feature of object-oriented programming.

  • Cautions:

    • Excessive Use of ABCs: While useful, overusing ABCs can lead to unnecessary complexity and rigidity, which runs counter to Python’s pragmatic and flexible nature.
    • Avoid Overcomplicating with Custom ABCs: It’s generally better to use existing ABCs rather than creating custom ones, which can complicate the codebase.

Subclassing an ABC: Implementing FrenchDeck2 as a Mutable Sequence

  • Overview:

    • Subclassing an existing Abstract Base Class (ABC) from Python’s standard library is recommended before creating custom ABCs. In this case, FrenchDeck2 is implemented as a subclass of collections.MutableSequence, making it an explicit mutable sequence with all necessary behaviors.

  • Implementation of FrenchDeck2:

    • Key Components:

      • The FrenchDeck2 class inherits from abc.MutableSequence, which requires the implementation of several methods to satisfy the mutable sequence protocol.
      from collections import namedtuple, abc

Card = namedtuple('Card', ['rank', 'suit'])

class FrenchDeck2(abc.MutableSequence):
    ranks = [str(n) for n in range(2, 11)] + list('JQKA')
    suits = 'spades diamonds clubs hearts'.split()

    def __init__(self):
        self._cards = [Card(rank, suit) for suit in self.suits for rank in self.ranks]

    def __len__(self):
        return len(self._cards)

    def __getitem__(self, position):
        return self._cards[position]

    def __setitem__(self, position, value):
        self._cards[position] = value

    def __delitem__(self, position):
        del self._cards[position]

    def insert(self, position, value):
        self._cards.insert(position, value)

  • Required Methods:

    • __len__: Returns the number of items in the deck.
    • __getitem__: Retrieves a card at a given position.
    • __setitem__: Allows assigning a card to a specific position, enabling mutability.
    • __delitem__: Deletes a card from a specific position, as required by mutable sequences.
    • insert: Inserts a card at a specified position within the deck.

  • Benefits of Subclassing MutableSequence: Inherited Methods:

    • From Sequence ABC:
      • __contains__: Checks if a card is in the deck.
      • __iter__: Allows iteration over the deck.
      • __reversed__: Supports reversed iteration.
      • index: Finds the position of a card.
      • count: Counts occurrences of a card.
    • From MutableSequence ABC:
      • append: Adds a card to the end of the deck.
      • reverse: Reverses the order of cards in the deck.
      • extend: Extends the deck with cards from another iterable.
      • pop: Removes and returns the last card.
      • remove: Removes the first occurrence of a specified card.
      • __iadd__: Supports in-place concatenation using +=.

  • Customization:

    • The inherited methods from ABCs are implemented in a generic way that works with any sequence. However, subclassers can override these methods with more efficient versions suited to the specific data structure.
    • Example: __contains__
      • The default implementation performs a sequential scan, which could be inefficient for large sequences. If the deck is sorted, overriding __contains__ to use binary search (via bisect) can significantly improve performance.

ABCs in the Python Standard Library

  • Introduction to ABCs:
    • Since Python 2.6, the standard library has included several Abstract Base Classes (ABCs) to define interfaces for various types of objects.
    • ABCs are primarily located in the collections.abc module, but others can be found in the io and numbers packages.
    • ABCs provide a way to enforce type constraints and establish a common interface across different implementations.
    • The goal of understanding and using ABCs is not to create new ones unnecessarily, but to be able to read and understand the ABCs in the standard library and other packages. This understanding helps in writing more robust, maintainable, and Pythonic code by leveraging established interfaces and behaviors.

  • Key Modules:

    • collections.abc:
      • Contains most of the widely used ABCs for collections, such as Iterable, Container, and Sized.
    • abc Module:
      • This module defines the ABC class, from which all ABCs inherit. It is typically used when creating custom ABCs.
      • Note: The collections.abc module is implemented in _collections_abc.py and is imported separately from the collections package for performance reasons.

  • Overview of Important ABCs:

    • Basic Collection ABCs:

      • Iterable:
        • Supports iteration using __iter__.
      • Container:
        • Supports membership testing using __contains__ (used with the in operator).
      • Sized:
        • Supports size queries using __len__.
      • Collection:
        • Added in Python 3.6, this ABC has no methods of its own but makes it easier to create classes that combine Iterable, Container, and Sized.

    • Immutable Collection Types:

      • Sequence:
        • Represents an immutable sequence, such as tuple or str.
      • Mapping:
        • Represents an immutable mapping, such as dict.
      • Set:
        • Represents an immutable set of unique items.

    • Mutable Collection Types:

      • Each immutable collection type has a corresponding mutable subclass:
        • MutableSequence (e.g., list)
        • MutableMapping (e.g., dict)
        • MutableSet (e.g., set)

    • MappingView ABCs:

      • These ABCs are used for the objects returned by dictionary methods like .items(), .keys(), and .values():
        • ItemsView
        • KeysView
        • ValuesView
      • ItemsView and KeysView also implement the Set interface, allowing set operations on dictionary views.

    • Other Important ABCs:

      • Iterator:
        • Subclass of Iterable, representing objects that can be iterated over once.
      • Callable:
        • Represents objects that can be called like a function (supports __call__).
        • More convenient to check using callable(obj) than isinstance(obj, Callable).
      • Hashable:
        • Represents objects that can be hashed (supports __hash__).
        • isinstance(obj, Hashable) can be misleading; it's more reliable to use hash(obj) and catch a TypeError if the object isn't hashable.

  • Common Pitfalls with ABCs:

    • isinstance with Hashable:

      • May return True for a tuple that contains unhashable elements, even though the tuple itself is not hashable.
      • Better Approach: Call hash(obj) to check hashability.

    • isinstance with Iterable:

      • May return False even if the object is iterable through the __getitem__ method using 0-based indices.
      • Better Approach: Call iter(obj) to reliably check if an object is iterable.

  • Practical Use of ABCs:

    • Subclassing and Implementing ABCs:

      • When creating new collection types or other data structures, it's often beneficial to subclass existing ABCs like MutableSequence, Mapping, or Set.
      • This ensures your class implements the expected methods and behaviors.

    • Goose Typing:

      • Use isinstance(obj, SomeABC) checks to enforce type constraints and ensure that objects adhere to the expected interface.
      • Registration of virtual subclasses allows flexibility in recognizing classes that implement the necessary interface without direct inheritance from the ABC.

Defining and Using an ABC in Python: The Tombola Example

  • Imagine building an ad management framework called ADAM, where you need to display ads in a random order without repetition until all ads have been shown. To support custom non-repeating random-picking classes, you can define an ABC.

  • The Tombola ABC represents a general-purpose interface for such random-picking containers, akin to bingo cages or lottery blowers.

  • Defining the Tombola ABC:

    import abc

class Tombola(abc.ABC):
    @abc.abstractmethod
    def load(self, iterable):
        """Add items from an iterable."""

    @abc.abstractmethod
    def pick(self):
        """Remove item at random, returning it.
        This method should raise `LookupError` when the instance is empty.
        """

    def loaded(self):
        """Return `True` if there's at least 1 item, `False` otherwise."""
        return bool(self.inspect())

    def inspect(self):
        """Return a sorted tuple with the items currently inside."""
        items = []
        while True:
            try:
                items.append(self.pick())
            except LookupError:
                break
        self.load(items)
        return tuple(sorted(items))

  • Key Features of the Tombola ABC:

    • Abstract Methods:

      • load(self, iterable):
        • Adds items from an iterable to the container.
        • Must be implemented by any subclass.
      • pick(self):
        • Removes and returns a random item from the container.
        • Should raise LookupError if the container is empty.

    • Concrete Methods:

      • loaded(self):
        • Returns True if the container has at least one item; otherwise, False.
        • Uses the inspect() method to check if the container is empty.
      • inspect(self):
        • Returns a sorted tuple of items in the container without modifying the container’s state.
        • Uses the pick() method to temporarily empty the container, and then reloads it to restore its original state.

  • Concrete Methods in ABCs:

    • Concrete methods in an ABC must rely only on the abstract methods or other concrete methods defined within the same ABC.
    • This ensures that any subclass implementing the abstract methods will automatically inherit the correct behavior for the concrete methods.

  • Creating a Subclass of Tombola:

    • When subclassing Tombola, you must implement the load and pick methods. Failure to do so will prevent the subclass from being instantiated.

  • Example of a Defective Subclass:

    • If you try to create a subclass of Tombola without implementing all required abstract methods, Python will raise a TypeError.

    • Example:

      from tombola import Tombola

class Fake(Tombola):
    def pick(self):
        return 13

f = Fake()  # Raises TypeError: Can't instantiate abstract class Fake with abstract method load

    • Explanation:

      • The Fake class fails to implement the load method, which is required by the Tombola ABC. As a result, Python prevents the instantiation of Fake, ensuring that the contract defined by the ABC is respected.

  • Why Use ABCs?:

    • Enforcing Interfaces: ABCs ensure that all subclasses adhere to a defined interface, making your code more predictable and easier to maintain.
    • Flexibility in Design: By defining methods in an ABC, you allow for flexibility in how those methods are implemented, while still ensuring that they exist.
    • Improved Type Checking: ABCs can be used in conjunction with type hints to improve static type checking, providing clearer contracts for function arguments and return types.

  • Side notes: exceptions

BaseException
├── SystemExit
├── KeyboardInterrupt
├── GeneratorExit
└── Exception
    ├── StopIteration
    ├── ArithmeticError
    │   ├── FloatingPointError
    │   ├── OverflowError
    │   └── ZeroDivisionError
    ├── AssertionError
    ├── AttributeError
    ├── BufferError
    ├── EOFError
    ├── ImportError
    ├── LookupError
    │   ├── IndexError
    │   └── KeyError
    ├── MemoryError
    ├── NameError
    │   └── UnboundLocalError
    ├── OSError
    │   ├── BlockingIOError
    │   ├── ChildProcessError
    │   ├── ConnectionError
    │   │   ├── BrokenPipeError
    │   │   ├── ConnectionAbortedError
    │   │   ├── ConnectionRefusedError
    │   │   └── ConnectionResetError
    │   ├── FileExistsError
    │   ├── FileNotFoundError
    │   ├── InterruptedError
    │   ├── IsADirectoryError
    │   ├── NotADirectoryError
    │   ├── PermissionError
    │   ├── ProcessLookupError
    │   └── TimeoutError
    ├── ReferenceError
    ├── RuntimeError
    │   ├── NotImplementedError
    │   └── RecursionError
    ├── SyntaxError
    │   └── IndentationError
    │       └── TabError
    ├── SystemError
    ├── TypeError
    ├── ValueError
    │   └── UnicodeError
    │       ├── UnicodeDecodeError
    │       ├── UnicodeEncodeError
    │       └── UnicodeTranslateError
    ├── Warning
        ├── DeprecationWarning
        ├── PendingDeprecationWarning
        ├── RuntimeWarning
        ├── SyntaxWarning
        ├── UserWarning
        ├── FutureWarning
        ├── ImportWarning
        ├── UnicodeWarning
        └── BytesWarning

ABC Syntax Details and Implementing Concrete Subclasses

  • Subclasses of abc.ABC:

    • The standard way to create an ABC in Python is to subclass abc.ABC or any other existing ABC.

  • Abstract Methods:

    • Abstract methods are methods that must be implemented by any subclass of the ABC.
    • These are defined using the @abstractmethod decorator.
    import abc

class MyABC(abc.ABC):
    @classmethod
    @abc.abstractmethod
    def an_abstract_classmethod(cls, ...):
        pass
    • Stacking Decorators:
      • When stacking decorators, the @abstractmethod decorator should be the innermost one, directly above the method definition.

  • Deprecated Decorators:

    • Python previously had decorators like @abstractclassmethod, @abstractstaticmethod, and @abstractproperty. These have been deprecated since Python 3.3, as the same effect can be achieved by stacking @abstractmethod with @classmethod, @staticmethod, or @property.

Subclassing an ABC: Concrete Implementations

Given the Tombola ABC (which represents a non-repeating random picker), here are two concrete subclasses that implement its required methods.

  1. BingoCage Class:

    • Purpose: This class uses a cryptographic randomizer to shuffle items before selecting them.

      import random
from tombola import Tombola

class BingoCage(Tombola):
    def __init__(self, items):
        self._randomizer = random.SystemRandom()
        self._items = []
        self.load(items)

    def load(self, items):
        self._items.extend(items)
        self._randomizer.shuffle(self._items)

    def pick(self):
        try:
            return self._items.pop()
        except IndexError:
            raise LookupError('pick from empty BingoCage')

    def __call__(self):
        return self.pick()

    • Details:

      • load(self, items): Adds items to the internal list and shuffles them using SystemRandom for cryptographic randomness.
      • pick(self): Removes and returns an item from the end of the shuffled list, raising a LookupError if the list is empty.
      • __call__(self): Allows the BingoCage instance to be callable, returning a picked item.

  2. LottoBlower Class:

    • Purpose: This class picks items by randomly selecting and removing them from a specific position in the list.

      import random
from tombola import Tombola

class LottoBlower(Tombola):
    def __init__(self, iterable):
        self._balls = list(iterable)

    def load(self, iterable):
        self._balls.extend(iterable)

    def pick(self):
        try:
            position = random.randrange(len(self._balls))
        except ValueError:
            raise LookupError('pick from empty LottoBlower')
        return self._balls.pop(position)

    def loaded(self):
        return bool(self._balls)

    def inspect(self):
        return tuple(self._balls)

    • Details:

      • __init__(self, iterable): Initializes the class by converting the iterable into a list.
      • pick(self): Selects and removes an item from a random position in the list. Raises a LookupError if the list is empty.
      • loaded(self): Overrides the loaded method to directly check if the internal list has items, improving efficiency.
      • inspect(self): Returns a tuple of the items remaining in the list, preserving their current order.

Key Points and Best Practices

  • Concrete Methods in ABCs:

    • Concrete methods in an ABC can rely only on the abstract methods or other concrete methods within the same ABC. This ensures that subclasses have all the necessary methods to support the inherited concrete methods.

  • Subclassing and Inheritance:

    • Concrete subclasses can override inherited methods to provide more efficient implementations, as seen with loaded() and inspect() in LottoBlower.

  • Practical Use of ABCs:

    • The use of ABCs, combined with concrete implementations like BingoCage and LottoBlower, ensures that different subclasses adhere to a consistent interface while allowing flexibility in how they fulfill the requirements of that interface.

Virtual Subclassing an ABC in Python

  • Virtual Subclasses Overview:

    • Goose Typing: In Python, "goose typing" refers to the ability to register a class as a virtual subclass of an Abstract Base Class (ABC) without actually inheriting from it. This allows the class to be recognized as a subclass by issubclass and isinstance checks, even though it doesn’t inherit any methods or attributes from the ABC.
    • Registration: You can register a class as a virtual subclass using the register method on the ABC. This method can be called directly or used as a decorator.
    • While virtual subclassing is powerful, it should be used judiciously. Ensure that the registered class fully implements the ABC’s interface to avoid runtime errors.

  • Key Characteristics:

    • No Inheritance: Virtual subclasses do not inherit from the ABC they are registered with. They are not checked for conformance to the ABC’s interface at the time of registration or instantiation.
    • Runtime Exceptions: If the virtual subclass does not actually implement the required interface, errors will be caught during runtime when the relevant methods are called.
    • Static Type Checkers: Current static type checkers, such as MyPy, do not support virtual subclasses.
    from random import randrange
from tombola import Tombola

@Tombola.register
class TomboList(list):
    def pick(self):
        if self:
            position = randrange(len(self))
            return self.pop(position)
        else:
            raise LookupError('pop from empty TomboList')

    load = list.extend

    def loaded(self):
        return bool(self)

    def inspect(self):
        return tuple(self)
    • Decorator Registration:
      • @Tombola.register registers TomboList as a virtual subclass of Tombola.
      • This means TomboList will pass issubclass(TomboList, Tombola) and isinstance(t, Tombola) checks, where t is an instance of TomboList.
    • Inheriting from list:
      • TomboList directly subclasses list and leverages its built-in methods.
    • pick(self) Method:
      • Implements the pick method required by Tombola. It removes a random item from the list.
    • load(self, iterable) Method:
      • Uses list.extend to implement the load method by directly extending the list with new items.
    • loaded(self) and inspect(self):
      • These methods check whether the list is non-empty and return its contents, respectively.

  • Checking the Virtual Subclass:

    • After registering TomboList as a virtual subclass, it behaves as if it were a subclass of Tombola:

      from tombola import Tombola
from tombolist import TomboList

print(issubclass(TomboList, Tombola))  # Output: True
t = TomboList(range(100))
print(isinstance(t, Tombola))  # Output: True

  • Understanding Method Resolution Order (MRO):

    • MRO: The Method Resolution Order (__mro__) of a class shows the hierarchy Python uses to search for methods.

    • In the case of TomboList, Tombola does not appear in its __mro__, confirming that TomboList does not inherit any methods from Tombola:

      print(TomboList.__mro__)
# Output: (<class 'tombolist.TomboList'>, <class 'list'>, <class 'object'>)

  • Usage of register in Practice:

    • Direct Registration:

      • The register method is often used to register built-in or external classes as virtual subclasses of ABCs.

      • Example:

        from collections.abc import Sequence

Sequence.register(tuple)
Sequence.register(str)
Sequence.register(range)
Sequence.register(memoryview)

      • These registrations occur in the _collections_abc.py module and are activated when the module is imported.

Structural Typing with ABCs

  • Nominal vs. Structural Typing:

    • Nominal Typing: Involves explicitly linking a class to an ABC through inheritance or registration. For example, a class Sub can inherit from AnABC, or it can be registered as a virtual subclass using AnABC.register(Sub).
    • Structural Typing: Determines the type of an object based on its structure, specifically the methods and attributes it implements, rather than its explicit inheritance hierarchy.

  • Structural Typing in ABCs:

    • Some Abstract Base Classes (ABCs) in Python support structural typing, allowing a class to be recognized as a subclass of an ABC if it implements certain methods, even if it doesn’t explicitly inherit from or register with that ABC.

    • Example:

      • The abc.Sized ABC considers any class that implements the __len__ method to be a virtual subclass, even if the class doesn’t inherit from Sized or register with it.
      class Struggle:
    def __len__(self): return 23
      • Struggle implements __len__, which is sufficient for it to be considered a subclass of abc.Sized.
      from collections import abc

print(isinstance(Struggle(), abc.Sized))  # Output: True
print(issubclass(Struggle, abc.Sized))    # Output: True
      • Both isinstance and issubclass return True because abc.Sized checks for the presence of the __len__ method using the __subclasshook__ mechanism.

  • How __subclasshook__ Works:

    • Definition of abc.Sized:

      • The __subclasshook__ method is defined in Sized to check if a class or any of its superclasses implements __len__.
      class Sized(metaclass=ABCMeta):
    __slots__ = ()

    @abstractmethod
    def __len__(self):
        return 0

    @classmethod
    def __subclasshook__(cls, C):
        if cls is Sized:
            if any("__len__" in B.__dict__ for B in C.__mro__):
                return True
        return NotImplemented
      • The __subclasshook__ method checks if __len__ is present in the __dict__ of any class in the method resolution order (__mro__) of the class being checked.
      • If __len__ is found, the method returns True, indicating that the class is a virtual subclass of Sized.

  • When to Use __subclasshook__:

    • Caution with Custom ABCs:
      • Implementing __subclasshook__ in your own ABCs can be risky because it assumes that any class with a specific method will behave as expected.
      • Example: A class might implement __len__, __getitem__, and __iter__, but that doesn’t necessarily mean it should be considered a Sequence. Thus, abc.Sequence does not implement __subclasshook__.

Static Protocols

  • Static protocols in Python, introduced in PEP 544, allow for structural typing (static duck typing) within Python’s type hinting system.

  • This enables you to specify that a function parameter must support a particular set of operations, rather than needing it to be of a specific type or subclass.

  • The double Function Example:

    • Dynamic Nature:

      • The double function demonstrates Python’s dynamic typing by working with various types (e.g., int, str, list, Fraction, custom objects) as long as the type supports multiplication (*).
      def double(x):
    return x * 2

print(double(1.5))  # Output: 3.0
print(double('A'))  # Output: 'AA'
print(double([10, 20, 30]))  # Output: [10, 20, 30, 10, 20, 30]

  • Challenges with Type Hints Before Static Protocols:

    • Before the introduction of static protocols, it was difficult to add type hints to such flexible functions without limiting their use to specific types.
    • This limitation arose because the type hint system initially relied on nominal typing, where types needed to be explicitly named.

  • Introducing typing.Protocol:

    • Purpose: typing.Protocol allows the creation of structural types, meaning you can define a type based on the presence of certain methods or attributes rather than its inheritance hierarchy.
    from typing import TypeVar, Protocol

T = TypeVar('T')

class Repeatable(Protocol):
    def __mul__(self: T, repeat_count: int) -> T: ...

RT = TypeVar('RT', bound=Repeatable)

def double(x: RT) -> RT:
    return x * 2
    • Repeatable Protocol:
      • This protocol defines a contract that any type implementing it must support multiplication (__mul__) with an int and return an instance of the same type.
    • Type Variables:
      • T: Used to ensure that the return type of __mul__ matches the type of self.
      • RT: A type variable bound by the Repeatable protocol, meaning RT must be a type that implements Repeatable.
    • Function double(x: RT) -> RT:
      • The type hint now tells the type checker that x must be of a type that implements the Repeatable protocol, ensuring it can be multiplied by 2 and return a result of the same type.

  • Advantages of Static Protocols:

    • Enhanced Flexibility: You can type-check functions like double without restricting them to specific types, as long as the type implements the required operations.
    • Future-Proofing: The function remains compatible with new types that implement the necessary operations, such as custom types introduced later in the codebase or by third-party libraries.
    • Static Duck Typing: Combines the dynamic flexibility of duck typing with the static safety of type checking, making it easier to write Pythonic code that is also type-safe.

  • Usage Example with Custom Types:

    • Consider a Vector class that supports multiplication:

      from vector_v7 import Vector

v = Vector([11.0, 12.0, 13.0])
print(double(v))  # Output: Vector([22.0, 24.0, 26.0])

    • The double function can work seamlessly with Vector instances because they conform to the Repeatable protocol.

Runtime Checkable Static Protocols in Python

  • While static protocols are primarily used for type checking during static analysis, they can be made runtime checkable by using the @runtime_checkable decorator. This allows them to be used with isinstance and issubclass checks at runtime.

    • The SupportsComplex protocol defines an interface for types that can be converted to a complex number. It requires the implementation of the __complex__ method.
    from typing import Protocol, runtime_checkable

@runtime_checkable
class SupportsComplex(Protocol):
    """An ABC with one abstract method __complex__."""
    def __complex__(self) -> complex:
        pass

  • Runtime Example:

    • You can use SupportsComplex to check if a type implements the __complex__ method, even if it does not inherit from SupportsComplex or a similar class.
    from typing import SupportsComplex
import numpy as np

c64 = np.complex64(3+4j)
print(isinstance(c64, SupportsComplex))  # Output: True
c = complex(c64)
print(c)  # Output: (3+4j)
    • NumPy’s complex64 type is not a subclass of Python's built-in complex, but it implements the __complex__ method. This makes it compatible with the SupportsComplex protocol, allowing it to be checked at runtime with isinstance and used as an argument to the complex() function.

  • Practical Use Case for SupportsComplex: Checking Compatibility

    from typing import SupportsComplex
from numbers import Complex

def use_as_complex(o: SupportsComplex) -> complex:
    if isinstance(o, SupportsComplex):
        return complex(o)
    raise TypeError('o must be convertible to complex')

# This works because complex64 is a SupportsComplex
c64 = np.complex64(3+4j)
print(use_as_complex(c64))  # Output: (3+4j)

Duck Typing vs. Explicit Type Checking

  • Duck Typing:

    • Instead of using isinstance checks, you can rely on Python's dynamic nature and attempt the operation directly, handling any exceptions that arise. This is known as the EAFP (Easier to Ask for Forgiveness than Permission) principle.
    def use_as_complex(o):
    try:
        return complex(o)
    except TypeError as exc:
        raise TypeError('o must be convertible to complex') from exc

# If o is not compatible, Python raises an appropriate exception.
c = use_as_complex(np.complex64(3+4j))

  • Comparison:

    • Explicit Checking:
      • Useful when you need to ensure an object meets certain interface requirements before performing operations on it. This can be particularly helpful in large codebases or when using external type checkers like Mypy.
    • Duck Typing:
      • Often more flexible and Pythonic, as it leverages the language’s dynamic typing capabilities. It can reduce boilerplate code and avoid unnecessary checks.

When to Use Runtime Checkable Protocols

  • Good Use Cases:

    • When you need to combine static type checking with runtime validation, particularly in codebases where both dynamic and static typing practices are used.
    • Useful in APIs where you want to ensure that objects passed to functions meet specific interface requirements, but without enforcing a strict inheritance hierarchy.

  • Limitations:

    • While runtime checkable protocols offer flexibility, they can introduce overhead if used excessively for runtime checks. Duck typing might be a better choice in performance-critical code.
    • Static type checkers like Mypy cannot always recognize virtual subclasses or runtime-registered protocols, limiting the effectiveness of runtime checkable protocols in some static analysis scenarios.

Limitations of Runtime Protocol Checks in Python

  • Python’s type hints are primarily used for static type checking and are generally ignored at runtime. This limitation also extends to runtime checks using isinstance or issubclass when dealing with static protocols defined using typing.Protocol.

  • SupportsFloat Protocol:

    • The SupportsFloat protocol checks if a class has a __float__ method. However, this check only verifies the existence of the method, not its behavior or return type.

  • Complex Type Example:

    • In Python 3.9, the complex type has a __float__ method, but this method is designed to raise a TypeError because a complex number cannot be converted to a float.

    • Console Session:

      import sys
print(sys.version)  # Output: '3.9.5 (v3.9.5:0a7dcbdb13, May 3 2021, 13:17:02)'

c = 3+4j
print(c.__float__)  # Output: <method-wrapper '__float__' of complex object>
print(c.__float__())  # Raises TypeError: can't convert complex to float

  • Misleading isinstance and issubclass Checks:

    • Despite the __float__ method being present, checking if complex is a subclass of SupportsFloat or if an instance of complex is an instance of SupportsFloat will incorrectly return True.

    • Example:

      from typing import SupportsFloat

c = 3+4j
print(isinstance(c, SupportsFloat))  # Output: True
print(issubclass(complex, SupportsFloat))  # Output: True

Limitations and Consequences

  • Method Presence vs. Method Behavior:

    • isinstance and issubclass checks against protocols like SupportsFloat only confirm the presence of the required method (__float__ in this case). They do not verify:
      • The method's signature.
      • The method's return type.
      • Whether the method functions correctly according to the protocol’s intent.

  • Impact of Ignoring Type Hints at Runtime:

    • Even if the __float__ method had proper type annotations (e.g., returning NoReturn to indicate it always raises an exception), Python’s runtime checks would not consider these annotations, leading to misleading results.

  • Performance Considerations:

    • Implementing comprehensive type checks at runtime would significantly degrade performance, which is why Python avoids verifying method signatures or annotations during isinstance or issubclass checks.

Resolution and Workarounds

  • Python 3.10 Fix:

    • The specific issue with the complex type’s __float__ method was resolved in Python 3.10 by removing the method entirely. This prevents complex from incorrectly being recognized as a SupportsFloat instance or subclass.

  • General Advice:

    • Awareness of Limitations:
      • Be aware of the limitations of runtime checks against protocols. These checks are useful but not foolproof, as they can lead to incorrect assumptions about an object’s capabilities.
    • Duck Typing:
      • Instead of relying solely on isinstance or issubclass checks, consider using duck typing with the EAFP (Easier to Ask for Forgiveness than Permission) principle. Attempt the operation directly and handle exceptions if they occur.

  • Example of Duck Typing:

    def convert_to_float(o):
    try:
        return float(o)
    except (TypeError, ValueError) as exc:
        raise TypeError("Object cannot be converted to float") from exc

c = 3+4j
try:
    print(convert_to_float(c))  # Will raise TypeError and provide a clear message
except TypeError as e:
    print(e)

Designing a Static Protocol in Python: The RandomPicker Example

Context

  • Protocols in Python: Protocols in Python allow you to define interfaces that classes can conform to without explicitly inheriting from them. This is particularly useful for static type checking and enhances the flexibility of duck typing.
  • Single-Method Protocols: Inspired by the Go programming language, single-method protocols are often more useful and flexible. You can later combine them into more complex protocols if needed.

Defining the RandomPicker Protocol

  • Purpose: The RandomPicker protocol defines a minimal interface that requires implementing a single method, pick, which returns an item.

  • Example 13-18: Defining RandomPicker:

    from typing import Protocol, runtime_checkable, Any

@runtime_checkable
class RandomPicker(Protocol):
    def pick(self) -> Any:
        ...
    • Key Points:
      • @runtime_checkable: This decorator allows RandomPicker to be used with isinstance and issubclass checks at runtime.
      • pick(self) -> Any: The pick method is expected to return an item, and its return type is currently set to Any. This makes the protocol flexible and applicable to various implementations.

Using the RandomPicker Protocol

  • Implementing the Protocol:

    • The SimplePicker class implements the RandomPicker protocol by providing a pick method that pops a shuffled item from a list.
    import random
from typing import Any, Iterable, TYPE_CHECKING
from randompick import RandomPicker

class SimplePicker:
    def __init__(self, items: Iterable) -> None:
        self._items = list(items)
        random.shuffle(self._items)

    def pick(self) -> Any:
        return self._items.pop()

def test_isinstance() -> None:
    popper: RandomPicker = SimplePicker([1])
    assert isinstance(popper, RandomPicker)

def test_item_type() -> None:
    items = [1, 2]
    popper = SimplePicker(items)
    item = popper.pick()
    assert item in items
    if TYPE_CHECKING:
        reveal_type(item)
    assert isinstance(item, int)
    • Implementing RandomPicker:
      • SimplePicker implements the RandomPicker protocol without explicitly subclassing it. This is a key feature of static duck typing in Python.
    • Runtime Check with isinstance:
      • The test_isinstance function demonstrates that you can use isinstance to check if an instance conforms to the RandomPicker protocol, thanks to the @runtime_checkable decorator.
    • Type Checking with reveal_type:
      • Inside test_item_type, the reveal_type(item) function, recognized by Mypy, helps reveal the type of item at runtime. This function is only invoked during static type checking and is skipped during actual execution.

Results of Running Mypy

  • Expected Output:

    • Running Mypy on the test code will reveal that Mypy understands SimplePicker as consistent with the RandomPicker protocol.

    • Output Example:

      $ mypy randompick_test.py
randompick_test.py:24: note: Revealed type is 'Any'

    • The output shows that the type of item is Any based on the pick method's return type, as expected.

Conclusion

  • Advantages of Single-Method Protocols:
    • Single-method protocols, like RandomPicker, make static duck typing more flexible. They allow you to define minimal interfaces that can be combined into more complex protocols as needed.
  • Runtime and Static Type Checking:
    • The combination of runtime checkability with static type checking allows you to write flexible yet type-safe code. Using @runtime_checkable, you can ensure your protocols work with isinstance and issubclass checks while benefiting from static type checking with tools like Mypy.
  • Mypy Integration:
    • The integration with Mypy provides additional safety by ensuring that your classes correctly implement the required methods and adhere to the expected interfaces, catching potential errors early in the development process.

Best Practices for Protocol Design in Python

  • Designing effective protocols in Python, particularly for static typing, requires careful consideration to ensure flexibility, maintainability, and adherence to key software design principles.
  • Below are some best practices inspired by the experience of other programming communities, particularly Go, and established design principles.

1. Favor Narrow Protocols

  • Definition: A narrow protocol defines a minimal interface, often with just one or two methods. This makes the protocol easy to implement and understand.
  • Benefits:
    • Flexibility: Narrow protocols are more adaptable to different use cases. They allow classes to implement only what is necessary without being burdened by a large, monolithic interface.
    • Extendability: If additional functionality is needed later, you can easily extend a narrow protocol by combining it with other protocols or creating a new derived protocol.
  • Example:
    • Instead of defining a protocol with multiple methods, create single-method protocols like Reader, Writer, or Closer, which can be combined as needed.

2. Design Protocols with the "Role Interface" Concept

  • Role Interface:
    • A concept where interfaces (or protocols) are designed to represent a specific role an object might play. This is particularly useful in systems where an object might serve multiple roles.
    • Example:
      • A Reader protocol for objects that can read data, and a Writer protocol for objects that can write data. An object could implement both if it plays both roles.

3. Define Protocols Close to Client Code

  • Client-Code Protocols:
    • Sometimes it’s beneficial to define a protocol directly within or near the function or module that uses it, rather than in a general-purpose library.
    • Advantages:
      • Ease of Extensibility: Makes it easier to create new types specifically for that function or module.
      • Testing: Facilitates testing with mocks, as the protocol is defined in the same context where it is used.
    • Example:
      • Define a protocol within a module that handles file I/O, rather than in a separate utilities module. This keeps the protocol tightly coupled to its usage context while remaining flexible.

4. Adhere to the Interface Segregation Principle (ISP)

  • ISP Summary: "Clients should not be forced to depend upon interfaces that they do not use."
    • Application in Protocol Design:
      • Avoid creating large protocols that require implementing methods unnecessary for certain clients. Instead, break down large interfaces into smaller, more specific protocols.
    • Example:
      • Instead of a monolithic FileHandler protocol that includes reading, writing, closing, and seeking methods, create separate protocols like Reader, Writer, and Seeker.

5. Naming Conventions

  • Use Clear, Descriptive Names:
    • Plain Names: For protocols that represent a clear concept (e.g., Iterator, Container).
    • SupportsX: For protocols that provide callable methods (e.g., SupportsInt, SupportsRead).
    • HasX: For protocols that involve readable/writable attributes or getter/setter methods (e.g., HasItems, HasFileno).
  • Go-Inspired Naming:
    • For single-method protocols, use a noun form of the verb by appending “-er” or “-or” (e.g., Reader, Writer, Formatter). This aligns with the convention used in the Go language and makes the purpose of the protocol immediately clear.
    • Example:
      • Instead of SupportsRead, use Reader.
      • Instead of SupportsWrite, use Writer.

6. Extendability through Derived Protocols

  • Extendability:
    • Start with a minimal protocol and extend it by creating derived protocols when more functionality is required.
    • Example:
      • Begin with a Reader protocol. If later you need a type that also resets its position, you can create an AdvancedReader protocol that extends Reader and adds a reset() method.

7. Example of Protocol Evolution

  • Initial Protocol:

    from typing import Protocol

class Reader(Protocol):
    def read(self) -> str:
        ...

  • Extended Protocol:

    class AdvancedReader(Reader, Protocol):
    def reset(self) -> None:
        ...

The Numbers ABCs and Numeric Protocols: A Summary

  • Python's numeric Abstract Base Classes (ABCs) in the numbers module work well for runtime type checking, but they present limitations when it comes to static type checking. The introduction of numeric protocols in the typing module offers a partial solution, but these protocols also have their own limitations, particularly when dealing with complex numbers.

1. Numbers ABCs for Runtime Type Checking

  • Purpose: The numbers module provides ABCs that help identify and work with different numeric types at runtime. These include:

    • numbers.Number: The base class for all numeric types.
    • numbers.Integral: For integer types (int, bool, and others).
    • numbers.Real: For floating-point types.
    • numbers.Complex: For complex number types.

  • Usage:

    • You can use these ABCs with isinstance and issubclass to perform runtime checks.

    • Example:

      import numbers

x = 42
print(isinstance(x, numbers.Integral))  # True

  • Limitations:

    • The numbers.Number ABC has no methods, making it unsuitable for static type checking. If you declare a variable as x: Number, tools like Mypy will not allow you to perform arithmetic or call any methods on x.
    • Certain numeric types, like decimal.Decimal, are intentionally not registered as subclasses of numbers.Real to prevent accidental mixing with less precise floating-point numbers.

2. Numeric Protocols in the typing Module

  • Purpose: Numeric protocols like SupportsFloat, SupportsComplex, and SupportsInt were introduced to improve static type checking by specifying that an object supports specific numeric operations.

    from typing import SupportsFloat

def to_float(value: SupportsFloat) -> float:
    return float(value)

  • Limitations:

    • Runtime Issues: These protocols can sometimes provide misleading results at runtime. For example, in Python 3.9, the complex type has a __float__ method that raises TypeError, but isinstance(x, SupportsFloat) returns True for a complex number.
    • Static vs. Runtime Discrepancies: While tools like Mypy can correctly identify types that support these protocols during static analysis, the runtime behavior may not align due to the way Python handles certain types, especially with complex numbers.

3. Static Type Checking Challenges

  • Static Type Checking Limitations:

    • The numbers ABCs do not work well for static type checking because they do not define any operations that static type checkers can verify.
    • Typeshed and Numeric Protocols: The Python typeshed project defines type hints like _Number (a union of float, Decimal, and Fraction) for use in the standard library. However, this approach doesn't account for numeric types from external libraries, which the numbers ABCs support at runtime.

  • Mypy Behavior:

    • Mypy, a popular static type checker, pragmatically recognizes that int and float can be converted to complex, even though they do not implement __complex__. This behavior helps bridge the gap between static type checking and Python's dynamic runtime behavior.
Previous ch11.a_pythonic_object Next ch14.inheritance