Ch 16. Operator Overloading

Operator Overloading 101

Definition:

• Operator overloading allows user-defined objects to interact with operators like +, |, -, ~, and more.
• Operators such as function invocation (), attribute access ., and item access/slicing [] are also considered operators in Python.

Concerns:

• Operator overloading is sometimes criticized due to potential abuse, leading to:
  • Programmer confusion
  • Bugs
  • Performance issues
• When used correctly, it can enhance APIs and improve code readability.

Python's Operator Overloading Rules:

• Operators for built-in types cannot be altered.
• No new operators can be created; only existing ones can be overloaded.
• Some operators, like is, and, or, not, cannot be overloaded, though bitwise operators (&, |, ~) can.

Unary Operators in Python

Unary Operators & Their Special Methods:

• - (Negation): Implemented by __neg__
  • Example: If x = -2, then -x == 2.
• + (Positive): Implemented by __pos__
  • Typically, x == +x, but there are exceptions (see below).
• ~ (Bitwise NOT): Implemented by __invert__
  • Defined as ~x == -(x + 1). Example: If x = 2, then ~x == -3.
• abs() (Absolute value): Implemented by __abs__
  • Returns the absolute value of a number or object.

Guidelines for Implementing Unary Operators:

• Special methods for unary operators take one argument: self.
• Operators should return a new object, not modify the original (self).
  • For - and +, return a new instance of the same class.
  • If the object is immutable, unary + should return self; otherwise, return a copy.
  • For abs(), return a scalar value (e.g., a number).
• Tilde (~): Not meaningful outside bitwise operations, but can be customized (e.g., in pandas it negates boolean filters).

Example of Unary Operators in Vector Class:

def __abs__(self):
    return math.hypot(*self)

def __neg__(self):
    return Vector(-x for x in self)

def __pos__(self):
    return Vector(self)

• __neg__: Creates a new Vector with every component negated.
• __pos__: Returns a new Vector with the same components as self.

Special Case: x != +x

1. decimal.Decimal Example:

   • The arithmetic context’s precision can affect whether x == +x holds true.

   • Example:

     import decimal
     ctx = decimal.getcontext()
     ctx.prec = 40
     one_third = decimal.Decimal('1') / decimal.Decimal('3')
     one_third == +one_third  # True
     ctx.prec = 28
     one_third == +one_third  # False

   • Explanation: +one_third is evaluated using the current precision, creating a new Decimal instance, which may differ from one_third.

2. collections.Counter Example:

   • Unary + on a Counter returns a new Counter that discards items with zero or negative counts.

   • Example:

     from collections import Counter
     ct = Counter('abracadabra')
     ct['r'] = -3
     ct['d'] = 0
     +ct  # Counter({'a': 5, 'b': 2, 'c': 1})

Overloading + for Vector Addition

1. Pairwise addition of components:
   • For vectors of different lengths, fill the shorter one with zeros.
2. Return a new Vector: The original vectors should not be modified.

Implementation of __add__ Method

def __add__(self, other):
    pairs = itertools.zip_longest(self, other, fillvalue=0.0)
    return Vector(a + b for a, b in pairs)

• itertools.zip_longest:
  • Combines the components of self and other into pairs.
  • If vectors have different lengths, missing values are filled with 0.0.
• Generator expression: Iterates through the pairs, adding each pair to create a new Vector.

Example of Vector Addition

v1 = Vector([3, 4, 5])
v2 = Vector([6, 7, 8])
v1 + v2  # Output: Vector([9.0, 11.0, 13.0])

v3 = Vector([3, 4, 5, 6])
v4 = Vector([1, 2])
v3 + v4  # Output: Vector([4.0, 6.0, 5.0, 6.0])

Supporting Non-Vector Operands

• Using __add__ with non-Vector types:

v1 + (10, 20, 30)  # Output: Vector([13.0, 24.0, 35.0])

Handling Reverse Addition with __radd__

• When the left operand does not support +, Python attempts the reverse operation using __radd__.

def __radd__(self, other):
    return self + other

• This ensures that expressions like (10, 20, 30) + v1 will work as expected.

Improved Error Handling

• TypeError is caught and NotImplemented is returned, allowing Python to try __radd__ for the reverse operation.

def __add__(self, other):
    try:
        pairs = itertools.zip_longest(self, other, fillvalue=0.0)
        return Vector(a + b for a, b in pairs)
    except TypeError:
        return NotImplemented

• Catches incompatible types: Returns NotImplemented, giving the other operand's method a chance to handle the operation correctly.

Summary

• Commutative operation (+) works for Vector + Vector, Vector + iterable, and iterable + Vector.
• Error handling allows for cleaner fallback behavior when operands are not compatible.

Overloading * for Scalar Multiplication in the Vector Class

Scalar Multiplication:

v1 = Vector([1, 2, 3])
v1 * 10      # Output: Vector([10.0, 20.0, 30.0])
11 * v1      # Output: Vector([11.0, 22.0, 33.0])

Basic __mul__ and __rmul__ Methods

def __mul__(self, scalar):
    return Vector(n * scalar for n in self)

def __rmul__(self, scalar):
    return self * scalar

• __mul__: Multiplies each component of self by the scalar.
• __rmul__: Handles reverse multiplication (scalar * Vector) by delegating to __mul__.

Handling Incompatible Operands

• Duck typing: Instead of checking the type explicitly, attempt to convert the scalar to a float. If conversion fails, return NotImplemented.

Improved Implementation (with error handling)

def __mul__(self, scalar):
    try:
        factor = float(scalar)
    except TypeError:
        return NotImplemented
    return Vector(n * factor for n in self)

def __rmul__(self, scalar):
    return self * scalar

Scalar Multiplication Examples

• Basic multiplication:

  v1 = Vector([1.0, 2.0, 3.0])
  v1 * 14      # Output: Vector([14.0, 28.0, 42.0])
  14 * v1      # Output: Vector([14.0, 28.0, 42.0])

• Multiplying with bool:

  v1 * True    # Output: Vector([1.0, 2.0, 3.0])

• Multiplying with fractions.Fraction:

  from fractions import Fraction
  v1 * Fraction(1, 3)    # Output: Vector([0.3333333333333333, 0.6666666666666666, 1.0])

Using @ as an Infix Operator for Dot Products

Introduction:

• Since Python 3.5, the @ operator can be used as an infix operator for matrix multiplication or dot products. It simplifies translating mathematical formulas into Python code, which previously required function calls like numpy.dot(a, b).

Special Methods for @

• __matmul__(self, other): Handles the a @ b operation.
• __rmatmul__(self, other): Handles the reverse operation b @ a.
• __imatmul__(self, other): Used for in-place matrix multiplication (e.g., a @= b).

Example of @ Operator with Vector Class

class Vector:
    # many methods omitted
    
    def __matmul__(self, other):
        if (isinstance(other, abc.Sized) and isinstance(other, abc.Iterable)):
            if len(self) == len(other):
                return sum(a * b for a, b in zip(self, other))
            else:
                raise ValueError('@ requires vectors of equal length.')
        else:
            return NotImplemented
    
    def __rmatmul__(self, other):
        return self @ other

Key Points

1. Conditions for @ to work:

   • Both operands must be sized (support __len__) and iterable (support __iter__).
   • Both must have equal lengths.

2. Dot product:

   • Uses sum(a * b for a, b in zip(self, other)) to compute the result.
   • If the lengths are not equal, a ValueError is raised.

3. __rmatmul__:

   • If other @ self is attempted and other doesn't support @, Python will call self.__rmatmul__(other).

Example Usage

va = Vector([1, 2, 3])
vz = Vector([5, 6, 7])

# Dot product of two vectors
va @ vz  # Output: 38.0  (1*5 + 2*6 + 3*7)

# Dot product with a list
[10, 20, 30] @ vz  # Output: 380.0

# Error when @ is used with incompatible types
va @ 3  # Raises: TypeError: unsupported operand type(s) for @: 'Vector' and 'int'

Python 3.10 Update

• zip(strict=True): Since Python 3.10, zip() can enforce strict length checking by raising a ValueError when the input iterables have different lengths. This aligns with Python’s "fail fast" philosophy.

try:
    return sum(a * b for a, b in zip(self, other, strict=True))
except ValueError:
    raise ValueError('@ requires vectors of equal length.')

Rich Comparison Operators in Python

Python handles rich comparison operators (==, !=, >, <, >=, and <=) in a way similar to other infix operators, but with some key differences:

• The same method handles both forward and reverse comparisons.
• For == and !=, if the reverse method is missing or returns NotImplemented, Python defaults to comparing object IDs instead of raising a TypeError.

Example: Improving Vector.__eq__

Initially, the Vector class had the following __eq__ method:

class Vector:
    def __eq__(self, other):
        return (len(self) == len(other) and
                all(a == b for a, b in zip(self, other)))

This method allows for comparisons between Vector and any other iterable, which might lead to unintended results. For instance:

va = Vector([1.0, 2.0, 3.0])
vb = Vector(range(1, 4))
va == vb   # True

t3 = (1, 2, 3)
va == t3   # True (not ideal)

Problem with Excessive Leniency

• This method considers a Vector equal to a tuple or other iterable with the same numbers. This excessive leniency may lead to surprising results, which violates the "Zen of Python" principle: "In the face of ambiguity, refuse the temptation to guess."

Solution: Type Checking in __eq__

To avoid such issues, we can ensure that the __eq__ method only compares Vector instances or instances of Vector subclasses. If the second operand is not a Vector, return NotImplemented, allowing Python to handle the comparison.

class Vector:
    def __eq__(self, other):
        if isinstance(other, Vector):
            return (len(self) == len(other) and
                    all(a == b for a, b in zip(self, other)))
        else:
            return NotImplemented

With the improved __eq__ method:

va = Vector([1.0, 2.0, 3.0])
vb = Vector(range(1, 4))

t3 = (1, 2, 3)

# Same result as before
va == vb   # True

# Same result, as Vector2d is also a Vector-like class
from vector2d_v3 import Vector2d
vc = Vector([1, 2])
v2d = Vector2d(1, 2)
vc == v2d  # True

# Different result, as t3 is not a Vector
va == t3   # False

Step-by-Step Explanation of vc == v2d

1. Python calls Vector.__eq__(vc, v2d).
2. Since v2d is not a Vector, __eq__ returns NotImplemented.
3. Python tries Vector2d.__eq__(v2d, vc).
4. Vector2d.__eq__ compares v2d and vc as tuples, returning True.

Step-by-Step Explanation of va == t3

1. Python calls Vector.__eq__(va, t3).
2. Since t3 is not a Vector, __eq__ returns NotImplemented.
3. Python tries tuple.__eq__(t3, va).
4. tuple.__eq__ doesn't recognize Vector, so it returns NotImplemented.
5. Python compares the object IDs of va and t3, resulting in False.

Handling !=

• We don't need to implement __ne__ because Python automatically handles it by negating the result of __eq__. For example:

va != vb   # False
vc != v2d  # False
va != t3   # True

Augmented Assignment Operators in Python

Python allows classes to support augmented assignment operators like += and *=. These operators work by modifying the object in place, which is why they behave differently from regular + or * operations, particularly for mutable objects.

Vector Class and Augmented Assignment

Our Vector class supports += and *= by default because Python evaluates a += b as a = a + b when the class doesn’t define the in-place operators (__iadd__ or __imul__). For immutable objects like Vector, this means a new instance is created.

Using += and *=

v1 = Vector([1, 2, 3])
v1_alias = v1    # Create an alias to inspect the original object

id(v1)           # Get the ID of the original Vector
v1 += Vector([4, 5, 6])    # Perform augmented addition
v1                # Output: Vector([5.0, 7.0, 9.0])
id(v1)           # The ID has changed; a new Vector was created
v1_alias         # Output: Vector([1.0, 2.0, 3.0]) - original object unchanged
v1 *= 11         # Perform augmented multiplication
v1                # Output: Vector([55.0, 77.0, 99.0])
id(v1)           # The ID has changed again, indicating a new object

• Explanation:
  • += and *= create a new instance of Vector since the original instance is immutable.
  • The alias (v1_alias) refers to the original object, which remains unchanged after the operation.

In-Place Operators (__iadd__, __imul__) for Mutable Objects

In contrast, mutable objects can implement in-place operators such as __iadd__ to modify the object without creating a new instance. In-place operators should return self.

Example: AddableBingoCage Class

The AddableBingoCage class is a mutable object that extends the BingoCage class to support + and +=.

Example 16-17: The + Operator for AddableBingoCage

class AddableBingoCage(BingoCage):
    def __add__(self, other):
        if isinstance(other, Tombola):
            return AddableBingoCage(self.inspect() + other.inspect())
        else:
            return NotImplemented

• Explanation:
  • __add__ creates a new AddableBingoCage by combining the items from self and other.
  • If other is not an instance of Tombola, NotImplemented is returned.

Example 16-18: The += Operator for AddableBingoCage

class AddableBingoCage(BingoCage):
    def __iadd__(self, other):
        if isinstance(other, Tombola):
            other_iterable = other.inspect()
        else:
            try:
                other_iterable = iter(other)
            except TypeError:
                msg = 'right operand in += must be Tombola or an iterable'
                raise TypeError(msg)
        
        self.load(other_iterable)
        return self

• Explanation:
  • __iadd__ modifies self in place by loading the items from other.
  • If other is not iterable or a Tombola, a TypeError is raised.
  • The method returns self, as required for in-place operators.

globe = AddableBingoCage('AEIOU')
globe2 = AddableBingoCage('XYZ')

# Use += to add items from globe2
globe += globe2    # globe now contains 'AEIOUXYZ'

# Use += to add items from an iterable
globe += ['M', 'N']   # globe now contains 'AEIOUXYZMN'

# Use += with a non-iterable raises TypeError
globe += 1   # Raises: TypeError: right operand in += must be Tombola or an iterable

Key Differences Between + and +=

• + Operator:

  • Creates a new object.
  • Typically requires both operands to be of the same type.

• += Operator:

  • Modifies self in place (for mutable objects).
  • More flexible with the second operand, allowing any iterable.