Python notes

ch06.object_references_mutability_recycling

Ch 6. Object References, Mutability and Recycling 

graph LR;
    A["Python Object"] -->|contains| B["Identity\nType\nValue"]
    B -.-> D["Only the value may change over time"]

    P["Variables hold references to objects"]
    P --> R["Simple Assignment\nDoes Not Create Copies"]
    P --> S["Augmented Assignment += or *=\nCreates New Objects if Immutable"]
    P --> T["May Modify Mutable Object In-Place"]
    P --> XX["Function parameters are passed as aliases\nwhich means the function may change\nany mutable object received as an argument"]
    XX --> XXX["Using mutable objects as default values\nfor function parameters is dangerous\nif the parameters are changed in place\nthen the default is changed\naffecting every future call that relies on the default."]
    P --> Y["Objects are discarded (garbage collected)\nonce references to them reaches 0"]
    P --> X["Assigning a new value to an existing variable creates\n\nrebinding\n\n - reduce reference for original object\n - changes on new variables won't affect the previous bounded one"]
    X -.-> Y




    I["immutable collection (eg. tuples)"]
    L["Identities of immutable objects never change"]
    F["Could hold references to mutable items\nits value may change\nwhen the value of a mutable item changes"]
    M["Frozenset Class won't have this problem\nIt only Holds hashable Elements\n(Hashable Values Cannot Change)"]

    I --> L
    I --> F
    I --> M

Variables Are Not Boxes

In many programming languages, the metaphor of variables as boxes can be helpful. However, this metaphor is misleading when applied to Python, where variables act more like labels or references to objects rather than containers.

The "Variables as Labels" Metaphor

In Python, variables are labels attached to objects. This means multiple variables can refer to the same object.

Variables Holding References

>>> a = [1, 2, 3]
>>> b = a
>>> a.append(4)
>>> b
[1, 2, 3, 4]
  • In the "variables as boxes" metaphor, b would hold a copy of [1, 2, 3], and appending 4 to a would not affect b.
  • However, since b is just another label for the same list, the list is modified regardless of whether we access it through a or b.

The Binding Process

Variables are bound to objects only after the objects are created. This means that the right-hand side of an assignment is evaluated first.

Example: Binding Variables to Objects

>>> class Gizmo:
...     def __init__(self):
...         print(f'Gizmo id: {id(self)}')
...
>>> x = Gizmo()
Gizmo id: 4301489152
>>> y = Gizmo() * 10
Gizmo id: 4301489432
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for *: 'Gizmo' and 'int'
>>> dir()
['Gizmo', '__builtins__', '__doc__', '__loader__', '__name__', '__package__', '__spec__', 'x']
  1. Class Definition:
    • Define a class Gizmo with an initializer that prints the object's ID.
  2. Create an Instance:
    • x = Gizmo(): This creates a Gizmo object and binds it to x.
  3. Attempt Invalid Operation:
    • y = Gizmo() * 10: This tries to create a new Gizmo and multiply it by 10, which raises a TypeError.
  4. Evaluate Before Binding:
    • The Gizmo object is instantiated (as indicated by the printed ID) before the multiplication is attempted and fails.
  5. Variable Existence:
    • dir(): Shows that y was never created because the assignment failed before completion.

Understanding Assignment in Python:

  • Evaluate Right-Hand Side First:
    • Read the right-hand side of an assignment first. This is where the object is created or retrieved.
  • Bind Variable to Object:
    • After the object is created, the variable on the left is bound to the object, like attaching a label to it.

Choosing Between == and is in Python

  • == (Equality Operator):

    • Compares the values of objects, i.e., the data they hold.
    • Frequently used to determine if two objects have the same value.
    • Example: a == b checks if the values of a and b are equal.
    • Implemented via the __eq__ method, which can be overloaded by classes.

  • is (Identity Operator):

    • Compares the identities of two objects.
    • Checks if two variables point to the exact same object in memory.
    • Faster than == because it does not involve method calls; simply compares memory addresses (IDs).
    • Commonly used to check if a variable is None.
    • Example: x is None checks if x is exactly None.

Advantages of Python's Approach

  1. Intuitive Comparisons:

    • == compares values, which is often what developers intend when checking for equality.
    • Avoids pitfalls of reference comparison when comparing objects like strings.

  2. Safe Handling of None:

    • None is a proper object, eliminating issues like null pointer exceptions in Java.

  3. Flexibility with Operator Overloading:

    • Python's operator overloading allows developers to define custom equality logic, enhancing the expressiveness of the language.

  4. Consistency Across Types:

    • Python's approach ensures consistent behavior across all types, including user-defined classes, providing a coherent and predictable model for equality.

Common Use Cases

  1. Comparing with None:

    • Use is when checking if a variable is None:

      if x is None:
    # Do something if x is None

  2. Singletons:

    • Use is for sentinel objects or unique instances meant to be singletons:

      END_OF_DATA = object()
if node is END_OF_DATA:
    # Handle end of data

  3. Comparing Values:

    • Use == when comparing values, especially for complex data structures:

      if a == b:
    # Do something if a and b have the same value

Performance Considerations

  • is Operator:

    • Faster because it directly compares object IDs.
    • Cannot be overloaded, making it more predictable and consistent.

  • == Operator:

    • May involve substantial computation if the objects are large or deeply nested.
    • Invokes the __eq__ method, which can be customized in user-defined classes to define equality behavior.

The Relative Immutability of Tuples

  • Immutability Definition: Tuples are immutable in terms of their references. They cannot change which objects they refer to once created.

  • Mutable Contents: If a tuple contains mutable objects, the contents of those objects can change. This means the overall "value" of the tuple can seem to change because the mutable items inside it can be modified.

Example

Consider two tuples, t1 and t2:

t1 = (1, 2, [30, 40])
t2 = (1, 2, [30, 40])

  • Equality Check: Initially, t1 and t2 are equal because their contents are the same:

    t1 == t2  # True

  • Identity Check: The identities of the list objects within the tuples are unique but they point to objects with equal values:

    id(t1[-1])  # ID of list in t1
id(t2[-1])  # Different ID of list in t2

  • Mutation Effect: If the list within t1 is mutated:

    t1[-1].append(99)

    Now t1 becomes:

    (1, 2, [30, 40, 99])

  • Post-Mutation Equality: t1 and t2 are no longer equal because the list within t1 has changed:

    t1 == t2  # False

Implications

  1. Immutability Caveats: Tuples' immutability pertains only to the structure (i.e., the references), not the referenced objects themselves.

  2. Hashability: A tuple containing mutable objects (like lists) is not hashable. Hashability requires that the object’s value is unchanging.

  3. Copying Considerations: When copying tuples with mutable objects, decide whether to create deep copies of the mutable objects or to share references. This decision depends on whether changes to the contents of the mutable objects should affect other tuples or copies.

  4. Aliasing and Equality: Understanding equality (==) and identity (is) is crucial for correctly using tuples in situations involving aliasing (e.g., same mutable object referenced by different tuples).

Copies Are Shallow by Default

In Python, creating a copy of a list or other mutable collection using its constructor (e.g., list()) or slicing (e.g., [:]) results in a shallow copy. This means:

  • Shallow Copy: The outer container is copied, but it references the same inner objects as the original.

  • Behavior:

    • Modifying mutable items (like lists) in the copy affects the original, and vice versa.
    • Immutable items (like numbers or strings) are safe from such side effects.

Example

Consider this Python code:

l1 = [3, [66, 55, 44], (7, 8, 9)]
l2 = list(l1)

  • Initial State: l2 is a shallow copy of l1, with both lists holding references to the same inner objects.

  • Modification Effects:

  • Appending to l1: l1.append(100) does not affect l2.

  • Removing from inner list: l1[1].remove(55) affects both because the inner list is shared.

l1: [3, [66, 44], (7, 8, 9), 100]
l2: [3, [66, 44], (7, 8, 9)]
  • Mutating shared list: l2[1] += [33, 22] modifies the shared list.
  • Modifying tuples: l2[2] += (10, 11) creates a new tuple, so l2[2] changes but l1[2] remains the same.
l1: [3, [66, 44, 33, 22], (7, 8, 9), 100]
l2: [3, [66, 44, 33, 22], (7, 8, 9, 10, 11)]
  • Conclusion
    • Shallow Copy Implications: Be cautious when copying lists containing mutable objects. Shared references can lead to unexpected modifications across copies.
    • Visualization: Using tools like Online Python Tutor can help visualize how shallow copies work and how references are shared.

Deep and Shallow Copies of Arbitrary Objects

Python’s copy module provides the copy() and deepcopy() functions to create shallow and deep copies of objects, respectively.

  • Shallow Copy (copy.copy()):

    • Creates a new object, but inserts references to the original nested objects.
    • Modifications to mutable nested objects in the copy affect the original object and vice versa.
    • Shallow copies are more memory-efficient

  • Deep Copy (copy.deepcopy()):

    • Recursively copies all objects, creating an entirely independent copy.
    • Changes to the copied object do not affect the original object.
    • Deep copies provide complete independence between the original and the copied objects

Example

Consider a Bus class:

class Bus:
    def __init__(self, passengers=None):
        if passengers is None:
            self.passengers = []
        else:
            self.passengers = list(passengers)

    def pick(self, name):
        self.passengers.append(name)

    def drop(self, name):
        self.passengers.remove(name)
import copy

# Original bus object
bus1 = Bus(['Alice', 'Bill', 'Claire', 'David'])

# Shallow copy of bus1
bus2 = copy.copy(bus1)

# Deep copy of bus1
bus3 = copy.deepcopy(bus1)

# IDs of the bus instances are different
print(id(bus1), id(bus2), id(bus3))  # (id1, id2, id3)

# Drop a passenger from the original bus
bus1.drop('Bill')

# Effects of the drop operation
print(bus1.passengers)  # ['Alice', 'Claire', 'David']
print(bus2.passengers)  # ['Alice', 'Claire', 'David'] (shallow copy affected)
print(bus3.passengers)  # ['Alice', 'Bill', 'Claire', 'David'] (deep copy unaffected)

# Check IDs of passengers lists
print(id(bus1.passengers), id(bus2.passengers), id(bus3.passengers))

Explanation

  • Shallow Copy:

    • bus2 is a shallow copy of bus1. They have different IDs, but they share the same passengers list.
    • Changes to the passengers list in bus1 are reflected in bus2.

  • Deep Copy:

    • bus3 is a deep copy of bus1. The passengers list is a new, independent list.
    • Changes to bus1 do not affect bus3.

Handling Cyclic References

The deepcopy() function can handle cyclic references, avoiding infinite loops by keeping track of objects already copied:

from copy import deepcopy

a = [10, 20]
b = [a, 30]
a.append(b)

# Cyclic structure: a contains a reference to itself via b
print(a)  # [10, 20, [...], 30]

# Deep copy handles cyclic references
c = deepcopy(a)
print(c)  # [10, 20, [...], 30]

Customizing Copy Behavior

  • Implement the __copy__() and __deepcopy__() special methods in your class to control the behavior of shallow and deep copies.
  • This is useful when objects reference external resources or singletons that should not be copied.

Function Parameters as References

Python uses call by sharing (also known as pass by object reference) for parameter passing. This means:

  • Reference Copying: When you pass an argument to a function, Python passes a reference to the object (i.e., the function receives a reference to the original object).

  • Mutable vs. Immutable:

    • Mutable objects (like lists or dictionaries) can be modified within the function.
    • Immutable objects (like integers, strings, or tuples) cannot be altered; only new objects can be created.

Example Analysis

Consider the function f defined as:

def f(a, b):
    a += b
    return a

Integers (Immutable) 

x = 1
y = 2
result = f(x, y)
print(result)  # 3
print(x, y)    # (1, 2)
  • Behavior: Integers are immutable. a += b creates a new integer rather than modifying x.

Lists (Mutable) 

a = [1, 2]
b = [3, 4]
result = f(a, b)
print(result)  # [1, 2, 3, 4]
print(a, b)    # ([1, 2, 3, 4], [3, 4])
  • Behavior: Lists are mutable. a += b modifies the list in place, changing a.

Tuples (Immutable) 

t = (10, 20)
u = (30, 40)
result = f(t, u)
print(result)  # (10, 20, 30, 40)
print(t, u)    # ((10, 20), (30, 40))
  • Behavior: Tuples are immutable. a += b creates a new tuple, leaving t unchanged.

Assignment on mutable in function create local object 

def replace_list(lst):
    lst = [100, 200, 300]  # This creates a new local variable and does not affect the original list

a = [1, 2, 3]
replace_list(a)
print(a)  # Outputs: [1, 2, 3]

Key Points

  • Assignment: You cannot reassign the object itself inside the function (i.e., a = new_value does not affect the caller's object).

  • Modifying Objects: You can modify mutable objects in place, affecting the original outside the function.

Mutable Types as Parameter Defaults: A Bad Idea

Using mutable objects as default parameter values can lead to unexpected behavior due to shared state across function calls. Here's an example using the HauntedBus class:

class HauntedBus:
    """A bus model haunted by ghost passengers"""
    def __init__(self, passengers=[]):
        self.passengers = passengers

    def pick(self, name):
        self.passengers.append(name)

    def drop(self, name):
        self.passengers.remove(name)

Explanation of the Problem

  • Mutable Default Issue: The passengers list is defined once when the function is loaded, not each time the function is called. As a result, all instances of HauntedBus that don't explicitly receive a passengers list will share the same default list.

  • Aliasing: self.passengers becomes an alias for the default passengers list when no argument is passed, leading to unexpected sharing between instances.

Example of Unexpected Behavior 

bus1 = HauntedBus(['Alice', 'Bill'])
bus1.pick('Charlie')
bus1.drop('Alice')
print(bus1.passengers)  # ['Bill', 'Charlie']

bus2 = HauntedBus()
bus2.pick('Carrie')
print(bus2.passengers)  # ['Carrie']

bus3 = HauntedBus()
bus3.pick('Dave')
print(bus3.passengers)  # ['Carrie', 'Dave']

print(bus2.passengers)  # ['Carrie', 'Dave']
print(bus2.passengers is bus3.passengers)  # True

To avoid these issues, use None as a default value and initialize the mutable object inside the function:

class SafeBus:
    """A bus model without ghost passengers"""
    def __init__(self, passengers=None):
        if passengers is None:
            self.passengers = []
        else:
            self.passengers = list(passengers)

    def pick(self, name):
        self.passengers.append(name)

    def drop(self, name):
        self.passengers.remove(name)
  • Use None for Defaults: Default parameters should be None for mutable types, with initialization occurring inside the function to ensure a new object is created for each call.
  • Avoid Shared State: This avoids the shared state across instances, preventing bugs and unexpected behavior.

Defensive Programming with Mutable Parameters

  • Considerations for Mutable Parameters:

    • When coding a function with a mutable parameter, assess if the caller expects the argument to be altered.
    • Determine if modifications to a mutable parameter should be visible outside the function. This depends on the context and requires aligning the coder's and caller's expectations.

  • TwilightBus Example:

    • Problem Demonstration:

      basketball_team = ['Sue', 'Tina', 'Maya', 'Diana', 'Pat']
bus = TwilightBus(basketball_team)
bus.drop('Tina')
bus.drop('Pat')
basketball_team  # Output: ['Sue', 'Maya', 'Diana']
      • Basketball team initially has five members.
      • After using TwilightBus, dropped passengers are removed from the original basketball team list.

  • Principle of Least Astonishment:

    • TwilightBus violates the principle by unexpectedly altering the basketball team list when dropping students.
    • The principle advises against surprising behavior in interface design.

  • Implementation and Problem Analysis:

    • Class Implementation:

      class TwilightBus:
    """A bus model that makes passengers vanish"""

    def __init__(self, passengers=None):
        if passengers is None:
            self.passengers = []
        else:
            self.passengers = passengers

    def pick(self, name):
        self.passengers.append(name)

    def drop(self, name):
        self.passengers.remove(name)

    • Issue Analysis:

      • self.passengers becomes an alias for the passengers argument, directly modifying the original list.
      • .remove() and .append() methods mutate the original list passed to __init__.

  • Solution to the Problem:

    • Fix by Copying the List:

      • Modify the __init__ method to create a copy of the passengers list:

        def __init__(self, passengers=None):
    if passengers is None:
        self.passengers = []
    else:
        self.passengers = list(passengers)

      • This approach ensures self.passengers does not alias the original list, preventing unintended side effects.

  • Advantages of the Solution:

    • Internal handling of the passenger list does not affect the original argument.
    • Increased flexibility: the passengers parameter can accept any iterable (e.g., tuples, sets, database results).
    • Ensures support for .remove() and .append() operations in pick() and drop() methods.

  • Best Practices:

    • Avoid aliasing objects received as arguments unless explicitly intended.
    • If unsure, make a copy to prevent unexpected mutations. This trade-off of increased resource usage is preferable to subtle bugs.

del and Garbage Collection in Python

Key Concepts

  • Object Destruction:

    • Objects in Python are never explicitly destroyed by the programmer.
    • Objects are garbage-collected when they become unreachable.

  • The del Statement:

    • del is a statement, not a function:
      • Syntax: del x (not del(x), although del(x) works because (x) is the same as x).
    • Purpose:
      • del deletes references, not the actual objects.
      • An object may be garbage-collected if del removes the last reference to it.
      • Rebinding a variable can also reduce the reference count to zero, leading to object destruction.

Example of del and References 

>>> a = [1, 2]
>>> b = a
>>> del a
>>> b
[1, 2]
>>> b = [3]
  • Explanation:
    • Creation and Binding:
      • Object [1, 2] is created and a is bound to it.
      • b is also bound to the same object [1, 2].
    • Deleting Reference:
      • del a deletes the reference a, but the object [1, 2] remains because b still points to it.
    • Rebinding:
      • Rebinding b to a different object [3] removes the last reference to [1, 2], allowing it to be garbage-collected.

Garbage Collection in CPython

  • Reference Counting:

    • The primary garbage collection algorithm in CPython is reference counting.
    • Each object maintains a count of how many references point to it.
    • When the reference count reaches zero, the object is immediately destroyed, and memory is freed.
    • The __del__ method is called before memory is freed, if defined.

  • Generational Garbage Collection:

    • CPython 2.0 introduced a generational garbage collector to handle reference cycles.
    • It detects groups of objects that reference each other, which would otherwise remain unreachable.

  • Alternative Implementations:

    • Other Python implementations (e.g., PyPy) use different garbage collectors that do not rely on reference counting.
    • In such cases, the __del__ method may not be called immediately when no more references exist.

Example of Garbage Collection with weakref 

>>> import weakref
>>> s1 = {1, 2, 3}
>>> s2 = s1
>>> def bye():
...     print('...like tears in the rain.')
...
>>> ender = weakref.finalize(s1, bye)
>>> ender.alive
True
>>> del s1
>>> ender.alive
True
>>> s2 = 'spam'
...like tears in the rain.
>>> ender.alive
False

  • Explanation:

    • Aliasing:
      • s1 and s2 both reference the same set {1, 2, 3}.
    • Callback Registration:
      • The bye function is registered as a callback using weakref.finalize, which will be called when the object is destroyed.
    • Reference Deletion:
      • del s1 deletes the reference s1 to the set, but it remains because s2 still points to it.
    • Rebinding and Garbage Collection:
      • Rebinding s2 makes {1, 2, 3} unreachable, triggering garbage collection, invoking the bye callback, and setting ender.alive to False.

  • Limitations of Reference Counting:

    • Memory Leaks with Cyclic References:

      • Reference counting alone cannot handle cyclic references where objects reference each other, creating a cycle that keeps their reference counts above zero.

      • Example of a cyclic reference:

        class Node:
    def __init__(self):
        self.next = None

node1 = Node()
node2 = Node()
node1.next = node2
node2.next = node1  # Cycle created

  • Generational Garbage Collection:

    • Introduced in Python 2.0, the generational garbage collector complements reference counting by identifying and collecting cyclic references.
    • It organizes objects into generations, optimizing for the common case where most objects have a short lifespan.

Safe Resource Management

  • Resource Management in CPython:

    • In CPython, objects with zero references are immediately cleaned up, which can be relied upon for certain resource management tasks.

    • Example:

      open('test.txt', 'wt', encoding='utf-8').write('1, 2, 3')
# The file is closed automatically after write() because its reference count goes to zero

  • Portability and Best Practices:

    • Although CPython's immediate cleanup might seem convenient, relying on it is not portable. Other Python implementations, like Jython and IronPython, use different garbage collection mechanisms.
    • The best practice for resource management is to explicitly close resources using context managers to ensure resources are freed promptly.

  • Using the with Statement:

    • The with statement provides a context manager that ensures resources are properly managed, even in the presence of exceptions.

    • Example with a context manager:

      with open('test.txt', 'wt', encoding='utf-8') as fp:
    fp.write('1, 2, 3')  # The file is automatically closed after the block

Weak References

  • Weak References:
    • weakref allows references to an object that do not increase its reference count.
    • Useful in caching applications to avoid keeping cached objects alive solely because they are referenced by the cache.

The __del__ Method

  • Purpose of __del__:

    • __del__ is a special method called by the Python interpreter before an object is destroyed.
    • It allows an object to release external resources.

  • Usage Caution:

    • Implementing __del__ is rarely needed and can be tricky.
    • Avoid using __del__ unless necessary, as improper use can lead to resource management issues.

  • Documentation:

    • For detailed information, refer to the __del__ method documentation in the "Data Model" chapter of The Python Language Reference.

Tricks Python Plays with Immutables

Tuples and Immutability

  • Tuple Slicing and Copying:

    • Using slicing on a tuple (t[:]) or creating a new tuple from an existing one (tuple(t)) does not create a new tuple object. Instead, it returns a reference to the original tuple.

  • Example with Tuples:

    >>> t1 = (1, 2, 3)
>>> t2 = tuple(t1)
>>> t2 is t1
True  # t2 is a reference to the same object as t1
>>> t3 = t1[:]
>>> t3 is t1
True  # t3 is also a reference to the same object as t1

  • Explanation:

    • Since tuples are immutable, Python optimizes memory usage by reusing existing objects. If an operation results in an identical tuple, Python will often return the original object.

Strings, Bytes, and Frozensets

  • Shared References:

    • Strings and frozensets also exhibit similar behavior where operations intended to copy or replicate them often return references to the original object.

  • Example with Frozensets:

    >>> fs1 = frozenset([1, 2, 3])
>>> fs2 = fs1.copy()
>>> fs2 is fs1
True  # fs2 is a reference to the same object as fs1

  • Example with Strings:

    >>> s1 = 'ABC'
>>> s2 = 'ABC'
>>> s2 is s1
True  # s1 and s2 refer to the same string object due to interning

Interning and Memory Optimization

  • String Interning:

    • CPython uses a technique called interning for strings, which involves storing only one copy of a particular string value. This is done to optimize memory usage and improve performance for frequently used strings.

  • Integer Interning:

    • Similar optimization is applied to small integers (e.g., 0, 1, -1), where Python maintains a pool of commonly used integer objects.

  • Example with Small Integers:

    >>> a = 100
>>> b = 100
>>> a is b
True  # a and b refer to the same integer object

  • Limitations:

    • Not all strings or integers are interned. Interning criteria are specific to the CPython implementation and can vary, making it an unreliable feature for application logic.

Best Practices

  • Use == for Equality Checks:

    • Always use == to compare strings or integers for equality, as is checks for object identity, not value equality.

  • Avoid Relying on Interning:

    • Since interning is an internal optimization and not guaranteed across different Python implementations or versions, do not rely on it for program logic.

Object Destruction and Garbage Collection in Python

  • No Direct Object Destruction:

    • Python does not provide a mechanism to explicitly destroy objects. This design decision prevents issues with dangling references, which could occur if an object were destroyed while other parts of the code still held references to it.

  • Implications:

    • If Python allowed explicit object destruction, it could lead to unstable behavior, where references would point to non-existent memory locations, causing errors and crashes.
Previous ch05.data_class_builders Next ch09.decorator_and_closures