Python notes

ch14.inheritance

Ch 14 Inheritance: For Better or for Worse

Inheritance and Its Challenges

Inheritance Issues

  • Instancing: A type of inheritance, where an instance of a class is created, which can blur the lines between pragmatics (like code reuse) and semantics (like specialization or generalization).
  • Tight Coupling: Superclasses and subclasses are tightly coupled, meaning changes in one can lead to unpredictable changes in the other. This tight coupling can make the codebase brittle and harder to maintain.
  • Complexity: Inheritance hierarchies can become complex, making it difficult to understand the relationships between classes and leading to unintended consequences when changes are made.

The Shift Away from Inheritance

  • Backlash Against Overuse: In recent years, there has been a significant shift away from overusing inheritance, favoring composition over inheritance. The composition design principle involves building classes by combining behaviors from other classes (often through interfaces or other means) rather than relying on a rigid inheritance hierarchy.
  • Composition over Inheritance: This principle advocates for designing systems where objects are composed of smaller, reusable parts rather than being extended through deep inheritance hierarchies. This approach reduces tight coupling and improves flexibility and maintainability.
    • Benefits:
      • Loose Coupling: Reduces the interdependencies between classes, making it easier to change and extend systems without unintended side effects.
      • Increased Flexibility: Composition allows for more flexible designs that can be easily adapted to changing requirements without needing to refactor entire inheritance hierarchies.

Dealing with Legacy Code and Frameworks

  • Legacy Systems: Despite the shift in best practices, many developers still need to maintain systems designed with complex class hierarchies or work within frameworks that enforce inheritance-based designs.
    • Maintaining Complex Inheritance: When dealing with legacy code, it's essential to carefully manage and understand the inheritance structures in place to avoid introducing bugs or creating more brittle systems.
    • Using Frameworks: Some frameworks still rely heavily on inheritance, including multiple inheritance. Developers working with these frameworks must be cautious, ensuring they understand the implications of changes to the class hierarchy and following best practices to minimize the risks associated with tight coupling.

The super() Function in Python

  • The super() function is an essential tool for writing maintainable object-oriented code in Python. It allows subclasses to call methods from their superclasses, which is crucial when overriding methods.

Purpose of super()

  • Method Overriding: When a subclass overrides a method of its superclass, the new method often needs to invoke the original method from the superclass to maintain the functionality provided by the superclass.

  • Example:

    class LastUpdatedOrderedDict(OrderedDict):
    """Store items in the order they were last updated"""
    def __setitem__(self, key, value):
        super().__setitem__(key, value)  # Call superclass method
        self.move_to_end(key)            # Custom logic
    • Steps:
      1. super().__setitem__(key, value): Calls the __setitem__ method from the superclass (OrderedDict).
      2. self.move_to_end(key): Custom behavior ensuring the updated key is moved to the end of the dictionary.

Importance in __init__ Method

  • In Python, the __init__ method of a subclass does not automatically call the __init__ method of its superclass. This contrasts with languages like Java, where a constructor automatically calls the no-argument constructor of the superclass.

  • Pattern:

    def __init__(self, a, b):
    super().__init__(a, b)  # Explicitly call the superclass __init__
    # Additional initialization code

Why super() is Preferred

  • Avoiding Hardcoding:

    • Using super() avoids hardcoding the name of the superclass within the method body. This makes the code more flexible and reduces the chance of errors if the class hierarchy changes.

    • Not Recommended:

      class NotRecommended(OrderedDict):
    def __setitem__(self, key, value):
        OrderedDict.__setitem__(self, key, value)  # Hardcoded base class
        self.move_to_end(key)
      • The above method hardcodes OrderedDict, which could cause issues if the class hierarchy changes.

  • Handling Multiple Inheritance:

    • The super() function is designed to work correctly in complex inheritance hierarchies that involve multiple inheritance. It follows the Method Resolution Order (MRO), ensuring that the correct method is called in cases where multiple classes are inherited.

super() in Python 2 vs. Python 3

  • Python 2:

    • The old syntax required two arguments: the class and the instance:

      class LastUpdatedOrderedDict(OrderedDict):
    def __setitem__(self, key, value):
        super(LastUpdatedOrderedDict, self).__setitem__(key, value)
        self.move_to_end(key)

    • Arguments:

      1. type: The class from which the method resolution search should begin.
      2. object_or_type: The instance (self) or class (cls) where the method should be bound.

  • Python 3:

    • Simplified Syntax: The super() function in Python 3 no longer requires these arguments; they are inferred automatically.

      class LastUpdatedOrderedDict(OrderedDict):
    def __setitem__(self, key, value):
        super().__setitem__(key, value)
        self.move_to_end(key)

Special Cases for super()

  • While the simplified super() is sufficient for most cases, the two-argument form can still be used in Python 3 for advanced scenarios, such as:
    • Skipping parts of the MRO: To bypass certain classes in the MRO during method resolution.
    • Testing or Debugging: To manually control which class’s method is called.

Subclassing Built-In Types Is Tricky

Challenges with Subclassing Built-In Types

  • Early Python Versions: Subclassing built-in types (like dict, list, and str) wasn't possible in early versions of Python. This was later introduced in Python 2.2.
  • Method Overriding Issues: Built-in types in Python are implemented in C, and their methods often do not call overridden methods in user-defined subclasses. This can lead to unexpected behavior when trying to extend or modify the functionality of these types.

Example of Overriding Methods in Subclasses

  • Ignored __setitem__ Override in dict:

    class DoppelDict(dict):
    def __setitem__(self, key, value):
        super().__setitem__(key, [value] * 2)

dd = DoppelDict(one=1)
print(dd)  # Output: {'one': 1}

dd['two'] = 2
print(dd)  # Output: {'one': 1, 'two': [2, 2]}

dd.update(three=3)
print(dd)  # Output: {'one': 1, 'two': [2, 2], 'three': 3}
    1. The __setitem__ method is overridden to store each value as a list with the value duplicated.
    2. When using [] (direct assignment), the overridden __setitem__ works as expected.
    3. However, methods like __init__ and update do not call the overridden method, resulting in the original behavior of the dict class.

  • Ignored __getitem__ Override in dict.update:

    class AnswerDict(dict):
    def __getitem__(self, key):
        return 42

ad = AnswerDict(a='foo')
print(ad['a'])  # Output: 42

d = {}
d.update(ad)
print(d['a'])  # Output: 'foo'
print(d)       # Output: {'a': 'foo'}
    1. The __getitem__ method is overridden to always return 42.
    2. Direct access (ad['a']) works as expected.
    3. However, when ad is used in dict.update(), the __getitem__ method of AnswerDict is bypassed, and the original value is used.

Why This Happens

  • Late Binding and Method Resolution: In Python, method resolution is supposed to start from the class of the instance (self), even when the method is called within a method of a superclass. However, the C implementation of built-in types often bypasses this dynamic dispatch, directly invoking methods of the built-in type rather than checking for overrides in subclasses. This breaks the principle of "late binding," where the exact method to be invoked is determined at runtime.

Best Practices: Using UserDict, UserList, and UserString

  • Recommended Approach:

    • Instead of subclassing built-in types directly (e.g., dict, list, str), it’s better to subclass from the corresponding classes in the collections module: UserDict, UserList, and UserString. These classes are implemented in Python and are designed to be easily extended.

  • Using UserDict:

    import collections

class DoppelDict2(collections.UserDict):
    def __setitem__(self, key, value):
        super().__setitem__(key, [value] * 2)

dd = DoppelDict2(one=1)
print(dd)  # Output: {'one': [1, 1]}

dd['two'] = 2
print(dd)  # Output: {'one': [1, 1], 'two': [2, 2]}

dd.update(three=3)
print(dd)  # Output: {'one': [1, 1], 'two': [2, 2], 'three': [3, 3]}
    • By using UserDict instead of dict, the overridden __setitem__ is respected by all methods, including __init__ and update, leading to consistent behavior.

Multiple Inheritance and Method Resolution Order (MRO)

The Diamond Problem

  • Multiple Inheritance: When a class inherits from more than one superclass, and those superclasses have methods with the same name, it can lead to naming conflicts known as the "diamond problem."

  • Example:

    class Root:
    def ping(self):
        print(f'{self}.ping() in Root')

    def pong(self):
        print(f'{self}.pong() in Root')

    def __repr__(self):
        cls_name = type(self).__name__
        return f'<instance of {cls_name}>'

class A(Root):
    def ping(self):
        print(f'{self}.ping() in A')
        super().ping()

    def pong(self):
        print(f'{self}.pong() in A')
        super().pong()

class B(Root):
    def ping(self):
        print(f'{self}.ping() in B')
        super().ping()

    def pong(self):
        print(f'{self}.pong() in B')

class Leaf(A, B):
    def ping(self):
        print(f'{self}.ping() in Leaf')
        super().ping()
    • Root provides basic methods ping, pong, and __repr__.
    • A and B inherit from Root and override ping and pong.
    • Leaf inherits from both A and B, overriding only ping.

Method Resolution Order (MRO)

  • MRO: Determines the order in which classes are checked for a method when it is called. Python uses the C3 linearization algorithm to calculate MRO.

  • Example:

    >>> Leaf.__mro__
(<class 'Leaf'>, <class 'A'>, <class 'B'>, <class 'Root'>, <class 'object'>)
    • The MRO for Leaf is Leaf → A → B → Root → object.

Cooperative Methods and super()

  • Cooperative Methods: Methods that call super() to ensure that all relevant methods in the MRO are activated.

  • Example of Method Calls:

    >>> leaf1 = Leaf()
>>> leaf1.ping()
<instance of Leaf>.ping() in Leaf
<instance of Leaf>.ping() in A
<instance of Leaf>.ping() in B
<instance of Leaf>.ping() in Root
    • Calling leaf1.ping() activates ping in Leaf, A, B, and Root, in that order, because each ping method calls super().ping().
    >>> leaf1.pong()
<instance of Leaf>.pong() in A
<instance of Leaf>.pong() in B
    • pong in Leaf uses A.pong, which calls super().pong() in B, but since B.pong does not call super(), Root.pong is not activated.

Impact of Class Declaration Order

  • Changing the Inheritance Order:

    • If Leaf is declared as Leaf(B, A) instead of Leaf(A, B), the MRO changes:

      >>> Leaf.__mro__
(<class 'Leaf'>, <class 'B'>, <class 'A'>, <class 'Root'>, <class 'object'>)

    • The activation order of methods changes, and this may cause some methods not to be called, depending on the behavior of super().

Dynamic Nature of super()

  • Dynamic MRO with super():

    • super() dynamically resolves methods based on the MRO at runtime.

    • Example:

      class U:
    def ping(self):
        print(f'{self}.ping() in U')
        super().ping()

class LeafUA(U, A):
    def ping(self):
        print(f'{self}.ping() in LeafUA')
        super().ping()

>>> leaf2 = LeafUA()
>>> leaf2.ping()
<instance of LeafUA>.ping() in LeafUA
<instance of LeafUA>.ping() in U
<instance of LeafUA>.ping() in A
<instance of LeafUA>.ping() in Root
      • The MRO of LeafUA is LeafUA → U → A → Root → object. super() in U.ping() and LeafUA.ping() activates methods in this order.
      • If LeafUA were defined as LeafUA(A, U), the order of activation would change, possibly skipping U.ping.

Cooperative Multiple Inheritance

  • Cooperation Required: For multiple inheritance to work properly, all methods in the hierarchy should call super() so that each class can participate in the method chain.
  • Non-Cooperative Methods: If a method does not call super(), it may prevent other methods in the MRO from being called, leading to subtle bugs.

Mixin Classes in Python

  • A mixin class is used to add functionality to other classes in a multiple inheritance scenario.
  • The key idea is that mixins are not designed to stand alone; they are combined with other classes to extend or customize behavior.
  • Unlike other languages with explicit support for mixins (e.g., Ruby, Rust, and C#), Python relies on conventions for their implementation.

Case-Insensitive Mappings: Example

  • Mixin Implementation: UpperCaseMixin adds case-insensitive key handling for mappings with string keys by converting keys to uppercase during insertion, lookup, and other operations.
import collections

def _upper(key):
    try:
        return key.upper()
    except AttributeError:
        return key

class UpperCaseMixin:
    def __setitem__(self, key, item):
        super().__setitem__(_upper(key), item)

    def __getitem__(self, key):
        return super().__getitem__(_upper(key))

    def get(self, key, default=None):
        return super().get(_upper(key), default)

    def __contains__(self, key):
        return super().__contains__(_upper(key))
  • Helper Function: _upper(key) ensures that string keys are converted to uppercase. If the key doesn’t have an upper method (i.e., non-string keys), the key remains unchanged.

Using the Mixin

  • Example Usage: UpperCaseMixin needs to be combined with a fully functional mapping class to work properly. The mixin modifies how keys are handled without implementing the full mapping behavior.
class UpperDict(UpperCaseMixin, collections.UserDict):
    """Dictionary with case-insensitive string keys."""
    pass

class UpperCounter(UpperCaseMixin, collections.Counter):
    """Specialized 'Counter' that uppercases string keys."""
    pass
  • Multiple Inheritance Order: UpperCaseMixin should be the first base class in the inheritance chain, ensuring its methods are prioritized over methods from the mapping class (e.g., UserDict, Counter) due to the method resolution order (MRO).

Doctests for UpperDict and UpperCounter

  • UpperDict (case-insensitive keys):
>>> d = UpperDict([('a', 'letter A'), (2, 'digit two')])
>>> list(d.keys())
['A', 2]
>>> d['b'] = 'letter B'
>>> 'b' in d  # Uppercases 'b' to check for presence.
True
>>> d['a'], d.get('B')
('letter A', 'letter B')
>>> list(d.keys())
['A', 2, 'B']
  • UpperCounter (case-insensitive character counting):
>>> c = UpperCounter('BaNanA')
>>> c.most_common()
[('A', 3), ('N', 2), ('B', 1)]

Key Considerations for Mixins

  • super(): All methods in the mixin use super() to ensure cooperation between multiple base classes. This ensures that other methods in the inheritance chain are called appropriately.
  • Order in Inheritance: MRO determines the method lookup order. For a mixin to function properly, it should come first in the inheritance list.
  • Brittleness of Inheritance: The mixin works by overriding key mapping methods. However, certain methods (like get) might need explicit implementation to ensure compatibility across different base classes (e.g., UserDict vs. Counter).

Best Practices

  • Minimalist Design: Keep mixins simple and focused on one responsibility (e.g., case-insensitive key handling). Avoid adding too much complexity to maintain ease of use and flexibility.
  • Dependency on super(): A mixin relies heavily on other base classes to provide core functionality. Therefore, its methods should delegate tasks to super().
  • Testing Across Multiple Classes: Ensure that the mixin works with different types of classes (e.g., UserDict, Counter) to catch potential issues in how different base classes handle method delegation.

Multiple Inheritance in the Real World

  • While multiple inheritance is not commonly used in many programming languages, it plays a significant role in Python, particularly with Abstract Base Classes (ABCs) and mixin classes.

1. ABCs as Mixins

  • In Python’s collections.abc module, multiple inheritance is widely used, and ABCs can act as mixins. They provide interface definitions along with mixin methods, which offer concrete implementations that can be reused.

    For instance:

    • collections.abc.MutableMapping defines the interface for mutable mappings and also provides mixin methods like __delitem__, pop, and update. These mixin methods are leveraged by other classes, such as UserDict, which is a part of Python’s collections module.

2. ThreadingMixIn and ForkingMixIn

  • ThreadingMixIn and ForkingMixIn from Python’s http.server and socketserver packages are key examples of multiple inheritance in real-world scenarios. These mixin classes add threading or forking capabilities to server classes.

ThreadingHTTPServer

  • The http.server.ThreadingHTTPServer class extends the regular HTTPServer and adds multi-threaded request handling using the ThreadingMixIn.
class ThreadingHTTPServer(socketserver.ThreadingMixIn, HTTPServer):
    daemon_threads = True
  • The key feature here is that ThreadingHTTPServer inherits from both ThreadingMixIn and HTTPServer. This enables it to process each HTTP request in a separate thread, making it suitable for handling multiple requests concurrently, like when web browsers pre-open sockets.

ThreadingMixIn Example

  • ThreadingMixIn adds support for multi-threaded request handling by overriding or extending some methods in the BaseServer class (inherited by HTTPServer).
class ThreadingMixIn:
    """Mixin class to handle each request in a new thread."""

    def process_request_thread(self, request, client_address):
        # Implementation to handle request in a new thread.
        pass

    def process_request(self, request, client_address):
        # Override to start a new thread for handling the request.
        pass

    def server_close(self):
        super().server_close()  # Calls the super() method to close the server.
        self._threads.join()    # Waits for all threads to finish.
  • Explanation:
    • process_request_thread is a new method introduced by ThreadingMixIn, which does not call super(), as it’s adding new functionality to handle each request in a new thread.
    • process_request overrides the method in BaseServer to start a thread and delegates the work to process_request_thread.
    • server_close overrides the method to stop accepting new requests and ensures all threads finish execution before shutting down the server.

ForkingMixIn

  • Similarly, ForkingMixIn is used in Unix-like systems for handling concurrent processes by forking a child process for each request using os.fork().

3. Django Generic Views and Mixins

  • In Django, views are responsible for handling HTTP requests and returning appropriate responses.
  • Django introduced class-based views (CBVs) in version 1.3, moving away from function-based views to offer greater flexibility and extensibility.
  • These CBVs are organized using base classes and mixins to modularize different responsibilities.

The Role of Class-Based Views in Django

  • Originally, Django's generic views (such as list and detail views) were functions that could not be extended easily. With the introduction of CBVs, these generic views became classes, which can now be extended and customized by users.

  • View: The base class for all views, providing core functionality like handling HTTP methods (e.g., GET, POST).

  • TemplateResponseMixin: A mixin that adds template rendering functionality to views that need it (e.g., ListView, DetailView).

Mixins in Django Generic Views

  • In Django’s CBVs, mixins are used to handle responsibilities like template rendering, pagination, and handling multiple objects.
  • For example, a ListView is used to display a list of objects, and much of its behavior is inherited from the MultipleObjectMixin, which handles things like pagination and querysets.
  • If a view does not need template rendering, you can use BaseListView—which inherits from View and MultipleObjectMixin but avoids the overhead of template machinery.

Example of Mixins in Django CBVs

  • UpperCaseMixin from a previous example works similarly to Django mixins, by modifying behavior when working with dictionary-like objects.
  • Django mixins follow this model, where they add or alter specific behaviors of a class without changing the core functionality of the view.
from django.views.generic import ListView
from myapp.models import Item

class ItemListView(ListView):
    model = Item
    template_name = 'item_list.html'
  • In this case, ListView aggregates the behavior of View, TemplateResponseMixin, and MultipleObjectMixin.
  • It handles database queries (via MultipleObjectMixin), template rendering (via TemplateResponseMixin), and dispatching HTTP requests (via View).

Multiple Inheritance in Django CBVs

  • Django’s CBV system uses multiple inheritance to combine functionality from base classes and mixins. For instance, a ThreadingMixIn could be added to a view to handle requests concurrently:
from django.http import HttpResponse
from django.views import View
from socketserver import ThreadingMixIn

class ThreadedView(ThreadingMixIn, View):
    def get(self, request):
        return HttpResponse('Threaded Response')
  • Here, the view inherits from both ThreadingMixIn and View. ThreadingMixIn handles the threading behavior, while View takes care of handling the HTTP request and response.

Advantages of Django's Class-Based Views

  • Modularity: CBVs are modular, allowing developers to mix and match behaviors using base classes and mixins.
  • Extensibility: Developers can extend or override specific behaviors without rewriting entire views.
  • Reduced Boilerplate: Common functionalities (like pagination or template rendering) are encapsulated in mixins, reducing the amount of code you need to write.

However, learning curve and opacity are often cited as drawbacks. Many developers prefer to use function-based views (FBVs) for their simplicity, while others feel that CBVs, though powerful, are harder to understand and debug.

Coping with Inheritance

Managing inheritance effectively in object-oriented programming can be challenging. The following tips help mitigate some of the common issues associated with inheritance, especially multiple inheritance:

Favor Composition over Inheritance

  • Reason: Composition allows for more flexible and maintainable designs. In contrast, inheritance tightly couples subclasses to superclasses, making changes harder to manage.
  • Example: Instead of having a widget in Tkinter inherit from all geometry managers (Pack, Place, Grid), a widget could hold a reference to a geometry manager and delegate layout tasks to it.

Understand Why Inheritance Is Used

  • Interface Inheritance: Creates a subtype representing an "is-a" relationship. Use abstract base classes (ABCs) for this.
  • Implementation Inheritance: Avoids code duplication through reuse, typically done with mixins.
  • Best Practice: Clearly distinguish between interface and implementation inheritance, using ABCs for the former and mixins for the latter.

Use ABCs to Define Interfaces

  • Tip: Use ABCs or typing.Protocol to define clear interfaces. This ensures the intent is explicit and avoids confusion.
  • Example: A class like collections.abc.MutableMapping defines the interface for a mapping but does not enforce any specific internal implementation.

Use Mixins for Code Reuse

  • Purpose: Mixins should be used to provide reusable methods without implying an "is-a" relationship. They should focus on providing specific behavior and should avoid managing internal state.
  • Naming: Use the Mixin suffix to make it clear that a class is designed for code reuse and not for instantiation.

Provide Aggregate Classes for Users

  • Definition: Aggregate classes combine ABCs or mixins in a way that makes it easier for users to employ multiple functionalities without manually combining them.
  • Example: Django’s ListView combines a base class and mixin to provide common functionality without additional code.

Subclass Only Classes Designed for Subclassing

  • Tip: Only subclass classes that are explicitly designed for it. Subclassing classes that are not designed to be extended can lead to unexpected behavior.
  • Documentation: Check class documentation and use cases to see if subclassing is expected or supported.
  • PEP 591: Python ≥ 3.8 introduces the @final decorator, which can be applied to methods or classes to indicate that they should not be overridden or subclassed.

Avoid Subclassing Concrete Classes

  • Reason: Concrete classes often have internal state that can easily be corrupted when methods are overridden. Overriding methods in concrete classes introduces risks, even if super() is used.
  • Best Practice: Scott Meyer’s principle from More Effective C++ advises that non-leaf classes should be abstract, meaning that they should not be instantiated or subclassed directly.

Tkinter: The Good, the Bad, and the Ugly

Tkinter, the Python interface to the Tk GUI toolkit, has been part of the standard library since Python 1.1. While it has stood the test of time, there are several areas where it violates many best practices in object-oriented design, especially with regard to inheritance.

The Good

  • Aggregate Classes: Tkinter provides aggregate classes like Widget, which integrate various functionalities such as geometry management, despite its overly complex structure. This aligns with the recommendation to provide aggregate classes that make it easier for users to use multiple behaviors without worrying about the inheritance structure.

  • Stability and Flexibility: Tkinter remains a stable and flexible GUI library, even though its design is outdated. Using tkinter.ttk, developers can achieve a modern look-and-feel with themed widgets, and original widgets like Canvas and Text are still powerful and versatile.

The Bad

  • Overuse of Inheritance: Tkinter heavily relies on deep inheritance trees, combining too many responsibilities in classes like Widget, which inherits from BaseWidget, Misc, Pack, Place, Grid, and more. This results in an unnecessarily complex design, violating the principle of favoring composition over inheritance.

  • Widget Class: Widget should ideally be an abstract base class (ABC) since it provides an interface without any methods of its own. Instead, it defines an overly broad interface by inheriting from multiple superclasses, leading to a cluttered and difficult-to-maintain design.

  • Misuse of Mixins: Classes like Misc and Wm are misused as "mixins" but are not properly designed as such. Misc, in particular, contains over 100 methods, many of which are unnecessary for most widgets (e.g., clipboard handling for buttons or scrollbars). This violates the best practice of using explicit mixins that offer specific, closely related methods.

The Ugly

  • Cluttered Inheritance: The extensive use of multiple inheritance makes Tkinter's class hierarchy difficult to navigate. For instance, tkinter.Button has 214 attributes due to its complex inheritance structure, making it overwhelming for developers to find the methods they need.

  • Monolithic Misc Class: Misc is a significant design flaw in Tkinter. It handles a wide range of unrelated responsibilities (e.g., clipboard, text selection, and timers), leading to unnecessary inheritance by all widgets. A better approach would be to split Misc into more focused, specialized mixins that are only inherited by relevant widgets.

Previous ch13.interface_protocols_abc Next ch16.operator_overloading