Ch 19. Concurrency model in python

• Concurrency: Managing overlapping tasks in time; supported on single-core systems.
• Parallelism: Simultaneous execution on multiple cores; requires multicore systems.
• Relationship: Parallelism is a special case of concurrency.
• Challenge: Managing threads/processes is complex and expensive.
• Coroutines: Lightweight, but challenging to track; limited by Python’s GIL for CPU-heavy tasks.
• Python GIL: Restricts true parallelism in threads but can be bypassed with processes for CPU-bound tasks.
• Multiprocessing: Suitable for CPU-intensive tasks, avoiding GIL limitations by using multiple processes.
• Asyncio: Provides cooperative multitasking for I/O-bound tasks but struggles with CPU-bound tasks without yielding control.
• Primality Check Performance: Multiprocessing significantly speeds up CPU-bound tasks, while threading is ineffective due to GIL.
• Task Queues: Distributed task queues like Celery improve scalability and efficiency for long-running background tasks.

Concurrency is about dealing with lots of things at once.

Parallelism is about doing lots of things at once.

Not the same, but related.

One is about structure, one is about execution.

Concurrency provides a way to structure a solution to solve a problem that may (but not necessarily) be parallelizable.

—Rob Pike, co-inventor of the Go language

Concurrency vs. Parallelism

• Concurrency:

  • Definition: Managing multiple tasks at once by structuring a solution so tasks can be dealt with in overlapping time periods.
  • Key Aspect: Structure.
  • Example: Running multiple processes on a single-core CPU by switching between tasks (time-slicing).

• Parallelism:

  • Definition: Executing multiple tasks simultaneously.
  • Key Aspect: Execution.
  • Example: Running multiple tasks on a multi-core CPU, with each core executing a task at the same time.

Relationship Between Concurrency and Parallelism

• Parallelism is a special case of concurrency, meaning all parallel systems are concurrent, but not all concurrent systems are parallel.
• Concurrency provides a structural approach to solve problems, which may or may not be parallelizable.
• In practical systems, many tasks are run concurrently rather than in true parallelism due to hardware limitations.

Key Concepts

• Single-core systems: In older systems, multiple processes were managed concurrently, but only one could execute at any moment.
• Multi-core systems: Modern laptops or systems can handle more parallel tasks, but even then, concurrent management by the operating system is crucial for handling many processes effectively.

The Big Picture

Basic Challenge:

• Starting threads or processes is easy, but tracking and managing them is hard.

Synchronous Function Calls

• In a typical synchronous function call, the calling code is blocked until the function returns.
• The caller easily retrieves the result or handles exceptions using try/except.

Concurrent Execution

• When a task runs in a separate thread or process, several issues arise:
  • Completion status: You don’t automatically know when the task finishes.
  • Result retrieval: Getting the result or catching exceptions requires setting up a communication channel (e.g., message queues).

Resource Management

• Starting threads or processes is costly: You want to avoid starting one for a single task.
  • Instead, create worker threads or processes that wait in a loop for tasks, amortizing startup costs.
  • This introduces further complications:
    • Graceful termination: How do you stop a worker without disrupting ongoing tasks or leaving resources (like open files) unmanaged?

Coroutines as a Solution

• Coroutines:
  • Lightweight to start compared to threads or processes.
  • Using await, it's easy to get the return value, handle exceptions, and safely cancel them.
  • However, when coroutines are started by the asynchronous framework, they can be hard to track, similar to threads or processes.

Limitations

• Coroutines and threads are not ideal for CPU-intensive tasks because of Python’s Global Interpreter Lock (GIL).

A Bit of Jargon

• Concurrency:

  • Ability to handle multiple tasks at once, progressing one task at a time or in parallel. Can occur even on a single-core CPU, where tasks are interleaved (multitasking).

• Parallelism:

  • Simultaneous execution of multiple tasks. Requires multicore CPUs, multiple CPUs, GPUs, or distributed systems/multiple computers in a cluster.

• Execution Unit:

  • Objects that execute code concurrently with independent states and call stacks.
  • Python natively supports 3 kinds of execution units:
    • Processes
    • Threads
    • Coroutines

• Process:

  • A running instance of a program, with its own memory space and CPU time slice.
  • Preemptive multitasking:
    • the OS scheduler preempts — i.e., suspends — each running process periodically to allow other processes to run (or say, to share CPU time between processes) that a frozen process can’t freeze the whole system—in theory.
  • Processes communicate via pipes, sockets, or memory mapped files—all of which can only carry raw bytes.
  • Python objects must be serialized (converted) into raw bytes to pass from one process to another, which can be costly.

• Thread:

  • A lightweight execution unit within a process. When a process starts, it uses a single thread: the main thread.
  • Multiple threads share memory space, enabling data sharing but risking data corruption if not properly synchronized.
  • Also supports preemptive multitasking under the supervision of the OS scheduler.

• Coroutine:

  • A function that can suspend itself and resume later, allowing cooperative multitasking (explicitly yielding control with yield or await).
  • In Python, classic coroutines are built from generator functions, and native coroutines are defined with async def.
  • Python coroutines usually run within a single thread under the supervision of an event loop, also in the same thread.
  • Cooperative multitasking: Each coroutine must explicitly cede control with the yield or await keyword, so that another may proceed concurrently (but not in parallel).
    • This means that any blocking code in a coroutine blocks the execution of the event loop and all other coroutines—in contrast with the preemptive multitasking supported by processes and threads.
    • On the other hand, each coroutine consumes less resources than a thread or process doing the same job.

• Queue:

  • A data structure used for passing items between execution units, usually in FIFO (First In, First Out) order.
  • Different Python libraries (e.g., queue, multiprocessing, asyncio) provide different implementations of queues.
  • The queue and asyncio packages also include queues that are not FIFO: LifoQueue and PriorityQueue.

• Lock:

  • A mechanism to synchronize actions and prevent data corruption by ensuring only one execution unit accesses a shared resource at a time.
  • Also known as a mutex (mutual exclusion).

• Contention:

  • Dispute over a limited asset.
  • Occurs when multiple execution units compete for the same resource, such as a lock or CPU time. Can lead to delays as units wait for access.

Processes, Threads, and Python’s Infamous GIL (Global Interpreter Lock)

1. Python Interpreter as a Process:

   • Each Python interpreter runs as a separate process.
   • You can spawn additional processes using libraries like multiprocessing and concurrent.futures.
   • The subprocess library is used to run external programs as new processes, regardless of the programming language.

2. Single-Threaded Python Interpreter:

   • The Python interpreter uses a single thread to execute code and handle garbage collection.
   • Additional threads can be created using the threading or concurrent.futures libraries.

3. The Global Interpreter Lock (GIL):

   • Python uses the GIL to ensure that only one thread executes Python bytecode at a time.
   • This restricts true parallelism in Python threads, even on multi-core processors.

4. GIL Releasing Mechanism:

   • To avoid one thread monopolizing the GIL, Python's interpreter pauses the active thread every 5ms (by default) and releases the GIL.
   • Other threads can then attempt to acquire the GIL.

5. No Direct Control Over the GIL:

   • Regular Python code cannot control the GIL directly.
   • However, functions written in C or extensions using the Python/C API can release the GIL during time-consuming operations.

6. GIL and I/O Operations:

   • Functions that make system calls (I/O operations like disk and network access, time.sleep()) automatically release the GIL while waiting.
   • This minimizes GIL contention during I/O-bound tasks.

7. GIL-Free Native Threads:

   • Native threads launched by C extensions are not bound by the GIL, though they typically cannot modify Python objects.
   • They can still interact with the memory of objects that use the buffer protocol (e.g., bytearray, array.array, NumPy arrays).

8. Network Programming and GIL:

   • The GIL has minimal effect on network programming because I/O operations release the GIL.
   • Network tasks, which are I/O-bound, can benefit from multi-threading despite the GIL.

9. GIL and Compute-Intensive Tasks:

   • The GIL negatively impacts the performance of CPU-bound tasks when run across multiple threads.
   • For CPU-heavy work, sequential single-threaded execution is often faster and simpler.

10. Multiple Processes for CPU-Bound Work:

• To fully utilize multiple CPU cores, Python processes (not threads) should be used.
• This can be achieved using libraries like multiprocessing or concurrent.futures.ProcessPoolExecutor.

Concurrent "Hello World" Example

Spinner with Threads

• a spinner animation that runs while a slow computation is performed, showcasing how threads can run concurrently in Python. The example highlights the coordination of threads using the threading.Event object.
• spin function: It uses an infinite loop that cycles through the characters \|/-. The loop is interrupted when the done event is set, allowing the thread to exit gracefully.
• slow function: Simulates a slow computation by sleeping for 3 seconds, releasing the GIL so the spinner can continue running concurrently.

import itertools
import time
from threading import Thread, Event

def spin(msg: str, done: Event) -> None: 
    for char in itertools.cycle(r'\|/-'): 
        status = f'\r{char} {msg}' 
        print(status, end='', flush=True)
        if done.wait(.1):  # Exit if the done event is set
            break 
    blanks = ' ' * len(status)
    print(f'\r{blanks}\r', end='')  # Clear the status line

def slow() -> int:
    time.sleep(3)  # Simulate a long computation
    return 42

def supervisor() -> int:
    done = Event()  # Create an Event object to signal when the task is done
    spinner = Thread(target=spin, args=('thinking!', done))  # Create a thread for the spinner
    print(f'spinner object: {spinner}') 
    spinner.start()  # Start the spinner thread
    result = slow()  # Perform the slow task on main
    done.set()  # Signal the spinner to stop
    spinner.join()  # Wait for the spinner thread to finish
    return result

def main() -> None:
    result = supervisor() 
    print(f'Answer: {result}')

if __name__ == '__main__':
    main()

• supervisor function: Manages the spinner thread and the main computation. It starts the spinner in a separate thread, runs the slow function in the main thread, and signals the spinner to stop once the computation is done.
• main function: Runs the program by calling the supervisor.

Key Points

1. Concurrency via Threads: The spinner and the slow computation run concurrently in different threads.
2. Thread Coordination with Event: The done event coordinates the threads, allowing the spinner to stop cleanly when the computation finishes.
3. GIL and Threading: While the slow() function sleeps (a blocking call), the GIL is released, allowing the spinner thread to run smoothly without being blocked.

Summary

• This example illustrates how Python can run multiple threads concurrently, even with the GIL in place, by releasing it during blocking operations like time.sleep.
• Threads can be coordinated using synchronization primitives like Event, which allows for clean termination of concurrent tasks.

Spinner with Processes: multiprocessing

import itertools
import time
from multiprocessing import Process, Event
from multiprocessing import synchronize

def spin(msg: str, done: synchronize.Event) -> None:
    for char in itertools.cycle(r'\|/-'):
        status = f'\r{char} {msg}'
        print(status, end='', flush=True)
        if done.wait(.1):
            break
    blanks = ' ' * len(status)
    print(f'\r{blanks}\r', end='')  # Clear the status line

def slow() -> int:
    time.sleep(3)  # Simulate a long computation
    return 42

def supervisor() -> int:
    done = Event()  # Create an Event object to signal when the task is done
    spinner = Process(target=spin, args=('thinking!', done))  # Create a new process for the spinner
    print(f'spinner object: {spinner}')
    spinner.start()  # Start the spinner process
    result = slow()  # Perform the slow task
    done.set()  # Signal the spinner to stop
    spinner.join()  # Wait for the spinner process to finish
    return result

def main() -> None:
    result = supervisor()
    print(f'Answer: {result}')

if __name__ == '__main__':
    main()

1. Process Creation: A Process object is used instead of Thread to run the spinner in a separate process, making better use of CPU cores.
2. Event Object: multiprocessing.Event() returns a synchronize.Event object. The multiprocessing API mimics the threading API closely, but with subtle differences such as needing to import multiprocessing.synchronize for type hints in Mypy.

Key Points

1. Independent GILs: Each process has its own GIL, meaning Python code can run concurrently on multiple cores.
2. Simple API Switch: Converting from threading to multiprocessing is easy, thanks to similar APIs.

Spinner with Coroutines: asyncio

• Coroutines provide cooperative multitasking within a single thread. The event loop drives the coroutines by interleaving their execution, handling I/O operations asynchronously.

import asyncio

def main() -> None: 
    result = asyncio.run(supervisor()) 
    print(f'Answer: {result}')

async def supervisor() -> int: 
    spinner = asyncio.create_task(spin('thinking!')) 
    print(f'spinner object: {spinner}') 
    result = await slow()  # Wait for slow() coroutine to complete
    spinner.cancel()  # Stop the spinner once slow() is done
    return result

if __name__ == '__main__':
    main()

Key Concepts

1. asyncio.run: Starts the event loop to drive the coroutine supervisor.
2. Coroutines: Defined using async def, allowing cooperative multitasking.
3. asyncio.create_task: Schedules a coroutine (spin) for execution, returning a Task object that runs concurrently.
4. await: Waits for the result of the coroutine slow, allowing other coroutines (like spin) to run in the meantime.
5. cancel(): Stops the spinner once the slow task is finished.

import itertools

async def spin(msg: str) -> None:
    for char in itertools.cycle(r'\|/-'):
        status = f'\r{char} {msg}'
        print(status, end='', flush=True)
        try:
            await asyncio.sleep(0.1)  # Yield control to event loop for 0.1s
        except asyncio.CancelledError:
            break
    blanks = ' ' * len(status)
    print(f'\r{blanks}\r', end='')  # Clear the status line

async def slow() -> int:
    await asyncio.sleep(3)  # Simulate a long computation asynchronously
    return 42

Key Concepts

1. await asyncio.sleep(0.1): Allows the event loop to continue driving other coroutines (like supervisor) while the spinner sleeps for 0.1 seconds.
2. CancelledError: Raised when cancel() is called on the spinner task, gracefully stopping the spinner loop.
3. slow coroutine: Uses await asyncio.sleep(3) to simulate a 3-second delay, without blocking other coroutines.

Never use time.sleep(…) in asyncio coroutines

• The supervisor coroutine schedules the spin coroutine as a background task and waits for the slow coroutine to complete.

• The event loop manages both coroutines concurrently.

• Once slow completes, supervisor cancels the spinner task to stop the animation.

• To understand how blocking affects coroutines, you can replace await asyncio.sleep(3) with time.sleep(3) in the slow coroutine.

async def slow() -> int:
    time.sleep(3)  # This will block the entire event loop
    return 42

When you run this experiment:

1. The spinner never appears because time.sleep(3) blocks the event loop, preventing any other coroutines from running.
2. After 3 seconds, the program prints the result: Answer: 42, without showing the spinner.

E.g. It blocks the entire event loop, halting all coroutines. Instead, always use await asyncio.sleep(DELAY) to allow other tasks to run while waiting.

Supervisor Function Comparison

Threaded Supervisor Function

def supervisor() -> int:
    done = Event()
    spinner = Thread(target=spin, args=('thinking!', done))
    print('spinner object:', spinner)
    spinner.start()
    result = slow()
    done.set()
    spinner.join()
    return result

Asynchronous Supervisor Coroutine

async def supervisor() -> int:
    spinner = asyncio.create_task(spin('thinking!'))
    print('spinner object:', spinner)
    result = await slow()
    spinner.cancel()
    return result

• Thread vs Task:

  • Thread: Used in the threaded version to execute a callable (spin function).
  • Task: Created using asyncio.create_task() to run a coroutine.

• Invocation:

  • Thread: Must explicitly start the thread using spinner.start().
  • Task: Automatically scheduled once created with asyncio.create_task().

• Function vs Coroutine:

  • In the threaded version, slow() is a regular function.
  • In the asynchronous version, slow() is a coroutine, called with await.

• Completion and Cleanup:

  • Threaded version: Signals completion with done.set() and waits for the thread to join using spinner.join().
  • Async version: Cancels the task using spinner.cancel() and handles coroutine cancellation with CancelledError.

• Control Flow:

  • Threading: Thread execution is managed by the operating system's thread scheduler.
  • Asyncio: Control is explicitly yielded with await, ensuring cooperative multitasking.

• Termination:

  • Threading: No API to terminate a thread externally; use signals (e.g., done.set()).
  • Asyncio: Coroutines can be cancelled externally using Task.cancel().

• Synchronization:

  • Threading: Requires locks to protect critical sections due to potential interruptions.
  • Asyncio: Coroutines are synchronized by default—only one runs at a time, and control is yielded explicitly using await.

• Safety in Cancellation:

  • Coroutines can only be cancelled when suspended at an await expression, allowing for safe cleanup.

• Concurrency:

  • The asynchronous approach avoids the complexity of thread management (e.g., locks) and provides clearer control over program flow via explicit suspension points (await).

Side note: Greenlet Package

• Cooperative Multitasking: Enables lightweight coroutines, called greenlets, for concurrent execution.
• No Special Syntax: Unlike yield or await, greenlets don't require special syntax, making them easier to integrate into sequential codebases.
• Usage in SQLAlchemy: SQLAlchemy 1.4 ORM uses greenlets internally to provide an asynchronous API compatible with asyncio.

Gevent Library

• Monkey Patching: Gevent modifies Python's standard socket module to make it non-blocking by replacing parts of it with greenlets.
• Transparent Integration: Gevent allows for easy adaptation of sequential applications and libraries, such as database drivers, to handle concurrent network I/O.
• Widely Used: Several open-source projects, including the popular Gunicorn WSGI server, use Gevent for concurrent networking.

The Real Impact of the GIL

• Networking Code:

  • Threading: If the time.sleep(3) call in slow() is replaced with an HTTP client request, the spinner continues spinning. This is because well-designed network libraries release the GIL when waiting for the network.
  • Asyncio: Similarly, replacing await asyncio.sleep(3) with an HTTP request in slow() allows the spinner to keep spinning because asynchronous network libraries yield control back to the event loop while waiting.

• CPU-Intensive Code:

  • Example Function: is_prime(n) checks if a number is prime. When used in place of sleep(3) in different concurrency models, the behavior of the spinner changes.

What Happens When Replacing sleep(3) with is_prime(n)?

1. Multiprocessing (spinner_proc.py):

   • The spinner continues spinning while the parent process computes the primality test, as multiprocessing runs each process independently, bypassing the GIL.

2. Threading (spinner_thread.py):

   • Contrary to what might be expected, the spinner continues spinning.
   • This is because the GIL is released by the main thread running is_prime every 5 milliseconds (by GIL mechanism), allowing the secondary thread controlling the spinner to briefly wake up and update.
   • Key Point: This works well in this simple case (with only two threads), but with more CPU-bound threads, contention for the GIL would slow the program down.

3. Asyncio (spinner_async.py):

   • The spinner does not appear at all. When is_prime(n) is called in the coroutine, control doesn’t return to the event loop until the computation is finished. The spinner never gets a chance to run because the program flow is blocked.

Solution: Refactor for Cooperative Yielding

• Power Napping with sleep(0):

  • To keep the spinner alive in the async version, the is_prime function can be refactored as a coroutine that periodically yields control back to the event loop with await asyncio.sleep(0). This allows the event loop to continue processing other tasks.

  • Example:

    async def is_prime(n):
        if n < 2:
            return False
        if n == 2:
            return True
        if n % 2 == 0:
            return False
        root = math.isqrt(n)
        for i in range(3, root + 1, 2):
            if n % i == 0:
                return False
            if i % 100_000 == 1:
                await asyncio.sleep(0)
        return True

    • Effect: This keeps the spinner responsive but increases the total execution time (e.g., from 3.3s to 4.9s in the example).
    • Drawback: Although this keeps the event loop active, it slows down the entire program, as yielding control frequently adds overhead.

Conclusion

• Using await asyncio.sleep(0) is a temporary solution for keeping the event loop responsive during CPU-bound tasks, but it slows down the overall computation.
• Best Practice: For CPU-intensive tasks, delegate the work to another process (e.g., with asyncio.loop.run_in_executor) to avoid blocking the event loop entirely, improving both responsiveness and performance.

A Homegrown Process Pool

Sequential Primality Check (sequential.py)

• Purpose: Provides a performance baseline for comparing sequential, multiprocessing, and threading code for CPU-intensive tasks like primality testing.
• Input: A sample of 20 integers ranging from small numbers to large primes and composites (up to $10^{16} - 1$).
• Output Format:
  • Each number with a prime flag ('P' for prime).
  • Time taken for each number's primality check.
  • Total elapsed time for all checks.
• Sample Output:
  • Each line: [Number] [Prime Status] [Elapsed Time]
  • Total time is approximately the sum of individual times.

from time import perf_counter
from typing import NamedTuple

from primes import is_prime, NUMBERS

class Result(NamedTuple):  # <1>
    prime: bool
    elapsed: float

def check(n: int) -> Result:  # <2>
    t0 = perf_counter()
    prime = is_prime(n)
    return Result(prime, perf_counter() - t0)

def main() -> None:
    print(f'Checking {len(NUMBERS)} numbers sequentially:')
    t0 = perf_counter()
    for n in NUMBERS:  # <3>
        prime, elapsed = check(n)
        label = 'P' if prime else ' '
        print(f'{n:16}  {label} {elapsed:9.6f}s')

    elapsed = perf_counter() - t0  # <4>
    print(f'Total time: {elapsed:.2f}s')

if __name__ == '__main__':
    main()

• check function: Determines if a number is prime and returns a Result tuple with primality and elapsed time.
• main function: Loops over all numbers, calls check, and prints the result for each number along with total elapsed time.

Process-Based Primality Check (procs.py)

• Purpose: Uses multiple processes to parallelize primality checking across CPU cores.
• Speed: Roughly 4.2 times faster than the sequential version using 12 processes on a 6-core machine.
• Explanation of Time Differences:
  • Times for individual numbers vary due to parallel execution.
  • Overhead exists from process setup and inter-process communication.

import sys
from time import perf_counter
from typing import NamedTuple
from multiprocessing import Process, SimpleQueue, cpu_count  # <1>
from multiprocessing import queues  # <2>

from primes import is_prime, NUMBERS

class PrimeResult(NamedTuple):  # <3>
    n: int
    prime: bool
    elapsed: float

JobQueue = queues.SimpleQueue[int]  # <4>
ResultQueue = queues.SimpleQueue[PrimeResult]  # <5>

def check(n: int) -> PrimeResult:  # <6>
    t0 = perf_counter()
    res = is_prime(n)
    return PrimeResult(n, res, perf_counter() - t0)

def worker(jobs: JobQueue, results: ResultQueue) -> None:  # <7>
    while n := jobs.get():  # <8>
        results.put(check(n))  # <9>
    results.put(PrimeResult(0, False, 0.0))  # <10>

def start_jobs(
    procs: int, jobs: JobQueue, results: ResultQueue  # <11>
) -> None:
    for n in NUMBERS:
        jobs.put(n)  # <12>
    for _ in range(procs):
        proc = Process(target=worker, args=(jobs, results))  # <13>
        proc.start()  # <14>
        jobs.put(0)  # <15>

def main() -> None:
    if len(sys.argv) < 2:  # <1>
        procs = cpu_count()
    else:
        procs = int(sys.argv[1])

    print(f'Checking {len(NUMBERS)} numbers with {procs} processes:')
    t0 = perf_counter()
    jobs: JobQueue = SimpleQueue()  # <2>
    results: ResultQueue = SimpleQueue()
    start_jobs(procs, jobs, results)  # <3>
    checked = report(procs, results)  # <4>
    elapsed = perf_counter() - t0
    print(f'{checked} checks in {elapsed:.2f}s')  # <5>

def report(procs: int, results: ResultQueue) -> int: # <6>
    checked = 0
    procs_done = 0
    while procs_done < procs:  # <7>
        n, prime, elapsed = results.get()  # <8>
        if n == 0:  # <9>
            procs_done += 1
        else:
            checked += 1  # <10>
            label = 'P' if prime else ' '
            print(f'{n:16}  {label} {elapsed:9.6f}s')
    return checked

if __name__ == '__main__':
    main()

• Queues:
  • JobQueue: Queue for submitting numbers to worker processes.
  • ResultQueue: Queue for collecting results from worker processes.
• check function: Performs primality test and returns a PrimeResult with the number, primality status, and elapsed time.
• worker function: Fetches jobs from JobQueue, computes primality, and posts results to ResultQueue. Terminates when it receives a "poison pill" (0).
• start_jobs function:
  • Adds numbers to JobQueue.
  • Launches worker processes and sends a poison pill to terminate them after completion.
• main function:
  • Sets up queues and starts the worker processes.
  • Collects and displays results.
• report function:
  • Retrieves results from the queue, tracks processes done, and prints results.

Key Learnings

• Sequential vs. Parallel: Parallelism yields significant speedup, but there is still overhead that limits the theoretical speed improvement.
• Queues for Coordination: Queues are used to distribute jobs and collect results efficiently between processes.
• Process Management: The number of processes can be dynamically set based on the system's CPU count or via command-line arguments.

Thread-Based Nonsolution

• threads.py Overview: This version of the prime-checking program is similar to procs.py, but it uses threading instead of multiprocessing.

  • Code Similarity: The structure of threads.py closely mirrors procs.py, demonstrating the API similarities between threading and multiprocessing in Python.

• Performance Issues:

  • Global Interpreter Lock (GIL): Python's GIL restricts threads from running CPU-bound tasks like is_prime in parallel, making the threaded version slower than the sequential code.
  • Increased Threads: As the number of threads increases, the performance worsens due to:
    • CPU Contention: Threads compete for the CPU's limited processing resources.
    • Context Switching: Switching between threads involves costly operations:
      • Saving CPU registers.
      • Updating the program counter and stack pointer.
      • Invalidating CPU caches.
      • Possibly swapping memory pages.
    • These overheads further degrade performance for compute-intensive tasks.

Python in the Multicore World

• Historical Context:

  • In the past, software performance improved automatically as CPU clock speeds increased, a trend referred to as the "free lunch."
  • Herb Sutter's Article: In "The Free Lunch Is Over" (2005), Sutter highlighted the end of this trend as CPU manufacturers shifted from increasing clock speeds to hyper-threading and multicore architectures.

• Modern Challenges:

  • Since 2004, significant performance improvements can only be achieved by leveraging multiple cores or hyperthreading, which requires software to be written for concurrent execution.

• Python's Evolution:

  • Early Python (1990s): The Global Interpreter Lock (GIL) was introduced when single-core processors dominated, making Python's interpreter faster and easier to implement on a single core.
  • GIL Benefits: Simplified writing Python extensions using the Python/C API and improved performance in single-core execution.

• Python's Current Success:

  • Despite the GIL, Python remains successful in concurrent and parallel applications due to:
    • Libraries like multiprocessing and concurrent.futures.
    • Architectures and tools that work around CPython's GIL limitations to enable efficient multicore usage.

System Administration with Python

• Usage: Python is widely employed in managing large fleets of servers, routers, load balancers, and NAS devices. It is also a leading tool in software-defined networking (SDN) and ethical hacking.
• Cloud Service Providers: Major cloud providers offer extensive support for Python via libraries and tutorials.
• Automation: Python scripts are used to automate configuration tasks on remote machines. These tasks are generally I/O-bound, making threads or coroutines ideal.
• Tools and Libraries:
  • concurrent.futures: Simplifies concurrent operations, such as running commands on multiple remote machines.
  • Ansible, Salt, Fabric: Python-based tools for managing server clusters.
  • AsyncIO: Popular for high-performance system administration tasks, embraced by companies like Facebook for coroutines and asynchronous programming.

Data Science and Scientific Computing with Python

• Compute-Intensive Tasks: Python is highly popular in data science and scientific computing, benefiting from a rich ecosystem of numeric libraries written in performant languages like C, C++, Fortran, and Cython.
• Leveraging Multicore and GPUs: Libraries in this space often take advantage of multicore CPUs, GPUs, and distributed parallel computing.
• Key Tools in the Python Ecosystem:
  • Jupyter Notebook & JupyterLab: Browser-based interfaces for running analytics, supporting multiple languages integrated through ZeroMQ.
  • TensorFlow & PyTorch: The leading deep learning frameworks, written in C++ and optimized for multicore machines, GPUs, and clusters. Both have strong Python support.
  • Dask: A parallel computing library for managing work across local processes or clusters, with APIs resembling NumPy, pandas, and scikit-learn. Dask integrates with Jupyter and Bokeh for real-time visualization and dashboarding.

Server-Side Web/Mobile Development with Python

• Widespread Usage: Python powers backend applications for major platforms like Google, YouTube, Dropbox, Instagram, Quora, and Reddit, supporting millions of users.
• Horizontal Scaling: At web scale, Python applications are designed with architectures that allow horizontal scaling, distributing workloads across multiple systems.
• Avoid Overengineering: As per Thoughtworks Technology Radar, many teams unnecessarily adopt complex architectures meant for "web-scale" without needing them, leading to complications.

Python Web Systems

• Application Caches: Tools like memcached, Redis, and Varnish speed up data access.
• Databases:
  • Relational: PostgreSQL, MySQL.
  • Document: MongoDB, CouchDB.
• Full-text Indexes: Elasticsearch, Solr for text search.
• Message Queues: RabbitMQ, Redis for task delegation and messaging.

WSGI Application Servers

• WSGI (Web Server Gateway Interface): Standard API allowing Python web applications to receive HTTP requests and send responses.

• Popular WSGI Servers:

  • mod_wsgi: Commonly used with Apache, well-maintained and supports Docker.
  • uWSGI: Offers extensive functionality but is complex to configure.
  • Gunicorn: A simpler, lightweight choice often paired with NGINX.
  • NGINX Unit: Supports multiple languages, including Python.

  All these servers maximize CPU utilization by running multiple processes to handle requests in parallel.

ASGI—Asynchronous Server Gateway Interface

• ASGI: Successor to WSGI, designed for asynchronous frameworks like aiohttp, Sanic, and FastAPI, supporting coroutines (async/await).
• Use Cases: Ideal for WebSockets and HTTP long polling, enabling more efficient handling of real-time communication.

Distributed Task Queues

• Problem: Some web requests require lengthy actions (e.g., sending emails, generating PDFs), which can slow down application performance.
• Solution: Use distributed task queues like Celery and RQ to offload these tasks, allowing quick responses to user requests.
  • Producers (e.g., a Django view) generate jobs and place them in a queue.
  • Consumers (worker processes) handle tasks asynchronously, often running on separate machines.
• Benefits:
  • Background processing: Tasks can run after a web request finishes.
  • Retry mechanisms: Ensures tasks complete with retry logic.
  • Horizontal scalability: Easily add more workers to handle increased task load.