Chapter 2. Data Types and Data Structures

Arrow Data Types

• Data comes in various shapes and sizes, all requiring memory storage for processing.
• Polars uses the Arrow memory specification to handle a wide range of data types.
• This chapter covers:
  • An introduction to Apache Arrow, the library Polars uses for in-memory data storage.
  • A walkthrough of the available data types, with focus on complex ones.
  • An explanation of the structures Polars uses to work with these data types.

Apache Arrow Overview

• Arrow is described as a cross-language development platform for in-memory analytics.
• It defines a language-independent columnar memory format for flat and hierarchical data.
• Optimized for modern hardware (CPUs, GPUs) with efficient analytics operations.

Advantages of Apache Arrow

• Columnar format:
  • Enables data adjacency for sequential access or scans, optimizing reading of large data blocks.
  • Supports storing and reading data in large, contiguous chunks.
• Vectorization-friendly:
  • Allows the use of modern Single Instruction, Multiple Data (SIMD) operations to process multiple data points simultaneously.

Filing Cabinet Metaphor (Explaining Columnar vs. Row-Based Formats)

• Row-based format: Each drawer contains all the data for a single sale (e.g., customer, items, price, date).
  • Ideal for retrieving complete sales data in a bundled manner.
• Columnar format: Each drawer contains one category of data for all customers (e.g., sale prices).
  • Ideal for analytical queries, such as identifying the top 5 customers by sales.
  • Faster, as you only need to open drawers containing relevant categories (e.g., sales and customer names), rather than every drawer.
  • In analytical queries, however, it’s more common to look for specific parts of the sales dossier, instead of all of it.

Constant-Time Random Access

• Arrow provides O(1) random access, meaning access time remains constant regardless of data size.
• Efficiency: Arrow's format lets you know exactly where data is stored, avoiding unnecessary searches.
• This significantly improves performance, especially for operations needing access to specific data points in large datasets.

Arrow Language Implementations

• Arrow supports multiple popular languages, including:
  • C/GLib, C++, C#, Go, Java, JavaScript, Julia, MATLAB, Python, R, Ruby, and Rust.
• Implementation differences:
  • Some data types, such as Float16, may not be available in all languages.

Cross-Language Data Sharing

• Normally, different languages have unique ways of representing data in memory, requiring serialization/deserialization for cross-language communication.
• Arrow eliminates this step by allowing all supported languages to access a shared mutable dataset in a unified format.
• This approach is known as Inter Process Communication (IPC), which saves time by avoiding data format translation.

Polars and Rust

• Polars' core is written in Rust for performance benefits.
• By using the Arrow Rust implementation, Polars supports various data types, many of which come in multiple bit-sizes for memory optimization.

Handling Arbitrary Python Data

• When creating a DataFrame with Python, sometimes arbitrary Python objects (e.g., machine-learning models) need to be stored.
• Object data type:
  • Allows arbitrary Python objects to be included in a DataFrame.
  • However, data in an Object column cannot be processed using standard functions or optimizations.
  • Objects are treated as passengers in the DataFrame, participating in operations like joins but not in optimized calculations.
  • Recommendation: Avoid using Object data type unless necessary, as other data types offer better performance.

Internal Data Types

• Unknown data type:
  • Used only as an internal placeholder in documentation.
  • Not recommended for use in code.

Nested Data Types Overview

• Nested data types contain other data types and require specific arguments like element type and count.
• Polars supports three nested data types:
  • Array
  • List
  • Struct

Array Data Type

• Similar to a Numpy ndarray, containing elements of the same data type.
• Key characteristics:
  • Fixed length: All rows must have arrays of the same length.
  • Arguments: Takes two arguments:
    1. Width (number of elements).
    2. Data type of elements.

import polars as pl

array_df = pl.DataFrame(
    [
        pl.Series("array_1", [[1, 3], [2, 5]]),
        pl.Series("array_2", [[1, 7, 3], [8, 1, 0]]),
    ],
    schema={
        "array_1": pl.Array(shape=2, inner=pl.Int64),
        "array_2": pl.Array(shape=3, inner=pl.Int64),
    },
)
array_df

List Data Type

• Similar to Array but does not require the same length for every row.
• Key differences:
  • Rows can have variable lengths.
  • Elements must be of the same data type (unlike Python lists which can mix types).
  • Argument: Only specifies the data type contained.

list_df = pl.DataFrame(
    {
        "integer_lists": [[1, 2], [3, 4]],
        "float_lists": [[1.0, 2.0], [3.0, 4.0]],
    }
)
list_df

Struct Data Type

• Struct allows working with multiple columns as a single column, containing different data types.
• Use case: Enables Polars expressions to take multiple columns as input, treating them as a single Series.
• Key characteristics:
  • Multiple data types allowed in different fields, as long as rows match in structure.
  • Constructed using Python dictionaries.

rating_series = pl.Series(
    "ratings",
    [
        {"Movie": "Cars", "Theatre": "NE", "Avg_Rating": 4.5},
        {"Movie": "Toy Story", "Theatre": "ME", "Avg_Rating": 4.9},
    ],
)
rating_series

Handling Missing Values in Polars

• Missing values in Polars are always represented as null, even for numerical data types.
• Information about missing values is stored in the metadata of the Arrow array, while the validity bitmap stores whether a value is missing (0 if missing, 1 if present).

Checking for Missing Values

• Use methods like:
  • null_count() to count the number of missing values.
  • is_null() to identify missing values.

Example: DataFrame with Missing Values

df = pl.DataFrame(
    {
        "value": [None, 2, 3, 4, None, None, 7, 8, 9, None],
    },
)
print(df)

Filling Missing Values (Nulls)

1. Filling with a Single Value

print(
    df.with_columns(
        pl.col("value").fill_null(-1).alias("filled_with_lit")
    )
)

2. Filling with a Fill Strategy

• Fill strategies:
  • None: Do not fill.
  • forward: Fill with the previous non-null value.
  • backward: Fill with the next non-null value.
  • min/max: Fill with the column's minimum or maximum value.
  • mean: Fill with the mean value.
  • zero: Fill with 0.
  • one: Fill with 1.

print(
    df.with_columns(
        pl.col("value").fill_null(strategy="forward").alias("forward"),
        pl.col("value").fill_null(strategy="backward").alias("backward"),
        pl.col("value").fill_null(strategy="min").alias("min"),
        pl.col("value").fill_null(strategy="max").alias("max"),
        pl.col("value").fill_null(strategy="mean").alias("mean"),
        pl.col("value").fill_null(strategy="zero").alias("zero"),
        pl.col("value").fill_null(strategy="one").alias("one"),
    )
)

3. Filling with an Expression

print(
    df.with_columns(
        pl.col("value").fill_null(pl.col("value").mean()).alias("expression_mean")
    )
)

4. Filling with Interpolation

print(df.interpolate())

NaN (Not a Number) vs. Null in Polars

• NaN values are not considered missing data in Polars.
• NaN values are used for Float data types to represent invalid or undefined results.
• Functions like null_count() do not count NaN values.
• Use is_nan() and fill_nan() to work with NaN values specifically.

Data Structures in Polars

• Series:

  • A single column of data of the same data type.

• DataFrame:

  • A two-dimensional table with rows and columns.
  • Internally represented as a collection of Series of the same length.

• ChunkedArray:

  • A container class for a sequence of arrays of data.
  • Benefits of using ChunkedArrays:
    • Optimized memory management: Data is added to the existing object without needing to copy or garbage collect old data.
    • Parallel processing: Data is split into chunks, each of which can be processed by a different CPU core to maximize performance.

Rechunking

• Rechunking: The process of changing the chunk size of a ChunkedArray.

  • In eager mode, data is rechunked after reading since the user likely wants to analyze it multiple times, making rechunking worthwhile.
  • In lazy evaluation, the query optimizer decides when to rechunk for efficiency.

• Key point: Setting the rechunk parameter to True means two operations are happening (rechunking and the primary operation). This should be considered during benchmarking.

LazyFrame

• LazyFrame: A DataFrame that is evaluated lazily, meaning it holds no actual data until it is needed.
  • Characteristics:
    • Unlike a DataFrame, a LazyFrame contains no data in memory.
    • It consists of a query graph, which stores the computational steps necessary to get the final result.
    • Operations and transformations are only evaluated when required, providing opportunities for query optimization.

Data Type Conversion in Polars

• cast() function:

  • Converts the data type of a Series from its current type to a specified one.
  • Can significantly optimize memory usage by switching to more appropriate data types.

• Example: Converting string data to a numeric type for better memory efficiency.

string_df = pl.DataFrame({"id": ["10000", "20000", "30000"]})
print(f"Estimated size: {string_df.estimated_size('b')} bytes")
# Output: Estimated size: 15 bytes

int_df = string_df.select(pl.col("id").cast(pl.UInt16))
print(f"Estimated size: {int_df.estimated_size('b')} bytes")
# Output: Estimated size: 6 bytes

• Benefit: In this example, casting to a more suitable data type (UInt16) reduces memory usage by over 60%.

Casting Multiple Columns in a DataFrame

• You can cast all columns in a DataFrame to a single data type or use a mapping for more control.

Cast all columns to a single type

df = pl.DataFrame({
    "id": [10000, 20000, 30000],
    "value": [1.0, 2.0, 3.0],
    "value2": ["1", "2", "3"]
})
df.cast(pl.UInt16)

• All columns will be cast to UInt16.

Cast specific columns with a mapping

df.cast({"id": pl.UInt16, "value": pl.Float32, "value2": pl.UInt8})

Cast specific data types

df.cast({pl.Float64: pl.Float32, pl.String: pl.UInt8})

Use column selectors for casting

import polars.selectors as cs
df.cast({cs.numeric(): pl.UInt16})

Limitations of Casting

• Basic casting: Works well in simple cases but may fail when more complex transformations are needed.
• Advanced casting: Some data (e.g., DateTime strings) requires special methods to parse and cast correctly.