Ch 1. Getting Started with Polars

Differences Between Polars and Pandas

Polars Data Structures

• Series and DataFrame

  • Like Pandas, Polars has both a Series and DataFrame structure.
  • Unlike Pandas, the Series in Polars is rarely used.

• Column Representation

  • Instead of a Series, Polars represents a single column using an expression via pl.col().
  • This allows for lazy evaluation, improving performance by deferring computations until necessary.

• Eager vs. Lazy Evaluation

  • Eager (Materialized) DataFrame: Directly evaluates operations immediately.
  • Lazy DataFrame: Represents data that is not yet evaluated, optimizing execution.

• Loading DataFrames

  • pl.read_csv("file.csv") → Returns an eager dataframe.
  • pl.scan_csv("file.csv") → Returns a lazy dataframe.

Laziness in Polars

• When to Use Eager vs. Lazy DataFrames

  • Eager (Materialized) DataFrames: Common in exploratory data analysis.
  • Lazy DataFrames: Used in data engineering or when optimizing performance.

• Working with Lazy DataFrames

  • Requires minimal code changes:
    • Use pl.scan_csv() instead of pl.read_csv().
    • Append .collect() at the end to materialize the results.
  • Provides significant speed benefits, especially for large datasets.
  • Optimizes the order of operations to improve performance.

• Limitations of Lazy DataFrames

  • Some operations require materialization, as they need knowledge of the underlying data.
  • The query engine optimizes execution order but cannot handle operations that require immediate data evaluation.

• Checking Unsupported Operations in Lazy DataFrames

  df_ops = set(x for x in dir(pl.DataFrame()) if not x.startswith('_'))
  lazy_ops = set(x for x in dir(pl.LazyFrame()) if not x.startswith('_'))
  print(sorted(df_ops - lazy_ops))  # Shows operations missing in lazy mode

  • These missing operations require materialization of data before execution.

• Checking Shared Operations

  print(sorted(lazy_ops & df_ops))  # Shows operations available in both modes

  • Many operations are supported in both eager and lazy modes.
  • Oddly, exporting JSON is available in lazy mode but not in other formats.

Contexts & Expressions in Polars

• Expressions vs. Direct Operations

  • Unlike Pandas, Polars encourages working with expressions instead of directly manipulating a Series or DataFrame.
  • Expressions are evaluated in a context (e.g., filter, select, group_by).
  • This allows for lazy execution and optimizations by the query engine.

• Understanding pl.col()

  • pl.col() represents a column without directly accessing its data.
  • Operations on pl.col() build an execution plan that Polars optimizes before execution.

• Checking Available Operations on pl.col()

  col = pl.col('sample')
  col_ops = set(x for x in dir(col) if not x.startswith('_'))
  print(sorted(col_ops))  # Shows available operations on expressions

• Method Chaining for Efficient Data Manipulation

  • Expressions return new expressions, allowing for efficient chaining:

    (pl.col("sample")
     .cast(pl.Int32)
     .fill_null(pl.col("sample").mean())
     .clip(upper_bound=100)
     .sample(10)
     .mean()
    )

  • In lazy mode, this chain is optimized before execution, improving performance.

• Alternative Assignment Style

  • Instead of chaining, expressions can be assigned step by step:

    ex1 = pl.col("sample").cast(pl.Int64)
    ex2 = ex1.fill_null(ex1.mean())
    ex3 = ex2.clip(upper_bound=100)
    ex4 = ex3.sample(10)
    ex5 = ex4.mean()

  • This style improves readability by clearly breaking down each transformation.

• Using Attribute Syntax for Expressions

  • Columns can be accessed via attributes, but this is not recommended:

    pl.col.a_column  # Equivalent to pl.col("a_column")

  • Requires column names to be valid Python attribute names.

Reading CSV Files in Polars

• Loading a CSV File

  • Polars reads CSV files using pl.read_csv().

• Handling ZIP Files

  • Polars cannot read ZIP files directly.

  • Use the zipfile module to extract the CSV before reading:

    import zipfile
    
    url = '../../pylib/datasets/data/vehicles.csv.zip'
    with zipfile.ZipFile(url) as z:
        z.extractall('data/')
    
    path = 'data/vehicles.csv'
    df = pl.read_csv(path, null_values=['NA'])

• Understanding DataFrame Output

  • Displays number of rows and columns at the top.
  • Shows column names and data types
  • Uses ... (ellipsis) to indicate omitted columns when there are too many.

• Configuring Display Settings

  • Adjust the number of displayed columns using:

    pl.Config.set_tbl_cols(6)  # Set to 6 columns for readability

  • Instead of increasing the number of visible columns, filter or visualize data for better analysis.

Lazy CSV Reading in Polars

• Using pl.scan_csv() Instead of pl.read_csv()

  • pl.scan_csv() returns a lazy dataframe, meaning the data is not immediately loaded.
  • Instead of displaying data, it shows a query plan.

  lazy = pl.scan_csv(path, null_values=['NA'])
  print(lazy)

  • Output:

    naive plan: (run LazyFrame.explain(optimized=True) to see the optimized plan)
    Csv SCAN [data/vehicles.csv]
    PROJECT */83 COLUMNS
    

  • "Project" means selecting all columns, but no data has been read yet.

• Filtering and Selecting Columns in Lazy Mode

  print(
      lazy
      .filter((pl.col('year') >= 1990) & (pl.col('year') < 2000))  # Filter rows
      .select(['year', 'make', 'model'])  # Select specific columns
  )

  • Updated plan now includes a filter and a select operation:

    naive plan:
    SELECT [col("year"), col("make"), col("model")] FROM
    FILTER [(col("year")) >= (1990)] FROM
    FILTER [(col("year")) < (2000)] FROM
    Csv SCAN [data/vehicles.csv]
    PROJECT */83 COLUMNS
    

• Optimized Query Execution

  • Polars optimizes queries by only reading required columns and rows.

  • Run .explain() to see the optimized plan:

    print(
        lazy
        .filter((pl.col('year') >= 1990) & (pl.col('year') < 2000))
        .select(['year', 'make', 'model'])
        .explain()
    )

  • Optimized output:

    simple π 3/3 ["year", "make", "model"]
    Csv SCAN [data/vehicles.csv]
    PROJECT 3/83 COLUMNS
    SELECTION: [([(col("year")) < (2000)]) & ([(col("year")) >= (1990)])]
    

  • Now, only 3 out of 83 columns are read, significantly improving performance.

Data Type Inference and Manual Overrides in Polars

• Challenges with CSV Files

  • CSV files store all data as text, without type information.
  • Numeric data is not optimized, leading to larger file sizes and slower reading.
  • Better alternatives: Formats like Apache Parquet store data efficiently with proper types.

• Checking Data Types in Polars

  print(df.dtypes)  # Returns a list of column data types
  print(df.schema)  # Returns a dictionary mapping column names to types

  • By default, Polars assigns 64-bit types to numeric values, which may be excessive (e.g., year could be Int16).

• Selecting Relevant Columns for Analysis

  cols = [
      'year', 'make', 'model', 'displ', 'cylinders', 'trany', 'drive',
      'VClass', 'fuelType', 'barrels08', 'city08', 'highway08', 'createdOn'
  ]

  • Use .select() to extract specific columns:

    print(df.select(cols))  # Direct column selection
    print(df.select(pl.col(cols)))  # Using expressions for flexibility

• Examining Integer Columns Before Type Casting

  print(df
        .select(pl.col(cols))
        .select(pl.col(pl.Int64))  # Selects only integer columns
        .describe())  # Displays min/max values

  • Helps determine the smallest suitable integer type (e.g., year can be Int16, others can use UInt8).

• Optimizing Integer Data Types with .with_columns()

  print(df
        .select(pl.col(cols))
        .with_columns(
            pl.col('year').cast(pl.Int16),       # Reduce to 16-bit integer
            pl.col('cylinders').cast(pl.UInt8),  # Reduce to 8-bit unsigned integer
            pl.col('highway08').cast(pl.UInt8),
            pl.col('city08').cast(pl.UInt8)
        ))

  • Preserves existing columns while modifying types.
  • Smaller types reduce memory usage and improve performance.

• Handling Type Mismatch Errors

  • Attempting to cast year to Int8 (too small) will raise an error:

    print(df.with_columns(pl.col('year').cast(pl.Int8)))  # Likely to fail

  • Use .describe() to validate column ranges before casting.

Automatic Type Shrinking in Polars

• Using .shrink_dtype() for Numeric Columns

  • Automatically reduces the data type size of numeric columns:

    print(df.select(pl.col(cols).shrink_dtype()))

  • Helps reduce memory usage without manual type selection.

• Limitations & Caution

  • Does not use unsigned integers (e.g., city08 could be UInt8, but it may be cast as Int16).
  • May not always choose optimal types, potentially leading to inefficiencies.
  • Manual type inspection is recommended:
    • Run .shrink_dtype(), check results, then manually apply proper types.
    • Helps avoid unintended casting issues.

• Preferred Approach: Manual Type Casting

  • Instead of relying on automatic shrinking, explicitly define types:

    print(df.with_columns(
        pl.col('year').cast(pl.Int16),
        pl.col('cylinders').cast(pl.UInt8),
        pl.col('highway08').cast(pl.UInt8),
        pl.col('city08').cast(pl.UInt8)
    ))

  • Ensures data types are optimized and appropriate for the dataset.

Extracting Transmission Information from the trany Column

Problem: Two Pieces of Information in One Column

• The trany column contains both transmission type (automatic/manual) and number of gears (e.g., "6-speed").
• To analyze the data effectively, it's best to split this into two separate columns.

1. Identifying Automatic Transmissions

• Use .str.to_lowercase() to normalize capitalization.
• Use .str.contains('automatic') to check for the word "automatic".
• The result is a boolean column (True/False).

print(df
      .select(
          'trany',
          pl.col('trany')
          .str.to_lowercase()
          .str.contains('automatic')
          .alias('is_automatic')
      )
)

2. Adding is_automatic to the DataFrame

• Use .with_columns() to modify/add a new column.
• Use keyword arguments instead of .alias() for better readability.

print(df
      .with_columns(
          is_automatic=pl.col('trany')
          .str.to_lowercase()
          .str.contains('automatic')
      )
)

3. Handling Missing Values in is_automatic

• Use .fill_null(True) to assume missing values as automatic.

print(df
      .with_columns(
          is_automatic=pl.col('trany')
          .str.contains('Automatic')
          .fill_null(True)
      )
)

4. Extracting Number of Gears

• Use .str.extract(r'(\d+)') to extract digits from trany.
• Convert extracted values to UInt8 (as gear numbers are small).

print(df
      .select(
          num_gears=pl.col('trany')
          .str.extract(r'(\d+)')
          .cast(pl.UInt8)
      )
)

5. Checking Extracted Values

• Ensure the regex captures all expected values using .unique().

print(df
      .select(
          num_gears=pl.col('trany')
          .str.extract(r'(\d+)')
          .cast(pl.UInt8)
          .unique()
      )
)

6. Identifying Missing Values in num_gears

• Filter rows where num_gears is null.

print(df
      .select(
          num_gears=pl.col('trany')
          .str.extract(r'(\d+)')
          .cast(pl.UInt8)
      )
      .filter(pl.col('num_gears').is_null())
)

7. Filling Missing Gear Values with 6

• Many modern automatic transmissions default to 6-speed, so fill nulls with 6.

print(df
      .with_columns(
          is_automatic=pl.col('trany')
          .str.contains('Automatic')
          .fill_null(True),
          
          num_gears=pl.col('trany')
          .str.extract(r'(\d+)')
          .cast(pl.UInt8)
          .fill_null(6)
      )
)

8. Exploring Unique trany Values

• Use .group_by('trany').len() to count occurrences.
• Sort in descending order to identify most frequent values.

print(df
      .group_by('trany')
      .len()
      .sort('len', descending=True)
)

9. Using .value_counts() Instead of Grouping

print(df
      .select(
          pl.col('trany').value_counts(sort=True)
      )
)

• To flatten the output, use .unnest():

print(df
      .select(
          pl.col('trany').value_counts(sort=True)
      )
      .unnest('trany')
)

10. Finding Rows Where trany is Missing

print(df
      .filter(pl.col('trany').is_null())
      .select('year', 'make', 'model', 'VClass')
)

Final Result

After these steps, we now have:
✅ is_automatic: Boolean column indicating if the car has an automatic transmission.
✅ num_gears: Integer column representing the number of gears.

Instead of storing unstructured text, we now have structured, analyzable data! 🚀