Chapter 4. Reading and Writing Data

Reading CSV Files in Polars

penguins = pl.read_csv("data/penguins.csv")

Common Arguments for pl.read_csv()

Common CSV Parsing Issues

1. Handling Missing Values (Null Values)

   penguins = pl.read_csv("data/penguins.csv", null_values="NA")
   print(penguins)

   • This will correctly mark "NA" entries in the input as missing data.

2. Changing the Delimiter

   penguins = pl.read_csv("data/penguins.csv", separator=";")

   • If your CSV file uses a delimiter other than a comma (e.g., semicolon ;, tab \t), you can specify the delimiter using the separator argument:

3. Skipping Initial Rows

   penguins = pl.read_csv("data/penguins.csv", skip_rows=2)

   • Sometimes the dataset contains metadata or comments at the top. You can skip rows before reading the data:

4. Selecting Specific Columns

   penguins = pl.read_csv("data/penguins.csv", columns=["species", "island"])

   • To load only specific columns by name or index, use the columns argument:

5. Handling Encoding Issues

   penguins = pl.read_csv("data/penguins.csv", encoding="utf8-lossy")

   • CSV files are typically encoded in UTF-8, but if your file has a different encoding, you can specify it. If you’re unsure of the encoding or need to handle invalid characters, use utf8-lossy, which replaces invalid characters with �.

Parsing Missing Values Correctly

• To correctly parse missing values in a CSV file using Polars, you can explicitly specify the values that should be treated as null using the null_values argument in the pl.read_csv() function.

1. Handling Missing Values

• By default, Polars interprets empty strings as null values, but in your dataset, other values like "NA" or "None" may also represent missing data. You need to provide these values explicitly to Polars.

  # Load the CSV and specify "NA" as null values
  penguins = pl.read_csv("data/penguins.csv", null_values="NA")

• In this example, "NA" values will now be parsed as null.

2. Handling Multiple Missing Value Representations

• If your dataset uses multiple representations for missing values (e.g., "NA", "None", "N/A"), you can pass a list of values to null_values:

  penguins = pl.read_csv("data/penguins.csv", null_values=["NA", "None", "N/A"])

3. Verifying Correct Parsing of Missing Values

• Once you've read in the dataset, you can verify if all missing values have been parsed correctly by using the null_count() method. This method counts the number of null values in each column.

  # Get the count of null values in each column
  null_counts = penguins.null_count()
  
  # Transpose the output for better readability
  null_counts.transpose(include_header=True, column_names=["null_count"])

Reading Files with Encodings Other than UTF-8

• When reading files with encodings other than UTF-8 in Polars, it's essential to correctly identify the encoding to avoid data corruption or errors during reading.
• Polars assumes UTF-8 by default, but many CSV files, especially those with non-Latin characters, may use other encodings like EUC-JP, ISO-8859-1, or Shift-JIS.

Key Steps to Handle Non-UTF-8 Encodings

1. Attempting to Read a File with UTF-8 Assumptions If you try to read a file that is not UTF-8 encoded, Polars may raise an error similar to:

   ComputeError: could not parse `����` as dtype `str` at column 'name' (column number 1)

   This indicates that Polars encountered invalid UTF-8 byte sequences. In such cases, manually specifying the correct encoding is necessary.

2. Using a Known Encoding

   If you already know the encoding (for example, if you expect the file to be encoded in EUC-CN), you can specify it using the encoding argument in the pl.read_csv() function:

   penguins = pl.read_csv("data/directors.csv", encoding="EUC-CN")

   However, as demonstrated earlier, guessing the encoding without verification may lead to incorrect character interpretation.

3. Using the chardet Package for Encoding Detection

   Instead of guessing the encoding, you can use the chardet package to automatically detect the encoding. chardet analyzes the file and provides the most likely character encoding.

   Here's how you can use chardet to detect the encoding of a file:

   import chardet
   
   def detect_encoding(filename: str) -> str:
       """Return the most probable character encoding for a file."""
       with open(filename, "rb") as f:
           raw_data = f.read()
           result = chardet.detect(raw_data)
           return result["encoding"]
   
   # Detect the encoding of the CSV file
   encoding = detect_encoding("data/directors.csv")
   print(encoding)  # For example: 'EUC-JP'

   Once the encoding is detected, you can pass it to Polars:

   penguins = pl.read_csv("data/directors.csv", encoding="EUC-JP")

Reading Excel Spreadsheets

• Polars supports reading Excel spreadsheets through the pl.read_excel() function, which uses the xlsx2csv package to convert the spreadsheet into CSV before parsing it into a Polars DataFrame.
• This is useful for working with data in .xlsx format, which is widely used in business and other contexts.

Basic Reading

To read this file into a Polars DataFrame, you can simply use:

import polars as pl

songs = pl.read_excel("data/top2000-2023.xlsx")
print(songs)

shape: (2_001, 4)
┌─────────┬───────────────────────┬────────────────┬──────┐
│ positie │ titel                 │ artiest        │ jaar │
│ ---     │ ---                   │ ---            │ ---  │
│ i64     │ str                   │ str            │ i64  │
╞═════════╪═══════════════════════╪════════════════╪══════╡
│ null    │ null                  │ null           │ null │
│ 1       │ Bohemian Rhapsody      │ Queen          │ 1975 │
│ ...     │ ...                    │ ...            │ ...  │
└─────────┴───────────────────────┴────────────────┴──────┘

Handling Header Issues

• In the example above, the first row contains only null values because the spreadsheet's header spans two rows.
• You can fix this by skipping the first row after the header using the read_options argument:

songs_fixed = pl.read_excel(
    "data/top2000-2023.xlsx", read_options={"skip_rows_after_header": 1}
)
print(songs_fixed)

Output:

shape: (2_000, 4)
┌─────────┬───────────────────────┬────────────────┬──────┐
│ positie │ titel                 │ artiest        │ jaar │
│ ---     │ ---                   │ ---            │ ---  │
│ i64     │ str                   │ str            │ i64  │
╞═════════╪═══════════════════════╪════════════════╪══════╡
│ 1       │ Bohemian Rhapsody      │ Queen          │ 1975 │
│ ...     │ ...                    │ ...            │ ...  │
└─────────┴───────────────────────┴────────────────┴──────┘

Common Arguments for pl.read_excel()

Polars provides several options to customize how Excel files are read, listed below.

Example: Reading a Specific Sheet by Name

• If the Excel file contains multiple sheets and you want to load a specific one by name, use the sheet_name argument:

songs_sheet2 = pl.read_excel("data/top2000-2023.xlsx", sheet_name="Sheet2")

Limitations

• File Format: Polars only supports .xlsx files. If you need to work with other Excel file formats like .xls, .xlsm, or .ods, consider using Pandas (pd.read_excel()), then convert the resulting DataFrame into Polars.

Working with Multiple Files

• If you have data spread across multiple files with the same format and schema, Polars provides efficient ways to read them all at once.
• You can use globbing patterns to match filenames or manually list and load files using pl.concat().

Scenario

• You have daily stock data for companies like ASML, NVIDIA, and TSM, each stored in separate CSV files for different years. The files are organized in directories like data/stock/<symbol>/<year>.csv.

1. Using a Globbing Pattern to Read Files

• If you want to read all CSV files for NVIDIA from 2010 to 2019, you can use a globbing pattern that matches those specific filenames:

  import polars as pl
  
  # Reading all NVIDIA stock data between 2010 and 2019
  nvda_data = pl.read_csv("data/stock/nvda/201[0-9].csv")
  print(nvda_data)

  shape: (2_516, 8)
  ┌────────┬────────────┬───────────┬───┬───────────┬───────────┬──────────┐
  │ symbol │ date       │ open      │ … │ close     │ adj close │ volume   │
  │ ---    │ ---        │ ---       │   │ ---       │ ---       │ ---      │
  │ str    │ str        │ f64       │   │ f64       │ f64       │ i64      │
  ╞════════╪════════════╪═══════════╪═══╪═══════════╪═══════════╪══════════╡
  │ NVDA   │ 2010-01-04 │ 4.6275    │ … │ 4.6225    │ 4.24115   │ 80020400 │
  │ NVDA   │ 2010-01-05 │ 4.605     │ … │ 4.69      │ 4.303082  │ 72864800 │
  │ …      │ …          │ …         │ … │ …         │ …         │ …        │
  └────────┴────────────┴───────────┴───┴───────────┴───────────┴──────────┘
  

2. Reading All CSV Files in Subdirectories

• To read all CSV files in the data/stock directory, regardless of the company or year, use a double asterisk ** for matching across subdirectories:

  all_stocks = pl.read_csv("data/stock/*/*.csv")
  print(all_stocks)

  shape: (18_476, 8)
  ┌────────┬────────────┬────────────┬───┬────────────┬────────────┬──────────┐
  │ symbol │ date       │ open       │ … │ close      │ adj close  │ volume   │
  │ ---    │ ---        │ ---        │   │ ---        │ ---        │ ---      │
  │ str    │ str        │ f64        │   │ f64        │ f64        │ i64      │
  ╞════════╪════════════╪════════════╪═══╪════════════╪════════════╪══════════╡
  │ ASML   │ 1999-01-04 │ 11.765625  │ … │ 12.140625  │ 7.5722     │ 1801867  │
  │ NVDA   │ 2010-01-04 │ 4.6275     │ … │ 4.6225     │ 4.24115    │ 80020400 │
  │ …      │ …          │ …          │ … │ …          │ …          │ …        │
  └────────┴────────────┴────────────┴───┴────────────┴────────────┴──────────┘
  

3. Manually Listing Files

• If you can't use a globbing pattern, you can manually create a list of filenames and read them. For example, to read ASML stock data only from leap years:

  import calendar
  import polars as pl
  
  # Manually constructing file names for leap years
  filenames = [
      f"data/stock/asml/{year}.csv"
      for year in range(1999, 2024)
      if calendar.isleap(year)
  ]
  
  # Reading and concatenating the data from these files
  asml_leap_years = pl.concat([pl.read_csv(f) for f in filenames])
  print(asml_leap_years)

  shape: (1_512, 8)
  ┌────────┬────────────┬────────────┬───┬────────────┬────────────┬─────────┐
  │ symbol │ date       │ open       │ … │ close      │ adj close  │ volume  │
  │ ---    │ ---        │ ---        │   │ ---        │ ---        │ ---     │
  │ str    │ str        │ f64        │   │ f64        │ f64        │ i64     │
  ╞════════╪════════════╪════════════╪═══╪════════════╪════════════╪═════════╡
  │ ASML   │ 2000-01-03 │ 43.875     │ … │ 43.640625  │ 27.218985  │ 1121600 │
  │ …      │ …          │ …          │ … │ …          │ …          │ …       │
  └────────┴────────────┴────────────┴───┴────────────┴────────────┴─────────┘
  

Key Points

• Globbing Patterns: Use wildcards like * or ** to match multiple files in directories.
• Manual Listing: If globbing isn't suitable, manually list filenames and use pl.concat() to merge the DataFrames.
• Efficiency: Polars can efficiently handle reading multiple files and concatenating large datasets in one go.

Reading Parquet Files in Polars

• Parquet is highly optimized for reading large datasets.
• The Parquet format is an efficient columnar storage file format commonly used in big data processing. It offers several advantages over row-based formats like CSV and Excel, particularly in terms of speed, compression, and support for complex data structures.

Key Advantages of Parquet

• Efficient columnar format: Only the columns requested are read, optimizing performance.
• Supports compression: Reduces storage and speeds up data transfer.
• Supports nested data: Can store complex structures like arrays, structs, and more.
• Schema inclusion: Parquet files include metadata that describes the schema, avoiding issues seen in formats like CSV.

Example: Reading NYC Taxi Trip Data

• You can read multiple Parquet files at once using globbing patterns, which can match multiple files by filename patterns.

  import polars as pl
  
  # Reading multiple parquet files using a globbing pattern
  trips = pl.read_parquet("data/taxi/yellow_tripdata_*.parquet")
  print(trips)
  

  
  shape: (39_656_098, 19)
  ┌──────────┬──────────────────────┬───┬──────────────────────┬─────────────┐
  │ VendorID │ tpep_pickup_datetime │ … │ congestion_surcharge │ airport_fee │
  │ ---      │ ---                  │   │ ---                  │ ---         │
  │ i64      │ datetime[ns]         │   │ f64                  │ f64         │
  ╞══════════╪══════════════════════╪═══╪══════════════════════╪═════════════╡
  │ 1        │ 2022-01-01 00:35:40  │ … │ 2.5                  │ 0.0         │
  │ 1        │ 2022-01-01 00:33:43  │ … │ 0.0                  │ 0.0         │
  │ 2        │ 2022-01-01 00:53:21  │ … │ 0.0                  │ 0.0         │
  │ …        │ …                    │ … │ …                    │ …           │
  │ 2        │ 2022-12-31 23:46:00  │ … │ null                 │ null        │
  └──────────┴──────────────────────┴───┴──────────────────────┴─────────────┘
  
  

Common Arguments for pl.read_parquet()

Example: Select Specific Columns

• If you only need certain columns from the dataset, you can pass a list of column names to the columns argument to optimize memory and performance:

  # Selecting specific columns while reading the Parquet file
  trips_selected_columns = pl.read_parquet(
      "data/taxi/yellow_tripdata_*.parquet",
      columns=["VendorID", "tpep_pickup_datetime", "passenger_count"]
  )
  print(trips_selected_columns)

  shape: (39_656_098, 3)
  ┌──────────┬──────────────────────┬────────────────┐
  │ VendorID │ tpep_pickup_datetime │ passenger_count │
  │ ---      │ ---                  │ ---            │
  │ i64      │ datetime[ns]         │ i64            │
  ╞══════════╪══════════════════════╪════════════════╡
  │ 1        │ 2022-01-01 00:35:40  │ 1              │
  │ 1        │ 2022-01-01 00:33:43  │ 2              │
  │ …        │ …                    │ …              │
  └──────────┴──────────────────────┴────────────────┘
  

• This reduces the amount of data loaded into memory, which is crucial when working with very large datasets.

Reading JSON and NDJSON in Polars

JSON

• JSON (JavaScript Object Notation) is a popular text format that can represent nested structures, which makes it flexible and widely used for APIs, NoSQL databases, and configurations.
• However, when working with complex structures like nested objects and arrays, reading JSON files into a tabular format such as a DataFrame can be challenging.

Example: Reading a Nested JSON File

• Let’s consider an example where we read the file data/pokedex.json containing information about Pokémon. Here's how to approach this using Polars:

  # Reading the JSON file
  pokedex = pl.read_json("data/pokedex.json")
  print(pokedex)

  shape: (1, 1)
  ┌──────────────────────────────────────────────────────────────────────────────┐
  │ pokemon                                                                      │
  │ ---                                                                          │
  │ list[struct[17]]                                                             │
  ╞══════════════════════════════════════════════════════════════════════════════╡
  │ [{1,"001","Bulbasaur","http://www.serebii.net/pokemongo/pokemon/001.png",["G │
  └──────────────────────────────────────────────────────────────────────────────┘
  

Flattening the JSON Structure

• To work with the data in a more tabular format, we can explode and unnest the nested structure:

  # Exploding and unnesting the JSON structure
  pokedex_flat = (
      pokedex.explode("pokemon")   # Flatten the 'pokemon' list
          .unnest("pokemon")    # Turn keys into columns
          .select("id", "name", "type", "height", "weight")  # Selecting some columns
  )
  print(pokedex_flat)

  shape: (151, 5)
  ┌─────┬────────────┬──────────────────────┬────────┬──────────┐
  │ id  │ name       │ type                 │ height │ weight   │
  │ --- │ ---        │ ---                  │ ---    │ ---      │
  │ i64 │ str        │ list[str]            │ str    │ str      │
  ╞═════╪════════════╪══════════════════════╪════════╪══════════╡
  │ 1   │ Bulbasaur  │ ["Grass", "Poison"]  │ 0.71 m │ 6.9 kg   │
  │ 2   │ Ivysaur    │ ["Grass", "Poison"]  │ 0.99 m │ 13.0 kg  │
  │ …   │ …          │ …                    │ …      │ …        │
  └─────┴────────────┴──────────────────────┴────────┴──────────┘
  

NDJSON (Newline Delimited JSON)

• NDJSON is a format where each line contains a separate JSON object.
• It’s efficient for streaming and large datasets because it allows processing line-by-line.

Example: Reading an NDJSON File

# Reading the NDJSON file
wikimedia = pl.read_ndjson("data/wikimedia.ndjson")
print(wikimedia)

shape: (100, 20)
┌─────────────────────┬─────────────────────┬───┬────────┬─────────────────────┐
│ $schema             │ meta                │ … │ wiki   │ parsedcomment       │
│ ---                 │ ---                 │   │ ---    │ ---                 │
│ str                 │ struct[9]           │   │ str    │ str                 │
╞═════════════════════╪═════════════════════╪═══╪════════╪═════════════════════╡
│ /mediawiki/recentc… │ {"https://en.wikip… │ … │ enwiki │ <span dir="auto"><… │
│ …                   │ …                   │ … │ …      │ …                   │
└─────────────────────┴─────────────────────┴───┴────────┴─────────────────────┘

Flattening NDJSON Data

Just like with JSON, we can unnest the structure to work with it in a more tabular form:

# Unnesting the 'meta' column and selecting some fields
wikimedia_flat = (
    wikimedia.rename({"id": "edit_id"})   # Rename 'id' to avoid conflicts
           .unnest("meta")                # Unnest the 'meta' struct
           .select("timestamp", "title", "user", "comment")  # Select columns
)
print(wikimedia_flat)

Output:

shape: (100, 4)
┌────────────┬────────────────────┬────────────────────┬────────────────────┐
│ timestamp  │ title              │ user               │ comment            │
│ ---        │ ---                │ ---                │ ---                │
│ i64        │ str                │ str                │ str                │
╞════════════╪════════════════════╪════════════════════╪════════════════════╡
│ 1690617099 │ EFL Championship   │ 87.12.215.232      │ /* League champio… │
│ 1690617102 │ Lim Sang-choon     │ Preferwiki         │                    │
│ …          │ …                  │ …                  │ …                  │
└────────────┴────────────────────┴────────────────────┴────────────────────┘

Summary of Common Arguments for Reading JSON & NDJSON

Other File Formats Supported by Polars

Polars supports several formats beyond the commonly used CSV, Parquet, and JSON. These include Arrow IPC (Feather), Apache Avro, Delta Lake Tables, and PyArrow Datasets.

1. Arrow IPC (Feather Version 2)

To read and write Arrow IPC (Feather v2) files, you can use pl.read_ipc() and pl.write_ipc():

# Reading Arrow IPC
df = pl.read_ipc("data/file.feather")
print(df)

# Writing to Arrow IPC
df.write_ipc("data/output.feather")

2. Apache Avro

To read Apache Avro files, Polars provides pl.read_avro():

# Reading Avro
df = pl.read_avro("data/file.avro")
print(df)

3. Delta Lake Tables

For working with Delta Lake tables, Polars supports pl.read_delta():

# Reading Delta Lake tables
df = pl.read_delta("data/delta_table")
print(df)

4. PyArrow Datasets

Polars can also read from PyArrow Datasets using pl.scan_pyarrow_dataset(). This is useful for querying large datasets partitioned across multiple files.

import pyarrow.dataset as ds

# Scanning PyArrow Dataset
dataset = ds.dataset("data/dataset_directory")
df = pl.scan_pyarrow_dataset(dataset)
print(df.collect())

Converting from Pandas to Polars

• If Polars doesn’t support a file format natively, you can use Pandas to read the data, then convert it to a Polars DataFrame using pl.from_pandas().

  import pandas as pd
  import polars as pl
  
  # Reading HTML table with Pandas
  url = "https://en.wikipedia.org/wiki/List_of_Latin_abbreviations"
  pandas_df = pd.read_html(url)[0]
  
  # Converting to Polars DataFrame
  polars_df = pl.from_pandas(pandas_df)
  print(polars_df)

File Formats Supported by Pandas

Pandas supports a variety of formats that Polars doesn't handle directly. You can use Pandas to read from these formats and then convert the DataFrame to Polars.

Some of the additional formats Pandas supports include:

• Feather
• Fixed-Width Text Files
• HDF5
• ORC
• SAS
• SPSS
• Stata
• XLS/XLSX
• XML
• Local clipboard

# Reading Stata file with Pandas
stata_df = pd.read_stata("data/file.dta")

# Converting to Polars DataFrame
polars_df = pl.from_pandas(stata_df)
print(polars_df)

Querying Databases

• Polars provides an efficient way to query relational databases using the pl.read_database() function (or pl.read_database_uri() for SQLite). This allows you to execute SQL queries directly on databases and retrieve the results as Polars DataFrames.
• Supported databases include PostgreSQL, MySQL, SQLite, MsSQL, Oracle, and BigQuery.

Example: Querying SQLite with Polars

• To demonstrate, let's use an SQLite database, specifically the Sakila sample database, which contains information about films and categories.
• The steps are similar for other databases, but the connection string and possibly the SQL dialect will differ.

Basic Query Example

The following query selects 10 film titles, their categories, ratings, and lengths from the Sakila database:

import polars as pl

# Execute SQL query using Polars
df = pl.read_database_uri(
    query="""
    SELECT
        f.film_id,
        f.title,
        c.name AS category,
        f.rating,
        f.length / 60.0 AS length
    FROM
        film AS f
    JOIN
        film_category AS fc ON fc.film_id = f.film_id
    JOIN
        category AS c ON fc.category_id = c.category_id
    LIMIT 10
    """,
    uri="sqlite:///data/sakila.db"
)

print(df)

Output:

shape: (10, 5)
┌─────────┬──────────────────┬─────────────┬────────┬──────────┐
│ film_id │ title            │ category    │ rating │ length   │
│ ---     │ ---              │ ---         │ ---    │ ---      │
│ i64     │ str              │ str         │ str    │ f64      │
╞═════════╪══════════════════╪═════════════╪════════╪══════════╡
│ 1       │ ACADEMY DINOSAUR │ Documentary │ PG     │ 1.433333 │
│ 2       │ ACE GOLDFINGER   │ Horror      │ G      │ 0.8      │
│ 3       │ ADAPTATION HOLES │ Documentary │ NC-17  │ 0.833333 │
│ 4       │ AFFAIR PREJUDICE │ Horror      │ G      │ 1.95     │
│ 5       │ AFRICAN EGG      │ Family      │ G      │ 2.166667 │
│ 6       │ AGENT TRUMAN     │ Foreign     │ PG     │ 2.816667 │
│ 7       │ AIRPLANE SIERRA  │ Comedy      │ PG-13  │ 1.033333 │
│ 8       │ AIRPORT POLLOCK  │ Horror      │ R      │ 0.9      │
│ 9       │ ALABAMA DEVIL    │ Horror      │ PG-13  │ 1.9      │
│ 10      │ ALADDIN CALENDAR │ Sports      │ NC-17  │ 1.05     │
└─────────┴──────────────────┴─────────────┴────────┴──────────┘

Querying Without SQL

• If SQL isn’t your preference, you can extract all the data from the tables and perform the operations in Polars. Here's how to replicate the previous query using Polars instead of SQL:

# Define database URI
db_uri = "sqlite:///data/sakila.db"

# Read tables into Polars DataFrames
films = pl.read_database_uri("SELECT * FROM film", db_uri)
film_categories = pl.read_database_uri("SELECT * FROM film_category", db_uri)
categories = pl.read_database_uri("SELECT * FROM category", db_uri)

# Perform the joins and data transformation in Polars
df = (
    films
    .join(film_categories, on="film_id", suffix="_fc")
    .join(categories, on="category_id", suffix="_c")
    .select(
        "film_id",
        "title",
        pl.col("name").alias("category"),
        "rating",
        (pl.col("length") / 60).alias("length")
    )
    .limit(10)
)

print(df)

• This code extracts the data from the database and performs the joins using Polars, allowing you to keep the processing within Polars if you prefer not to write SQL queries.

Connection Strings

Each database type has its own connection string format. Below are a few examples:

• PostgreSQL: postgresql://username:password@host:port/database

• MySQL: mysql://username:password@host:port/database

• MsSQL: mssql://username:password@host:port/database

• SQLite: sqlite:///path_to_database.db

• For SQLite, the database is just a local file, so the connection string simply points to that file.

Performance Considerations

While pulling all data into Polars and manipulating it there is possible, it’s often more efficient to allow the database to handle as much of the querying and filtering as possible. Transferring large amounts of data can slow down performance, so try to:

• Select only the necessary columns using SQL.
• Filter rows as much as possible within the SQL query.

Writing Data

Polars offers several options for writing DataFrames to files, depending on the format you need. Each format has its advantages and disadvantages, making it suitable for different scenarios.

1. CSV Format

all_stocks.write_csv("data/all_stocks.csv")

Common Arguments

Pros:

• Universal compatibility.
• Easy to read and edit.

Cons:

• Can be inefficient with large datasets.
• Doesn't preserve data types.

2. Excel Format

all_stocks.write_excel("data/all_stocks.xlsx")

Common Arguments

Pros:

• Supports styling, formulas, and multiple sheets.
• Familiar to many business users.

Cons:

• Binary format, not human-readable in raw form.
• Performance issues with very large datasets.

3. Parquet Format

Parquet is a highly efficient columnar storage format that’s ideal for large datasets. It compresses data effectively, retains schema information, and is designed for high-performance read/write operations.

all_stocks.write_parquet("data/all_stocks.parquet")

Common Arguments

Pros:

• Efficient storage and retrieval for large datasets.
• Retains schema and supports nested data structures.
• Ideal for big-data processing.

Cons:

• Less universally recognized compared to CSV and Excel.
• Requires specific tools or libraries for reading.

4. Other Formats

Polars also supports other formats, such as Avro and JSON.

Example (JSON)

all_stocks.write_json("data/all_stocks.json")

Example (Avro)

all_stocks.write_avro("data/all_stocks.avro")

When to use:

• JSON: Suitable for web applications and APIs.
• Avro: Great for efficient data serialization, especially in distributed systems.

Choosing the Right Format

• CSV: Best for portability and human-readability across platforms.
• Excel: Ideal for business users and multi-sheet, styled workbooks.
• Parquet: Optimal for large, complex datasets needing efficient storage and retrieval.
• Other formats (Avro, JSON): Consider when working with web services, APIs, or distributed systems.