polars

Polars

Overview

Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.

Quick Start

Installation and Basic Usage

Install Polars:

uv pip install polars

Basic DataFrame creation and operations:

import polars as pl
 
# Create DataFrame
df = pl.DataFrame({
    "name": ["Alice", "Bob", "Charlie"],
    "age": [25

Core Concepts

Expressions

Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.

Key principles:

• Use pl.col("column_name") to reference columns
• Chain methods to build complex transformations
• Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)

Example:

Lazy vs Eager Evaluation

Eager (DataFrame): Operations execute immediately

Lazy (LazyFrame): Operations build a query plan, optimized before execution

When to use lazy:

• Working with large datasets
• Complex query pipelines
• When only some columns/rows are needed
• Performance is critical

Benefits of lazy evaluation:

• Automatic query optimization
• Predicate pushdown
• Projection pushdown
• Parallel execution

For detailed concepts, load references/core_concepts.md.

Common Operations

Select

Select and manipulate columns:

Filter

Filter rows by conditions:

With Columns

Add or modify columns while preserving existing ones:

Group By and Aggregations

Group data and compute aggregations:

For detailed operation patterns, load references/operations.md.

Aggregations and Window Functions

Aggregation Functions

Common aggregations within group_by context:

• pl.len() - count rows
• pl.col("x").sum() - sum values
• pl.col("x").mean() - average
• pl.col("x").min() / pl.col("x").max() - extremes
• pl.first() / pl.last() - first/last values

Window Functions with over()

Apply aggregations while preserving row count:

Mapping strategies:

• group_to_rows (default): Preserves original row order
• explode: Faster but groups rows together
• join: Creates list columns

Data I/O

Supported Formats

Polars supports reading and writing:

• CSV, Parquet, JSON, Excel
• Databases (via connectors)
• Cloud storage (S3, Azure, GCS)
• Google BigQuery
• Multiple/partitioned files

Common I/O Operations

CSV:

Parquet (recommended for performance):

JSON:

For comprehensive I/O documentation, load references/io_guide.md.

Transformations

Joins

Combine DataFrames:

Concatenation

Stack DataFrames:

Pivot and Unpivot

Reshape data:

For detailed transformation examples, load references/transformations.md.

Pandas Migration

Polars offers significant performance improvements over pandas with a cleaner API. Key differences:

Conceptual Differences

• No index: Polars uses integer positions only
• Strict typing: No silent type conversions
• Lazy evaluation: Available via LazyFrame
• Parallel by default: Operations parallelized automatically

Common Operation Mappings

Key Syntax Patterns

Pandas sequential (slow):

Polars parallel (fast):

For comprehensive migration guide, load references/pandas_migration.md.

Best Practices

Performance Optimization

1. Use lazy evaluation for large datasets:

   lf = pl.scan_csv("large.csv")  # Don't use read_csv
   result = lf.filter(...).select(...).collect()

2. Avoid Python functions in hot paths:

   • Stay within expression API for parallelization
   • Use .map_elements() only when necessary
   • Prefer native Polars operations

3. Use streaming for very large data:

   lf.collect(streaming=True)

4. Select only needed columns early:

   # Good: Select columns early
   lf.select("col1", "col2").filter(...)
    
   # Bad: Filter on all columns first
   lf.filter(...).select("col1", "col2")

5. Use appropriate data types:

   • Categorical for low-cardinality strings
   • Appropriate integer sizes (i32 vs i64)
   • Date types for temporal data

Expression Patterns

Conditional operations:

pl.when(condition).then(value).otherwise(other_value)

Column operations across multiple columns:

df.select(pl.col("^.*_value$") * 2)  # Regex pattern

Null handling:

pl.col("x").fill_null(0)
pl.col("x").is_null()
pl.col("x").drop_nulls()

For additional best practices and patterns, load references/best_practices.md.

Resources

This skill includes comprehensive reference documentation:

references/

• core_concepts.md - Detailed explanations of expressions, lazy evaluation, and type system
• operations.md - Comprehensive guide to all common operations with examples
• pandas_migration.md - Complete migration guide from pandas to Polars
• io_guide.md - Data I/O operations for all supported formats
• transformations.md - Joins, concatenation, pivots, and reshaping operations
• best_practices.md - Performance optimization tips and common patterns

Load these references as needed when users require detailed information about specific topics.