Narwhals

Narwhals - DataFrame Agnostic API

Narwhals is a lightweight, zero-dependency compatibility layer for dataframe libraries in Python that provides a unified interface across different backends.

Docs: https://narwhals-dev.github.io/narwhals/

What is Narwhals?

Narwhals enables writing dataframe-agnostic code that works seamlessly across multiple Python dataframe libraries:

Full API Support:

• cuDF
• Modin
• pandas
• Polars
• PyArrow

Lazy-Only Support:

• Dask
• DuckDB
• Ibis
• PySpark
• SQLFrame

Core Philosophy

Why Narwhals?

• Resolves subtle differences between libraries (e.g., pandas checking index vs Polars checking values)
• Provides unified, simple, and predictable API
• Handles backwards compatibility internally
• Tests against nightly builds of supported libraries
• Maintains negligible performance overhead
• Full static typing support
• Zero dependencies

Target Use Case: Anyone building libraries, applications, or services that consume dataframes and need complete backend independence.

Key Features

1. Backend Agnostic: Write once, run on any supported dataframe library
2. Polars-Like API: Uses a subset of the Polars API for consistency
3. Lazy & Eager Execution: Separate APIs for both execution modes
4. Expression Support: Full expression API for complex operations
5. Type Safety: Perfect static typing support
6. 100% Branch Coverage: Thoroughly tested

Basic Usage Pattern

Three-Step Workflow

import narwhals as nw

# 1. Convert to Narwhals
df_nw = nw.from_native(df)  # Works with pandas, Polars, PyArrow, etc.

# 2. Perform operations using Polars-like API
result = df_nw.select(a_sum=nw.col("a").sum(), a_mean=nw.col("a").mean(), b_std=nw.col("b").std())

# 3. Convert back to original library
result_native = result.to_native()

Using the @narwhalify Decorator

Simplifies function definitions for automatic conversion:

@nw.narwhalify
def my_func(df: IntoDataFrameT):
    return df.select(nw.col("a").sum(), nw.col("b").mean()).filter(nw.col("a") > 0)


# Automatically handles conversion to/from Narwhals
result = my_func(pandas_df)  # Works!
result = my_func(polars_df)  # Also works!

Top-Level Functions

Conversion Functions

• from_native(df, ...): Convert native DataFrame/Series to Narwhals object
  • Parameters: pass_through, backend, eager_only, allow_series
• to_native(nw_obj): Convert Narwhals object back to native library type
• narwhalify(): Decorator for automatic dataframe-agnostic functions

Data Creation

• new_series(name, values, dtype): Create a new Series
• from_dict(data): Create DataFrame from dictionary
• from_dicts(data): Create DataFrame from sequence of dictionaries

File I/O

Eager Loading:

• read_csv(source, **kwargs): Read CSV file into DataFrame
• read_parquet(source, **kwargs): Read Parquet file into DataFrame

Lazy Loading:

• scan_csv(source, **kwargs): Lazily scan CSV file
• scan_parquet(source, **kwargs): Lazily scan Parquet file

Aggregation Functions

• sum(), mean(), min(), max(), median()
• sum_horizontal(), mean_horizontal(), etc.

Expression Creation

• col(name): Reference column by name
• lit(value): Create literal expression
• when(condition): Create conditional expression
• format(template, *args): Format expression as string

Utilities

• generate_temporary_column_name(): Generate unique column names
• get_native_namespace(obj): Get the native library of an object
• show_versions(): Print debugging information

DataFrame Methods

Properties

• columns: List of column names
• schema: Ordered mapping of column names to dtypes
• shape: Tuple of (rows, columns)
• implementation: Name of native implementation

Column Operations

• select(*exprs): Select columns using expressions
• with_columns(*exprs): Add or modify columns
• drop(*columns): Remove specified columns
• rename(mapping): Rename columns

Row Operations

• filter(predicate): Filter rows based on conditions
• head(n): Get first n rows
• tail(n): Get last n rows
• sample(n): Randomly sample n rows
• drop_nulls(): Drop rows with null values
• unique(): Remove duplicate rows

Inspection

• is_empty(): Check if DataFrame has no rows
• is_duplicated(): Identify duplicated rows
• is_unique(): Identify unique rows
• null_count(): Count null values per column
• estimated_size(): Estimate memory usage

Transformations

• sort(*by): Sort by one or more columns
• group_by(*by): Group by columns for aggregation
• join(other, on, how): Perform SQL-style joins
• pivot(on, index, values): Create pivot table
• explode(*columns): Expand list columns to long format
• lazy(): Convert to LazyFrame

Export

• to_native(): Convert to original library type
• to_numpy(): Convert to NumPy array
• to_pandas(): Convert to pandas DataFrame
• to_polars(): Convert to Polars DataFrame
• clone(): Create a copy

LazyFrame Methods

LazyFrame provides the same API as DataFrame but with lazy evaluation:

Key Differences

• Operations build an execution plan without computing
• collect(): Materialize the LazyFrame into a DataFrame
• collect_schema(): Get schema without collecting data
• sink_parquet(path): Write results directly to Parquet

Common Methods

All DataFrame methods are available on LazyFrame:

• select(), filter(), with_columns(), drop()
• group_by(), join(), sort(), unique()
• head(), tail(), top_k()
• gather_every(): Select rows at regular intervals
• unpivot(): Convert from wide to long format
• with_row_index(): Add row index column
• pipe(): Apply function to LazyFrame

Expression (Expr) API

Expressions are the building blocks for column operations.

Creation

nw.col("column_name")  # Reference column
nw.lit(42)  # Literal value

Filtering

• filter(predicate): Filter elements
• is_in(values): Check membership
• is_between(lower, upper): Check range
• drop_nulls(): Remove nulls

Aggregations

• count(): Count non-null elements
• null_count(): Count null values
• n_unique(): Count unique values
• sum(), mean(), median(): Statistical aggregations
• min(), max(): Extremes
• std(), var(): Spread measures
• quantile(q): Quantile values