Pandas, the widely used Python library for data analysis, is often the default choice for handling datasets up to several gigabytes. However, as data sizes approach the 10s or 100s of gigabytes, users frequently encounter memory limitations, slow computations, and a complex API. The common solution has been to switch to distributed systems such as Spark or Snowflake, but these introduce significant complexity that many workloads do not justify.
Recent analysis of Amazon Redshift usage reveals that most tables and queries operate on datasets smaller than 100GB. Specifically, over 94% of tables contain less than 100GB, and nearly 87% of queries involve 80GB or less. This suggests that the majority of real-world data problems fall into a "medium data" category, where distributed systems may be overkill.
Emerging tools like Polars and DuckDB offer compelling alternatives for these medium-sized datasets. Polars is a Rust-based DataFrame library designed to be familiar to Pandas users but optimized for performance through lazy evaluation and multi-threading. DuckDB is an in-memory analytics database that provides an SQL interface for querying data efficiently.
Benchmark tests on a 1 billion row CSV dataset demonstrate that Polars and DuckDB significantly outperform Pandas, using less memory and running faster. Polars uses about half the memory of Pandas, while DuckDB uses nearly 20 times less memory. Both tools leverage lazy evaluation and chunk-wise processing to optimize resource use and speed.
On typical laptop hardware, Polars and DuckDB maintain strong performance, automatically utilizing multiple CPU cores and managing memory efficiently. In contrast, Pandas struggles with high memory consumption and slower execution.
These tools also support Apache Arrow, enabling seamless data interchange with Pandas without costly memory copies. This interoperability lowers the barrier to adopting Polars or DuckDB in existing workflows.
For example, analyzing 3GB of NYC Taxi Parquet data to study payment trends during the pandemic can be done with both Pandas and DuckDB. Benchmarks show DuckDB executes these queries faster and with lower memory usage.
While Pandas continues to evolve, its limitations for larger datasets are increasingly apparent. Polars and DuckDB offer less complex, high-performance options that can handle medium-sized data efficiently without resorting to distributed systems.
Ultimately, the choice depends on workload, user preference, and ecosystem integration. Data engineers may prefer DuckDB’s SQL interface, while software engineers might favor Polars’ API. The key takeaway is to avoid prematurely adopting complex distributed systems when modern single-machine tools can meet most needs more effectively.