Pandas is essential in Python data science — but its limits show up quickly: excessive memory use, single-threaded execution, a sometimes inconsistent API. This course gives your teams the tools to go beyond these limits.
Polars redefines data manipulation in Python: an engine written in Rust, native parallelism, lazy evaluation, and the ability to process datasets larger than the available RAM thanks to its streaming engine. It is today the best choice to take a data pipeline from prototype to production.
I use Polars every day in my own research, in particular to analyze the entire Bitcoin transaction graph — 16.5 billion links between 1.5 billion addresses, stored in partitioned Parquet files. This course is built on that practice: what I wish I had known when my Pandas jobs took weeks and saturated the RAM — and moving to Polars brought them down to a few hours.
After this course, your staff will be able to
- Write or rewrite their data processing in Polars for significant speed gains
- Process datasets larger than the available RAM with the Polars streaming engine
- Read and write Parquet files with Polars and PyArrow for efficient storage and exchange
- Speed up their numerical computations with Numba and JIT compilation
- Parallelize independent CPU-bound tasks with the
multiprocessingmodule - Choose the right performance strategy for the context: Polars, Numba or multiprocessing
- Apply best practices for high-performance data manipulation
Detailed program
Module 1 — Polars, the modern replacement for Pandas
Polars architecture: Rust engine, native parallelism, Apache Arrow. The core API: Series, DataFrame, essential operations. Lazy evaluation and query optimization. The streaming engine: processing data larger than RAM. Polars expressions: the most powerful API for data manipulation. Handling missing data. Aggregations and windows: group_by, over. Joins and concatenations. Reshaping: pivot and unpivot. Time series: resampling, rolling windows, time zone handling. The Parquet format: reading/writing, compression, partitioning, interoperability with PyArrow and Pandas. Migrating from Pandas to Polars: strategies and equivalences. Comparative benchmark on real cases.
Module 2 — Numba, JIT acceleration for numerical computing
The principle of JIT compilation with Numba. The @jit decorator: when and how to use it. Speeding up loops and NumPy functions. Parallelization with @jit(parallel=True) and prange. Use cases: numerical simulation, signal processing, scientific computing. Numba's limits and best practices.
Module 3 — Multiprocessing, multi-process parallelism in Python
The Python GIL and its implications for CPU-bound parallelism. Python without the GIL (the free-threaded build, officially supported since Python 3.14): what changes, and why multiprocessing remains the standard solution today. The multiprocessing module: Process, Pool, Queue, Pipe. Pool.map() and Pool.starmap(): parallelizing independent tasks. Process start methods (fork, spawn, forkserver, the default on Linux since Python 3.14) and their consequences for existing code. Memory sharing and inter-process communication. multiprocessing vs Numba vs Polars: when to use which. Common pitfalls and debugging multiprocessing code.
Do your teams have the required Python level?
The data science courses assume a good command of Python: functions, iteration, modules, namespaces. If this is not the case for all your staff, I can add the Python — level 1 module before the data science days. For teams that develop complex code, the Python — advanced module (object-oriented programming, decorators, metaclasses) is also available.
These complements are offered case by case — feel free to mention them in your quote request.
Audience and prerequisites
Target audience
Data scientists, engineers, researchers, postdocs and Ph.D. students who use Pandas and want to improve the performance of their data processing.
Prerequisites
Intermediate level in Python (functions, iteration, namespaces, importing modules) and in Pandas/NumPy. This course is designed as the natural follow-up to the Python data science level 1 course.
Teaching method
The course alternates short theoretical presentations and hands-on exercises in Jupyter notebooks. Each concept is put into practice immediately. I pay particular attention to making sure every participant keeps up, whatever their starting level, and I calibrate the depth of each topic to the group.
The material is given to the participants at the end of the course, along with the exercise notebooks and their solutions.
Official documentation
Polars, Numba, the multiprocessing module (Python 3), the Apache Parquet format.
Frequently asked questions
Does Polars completely replace Pandas?
Polars is today the best choice to take a data pipeline from prototype to production. The course covers migrating from Pandas to Polars, with strategies and equivalences. In practice, many teams keep both during a transition period — the course prepares them to manage this coexistence.
What is the Polars streaming engine and when do you need it?
The Polars streaming engine processes datasets larger than the available RAM by transparently handling the data in chunks. It is covered in module 1. A typical case: a 200 GB Parquet file on a machine with 32 GB of RAM.
What is the difference between Numba and multiprocessing, and when should you choose one or the other?
Numba speeds up intensive numerical computations through JIT compilation — useful for loops or NumPy functions. Multiprocessing parallelizes independent tasks by working around the Python GIL — useful for "map"-style CPU-bound processing. Module 3 explicitly covers the comparison and the selection criteria.
Do I need to take the level 1 course first?
The level 2 course assumes an intermediate command of Python and practice with Pandas/NumPy. The Python data science level 1 course is the natural way in, but an equivalent level acquired through practice is also fine.
Is this course suitable for researchers who process large volumes of data?
Yes, this is precisely the target audience. I use Polars every day in my own research, in particular to analyze 16.5 billion Bitcoin transactions stored in partitioned Parquet files — the course is built on this real practice.
Is the remote format suitable for this kind of training?
Yes. 90% of participants rate the remote format as equivalent or better than in-person. I have been running remote training since 2020 and have built a solid practice of online teaching. I use Zoom, which makes it possible to record the sessions and share them with participants if desired, and CoCalc, an online platform that gives access to preconfigured servers with all the training material and libraries installed. CoCalc also lets me follow each participant's progress in real time and step in directly to correct or unblock.