A nightly job that spills over into the morning. A notebook that runs out of memory on a perfectly reasonable volume. A cloud bill that grows with every new use case. Faced with these symptoms, the usual answer is to add: a bigger instance, a cluster, a distributed platform. It is almost always the most expensive answer, and rarely the right one.
This audit answers two questions. Where does your data processing really lose time, and which levers can win it back? Where are you losing money, and what infrastructure really matches your workload?
I sell no platform, no license and no migration project. My recommendations serve no interest other than yours.
The wrong reflex: Google's tools for data that is not Google's
Spark, Hadoop, and the platforms built on top of them were designed for companies that process data their machines cannot hold. It is a real problem, and a rare one. Most teams that adopt these tools have volumes several orders of magnitude smaller, but reason this way: if it is good enough for Google or Amazon, it is the best there is. They then inherit a layer of operations, skills and cost that they never pay back, and processing that is often slower than on a single, well-used machine.
Research describes this mismatch. Baziotis, Kang and Mendis find that a large share of real data science work runs on a single machine, with ad hoc code, and that distributed frameworks were not designed for these cases; they obtain significant speedups by rewriting existing pandas code, without touching the infrastructure. Petersohn and coauthors show that dataframe performance problems appear at moderate volumes already, and stem from the way operations are expressed, not from the machine.
My experience confirms it at a scale few teams reach. I have analyzed the entire Twitter social graph, more than five hundred million accounts and twenty-three billion links; the entire Bitcoin transaction chain, more than sixteen billion links; years of BitTorrent exchanges involving hundreds of millions of addresses. All of it on a single machine, in Python. Your data is almost certainly smaller.
What I bring. The view of someone who has already met these problems, solved them, and sells neither infrastructure nor licenses. By reading your code and examining your architecture, I can tell which part of the slowness comes from the tooling, which part comes from the code, and which part, rarely, really comes from the volume.
Where the time goes, where the money goes
Performance gains in data science rarely come from computing power. They come from the way the code is written: loops where a vectorized operation would do, joins that recompute what could be kept, data types that multiply the memory footprint by ten, full reads of files of which three columns are used. Moving from pandas to Polars, compiling a kernel with Numba, or rewriting ten critical lines brings orders of magnitude of performance improvement — provided you know where to apply them, and use these tools as they are designed: pandas code translated line by line into Polars gains nothing.
On the infrastructure side, the surveys agree: Flexera reports in 2026 that 85% of organizations see managing cloud costs as a major challenge and estimate that they waste 29% of their infrastructure spending. On-demand cloud is cost-effective for an irregular workload and ruinous for a continuous one. For many teams, a dedicated machine, bought and depreciated over several years, costs less than six months of subscription, fits in a closet, and depends on no vendor. For others, it is the opposite. The trade-off is made on your real workload profile, not on principles.
What I bring. I have been training teams in Python performance since 2011, and my “High-performance Python” course grew out of my own large-scale processing. I read an execution profile, I recognize the patterns that cost, and I know how to decide between optimizing the existing code, migrating a section, compiling, rewriting, or changing the infrastructure. I also know when to say that nothing should be done.
How I work
The audit lasts three to five days depending on the scope. I do not rewrite your code and I do not size your infrastructure in your place: I establish a diagnosis and give your team the means to act.
- Scoping with the manager: the processing that causes problems, its frequency, its current cost, what a gain would change for the business.
- Code review of the critical processing: data structures, types, order of operations, file formats, use of libraries.
- Architecture review: what runs where, why, and what it costs.
- Sessions with the team on the identified processing, to locate together where time and memory go and check assumptions on your data.
- Workload profile analysis with you: current and expected volumes, regularity, real constraints.
What you receive
A report of findings and recommendations: the identified causes of slowness, ranked by effort versus expected gain; the places where a migration to Polars, a Numba compilation or a rewrite is justified, and those where it is not; a reasoned position on the infrastructure — keep, reduce, change, or buy — with orders of magnitude of cost over several years; and an action plan that the team carries out itself. If the plan requires skills the team does not have yet, I say so, and targeted training can take over.
Ongoing advisory
After an audit, gains are sometimes lost: new use cases, new dependencies, code written in a hurry, and the bill going up again without anyone knowing why. Half a day to a day per month makes it possible to check that the choices made still hold, to settle new ones, and to spot a drift before it costs.
Who it is for
Data science, data engineering and R&D teams in SMEs, mid-sized companies and growing startups, whose Python processing — pandas, NumPy, scikit-learn — has become too slow or too costly for the business it supports, or who are about to invest in a distributed platform and want an independent opinion before signing.
Why me
As a senior research scientist at Inria, I have been doing large-scale data analysis for more than twenty years, in Python, on research problems where reproducibility matters as much as speed. I was also CTO assistant at one startup, then founder of another: I know what oversized infrastructure costs a company that counts every euro, and I know that a server depreciated over seven years is sometimes the best data investment it will make.
Preliminary diagnostic
First of all, a free, no-commitment one-hour call, to understand your situation and check that I can be useful to you. If there is something to work on, a one- to two-day diagnostic follows: scoping with the manager, a quick review of one or two representative processing jobs, an oral debrief and written recommendations. It concludes on what deserves an audit, a training course, or nothing.