A model is about to go into production. A metric is about to be presented to your investors. An A/B test is about to set a price. A forecast is about to commit a budget. In all these cases, a decision rests on an analysis that nobody, apart from its author, has really examined. It is probably correct. But a wrong analysis looks exactly like a right one, and the cost of an error is paid long afterwards.
In science, no result is accepted until a peer has reviewed and challenged it. This is what I offer your team: an independent review, by someone whose job it is, before you decide.
Why even good teams get it wrong
It is not a question of competence. Silberzahn and coauthors gave the same dataset and the same question to twenty-nine teams of analysts: their conclusions diverged, down to the sign of the effect, solely through their methodological choices. Botvinik-Nezer and coauthors repeated the experiment with seventy teams: the same dispersion. Kapoor and Narayanan identified, across seventeen disciplines, hundreds of published papers whose results rest on information leaking between training and test data — and in one emblematic case, the announced advantage of machine learning disappears once the error is fixed. Kohavi and coauthors, drawing on the thousands of experiments run at Microsoft, show that significant A/B tests regularly lead to wrong conclusions, for reasons that only a trained eye notices.
These errors have one thing in common: they do not crash the code. A badly built cross-validation, a variable that already contains the answer, a metric that measures something other than what you think, a comparison between populations that are not comparable, an effect that is real but too small to justify the decision. The result is plausible, the chart is clean, and the company moves ahead on it.
The problem often starts before the analysis. A business question has to become a measurable question, with data that can answer it and a suitable method. Many analyses are technically right and answer a question close to the one that mattered.
What I bring. Peer review has been my activity for more than twenty years: program committees of the leading conferences in my field, a journal editorial board, Ph.D. supervision. Defining a research question that holds up against international competition, then checking that the data and the method answer it, is exactly what a senior research scientist does. I apply it to your analyses with the same standards, and with no stake in the answer.
What I review
- The question: what you really want to know, what the analysis actually measures, and the gap between the two.
- The data: where it comes from, what it contains and does not contain, collection biases, representativeness.
- The method: the choice of statistical or learning tools, their assumptions, their fit to the question; the validation protocols; information leaks; comparisons; the uncertainty of the results and what it allows you to conclude.
- The interpretation: what the results really allow you to claim, and what is extrapolation.
- Reproducibility: can the analysis be redone in six months and give the same result?
How I work
A review covers one analysis, one model or one decision, and lasts two to four days. I read the code, the data and the documents produced; I talk with the authors, as a reviewer talks with an author, to understand the choices and the constraints; I check the critical points on your data; I give a written opinion. When the analysis holds, I say so, and this opinion has value for those you present it to. When it does not, I say why, and what it would take for it to hold.
What you receive
A written opinion, structured like a scientific review: what is solid, what is fragile, what is wrong, and for each point the reason and a possible correction; an assessment of whether the decision can rely on these results; and, when the original question deserves to be reformulated, a proposed question and method. The opinion is about the analysis, never about the people: that is the principle of peer review, and it is what makes it useful.
Ongoing advisory
For startups and R&D teams that regularly make decisions based on their analyses, I offer a scientific advisory role: half a day to a day per month, with review of ongoing analyses, participation in methodological choices, and someone to ask “can we trust this?” before deciding.
Who it is for
Data science and R&D teams in SMEs, mid-sized companies and growing startups, whose analyses underpin decisions that commit the company: putting a model into production, metrics presented to investors or customers, product experimentation, forecasts, strategic choices.
Why me
As a senior research scientist at Inria, with a habilitation to supervise research, I have built my career on the experimental analysis of large-scale data, published in the most selective conferences of my field and covered by the international press. My work established in particular, on the whole Twitter graph, that nearly six people in ten share an article without having read it (The Washington Post, Fortune, Les Échos, Slate.fr) — the kind of result that is only worth something if the method is flawless. I was also CTO assistant at one startup, then founder of another: I know the pressure to decide quickly on an analysis nobody had time to check.
Preliminary diagnostic
First of all, a free, no-commitment one-hour call, to understand your situation and check that I can be useful to you. If there is something to work on, a one- to two-day diagnostic follows: presentation of the analysis by its authors, a quick read, an oral debrief of the points to watch and written recommendations. It concludes on what deserves an in-depth review, a training course, or nothing.