For the past two years, most of your data team's code has been written with Copilot, Cursor, Claude Code or ChatGPT. Nobody formally decided it; it happened one workstation at a time. Everyone will tell you they are faster, and for part of the work that is true.
Three questions remain unanswered, though. Is the quality of the code improving or degrading? Is the productivity gain real or perceived? And does what goes out to the models — your code, your data, your methods — pose a risk to the company?
This audit answers them based on your code and your practices, not on generalities about AI.
Quality and productivity: what agents really produce
Research results are more nuanced than vendors' promises. In a randomized trial by METR, experienced developers working with AI tools on their own projects took 19% longer, while believing they were 20% faster. At Stanford, Perry and coauthors showed that assisted participants wrote less secure code and were nevertheless more confident in it. Prather and coauthors describe a widening gap: assistance benefits those who master the subject and harms the others, by giving them an illusion of competence. Bastani and coauthors establish the mechanism on nearly a thousand participants: AI without guardrails improves immediate results and degrades learning, because people stop understanding what they get.
The principle is simple. An agent saves time for those who already know how, because they can judge and correct the suggestion. For those who do not master the subject, it proposes a solution that drifts away from the right one without anything signaling it. Final quality comes from the expert's small adjustments; degradation comes from small surrenders of understanding.
In data science, this mechanism is more dangerous than in software development. An application bug is visible: the program crashes, a test fails. An analysis error is not. Information leaking between training and test, a badly built cross-validation, an ignored statistical assumption produce a plausible result, presented with confidence, and carried into a decision. Notebooks have no tests, exploratory code ends up in production, and the model that wrote the analysis knew nothing about your data.
What I bring. I have been training teams in Python since 2011 (arnaudlegout.fr/en/training) and I code with these agents myself every day. I recognize code that a competent practitioner would not have written, and analysis that looks right without being right. The audit establishes, by type of task and by profile, where the agent speeds the team up and where it loses control of what it produces.
Data, code and methods: what leaves the company, and what it costs you
Here too, research is ahead of practice. Carlini and coauthors showed that verbatim extracts of a model's training data, including code, can be extracted from it, and that this memorization increases with model size. Greshake and coauthors established that an agent reading documents or web pages can receive hidden instructions and exfiltrate what it has access to — a risk that grows as agents gain autonomy and tools. Schuster and coauthors showed that a code completion assistant can be poisoned to suggest vulnerable code.
In practice, every request sends context: code snippets, column names, data samples, sometimes revenue figures or customer data. Depending on the tool, the subscription and the configuration, this context is retained or not, reused for training or not, hosted in Europe or not. Usage often goes beyond the contractual framework: a personal account to get by, an extension installed without approval.
The question is not only what goes out, but whether it harms you. A business algorithm that sets you apart from your competitors, sent to a model that trains on it, becomes a suggestion available to everyone. Customer data sent outside Europe engages your liability under the GDPR. An autonomous agent with access to your databases and to the Internet is an exfiltration target that your security controls do not cover. And dependence on a vendor whose terms change is a sovereignty question that few teams have asked.
What I bring. A large part of my research, published in the leading conferences on security and privacy, deals with measuring data leaks and threats on the Internet. The audit establishes what actually goes out, to whom, under which conditions, and qualifies the risk according to your business: what is harmless, what is regulatory, what puts your competitive edge at stake.
How I work
The audit lasts three to five days, partly on site.
- Individual interviews on the real use of agents: for which tasks, with what level of review, with which tools and which accounts.
- Review of a sample of notebooks, scripts and pipelines produced over recent months, with the Git history, to tell what was generated, understood and reworked.
- Observation of work sessions, to see how suggestions are accepted, modified or rejected.
- Validity check of two or three analyses that are critical for your business.
- Mapping of data flows: tools, subscriptions, configuration, retention and training clauses, data actually sent, and how sensitive each is for the company.
What you receive
A written report that answers the three questions, with the observations they rest on; an assessment of practices by type of task and profile; usage, review and verification rules adapted to your team; the measures to protect your code, data and methods, in order of priority. Two debriefs: one for the team, focused on practices; the other for management, focused on risks and decisions.
The goal is not to restrict the use of agents. It is for your team to get the productivity they allow, without losing control of what it produces or ownership of what it knows.
Ongoing advisory
Tools and practices evolve fast, and the rules adopted after an audit loosen or become obsolete within a few months. Half a day to a day per month makes it possible to review a recent sample, check what the team has really retained, and adjust the rules to the tools of the moment.
Who it is for
Data science, data engineering and R&D teams in SMEs, mid-sized companies and growing startups, whose analyses, models and code are company assets.
Why me
As a senior research scientist at Inria, with a habilitation to supervise research, I have built my career on the experimental analysis of very large volumes of data: the complete Twitter graph, all Bitcoin transactions, millions of IP addresses observed continuously. A large part of this work deals with data protection and measuring what leaks on the Internet. I was also CTO assistant at one startup, then founder of another: I know the trade-offs of a team that has to deliver.
Preliminary diagnostic
First of all, a free, no-commitment one-hour call, to understand your situation and check that I can be useful to you. If there is something to work on, a one- to two-day diagnostic follows: interviews with the manager and two or three team members, a quick review of a code sample, an oral debrief and written recommendations. It concludes on what deserves to be explored further, through an audit or a training course.