France’s Digital Inquisition

Taking apart the secretive fraud detection algorithm that scores half of France’s population but pursues the most vulnerable.

In 2022, Juliette, a single mother on welfare, received money from her family to visit her critically ill father. A few months after her father died, a fraud investigator from France’s social security agency, CNAF, knocked on her door. The investigation determined that she owed thousands of euros that would be deducted from her monthly welfare payments.

What Juliette did not know at the time was that she was one of hundreds of thousands of people on welfare in France being flagged by an algorithm.

For more than a decade and without any public consultation, CNAF has deployed machine learning at a massive scale in a hunt for welfare fraud. Each year, nearly half of France’s population is silently ranked by a secretive risk score between 0 and 1. The only way to obtain the machine score is to make a subject access request under the General Data Protection Regulation.

People with the highest scores are subject to the CNAF’s most invasive investigations, where fraud controllers are empowered to search people’s homes, count toothbrushes to guess how many people live there, question their neighbours and scour bank records. Seven out of every ten people investigated in this way are flagged by the algorithm.

Over the course of six months Lighthouse and Le Monde investigated the algorithm deployed by CNAF. We used freedom-of-information laws to obtain far-reaching access to the building blocks of the system that enabled us to take it apart and analyse how it scores millions of people. We found that the algorithm hones in on people’s vulnerabilities — which have no apparent connection with fraud — such as being a single parent, having a low income or a disability.

METHODS

France is an early member of a growing club of nations that experiment with predictive technology on vulnerable populations with profound consequences. For more than two years, Lighthouse has investigated the growing deployment of this technology across European welfare systems as part of our Suspicion Machines reporting.

In 2021, Lighthouse worked with Algorithm Watch’s Nicolas Kayser-Bril to send a freedom-of-information request asking for the source code of the CNAF’s algorithm. The agency disclosed the source code, but redacted the variables that would allow us to understand how it actually scores people. After an appeal was rejected, work on the investigation paused as we pursued access elsewhere.

In parallel, digital rights groups La Quadrature du Net and Changer de Cap mounted a series of campaigns calling for the CNAF to release the code of the algorithm. Earlier this year, they successfully argued in front of France’s Commission for Access to Administrative Documents (CADA) that the CNAF should have to release the source code for previous versions of its risk assessment algorithm. The CNAF independently disclosed unredacted source code to La Quadrature and Le Monde. La Quadrature’s analysis can be found here.

Obtaining runnable code is rare and the far-reaching access we acquired enabled us to test how different kinds of people are scored (see this blog for a detailed discussion of the methodology). With nearly half of France’s population scored every year by the model, in principle a wide variety of people are at risk of being flagged. Yet when we tested the model we found that in practice it only flags the most vulnerable while it is nearly impossible for the better-off to score high enough to be investigated. Meanwhile poorly constructed criteria mean that arbitrary changes in behaviour — like sending an email one month instead of two months ago — can land people flagged

Interviews with insiders and CNAF officials allowed us to trace the system’s history back to the growing hysteria around welfare fraud in late 2000s as Nicolas Sarkozy vowed to “punish fraudsters.” Months of community-level reporting and interviews with lawyers allowed us illustrate the drastic consequences for vulnerable people.

STORYLINES

We worked with Le Monde’s Les Decodeurs desk to produce a three-part series based on our joint-reporting breaking down the technology, the people it affects and the system that has grown up.

The technology story provides an interactive explanation of our joint audit and how the algorithm makes broad, prejudiced generalisations about millions of people. It finds evidence that the system both directly and indirectly discriminates against groups protected under French discrimination law.

The people story traces the human consequences of being flagged for investigation and its dramatic toll on people’s lives. Readers follow Juliette as she types her details into a prototype of the algorithm and watch as her risk score rises. It ranges across France — where the level of unclaimed benefits is estimated to exceed 30 percent — and follows welfare recipients who struggle to navigate the complex rules of a system where mistakes are cast as deliberate fraud.

The system story traces the history of the system and how a small experiment at a local CNAF in Bordeaux went national amidst a growing moral-panic over welfare fraud. It shows how even key figures close to the algorithm’s development are beginning to sound the alarm, including former CNAF director Daniel Lenoir.