Inside a Fraud Prediction Algorithm

Obtaining the source code Rotterdam uses to predict benefits cheating

Every year, hundreds of intimate details relating to every welfare recipient in the Dutch port city of Rotterdam are fed into a machine learning algorithm. These details — which range from address to mental health history to hobbies — ultimately result in each recipient being assigned one number: a fraud risk score.

This risk assessment can have life-changing implications for the people being scored even though they themselves have no knowledge of how it is calculated. Rotterdam is one of many European cities that are turning to algorithms to inform critical decisions that shape the allocation of public resources.

A joint investigation by Lighthouse Reports and Dutch public broadcaster VPRO obtained the source code, inputs and outputs, as well as technical documentation of Rotterdam’s predictive algorithm. Disclosed documents reveal that since 2018, the municipality of Rotterdam has automatically put thousands of welfare recipients with high risk scores under investigation; hundreds have had their benefits terminated.

Blending data journalism with traditional reporting methods, Lighthouse is working to understand as concretely as possible the technical and human dimensions of Rotterdam’s predictive algorithm.

METHODS

In a welcome step, the city of Rotterdam voluntarily disclosed some of the code and technical information required for an independent assessment of their algorithm to Lighthouse and VPRO.

Unfortunately, key material — such as training data — necessary to conduct a comprehensive technical analysis have not been disclosed by Rotterdam as of writing. Lighthouse’s data scientists have so far conducted a partial audit based on the disclosed material evaluating data quality, reliability, transparency, and accuracy.

In order to better understand the human consequences of a high risk score, we have spoken to welfare recipients either definitively or potentially flagged for investigation by the algorithm. Some of these welfare recipients have submitted subject access requests under the GDPR to understand how their personal data has been processed.

To keep up to date with Lighthouse investigations sign up for our monthly newsletter