Junk Science Underpins Fraud Scores

Crude software scoring vulnerable benefit claimants is random and prejudiced

For two decades, more than a hundred municipalities in the Netherlands deployed crude software to calculate the fraud risk of benefits recipients. The software—called the “fraudescorekaart”—consists of a spreadsheet and programming script that create risk profiles. Benefits recipients flagged as ‘high risk’ by the fraudescorekaart are frequently subject to extra checks, including unannounced home visits. In some municipalities, the difference in treatment between those assigned low and high risk scores is drastic.

Lighthouse Reports was able to obtain the fraudescorekaart and reconstruct the underlying calculations. With partners VPRO and NRC, we outlined the various fraud risk indicators, which range from working as a taxi driver to living in a low-income neighbourhood to level of education. Lighthouse Reports made an interactive reconstruction where users can input data and see the corresponding risk scores. This can be found here.

METHODS

Over the course of four months, Lighthouse Reports made extensive use of freedom of information laws to obtain the fraudescorekaart and confirm that other municipalities had continued to use it. Internal emails received via FOIA provided insight into internal thinking on the scorecard’s ethics and legality.

Meanwhile, reporting partners VPRO and NRC tracked down the original developers of the Fraud scorecard and DPS-Matrix to confirm that they use the same risk scoring calculations.
With VPRO, we found and interviewed a benefits recipient likely marked as high risk by the scorecard to explore the human implications of a high score.

While the fraudescorekaart was phased out nationally in 2020, documents obtained by Lighthouse Reports reveal that a number of municipalities have continued to use separate software called DPS-Matrix that integrated the exact same scoring algorithm and spreadsheet. Four of these municipalities have said they will stop using the system after being approached for comment.

The Dutch Employment Agency (UWV) made the fraudescorekaart nationally available more than two decades ago. Since then, it has been used to calculate risk scores for hundreds of thousands of benefits recipients in the Netherlands. Despite this, the fraudescorekaart’s risk variables and weights are not backed up by scientific evidence and its calculations have never been externally validated. Many of the indicators, such as ‘risk neighbourhood,’ are based on subjective interpretation rather than statistical substantiation.

Nevertheless, the risk scores produced by the fraudescorekaart can have life changing implications. We spoke to Tamara, a young woman living in Horst aan de Maas, a municipality in the southeast of the Netherlands and one of the most fervent users of the fraudescorekaart according to statistics we obtained. When she filled out the scorecard, she received a high risk score based on her debt position, among other things. Throughout receiving benefits, Tamara had been subject to frequent checks and home visits, and the constant surveillance negatively affected her mental health. While we cannot know for certain how Horst aan de Maas’ logged her data in the fraudescorekaart, her treatment closely matched the municipality’s protocol for high risk scores.

Internal emails reveal that public officials phased out the fraudescorekaart because of optics concerns following the Dutch childcare benefits scandal. Yet, documents we obtained reveal that a group of five municipalities near the Dutch city of Utrecht—and possibly others—have continued to use an identical system to categorise benefits applicants into risk groups.

To keep up to date with Lighthouse investigations sign up for our monthly newsletter