Bristol data tools risked wrongly flagging victims and suspects, Children’s Commissioner ‘deeply concerned’
This investigation was reported by the Cable in partnership with Liberty Investigates, Lighthouse Reports and WIRED, whose reporting on this issue you can also find here.
Children at risk of exploitation could have “slipped through the net” because authorities were using an AI tool to identify them that was later deemed “not fit” for use.
These algorithms, developed by Bristol Council and Avon and Somerset police, were used for years before being quietly decommissioned.
The programs were designed to identify children at risk of criminal or sexual exploitation and based on a host of data from public agencies as part of the controversial Think Family Database.
They were used by social workers to prioritise which children needed support, and by police to identify potential predators. The risk scores they produced could also be seen by safeguarding leads at Bristol schools via an app connected to the database.
Internal council documents and testimony seen by the Cable reveal concerns that the models were wrongly identifying potential suspects and victims and missing vulnerable children in need of support — raising concerns about the historic harm they may have caused.
The Children’s Commissioner for England, Dame Rachel de Souza, responsible for protecting the rights of children in England, said our findings were “deeply concerning” and highlighted the need for scrutiny.
A lack of transparency, and the absence of any clear way to trace potential harms, raises urgent questions about the use of algorithms to assess children’s risk of exploitation.
More than that, our findings are set in the wider context of a national drive to embrace artificial intelligence and predictive algorithms across public services – plans for which look strikingly similar to this Bristol experiment.
Insight Bristol is a joint team of data scientists originally formed by Bristol City Council and Avon and Somerset Police. It was launched in 2015 as part of a regional drive to invest in predictive analytics across public services.
At that time, the team was headed by Gary Davies, a former police chief superintendent at Avon and Somerset Police who later became head of early intervention at Bristol City Council.
Together, they created the Think Family Database, which launched in 2016 and is still in operation today. The Cable understands it contains records relating to around 475,000 current and former Bristol residents, bringing together information held across public agencies on families in the city.
The system was designed to give frontline workers a fuller picture of children and families, combining data from housing, education, criminal records and policing to inform safeguarding decisions and improve support.
To gather and use this information, it relies on what are known as “legal gateways” — statutory provisions that allow public bodies to process and share personal data where it is deemed necessary to meet legal obligations. In this case: child protection.
Davies defends the approach: “If you were to give the impression that people had consent then it creates a false illusion because, actually, as local authority or police or whoever, we have to actually keep those records,” he said.
The Child Sexual Exploitation (CSE) and Child Criminal Exploitation (CCE) models were developed within Insight Bristol to generate “risk scores” for individual children and young people in the city.
Introduced in 2016, the CSE model drew on datasets held by Avon and Somerset Police, Bristol City Council and other public agencies. It was also trained on data from charity Barnardo’s, covering around 1,000 children known to have experienced child sexual abuse.
The model was designed to identify patterns associated with exploitation risk. Factors such as being reported missing, persistent school absence, or concerns around mental health would all increase a child’s risk score.
The CCE model also drew on a wide set of administrative data that represent markers of poverty or financial insecurity. Both models drew on variables such as housing support, rent arrears and whether a child was on free school meals.
Gender and ethnicity data was not used to create the risk scores, the Cable understands, but a child’s social connections were analysed to identify links to others deemed vulnerable to, or perpetrators of, exploitation.
Together, the models produced numerical scores intended to indicate a child’s likelihood of experiencing criminal or sexual exploitation.
But our investigation found that the council decided the models were “not fit for operational use” and shut them down in 2023.
Documents seen by the Cable suggest that confidence in the models, and in the Think Family Database as a whole, was low among staff who used them. Concerns about the project were tabled internally in the early days of its roll-out.
In 2021, the government’s Centre for Data Ethics and Innovation reported concerns about the use of “legal gateways” to collect large amounts of sensitive data, instead of building trust with the people they were gathering data from.
“Legality is not the same as legitimacy,” the authors wrote, in a report not made public until obtained through a record request by Cable’s reporting partner WIRED.
But questions were not only being raised about how the data was collected. There were growing concerns about whether the models themselves were working.
Two years later, an independent review by the non-profit Social Finance assessed the Think Family database and Insight Bristol’s work more broadly. Commissioned by Bristol City Council and Somerset Council, it was intended to inform Somerset’s plans for a similar system.
The review found that Think Family could lead to “timelier responses” by child protection staff. But it described the risk-scoring models as the “weakest element” of the Think Family database.
Bristol City Council staff raised a series of concerns about how the models performed in practice.
Children expected to appear as vulnerable were not showing up in the system’s results. “We know there’s young girls that get criminally exploited, but they don’t come up, we don’t talk about them cause they don’t fit,” one member of staff told reviewers.
In some cases, victims of sexual exploitation were scoring lower than people involved in burglary – raising serious questions about how the model calculated risk.
Staff also questioned whether the scores could be relied upon. “I wouldn’t go into a meeting saying I’ve seen this on TFD TFE because I wouldn’t be confident that that is accurate enough,” one said.
Others said they did not understand how the models worked. “Personally, I feel uncomfortable using it to guide our work because of the lack of transparency on where the numbers come from and how it was developed,” another told reviewers.
For some, confidence in the system eroded over time.
One member of staff said: “I used to spend a lot of time methodically going through the 30 names, emailing people, and checking all the details, but it took up so much of my time that I kind of stopped doing that.”
Questions about the models’ performance were compounded by a lack of information about how they had been built, tested and ultimately abandoned.
The Social Finance review found it was difficult to scrutinise the systems because key technical information was missing. “Source code and variables that detail how these models were created were unable to be found,” the report said, preventing a full evaluation.
Social Finance declined to comment on its report.
Responses to public record requests from Bristol City Council and Avon and Somerset Police suggest neither body holds records explaining why the CSE and CCE models were decommissioned in 2023.
Rob Procter, professor of social informatics at Warwick University and an expert adviser to the Social Finance review, said the lack of documentation made it difficult to properly assess the models.
“The process to build the models was not documented in anything like sufficient detail,” he told WIRED. He said the Bristol case demonstrated the need for greater transparency and public scrutiny whenever predictive systems are used in public services.
“This really raises the question of how you involve the public in deciding to develop and deploy these kinds of tools,” he said. “You have to consider the impact that even one false positive has on a family if a child is flagged as at risk of criminal or sexual exploitation.”
Debbie Watson, a professor of child and family welfare at the University of Bristol who has been researching the Think Family project since 2022, raised similar concerns: “Whilst they may no longer be in operation, their use appears to have been significant in ways that have seriously impacted some young people in the city.”
Watson said she has concerns about “historic harms” that may have been caused by the risk-scoring models.
Davies, who led the Insight Bristol project but has recently retired, sought to downplay the models’ impact, arguing practitioners did not place significant weight on the scores: “They weren’t really using it to support their judgments, because they didn’t understand it and they didn’t value it.”
But the absence of records about how the models worked, how they were used and why they were decommissioned makes those claims difficult to independently verify.
The findings have prompted concern from the Children’s Commissioner for England, Dame Rachel de Souza. Responding to our investigation, she said: “Any technology used in child safeguarding must be transparent, well understood by professionals and clearly shown to help protect children.
“It is deeply concerning that tools used by staff were not fully understood, trusted, and may have compromised their professional judgement.
“Technology can play a role in protecting children, but only if it is safe, transparent and subject to rigorous scrutiny. This case highlights the need for clear national standards and independent oversight on the use of data and algorithms in children’s services.”
Frontline youth workers in Bristol also questioned whether risk scores could be relied upon to direct support.
Amy Newcombe, service co-ordinator at the Mwanzo Project, which works with young people affected by criminal exploitation, warned that predictive scores risk reinforcing existing assumptions about vulnerable children.
“A young person with ‘good’ data could be struggling with a multitude of issues that are not flagged because it doesn’t fit the algorithm and therefore they could ‘slip through the net’ whilst the eyes, meetings and strategies are being pursued for the ‘high risk’ individuals who may not be at risk at all,” she said.
Bristol City Council declined interview requests about its use of predictive risk-scoring systems. In a statement, Christine Townsend, chair of the council’s Children and Young People Committee, said: “This administration does not use any predictive analytics apart from helping to identify children who are at risk of becoming not in education, employment, or training after finishing school (NEET)… The use of analytics has never replaced professional human judgement or decision-making.”
“The use of analytics has never replaced professional human judgement or decision-making.”
While Townsed acknowledged that risk-scoring models are in use to assess the likelihood of a child becoming NEET, she would not respond to questions about how the CSE and CCE models that did may have impacted those it scored.
Predictive analytics like the NEET model continue to play a significant role in Bristol. Like the city council, Avon and Somerset Police relies on multiple predictive models to inform policing – including on the Offender Management App.
But this is not just a regional issue. Bristol has been the testbed for a national drive to embrace the use of this kind of technology across public services including children’s services.
In February, the Ministry of Justice announced it would deploy machine learning to identify at-risk children for early intervention. The technologies it plans to use are strikingly similar to those already used in Bristol.
At the forefront of a drive to embed AI across the criminal justice system is former Avon and Somerset Police chief constable Andy Marsh, who says the technology should be “injected like heroin” into policing.
In a recent interview, Marsh said the role of his organisation in deploying AI tools: “test the ones that work properly… and then spread them like wildfire through policing.”
