A new study has found that AI used to analyze medical scans can identify the patient’s race – even if the scan is blurry or presented at a low-resolution

The study’s 20 authors found deep learning models can identify race in chest and hand x-rays and mammograms among patients who identified as black, white and Asian.

The algorithms correctly identify which images were from a black person more than 90% of the time, but also showed it was able to identity race with 99% accuracy at times.

However, the team was unable to explain how the AI systems were making accurate predictions, some of which were done with scans that were blurry or low-resolution.

“We emphasize that model ability to predict self-reported race is itself not the issue of importance,’ Ritu Banerjee, associate professor of pediatrics at Washington University School of Medicine and lead author of the study, and collages wrote in the study published in arXiv.

“However, our findings that AI can trivially predict self-reported race – even from corrupted, cropped, and noised medical images – in a setting where clinical experts cannot, creates an enormous risk for all model deployments in medical imaging: if an AI model secretly used its knowledge of self-reported race to misclassify all Black patients, radiologists would not be able to tell using the same data the model has access to.”

The team set out to understand how these deep learning models were able to identify the patient’s race.

This included looking at if predictions were based on biological differences such as denser breast tissue.

Scientists also investigated the images themselves to see whether the AI models were picking up on differences in quality or resolution to make their predictions, perhaps because images of black patients came from lower-quality machines.

However, none of these leads led to an explanation behind the technology’s ability.  

The team notes that such abilities in AI could lead to further disparities in treatment.

‘These findings suggest that not only is racial identity trivially learned by AI models, but that it appears likely that it will be remarkably difficult to debias these systems,’ reads the study.

‘We could only reduce the ability of the models to detect race with extreme degradation of the image quality, to the level where we would expect task performance to also be severely impaired and often well beyond that point that the images are undiagnosable for a human radiologist.’

The full study can be read here