Epic, one of the electronic health records (EHRs) providers in the US uses machine learning to produce a deterioration index which gives clinicians a quick overview of the risks each patient faces. The system is now being used by dozens of hospitals in the country to forecast which COVID-19 patient will turn for the worst. However, these healthcare institutions are arriving at different conclusions if the tool is effective and accurate.
A deterioration index in question
The algorithm takes in patients’ vital signs, laboratory results and primary care assessments to generate a score of zero to 100 for every patient. The higher the score, the more patient likely the patient is at risk of deterioration. Under normal circumstance, any artificial intelligence (AI) based model will have to be validated and refined before it goes into practice but the pandemic is not giving technology that luxury.
Some doctors found that 75% of hospitalized patients who received a score between 38 and 55 will eventually be transferred to the intensive care unit. Others believe the index is only helpful when patients’ scores fall on the extreme end of the scale. Clinicians urged AI should not replace their professional judgement since it has not been tested on enough patients to evaluate its actual performance. Although healthcare providers are racing to get more technology into the system to combat against the pandemic, whether they truly save lives become a question that cannot be answered.
Mark Pierce, Physician and Chief Medical Informatics Officer at Parkview Health told Stat, “Nobody has amassed the numbers to do a statistically valid test of the AI but in times like this that are unprecedented in US healthcare, you really do the best you can with the numbers you have, and err on the side of patient care”.
Some institutions, including Stanford University, are testing the deterioration index but researchers feel there isn’t sufficient patients to thoroughly examine whether such indication will work on a vast majority of the patients. Executives at Epic explained the index was originally introduced two years ago to keep track of the progress of hospitalized patients. They are confident, from a machine learning perspective, that it’s able to pick up patients in rapid deterioration.
Acting on the predictions
Indeed, Epic trained and tested its machine learning driven index with over 100,000 patient data in three hospital systems between years 2012 and 2016. It was found to be trustworthy on characterizing risks faced by patients. Nevertheless, the index was rapidly remodelled at the wake of coronavirus outbreak in the US, so that healthcare institutions can estimate and manage its resources.
The challenge now becomes, if hospitals are able to act on the index to change disease trajectory and improve patient outcomes? If hospitals are unable to do so, does the index give them adequate warning to respond? Some physicians pointed out because the index is being calculated once every 15 minutes, hence, a patient’s score may change from 70 to 30, turning them from high risk to low risk in a short period of time. Sometimes, patients may hover in the middle zone for days but require intensive care all of a sudden.
Despite the uncertainties, some physicians find the index does give them up to 40 hours of warning of any life-threatening event which may occur. Yet, since an overwhelming number of patients are coming into the hospitals at any minute during the pandemic, there is no clear sign if this alone, can save a life.