“What is measured improves.”

Peter Drucker, American business consultant

Over the past few weeks, we have discussed the steps of your journey towards becoming a center of AI in a health system. The following are some helpful metrics to measure the artificial intelligence capabilities of the health system in the context of an individual AI project:

AI project score

Projects that involve machine learning and artificial intelligence, either clinical or administrative, can be followed in stages (with each stage being scored 1 point each to a maximum of 5 points) and scored to keep track as well as maintain momentum:

Stage 1: Ideation

The project is initially discussed and brought to a regular meeting for input from all stakeholders. This is perhaps the most important part of an AI project that is often not done with sufficient discussion and consideration.

Stage 2: Preparation

After approval from the group, data access and curation takes place in order to perform the ML/AI steps that will ensue. The team should appreciate that this stage takes the most effort and will require sufficient resources.

Stage 3: Operation

After the data is curated and managed, this stage entails a collaborative effort during the feature engineering and selection process. Using the ML/AI tools, the team then creates the algorithms that will lead to the models that will be used later on in the project.

Stage 4: Presentation

Upon completion of the model with real world data, the project is presented in front of the group and, depending on the nature of the project, it may also be presented at a regional or national meeting, or advanced to be published in a journal.

Stage 5: Implementation

Beyond presentation and publication, it is essential for an AI project to be implemented in the real world setting using real world data. This project still requires continual surveillance and maintenance, as models and data often fatigue.

AI portfolio score

This relatively simple “stage to score” system can be effective for a single AI project,  as well as for the entire portfolio of AI projects in a center. For example, an AI effort in a health system with 5 projects can be scored as: Project 1: stage 3; Project 2: stage 1; Project 3: stage 4; Project 4: stage 5; Project 5: stage 1 for a total AI project portfolio score of 3+1+4+5+1 = 14. If the projects all successfully move one stage each during a quarter, then the total AI portfolio score progression is 19-14 = 5 for the quarter.

The AI portfolio score is therefore proportional to the number of AI projects, but concomitantly holds everyone accountable to progress (and increased score) of these projects. This simple scoring metric is useful to help allocate appropriate resources, as well as building team morale as the center matures.

Next week, for the final part of this series on AI in a health system, we will discuss briefly how the entire AI effort in a health system (beyond AI projects) can be measured and followed to assess AI readiness and capability.