In January 2020, radiologists at Vancouver General Hospital initiated an engagement with the UBC Cloud Innovation Centre to collaborate on the development of an AI model to analyze medical scans in response to COVID-19 pandemic. The result is L3-Net, an AI model developed and evolved collaboratively by radiologists and Machine Learning specialists, which is now in clinical use at Vancouver General Hospital.

THE JOURNEY

Early in 2020, radiologists from Vancouver General Hospital began looking into the feasibility of developing an AI model, focusing on analyzing CT scans of COVID-19 infected patients. The goal became to empower radiologists and provide metrics and statistical information about COVID-19 that could typically not be assessed by the human eye alone.

At that time, 3 in 10 people with the COVID-19 disease were testing negative and being sent home due to massive overcrowding in hospitals and a lack of resources for care. Infection rates were high and there was a palpable demand for additional tools or tests to help provide supplemental information and augment a physician’s decision-making toolbox.

The project started an initial phase of work that involved an ensemble cast made from a collaborative community including Vancouver General Hospital, Vancouver Imaging, SapienML, Amazon Web Services, Xtract.ai, the University of British Columbia, Element ai, and md.ai. This collaboration allowed the team to reach out to healthcare centers around the world to collect CT scan data from USA, China, South Korea, Australia, Canada, Italy and the Middle East of patients that ranged in diversity and infection severity.

This first phase resulted in Version 1 of L3-Net and enabled the new, improved, scalable L3-Net model of today.

TIMELINE

DATA

As quality ingredients are the key to a delicious meal, quality data is the building block of powerful AI. SapienSecure prepared a dataset that is comprised of COVID-19 positive scans as well as scans of patients that present with similar symptoms but had a different type of pneumonia.

The dataset contains CT studies from around the globe, including USA, China, South Korea, Australia, Canada, Italy and the Middle East. The variety in the data set was gathered to increase model generalizability, minimize bias and establish an accurate model for any site to either pull off the shelf and use, or reinforce with data from their own local population. As the goal of this project was to foster a collaborative development and research community, it was important that the dataset was not highly specific to any one region. The dataset also contained images from scanners made by many different manufacturers, including ones that are not common to the sites local to the developers. Not only does scanner/image quality vary, but so does the collection techniques and image post-processing methods. The scans vary in thickness as well as reconstruction techniques, which our model tackles flexibly.

“Fostering an open-source community and mentality around treating this illness may be the fastest and safest way to… foster enhanced collaboration between sites arounds the world.”

Brian Lee, SapienSecure, CTO

One of the key elements of this project was to empower doctors with more information than they ever had before. The model provides analytics that help a Doctor make critical decisions.

The model was developed as open-source, providing the opportunity for centres around the world to develop the model with their own patient data and enhance the model’s performance. The open-source nature of the model also enables the community to share their findings and foster enhanced collaboration between sites around the world.

AI is no good if it’s not accessible. One of the core tenets was to release everything open source. The architecture, the training algorithms, and even the weights were made publicly available to download. We promote discussion and criticism to foster a community that comes together to fight back against CoVID-19 and other infections.

If your team wishes to label your data to prepare for training on top of L3-Net, you can reach out to the team at Sapien ML: hello@sapienml.com

LABELS

SapienSecure lives and breathes medicine so when it comes to labeling high quality datasets, the organization turns to its medical experts. Dozens of radiologists, residents, fellows and medical students were brought onboard from Vancouver, Canada to create this model.

These Radiologists provide expert opinions on the presence and location of lung abnormalities caused by pneumonia (including COVID-19), as well as analyze the severity of the infection. As multiple radiologists gave their opinions, we aimed to eliminate bias in our annotations, as well as decrease variability by a radiologist’s interpretation of a scan.

Annotations were broken into three broad categories:

L3-NET

L3-net is not just another CT AI model. L3-Net was created to empower medical personnel with tools to augment their ability to make decisions involving patients with severe pneumonia (such as COVID-19).

L3-net is a clinically relevant approach to AI in medicine and is a continually evolving project that aims to provide quantitative measurements to radiologists.

The team behind L3-net worked closely with Medical Doctors called Radiologists to identify the features and characteristics in CT scans that correlate most strongly with poor outcomes in patients. Every facet of the model has grown organically from the needs of the radiologists involved with this project. Daily interaction between the Engineering team and Medical staff has shaped the development of L3-net every step of the way.

L3-Net performs well. The following are preliminary results:

PERFORMANCE

We have worked with health centers around the world to put together one of the largest international CT chest datasets.

EXPLORE THE TOOL AND THE CODE

The amazing Team that worked on the project are listed here. The models and code are available on Github. We encourage you to explore both. If you have questions about our approach, code, and/or labels, please reach out!  hello@sapienml.com

References – https://www.sciencedirect.com/science/article/pii/S1076633222002021