0:02

Hello, and welcome to today's Noon Conference co-presented

0:05

by MRI online and A A WR.

0:08

The A A WR was founded in 1981 to provide a form

0:12

for issues unique to women in radiology, radiation oncology,

0:15

and related professions.

0:17

The association sponsors programs that promote opportunities

0:20

for women and facilitates networking among members

0:24

and other professionals.

0:25

They have membership opportunities for those

0:27

who have completed their training.

0:29

Members in training

0:30

and international radiologist learn more about their mission

0:33

and membership@a.org.

0:36

We're thrilled to partner with a WR on these lectures

0:39

as part of our shared commitment to advancing

0:41

and supporting women in radiology

0:43

and transforming the way radiologists learn and thrive.

0:46

Today, we are honored to welcome Dr.

0:48

Bach Dewan for a lectured entitled AI at the Heart

0:51

of Breast Imaging Innovations and Insights.

0:54

Dr. Dewan is a clinical professor of radiology

0:57

and Eugene BP Frankel Endowed Scholar in Clinical Medicine

1:01

at UT Southwestern Medical Center, where she serves

1:05

as a member of its breast imaging division

1:07

and the Director of Breast Imaging Research.

1:10

She earned her me medical degree in Ankara, Turkey

1:13

and completed her residency training in diagnostic radiology

1:16

at Inca University Medical School, followed

1:19

by Breast Imaging Fellowship at University of Texas,

1:22

MD Anderson Cancer Center in Houston.

1:25

At the end of the lecture, please join her in a q

1:27

and a session where she will address questions you may

1:29

have on today's topic.

1:31

Please remember to use the q

1:32

and a feature to submit your questions so we can get to

1:35

as many as we can before our time is up.

1:37

With that, we are ready to begin today's lecture. Dr.

1:40

Dwan, please take it from here.

1:43

Thank you very much for hosting me

1:45

to share my perspective on artificial

1:48

intelligence and breast imaging.

1:49

Where are we now and where are we going?

1:51

I hope we will have a robust discussion

1:54

after, after the talk.

1:57

All right, so these are my disclosures for this talk.

2:02

I'm gonna start with, uh, a brief timeline of how we came

2:07

to be where we were at in, uh, breast imaging.

2:10

Um, so AI was actually born, uh,

2:13

back in 1956 in Dartmouth College,

2:17

and the initial attempts at computer, uh,

2:19

based image analysis dates back to 1960s.

2:23

And, um, very quickly, uh, development of CAD

2:28

for mammography started in 1980s and nineties,

2:31

and in 1998, FDA approved CAD for mammography, uh,

2:36

which was R two IME checker at the time.

2:40

And then, um, early on, the study suggested

2:43

that this CAD improved cancer detection rates by two to 10%.

2:47

So there were very promising,

2:49

but later down the line, uh,

2:51

subsequent data emerged suggesting that CAD is maybe not

2:55

that helpful in clinical practice.

2:58

Parallel to that, we had, um, development of, uh,

3:05

first applications of convoluted neural networks

3:08

to mammography, which first, uh,

3:10

focused on mass identification.

3:13

And later on in, uh, late two thousands and early 2000

3:17

and tens, advanced methodology

3:20

and, uh, availability of especially digital mammography

3:24

and hardware led to renewed interest in, um,

3:28

CNN architectures and deep learning.

3:31

And over the past decade, really the application

3:34

of deep learning to detection, diagnosis, tasks,

3:37

and mammography and other breast imaging

3:39

modalities absolutely exploded.

3:42

Uh, this includes segmentation, le lesion identification

3:46

and classification of, uh, masses

3:49

and microcalcifications in addition to, uh,

3:52

breast cancer risk prediction of patients

3:54

and density determination.

3:58

And, uh, one of the key, uh,

4:00

events in this timeline is the dream challenge,

4:03

which occurred, uh, in 2016 2017,

4:08

and the front runners really led the artificial intelligence

4:12

applications in mammography.

4:14

And, uh, since, uh, late 2000 tens,

4:19

multiple studies now suggest

4:20

that deep learning can enhance the diagnostic accuracy.

4:25

So when we say, uh, ai, there's so many uses.

4:28

I mean, it's infiltrated our lives in so many ways.

4:31

Uh, when you're scrolling in internet, um,

4:34

if you click on different, um, areas of interest,

4:37

you will see that, uh, the ads actually start showing you

4:41

what, uh, you're interested in.

4:43

And, you know, that's really AI at work, just trying

4:46

to determine your areas of interest.

4:48

When you're in Netflix, it will show you, um, you know,

4:52

your, the, the movies

4:53

that you were interested in in the past.

4:55

It'll show you more of that.

4:56

So, um, this actually goes to, uh, different aspects

5:01

of breast imaging in terms of appointment scheduling

5:04

and fight coordination, dose regulation, image positioning,

5:07

and quality assurance.

5:08

But today, uh, I just want to focus on narrow ai,

5:13

which is detection, diagnosis, triage, segmentation,

5:17

treatment response prediction,

5:19

and risk prediction, which are really tasks related to, uh,

5:24

what we radiologists do in our daily lives.

5:30

Um, a little bit on tech technical terms of, uh,

5:34

artificial intelligence, uh, terminology, machine learning.

5:39

Uh, you may have heard this before, um, machine learning

5:42

or models that, that derive parameters through being trained

5:46

by existing data, meaning we are showing

5:49

the computer what to find.

5:51

And this was essentially what conventional CAD was based on.

5:55

Um, you provide an input

5:58

and then it basically starts to try to find that input in,

6:02

uh, subsequent, uh, images.

6:05

Artificial neural networks

6:07

or is, is a model framework, which consists

6:09

of interconnected units

6:10

or neurons organized into different layers.

6:13

So there's, um, prioritization

6:16

of certain information over others.

6:19

For example, um, patients over the age of 40 are more likely

6:23

to develop breast cancer.

6:25

So, um, and a NN may actually prioritize patient age over

6:30

other findings that it may find on the image.

6:33

Deep learning refers to, uh,

6:36

these artificial neural networks that features

6:39

multiple intermediate or hidden layers

6:41

that we don't necessarily teach the, um, the,

6:45

uh, the computer.

6:46

So, uh, the most widely used, uh, deep learning architecture

6:50

for image classification, operating,

6:53

bypassing filters over on input

6:55

to extract these higher order features

6:58

that are highly predictive of.

6:59

What we're looking for is called

7:01

convolutional neural network.

7:03

So, um, so these are the different types of AI that, uh,

7:08

we use or that are tested in breast imaging.

7:14

Um, so I've been practicing breast imaging since 2001.

7:18

And, you know, since then, we're not foreigners to CAD

7:23

or computer aided detection.

7:25

So when I, I pressed on the CAD function,

7:28

I would see something like this, the triangles, um,

7:32

and the, the stars here would indicate mass

7:36

or asymmetries

7:37

or focal asymmetries on the triangles would indicate,

7:40

you know, calcifications.

7:42

So, uh, but there would be

7:44

multiple such annotations on a mammogram,

7:46

which could get sometimes very confusing.

7:50

So this is an example of, uh, a patient that we've seen at

7:54

that time, see down there, actually she has a mass,

7:57

I circled it, but this was not even, uh, found by the CAT

8:01

or indicated to me to look at by the cat.

8:05

So CAT has been FDA approved since 1998.

8:09

And by 2018, 92%

8:11

of screening mammograms were being interpreted using cad.

8:15

'cause that was, uh, mainly

8:16

because there was an additional code for it,

8:18

and we could bill separately for it.

8:20

But despite initial optimism, the largest, uh, studies, uh,

8:26

down the line failed

8:27

to demonstrate a very significant improvement in diagnostic

8:31

performance with cad.

8:33

Um, this is, uh, one of those landmark studies by, um,

8:38

Lehman and colleagues, which was published in JAMA

8:42

that showed really, um, no improvement in sensitivity,

8:46

specificity or cancer detection rate with cad.

8:48

Um, if you look up there, the cancers detected

8:51

for a thousand exams with CAD was 4.1 in a thousand.

8:55

And without cad, it was exactly the same.

8:58

Um, and same thing with invasive cancers.

9:01

There was some marginal improvements

9:03

with di carcinoma situ detected with cad.

9:07

Um, but if you look at the overall sensitivity, uh,

9:10

it was 85% with cad, 87% without cad.

9:14

And this was based on large national data

9:18

provided by breast cancer civilian consortium.

9:22

Um, so it was, it was pretty, uh, symptomatic of

9:26

what was going on nationwide.

9:29

So why was that happening?

9:31

The traditional CAD systems rely on detection of

9:35

hand engineered features such,

9:38

or, for example, masses, uh, defined by expert knowledge.

9:42

So they were delineated by radiologists at the outset.

9:46

So it primarily functions as a spell checker.

9:50

Uh, what does that mean? It, it,

9:52

it means really we have no hope of, uh, thinking that

9:56

Cath would actually find a cancer that we haven't defined

10:00

to it at the outset when it was being trained,

10:03

and there was no ongoing training in, in other words,

10:07

it didn't self-train as it was seeing more and more cases.

10:11

Um, well, how can deep learning, uh, be beneficial here?

10:16

Deep learning models learn directly from the data without

10:19

the need for explicit delineation

10:22

or feature engineering by the radiologists.

10:24

So they have the potential to learn from features

10:27

that are unseen

10:29

and unknown by radiologists,

10:31

so potentially they can improve the

10:33

radiologist's performance.

10:37

Um, and breast imaging was the per poster child for that,

10:40

because first of all,

10:42

we produced large scale categorical outcomes.

10:46

We have been using birads for many decades

10:49

before it was, it's implemented for other, um,

10:54

organ systems, really.

10:56

And, uh, we generated a large amount of data that is, um,

11:01

audited every year.

11:02

So we were actually recording our outcomes.

11:06

So we, uh, naturally became the most suitable

11:10

first applicators to train, validate,

11:12

and implement robust AI algorithms.

11:16

Um, this is a summary of early studies of deep learning,

11:21

so don't worry, I'm not gonna go over all of them.

11:22

But, um, just in summary, the, uh,

11:27

performance of even the very earliest applications

11:31

of deep learning were quite successful,

11:34

and they were pretty close to radiologist performance.

11:38

Um, I, I mentioned the dream, um, trial

11:42

or the dream challenge, which was, uh, also, uh,

11:47

funded by IBM

11:49

and lots of, um, other technology, uh, pioneers at the time.

11:54

And, uh, those were the, that's really what opened, uh,

11:58

the door to really wide scale application of ai.

12:01

And these are the, uh, the outcomes of the green challenge.

12:04

There were, um, three, um, top contenders.

12:09

Um, there were data sets submitted from both North America

12:13

and Sweden, and it seemed that the, um,

12:17

the performance a UC was, um, pretty good overall.

12:21

And when compared to single radiologists, um, this is,

12:26

um, which is, uh, Kaiser Permanente

12:29

and, uh, the consensus evaluation, uh, the performances

12:33

of those were, uh, pretty close, not quite at the level

12:37

of the radiologist performance,

12:38

but they were pretty close to the radiologist.

12:41

So, uh, no single algorithm reached the published, uh,

12:45

US Community Radiologist average, which at the time

12:49

was benchmarks were, uh, determined, um,

12:53

in, in a recent study.

12:55

Uh, and, and,

12:56

but an ensemble of AI algorithms combined

12:59

with a single radiologist assessment was associated

13:02

with an improved overall mammography performance over

13:05

a single radiologist alone.

13:08

What does an ensemble of AI algorithms means?

13:10

It's a combination of, um, the top performing algorithms.

13:15

So it wasn't a single one,

13:17

but multiple ones that was combined that, uh, was able

13:21

to improve a single radiologist assessments performance.

13:28

And, um, in a reader study of 14 radiologists, actually,

13:32

they just recently showed that, uh, reading

13:35

with AI support increased the main a UC by 0.2% as well.

13:40

So there's a marginal improvement,

13:42

but a, not a huge amount of improvement for, uh,

13:46

for single AI use.

13:49

So, um, can we actually, uh, do superhuman performance?

13:54

Meaning can we, can AI alone do better than the radiologist?

13:59

So, um, there are different reading algorithms, uh,

14:03

implemented in different countries.

14:06

So the settings are different in that in Europe, there's,

14:10

uh, they implement more, uh, double reading,

14:12

meaning a radiologist reads the, uh, mammogram first,

14:16

and the recalls are over read by a second, usually a more,

14:22

uh, experience radiologists.

14:24

And, um, most of the time they're on called

14:28

and in, in the USA, uh,

14:31

we have a single radiologist setting.

14:33

So we have to really test

14:36

how this fits into these two different scenarios.

14:39

So when they looked at that, uh,

14:42

a deep learning based ensemble algorithm, uh, they,

14:47

it did demonstrate superior sensitivity

14:49

and specificity compared to

14:51

the original interpreting radiologists.

14:54

Uh, there was about a 10% increase

14:57

and a, uh, a 5% increase in specificity

15:01

and a 10% increase in sensitivity in the US data.

15:05

And, uh, also, AI outperformed radiologists

15:10

with an overall increase in both

15:12

sensitivity and specificity.

15:14

But if you, if you look at the improvements,

15:16

these were mainly in the single radiologist setting.

15:21

When you look at the double radiologist setting, uh,

15:24

the red.here represents, uh, the consensus reading.

15:28

Um, and the, the, um,

15:33

green dot represents the single radiologist.

15:36

So there was really, uh, 1% improvement in specificity

15:40

and 2% in sensitivity here,

15:42

whereas the consensus was much better than, um,

15:46

both the single reader and the AI here.

15:49

So it depends on how we are reading it.

15:52

So it seems like, uh,

15:54

if there's a single radiologist reading

15:57

AI will be a huge help, uh, which is relevant to us, uh,

16:01

practicing in the, the us.

16:05

So we talked just about mammography.

16:07

What about AI performance and ultrasound?

16:11

Breast ultrasound received, uh,

16:13

less attention than mammography in the literature, uh,

16:16

because it's, um, it's more user dependent

16:20

and, um, equipment dependent.

16:22

And, um, you know, how you scan a patient can make a huge

16:25

difference in the imaging findings.

16:28

But the available data suggests

16:30

that DL can achieve satisfactory performance in

16:33

classification of masses seen on breast ultrasound.

16:37

There are fewer, uh, published comparison studies here,

16:40

but, um, deep learning for breast ultrasound,

16:44

it looks like it may outperform the traditional CAD

16:47

approach, and, uh, performs comparably

16:50

to human radiologists.

16:53

And again, these are, um,

16:55

the published literature on the subject as well.

16:57

But I just want to focus on one really large, uh, study

17:02

which showed, which was a reader study of deep learning

17:05

for benign versus cancer classification versus three

17:08

human readers.

17:10

And the deep learning, uh, achieved a uc of 0.84,

17:15

uh, versus zero point 89, 89,

17:17

and 0.79 for human readers,

17:20

which was pretty much equivalent.

17:22

So this could be really relevant in centers that, um,

17:27

do screening ultrasound for, uh,

17:30

for breast cancer as a supplement.

17:33

What about breast MRI?

17:35

The diagnostic AI development is similarly complicated

17:39

by a 3D problem space

17:41

because we have multiple sequences there to think about

17:45

and the, uh, the dynamic setting.

17:48

So which, um,

17:51

phase in the dynamic setting is the most valuable here,

17:54

and do we discard the rest?

17:55

So, um, for that reason, most classification, uh,

17:59

studies have focused on the subtraction series

18:03

or the early subtraction series

18:05

and targeted classification of 2D or 3D ROIs.

18:10

Um, so comparison studies are rare,

18:12

but again, um, the more recent studies show

18:15

that deep learning may outperform other machine learning

18:19

approaches, while the results

18:20

for the human comparisons are more mixed here.

18:25

Um, so for example, I chose this

18:27

because it's, it's, uh, more representative of, um,

18:32

the what's published in the literature.

18:35

There is a comparison of both radios as well as the humans.

18:40

And, uh, the comparison between the deep learning radios

18:44

and radiologists for classification of lesions.

18:47

On dynamic MRI, it showed that, uh,

18:50

deep learning outperforms the radis models,

18:53

but not quite the radiologist.

18:55

Uh, the red line here shows the radiologist performance,

18:59

whereas the deep learning is the green line.

19:02

So those are two are most close to each other,

19:05

and, uh, the radios approach didn't do quite as well.

19:12

So we have the data out there,

19:15

but how do we actually use it in our clinical practice?

19:19

So one idea is to use AI as second reader,

19:22

and this one is the most popular one, uh,

19:25

AI enterprise the case.

19:27

And then radiologists enterprise the abnormal cases,

19:31

and then radiologists determines the final interpretation.

19:34

BI scan has been shown to improve diagnostic accuracy,

19:37

as I've shown before, and, uh, it would

19:42

translate into double reading the mammogram in

19:45

resource limited settings.

19:47

The other way we can use it is as a triage method.

19:51

AI flags just the abnormal cases.

19:54

Um, you can generate a work list sorted based on

19:58

AI assigned priorities.

20:00

I'm gonna show you an example of that.

20:02

And the radiologist reads cases in order of priority,

20:05

and you can prioritize the negative cases if you like,

20:09

or the, um, highest, uh, probability

20:13

of malignancy cases.

20:15

And, um, theory, this can improve the workflow

20:19

and it can lead to a faster turnaround for protocol results,

20:23

especially if this is done at a mobile setting

20:27

or, uh, in a, in a resource, uh,

20:32

scarce setting where patients have difficulty, uh,

20:35

coming back for their additional workups.

20:38

Another way we can use AI triaging is, uh,

20:42

you can triage negative cases,

20:44

and if you can, um, trust AI sufficiently,

20:48

you can actually just give them a negative report.

20:51

And radiologists can only read the cases flagged

20:55

as abnormal, or you can let AI read all of the cases

21:00

and radiologists reviews only the abnormal

21:04

or uncertain cases, which are not highly, uh,

21:10

ranked as abnormal.

21:11

Um, there's full somewhere in between, um, 40 to 40,

21:15

50% chance of malignancy.

21:18

So in both of these scenarios, um, there's a potential

21:21

to decrease the radiologist workload

21:23

and allow more time for reading positive cases

21:27

and performing other duties for the radiologist.

21:32

Um, so they actually tested the simulation

21:35

of a deep learning triaging scenario,

21:38

and they, uh, this was by a homegrown model

21:42

by MGH

21:43

and MIT, uh, which are really, they, they produce a lot of,

21:48

uh, research on this, on this topic, on large data.

21:53

And, uh, in a retrospective simulation study

21:56

of a triage scenario in which you

21:59

basically give the patient a negative results based on

22:03

just AI prediction.

22:05

Um, then they said, well, 19%

22:08

of the mammograms would've been excluded from review without

22:12

compromising any diagnostic performance

22:15

or missing cancers that would be

22:17

otherwise detected by the radiologist.

22:20

So you have, uh, 90.6%, uh,

22:25

sensitivity with the radiologist and 90.1%.

22:29

So there's a bit of a drop

22:30

with sensitivity there if you are tolerant to that.

22:34

Um, and, uh, about the same specificity, maybe a bit, uh,

22:39

higher specificity.

22:41

And, you know, you could basically OMI reading 19%

22:44

of the mammograms, which doesn't seem like a,

22:48

um, a huge savings.

22:49

But, um, there it is.

22:51

And more recently, uh, AI

22:54

as a standalone mammography reader was tested

22:57

in another homegrown model by, uh, NYU.

23:01

And then they said, well, at least 40%

23:04

of true negative screens could have been read by AI

23:08

with zero basis.

23:09

So this is really, really promising,

23:11

and it seems that the, the higher the number of exams

23:15

that you actually give ai, the better the prediction,

23:20

uh, performance.

23:25

And, uh, so the question here was also I was interested in

23:29

was, um, what about the callbacks?

23:31

Do we have an increase in callbacks?

23:34

And there was actually, um, not an increase in the callbacks

23:38

by AI use, but a decrease of, uh,

23:41

20% if you set the AI triaging to 30%.

23:46

And in the 40% that they were presenting,

23:48

there was an actual savings of the callbacks of about 30%.

23:52

So this was a really, really good forming model.

23:55

Um, so how do we use it in daily practice?

23:59

We do have an integrated, um, ai, um,

24:04

model in mammography model in our, uh, practice.

24:07

And when I first log into my workstation, this is

24:10

what it looks like, and it, on the right,

24:13

you have a case score here.

24:14

These are the screening mammograms of the day.

24:17

And, um, there are the higher case scores of,

24:20

I actually start looking at the higher case scores first,

24:23

so you can, uh, rank them the way you like it, uh,

24:28

order them the way it's ranked,

24:30

and you have the zeros down here.

24:33

So, um, for example, if, um, this is from another day,

24:37

you have these really high ranked, uh,

24:41

screening mammograms.

24:43

Um, I clicked on one of those just to check, uh, the case,

24:48

and it seemed that it actually highlighted these segmental

24:51

calcifications in the right breast,

24:53

which really I would've called back.

24:55

And, uh, since then, the patients had, um, uh,

24:58

a stereotactic biopsy confirming di gluc

25:01

carcinoma CY as well.

25:03

So this is really helpful in getting the more abnormal

25:07

cases read early on

25:09

and working out the patient

25:11

potentially while she's still there.

25:14

So what about, uh, mammograms as predictors of risk?

25:18

This is actually for me a more, uh, exciting area

25:23

of, uh, developments

25:25

because I mean, yes, there's improvements

25:28

and there's triaging available for, um,

25:32

for image interpretation,

25:34

but we could do things, AI can actually do things

25:37

that we can't do with our naked eye.

25:41

So density has long been a big issue

25:45

with us reading screening mammograms, and, um, not just

25:50

because it masks cancer,

25:51

but mammography breast density is associated with

25:55

over two times increased relative risk for cancer in women

25:59

with extremely dense breast.

26:01

Um, and we do have automated quantitative tissue density

26:05

tools that provide objective measurements

26:08

that are commercially and publicly available.

26:10

It's, I think we widely use it,

26:12

but it's not just about quantification.

26:15

And there is, um, density signatures in there

26:19

that actually predict the patient's future breast, uh,

26:22

breast cancer risk,

26:24

the parenchymal complexity features which are independently

26:28

associated with the breast cancer risk.

26:30

This is not about the percent of breast density,

26:33

but the quality

26:35

or the texture of the, um, the breast density,

26:38

which we can't always quantify with our naked eye.

26:42

So, um, this is really, really interesting

26:46

because, uh, it, again, the,

26:48

the very productive group from MIT

26:50

and MGH have come up with, uh, uh,

26:54

predictive, uh, model that used mammograms,

26:59

and they took, uh, patients traditional, uh, risk,

27:04

uh, features, and then they fed that to the model.

27:07

And then they also used the, uh, the normal mammograms.

27:12

And they also used, um, compare that

27:15

with Tyro CIC version eight,

27:17

which is the most common model we use

27:19

to predict the patient's future breast cancer risk.

27:22

And we base our, uh, our decisions as

27:25

to whether patient needs supplemental imaging

27:27

or supplemental intense screening based on

27:30

this clinical model.

27:32

Um, they found that, uh, traditional risk factors

27:35

didn't do very well.

27:38

Um, they were better than tyro kic, though Kuzi did

27:43

not very well at all, uh,

27:44

to predict the patient's five year breast cancer risk.

27:47

But if you looked at mammograms alone, um,

27:51

it actually outperformed both tyro CIC

27:54

and the traditional risk factors.

27:56

Um, and if you actually combine those traditional risk

28:00

factors with imaging that actually perform the best.

28:04

So I think this is very, uh, tra groundbreaking, uh,

28:08

developments in

28:09

that we could do better actually at predicting which

28:12

patients would benefit from, uh, supplemental screening

28:16

or more in intense screening, um, in this patient group.

28:21

Um, and again, uh, you could use AI for,

28:25

for different reasons as well

28:27

besides that of predicting risk.

28:30

So could we predict, uh, the presence

28:34

of external lymph node metastasis?

28:36

So this was, uh, a study that used ultrasound ROI images,

28:40

and, you know, they, they found much, much,

28:44

much better prediction than what you would normally do, uh,

28:48

by naked eye or radiologist evaluation of traditional, um,

28:53

uh, values or parameters that we would use.

28:57

Um, another, um, clinical decision making use for AI is,

29:03

uh, preoperative prediction of pathologic complete response.

29:07

So her two cancers, her two positive cancers, as you know,

29:11

they have a very high response rate.

29:14

Um, but having HER two positive cancer alone could be a huge

29:17

predictor of response,

29:19

but that did only, uh, perform with an a UC of 0.7.

29:24

Uh, but when you added that to imaging based, um,

29:28

or AI based, uh, prediction

29:30

and the HER two status, it certainly improved

29:34

that prediction because this is an area we're not

29:36

very, very good at.

29:38

In, uh, in, in breast imaging, unfortunately,

29:43

we also used it for a prediction of axillary metastasis, um,

29:48

predicting that on breast MRI here in UT Southwestern, uh,

29:52

we developed both a clinical model

29:54

and an imaging based model.

29:56

And, um, so you're seeing our results based on test data,

30:01

um, for clinical node prediction

30:04

and, uh, pathologic node prediction in patients

30:08

who actually had surgery.

30:10

And we had a pretty, uh, respectable outcome with this.

30:14

And more importantly, we didn't really have

30:17

to look at the patient's axilla at all.

30:20

These were just made based on, uh,

30:23

just looking at the tumor characteristics

30:26

or derived from, uh, from MRI features of the tumor alone.

30:32

Um, so when we looked at the, uh, on the,

30:36

on the test set performance, we had a zero point 87.

30:41

Um, again, this means that it's actually applicable, uh,

30:45

clinically and, uh, overall, um, median sensitivity

30:50

of 89%

30:51

and the pretty high specificity of 76%,

30:57

um, key points from that study was

31:01

MRI based hybrid model showed an improvement over

31:06

the 77.6% radiologist sensitivity

31:09

that we had in our institution,

31:12

and we had a 55% specificity in that half

31:16

of our patients actually underwent, um, uh,

31:21

uh, a biopsy.

31:23

But, you know, half

31:25

of them actually had a positive result after that.

31:28

So we were sending patients to biopsy for, um,

31:32

finding not a lot of cancers there.

31:35

Uh, and one of the benefits of the model was scalability.

31:38

So when with all AI models, really, um, if you

31:43

set the specificity rate to 71%,

31:46

our model yielded 91% sense,

31:50

which means we only missed at 9% of the metastasis.

31:56

And if we had implemented our model,

31:59

it would've helped avoid more than health of

32:02

benign operative sentinel lymph node biopsies,

32:04

which unfortunately most of the, uh,

32:07

invasive cancer patients have, while we were able

32:10

to detect 95% of patients with OID metastasis.

32:15

Um, so one of the concerns about using these models is,

32:20

well, I mean, it performs good in your institution,

32:23

but what about if I implemented in my institution,

32:27

how would it do then?

32:28

So we have an advantage here, uh, UT Southwestern

32:32

where we are affiliated in both a tertiary care center,

32:36

the university hospital, and a safety net hospital.

32:41

So we have collected data from both about 400 patients.

32:46

And what we did was we did an exercise in which we developed

32:50

the model in one, uh, hospital's, uh, data,

32:54

which are completely, uh, different, uh,

32:58

patient characteristics, even cancer age, even the types

33:01

of cancers are slightly different in the safety net hospital

33:05

versus the university hospital.

33:08

And we were able to, the performance, see

33:11

that the performance of the date of the models were the same

33:14

when we developed it in one and applied it to the other.

33:17

And that's really what we ought to look at in all, um,

33:21

AI vendors that are, uh, marketing their, um,

33:24

their product out there, how would it perform, um, not in a,

33:28

in a test or a lab environment,

33:30

but in a real life environment.

33:32

So, um, this is the promise,

33:36

and now I'm going to shift gears

33:38

and, uh, explore, you know, what are some of the problems

33:42

that we're dealing with daily, um, clinical implementation.

33:47

So, uh, the legal re responsibility is a big one.

33:50

Uh, who takes the responsibility of the decisions

33:54

of the AI systems hospital vendor

33:57

or the radiologist, um, who would handle the disagreements

34:01

between the AI and the radiologist?

34:04

Um, so if I overruled a lesion flagged by ai,

34:08

what would be the liability of that?

34:10

And if a radiologist cannot identify a specific lesion,

34:14

how will that case will be managed?

34:17

So if ai, you know, flags something,

34:20

and then you work it up,

34:21

and then there's, there's no correlate for that.

34:24

So how would we go about that?

34:26

Um, and the AI detected cases, um,

34:31

so would they lead to more recalls for the radiologist?

34:34

And that's a, that's a a real question

34:37

because there's some data out there that suggests that if,

34:41

uh, AI does the tech cases

34:44

and, you know, in a triaging environment where, uh,

34:47

radiologists are only looking at the positive cases

34:51

and no negative cases, that would actually

34:55

increase our callback rates,

34:56

which are already under fire right now.

35:00

Um, transparency and reliability are also major ones.

35:05

Uh, interpretability

35:06

of an AI algorithm is key in under understanding

35:08

how prediction of are made,

35:11

and some black box models are generated

35:14

directly from the data by an algorithm.

35:15

So we don't know how, uh, the, uh,

35:19

AI model finds these cancers

35:22

because we haven't really shown it how to find it.

35:26

So it is very imperative to make model development code

35:29

and testing data sets accessible to the public.

35:32

And this is also one of the things we ought to look for

35:36

when, uh, we are looking at marketed products.

35:40

There is a true concern about, uh, model drift,

35:43

which is a concept that is

35:48

degradation of a model's prediction power due

35:51

to changes in the environment over time.

35:53

So it's very important to do real time surveillance

35:56

to monitor for unexpected model performances,

36:00

to make sure the algorithm, uh, meets the standards

36:03

and doing exactly what it's supposed to do

36:06

at the outset when you, we are first, uh, installing them.

36:10

And, uh, there's now, uh, uh, a recommendation by a CR to do

36:15

actually, um, a periodic checking on this.

36:20

And this is an example comparing three commercial artificial

36:24

intelligence algorithms for, uh, independent assessment

36:27

of screening mammograms.

36:29

And, you know, these were done at the patient level on non

36:33

annotated mammograms, and the counselors were not localized

36:37

and without insight into why AI decisions were made.

36:41

So we don't actually know that the cancers reported here.

36:46

Were actually the same ones that were, um,

36:50

detected down the line at diagnostic workup.

36:53

There was a, there was a cancer prediction done,

36:56

but it wasn't, we don't know if it's the same one

36:59

as the actual diagnosed cancer.

37:02

It could be elsewhere in the breast,

37:04

or it could be the opposite breast.

37:06

So, um, when vendors are touting their, uh, performances,

37:10

it's very important to be careful

37:12

and look, um, between the lines there.

37:16

So approval and regulation of AI systems, um, this is,

37:21

um, this is a, a developing area.

37:24

Um, there's increased FDA evidentially, uh,

37:28

regulatory standards recently,

37:31

but, um, development

37:33

of more improved course marketing surveillance

37:36

and trials are needed.

37:38

Um, currently, uh, they're regulated as software

37:41

as a medical device, um,

37:44

and which means software intended to be used for one

37:47

or more medical purposes that performs

37:50

these purposes without being part

37:52

of a hardware medical device.

37:53

So that's, that's just the code that they have.

37:57

And, uh, the concern with that is AI use adaptive algorithms

38:01

that are dynamic can change in, respond to new information.

38:06

So adaptive algorithms are very difficult

38:08

to regulate in the current framework

38:11

because the product initially reviewed may be different from

38:14

the one the consumer ultimately encounters.

38:18

So most products are actually locked

38:21

after FDA, um, approval that's obtained.

38:25

So this is good and bad.

38:26

So you can actually look at the data presented to A FDA

38:31

and you know, you can assume

38:33

that's the same performance you're gonna get,

38:35

but, uh, it could be bad

38:37

because there's new data that

38:40

could have developed over time.

38:42

And, um, for example, you know, tomosynthesis

38:45

or synthetic, uh, mammograms.

38:49

Um, so you don't know that they're doing, um,

38:53

as well on that new data

38:55

because there hasn't really been an interval training on

38:58

that, that algorithm.

39:01

So there has been discussions about this where, you know,

39:04

clinical prospective trials, uh, may be needed

39:09

for AI products before clearance or approval,

39:12

but it, it's not in implementation at this point.

39:17

Um, I'm a research director in my institution,

39:20

so I always look at what are the barriers and research.

39:24

Um, we need large, high quality data sets.

39:27

I showed you the, the higher, the more the data,

39:30

the better the algorithm, uh,

39:33

with accurate ground truth is needed, which, um, you know,

39:37

we, I think we have a lot with mammography,

39:39

but not so much with the other, um, uh, modalities.

39:44

So we gotta be very careful with those.

39:47

And the training data sets

39:48

that are often collected from the same institution.

39:52

Uh, and cross validation is now mandatory in

39:55

publication standards.

39:56

I mean, if you go and look at, uh, majority of the, uh,

40:01

respectable, um, journals, they, they request this now.

40:07

So, um, and the cyber attacks, of course, um,

40:12

you need to invest in a strong security infrastructure.

40:15

When we are dealing with these third party vendors.

40:19

Um, there are even more biases that I wanna touch on,

40:22

which is, um, bias is a big one,

40:25

and there may be several biases in these algorithms

40:29

that are developed.

40:31

Representation is a big one.

40:33

Um, FDA clearance of AI products for mammography screening

40:38

did not necessarily report the race

40:40

and ethnicity of the patients included.

40:42

And you know, when you look at the most available training

40:45

data sets used in AI development of vendors,

40:49

they actually contain mostly mammograms of, uh, white women.

40:54

And, uh, this was a very interesting study in JAMA published

40:58

a few years back, and it showed that patient cohorts used

41:01

to train AI algorithms in the us, uh,

41:05

originated mainly from three states, California,

41:09

Massachusetts, and Europe.

41:11

So this is really important when you're looking

41:14

to implement it in your own institution.

41:16

If it's not representative,

41:18

if your population is not representative, then um,

41:22

you may not be, you may, it may not perform as well as, uh,

41:27

what's published out there.

41:30

There are socioeconomic disadvantages.

41:32

There's missing data on disparate

41:34

or socioeconomically disadvantaged populations,

41:37

which may lead to underperformance

41:39

of the AI in the underrepresented individuals.

41:42

And this is primarily why we did that

41:45

exercise in our own data, uh, in our own, uh, publications

41:48

with, uh, lymph node metastasis prediction, uh,

41:52

comparing the safety net versus, um, the CUH,

41:57

because we know we rotate at both sites,

42:00

and we knew that the patient profiles were very different.

42:03

And also our resource poor communities may not be able

42:06

to take advantage of AI's potential patient care benefits

42:10

because they actually lack the resources

42:13

to purchase these commercially available AI products.

42:16

I mean, the promise is always that these, uh,

42:20

will actually help with the disparities

42:22

and, um, you know, where radiologists are not available,

42:25

you can deploy them, but really they're being marketed

42:29

and you, you know, it's, it may not be possible

42:32

for resource poor communities to access these tools.

42:37

Um, this is a very, very interesting, uh, study

42:39

that I wanted to share with you.

42:41

Um, I had the opportunity to actually work with one of the,

42:45

um, PI BIS of this study, um, the,

42:50

and it is on AI recognition

42:53

of patient race in medical imaging.

42:56

So they actually looked at the capability of AI

43:00

to predict patient race without actually giving

43:03

the AI this information.

43:05

And if you look at it, I mean,

43:08

the AI can accurately predict race, um, quite well,

43:12

especially in chest x-ray.

43:14

I was actually gratified to see mammography is kind of

43:17

down there in the rankings, but still pretty high.

43:21

Um, so that actually tells you there is a bias right there

43:25

because AI can actually predict what the patient's, um,

43:30

race is without you realizing that it did.

43:34

Um, again, uh, more implementation challenges,

43:38

translation from proof of concept

43:40

to a commercially available products is a problem.

43:42

I showed you a lot of homegrown, uh,

43:47

AI models, but, um, a lot of them are not actually available

43:51

for patient use at this time.

43:54

And again, recognition of biases that may arise

43:58

with human AI interaction.

43:59

What does that mean? If AI says,

44:02

and a mammogram is negative, I may be more inclined

44:05

to call it negative and ignore, uh, cancer that's,

44:09

I may have otherwise called back,

44:11

or if AI says a mammogram is, uh, has cancer in it,

44:16

I may actually look for something that I haven't,

44:19

otherwise I wouldn't otherwise call back.

44:22

And this is called automation bias.

44:25

And then there is the reimbursement considerations.

44:28

There's, uh, currently no, uh, existing CPT code for,

44:32

uh, AI in mammo.

44:34

There are two category three codes for quantitative analysis

44:37

with ultrasound applicable to ai.

44:40

So we don't know if there will be reimbursement

44:44

and, uh, if that reimbursement happens,

44:48

if it's gonna be a positive reimbursement

44:50

or a negative reimbursement.

44:51

And we can actually maybe chat more about that

44:54

after, um, after the lecture.

44:58

And, uh, incorporation into existing PACS

45:01

and EMR systems requires a substantial, um,

45:05

infrastructure sometimes

45:08

and the real time surveillance to monitor

45:11

for unexpected model performance that may be secondary

45:14

to underlying biases.

45:16

Um, I don't know that a lot

45:18

of institutions are prepared for this.

45:20

I know that we are starting to, um,

45:23

do these routine audits in my institution.

45:25

We're getting ready, but it's really difficult

45:28

to find expertise and allocate time

45:32

and, um, resources to, to this, to make sure

45:36

that the AI is performing as it should, uh,

45:39

not at the outset, but after, uh, multiple years.

45:45

Um, these are the training data sets available

45:47

for breast imaging and AI algorithms.

45:50

Again, this is really relevant, uh,

45:53

to the resources out there, but

45:56

whenever, uh, someone tries to develop an algorithm,

45:58

usually this is the first go-to, uh, resource, they go

46:02

to one of these, um, you know, the high number of image

46:07

data sets, free data sets available,

46:10

and, you know, this is where they start out.

46:15

And lastly, I wanted to touch on what about the patients?

46:19

What do they think? We don't have much data on, um,

46:24

the US population,

46:25

but there was a recent Dutch population which was published

46:29

in JACR.

46:31

And it interestingly said that most patients did not fully,

46:36

did not support fully independent use

46:38

of AI-based screening interpretation,

46:40

meaning they didn't want to just get their results from, uh,

46:44

the computer right after they have their mammograms.

46:48

About, uh, 42% were opposed to the idea of using AI

46:53

to select cases that require a second reading.

46:56

And there was really no agreement on

46:58

who would be responsible for a diagnostic error.

47:01

And, uh, I mean, I think if we polled the US population,

47:06

I don't think we would get very, very different results,

47:09

but this was what the patients thought.

47:13

In closing, I want to say

47:15

that AI could improve radiology accuracy

47:17

and workload by serving as a second reader

47:20

or by excluding the normal findings from human review,

47:24

which could, um, improve our workload.

47:28

Um, it holds tremendous potential to improve the risk

47:30

and prognostic prediction,

47:32

which is really impressive in my opinion.

47:35

However, there are various ethical medicolegal

47:38

and logistical issues pertaining to its full implementation,

47:42

which remain unsolved.

47:44

And we need more high quality research,

47:47

especially prospective data, to see

47:51

how it performs in real life setting

47:54

and not just in the lab setting.

47:58

Uh, radiologists are poised

47:59

to take a prominent role in the ongoing discourse about

48:02

incorporation of AI into clinical practices.

48:05

What I believe, and there are, um,

48:10

in the future, there are, um, opinions that, uh,

48:14

if AI lives up to its potential breast imaging,

48:17

radiologists could have more time

48:18

to pursue the more personally

48:20

and professionally fulfilling aspects of their jobs,

48:23

including a patient interaction collaboration

48:26

with colleagues and teaching of trainees.

48:29

This is a quote from, uh, Dr.

48:31

Smitherman who, uh, brought on the economic impact

48:34

of AI on breast imaging and JBI.

48:37

Um, I, I think that could very much happen,

48:40

but, um, it's all about how we, uh, implemented.

48:46

I wanna thank all of my research team here

48:49

and, um, I,

48:52

we wouldn't have generated our data without their invaluable

48:55

help and also point that, uh, a new really good summary

49:00

that we published on this topic in JBI.

49:04

Thank you very much.

49:06

Thank you so much for that awesome lecture, Dr. Dewan.

49:09

And we will open up the floor for any questions folks have,

49:14

and if you could submit those through the q

49:16

and a feature, that would be great.

49:20

Um, we've got one in there now.

49:22

Um, did AI decrease the workload for prediction

49:26

or does it increase the workload?

49:31

So, um, with prediction, uh, I understand that.

49:37

So when you say prediction, I guess my,

49:40

that question would be, would that be prediction

49:43

of versus malignant,

49:45

or would this be prediction of, um,

49:49

future risk of the patient?

49:53

So for future risk assessment, um, of breast cancer risk,

49:58

um, I think we might consider that it,

50:01

it may increase the workload of the, in that, uh,

50:05

if the pa if we the patient is, uh, average risk,

50:09

and you know, we're thinking that the patient's

50:12

risk prediction increases, then we can expect to see more,

50:16

um, additional imaging or supplemental screening

50:19

or MRI, ultrasound et cetera, for those patients.

50:23

But if we are, um,

50:25

talking about benign versus malignant classification, um,

50:29

it's predicted to decrease it

50:31

because the idea is to really not look at those ones

50:36

that are predicted as zero

50:38

and chance of malignancy by AI at all.

50:40

Just like the workflow I showed

50:42

that we have here in UT Southwestern.

50:46

I actually have a question. The, um,

50:48

that patient slide you had about folks not

50:53

feeling super comfortable getting a report, uh, using ai,

50:58

is there something to be done

51:00

with helping folks be more comfortable with that?

51:03

Is it patient education?

51:04

Is it attending more lectures like these?

51:07

What, what is, what is something

51:09

that could make patients feel a little more comfortable

51:11

with this new landscape?

51:15

Um, that's a good question.

51:16

So we, in that study,

51:18

and we also did a study here, uh, in our center as well,

51:22

it seems that they hire the higher the patient's education,

51:27

uh, there there is an increase in, uh, trusting

51:31

or, uh, relying more on ai.

51:35

So whether that's, um, whether that's relevant to,

51:40

uh, a higher interaction with AI

51:42

or not in their daily lives, um, you know, that's,

51:46

that's a question we didn't ask

51:48

and they didn't ask in that study either.

51:49

But, uh, patient education seems to be one factor.

51:54

And, um, here in our, uh, data set, we found

52:00

that, uh, patient race did play a role, um, with, um,

52:06

non-Hispanic African American patients

52:09

and, um, Hispanic patients showing, um, more, um,

52:15

of distrust towards AI than, um,

52:19

non-Hispanic white patients.

52:20

So, so there are a lot of factors that play into this,

52:24

but, um, maybe it is patient education

52:27

and it could be that, uh, sometimes patients' interaction

52:32

with AI are not positive at all in daily life,

52:35

and they think that's gonna, what's gonna happen here.

52:38

Hmm.

52:39

Yeah. Interesting. I'm, I'm curious to see

52:41

how this evolves as, as this becomes implemented. Other

52:45

Component of that, sorry to interrupt you,

52:47

is they don't want

52:48

to lose their interaction with the doctor.

52:52

That's a big, um, factor in the patients not wanting

52:56

to just interact with the AI

52:57

because that interaction is very valuable to them.

53:04

Yeah. Interesting. Thank you.

53:09

Do you have any experience in using AI

53:11

and mammography with genetical tests as a predictor of risk?

53:15

Or what is your opinion about that?

53:18

That's a good question. We haven't done that.

53:20

We don't have that, uh, software I showed in my talk,

53:25

but I believe it has now been made available into a

53:30

commercially available product, um, by the group

53:34

that developed it.

53:35

But we don't have that implemented here.

53:40

Does your practice currently allow AI negative readings

53:42

to not be reviewed by a rad radiologist?

53:46

We don't because, uh,

53:47

the regulations don't allow for that at this time.

53:50

So how is this gonna play out?

53:52

Is really, it's gonna be very,

53:54

very interesting in the future

53:56

because, um, of the liability situation,

54:00

we don't know who is to take the liability for an,

54:04

for wrong AI interpretation or a wrong AI recall.

54:09

So, uh, at this time we are

54:12

basically using it as a second reader.

54:15

Um, but uh, there, I think there's implementation in Europe

54:20

actually for, to that effect

54:21

because it, they're doing double reading

54:25

and, uh, they're really, really behind on screening, reading

54:29

and, uh, their payment model is

54:32

based on that double reading.

54:33

So, uh, because it, on calls, the,

54:37

their call right back rates are really, really low,

54:40

much lower than average American, right?

54:42

So, uh, they have, uh, they can maintain their, um,

54:48

national mammography, uh, programs

54:51

where they offer it to everybody.

54:53

So in, in the US it's opportunistic

54:55

mammography screening as you know.

54:57

So, uh, with them, actually, I think a lot

55:01

of them are thinking about going live with, uh, AI reading

55:05

and only having, uh, the, the positive

55:09

or abnormal ones being looked at by the radiologist

55:12

because they're really behind and backlogged

55:14

and there's really not a whole lot

55:15

of press radiologists in the pipeline being trained.

55:19

So, um, I think that's the situation there.

55:22

So we may see it roll out in Europe

55:24

before it does in the UQS.

55:27

Gotcha. Let's say the AI gets something wrong.

55:31

How do you rectify it

55:34

or retrain it to correct itself in the future

55:38

for any other case it might see?

55:42

So at the time we received, um,

55:46

the AI model, it's actually locked.

55:50

So because, you know, I told you about the model drift

55:52

and model drift can occur even after being locked,

55:55

but, um, the, the, the changes

55:59

and, you know, the additional developments in the,

56:03

in the model is not allowed.

56:06

So that's both a good thing and a bad thing.

56:09

It's a, it is a, it's a good thing

56:10

because you know what to expect, but it's a bad thing

56:13

because, um, I mean, it would not be bad for you

56:17

to actually train it on your own dataset

56:19

and make sure it performs better in your own dataset,

56:23

but that's not currently allowed, uh,

56:26

because FDA requests that from the vendors.

56:29

So in answer to your question, I don't,

56:32

we can't really improve

56:34

or change, uh, commercial models, um, performance.

56:38

We can't, we might be able

56:39

to do it without commercialization if we have a homegrown

56:43

model that we're using, um, it might be possible,

56:47

but, um, it, it's not possible

56:50

for a commercial model at this time.

56:53

Gotcha. Alright, I think we're going

56:55

to wrap up today's noon conference.

56:56

Thank you all for participating in our noon conference

56:59

and asking such great questions.

57:00

You can access the recording of today's conference

57:03

and all our previous noom conferences

57:04

by creating a free MRI online account.

57:07

We'll also email out a link to the replay later today.

57:10

Be sure to join us next week on August 22nd at 12:00 PM

57:14

Eastern, where Dr. Reed Omari,

57:16

we'll deliver a lecture in entitled Radiology's Opportunity

57:20

to Benefit Patients and the Planet.

57:22

You can register for that@mrionline.com

57:24

and follow us on social media

57:26

for updates on future NOOM conferences.

57:28

Thanks again for learning with us and have a great day.