Mole or Cancer? How Much Can We Trust AI Applications?

Using artificial intelligence (AI) to diagnose skin cancer with dedicated devices or smartphone applications has become popular in recent years. However, there are concerns about their diagnostic accuracy .

So far, two computer-aided skin lesion classification devices designed to help doctors assess suspected melanoma have received FDA approval. Only one, SciBase AB’s Nevisense, is still supported. No smartphone application, however, has been approved by the FDA or another recognized health authority to classify skin lesions. 

In May 2020, Swedish company SciBase AB announced FDA approval of AI-based Nevisense 3.0 to assess whether irregular/high-risk moles were cancerous without biopsy. The other device, MelaFind, was acquired by Stara Skin Sciences, which said it stopped research/development and sales/support activities for it in 2017.

The Nevisense device is shown below:

The Growing Number of AI Applications

Recent years have seen more AI and machine-learning publications that could lead to tools helping diagnose skin diseases , including cancer. Google is among the companies involved. The FDA therefore asked an expert panel to discuss questions about assessing the risks of companies’ computer-aided melanoma diagnostic devices, and requested feedback on how those risks could be managed.

A July 29, 2022 panel focused on reclassifying current Class III devices. A key question was whether the FDA would impose safety and effectiveness requirements and restrictions on computer-aided devices helping dermatologists assess suspicious melanoma lesions. Potential special controls could include clinical performance tests of sensitivity (true-positive rate) and specificity (true-negative rate), as well as software validation and cybersecurity testing to reduce risk.

Experts also highlighted the need for studies answering how accurately new tools identify relevant lesions.

Binita Ashar of the FDA’s Center for Devices and Radiological Health said about future products: “Innovation continues. Medical device developers are concerned about how to achieve the level of evidence you expect.”

Medical Device Classification

Simon Grant, CEO of SciBase, maker of Nevisense, the only skin cancer diagnostic device currently on the US market, said Class III regulations should remain and that he would pursue legal action. Reclassification of Nevisense from Class III to the less restrictive Class II had been proposed.

Grant said moving to Class II would weaken the standards used to clear products analyzing skin lesions, placing patients at risk.

1. Under FDA rules, devices presenting potential disease or injury risk are designated Class III. Approximately 10% of medical devices are in this category, including implantable pacemakers, breast implants and Nevisense.
2. Approximately 43% of medical devices are Class II, such as powered wheelchairs and some pregnancy test kits.
3. Class I devices have minimal potential for harm and simple designs. They include enema kits and elastic bandages.

Grant said such a change would have unintended consequences. “Small changes in AI systems can have large, nonlinear and even random effects. If a product is already on the market and harm has occurred, it is too late.”

The American Society for Dermatologic Surgery (ADCD) also protested reclassifying approved Class III computer-aided melanoma diagnostic devices. It raised concerns about expanded conditions of use, including products marketed directly to consumers.

“Algorithms and devices for patients to diagnose themselves by examining skin lesions are already widely available, but they are not diagnostic devices and are not endorsed by dermatologists or recognized medical regulators. They should carry detailed disclaimers saying they are primarily for entertainment and education. Devices and algorithms in screening tools are not highly regulated or proven. They may cause incorrect, missed or incomplete diagnoses. Both low- and high-risk patients are better served by an in-person examination with a board-certified dermatologist who can help determine their most appropriate skin screening program.”

Consumers Want Diagnostic Applications as an Intermediate Step

Many patients also face barriers to seeing a dermatologist. Google health AI team member Lily Peng said, “Google research shows consumers are looking for an intermediate step between information readily available online and what they can obtain from a medical professional” .

“Consumers often want more information about common conditions such as acne and sometimes struggle to access doctors. Many needs remain unmet for people with skin problems, most of which are low-acuity conditions. Consumers have told us they want a tool to help with nonserious conditions and decide when to seek medical attention.”

Google researchers have published several studies in recent years on using AI to improve dermatological diagnosis.

A paper published in 2020 reported testing a form of AI called a deep-learning system to provide differential diagnoses of skin conditions. According to its abstract, in 963 validation cases assessed by a panel of three board-certified dermatologists, the system outperformed 6 dermatologists, six primary care physicians and 6 nurses.

Image quality is also crucial: professional photographs should not be compared with variable-quality phone images. Yet diagnostic applications may be tested on nonstandardized images in everyday life.

We Are Not Ready Yet

Panelist Maral Skelsey, a skin cancer specialist in Chevy Chase, Maryland, said she had used both Nevisense and MelaFind for years but that they were not ready for initial use. She particularly cited MelaFind’s high misdiagnosis rate and restrictions on the body sites where it can be used.

She nevertheless looked forward to better devices for diagnosing skin lesions. “This is an area where we are on the threshold, and we really need these devices. We need patients to examine themselves and nondermatologists to assess lesions.”

Skelsey warned that rapidly relaxing approval standards could cause harm. Chicago dermatologist and panelist Renata Block agreed, insisting on retaining Class III status. Weill Cornell Medicine panelist Karla V. Ballman said a clear clinical performance standard was needed before considering device classification. “At this point, I do not think we are ready for initial use. These should remain Class III.”

Others disagreed. Panelist Katalin Roth believed these risks could be managed and future tools could help more patients move toward critical diagnoses. “As a palliative care doctor, I care for many cancer patients. I think what we are overlooking is time. Melanoma is terrible, and late diagnosis is terrible. But I believe special controls would adequately address my colleagues’ concerns.”

Finally, panelist Veronica Rotember of Memorial Sloan Kettering Cancer Center in New York spoke. Rotemberg has long researched AI and other computer-based tools for diagnosing melanoma, the most aggressive skin cancer. She questioned the performance of commercial applications claiming to scan moles and other lesions and identify cancer. British Journal of Dermatology published a review coauthored by Rotemberg reporting that these applications were only 59% accurate. She also emphasized periodic monitoring of algorithms in marketed skin cancer detection tools. “We need a sampling mechanism. Most data are already electronic, so comparing an algorithm and its performance with biopsy results should not be very difficult.”