Next Article in Journal
Risk Assessment for Heroin Use and Craving Score Using Polygenic Risk Score
Previous Article in Journal
Diagnosis and Management of Glioblastoma: A Comprehensive Perspective
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JPM
Volume 11
Issue 4
10.3390/jpm11040257
jpm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

The BRAF P.V600E Mutation Status of Melanoma Lung Metastases Cannot Be Discriminated on Computed Tomography by LIDC Criteria nor Radiomics Using Machine Learning

by
Lindsay Angus
1,2,*,†,
Martijn P. A. Starmans
2,3,†,
Ana Rajicic
1,
Arlette E. Odink
2,
Mathilde Jalving
4,
Wiro J. Niessen
2,3,5,
Jacob J. Visser
2,
Stefan Sleijfer
1,
Stefan Klein
2,3,‡ and
Astrid A. M. van der Veldt
1,2,‡
1
Department of Medical Oncology, Erasmus MC Cancer Institute, 3015 GD Rotterdam, The Netherlands
2
Department of Radiology and Nuclear Medicine, Erasmus MC, 3015 GD Rotterdam, The Netherlands
3
Department of Medical Informatics, Erasmus MC, 3015 GD Rotterdam, The Netherlands
4
Department of Medical Oncology, University Medical Center Groningen, University of Groningen, 9713 GZ Groningen, The Netherlands
5
Faculty of Applied Sciences, Delft University of Technology, 2628 CJ Delft, The Netherlands
*
Author to whom correspondence should be addressed.
†
These authors contributed equally.
‡
These authors contributed equally.
J. Pers. Med. 2021, 11(4), 257; https://doi.org/10.3390/jpm11040257
Submission received: 27 January 2021 / Revised: 18 March 2021 / Accepted: 24 March 2021 / Published: 1 April 2021
(This article belongs to the Section Omics/Informatics)
Browse Figures
Review Reports Versions Notes

Abstract

:
Patients with BRAF mutated (BRAF-mt) metastatic melanoma benefit significantly from treatment with BRAF inhibitors. Currently, the BRAF status is determined on archival tumor tissue or on fresh tumor tissue from an invasive biopsy. The aim of this study was to evaluate whether radiomics can predict the BRAF status in a non-invasive manner. Patients with melanoma lung metastases, known BRAF status, and a pretreatment computed tomography scan were included. After semi-automatic annotation of the lung lesions (maximum two per patient), 540 radiomics features were extracted. A chest radiologist scored all segmented lung lesions according to the Lung Image Database Consortium (LIDC) criteria. Univariate analysis was performed to assess the predictive value of each feature for BRAF mutation status. A combination of various machine learning methods was used to develop BRAF decision models based on the radiomics features and LIDC criteria. A total of 169 lung lesions from 103 patients (51 BRAF-mt; 52 BRAF wild type) were included. There were no features with a significant discriminative value in the univariate analysis. Models based on radiomics features and LIDC criteria both performed as poorly as guessing. Hence, the BRAF mutation status in melanoma lung metastases cannot be predicted using radiomics features or visually scored LIDC criteria.

Graphical Abstract

1. Introduction

Cutaneous melanoma is an aggressive skin cancer most commonly occurring on the ultra-violet light exposed skin of Caucasians [1,2]. In Europe, it is the 8th most common malignancy in men and the 5th most common in women, with an annual incidence of 144,200 new cases and 27,100 deaths [3]. In the coming years, the incidence of melanoma is expected to increase rapidly, resulting in an increased melanoma-associated mortality [4].
The introduction of new systemic treatment modalities, including immunotherapy and BRAF inhibitors, has significantly improved the prognosis of patients with metastatic melanoma [5]. Approximately 50% of melanomas harbor a mutation in the BRAF gene, with p.V600E being the most common variant [6,7,8]. Patients with BRAF-mutant (BRAF-mt) melanoma benefit significantly from treatment with BRAF inhibitors and onset of response is often rapid [9]. To enhance response rates and duration of response, patients are usually treated with a combination of a BRAF and a MEK inhibitor [10,11,12,13]. Due to the therapeutic consequences, determination of the BRAF mutation status in patients with metastatic melanoma is mandatory according to the European Society of Medical Oncology guidelines [14].
Currently, the BRAF mutation status is usually determined by molecular analysis of a metastatic lesion [15]. However, tissue biopsies are invasive, thereby exposing patients to potential risks including bleeding, infection and in case a lung biopsy is taken the risk of pneumothorax. In addition, molecular analyses can be time-consuming, especially when the tumor specimen has been archived at another hospital. Since patients with metastatic melanoma can experience rapidly progressive disease with life-threatening symptoms and an urgent medical need for systemic therapy, faster and less invasive diagnostics to determine the BRAF mutation status may significantly improve patient management.
Recently, various tumor characteristics have been predicted non-invasively using quantitative imaging features, also referred to as “radiomics”. In non-small cell lung cancer, radiomics on computed tomography (CT) can predict tumor stage and epidermal growth factor receptor (EGFR) mutation status [16,17,18,19,20,21,22,23,24]. In patients with primary colorectal cancer, a CT radiomics signature that was associated with BRAF mutation status [25]. CT-based radiomics has been applied to predict response to immunotherapy in melanoma lymph node metastases [26], but with little success (area under the curve (AUC) of 0.64). The value of radiomics for predicting BRAF mutation status has not been investigated. If CT-based radiomics could predict BRAF mutation status with a high positive predictive value, this may provide a faster and more patient-friendly alternative to determine the BRAF mutation status in metastatic melanoma.
The aim of this study was to evaluate the utility of CT-based radiomics to predict BRAF mutation status (mutant versus wild type) in metastatic melanoma. In metastatic melanoma, lung metastases are relatively easy to annotate on CT as compared to other metastases since they can be clearly distinguished from healthy lung tissue. Therefore, the aim of this study was to evaluate the utility of CT-based radiomics to predict BRAF mutation status (mutant versus wild type) in melanoma lung metastases.

2. Materials and Methods

2.1. Data Collection

This study was approved by the Erasmus MC institutional research board (MEC-2019-0693). Anonymized patient data was used and therefore need for written informed consent was waived by the Institutional Review Board. All patients diagnosed with metastatic melanoma at the Erasmus MC between January 2012 and February 2018 were included retrospectively if they met the following pre-specified criteria: known tumor BRAF mutation, diagnostic contrast-enhanced thoracic CT scan prior to commencement of any systemic therapy, and at least one lung metastasis of ≥10 mm evaluable according to Response Evaluation Criteria In Solid Tumors (RECIST) v1.1 [27]. Patients with BRAF mutations other than p.V600E were excluded from the analysis, since BRAF inhibitors may be less effective in patients with other BRAF mutations [28]. Formalin-fixed paraffin-embedded material of the primary tumor and/ or metastasis is tested for BRAF (exon 15) using a polymerase chain reaction based assay or next generation sequencing as part of standard care.

2.2. Radiomics

Lung metastases were measured according to RECIST v1.1 [27]. For 3D segmentation, up to two lung lesions ≥ 10 mm were selected by a clinician supervised by an experienced chest radiologist. In patients with >2 lung metastases of ≥10 mm, either the two largest or the two most easily distinguishable lesions were segmented (i.e., two separate lesions were preferred over two adjacent lesions). Using in-house developed software [29], selected lung metastases were segmented semi-automatically using a lung window for visualization. The result was visually inspected and manually corrected when necessary by an experienced chest radiologist to ensure that the semi-automatic segmentation resembled the manual segmentation. The clinician and chest radiologist were both blinded for BRAF mutation status. From each segmented lesion, 540 radiomics features were extracted to quantify intensity, shape and texture. Details are described in Supplementary Materials 1. To create a decision model using these features, the Workflow for Optimal Radiomics Classification (WORC) toolbox was used (Figure 1) [30,31,32]. Details are described in Supplementary Materials 2. In brief, the creation of a decision model in WORC consists of several steps, including selection of relevant features, resampling and machine learning techniques to identify patterns to distinguish BRAF-mt from BRAF wild type (BRAF-wt) lesions. WORC performs an automated search including a variety of algorithms for each step and determines which combination of algorithms maximizes the predictive performance on the training set. The open-source code for the feature extraction and model optimization has been published [33].

2.3. Scoring by Radiologist

An experienced chest radiologist (certified for 8 years) scored the segmented lung lesions. There are no guidelines to differentiate histologic subtypes in lung metastases; therefore, the Lung Image Database Consortium (LIDC) criteria were used. These criteria were developed to standardize the description of radiological features of lung abnormalities in clinical practice [34]. The following LIDC features were rated: subtlety, calcification, internal structure, lobulation, likelihood of malignancy, margin, sphericity, spiculation and texture (see Table S1 for the rating system). The radiologist was blinded for the BRAF status, but not to the diagnosis of metastatic melanoma and had access to the CT scan, age and sex of the patient.

2.4. Experimental Setup

To assess the predictive value of quantitative imaging features (i.e., radiomics features) and LIDC features, five models were trained and tested using WORC based on: (1) automatically extracted radiomics features only (2) similar to model 1, but only including the largest lesion per patient; (3) similar to model 1, but only including patients with NRAS and BRAF wild type melanoma for the comparison with BRAF-mt; (4) manually scored LIDC features only; and (5) a simple benchmark model. Model 2 was applied to assess a potential bias for patients with multiple lesions. Model 3 was included because activating NRAS mutations could potentially result in a similar phenotype as BRAF-mt, since mutations in both genes lead to activation of the mitogen-activated protein kinase (MAPK) pathway. The simple benchmark model was evaluated in a similar way as model 1, i.e., using all lesions and automatically extracted radiomics features. Model 5 was applied to compare the performance of WORC to a simple benchmark machine learning model, which uses binary logistic regression with LASSO (least absolute shrinkage and selection operator) feature selection (i.e., ElasticNet).

2.5. Statistics

To assess the predictive value of the individual features, the Mann–Whitney U test was performed for univariate analyses of continuous variables and Pearson’s chi-squared test was used for categorical variables. For radiomics, p-values were corrected for multiple testing using the Bonferroni correction according to the default in WORC. A p-value of <0.05 was considered to be statistically significant.
Evaluation of the radiomics models was performed using a 100× random-split cross-validation. In each iteration, the data was randomly split into 80% for training and 20% for testing in a stratified manner to guarantee a similar distribution of the classes in the training and test set as compared to the original set. Metastases from the same patients were always grouped together in either the training or test set. To eliminate the risk of overfitting, in each iteration, all model optimization was performed strictly within the training set by using a second internal 5× random-split cross-validation (see Figure S1). The final model consists of an ensemble of the 50 best workflows, i.e., combination of methods and parameters, each defined by a specific set of hyperparameters. This final model may be different in each of the 100× random-split cross-validation iterations. For each of the five models described in the experimental setup, these sets hyperparameters are included with the code [33]. Details are described in Supplementary Materials 2.
The performance of all four models was described by the AUC of the receiver operating characteristic (ROC) curve, accuracy, sensitivity, specificity, negative predictive value (NPV) and positive predictive value (PPV). The positive class was defined as BRAF-mt. For each metric, the average over the 100 cross-validation iterations and a 95% confidence interval (CI) were reported. The 95% CIs were constructed using the corrected resampled t-test based on the results from all 100 cross-validation iterations, thereby taking into account that the samples in the cross-validation splits are not statistically independent [35]. ROC confidence bands were constructed using fixed-width bands [36].

3. Results

3.1. Study Population

In total, 103 patients were included, see Figure S2 for a flowchart of patient inclusion. Characteristics of these patients and their CT scans are summarized in Table 1. The median age was 65 years (interquartile range (IQR) 52–72) and 50.5% of the patients were men. BRAF mutation status was either determined on the primary tumor (N = 20), local recurrence (N = 3), or metastasis (N = 79). In these lesions, BRAF p.V600E was detected in 51 patients, whereas 52 patients had BRAF-wt melanomas. In total, 103 CT scans were acquired from 10 different CT scanners, resulting in in the inclusion of data acquired with different acquisition protocols (Table 1). Although for all acquisition parameters the difference between BRAF-mt and BRAF-wt was not statistically significant, the difference in tube current reached almost statistical significance (p = 0.05).

3.2. Radiomics and LIDC Features and Models

In total, 169 lung metastases in 103 patients were segmented. Figure 2 illustrates randomly selected segmentations of lung metastases from patients with BRAF-mt and BRAF-wt metastatic melanoma. Median volume of segmented lung lesions was 18.3 mL (IQR: 7.3–48.6 mL). None of the radiomics or LIDC features were significantly different between BRAF-mt and BRAF-wt lung metastases, as none of the features had a p-value < 0.05 after Bonferroni correction. LIDC criteria scores are shown in Table S2. Using all 169 lung metastases, the radiomics model (model 1) resulted in a mean AUC of 0.49, sensitivity of 0.61 and specificity of 0.37 (Figure 3A, Table 2). Model 2, i.e., only inclusion of the largest lesion per patient, slightly improved the performance (AUC of 0.65), whereas model 3, i.e., only inclusion of BRAF-wt melanoma who were also NRAS wild type, still had a poor performance (AUC of 0.49) (Figure 3B,C, Table 2). In addition, model 4, i.e., based on the LIDC features scored by a radiologist, resulted in an AUC of 0.46 (Figure 3D). The simple benchmark (model 5) resulted in a similar performance (AUC of 0.50).

4. Discussion

The results of this study show that there is no association between radiomics features of lung metastases and the BRAF mutation status in patients with metastatic melanoma. Our model using only the largest lesion per patient performed best with a moderate mean AUC, but still none of the features had any individual discriminative value. In addition, the performance confidence intervals (e.g., the sensitivity and specificity) still included many values below the performance of guessing. The LIDC criteria as scored by a thorax radiologist also failed to discriminate the BRAF mutation status in melanoma lung metastases.
Despite the remarkable success of BRAF inhibitors and immunotherapy in patients with metastatic melanoma, only a subset of patients benefits from these therapies [11,37]. Tools to select the patients most likely to benefit are of great interest and this has resulted in several radiomics studies aiming to predict tumor response. Similar to our study, previous radiomics models, either to predict therapy response or survival, had a low to moderate performance in metastatic melanoma [26,38,39]. In the largest radiomics study in melanoma thus far, 483 lesions from 80 melanoma patients were included and a greater morphological heterogeneity of lymph nodes determined by CT was associated with immunotherapy response, resulting in a moderate AUC of 0.64 [26]. However, the model performed poorly in lung and liver lesions (AUC of 0.55). Comparable to our CT-based findings, a recent study showed that radiomics features derived from 18F-FDG positron emission tomography (PET) to determine the BRAF p.V600E mutation status also had a moderate performance (AUC of 0.62). They studied 176 lesions, including 18 lung lesions from 70 patients with melanoma (35 BRAF-mt and 35 BRAF-wt) [40]. To the best of our knowledge, this PET study [40] and our CT study are the first melanoma studies aiming to predict BRAF p.V600E mutation status, showing that neither PET nor CT radiomics features can discriminate between patients with BRAF-mt and BRAF-wt melanomas. We therefore believe that our comprehensive study provides insight into the potential of radiomics in this area, which can guide future research [41].
The lack of discrimination between BRAF-mt and BRAF-wt melanoma could potentially be explained by activating mutations in the NRAS gene in BRAF-wt melanoma. Since NRAS and BRAF are involved in the same pathway, i.e., the MAPK pathway, activating NRAS and BRAF mutations could result in a similar phenotype. Therefore, we evaluated an additional model which only included NRAS wild type lesions in patients with BRAF-wt melanoma (model 3). In our cohort of patients with BRAF-wt melanoma, 22 out of 45 (49%) patients—with known NRAS mutation status—had a NRAS mutation. Exclusion of all patients with NRAS mutation or unknown NRAS mutation status resulted in an AUC of 0.54 (95% CI 0.44–0.64). Based on these findings, it is very unlikely that inclusion of NRAS mutant melanomas negatively impacted our results. In addition, our findings are supported by the low predictive value of PET radiomics in the same setting in which patients with NRAS mutations were also excluded [40].
Our study was designed for a comprehensive evaluation of the relationship between CT imaging features and the BRAF mutation status in melanoma lung metastases. To our knowledge this is currently the largest CT-based radiomics study on the BRAF mutation status in patients with metastatic melanoma and with 103 subjects even large for a radiomics study [42]. It is unlikely that treatment-related resistance mechanisms influenced the outcome, since the study population was treatment-naïve, thereby reflecting the appearance of untreated melanoma lung metastases. The investigated patient population only included melanoma patients for whom correct determination of the BRAF status is of utmost importance for rapid treatment stratification. The WORC radiomics method applied has been previously validated to predict mutation status of several genes in other tumor types, such as lipoma and liposarcoma [32], desmoids [43], gastrointestinal stromal tumors [44], liver cancer [29,45], prostate cancer [46] and mesenteric fibrosis [47]. In these previous studies, the radiomics models had a much better performance (mean AUCs between 0.71–0.89) and multiple features were statistically significant in univariate statistical testing. In the current study, none of the radiomics features had any discriminative value; therefore, it can be concluded that radiomics features of melanoma lung metastases are not related to the BRAF mutation status. WORC includes a wide variety of radiomics approaches and automatically optimizes the combination, thereby evaluating many different approaches. Moreover, a different normalization method, combining z-scoring with a logarithmic transform and a correction term to better cope with outliers and non-normally distributed features [48], yielded similar negative results (model 1: AUC of 0.49). Hence, it is unlikely that a different radiomics approach will lead to a positive result.
In addition to the radiomics analysis, a radiologist visually evaluated the lesions. Similar to radiomics results, the radiologist could not discriminate between BRAF-wt and BRAF-mt lesions by applying the LIDC criteria. Although radiomics can potentially correlate imaging features with clinical outcome even in cases where a radiologist cannot, the relation between quantitative imaging features and clinical outcome is considered stronger when clinical outcomes can be discriminated visually by a radiologist. This was not evident in the current study and it can be considered additional evidence that a CT-based radiomics signature probably does not exist for the BRAF mutation status in melanoma lung metastases. Although radiomics is promising in other fields of research, it is not expected that all cytogenetic changes are associated with morphological changes. Consequently, it is unlikely that every DNA alteration can be detected by radiomics.
Our study has several limitations. Firstly, the BRAF mutation status was often determined on other tumor tissue than the segmented lung metastases. The BRAF status was determined on biopsy material from a lung metastasis, which did not necessarily match the segmented lung lesion, in only 12 patients. Although the concordance rate of the BRAF mutation status between primary melanoma and metastases is quite high [8,49,50], a recent meta-analysis showed a pooled discrepancy rate of 13.4% between primary melanomas and metastases and a 7.3% discrepancy rate between metastatic sites [51]. Hence, tumor heterogeneity might have caused misclassification of BRAF mutation status, thereby negatively affecting the results. Ideally, in prospective radiomics studies, genomic and radiomics analyses are performed on the same tumor site. Secondly, the segmentation of regions of interest (ROI) was performed semi-automatically. Automatic segmentation methods may improve the consistency of the segmentations and thus affect the radiomics model. However, due to the clear distinction of lung lesions and their surroundings, it is not expected that automatic segmentation will substantially alter the results. Thirdly, the heterogeneity in the acquisition protocols may have negatively affected the performance or our radiomics model. These variations may have led to variations in the imaging features, which complicate the recognition of patterns. Using a single acquisition protocol would give an estimate of the performance unaffected by such variations. However, the variations in the acquisition protocols were small, making it unlikely this significantly affected the results of the current study. Feature selection methods based on feature test-retest reproducibility could be investigated in future work [52,53]. The difference in tube current between BRAF-mt and BRAF-wt almost reached statistical significance and could have been implicitly used by the model to distinguish these lesions. However, our results show that, despite this difference, the performance of the model was similar to guessing. Lastly, although training data were strictly separated from test data in cross-validation, we did not validate our findings on an independent, external dataset.

5. Conclusions

In summary, our study demonstrates that neither CT-based radiomics features, nor CT-derived LIDC features scored by a radiologist can discriminate between BRAF mutant and BRAF wild type lung metastases in patients with metastatic melanoma. Therefore, CT-based parameters cannot replace determination of BRAF mutation status on tumor tissue.

Supplementary Materials

The following are available online at https://www.mdpi.com/article/10.3390/jpm11040257/s1, Figure S1: Visualization of the 100× random split-cross validation, including a second cross validation within the training set for the model optimization., Figure S2: Flowchart of patient inclusion., Table S1: LIDC Nodule Characteristics, Definitions and Ratings., Table S2: LIDC criteria scored by a chest radiologist., Table S3: Overview of the 540 features used in this study. Supplementary Materials 1., Supplementary Materials 2.

Author Contributions

Conceptualization, L.A., M.P.A.S., J.J.V., S.S., S.K. and A.A.M.v.d.V.; methodology, L.A., M.P.A.S., J.J.V., S.S., S.K. and A.A.M.v.d.V.; software, M.P.A.S., S.K. and W.J.N.; formal analysis, L.A. and M.P.A.S.; investigation, L.A., A.R., M.P.A.S. and A.E.O.; resources, A.A.M.v.d.V. and S.K.; data curation, L.A. and M.P.A.S.; writing—original draft preparation, L.A., A.R. and M.P.A.S.; writing—review and editing, L.A., M.P.A.S., A.R., A.E.O., M.J., W.J.N., J.J.V., S.S., S.K., A.A.M.v.d.V.; visualization, M.P.A.S.; supervision, A.A.M.v.d.V. and S.K.; project administration, L.A.; funding acquisition, not applicable. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding. M.P.A.S. acknowledges funding from the research program STRaTeGy (project number 14929-14930), which is (partly) financed by the Netherlands Organization for Scientific Research (NWO). This work was partially carried out on the Dutch national e-infrastructure with the support of SURF Cooperative.

Institutional Review Board Statement

The study was conducted according to the guidelines of the Declaration of Helsinki and approved by the Institutional Review Board (or Ethics Committee) of the Erasmus MC (MEC-2019-0693, date of approval 29 November 2019).

Informed Consent Statement

Since we used anonymized patient data, written informed consent was waived by the Institutional Review Board.

Data Availability Statement

Imaging and clinical research data are not available at this time. Programming code is openly available on Zenodo at https://doi.org/10.5281/zenodo.4644067 (accessed on 27 January 2021).

Conflicts of Interest

L.A. reports receiving a consulting honorarium from Merck and a speaking honorarium from Pfizer; and A.A.M.v.d.V., receiving consulting honoraria from Sanofi, Roche, Merck Sharp & Dohme, Pfizer, Eisai, Ipsen, Novartis, Pierre Fabre and Bristol-Myers Squibb. W.J.N. is founder, scientific lead and stock holder of Quantib BV. M.J. reports consulting honoraria to institution from Merck, BMS, Novartis, Pierre Fabre, Tesaro, AstraZeneca and performs clinical studies with BMS, AbbVie, Merck, Cristal Therapeutics. No other potential conflict of interest relevant to this publication was reported.

References

  1. MacKie, R.M.; Hauschild, A.; Eggermont, A.M.M. Epidemiology of invasive cutaneous melanoma. Ann. Oncol. 2009, 20, vi1–vi7. [Google Scholar] [CrossRef] [PubMed]
  2. Shain, A.H.; Bastian, B.C. From melanocytes to melanomas. Nat. Rev. Cancer 2016, 16, 345–358. [Google Scholar] [CrossRef] [PubMed]
  3. Ferlay, J.; Colombet, M.; Soerjomataram, I.; Dyba, T.; Randi, G.; Bettio, M.; Gavin, A.; Visser, O.; Bray, F. Cancer incidence and mortality patterns in Europe: Estimates for 40 countries and 25 major cancers in 2018. Eur. J. Cancer 2018, 103, 356–387. [Google Scholar] [CrossRef] [PubMed]
  4. Whiteman, D.C.; Green, A.C.; Olsen, C.M. The Growing Burden of Invasive Melanoma: Projections of Incidence Rates and Numbers of New Cases in Six Susceptible Populations through 2031. J. Investig. Dermatol. 2016, 136, 1161–1171. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  5. Luke, J.J.; Flaherty, K.T.; Ribas, A.; Long, G.V. Targeted agents and immunotherapies: Optimizing outcomes in melanoma. Nat. Rev. Clin. Oncol. 2017, 14, 463–482. [Google Scholar] [CrossRef] [Green Version]
  6. Davies, H.; Bignell, G.R.; Cox, C.; Stephens, P.; Edkins, S.; Clegg, S.; Teague, J.; Woffendin, H.; Garnett, M.J.; Bottomley, W.; et al. Mutations of the BRAF gene in human cancer. Nature 2002, 417, 949–954. [Google Scholar] [CrossRef]
  7. Curtin, J.A.; Fridlyand, J.; Kageshita, T.; Patel, H.N.; Busam, K.J.; Kutzner, H.; Cho, K.H.; Aiba, S.; Brocker, E.B.; LeBoit, P.E.; et al. Distinct sets of genetic alterations in melanoma. N. Engl. J. Med. 2005, 353, 2135–2147. [Google Scholar] [CrossRef]
  8. Colombino, M.; Capone, M.; Lissia, A.; Cossu, A.; Rubino, C.; De Giorgi, V.; Massi, D.; Fonsatti, E.; Staibano, S.; Nappi, O.; et al. BRAF/NRAS mutation frequencies among primary tumors and metastases in patients with melanoma. J. Clin. Oncol. 2012, 30, 2522–2529. [Google Scholar] [CrossRef]
  9. Flaherty, K.T.; Puzanov, I.; Kim, K.B.; Ribas, A.; McArthur, G.A.; Sosman, J.A.; O’Dwyer, P.J.; Lee, R.J.; Grippo, J.F.; Nolop, K.; et al. Inhibition of mutated, activated BRAF in metastatic melanoma. N. Engl. J. Med. 2010, 363, 809–819. [Google Scholar] [CrossRef] [Green Version]
  10. Long, G.V.; Stroyakovskiy, D.; Gogas, H.; Levchenko, E.; de Braud, F.; Larkin, J.; Garbe, C.; Jouary, T.; Hauschild, A.; Grob, J.J.; et al. Combined BRAF and MEK inhibition versus BRAF inhibition alone in melanoma. N. Engl. J. Med. 2014, 371, 1877–1888. [Google Scholar] [CrossRef] [Green Version]
  11. Long, G.V.; Flaherty, K.T.; Stroyakovskiy, D.; Gogas, H.; Levchenko, E.; de Braud, F.; Larkin, J.; Garbe, C.; Jouary, T.; Hauschild, A.; et al. Dabrafenib plus trametinib versus dabrafenib monotherapy in patients with metastatic BRAF V600E/K-mutant melanoma: Long-term survival and safety analysis of a phase 3 study. Ann. Oncol. 2017, 28, 1631–1639. [Google Scholar] [CrossRef]
  12. Larkin, J.; Ascierto, P.A.; Dreno, B.; Atkinson, V.; Liszkay, G.; Maio, M.; Mandala, M.; Demidov, L.; Stroyakovskiy, D.; Thomas, L.; et al. Combined vemurafenib and cobimetinib in BRAF-mutated melanoma. N. Engl. J. Med. 2014, 371, 1867–1876. [Google Scholar] [CrossRef] [Green Version]
  13. Ascierto, P.A.; McArthur, G.A.; Dreno, B.; Atkinson, V.; Liszkay, G.; Di Giacomo, A.M.; Mandala, M.; Demidov, L.; Stroyakovskiy, D.; Thomas, L.; et al. Cobimetinib combined with vemurafenib in advanced BRAF(V600)-mutant melanoma (coBRIM): Updated efficacy results from a randomised, double-blind, phase 3 trial. Lancet Oncol. 2016, 17, 1248–1260. [Google Scholar] [CrossRef]
  14. Michielin, O.; van Akkooi, A.C.J.; Ascierto, P.A.; Dummer, R.; Keilholz, U.; Committee, E.G. Cutaneous melanoma: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann. Oncol. 2019, 30, 1884–1901. [Google Scholar] [CrossRef] [Green Version]
  15. Cheng, L.; Lopez-Beltran, A.; Massari, F.; MacLennan, G.T.; Montironi, R. Molecular testing for BRAF mutations to inform melanoma treatment decisions: A move toward precision medicine. Mod. Pathol. 2018, 31, 24–38. [Google Scholar] [CrossRef]
  16. Rios Velazquez, E.; Parmar, C.; Liu, Y.; Coroller, T.P.; Cruz, G.; Stringfield, O.; Ye, Z.; Makrigiorgos, M.; Fennessy, F.; Mak, R.H.; et al. Somatic Mutations Drive Distinct Imaging Phenotypes in Lung Cancer. Cancer Res. 2017, 77, 3922–3930. [Google Scholar] [CrossRef] [Green Version]
  17. Aerts, H.J.; Velazquez, E.R.; Leijenaar, R.T.; Parmar, C.; Grossmann, P.; Carvalho, S.; Bussink, J.; Monshouwer, R.; Haibe-Kains, B.; Rietveld, D.; et al. Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach. Nat. Commun. 2014, 5, 4006. [Google Scholar] [CrossRef]
  18. Gevaert, O.; Xu, J.; Hoang, C.D.; Leung, A.N.; Xu, Y.; Quon, A.; Rubin, D.L.; Napel, S.; Plevritis, S.K. Non-small cell lung cancer: Identifying prognostic imaging biomarkers by leveraging public gene expression microarray data--methods and preliminary results. Radiology 2012, 264, 387–396. [Google Scholar] [CrossRef]
  19. Huang, Y.; Liu, Z.; He, L.; Chen, X.; Pan, D.; Ma, Z.; Liang, C.; Tian, J.; Liang, C. Radiomics Signature: A Potential Biomarker for the Prediction of Disease-Free Survival in Early-Stage (I or II) Non-Small Cell Lung Cancer. Radiology 2016, 281, 947–957. [Google Scholar] [CrossRef]
  20. Nair, V.S.; Gevaert, O.; Davidzon, G.; Napel, S.; Graves, E.E.; Hoang, C.D.; Shrager, J.B.; Quon, A.; Rubin, D.L.; Plevritis, S.K. Prognostic PET 18F-FDG uptake imaging features are associated with major oncogenomic alterations in patients with resected non-small cell lung cancer. Cancer Res. 2012, 72, 3725–3734. [Google Scholar] [CrossRef] [Green Version]
  21. Parmar, C.; Leijenaar, R.T.; Grossmann, P.; Rios Velazquez, E.; Bussink, J.; Rietveld, D.; Rietbergen, M.M.; Haibe-Kains, B.; Lambin, P.; Aerts, H.J. Radiomic feature clusters and prognostic signatures specific for Lung and Head & Neck cancer. Sci. Rep. 2015, 5, 11044. [Google Scholar]
  22. Scrivener, M.; de Jong, E.E.C.; van Timmeren, J.E.; Pieters, T.; Ghaye, B.; Geets, X. Radiomics applied to lung cancer: A review. Transl. Cancer Res. 2016, 5, 398–409. [Google Scholar] [CrossRef]
  23. van Timmeren, J.E.; van Elmpt, W.; Leijenaar, R.T.H.; Reymen, B.; Monshouwer, R.; Bussink, J.; Paelinck, L.; Bogaert, E.; De Wagter, C.; Elhaseen, E.; et al. Longitudinal radiomics of cone-beam CT images from non-small cell lung cancer patients: Evaluation of the added prognostic value for overall survival and locoregional recurrence. Radiother. Oncol. 2019, 136, 78–85. [Google Scholar] [CrossRef]
  24. Yamamoto, S.; Korn, R.L.; Oklu, R.; Migdal, C.; Gotway, M.B.; Weiss, G.J.; Iafrate, A.J.; Kim, D.W.; Kuo, M.D. ALK molecular phenotype in non-small cell lung cancer: CT radiogenomic characterization. Radiology 2014, 272, 568–576. [Google Scholar] [CrossRef]
  25. Yang, L.; Dong, D.; Fang, M.; Zhu, Y.; Zang, Y.; Liu, Z.; Zhang, H.; Ying, J.; Zhao, X.; Tian, J. Can CT-based radiomics signature predict KRAS/NRAS/BRAF mutations in colorectal cancer? Eur. Radiol. 2018, 28, 2058–2067. [Google Scholar] [CrossRef]
  26. Trebeschi, S.; Drago, S.G.; Birkbak, N.J.; Kurilova, I.; Calin, A.M.; Delli Pizzi, A.; Lalezari, F.; Lambregts, D.M.J.; Rohaan, M.W.; Parmar, C.; et al. Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers. Ann. Oncol. 2019, 30, 998–1004. [Google Scholar] [CrossRef] [Green Version]
  27. Eisenhauer, E.A.; Therasse, P.; Bogaerts, J.; Schwartz, L.H.; Sargent, D.; Ford, R.; Dancey, J.; Arbuck, S.; Gwyther, S.; Mooney, M.; et al. New response evaluation criteria in solid tumours: Revised RECIST guideline (version 1.1). Eur. J. Cancer 2009, 45, 228–247. [Google Scholar] [CrossRef]
  28. Menzer, C.; Menzies, A.M.; Carlino, M.S.; Reijers, I.; Groen, E.J.; Eigentler, T.; de Groot, J.W.B.; van der Veldt, A.A.M.; Johnson, D.B.; Meiss, F.; et al. Targeted Therapy in Advanced Melanoma With Rare BRAF Mutations. J. Clin. Oncol. 2019, 37, 3142–3151. [Google Scholar] [CrossRef] [Green Version]
  29. Starmans, M.P.; Miclea, R.; van der Voort, S.; Niessen, W.; Thomeer, M.; Klein, S. Classification of Malignant and Benign Liver Tumors Using a Radiomics Approach; SPIE: Bellingham, WA, USA, 2018; Volume 10574. [Google Scholar]
  30. Starmans, M.P.A.; van der Voort, S.R.; Phil, T.; Klein, S. Workflow for Optimal Radiomics Classification (WORC). Zenodo 2018. Available online: https://github.com/MStarmans91/WORC (accessed on 27 January 2021). [CrossRef]
  31. Starmans, M.; van der Voort, S.; Vos, M. Fully automatic construction of optimal radiomics workflows. In ECR 2019: Book of Abstracts (B-0908); Insights Imaging; Springer: Vienna, Austria, 2019; Volume 10, p. S379. [Google Scholar] [CrossRef]
  32. Vos, M.; Starmans, M.P.A.; Timbergen, M.J.M.; van der Voort, S.R.; Padmos, G.A.; Kessels, W.; Niessen, W.J.; van Leenders, G.; Grunhagen, D.J.; Sleijfer, S.; et al. Radiomics approach to distinguish between well differentiated liposarcomas and lipomas on MRI. Br. J. Surg 2019, 106, 1800–1809. [Google Scholar] [CrossRef] [Green Version]
  33. Martijn, P.A. Starmans. MelaRadiomics. Zenodo 2021. Available online: https://github.com/MStarmans91/MelaRadiomics (accessed on 27 January 2021). [CrossRef]
  34. Opulencia, P.; Channin, D.S.; Raicu, D.S.; Furst, J.D. Mapping LIDC, RadLex, and lung nodule image features. J. Digit. Imaging 2011, 24, 256–270. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  35. Nadeau, C.; Bengio, Y. Inference for the Generalization Error. Mach. Learn. 2003, 52, 239–281. [Google Scholar] [CrossRef] [Green Version]
  36. Macskassy, S.A.; Provost, F.; Rosset, S. ROC confidence bands: An empirical evaluation. In Proceedings of the 22nd International Conference on Machine Learning, Bonn, Germany, 7–11 August 2005. [Google Scholar]
  37. Robert, C.; Ribas, A.; Schachter, J.; Arance, A.; Grob, J.J.; Mortier, L.; Daud, A.; Carlino, M.S.; McNeil, C.M.; Lotem, M.; et al. Pembrolizumab versus ipilimumab in advanced melanoma (KEYNOTE-006): Post-hoc 5-year results from an open-label, multicentre, randomised, controlled, phase 3 study. Lancet Oncol. 2019, 20, 1239–1251. [Google Scholar] [CrossRef]
  38. Durot, C.; Mule, S.; Soyer, P.; Marchal, A.; Grange, F.; Hoeffel, C. Metastatic melanoma: Pretreatment contrast-enhanced CT texture parameters as predictive biomarkers of survival in patients treated with pembrolizumab. Eur. Radiol. 2019, 29, 3183–3191. [Google Scholar] [CrossRef]
  39. Sun, R.; Limkin, E.J.; Vakalopoulou, M.; Dercle, L.; Champiat, S.; Han, S.R.; Verlingue, L.; Brandao, D.; Lancia, A.; Ammari, S.; et al. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: An imaging biomarker, retrospective multicohort study. Lancet Oncol. 2018, 19, 1180–1191. [Google Scholar] [CrossRef]
  40. Saadani, H.; van der Hiel, B.; Aalbersberg, E.A.; Zavrakidis, I.; Haanen, J.; Hoekstra, O.S.; Boellaard, R.; Stokkel, M.P.M. Metabolic Biomarker-Based BRAFV600 Mutation Association and Prediction in Melanoma. J. Nucl. Med. 2019, 60, 1545–1552. [Google Scholar] [CrossRef]
  41. Buvat, I.; Orlhac, F. The Dark Side of Radiomics: On the Paramount Importance of Publishing Negative Results. J. Nucl. Med. 2019, 60, 1543–1544. [Google Scholar] [CrossRef]
  42. Jiangdian, S.; Yanjie, Y. A review of original articles published in the emerging field of radiomics. Eur. J. Radiol. 2020, 127, 108991. [Google Scholar] [CrossRef]
  43. Timbergen, M.J.M.; Starmans, M.P.A.; Padmos, G.A.; Grünhagen, D.J.; van Leenders, G.J.L.H.; Hanff, D.; Verhoef, C.; Niessen, W.J.; Sleijfer, S.; Klein, S.; et al. Differential diagnosis and mutation stratification of desmoid-type fibromatosis on MRI using radiomics. Eur. J. Radiol. 2020, 131, 109266. [Google Scholar] [CrossRef]
  44. Starmans, M.P.A.; Timbergen, M.J.; Vos, M.; Renckens, M.; Grünhagen, D.J.; van Leenders, G.J.L.H.; Dwarkasing, R.S.; Willemssen, F.E.J.A.; Niessen, W.J.; Verhoef, C.; et al. Differential diagnosis and molecular stratification of gastrointestinal stromal tumors on CT images using a radiomics approach. arXiv 2020, arXiv:2010.06824. [Google Scholar]
  45. Starmans, M.P.A.; Buisman, F.E.; Willemssen, F. Prediction of histopathological growth patterns by radiomics and CT-imaging in patients with operable colorectal liver metastases: A proof-of-concept study. In ECR 2020 Book of Abstracts (RPS 1001c-2); Insights Imaging; Springer: Vienna, Austria, 2020; Volume 11, p. 419. [Google Scholar] [CrossRef]
  46. Castillo, T.J.; Starmans, M.P.; Niessen, W.J.; Schoots, I.; Klein, S.; Veenland, J.F. Classification of Prostate cancer: High grade versus low grade using a radiomics approach. In Proceedings of the IEEE 16th International Symposium on Biomedical Imaging, Venice, Italy, 8–11 April 2019; pp. 1319–1322. [Google Scholar]
  47. Starmans, M.P.A.; Blazevic, A.; van der Voort, S.R. Prediction of surgery requirement in mesenteric fibrosis on CT using a radiomics approach. In ECR 2019: Book of Abstracts (B-1279); Insights Imaging; Springer: Vienna, Austria, 2019; Volume 10, p. S457. [Google Scholar] [CrossRef]
  48. Chen, J.; Cheung, H.; Milot, L.; Martel, A.L. AMINN: Autoencoder-based Multiple Instance Neural Network for Outcome Prediction of Multifocal Liver Metastases. arXiv 2020, arXiv:2012.06875. [Google Scholar]
  49. Manca, A.; Paliogiannis, P.; Colombino, M.; Casula, M.; Lissia, A.; Botti, G.; Caraco, C.; Ascierto, P.A.; Sini, M.C.; Palomba, G.; et al. Mutational concordance between primary and metastatic melanoma: A next-generation sequencing approach. J. Transl. Med. 2019, 17, 289. [Google Scholar] [CrossRef] [Green Version]
  50. Riveiro-Falkenbach, E.; Villanueva, C.A.; Garrido, M.C.; Ruano, Y.; Garcia-Martin, R.M.; Godoy, E.; Ortiz-Romero, P.L.; Rios-Martin, J.J.; Santos-Briz, A.; Rodriguez-Peralto, J.L. Intra- and Inter-Tumoral Homogeneity of BRAF(V600E) Mutations in Melanoma Tumors. J. Investig. Dermatol. 2015, 135, 3078–3085. [Google Scholar] [CrossRef] [Green Version]
  51. Valachis, A.; Ullenhag, G.J. Discrepancy in BRAF status among patients with metastatic malignant melanoma: A meta-analysis. Eur. J. Cancer 2017, 81, 106–115. [Google Scholar] [CrossRef]
  52. Traverso, A.; Wee, L.; Dekker, A.; Gillies, R. Repeatability and Reproducibility of Radiomic Features: A Systematic Review. Int. J. Radiat. Oncol. 2018, 102, 1143–1158. [Google Scholar] [CrossRef] [Green Version]
  53. Rossi, G.; Barabino, E.; Fedeli, A.; Ficarra, G.; Coco, S.; Russo, A.; Adamo, V.; Buemi, F.; Zullo, L.; Dono, M.; et al. Radiomic Detection of EGFR Mutations in NSCLC. Cancer Res. 2021, 81, 724. [Google Scholar] [CrossRef]
Jpm 11 00257 g001 550
Figure 1. Schematic overview of the radiomics approach: adapted from Vos and Starmans et al. [32]. Inputs to the algorithm are (1) contrast-enhanced thoracic CT images of patients with BRAF mutated or BRAF wild type metastatic melanoma and (2) a segmentation of the lung metastasis. Processing steps include (3) feature extraction and (5) the creation of a machine learning decision model, using (4) an ensemble of the best 50 workflows from 100,000 candidate workflows, which are different combinations of the different processing and analysis steps (e.g., the classifier used).
Figure 1. Schematic overview of the radiomics approach: adapted from Vos and Starmans et al. [32]. Inputs to the algorithm are (1) contrast-enhanced thoracic CT images of patients with BRAF mutated or BRAF wild type metastatic melanoma and (2) a segmentation of the lung metastasis. Processing steps include (3) feature extraction and (5) the creation of a machine learning decision model, using (4) an ensemble of the best 50 workflows from 100,000 candidate workflows, which are different combinations of the different processing and analysis steps (e.g., the classifier used).
Jpm 11 00257 g001
Jpm 11 00257 g002 550
Figure 2. Examples of BRAF wild type (A,B) and BRAF mutant (C,D) lung metastases of four patients with metastatic melanoma. Contours of the segmentations of the selected metastases are shown in red.
Figure 2. Examples of BRAF wild type (A,B) and BRAF mutant (C,D) lung metastases of four patients with metastatic melanoma. Contours of the segmentations of the selected metastases are shown in red.
Jpm 11 00257 g002
Jpm 11 00257 g003 550
Figure 3. Receiver operating characteristic (ROC) curve of the radiomics model of all lesions (A), only the largest lesion (B), only BRAF wild type lesions with NRAS wild type (C) and LIDC features (D). The crosses identify the 95% confidence intervals of the 100× random-split cross-validation; the blue curve is fit through their means.
Figure 3. Receiver operating characteristic (ROC) curve of the radiomics model of all lesions (A), only the largest lesion (B), only BRAF wild type lesions with NRAS wild type (C) and LIDC features (D). The crosses identify the 95% confidence intervals of the 100× random-split cross-validation; the blue curve is fit through their means.
Jpm 11 00257 g003
Table
Table 1. Patient and imaging characteristics. Values in parentheses are percentages unless indicated otherwise.
Table 1. Patient and imaging characteristics. Values in parentheses are percentages unless indicated otherwise.
PatientBRAF-mt (N = 51)BRAF-wt (N = 52)p-Value
Age (years)†59 (50–69)66 (57–74)0.048
Sex 0.768
 Male25 (49)27 (52)
 Female26 (51)25 (48)
Primary tumor localization 0.027
 Skin49 (96)42 (81)
 Mucosal0 (0)6 (11)
 Unknown2 (4)4 (8)
Determination of BRAF-mutation status 0.851
 Primary tumor9 (18)11 (21)
 Local recurrence1 (2)2 (4)
 Metastasis40 (78)39 (75)
 Unknown1 (2)0 (0)
NRAS mutation status $ Not applicable
 Mutant-22 (42)
 Wild type-23 (44)
 Unknown-7 (2)
Imaging
Acquisition protocol
 Slice thickness (mm) †,11.5 (1.5, 1.5)1.5 (1.5, 1.5)0.23
 Pixel spacing (mm) †0.68 (0.64, 0.74)0.67 (0.61, 0.73)0.16
 Tube current (mA) †405 (278, 553)333 (210, 490)0.05
 Peak kilovoltage †,1120 (120, 120)120 (118, 120)0.44
 Contrast Agent 0.84
  Visipaque 3203537
  Ultravist10
  Omnipaque11
  Optiray01
  Unknown1413
Number of segmented lesions per patient 0.54
 One20 (39)17 (33)
 Two31 (61)35 (67)
Values in parentheses are percentages unless stated otherwise. † Values are median (Inter quartile range). $ NRAS and BRAF mutations are mutually exclusively occurring; hence, we did not test for significance between BRAF wild type versus mutant cases. 1 Other values than those given in the median and inter quartile range do occur.
Table
Table 2. Performance of the models for BRAF mutation prediction based on different sets of features and lesions.
Table 2. Performance of the models for BRAF mutation prediction based on different sets of features and lesions.
Model 1
Radiomics All Lesions—WORC		Model 2
Radiomics Largest Lesion		Model 3
Radiomics NRAS Wild Type		Model 4
LIDC All Lesions		Model 5
Radiomics All Lesions—Benchmark
AUC0.49 [0.38, 0.59]0.65 [0.51, 0.79]0.49 [0.37, 0.61]0.46 [0.38, 0.55]0.50 [0.42, 0.58]
Accuracy0.48 [0.39, 0.57]0.61 [0.50, 0.72]0.65 [0.58, 0.71]0.49 [0.42, 0.56]0.50 [0.43, 0.57]
Sensitivity0.61 [0.44, 0.77]0.61 [0.42, 0.80]0.94 [0.87, 1.00]0.29 [0.11, 0.48]0.56 [0.32, 0.80]
Specificity0.37 [0.22, 0.52]0.60 [0.38, 0.82]0.08 [0.00, 0.17]0.66 [0.46, 0.86]0.44 [0.20, 0.69]
NPV0.53 [0.39, 0.66]0.61 [0.46, 0.76]0.35 [0.00, 0.75]0.52 [0.42, 0.61]0.43 [0.21, 0.66]
PPV0.45 [0.37, 0.53]0.63 [0.48, 0.77]0.67 [0.62, 0.72]0.44 [0.30, 0.58]0.47 [0.37, 0.56]
Abbreviations: AUC: area under the receiver operating characteristic curve; NPV: negative predictive value; PPV: positive predictive value.
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Share and Cite

MDPI and ACS Style

Angus, L.; Starmans, M.P.A.; Rajicic, A.; Odink, A.E.; Jalving, M.; Niessen, W.J.; Visser, J.J.; Sleijfer, S.; Klein, S.; van der Veldt, A.A.M. The BRAF P.V600E Mutation Status of Melanoma Lung Metastases Cannot Be Discriminated on Computed Tomography by LIDC Criteria nor Radiomics Using Machine Learning. J. Pers. Med. 2021, 11, 257. https://doi.org/10.3390/jpm11040257

AMA Style

Angus L, Starmans MPA, Rajicic A, Odink AE, Jalving M, Niessen WJ, Visser JJ, Sleijfer S, Klein S, van der Veldt AAM. The BRAF P.V600E Mutation Status of Melanoma Lung Metastases Cannot Be Discriminated on Computed Tomography by LIDC Criteria nor Radiomics Using Machine Learning. Journal of Personalized Medicine. 2021; 11(4):257. https://doi.org/10.3390/jpm11040257

Chicago/Turabian Style

Angus, Lindsay, Martijn P. A. Starmans, Ana Rajicic, Arlette E. Odink, Mathilde Jalving, Wiro J. Niessen, Jacob J. Visser, Stefan Sleijfer, Stefan Klein, and Astrid A. M. van der Veldt. 2021. "The BRAF P.V600E Mutation Status of Melanoma Lung Metastases Cannot Be Discriminated on Computed Tomography by LIDC Criteria nor Radiomics Using Machine Learning" Journal of Personalized Medicine 11, no. 4: 257. https://doi.org/10.3390/jpm11040257

APA Style

Angus, L., Starmans, M. P. A., Rajicic, A., Odink, A. E., Jalving, M., Niessen, W. J., Visser, J. J., Sleijfer, S., Klein, S., & van der Veldt, A. A. M. (2021). The BRAF P.V600E Mutation Status of Melanoma Lung Metastases Cannot Be Discriminated on Computed Tomography by LIDC Criteria nor Radiomics Using Machine Learning. Journal of Personalized Medicine, 11(4), 257. https://doi.org/10.3390/jpm11040257

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop