Next Article in Journal
The Impact of Anatomical Predisposition and Mechanism of Trauma on Dislocation of the Patella: A Retrospective Analysis of 104 Cases
Previous Article in Journal
Comparison of the Effectiveness of Palonosetron and Ramosetron in Preventing Postoperative Nausea and Vomiting: Updated Systematic Review and Meta-Analysis with Trial Sequential Analysis
 
 
Search for Articles:
Title / Keyword
Author / Affiliation / Email
Journal
Article Type
 
 
Advanced Search
 
Section
Special Issue
Volume
Issue
Number
Page
 
Journals
JPM
Volume 13
Issue 1
10.3390/jpm13010083
jpm-logo
Submit to this Journal Review for this Journal Propose a Special Issue
► Article Menu

Article Menu

share Share announcement Help format_quote Cite question_answer Discuss in SciProfiles thumb_up
...
Endorse textsms
...
Comment
first_page
settings
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Robustness of Radiomics in Pre-Surgical Computer Tomography of Non-Small-Cell Lung Cancer

by
Maria Paola Belfiore
1,*,
Mario Sansone
2,
Riccardo Monti
1,
Stefano Marrone
2,
Roberta Fusco
3,
Valerio Nardone
1,
Roberto Grassi
1 and
Alfonso Reginelli
1
1
Departement of Precision Medicine, Campania University “Luigi Vanvitelli”, 80138 Naples, Italy
2
Department of Electrical Engineering and Information Technology, University ‘Federico II’, 80131 Naples, Italy
3
Department of Research & Development IGEA Span, 80013 Carpi, Italy
*
Author to whom correspondence should be addressed.
J. Pers. Med. 2023, 13(1), 83; https://doi.org/10.3390/jpm13010083
Submission received: 22 November 2022 / Accepted: 20 December 2022 / Published: 29 December 2022
(This article belongs to the Section Omics/Informatics)
Browse Figures
Versions Notes

Abstract

:
Background: Radiomic features are increasingly used in CT of NSCLC. However, their robustness with respect to segmentation variability has not yet been demonstrated. The aim of this study was to assess radiomic features agreement across three kinds of segmentation. Methods: We retrospectively included 48 patients suffering from NSCLC who underwent pre-surgery CT. Two expert radiologists in consensus manually delineated three 3D-ROIs on each patient. To assess robustness for each feature, the intra-class correlation coefficient (ICC) across segmentations was evaluated. The ‘sensitivity’ of ICC upon some parameters affecting features computation (such as bin-width for first-order features and pixel-distances for second-order features) was also evaluated. Moreover, an assessment with respect to interpolator and isotropic resolution was also performed. Results: Our results indicate that ‘shape’ features tend to have excellent agreement (ICC > 0.9) across segmentations; moreover, they have approximately zero sensitivity to other parameters. ‘First-order’ features are in general sensitive to parameters variation; however, a few of them showed excellent agreement and low sensitivity (below 0.1) with respect to bin-width and pixel-distance. Similarly, a few second-order features showed excellent agreement and low sensitivity. Conclusions: Our results suggest that a limited number of radiomic features can achieve a high level of reproducibility in CT of NSCLC.

1. Introduction

Radiomics is gaining increasing interest as a tool for objective and quantitative radiological image evaluation [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16]; radiomic features have also been recently used in relation to genomic data (radiogenomics) [11,12,17,18,19,20,21,22]. However, in spite of the claimed ‘objectivity’, there is still an operator-dependent component in radiomic features. In particular, it has been observed that radiomic features are dependent on image perturbation [1]; moreover, radiomic features reproducibility might depend on region-of-interest (ROI) segmentation performed by different physicians.
A number of studies have reported on radiomics reproducibility. For example, in [3], the authors focused on reproducibility across different computerised tomography (CT) equipment or reconstruction techniques. However, they used only one segmentation obtained in consensus by three radiologists which might not be available in clinical routine. Many studies [1,2,3,4,23,24,25,26,27,28,29,30,31] highlighted that from a clinical point of view, before radiomic features can be introduced into the routine clinical evaluation of non-small-cell lung cancer (NSCLC) patients, robustness with respect to segmentation variability must also be assessed [31,32,33,34].
In this study, we evaluated the robustness of radiomic features across segmentations and evaluated the influence of other parameters (specifically, bin-width, pixel-distance in second-order features, interpolator and isotropic resolution) on a set of CT volumes from patients suffering from NSCLC.

2. Methods

2.1. Patients

In this retrospective study, we enrolled 48 patients who underwent high resolution CT (HRCT) for NSCLC evaluation before surgery.

2.2. Image Acquisition

Images were acquired with one single scanner at our institution with standard reconstruction kernel. Equipment data are reported in Table 1. For every patient, a high-resolution chest CT (GE Revolution 128 MDCT) without contrast was carried out on the patient in the supine position to determine the position, the dimension and number of the lesion. The protocol of the HRCT for every patient was acquired with thin sections defined as <1.5, rotation time 200–500 ms, matrix size 512 × 512, collimation: 1.5–3 mm, slice thickness < 1.5 mm and FOV 35 cm. Reconstruction algorithm: high spatial frequency. Every protocol was acquired in full inspiration.

2.3. ROIs Delineation

Two expert radiologists in consensus (MB and RM) manually delineated three segmentations on each patient. The software used was ITK-snap [35,36]. The first segmentation was free-hand and was delineated as accurately as possible. The second segmentation was a free-hand rough one. The third was delineated using a rough polygonal. The second and third segmentations were drawn including the lesion and some tissues around it.

2.4. Features Extraction

Radiomic features might support radiologists in quantifying medical image characteristics. However, given the difficulties of standardisation in this field, recently [4] the Imaging Biomarker Standardisation Initiative (IBSI) has assessed reproducibility. Therefore, we used radiomic features that have been approved by the IBSI committee.
Specifically, Pyradiomics [22] is an open-source python library allowing the calculation of radiomic features compatible and approved within the IBSI. In this study, we used Pyradiomics version 3.0.1.
We computed the following categories of radiomic features: shape 3D, first-order and second-order (glcm, glrlm, glszm, ngtdm and gldm). Each feature has been computed on original and wavelet (coif1) images. See [4,22] for a detailed account for features formulas and interpretation. A total of 851 features were computed.
In order to assess the dependence on important parameters used for features computation, we computed features using several values of bin-width and pixel-distance. Bin-width is the gray-level bin-width of the histogram and it can affect mainly features which require histogram computation: gray-level co-occurrence matrix (GLCM) and gray-level run length matrix. We repeated the feature computation for bin-width = 10, 20 and 40. Pixel-distance specifies the distances between the centre voxel and the neighbourhood, for which angles should be generated; it affects mainly the gray-level co-occurrence matrix (GLCM) and neighbouring gray-tone difference matrix (NTGDM). We repeated the feature computation using distance = 1, 2 and 4 pixels.
Moreover, we assessed robustness also with respect to isotropic resolution and type of interpolator. In particular, we used nearest-neighbour, linear and spline interpolators as the default implemented in pyradiomics. Resolutions were fixed at 1, 2 and 2.5 mm.

2.5. Robustness Evaluation

In order to quantify ‘robustness’ of a feature across segmentations, we used the intra-class correlation coefficient (ICC). The ICC is described in [37]. In particular, per each feature, we computed the lower bound of the 95% confidence interval for ICC. As a reference value for robustness, in line with other studies [3,4] we used ICC = 0.9, which indicates ‘excellent’ agreement. The computation of ICC and subsequent statistical analysis was performed in R version 3.6.3 [38] using package ‘irr’ version 0.84.1.
As regards to the ‘sensitivity’ to other parameters (bin-width, pixel-distances, type of interpolator and isotropic resolution), it was assessed per each parameter within each feature using the range of variation of ICC having other fixed parameters. In lack of previous studies, we arbitrarily considered a variation below 0.1 as low sensitivity. In Figure 1 there is a graphical summary of our analysis.
Lastly, the correlation between each feature reproducibility and lesion size (shape feature Voxel Volume) was analysed. To do this kind of analysis, it was not possible to use ICC (because patient information are flattened); we instead used the coefficient of variation (CV) which is the ratio between a measure of dispersion (standard deviation) of the distribution of one feature over a patient and the average value of the same feature over the same patient. The correlation between CV and tumour size is then computed.

3. Results

Patient demographic data and lesion type information are presented in Table 2. Figure 1 reports some illustrative examples of ROIs delineated on five randomly chosen patients. Flowchart of the method used in this study to evaluate robustness of radiomic features is shown in Scheme 1.
Figure 2, Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14 report the main results of our study. In particular, for each feature, we reported the ICC value with respect to the three segmentations. We grouped robustness per category of features. In particular, we considered ‘shape’, ‘first-order’, ’second-order-original’ and ‘second-order-wavelet’. As an example, we comment in detail on Figure 2; we report robustness (ICC) evaluated on shape features. These features (shape) seem to have generally high robustness. It is not less than 0.6 and many of them have robustness higher than 0.9; only three of them have an ICC below 0.9. This kind of feature does not depend on bin-width, distance, resolution or interpolator.
We can observe similar behaviours for other features (Figure 3, Figure 4, Figure 5, Figure 6, Figure 7, Figure 8, Figure 9, Figure 10, Figure 11, Figure 12, Figure 13 and Figure 14). However, in some cases, the ICC can be well below 0.9 indicating poor robustness of that feature. Moreover, in most wavelet-based and second-order features, there is a dependence on bin-width, distance, resolution or interpolator.
As regards to sensitivity of the ICC of features to other parameters such as bin-width, pixel-distance, isotropic resolution and interpolator, we report in Figure 15 the sensitivity per each feature. The first 107 features were computed from the original image; the other 744 were computed from wavelet images. The index in the figure refers to a feature list given as a separate file. We note that ‘shape’ features (index from 94 to 107) have very small (approximately zero) sensitivity.
Figure 16 reports the correlation between each feature CV and lesion size. The Spearman method was used. Red circles represent correlations with ‘shape’ features; only a weak correlation exists.

4. Discussion

The aim of this study was to evaluate the robustness of radiomic features with respect to ROI segmentation in the case of pre-surgical CT of NSCLC. We quantified robustness as ‘agreement’ among three segmentations using the intra-class correlation coefficient (ICC). We chose ICC = 0.9 as a reference value indicating an ‘excellent’ agreement [37]. The sensitivity of ICC upon bin-width, distance, resolution or interpolator was also evaluated. We assessed the following feature categories: shape (14 features), first-order (18), gray-level co-occurrence matrix (24), GLDM (14), GLRLM (16), GLSZM (16) and NGTDM (5): a total of 107 features. Moreover, the same features were assessed on wavelet images (coif1) using High/Low filtering in x-y-z directions for a total of 851 features.
Our results showed that some features were robust having high ICC; however, a number of features were sensitive to segmentation, having low ICC. Moreover, a dependence on bin-width, distance, resolution or interpolator has been observed for many features.
Most importantly, we observed that shape features showed high robustness (ICC > 0.90). This is in line with intuition, as shape features (e.g., VoxelVolume, maximum diameter or surface area) should not vary too much across segmentations. It should be noticed, however, that surface-volume ratio and sphericity can reach very low ICC values (down to 0.75).
Another important result of our study is that we observed a low number of features robust with respect to segmentations and to bin-width, distance, resolution or interpolator. In fact, our results showed that a large number of first- and second-order features have an ICC lower than 0.9. For these features the agreement ICC, across segmentations and other source of variability, is less than excellent.
Lastly, the correlation analysis showed that ‘shape’ features CV is only weakly negatively correlated with lesion size: as tumour size increases, CV weakly decreases. This might be explained by the fact that the larger the tumour size, the larger the segmentation variability by manual delineation. Moreover, Figure 16 shows that there is a large variability among feature CV vs lesion size.
Assessment of radiomics robustness is an important task for guaranteeing reproducibility of radiomic analysis across segmentations; it has been also addressed in other papers [3,4]. A very important paper in this field is [4] in which a large number of radiomics features was evaluated for reproducibility across 25 teams. However, they used only one CT of a patient suffering from lung cancer. Subsequently, in [3], the authors evaluated radiomics reproducibility of NSCLC across different CT equipment and reconstruction algorithms. In this last paper, the authors analysed 103 patients. However, the authors did not explicitly address the dependence on segmentations and feature parameters.
The results of our study are in line with previous studies [3,4,13,15,23,25,26,30,39,40,41,42,43,44,45,46,47]; however, as a difference, we specifically addressed segmentation variability with a fixed CT device but analysed the dependence on bin-width, distance, resolution or interpolator.
Commentary is in order regarding the applicability of our results in clinical settings. The fact that a feature might be suitable for clinical application should be assessed, not only looking for high reproducibility; in fact, a certain variability of a feature might be tolerated given that the variability in the patients is larger. Further, in ultimate analysis, features are used as an input to classifiers; therefore, also the specific combination of classifier plus feature must be optimised. Specifically, our study addressed the problem of features selection only from the point of view of high reproducibility but did not address the points just mentioned.
One important limitation of this study is the reduced sample size (48 patients). In order to gain insight into the reproducibility of radiomics features, large databases should be analysed.

5. Conclusions

Radiomics analysis has the potential for improving diagnosis in many pathologies and in particular in NSCLC. However, the reproducibility of features with respect to variability in 3D ROI segmentations and other sources of variability in feature computation (such as bin-width, pixel-distance, interpolator and isotropic resolution) must be assessed before introduction into the clinical practice. In our study, we observed that ‘excellent’ agreement (ICC > 0.9) across different sources of variability can be achieved by a relatively small number of features: in particular, ‘shape’ features, some second-order features and some wavelet-based features.

Author Contributions

M.P.B. and R.M., expert radiologists, performed manual segmentation. Quality assessment was provided by M.P.B. and data analysis were provided from M.S. and S.M. Possible disagreements were resolved after a panel discussion with two other authors familiar with the project (V.N. and A.R.). All articles were identified by two reviewers (R.G. and R.F.) with an experience of at least 5 years in the radiomic field. All authors have read and agreed to the published version of the manuscript.

Funding

The research leading to these results has received funding from Ministero dell’Universita e della Ricerca (MIUR) under PRIN 2021 (2020YL3FB3_002) “REASONING: foRmal mEthods for computAtional analySis for diagnOsis and progNosis in imaging” (University of Campania “L. Vanvitelli”).

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Image and segmentations data might be shared upon reasonable request.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Zwanenburg, A.; Leger, S.; Agolli, L.; Pilz, K.; Troost, E.G.C.; Richter, C.; Löck, S. Assessing robustness of radiomic features by image perturbation. Sci. Rep. 2019, 9, 614. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  2. Defeudis, A.; Mazzetti, S.; Panic, J.; Micilotta, M.; Vassallo, L.; Giannetto, G.; Gatti, M.; Faletti, R.; Cirillo, S.; Regge, D.; et al. MRI-based radiomics to predict response in locally advanced rectal cancer: Comparison of manual and automatic segmentation on external validation in a multicentre study. Eur. Radiol. Exp. 2022, 6, 19. [Google Scholar] [CrossRef] [PubMed]
  3. Rinaldi, L.; De Angelis, S.P.; Raimondi, S.; Rizzo, S.; Fanciullo, C.; Rampinelli, C.; Mariani, M.; Lascialfari, A.; Cremonesi, M.; Orecchia, R.; et al. Reproducibility of radiomic features in CT images of NSCLC patients: An integrative analysis on the impact of acquisition and reconstruction parameters. Eur. Radiol. Exp. 2022, 6, 2. [Google Scholar] [CrossRef] [PubMed]
  4. Zwanenburg, A.; Vallières, M.; Abdalah, M.A.; Aerts, H.J.W.L.; Andrearczyk, V.; Apte, A.; Ashrafinia, S.; Bakas, S.; Beukinga, R.J.; Boellaard, R.; et al. The Image Biomarker Standardization Initiative: Standardized Quantitative Radiomics for High-Throughput Image-based Phenotyping. Radiology 2020, 295, 328–338. [Google Scholar] [CrossRef] [Green Version]
  5. Staal, F.C.R.; van der Reijd, D.J.; Taghavi, M.; Lambregts, D.M.J.; Beets-Tan, R.G.H.; Maas, M. Radiomics for the Prediction of Treatment Outcome and Survival in Patients With Colorectal Cancer: A Systematic Review. Clin. Color. Cancer 2021, 20, 52–71. [Google Scholar] [CrossRef]
  6. Di Re, A.M.; Sun, Y.; Sundaresan, P.; Hau, E.; Toh, J.W.T.; Gee, H.; Or, M.; Haworth, A. MRI radiomics in the prediction of therapeutic response to neoadjuvant therapy for locoregionally advanced rectal cancer: A systematic review. Expert Rev. Anticancer Ther. 2021, 21, 425–449. [Google Scholar] [CrossRef]
  7. Yan, R.; Hao, D.; Li, J.; Liu, J.; Hou, F.; Chen, H.; Duan, L.; Huang, C.; Wang, H.; Yu, T. Magnetic Resonance Imaging-Based Radiomics Nomogram for Prediction of the Histopathological Grade of Soft Tissue Sarcomas: A Two-Center Study. J. Magn. Reson. Imaging 2021, 53, 1683–1696. [Google Scholar] [CrossRef]
  8. Yuan, Z.; Frazer, M.; Rishi, A.; Latifi, K.; Tomaszewski, M.R.; Moros, E.G.; Feygelman, V.; Felder, S.; Sanchez, J.; Dessureault, S.; et al. Pretreatment CT and PET radiomics predicting rectal cancer patients in response to neoadjuvant chemoradiotherapy. Rep. Pract. Oncol. Radiother. 2021, 26, 29–34. [Google Scholar] [CrossRef]
  9. Rizzo, S.; Manganaro, L.; Dolciami, M.; Gasparri, M.L.; Papadia, A.; Del Grande, F. Computed Tomography Based Radiomics as a Predictor of Survival in Ovarian Cancer Patients: A Systematic Review. Cancers 2021, 13, 573. [Google Scholar] [CrossRef]
  10. Qin, H.; Que, Q.; Lin, P.; Li, X.; Wang, X.-R.; He, Y.; Chen, J.-Q.; Yang, H. Magnetic resonance imaging (MRI) radiomics of papillary thyroid cancer (PTC): A comparison of predictive performance of multiple classifiers modeling to identify cervical lymph node metastases before surgery. La Radiol. Med. 2021, 126, 1312–1327. [Google Scholar] [CrossRef]
  11. Nardone, V.; Reginelli, A.; Grassi, R.; Vacca, G.; Giacobbe, G.; Angrisani, A.; Clemente, A.; Danti, G.; Correale, P.; Carbone, S.F.; et al. Ability of Delta Radiomics to Predict a Complete Pathological Response in Patients with Loco-Regional Rectal Cancer Addressed to Neoadjuvant Chemo-Radiation and Surgery. Cancers 2022, 14, 3004. [Google Scholar] [CrossRef] [PubMed]
  12. Mazzei, M.A.; Di Giacomo, L.; Bagnacci, G.; Nardone, V.; Gentili, F.; Lucii, G.; Tini, P.; Marrelli, D.; Morgagni, P.; Mura, G.; et al. Delta-radiomics and response to neoadjuvant treatment in locally advanced gastric cancer—A multicenter study of GIRCG (Italian Research Group for Gastric Cancer). Quant. Imaging Med. Surg. 2021, 11, 2376–2387. [Google Scholar] [CrossRef] [PubMed]
  13. Brunese, L.; Mercaldo, F.; Reginelli, A.; Santone, A. Radiomics for gleason score detection through deep learning. Sensors 2020, 20, 5411. [Google Scholar] [CrossRef] [PubMed]
  14. Nardone, V.; Reginelli, A.; Guida, C.; Belfiore, M.P.; Biondi, M.; Mormile, M.; Buonamici, F.B.; Di Giorgio, E.; Spadafora, M.; Tini, P.; et al. Delta-radiomics increases multicentre reproducibility: A phantom study. Med. Oncol. 2020, 37, 38. [Google Scholar] [CrossRef] [PubMed]
  15. Reginelli, A.; Nardone, V.; Giacobbe, G.; Belfiore, M.P.; Grassi, R.; Schettino, F.; Del Canto, M.; Grassi, R.; Cappabianca, S. Radiomics as a new frontier of imaging for cancer prognosis: A narrative review. Diagnostics 2021, 11, 1796. [Google Scholar] [CrossRef] [PubMed]
  16. Nardone, V.; Reginelli, A.; Grassi, R.; Boldrini, L.; Vacca, G.; D’Ippolito, E.; Annunziata, S.; Farchione, A.; Belfiore, M.P.; Desideri, I.; et al. Delta radiomics: A systematic review. La Radiol. Med. 2021, 126, 1571–1583. [Google Scholar] [CrossRef]
  17. Shui, L.; Ren, H.; Yang, X.; Li, J.; Chen, Z.; Yi, C.; Zhu, H.; Shui, P. The Era of Radiogenomics in Precision Medicine: An Emerging Approach to Support Diagnosis, Treatment Decisions, and Prognostication in Oncology. Front. Oncol. 2020, 10, 570465. [Google Scholar] [CrossRef]
  18. Ninatti, G.; Kirienko, M.; Neri, E.; Sollini, M.; Chiti, A. Imaging-Based Prediction of Molecular Therapy Targets in NSCLC by Radiogenomics and AI Approaches: A Systematic Review. Diagnostics 2020, 10, 359. [Google Scholar] [CrossRef]
  19. Wong, C.W.; Chaudhry, A. Radiogenomics of lung cancer. J. Thorac. Dis. 2020, 12, 5104–5109. [Google Scholar] [CrossRef]
  20. Saha, A.; Harowicz, M.R.; Grimm, L.J.; Kim, C.E.; Ghate, S.V.; Walsh, R.; Mazurowski, M.A. A machine learning approach to radiogenomics of breast cancer: A study of 922 subjects and 529 DCE-MRI features. Br. J. Cancer 2018, 119, 508–516. [Google Scholar] [CrossRef]
  21. Santone, A.; Belfiore, M.P.; Mercaldo, F.; Varriano, G.; Brunese, L. On the Adoption of Radiomics and Formal Methods for COVID-19 Coronavirus Diagnosis. Diagnostics 2021, 11, 293. [Google Scholar] [CrossRef] [PubMed]
  22. van Griethuysen, J.J.M.; Fedorov, A.; Parmar, C.; Hosny, A.; Aucoin, N.; Narayan, V.; Beets-Tan, R.G.H.; Fillion-Robin, J.-C.; Pieper, S.; Aerts, H.J.W.L. Computational Radiomics System to Decode the Radiographic Phenotype. Cancer Res. 2017, 77, e104–e107. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  23. Xue, C.; Yuan, J.; Poon, D.M.; Zhou, Y.; Yang, B.; Yu, S.K.; Cheung, Y.K. Reliability of MRI radiomics features in MR-guided radiotherapy for prostate cancer: Repeatability, reproducibility, and within-subject agreement. Med. Phys. 2021, 48, 6976–6986. [Google Scholar] [CrossRef] [PubMed]
  24. Liu, S.; He, J.; Liu, S.; Ji, C.; Guan, W.; Chen, L.; Guan, Y.; Yang, X.; Zhou, Z. Radiomics analysis using contrast-enhanced CT for preoperative prediction of occult peritoneal metastasis in advanced gastric cancer. Eur. Radiol. 2020, 30, 239–246. [Google Scholar] [CrossRef] [PubMed]
  25. Hu, H.-T.; Shan, Q.-Y.; Chen, S.-L.; Li, B.; Feng, S.-T.; Xu, E.-J.; Li, X.; Long, J.-Y.; Xie, X.-Y.; Lu, M.-D.; et al. CT-based radiomics for preoperative prediction of early recurrent hepatocellular carcinoma: Technical reproducibility of acquisition and scanners. La Radiol. Med. 2020, 125, 697–705. [Google Scholar] [CrossRef] [PubMed]
  26. Scalco, E.; Belfatto, A.; Mastropietro, A.; Rancati, T.; Avuzzi, B.; Messina, A.; Valdagni, R.; Rizzo, G. T2w-MRI signal normalization affects radiomics features reproducibility. Med. Phys. 2020, 47, 1680–1691. [Google Scholar] [CrossRef] [PubMed]
  27. Li, Q.; Qi, L.; Feng, Q.-X.; Liu, C.; Sun, S.-W.; Zhang, J.; Yang, G.; Ge, Y.-Q.; Zhang, Y.-D.; Liu, X.-S. Machine Learning-Based Computational Models Derived From Large—Scale Radiographic-Radiomic Images Can Help Predict Adverse Histopathological Status of Gastric Cancer. Clin. Transl. Gastroenterol. 2019, 10, e00079. [Google Scholar] [CrossRef]
  28. Pavic, M.; Bogowicz, M.; Würms, X.; Glatz, S.; Finazzi, T.; Riesterer, O.; Roesch, J.; Rudofsky, L.; Friess, M.; Veit-Haibach, P.; et al. Influence of inter-observer delineation variability on radiomics stability in different tumor sites. Acta Oncol. 2018, 57, 1070–1074. [Google Scholar] [CrossRef] [Green Version]
  29. Belfiore, G.; Tedeschi, E.; Ronza, F.M.; Belfiore, M.P.; Borsi, E.; Ianniello, G.P.; Rotondo, A. CT-guided radiofrequency ablation in the treatment of recurrent rectal cancer. Am. J. Roentgenol. 2009, 192, 137–141. [Google Scholar] [CrossRef]
  30. Sansone, M.; Grassi, R.; Belfiore, M.P.; Gatta, G.; Grassi, F.; Pinto, F.; La Casella, G.V.; Fusco, R.; Cappabianca, S.; Granata, V.; et al. Radiomic features of breast parenchyma: Assessing differences between FOR PROCESSING and FOR PRESENTATION digital mammography. Insights Into Imaging 2021, 12, 147. [Google Scholar] [CrossRef]
  31. Reginelli, A.; Belfiore, M.P.; Monti, R.; Cozzolino, I.; Costa, M.; Vicidomini, G.; Grassi, R.; Morgillo, F.; Urraro, F.; Nardone, V.; et al. The texture analysis as a predictive method in the assessment of the cytological specimen of CT-guided FNAC of the lung cancer. Med. Oncol. 2020, 37, 54. [Google Scholar] [CrossRef] [PubMed]
  32. Brunese, L.; Mercaldo, F.; Reginelli, A.; Santone, A. Neural Networks for Lung Cancer Detection through Radiomic Features. In Proceedings of the 2019 International Joint Conference on Neural Networks (IJCNN), Budapest, Hungary, 14–19 July 2019. [Google Scholar]
  33. Nardone, V.; Nanni, S.; Pastina, P.; Vinciguerra, C.; Cerase, A.; Correale, P.; Guida, C.; Giordano, A.; Tini, P.; Reginelli, A.; et al. Role of perilesional edema and tumor volume in the prognosis of non-small cell lung cancer (NSCLC) undergoing radiosurgery (SRS) for brain metastases. Strahlenther. Und Onkol. 2019, 195, 734–744. [Google Scholar] [CrossRef] [PubMed]
  34. Brunese, L.; Mercaldo, F.; Reginelli, A.; Santone, A. Lung Cancer Detection and Characterisation through Genomic and Radiomic Biomarkers. In Proceedings of the 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, 19–24 July 2020. [Google Scholar]
  35. Yushkevich, P.A.; Piven, J.; Hazlett, H.C.; Smith, R.G.; Ho, S.; Gee, J.C.; Gerig, G. User-guided 3D active contour segmentation of anatomical structures: Significantly improved efficiency and reliability. NeuroImage 2006, 31, 1116–1128. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  36. Reginelli, A.; Grassi, R.; Feragalli, B.; Belfiore, M.P.; Montanelli, A.; Patelli, G.; La Porta, M.; Urraro, F.; Fusco, R.; Granata, V.; et al. Coronavirus Disease 2019 (COVID-19) in Italy: Double Reading of Chest CT Examination. Biology 2021, 10, 89. [Google Scholar] [CrossRef] [PubMed]
  37. Koo, T.K.; Li, M.Y. A Guideline of Selecting and Reporting Intraclass Correlation Coefficients for Reliability Research. J. Chiropr. Med. 2016, 15, 155–163. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  38. R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for Statistical Computing: Vienna, Austria; Available online: https://www.R-project.org/ (accessed on 3 November 2022).
  39. Reginelli, A.; Capasso, R.; Petrillo, M.; Rossi, C.; Faella, P.; Grassi, R.; Belfiore, M.P.; Rossi, G.; Muto, M.; Muto, P.; et al. Looking for Lepidic Component inside Invasive Adenocarcinomas Appearing as CT Solid Solitary Pulmonary Nodules (SPNs): CT Morpho-Densitometric Features and 18-FDG PET Findings. BioMed Res. Int. 2019, 2019, 7683648. [Google Scholar] [CrossRef] [Green Version]
  40. Nardone, V.; Boldrini, L.; Grassi, R.; Franceschini, D.; Morelli, I.; Becherini, C.; Loi, M.; Greto, D.; Desideri, I. Radiomics in the Setting of Neoadjuvant Radiotherapy: A New Approach for Tailored Treatment. Cancers 2021, 13, 3590. [Google Scholar] [CrossRef]
  41. Urraro, F.; Nardone, V.; Reginelli, A.; Varelli, C.; Angrisani, A.; Patanè, V.; D’Ambrosio, L.; Roccatagliata, P.; Russo, G.M.; Gallo, L.; et al. MRI Radiomics in Prostate Cancer: A Reliability Study. Front. Oncol. 2021, 11, 805137. [Google Scholar] [CrossRef]
  42. Santone, A.; Brunese, M.C.; Donnarumma, F.; Guerriero, P.; Mercaldo, F.; Reginelli, A.; Miele, V.; Giovagnoni, A.; Brunese, L. Radiomic features for prostate cancer grade detection through formal verification. La Radiol. Med. 2021, 126, 688–697. [Google Scholar] [CrossRef]
  43. Anjari, M.; Guha, A.; Burd, C.; Varela, M.; Goh, V.; Connor, S. Apparent diffusion coefficient agreement and reliability using different region of interest methods for the evaluation of head and neck cancer post chemo-radiotherapy. Dentomaxillofacial Radiol. 2021, 50, 20200579. [Google Scholar] [CrossRef]
  44. Fusco, R.; Granata, V.; Sansone, M.; Rega, D.; Delrio, P.; Tatangelo, F.; Romano, C.; Avallone, A.; Pupo, D.; Giordano, M.; et al. Validation of the standardized index of shape tool to analyze DCE-MRI data in the assessment of neo-adjuvant therapy in locally advanced rectal cancer. La Radiol. Med. 2021, 126, 1044–1054. [Google Scholar] [CrossRef] [PubMed]
  45. Cappabianca, S.; Porto, A.; Petrillo, M.; Greco, B.; Reginelli, A.; Ronza, F.; Setola, F.; Rossi, G.; Di Matteo, A.; Muto, R.; et al. Preliminary study on the correlation between grading and histology of solitary pulmonary nodules and contrast enhancement and [18F]fluorodeoxyglucose standardised uptake value after evaluation by dynamic multiphase CT and PET/CT. J. Clin. Pathol. 2011, 64, 114–119. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  46. Russo, U.; Sabatino, V.; Nizzoli, R.; Tiseo, M.; Cappabianca, S.; Reginelli, A.; Carrafiello, G.; Brunese, L.; De Filippo, M. Transthoracic computed tomography-guided lung biopsy in the new era of personalized medicine. Future Oncol. 2019, 15, 1125–1134. [Google Scholar] [CrossRef] [PubMed]
  47. Sansone, M.; Marrone, S.; Di Salvio, G.; Belfiore, M.P.; Gatta, G.; Fusco, R.; Vanore, L.; Zuiani, C.; Grassi, F.; Vietri, M.T.; et al. Comparison between two packages for pectoral muscle removal on mammographic images. La Radiol. Med. 2022, 127, 848–856. [Google Scholar] [CrossRef]
Jpm 13 00083 sch001 550
Scheme 1. Flowchart of the method used in this study to evaluate robustness of radiomic features. Three 3D ROIs have been delineated on each subject included in the study. Radiomic features from the original image and from the wavelet transformed have been computed using different values of four parameters: bin-width, pixel-distance, interpolator and isotropic resolution. ICC has been used for robustness evaluation and variability of ICC across a parameter was used as an index of ‘sensitivity’ of the feature to the parameter.
Scheme 1. Flowchart of the method used in this study to evaluate robustness of radiomic features. Three 3D ROIs have been delineated on each subject included in the study. Radiomic features from the original image and from the wavelet transformed have been computed using different values of four parameters: bin-width, pixel-distance, interpolator and isotropic resolution. ICC has been used for robustness evaluation and variability of ICC across a parameter was used as an index of ‘sensitivity’ of the feature to the parameter.
Jpm 13 00083 sch001
Jpm 13 00083 g001a 550Jpm 13 00083 g001b 550
Figure 1. Illustrative examples of manual segmentations. For each patient on a row, different segmentations are showed superimposed on CT data: (a) polygonal, (b) free-hand accurate and (c) free-hand rough.
Figure 1. Illustrative examples of manual segmentations. For each patient on a row, different segmentations are showed superimposed on CT data: (a) polygonal, (b) free-hand accurate and (c) free-hand rough.
Jpm 13 00083 g001aJpm 13 00083 g001b
Jpm 13 00083 g002 550
Figure 2. Robustness of shape features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 shape features. The robustness is above 0.60 for all of them. However, three of them are less robust (<0.90) across segmentations (Sphericity, SurfaceVolumeRatio and Elongation); this might be because they largely depend on the precise ROI contours. They do not depend on bin-width, distance, resolution or interpolator. With the exception of Flatness, the other features have a robustness greater than 0.95.
Figure 2. Robustness of shape features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 shape features. The robustness is above 0.60 for all of them. However, three of them are less robust (<0.90) across segmentations (Sphericity, SurfaceVolumeRatio and Elongation); this might be because they largely depend on the precise ROI contours. They do not depend on bin-width, distance, resolution or interpolator. With the exception of Flatness, the other features have a robustness greater than 0.95.
Jpm 13 00083 g002
Jpm 13 00083 g003 550
Figure 3. Robustness of first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for skewness, kurtosis, uniformity and 90 percentile. The latter has ICC > 0.90. They are not dependent on bin-width (only entropy has slight dependence). Because they are first-order, they are not dependent on distance, neither do they depend on resolution or interpolator.
Figure 3. Robustness of first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for skewness, kurtosis, uniformity and 90 percentile. The latter has ICC > 0.90. They are not dependent on bin-width (only entropy has slight dependence). Because they are first-order, they are not dependent on distance, neither do they depend on resolution or interpolator.
Jpm 13 00083 g003
Jpm 13 00083 g004 550
Figure 4. Robustness of second-order GLCM features across segmentations. Pyradiomics computes 24 second-order GLCM features. The robustness (ICC) is above 0.8 per all values of distance, only for Autocorrelation, Idn, Idmn, SumAverage, JointAverage, InverseVariance, JointEnergy, MaximumProbability, Id and Idm. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line on the right indicates the 0.9 threshold).
Figure 4. Robustness of second-order GLCM features across segmentations. Pyradiomics computes 24 second-order GLCM features. The robustness (ICC) is above 0.8 per all values of distance, only for Autocorrelation, Idn, Idmn, SumAverage, JointAverage, InverseVariance, JointEnergy, MaximumProbability, Id and Idm. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line on the right indicates the 0.9 threshold).
Jpm 13 00083 g004
Jpm 13 00083 g005 550
Figure 5. Robustness of second-order GLDM features across segmentations. Pyradiomics computes 14 second-order GLDM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for nine of them. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line indicates the 0.9 threshold).
Figure 5. Robustness of second-order GLDM features across segmentations. Pyradiomics computes 14 second-order GLDM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for nine of them. Robustness of many features across segmentations is dependent on bin-width and distance (the vertical dashed red line indicates the 0.9 threshold).
Jpm 13 00083 g005
Jpm 13 00083 g006 550
Figure 6. Robustness of second-order GLRLM features across segmentations. Pyradiomics computes 16 second-order GLRLM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).
Figure 6. Robustness of second-order GLRLM features across segmentations. Pyradiomics computes 16 second-order GLRLM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).
Jpm 13 00083 g006
Jpm 13 00083 g007 550
Figure 7. Robustness of second-order GLSZM features across segmentations. Pyradiomics computes 16 second-order GLSZM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).
Figure 7. Robustness of second-order GLSZM features across segmentations. Pyradiomics computes 16 second-order GLSZM features. The robustness (ICC) is above 0.8, per all values of distance and binw, only for 11 of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).
Jpm 13 00083 g007
Jpm 13 00083 g008 550
Figure 8. Robustness of second-order NGTDM features across segmentations. Pyradiomics computes five second-order NGTDM features. The robustness (ICC) is above 0.9, for all values of distance and binw, only for one of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).
Figure 8. Robustness of second-order NGTDM features across segmentations. Pyradiomics computes five second-order NGTDM features. The robustness (ICC) is above 0.9, for all values of distance and binw, only for one of them. Robustness of many features across segmentations is dependent on bin-width but not on distance (the vertical dashed red line indicates the 0.9 threshold).
Jpm 13 00083 g008
Jpm 13 00083 g009 550
Figure 9. Robustness of wavelet-based first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for Skewness, Kurtosis, Uniformity and 90Percentile. The latter has ICC > 0.90. They are not much dependent on bin-width (only entropy and uniformity have slight dependence). Because they are first-order, they are not dependent on distance neither do they depend on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Figure 9. Robustness of wavelet-based first-order histogram features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 18 first-order features. The robustness (ICC) is above 0.8 only for Skewness, Kurtosis, Uniformity and 90Percentile. The latter has ICC > 0.90. They are not much dependent on bin-width (only entropy and uniformity have slight dependence). Because they are first-order, they are not dependent on distance neither do they depend on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Jpm 13 00083 g009
Jpm 13 00083 g010 550
Figure 10. Robustness of wavelet-based second-order GLCM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 24 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Figure 10. Robustness of wavelet-based second-order GLCM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 24 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Jpm 13 00083 g010
Jpm 13 00083 g011 550
Figure 11. Robustness of wavelet-based second-order GLDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Figure 11. Robustness of wavelet-based second-order GLDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 14 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Jpm 13 00083 g011
Jpm 13 00083 g012 550
Figure 12. Robustness of wavelet-based second-order GLRLM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Figure 12. Robustness of wavelet-based second-order GLRLM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Jpm 13 00083 g012
Jpm 13 00083 g013 550
Figure 13. Robustness of wavelet-based second-order GLSZM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Figure 13. Robustness of wavelet-based second-order GLSZM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides 16 such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Jpm 13 00083 g013
Jpm 13 00083 g014 550
Figure 14. Robustness of wavelet-based second-order NGTDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides five such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Figure 14. Robustness of wavelet-based second-order NGTDM features across segmentations. The vertical dashed red line indicates the ICC > 0.9 threshold. Pyradiomics provides five such features. The robustness (ICC) is above 0.9 only for few of them. They are dependent on bin-width (binw: 10, 20, 40) and on (dst: 1, 2, 4). Because they are second-order, they are dependent on distance (dst) but they do not depend much on resolution or interpolator. However, the wavelet features are dependent on the direction used for wavelet computation (each circle on a line is a different gradient direction).
Jpm 13 00083 g014
Jpm 13 00083 g015 550
Figure 15. Sensitivity to other parameters. Per each feature, the range of variation of ICC is reported per each parameter (bin-width, pixel distance, interpolator and isotropic resolution) having fixed the other three.
Figure 15. Sensitivity to other parameters. Per each feature, the range of variation of ICC is reported per each parameter (bin-width, pixel distance, interpolator and isotropic resolution) having fixed the other three.
Jpm 13 00083 g015
Jpm 13 00083 g016 550
Figure 16. Correlation between CV for each feature (on x-axis) and lesion size (shape feature Voxel Volume). Spearman method was used. Red circles indicate correlation for ‘shape’ features; only a weak correlation exists.
Figure 16. Correlation between CV for each feature (on x-axis) and lesion size (shape feature Voxel Volume). Spearman method was used. Red circles indicate correlation for ‘shape’ features; only a weak correlation exists.
Jpm 13 00083 g016
Table
Table 1. CT equipment and acquisition parameters.
Table 1. CT equipment and acquisition parameters.
Pixel spacing0.578–0.976 mm
Slice thickness1.5 mm
KVP120 kV
EquipmentRevolution HD—GE MEDICAL SYSTEMS
Scan optionsHelical Mode
Table
Table 2. Patient characteristics.
Table 2. Patient characteristics.
Sex14 F, 34 M
Age49–87 (mean 70, std 10)
Diagnosis 32 Adenocarcinoma,
14 Squamous,
1 Neuroendocrine
1 atypical carcinoid
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

MDPI and ACS Style

Belfiore, M.P.; Sansone, M.; Monti, R.; Marrone, S.; Fusco, R.; Nardone, V.; Grassi, R.; Reginelli, A. Robustness of Radiomics in Pre-Surgical Computer Tomography of Non-Small-Cell Lung Cancer. J. Pers. Med. 2023, 13, 83. https://doi.org/10.3390/jpm13010083

AMA Style

Belfiore MP, Sansone M, Monti R, Marrone S, Fusco R, Nardone V, Grassi R, Reginelli A. Robustness of Radiomics in Pre-Surgical Computer Tomography of Non-Small-Cell Lung Cancer. Journal of Personalized Medicine. 2023; 13(1):83. https://doi.org/10.3390/jpm13010083

Chicago/Turabian Style

Belfiore, Maria Paola, Mario Sansone, Riccardo Monti, Stefano Marrone, Roberta Fusco, Valerio Nardone, Roberto Grassi, and Alfonso Reginelli. 2023. "Robustness of Radiomics in Pre-Surgical Computer Tomography of Non-Small-Cell Lung Cancer" Journal of Personalized Medicine 13, no. 1: 83. https://doi.org/10.3390/jpm13010083

APA Style

Belfiore, M. P., Sansone, M., Monti, R., Marrone, S., Fusco, R., Nardone, V., Grassi, R., & Reginelli, A. (2023). Robustness of Radiomics in Pre-Surgical Computer Tomography of Non-Small-Cell Lung Cancer. Journal of Personalized Medicine, 13(1), 83. https://doi.org/10.3390/jpm13010083

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop