Next Article in Journal
Verification of an Accommodative Response for Depth Measurement of Floating Hologram Using a Holographic Optical Element
Next Article in Special Issue
Risk-Averse Food Recommendation Using Bayesian Feedforward Neural Networks for Patients with Type 1 Diabetes Doing Physical Activities
Previous Article in Journal
Total and Sustainable Valorisation of Olive Pomace Using a Fractionation Approach
Previous Article in Special Issue
In-Silico Evaluation of Glucose Regulation Using Policy Gradient Reinforcement Learning for Patients with Type 1 Diabetes Mellitus
 
 
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Development of a Mortality Risk Model in Elderly Hip Fracture Patients by Different Analytical Approaches

1
Department of Health-Business Administration, Fooyin University, Kaohsiung 83102, Taiwan
2
Department of Orthopedics, Kaohsiung Veterans General Hospital, Kaohsiung 813414, Taiwan
3
Section of Critical Care and Cardiovascular Medical Center, Kaohsiung Veterans General Hospital, Kaohsiung 813414, Taiwan
4
Research Center of Medical Informatics, Kaohsiung Veterans General Hospital, Kaohsiung 813414, Taiwan
*
Author to whom correspondence should be addressed.
Appl. Sci. 2020, 10(19), 6787; https://doi.org/10.3390/app10196787
Submission received: 3 September 2020 / Revised: 17 September 2020 / Accepted: 24 September 2020 / Published: 28 September 2020
(This article belongs to the Special Issue Medical Artificial Intelligence)

Abstract

:
Hip fracture is a major health issue that accompanies community aging. The most critical time after a hip fracture should be the first year. Care systems and surgical techniques for hip fractures have improved, so the trend of mortality in elderly hip fracture could be changed with them. Therefore, we observed the changes in the trend and critical factors for first-year mortality for the hip fractures in an elderly population in Taiwan, and mortality of prognosis prediction model was developed for the early diagnosis using a population-based database in Taiwan (National Health Insurance Research Database, NHIRD). A total of 166,274 elderly subjects with an age greater than 60-years-old from 2001 to 2010 were collected for this study. Cox proportional-hazards (PH) regression and logistic regression were calculated to odds ratio and hazard ratio for mortality of those patients and compared it. Data mining algorithms were also used to generate a risk stratification prediction model. The first-year mortality rate of the overall study group was 21.5% in 2001 and 15.0% in 2010 (p for trend < 0.001). In the male subgroup, the first-year mortality rate was 29.3% in 2001 and decreased to 17.3% in 2010; the trend of standardized mortality ratio was significantly decreased from 4.4 to 2.6 (p for trend < 0.001). By logistic regression, mortality significantly increased with age and male gender. Furthermore, gender, age, patients with diabetes mellitus (DM), cardiovascular (CV), and renal comorbidity, and surgical intervention can be variables for constructing the risk stratification model. The findings of the study will be used for helping related field physicians to predict the prognosis risk of hip fracture patients, and provide evidence-based tailored treatment recommendations for those patients. It may consider to build various models for predicting the prognosis of hip fracture or integrating prediction algorithms into the computerized physician order entry system, thus creating a practical clinical decision support system with warning functions.

1. Introduction

The aging of society has become a crucial concern in many countries. It leads not only to care pressure in small families, but also increased in expenses for elderly welfare and medical care. A major aging-related health problem hip-fracture, which is common cause of disability and mortality in older people [1]. According to a survey, the number of hip fractures will increase to 6.26 million worldwide by 2050 [2]. Older patients with hip fracture impose a high care burden on their families and health-care services, which can result in numerous other negative consequences in these patients’ lives.
Higher mortality rates, decreased functional status, and reduced self-care ability in order people following hip fracture are well-discussed and well-known factors that have been discussed in relevant studies [3,4,5,6]. Hip fractures in older people are always severe problems in public health, and many studies have explored this topic and analyzed the factors contributing to hip fracture [7,8,9,10].
The trend of mortality in older patients with hip fracture is likely to change with improvements in care systems and surgical techniques [11,12]. According to research [13], the long-term trend of first-year mortality in older patients with hip fracture has declined by 8.8% for women and 20.0% for men in the United States. In general, the most critical time after a hip fracture is the first year [12]. Three studies have indicated that the long-term trend of first-year mortality in older patients with hip fracture remained unchanged [14,15,16]. To improve the quality of treatment older patients with hip fracture, a thorough understanding of the underlying deterioration factors is necessary before clinical treatment.
The aforementioned factors include comorbidities, patient demographics, and the physician. Making a decision about the type of specific intervention to administer is difficult before the factors related to the mortality of such patients are known. Wang et al. published a 10-year trend analysis of hip fracture mortality. However, to our knowledge, no further survival-related analysis has been conducted for the first-year trend, no population-based study has reported the long-term trend in older patients with hip fracture, and no study has used a population-based approach to examine the association between the characteristics and mortality of older patients with hip fracture [17].
For multivariate survival analysis, semiparametric models have been used in related studies, among which one of the popular approaches is the Cox regression model. Even though the model is the most widely used survival analysis method in medical literature, it still has numerous drawbacks and limitations [18,19]. The Cox proportional hazard method relies on assumptions that can easily be violated in the presence of time-dependent covariates. Moreover, the potential for covariate bias is considerable, and the method lacks individual predictions [20]. Even when fixed covariate values are used in the simpler Cox model to make individual predictions, this is computationally expensive [21]. Furthermore, the Cox proportional hazards method produces poor results with many input variables [22]. Multicollinear, continuous, and dichotomized data entered together in the model may produce unstable results [23]. Finally, this model may not be capable of modeling interaction terms when data are dispersed in multidimensional space with nonlinear interactions [22,23].
With recent developments in data science technology, data classification methods have been used in many important research fields. Such methods have also been used to represent and develop some useful tools to support clinical diagnostic decisions and guidelines in medical care. Machine learning is a commonly used data-mining method, and it has been applied to analyze critical information hidden in medical databases. Many different machine-learning methods exist, and they are used for building predictive models for disease prognosis.
In this study, we conducted a population-based study in Taiwan using the National Health Insurance Research Database (NHIRD) to observe the changes in the trends of the older patients with hip fractures. We also identified risk factors for mortality in patients with hip fracture among an older population and examined the association between patient characteristics and risk factors for patients in Taiwan. Moreover, we developed prognostic risk models for patients with hip fracture through data mining and the traditional Cox proportional hazards model, and we compared the performance results. The models were developed and validated using a large population of older patients with hip fracture retrieved from the NHIRD.

2. Subjects and Methods

2.1. Database

This retrospective cohort study was performed from the National Health Insurance Research Database (NHIRD) in Taiwan. NHIRD is the one of the largest nationwide population databases in the world and provides health insurance with a coverage rate from 99% to 99.5% for the residents of Taiwan [24,25]. The dataset named Longitudinal Health Insurance Database 2000 (LHID2000) of the NHIRD was used for this study (registered number NHIRD 104-071). LHID2000 is a longitudinal cohort dataset and also contains completed all the original claim data of health insurance of one million individuals randomly sampled from the year 2000 registry of beneficiaries of the NHIRD and is created by the Taiwan National Health Research Institutes [25]. This study was exempt and approved from full review by the Institutional Review Board of the Kaohsiung Veterans General Hospital Institutional Review Board (VGHKS14-CT7-09).

2.2. Study Population

We reviewed related literature to know potential variables affecting hip fracture deterioration, then identified all possible variable collected from the NHIRD. We selected the period of this study was from 1 January 2000 to 31 December 2010 to get the samples by following procedure and diagnosis codes which are the International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM). We selected the elderly subjects with an age equal or greater than 60-years-old in each year from 2001 to 2010. In total, there were 166,274 subjects enrolled in this study. From these subjects, hip fracture group were selected (ICD-9-CM codes: 820.xx) from the data of inpatient expenditures by admissions. Operative treatment for hip fracture was defined by the following ICD-9-CM procedure codes: 78.55, 79.05, 79.15, 79.25, 79.35, 81.51, and 81.52. The cases of hip fracture in 2000 were as baseline. In this study, the definition of hip fracture was defined as the new hip fracture from 2001 to 2010; the definition of death was defined as the duration from the admission date for hip fracture to the date of death. In addition, the patients who left without being permission, or were transferred to another hospital by physician asked were excluded from the analysis. The records with unknown disposition and miscellaneous missing data, such as unknown time of registration or disposition, unrecorded prognosis of disease, missing data of birth, and ambiguous and unreasonable data were also dropped out. A total of 1166 hip fracture patients were excluded from our study (as Figure 1). The main outcome of interest was if these patients were dead within five years. Time of dead was determined from the time of dead time of mark from hospital. Further, index of outcome was classified to with dead within five years’ group, otherwise to alive after five years’ group.

2.3. Covariates

Potential variables affecting these patients dead within 5 years were comorbidities which also collected from NHIRD. Comorbidities consisted of the following ICD-9-CM codes: (1) diabetes mellitus (ICD-9-CM codes: 250.xx); (2) cardiovascular diseases, which include hypertensive disease (secondary hypertension were excluded) (ICD-9-CM codes: 401.xx-404.xx), ischemic heart disease (ICD-9-CM codes: 410.xx-414.xx), cardiomyopathy (ICD-9-CM codes: 425.xx) and heart failure (ICD-9-CM codes: 428.xx); (3) cerebrovascular disease, which include subarachnoid hemorrhage (ICD-9-CM codes: 430.xx), intracerebral and other intracranial hemorrhage (ICD-9-CM codes: 431.xx-432.xx), occlusion and stenosis of precerebral arteries (ICD-9-CM codes: 433.xx) and other and ill-defined cerebrovascular disease (ICD-9-CM codes: 437.xx); (4) renal failure unspecified, which range from acute renal failure to unspecified renal failure (ICD-9-CM codes: 584.xx-586.xx).

2.4. Statistical Analysis

First of all, the incidence, first-year mortality rate and first-year standardized mortality ratio (SMR) were calculated by each year. The incidence of new hip fracture was calculated as the number of patients with new hip fracture divided by the live subjects in the same year and stratified them by gender. The first-year mortality rate of new hip fracture was calculated as the number of deaths divided by patients with new hip fracture. The first-year SMR was the ratio of observed deaths in patients with new hip fractures to expected deaths in the study population.
An independent t-test was tested between the means in genders. Chi-square test was used to test the association between fracture numbers, death, first-year mortality rate and genders. We performed the linear regression for the long-term trend for incidence, first-year mortality rate and first-year SMR in elderly hip-fracture. Regression analysis was performed with logistic regression. Significance was defined if p-value < 0.05.
Next, we used first univariate Cox regression analysis, controlling that the proportionality of hazards was not violated, to test the different variables for their ability to predict mortality at 5 years from hip fracture patient dataset. On the other hands, the hazard ratios (HR) can be interpreted as the relative risk of dying due to hip fracture, are presented with their 95% confidence intervals as Cox regression analysis. Then, the mortality probability for any case at 5 years from diagnosis was calculated [23]. Cox analysis was performed by the SPSS ver. 22.0 software.
Moreover, the following rules of the classifies with data mining classifiers as below: hip fracture patients alive ≥5 years from hip fracture patients who died within 5 years from disease dataset of whole NHIRD were classified as “dead” while patients alive but with a disease duration <5 years were excluded from the analysis. Thus, we incorporated the following three steps in building our data mining based model: (a) attribute selection based on the literature; (b) comparative evaluation of different classifiers or learning algorithms to select one classifier is the best classifier in this study; and (c) decision tree were generated through data mining tool and identified variables respectively for helping the prognosis of hip fracture patients. The classifiers are re-learned from the resampled datasets to further improve the performance on the hip fracture survival task.

2.5. Data Mining Learning Algorithms

The following learning algorithms/data mining classifiers were used for analysis: we applied several well-known single classification techniques, including decision tree (DT), support vector machines (SVM), and multilayer perceptron (MLP). We used their implementations in the Weka 3.7.3 open-source data mining software (www.cs.waikato.ac.nz/ml/weka) for all the analyses.
For the DT, the most commonly used DT-based learning techniques areC4.5 and random forest (RF). DT is a model utilizing classification technology and induction methods to generate a tree-like decision structure which can be learned by the inductive method of the known examples of each class. Moreover, nodes of DT model consist of branches and leaves, the decision node indicates the test to be performed. To classify the input data, each node of the DT is a predicate and each predicate can determine whether the variable is greater than or equal to or less than types. When analyzing the data, if the selected data variable belongs to the categorical data, it is called a classification tree. If the selected data variable belongs to the continuous pattern, it is called the regression tree. Since it is not affected by linear regression and interactions between independent variables, therefore, especially in filed with complex data and not easy to make decision easier, DT is a very useful data mining model for the decision maker.
The classification of the data using the decision tree technique is a two-step process. The first is the learning process, training data is analyzed by DT algorithm in order to create a model to show as classification rules or a decision tree. The next step is used to determine the accuracy of the classification rules or decision tree. If the accuracy is acceptable, rules can be reused to classify new data in the same scenario of the practical field [26]. It is well known the DT structure is very similar to the clinical decision-making process of the doctor. After the DT has been modelized, it can provide a good way to explain the problem that we want to find it. Therefore, the DT has been preferred in this study.
The aim of SVM is a controlled classification algorithm based on fictional based statistical learning techniques. It is devise a computationally efficient way of learning separating hyperplanes in a high dimensional feature space based on statistical learning theory. There are two cases for SVM: linear SVM and non-linear SVM. The working principle of the SVM is based on the principle of predicting the most appropriate decision function that separates the two classes in the most appropriate way, so as to achieve the best classification effect [27]. It is major functional based learning techniques to generate the classification problems encountered during data mining.
In MLP, neurons are organized in layers, which is a mathematical and integrated based model that imitates the functionality of biological neural systems. MLP consists of an input and an output layer with one or more hidden layers. Neurons between two adjacent layers are fully connected, and each of them receives inputs and converts them into a higher level of combinations through both the combination and the transfer functions [28].

2.6. Measures for Performance Evaluation

To build an alive prediction model for hip fraction patients, this study adopts Weka to investigate the performance of the classification techniques, including J48 (C4.5 in Weka), random forest (RF in Weka), SVM (SMO in Weka), multilayer perceptron (MLP in Weka). The performance of the classifiers was evaluated by means of sensitivity, specificity, accuracy and area under ROC curve (AUC). The evaluation of the performance of AUC that we are using, relies on the rules provided by Hosmer and Lemeshow [29], and these are as follows: “excellent” if AUC ≧ 0.9; “good” if 0.9 > AUC ≧ 0.8; “fair” if 0.8 > AUC ≧ 0.7; “poor” if 0.7 > AUC ≧ 0.6; and “very poor” if AUC < 0.6. To label the prediction form each classifier, among all the possible thresholds T that constitute the coordinates of the ROC curves, the value that resulted in the highest classification accuracy was then chosen [30]. Moreover, avoid the data class imbalance problem deteriorates the performance of classification techniques, resample module [31]; thus a resample module in Weka is adopted to modify the proportions of two classes to be almost identical. In addition, ten-fold cross-validation is then applied in all the experimental evaluations for each generated dataset. The specific parameter values setting selected for each classification technique of this study in Weka Software is list in Table 1.

3. Empirical Results

3.1. Patient Selection

After exclude younger than 60 years old samples from the database population (n = 990,801), there were 166,274 persons older than 60 years old as our study samples during the period of this study where 85,593 (51.4%) persons were male and 80,681 (48.6%) persons were female. In, the mean age was 71.5 years old and it had no significant difference between both genders in this study (as Table 2).
The incidence of new hip fractures was 4.0% from 2001 to 2010, where female incidence was significantly higher than male is plotted in Figure 2 (5.0% vs. 3.1%, p < 0.001). The situation could also be observed in both the male and female subgroups. In the male subgroup, the incidence was 435/100,000 in 2001 and increased to 486/100,000 in 2010. In the female subgroup, the incidence was increased from 683/100,000 to 685/100,000 from 2001 to 2010. Among them, Figure 3 illustrated the first-year mortality rate in elderly hip fracture patients was 21.46% (1166/6679), on the contrary, the first-year mortality rate in male (29.32%) was significantly higher than female (15.91%).
As shown in Table 3, the first-year mortality rate of the overall study group was 21.5% in 2001 and 15.0% in 2010. There was a significant decreasing tendency from 2001 to 2010 (p for trend < 0.001). In the male subgroup, the first-year mortality rate was 29.3% in 2001 and decreased to 17.3% in 2010. There was also a significant decreasing tendency from 2001 to 2010 (p for trend < 0.001). In the female group, the first-year mortality rate was from 15.9% in 2001 and decreased to 13.5% in 2010. The tendency towards a decrease from 2001 to 2010 was significant (p for trend = 0.01). The long term trend of first-year mortality in elderly hip fracture declined by 15.1% for women and by 41.0% for men, respectively.

3.2. Model Construction and Evaluation

Table 4 shows the results of a logistic regression on mortality related to age, gender, comorbidity number, year of fracture and surgical or conservative treatment. In this study group, mortality significantly increased with age (p < 0.001) and male gender (p < 0.001). Fractures treated surgically versus those treated conservatively (p < 0.001) and fractures occurring later in the year (p < 0.001) had significantly lower first-year mortality.
Table 5 shows the results of the logistic regression analysis and Cox PH regression analysis. Logistic regression revealed that hip fractures patients’ gender, age and they have comorbidity (including diabetes mellitus (DM), cardiovascular (CV), and renal disease) increase in the risk of mortality (odds ratio = 0.678, 1.339, 0.159). Meanwhile, the patients with adopting surgical intervention were less likely to mortality compared to patients who did not (OR = 1.763).
Cox PH regression results, similar to the logistic regression analysis results, also showed the patients with DM, CV, and renal comorbidity would increase mortality compared to patients without these comorbidities. Gender, age of patients, and surgical intervention were also consistent with the results of the logistic regression analysis.
Table 6 presents the predicative performance of the four models developed by data mining algorithms in this study. Across all predictive indices, random forest had the highest predictive performance. According to comparisons of predicting performance among all selected classifiers, we can conclude that the decision tree based classifiers (C4.5, RF) out-perform the functional based classifiers (SVM, MLP), showing that decision tree-based approaches can make a better prediction on the mortality of the patients with hip fracture. In addition to the comparison of the prediction performance from different techniques, we further plotted the decision tree which generated through data mining analysis is shown in Figure 4. The decision tree methods identified variables respectively by its importance degree in the order of comorbidity on renal, patient’s age, surgical intervention, gender, comorbidity on cardiovascular disease, and comorbidity on diabetes mellitus.

4. Discussion

This study examined the 10-year trend in hip-fracture incidence and first-year mortality among older patients. It is the first study to report the long-term trend of first-year mortality in older patients with hip fracture in Taiwan. From 2001 to 2010, hip-fracture incidence did not significantly increase, but first-year mortality decreased significantly.

4.1. Incidence of Hip Fracture

In the 1990s, Gullberg et al. estimated the number of hip fractures would increase from 2.6 million in 2025 to 4.5 million in 2050 [32]. In a nationwide study in Romania, the annual incidence of hip fracture increased from 184 to 214 per 100,000 persons between 2005 and 2009 [33]. However, in some cross-sectional observational studies, the incidence of hip fracture has decreased in recent years [13,34]. The present study revealed that the annual incidence of new hip fractures among patients aged older than 60 years increased from 552 to 587 per 100,000 persons from 2001 to 2010. Moreover, the incidence increased from 435 to 486 per 100,000 persons in the male subgroup and from 683 to 689 per 100,000 persons in the female subgroup from 2001 to 2010. The increasing trend may at least be partially explained by population aging during the study period. Other studies have indicated that as populations age, the rate of hip fractures increases without additional interventions [35,36,37]. In Taiwan, hip-fracture prevention measures have been implemented. Since 2003, Taiwan’s National Health Insurance (NHI) program has funded a drug to prevent osteoporosis; furthermore, since 2005, the Health Promotion Administration has cooperated with the Taiwanese Osteoporosis Association to promote osteoporosis prevention through measures such as publishing clinical guidelines and educating medical personnel. These measures may have prevented the rate of hip fracture among older adults from increasing significantly.
Similar to this study, many authors have reported a higher incidence of hip fracture in women than in men across various countries and regions [13,38,39,40,41,42]. Brauer et al. reported annual incidence rates in the United States in 2005 of 793.5 per 100,000 persons in female and 369 per 100,000 persons in male patients [13]. In our series, the incidence rates in 2005 were 740 per 100,000 persons in the female subgroup and 462 per 100,000 persons in the male subgroup. Comparing our 2005 data with those from the United States, the incidence rates of hip fracture in female patients are similar, but that in male patients is higher. In a systematic review of hip fracture, Kanis et al. reported that the rate for men and for men and women combined in Taiwan were higher than those in the United States, and our findings were similar [38].

4.2. First-Year Mortality

In the United States, the long-term trend of first-year mortality in older patients with hip fracture declined by 8.8% for women and 20.0% for men [13]. However, the mortality of hip fracture was unchanged after 1995 [13]. Wang et al. reported a similar finding for a standardized mortality rate in Taiwan using NHI data from 1999 to 2009 [17]. In the present study, the first-year mortality rate after hip fracture was 21.5% in 2001, and it declined to 15.0% in 2010. Though there was a minor fluctuation between 2001 and 2010, the trend of decreasing first-year mortality rate after hip fracture was significant. There was a similarly significant finding in the male subgroup, decreasing from 29.3% in 2001 to 17.3% in 2010. Among females, the decrease was less—from 15.9% in 2001 to 13.5% in 2010. Although a minor fluctuation occurred between 2001 and 2010, the decreasing trend in first-year mortality rate after hip fracture was significant. A similar significant finding was observed in the male subgroup, where the rate decreased from 29.3% in 2001 to 17.3% in 2010. Among women, the decrease was less substantial—from 15.9% in 2001 to 13.5% in 2010. The integration of care systems, progress in medical and surgical care and techniques, and improvements in self-care ability and general health status in the older population may explain this phenomenon [11,43].
Moreover, the female subgroup exhibited lower first-year mortality than the male subgroup during each year of the study. Ariza-Vega et al., Endo et al. and Kannegaard et al. reported the similar finding [44,45,46]. Brauer et al. reported a higher 30-, 180-, and 360-day mortality in males than in women [13]. Hasegawa et al. reported a higher 120-day mortality in Japanese males than in women [1]. Mortality after hip fracture was also sex-dependent in Singapore Chinese people and people covered by Singapore’s Medicare program [47,48]. When we adjusted for age and sex, the SMRs in the overall population and men significantly decreased from 2001 to 2010. However, the decreasing trend of SMR in women was nonsignificant. Even when we adjusted for age and sex, the SMR in men was still higher than that in women. The reason for the higher hip-fracture mortality in men is unknown [45,49]. After a hip fracture, male sex is a risk factor for death within the first year in itself [46], which warrants further exploration.

4.3. Risk Factor of First-Year Mortality after Hip Fracture

In this study, for the general population of patients with hip fracture, risk factors for first-year mortality after hip fracture were age, sex, fracture occurring later in the year, and surgical intervention, but the number of comorbidities was not a significant risk factor. Patients with fractures occurring later in the year had lower first-year mortality following hip fracture, which confirmed the decreasing trend of first-year mortality among older patients. In the population of older patients with hip fracture, male sex was a risk factor for first-year mortality, as discussed in the section above.
Age was reported to be a risk factor for loss of bone strength over the femoral neck and hip fracture [50,51,52,53]. However, Richmond et al. reported that the mortality was higher in older patients (64–85 years) than it was in very old patients (85 years and older). Older age and more comorbidities (i.e., low health status) were reported to be common risk factors for in-hospital [6,54], acute [6,34,55,56], and late mortality after hip fracture [1,44,46,56,57,58]. Jou et al. reported that the Charlson comorbidity index score was associated with in-hospital mortality in women aged older than 50 years with hip fracture [59]. Nonoperative treatment was another risk factor for first-year mortality following hip fracture in this study. Parker et al. reported no significant difference in mortality between operative and nonoperative treatment of hip fracture [60]. Jain et al. reported that 30-day mortality was higher in their nonoperative group than in their operative group [61]. Neuman et al. reported a correlation between race and 7- and 30-day mortality following hip fracture [62]. The findings of this study were similar to those of the abovementioned relevant studies. In the present study, data-mining approaches were adopted to identify significant predictors of mortality in patients with hip fracture. This model uses a simple process for building a decision tree, and it can be easily and accurately applied in clinical practice. It could help to identify patients at risk of hip-fracture prognosis at an early stage and to arrange appropriate treatment for them.

5. Conclusions

In this retrospective cohort study using the secondary NHIRD, we found that the first-year mortality following hip fracture in older people exhibited a decreasing trend from 2001 to 2010. Risk factors for first-year mortality following hip fracture among older adults included male sex, older age, nonoperative treatment, and fractures occurring later in the year. We developed a framework to build a prognostic model for predicting 5-year mortality in patients with hip fracture using data-mining algorithms. Furthermore, we demonstrated for the first time that accurate individual predictions can be made. The advantages of our model include its logical simplicity, its biological plausibility, and its capability for generalization and applicability, which would ultimately support its implementation in clinical practice. It may consider to build various models for predicting the prognosis of hip fracture or integrating prediction algorithms into the computerized physician order entry system, thus creating a practical clinical decision support system with warning functions.
Some constraints must be addressed here because they may restrict the implications of the study; these limitations may provide clues for future studies. First, we collected cases from the NHIRD, which is only used for NHI claims. Information regarding the caregiver, family history, lifestyle factors, and environmental factors are not included in the NHIRD, all of which might be associated with the risk. Second, studies using the NHIRD have indicated a lack of clarity in diagnostic classification. Therefore, some diagnostic accuracy and the prognostic situation of the cases could not be ascertained.
Despite these limitations, we used decision-tree algorithms to build a robust tool that is easy to use and has excellent discriminative ability for the prediction of the prognosis of older patients with hip fracture. Prediction ability may be improved if models can include more clinical and lifestyle variables, which is often the least expensive way to improve care quality for patients with hip fracture. Finally, future population-based prospective studies are required to further validate the present findings. Alternatively, an interventional study in which patients are recruited based on the rules of expert systems in a Hospital Information System may be conducted.

Author Contributions

C.-L.L. and Y.-H.Y. writing draft, review & editing, formal analysis, and methodology design; C.-L.L. Funding support English Editing; Y.-H.Y., C.-Y.C., W.-C.H. and P.-L.T. performed the conceptualization, data, writing—original draft preparation, and visualization; J.-H.R. and C.-J.H. contributed the software, investigation and project administration for this study. All authors have read and agreed to the published version of the manuscript.

Funding

The work was supported by the Ministry of Science and Technology (MOST) of Taiwan under grant MOST 109-2637-H-242-001.

Conflicts of Interest

The authors declare no conflict of interest.

References

  1. Hasegawa, Y.; Suzuki, S.; Wingstrand, H. Risk of mortality following hip fracture in Japan. J. Orthop. Sci. 2007, 12, 113–117. [Google Scholar] [CrossRef] [PubMed]
  2. Cooper, C.; Campion, G.; Melton, L., III. Hip fractures in the elderly: A world-wide projection. Osteoporos. Int. 1992, 2, 285–289. [Google Scholar] [CrossRef] [PubMed]
  3. Man, L.P.; Ho, A.W.; Wong, S.H. Excess mortality for operated geriatric hip fracture in Hong Kong. Hong Kong Med. J. 2016, 22, 6–10. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  4. Boufous, S.; Finch, C.F.; Lord, S.R. Incidence of hip fracture in New South Wales: Are our efforts having an effect? Med. J. Aust. 2004, 180, 623–626. [Google Scholar] [CrossRef]
  5. White, S.M.; Griffiths, R. Projected incidence of proximal femoral fracture in England: A report from the NHS Hip Fracture Anaesthesia Network (HIPFAN). Injury 2011, 42, 1230–1233. [Google Scholar] [CrossRef]
  6. Smith, T.; Pelpola, K.; Ball, M.; Ong, A.; Myint, P.K. Pre-operative indicators for mortality following hip fracture surgery: A systematic review and meta-analysis. Age Ageing 2014, 43, 464–471. [Google Scholar] [CrossRef]
  7. Castronuovo, E.; Pezzotti, P.; Franzo, A.; Di Lallo, D.; Guasticchi, G. Early and late mortality in elderly patients after hip fracture: A cohort study using administrative health databases in the Lazio region, Italy. BMC Geriatr. 2011, 11, 37. [Google Scholar] [CrossRef] [Green Version]
  8. Dhanwal, D.K.; Dennison, E.M.; Harvey, N.C.; Cooper, C. Epidemiology of hip fracture: Worldwide geographic variation. Indian J. Orthop. 2011, 45, 15–22. [Google Scholar] [CrossRef]
  9. Hall, S.E.; Williams, J.A.; Senior, J.A.; Goldswain, P.R.T.; Criddle, R.A. Hip fracture outcomes: Quality of life and functional status in older adults living in the community. Aust. N. Z. J. Med. 2000, 30, 327–332. [Google Scholar] [CrossRef]
  10. Randell, A.G.; Nguyen, T.V.; Bhalerao, N.; Silverman, S.L.; Sambrook, P.N.; Eisman, J.A. Deterioration in quality of life following hip fracture: A prospective study. Osteoporos. Int. 2000, 11, 460–466. [Google Scholar] [CrossRef]
  11. Omsland, T.K.; Emaus, N.; Tell, G.S.; Magnus, J.H.; Ahmed, L.A.; Holvik, K.; Vestergaard, P. Mortality following the first hip fracture in Norwegian women and men (1999–2008). A NOREPOS study. Bone 2014, 63, 81–86. [Google Scholar] [CrossRef] [PubMed]
  12. Schnell, S.; Friedman, S.M.; Mendelson, D.A.; Bingham, K.W.; Kates, S.L. The 1-year mortality of patients treated in a hip fracture program for elders. Geriatr Orthop. Surg. Rehabil. 2010, 1, 6–14. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  13. Brauer, C.A.; Coca-Perraillon, M.; Cutler, D.M.; Rosen, A.B. Incidence and mortality of hip fractures in the United States. JAMA 2009, 302, 1573–1579. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  14. Haleem, S.; Lutchman, L.; Mayahi, R.; Grice, J.E.; Parker, M.J. Mortality following hip fracture: Trends and geographical variations over the last 40 years. Injury 2008, 39, 1157–1163. [Google Scholar] [CrossRef]
  15. Giversen, I.M. Time trends of mortality after first hip fractures. Osteoporos. Int. 2007, 18, 721–732. [Google Scholar] [CrossRef]
  16. Roberts, S.E.; Goldacre, M.J. Time trends and demography of mortality after fractured neck of femur in an English population, 1968–1998: Database study. BMJ 2003, 327, 771–775. [Google Scholar] [CrossRef] [Green Version]
  17. Wang, C.B.; Lin, C.F.; Liang, W.M.; Cheng, C.F.; Chang, Y.J.; Wu, H.C.; Leu, T.H. Excess mortality after hip fracture among the elderly in Taiwan: A nationwide population-based cohort study. Bone 2013, 56, 147–153. [Google Scholar] [CrossRef] [Green Version]
  18. Fisher, L.D.; Lin, D.Y. Time-dependent covariates in the Cox proportional hazards regression model. Annu. Rev. Public Health 1999, 20, 145–157. [Google Scholar] [CrossRef] [Green Version]
  19. Ohno, M.L. A comparison of Cox proportional hazards and artificial neural network models for medical prognosis. Comput. Biol. Med. 1997, 27, 55–65. [Google Scholar] [CrossRef]
  20. Bakker, B.; Heskes, T.; Neijt, J.; Kappen, B. Improving Cox survival analysis with a neural-Bayesian approach. Stat. Med. 2004, 23, 2989–3012. [Google Scholar] [CrossRef]
  21. Bakker, S.J. Crippling of inflammatory markers as predictors of death by dichotomization and multicollinearity. Nephrol. Dial. Transplant. 2006, 21, 2990–2991. [Google Scholar] [CrossRef] [PubMed]
  22. Fayyad, U.; Piatetsky, S.G.; Smyth, P. From Data Mining to Knowledge Discovery in Databases (a survey). AI Mag. 1996, 17, 37–54. [Google Scholar]
  23. Beretta, L.; Santaniello, A.; Cappiello, F.; Chawla, N.V.; Vonk, M.C.; Carreira, P.E.; Allanore, Y.; Popa-Diaconu, D.A.; Cossu, M.; Bertolotti, F.; et al. Development of a five-year mortality model in systemic sclerosis patients by different analytical approaches. Clin. Exp. Rheumatol. 2010, 28 (Suppl. 58), 18–27. [Google Scholar]
  24. Hsing, A.W.; Ioannidis, J.P. Nationwide Population Science: Lessons From the Taiwan National Health Insurance Research Database. JAMA Intern. Med. 2015, 175, 1527–1529. [Google Scholar] [CrossRef] [PubMed]
  25. The National Health Research Institutes. Data Subsets. Available online: Nhird.nhri.org.tw/en/Data_Subsets.html#S3 (accessed on 1 August 2020).
  26. Quinlan, J.R. C4. 5: Programs for Machine Learning; Morgan Kaufmann: San Francisco, CA, USA, 1993. [Google Scholar]
  27. Vapnik, V. The Nature of Statistical Learning Theory; Springer: New York, NY, USA, 1995. [Google Scholar]
  28. Rumelhart, D.E.; Hinton, G.E.; Williams, R.J. Learning representations by backpropagating errors. Nature 1986, 323, 533–536. [Google Scholar] [CrossRef]
  29. Hosmer, D.W.; Lemeshow, S. Applied Logistic Regression; John Wiley & Sons: New York, NY, USA, 1989. [Google Scholar]
  30. Vuk, M.; Curk, T. ROC curve, Lift Chart and Calibration Plot. Metodološki Zvezki 2006, 3, 89–108. [Google Scholar]
  31. Tan, P.N.; Steinbach, M.; Kumar, V. Introduction to Data Mining; Addison Wesley: Boston, MA, USA, 2006; p. 769. [Google Scholar]
  32. Gullberg, B.; Johnell, O.; Kanis, J.A. World-wide projections for hip fracture. Osteoporos. Int. 1997, 7, 407–413. [Google Scholar] [CrossRef]
  33. Grigorie, D.; Sucaliuc, A.; Vladescu, C. Hip fracture epidemiology in Romania: Nationwide study 2005–2009. Osteoporos. Int. 2011, 22, S323. [Google Scholar]
  34. Williams, N.; Hardy, B.M.; Tarrant, S.; Enninghorst, N.; Attia, J.; Oldmeadow, C.; Balogh, Z.J. Changes in hip fracture incidence, mortality and length of stay over the last decade in an Australian major trauma centre. Arch. Osteoporos. 2013, 8, 150. [Google Scholar] [CrossRef]
  35. Burge, R.; Dawson-Hughes, B.; Solomon, D.H.; Wong, J.B.; King, A.; Tosteson, A. Incidence and economic burden of osteoporosis-related fractures in the United States, 2005–2025. J. Bone Miner. Res. 2007, 22, 465–475. [Google Scholar] [CrossRef]
  36. Lönnroos, E.; Kautiainen, H.; Karppi, P.; Huusko, T.; Hartikainen, S.; Kiviranta, I.; Sulkava, R. Increased incidence of hip fractures: A population based-study in Finland. Bone 2006, 39, 623–627. [Google Scholar] [CrossRef] [PubMed]
  37. Melton, L.J., 3rd; Kearns, A.E.; Atkinson, E.J.; Bolander, M.E.; Achenbach, S.J.; Huddleston, J.M.; Leibson, C.L. Secular trends in hip fracture incidence and recurrence. Osteoporos. Int. 2009, 20, 687–694. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  38. Kanis, J.A.; Odén, A.; McCloskey, E.V.; Johansson, H.; Wahl, D.A.; Cooper, C. A systematic review of hip fracture incidence and probability of fracture worldwide. Osteoporos. Int. 2012, 2, 2239–2256. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  39. Black, D.M.; Cooper, C. Epidemiology of fractures and assessment of fracture risk. Clin. Lab. Med. 2000, 20, 439–453. [Google Scholar] [CrossRef]
  40. Grigorie, D.; Sucaliuc, A.; Johansson, H.; Kanis, J.A.; McCloskey, E. Incidence of hip fracture in Romania and the development of a Romanian FRAX model. Calcif. Tissue Int. 2013, 92, 429–436. [Google Scholar] [CrossRef]
  41. Formiga, F.; Lopez-Soto, A.; Duaso, E.; Ruiz, D.; Chivite, D.; Perez-Castejon, J.M.; Pujol, R. Differences in the characteristics of elderly patients suffering from hip fracture due to falls according to place of residence. J. Am. Med. Dir. Assoc. 2007, 8, 533–537. [Google Scholar] [CrossRef]
  42. Jaller-Raad, J.J.; Jaller-Char, J.J.; Lechuga-Ortiz, J.A.; Navarro-Lechuga, E.; Johansson, H.; Kanis, J.A. Incidence of hip fracture in Barranquilla, Colombia, and the development of a Colombian FRAX model. Calcif. Tissue Int. 2013, 93, 15–22. [Google Scholar] [CrossRef]
  43. Doshi, H.K.; Ramason, R.; Azellarasi, J.; Naidu, G.; Chan, W.L.W. Orthogeriatric model for hip fracture patients in Singapore: Our early experience and initial outcomes. Arch. Orthop. Trauma Surg. 2014, 134, 351–357. [Google Scholar] [CrossRef]
  44. Ariza-Vega, P.; Kristensen, M.T.; Martín-Martín, L.; Jiménez-Moleón, J.J. Predictors of long-term mortality in older people with hip fracture. Arch. Phys. Med. Rehabil. 2015, 96, 1215–1221. [Google Scholar] [CrossRef]
  45. Endo, Y.; Aharonoff, G.B.; Zuckerman, J.D.; Egol, K.A.; Koval, K.J. Gender differences in patients with hip fracture: A greater risk of morbidity and mortality in men. J. Orthop. Trauma 2005, 19, 29–35. [Google Scholar] [CrossRef]
  46. Kannegaard, P.N.; van der Mark, S.; Eiken, P.; Abrahamsen, B.O. Excess mortality in men compared with women following a hip fracture. National analysis of comedications, comorbidity and survival. Age Ageing 2010, 39, 203–209. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  47. Koh, G.C.; Tai, B.C.; Ang, L.W.; Heng, D.; Yuan, J.M.; Koh, W.P. All-cause and cause-specific mortality after hip fracture among Chinese women and men: The Singapore Chinese Health Study. Osteoporos. Int. 2013, 24, 1981–1989. [Google Scholar] [CrossRef] [PubMed]
  48. Jacobsen, S.J.; Goldberg, J.; Miles, T.P.; Brody, J.A.; Stiers, W.; Rimm, A.A. Race and sex differences in mortality following fracture of the hip. Am. J. Public Health 1992, 82, 1147–1150. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  49. Alegre-López, J.; Cordero-Guevara, J.; Alonso-Valdivielso, J.L.; Fernández-Melón, J. Factors associated with mortality and functional disability after hip fracture: An inception cohort study. Osteoporos. Int. 2005, 16, 729–736. [Google Scholar] [CrossRef] [PubMed]
  50. Keaveny, T.M.; Kopperdahl, D.L.; Melton, L.J., 3rd. Age-dependence of femoral strength in white women and men. J. Bone Miner. Res. 2010, 25, 994–1001. [Google Scholar] [CrossRef] [PubMed]
  51. Hedlund, R.; Lindgren, U. Trauma type, age, and gender as determinants of hip fracture. J. Orthop. Res. 1987, 5, 242–246. [Google Scholar]
  52. Lin, W.P.; Wen, C.J.; Jiang, C.C.; Hou, S.M.; Chen, C.Y.; Lin, J. Risk factors for hip fracture sites and mortality in older adults. J. Trauma Acute Care Surg. 2011, 71, 191–197. [Google Scholar] [CrossRef]
  53. Gerdhem, P.; Ringsberg, K.; Akesson, K.; Obrant, K.J. Just one look, and fractures and death can be predicted in elderly ambulatory women. Gerontology 2004, 50, 309–314. [Google Scholar] [CrossRef]
  54. Valizadeh, M.; Mazloomzadeh, S.; Golmohammadi, S.; Larijani, B. Mortality after low trauma hip fracture: A prospective cohort study. BMC Musculoskelet. Disord. 2012, 13, 143. [Google Scholar] [CrossRef] [Green Version]
  55. Pugely, A.J.; Martin, C.T.; Gao, Y.; Klocke, N.F.; Callaghan, J.J.; Marsh, J.L. A risk calculator for short-term morbidity and mortality after hip fracture surgery. J. Orthop. Trauma 2014, 28, 63–69. [Google Scholar] [CrossRef]
  56. Lin, J.C.; Wu, C.C.; Lo, C.; Liang, W.M.; Cheng, C.F.; Wang, C.B.; Leu, T.H. Mortality and complications of hip fracture in young adults: A nationwide population-based cohort study. BMC Musculoskelet. Disord. 2014, 15, 362. [Google Scholar] [CrossRef] [Green Version]
  57. Ireland, A.W.; Kelly, P.J.; Cumming, R.G. Risk factor profiles for early and delayed mortality after hip fracture: Analyses of linked Australian Department of Veterans’ Affairs databases. Injury 2015, 46, 1028–1035. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  58. Poor, G.; Atkinson, E.J.; O’fallon, W.M.; Melton, L.J., 3rd. Determinants of reduced survival following hip fractures in men. Clin. Orthop. Relat. Res. 1995, 260–265. [Google Scholar]
  59. Jou, H.J.; Siao, R.Y.; Tsai, Y.S.; Chen, Y.T.; Li, C.Y.; Chen, C.C. Postdischarge rehospitalization and in-hospital mortality among Taiwanese women with hip fracture. Taiwan J. Obstet. Gynecol. 2014, 53, 43–47. [Google Scholar] [CrossRef] [PubMed] [Green Version]
  60. Parker, M.J.; Handoll, H.H.; Bhargara, A. Conservative versus operative treatment for hip fractures in adults. Cochrane Database Syst. Rev. 2000, Cd000337. [Google Scholar] [CrossRef]
  61. Jain, R.; Basinski, A.; Kreder, H.J. Nonoperative treatment of hip fractures. Int. Orthop. 2003, 27, 11–17. [Google Scholar] [CrossRef] [Green Version]
  62. Neuman, M.D.; Fleisher, L.A.; Even-Shoshan, O.; Mi, L.; Silber, J.H. Nonoperative care for hip fracture in the elderly: The influence of race, income, and comorbidities. Med. Care 2010, 48, 314–320. [Google Scholar] [CrossRef]
Figure 1. Diagram of study population.
Figure 1. Diagram of study population.
Applsci 10 06787 g001
Figure 2. Incidence (per 100,000 persons per year) of hip fracture in across years.
Figure 2. Incidence (per 100,000 persons per year) of hip fracture in across years.
Applsci 10 06787 g002
Figure 3. First year mortality of elder hip fracture patient across years.
Figure 3. First year mortality of elder hip fracture patient across years.
Applsci 10 06787 g003
Figure 4. The decision tree for predicting a mortality risk of hip fracture patients.
Figure 4. The decision tree for predicting a mortality risk of hip fracture patients.
Applsci 10 06787 g004
Table 1. Parameter setting in Weka.
Table 1. Parameter setting in Weka.
MethodParametersValue Setting
C4.5Confidence factor0.25
Minimum number of instances per leaf20
Random ForestNumber of trees10
Number of attributes to be used in random selection4
Support Vector MachinesKernelPolyKernel
Multilayer PerceptronNumber of hidden nodes7
Learning rate0.3
Momentum factor0.2
Maximum number of epochs500
Table 2. Demographics data of random sample from National Health Insurance Research Database.
Table 2. Demographics data of random sample from National Health Insurance Research Database.
No.%
Gendermale50908151.4
female48172048.6
total990801100
meanstd. err95% C.I.p value
Agemale34.200.0334.1434.250.5093
female34.170.0334.1134.23
total34.180.0234.1434.22 ioii
Table 3. First year mortablity following hip fracture from 2000 to 2010.
Table 3. First year mortablity following hip fracture from 2000 to 2010.
Year20002001200220032004200520062007200820092010TotalTrend p (* t-Test **)
All hip fractures7596016136746386836036927337037397438
survival6654724915475155615045716196056286178
death9412912212712312299121114981111260
First year mortality rate12.38%21.46%19.90%18.84%19.28%17.86%16.42%17.49%15.55%13.94%15.02%16.94%0.042
Male hip fractures2192492522742762702402602852713012678
survival1701761922062102101912072292222492092
death4973606866604953564952586
First year mortality rate22.37%29.32%23.81%24.82%23.91%22.22%20.42%20.83%19.65%18.08%17.28%21.88%0.000
Female hip fractures5403523614003624133634324484324384541
survival4952962993413053513133643903833793916
death4556625957625068584959580
First year mortality rate8.33%15.91%17.71%14.75%15.75%15.01%13.77%15.74%12.95%11.34%13.47%%12.77%0.769
* Chi-square test for trend. ** t-test for first year mortality rate between male and female group.
Table 4. Results of logistic regression of first-year mortality odds.
Table 4. Results of logistic regression of first-year mortality odds.
RatioStd. Err.tp Value[95% Conf.Interval]
Entire study group
age0.005 0.000 19.460.000 0.004 0.005
gender (male vs. female)0.075 0.008 9.610.000 0.060 0.090
number of comorbidity0.004 0.006 0.690.488 −0.008 0.016
the year of fracture −0.006 0.001 −4.560.000 −0.008 −0.003
surgical intervention (yes vs. no)−1.554 0.122 −12.740.000 −0.179 −0.132
Male group
age0.005 0.000 15.750.000 0.005 0.006
number of comorbidity0.007 0.010 0.680.497 −0.013 −0.027
the year of fracture −0.009 0.002 −4.590.000 −0.125 −0.005
surgical intervention−0.141 −0.019 −7.340.000 −0.178 −0.103
Female group
age0.004 0.000 10.660.000 0.003 0.005
number of comorbidity0.002 0.007 0.230.822 −0.012 0.016
The year of fracture −0.003 0.002 −1.820.069 −0.006 0.000
surgical intervention−0.170 0.016 −10.830.000 −0.201 −0.139
Table 5. Summary of mortality perdition models.
Table 5. Summary of mortality perdition models.
Logistic Regression Analysis Cox PH Regression Analysis
FeaturesOR95% CIpFeaturesHR95% CIp
Gender0.590.536–0.649 Gender1.4551.375–1.539
Age0.9320.926–0.937 Age1.0471.044–1.051
Surgical intervention1.7631.523–2.040 Surgical intervention0.6820.631–0.736
Comorbidity Comorbidity
DM0.6780.603–0.762 DM1.2371.154–1.327
CV1.3391.214–1.476 CV0.8760.825–0.930
CVA1.0470.487–2.2520.905CVA0.9850.640–1.5140.944
RENAL0.1590.071–0.355 RENAL2.9112.267–3.738
DM: diabetes mellitus; CV: cardiovascular; CVA: cerebrovascular accident; RENAL: renal related disease.
Table 6. The predictive validity of the hip-fracture mortality model.
Table 6. The predictive validity of the hip-fracture mortality model.
ModelSensitivitySpecificityAccuracyAUC
C4.50.6740.6720.6740.723
Random Forest0.7250.7310.7240.790
SVM0.6370.6360.6360.637
MLP0.6260.6390.6200.674

Share and Cite

MDPI and ACS Style

Lo, C.-L.; Yang, Y.-H.; Hsu, C.-J.; Chen, C.-Y.; Huang, W.-C.; Tang, P.-L.; Renn, J.-H. Development of a Mortality Risk Model in Elderly Hip Fracture Patients by Different Analytical Approaches. Appl. Sci. 2020, 10, 6787. https://doi.org/10.3390/app10196787

AMA Style

Lo C-L, Yang Y-H, Hsu C-J, Chen C-Y, Huang W-C, Tang P-L, Renn J-H. Development of a Mortality Risk Model in Elderly Hip Fracture Patients by Different Analytical Approaches. Applied Sciences. 2020; 10(19):6787. https://doi.org/10.3390/app10196787

Chicago/Turabian Style

Lo, Chia-Lun, Ya-Hui Yang, Chien-Jen Hsu, Chun-Yu Chen, Wei-Chun Huang, Pei-Ling Tang, and Jenn-Huei Renn. 2020. "Development of a Mortality Risk Model in Elderly Hip Fracture Patients by Different Analytical Approaches" Applied Sciences 10, no. 19: 6787. https://doi.org/10.3390/app10196787

APA Style

Lo, C. -L., Yang, Y. -H., Hsu, C. -J., Chen, C. -Y., Huang, W. -C., Tang, P. -L., & Renn, J. -H. (2020). Development of a Mortality Risk Model in Elderly Hip Fracture Patients by Different Analytical Approaches. Applied Sciences, 10(19), 6787. https://doi.org/10.3390/app10196787

Note that from the first issue of 2016, this journal uses article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop