Special Issue on Applied Machine Learning

1. Introduction

Machine learning (ML) is one of the most exciting fields of computing today. Over the past few decades, ML has become an entrenched part of everyday life and has been successfully used to solve practical problems. An application area of ML is very broad, including engineering, industry, business, finance, medicine, and many other domains. ML covers a wide range of learning algorithms, including classical ones such as linear regression, k-nearest neighbors, decision trees, support vector machines and neural networks, and newly developed algorithms such as deep learning and boosted tree models. In practice, it is quite challenging to properly determine an appropriate architecture and parameters of ML models so that the resulting learner model can achieve sound performance for both learning and generalization. Practical applications of ML bring additional challenges, such as dealing with big, missing, distorted, and uncertain data. In addition, interpretability is a paramount quality that ML methods should aim to achieve if they are to be applied in practice. Interpretability allows us to understand ML model operation and raises confidence in its results.

This book compiles 41 papers published in the Special Issue titled “Applied Machine Learning”. The papers focus on applications of ML models in a diverse range of fields and problems. They report substantive results on a wide range of learning methods, discuss conceptualization of problems, data representation, feature engineering, ML models, critical comparisons with existing techniques, and interpretation of results.

2. Summary of the Contributions

There were 116 papers submitted to this special issue, and 41 papers were accepted. Although each paper covers different topics, we can identify six categories where the papers can be classified according to their main focus: computer vision, teaching and learning, social media, forecasting, basic problems of ML, and other topics.

2.1. Computer Vision

Image processing and analysis are a basis of computer vision problems such as semantic segmentation, object classification, localization, and detection, optical character recognition, facial recognition etc. An appropriate representation of image content is a crucial problem. In [1], to deal with this problem, a novel type of representation is proposed where an image is reduced to a set of highly sparse matrices representing detected key-points. The authors express intensity of features extracted from a dedicated convolutional neural network (CNN) autoencoder. The new features have many advantages such as they are not manually designed but learned, they are expected to minimize information loss and they are relatively easy to interpret.

In [2], a fast-self-adaptive digital camouflage method based on deep learning is proposed. It is designed for the new generation of adaptive optical camouflage which can change with the environment in real-time. The system is composed of a YOLOv3 model that identifies military targets, a pre-trained deepfillv1 model that designs the preliminary camouflage texture, and a k-means algorithm for standardization of the texture. The experimental results show that the camouflage pattern designed by the proposed method is consistent with the background in texture and semantics, and has excellent camouflage performance.

A problem of classification of remote sensing images for disaster investigation, traffic control, and land-use resource management is considered in [3]. A new remote sensing scene classification network is proposed and a two-stage cyclical learning is developed to speed up model training and enhance accuracy. A t-distributed stochastic neighbor embedding algorithm was used to verify the effectiveness of the proposed model, and a local interpretable model-agnostic explanation algorithm was applied to improve the results.

In [4], tracking pedestrian workers on construction sites is considered to improve efficiency and safety management. Vision-based tracking approaches, suitable in this case, require a large amount of data originating from construction sites. These data are hardly available, so the authors propose to use a small general dataset and combine a deep learning detector with an approach based on classical ML techniques. They use YOLOv3 detector for identifying workers and compare its performance with an approach based on a soft cascaded classifier. They found that both approaches generally yield satisfying tracking performances but feature different characteristics. To augment a self-recorded real world dataset for learning the vision-based tracking system, in [5] virtual construction site scenarios are modeled using 3D computer graphics software. The detector’s performance is examined when using synthetic data of various environmental conditions for training. The findings showed that a synthetic extension is beneficial for otherwise small datasets. It is an alternative to evaluate vision-based tracking systems on hazardous scenes without exposing workers to risks.

In [6], a problem of environment classification for unmanned aerial vehicles (UAV) is addressed. Images obtained from video and photographic cameras mounted on a UAV are recognized to detect ground, sky, and clouds. The proposed recognition system includes CNN trained with a dataset generated by both, a human expert and a Support Vector Machine (SVM) to capture context and precise localization.

2.2. Teaching and Learning

Student grade prediction is an important educational problem for designing personalized strategies of teaching and learning. To solve this problem, in [7], a graph regularized robust matrix factorization is proposed optimized by a majorization minimization algorithm. This method integrates two side graphs built on the side data of students and courses into the objective of robust low-rank matrix factorization. As a result, the learned features of students and courses can grasp more priors from educational situations to achieve higher grade prediction results. This facilitates personalized teaching and learning in higher education.

For developing adaptive e-learning systems, it is very helpful to provide information on how students recognize, process and store information. To improve students’ learning evaluation, in [8], a method based on deep multi-target prediction algorithm using Felder-Silverman learning styles model is proposed. It uses feature selection, learning styles models, and multiple target classification to investigate the possibility of improving the accuracy of automatic learning styles identification. The obtained results show that learning styles allow adaptive e-learning systems to improve the learning processes of students.

Students’ performance prediction in higher education was considered in [9]. To exploit the knowledge retrieved from one problem for improving the predictive performance of a learning model for a different but related problem, the authors use transfer learning. The experimental results demonstrate that the prognosis of students at risk of failure can be achieved with satisfactory accuracy in most cases, provided that datasets of students who have attended other related courses are available.

Paper [10] was conducted with the aim of identifying the interrelationships among topics based on the understanding of various bodies of knowledge. The study provides a foundation for topic compositions to construct an academic body of knowledge of AI. To this end, ML-based sentence similarity measurement models used in machine translation, chatbots, and document summarization were applied to the body of knowledge of AI. Consequently, several similar topics related to agent designing in AI were identified. The results of this study can be applied in the edutech field.

Predicting the academic standing of a student at the graduation time can be very useful for institutions to select among candidates or in helping potentially weak students in overcoming educational challenges. In [11], this problem is solved using several ML algorithms based on different student data including individual course grades and grade point averages. This approach can be applied to any dataset to determine when to use which college performance representation for enhanced prediction. For predicting the grades of undergraduate students in the final exams, in [12], multi-view learning is applied to exploit the knowledge retrieved from data, represented by multiple feature subsets known as views. A semi-supervised regression algorithm is proposed which exploits three independent and naturally formed feature views, derived from different sources. The experimental results demonstrate that the early prognosis of students at risk of failure can be accurately achieved and could highly benefit the educational domain.

2.3. Social Media

One prominent dark side of online information behavior is the spreading of rumors on social media. Paper [13] analyses the association between user features and rumor refuting behavior in different rumor categories. Natural language processing (NLP) techniques are applied to quantify the user’s sentiment tendency and recent interests. The users’ personalized features are used to train XGBoost classification model to identify potential refuters. The results revealed that there are significant differences between rumor stiflers and refuters, as well as between refuters for different categories.

The objective of [14] is to detect variables that allow organizations to manage their social network services efficiently. This study, applying ML algorithms and multiple linear regression, reveals which aspects of published content increase the recognition of publications through retweets and favorites. The findings of this research provide new knowledge about trends and patterns of use in social media, providing academics and professionals with the necessary guidelines to efficiently manage these technologies in the organizational field.

Paper [15] concerns the tourists’ sentiments regarding travel destinations based on online travel review texts. The authors transformed sentiment analysis into a multi-classification problem based on ML methods, and further designed a keyword semantic expansion method based on a knowledge graph. The method extracts keywords from online travel review texts and obtains the concept list of keywords through the knowledge graph. This list is then added to the review text to facilitate the construction of semantically expanded classification data. The results of sentiment analysis form an important basis for tourism decision making.

Micro-blogs, such as Twitter, have become important tools to share opinions and information among users. The authors of [16] wonder how a user can discover influencers concerned with their interest. They propose a classification model trained on messages labeled with topical classes, so as this model is able to classify unlabeled messages. This model can be used to reveal the hidden topic the messages are talking about.

With the widespread use of over-the-top (OTT) media, such as YouTube and Netflix, network markets are changing and innovating rapidly, making it essential for network providers to quickly and efficiently analyze OTT traffic with respect to pricing plans and infrastructure investments. In [17], a time-aware deep learning method of analyzing OTT traffic to classify users for this purpose is presented. A novel framework to better exploit accuracy, while dramatically reducing classification time is proposed. The resultant approach provides a simple method for customizing pricing plans and load balancing by classifying OTT users more accurately.

Recommendation systems aim to decipher user interests, preferences, and behavioral patterns automatically. The credibility of the recommendation is of magnificent importance in crowdfunding project recommendations. Paper [18] devises a hybrid ML-based approach for credible crowdfunding projects’ recommendations by wisely incorporating backers’ sentiments and other influential features. The proposed model has four modules: a feature extraction module, a hybrid latent Dirichlet allocation and LSTM-based latent topics evaluation module, credibility formulation, and recommendation module. The proposed model’s evaluation depicts that credibility assessment based on the hybrid ML approach contributes more efficient results than existing recommendation models.

2.4. Forecasting

Stock performance prediction is one of the most challenging issues in time series data analysis. Paper [19] proposes to build an automated trading system by integrating AI and the proven method invented by human stock traders. The knowledge and experience of the successful stock traders are extracted from their related publications. After that, an LSTM-based deep NN is developed to use the human stock traders’ knowledge in the automatic trading system. Experimental results indicate that the proposed ranking-based stock classification considering historical volatility strategy outperforms conventional methods.

In [20], the authors study the volatility forecasts in the Bitcoin market, which has become popular in the global market in recent years. For the improvement of the forecasting accuracy of Bitcoin’s volatility, they develop hybrid forecasting models combining the GARCH family models with the ML approach including NNs.

Paper [21] is about forecasting the Key Performance Indicators (KPIs), usually in the form of time series data, related to the COVID-19 pandemic. Making reliable predictions of these indicators, particularly for emergency departments, can facilitate acute unit planning, enhance the quality of care and optimise resources. The authors compare the KPI forecasting models including classical ARIMA, Prophet and General Regression NN.

A development of the intelligent transport systems has created conditions for solving the supply-demand imbalance of public transportation services. In [22], a method to forecast real-time online taxi-hailing demand is introduced. It is based on NNs and extreme gradient boosting. The proposed method can help to schedule online taxi-hailing resources in advance.

Climate change increases the frequency and intensity of heatwaves, causing significant human and material losses every year. Big data, whose volumes are rapidly increasing, are expected to be used for preemptive responses. In [23], for weekly prediction of heat-related damages, a random forest model was developed using statistical, meteorological, and floating population data. The results show that the proposed model outperforms existing ones.

One of the hottest topics in today’s meteorological research is weather nowcasting, which is the weather forecast for a short time period such as one to six hours. With the main goal of helping meteorologists in analyzing radar data for issuing nowcasting warnings, in [24], a regression model based on an ensemble of deep NNs for predicting the values for radar products is proposed. The proposed model is intended to be a proof of concept for the effectiveness of learning from radar data relevant patterns that would be useful for predicting future values for radar products based on their historical values.

2.5. Basic Problems of ML

Paper [25] deals with the problem of instance selection for classifiers. The main goal is to improve the performance of a classifier (its speed and accuracy) by eliminating redundant and noisy samples. The obtained results indicate that for the most of the classifiers compressing the training set affects prediction performance and only a small group of instance selection methods can be recommended as a general purpose preprocessing step. These are learning vector quantization based algorithms, along with the Drop2 and Drop3.

Support vector machines are a well-known classifiers due to their superior classification performance. To decrease the large-scale SVM complexity, in [26], a novel data reduction method for reducing the training time by combining decision trees and relative support distance is proposed. The method selects good support vector candidates in each partition generated by the decision trees. The selected candidates reduced the training time while maintaining good classification performance in comparison with existing approaches

Paper [27] deals with the problem of solving of partial differential equations, which is a hot topic of mathematical research. The authors introduce an improved Physics Informed Neural Network (PINN) for solving partial differential equations. PINN takes the physical information that is contained in partial differential equations as a regularization term, which improves a performance of NNs. The experimental results show that PINN is effective in solving partial differential equations and deserves further research.

Machine learning of automata and grammars has a wide range of applications in such fields as syntactic pattern recognition, computational biology, systems modeling, natural language acquisition, and knowledge discovery. In [28], an approach to non-deterministic finite automaton inductive synthesis that is based on answer set programming (ASP) solvers are proposed. They consist of preparing logical rules before starting the searching process. The authors show how the proposed ASP solvers help to tackle the regular inference problem for large-size instances and compare their approach with the existing ones. Experiments indicated that the proposed approach clearly outperforms the current state-of-the-art satisfiability-based method and all backtracking algorithms proposed in the literature.

Paper [29] sits in the scientific field known as grammatical inference (GI), automata learning, grammar identification, or grammar induction. The matter under consideration is the set of rules that lie behind a given sequence of words and the main task is to discover the rules that can help to evaluate new, unseen words. The authors propose a new grammatical inference method and applied it to a real bioinformatics task, i.e., classification of amyloidogenic sequences. In the experimental evaluation, they showed that the new grammatical inference algorithm gives the best results in comparison to other automata or grammar learning methods as well as ML approach combining an unsupervised data-driven distributed representation and SVM.

Paper [30] is about anticipatory classifier systems, i.e., the classifier systems that learn by using a cognitive mechanism of anticipatory behavioral control which was introduced in cognitive psychology. The authors note that the learning classifier systems revealed many real-world sequential decision problems where the preferred objective is the maximization of the average of successive rewards. To address such problems, they proposes a modification toward a learning component: a new average reward criterion. In the experimental study, they showed that the anticipatory classifier systems with an averaged reward criterion can be used successively in multi-step environments.

2.6. Other Topics

A medical care application of ML is considered in [31]. This work is on a sleep apnea which is a common sleep-related disorder that significantly affects the population. It is characterized by repeated breathing interruption during sleep. The authors propose a new probabilistic algorithm based on oronasal respiration signal for automated detection of apnea events during sleep. Unlike classical threshold-based classification models, they use a Gaussian mixture probability model for detecting sleep apnea based on the posterior probabilities of the respective events. The results show significant improvement in the ability to detect sleep apnea events compared to a rule-based classifier that uses the same classification features and also compared to the previously published studies.

Paper [32] deals with a discrete optimization problem of product placement and of order picking routes in a warehouse. The authors propose a genetic algorithm that minimizes the sum of the order picking times. The product placement is optimized by another genetic algorithm. To improve and accelerate an optimization process, several ideas are proposed such as a multi-parent crossover, caching procedure, multiple restart and order grouping. A proposed solution decreases significantly the total order picking times.

ML techniques have been actively applied to the meteorology and climatology fields in recent years. They are used for forecasting in different horizons, modelling climatic data, quality control and correction of observed weather data. Paper [33] deals with the topic of the climate change. It presents a framework for selecting general circulation models (GCMs) in homogeneous climatic zones and detecting future climate change trends. With the support of ML techniques, long records of climate data, from numerous gauging sites and web sources, were analyzed and used to determine historical and projected trends of climate change. In [34], to detect the weather phenomena such as precipitation and fog from the backscatter data obtained from the lidar ceilometer, three ML models were applied: random forest, SVM, and NN. The prediction results showed the potential for precipitation detection, but fog detection was found to be very difficult.

The emission of carbon dioxide caused by various sectors, including construction and industrial processes, has emerged as a severe problem that dramatically affects global climate change. A portland cement production process accounts for a large part global anthropogenic CO₂ emission. A fly ash-based geopolymer concrete (FAGP) offers a favourable alternative to conventional Portland concrete due to its reduced embodied carbon dioxide content. In [35], ML methods including artificial NN, deep NN and ResNet were employed to predict mechanical properties of FAGP concrete. The obtained results indicate that the proposed approaches offer reliable methods for FAGP design and optimisation.

Paper [36] deals with a vibration test in the space structure testing. During the physical tests, the structure must not be overtested to avoid any risk of damage. In order to solve the issues associated with existing methods of live monitoring of the structure’s response, the authors investigated the use of artificial NNs to predict the system’s responses during the test. The conducted research accounts for a novel method for live prediction of stresses, allowing failure to be evaluated for different types of material via yield criteria.

Software vulnerabilities are one of the main causes of cybersecurity problems, resulting in huge losses. Existing solutions to automated vulnerability detection are mostly based on features that are defined by human experts and directly lead to missed potential vulnerability. Deep learning is proposed in [37] as an effective method for automating the extraction of vulnerability characteristics. Word2vec continuous bag-of-words, multiple structural CNNs, and stacking classifiers were found to be the best combination for automated vulnerability detection by comparing classification results.

Paper [38] is on information privacy which is a critical design feature for any exchange system, with privacy-preserving applications requiring the identification and labelling of sensitive information. The authors propose a predictive context-aware model based on a Bidirectional Long Short Term Memory network with Conditional Random Fields (BiLSTM + CRF) to identify and label sensitive information in conversational data. The results demonstrate that the BiLSTM + CRF model architecture with BERT embeddings and WordShape features is very effective and outperforms competitive solutions.

Natural language processing has enormous areas of applications including sentiment analysis, machine translation, text classification and extraction. In [39], the problem of developing a deep learning-based language model that helps software engineers write code faster is considered. This research proposes a hybrid approach that harnesses the synergy between ML techniques and advanced design methods aiming to develop a code auto-completion framework that helps firmware developers write code in a more efficient manner. The proposed framework can save numerous hours of productivity by eliminating tedious parts of writing code.

In [40], the problem of predicting the movement of a drifter on the ocean is considered. The authors estimated drifter tracking over seawater using ML and evolutionary search techniques including differential evolution, particle swarm optimization, multi-layer perceptron, SVM, deep NNs, LSTM and others. Extensive comparative research allows us to evaluate the suitability of various ML algorithms for solving this type of problem.

A salesperson performance measurement is a process that occurs multiple times per year on a company. During this process, the salesperson is evaluated how he or she performed on numerous KPIs. In [41], several data mining techniques are proposed to allow managers to make a better decision about salespeople performance measurement based on metrics defined by the business. The authors applied a naive Bayes model to classify salespeople into pre-defined categories provided by the business. They showed that the proposed approach can be applied in many companies using different KPIs.