Submit to Applied Sciences Review for Applied Sciences Propose a Special Issue

Journal Menu

Journal Browser

Pattern Recognition in Multimedia Signal Analysis

Print Special Issue Flyer
Special Issue Editors
Special Issue Information
Keywords
Benefits of Publishing in a Special Issue
Published Papers

A special issue of Applied Sciences (ISSN 2076-3417). This special issue belongs to the section "Computing and Artificial Intelligence".

Deadline for manuscript submissions: closed (30 April 2022) | Viewed by 38459

Share This Special Issue

Special Issue Editors

Dr. Theodoros Giannakopoulos

E-Mail Website
Guest Editor

Institute of Informatics and Telecommunications, National Center for Scientific Research, Athens, Greece
Interests: audio/speech analysis; multimodal information retrieval

Prof. Dr. Evaggelos Spyrou

E-Mail Website
Guest Editor

Department of Informatics and Telecommunications, University of Thessaly, Thessaly, Greece
Interests: multimedia analysis; computer vision; human activity recognition; emotion recognition; deep learning
Special Issues, Collections and Topics in MDPI journals

Special Issue Information

Dear Colleagues,

Huge amounts of multimedia data have been generated in recent years, either through profesional “content providers” (TV, movies, internet TV, and music videos) or user-generated content (vlogs, social media multimodal content, and multisensor data). Therefore, the need for automatic indexing, classification, content visualization, and recommendation, through multimodal pattern recognition, is obvious for various applications. In addition, multimedia data exhibit much richer structures and representations than simple forms of data and, as a result, the related pattern recognition approaches must take that into consideration.

This Special Issue focuses on novel approaches for analyzing multimodal content using pattern recognition and signal analysis algorithms. Application areas include but are not limited to video summarization, content-based multimedia indexing and retrieval, content-based recommender systems, multimodal behavior and emotion recognition, patient/elderly home monitoring based on multimodal sensors, mental health monitoring, and autonomous driving.

In this Special Issue, we invite submissions that report on cutting-edge research in the broad spectrum of pattern recognition in multimedia analysis, related to the aforementioned areas. Survey papers and reviews in a specific research and/or application area are also welcome. All submitted papers will undergo our standard peer-review procedure. Accepted papers will be published in open-access format in Applied Sciences and collected together on this Special Issue website.

Dr. Theodoros Giannakopoulos
Prof. Dr. Evaggelos Spyrou
Guest Editor

Manuscript Submission Information

Manuscripts should be submitted online at www.mdpi.com by registering and logging in to this website. Once you are registered, click here to go to the submission form. Manuscripts can be submitted until the deadline. All submissions that pass pre-check are peer-reviewed. Accepted papers will be published continuously in the journal (as soon as accepted) and will be listed together on the special issue website. Research articles, review articles as well as short communications are invited. For planned papers, a title and short abstract (about 100 words) can be sent to the Editorial Office for announcement on this website.

Submitted manuscripts should not have been published previously, nor be under consideration for publication elsewhere (except conference proceedings papers). All manuscripts are thoroughly refereed through a single-blind peer-review process. A guide for authors and other relevant information for submission of manuscripts is available on the Instructions for Authors page. Applied Sciences is an international peer-reviewed open access semimonthly journal published by MDPI.

Please visit the Instructions for Authors page before submitting a manuscript. The Article Processing Charge (APC) for publication in this open access journal is 2400 CHF (Swiss Francs). Submitted papers should be well formatted and use good English. Authors may use MDPI's English editing service prior to publication or during author revisions.

Keywords

Multimodal signal processing;
Multimedia pattern recognition;
Audio-visual fusion.

Benefits of Publishing in a Special Issue

Ease of navigation: Grouping papers by topic helps scholars navigate broad scope journals more efficiently.
Greater discoverability: Special Issues support the reach and impact of scientific research. Articles in Special Issues are more discoverable and cited more frequently.
Expansion of research network: Special Issues facilitate connections among authors, fostering scientific collaborations.
External promotion: Articles in Special Issues are often promoted through the journal's social media, increasing their visibility.
e-Book format: Special Issues with more than 10 articles can be published as dedicated e-books, ensuring wide and rapid dissemination.

Further information on MDPI's Special Issue polices can be found here.

Published Papers (7 papers)

Download All Papers

Order results

Result details

Show export options Show export options

Select all

Export citation of selected articles as:

Research

Jump to: Review

31 pages, 1145 KiB

Open AccessArticle

Multimodal Classification of Safety-Report Observations

by Georgios Paraskevopoulos, Petros Pistofidis, Georgios Banoutsos, Efthymios Georgiou and Vassilis Katsouros

Appl. Sci. 2022, 12(12), 5781; https://doi.org/10.3390/app12125781 - 7 Jun 2022

Cited by 5 | Viewed by 3106

Abstract

Modern businesses are obligated to conform to regulations to prevent physical injuries and ill health for anyone present on a site under their responsibility, such as customers, employees and visitors. Safety officers (SOs) are engineers, who perform site audits to businesses, record observations regarding possible safety issues and make appropriate recommendations. In this work, we develop a multimodal machine-learning architecture for the analysis and categorization of safety observations, given textual descriptions and images taken from the location sites. For this, we utilize a new multimodal dataset, Safety4All, which contains 5344 safety-related observations created by 86 SOs in 486 sites. An observation consists of a short issue description, written by the SOs, accompanied with images where the issue is shown, relevant metadata and a priority score. Our proposed architecture is based on the joint fine tuning of large pretrained language and image neural network models. Specifically, we propose the use of a joint task and contrastive loss, which aligns the text and vision representations in a joint multimodal space. The contrastive loss ensures that inter-modality representation distances are maintained, so that vision and language representations for similar samples are close in the shared multimodal space. We evaluate the proposed model on three tasks, namely, priority classification of input observations, observation assessment and observation categorization. Our experiments show that inspection scene images and textual descriptions provide complementary information, signifying the importance of both modalities. Furthermore, the use of the joint contrastive loss produces strong multimodal representations and outperforms a baseline simple model in tasks fusion. In addition, we train and release a large transformer-based language model for the Greek language based on the Electra architecture. Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Figure 1

9 pages, 1445 KiB

Open AccessArticle

Multi-Modal Emotion Recognition Using Speech Features and Text-Embedding

by Sung-Woo Byun, Ju-Hee Kim and Seok-Pil Lee

Appl. Sci. 2021, 11(17), 7967; https://doi.org/10.3390/app11177967 - 28 Aug 2021

Cited by 9 | Viewed by 3509

Abstract

Recently, intelligent personal assistants, chat-bots and AI speakers are being utilized more broadly as communication interfaces and the demands for more natural interaction measures have increased as well. Humans can express emotions in various ways, such as using voice tones or facial expressions; therefore, multimodal approaches to recognize human emotions have been studied. In this paper, we propose an emotion recognition method to deliver more accuracy by using speech and text data. The strengths of the data are also utilized in this method. We conducted 43 feature vectors such as spectral features, harmonic features and MFCC from speech datasets. In addition, 256 embedding vectors from transcripts using pre-trained Tacotron encoder were extracted. The acoustic feature vectors and embedding vectors were fed into each deep learning model which produced a probability for the predicted output classes. The results show that the proposed model exhibited more accurate performance than in previous research. Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Figure 1

19 pages, 669 KiB

Open AccessArticle

Emotion Identification in Movies through Facial Expression Recognition

by João Almeida, Luís Vilaça, Inês N. Teixeira and Paula Viana

Appl. Sci. 2021, 11(15), 6827; https://doi.org/10.3390/app11156827 - 25 Jul 2021

Cited by 12 | Viewed by 6147

Abstract

Understanding how acting bridges the emotional bond between spectators and films is essential to depict how humans interact with this rapidly growing digital medium. In recent decades, the research community made promising progress in developing facial expression recognition (FER) methods. However, no emphasis has been put in cinematographic content, which is complex by nature due to the visual techniques used to convey the desired emotions. Our work represents a step towards emotion identification in cinema through facial expressions’ analysis. We presented a comprehensive overview of the most relevant datasets used for FER, highlighting problems caused by their heterogeneity and to the inexistence of a universal model of emotions. Built upon this understanding, we evaluated these datasets with a standard image classification models to analyze the feasibility of using facial expressions to determine the emotional charge of a film. To cope with the problem of lack of datasets for the scope under analysis, we demonstrated the feasibility of using a generic dataset for the training process and propose a new way to look at emotions by creating clusters of emotions based on the evidence obtained in the experiments. Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Figure 1

17 pages, 1182 KiB

Open AccessArticle

Multimodal Summarization of User-Generated Videos

by Theodoros Psallidas, Panagiotis Koromilas, Theodoros Giannakopoulos and Evaggelos Spyrou

Appl. Sci. 2021, 11(11), 5260; https://doi.org/10.3390/app11115260 - 5 Jun 2021

Cited by 9 | Viewed by 3501

Abstract

The exponential growth of user-generated content has increased the need for efficient video summarization schemes. However, most approaches underestimate the power of aural features, while they are designed to work mainly on commercial/professional videos. In this work, we present an approach that uses both aural and visual features in order to create video summaries from user-generated videos. Our approach produces dynamic video summaries, that is, comprising the most “important” parts of the original video, which are arranged so as to preserve their temporal order. We use supervised knowledge from both the aforementioned modalities and train a binary classifier, which learns to recognize the important parts of videos. Moreover, we present a novel user-generated dataset which contains videos from several categories. Every 1 s part of each video from our dataset has been annotated by more than three annotators as being important or not. We evaluate our approach using several classification strategies based on audio, video and fused features. Our experimental results illustrate the potential of our approach. Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Figure 1

15 pages, 624 KiB

Open AccessArticle

Face Morphing, a Modern Threat to Border Security: Recent Advances and Open Challenges

by Erion-Vasilis Pikoulis, Zafeiria-Marina Ioannou, Mersini Paschou and Evangelos Sakkopoulos

Appl. Sci. 2021, 11(7), 3207; https://doi.org/10.3390/app11073207 - 2 Apr 2021

Cited by 10 | Viewed by 8272

Abstract

Face morphing poses a serious threat to Automatic Border Control (ABC) and Face Recognition Systems (FRS) in general. The aim of this paper is to present a qualitative assessment of the morphing attack issue, and the challenges it entails, highlighting both the technological and human aspects of the problem. Here, after the face morphing attack scenario is presented, the paper provides an overview of the relevant bibliography and recent advances towards two central directions. First, the morphing of face images is outlined with a particular focus on the three main steps that are involved in the process, namely, landmark detection, face alignment and blending. Second, the detection of morphing attacks is presented under the prism of the so-called on-line and off-line detection scenarios and whether the proposed techniques employ handcrafted features, using classical methods, or automatically generated features, using deep-learning-based methods. The paper, then, presents the evaluation metrics that are employed in the corresponding bibliography and concludes with a discussion on open challenges that need to be address for further advancing automatic detection of morphing attacks. Despite the progress being made, the general consensus of the research community is that significant effort and resources are needed in the near future for the mitigation of the issue, especially, towards the creation of datasets capturing the full extent of the problem at hand and the availability of reference evaluation procedures for comparing novel automatic attack detection algorithms. Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Figure 1

22 pages, 2895 KiB

Open AccessFeature PaperArticle

Distracted and Drowsy Driving Modeling Using Deep Physiological Representations and Multitask Learning

by Michalis Papakostas, Kapotaksha Das, Mohamed Abouelenien, Rada Mihalcea and Mihai Burzo

Appl. Sci. 2021, 11(1), 88; https://doi.org/10.3390/app11010088 - 24 Dec 2020

Cited by 12 | Viewed by 3880

Abstract

In this paper, we investigated various physiological indicators on their ability to identify distracted and drowsy driving. In particular, four physiological signals are being tested: blood volume pulse (BVP), respiration, skin conductance and skin temperature. Data were collected from 45 participants, under a simulated driving scenario, through different times of the day and during their engagement on a variety of physical and cognitive distractors. We explore several statistical features extracted from those signals and their efficiency to discriminate between the presence or not of each of the two conditions. To that end, we evaluate three traditional classifiers (Random Forests, KNN and SVM), which have been extensively applied by the related literature and we compare their performance against a deep CNN-LSTM network that learns spatio-temporal physiological representations. In addition, we explore the potential of learning multiple conditions in parallel using a single machine learning model, and we discuss how such a problem could be formulated and what are the benefits and disadvantages of the different approaches. Overall, our findings indicate that information related to the BVP data, especially features that describe patterns with respect to the inter-beat-intervals (IBI), are highly associates with both targeted conditions. In addition, features related to the respiratory behavior of the driver can be indicative of drowsiness, while being less associated with distractions. Moreover, spatio-temporal deep methods seem to have a clear advantage against traditional classifiers on detecting both driver conditions. Our experiments show, that even though learning both conditions jointly can not compete directly to individual, task-specific CNN-LSTM models, deep multitask learning approaches have a great potential towards that end as they offer the second best performance on both tasks against all other evaluated alternatives in terms of sensitivity, specificity and the area under the receiver operating characteristic curve (AUC). Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Figure 1

Review

Jump to: Research

21 pages, 429 KiB

Open AccessReview

Deep Multimodal Emotion Recognition on Human Speech: A Review

by Panagiotis Koromilas and Theodoros Giannakopoulos

Appl. Sci. 2021, 11(17), 7962; https://doi.org/10.3390/app11177962 - 28 Aug 2021

Cited by 30 | Viewed by 7284

Abstract

This work reviews the state of the art in multimodal speech emotion recognition methodologies, focusing on audio, text and visual information. We provide a new, descriptive categorization of methods, based on the way they handle the inter-modality and intra-modality dynamics in the temporal dimension: (i) non-temporal architectures (NTA), which do not significantly model the temporal dimension in both unimodal and multimodal interaction; (ii) pseudo-temporal architectures (PTA), which also assume an oversimplification of the temporal dimension, although in one of the unimodal or multimodal interactions; and (iii) temporal architectures (TA), which try to capture both unimodal and cross-modal temporal dependencies. In addition, we review the basic feature representation methods for each modality, and we present aggregated evaluation results on the reported methodologies. Finally, we conclude this work with an in-depth analysis of the future challenges related to validation procedures, representation learning and method robustness. Full article

(This article belongs to the Special Issue Pattern Recognition in Multimedia Signal Analysis)

► Show Figures

Journal Menu

Journal Browser

Pattern Recognition in Multimedia Signal Analysis

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Published Papers (7 papers)

Research

Review

Further Information

Guidelines

MDPI Initiatives

Follow MDPI