Spatial Footprints of Human Perceptual Experience in Geo-Social Media

Abstract

Analyses of social media have increased in importance for understanding human behaviors, interests, and opinions. Business intelligence based on social media can reduce the costs of managing customer trend complexities. This paper focuses on analyzing sensation information representing human perceptual experiences in social media through the five senses: sight, hearing, touch, smell, and taste. First a measurement is defined to estimate social sensation intensities, and subsequently sensation characteristics on geo-social media are identified using geo-spatial footprints. Finally, we evaluate the accuracy and F-measure of our approach by comparing with baselines.

1. Introduction

Social media has become a critical part of the modern information ecosystem in various business fields, such as marketing, reputation, sales, and price competition [1,2]. Many companies have actively used social media in their marketing activities by uploading multimedia adverising stops on YouTube (https://www.youtube.com/), running real-time promotions on Facebook (https://www.facebook.com/) and Twitter (https://twitter.com/), or posting detail information about their products on Wikipedia (https://www.wikipedia.org/) although social media marketing has drawbacks such as backfires, a user abusing, or a malicious rumor [3,4]. Accordingly, business intelligence based on social media big data analysis has been emerging in both the academic and business communities over twhe past decade [5,6,7,8]. Social media analysis helps companies understand what customers are thinking, how the market is changing, and even what their competitors are doing [9,10,11,12].

Comprehending individual preferences and public popularity results in successful marketing and product strategies. For this reason, opinion mining and sentiment analyses are the most active topics in social media big data as shown in [13,14,15,16]. Opinion is a primary influencer of human activities and decision-making. Subjectivity classifications indicating whether opinions are positive, negative, or neutral help companies identify how consumers perceive their products, services, and brands as well as indirectly obtain feedback from consumers. However, it was found that human sensory perception, called sensation information, is a primal factor affecting human behaviors preceding opinions, sentiments, emotions, and evaluations [17,18,19,20]. Sensation information has been applied to marketing to design new products and imprint company brands [21,22,23,24,25,26]. For example, human interaction devices such as Oculus lift (https://www.oculus.com/rift/) and Gloveone (https://www.neurodigital.es/gloveone/) fundamentally considered human sensory experiences in terms of their design and functionality. In [27], a language processor bridging the gap between perception and action was created to enable intelligent human–robot interactions during sensorimotor tasks. Although sensory experiences directly affect decision-making, actions, and sentiment formation in humans, few studies have considered sensation information itself in text mining.

This paper focused on sensation analysis by extracting, quantifying, and identifying sensation information from sight (ophthalmoception), hearing (audioception), touch (tactioception), smell (olfacception), and taste (gustaoception) as introduced in [28]. Human perception is the process of selecting, organizing, and interpreting sensory information into meanings or concepts on an individualized basis. For example, the statement, “I went to a restaurant with my friend a few days ago. It was so crowded, and music was played at a high volume. Although the food was great, I could not clearly hear my friend’s voice. I do not want to go there again”, presents a negative opinion regarding the restaurant. When we look at the reasoning behind this negative evaluation, auditory perception is an important criterion. Human behavior is influenced by unconscious states as well as conscious thinking, as shown in Figure 1.

However, finding sensory perceptions in natural language is a rudimentary task with a small set of lexicons and training corpora. Furthermore, word sense disambiguation (WSD) and metaphor expression create difficulties in distinguishing sensations from opinions or sentiments, such as in, “He gave me a cool reception” and “All people are blind to their own mistakes”. Here, quantitative features of sensation were extracted from social media with geographic locations based on a previous study in [28] and geo-spatial pattern differences in social perceptual experiences were investigated. The main contributions of this paper are threefold:

• Sensation feature analysis: The sensation measurements assigned to sight, hearing, touch, smell, and taste in sentences were addressed with sensation word sets and WordNet [29]. Based on the five senses, unstructured text was turned into structure data for sensation classification. This classification contributed to the construction of a sensation corpus for studying social perceptual experiences by lowering costs and complexity.
• Geo-spatial footprint analysis: The natural language of social media was considered as a kind of sensor data, and we tried to exploit this data to discover a geo-spatial knowledge. For this purpose, a domain area was divided into several sub-areas to identify sensation patterns from geo-spatial footprints on social media. Additionally, strong trends in sensation features were identified for each area and their patterns were compared.
• Comprehensive evaluation: To evaluate the proposed classification approach, available alternatives (random forest, support vector machine (SVM), multilayer perceptron (MLP), convolutional neural network (CNN), and recurrent convolutional neural network (RCNN)) were performed with the proposed sensation features and general word-based features. These results identified the best-performing combinations for classification.

The remainder of this paper is organized as follows. Section 2 briefly reviews related work encompassing social media analysis methods and several related techniques. In Section 3, a measurement for sensation classification is presented followed by the sensation features of geo-social media data and the resulting evaluations of sensation classification using several baselines, including neural network methods in Section 4. Finally, Section 5 concludes the paper and outlines future work.

2. Related Work

Social networks play a crucial role in understanding daily lives as information channels outside of traditional communication tools. Social media and mobile technologies have begun delivering human experiences such as observations, emotions, judgments, and even behaviors. The popularity of social media has boosted its utilization in advancing business information technologies to increase sales functions by analyzing unstructured contents. A comprehensive survey of social media analysis is addressed in [30]. In particular, sentiment analysis (known as opinion mining or emotion AI) from unstructured textual data was adapted early in business intelligence for garnering feedback on human experiences with products and services. Sentiment analysis is intended to classify polarities from text, such as positive, negative, or neutral, as addressed in [31]. One early work in sentiment analysis by Turney [32] and Pang et al. [33] attempted to detect polarities in product and movie reviews using machine learning techniques. More recently, fine-grained emotion (happy, sad, etc.), mood (cheerful, depressed, etc.), interpersonal stance (friendly, distant, etc.), attitude (like, love, etc.), and personality trait (nervous, anxious, etc.) analyses have been conducted as machine learning techniques developed. These have been widely applied to real-life services such as personal recommendation [34,35,36], stock prediction [37,38,39], and political campaigns [40,41,42]. Lisette et al. [43] investigated the relation between brand popularity and numbers of fans as an indication of brand recognition on social media. In [44], Linda et al. attempted to identify consumer-brand engagement (CBE) to reflect the nature of interactive brand relationships (including social interactions, emotional, and cognitive clues, and brand-related behaviors) among particular consumers in social media. Furthermore, Jari et al. [45] and Carolyn et al. [46] addressed the opportunities and challenges of utilizing social media in business-to-business relationships and customer relationship management, respectively.

There exist two main approaches to sentiment analysis: learning- and lexicon-based. In learning-based approaches, sentiment classification can be formulated as a supervised learning problem with three classes: positive, negative, and neutral [31]. Approximately all supervised learning methods can be used for sentiment classification, including naive Bayesian and SVM. Pang et al. [33] took this approach to classify movie reviews into two categories: positive and negative. Their work demonstrated that using unigrams as features in classification performed well with either naive Bayesian or SVM. Tan et al. [47] used opinion words to label a part of valuable examples and then learn a new supervised classifier based on labeled ones.

Lately, deep learning has become a highly important method for analyzing text and image data. State-of-the-art deep learning methods such as the CNN, RNN, and recurrent convolutional neural network (RCNN) are continually being developed for sentiment analysis. Previously, RNN techniques were the most popular ones for sentiment analysis and were dramatically improved by using long short-term memory (LSTM), developed by S. Hochreiter et al. [48]. D. Tang et al. [49] introduced a neural network model (LSTM-GRNN) for document level sentiment classification. Their model performed end-to-end learning more effectively than SVM and CNN on four review datasets. In past research, CNNs have primarily been applied to image processing; however, Y. Kim [50] proposed a CNN for sentiment text classification with remarkable experiment results. S. Poria et al. [51] employed a deep CNN to extract features from text for utterance level sentiment analysis. Furthermore, S. Lai et al. [52] combined recurrent and convolutional sentiment analysis methods for short texts. They took advantage of the local features generated by CNNs and long-distance dependencies learned via RNNs to overcome constraints caused by short text.

Word sense disambiguation (WSD) is worthy of notice in natural language processing to identify the meaning of words in context in a computational manner [53]. There are several approaches to solve WSD problems nowadays, including polysemic, metonymic, and metaphorical extensions [54,55,56]. Of course, the sentence that contains sensation words has the ambiguities of meanings. For example, there are two example sentences: (1) “Blue Cheese is quite pungent but it’s delicious”; (2) “She is the author of a pungent political comedy”. The pungent word is commonly used in the representation of sensory information having “a sharply strong taste or smell”. However, it is used to metaphorically represent “a sharp and caustic quality of comment, criticism, and humor”. This demonstrates that words can be interpreted in multiple ways depending on their linguistic context. In order to address the WSD to identify the sense of a polysemic and metaphorical word, there are knowledge based approaches, machine learning based approaches, and hybrid approaches. Knowledge based approaches rely on knowledge resources or dictionaries such as WordNet and thesaurus and mainly use hand coded and grammar rules for disambiguation. However, machine learning based approaches rely on corpus evidence and train a probabilistic or statistical model using tagged or untagged corpus. Hybrid approaches mix them to use corpus evidence as well as semantic relations from the knowledge resources.

Furthermore, sentiment lexicons are crucial resources for analyzing polarities in sentiment analysis. Lexicon-based approaches are based on the sentiment orientation sum of each word or phrase. P. Palanisamy et al. [57] described a method of building lexicons from Serendio taxonomy, and then classified tweets as positive or negative based on the contextual sentiment orientation of words from their lexicon. Melville et al. [58] suggested a framework incorporating lexical knowledge into supervised learning to improve accuracy; additionally, Taboada et al. [59] used known opinion words for sentiment classification. Several open lexicons such as Sentiwordnet [60], Q-wordnet [61], WordNet-Affect [62] have already been developed to support sentiment analysis. Such lexicon-based approaches are highly effective, especially when no supervised datasets are available.

This study proposed a lexicon-based approach for the sensation classification of Twitter posts, because few labeled examples exist. Sensation information is a property dependent on the perception of human sensory organs and the processing of their information in the brain. According to the psychology studies shown in [63], decision-making is based on sensation information from the unconscious mind. Hence, sensation information affects human decision-making prior to cognitive thinking. As the usage of sensation information emerges in marketing and business areas, some companies strive to improve their brand images using a method called sensory branding. For example, pairing sound with food and drink has been scientifically proven to enhance the human experience of flavor. The study in [64] discovered that high-frequency sounds help to stimulate sweet taste sensations, whereas low sounds bring out bitterness. British Airways (https://www.britishairways.com) is banking on the sensory science to stand out in the premium market. Furthermore, Dunkin’ Donuts (https://www.dunkindonuts.com) stimulates the sense of smell to entice visitors by releasing a coffee aroma into the bus whenever the company jingle played. Even though sensation information may clearly be considered an important feature reflecting the five senses using human language, very few works have been published estimating or discovering the sensation information from text. If the trends of human perceptual experiences can be understood using social media, these trends can be rapidly adopted to develop new human-centered designs and systems.

3. Methodology

This section explains how sensation sentences representing social perceptual experiences are classified. First, a dictionary of representative seed words was built to include all five senses: sight (ophthalmoception), hearing (audioception), touch (tactioception), smell (olfacception), and taste (gustaoception). Next, the intensity of each sensation feature was calculated based on this dictionary and WordNet, and then, finally, sensation sentences were identified. Figure 2 shows the classification procedures of Geo-partitioning, Tagging, Enriching, and Scoring.

3.1. Lexicon Resources

One important lexicon resource is a set of seed words containing sensory activities such as sight, hearing, touch, smell, and taste. For this paper, the sensation word set was built from words manually collected using Google search results from the queries “sensation word” and “sensory word”. The resulting word set included 1222 words such as lucent and murky (for sight), sizzle and whir (for hearing), chill and tingle (for touch), reek and stench (for smell), and sour and ambrosial (for taste).

WordNet, a computational lexicon of English based on psycholinguistic principles, was used as a resource for dealing with WSD. It encodes concepts in terms of sets of synonyms (called synsets), and the latest version contains approximately 155,000 words organized in over 117,000 synsets. In order to address WSD and identify the sense of a polysemic word, each word in this paper was enriched with WordNet synsets.

3.2. Sensation Feature Extraction

3.2.1. Geo-Partitioning

The primary task of this work was to partition social media data into several small geographical areas. One aim was to identify sensation intensities in terms of geo-spatial location, and thus data had to be grouped regarding spatial coordinates. For this work, the whole world was partitioned equally into uniquely numbered $C_{id}$ fixed-size cells.

With development of mobile applications, the social media users could generate social data, which have attached GPS coordinate location information, including latitude and longitude. Among social media data, we focused on the tweets from Twitter, which is one of most popular social media service. Thus, Twitter data could be partitioned off into cells using the naming protocol $G_{id}=\{DT,C_{id}\}$, where $G_{id}$ was a partitioned Twitter datum associated with $DT$ (the tweet text) and $C_{id}$ (the correspoding spatial cell); furthermore, $DT=\{T_1,T_2,\dots ,T_n\}$ represented a set of n number geo-partitioned tweet texts.

3.2.2. Tagging

In order to retrieve a corresponding part-of-speech (POS) word synset, tweet texts were preprocessed via morpheme analysis using the POS tagger for Twitter [65]. This study accounted only for Noun, Verb, Adjective, and Adverb classifications among the eight parts of speech in English. These are the four parts of speech primarily used in sentences associated with sensation features: sight, hearing, touch, smell, and taste. For example, the quotation “I smell a subtle fragrance of perfume. It is never smelled so sweet!” passes only the following POS results on to the next step: {(smell, Verb), (subtle, Adj), (fragrance, Noun), (perfume, Noun), (never, Adv), (smell, Verb), (sweet, Adj)} (hereafter referred to as tagging-pairs).

3.2.3. Enriching

The third step of sensation feature extraction was enriching POS results by attaching WordNet synsets to decrease the effect of WSD and reveal the appropriate meaning of words. In this process, the most useful semantic relations were selected from WordNet in accordance to the POS of each word.

Table 1. Selected semantic relation from WordNet synset.

Part-of-Speech	Selected Semantic Relation
Noun, Verb	{direct hypernym, sister term}
Adjective	{see also, similar term}
Adverb	{synonyms}

shows the selection of semantic relationships from WordNet corresponding to each POS.

Let $T_e=\{t_1,t_2,\dots ,t_k\}$ be a set of tagging-pairs with k elements and $t_i=(w_i,po{s}_i):1\le i\le k$ be the ith word (w) and POS tag (pos) pairing. The function $H_t$ was defined to return a set of words containing similar meanings from the WordNet synsets of w via the pos selected semantic relationships, including the w from tagging-pair t. For example, the H function enriches the word smell in tagging-pair t = (smell, Verb) into the set of words $H_t$ = {smell, odorize, odourise, scent, radiate, project, ache, smack, reek, perceive} by traversing the selected semantic relation {direct hypernym, sister term} corresponding to the POS tag Verb.

3.2.4. Scoring

The scoring task employed the Lesk Algorithm [66] to calculate the gloss overlap between the sense definitions of two or more target words. Given the context of two words, (w1, w2), the senses of target words whose definitions had the highest overlap (specifically, words in common) were assumed to be the correct ones. As seen in much of the latest research based on this method, sensation measures were defined by optimizing the use of WordNet and the Lesk algorithm.

A single sensation word set was referred to as $Sensatio{n}_f$, where f was a feature of sight, hearing, touch, smell, or taste. Given a sensation word set $Sensatio{n}_f$ of a feature f, the sensation intensity of the f feature for a tagger-pair, t, with the results of the enriching task, $H_t$, was calculated as follows:

• Definition 1. Sensation Intensity

  $$I_{Sensatio{n}_f}\left(t\right)={\sum }_{e\in H_t}{S}_e,$$

  (1)

where e was an element (enriched word) from function $H_t$ and $S_e$ was the indicator function defined below:

$$S_e=\left\{\begin{array}{cc}1\ast \sigma ,\hfill & \mathrm{if} e.w\in Sensatio{n}_f\hfill \\ 0,\hfill & \mathrm{otherwise},\hfill \end{array}\right.$$

(2)

where $\sigma$ was a weight factor for normalizing the occurrences of words in the sensation word sets and text. First the number of each sensation word set, $|Sensatio{n}_f|$ was accounted for because a sensation word set containing more words may bring a higher value of sensation intensity. Specifically, the probability of an enriched word being included in the word set $Sensatio{n}_f$ was increased. To normalize this imbalance when computing the sensation intensities of each feature, the weight factor in the indicator function was adjusted using the ratio between the number of the word in each sensation word set using the following equation:

$$\sigma ={\sigma }_S=\overline{|Sensatio{n}_f|}/\left|Sensatio{n}_f\right|,$$

(3)

where $|Sensatio{n}_f|$ was the number of the targeted word in feature f (the sensation word set) and $\overline{|Sensatio{n}_f|}$ was the average word count from all sensation word sets. Thus, the weight was the reciprocal of the ratio giving high weights to a small sensation word set.

Second, the importance of a word was regarded as a weight factor using tf-idf [67]. The tf-idf measure is a popular term weighting schemes for assessing imbalances in word occurrences in a document corpuses. For a copus of tagging-pair sets $DT=\{T_1,T_2,\dots ,T_n\}$, the scoring in this paper used a weight factor defined by the following equation:

$$\sigma ={\sigma }_S\times (t{f}_{w,T_i}\times id{f}_w),$$

(4)

where $t{f}_{w,T_i}$ was the number of occurrences of a word, w, in a tagging-pair set, $T_i$, in $DT$, representing a text, and $id{f}_w$ was the number of tagging-pair sets containing the word in $DT$.

Given a sensation word set, $\forall f\in F=$ ($sight$, $hearing$, $touch$, $smell$, $taste$) : $Sensatio{n}_f$, the sensation intensity for a set of tagging-pairs, $T_i=\{t_{i1},t_{i2},\dots ,t_{ik}\}$, was calculated based on the Definition 1 and indicator function in Equation (2) as follows:

$$I_{Sensatio{n}_f}\left(T_i\right)={\sum }_{t\in T_i}{\sum }_{e\in H_t}{S}_e.$$

(5)

Finally, text were represented by vectors in a five-dimensional space at each sensation feature, for instance, $I\left(\right\{(honey,Adj),$ $(smell,Verb)\left\}\right)$ = $(I_{sight},$ $I_{hearing},$ $I_{touch},$ $I_{smell},$ $I_{taste})$ = $(4,3,10,3,8)$. Namely, the sensation intensity could indicate how the five sensation features were reflected in a sentence or corpus. An experimental result demonstrating the differences in sensation feature intensities with respect to a corpus topic is shown in Section 4.

3.3. Sensation Classification

This study simply classified texts into two groups: True if they represented the sensation information and False otherwise. This binary classification was considered as there were insufficient data to train for individual sensation features. Classifying each sensation feature individually remains a task for future work after a sufficient amount of corpus evidence is collected. In order to identify the existence of sensation representation in a text with the described sensation intensity features, an SVM variant was employed. The SVM classifier has shown substantial performance gains in many binary classification problem domains [68]. First, a labeled training set m was manually prepared: $TS=\{(\overrightarrow{x_1},y_1),$ $(\overrightarrow{x_2},y_2),$ $\dots ,$ $(\overrightarrow{x_m},y_m)\}$, where the label $y_i=$ ±1 indicated one of two classes (True:+1, False:−1) to which the ith input feature $\overrightarrow{x_i}$ belonged. It was assumed that the training set was linearly separable by a hyperplane (decision boundary) of the form: $w^{T}x+b=0$, where w and b represented the weight vector and bias of the plane, respectively. The SVM classifier then found the best choice of hyperplane to maximize margin 2/$\parallel w\parallel$ from both groups (sensation and non-sensation).

For the input features $\overrightarrow{x}$, an additional dimension (accumulation of intensities) was included with the five features (sight, hearing, touch, smell, and taste) because of the binary classification problem. Each $\overrightarrow{x}$ was therefore a six-dimensional vector found by computing the intensity of each sensation feature, $\overrightarrow{x}$ = ($I_{sight}$, $I_{hearing}$, $I_{touch}$, $I_{smell}$, $I_{taste}$, $I_{sum}$), where $I_{sum}={\sum }_{f\in (sight,hearing,touch,smell,taste)}{I}_f$. However, two issues occurred when using the original sensation intensity values for classification. First, the number of synsets in WordNet has a side effect. Words with more synsets may have increased sensation intensities due to the larger enriched word set calculated by Equation (1). For example, the word clear can be used as a Noun, Verb, Adjective, or Adverb and, has multiple meanings for the same POS (i.e., 24 meanings under Verb in WordNet). In order to reduce this effect, the sensation intensity was redefined from Equation (1) as follows:

• Definition 2. Normalized Sensation Intensity

  $$I_{Sensatio{n}_f}^{\ast }\left(t\right)={\sum }_{e\in H_t}{S}_e/\left|H_t\right|,$$

  (6)

where $|H_t|$ was the number of words enriched by function $H_t$ with the relevant semantic relationships in the enriching task.

Second, the length of each text was considered because the sensation intensity is also influenced by the number of words in this measurement per Equation (5). The more words a text includes, the larger the derived sensation feature intensity score becomes. Thus, the equation was modified by dividing the sensation feature intensity score by the number of elements in the tagging-pair set T as follows:

$$I_{Sensatio{n}_f}^{\ast }\left(T\right)={\sum }_{t\in T}{I}_{Sensatio{n}_f}^{\ast }\left(t\right)/\left|T\right|,$$

(7)

where $\left|T\right|$ was the number of elements in the tagging-pair set. Accordingly, the training and test data were transformed into six-dimensional vectors using the normalized sensation intensity of features for this classification problem.

4. Experiments and Evaluations

The following experiments were performed with two purposes of assessment. One was to evaluate the accuracy of sensation features derived from the Twitter corpus (http://noisy-text.github.io/2016/index.htm) with several machine learning techniques and baselines. The other was to reveal the regional characteristics of social perceptual experiences. In concrete terms, this study aimed to observe how social sensation reacts to, among the five senses, trends in the same time period among geo-spatial groups. Particularly, we tried to discover a usage pattern of sensation expression between geo-spatial groups. The tweet text was considered as a kind of human sensor data to reveal a geo-spatial knowledge in terms of the sensation information.

4.1. Dataset

This experiment used the Twitter corpus among other social medias. This dataset consisted of approximately seven million tweets including tweet text, tags, urls, time, and geo-location information. Two hundred thousand tweets were randomly selected from all tweets posted between 2007 and 2015 with tweet text, time and geo-location information. As there is no corpus for sensation classification, a judgment was made as to whether these tweets contained sensation information or not. Only tweets annotated as the same decision from more than two out of three individuals were taken into account. As a result, approximately fifteen thousand tweets were selected for the training and test datasets from the overall dataset.

4.2. Classification Results

First, the performance of the proposed classification based on sensation features was determined. To simplify comparisons with the selected baselines, the Twitter corpus was rearranged into two types of input features: sensation features and general word-based features. Concretely, we identified that the sensation feature from the proposed method can bring forth a good performance using simple classification algorithms (i.e., Naive Bayes, decision tree, and SVM) by comparing with several state-of-the-art techniques (i.e., neural network algorithms), when we have not enough a training data. For experiments with word-based feature inputs, baselines such as the random forest, SVM, MLP, CNN, and RCNN were employed. To evaluate our sensation feature, the naive Bayes [69], decision tree (C4.5) [70], random forest [71], and SVM algorithms were utilized. In this research, the naive Bayes, C4.5 and random forest algorithms were employed and evaluated using the Weka [72] toolkit. Conversely, the Tensorflow (https://www.tensorflow.org/) library was used to implement the MLP [73], CNN, and RCNN algorithms. In this study, the CNN and RCNN algorithms were adopted from Y. Kim [50] and S. Lai et al. [52] respectively. Additionally, the libsvm [74] algorithm was applied as an SVM classifier for sensation binary classification. In SVM classification, higher sensation intensities were classified as the positive (or sensation) class. To verify these classification results, all experiments were evaluated using 10-fold cross validation as there was a lack of supervised sensation data.

Table 2. Comparison with results from sensation and word features.

Features		Classifier	F1 Measure	Accuracy
		Random Forest	60.7	61.24
Word		SVM	64.4	65.68
feature		MLP	68.2	69.88
		CNN	79.1	79.33
		RCNN	80.1	80.39
		Naive Bayes	61.2	62.42
Sensation	Weight	C4.5	68.9	68.97
feature	(4)	Random Forest	78.4	78.56
		SVM	80.1	80.24

summarizes the classification results in terms of Accuracy measures, such as Accuracy and F1 measure, which considers both precision and recall. In this table, the results of using general word features are in the upper column, whereas the lower column illustrates sensation feature results using the TF-IDF×sensation weight. In the upper results, the neural network classifiers, (MLP, CNN, and RCNN), achieved better performance than the traditional methods, for both F1 and accuracy. However, the greatest encouraging result is the TF/IDF×sensation weight classifier given by Equation (4), which showed 80.24% accuracy and an 80.1% F1 measurement in the averaged evaluation values. Although the word-feature accuracies were slightly better when the RCNN classifier was used relative to the proposed sensation features (80.1% of F1 measure and 80.39% of accuracy), the sensation feature classifiers generally demonstrated good performance even with traditional methods, such as C4.5 and random forest. These experiments were performed with insufficient data to fully assess classification performance; however, it was shown that sensation features may be entitled to further consideration as input features for sensation classification.

4.3. Geo-Spatial Analysis of Sensation Intensity

The geo-spatial sensation intensity distribution was investigated with approximately 80% accuracy across the glove using seven million tweets. Before geo-spatial analysis, it was verified that the proposed measure appropriately reflected real-world sensation intensities. Figure 3 includes two pie-charts showing tweet sensation intensities and the sense importance ratio of human behaviors from survey results [75]. From Figure 3b, sight has the greatest effect on human behavior (74%), followed by hearing (8%), touch (7%), taste (6%), and smell (5%). The sensation intensity ratio found using the proposed measurement, as shown in Figure 3a, is highly similar to Figure 3b. Thus, the proposed measurement can be considered a reasonable sensation intensity measurement within sensation classified tweets, as in the following examples:

• Sight: My eyes are on black haired girls with blue and green eyes.
• Hearing: Music really loud from the next door.
• Touch: It’s such a cold morning.
• Smell: The room smelt odour like rotten eggs and spoiled tomatoes.
• Taste: This root beer is definitely sweet but doesn’t contain alcohol.

Additionally, it was verified that the weight functions (above) could appropriately normalize biased sensation intensities. Sensation intensity normalization was essential for identifying the importance of each intensity. In the Section 3, two weight functions were proposed to normalize the occurrences of words in sensation word sets and tweet texts. As shown in Figure 3a, the sensation intensity ratio is extremely biased toward sight sensations. However, sensation intensities, as shown in Figure 4a, demonstrated that overvalued sensation intensities were reduced more than the non-weight case in Figure 3a. In addition, the superpositions of sensation weights calculated with TF/IDF weights had approximately equal sensation intensities proportional to each other, as shown in Figure 4b. As a result, it was concluded that the weight functions were able to properly calibrate the sensation intensity biases. Thus, the geo-spatial analysis experiments utilized sensations with the TF/IDF weight function applied to clearly distinguish the ratio of sensation intensities.

In our experiment, we discovered the usage pattern of sensation expressions in two aspects. First, the geo-spatial sensation intensity distribution was estimated between English-speaking region and non-English-speaking region to identify a different usage proportion of sensation expression. Second, we revealed how sensation expressions react to natural phenomena, such as the temperature during summer season in United States.

For the first purpose, the experiment was conducted to reveal the sensation intensities of each geo-spatial group. The top seven groups selected in terms of tweet volume was considered as English-speaking region, such as the United States, Australia, and Western Europe (including Britain). From Figure 5, there was no significant difference in sensation intensity between English-speaking geo-spatial groups. However, one remarkable aspect of this experiment was that the sense of touch was more verbalized than hearing in social media, dissimilar to general statistics. Next, the scope of sensation intensity analysis was extended to the rest of the world, including non-English-speaking region. In the geo-partitioning task, the globe was separated into uniquely numbered cells with corresponding tweets. As shown in Figure 6a, some geo-spatial groups manifested very different sensation intensities, especially groups 6, 7, and 8. These groups were usually located in non-English-speaking countries such as Russia and Eastern European countries. They showed a distinctive pattern containing a large proportion of smell and taste sensation features relative to English-speaking groups. Furthermore, Latin America and Africa (including groups 21, 22, 23, and 24) showed a greater sensitivity to touch relative to other senses, contrasting with English-speaking countries. As a result, it can be hypothesized that there are distinctive differences in the usage of sensation expressions between English-speaking and non-English-speaking countries, at least in social media. However, the experiment data, i.e., tweets, had serious inequalities for each region as shown in Figure 6b. This imbalance is an important factor affecting validity and reliability of the experiment. To identify misgivings from data volume, another experiment was proceeded with English tweets in the United States.

Now, the experiment focused on the English-speaking country (i.e., United States) to reveal how sensation expressions react to natural phenomena in different geo-spatial groups. A hypothesis of this experiment was that the hotter a place is, the more touch sensation intensity a place will include. This assumption seems reasonable, since touch sensation is usually appeared by such as hot, cool, warm, and humid words when we feel a temperature deviation. To verify this hypothesis, tweets were separated into the geo-spatial groups (i.e., the United States by States), and then sensation intensities were estimated for the comparison with the average temperatures for every state in the United States. We collected the average temperature of the summer season (i.e., June to August) during 2010 to 2014 from the National Centers for Environmental Information (NCEI) of National Oceanic and Atmospheric Administration (NOAA) (https://www.ncdc.noaa.gov/cag/). Moreover, the usage statistics of Twitter was probed in terms of the tweet volume and the ratio of sensation expression to determine whether the tweet volume contributes to the sensation expression.

From Figure 7, the hypothesis proved to be reasonable, since the usage rate of the touch sensation intensity looked very similar to the average temperatures in most states. For example, California, Texas, and Florida showed higher temperatures than the northern states (e.g., Montana, North Dakota, or Minnesota) as shown in Figure 7a. Likewise, the usage rate of the touch sensation intensity strongly appeared in the same states (i.e., California, Texas and Florida) as shown in Figure 7b. From this result, we could consider that the sensation intensity has the potential to be used for the analysis of natural phenomena. Meanwhile, the usage statistics of Twitter could definitely influence this kind of experiments. This result could be written off as untrustworthy, if there was a critical imbalance in the usage statistics of Twitter or data volume, such as Figure 7b. To determine the reliability of our experiments, we surveyed the volume of tweet as well as the rate of sensation expression over all states during that time period. In Figure 8a, the green color indicated the tweet volume, and the displayed percentage values on the states showed the rate of sensation expression in the total volume of tweet. The average rate of sensation expression usage was 19.3%, and the standard deviation is 3.78 as denoted by the box plot graph in Figure 8b. The remarkable thing was that the sensation expression of all states remained the similar ratios, even if the total volumes were quite different between states. In addition, one interesting fact discovered is that the high temperature states (i.e., California, Texas, and Florida) represented high ratios of the sensation expression, such as 32%, 25%, and 30% respectively.

From our experiments, we could make an assumption from these results that sensation expression reacts to natural phenomena. Thus, the sensation information could be considered as a fine measure to estimate human behaviors. This may be worth considering in some fields of practical application such as marketing or business. For example, a marketing company could optimize advertisement themes or catch phrase designs following the sensation intensities of specific geo-spatial locations.

5. Conclusions and Future Work

Sensation information is not only closely connected to human behavior but also makes it easier to understand social perceptual experiences. This paper focused on representations of sensation representations in social media, including sight (ophthalmoception), hearing (audioception), touch (tactioception), smell (olfacception), and taste (gustaoception). A knowledge-based approach was proposed utilizing the Lesk algorithm with WordNet and a manually collected sensation word set (including terms such as angular, breezy, cry, delicious, and sweet). A measurement was defined for sensory representation in social media sentences, called “sensation intensity” and machine learning techniques were applied to conduct a binary classification using the defined sensation features. Experiments were conducted using data from Twitter to determine the performance of our classification approach relative to several traditional and neural network baselines, including SVM, random forest, MLP, CNN, and RCNN. Furthermore, the importance of sensation intensities according to geo-spatial location was analyzed. This analysis determined that sensitivity to different senses varies distinctively between English-speaking and non-English-speaking countries, including Latin America and Eastern European countries. In addition, we identified geo-spatial characteristic of sensation intensities on the United States by comparing with natural phenomena and sensation expression. From this, we identified that geo-spatial characteristic can be represented by the sensation intensities. This also could be applied to marketing or business strategies for optimizing target marketing.

In future, these experiments will continue with a larger dataset to achieve more robust results. Additionally, fine-grained classifications will be extended to identify each of the five sensory types of social perception from text. Furthermore, an elaborate sensation measurement based on neural network techniques combined with sensation pre-training will be performed. Finally, a social sensory knowledge or database will be constructed for sensation analyses in the same vein as SentiWordNet [60].