UPGCN: User Perception-Guided Graph Convolutional Network for Multimodal Recommendation

Abstract

To tackle the challenges of cold start and data sparsity in recommendation systems, an increasing number of researchers are integrating item features, resulting in the emergence of multimodal recommendation systems. Although graph convolutional network-based approaches have achieved significant success, they still face two limitations: (1) Users have different preferences for various types of features, but existing methods often treat these preferences equally or fail to specifically address this issue. (2) They do not effectively distinguish the similarity between different modality item features, overlook the unique characteristics of each type, and fail to fully exploit their complementarity. To solve these issues, we propose the user perception-guided graph convolutional network for multimodal recommendation (UPGCN). This model consists of two main parts: the user perception-guided representation enhancement module (UPEM) and the multimodal two-step enhanced fusion method, which are designed to capture user preferences for different modalities to enhance user representation. At the same time, by distinguishing the similarity between different modalities, the model filters out noise and fully leverages their complementarity to achieve more accurate item representations. We performed comprehensive experiments on the proposed model, and the results indicate that it outperforms other baseline models in recommendation performance, strongly demonstrating its effectiveness.

1. Introduction

The swift advancement of the internet has facilitated the accessibility of multimedia content in our daily lives. Consequently, the extensive volume of multimedia content, particularly within e-commerce enterprises, has led to the problem of information overload. Recommendation systems are essential for efficiently using the vast array of multimedia content in e-commerce, assisting consumers in identifying products or services of interest among millions of offerings. Collaborative filtering (CF) is a traditional method in recommendation systems that generates personalized suggestions by analyzing the similarities in user behaviors. It has been thoroughly advanced in the domain of recommendations. For instance, Bayesian Personalized Ranking (BPR) [1], a widely recognized collaborative filtering technique that enhances user representation and item representation through matrix factorization, utilizes a BPR loss for optimization. The method of neural graph collaborative filtering (NGCF) [2] integrates user–item interactions into the embedding process, connecting users and items through higher-order connectivity, and utilizes graph convolutional networks (GCNs) to extract valuable information from each layer. MM-Rec [3] combines textual and visual information from news articles to learn multimodal news representations. LightGCN [4] enhances the information aggregation technique of the original GCN, simplifying the model while augmenting recommendation performance.

Collaborative filtering’s advantage rests in its independence from prior knowledge regarding item or user attributes, as it utilizes user behavior information to generate recommendations. However, it has other limitations and challenges, such as the cold start problem and data sparsity. As e-commerce continues to advance rapidly, the accessibility of multimodal item features (including images, text, and videos) has surged. Numerous researchers have integrated these multimodal features into recommendation methods as supplementary information for items, effectively addressing the cold start issue and data sparsity. Consequently, multimodal recommendation methods that incorporate multimodal features into conventional collaborative filtering methods have arisen. Early multimodal recommendation methods [4,5] regarded multimodal features as supplementary information for items, with the objective of mining users’ preferences for those items.

For example, Visual Bayesian Personalized Ranking (VBPR) [5] employs a convolutional neural network (CNN) to derive visual features from images, subsequently integrating these features with item ID embeddings to create a complete item representation. The extensive utilization of graph neural networks (GNNs) in recommendation methods [6,7,8,9,10,11,12] has led to the adoption of GNNs in multimodal recommendation methods to improve the performance of multimodal recommendation. For instance, MMGCN [13] employs three distinct modalities of micro-video information (visual, acoustic, and textual) to construct user–item interaction graphs across various modalities. Graph convolutional networks are then utilized for message passing to derive embedding representations of users and items in these modalities, which are then concatenated to produce the final user preference prediction. GRCN [14] dynamically modifies and enhances the configuration of the user–item interaction graph according to the model training status, thereby removing extraneous edges and acquiring more pertinent information regarding user preferences. EgoGCN [15] uses edgewise modulation (EGO) fusion technology to adaptively transfer multimodal information on the boundary, solving the problem of insufficient fusion or noise information in existing methods. EgoGCN is different from other graph fusion methods because it improves the transfer of information between modalities by changing the boundary information of local modalities. This keeps the processing within modalities and lets information flow between multiple modalities.

DARGON [16] learns dual representations of users and items by constructing a multimodal recommendation homogeneous graph, thus enhancing the binary relationship. MARIO [17] predicts users’ preferences by considering the individual impact of each modality on each interaction, while obtaining item embeddings that preserve the inherent modal-specific properties. MKGCN [18] builds three aggregators, where the multimodal aggregator aggregates the multimodal features of the item into a multimodal knowledge graph, the user aggregator and item aggregator use graph convolutional networks to aggregate multimodal knowledge graph nodes of multihop adjacent nodes, respectively, to model user preferences and item high-order representations. Finally, the recommendation is made using the aggregated embedding representation. PMGT [19] is the pre-trained model that leverages the fused multimodal feature and interactions. The model is pre-trained by setting two graph reconstruction tasks, graph structure reconstruction and masked node feature reconstruction, as the learning objectives. TMKG [20] utilizes multitask learning to end-to-end integrate the two-side information of the trust graph and knowledge graph, in order to capture implicit relationships in a more fine-grained manner. DMRL [21] can capture the different modalities of users’ attention to each factor in user preference modeling and adopts a decoupled representation technique to ensure that the features of different factors in each modality are independent of each other. MEGCF [7] captures the semantic relevance between user and item interactions and richer multimodal features by constructing a symmetric linear graph convolutional network, thereby achieving the synchronous capture of higher-order multimodal semantic relevance and collaborative signals. LUDP [22] uses modal information to construct modal similarity graphs, so as to mine potential relationships between items with similar patterns and learn deeper user preferences based on the characteristics of the different modes of interaction the user has had with items in the past.

Furthermore, certain studies construct auxiliary graphs based on user or item relationships. For instance, DualGNN [23] develops a user co-occurrence graph to extract more precise representations of users by leveraging the interrelations across users. LATTICE [24] constructs item–item relation graphs for each modality utilizing multimodal features, subsequently fusing these graphs to derive a latent item–item graph, resulting in a more holistic latent item representation and augmenting recommendation performance. FREEDOM [25] is based on LATTICE, where the latent item–item graph updates are frozen, and a degree-sensitive edge pruning method is applied to reduce the noise in the interaction graph between users and items. Other approaches employed attention mechanisms to extract user preferences for items’ multimodal features, hence enhancing recommendation performance. MGAT [26], built upon the MMGCN framework, uses the standard graph convolutional network (GCN) for aggregation and combines the aggregated results. To improve information flow across different modalities, it introduced a new gated attention mechanism.

Despite the notable achievements of current multimodal recommendation methods, many limitations persist. Primarily, some approaches utilize the multimodal item features directly as supplementary information for evaluating user preferences [5,14,23]. In reality, the influence of these features on user preferences are variable. To address this issue, some methods use multimodal features as node representations. For instance, MMGCN [13] builds the user–item interaction graphs for each modality to derive the modality-specific representations of users and items, subsequently merging the GCN outputs from different modalities to achieve the final representations. Despite employing analogous methodologies that consider the user’s individual interests across several modalities, they just utilize GCNs within parallel interaction graphs and uniformly process all nearby information, failing to adaptively capture user modality preferences and ignoring the mutual relationships between different modalities, while neglecting the similarity and complementarity between the features of items in different modalities in the embedding vector fusion.

In this paper, we present a user perception-guided graph convolutional network for multimodal recommendation, referred to as UPGCN. Specifically, in the UPGCN model, we first use the multimodal features of items as input and perform graph convolution on the user–item interaction graph to extract user representations under different modalities. Then, we further use the output user representations and the user–item interaction graph to mine the item representations. To accurately assess the potential impact of item features across various modalities on user preferences, we developed a user perception guided representation enhancement module (UPEM), which utilizes user representation with modal preferences as a guiding signal to augment the user ID Embedding. Additionally, to effectively utilize the similarity and complementarity between different modalities of the items’ features, a two-step enhanced fusion method was designed, in which the original modalities of item features and the modalities of item features with user modal preferences were used as enhanced signals to enhance the item ID Embedding. Finally, the enhanced user ID Embedding and item ID Embedding are learned through LightGCN to obtain the final user preference prediction. In addition, we chose graph convolutional networks (GCNs) as the basis of our method because of their effectiveness in handling graph structured data, especially in modeling user–item interactions in recommendation systems. GCNs can capture complex user–item relationships by aggregating information from neighboring nodes, thereby improving the accuracy of recommendations. The main contributions of our work can be outlined as follows:

• A user perception-guided graph convolutional network for multimodal recommendation is proposed, which combines the multimodal features of items into the GCN model and utilizes the user–item interaction graph and GCN to extract the modal preferences of users better. It obtains a more accurate user representation and effectively fuses the item representation in different modalities.
• A user perception-guided representation enhancement module (UPEM) is developed, which first utilizes the different modal features of the item as inputs, performs graph convolution on the user–item interaction graph to obtain user representations in different modalities, and then utilizes the user representation as a guidance signal to augment the embedding of user ID, thus obtaining a more accurate user representation.
• A multimodal two-step enhanced fusion method is proposed, which extracts useful information from item features with user modality preference and the original item’s multimodal features for enhancing item representation.
• We performed comprehensive experiments on three public datasets to demonstrate the effectiveness of our proposed model.

The rest of this paper is organized as follows. We introduce related work in Section 2. Section 3 presents a model of the user perception-guided graph convolutional network for multimodal recommendation. In Section 4, we provide a detailed explanation of the experimental design, including the datasets used, evaluation metrics, and parameter settings, along with comparative experiments and results analysis against baseline methods. Additionally, ablation experiments and their results analysis, as well as experiments related to hyperparameters, are also conducted. Section 5 summarizes the model and method we have proposed and points out the future research goals.

2. Related Work

2.1. Deep Learning-Based Multimodal Models

With the popularity of deep learning, many multimodal recommendation approaches based on convolutional neural networks (CNNs) have been developed. For instance, DeepCoNN [27] introduces a novel deep learning architecture that leverages user and item reviews to improve recommendation accuracy. It utilizes two parallel CNNs, one for users and one for items, to extract features from review text, which are then combined through a shared layer to model user–item interactions. This approach addresses the data sparsity issue common in CF by incorporating rich semantic information from reviews. DVBPR [28] utilizes a Siamese CNN framework to extract visual features of items from its images, and these features serve as item representations for user preference prediction. The item representation generated by the CNN architecture can be trained end-to-end with the recommendation system. MRLM [29] addresses the challenges of heterogeneous and multimodal item descriptions in the Internet of Things (IoT) environment. It integrates global feature representation learning (GFRL) and multimodal feature representation learning (MFRL) to jointly capture user–item interactions and multimodal features. Additionally, to capture users’ different preferences for multimodal features, some multimodal recommendation models also use attention mechanisms. For example, VECF [30] addresses the problem that different users may have different focus points on the same item by using the VGG model to pre-segment the item image and employing attention mechanisms to capture the users’ attention on different image regions, which are used to enhance item representation in the image modality. ACF [31] incorporates component-level and item-level attention mechanisms into traditional collaborative filtering models to allocate attentive weights for deducing the underlying user preferences reflected in implicit feedback. Deep learning-based models have excellent recommendation performance, but these models mostly tend to focus more on the modal information of items and neglect the rich interaction information between users and items.

2.2. Graph-Based Multimodal Models

Graph neural networks (GNNs) have proven effective in learning representations for graph data across various domains [32,33,34]. The core concept of GNNs is to update the representation of the current node by aggregating the representations of its neighboring nodes during the propagation process. Recently, more and more researchers have introduced GNNs into recommendation systems. In collaborative filtering recommendations, there are only two entities: users and items. The relationships between them can be represented by an interaction graph, allowing for the application of GNNs in this recommendation context. Specifically, graph-based approaches represent users and items by transforming user interaction histories into an interaction graph between users and items.

In order to better utilize multimodal information and improve recommendation performance, multimodal recommendation methods have emerged. MMGCN [13] constructs user–item interaction graphs across multiple modalities, then employs GNN to combine information from neighboring nodes, fusing the user and item representations across multiple modalities, thereby enhancing the understanding of user preferences. Based on MMGCN, DualGNN [23] obtains user representations under different modalities through GNN and then establishes a user co-occurrence graph to capture users’ modal preferences across different modalities. By identifying false-positive edges and noisy edges, GRCN [14] has created a graph refining layer that further refines the interaction graph between users and items, therefore, preventing the propagation and aggregation of noise in the GNN. However, the aforementioned method overlooks the relationships between items. To more accurately obtain the relationship between items, LATTICE [24] establishes the item–item graphs for different modalities and then integrates them to obtain a latent item–item graph. Additionally, during the backpropagation process to optimize parameters, the item–item relationship graph will be updated. MICRO [35] is an extension of LATTICE, that proposes a contrastive modality fusion method to solve the inconsistency problem in multimodal information fusion by combining contrastive learning and latent structure mining. FREEDOM [25] finds that learning the potential structure was inefficient, so it freezes the updates of the item–item graphs across various modalities, allowing for better item semantic representation and improving the algorithm’s efficiency. BM3 [36] utilizes self-supervised learning methods to solve the negative sample sampling problem and the complexity problem of large-scale graph computation in multimodal recommendation. The model generates a contrastive view by using a simple dropout data augmentation technique, replacing the existing graph structure enhancement methods and thus reducing the computational and memory overhead. Following this line of research, we utilize graph convolutional networks and multimodal features to initially obtain the users’ modality preferences and then adaptively learn user representations based on these preferences to achieve more accurate user preferences and improve recommendation performance.

2.3. Multimodal Fusion

For multimodal recommendation systems, finding a multimodal fusion method that excludes modality similarity and can achieve information complementarity between modalities is crucial for enhancing recommendation performance. Multimodal fusion methods mainly include early fusion, hybrid fusion, and late fusion [37]. Early fusion refers to the process of merging the modal features extracted at the beginning and then inputting this combined feature representation into the model for processing. On the contrary, late fusion involves merging after the model has made decisions for each modality. Hybrid fusion combines these two methods. For example, UVCAN [38] uses an early fusion method based on attention mechanisms. In the late fusion approach, SLMRec [39] presents an innovative self-supervised learning framework for multimedia recommendation, leveraging graph neural networks alongside three novel self-supervised tasks: feature dropping, feature masking, and fine- and coarse-grained feature learning. By implementing these tasks, SLMRec generates multiple views that facilitate contrastive learning, thereby uncovering implicit relationships across different modalities and performing late fusion on the generated item views. CELFT [40] has developed a hybrid fusion method where fusion is performed both before and after modal features are encoded, combining early and late fusion together. However, these fusion methods only concatenate or add the item representations obtained from different modalities, failing to effectively distinguish the similarities among modal features and fully leverage their complementarity. Therefore, we propose a two-step enhanced fusion method that takes a strategy of first enhancing and then fusing item representations to effectively distinguish the similarity between multimodal features and fully utilize their complementarity.

3. Methods

In this section, we first provide some descriptive information about the proposed model and then introduce the model we propose, which consists of three main aspects: (a) the user perception-guided representation enhancement module, (b) multimodal two-step enhanced fusion method, and (c) the loss function used.

3.1. Relevant Description

3.1.1. User–Item Interaction Graph

In our research, we have users $U=\{u_1,u_2\dots u_X\}$ and items $I=\{i_1,i_2\dots i_Y\}$, where $X$ is the number of users and $Y$ is the number of items. We formalize the user–item interaction graph through the following formula:

$$G=\left\{\left(u,y_{ui},i\right)|\left(u\in U,i\in I\right)\right\}$$

(1)

where $U$ denote the user set and $I$ the item set, while $y_{ui}$ signifies the interaction between a user and an item. The equation defining $y_{ui}$ is presented below:

$$y_{ui}=\left\{\begin{array}{c}1,\mathrm{i}\mathrm{f}u\mathrm{i}\mathrm{n}\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{c}\mathrm{t}\mathrm{e}\mathrm{d}\mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}i;\hfill \\ 0,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\right.$$

(2)

When a user interacts (such as purchasing, liking, or browsing) with an item, $y_{ui}=1$, otherwise, $y_{ui}=0$. Follow the mainstream approach [13,24], we also embed the users and item in the interaction graph as ID embeddings $e_u,e_i\in {\mathbb{R}}^d$, where $e_u$ denotes the user ID embedding, $e_i$ denotes the item ID embedding, and $d$ denotes the dimension of the embedded vector.

3.1.2. Item–Item Graph

Following the approach outlined in [24], we utilize the k-nearest neighbors (kNN) method to construct an item–item graph $S^m$ for modality $m$ based on the original modal features. Initially, we assess the similarity between items by calculating the similarity of their respective original features, which results in the formation of a similarity matrix $S^m\in {\mathbb{R}}^{N\times N}$ representing the item–item graph. The cosine similarity function is employed to compute the values within the similarity matrix $S^m$:

$$S_{ab}^m={\displaystyle \frac{{\left(x_a^m\right)}^T{x}_b^m}{‖x_a^m‖‖x_b^m‖}}$$

(3)

where $S_{ab}^m$ refers to the similarity between item $a$ and $b$ in modality $m$. $x_a^m$ denotes the original features of item $a$ in modality $m$, and $x_b^m$ denotes the original features of item $b$ in modality $m$. To obtain more precise node representations and more relevant neighboring features in the execution of graph convolution procedures, we will use the kNN sparse method [41] to sparse the original adjacency matrix $S^m$, which means that for each item, only the top-k edges are retained:

$${\widehat{S}}_{ab}^m=\left\{\begin{array}{c}1,S_{ab}^m\in \mathrm{top-k}\left(S_a^m\right),\hfill \\ 0,\mathrm{o}\mathrm{t}\mathrm{h}\mathrm{e}\mathrm{r}\mathrm{w}\mathrm{i}\mathrm{s}\mathrm{e}.\hfill \end{array}\right.$$

(4)

where ${\widehat{S}}_{ab}^m$ is the element of the resulting sparse matrix ${\widehat{S}}^m$. Different from [24], we do not use the similarity between items as the element value of the sparse matrix. Instead, we follow the approach of [25] and set the top-k highest-correlated edges to 1, while setting the remaining lower-correlated or unrelated edges to 0. Furthermore, we normalize the discretized matrix ${\widehat{S}}^m$ to avoid the gradient explosion problem:

$${\tilde{S}}^m={\left(D^m\right)}^{-{\displaystyle \frac{1}{2}}}{\widehat{S}}^m{\left(D^m\right)}^{-{\displaystyle \frac{1}{2}}}$$

(5)

where $D^m\in {\mathbb{R}}^{N\times N}$ is the diagonal degree matrix of ${\widehat{S}}^m$ and $D_{aa}^m={\sum }_b{\widehat{S}}_{ab}^m$. At this point, we have obtained the normalized item–item graph ${\tilde{S}}^m$ for $m$ modality. The next step is to aggregate the item–item graphs across various modalities to obtain the latent item–item graph. Following [25], we also freeze the updates of the latent item–item graph. To be specific, we have modal set $M=\left\{v,t\right\}$ where $v$ and $t$, respectively, denote the visual modality and textual modality. The latent item–item graph can be derived from the normalized item–item graphs for different modalities using the following formula:

$$S={\alpha }_v{\tilde{S}}^v+{\alpha }_t{\tilde{S}}^t$$

(6)

where $S\in {\mathbb{R}}^{N\times N}$, ${\alpha }_v$, and ${\alpha }_t$ are the weight coefficients for the visual modality and textual modality, respectively. Additionally, we set ${\alpha }_t=1-{\alpha }_v$, and based on the findings of [25], we empirically set ${\alpha }_v$ to $0.1$.

3.2. Overview of the Proposed Model

Figure 1 illustrates the comprehensive structure of the UPGCN model. UPGCN primarily consists of the UPEM and the multimodal two-step enhanced fusion method, which work together to achieve more precise representations of users and items. First, the enhanced signal is obtained by enhancing the item ID embedding using the transformed multimodal features of the item as the enhancing signal, thus obtaining ${\overline{e}}_v$ and ${\overline{e}}_t$. Then, the user representation is obtained by performing graph convolution on the user–item interaction graph using the multimodal features of the item as input, obtaining $h_u^v$ and $h_u^t$. Subsequently, the item representation is obtained by performing graph convolution on the same interaction graph using the user representation $h_u^v$, $h_u^t$ as input, obtaining $h_i^v$ and $h_i^t$ in different modalities. These item representations, along with the enhanced item ID embeddings ${\overline{e}}_v$ and ${\overline{e}}_t$, are fed into the enhancement module for further processing and fusion, producing ${\overline{e}}_i$. Then, use the item representations $h_i^v$ and $h_i^t$ from different modalities as inputs, and perform graph convolution on the item–item graph in different modalities to obtain preliminary disentangled item representations ${\tilde{h}}_i^v$ and ${\tilde{h}}_i^t$ with reduced similarity. Finally, ${\overline{e}}_i$, and the item representation ${\tilde{h}}_i^v$, ${\tilde{h}}_i^t$ obtained in different modalities are fused, and the final item representation ${\tilde{e}}_i$ generated by the multimodal two-step enhanced fusion method is output. Furthermore, to effectively extract user preference information across different modalities, the user representations $h_u^v$ and $h_u^t$ derived from these modalities serve as guiding signals for enhancing the user ID embedding. An attention mechanism is employed to quantify the user’s focus on each modality’s information, which is subsequently integrated into the user ID embedding for subsequent embedding learning. Building upon this framework, embedding learning is conducted on the user–item interaction graph to derive final representations of users and items that facilitate accurate predictions of user preferences.

3.3. User Perception-Guided Representation Enhancement Module

To effectively leverage the multimodal features of the item and mine user preferences across various modalities, we developed the user perception-guided representation enhancement module, which enriches user representations and enhances recommendation performance.

First of all, given that the original modal features of items vary in dimensionality, a multilayer perceptron (MLP) is employed to map these different modal features into a unified, low-dimensional space:

$$h_m=x_m{W}_m+b_m$$

(7)

where $W_m\in {\mathbb{R}}^{d_m\times d}$ and $b_m\in {\mathbb{R}}^d$ represent the learnable transformation matrix and bias in the MLP, respectively. Here, $d_m$ denotes the dimensionality of the item features in $m$ modality, while $d$ corresponds to the dimension of the embedding vector. After applying the dimensionality transformation, all modal features are in the same dimension as the embedding vector.

After acquiring the modal features of the item following dimensionality transformation, we derive the user features of different modalities by aggregating the item modal features on the user–item interaction graph. Analogous to MMGCN [13], we employ the item modal features to conduct GCN on the interaction graph between users and items in order to capture the user features across different modalities. Similarly, upon acquiring the user modal features, these features can be employed as inputs to conduct GCN on the same interaction graph in order to derive the item features of different modalities. This approach effectively leverages user modality information and integrates it with item modality data, thereby facilitating the capture of user preferences across various modalities. Additionally, LightGCN [4] posits that the feature transformation and nonlinear transformation inherent in traditional GCN do not yield significant benefits for collaborative filtering tasks. Building on this premise, it streamlines the graph convolution operation for multimodal recommendation. In alignment with LightGCN [4], we eliminated the circular propagation of self-information, as well as the nonlinear transformation and feature transformation of neighboring nodes during the graph convolution operation. This approach aims to simplify the model while enhancing both inference speed and training efficiency. Consequently, the GCN operation can be defined as follows:

$$h_{u,m}^{(l+1)}={\sum }_{i\in {\mathcal{N}}_u}{\displaystyle \frac{1}{\sqrt{\left|{\mathcal{N}}_u\right|}\sqrt{\left|{\mathcal{N}}_i\right|}}}{h}_{i,m}^{\left(l\right)}$$

(8)

$$h_{i,m}^{(l+1)}={\sum }_{u\in {\mathcal{N}}_i}{\displaystyle \frac{1}{\sqrt{\left|{\mathcal{N}}_u\right|}\sqrt{\left|{\mathcal{N}}_i\right|}}}{h}_{u,m}^{\left(l\right)}$$

(9)

where ${\mathcal{N}}_u$ and ${\mathcal{N}}_i$ denote the neighborhoods of $u$ and $i$ within the interaction graph between users and items, respectively. $h_{u,m}^{\left(l\right)}\in {\mathbb{R}}^d$ and $h_{i,m}^{\left(l\right)}\in {\mathbb{R}}^d$, respectively, denote the representations of users and items derived from the modalities $m$ in the preceding layer. It is crucial to emphasize that $h_{i,m}^{\left(0\right)}=h_m$. Following the acquisition of user representations across various modalities, the subsequent task involves inputting these representations into the UPEM to extract users’ preferences for different modalities of information and to incorporate this preference data into the embedding of user ID. Initially, based on the user representations for each modality, the attention weights corresponding to the user’s preferences across different modalities are computed using the following formula:

$${\gamma }_u^m={softmax(q}_m^T\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(V_m{H}_m^u+b_m))$$

(10)

where $q_m\in {\mathbb{R}}^d$ is the attention vector, and $V_m\in {\mathbb{R}}^{d\times d}$ and $b_m\in {\mathbb{R}}^d$, respectively, represent the transformation matrix and bias vector; all of these are learnable parameters within a particular modality. Consequently, the attention weights across different modalities remain independent, indicating that the parameters between modalities do not share weights. $H_m^u\in {\mathbb{R}}^{d\times \left|U\right|}$ is the user representation matrix in modality m, which is obtained by concatenating the $h_{u,m}^{(l+1)}$ values for each user as derived from Equation (9). Then, by multiplying with the attention weights ${\gamma }_u^m$, the embedding of user ID ${\widehat{e}}_u^{\left(0\right)}$ that incorporates the user’s preference information is obtained:

$${\widehat{e}}_u^{\left(0\right)}={\sum }_{m\in M}{\gamma }_u^m{e}_u^{\left(0\right)}$$

(11)

where $M=\{v,t\}$ is the set of modalities, and $v$ and $t$ represent the visual modality and textual modality, respectively.

Finally, ${\widehat{e}}_u^{\left(0\right)}$ is normalized prior to executing the residual connections; an initial user representation is obtained:

$${\overline{e}}_u^{\left(0\right)}={\displaystyle \frac{{\widehat{e}}_u^{\left(0\right)}}{{‖{\widehat{e}}_u^{\left(0\right)}‖}_2}}+e_u^{\left(0\right)}$$

(12)

3.4. Multimodal Two-Step Enhanced Fusion Method

To effectively differentiate the similarities among multimodal features while fully utilizing their complementarity, we propose the multimodal two-step enhanced fusion method. This approach employs a strategy of pre-enhancement followed by fusion, wherein the embedding of item ID is initially enhanced based on modal features across different modalities to separate the similarities between these diverse modal features. The second enhanced signal is derived from Equation (9), which aims to enhance the item ID embedding by utilizing the item modality information $h_{i,m}^{(l+1)}$ after fusing the user modality information. In summary, the enhancement process involves extracting modality-specific information from both the original item modal features and the item modal features that incorporate user modality features, subsequently reflecting this information in the embedding of item ID.

Specifically, after deriving the modal feature $h_m$ with dimensions matching those of the ID embedding using Equation (7), we utilize this modality feature $h_m$ as a guiding signal to enhance the initial embedding of item ID $e_u^{\left(0\right)}$, thereby distinguishing similarities between modalities and emphasizing the unique characteristics inherent to each modality. The calculation equation is as follows:

$${\overline{e}}_m=e_i^{\left(0\right)}⨀\sigma (h_m{W}_{id}^{item}+b_{id}^{item})$$

(13)

where $W_{id}^{item}\in {\mathbb{R}}^{d\times d}$ and $b_{id}^{item}\in {\mathbb{R}}^d$ represent the transformation matrix and bias, respectively, both of which are learnable parameters, $\sigma$ denotes the sigmoid function, while $⨀$ represents the element-wise product. Guided by modal feature $h_m$, the initial embedding of item ID $e_i^{\left(0\right)}$ was enhanced to yield an improved embedding of item ID ${\overline{e}}_m$.

Following the calculation of user modal information $h_{u,m}$ using Equation (7) and given that we continue to utilize $h_{u,m}$ as the neighbor node, the item modal information $h_{i,m}^{(l+1)}$ obtained through the GCN aggregation of neighboring nodes contains certain user preference information. To fully leverage this information, we opt to enhance ${\overline{e}}_m$ using item modal information $h_{i,m}^{(l+1)}$ after the initial enhancement step. The enhancement process is analogous to the methodologies presented in Equations (10) and (11), wherein $h_{i,m}^{(l+1)}$ serves as the input for computing the attention weights ${\gamma }_i^m$, followed by a weighted summation of ${\overline{e}}_m$ to derive ${\overline{e}}_i$:

$${\overline{e}}_i={\sum }_{m\in M}{\gamma }_i^m{\overline{e}}_m$$

(14)

To further distinguish the similarities between items across different modalities and facilitate subsequent multimodal fusion, we apply graph convolution to the item–item graph across various modalities established in Section 3.1.2, thereby obtaining a more precise representation of an item within a single modality. The graph convolution executed on the item–item relationship graphs across various modalities is defined as follows:

$${\tilde{h}}_{i,m}^{(l+1)}={\sum }_{j\in {\mathcal{N}}_i}{\tilde{S}}_{ij}^m{\tilde{h}}_{j,m}^{\left(l\right)}$$

(15)

where ${\mathcal{N}}_i$ denotes the set of all neighboring nodes for item $i$, ${\tilde{h}}_{j,m}^{\left(l\right)}$ signifies the node representation of item $j$ within the graph convolutional network at layer $l$ in modality $m$, and ${\tilde{h}}_{j,m}^{\left(0\right)}$ is derived from Equation (9) as the item modality information $h_{i,m}$.

To fully leverage the complementarity inherent in multimodal features, the subsequent step involves effectively integrating item information from various modalities to derive a unified item representation, thereby facilitating subsequent embedding learning and further uncovering user preferences.

Initially, we integrate the item representations ${\tilde{h}}_{i,m}$ derived from Equation (15) across different modalities. To achieve this, we employ an attention mechanism to perform self-enhancement on the item representation of each modality, thereby emphasizing the distinctive features of individual modalities and adaptively adjusting the importance of each modality to facilitate a more accurate item representation of in subsequent embedding learning. The definition of self-enhancement and the fusion process for multimodal item representations is as follows:

$${\epsilon }_i^m={softmax(q}_{i,m}^T\mathrm{t}\mathrm{a}\mathrm{n}\mathrm{h}(W_{i,m}{\tilde{h}}_{i,m}+b_{i,m}))$$

(16)

$${\tilde{h}}_i={\sum }_{m\in M}{\epsilon }_i^m{\tilde{h}}_{i,m}$$

(17)

where $W_{i,m}\in {\mathbb{R}}^{d\times d}$ and $b_{i,m}\in {\mathbb{R}}^d$ denote the transformation matrix and bias, respectively, $q_m\in {\mathbb{R}}^d$ is the attention vector, and ${\epsilon }_i^m$ denotes attention weight for modality $m$, which reflects the importance level of modality $m$ in item $i$ as derived from the multimodal representation. We employ Equation (17) along with this attention weight to perform a weighted fusion of the modality representation ${\tilde{h}}_{i,m}$ for the item. Following the acquisition of the self-enhanced multimodal item representation ${\tilde{h}}_i$ (Equation (17)) and the enhanced item behavior representation ${\overline{e}}_i$ (Equation (14)), the subsequent step involves fusing these representations to produce an initial item representation ${\tilde{e}}_i$ used for embedding learning. The fusion process is defined as follows:

$${\tilde{e}}_i={\lambda }_i{\overline{e}}_i+(1-{\lambda }_i){\tilde{h}}_i$$

(18)

where ${\lambda }_i$ represents the fusion weight, serving as a hyperparameter within UPGCN, and its value determines the balance between ${\overline{e}}_i$ and ${\tilde{h}}_i$ during the final fusion process.

3.5. Model Prediction

Following the completion of module propagation as outlined in Section 3.3 and Section 3.4, we acquire the initial user representation ${\overline{e}}_u^{\left(0\right)}$ (Equation (12)) and the initial item representation ${\tilde{e}}_i$ (Equation (18)). Subsequently, we utilize the initial user representation ${\overline{e}}_u^{\left(0\right)}$ along with the initial embedding of item ID $e_i^{\left(0\right)}$ to conduct graph convolution on the interaction graph between users and items, thereby obtaining the final user representation $y_u$ and the transitional item representation $e_i$. Subsequently, we will integrate $e_i$ and ${\tilde{e}}_i$ (Equation (18)), with the integration process defined as follows:

$${\overline{e}}_i={\displaystyle \frac{{\tilde{e}}_i}{{‖{\tilde{e}}_i‖}_2}}+e_i$$

(19)

Finally, we utilize ${\overline{e}}_i$ as input to apply GCN on the latent item–item graph $S$, constructed in Section 3.1.2, employing the GCN operation defined as follows:

$${\widehat{e}}_i^{(l+1)}={\sum }_{j\in {\mathcal{N}}_i}{S}_{ij}{\widehat{e}}_j^{\left(l\right)}$$

(20)

where ${\mathcal{N}}_i$ denotes the set of all neighboring nodes for item $i$, ${\widehat{e}}_i^{\left(0\right)}$ is initialized as ${\overline{e}}_i$, and the item–item latent graph $S$ output item representation is denoted as ${\widehat{e}}_i$. The final item representation $y_i$ is obtained by fusing ${\widehat{e}}_i$ with the transitional item representation $e_i$ output from the user–item interaction graph. The integration operation defined as follows:

$$y_i={\widehat{e}}_i+e_i$$

(21)

At this point, we have obtained the final user representation $y_u$ and the final item representation $y_i$ for prediction. To generate suggestions for users, we first compute the inner product between $y_u$ and $y_i$ to evaluate the affinity between the user and the candidate items:

$$s(y_u,y_i)={\left(y_u\right)}^T{y}_i$$

(22)

Subsequently, the candidate items are ordered in descending fashion according to the predicted affinity scores, and the top K items are selected as recommendations for the user.

3.6. Optimization

To optimize the recommendation task parameters in UPGCN, we utilize the BPR loss [1] as the primary optimization objective, which assumes that users are more inclined to select items they have previously interacted with rather than those they have not:

$${\mathcal{L}}_{bpr}={\sum }_{(u,i,j)\in D}-log\sigma (y_u^T{y}_i-y_u^T{y}_j)$$

(23)

where $D=\left\{\right(u,i,j\left)\right|(u,i)\in P,(u,j)\in N\}$ denotes the triplet instances within the training set, $u$ represents a user, $i$ is an item the user $u$ has interacted with, and $j$ is an item the user $u$ has not interacted with. The set $P$ includes positive samples, while $N$ contains negative samples, and $\sigma$ represents the sigmoid function.

To effectively preserve the modal features that have a significant influence on user preferences, we developed a multimodal BPR loss aimed at optimizing the parameters utilized in the modal feature extraction process (Equation (7)):

$${\mathcal{L}}_{mmbpr}={\sum }_{(u,i,j)\in D}{\sum }_{m\in M}-log\sigma (y_u^T{h}_m^i-y_u^T{h}_m^j)$$

(24)

where $h_m^i$ denotes the modal features of positive samples and $h_m^j$ denotes the modal features of negative samples. Ultimately, combining the BPR loss, we derive the final loss function:

$$\mathcal{L}={\mathcal{L}}_{bpr}+{{\lambda }_{reg}\mathcal{L}}_{mmbpr}$$

(25)

where ${\lambda }_{reg}$ is a hyperparameter in UPGCN that serves to weight the significance of the multimodal BPR loss in $\mathcal{L}$.

4. Experiments

4.1. Datasets

We select three categories from the Amazon review dataset [42] for our experimental evaluation: (a) Baby, (b) Sports and Outdoors, and (c) Clothing, Shoes and Jewelry, abbreviated as Baby, Sports, and Clothing. These datasets provide both textual and visual modalities for items and are publicly available. The size of the datasets for different item categories varies. The statistical information for these three datasets is presented in

Table 1. Statistics of three datasets used in experiments.

Dataset 1	Users	Items	Interactions	Sparsity
Baby	19,445	7050	160,792	99.88%
Sports	35,598	18,357	296,337	99.95%
Clothing	39,387	23,033	278,677	99.97%

¹ Dataset can be accessed at http://jmcauley.ucsd.edu/data/amazon/links.html (accessed on 4 November 2024).

, where data sparsity $\rho$ is defined as follows:

$$\rho =\left(1-{\displaystyle \frac{\left|A\right|}{\left|U\right|\times \left|I\right|}}\right)\times 100\%$$

(26)

where $\left|U\right|$, $\left|I\right|$, and $\left|A\right|$ denote the number of users, the number of items, and the number of interactions, respectively. Regarding modality features, we utilize the pre-extracted 4096-dimensional visual features and 384-dimensional text features that are available in [24].

4.2. Baselines

To demonstrate the effectiveness of the proposed model, we compare it with several representative baseline recommendation models in two categories. The first category consists of general collaborative filtering (CF) models, which rely solely on historical interaction relationships between users and items to generate recommendations. The second category encompasses several representative multimodal models that leverage both historical interaction information and the multimodal features of items to provide recommendations for users.

1.

General CF Models:

• BPR [1] is a classic collaborative filtering method that optimizes the latent representations of users and items within a matrix factorization (MF) framework through a BPR loss;
• LightGCN [4] is a simplified graph convolution network that omits feature transformation and nonlinear activation layers, focusing on linear propagation and neighbor aggregation.

2.

Multimodal Models:

• VBPR [5] employs the CNN to obtain visual features from product images, subsequently integrating these visual features and ID embeddings of each item as its representation.
• MMGCN [13] constructs the user–item interaction graphs for different modalities based on the features of different modalities and user–item interaction information. It subsequently employs graph convolutional on this graph to obtain user and item representations in different modalities. The user and item representations utilized for the final recommendation are derived from concatenating the user and item representations learned from the above steps across different modalities.
• GRCN [14] has designed a graph refining layer that further refines the interaction graph between user and items by detecting false-positive edges and noisy edges, thereby preventing the propagation and aggregation of noise in the GNN.
• DualGNN [23] initially extracts user representations from the user–item interaction graph across various modalities, subsequently integrates these representations, and constructs an additional user co-occurrence graph.
• LATTICE [24] constructs distinct modality-specific item–item graphs and obtains a latent item–item graph by integrating item–item graphs from all modalities.
• SLMRec [39] is a self-supervised learning framework for multimedia recommendation, which designs three different granularity data augmentation methods to build auxiliary tasks for contrastive learning.
• FREEDOM [25], based on the LATTICE framework, freezes the graph prior to training and employs a degree-sensitive edge pruning method to reduce noise in the interaction graph between users and items.

4.3. Evaluation Metrics

We employed the widely recognized metrics Recall@K and NDCG@K, abbreviated as R@K and N@K, to assess the recommendation effectiveness of UPGCN and baseline models. In the experiments, we present empirical results for K values of 10 and 20 on the test set.

Recall@K is a metric in recommendation systems that evaluates the proportion of actual relevant items in the top-K recommended items. It measures the model’s effectiveness in retrieving items that a user is likely to find interesting among the initial K suggestions. The formula for calculating Recall@K is presented below:

$$\mathrm{Recall}@\mathrm{K}={\displaystyle \frac{\left|{Rel}_u\cap {Rec}_u\right|}{\left|{Rel}_u\right|}}$$

(27)

where ${Rel}_u$ denotes the set of items that user $u$ has interacted with, ${Rec}_u$ denotes the set of the top K recommended items for user $u$.

NDCG@K is a metric employed by recommendation systems to assess the quality of the top-K recommended items. The first step involves calculating the Discounted Cumulative Gain (DCG), which is based on the rankings and relevance scores of pertinent items found in the recommendations. The formula is as follows:

$${DCG}_u@\mathrm{K}={\sum }_{i=1}^K{\displaystyle \frac{2^{r\left(i\right)}-1}{{\log }_2(i+1)}}$$

(28)

where $r\left(i\right)$ denotes the relevance score of the $i$-th recommended item. If the recommended list matches item $i$, then $r\left(i\right)$ is 1, otherwise it is 0. Subsequently, the calculation of ${IDCG}_u@\mathrm{K}$ will be conducted:

$${IDCG}_u@\mathrm{K}={\sum }_{i=1}^K{\displaystyle \frac{1}{{\log }_2(i+1)}}$$

(29)

The definition of NDCG@K is as follows:

$${NDCG}_u@\mathrm{K}={\displaystyle \frac{{DCG}_u@K}{{IDCG}_u@K}}$$

(30)

$$NDCG@\mathrm{K}={\displaystyle \frac{{\sum }_{u\in U^{te}}{NDCG}_u@K}{\left|U^{te}\right|}}$$

(31)

where $U^{te}$ is the set of all users in the test dataset, and $\left|U^{te}\right|$ is the number of users in the test dataset.

4.4. Parameter Setup

For a fair comparison, the embedding size was fixed at 64 across all models, with the embedding and network parameters initialized using the Xavier method [43], while Adam [44] served as the optimizer. Following the existing methods [24,25], we fixed the number of GCN layers in the user–item interaction graph and item–item graph at $L_{ui}=2$ and $L_{ii}=1$, respectively. Additionally, the optimal hyperparameters were identified through grid searches conducted on the validation data, specifically focusing on the fusion weight ${\lambda }_i$ in $\{0.35,0.40,0.45,0.50,0.55\}$ and the weight of multimodal BPR loss ${\lambda }_{reg}$ in {1 × 10⁻⁵, 1 × 10⁻⁴, 1 × 10⁻³, 0.01, 0.1}. We set the early stopping and total epochs to 20 and 1000, respectively, utilizing Recall@20 on the validation data as the indicator for terminating training.

4.5. Performance Comparison with Baselines

Table 2. Performance comparison of our UPGCN with baseline models (adapted from ref. [25]). Best results are underlined. The line below LightGCN serves as the boundary between the two types of methods mentioned in Section 4.2.

Dataset	Model	R@10	R@20	N@10	N@20
Baby	BPR	0.0357	0.0575	0.0192	0.0249
	LightGCN	0.0479	0.0754	0.0257	0.0328
	VBPR	0.0423	0.0663	0.0223	0.0284
	MMGCN	0.0421	0.0660	0.0220	0.0282
	GRCN	0.0532	0.0824	0.0282	0.0358
	DualGNN	0.0513	0.0803	0.0278	0.0352
	SLMRec	0.0521	0.0772	0.0289	0.0354
	LATTICE	0.0547	0.0850	0.0292	0.0370
	FREEDOM	0.0627	0.0992	0.0330	0.0424
	UPGCN	0.0664	0.1043	0.0351	0.0449
	Improv. (%)	4.31	4.53	5.76	5.42
Sports	BPR	0.0432	0.0653	0.0241	0.0298
	LightGCN	0.0569	0.0864	0.0311	0.0387
	VBPR	0.0558	0.0856	0.0307	0.0384
	MMGCN	0.0401	0.0636	0.0209	0.0270
	GRCN	0.0599	0.0919	0.0330	0.0413
	DualGNN	0.0588	0.0899	0.0324	0.0404
	SLMRec	0.0663	0.0990	0.0335	0.0421
	LATTICE	0.0620	0.0953	0.0335	0.0421
	FREEDOM	0.0717	0.1089	0.0385	0.0481
	UPGCN	0.0740	0.1114	0.0402	0.0498
	Improv. (%)	3.21	2.29	4.41	3.53
Clothing	BPR	0.0206	0.0303	0.0114	0.0138
	LightGCN	0.0361	0.0544	0.0197	0.0243
	VBPR	0.0281	0.0415	0.0158	0.0192
	MMGCN	0.0227	0.0361	0.0120	0.0154
	GRCN	0.0421	0.0657	0.0224	0.0284
	DualGNN	0.0452	0.0675	0.0242	0.0298
	SLMRec	0.0442	0.0659	0.0241	0.0296
	LATTICE	0.0492	0.0733	0.0268	0.0330
	FREEDOM	0.0629	0.0941	0.0341	0.0420
	UPGCN	0.0644	0.0959	0.0348	0.0428
	Improv. (%)	2.38	1.91	2.05	1.90

illustrates a comparison of the recommendation performance between the presented methods and our proposed model, UPGCN. Specifically, we used FREEDOM as the baseline method for comparison. UPGCN made an improvement in effectiveness compared to FREEDOM, which is expressed as a percentage of the improvement. The table reveals the subsequent observations:

The UPGCN model outperforms the presented models in terms of recommendation performance on each dataset. Specifically, UPGCN improves FREEDOM by 4.31%, 3.21%, and 2.38% in terms of R@10 on Baby, Sports, and Clothing, respectively. This indicates that our proposed model is well-designed for the multimodal recommendation. Specifically, to capture users’ preference information for different modalities, we adopted a user-perception-guided representation enhancement module to obtain more accurate user representations. Additionally, by using a multimodal two-step enhanced fusion method, we separated the similarity between different modalities and highlighted the features of a single modality, thus avoiding contamination from modality noise and fully utilizing their complementarity to further enrich the item representation.

The integration of multimodal information into the recommendation system has substantially improved the model’s effectiveness in generating recommendations. For example, VBPR, which utilizes visual information of items, achieves a 15.3%, 31.1%, and 36.9% performance improvement in terms of R@20 compared to BPR on the Baby, Sports, and Clothing datasets, respectively. Furthermore, graph-based models (e.g., LightGCN) have also achieved improved recommendation performance by combining multimodal information. For example, DualGNN uses LightGCN to aggregate node information in the interaction graph between users and items and the user co-occurrence graph in each modality to obtain user and item representations, which have better recommendation performance than LightGCN. Moreover, most graph-based models have better performance than general models (e.g., BRP). However, as shown in Table 2, the MMGCN based on LightGCN has not achieved better performance than LightGCN. We believe the reason for this is that MMGCN directly fuses the user and item representations in different modalities, which summarizes all modalities’ features but ignores the important contribution of a single modality to improving performance. Meanwhile, multimodal information usually has similarities, so how to distinguish these similarities to highlight the single-modality features is crucial. Therefore, we designed a two-step enhanced fusion method to distinguish the similarities among multimodal features and fully utilize their complementarity. As a result, the model we proposed, UPGCN, has attained best recommendation performance.

User modal preferences exert a substantial influence on recommendation performance. Building upon MMGCN, DualGNN constructs a user co-occurrence graph to extract user preferences across various modalities, thereby significantly enhancing model’s performance. However, some models overlook the critical role of user preference information in a specific modality for mining more accurate user representations. LATTICE constructs item–item graphs in different modalities and a latent item–item graph to augment item representations in each modality. FREEDOM introduces improvements to LATTICE by freezing the item–item graph and denoising the user–item interaction graph. The aforementioned models primarily emphasize methodological enhancements to improve item representations; however, it inadequately accounts for user preferences in different modalities, resulting in the model’s failure to accurately capture user representations. Consequently, we developed UPEM in response to this issue, aiming to extract users’ preference information across various modalities to enhance their representations. As shown in Table 2, our model exhibits competitive recommendation performance, thereby validating the effectiveness of the proposed method. However, if the extraction of multimodal features from the items is insufficient, the recommendation performance of our proposed UPGCN may be weaker than that of other state-of-the-art approaches. This is because we need to use the modal features to obtain user modal representations across different modalities, thereby capturing users’ preferences for various modalities. If the initially extracted multimodal features are not accurate enough, it may affect the subsequent embedding learning. In contrast, other methods primarily use modal features as auxiliary information, which makes them less susceptible to this issue.

4.6. Ablation Studies

To thoroughly examine the impacts of various factors, we conducted ablation studies on both the modules of UPGCN and their modality features.

4.6.1. Effect of Multimodal Features

We evaluated the recommendation performance of UPGCN by feeding each modal feature into the model sequentially. Specifically, we designed the following UPGCN variants:

• UPGCN_w/o-v&t: In this variant, UPGCN eliminates the input of two modalities, thereby transforming into a general recommendation model;
• UPGCN_w/o-v: This variant denotes that UPGCN utilizes exclusively the textual modality of the items;
• UPGCN_w/o-t: This variant indicates that UPGCN has eliminated the textual modality of the item, thereby utilizing solely its visual modality.

As is shown in

Table 3. Ablation study of UPGCN on multimodal features.

Dataset	Variants	R@10	R@20	N@10	N@20
Baby	UPGCNw/o-v&t	0.0479	0.0754	0.0257	0.0328
	UPGCNw/o-t	0.0527	0.0827	0.0282	0.0359
	UPGCNw/o-v	0.0616	0.0984	0.0328	0.0422
	UPGCN	0.0664	0.1043	0.0351	0.0449
Sports	UPGCNw/o-v&t	0.0569	0.0864	0.0311	0.0387
	UPGCNw/o-t	0.0610	0.0931	0.0332	0.0415
	UPGCNw/o-v	0.0716	0.1079	0.0391	0.0484
	UPGCN	0.0740	0.1114	0.0402	0.0498
Clothing	UPGCNw/o-v&t	0.0340	0.0526	0.0188	0.0236
	UPGCNw/o-t	0.0430	0.0633	0.0229	0.0281
	UPGCNw/o-v	0.0616	0.0927	0.0336	0.0415
	UPGCN	0.0644	0.0959	0.0348	0.0428

, in all evaluation metrics (R@10, R@20, N@10, and N@20) and datasets (Baby, Sports, and Clothing), UPGCN_w/o-v, which only utilizes textual features, achieved higher recommendation performance than UPGCN_w/o-t, which only relies on visual features, indicating that the text modality contains more information. Furthermore, the two variants that integrate single-modal features demonstrate superior recommendation performance compared to UPGCN_w/o-v&t, which does not incorporate any modal features, thereby further substantiating the significant role of multimodal features. Additionally, the modal features derived from the texts and images of products in the Sports dataset are less informative compared to those obtained from the Baby and Clothing datasets. The evidence is that the average performance improvement of UPGCN_w/o-t in the Sports dataset is 7.23%, while it is 9.71% and 21.91% in the Baby and Clothing datasets, respectively. In the Clothing dataset, characterized by the highest level of data sparsity (as is shown in Table 1), the performance improvement achieved through the integration of multimodal features is particularly significant. This further suggests that incorporating multimodal features within recommendation systems can effectively alleviate the data sparsity issue.

Finally, UPGCN utilizing multimodal features demonstrates higher recommendation performance compared to the other two variants employing single-modal features, indicating that the integration of multimodal features can significantly enhance UPGCN’s recommendation performance; thus, our proposed fusion method is essential.

4.6.2. Effect of Modules

To assess the effectiveness and robustness of the proposed modules, we decoupled UPGCN. Specifically, we separated the designed modules and methods from UPGCN to investigate their impact on recommendation performance; consequently, we constructed the following variants:

• UPGCN-B: This variant removes the UPEM and multimodal enhanced fusion method that we designed;
• UPGCN-E: This variant only uses UPEM to enhance the user representation, and removes the two-step enhanced fusion method;
• UPGCN-T: This variant only uses the multimodal two-step enhanced fusion method to enhance and fuse the item representations in different modalities, removing UPEM.

The comparison results for R@20 and N@20 are presented in Figure 2. By comparing the performance of UPGCN-E and UPGCN-B, it is evident that incorporating the UPEM has resulted in significant enhancements in both Recall@20 and NDCG@20 across all datasets. This indicates that UPEM has played a positive role in capturing user preference information and improving recommendation performance. Comparing UPGCN-T with UPGCN-B shows that the multimodal enhanced fusion method has also significantly improved the model’s performance in Recall@20 and NDCG@20. In particular, the multimodal enhanced fusion method has a particularly noticeable impact on improving recommendation performance in the Clothing dataset, further indicating that the method can effectively improve recommendation results on large-scale datasets.

UPGCN outperforms all other variants across all datasets regarding Recall@20 and NDCG@20, demonstrating its superior performance. This indicates that the combination of UPEM and multimodal enhanced fusion methods can synergistically enhance model performance. Moreover, the integration of these two methods is an important component of our model, which helps to better capture user preferences and effectively distinguish the similarity between multimodal features, thereby highlighting the single-modal features while fully utilizing the complementarity of different modalities to obtain a more accurate item representation and ultimately improve recommendation performance. Furthermore, in the Baby and Clothing datasets, UPGCN-T performs slightly better than UPGCN-E, indicating that the multimodal enhanced fusion method has a relatively greater impact on these two datasets. However, in the Sports dataset, UPGCN-T performs slightly worse than UPGCN-E in terms of Recall@20 and NDCG@20. We believe that one potential reason for this is that the Sports dataset extracts less information from product text descriptions and images compared to the other two datasets, and therefore, the two-step enhanced fusion method that utilizes multimodal features does not perform well in this dataset. However, despite the performance gap between UPGCN-T and UPGCN-E in the Sports dataset, their results remain relatively close to each other, further underscoring the significance of UPEM and the robustness of UPGCN.

4.7. Hyperparameter Sensitivity Study

4.7.1. Effects of the Fusion Weight ${\mathsf{\lambda }}_{\mathrm{i}}$

In our multimodal enhanced fusion method, the first enhancement utilizes dimensionally transformed modal features and item modal features that contain user modal preference information as guiding signals to extract the parts of item features that users particularly focus on. Subsequently, the enhanced item representations across different modalities are fused. Before the second fusion, the modal features undergo self-enhancement to highlight the single-modal features, thus distinguishing the similarities between different modalities and removing noise. Then, using Equation (18), the two enhanced results, ${\overline{e}}_i$ and ${\tilde{h}}_i$, are fused to fully leverage the complementarity of these two outcomes. Theoretically, this linear combination helps the model make full use of both types of information, thereby improving recommendation performance. As shown in Figure 3, in two datasets, by observing changes in Recall@20 and NDCG@20, when ${\lambda }_i=0.45$, both Recall@20 and NDCG@20 reached their maximum values. This suggests that in the Baby and Sports datasets, a moderate fusion weight can find the optimal balance between the two types of information, thereby maximizing recommendation performance. However, when ${\lambda }_i=0.35$ or ${\lambda }_i=0.55$, both Recall@20 and NDCG@20 slightly decreased, indicating that excessively high or low weights lead to reduced recommendation performance, indirectly suggesting that the two types of extracted information positively contribute to improving recommendation performance.

4.7.2. Effects of the Weight for Multimodal BPR Loss ${\mathsf{\lambda }}_{\mathrm{r}\mathrm{e}\mathrm{g}}$

In the experiment of hyperparameter ${\lambda }_{reg}$, we systematically explored the impact of different ${\lambda }_{reg}$ values on Recall@20 and NDCG@20, aiming to study the role of balancing multimodal BPR loss and standard BPR loss on recommendation performance. As shown in Figure 4, in the Baby and Clothing datasets, as ${\lambda }_{reg}$ is increased from 1 × 10⁻⁵ to 0.1 in steps, the changes in Recall@20 and NDCG@20 exhibit a complex nonlinear trend, reflecting the model’s sensitivity to the weight of the multimodal loss.

In the Baby dataset, the Recall@20 and NDCG@20 metrics show a noticeable fluctuation within a small range of ${\lambda }_{reg}$. Initially, Recall@20 is close to 0.1035 when ${\lambda }_{reg}$ is 1 × 10⁻⁵, and both Recall@20 and NDCG@20 slightly decrease as ${\lambda }_{reg}$ increases to 1 × 10⁻³, indicating that the model’s dependence on multimodal features has not been fully stimulated at this point. When ${\lambda }_{reg}$ is increased to 1 × 10⁻², both Recall@20 and NDCG@20 reach relatively high values simultaneously, with Recall@20 at 0.1043 and NDCG@20 at 0.0451. This result shows that the introduction of multimodal BPR loss with moderate weight strength effectively enhances the model’s recommendation performance, especially when user and item features from different modalities are fully fused. However, when ${\lambda }_{reg}$ is increased to 0.1, performance slightly declines again, indicating that the model may suppress multimodal features if it pays too much attention to feature extractor optimization, preventing the model from fully learning the multimodal information that is effective in predicting user preferences.

In the Clothing dataset, Recall@20 and NDCG@20 also show similar trends of change. When ${\lambda }_{reg}={1\times 10}^{-5}$, Recall@20 and NDCG@20 are 0.0946 and 0.0423, respectively. As ${\lambda }_{reg}$ increases to 1 × 10⁻⁴, Recall@20 slightly increases to 0.0959, which is the peak of Recall@20 on the Clothing dataset. For NDCG@20, the optimal value also appears at ${\lambda }_{reg}={1\times 10}^{-4}$, with NDCG@20 reaching 0.0428. This indicates that when ${\lambda }_{reg}$ is small, the model can better balance the standard BPR loss and multimodal BPR loss, allowing the model to effectively utilize multimodal features to enhance recommendation accuracy. However, when ${\lambda }_{reg}$ is further increased to 1 × 10⁻² and 0.1, Recall@20 and NDCG@20 both decrease, indicating that excessive optimization of the feature extractor has the opposite effect on the utilization of multimodal features.

Overall, the experimental results show that moderate ${\lambda }_{reg}$ can achieve a good balance between BPR loss and multimodal BPR loss, thereby improving the recommendation performance of the model. In particular, when ${\lambda }_{reg}$ is moderate, the model can fully utilize multimodal information to enhance the representation of users and items, thereby improving the recommendation accuracy. However, when ${\lambda }_{reg}$ is too large, the excessively high weight of the multimodal BPR loss may lead to the excessive suppression of multimodal features, thereby reducing the learning effect of the model. Therefore, the experimental results suggest that ${\lambda }_{reg}$ should be set to a medium range of values to ensure that the model can effectively integrate information across different modalities while avoiding the negative effects of excessively optimizing the feature extractor.

5. Conclusions

In this paper, we have proposed a user perception-guided graph convolutional network for multimodal recommendation. Specifically, we developed a user perception-guided representation enhancement module, which initially employs various modal features of the item as inputs, conducts GCN on the interaction graph between users and items to derive user representations across different modalities, and subsequently utilizes these representations as guidance signals to enhance the embedding of user ID, thereby obtaining a more accurate user representation by embedding learning. Furthermore, in order to effectively separate the similarity between different modalities and highlight the unique features of each modality while fully utilizing their complementarity, we design a multimodal two-step enhanced fusion method that takes a “first enhancement and then fusion” strategy for multimodal features to obtain a more accurate item representation for recommendation prediction. We conducted extensive experiments on three real-world datasets to evaluate the effectiveness of the proposed model in the recommendation task. The experimental results show that the proposed model outperforms the baseline model in terms of recommendation performance. Furthermore, we believe that UPGCN can be extended to other domains, such as the short video domain. In current collaborative filtering methods, the relationships between users and items are primarily used to identify items of interest to users. However, items typically possess multimodal features, such as images and reviews for products, as well as audio, video images, and comments in short videos. As long as the initial work effectively extracts the multimodal features of items using deep learning techniques, our model can utilize these features along with user–item historical interaction data as input, leading to effective recommendations. This is also one of the directions we aim to pursue in the future.

While UPGCN effectively uses multimodal features, it assumes that all modalities are available and of high quality. However, in real-world scenarios, certain modalities might be missing or noisy. Future work could address this limitation by developing methods to handle incomplete or unreliable multimodal data, perhaps through modality dropout or denoising mechanisms. Furthermore, the UPGCN model primarily focuses on the accuracy of recommendations but does not fully consider the efficiency of recommendations. The model’s graph convolutional structure, combined with the multimodal feature fusion, can lead to higher computational costs, particularly when scaling to very large graphs. Specifically, our model may involve more machine learning processes related to graphs, including user–item interaction graphs, item–item graphs in each modality, and latent item–item graphs, and the graph involved in real-world applications is much larger, which would result in higher computational costs when performing graph convolution operations on these large graphs. In future work, we could focus on optimizing the architecture to improve efficiency without sacrificing recommendation quality, perhaps through more lightweight graph processing techniques, pruning methods, or knowledge distillation techniques [45].