Generative Transformer with Knowledge-Guided Decoding for Academic Knowledge Graph Completion

Abstract

Academic knowledge graphs are essential resources and can be beneficial in widespread real-world applications. Most of the existing academic knowledge graphs are far from completion; thus, knowledge graph completion—the task of extending a knowledge graph with missing entities and relations—attracts many researchers. Most existing methods utilize low-dimensional embeddings to represent entities and relations and follow the discrimination paradigm for link prediction. However, discrimination approaches may suffer from the scaling issue during inference with large-scale academic knowledge graphs. In this paper, we propose a novel approach of a generative transformer with knowledge-guided decoding for academic knowledge graph completion. Specifically, we introduce generative academic knowledge graph pre-training with a transformer. Then, we propose knowledge-guided decoding, which leverages relevant knowledge in the training corpus as guidance for help. We conducted experiments on benchmark datasets for knowledge graph completion. The experimental results show that the proposed approach can achieve performance gains of 30 units of the MRR score over the baselines on the academic knowledge graph AIDA.

1. Introduction

Knowledge graphs (KGs), also known as semantic networks, can represent much symbolic knowledge, including entities, relations, and events; thus, they appeal to many researchers. With the fast development of artificial intelligence (AI), knowledge graphs can provide back-end support for widespread tasks and be applied to real-world applications, including search engines [1,2], recommendation systems [3], medical health [4], natural language understanding [5,6,7,8], commonsense reasoning [9], time-serious prediction [10], and cross-discipline scenarios [11]. Note that most of the existing knowledge graphs are far from completion [12]; thus, knowledge graph completion—the task of extending a knowledge graph with missing entities and relations—attracts many researchers.

In the academic domain, knowledge graphs are essential resources due to their potential value for scientific research. With the fast development of knowledge graphs, we have witnessed many popular knowledge graphs, including OpenCitations [13], Core [14], Microsoft Academic Graph [15], Aminer [16], Open Research Knowledge Graph [17], and so on. These resources provide essential advantages for various research studies and can help research policymakers and funding agencies. Moreover, academic knowledge graphs can provide support for scientific literature research, can recommend related papers and authors, and can help discover future research trends. Nevertheless, with the fast evolution of the academic domain, most of those knowledge graphs are far from complete, which is a major obstacle for their application. For example, as shown in Figure 1, the relation type between “author: Yann Lecun” and “paper: Convolutional Networks and Applications in Vision” is missing, which makes the relational triple less informative and even ambiguous. Therefore, it is crucial for models to be able to complete missing entities/relations for academic knowledge graphs.

Conventionally, information extraction technologies [18,19,20,21] such as named entity recognition [22], relation extraction [23], and event extraction [24,25], can extract knowledge from a text corpus and help to complete the missing academic knowledge. However, when facing large-scale real-world academic knowledge graphs, it is necessary to develop efficient knowledge graph completion methodologies. Concretely, many knowledge-graph-embedding approaches (e.g., TransE [26], RotatE [27]) that utilize low-dimensional representations to embed entities and relations, which are dubbed knowledge graph embeddings, and obtain target predictions via score functions for those embeddings have been developed. More recently, some approaches (e.g., KG-BERT [28]) have tried to leverage pre-trained language models in natural language processing and encode relational triples with transformers to obtain the final score for a missing target. Note that most of those methodologies followed the discrimination paradigm with a pre-defined scoring function for knowledge graph representation learning. We argue that these discrimination approaches are time-consuming due to the costly scoring of candidate relational triples when inferring missing knowledge. Moreover, discrimination approaches lack rich interactions between relations and entities and suffer from the issue of the instability of negative sampling.

In this paper, to address the above-mentioned challenges, we propose a new technical solution for academic knowledge graph completion, which is named a generative transformer with knowledge-guided decoding (GTK). Specifically, we introduce generative academic knowledge graph pre-training with a transformer. We formulate knowledge graph completion as a sequence-to-sequence task and leverage BART-style [29] pre-training. We further propose knowledge-aware demonstrations inspired by GPT-3 [30], which concatenate some selected samples to the input sequence. Since there exist rich semantics and types of constraints for knowledge graph completion, we propose knowledge-guided decoding, which leverages relevant knowledge in the training corpus as guidance for help. We further utilize a policy gradient [31] to learn the generation strategy for maximizing the correctness of the generated knowledge and mitigating exposure bias.

We evaluate the proposed model GTK against previous baselines, including TransE [26] and KG-BERT [28], on the benchmark datasets of AIDA, MAG, and other popular knowledge graph completion datasets, namely, FB15K-237, WN18RR, and OpenBG500 [32,33].

The extensive experimental results illustrate that the proposed approach can yield better performance than that of the baselines with a faster inference speed. In total, we summarize the contributions of this work as follows:

• We propose a novel model of a generative transformer with knowledge-guided decoding (GTK) for academic knowledge graph completion.
• We propose knowledge-aware demonstration and knowledge-guided decoding for knowledge graph completion.
• We evaluate the model on various benchmark datasets for knowledge graph completion, which demonstrates the effectiveness of the proposed approach.

In the following sections, we will introduce the related work (Section 2) and background (Section 3). Then, we will introduce the technical details of the proposed approach, including link prediction as Seq2Seq generation (Section 4.1), knowledge-aware demonstration (Section 4.2), and knowledge-guided decoding (Section 4.3). Finally, we will introduce the experiments (Section 5) and conclude the paper (Section 6).

2. Related Work

Constructing knowledge graphs from scratch is intractable and requires the efficient completion of missing triples. Given the trait of incompleteness, knowledge graph completion (KGC) approaches can be divided into embedding-based methods and text-based methods.

In preliminary works, embedding-based methods showed many advantages in handling KGC tasks [34]. By converting triples into continuous embedding vectors, relations between entities are treated as special mapping functions that score pairs of top-k entities in a low-dimensional space. Such methods include TransE [26], TransR [35], and RotatE [27]. The Trans-series models, which are translation based, are equipped with both simplicity and strong interpretability in practice. Another type of embedding model, a semantic matching model, utilizes semantic similarity functions to capture the plausibility of latent triples in knowledge graphs. These methods emphasize the structural information observed in triples rather than contextualized information, and they include DistMult [36], ComplEx [37], Tucker [38], and Trans4E [39], which is an embedding model fit for academic knowledge graphs with N to M relations.

In contrast, text-based methods attempt to incorporate available texts for representation by leveraging state-of-the-art pre-trained language models [28,40,41,42]. When the architecture of a transformer occurs, it is employed in CoKE [43] to encode the path and edge sequences in graphs. Before long, pre-trained language models advanced contextualized text representation learning. KG-BERT [28], which took the first step in applying BERT [44] for KGC, concatenated the text of triples as sequences and handled the sequence classification problem with binary cross-entropy optimization. Referring to KG-BERT, StAR [45] followed the textual encoding paradigm and applied a Siamese-style textual encoder that partitioned each triplet into two asymmetric parts. kNN-KGE [46] derived knowledge graph embeddings by interpolating the entity distribution linearly from the knowledge store with the k-nearest neighbors (kNN) model. Apart from switching to the InfoNCE loss, SimKGC [47] introduced three types of negatives: pre-batch negatives, self-negatives, and in-batch negatives. LMKE [48] formulated a contrastive learning framework with the goal of augmenting long-tail entity representations with the inductive capabilities of description-based approaches. Motivated by prompt-based models, PKGC [49] manually defined each triple as a natural prompt sentence and further introduced soft prompts for better representation. Considering both the structure of a knowledge graph and the underlying textual description, StATIK [50] used language models to extract semantic information while incorporating a structure via message-passing neural networks. LASS [51], a joint embedding approach, embedded knowledge graphs via fine-tuned BERT or RoBERTa with respect to a probabilistic structured loss.

Rather than using discriminative methods, a line of recent work changed KGC into sequence-to-sequence generation problems. Posing link prediction as a sequence-to-sequence task, KGT5 [52] adopted a transformer model and was further fine-tuned; it used autoregressive decoding to reduce the model size by up to 98% in comparison with that of KGE models. For better representation learning and fast inference, GenKGC [33] proposed entity-aware hierarchical decoding to generate target entities. In addition, KG-S2S [53], which uniformly codified the representation of KG facts into “flat” text, was capable of handling a variety of verbalizable graph structures. However, most previous works regarded entities and relations as plain text, ignoring the structural bias when generating target knowledge, which hindered their knowledge graph completion performance.

3. Background

A knowledge graph represents the fact that knowledge exists in the form of a triple. In this paper, we target the knowledge graph completion task with the aim of predicting missing parts (entities or relations) based on existing triples.

A knowledge graph $G=(\mathcal{E},\mathcal{R})$ composed of an entity set $\mathcal{E}=\{e_1,e_2,e_3,\dots \}$ and an edge set $\mathcal{R}=\{r_1,r_2,r_3,\dots \}$ is given. The node set V consists of both entities and relations, that is, $V=\mathcal{E}\cup \mathcal{R}$. The graph structure can be represented as an adjacency matrix $\mathbf{A}\in {\{0,1\}}^{\left|V\right|\times \left|V\right|}$, where $\left|V\right|$ denotes the number of nodes in the graph. The element $\mathbf{A}[i,j]$ in the adjacency matrix equals 1 if there exists a link between nodes $v_i$ and $v_j$; otherwise, $\mathbf{A}[i,j]=0$, where i and j refer to the ID of the node v.

For the knowledge graph completion task, a triple is denoted as $(v_{src},v_p,v_{target})$, or $\mathcal{T}$ for short. We define the contextualized sub-graph ${\mathcal{T}}_G$ as the subgraph surrounding the triple $\mathcal{T}$ (including the triple itself). Meanwhile, the corresponding graph structure of the contextualized sub-graph sequence is preserved in matrix ${\mathbf{A}}_G$. ${\mathcal{T}}_M$ denotes the masked contextualized sub-graph. Therefore, the goal of knowledge graph completion is to learn a mapping $f:{\mathcal{T}}_M,{\mathbf{A}}_G\to \mathbf{Y}$, where $\mathbf{Y}\in {\mathbb{R}}^{\left|\mathcal{E}\right|+\left|\mathcal{R}\right|}$ are the label sets of the triple $\mathcal{T}$. f refers to the score function, which maps facts into embeddings to generate label sets.

3.1. Knowledge-Graph-Embedding-Based Methods

Knowledge-graph-embedding-based methods are the approaches for knowledge graph completion that have recently been dominant. Given a triple $(h,r,t)$, the key is to find the relationships between relations and entities via a score function as follows:

$$score(h,r,t)=f_r(h,t)$$

(1)

The relevance or distance is calculated with a pre-defined score function f, and there are many score functions [26,27,35]. It should be noted that the translational distance method is essentially a ranking model, and it requires negative samples to maximize the difference between the positive and negative triples. A straightforward approach to getting negative triples $(h^{\prime },r,t)$ or $(h,r,t^{\prime })$ is randomly replacing head or tail entities. Thus, we can optimize knowledge-graph-embedding-based knowledge graph completion as follows:

$$\begin{array}{c}L=-log\sigma (\gamma -scor{e}_{pos}(h,r,t))\hfill \\ -\sum log\sigma (scor{e}_{neg}(h,r,t)-\gamma )\hfill \end{array}$$

(2)

where $\gamma$ is a marginal hyper-parameter, and $\sigma$ is a sigmoid function.

3.2. Pre-Trained Language-Model-Based Methods

With the fast development of pre-training, there are many approaches that leverage pre-trained language models for the knowledge graph completion task. For example, KG-BERT [28] utilizes a single encoder to encode relational triples in a knowledge graph via text description. Formally, we have the score of the relational triple as follows:   

$$Score(h,r,t)=\mathrm{TransformerEnc}(X^h,X^r,X^t),$$

(3)

where TransformerEnc is the BERT [44] model followed by a binary classifier. Note that the previous studies mostly followed the discrimination paradigm; however, they may suffer from inefficiency and inflexibility issues.

There are several reasons for this: (1) A ranking model is not a straightforward way to predict the missing part of a triple, since it needs to repeatedly compute all scores of the candidates. (2) A score function is essentially an artificial constraint, and handcrafting the best-performing score function is like finding a needle in a haystack. (3) The negative sampling process is full of uncertainty with different batch sizes or sample difficulties. Thus, it is intuitive to develop a new solution for better knowledge graph completion, and the generation paradigm is a good fit for that role.

4. Approach

4.1. Link Prediction as Seq2Seq Generation

In this paper, we follow BART [29] in formulating knowledge graph completion as sequence-to-sequence generation. Specifically, we regard the entities and relations as token sequences and utilize an encoder–decoder architecture for generation. As shown in Figure 2, we follow [28] in leveraging text descriptions rather than specific embeddings to represent entities and relations. Given an input triple with a missing tail entity $(e_i,r_j,?)$, we get the descriptions $d{r}_j$ and $d_{e_i}$ of $r_j$ and $e_i$, respectively. Then, we concatenate these embeddings to obtain the input sequence of $<e_i,r>$. Note that we aim to generate the token sequence of $e_k$, which is similar to natural language generation. For example, given the text sequence regarding the Query <Steve Jobs, founded, ?>, we aim to generate a target entity with the text sequence Apple Inc. It is intuitive to pre-train the generation model to boost performance.

We leveraged a large-scale corpus (while filtering the triples, such as FB15K-237, during the evaluation to avoid data leakage) from Wikipedia and aligned it with the Wikidata knowledge graph for pre-training.We pre-trained the model with 16 A100 for one week. During inference, we used autoregressive generation, and more details can be found in Section 4.3.

4.2. Knowledge-Aware Demonstration

Inspired by GPT-3 [30], we propose the utilization of relevant triples in the training set as demonstrations to boost the generation performance. Regarding the long-tailed distribution in a large-scale knowledge graph, for example, only 37 instances of the relation film/type_of_appearance exist in the popular FB15k-237 dataset. We utilized the prior knowledge distribution of entities and relations as guidance for sampling demonstrations. To be specific, we first filtered the relations based on the input sequence and then sampled the triples according to the distribution of entity types. Thus, those long-tailed relational triples have more representative demonstrations for help. Finally, we can construct the demonstration examples $\{r_{\mathrm{in}},t_{\mathrm{train}}\}$ and obtain the final input sequence as follows:

$$x=<\mathtt{bos}>\mathrm{demonstration}\left(r_j\right)<\mathtt{sep}>d_{e_i}{d}_{r_j}<\mathtt{sep}>$$

4.3. Knowledge-Guided Decoding

During inference, we can use the Beam Search to obtain the top k entities in $\mathcal{E}$ (k is the hyper-parameter of the beam size). Note that there is no native sampling, as we directly optimize via the generation of the right entities. Since academic knowledge graphs contain rich semantic and schematic knowledge, such as entity types, we propose knowledge-guided decoding to boost performance. Firstly, we sample the lowest-frequency entity types and construct a trie tree due to the fact that these low-frequency entities are challenging to decode. We add some special tokens as entity types in the vocabulary of the language models for knowledge-guided decoding. We first utilize the standard cross-entry loss to optimize the log-likelihood for generation as follows:

$$\mathcal{L}=-log{p}_{\theta }(y\mid x)$$

(4)

Inspired by [54], we then formulate sequence generation as a reinforcement learning problem. The action is the current step’s output token (i.e., $a_t=y_t$), and the state is the same as that of the tokens generated before t. Then, we utilize the policy ${\pi }_t$ to pick token $y_t$ (action $a_t$) given the state $s_t=y_{<t}$ through multi-hop imagination to constrain the output of generation. We use a policy gradient [31] to learn optimized strategies.

5. Experiments

5.1. Datasets

We followed [39] and use the AIDA+MAG knowledge graph for evaluation. It contains 180 K triples with 68,906 entities, including affiliated organizations, authors, publications, and so on. The subset of the hasGRIDType relation contains research paper entities. The subset of the hasTopic relation mainly consists of triples of articles with associated topics. The datasets were split according to a ratio of 8/1/1 (80% for the training set, 10% for the validation set, 10% for the test set) to construct the training, validation, and test sets.

We also evaluated the proposed approach on FB15k-237, WN18RR, and a real-world knowledge graph for e-commerce, OpenBG500 (OpenBG500 is a subset of the open business KG from https://kg.alibaba.com/ accessed on 1 September 2022). For FB15k-237, we leveraged descriptions from the Wikipedia page for entities and relations. For WN18RR, each entity contained a word sense, and we leveraged the word definitions for descriptions. For OpenBG500, we leveraged the descriptions from the e-commerce description page for entities and relations. We illustrate the statistical details of FB15k-237, WN18RR, and OpenBG500 in

Table 1. Statistics of FB15k-237, WN18RR, and OpenBG500. # refers to the number.

Dataset	# Ent	# Rel	# Train	# Dev	# Test
AIDA	68,906	2	144,000	18,000	18,000
WN18RR	40,943	11	86,835	3034	3134
FB15k-237	14,541	237	272,115	17,535	20,466
OpenBG500	269,658	500	1,242,550	5000	5000

.

5.2. Metrics

We used the metrics of hits@1, hits@3, hits@10, and MRR for evaluation. We sorted the scores of the entities in the candidate set to obtain a ranked list. We obtained the hits at k metric (Hits@k) by counting the number of times that the correct triple appeared at position k. We defined $ran{k}_{\mathrm{head}}$ as the position of the correct entity in the rank list. We calculated the reciprocal rank with $1/ran{k}_{\mathrm{head}}$ and repeated the procedure by predicting the tail entity to obtain the reciprocal rank $1/ran{k}_{\mathrm{tail}}$. Additionally, the mean rank (MR) refers to the $ran{k}_{\mathrm{head}}$ and $ran{k}_{\mathrm{tail}}$ averaged across all triples in the KG. The mean reciprocal rank (MRR) refers to the mean of these two values. Formally, we have:

$$MRR=\frac{1}{2\left|\mathcal{T}\right|}\sum _{t\in \mathcal{T}}\left(\frac{1}{ran{k}_{\mathrm{head}}}+\frac{1}{ran{k}_{\mathrm{tail}}}\right)$$

(5)

It is to be noted that the entities in the test triple were assumed to be seen in the training set.

5.3. Settings

We used 16 Nvidia A 100 GPUs for model pre-training and fine-tuning. We developed our model with Pytorch and utilized the BART-base as a backbone. The learning rate was set to $5\times {10}^{-5}$. We leveraged a grid with a validation set search to find the optimized hyper-parameters. We also utilized early stooping for each dataset. Specifically, we recorded the test-set performance of the best-performing model on the development set for each random seed and reported the final performance as the average of the results across the five seeds.

5.4. Results

On the one hand, from

Table 2. Evaluation results on the AIDA academic knowledge graph.

Model Type	hasTopic					hasGRIDType
	MR	MRR	Hits@1	Hits@3	Hits@10	MR	MRR	Hits@1	Hits@3	Hits@10
TransE	3982	0.400	0.294	0.462	0.592	1	0.968	0.944	0.990	1.000
RotatE	4407	0.433	0.332	0.492	0.622	1	0.953	0.933	0.975	0.996
QuatE	1353	0.426	0.341	0.472	0.581	1	0.957	0.928	0.983	0.998
ComplEx	5855	0.099	0.077	0.109	0.129	1566	0.566	0.531	0.596	0.609
Trans4E	3904	0.426	0.318	0.492	0.628	1	0.968	0.944	0.995	0.998
GTK	3835	0.456	0.355	0.488	0.650	1	0.975	0.967	0.998	1

, we observed that on the subsets of hasTopic and hasGRIDType, the previous baselines TransE, RotatE, and QuatE were able to obtain better performance than that of ComplEx, while Trans4E was able to achieve better performance than all of them. We argue that Trans4E can better represent academic knowledge graphs with N to M relations, thus leading to better performance. We further found that the performance of knowledge-embedding models in the hasGRIDType subset was much higher than that in the hasTopic subset. We think that this was because the hasGRIDType subset is a relatively easy subset of academic knowledge graphs. We also noticed that the proposed GTK model was able to obtain the best performance on both academic knowledge graph datasets. On the other hand, note that previous knowledge-graph-embedding approaches regard relations and entities as dense vectors in the same space; thus, they struggle with the memory cost for large-scale academic knowledge graphs. However, our GTK has a fixed memory size with a pre-trained language model, which contains 110 M parameters, and will not scale with the size of entities.

From

Table 3. Evaluation results on WN18RR, FB15k-237, and OpenBG500. Note that the ⋄ resulting numbers were reported by previous works, which means that we reproduced the experimental results on OpenBG500 and took other results from the original papers.

	WN18RR			FB15k-237			OpenBG500
Method	Hits@1	Hits@3	Hits@10	Hits@1	Hits@3	Hits@10	Hits@1	Hits@3	Hits@10
Graph-embedding approach
TransE [26] ⋄	0.043	0.441	0.532	0.198	0.376	0.441	0.207	0.340	0.513
DistMult [36] ⋄	0.412	0.470	0.504	0.199	0.301	0.446	0.049	0.088	0.216
ComplEx [37] ⋄	0.409	0.469	0.530	0.194	0.297	0.450	0.053	0.120	0.266
RotatE [27]	0.428	0.492	0.571	0.241	0.375	0.533	-	-	-
TuckER [38]	0.443	0.482	0.526	0.226	0.394	0.544	-	-	-
ATTH [55]	0.443	0.499	0.486	0.252	0.384	0.549	-	-	-
Textual encoding approach
KG-BERT [28]	0.041	0.302	0.524	-	-	0.420	0.023	0.049	0.241
StAR [45]	0.243	0.491	0.709	0.205	0.322	0.482	-	-	-
GenKGC [33]	0.287	0.403	0.535	0.192	0.355	0.439	0.203	0.280	0.351
GTK	0.449	0.501	0.616	0.291	0.402	0.550	0.210	0.366	0.551

, we also observe that GTK obtained better performance than that of KG-BERT [28] on WN18RR, FB15k-237, and OpenBG500. Moreover, the proposed model had a faster inference speed, since it directly generated the target entities rather than engaging in the costly scoring of candidate relational triples. For example, when the number of target candidate entities was very large, it cost lots of time (KG-BERT took about 91,100 s) for inference, as shown in

Table 4. Training and inference speed comparison. $\left|d\right|$ refers to the length of the entity description. k refers to the number of negative samples of KG-BERT and the beam size for the proposed model. $\left|\mathcal{E}\right|$ refers to the number of unique entities in the knowledge graph. We used time under A100 to estimate the training and inference speed on OpenBG500.

For One Triple	Method	Complexity	Time under A100
Training	KG-BERT	$O\left(\right|d{|}^2\times (k+1))$	72 ms
	GTK	$O\left(\right|d{|}^2)$	2.01 ms
Inference	KG-BERT	$O\left(\right|d{|}^2\times \left|\mathcal{E}\right|)$	91,100 s
	GTK	$O\left(\right|d{|}^2\times \left|d{|}^k\right)$	0.73 s

. However, the proposed model only had to generate the top k entities with knowledge-guided decoding, which was faster.

5.5. Case Study

To further analyze the proposed model, we randomly sampled some instances and provide case studies. We provide the empirical results with and without knowledge-guided decoding, which indicates that we leveraged the vanilla Beam Search to generate the target entities. As shown in

Table 5. We list a query and the first five entities with their probabilities predicted by GTK without entity-aware decoding and the reranking with GTK.

Query:(?,student, Michael Chabon)
Rank	GTK w/o hierarchical decoding	Probability
1	University of California
5	University of California, Santa Cruz
2	University of California, Irvine
3	University of California, San Francisco
4	University of California, Davis
Rank	GTK	Probability
1	University of California, Irvine
2	University of California, San Francisco
5	University of Calgary
4	University of California, Santa Cruz
3	University of California, Davis

, we noticed that the proposed model could achieve better generation performance than that of a vanilla Beam Search. For example, the proposed GTK model could generate University of California, Irvine with the highest probability given the Query (?, student, Michael Chabon). Moreover, we found that the proposed model without knowledge-guided decoding could stop early with the correct—but not very precise—target entity. We think that this was due to the bias from the pre-trained language models, as high-frequency terms may lead to biased predictions. However, the model with knowledge-guided decoding could generate the correct decoding sequence due to prior knowledge.

6. Conclusions and Future Work

In this paper, we proposed a novel model of a generative transformer with knowledge-guided decoding for academic knowledge graph completion. Specifically, we proposed knowledge-aware demonstration and knowledge-guided decoding. We evaluated the proposed approach on four datasets. The extensive experimental results indicate that the proposed model can obtain better performance in academic knowledge graph completion and on popular benchmark datasets, and it can yield a faster inference speed than that of KG-BERT. The success of the proposed approach reveals that treating academic knowledge graph completion as a sequence-to-sequence generation task can be beneficial because knowledge can be inferred directly rather than engaging in the costly scoring of candidate triples. However, the proposed approach still has the limitations of trie tree construction.

In the future, we plan to explore the following: (1) how to obtain the suitable trie tree for candidate knowledge in decoding; (2) how to design specific pre-training objects for knowledge graphs.