KETG: A KNOWLEDGE ENHANCED TEXT GENERA-TION FRAMEWORK

Abstract

Embedding logical knowledge information into text generation is a challenging NLP task. In this paper, we propose a knowledge enhanced text generation (KETG) framework, which incorporates both the knowledge and associated text corpus to address logicality and diversity in text generation. Specifically, we validate our framework on rhetorical text generation from our newly built rhetoric knowledge graph. Experiments show that our framework outperforms baseline models such as Transformer and GPT-2, on rhetorical type control, semantic comprehensibility and diversity.

1. INTRODUCTION

Recent pre-trained language models such as GPT-2 can capture clear semantic and syntactic features (Radford, 2018) , performing well in machine translation and abstract generation tasks (Li et al., 2016; Wang et al., 2016) . However, the application of language models in text generation still needs to be explored. The logic in text generation, especially literature creation, is always obscure, which means they are usually low-frequency, causing the difficulty of modeling by current language models. On the other hand, too much limits of prior information will lead to homogenization in generated texts. To address these issues, (Guan et al., 2020) proposes a knowledge-enhanced pretraining model for commonsense story generation, by transforming the commonsense triples into sentences using a template-based method. However, the template-based transformed sentences from commonsense triples for post-training are rather homogeneous. In this paper, we introduce an innovative knowledge enhanced text generation (KETG) framework, which incorporates knowledge tuples and their associated sentences in training, such that the logic relation lying in the knowledge tuples can be effectively addressed. Regarding the sentences associated with the knowledge tuples, we may generate the sentences from the tuples by template-based method as in (Guan et al., 2020) . However, incorporating real corpus sentences would be more beneficial as they generally exhibit more diversity than those generated from templates, if they are available. In this way, the generation model can learn the both logicality and diversity in the knowledge tuples and sentences. We validate our KETG framework on rhetorical text generation, which is an important and essential part in modern literature (Tu et al., 2013) . Rhetoric is quite obscure, requiring strong logical correlation, and a rhetoric knowledge graph with explicit logical information (rather than the commonsense knowledge graph) would be helpful to rhetorical text generation. Unfortunately, to the best of our knowledge, we are not aware of any rhetoric knowledge graph. Hence by using relation extraction methods, we build a rhetoric (specifically, here we refer to metaphor and personification) knowledge graph from a collection of Chinese poems and compositions. With the newly built rhetoric knowledge graph and the corpus from which the knowledge graph is extracted, we train a rhetorical text generation model. Both automatic and manual evaluations show that our KETG model outperforms baseline models on rhetorical type control, semantic comprehensibility and diversity. Experiments also illustrate that incorporating sentences by template-based method in training results in rather similar generated text as the template, while incorporating real corpus sentences brings more diversity in text generation. To sum up ,the main contributions of this paper are summarized as follows: 1. We propose a KETG framework, which includes both knowledge information and associated sentences in training to address logicality and diversity. 2. We validate our KETG framework on rhetorical (metaphor and personification) text generation. Results show that our KETG framework can generate more reasonable and diverse rhetorical texts, and the rhetoric types can be controlled implicitly. 3. To the best of our knowledge, we build the first Chinese rhetoric (metaphor and personification) graph with 35228 tuples.

2. RELATED WORK

Language Model(LM) In order to use as much semantic information as possible, several research work has been conducted. In early stage, researchers focused on the feature-based method to express syntactic and semantic information in texts. However, this kind of method can not solve the problem of polysemy. Knowledge Enhanced LM To mimic human's writing manner, the most basic thing is to ensure that the generated text fluent and semantically understandable. Secondly, the common sense of humankind is also indispensable. Furthermore, aesthetics and logicality make language expressions more vivid, novel and apt. However, it's hard to meet these requirements merely by language models. ( Bowman Rhetorical Text Generation Rhetoric is an important and essential part in modern literature (Tu et al., 2013) . It can express author's passion and grace, improving the aesthetic merit of creations. (Liu et al., 2019) proposed a rhetorically controlled generation model for Chinese poetry generation to govern the rhetorical modes. Through a classifier inserted in the encoder, they can control the rhetorical modes of generated poems. However, it does not include knowledge graph and hence might generate illogical sentences, like "Flakes of snow are flying like snow", which appears to be a metaphor, but includes illogical 'snow like snow'.

3. OUR KETG FRAMEWORK

We propose an innovative KETG framework, to combine the knowledge information with text generation models, just like the external device to computer. The architecture could be used to combine different types of knowledge graph with text generation model. As depicted in Figure 1 , we query the keyword in knowledge graph firstly, getting a context vector containing knowledge information. Then, we concatenate the context knowledge vector and the keyword vector, input them together with associated sentence to the language model. In this way, we can highlight the topic in the sentence and potential logical relationship between the entities, forcing the model pay more attention to them. When generating texts, with a given topic word, we get the context knowledge vector in the same way, which then serve as input to the trained model to generate the whole sentence in an auto-regressive manner. Compared with single topic word, the expanded context knowledge vector can also take the diversity advantage of knowledge graph, make sure the generated sentences full of variety. It's worth



et al., 2015)  used common-sense knowledge base in natural language inference(NLI) and NLG. As mentioned in(Zhou et al., 2018), common sense knowledge can promote performance in dialogue generation.(Mihaylov & Frank, 2018)  introduced a neural reading comprehension model that encodes external common sense knowledge as key-value memory.(Zhang  et al., 2019)  introduced a knowledge enhanced pre-trained language framework ERNIE, trying to increase the knowledge representation by masking semantic units such as words and entities.(Guan  et al., 2020)  proposes a knowledge-enhanced pretraining model for commonsense story generation. They post-train the model on knowledge-augmented data by transforming the commonsense triples into sentences.

To improve, (Peters et al., 2018)  Elmo is proposed to capture complex word characteristics in texts. Meanwhile, in NLP tasks, massive texts are often unlabeled. To solve this, fine-tuning models are raised, which can learn "common sense" from unlabeled texts. Both Bert and GPT-2 are representative models.(Wang et al., 2019; Ferreira et al., 2019)  They have achieved good evaluation results in multiple NLP tasks, such as named entity recognition, Q&A, text classification and text generation.

