CREATIVE SKETCH GENERATION

Abstract

Sketching or doodling is a popular creative activity that people engage in. However, most existing work in automatic sketch understanding or generation has focused on sketches that are quite mundane. In this work, we introduce two datasets of creative sketches -Creative Birds and Creative Creatures -containing 10k sketches each along with part annotations. We propose DoodlerGANa part-based Generative Adversarial Network (GAN) -to generate unseen compositions of novel part appearances. Quantitative evaluations as well as human studies demonstrate that sketches generated by our approach are more creative and of higher quality than existing approaches. In fact, in Creative Birds, subjects prefer sketches generated by DoodlerGAN over those drawn by humans!

1. INTRODUCTION

The true sign of intelligence is not knowledge but imagination. -Albert Einstein From serving as a communication tool since prehistoric times to its growing prevalence with ubiquitous touch-screen devices -sketches are an indispensable visual modality. Sketching is often used during brainstorming to help the creative process, and is a popular creative activity in itself. Sketch-related AI so far has primarily focused on mimicking the human ability to perceive rich visual information from simple line drawings (Yu et al., 2015; Li et al., 2018) and to generate minimal depictions that capture the salient aspects of our visual world (Ha & Eck, 2018; Isola et al., 2017) . Most existing datasets contain sketches drawn by humans to realistically mimic common objects (Eitz et al., 2012; Sangkloy et al., 2016; Jongejan et al., 2016; Wang et al., 2019) . In this work we focus on creative sketches. AI systems that can generate and interpret creative sketches can inspire, enhance or augment the human creative process or final artifact. Concrete scenarios include automatically generating an initial sketch that a user can build on, proposing the next set of strokes or completions based on partial sketches drawn by a user, presenting the user with possible interpretations of the sketch that may inspire further ideas, etc. AI for creative sketches is challenging. They are diverse and complex. They are unusual depictions of visual concepts while simultaneously being recognizable. They have subjective interpretations like aesthetics and style, and are semantically rich -often conveying a story or emotions. To facilitate progress in AI-assisted creative sketching, we collect two datasets -Creative Birds and Creative Creatures (Figure 1 ) -containing 10k creative sketches of birds and generic creatures respectively, along with part annotations (Figure 2 right columns). To engage subjects in a creative exercise during data collection, we take inspiration from a process doodling artists often follow. We setup a sketching interface where subjects are asked to draw an eye arbitrarily around a random initial stroke generated by the interface. Subjects are then asked to imagine a bird or generic creature that incorporates the eye and initial stroke, and draw it one part at a time. Figure 2 shows example sketches from our datasets. Notice the larger diversity and creativity of birds in our dataset than those from existing datasets with more canonical and mundane birds. We focus on creative sketch generation. Generating novel artifacts is key to creativity. To this end we propose DoodlerGAN -a part-based Generative Adversarial Network (GAN) that generates novel part appearances and composes them in previously unseen configurations. During inference, the model automatically determines the appropriate order of parts to generate. This makes the model well suited for human-in-the-loop interactive interfaces where it can make suggestions based on user drawn partial sketches. Quantitative evaluation and human studies show that our approach generates more creative and higher quality sketches than existing approaches. In fact, subjects prefer sketches generated by DoodlerGAN over human sketches from the Creative Birds dataset! Our datasets, code, and a web demo are publicly availablefoot_0 .

2. RELATED WORK

Sketches have been studied extensively as a visual modality that is expressive yet minimal. The sparsity of sketches compared to natural images has inspired novel modelling techniques. We discuss existing sketch datasets and sketch generation approaches. Other related work includes sketch recognition (Yu et al., 2015; Li et al., 2018) , sketch-based image retrieval (Yu et al., 2016; Liu et al., 2017; Ribeiro et al., 2020) and generation (Gao et al., 2020; Lu et al., 2018; Park et al., 2019) et al., 2016) were collected in a Pictionary-like game setting -they were drawn under 20 seconds to be easily recognized as a target object category. This is in stark contrast with how people engage in doodling as a creative activity, where they take their time, engage their imagination, and draw previously unseen depictions of visual concepts. These depictions may be quite unrealistic -including exaggerations or combinations of multiple categories. Our datasets contain such creative sketches of birds and generic creatures. See Figures 1 and 2 . Our data collection protocol was explicitly designed to engage users in a creative process. Also note that while not the focus of this paper, our datasets are a valuable resource for sketch segmentation approaches. See Section A in the Appendix for further discussion.



songweige.github.io/projects/creative_sketech_generation/home.html



Figure 1: Cherry-picked example sketches from our proposed datasets: Creative Birds (left) and Creative Creatures (right). See random examples in Figure 2 and Figures 12 and 13 in the Appendix.

Figure 2: Random sketches from existing datasets (left) and our creative sketches datasets (right).

. An overview of deep learning approaches for sketches can be found in this survey (Xu et al., 2020) Sketch datasets. Existing sketch datasets such as TU-Berlin (Eitz et al., 2012), Sketchy (Sangkloy et al., 2016), ImageNet-Sketch (Wang et al., 2019) and QuickDraw (Jongejan et al., 2016) are typically focused on realistic and canonical depictions of everyday objects. For instance, sketches in the Sketchy dataset (Sangkloy et al., 2016) were drawn by humans mimicking a natural image. Sketches in the QuickDraw dataset (Jongejan

