SEQUENTIAL BRICK ASSEMBLY WITH EFFICIENT CONSTRAINT SATISFACTION

Abstract

We tackle the problem of assembling LEGO bricks, which can be considered as an instance of combinatorial optimization problems. Such a problem is challenging, since the number of possible structures increases exponentially with the number of available bricks due to complex physical constraints between bricks. To solve this problem, our method assesses a brick structure to predict the next brick position and its confidence by employing a U-shaped sparse 3D convolutional network. A convolution filter efficiently validates the physical constraints in a parallelizable and scalable manner and effectively allows us to process different brick types. To generate a novel structure, we devise a sampling strategy to determine the next brick position by considering attachable positions under the physical constraints. Instead of using handcrafted brick assembly datasets, our model is trained with a large number of 3D objects that enable to create a new high-fidelity structure. We demonstrate that our method successfully generates diverse brick structures while handling two different brick types and outperforms existing methods based on Bayesian optimization, deep graph generative model, and reinforcement learning, all of which consider assembling a single brick type. To show the validity of our method, elaborate studies on various assembly scenarios are also presented.

1. INTRODUCTION

Most real-world 3D structures are constructed with smaller primitives. A broad range of studies have tackled an interesting assembly problem such as molecule generation (Ertl et al., 2017; Neil et al., 2018; You et al., 2018 ), building construction (Talton et al., 2011; Martinovic & Van Gool, 2013; Ritchie et al., 2015) , and part assembly generation (Sung et al., 2017; Lee et al., 2021; Jones et al., 2021) . However, if a primitive unit (e.g., LEGO bricks) is used to construct a 3D structure, it is thought of as an instance of combinatorial optimization problems. In particular, the search space of the primitive assembly problem increases exponentially as a search depth increases: given n primitives with k possible combinations, the search space increases as O(n k ). In addition, supposing that a constraint between unit primitives should be satisfied, the problem becomes more challenging. In this work, we tackle the problem of assembling LEGO bricks into a 3D structure. The generation problem of brick assembly is defined as a sequential decision-making process that sequentially appends a new brick by determining the brick type, position, and direction of the new brick. Moreover, a brick to assemble has to satisfy the physical constraints related to the disallowance of overlap, no isolated bricks, and LEGO-like brick connections; see Section 2 for the details of such constraints. Compared to general 3D generation methods such as previous work (Wu et al., 2016; Gadelha et al., 2018; Achlioptas et al., 2018) , the advantages of sequential brick assembly as a generation method are that it can generate a structure in an open space and provide instructions to build the structure. Several attempts have been conducted on the generation of sequential brick assembly by utilizing Bayesian optimization (Kim et al., 2020) , deep graph generative model (Thompson et al., 2020) , and reinforcement learning (Chung et al., 2021) . Those methods are capable of assembling 2 × 4 LEGO bricks with consideration of the physical constraints. However, the previous literature has several limitations. A Bayesian optimization method proposed by Kim et al. (2020) requires heavy computations due to their iterative optimization process for each brick position. Thompson et al. (2020) propose to use masks to filter out invalid actions along with their method, but the utilization of masks degrades assembly performance. To predict a valid action, Chung et al. (2021) propose

