TRANSFORMERS SATISFY

Abstract

The Propositional Satisfiability Problem (SAT), and more generally, the Constraint Satisfaction Problem (CSP), are mathematical questions defined as finding an assignment to a set of variables such that all the constraints are satisfied. The modern approach is trending to solve CSP through neural symbolic methods. Most recent works are sequential model-based, and adopt neural embedding, i.e., reinforcement learning with graph neural networks, and graph recurrent neural networks. In this work, we propose Heterogeneous Graph Transformer (HGT), a one-shot model derived from the eminent Transformer architecture for factor graph structure to solve the CSP problem. We define the heterogeneous attention mechanism based on meta-paths for the self-attention between literals, the cross-attention based on the bipartite graph links between literals and clauses. Exploiting highlevel parallelism, our model is able to achieve exceptional speed and accuracy on the factor graph for CSPs with arbitrary size. The experimental results have demonstrated the competitive performance and generality of HGT compared to the most recent baseline approaches.

1. INTRODUCTION

The Constraint Satisfaction Problems (CSP) is of central importance in several aspects of computer science, including theoretical computer science, complexity theory, algorithmics, cryptography, and artificial intelligence. CSP aims at finding a consistent assignment of values to variables such that all constraints, which are typically defined over a finite domain, are satisfied. In particular, there is an assortment of problems arising from artificial intelligence and circuit design that can be reduced to CSP subtypes, including map coloring, vertex cover, independent set, dominating set, and clique detection. Solving a CSP on a finite domain is often an NP-complete problem with respect to the domain size. The conventional CSP-solvers rely on handcrafted heuristics that guide the search for satisfying assignments. These algorithms are focused on solving CSP via backtracking or local search. Hence, the resulted model is bounded by the greedy strategy, which is generally sub-optimal. With the advent of Graph Neural Networks (Scarselli et al. ( 2009 2019), attempted to solve CSP through different deep learning approaches. However, most pioneering works, such as neural approaches utilizing RNN or Reinforcement Learning, are still restricted to sequential algorithms, while clauses are parallelizable even though they are strongly correlated through shared variables. In this work, we propose a hybrid model of the Transformer architecture (Vaswani et al. (2017) ) and the Graph Neural Network for solving combinatorial problems, especially CSP. Our main contributions in this work are: (a) We derived meta-paths adopted from Sun et al. (2011) to formulate the message passing mechanism between homogeneous nodes (i.e., variable to variable, or clause to clause), which enable us to perform self-attention and let message pass through either variables sharing the same clauses, or clauses that include the same variables. We apply the cross-attention mechanism to optimize message exchanges between heterogeneous nodes(i.e., clause to variable, or variable to clause). (b) With the combination of homogeneous attention and heterogeneous atten-



)), the geometric deep learning (Bronstein et al. (2017)) for Non-Euclidean data has become one of the most emerging fields of machine learning. In particular, it brought deep learning solutions to one of the most dominant combinatorial optimization problems, the Constraint Satisfaction Problem (CSP) (Khalil et al. (2017)). Works including NeuroSAT (Selsam et al. (2018)) and Circuit-SAT (Amizadeh et al. (2018)) commenced the study of neural methods targeted at CSP. Later works, such as Yolcu & Póczos (2019) and You et al. (

