LEARNING LATENT TOPOLOGY FOR GRAPH MATCHING

Abstract

Graph matching (GM) has been traditionally modeled as a deterministic optimization problem characterized by an affinity matrix under pre-defined graph topology. Though there have been several attempts on learning more effective node-level affinity/representation for matching, they still heavily rely on the initial graph structure/topology which is typically obtained through heuristic ways (e.g. Delaunay or k-nearest) and will not be adjusted during the learning process to adapt to problem-specific patterns. We argue that such a mechanism for learning on the fixed topology may restrict the potential of a GM solver for specific tasks, and propose to learn latent graph topology in replacement of the fixed topology as input. To this end, we devise two types of latent graph generation procedures in a deterministic and generative fashion, respectively. Particularly, the generative procedure emphasizes the across-graph consistency and thus can be viewed as a matching-guided generative model. Our methods show superior performance over previous state-of-the-arts on public benchmarks.

1. INTRODUCTION

Being a long standing NP-hard problem (Loiola et al., 2007) , graph matching (GM) has received persistent attention from the machine learning and optimization communities for many years. Concretely, for two graphs with n nodes for each, graph matching seeks to solvefoot_0 : max where the affinity matrix M ∈ R n 2 ×n 2 + encodes node (diagonal elements) and edge (off-diagonal) affinities/similarities and z is the column-wise vectorization form of the permutation matrix Z. H is a selection matrix ensuring each row and column of Z summing to 1. 1 is a column vector filled with 1. Eq. ( 1) is the so-called quadratic assignment problem (QAP) (Cho et al., 2010) . Maximizing Eq. (1) amounts to maximizing the sum of the similarity induced by matching vector Z. While Eq. (1) does not encode the topology of graphs, Zhou & Torre (2016) further propose to factorize M to explicitly incorporate topology matrix, where a connectivity matrix A ∈ {0, 1} n×n is used to indicate the topology of a single graph (A ij = 1 if there exists an edge between nodes i and j; A ij = 0 otherwise). To ease the computation, Eq. ( 1) is typically relaxed by letting z ∈ [0, 1] n 2 and keeping other parts of Eq. ( 1) intact. Traditional solvers to such relaxed problem generally fall into the categories of iterative update (Cho et al., 2010; Jiang et al., 2017) or numerical continuation (Zhou & Torre, 2016; Yu et al., 2018) , where the solvers are developed under two key assumptions: 1) Affinity M is pre-computed with some non-negative metrics, e.g. Gaussian kernel, L 2 -distance or Manhattan distance; 2) Graph topology is pre-defined as input either in dense (Schellewald & Schnörr, 2005) or sparse (Zhou & Torre, 2016) fashion. There have been several successful attempts towards adjusting the first assumption by leveraging the power of deep networks to learn more effective graph representation for GM (Wang et al., 2019a; Yu et al., 2020; Fey et al., 2020) . However, to our best knowledge, there is little previous work questioning and addressing the problem regarding the second assumption in the context of learning-based graph matching 2 . For example, existing standard pipeline of keypoint matching in computer vision will construct initial topology by Delaunay triangulation or k-nearest neighbors. Then this topology will be freezed throughout the subsequent learning and matching procedures. In this sense, the construction of graph topology is peeled from matching task as a pre-processing stage. More examples can be found beyond the vision communities such as in social network alignment (Zhang & Tong, 2016; Heimann et al., 2018; Xiong & Yan, 2020) assuming fixed network structure for individual node matching in two networks. We argue that freezing graph topology for matching can hinder the capacity of graph matching solvers. For a pre-defined graph topology, the linked nodes sometimes result in less meaningful interaction, especially under the message-passing mechanism in graph neural networks (Kipf & Welling, 2017) . We give a schematic demonstration in Fig. 1 . Though some earlier attempts (Cho & Lee, 2012; Cho et al., 2013) seek to adjust the graph topology under traditional non-deep learning setting, such procedures cannot be readily integrated into end-to-end deep learning frameworks due to undifferentiable nature. Building upon the hypothesis that there exists some latent topology better than heuristically created one for GM, our aim is to learn it (or its distribution) for GM. Indeed, jointly solving matching and graph topology learning can be intimidating due to the combinatorial nature, which calls for more advanced approaches. In this paper, we propose an end-to-end framework to jointly learn the latent graph topology and perform GM, termed as deep latent graph matching (DLGM). We leverage the power of graph generative model to automatically produce graph topology from given features and their geometric relations, under specific locality prior. Different from generative learning on singleton graphs (Kipf & Welling, 2016; Bojchevski et al., 2018) , our graph generative learning is performed in a pairwise fashion, leading to a novel matching-guided generative paradigm. The source code will be made publicly available. Contributions: 1) We explore a new direction for more flexible GM by actively learning latent topology, in contrast to previous works using fixed topology as input; 2) Under this setting, we propose a deterministic optimization approach to learn graph topology for matching; 3) We further present a generative way to produce latent topology under a probabilistic interpretation by Expectation-Maximization. This framework can also adapt to other problems where graph topology is the latent structure to infer; 4) Our method achieves state-ofthe-art performance on public benchmarks.

2. RELATED WORKS

In this section, we first discuss existing works for graph topology and matching updating whose motivation is a bit similar to ours while the technique is largely different. Then we discuss relevant works in learning graph matching and generative graph models from the technical perspective. Topology updating and matching. There are a few works for joint graph topology updating and matching, in the context of network alignment. Specifically, given two initial networks for matching, Du et al. (2019) show how to alternatively perform link prediction within each network and node matching across networks based on the observation that these two tasks can benefit to each other. In their extension (Du et al., 2020) , a skip-gram embedding framework is further established under the same problem setting. In fact, these works involve a random-walk based node embedding updating and classification based link prediction modules and the whole algorithm runs in a one-shot optimization fashion. There is neither explicit training dataset nor trained matching model (except



Without loss of generality, we discuss graph matching under the setting of equal number of nodes without outliers. The unequal case can be readily handled by introducing extra constraints or dummy nodes. Bipartite matching and graph isomorphism are subsets of this quadratic formulation(Loiola et al., 2007).2 There are some loosely related works(Du et al., 2019; 2020) on network alignment and link prediction without learning, which will be discussed in detail in the related works.



. Z ∈ {0, 1} n×n , Hz = 1 (1)

Figure 1: Matching of BBGM (Rolínek et al., 2020) 11/13 with Delaunay triangulation and our DLGM-G 13/13 using generated graph (Pascal VOC). DLGM-G generates graph with 4 more edges than Delaunay (33 vs 29) for both source and target. But with 4 more common edges across source and target than Delaunay triangulation (26 vs. 22), it leads to better accuracy. Blue and red edges denote common edges in Delaunay and learned graph pairs.

