GOING BEYOND APPROXIMATION: ENCODING CON-STRAINTS FOR EXPLAINABLE MULTI-HOP INFERENCE VIA DIFFERENTIABLE COMBINATORIAL SOLVERS

Abstract

Integer Linear Programming (ILP) provides a viable mechanism to encode explicit and controllable assumptions about explainable multi-hop inference with natural language. However, an ILP formulation is non-differentiable and cannot be integrated into broader deep learning architectures. Recently, Thayaparan et al. (2022) proposed a novel methodology to integrate ILP with Transformers to achieve end-to-end differentiability for complex multi-hop inference. While this hybrid framework demonstrates to deliver better answer and explanation selection than transformer-based and ILP solvers, the neuro-symbolic integration still relies on a convex relaxation of the ILP formulation, which can produce suboptimal solutions. To improve these limitations, we propose Diff-Comb Explainer, a novel neuro-symbolic architecture based on Differentiable BlackBox Combinatorial solvers (DBCS) (Pogančić et al., 2019). Unlike existing differentiable solvers, the presented model does not require the transformation and relaxation of the explicit semantic constraints, allowing for a direct and a more efficient integration of ILP formulations. Diff-Comb Explainer demonstrates improved accuracy in answer and explanation selection over non-differentiable solvers, Transformers and constraint-based differentiable multi-hop inference frameworks.

1. INTRODUCTION

Given a question expressed in natural language, ILP-based Multi-hop Question Answering (QA) aims to construct an explanation graph of interconnected facts (i.e., natural language sentences) to support the answer (see Figure 1 ). This framework provides a viable mechanism to encode explicit and controllable assumptions about the structure of the inference (Khashabi et al., 2018; Khot et al., 2017; Khashabi et al., 2016) . For this reason, inference based on constrained optimization is generally regarded as interpretable and transparent, providing structured explanations in support of the underlying reasoning process (Thayaparan et al., 2020) . However, ILP solvers are non-differentiable and cannot be integrated as part of a broader deep learning architecture (Paulus et al., 2021; Pogančić et al., 2019) . Moreover, these approaches are often limited by the exclusive adoption of hard-coded heuristics for the inference and cannot be optimised end-to-end on annotated corpora to achieve performance comparable to deep learning counterparts (Thayaparan et al., 2022; Khashabi et al., 2018) . In an attempt to combine the best of both worlds, Thayaparan et al. ( 2022) proposed a novel neurosymbolic framework (Diff-Explainer) that integrates explicit constraints with neural representations via Differentiable Convex Optimization Layers (Agrawal et al., 2019) . Diff-Explainer combines constraint optimization solvers with Transformers-based representations, enabling end-to-end training for explainable multi-hop inference. The non-differenitability of ILP solvers is alleviated by approximating the constraints using semi-definite programming (Helmberg, 2000) . This approximation usually requires non-trivial transformations of ILP formulations into convex optimization problems. Since constraint-based multi-hop inference is typically framed as optimal subgraph selection via binary optimization (0, 1), The semi-definite relaxation employed in Diff-Explainer necessitates a continuous relaxation of the discrete variables (from {0, 1} to [0, 1]). While this process can provide 1

