AUTOBAYES: AUTOMATED BAYESIAN GRAPH EXPLO-RATION FOR NUISANCE-ROBUST INFERENCE

Abstract

Learning data representations that capture task-related features, but are invariant to nuisance variations 1 remains a key challenge in machine learning. We introduce an automated Bayesian inference framework, called AutoBayes, that explores different graphical models linking classifier, encoder, decoder, estimator and adversarial network blocks to optimize nuisance-invariant machine learning pipelines. AutoBayes also enables learning disentangled representations, where the latent variable is split into multiple pieces to impose various relationships with the nuisance variation and task labels. We benchmark the framework on several public datasets, and provide analysis of its capability for subject-transfer learning with/without variational modeling and adversarial training. We demonstrate a significant performance improvement with ensemble learning across explored graphical models.

1. INTRODUCTION

The great advancement of deep learning techniques based on deep neural networks (DNN) has enabled more practical design of human-machine interfaces (HMI) through the analysis of the user's physiological data (Faust et al., 2018) , such as electroencephalogram (EEG) (Lawhern et al., 2018) and electromyogram (EMG) (Atzori et al., 2016) . However, such biosignals are highly prone to variation depending on the biological states of each subject (Christoforou et al., 2010) . Hence, frequent calibration is often required in typical HMI systems. Toward resolving this issue, subject-invariant methods (Özdenizci et al., 2019b) , employing adversarial training (Makhzani et al., 2015; Lample et al., 2017; Creswell et al., 2017) with the Conditional Variational AutoEncoder (A-CVAE) (Louizos et al., 2015; Sohn et al., 2015) shown in Fig. 1(b) , have emerged to reduce user calibration for realizing successful HMI systems. Compared to a standard DNN classifier C in Fig. 1 (a), integrating additional functional blocks for encoder E, nuisanceconditional decoder D, and adversary A networks offers excellent subject-invariant performance. The DNN structure may be potentially extended with more functional blocks and more latent nodes as shown in Fig. 1(c ). However, such a DNN architecture design may rely on human effort and insight to determine the block connectivity of DNNs. Automation of hyperparameter and architecture exploration in the context of AutoML (Ashok et al., 2017; Brock et al., 2017; Cai et al., 2017; He et al., 2018; Miikkulainen et al., 2019; Real et al., 2017; 2020; Stanley & Miikkulainen, 2002; Zoph et al., 2018) can facilitate DNN design suited for nuisance-invariant inference. Nevertheless, without proper reasoning, most of the search space for link connectivity will be pointless. In this paper, we propose a systematic automation framework called AutoBayes, which searches for the best inference graph model associated with a Bayesian graph model (also a.k.a. Bayesian network) well-suited to reproduce the training datasets. The proposed method automatically formulates various different Bayesian graphs by factorizing the joint probability distribution in terms of data, class label, subject identification (ID), and inherent latent representations. Given Bayesian graphs, some meaningful inference graphs are generated through the Bayes-Ball algorithm (Shachter, 2013) for pruning redundant links to achieve high-accuracy estimation. In order to promote robustness against nuisance variations such as inter-subject/session factors, the explored Bayesian graphs can provide



For example of speech recognition, nuisance factors such as speaker's attributes and recording environment may change the task accuracy. For image recognition, ambient light conditions and image sensor conditions may become inherent nuisance factors. In the context of this paper, nuisance variations mainly refer to subject identities and biological states during recording sessions for physiological data learning.

