ACCURACY BOOSTERS: EPOCH-DRIVEN MIXED-MANTISSA BLOCK FLOATING-POINT FOR DNN TRAINING

Abstract

The unprecedented growth in DNN model complexity, size and the amount of training data have led to a commensurate increase in demand for computing and a search for minimal encoding. Recent research advocates Hybrid Block Floating-Point (HBFP) as a technique that minimizes silicon provisioning in accelerators by converting the majority of arithmetic operations in training to 8-bit fixed-point. In this paper, we perform a full-scale exploration of the HBFP design space including minimal mantissa encoding, varying block sizes, and mixed mantissa bit-width across layers and epochs. We propose Accuracy Boosters, an epoch-driven mixedmantissa HBFP that uses 6-bit mantissa only in the last epoch and converts 99.7% of all arithmetic operations in training to 4-bit mantissas. Accuracy Boosters enable reducing silicon provisioning for an HBFP training accelerator by 16.98× as compared to FP32, while preserving or outperforming FP32 accuracy.

1. INTRODUCTION

Improvements in Deep Neural Network (DNN) algorithms over the past decade have led to an unprecedented growth in model complexity and dataset size and consequently the required computational resources to train DNN models. One of the largest DNN models (GPT-3) (Brown et al., 2020) has 175 billion parameters and requires 3.14×10 23 FLOPs to train. With the slowdown in Moore's law, researchers and vendors have begun to search for ways to improve the arithmetic density of the underlying hardware platforms. Narrow bit-width (with lower precision) number formats (Wang & Kanwar, 2019; Micikevicius et al., 2018; Sun et al., 2019; Mellempudi et al., 2019; Sun et al., 2020) have emerged as a promising approach to increase arithmetic density, as well as, reduce the required operand storage and communication bandwidth while maintaining high accuracy for training. Many have tried fixed-point formats, often used in inference, to further reduce silicon logic complexity for arithmetic (Courbariaux et al., 2015a; Hubara et al., 2016; Rastegari et al., 2016; Dettmers et al., 2022a; b) . Fixed-point formats, unfortunately, dramatically suffer from a limited range in numerical representation especially for arithmetic in backward propagation. As such, researchers have tried mixed-precision training to trade off accuracy for efficiency (Zhang et al., 2021; Fu et al., 2020; 2021) . Recently there have been several proposals for block floating point (Köster et al., 2017; Das et al., 2018; Zhang et al., 2021) , a numerical encoding that groups a block of mantissas which rely on only fixed-point arithmetic with a single exponent. Block floating point asympototically approaches the arithmetic density of fixed point with larger block sizes and naturally lends itself well to mixedprecision hardware where a block with the same number of exponent bits can have a fixed-point datapath which is bitsliced for various multiples of mantissa bit encodings (e.g., the same today's CPU cores implement SIMD). While block floating point has been promising in use for inference (e.g., Microsoft Floating Point (Darvish Rouhani et al., 2020) ), most proposals to train with block floating point have either failed to reach its full potential by requiring small blocks and/or just fall short of reaching FP32 accuracy. One specific proposal, Hybrid Block Floating Point (HBFP) (Drumond et al., 2018) , uses a mixedprecision format where the dominant fraction of training which is the dot products happens in block floating point (e.g., convolutions, matrix multiplications, outer products), and higher precision (e.g.,

