RoundClip

Topics

• BitNet b1.58

To constrain the weights to -1, 0, or +1, in BitNet b1.58 we adopt an absmean quantization function. It first scales the weight matrix by its average absolute value, and then round each value to the nearest integer among {-1, 0, +1}:

$$\begin{array}{rl}\tilde{W}& =\mathrm{RoundClip}(\frac{W}{\gamma +ϵ},-1,1)\\ \mathrm{RoundClip}& =\max (a,\min (b,\mathrm{round}(x)))\\ \gamma =& \frac{1}{nm}\sum _{ij}|W_{ij}|\end{array}$$

Related