quantize activation functions

Topics

• quantization
• activation functions

Activations are quantized to a specified bit-width (8-bit, in the case of BitNet b1.58) using absmax per token quantization. This involves scaling the activations into the range [−128, 127] for an 8-bit bit-width. The quantization formula is:

$$\begin{array}{rl}{s}_i& =\frac{127}{{\max }_j|X_{i,j}|}\\ X_i^{quant}& ={\text{clamp}}_{[-128,127]}(\text{round}(X_i\cdot s_i))\\ X_i^{dequant}& =\frac{X_i^{quant}}{s_i}\end{array}$$

Breaking down each component:

1. The scale factor $s_i$ which is a scalar for row $i$ is computed as: divide 127 by the maximum absolute value in that row
2. Create a new quantized matrix $X_i^{quant}$ by applying round clamp/clip as per formula
3. Finally, dequantize by simply dividing each row by its scale factor $s_i$

The process maintains precision per row while ensuring all values fit within the 8-bit integer range.

Note

We apply layer normalization (LN) before quantizing the activations to maintain the variance of the output:

Related

• quantize activation functions in pytorch