Bit manipulation to quantization

A model's weights are 32-bit floats. Quantization is the bit manipulation turned on those weights - snap each one to a coarse integer grid, pack several into a word, and a datacentre model fits on one GPU.

A trained network is millions or billions of numbers, each stored as a 32-bit float - 4 bytes of sign, exponent, and mantissa buying about seven decimal digits of precision. But inference barely notices that precision. So we quantize: throw away most of the bits, keep a few. Pick a range, chop it into a handful of evenly spaced rungs, and store each weight as the index of its nearest rung - an int8 or an int4. That index is a plain integer, and integers pack: eight int4 codes slide into one 32-bit word using the exact shift, mask, and OR you learned as bit tricks. The reward is 4-8× less memory, for a sliver of accuracy.

Continuous weights, a handful of rungs

Fixed vs floating point is really a question of where the bits go. A float spends its 32 bits on a movable decimal point - huge dynamic range, fine resolution near zero. An integer spends all its bits on magnitude, so to represent a fractional weight you must agree on a scale: one integer step equals some fixed real amount. Choose fewer bits and you get fewer rungs, coarser steps, and - the whole point - a smaller number to store. The mapping from a real weight to its nearest integer rung, and back, is called affine quantization.

The affine map: scale, round, clamp

Fix the weight range $[w_{\min },w_{\max }]$ and a bit width $b$, giving $N=2^b$ rungs. The scale is the real-world width of one rung; the integer code $q$ is the rounded, clamped rung index:

Snap a weight, watch the word fill

Below is one real weight sitting on the number line. Drag the bit-width slider from fp32 down to int8 to int4 and the ruler grows coarser: the weight snaps to the nearest teal rung, and the red gap is the quantization error you just accepted. Drag the weight slider (or the number line itself) to move it between rungs. Watch the memory bar shrink and the 32-bit word fill with more, smaller weights as the bits drop.

The trade you're making, in code

The whole pipeline is a scale, a round, a clamp - then the bit-packing you met as a party trick. Everything here is one expression doing what precision-preserving code cannot: forget information on purpose, cheaply.

import numpy as np

def quantize(w, bits, wmin, wmax):
    N     = 2 ** bits                  # levels: 16 for int4, 256 for int8
    scale = (wmax - wmin) / (N - 1)   # real width of one rung
    q     = np.round((w - wmin) / scale) # snap to the nearest integer code
    return np.clip(q, 0, N - 1).astype(np.uint8), scale

def dequantize(q, scale, wmin):
    return wmin + scale * q            # read the float back off the grid

# --- pack two int4 codes into one byte: the shift + mask + OR bit tricks ---
hi, lo  = 12, 9                    # two 4-bit weights, each 0..15
byte    = (hi << 4) | lo             # 1100 1001  ->  201
back_hi = (byte >> 4) & 0xF          # 12  (shift down, mask a nibble)
back_lo =  byte        & 0xF          # 9   (mask the low nibble)

Check yourself