Add README.md + measurement files

README.md

@@ -0,0 +1,161 @@
+---
+license: mit
+base_model: zai-org/GLM-4.6
+base_model_relation: quantized
+quantization: exl3
+pipeline_tag: text-generation
+tags:
+  - exl3
+library_name: exllamav3
+---
+
+# GLM 4.6 (EXL3 Quants)
+
+- Original Model:
+  - [zai-org/GLM-4.6](https://huggingface.co/zai-org/GLM-4.6)
+
+This repo contains:
+- base quants (3, 4, 5, 6, 8 bits) for Exllamav3 (using SOTA random Hadamard transforms and Trellis quantization for high-quality reconstruction)
+- layer and tensor level KL-divergence measurements for bit-allocation optimization given a target size
+- theoretical research related to quantization, in particular MoE quantization
+
+## Motivation
+
+The goals are:
+- to provide the best possible quants for what is arguably the top general model of 2025
+- to serve as a reference for quantization strategies (as of 2025 knowledge)
+
+The base model is 355B parameters, which when 4-bit quantized should take 177GiB, leaving almost 20GB for context, a perfect situation when you have 196GiB of VRAM (i.e. 8x 3090/4090, 6x 5090, $x RTX A6000, 4x RTX 6000 Ada or 2x RTX Pro 6000 Blackwell). Too bad all the 4-bit quants for my usual framework of choice, vllm, start at 191~200GiB of VRAM.
+
+So while looking for a new backend that could leverage tensor parallelism, I landed on Exllamav3. And even better it had already in place the proper tools to fully quantized Mixture-of-Experts (MoE) models, unlike vllm/llmcompressor that requires you extra code to ensure all experts are activated (or their activation might be quantized away as unimportant if you have a non-comprehensive calibration dataset).
+
+## Artifacts
+
+### Base Quants
+
+The base quants use the new "MCG" multiplier from https://github.com/turboderp-org/exllamav3/pull/26#issuecomment-3395345415
+
+- Size measured through: https://github.com/turboderp-org/exllamav3/pull/103
+- Kullback-Leibler divergence (KL-div) and Top-K agreement measured through: https://github.com/turboderp-org/exllamav3/blob/v0.0.14/eval/model_diff.py
+- Perplexity measured through: https://github.com/turboderp-org/exllamav3/blob/v0.0.14/eval/model_diff.py
+  - Caveat both quantization calibration and perplexity use the same dataset in EXL3, hence we have overfitting.\
+    The most appropriate measure for quality is KL-divergence (i.e. how well the quant reproduces the original probability distribution of token output, before samplers)\
+    For example the 3-bit quant have lower perplexity than the original FP16.\
+
+| Quant                                                            | Size    | KL-div (quant, FP16) | KL-div (FP16, quant) | Perplexity | Top-1  | Top-2  | Top-3  | Top-4  | Top-5  |
+| ---------------------------------------------------------------- | ------- | -------------------- | -------------------- | ---------- | ------ | ------ | ------ | ------ | ------ |
+| [3bpw](https://huggingface.co/mratsim/glm-4.6-exl3/tree/3bpw_H6) | 124 GiB | 0.32625636           | 0.30842110           | 4.36145115 | 0.8409 | 0.5497 | 0.3022 | 0.1527 | 0.0695 |
+| [4bpw](https://huggingface.co/mratsim/glm-4.6-exl3/tree/4bpw_H6) | 165 GiB | 0.15579397           | 0.15313307           | 4.64835933 | 0.8969 | 0.6892 | 0.4609 | 0.2840 | 0.1611 |
+| [5bpw](https://huggingface.co/mratsim/glm-4.6-exl3/tree/5bpw_H6) | 206 GiB | 0.11346048           | 0.10777174           | 4.46847223 | 0.9172 | 0.7553 | 0.5610 | 0.3868 | 0.2486 |
+| [6bpw](https://huggingface.co/mratsim/glm-4.6-exl3/tree/6bpw_H6) | 247 GiB | 0.08243355           | 0.07828716           | 4.46603787 | 0.9336 | 0.7970 | 0.6218 | 0.4600 | 0.3226 |
+| [8bpw](https://huggingface.co/mratsim/glm-4.6-exl3/tree/8bpw_H8) | 328 GiB | 0.06771311           | 0.06660905           | 4.61223994 | 0.9441 | 0.8221 | 0.6663 | 0.5155 | 0.3780 |
+| FP16                                                             | 656 GiB |                      |                      | 4.62864232 |        |        |        |        |        |
+
+### Optimized Quants
+
+Coming soon.
+
+- "opt🂡" for automatically optimized quants
+- "tuned🂱" for hand-tuned quants
+
+### Detailed measurements of KL-div improvements
+
+Exllamav3 offers tools to measure per layer (with `-l2`) or even per-tensor (with `-l3`) contributions to KL-div improvements.
+They might take 2 hours to 5 hours, if comparing 2 quants -- to 12 hours if comparing 3 quants -- to over a day of compute if comparing all quants.
+
+Currently available are:
+- 3vs4vs5 `-l3`: [json](glm-4.6-measurements-3vs4vs5.json), [markdown](glm-4.6-measurements-3vs4vs5.md)
+- 4vs5vs6 `-l3`: [json](glm-4.6-measurements-4vs5vs6.json), [markdown](glm-4.6-measurements-4vs5vs6.md)
+- 6vs8 `-l3`: [json](glm-4.6-measurements-6vs8.json)
+
+The json file can be fed to https://github.com/turboderp-org/exllamav3/blob/v0.0.14/util/optimize.py with a target `bpw` to output an optimized quant.
+
+Please note that from experimentations, manual tuning using the heuristics below can achieve better KL-divergence than optimizing by only mixing 3 quants and is less likely to overfit the calibration set. Having `shared experts` or `self_attn` layers use 6 or even 8-bit provide a very large improvement to KL-divergence. A measurement with all available quants might achieve better than manual tuning though overfitting becomes a bigger risk.
+
+## Quantization theory and heuristics for manual tuning
+
+### Layers to quantize
+
+Quantization should be focused on Linear layers (also called Dense or Fully-Connected layers i.e. MatMul+Bias)
+In particular quantizing LayerNorm/RMSnorm layer is strongly discouraged, see [1]
+> LayerNorm in Quantization. Kovaleva et al. (2021); Wei et al. (2022) find that outliers in the
+> LayerNorm parameters of BERT (Devlin et al., 2019) cause difficulties in model compression.
+> Given the importance of LayerNorm, all the quantization methods we discuss above leave LayerNorm unquantized.
+
+This is also reported in Intel and Nvidia repo:
+- https://github.com/intel/neural-compressor/issues/1963#issuecomment-2274873441
+- https://github.com/NVIDIA/TensorRT/issues/4084#issuecomment-2294513950
+
+EXL3 can only quantize linear layers
+
+### Tensors to up-quantize
+
+If there is enough bits, down projections should be prioritized.
+
+According to [4]
+> Fig. 3: Maximum absolute value over layers for a LLaMA3-8B.
+> Each color represent a different projection and we clearly see that down_proj has the biggest
+> spikes in input and output. We also observe that RMSNorm propagate spikes through the entire model
+
+According to [5]
+> Figure 5(a) illustrates the extremal ratio across layers and modules in LLaMA2-7B, highlighting
+> that weight outliers are concentrated in the down-projection matrices Wdown
+> ℓ of the second layer and
+> the last two layers. Figures 5(b) and 5(c) provide detailed visualizations of these outliers in the last
+> two layers.
+
+### Mixture-of-Experts quantization (MoE)
+
+Mixture-of-Experts require specific quantization techniques.
+
+#### Mixed-precision quantization
+
+Some layers have a higher impact on LLM performance.
+According to [2], spending more bits in attention layers results in large gain compared to spending them in FFN layers.
+According to [3] on 2-bit quantization:
+- quantizing expert FFN layers do not seriously impact model quality
+- quantizing cross-attention has some impact
+- quantizing self-attention has a large impact
+- quantizing dense FFN has a very significant impact
+
+Hence to preserve model quality we should choose not to quantize dense FFN layers and self-attention layers.
+
+We notice that:
+- official MXFP4 weights of gpt-oss-120b from OpenAI keep self-attention in BF16:
+  - https://huggingface.co/openai/gpt-oss-120b/blob/main/model.safetensors.index.json
+- NVFP4 weights of DeepSeek-R1 quantized by Nvidia also keep self-attention in BF16:
+  - https://huggingface.co/nvidia/DeepSeek-R1-0528-FP4/blob/main/model.safetensors.index.json
+
+#### Layers with high-impact
+
+According to [2], giving more bits to the first `k` blocks have a significantly higher impact on model quality than for the same last `k` blocks.
+
+#### Expert quantization
+
+When quantizing MoE, quantizing activations is tricky as only a subset of experts are activated per request.
+
+EXL3 has the tooling in-place to ensure all experts are activated during quantization, though it is unsure if the dataset should be expanded to be diverse enough so that all experts have a high likelyhood of taking the full range of values they can exhibit to avoid clipping.
+
+## References
+
+1. Why Do Some Inputs Break Low-Bit LLM Quantization? (2025)\
+  Ting-Yun Chang, Muru Zhang, Jesse Thomason, Robin Jia\
+  https://arxiv.org/pdf/2506.12044
+
+2. Examining Post-Training Quantization for Mixture-of-Experts: A Benchmark (2024)\
+  Pingzhi Li, Xiaolong Jin, Yu Cheng, Tianlong Chen\
+  https://arxiv.org/pdf/2406.08155v1
+
+3. Mixture of Quantized Experts (MoQE): Complementary Effect of Low-bit Quantization and Robustness (2023)\
+  Young Jin Kim, Raffy Fahim, Hany Hassan Awadalla\
+  https://arxiv.org/pdf/2310.02410
+
+4. Precision Where It Matters: A Novel Spike\
+   Aware Mixed-Precision Quantization Strategy for\
+   LLaMA-based Language Models (2025)\
+   Lucas Maisonnave, Cyril Moineau, Olivier Bichler, and Fabrice Rastello\
+   https://arxiv.org/pdf/2504.21553
+
+5. Systematic Outliers in Large Language Models (2025)\
+   Yongqi An, Xu Zhao, Tao Yu, Ming Tang, Jinqiao Wang\
+   https://arxiv.org/pdf/2502.06415v2