LLM Quantization - Running Large Models on Limited Hardware

Quantization reduces the memory footprint of LLMs by representing model weights in lower-precision formats such as INT8 or INT4 instead of the default FP32 or FP16. A 7B parameter model that normally requires 14GB of GPU memory can run in under 5GB with 4-bit quantization, making local LLM deployment practical for teams without expensive hardware.

The two most popular quantization approaches are GPTQ (post-training quantization) and BitsAndBytes (dynamic quantization). BitsAndBytes integrates directly with the Hugging Face transformers library and requires a single configuration change. GPTQ requires a calibration dataset but delivers higher quality at lower bit widths.

The below example loads a quantized model using BitsAndBytes 4-bit quantization to run a ShopMax India product FAQ assistant on a machine with limited GPU memory.

It gives the following output,

Model memory footprint: 4.17 GB
Response: What is the return policy at ShopMax India for electronics?
ShopMax India offers a 10-day return policy on all electronics.
Items must be in original packaging with all accessories included.
Initiate returns via the ShopMax app or visit any store in
Mumbai, Bangalore, or Hyderabad.

The 4-bit model uses 4.17 GB versus 14 GB for FP16, a 70% memory reduction with minimal quality degradation. Use double quantization and NF4 data type for best results. For production deployments at ShopMax, run INT8 quantization for better accuracy on customer-facing tasks, reserving INT4 for internal tools where slight quality tradeoffs are acceptable.