Welcome to llmc’s documentation!

llmc is a tool for large model compression, supporting a variety of models and a variety of compression algorithms.

Quick Start

• Installation of llmc
• Prepare the model
• Download the datasets
• Set Configs
• Start to run
• FAQ

Configs

• Configs’ brief description
• Configs’ detailed description
  • base
  • model
  • calib
  • eval
  • quant
  • save

Advanced

• Model accuracy test V1
  • Accuracy test pipeline
  • Use of the lightLLM inference engine
  • Use of the opencompass evaluation tool
  • Use of the lm-evaluation-harness evaluation tool
  • FAQ
• Model accuracy test V2
  • Multi-GPU parallel test
• Custom calibration datasets
• Layerwise mixed bits quantization
• Model Sparsification

Best Practice

• AWQ
  • 1.1 Weight-only Quantization
  • 1.2 Weight-Activation Quantization
• AWQ + OmniQuant
  • 1.1 Weight-only Quantization
  • 1.2 Weight-Activation Quantization
• Quarot + GPTQ
  • 1.1 Weight-Activation Quantization

Backbend

• VLLM Quantized Inference
  • 1.1 Environment Setup
  • 1.2 Quantization Formats
  • 1.3 Using LLMC for Model Quantization
  • 1.4 Using VLLM for Inference