This paper addresses the growing concern of computational cost associated with state-of-the-art Machine Translation (MT) evaluation metrics. While models like xCOMET achieve high correlation with human judgment, their large size (up to 10.7B parameters) makes them inaccessible for researchers with limited resources.
The authors investigate three main compression techniques to create efficient alternatives to xCOMET:
1. Quantization: Reducing the precision of model parameters and activations from 32/16 bits to lower bit representations (8, 4, 3, 2 bits). This reduces memory footprint and allows for faster computations.
- Methods: GPTQ (data-aware, weight-only), LLM.int8() (data-free, dynamic), QLoRA (data-free, double quantization).
- Advantages: Significant speed improvements with minimal quality degradation.
- Limitations: Requires careful selection of quantization method and bit precision to balance speed and accuracy.
2. Pruning: Removing less significant parts of the model, such as specific parameters, blocks, or entire layers.
- Methods: Layer pruning with parameter-efficient fine-tuning (tuning only biases, LayerNorm parameters, attention weights, and head parameters).
- Advantages: Reduces model size and can improve inference speed.
- Limitations: Removing too many layers can significantly impact performance. Careful fine-tuning is crucial to regain lost accuracy.
3. Distillation: Training a smaller "student" model to mimic the behavior of the larger "teacher" model (xCOMET-XXL).
- Methods: Black-box distillation using a large dataset labeled by the teacher model.
- Advantages: Can significantly reduce model size while retaining most of the teacher's performance.
- Limitations: Requires a large, high-quality dataset for training the student model.
Key findings:
- Quantization: 3-bit quantization effectively reduces hardware requirements for xCOMET without compromising quality.
- Pruning: Pruning up to 25% of layers provides speed improvements with minor quality loss, but removing more layers significantly degrades performance.
- Distillation: The authors introduce xCOMET-lite, a distilled version of xCOMET with only 2.6% of the original parameters. xCOMET-lite achieves 92.1% of xCOMET-XXL's quality, outperforming other small-scale metrics.
- Interaction between methods: Distillation combines well with quantization, but not with pruning in the conducted experiments.
Novel contributions:
- Comprehensive study of compression methods for a large-scale MT evaluation metric like xCOMET.
- Introduction of a novel data collection pipeline for black-box distillation, resulting in a 14M example dataset.
- Development of xCOMET-lite, a highly efficient and accurate distilled metric.
- Analysis of interactions between different compression methods.
Impact:
This work significantly contributes to making advanced MT evaluation metrics more accessible. xCOMET-lite and the insights on quantization and pruning provide valuable tools for researchers and practitioners with limited resources, enabling them to benefit from state-of-the-art evaluation techniques.
Furthermore, the paper promotes environmentally conscious research by highlighting the computational cost and carbon footprint associated with large models and offering efficient alternatives.