ELO-Mask: Effective and Layerwise Optimization of Mask for sparse LLMs

To address the issue of the substantial computational resource consumption during the inference phase of large language models due to their vast number of parameters, model sparsification is an effective solution. However, current sparsification methods for large models are costly. We propose a comprehensive two-stage approach called ELO-Mask for the rapid sparsification of large language models using a small calibration dataset. The approach consists of two steps: 1) Mask Reordering Step, this step involves initializing the mask using predefined parameter importance metrics, followed by reordering the model masks in blocks using the Straight-Through Estimator method with a small sample dataset. 2) Mask Fine-Tuning Step, this step involves further fine-tuning the masks obtained from the first step in blocks, using the same small sample dataset. Our experiments demonstrate the effectiveness of this approach. When sparsifying the Llama-7B model, our method shows significant superiority over the standard sparsification plus LoRA fine-tuning approach. It achieves comparable performance in the final sparse model while consuming less computational power, using a smaller dataset, occupying less GPU memory, and not affecting the inference speed of the sparse model.