Reducing Resource Consumption in Training Large Language Models through Multi-Objective Optimization

The rapid scaling of large language models (LLMs) has led to unprecedented computational costs and environmental impact. We address the problem of multi-objective optimization for LLM training, balancing model performance against resource usage (training time, energy, GPU-hours, carbon footprint). We survey and evaluate state-of-the-art techniques—model pruning, quantization, knowledge distillation, neural architecture search (NAS), and hyperparameter tuning via evolutionary or reinforcement learning—in terms of their trade-offs between accuracy (or loss) and efficiency. Using recent experimental data from public benchmarks (e.g. BERT fine-tuning on GLUE tasks, GPT-family training), we analyze how each method shapes the Pareto frontier of accuracy vs. cost. For example, static ۸-bit quantization has been shown to cut energy use by ~۲۹% with negligible accuracy loss, while structured pruning can speed up inference by ~۶۳% for minor accuracy degradation. Advanced LLM compression methods achieve even larger gains: a multi-objective shift-add reparameterization method achieved over ۸۰% reduction in memory and energy usage compared to full models. We include Pareto-plots (accuracy vs energy) to visualize these trade-offs (Figure ۱–۴). Overall, we find that multi-objective search (e.g. Bayesian optimization or genetic algorithms) can systematically identify configurations that lie on the Pareto-optimal front, enabling practitioners to choose models that best fit their constraints. Our paper highlights that, consistent with “Green AI” principles, moderate sacrifices in accuracy can yield large efficiency gains, and we provide actionable recommendations for training sustainable LLMs.

کلیدواژه ها:

Multi-objective optimization ، Language model compression ، Energy efficiency ، Accuracy-cost trade-off ، Green AI