Reducing Resource Consumption in Training Large Language Models through Multi-Objective Optimization
سال انتشار: 1404
نوع سند: مقاله کنفرانسی
زبان: انگلیسی
مشاهده: 4
فایل این مقاله در 9 صفحه با فرمت PDF قابل دریافت می باشد
- صدور گواهی نمایه سازی
- من نویسنده این مقاله هستم
استخراج به نرم افزارهای پژوهشی:
شناسه ملی سند علمی:
CMELC02_083
تاریخ نمایه سازی: 16 خرداد 1404
چکیده مقاله:
The rapid scaling of large language models (LLMs) has led to unprecedented computational costs and environmental impact. We address the problem of multi-objective optimization for LLM training, balancing model performance against resource usage (training time, energy, GPU-hours, carbon footprint). We survey and evaluate state-of-the-art techniques—model pruning, quantization, knowledge distillation, neural architecture search (NAS), and hyperparameter tuning via evolutionary or reinforcement learning—in terms of their trade-offs between accuracy (or loss) and efficiency. Using recent experimental data from public benchmarks (e.g. BERT fine-tuning on GLUE tasks, GPT-family training), we analyze how each method shapes the Pareto frontier of accuracy vs. cost. For example, static ۸-bit quantization has been shown to cut energy use by ~۲۹% with negligible accuracy loss, while structured pruning can speed up inference by ~۶۳% for minor accuracy degradation. Advanced LLM compression methods achieve even larger gains: a multi-objective shift-add reparameterization method achieved over ۸۰% reduction in memory and energy usage compared to full models. We include Pareto-plots (accuracy vs energy) to visualize these trade-offs (Figure ۱–۴). Overall, we find that multi-objective search (e.g. Bayesian optimization or genetic algorithms) can systematically identify configurations that lie on the Pareto-optimal front, enabling practitioners to choose models that best fit their constraints. Our paper highlights that, consistent with “Green AI” principles, moderate sacrifices in accuracy can yield large efficiency gains, and we provide actionable recommendations for training sustainable LLMs.
کلیدواژه ها:
نویسندگان
Alireza Rahimipour Anaraki
Department of Computer Engineering, CT.C., Islamic Azad University, Tehran, Iran