Adaptive-batch stochastic gradient descent for constrained optimization based on relaxed barrier functions

Stochastic Gradient Descent (SGD) is the cornerstone of large-scale optimization; however, its application to problems with a vast number of constraints remains a significant challenge. Methods based on relaxed logarithmic barrier functions have enabled the use of SGD by sampling constraints, yet the reliance on single samples in these methods often leads to high variance and oscillations that hinder progress near the optimal solution.In this paper, we propose a novel hybrid algorithm, AdaBS-RBSGD, which integrates the relaxed barrier framework with an adaptive batch size strategy. Our algorithm dynamically adjusts the batch size Bk to be inversely proportional to the estimated optimization loss, systematically reducing gradient variance as the iterations approach the solution.We provide a rigorous mathematical proof establishing the almost-sure convergence of this hybrid algorithm. Furthermore, our theoretical analysis of error bounds demonstrates that active variance management allows the error term associated with stochastic noise to vanish more rapidly compared to the baseline method. Empirically, results on a benchmark problem show that the proposed algorithm significantly outperforms its single-sample counterpart, successfully breaking the “noise floor,” achieving higher stability, and reaching a more accurate final solution. This work presents a practical and theoretically sound methodology for accelerating and improving the precision of stochastic constrained optimization.

کلیدواژه ها:

Constrained optimization ، Stochastic Gradient Descent (SGD) ، Relaxed Logarithmic Barrier Functions ، Adaptive Batch Sizes ، Variance Reduction ، Large-Scale Optimization