Template Class MinibatchOptimizer

Class Documentation

template<typename BATCH_ELEMENT>
class MinibatchOptimizer

Public Functions

inline std::tuple<int, double, std::vector<double>> optimize(SummedGradientFunction<BATCH_ELEMENT> const &f, gsl::span<const BATCH_ELEMENT> data, gsl::span<const double> initialParameters, gsl::span<const double> lowerParameterBounds, gsl::span<const double> upperParameterBounds, OptimizationReporter *reporter, Logger *logger_)

Minimize the given function using mini-batch gradient descent.

Parameters:

  • f – Function to minimize

  • data – Full data set on which f will be evaluated

  • initialParameters – Starting point for optimization

  • reporter – OptimizationReporter instance for tracking progress

  • logger – Logger instance for status messages

Returns:

Tuple (exit code, final cost, final parameters)

inline FunctionEvaluationStatus evaluate(SummedGradientFunction<BATCH_ELEMENT> const &f, gsl::span<const double> parameters, std::vector<BATCH_ELEMENT> datasets, double &cost, gsl::span<double> gradient, Logger *logger, OptimizationReporter *reporter) const
inline std::tuple<int, double, std::vector<double>> finish(double cost, std::vector<double> const &parameters, minibatchExitStatus status, OptimizationReporter *reporter, Logger *logger)
inline FunctionEvaluationStatus rescueInterceptor(gsl::span<double> parameters, gsl::span<double> oldParameters, gsl::span<double> gradient, gsl::span<double> oldGradient, gsl::span<const double> lowerParameterBounds, gsl::span<const double> upperParameterBounds, double &cost, int &subsequentFails, int iteration, SummedGradientFunction<BATCH_ELEMENT> const &f, std::vector<BATCH_ELEMENT> datasets, Logger *logger, OptimizationReporter *reporter)

Try to rescue optimization run at cost function failure.

Parameters:

  • parameters – current parameter vector

  • oldParameters – parameter vector before last step

  • gradient – current cost function gradient

  • oldGradient – cost function gradient before last step

  • cost – new cost function value after interception

  • subsequentFails – number of iterations during rescue interceptor

  • f – Function to minimize

  • data – Full data set on which f will be evaluated

  • logger – Logger instance for status messages

  • reporter – OptimizationReporter instance for tracking progress

Returns:

FunctionEvaluationStatus

inline void handleStep(gsl::span<double> parameters, gsl::span<double> oldParameters, gsl::span<double> gradient, gsl::span<const double> lowerParameterBounds, gsl::span<const double> upperParameterBounds, double cost, int iteration, SummedGradientFunction<BATCH_ELEMENT> const &f, std::vector<BATCH_ELEMENT> datasets, Logger *logger, OptimizationReporter *reporter)

Try to determine a good step length in descent direction.

Parameters:

  • parameters – current parameter vector

  • oldParameters – parameter vector before last step

  • gradient – current cost function gradient

  • oldGradient – cost function gradient before last step

  • cost – new cost function value after interception

  • subsequentFails – number of iterations during rescue interceptor

  • f – Function to minimize

  • data – Full data set on which f will be evaluated

  • logger – Logger instance for status messages

  • reporter – OptimizationReporter instance for tracking progress

Returns:

FunctionEvaluationStatus

inline void performLineSearch(double alpha1, double alpha2, double cost, double cost1, double cost2, double dirGradient, std::function<double(double)> costFunEvaluate)

Perform line search according to interpolation algorithm by [Dennis and Schnabel, Numerical Methods for Unconstrained Optimization and Non-linear Equations, 1993].

Parameters:

  • alpha1 – step length of first step

  • alpha2 – step length of second step

  • cost – objective function value before update

  • cost1 – objective function value after first step

  • cost2 – objective function value after second step

  • dirGradient – alignment of gradient and step direction

  • costFunEvaluate – objective function wo gradient

Public Members

std::unique_ptr<ParameterUpdater> parameterUpdater = std::make_unique<ParameterUpdaterVanilla>()
interceptType interceptor = interceptType::reduceStepAndRestart
int lineSearchSteps = 0
int batchSize = 1
int maxEpochs = 1
double gradientNormThreshold = 0.0
double learningRate = 0.001
std::unique_ptr<LearningRateUpdater> learningRateUpdater = std::make_unique<LearningRateUpdater>(maxEpochs, learningRateInterp::linear)