Template Class MinibatchOptimizer

Class Documentation

template<typename BATCH_ELEMENT>
class parpe::MinibatchOptimizer

Public Functions

inline std::tuple<int, double, std::vector<double>> optimize(SummedGradientFunction<BATCH_ELEMENT> const &f, gsl::span<const BATCH_ELEMENT> data, gsl::span<const double> initialParameters, gsl::span<const double> lowerParameterBounds, gsl::span<const double> upperParameterBounds, OptimizationReporter *reporter, Logger *logger_)

Minimize the given function using mini-batch gradient descent.

Return

Tuple (exit code, final cost, final parameters)

Parameters

  • f: Function to minize

  • data: Full data set on which f will be evaluated

  • initialParameters: Starting point for optimization

  • reporter: OptimizationReporter instance for tracking progress

  • logger: Logger instance for status messages

inline FunctionEvaluationStatus evaluate(SummedGradientFunction<BATCH_ELEMENT> const &f, gsl::span<const double> parameters, std::vector<BATCH_ELEMENT> datasets, double &cost, gsl::span<double> gradient, Logger *logger, OptimizationReporter *reporter) const
inline std::tuple<int, double, std::vector<double>> finish(double cost, std::vector<double> const &parameters, minibatchExitStatus status, OptimizationReporter *reporter, Logger *logger)
inline FunctionEvaluationStatus rescueInterceptor(gsl::span<double> parameters, gsl::span<double> oldParameters, gsl::span<double> gradient, gsl::span<double> oldGradient, gsl::span<const double> lowerParameterBounds, gsl::span<const double> upperParameterBounds, double &cost, int &subsequentFails, int iteration, SummedGradientFunction<BATCH_ELEMENT> const &f, std::vector<BATCH_ELEMENT> datasets, Logger *logger, OptimizationReporter *reporter)

Try to rescue optimization run at cost function failure.

Return

FunctionEvaluationStatus

Parameters

  • parameters: current parameter vector

  • oldParameters: parameter vector before last step

  • gradient: current cost function gradient

  • oldGradient: cost function gradient before last step

  • cost: new cost function value after interception

  • subsequentFails: number of iterations during rescue interceptor

  • f: Function to minize

  • data: Full data set on which f will be evaluated

  • logger: Logger instance for status messages

  • reporter: OptimizationReporter instance for tracking progress

inline void handleStep(gsl::span<double> parameters, gsl::span<double> oldParameters, gsl::span<double> gradient, gsl::span<const double> lowerParameterBounds, gsl::span<const double> upperParameterBounds, double cost, int iteration, SummedGradientFunction<BATCH_ELEMENT> const &f, std::vector<BATCH_ELEMENT> datasets, Logger *logger, OptimizationReporter *reporter)

Try to determine a good step length in descent direction.

Return

FunctionEvaluationStatus

Parameters

  • parameters: current parameter vector

  • oldParameters: parameter vector before last step

  • gradient: current cost function gradient

  • oldGradient: cost function gradient before last step

  • cost: new cost function value after interception

  • subsequentFails: number of iterations during rescue interceptor

  • f: Function to minize

  • data: Full data set on which f will be evaluated

  • logger: Logger instance for status messages

  • reporter: OptimizationReporter instance for tracking progress

inline void performLineSearch(double alpha1, double alpha2, double cost, double cost1, double cost2, double dirGradient, std::function<double(double)> costFunEvaluate)

Perform line search according to interpolation algo by [Dennis and Schnabel, Numerical Methods for Unconstrained Optimization and Non-linear Equations, 1993].

Parameters

  • alpha1: step length of first step

  • alpha2: step length of second step

  • cost: objective function value before update

  • cost1: objective function value after first step

  • cost2: objective function value after second step

  • dirGradient: alignment of gradient and step direction

  • costFunEvaluate: objective function wo gradient

Public Members

std::unique_ptr<ParameterUpdater> parameterUpdater = std::make_unique<ParameterUpdaterVanilla>()
interceptType interceptor = interceptType::reduceStepAndRestart
int lineSearchSteps = 0
int batchSize = 1
int maxEpochs = 1
double gradientNormThreshold = 0.0
double learningRate = 0.001
std::unique_ptr<LearningRateUpdater> learningRateUpdater = std::make_unique<LearningRateUpdater>(maxEpochs, learningRateInterp::linear)