Brief of optim Function

Reading through the optim function brief from the previous post it will help the reader regarding operational questions of the function.

So for the initial parameters of a=0.1 and b=0.2 we will be finding estimated parameters from different analytical methods which would minimize the Negative Log Likelihood value of the Beta-Binomial distribution.

First we are transforming the given EstMLEBetaBin function to satisfy the optim function conditions.

# new function to facilitate optim criteria
# only one input but has two elements
foroptim<-function(a)
  {
  EstMLEBetaBin(x=Alcohol_data$Days, freq=Alcohol_data$week1,a=a[1],b=a[2])
  }

So the foroptim function can be used as above and parameters are estimated for \(\alpha\) and \(\beta\) (or a, b) for the Alcohol Consumption data week 1. Further the optim function can be scrutinized as below.

• package : stats
• No of Inputs: 7
• Minimum required Inputs : 2
• Class of output : list
• No of outputs: 5
• No of Analytical Methods : 6
• Default Method : Nelder-Mead

Nelder and Mead method

Default analytical method is Nelder and Mead method. According to the documentation it uses only function values and is robust but relatively slow. It will work reasonably well for non-differential functions.

Reference : Nelder, J.A. and Mead, R., 1965. A simplex method for function minimization. The computer journal, 7(4), pp.308-313.

Below is the code of using optim function with Nelder and Mead analytical method.

# optimizing values for a,b using default analytical method or Nelder and Mead
NM_answer<-optim(par=c(0.1,0.2),fn=foroptim)

# the outputs
NM_answer$par # estimated values for a, b
NM_answer$value # minimized function value 
NM_answer$counts  # see the documentation to understand
NM_answer$convergence # indicates successful completion
NM_answer$message # additional information

# fitting the Beta-Binomial distribution with estimated shape parameter  values
fitBetaBin(Alcohol_data$Days,Alcohol_data$week1,NM_answer$par[1],NM_answer$par[2])

BFGS method

The documentation indicates that BFGS is a Quasi-Newton method (also known as a variable metric algorithm), specifically that published simultaneously in 1970 by Broyden, Fletcher, Goldfarb and Shanno. This uses function values and gradients to build up a picture of the surface to be optimized.

Reference : Broyden, C.G., 1967. Quasi-Newton methods and their application to function minimization. Mathematics of Computation, 21(99), pp.368-381.

Below is the code for using optim function with BFGS analytical method

# optimizing values for a,b using BFGS inputs
BFGS_answer<-optim(par=c(0.1,0.2),fn=foroptim,method = "BFGS")

# the outputs
BFGS_answer$par # estimated values for a, b
BFGS_answer$value # minimized function value 
BFGS_answer$counts  # see the documentation to understand
BFGS_answer$convergence # indicates successful completion
BFGS_answer$message # additional information

# fitting the Beta-Binomial distribution with estimated shape parameter  values
fitBetaBin(Alcohol_data$Days,Alcohol_data$week1,
           BFGS_answer$par[1],BFGS_answer$par[2])

CG method

The documentation indicates the Method CG is a conjugate gradients method based on that by Fletcher and Reeves (1964) (but with the option of Polak–Ribiere or Beale–Sorenson updates). Conjugate gradient methods will generally be more fragile than the BFGS method, but as they do not store a matrix they may be successful in much larger optimization problems.

Reference : Fletcher, R. and Reeves, C.M., 1964. Function minimization by conjugate gradients. The computer journal, 7(2), pp.149-154.

Using CG method with optim function is explained below

# optimizing values for a,b using CG inputs
CG_answer<-optim(par=c(0.1,0.2),fn=foroptim,method = "CG")

# the outputs
CG_answer$par # estimated values for a, b
CG_answer$value # minimized function value 
CG_answer$counts  # see the documentation to understand
CG_answer$convergence # indicates successful completion
CG_answer$message # additional information

# fitting the Beta-Binomial distribution with estimated shape parameter  values
fitBetaBin(Alcohol_data$Days,Alcohol_data$week1,CG_answer$par[1],CG_answer$par[2])

L-BFGS-B method

Method L-BFGS-B is that of Byrd et. al. (1995) which allows box constraints, that is each variable can be given a lower and/or upper bound. The initial value must satisfy the constraints. This uses a limited-memory modification of the BFGS quasi-Newton method. If non-trivial bounds are supplied, this method will be selected, with a warning.

Reference : Byrd, R.H., Lu, P., Nocedal, J. and Zhu, C., 1995. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing, 16(5), pp.1190-1208.

Refer the below code chunk to under the L-BFGS-B method from optim function

# optimizing values for a,b using L-BFGS-B inputs
L_BFGS_B_answer<-optim(par=c(0.1,0.2),fn=foroptim,method = "L-BFGS-B")

# the outputs
L_BFGS_B_answer$par # estimated values for a, b
L_BFGS_B_answer$value # minimized function value 
L_BFGS_B_answer$counts  # see the documentation to understand
L_BFGS_B_answer$convergence # indicates successful completion
L_BFGS_B_answer$message # additional information

# fitting the Beta-Binomial distribution with estimated shape parameter  values
fitBetaBin(Alcohol_data$Days,Alcohol_data$week1,
           L_BFGS_B_answer$par[1],L_BFGS_B_answer$par[2])

SANN method

Method SANN is by default a variant of simulated annealing given in Belisle (1992). Simulated-annealing belongs to the class of stochastic global optimization methods. It uses only function values but is relatively slow. It will also work for non-differential functions. This implementation uses the Metropolis function for the acceptance probability.

By default the next candidate point is generated from a Gaussian Markov kernel with scale proportional to the actual temperature. If a function to generate a new candidate point is given, method SANN can also be used to solve combinatorial optimization problems. Temperatures are decreased according to the logarithmic cooling schedule as given in Belisle (1992, p.890); specifically, the temperature is set to \(temp / log(((t-1) %/% tmax)*tmax + exp(1))\), where \(t\) is the current iteration step and temp and tmax are specifiable via control.

Note that the SANN method depends critically on the settings of the control parameters. It is not a general-purpose method but can be very useful in getting to a good value on a very rough surface.

Reference : Belisle, C.J., 1992. Convergence theorems for a class of simulated annealing algorithms on R d. Journal of Applied Probability, 29(4), pp.885-895.

Below mentioned code chunk is simply using SANN method for optim function

# optimizing values for a,b using default inputs
SANN_answer<-optim(par=c(0.1,0.2),fn=foroptim,method = "SANN")

# the outputs
SANN_answer$par # estimated values for a, b
SANN_answer$value # minimized function value 
SANN_answer$counts  # see the documentation to understand
SANN_answer$convergence # indicates successful completion
SANN_answer$message # additional information

# fitting the Beta-Binomial distribution with estimated shape parameter  values
fitBetaBin(Alcohol_data$Days,Alcohol_data$week1,
           SANN_answer$par[1],SANN_answer$par[2])

Brent method

Brent Method is for one-dimensional problems only, using optimize(). It can be useful in cases where optim() is used inside other functions where only method can be specified, such as in mle from package stats4. Brent method does not work for our situation.

Reference : Brent, R.P., 2013. Algorithms for minimization without derivatives. Courier Corporation.