Introduction to Optimization Methods and their Application in Statistics

Optimization techniques are used to find the values of a set of parameters which maximize or minimize some objective function of interest. Such methods have become of great importance in statistics for estimation, model fitting, etc. This text attempts to give a brief introduction to optimization methods and their use in several important areas of statistics. It does not pretend to provide either a complete treatment of optimization techniques or a comprehensive review of their application in statistics; such a review would, of course, require a volume several orders of magnitude larger than this since almost every issue of every statistics journal contains one or other paper which involves the application of an optimization method. It is hoped that the text will be useful to students on applied statistics courses and to researchers needing to use optimization techniques in a statistical context. Lastly, my thanks are due to Bertha Lakey for typing the manuscript.

1 An introduction to optimization methods.- 1.1 Introduction.- 1.2 The optimization problem.- 1.3 Some simple examples.- 1.4 Minimization procedures.- 1.5 Constrained minimization.- 1.6 Summary.- 2 Direct search methods.- 2.1 Introduction.- 2.2 Univariate search methods.- 2.3 Multiparameter search methods.- 2.4 Summary.- 3 Gradient methods.- 3.1 Introduction.- 3.2 The method of steepest descent.- 3.3 The Newton—Raphson method.- 3.4 The Davidon—Fletcher—Powell method.- 3.5 The Fletcher—Reeves method.- 3.6 Summary.- 4 Some examples of the application of optimization techniques to statistical problems.- 4.1 Introduction.- 4.2 Maximum likelihood estimation.- 4.3 Maximum likelihood estimation for incomplete data.- 4.4 Summary.- 5 Optimization in regression problems.- 5.1 Introduction.- 5.2 Regression.- 5.3 Non-linear regression.- 5.4 Log-linear and linear logistic models.- 5.5 The generalized linear model.- 5.6 Summary.- 6 Optimization in multivariate analysis.- 6.1 Introduction.- 6.2 Maximum likelihood factor analysis.- 6.3 Cluster analysis.- 6.4 Multidimensional scaling.- 6.5 Summary.- Appendix: exercises.- References.