Regression Analysis Theory, Methods, and Applications

Any method of fitting equations to data may be called regression. Such equations are valuable for at least two purposes: making predictions and judging the strength of relationships. Because they provide a way of em­ pirically identifying how a variable is affected by other variables, regression methods have become essential in a wide range of fields, including the social sciences, engineering, medical research and business. Of the various methods of performing regression, least squares is the most widely used. In fact, linear least squares regression is by far the most widely used of any statistical technique. Although nonlinear least squares is covered in an appendix, this book is mainly about linear least squares applied to fit a single equation (as opposed to a system of equations). The writing of this book started in 1982. Since then, various drafts have been used at the University of Toronto for teaching a semester-long course to juniors, seniors and graduate students in a number of fields, including statistics, pharmacology, engineering, economics, forestry and the behav­ ioral sciences. Parts of the book have also been used in a quarter-long course given to Master's and Ph.D. students in public administration, urban plan­ ning and engineering at the University of Illinois at Chicago (UIC). This experience and the comments and criticisms from students helped forge the final version.

1 Introduction.- 1.1 Relationships.- 1.2 Determining Relationships: A Specific Problem.- 1.3 The Model.- 1.4 Least Squares.- 1.5 Another Example and a Special Case.- 1.6 When Is Least Squares a Good Method?.- 1.7 A pleasure of Fit for Simple Regression.- 1.8 Mean and Variance of b0 and b1.- 1.9 Confidence Intervals and Tests.- 1.10 Predictions.- 2 Multiple Regression.- 2.1 Introduction.- 2.2 Regression Model in Matrix Notation.- 2.3 Least Squares Estimates.- 2.4 Examples 31 2..- 2.6 Mean and Variance of Estimates Under G-M Conditions.- 2.7 Estimation of ?.- 2.8 Measures of Fit 39?2.- 2.9 The Gauss-Markov Theorem.- 2.10 The Centered Model.- 2.11 Centering and Scaling.- 2.12 *Constrained Least Squares.- 3 Tests and Confidence Regions.- 3.1 Introduction.- 12 Linear Hypothesis.- 3.3 *Likelihood Ratio Test.- 3.4 *Distribution of Test Statistic.- 3.5 Two Special Cases.- 3.6 Examples.- 3.7 Comparison of Repression Equations.- 3.8 Confidence Intervals and Regions.- 4 Indicator Variables.- 4.1 Introduction.- 4.2 A Simple Application.- 4.3 Polychotomous Variables.- 4.4 Continuous and Indicator Variables.- 4.5 Broken Line Regression.- 4.6 Indicators as Dependent Variables.- 5 The Normality Assumption.- 5.1 Introduction.- 5.2 Checking for Normality.- 5.3 Invoking Large Sample Theory.- 5.4 *Bootstrapping.- 5.5 *Asymptotic Theory.- 6 Unequal Variances.- 6.1 Introduction.- 6.2 Detecting Heteroscedasticity.- 6.3 Variance Stabilizing Transformations.- 6.4 Weighing.- 7 *Correlated Errors.- 7.1 Introduction.- 7.2 Generalized Least Squares: Case When ? Is Known.- 7.3 Estimated Generalized Least Squares.- 7.4 Nested Errors.- 7.5 The Growth Curve Model.- 7.6 Serial Correlation.- 7.7 Spatial Correlation.- 8 Outliers and Influential Observations.- 8.1 Introduction.- 8.2 The Leverage.- 8.3The Residuals.- 8.4 Detecting Outliers and Points That Do Not Belong to the Model 157.- 8.5 Influential Observations.- 8.6 Examples.- 9 Transformations.- 9.1 Introduction.- 9.2 Some Common Transformations.- 9.3 Deciding on the Need for Transformations.- 9.4 Choosing Transformations.- 10 Multicollinearity.- 10.1 Introduction.- 10.2 Multicollinearity and Its Effects.- 10.3 Detecting Multicollinearity.- 10.4 Examples.- 11 Variable Selection.- 11.1 Introduction.- 11.2 Some Effects of Dropping Variables.- 11.3 Variable Selection Procedures.- 11.4 Examples.- 12 *Biased Estimation.- 12.1 Introduction 2..- 12.2 Principal Component. Regression.- 12.3 Ridge Regression.- 12.4 Shrinkage Estimator.- A Matrices.- A.1 Addition and Multiplication.- A.2 The Transpose of a Matrix.- A.3 Null and Identity Matrices.- A.4 Vectors.- A.5 Rank of a Matrix.- A.6 Trace of a Matrix.- A.7 Partitioned Matrices.- A.8 Determinants.- A.9 Inverses.- A.10 Characteristic Roots and Vectors.- A.11 Idempotent Matrices.- A.12 The Generalized Inverse.- A.13 Quadratic Forms.- A.14 Vector Spaces.- Problems.- B Random Variables and Random Vectors.- B.1 Random Variables.- B.1.1 Independent. Random Variables.- B.1.2 Correlated Random Variables.- B.1.3 Sample Statistics.- B.1.4 Linear Combinations of Random Variables.- B.2 Random Vectors.- B.3 The Multivariate Normal Distribution.- B.4 The Chi-Square Distributions.- B.5 The F and t Distributions.- B.6 Jacobian of Transformations.- B.7 Multiple Correlation.- Problems.- C Nonlinear Least Squares.- C.1 Gauss-Newton Type Algorithms.- C.1.1 The Gauss-Newton Procedure.- C.1.2 Step Halving.- C.1.3 Starting Values and Derivatives.- C.1.4 Marquardt Procedure.- C.2 Some Other Algorithms.- C.2.1 Steepest Descent Method.- C.2.2 Quasi-Newton Algorithms.- C.2.3 The Simplex Method.- C.2.4 Weighting.- C.3 Pitfalls.- C.4 Bias, Confidence Regions and Measures of Fit.- C.5 Examples.- Problems.- Tables.- References.- Author Index.