Bayes Theory

This book is based on lectures given at Yale in 1971-1981 to students prepared with a course in measure-theoretic probability. It contains one technical innovation-probability distributions in which the total probability is infinite. Such improper distributions arise embarras­ singly frequently in Bayes theory, especially in establishing correspondences between Bayesian and Fisherian techniques. Infinite probabilities create interesting complications in defining conditional probability and limit concepts. The main results are theoretical, probabilistic conclusions derived from probabilistic assumptions. A useful theory requires rules for constructing and interpreting probabilities. Probabilities are computed from similarities, using a formalization of the idea that the future will probably be like the past. Probabilities are objectively derived from similarities, but similarities are sUbjective judgments of individuals. Of course the theorems remain true in any interpretation of probability that satisfies the formal axioms. My colleague David Potlard helped a lot, especially with Chapter 13. Dan Barry read proof. vii Contents CHAPTER 1 Theories of Probability 1. 0. Introduction 1 1. 1. Logical Theories: Laplace 1 1. 2. Logical Theories: Keynes and Jeffreys 2 1. 3. Empirical Theories: Von Mises 3 1. 4. Empirical Theories: Kolmogorov 5 1. 5. Empirical Theories: Falsifiable Models 5 1. 6. Subjective Theories: De Finetti 6 7 1. 7. Subjective Theories: Good 8 1. 8. All the Probabilities 10 1. 9. Infinite Axioms 11 1. 10. Probability and Similarity 1. 11. References 13 CHAPTER 2 Axioms 14 2. 0. Notation 14 2. 1. Probability Axioms 14 2. 2.

1 Theories of Probability.- 1.0. Introduction.- 1.1. Logical Theories: Laplace.- 1.2. Logical Theories: Keynes and Jeffreys.- 1.3. Empirical Theories: Von Mises.- 1.4. Empirical Theories: Kolmogorov.- 1.5. Empirical Theories: Falsifiable Models.- 1.6. Subjective Theories: De Finetti.- 1.7. Subjective Theories: Good.- 1.8. All the Probabilities.- 1.9. Infinite Axioms.- 1.10. Probability and Similarity.- 1.11. References.- 2 Axioms.- 2.0. Notation.- 2.1. Probability Axioms.- 2.2. Prespaces and Rings.- 2.3. Random Variables.- 2.4. Probable Bets.- 2.5. Comparative Probability.- 2.6. Problems.- 2.7. References.- 3 Conditional Probability.- 3.0. Introduction.- 3.1. Axioms of Conditional Probability.- 3.2. Product Probabilities.- 3.3. Quotient Probabilities.- 3.4. Marginalization Paradoxes.- 3.5. Bayes Theorem.- 3.6. Binomial Conditional Probability.- 3.7. Problems.- 3.8. References.- 4 Convergence.- 4.0. Introduction.- 4.1. Convergence Definitions.- 4.2. Mean Convergence of Conditional Probabilities.- 4.3. Almost Sure Convergence of Conditional Probabilities.- 4.4. Consistency of Posterior Distributions.- 4.5. Binomial Case.- 4.6. Exchangeable Sequences.- 4.7. Problems.- 4.8. References.- 5 Making Probabilities.- 5.0. Introduction.- 5.1. Information.- 5.2. Maximal Learning Probabilities.- 5.3. Invariance.- 5.4. The Jeffreys Density.- 5.5. Similarity Probability.- 5.6. Problems.- 5.7. References.- 6 Decision Theory.- 6.0. Introduction.- 6.1. Admissible Decisions.- 6.2. Conditional Bayes Decisions.- 6.3. Admissibility of Bayes Decisions.- 6.4. Variations on the Definition of Admissibility.- 6.5. Problems.- 6.6. References.- 7 Uniformity Criteria for Selecting Decisions.- 7.0. Introduction.- 7.1. Bayes Estimates Are Biased or Exact.- 7.2. Unbiased Location Estimates.- 7.3. Unbiased Bayes Tests.- 7.4. Confidence Regions.- 7.5. One-Sided Confidence Intervals Are Not Unitary Bayes.- 7.6. Conditional Bets.- 7.7. Problems.- 7.8. References.- 8 Exponential Families.- 8.0. Introduction.- 8.1. Examples of Exponential Families.- 8.2. Prior Distributions for the Exponential Family.- 8.3. Normal Location.- 8.4. Binomial.- 8.5. Poisson.- 8.6. Normal Location and Scale.- 8.7. Problems.- 8.8. References.- 9 Many Normal Means.- 9.0. Introduction.- 9.1. Baranchik’s Theorem.- 9.2. Bayes Estimates Beating the Straight Estimate.- 9.3. Shrinking towards the Mean.- 9.4. A Random Sample of Means.- 9.5. When Most of the Means Are Small.- 9.6. Multivariate Means.- 9.7. Regression.- 9.8. Many Means, Unknown Variance.- 9.9. Variance Components, One Way Analysis of Variance.- 9.10. Problems.- 9.11. References.- 10 The Multinomial Distribution.- 10.0. Introduction.- 10.1. Dirichlet Priors.- 10.2. Admissibility of Maximum Likelihood, Multinomial Case.- 10.3. Inadmissibility of Maximum Likelihood, Poisson Case.- 10.4. Selection of Dirichlet Priors.- 10.5. Two Stage Poisson Models.- 10.6. Multinomials with Clusters.- 10.7. Multinomials with Similarities.- 10.8. Contingency Tables.- 10.9. Problems.- 10.10. References.- 11 Asymptotic Normality of Posterior Distributions.- 11.0. Introduction.- 11.1. A Crude Demonstration of Asymptotic Normality.- 11.2. Regularity Conditions for Asymptotic Normality.- 11.3. Pointwise Asymptotic Normality.- 11.4. Asymptotic Normality of Martingale Sequences.- 11.5. Higher Order Approximations to Posterior Densities.- 11.6. Problems.- 11.7. References.- 12 Robustness of Bayes Methods.- 12.0. Introduction.- 12.1. Intervals of Probabilities.- 12.2. Intervals of Means.- 12.3. Intervals of Risk.- 12.4. Posterior Variances.- 12.5. Intervals ofPosterior Probabilities.- 12.6. Asymptotic Behavior of Posterior Intervals.- 12.7. Asymptotic Intervals under Asymptotic Normality.- 12.8. A More General Range of Probabilities.- 12.9. Problems.- 12.10. References.- 13 Nonparametric Bayes Procedures.- 13.0. Introduction.- 13.1. The Dirichlet Process.- 13.2 The Dirichlet Process on (0, 1).- 13.3. Bayes Theorem for a Dirichlet Process.- 13.4. The Empirical Process.- 13.5. Subsample Methods.- 13.6. The Tolerance Process.- 13.7. Problems.- 13.8. References.- Author Index.