Bayesian Programming

Probability as an Alternative to Boolean LogicWhile logic is the mathematical foundation of rational reasoning and the fundamental principle of computing, it is restricted to problems where information is both complete and certain. However, many real-world problems, from financial investments to email filtering, are incomplete or uncertain in nature. Probability theory and Bayesian computing together provide an alternative framework to deal with incomplete and uncertain data. Decision-Making Tools and Methods for Incomplete and Uncertain DataEmphasizing probability as an alternative to Boolean logic, Bayesian Programming covers new methods to build probabilistic programs for real-world applications. Written by the team who designed and implemented an efficient probabilistic inference engine to interpret Bayesian programs, the book offers many Python examples that are also available on a supplementary website together with an interpreter that allows readers to experiment with this new approach to programming. Principles and Modeling Only requiring a basic foundation in mathematics, the first two parts of the book present a new methodology for building subjective probabilistic models. The authors introduce the principles of Bayesian programming and discuss good practices for probabilistic modeling. Numerous simple examples highlight the application of Bayesian modeling in different fields. Formalism and AlgorithmsThe third part synthesizes existing work on Bayesian inference algorithms since an efficient Bayesian inference engine is needed to automate the probabilistic calculus in Bayesian programs. Many bibliographic references are included for readers who would like more details on the formalism of Bayesian programming, the main probabilistic models, general purpose algorithms for Bayesian inference, and learning problems. FAQsAlong with a glossary, the fourth part contains answers to frequently asked questions. The authors compare Bayesian programming and possibility theories, discuss the computational complexity of Bayesian inference, cover the irreducibility of incompleteness, and address the subjectivist versus objectivist epistemology of probability. The First Steps toward a Bayesian ComputerA new modeling methodology, new inference algorithms, new programming languages, and new hardware are all needed to create a complete Bayesian computing framework. Focusing on the methodology and algorithms, this book describes the first steps toward reaching that goal. It encourages readers to explore emerging areas, such as bio-inspired computing, and develop new programming languages and hardware architectures.

Introduction Probability an alternative to logic A need for a new computing paradigm A need for a new modeling methodology A need for new inference algorithms A need for a new programming language and new hardware A place for numerous controversies Running real programs as exercises Bayesian Programming PrinciplesBasic Concepts Variable Probability The normalization postulate Conditional probability Variable conjunction The conjunction postulate (Bayes theorem) Syllogisms The marginalization rule Joint distribution and questions Decomposition Parametric forms Identification Specification = variables + decomposition + parametric forms Description = specification + identification Question Bayesian program = description + question Results Incompleteness and Uncertainty Observing a water treatment unitLessons, comments, and notes Description = Specification + Identification Pushing objects and following contoursDescription of a water treatment unitLessons, comments, and notes The Importance of Conditional Independence Water treatment center Bayesian model (continuation) Description of the water treatment centerLessons, comments, and notes Bayesian Program = Description + Question Water treatment center Bayesian model (end) Forward simulation of a single unitForward simulation of the water treatment centerControl of the water treatment centerDiagnosisLessons, comments, and notes Bayesian Programming Cookbook Information Fusion "Naive" Bayes sensor fusion Relaxing the conditional independence fundamental hypothesis Classification Ancillary clues Sensor fusion with false alarmInverse programming Bayesian Programming with Coherence Variables Basic example with Boolean variablesBasic example with discrete variablesChecking the semantic of ?Information fusion revisited using coherence variablesReasoning with soft evidence Switch Cycles Bayesian Programming Subroutines The sprinkler modelCalling subroutines conditioned by values Water treatment center revisited (final) Fusion of subroutines Superposition Bayesian Programming Conditional Statement Bayesian if-then-else Behavior recognitionMixture of models and model recognition Bayesian Programming Iteration Generic iteration Generic Bayesian filters Markov localization Bayesian Programming Formalism and Algorithms Bayesian Programming FormalismLogical propositions Probability of a proposition Normalization and conjunction postulates Disjunction rule for propositions Discrete variables Variable conjunction Probability on variables Conjunction rule for variables Normalization rule for variables Marginalization rule Bayesian program Description Specification Questions Inference Bayesian Models Revisited General purpose probabilistic modelsEngineering-oriented probabilistic modelsCognitive-oriented probabilistic models Bayesian Inference Algorithms RevisitedStating the problem Symbolic computationNumerical computation: General sampling algorithms for approximate Bayesian inferenceApproximate inference in ProBT Bayesian Learning Revisited Parameter identification Expectation-Maximization (EM)Learning structure of Bayesian networks Frequently Asked Questions and Frequently Argued Matter Frequently Asked Question and Frequently Argued Matter Alternative Bayesian inference enginesBayesian programming applications Bayesian programming vs. Bayesian networks Bayesian programming vs. Bayesian modeling Bayesian programming vs. possibility theories Bayesian programming vs. probabilistic programming Computational complexity of Bayesian inference Cox theorem Discrete vs. continuous variables Incompleteness irreducibility Maximum entropy principle justifications Noise or ignorance? Objectivism vs. subjectivism controversy and the "mind projection fallacy" Unknown distribution Glossary Bayesian filter Bayesian inference Bayesian network Bayesian program Coherence variable Conditional statement Decomposition Description Forms Incompleteness Mixture Noise Preliminary knowledge Question SpecificationSubroutines Variable Index

Pierre Bessiere is with CNRS, the French National Centre for Scientific Research. Juan-Manuel Ahuactzin, Kamel Mekhnacha, and Emmanuel Mazer are with Probayes Inc., France.