An excursion into Markov chains

This textbook will present, in a rigorous way, the basic theory of the discrete-time and the continuous-time Markov chains, along with many examples and solved problems. For both the topics a simple model, the Random Walk and the Poisson Process respectively, will be used to anticipate and illustrate the most interesting concepts rigorously defined in the following sections. A great attention will be paid to the applications of the theory of the Markov chains and many classical as well as new results will be faced in the book. This textbook is intended for a basic course on stochastic processes at an advanced undergraduate level and the background needed will be a first course in probability theory. A big emphasis is given to the computational approach and to simulations.

1. Introduction.- 2. Discrete-time Markov chains.- 2.1. Motivation: the random walk.- 2.2. Definitions. Basic properties. Transition matrix.- 2.3. Stopping time. Strong Markov property.- 2.4. Recurrent and transient states. Equivalence classes.- 2.5. Asymptotic behaviour. Invariant distribution.- 2.6. Ergodic theorem.- 2.7. Mores aspects of the random walk.- 3. Continuous-time Markov chains.- 3.1. Motivation: the Poisson process.- 3.2. Basic properties. Transition matrix. Recurrent and transient states.- 3.3. Invariant distribution.- 3.4. Ergodic theorem.- 3.5. More aspects of the Poisson process.- 4. Applications.- 4.1. Sport modelling.- 4.2. Information retrieval.- 4.3. Weather forecast.