Preface

Stochastic models have become a very useful tool, indeed fundamental, for a lot of sciences and applied engineering. Whether it be in theoretical or applied physics, economy or social sciences, engineering or even music, stochastic models are there to help.

Stochastic modeling is mainly based on Markov processes, characterized by the memoryless property, conditioned on their position at present. This is the probabilistic analogue of dynamical systems, where the future position of a moving particle depends only on the current position and speed. The natural generalization of Markov processes to semi-Markov processes offers much more generality, as the sojourn time in a state can have any arbitrary distribution on the real half-line x ≥ 0, and not only an exponential distribution as in the Markovian case. Let us also notice that martingales are increasingly used for real system modeling, especially in financial mathematics and statistics.

This book is based on the book [IOS 84] published in 1984, in Romanian. We will follow the main lines of that book, but will replace quite a lot of the material. We would like to pay tribute to the memory of our co-authors of [IOS 84], Serban Grigorescu (1946–1997) and Gheorghe Popescu (1944–1989), who unfortunately died before fulfilling their creative potential.

Throughout this book, composed of seven chapters, we will focus on Markov and semi-Markov processes and on their particular cases: Poisson and renewal processes.

The aim of our book is to present stochastic models in a simple context, without complicated mathematical tools, but precise nonetheless. Emphasis is placed on comprehension of the main issues, on the development of the results linked to the phenomena discussed, as well as on the specific needs giving rise to these models.

Chapter 1 presents useful families of stochastic processes. We give special attention to martingales, Markov chains, Markov processes, and semi-Markov processes, which will be the “protagonists” of the following chapters.

In Chapter 2 we present some simple stochastic models, like urn models, Brownian motion, Poisson processes and birth and death processes, models which are used in applications individually or combined with other stochastic processes. We should stress that these models alone might be a good introduction to stochastic processes. Although these processes are particular cases of the Markov processes presented in the following chapters, they are studied separately by way of more direct techniques.

Chapter 3 is devoted to the Markovian modeling from a more systematic point of view, starting from the Markov property. The presentation is focused on models like the Ehrenfest chain and various models in genetics, storage problems, and system reliability.

The basic results on renewal models are presented in Chapter 4, together with their main applications, such as replacement and reward models, risk models in insurance and counter models.

In Chapter 5 we describe semi-Markov processes. After several basic theoretical results, we study some of the best known applications for semi-Markov models in fields like: system reliability, reservoir models, queueing systems, and digital communication channels.

In Chapter 6 the branching models are presented, especially the Bienyamé-Galton-Watson model, with the associated computation of extinction probability or of absorption time distribution and the analysis of related asymptotic properties. Some generalizations of this model, as well as some models in continuous time, are also presented.

Finally, Chapter 7 is devoted to optimal stopping models. After a description of the classic problem, the optimal stopping problem for a

Markovian structure is also presented and different resolution methods are proposed. Then we give the optimal stopping problem for a renewal structure.

This book is mainly intended for applied science and engineering students and researchers in applied sciences, but also for anybody interested in an accessible introduction to stochastic models.

We would like to thank our colleagues for their contributions to our discussions, as well as our students from Bucharest and Compiègne, whose questions helped us advance.