# Discrete Probability Distributions

There are some probability distributions that occur frequently. This is because they either have a particularly natural or simple construction. Or they arise as the limit of some simpler distribution. Here we cover

• Bernoulli random variables
• Binomial distribution
• Geometric distribution
• Poisson distribution.

(This is a section in the notes here.)

Of course there are many other important distributions. Like with counting, it is often easy when first learning about probability to think of different probability distributions as being a main destination for probability. However, it is perhaps better to think of the probability distributions that we cover now as simple building blocks that we can then be later used to construct more expressive probabilistic and statistical models.

Again we focus on probability distributions that take discrete values though shortly we will begin to discuss their continuous counterpart.

Notation. There are many different distributions which often have different parameters. For instance, shortly we will define the Binomial Distribution, which has two parameters $n$ and $p$ and we denote by $\text{Bin}(n,p)$. If a random variable $X$ has a specific distribution then we use “$\sim$” to denote this. E.g. if $X$ is a random variable with parameters $n=4$ and $p=0.2$ then we write

## Bernoulli Distribution.

Definition [Bernoulli Distribution] A random variable that is either zero or one is a Bernoulli random variable. That is we we write $X \sim \text{Bern}(p)$ if

It is a straight-forward calculation to show that

for $X \sim \text{Bern}(p)$

## Binomial Distribution.

If we take $X_1,X_2,...,X_n$ to be independent Bernoulli random variables with parameter $p$, and we add then together

then we get a Binomial distribution with parameters $n$ and $p$.

So if we consider an experiment with probability of success $p$ and we repeat an experiment $n$ times and count up the number of successes, then the resulting probability distribution is a Binomial distribution.

Let’s briefly consider the probability that $X=k$. One event where $\{ X=k \}$ occurs is when the first $k$ experiments end in success and the rest fail, $\{X_1=1, ..., X_k=1, X_{k+1} = 0 ,..., X_n = 0 \}$. Note that by independence

Indeed the probability of any individual sequence $X_1,...,X_n$ where $\sum_i X_i = k$. So how may such sequences are there? Well we have seen this before. It is the number of ways we can label $n$ points with $k$ ones (see additional remarks on combinations in Section [sec:Counting]), that is the combination $C^n_k$. Thus we have

This motivates the following definition.

Definition [Binomial Distribution] A random variable $X$ has a binomial distribution with parameters $n$ and $p$, if it has probability mass function

for $k=0,1,...n$, and we write $X \sim \text{Bin}(n,p)$.

Here are some results on Binomial distributions that might be handy.

Lemma 1. If $X \sim \text{Bin}(n,p)$, then

Lemma 2. If $X \sim \text{Bin}(n,p)$ and $Y \sim \text{Bin}(m,p)$ and are independent then

Proof. $X= \sum_{i=1}^n X_i$ for $X_i$ independent and $X_i \sim \text{Bern}(p)$, and $X= \sum_{i=n+1}^m X_i$ for $X_i$ independent and $X_i \sim \text{Bern}(p)$. So $X+Y = \sum_{i=1}^{n+m} X_i$ thus is Bernoulli with parameters $n+m$ and $p$.

Lemma 3. If $X \sim Bin(n,p)$ and $Y_1,...,Y_n$ are independent Bernoulli random variables with parameter $q$ then

Proof. Since we can write $X= \sum_{i=1}^n X_i$ for $X_i \sim \text{Bern}(p)$. Note that an equivalent way to represent the above random variable is Since $X_i Y_i \sim \text{Bern}(pq)$, then the above random variable must be $\text{Bin}(n,pq)$. $\square$

## Geometric Distribution

Suppose we throw a biased coin until the first time that it lands on heads. The distribution of the number of throws is a geometric distribution. For instance, the probability that it takes $X=5$ coin throws is the same as the probability of $4$ tails in a row and then one heads which is

where $p$ is the probability of heads. In general, the probability we need $k$ throws is

This gives the geometric distribution.

Definition [Geometric distribution] The geometric distribution with success probability $p$ is the distribution with probability mass function

for $k=1,2,...$, and we write $X \sim \text{Geo}(p)$.

The following lemma is useful for geometrics distributions but also various forms of compound interest and other applications.

Lemma 4. [Geometric Series]For $|x| <1$,

Proof.

Now subtracting gives

Thus

Differentiating the above with respect to $x$ gives

Differentiating again gives

$\square$

Lemma 4. If $X \sim \text{Geo}(p)$ then

If we throw a coin and get $8$ tails in a row, and we ask how long should we wait until we next get a heads, then (even though it might feel like we are now due a heads) it is the same as the time we would have expected when we first started throwing the coin. This is key property of the geometric distribution and its called memoryless property.

Lemma 5. [Memoryless Property] If $T \sim Geo(p)$ then, conditional on $\{ T > t \}$, the distribution of $T-t$ is geometrically distributed with parameter $p$. In otherwords $(T-t | T \geq t) \sim Geo(p)$.

Proof.

$\square$

Example [Waiting for a bus] At a bus stop, the probability that a bus arrives at any given minute is $p$ and is independent from one minute to the next.

1. What is the expect gap in the time between any two busses?
2. You arrive at the bus stop and there is no bus there. What is the expected gap between the last time a bus arrived and the next bus to arrive?

Answer. 1. The time from one bus to the next is geometric $p$, so the expected wait is $1/p$.

2. Given you at a time with no bus the time until the last bus too arrive is geometrically distributed with parameter $p$ and so is the time until the next bus to arrive. The time between this bus arrivals is thus the sum of these geometeric distributions, and so the expected time is $2/p$.

This is sometimes called the waiting time paradox. Here we see that when we turn up at the bus station the gap between the buses is twice as long as the mean time between the buses. This is because when we turn up and there is no bus there then we are more likely to have chosen a time with a bigger gap between the buses.

## Poisson Distribution.

The Poisson distribution arises when we count the number of successes of an unlikely event over a large population. This occurs in all manner of settings from nuclear decay, to insurance, to call over a telephone line.

We present a definition first and then we will motivate the Poisson distribution.

Definition [Poisson distribution] For a parameter $\lambda >0$, the Poisson distribution has probability mass function

for $k=0,1,2,...\,$ and we write $X \sim Po(\lambda)$

Motivation for Poisson Distribution. If we take a Binomial distribution where the number of trails $n$ is large but the probability of success in each trail is small, specifically $p=\lambda /n$, then the Binomial distribution is well approximated by a Poisson distribution.

This is the reason the Poisson distribution is a reasonable distribution to represent pheonomena like nuclear decay. In nuclear decay, there are a large number of atoms in a radio-active substance, and, in any given time interval, there is a very small probability of one of these atom undergoing nuclear decay and the emitting a particle (e.g. a gamma-ray). For this reason the distribution of the number of observed gamma-rays over a time interval is well approximated by a Poisson distribution.

The following lemma sets out how the Poisson distribution approximates the Binomial distribution (again students primarily interested in assessment can skip with argument).

Theorem 1 [Binomial to Poisson Limit] Consider a sequence of Binomial random variables $X^{(n)} \sim \text{Bin}(n , \frac{\lambda}{n} )$ for $n \in \mathbb N$, and let $X \sim Po(\lambda)$. Then

That is as $n$ gets large the probability of $X^{(n)}=k$ approaches the probability that $X=k$ for each $k$.

Now for some more standard facts about the Poisson distribution.

Lemma 6 [Poisson Summation Property] If $X \sim Po(\lambda)$, $Y \sim Po (\mu)$, and $X$ and $Y$ are independent then

Lemma 7 [Poisson Thinning Property] If $N \sim Po(\lambda)$ and, independent of $N$, we let $X_1,X_2,...$ be independent Bernoulli random variables with parameter $p$ then

In Lemma 6, we can begin to see how we can think of a Poisson distribution as part of a process that evolves in time. For instance we might say that the number of calls on a set of telephone lines in each minute is Poisson distributed with mean $4$, then the number of calls per hour is Poisson mean $4\times 60 = 240$.

In Lemma 7, we can see that if we exclude points according to an independent random variable then the resulting random variable is still Poisson. This is useful for instance in insurance. Here the number of claims an insurance company receives in a given day might be Poisson with mean $20$. The company might split the claims into big and small claims (say on average half the claims are big and half small). Since there is some fixed probability that each claim is, say, big then the resulting number of big claims is Poisson mean $10$. This is useful for an insurance company as they can divide up, reinsure or resell some of their risk.

Both lemmas can be proved directly by summing things but is a bit of a messy calculation. Intuitively the above lemmas holds because an equivalent results, Lemma 2 and Lemma 3, hold for Binomial random variables. So the both properties persists when we take the limit to a Poisson random variable (like in Theorem 1). The cleanest proof (using moment generating functions) is beyond the scope of this course, so we omit the proof for now.