ch4: Discrete random variables

Random variable

A random variable is called discrete if it can take only a finite or countably infinite number of values. The range $S_x$ of a random variable is the set of all possible realizations (values) $X(s)$. A random variable is discrete if the range is denumerable.

Range

The range of a variable (not neccesarily random) can be:

• Finite
• Countably infinite
• Not countably infinite

The probability function of a discrete random variable

If $x$ is a discrete random variable we'll call the function that assigns a probablilty $P(X=x)$ to each $x$ which is an elemant of $S_x$ the probab lity function of $X$. The sum of all the probablities from this function equal 1.

Properties Probability function

• $\mathit{P}(\mathit{X}=\mathit{x})\ge \mathbf{0}\text{ for }x\in S_X$
• $\sum _{x\in S_X}\mathit{P}(\mathit{X}=\mathit{x})=\mathbf{1}$

The above also means that a function which statisfies both properties is a probablity function.

The probablities $$P(X \in B)$$ for each $$B \subset S_{X}$$ are, all together, called the (probability) distribution of the random variable $X$. If all probablities are equal we would say that $X$ has a homogenous distribution.

Geometric series

$$\sum \mathrm{\_}{k=0}^{\mathrm{\infty }}{x}^k=\frac{1}{1-x}$$

The expectation of a discrete random variable

The expectation or expected value $E(X)$ of a discrete random variable 𝑋 is given by:

$$\mathit{E}(\mathit{X})=\sum _{x\in S_X}x\mathit{P}(\mathit{X}=\mathit{x})$$

provided that this summation absolute convergent is (that is: $$\sum_{x \in S_{X}}|x| \cdot P(X=x)<\infty$$).

TIP

Expectation $E(X)$ is the average or mean value of $X$

If this the summation converges (absolutely) then the expected value exists, if the summations doesn't converge (absolutely) then the expected value doesn't exist.

letter	description
$\mu$	(greek letter m, for mean) sometimes is used instead of $E(X)$. $E(X)$ is often referred to as the sample or population mean
$\overline{x}$	standing for sample mean and $\mu$ for population mean.

Functions of a discrete random variable; variance

Building further on the expectation we can define multiple imporant properties:

Functions

If $X$ is a discrete random variable and $g$ a (real) function, then:

$$\mathit{E}(\mathit{g}(\mathit{X}))=\sum _{x\in S_X}\mathit{g}(\mathit{x})\mathit{P}(\mathit{X}=\mathit{x})$$

So if $Y$ is a linear function of $X$, that is $Y=aX+b$ for any real constants $a,b\in \mathbb{R}$ then we have:

$$E(aX+b)=\sum _{x\in S_X}(ax+b)\cdot P(X=x)$$$$=\sum _{x\in S_X}ax\cdot P(X=x)+\sum _{x\in S_X}b\cdot P(X=x)$$$$=a\cdot \sum _{x\in S_X}x\cdot P(X=x)+b\cdot \sum _{x\in S_X}P(X=x)$$$$=a\cdot E(X)+b\cdot 1$$

The average can be considered a measure for the center of a distribution $X$ while the median is the value $M$ such that $$P(X \leq M) \geq 50 % \text { and } P(X \geq M) \geq 50 %$$. This however tells us nothing about the magnitude of the differences in the values of $X$. These differences a measured in a different way:

Variance and standard deviation

Notation	Name	Definition
$\mathit{v}\mathit{a}\mathit{r}\mathbf{(}\mathit{X}\mathbf{)}$	The variance of $X$	$\mathrm{var}(\mathit{X})=\mathit{E}{(\mathit{X}-{\mathit{\mu }}_X)}^2$
${\mathit{\sigma }}_{\mathit{X}}$	The standard deviation of $X$	is the square root of the variance: ${\mathit{\sigma }}_{\mathit{X}}\mathbf{=}\sqrt{\mathbf{var}\mathbf{(}\mathit{X}\mathbf{)}}$

Properties of variance and standard deviation

• $\mathrm{var}(X)\ge 0\text{ and }{\sigma }_X\ge 0$
• $\mathrm{var}(X)=E\left(X^2\right)-{\mu }_X^2$ (the computational formula)
• if $var(X)>0$, so if $X$ is not degenerate, we have $E\left(X^2\right)>(EX{)}^2$
• $\mathrm{var}(aX+b)=a^2\cdot \mathrm{var}(X)$ and ${\sigma }_{aX+b}=|a|\cdot {\sigma }_X$

Chebysshev's inequality and the Empirical rule

Formula

For any real number $c>0$, we have: $P(|X-{\mu }_X|\ge c)\le \frac{\mathrm{var}(X)}{c^2}$

Valid for any random variable $X$ and gives us an upper bound of the probability of values outside the interval $({\mathit{\mu }}_X-\mathit{c},{\mathit{\mu }}_X+\mathit{c})$

In essence Chebyshev's inequality guarantees that, for a wide class of probability distributions, no more than a certain fraction of values can be more than a certain distance from the mean. Using the inequality and standard deviation a standard interval can be build: $({\mu }_X-k\cdot {\sigma }_X,{\mu }_X+k\cdot {\sigma }_X)$ and the upper bound of probability of observing values outside this interval is $\frac{\mathrm{var}(X)}{c^2}=\frac{\mathrm{var}(X)}{k^2{\sigma }_X^2}=\frac{1}{k^2}$.

Empircal rule

If the graph of the distribution of 𝑋 shows a bell shape, then the approximately probabilities for 𝑋 having a value within the interval

• $(\mu -\sigma ,\mu +\sigma )$ is 68%
• $(\mu -2\sigma ,\mu +2\sigma )$ is 95%
• $(\mu -3\sigma ,\mu +3\sigma )$ is 99.7%

Chebyshev's rule is valid for any distribution, but the so called Empirical Rule is only valid for distributions that are (approximately) symmetric and bell (hill) shaped.

The binomial, hypergeometric, geometric and Poisson distribution

The Binomial distribution

Definition

$X$ is binomially distributed with parameters $n$ and $p$, for all $n = 1, 2, … $ and $p\in [0,1]$, if the probability function of $X$ is given by: $\mathit{P}(\mathit{X}=\mathit{k})=(\genfrac{}{}{0ex}{}{n}{k})){\mathit{p}}^k(\mathbf{1}-\mathit{p}{)}^{n-k}$, where $k=0,1,2,\dots ,n$

Short notations: $X$ is $B(n,p)$-distributed, or: $X\sim B(n,p)$

One can apply the binomial distribution as a probability model of real life situations, whenever there is a series of 𝑛 similar experiments for which the conditions of Bernoulli trials hold, i.e.:

• A phenomenon occurs (or does not occur) at a fixed success rate $p$ (or failure rate $1-p$)
• Independence of the trials.

If $X$ is $B(n,p)$-distributed, then expected value and variance are given by: $\mathit{E}(\mathit{X})=\mathit{n}\mathit{p}$ and $\mathrm{var}(\mathit{X})=\mathit{n}\mathit{p}(\mathbf{1}-\mathit{p})$

Special values of n and p, the parameters of the B(n,p)-distribution

• If $\mathit{p}=\mathbf{1}$ ("success guaranteed"), then $P(X=n)=1$ and $E(X)=n:X$ has a degenerate distribution in $n$. Similarly, if $p=0$, then $P(X=0)=1$ and $E(X)=0$.
• If $\mathit{n}=\mathbf{1}$, that is, if only one trial is conducted (one shot on the basket, the quality of one product is assessed, etc.), $X$ is said to have an alternative distribution with success probability $\mathit{p}$, which is a $B(1,p)$ -distribution.

It follows that: $P(X=1)=p$, and, $P(X=0)=1-p$

so:

$E(X)=\sum _{x}xP(X=x)=1\cdot p+0\cdot (1-p)=p$

And:

$E\left(X^2\right)=\sum _x{x}^{2}P(X=x)=1^2\cdot p+0^2\cdot (1-p)=p$

We find:

$\mathrm{var}(X)=E\left(X^2\right)-(EX{)}^2=p(1-p)$,

the variance of a $B(1,p)$ -distribution.

The Hypergeometric distribution

Definition

$X$ is hypergeometricly distributed (with parameters $N$, $R$ and $n$) if:

$$P(A_k)=\frac{(\genfrac{}{}{0ex}{}{R}{k})(\genfrac{}{}{0ex}{}{N-R}{n-k})}{(\genfrac{}{}{0ex}{}{N}{n})}$$

If the probability function of the random variable $X$ can be given by the hypergeometric formula, $X$ is said to have a hypergeometric distribution. We can apply this distribution whenever we consider a number of random draws without replacement from a so called dichotomous population: consisting of elements which do or do not have a specific property.

Random draws from a dichotomous population lead to the hypergeometric distribution of the number of “successes” if we draw without replacement, but on the other hand, if the draws are with replacement, we can use the binomial distribution: in that case the draws should be independent.

Other properties

For relatively large $R$ and $N-R$ and relatively small $n$ the hypergeometric distribution with parameters $N$, $R$ and $n$ can be approximated by a $$B\left(n, \frac{R}{N}\right)-\text { distribution }$$.

Note that the variances of the hypergeometric and binomial distributions under these conditions are almost equal: $np(1-p)\cdot \frac{N-n}{N-1}\approx np(1-p)$.

A (quite strict) rule of thumb for approximating by the binomial distribution is $N>5n^2$.

The Geometric distribution

Definition

$X$ has a geometric distribution with parameter $p\in (0,1]$, if:

$\mathit{P}(\mathit{X}=\mathit{k})=(\mathbf{1}-\mathit{p}{)}^{k-1}\mathit{p}$, where $k=1,2,\dots$

If $p=1$ the distribution is degenerate: $P(X=1)=1$. Using the properties of the geometric series (which you can find in the appendix “Mathematical Techniques” of Probability Theory for Engineers, see canvas) the following can be proven:

$$\mathit{E}(\mathit{X})=\frac{1}{\mathit{p}}\text{ en }\mathrm{var}(\mathit{X})=\frac{\mathbf{1}-\mathit{p}}{{\mathit{p}}^2}$$

The following formula is convenient whenever we have to compute a summation of geometric probabilities:

$$\mathit{P}(\mathit{X}>\mathit{k})=(\mathbf{1}-\mathit{p}{)}^k$$

The reasoning is as follows: the probability that we need more than $k$ trials to score a success equals the probability that we are not successful in the first $k$ trials.

The Poisson distribution

Definition

$X$ has a Poisson distribution with parameter $\mu >0$ if

$$\mathit{P}(\mathit{X}=\mathit{k})=\frac{{\mathit{\mu }}^k{\mathit{e}}^{-\mu }}{\mathit{k}!},\text{ for }k=0,1,2\dots$$

This is a probability function: all probabilities are at least 0 and the sum of all probabilities is 1.

Poisson probabilities are given in (cumulative) probability tables for $$ P(X \leq c)$$

Other properties

If $X$ has a $B(n,p)$-distribution with “large $n$ and small $p$”, then $X$ has approximately a Poisson distribution with parameter $\mu =np$.

A rule of thumb for applying this approximation is:

$$n>25\text{ and }np<10\text{ or }n(1-p)<10$$

These approximations are also applicable in case of "large $n$ and large $p$ "(p close to 1) ", because we noticed before that if the number of successes $X$ is $B(n,p)$ with $p$ close to 1, then the number of failures, $n-X$, is $B(n,1-p)$, with $1-p$ close to 0.