◐

on 

Random Variable

Random variable

Probability theory defines a probability space as a triple ${\Omega,X,p}$, where $\Omega$ is the space of all elementary events, $X$ are disjoint subsets of $\Omega$, called events and $p$ is a probability function that maps events in $X$ to the closed unit interval $p:X\rightarrow [0,1]$. A random variable $x:\Omega\rightarrow X$ maps from the sample space to events. The probability $p$ of a discrete random variable $x$ belongingto the event $x_i ∈ X$ as the result of a statistical process is: $p({ω ∈ Ω : x(ω) ∈ x_i}) = |x_i|/{|\Omega|}$ assuming there is a uniform probability of any elementary event $\omega$ occurring.

For any element $\omega\in \Omega$ probability of an event ${\omega|X(\omega)\leq x}$ is called a probability distribution function $Pr(x)$ of $X$.

The probability density function is defined as $p_X(x) = \frac{dPr(x)}{dx}$.

For some three events $a,b,\omega\in\Omega$, $x_i$ is a subset of the sample space $x_i={\omega|a \leq x(\omega)\leq b }$ and the probability of this event is as before, but over a continuous probability function is defined as $Pr({ω ∈ Ω : x(ω) ∈ x_i}) = \int_a^bp(x)dx$ where $p(x)$ is the probability density function.

Examples of PDFs

Gaussian distribution

$p(x)=\frac{1}{\sqrt{2\pi\sigma^2}}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$.

Uniform ($b>a$)

$p(x) =\frac{1}{b-a}$ if $a < x < b$, $p(x)=0$ otherwise.

Exponential ($\mu>0$)

$p(x) = \frac{1}{\mu}e^{-x/\mu}u(x)$

Joint, marginal, conditional probabilities

For $A,B \in X$, we have :

  1. $A=\Omega \rightarrow p(A)=1$.
  2. $A=B \rightarrow p(A)=p(B)$.
  3. $A\subset \Omega \rightarrow p(A)<1$.
  4. $A\subset B \rightarrow P(A)<P(B)$.

The outcome of a process is an event $x_i$ that is the intersection of two sets A,B: $A,B,x_i\in X, p(\omega\in A\cap B:x(\omega)\rightarrow x_i)= p(A\cap B)$.

Conditional probability is defined as $P(A|B) = \frac{P(A\cap B)}{P(B)}$. Two probabilities are considered independent of one another iff their joint probability is equal to the product of their individual probabilities: $p(x,y)=p(x)p(y)$.

In case of joint probabilities given more than one random process, multiple probability spaces are consitered: ${\Omega_x,X,p_x},{\Omega_y,Y,p_y}$. Their joint probability space is ${\Omega = \Omega_x\times \Omega_y,XY,p}$.

Marginalisation: $p(x) = \int_{-∞}^{∞} p(x,y)dy$.

Bayes’ theorem: $p(y|x) = \frac{p(x|y)p(y)}{p(x)}$. This leads to a remark that for independent variables, conditioning does not change their probabilities.

Expected value and moments of a random variable

$r^{\text{th}}$ moment of a random variable $X$ with a PDF $p(x)$ is defined as $$ E[X^r] = \int_{-∞}^{∞}x^rp(x)dx. $$ The first moment is known as expectation value $E[X]=\mu$ and has properties:

  1. $E[X+Y] = E[X] + E[Y]$
  2. $E[aX] = aE[X]$
  3. If $E[Y] = g(X)$, then $E[Y] = \int_{-∞}^{∞} g(x)p(x)dx$
  4. $E[X|Y] = \int_{-∞}^{∞}xp(x|y)dx$

The moments $E[(X-\mu)^r]$ are known as central moments. $r=2$ is known as variance $\sigma^2$ or $var(X)$.

Properties of variance:

  1. $var(X) = E[X^2]-E[X]$
  2. $var(X)>0$
  3. $var(a) = 0$

Covariance

Covariance is a measure of the joint variability of two random variables. Covariance between two RV is defined as $\sigma_{xy} = E[(X-\mu_x)(Y-\mu_y)]=\int_{-∞}^{∞}\int_{-∞}^{∞} (x-\mu_x)(y-\mu_y)p(x,y)dxdy$

  1. $var(X+Y) = var(X)+var(Y) + 2\sigma_{xy}$
  2. $var(aX) = a^2var(X)$.
  3. If $\sigma_{xy}=0$, $E[XY]=E[X]E[Y]$.
  4. If $E[XY]=0$, $X,Y$ are independent and therefore uncorrelated.
  5. If they are uncorrelated, they can still be dependent (non-linearly?).
  6. For random vectors, one has vectors of moments, .e.g. expectation vector, and covariance matrix.

Discrete random process $x_n$ is a sequence of random variables defined for every integer $n$. Auto-covariance is defined as $ACF_k = E[(x_n-\mu_x)(x_{n+k}-\mu_x)]$. Cross-correlation is defined as $CCF_k = E[x_n y_{n+k}]$.

Power spectral density is the FT of ACF, CCF: $P_{xy}(f) = \sum_{k=-∞}^{∞}CCF_ke^{-j2\pi fk}$.