Statistical basis for pharmacometrics: random variables and their distribution functions, expected values, and correlation coefficient

When the population is too big to study all, it is necessary to randomly sample a proper size of data in order to figure out its distribution including center and dispersion (Fig. 1). Let us define two important parameters µ and σ² as the true mean and variance of the population which are unknown unless the whole population is studied. When µ and σ² are estimated from the sample, the probability theory provides their estimators with the theoretical background like how close the estimators are to the true values.

Suppose X₁,X₂, ..., X_n are a random sample of size n. Since each of them is supposed to be independent and identically represent the population while each observation X_i,i=1, 2, ···, n varies. The histogram of a sample is supposed to predict the distribution curve of the population. As an analogy of the population mean and variance, we can define the sample mean X̄ and variance S² as follows:

For the notation of estimator, we often use hats such as µ̂ = X̄, σ̂² = S². Naturally we expect that X̄ converges to µ and S² converges to σ² as n is large enough. Then, the probability theory provides the mathematical basis to those asymptotic properties.

Whenever the readers want more details, they can refer to the following excellent literatures on mathematical statistics: Hogg and Craig 4th ed. (pp 1-149),[1] Hogg and Tanis 3rd ed. (pp 1-60),[2] Kim et al. 4th ed. (pp 1-93),[3] Mood 3rd ed. (pp 1-84),[4] Peebles 4th (pp 1-202),[5] Rice 3rd ed. (pp 1-176),[6] Rosner 6th ed. (pp 1-121),[7] and Ross 8th ed.[8]

In this tutorial, we start from defining the probability. The sections are composed of probability, expectations, the mean and the variance, the moment generating function, and finally the correlation coefficient. Still, the readers are encouraged to review derivative and integral of polynomial function and exponential funcion if they want to follow the examples of continuous cases. Otherwise, they can skip those examples. The contents are proper for a lecture of two or three hours. The readers who want more subject related examples can look into the biostatistics book by Rosner.[7] This tutorial focuses on mathematics and logic.

Let us first define some words in view of experiments as follows [2]:

Here are some examples.

Example 1 Sample space and events

Suppose we toss a die and observe the upface number. "Toss a die" is a trial, "the number of the upface" is an outcome. The sample space is S = {1, 2, 3, 4, 5, 6}. Let the event A be multiples of 3 in a die toss, and the event B be the minimum outcome. Then A = {3, 6}, B = {1}, and the two events A and B are exclusive.

Let us now define the probability of an event A, P(A).

Definition 1 For a given event A the probability is defined by

where n(A) is the size or number of outcomes of the event A.

Then the three axioms of probability hold for P.

Axiom 1 Three axioms of probability

The associated properties are as follows:

Definition 2 For A, B ⊆ S, the conditional probability P(A|B) is said to be the probability of the event A given that the event B has occurred and is defined by

if P(B) > 0.

Example 2 Probability and conditional probability

Suppose we roll a die and observe the number of the upface. Then S = {1, 2, 3, 4, 5, 6}. Let A = {1} and B = {1, 3, 5}, a set of odd numbers, and C = {2, 3}. Then, P(A)=16, P(B)=12, P(C)=13,P(B∩C) = 1/6 .

If we have an information that the outcome is odd, then P({1}|odd) is one of the three. Thus Note that we divide n(A∩B) by n(B) because the sample space S becomes B. Here,[2] Therefore,

From the definition of the conditional probability, the multiplication rule is obtained.

Theorem 1 Multiplication rule

Example 3 Sensitivity and specificity [7]

Conditional probability is very useful in screening tests. Suppose we have a table of a new test result for a disease. There are two widely used conditional probabilities to measure test abilities.

There are two types of error probabilities False Positive (FP) rate and False Negative (FN) rate as follows:

Let us also think about the conditional probability of having the disease given the + test result.

Similarly, the conditional probability of having the disease given the - test result is as follows: The last two probabilities are related to the famous Bayes' formula which we want to skip its derivation. Let us still take a look at an interesting small example in a signal trasmission problem through a channel.

Example 4 Bayes' formula. [4]

Let us transmit a binary signal through a channel. There are four cases: (1) send 0 and receive 0, (2) send 0 and receive 1, (3) send 1 and receive 0, (4) send 1 and receive 1. Let us dene events S0 as send 0, S1 as send 1, R0 as receive 0, R1 as receive 1. Suppose that P(S0) = 0.3, P(S1) = 0.7, and P(R0|S0) = P(R1|S1) = 0.99, P(R1|S0) = P(R0|S1) = 0.01. Then what is the probability that the received signal 0 is the true signal? Then we need to calculate

Note that there are only two cases of receiving 0 for the denominator: (1) send 0 and receive 0 given that 0 is sent (2) send 1 and receive 0 given that 1 is sent.

If events A and B are independent each other, then the conditional probability of A given B would not depend on whether B has occurred or not. In other words, the occurrence of B may not change the probability of the occurrence of A. The concept of independence is very important in statistics because the assumption of independence makes all calculation much simpler and easier.

Definition 3 Events A and B are independent if P(A|B) = P(A) or P(B|A) = P(B).

From the definition of conditional probability, the following important theorem holds right away.

Theorem 2 Events A and B are independent if and only if P(A∩B) = P(A)P(B).

Proof. P(A|B) = P(A∩B)P(B) = P(A), so P(A∩B) = P(A)P(B).

Example 5 independent events [4]

Suppose we have a signal transmission system composed of two parts connected in parallel: upper part (UP) and lower part (LP). The UP has one router and the LP has two routers serially connected: 1 in the UP and 2 and 3 in the LP (Fig. 2). Let R_i be the ith router failure, and assume that each failure is independent with the failure probabilities P(R₁) = 0.005, P(R₂) = P(R₃) = 0.008. What is the failure probability of transmitting a signal from a to b?

    P(transmission failure of a signal)
  = P(UP fails ∩ LP fails)
  = P(UP fails)P(LP fails)
    by independence of UP and LP
  = P(R1)P(R2 ∪ R3)
  = P(R1)(P(R2) + P(R3) - P(R2 ∩ R3))
  = P(R1)(P(R2) + P(R3) - P(R2)P(R3))
    by independence of the three routers's failure
  = (0.005)(0.008 + 0.008 - (0.008)2)
  = 0.00007968
  

Let us now define a random variable (rv) X as a function from the sample space S to a set of real numbers R with its corresponding probability density function (pdf).

Definition 4 X(s) : S→R for s ∈ S. The probability density function (pdf) for a discrete rv X is defined by

Example 6 a rv and its pdf

Let X be the number of heads in two coins. Then

The pdf is uniquely defined for a rv. Note that the difference between a variable and a random variable depends on whether or not it has its corresponding pdf. For example, in the function for a simple line y = a+bx, both x and y are variables since their values change, but not random variables since they don't have corresponding pdf's. The rv is called discrete if its values are countable or continuous if its values are not countable.

Definition 5 For f(x) = P(X = x) to be the pdf of a discrete rv, it should satisfy the following two conditions:

Calculate the probability of event A as follows: For a discrete rv, the probability mass function is more widely used than pdf. In this tutorial, we will stick to pdf just for convenience.

Example 7 the pdf of a discrete rv

Let fx=x6, for x = 1, 2, 3 and 0 otherwise. Then PX=3=12, P23<X<94=f1+f2=16+26=12.

Definition 6 For f(x) to be a pdf of a continuous rv it should satisfy the following two conditions:

Calculate the probability of set A as follows: Note that the area under the curve f(x) is 1 and P(A) is the area under the curve f(x) for x∈A. Also P(X = x) = 0 since the area at a point is zero, so that

Example 8 the pdf of a continuous rv

For f(x) = cx, 0 < x < 1 to be a pdf, what is c?

therefore c = 2.

Definition 7 The cumulative distribution function (cdf) of a rv X is defined by

The distribution function is often used instead of cdf.

Example 9 the cdf of a discrete rv

Suppose a rv X has a pdf P(X = 0) = 0.3, P(X = 1) = 0.2, P(X = 2) = 0.5.

Then its cdf is as follows (Fig. 3):

Example 10 the cdf of a continuous rv

Suppose a rv X has a pdf f(x) = 1, 0 < x < 1. Then its cdf is as follows (Fig. 3):

From the two examples, we can easily derive the following properties of the cdf F(x):

Let us extend it to the two dimensional sample space, where we have two random variables X and Y.

Definition 8 The joint pdf of the two discrete random variables X and Y is defined by

which is positive and should satisfy For the event A, P(A) can be evaluated by For the discrete random variables X and Y with the given joint pdf f(x, y), the marginal pdf's f_X(x) and f_Y(y) are defined by If a vector (x, y) is given, then a value of the joint pdf f(x, y) is determined since it is a bivariate function. In other words, one joint pdf f(x, y) corresponds to a vector (X, Y) where both X and Y are random variables.

For continuous random variables we replace ∑ with ∫.

Definition 9 For the function f(x, y) to be the joint pdf of two continuous random variables X and Y, it should satisfy the following two conditions:

For calculation of the probability of the set A, use Its marginal pdfs are defined by More details about the joint distribution functions F(x, y) = P(X ≤ x, Y ≤ y) are found in Peebles.[5]

Theorem 3 Two random variables X and Y are independent if and only if

Proof. Let us consider the discrete case only. X and Y are independen if and only if f(x, y) = P(X = x, Y = y) = P(X = x)P(Y = y) = f_X(x)f_Y(y).

Example 11 independence of discrete random variables

Suppose we flip a coin twice and define the random variables X and Y as follows:

Then Therefore the joint pdf is given as the following table.

Let us calculate some probabilities.

To check the independence of X and Y, we calculate P(X = 0, Y = 0) and P(X = 0)P(Y = 0) and compare them. Thus, X and Y are not independent.

Example 12 independence of continuous rv's

The joint pdf f(x, y) of two continuous random variables X and Y is given by

Then the marginal pdfs are obtained as follows: Since f(x, y) = f_X(x)f_Y(y), X and Y are independent.

Let us now consider how we can estimate the center of the population distribution. The top priority estimator is the population mean µ mentioned at the beginning. The µ is the simple average of the whole observations for a rv X in the population, which is said to be the expected value of a rv X. Since we often do not know the whole observations for X and we assume its pdf, we need to define the expected value of a rv X based on its pdf.

Definition 10 The expected value of a rv X is defined by

If X is a rv, then a function g(X) is also a rv. Thus the following theorem holds immediately.

Theorem 4 For a rv X with its pdf f(x), the expected value of g(X) is obtained by

Let us look at a very simple example.

Example 13 µ of a rv

Let a rv X have a pdf as the following table.

Then

Now let us define another rv Y which is a function of X. Let Then

Due to the linearity of summation and integration, the following properties can be derived.

Theorem 5 Let a rv X have the pdf f(x).

Proof of 2.

As a measure of the data dispersion from the center µ, the variance σ² is used.

Definition 11 The variance of a rv X with its pdf f(x) is defined by

One easy way of calculating σ² is The standard deviation σ is defined by σ2. Some useful properties of Var(X) are as follows:

Example 14 the mean and the variance of a rv

Let a rv X have a distribution as the following table.

Then, µ = E[X] = 0, E[X²] = 0.4, σ² = E[X²] - (E[X])² = 0.4

Example 15 the expected values of a tranformed rv

Let E[X] = µ, Var(X) = σ²,Z=X-µσ, and T = 5Z + 1. Then, E[T] = 1, Var(T) = 5².

Let us now define the expectation of two random variables.

Definition 12 Let the rv's X and Y have a joint pdf f(x, y). Then, for the function r(X, Y),

As before the linearity of the expectation holds because of the linearity of summation and integral.

Theorem 6 Let the rv's X and Y have the joint pdf f(x, y). Then

Proof of 1.

The expectation of the linear combination of g(X) and h(Y) is the linear combination of E[g(X)] and E[h(Y)].

Proof of 2.

The expectation of product of g(X and h(Y) is the product of each expectation E[g(X)] and E[h(Y)]. We will take a look at the examples at the last section.

In order to measure the location center µ and the dispersion of the distribution σ², we need E[X] and E[X²], which we call the first moment and the second moment of the distribution.

Definition 13 The two types of moments of a rv X are defined by

where m₁ = µ, the population mean and where the variance is σ² = µ₂, the skewness of the distribution is µ₃/σ³, and the kurtosis, thickness of the distribution tails, is µ₄/σ⁴ - 3.

We define a moment generating function from which we can generate moments of the distribution.

Definition 14 The moment generating function (MGF) of a rv X is given by

Then Similarly, the nth derivative of MGF at 0 generates the nthe moment of X Since a MGF corresponds to a rv as in pdf, the MGF uniquely defines the distribution of a rv.

Example 16 the MGF of a discrete rv

Let us consider the previous rv X that has a pdf as the following table.

Then its MGF is

and

Example 17 the MGF of a continuous rv

Suppose a rv X have pdf f(x) = e^-x, x > 0. Then its MGF is for t < 1,

since limx→∞e-1-tx for t < 1. After differentiating M_X(t), we get Therefore, For the calculation, the readers should be familiar with the following integral and derivative: ∫e^axdx = (1/a)e^ax + C, (f/g)' = (f'g-fg')/g².

Two random variables X and Y can be correlated. For example, Y increases as X increases, or vise versa. As a measure of their relationship, covariance and correlation coefficient are often used.

Definition 15 The covariance of rv's X and Y are defined by

If both X and Y increase, then the covariance is positive. If Y decreases as X increases, then the covariance is negative. If X and Y are independent, then E[XY] = E[X]E[Y]. Thus, Cov(X, Y) = 0.

Theorem 7 Let X and Y be two random variables. Then

The readers are recommended to prove this theorem. If X and Y are independent, then

Note that the amount of covariance depends on the measurement units of X and Y, which means that the covariance in km is greater than that in cm. In order to make the covariance be free of measurement units, we divide it by the standard deviations of the two random variables.

Definition 16 The correlation coefficient of the two random variables X and Y is defined by

Then ρ does not depend on the measurement units and ρ = 0 if X and Y are independent.

Example 18 The correlation coefficient between two discrete random variables

Let us go back to Example 11 of flipping two coins.

Then,

Therefore The following example would be a challenging example to the readers who are familiar with double integrals.

Example 19 The correlation coefficient between two continuous random variables

Let f(x, y) = 2, 0 ≤ x + y ≤ 1. Then,

Thus Similarly, Also, Therefore,

Since the contents of this tutorial is more like a lecture note and limited, readers are strongly encouraged to read references for further contents. Hopefully, pharmacometricians who read this tutorial get more familiar with statistical notations and probability theories.