A Short Note on Generating a Random Sample from Finite Mixture Distributions

Abstract

Computational statistics is a critical skill for professionals in fields such as data science, statistics, and related disciplines. One essential aspect of computational statistics is the ability to simulate random variables from specified probability distributions. Commonly employed techniques for sampling random variables include the inverse transform method, acceptance–rejection method, and Box–Muller transformation, all of which rely on sampling from the uniform $(0,1)$ distribution. A significant concept in statistics is the finite mixture model, characterized by a convex combination of multiple probability density functions. In this paper, we introduce a modified version of the composition method, a standard approach for sampling finite mixture models. Our modification offers the advantage of relying on sampling from the uniform $(0,1)$ distribution, aligning with prevalent methods in computational statistics. This alignment simplifies teaching computational statistics courses, as well as having other benefits. We offer several examples to illustrate the approach.

1. Introduction

Computational statistics has gained significant importance in recent years due to the exponential growth of data and the increasing complexity of data-driven problems. Within computational statistics, the ability to simulate or generate random samples from a probability distribution is fundamental. These generated random samples are utilized for estimating probabilities, expectations, and testing hypotheses. The inverse transform method and the acceptance–rejection method are two of the most fundamental techniques for generating random samples, and these can be found in well-known computational statistics textbooks such as Statistical Computing with R by [1]. These methods rely on generating numbers from the uniform $(0,1)$ distribution. The choice of method depends on the specific distribution being generated and the desired properties of the generated sample, such as efficiency or accuracy.

In certain cases, the data may not conform to commonly known distributions such as the normal or exponential distributions. Instead, they can be represented as a finite mixture model, which combines multiple probability density functions in a convex manner. These models find applications in various scientific domains. For instance, normal mixture distributions are used as parametric density estimators [2], whereas finite mixture models are employed in medical studies [3] and financial analyses [4]. Finite mixture models have also been used by [5] in the analysis of wind speeds, and Ref. [6] have demonstrated their usefulness in Bayesian density estimation. Furthermore, Ref. [7] provide a comprehensive overview of the different applications of mixture models.

Sampling from finite mixture models is a standard topic covered in many computational statistics textbooks, including works by [1,8], among others. In these texts, the primary approach for sampling from finite mixture models is typically the composition method. However, although the composition method is effective, it does not directly use the uniform distribution.

The goal of this paper is to modify the standard composition algorithm by incorporating sampling from the uniform $(0,1)$ distribution to ensure consistency with primary sampling algorithms such as the inverse transform method and the acceptance–rejection method. This aspect could prove beneficial in teaching computational statistics courses, as sampling from the uniform $(0,1)$ distribution becomes a standard step in various sampling algorithms.

The remainder of this paper is organized as follows. Section 2 provides a relevant background on finite mixture models and discusses the proposed modification. Section 3 presents several examples demonstrating the effectiveness of the proposed method. Finally, Section 4 offers concluding remarks.

2. Finite Mixture Models and Simulation Theorem

In this section, we define a finite mixture model and introduce a theorem for sampling this model via an adaptation of the composition method. The proof of this theorem is also included.

A finite mixture model is a statistical model that represents a probability distribution as a mixture of several component distributions. Mathematically, given k component distributions $f_1\left(x\right),\dots ,f_k\left(x\right)$, each with associated mixing probabilities (also known as mixing weights) ${\pi }_1,\dots ,{\pi }_k$, a finite mixture model $f\left(x\right)$ is defined as:

$$f\left(x\right)=\sum _{i=1}^k{\pi }_i{f}_i\left(x\right),$$

(1)

where $0\le {\pi }_i\le 1$ and ${\sum }_{i=1}^k{\pi }_i=1$. Further insights into Equation (1) can be found in studies by [9,10].

In the literature, simulating a variable from a finite k-mixture distribution is typically carried out by the composition method [1,11]:

• Generate an integer $I\in \{1,\dots ,k\}$ such that

  $$P(I=i)={\pi }_i,\mathrm{for} i=1,\dots ,k;$$
• Deliver X with cumulative distribution function $F_I.$

The following theorem introduces an algorithm for generating a sample from (1). This theorem presents a modified version of the composition method, utilizing the uniform distribution. Aligning with well-established algorithms such as the inverse transform and acceptance–rejection method enhances accessibility for learners.

Theorem 1.

Consider $F\left(x\right)$ as defined in (1). The following algorithm generates a random variate from X with the cumulative distribution function $F\left(x\right)$:

1. 

Generate a random u from the uniform (0, 1) distribution;

2. 

If ${\displaystyle \sum _{i=1}^{l-1}}{\pi }_i\le u<{\displaystyle \sum _{i=1}^l}{\pi }_i$, generate a random x from $F_l\left(x\right)$, where $l=1,\dots ,k$, with the convention that ${\displaystyle \sum _{i=1}^0}{\pi }_i=0$.

Proof. 

We show that the generated sample has the same distribution as X. By the law of total probability, we have

$$\begin{array}{ccc}\hfill P(X\le x)& =& {\int }_0^{1}P(X\le x|U=u)du\hfill \\ & =& {\int }_0^{{\pi }_1}P(X\le x|U=u)du+{\int }_{{\pi }_1}^{{\pi }_1+{\pi }_2}P(X\le x|U=u)du+\hfill \\ & & \cdots +{\int }_{{\sum }_{i=1}^{l-1}{\pi }_i}^{{\sum }_{i=1}^l{\pi }_i}P(X\le x|U=u)du+\hfill \\ & & \cdots +{\int }_{{\sum }_{i=1}^{k-1}{\pi }_i}^{1}P(X\le x|U=u)du\hfill \\ & =& {\int }_0^{{\pi }_1}{F}_1\left(x\right)du+{\int }_{{\pi }_1}^{{\pi }_1+{\pi }_2}{F}_2\left(x\right)du+\hfill \\ & & \cdots +{\int }_{{\sum }_{i=1}^{l-1}{\pi }_i}^{{\sum }_{i=1}^l{\pi }_i}{F}_l\left(x\right)du+\cdots +{\int }_{{\sum }_{i=1}^{k-1}{\pi }_i}^1{F}_k\left(x\right)du\hfill \\ & =& {\pi }_1{F}_1\left(x\right)+{\pi }_2{F}_2\left(x\right)+\cdots +{\pi }_l{F}_l\left(x\right)+\cdots +{\pi }_k{F}_k\left(x\right)\hfill \\ & =& \sum _{i=1}^k{\pi }_i{F}_i\left(x\right)=F\left(x\right).\hfill \end{array}$$

   □

The proof of Theorem 1 reveals that the approach is overly general, encompassing not only mixtures of continuous distributions but also extending to other scenarios. This includes mixtures involving continuous and discrete distributions, as well as mixtures comprising only discrete distributions. Additionally, the framework can be extended to sample mixtures of multivariate distributions. In the following section, we explore specific examples that illustrate these various cases.

3. Examples

In this section, we demonstrate the proposed algorithm outlined in Theorem 1 with six illustrative examples. The R code is provided in the Supplementary Materials.

Example 1.

Mixture of three normal distributions [10].

Suppose $X_1\sim N(\mu =0,{\sigma }^2=1)$, $X_2\sim N(\mu =5,{\sigma }^2=0.25)$, and $X_3\sim N(\mu =2,{\sigma }^2=9)$ are independent. Let

$$F\left(x\right)=0.3F_1\left(x\right)+0.5F_2\left(x\right)+0.2F_3\left(x\right).$$

Using Theorem 1, we generated a sample of size ${10}^6$ from $F\left(x\right)$. Figure 1 shows the histogram of the generated sample with the true density superimposed. It is evident from Figure 1 that the proposed method performs exceptionally well in this example.

Example 2.

Mixture of five gamma distributions: different shapes with same scale parameters [1].

Consider $F\left(x\right)={\sum }_{i=1}^5{\pi }_i{F}_i\left(x\right),$ where $X_i\sim g$amma$(r=3,{\beta }_i=i)$ are independent and the mixing probabilities are ${\pi }_i=i/15,i=1,\dots ,5$. Using Theorem 1, we generated a sample of size ${10}^6$ from $F\left(x\right)$. Figure 2 displays the histogram plot of the generated sample with the true density superimposed. The proposed procedure also performs well in this example.

Example 3.

Mixture of five gamma distributions: different scale with same shape parameters.

Let $F\left(x\right)$ be as described in Example 2, with $X_i\sim \mathrm{gamma}(r_i=i,{\beta }_i=3)$. Employing Theorem 1, we generated a sample of size ${10}^6$ from $F\left(x\right)$. Figure 3 presents the histogram plot of the generated sample with the true density superimposed. The proposed procedure demonstrates effective performance in this example as well.

Example 4.

Comparing empirical and true mixed distributions.

In this example, we compare $F_n\left(x\right)$, the empirical cumulative distribution function (ECDF) of the simulated data, with the true mixed distribution

$$F\left(x\right)=\sum _{i=1}^3{\pi }_i{F}_i\left(x\right),$$

where $F_i$ represents three cases:

• Case 1: $X_1\sim t\left(5\right)$, $X_2\sim t\left(10\right)$, and $X_3\sim t\left(15\right)$. Here, $t\left(\nu \right)$ represents the t distribution with $\nu$ degrees of freedom;
• Case 2: $X_1\sim \mathrm{beta}(2,5)$, $X_2\sim \mathrm{beta}(2,10)$, and $X_3\sim \mathrm{beta}(2,15)$;
• Case 3: $X_1\sim \mathrm{Pareto}(1,1)$, $X_2\sim \mathrm{Pareto}(2,2.5)$, and $X_3\sim \mathrm{Pareto}(3,3)$. Here, $\mathrm{Pareto}(x_m,\alpha )$ is the Pareto distribution with $x_m$ as the minimum possible value (scale parameter) and $\alpha$ as the shape parameter.

In all three cases, we let ${\pi }_1=9/20,{\pi }_2=9/20,$ and ${\pi }_3=1/10$. As a measure of proximity, we utilize the Cramér–von Mises distance defined as

$$D=\int {\left(F_n\left(x\right)-F\left(x\right)\right)}^{2}dF\left(x\right).$$

We examine various sample sizes $n\in \left\{20,50,100,1000\right\}$. For each generated sample $X_1,\dots ,X_n$, we estimate D using

$$\widehat{D}=\frac{1}{n}\sum _{i=1}^n{\left(F_n\left(X_i\right)-F\left(X_i\right)\right)}^2.$$

For each sample, we compute ${10}^4$ values of $\widehat{D}$ and report $\overline{\widehat{D}}$ and $\mathrm{sd}\left(\widehat{D}\right)$, representing the mean and standard deviation of the ${10}^4$ values of $\widehat{D}$. Additionally, for comparison, we include results obtained using samples generated from the composition method described in Section 2. The results are reported in

Table 1. Comparison of proposed and composition methods.

		Proposed		Composition
$\mathit{F}\left(\mathit{x}\right)$	$\mathit{n}$	$\overline{\widehat{\mathit{D}}}$	$\mathit{sd}\left(\widehat{\mathit{D}}\right)$	$\overline{\widehat{\mathit{D}}}$	$\mathit{sd}\left(\widehat{\mathit{D}}\right)$
Case 1	20	0.008771	0.007934	0.008814	0.008036
	40	0.004291	0.003903	0.004295	0.003871
	60	0.002812	0.002515	0.002851	0.002566
	80	0.002126	0.00188	0.002109	0.001861
	100	0.001687	0.001539	0.001674	0.001517
Case 2	20	0.008795	0.008048	0.008814	0.008207
	40	0.004269	0.003854	0.004325	0.003945
	60	0.002822	0.00256	0.002842	0.002528
	80	0.002087	0.00189	0.002108	0.001951
	100	0.001692	0.001546	0.0017	0.001518
Case 2	20	0.008813	0.008194	0.008857	0.008049
	40	0.004299	0.003889	0.004293	0.003844
	60	0.002842	0.002514	0.00285	0.002601
	80	0.002148	0.001903	0.002173	0.002008
	100	0.001678	0.001485	0.001711	0.001569

. It is clear that both simulation algorithms work well as both $\overline{\widehat{D}}$ and $\mathrm{sd}\left(\widehat{D}\right)$ approach zero, especially as we increase the sample size.

Example 5.

Mixture of four binomial distributions [12].

Consider

$$F\left(x\right)=\sum _{i=1}^4{\pi }_i{F}_i\left(x\right),$$

where $X_i\sim \mathrm{binomial}(m=10,{\theta }_i)$ are independent, with ${\theta }_1=0.1$, ${\theta }_2=0.2$, ${\theta }_3=0.6$, and ${\theta }_4=0.9$. The mixing probabilities are ${\pi }_1={\pi }_2={\pi }_3=0.2$ and ${\pi }_4=0.4$. Using Theorem 1, a sample of size ${10}^6$ was generated from $F\left(x\right)$. For comparison, we analyzed the theoretical mean and variance alongside the sample mean and variance. As stated by [9], we have $E\left[X\right]=\mu ={\sum }_{i=1}^4{\pi }_i{\mu }_i$ and $V\left[X\right]={\sum }_{i=1}^4{\pi }_i{\sigma }_i^2+{\sum }_{i=1}^4{\pi }_i{({\mu }_i-\mu )}^2$. In this example, ${\mu }_i=m{\theta }_i$ and ${\sigma }_i^2=m{\theta }_i(1-{\theta }_i)$. Thus, $\mu =16.20$ and ${\sigma }^2=106.98$. Additionally, the sample mean and variance are 16.2050 and 106.9789, respectively. This indicates a close correspondence between the theoretical and sample statistics.

Example 6.

Mixture of normal and Poisson distributions.

Consider the mixture distribution given by

$$F\left(x\right)=0.7F_1\left(x\right)+0.3F_2\left(x\right),$$

where $X_1$ follows a normal distribution with mean 10 and variance 4 and $X_2$ follows a Poisson distribution with mean 4. $X_1$ and $X_2$ are independent. Utilizing Theorem 1, a sample of size ${10}^6$ was generated from $F\left(x\right)$. As in Example 4, the exact mean and the exact variance of the mixture distribution are $\mu =0.7\times 10+0.3\times 5=8.50$ and ${\sigma }^2=0.7(4+{(10-\mu )}^2)+0.3(5+{(5-\mu )}^2)=9.55$. Additionally, the simulated mean and variance of the mixture distribution are 8.4963 and 9.5485, respectively. This demonstrates a close correspondence between the theoretical and sample statistics.

4. Conclusions

This paper introduces a modified version of the composition method for sampling finite mixture distributions. By incorporating sampling from the uniform $(0,1)$ distribution, our modification aligns with prevalent methods in computational statistics, such as the inverse transform and acceptance–rejection methods. This modification not only enhances the consistency and accuracy of sampling procedures but also simplifies the teaching of computational statistics courses, where sampling from the uniform $(0,1)$ distribution is a common step in various algorithms.

The effectiveness of the proposed modification is demonstrated through several illustrative examples, showcasing its robust performance across different scenarios. From mixtures of normal and gamma distributions to binomial and Poisson mixtures, the proposed algorithm consistently generates samples that closely match the theoretical distributions. Moreover, comparison metrics such as the Cramér–von Mises distance provide quantitative evidence of the algorithm’s efficiency and accuracy, especially as sample sizes increase.

Overall, the modified composition method presented in this paper offers a valuable addition to the toolkit of computational statisticians and educators alike. Its simplicity, consistency, and performance make it a practical choice for sampling finite mixture distributions in various applications.