Abstract
By minimizing the energy distance, the support points (SP) method can efficiently compact big training sample into a representative point set with small size. However, when the training sample is deficient, the quality of SP will be greatly reduced. In this paper, a sequential version of SP, called sequential support point (SSP), is proposed. The new method has two appealing features. First, the construction algorithm of SSP can adaptively update the proposal density in importance sampling process based on the existing information. Second, a hyperparameter is introduced to balance the representativeness of sequentially added points with the representativeness of overall points, so that some special purpose experimental designs, such as augmented design and sliced designs, can be efficiently constructed by setting the hyperparameter.
Similar content being viewed by others
References
Ba S (2015) SLHD: Maximin-Distance (Sliced) Latin Hypercube Designs. R package version 2.1-1.https://CRAN.R-project.org/package=SLHD
Borodachov S, Hardin D, Saff E (2014) Low complexity methods for discretizing manifolds via Riesz energy minimization. Found Comput Math 14(6):1173–1208
Bugallo MF, Elvira V, Martino L, Luengo D, Miguez J, Djuric PM (2017) Adaptive importance sampling: the past, the present, and the future. IEEE Signal Process Mag 34(4):60–79
Chivers C (2012) MHadaptive: general Markov Chain Monte Carlo for Bayesian inference using adaptive Metropolis-Hastings sampling. R package version 1.1-8. https://CRAN.R-project.org/package=MHadaptive
Dick J, Kuo FY, Sloan IH (2013) High-dimensional integration: the quasi-Monte Carlo way. Acta Numer 22:133–288
Fang KT, Wang Y (1994) Number-theoretic methods in statistics. Chapman and Hall, London
Fang KT, Liu MQ, Qin H, Zhou YD (2018) Theory and Application of Uniform Experimental Designs. Springer, Singapore
Givens GH, Hoeting JA (2013) Computational statistics, 2nd edn. Wiley, Hoboken
He X (2019) Sliced rotated sphere packing designs. Technometrics 61(1):66–76
Huang C, Joseph VR, Mak S (2020) Population quasi-Monte Carlo. arXiv Preprint. arXiv: 2012.13769
Joseph VR, Dasgupta T, Tuo R, Wu CFJ (2015) Sequential exploration of complex surfaces using minimum energy designs. Technometrics 57(1):64–74
Joseph VR, Wang D, Gu L, Lyu S, Tuo R (2019) Deterministic sampling of expensive posteriors using minimum energy designs. Technometrics 61(3):297–308. https://doi.org/10.1080/00401706.2018.1552203
Liu HY, Liu MQ (2015) Column-orthogonal strong orthogonal arrays and sliced strong orthogonal arrays. Stat Sin 25(4):1713–1734
Mak S, Joseph VR (2018) Support points. Ann Stat 46(6A):2562–2592
Owen AB, Tribble SD (2005) A quasi-Monte Carlo metropolis algorithm. Proc Natl Acad Sci U S A 102(25):8844–8849. https://doi.org/10.1073/pnas.0409596102
Qian PZG (2012) Sliced Latin hypercube designs. J Am Stat Assoc 107(497):393–399
Qian PZG, Wu CFJ (2009) Sliced space-filling designs. Biometrika 96(4):945–956
Razaviyayn M, Hong M, Luo Z (2013) A unified convergence analysis of block successive minimization methods for nonsmooth optimization. SIAM J Optim 23(2):1126–1153
Santner TJ, Williams BJ, Notz WI (2018) The design and analysis of computer experiments. Springer, New York
Sun Y, Babu P, Palomar DP (2017) Majorization-minimization algorithms in signal processing, communications, and machine learning. IEEE Trans Signal Process 65(3):794–816
Székely GJ, Rizzo ML (2013) Energy statistics: a class of statistics based on distances. J Stat Plan Inference 143(8):1249–1272
Yang F, Zhou YD, Zhang XR (2017) Augmented uniform designs. J Stat Plan Inference 182:61–73
Yuan R, Guo B, Liu M (2019) Flexible sliced Latin hypercube designs with slices of different sizes. Stat Pap 62:1117–1134
Zhang AJ, Yang ZB (2020) Hyperparameter tuning methods in automated machine learning. Sci Sin Math 50(05):695–710 (in Chinese)
Acknowledgements
The authors thank the Editor, AE, and two referees for their valuable comments. This study is partially supported by the National Natural Science Foundation of China (No.11571133, 11871237).
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendix
Appendix
To prove Theorem 1, we require following Lemmas 1 and 2.
Lemma 1
Let \(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}\) and \(q_1(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N)\) is defined as (5). Then function
is the surrogate function of \(q_1\) at the current iteration \(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\), where \({\mathbf {y}}_m\ne {\mathbf {x}}_{n_c+i}^{(t)},\ \forall \ i=1,\ldots ,n_k,\ m=1,\ldots ,N\).
Proof of Lemma 1
On the one hand, according to Arithmetic-Geometric mean inequality,
Therefore, \(g_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})\ge q_1(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {y}}_m\}_{m=1}^N)\), \(\forall \ \{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}\). On the other hand,
In the light of Definition 5, the proof is complete.\(\square \)
Lemma 2
Let \(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\subseteq {\mathcal {X}}\) and \(q_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})\) is defined as (6). Then function
is the surrogate function of \(-q_2\) at the current iteration \(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\), where \(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\) is pairwise distinct and \({\mathbf {x}}_j\ne {\mathbf {x}}_{n_c+i}^{(t)},\ \forall \ i=1,\ldots ,n_k,\ j=1,\ldots ,n_c\).
Proof of Lemma 2
The tangent plane of \(-q_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c})\) at the current iteration \(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\) is
where
Therefore, \(t(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k})=u_2(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\ |\ \{{\mathbf {x}}_j\}_{j=1}^{n_c}; \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k})\). As \(q_2\) is convex about \(\{{\mathbf {x}}_{n_c+i}\}_{i=1}^{n_k}\), then \(-q_2\) is concave. According to property of concave function,
In addition, it is obvious that
In the light of Definition 5, the proof is complete.\(\square \)
Proof of Theorem 1
Based on Lemmas 1 and 2, it is easy to show that \(u^*\) is the the surrogate function of \(q_1-q_2\) at the current iteration \(\{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\), we omit it here. Noting that \(u^*\) is a separable and quadratic function for variables, then take the partial derivatives of the variables and set them equal to 0,
After a simple calculation, we can get the closed-form in (7).\(\square \)
Proof of Theorem 2
To prove this theorem, we require a lemma in Razaviyayn et al. (2013):
Lemma 3
Assume \(u({\mathbf {x}}|{\mathbf {x}}_t)\) is a surrogate function of \(f({\mathbf {x}})\) at the current iteration \({\mathbf {x}}_t\) on a closed convex set \({\mathcal {X}}\subseteq {\mathbb {R}}^p\). If \(u({\mathbf {x}}|{\mathbf {x}}_t)\) is continuous and their directional derivatives with respect to \({\mathbf {x}}\) satisfy
Then every limit point of the iterates in MM algorithm is a stationary point of \(f({\mathbf {x}})\).
Proof
(Theorem 2) According to Lemma 3 and Theorem 1, we only need to prove that \(u^*\) is continuous and the condition of directional derivative holds. As \(u^*\) is a quadratic function for variables, it is continuous. For any feasible direction set \({\mathbf {d}}=\{{\mathbf {d}}_i\}_{i=1}^{n_k} \in {\mathbb {R}}^{n_kp}\), and \({\mathbf {x}}_{n_c+i}+{\mathbf {d}}_i\in {\mathcal {X}},\ i=1,\ldots ,n_k\), then directional derivative (let \({\mathcal {M}}=\{{\mathbf {y}}_m\}_{m=1}^N\bigcup \{{\mathbf {x}}_j\}_{j=1}^{n_c}\bigcup \{{\mathbf {x}}_{n_c+i}^{(t)}\}_{i=1}^{n_k}\) for brevity)
This proves the stationary convergence of Algorithm 1.\(\square \)
Rights and permissions
About this article
Cite this article
Xiong, Z., Liu, W., Ning, J. et al. Sequential support points. Stat Papers 63, 1757–1775 (2022). https://doi.org/10.1007/s00362-022-01294-z
Received:
Revised:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00362-022-01294-z
Keywords
- Representative points
- Support points
- Importance sampling
- Sequential experimental design
- Augmented design
- Sliced design