Abstract
Kriging metamodels (also called Gaussian process or spatial correlation models) approximate the Input/Output functions implied by the underlying simulation models. Such metamodels serve sensitivity analysis, especially for computationally expensive simulations. In practice, simulation analysts often know that this Input/Output function is monotonic. To obtain a Kriging metamodel that preserves this characteristic, this article uses distribution-free bootstrapping assuming each input combination is simulated several times to obtain more reliable averaged outputs. Nevertheless, these averages still show sampling variation, so the Kriging metamodel does not need to be an exact interpolator; bootstrapping gives a noninterpolating Kriging metamodel. Bootstrapping may use standard Kriging software. The method is illustrated through the popular M/M/1 model with either the mean or the 90% quantile as output; these outputs are monotonic functions of the traffic rate. The empirical results demonstrate that monotonicity-preserving bootstrapped Kriging gives higher probability of covering the true outputs, without lengthening the confidence interval.
Similar content being viewed by others
References
Ankenman B, Nelson B and Staum J (2010). Stochastic Kriging for simulation metamodeling. Operations Research 58 (2): 371–382.
Antognini B and Zagoraiou M (2010). Exact optimal designs for computer experiments via Kriging metamodelling. Journal of Statistical Planning and Inference 140 (9): 2607–2617.
Batur D and Choobineh F (2010). A quantile-based approach to system selection. European Journal of Operational Research 202 (3): 764–772.
Chen EJ (2008). Some procedures of selecting the best designs with respect to quantile. Simulation 84: 275–284.
Chen X, Ankenman B and Nelson BL (2010). The effects of common random numbers on stochastic Kriging metamodels. Working Paper, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL USA.
Den Hertog D, Kleijnen JPC and Siem AYD (2006). The correct Kriging variance estimated by bootstrapping. Journal of the Operational Research Society 57 (4): 400–409.
Dette H, Neumeyer N and Pilz KF (2006). A simple nonparametric estimator of a strictly monotone regression function. Bernoulli 12 (3): 469–490.
Efron B and Tibshirani RJ (1993). An Introduction to the Bootstrap. Chapman & Hall: New York.
Feelders AJ (2000). Prior knowledge in economic applications of data mining. Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases, Springer: New York, pp 395–400.
Forrester A, Sobester A and Keane A (2008). Engineering Design via Surrogate Modelling: A Practical Guide. Wiley: Chichester, UK.
Frazier P, Powell W and Dayanik S (2009). The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal on Computing 21: 599–613.
Gross D and Harris CM (1998). Fundamentals of Queueing Theory. Wiley: New York.
Hong LJ (2009). Estimating quantile sensitivities. Operations Research 57 (1): 118–130.
Jin X, Fu MC and Xiong X (2003). Probabilistic error bounds for simulation quantile estimators. Management Science 14 (2): 230–246.
Kleijnen JPC (2008). Design and Analysis of Simulation Experiments. Springer Science+Business Media, New York.
Kleijnen JPC (2009). Factor screening in simulation experiments: Review of sequential bifurcation. In: Alexopoulos C, Goldsman D and Wilson JR (eds). Advancing the Frontiers of Simulation: A Festschrift in Honor of George S. Fishman. Springer: New York, pp 169–173.
Kleijnen JPC, van Beers W and van Nieuwenhuyse I (2010). Constrained optimization in expensive simulation: Novel approach. European Journal of Operational Research 202: 164–174.
Law AM (2007). Simulation Modeling and Analysis. Fourth edn, McGraw-Hill: Boston.
Lehmann EL (1999). Elements of Large-Sample Theory. Springer: New York.
Lim E and Glynn PW (2006). Simulation-based response surface computation in the presence of monotonicity. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM and Fujimoto RM (eds). Proceedings of the 2006 Winter Simulation Conference, The Society for Computer Simulation International (SCS): San Diego, CA, pp 264–271.
Loeppky JL, Sacks J and Welch W (2009). Choosing the sample size of a computer experiment: A practical guide. Technometrics 51: 366–376.
Lophaven SN, Nielsen HB and Sondergaard J (2002). DACE: a Matlab Kriging Toolbox, version 2.0. IMM Technical University of Denmark: Lyngby.
Sacks J, Welch WJ, Mitchell TJ and Wynn HP (1989). Design and analysis of computer experiments. Statistical Science 4 (4): 409–435.
Santner TJ, Williams BJ and Notz WI (2003). The Design and Analysis of Computer Experiments. Springer-Verlag: New York.
Siem AYD (2007). Property preservation and quality measures in meta-models. PhD dissertation, Tilburg University, Tilburg, the Netherlands.
Simpson TW, Booker AJ, Ghosh D, Giunta AA, Koch PN and Yang R-J (2004). Approximation methods in multidisciplinary analysis and optimization: A panel discussion. Structural and Multidisciplinary Optimization 27 (5): 302–313.
Van Beers W and Kleijnen JPC (2003). Kriging for interpolation in random simulation. Journal of the Operational Research Society 54 (3): 255–262.
Velikova M (2006). Monotone models for prediction in data mining. PhD dissertation, Tilburg University, Tilburg, the Netherlands.
Yin J, Ng SH and Ng KM (2009). A study on the effects of parameter estimation on Kriging model's prediction error in stochastic simulations. In: Rossini MD, Hill RR, Johansson B, Dunkin A and Ingalls RG (eds). Proceedings of the 2009 Winter Simulation Conference, The Society for Computer Simulation International (SCS): San Diego, CA, pp 674–685.
Acknowledgements
We thank two (anonymous) referees and Dick den Hertog (Tilburg University) for their comments on earlier versions, which lead to drastic revisions.
Author information
Authors and Affiliations
Corresponding author
Additional information
Supplementary Information accompanies the paper on Journal of the Operational Research Society website (http://www.palgrave-journals.com/jors/)
Electronic supplementary material
Rights and permissions
About this article
Cite this article
Kleijnen, J., van Beers, W. Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations. J Oper Res Soc 64, 708–717 (2013). https://doi.org/10.1057/jors.2011.148
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1057/jors.2011.148