Skip to main content
Log in

Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations

  • General Paper
  • Published:
Journal of the Operational Research Society

Abstract

Kriging metamodels (also called Gaussian process or spatial correlation models) approximate the Input/Output functions implied by the underlying simulation models. Such metamodels serve sensitivity analysis, especially for computationally expensive simulations. In practice, simulation analysts often know that this Input/Output function is monotonic. To obtain a Kriging metamodel that preserves this characteristic, this article uses distribution-free bootstrapping assuming each input combination is simulated several times to obtain more reliable averaged outputs. Nevertheless, these averages still show sampling variation, so the Kriging metamodel does not need to be an exact interpolator; bootstrapping gives a noninterpolating Kriging metamodel. Bootstrapping may use standard Kriging software. The method is illustrated through the popular M/M/1 model with either the mean or the 90% quantile as output; these outputs are monotonic functions of the traffic rate. The empirical results demonstrate that monotonicity-preserving bootstrapped Kriging gives higher probability of covering the true outputs, without lengthening the confidence interval.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Figure 1
Figure 2
Figure 3
Figure 4

Similar content being viewed by others

References

  • Ankenman B, Nelson B and Staum J (2010). Stochastic Kriging for simulation metamodeling. Operations Research 58 (2): 371–382.

    Article  Google Scholar 

  • Antognini B and Zagoraiou M (2010). Exact optimal designs for computer experiments via Kriging metamodelling. Journal of Statistical Planning and Inference 140 (9): 2607–2617.

    Article  Google Scholar 

  • Batur D and Choobineh F (2010). A quantile-based approach to system selection. European Journal of Operational Research 202 (3): 764–772.

    Article  Google Scholar 

  • Chen EJ (2008). Some procedures of selecting the best designs with respect to quantile. Simulation 84: 275–284.

    Article  Google Scholar 

  • Chen X, Ankenman B and Nelson BL (2010). The effects of common random numbers on stochastic Kriging metamodels. Working Paper, Department of Industrial Engineering and Management Sciences, Northwestern University, Evanston, IL USA.

  • Den Hertog D, Kleijnen JPC and Siem AYD (2006). The correct Kriging variance estimated by bootstrapping. Journal of the Operational Research Society 57 (4): 400–409.

    Article  Google Scholar 

  • Dette H, Neumeyer N and Pilz KF (2006). A simple nonparametric estimator of a strictly monotone regression function. Bernoulli 12 (3): 469–490.

    Article  Google Scholar 

  • Efron B and Tibshirani RJ (1993). An Introduction to the Bootstrap. Chapman & Hall: New York.

    Book  Google Scholar 

  • Feelders AJ (2000). Prior knowledge in economic applications of data mining. Proceedings of the Fourth European Conference on Principles and Practice of Knowledge Discovery in Databases, Springer: New York, pp 395–400.

  • Forrester A, Sobester A and Keane A (2008). Engineering Design via Surrogate Modelling: A Practical Guide. Wiley: Chichester, UK.

    Book  Google Scholar 

  • Frazier P, Powell W and Dayanik S (2009). The knowledge-gradient policy for correlated normal beliefs. INFORMS Journal on Computing 21: 599–613.

    Article  Google Scholar 

  • Gross D and Harris CM (1998). Fundamentals of Queueing Theory. Wiley: New York.

    Google Scholar 

  • Hong LJ (2009). Estimating quantile sensitivities. Operations Research 57 (1): 118–130.

    Article  Google Scholar 

  • Jin X, Fu MC and Xiong X (2003). Probabilistic error bounds for simulation quantile estimators. Management Science 14 (2): 230–246.

    Article  Google Scholar 

  • Kleijnen JPC (2008). Design and Analysis of Simulation Experiments. Springer Science+Business Media, New York.

    Google Scholar 

  • Kleijnen JPC (2009). Factor screening in simulation experiments: Review of sequential bifurcation. In: Alexopoulos C, Goldsman D and Wilson JR (eds). Advancing the Frontiers of Simulation: A Festschrift in Honor of George S. Fishman. Springer: New York, pp 169–173.

    Google Scholar 

  • Kleijnen JPC, van Beers W and van Nieuwenhuyse I (2010). Constrained optimization in expensive simulation: Novel approach. European Journal of Operational Research 202: 164–174.

    Article  Google Scholar 

  • Law AM (2007). Simulation Modeling and Analysis. Fourth edn, McGraw-Hill: Boston.

    Google Scholar 

  • Lehmann EL (1999). Elements of Large-Sample Theory. Springer: New York.

    Book  Google Scholar 

  • Lim E and Glynn PW (2006). Simulation-based response surface computation in the presence of monotonicity. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM and Fujimoto RM (eds). Proceedings of the 2006 Winter Simulation Conference, The Society for Computer Simulation International (SCS): San Diego, CA, pp 264–271.

  • Loeppky JL, Sacks J and Welch W (2009). Choosing the sample size of a computer experiment: A practical guide. Technometrics 51: 366–376.

    Article  Google Scholar 

  • Lophaven SN, Nielsen HB and Sondergaard J (2002). DACE: a Matlab Kriging Toolbox, version 2.0. IMM Technical University of Denmark: Lyngby.

    Google Scholar 

  • Sacks J, Welch WJ, Mitchell TJ and Wynn HP (1989). Design and analysis of computer experiments. Statistical Science 4 (4): 409–435.

    Article  Google Scholar 

  • Santner TJ, Williams BJ and Notz WI (2003). The Design and Analysis of Computer Experiments. Springer-Verlag: New York.

    Book  Google Scholar 

  • Siem AYD (2007). Property preservation and quality measures in meta-models. PhD dissertation, Tilburg University, Tilburg, the Netherlands.

  • Simpson TW, Booker AJ, Ghosh D, Giunta AA, Koch PN and Yang R-J (2004). Approximation methods in multidisciplinary analysis and optimization: A panel discussion. Structural and Multidisciplinary Optimization 27 (5): 302–313.

    Article  Google Scholar 

  • Van Beers W and Kleijnen JPC (2003). Kriging for interpolation in random simulation. Journal of the Operational Research Society 54 (3): 255–262.

    Article  Google Scholar 

  • Velikova M (2006). Monotone models for prediction in data mining. PhD dissertation, Tilburg University, Tilburg, the Netherlands.

  • Yin J, Ng SH and Ng KM (2009). A study on the effects of parameter estimation on Kriging model's prediction error in stochastic simulations. In: Rossini MD, Hill RR, Johansson B, Dunkin A and Ingalls RG (eds). Proceedings of the 2009 Winter Simulation Conference, The Society for Computer Simulation International (SCS): San Diego, CA, pp 674–685.

Download references

Acknowledgements

We thank two (anonymous) referees and Dick den Hertog (Tilburg University) for their comments on earlier versions, which lead to drastic revisions.

Author information

Authors and Affiliations

Authors

Corresponding author

Correspondence to J P C Kleijnen.

Additional information

Supplementary Information accompanies the paper on Journal of the Operational Research Society website (http://www.palgrave-journals.com/jors/)

Electronic supplementary material

Rights and permissions

Reprints and permissions

About this article

Cite this article

Kleijnen, J., van Beers, W. Monotonicity-preserving bootstrapped Kriging metamodels for expensive simulations. J Oper Res Soc 64, 708–717 (2013). https://doi.org/10.1057/jors.2011.148

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/jors.2011.148

Keywords

Navigation