Skip to main content
SpringerLink
Log in

Automating warm-up length estimation

  • Theoretical Paper
  • Published:
Journal of the Operational Research Society

Abstract

There are two key issues in assuring the accuracy of estimates of performance obtained from a simulation model. The first is the removal of any initialisation bias, the second is ensuring that enough output data is produced to obtain an accurate estimate of performance. This paper is concerned with the first issue, and more specifically warm-up estimation. Our aim is to produce an automated procedure, for inclusion into commercial simulation software, for estimating the length of warm-up and hence removing initialisation bias from simulation output data. This paper describes the extensive literature search that was carried out in order to find and assess the various existing warm-up methods, the process of short-listing and testing of candidate methods. In particular it details the extensive testing of the warm-up MSER-5 method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10
Figure 11
Figure 12
Figure 13

Similar content being viewed by others

References

  • Alexopoulos C and Seila AF ( 1998 ). Output data analysis Handbook of simulation . Wiley: New York pp 225–272 .

    Google Scholar 

  • Banks J, Carson JS, Nelson BL and Nicol DM ( 2001 ). Discrete-event system simulation, 4th edn . Prentice-Hall: New Jersey .

    Google Scholar 

  • Bause F and Eickhoff M ( 2003 ). Truncation point estimation using multiple replications in parallel . In: Chick S, Sánchez PJ, Ferrin D and Morrice DJ (eds) . Proceedings of the 2003 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 414 – 421 .

    Chapter  Google Scholar 

  • Beck AD ( 2004 ). Consistency of warm up periods for a simulation model that is cyclic in nature . In: Brailsford SC, Oakshott L, Robinson S and Taylor SJE (eds) . Proceedings of the 2004 Operational Research Society Simulation Workshop (SW04) . OR Society: Birmingham, UK, pp . 105 – 108 .

    Google Scholar 

  • Box GE, Jenkins GM and Reinsel GC ( 1994 ). Time series analysis: Forecasting and control, 3rd edn . Prentice-Hall: New Jersey .

    Google Scholar 

  • Bratley P, Fox B and Schrage L ( 1987 ). A guide to simulation, 2nd edn . Springer-Verlag: New York .

    Book  Google Scholar 

  • Cash CR, Dippold DG, Long JM and Pollard WP ( 1992 ). Evaluation of tests for initial-condition bias . In: Cash CR, Nelson BL, Dippold DG, Long MJ and Pollard WP (eds) . Proceedings of the 1992 Winter Simulation Conference . ACM: New York, NY, USA, pp . 577 – 585 .

    Google Scholar 

  • Conway RW ( 1963 ). Some tactical problems in digital simulation . Mngt Sci 10 ( 1 ): 47 – 61 .

    Article  Google Scholar 

  • Delaney PJ ( 1995 ). Control of initialisation bias in queuing simulations using queuing approximations . MS thesis, Department of Systems Engineering, University of Virginia. .

  • Fishman GS ( 1971 ). Estimating sample size in computing simulation experiments . Mngt Sci 18 : 21 – 38 .

    Article  Google Scholar 

  • Fishman GS ( 1973 ). Concepts and methods in discrete event digital simulation . Wiley: New York .

    Google Scholar 

  • Fishman GS ( 2001 ). Discrete-event simulation, modeling, programming, and analysis . Springer-Verlag: New York .

    Book  Google Scholar 

  • Gafarian AV, Ancker CJ Jr and Morisaku T ( 1978 ). Evaluation of commonly used rules for detecting ‘steady-state' in computer simulation . Nav Res Logist Q 25 : 511 – 529 .

    Article  Google Scholar 

  • Gallagher MA, Bauer KW and Maybeck PS ( 1996 ). Initial data truncation for univariate output of discrete-event simulations using the Kalman Filter . Mngt Sci 42 ( 4 ): 559 – 575 .

    Article  Google Scholar 

  • Glynn PW and Iglehart DL ( 1987 ). A new initial bias deletion rule . In: Thesen A, Grant H and Kelton DW (eds) . Proceedings of the 1987 Winter Simulation Conference . ACM: New York, NY, USA, pp . 318 – 319 .

    Google Scholar 

  • Goldsman D, Schruben LW and Swain JJ ( 1994 ). Tests for transient means in simulated time series . Nav Res Log 41 : 171 – 187 .

    Article  Google Scholar 

  • Gordon G ( 1969 ). System simulation . Prentice-Hall: New Jersey .

    Google Scholar 

  • Jackway PT and deSilva BM ( 1992 ). A methodology for initialisation bias reduction in computer simulation output . Asia Pac J Opl Res 9 : 87 – 100 .

    Google Scholar 

  • Kelton WD and Law AM ( 1983 ). A new approach for dealing with the startup problem in discrete event simulation . Nav Res Logist Q 30 : 641 – 658 .

    Article  Google Scholar 

  • Kimbler DL and Knight BD ( 1987 ). A survey of current methods for the elimination of initialisation bias in digital simulation . Annu Simul Symp 20 : 133 – 142 .

    Google Scholar 

  • Lada EK and Wilson JR ( 2006 ). A wavelet-based spectral procedure for steady-state simulation analysis . Eur J Opl Res 174 : 1769 – 1801 .

    Article  Google Scholar 

  • Lada EK, Wilson JR and Steiger NM ( 2003 ). A wavelet-based spectral method for steady-state simulation analysis . In: Chick S, Sánchez PJ, Ferrin D and Morrice DJ (eds) . Proceedings of the 2003 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 422 – 430 .

    Chapter  Google Scholar 

  • Lada EK, Wilson JR, Steiger NM and Joines JA ( 2004 ). Performance evaluation of a wavelet-based spectral method for steady-state simulation analysis . In: Lada EK, Wilson JR, Steiger NM and Joines JA (eds) . Proceedings of the 2004 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 694 – 702 .

    Google Scholar 

  • Law AM ( 1983 ). Statistical analysis of simulation output data . Opns Res 31 : 983 – 1029 .

    Article  Google Scholar 

  • Law AM ( 2007 ). Simulation modelling and analysis . McGraw-Hill: New York .

    Google Scholar 

  • L'Ecuyer P ( 1999 ). Good parameters and implementations for combined multiple recursive random number generators . Opns Res 47 : 159 – 164 .

    Article  Google Scholar 

  • Lee Y-H and Oh H-S ( 1994 ). Detecting truncation point in steady-state simulation using chaos theory . In: Tew JD, Manivannan S, Sadowski DA and Seila AF (eds) . Proceedings of the 1994 Winter Simulation Conference . Society for Computer Simulation International: San Diego, CA, USA, pp . 353 – 360 .

    Google Scholar 

  • Lee Y-H, Kyung K-H and Jung C-S ( 1997 ). On-line determination of steady-state in simulation outputs . Comput Ind Eng 33 ( 3 ): 805 – 808 .

    Article  Google Scholar 

  • Linton JR and Harmonosky CM ( 2002 ). A comparison of selective initialization bias elimination methods . In: Yücesan E, Chen C-H, Snowdon JL and Charnes JM (eds) . Proceedings of the 34th Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 1951 – 1957 .

    Chapter  Google Scholar 

  • Ma X and Kochhar AK ( 1993 ). A comparison study of two tests for detecting initialization bias in simulation output . Simulation 61 ( 2 ): 94 – 101 .

    Article  Google Scholar 

  • Mahajan PS and Ingalls RG ( 2004 ). Evaluation of methods used to detect warm-up period in steady-state simulation . In: Lada EK, Wilson JR, Steiger NM and Joines JA (eds) . Proceedings of the 2004 Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 663 – 671 .

    Google Scholar 

  • Nelson BL ( 1992 ). Statistical analysis of simulation results, Handbook of industrial engineering, 2nd edn . John Wiley: New York .

    Google Scholar 

  • Ockerman DH and Goldsman D ( 1999 ). Student t-tests and compound tests to detect transients in simulated time series . Eur J Opl Res 116 : 681 – 691 .

    Article  Google Scholar 

  • Pawlikowski K ( 1990 ). Steady-state simulation of queueing processes: A survey of problems and solutions . Comput Surv 122 ( 2 ): 123 – 170 .

    Article  Google Scholar 

  • Robinson S ( 2004 ). Simulation. The practice of model development and use . John Wiley & Sons Ltd: England .

    Google Scholar 

  • Robinson S ( 2005 ). A statistical process control approach to selecting a warm-up period for a discrete-event simulation . Eur J Opl Res 176 : 332 – 346 .

    Article  Google Scholar 

  • Rossetti MD, Li Z and Qu P ( 2005 ). Exploring exponentially weighted moving average control charts to determine the warm-up period . In: Kuhl ME, Steiger NM, Armstrong FB and Joines JA (eds) . Proceedings of the Winter Simulation Conference . IEEE: Piscataway, NJ, pp . 771 – 780 .

    Google Scholar 

  • Roth E ( 1994 ). The relaxation time heuristic for the initial transient problem in M/M/k queueing systems . Eur J Opl Res 72 : 376 – 386 .

    Article  Google Scholar 

  • Roth E and Josephy N ( 1993 ). A relaxation time heuristic for exponential-Erlang queueing systems . Comput Opns Res 20 ( 3 ): 293 – 301 .

    Article  Google Scholar 

  • Sandikci B and Sabuncuoglu I ( 2006 ). Analysis of the behaviour of the transient period in non-terminating simulations . Eur J Opl Res 173 : 252 – 267 .

    Article  Google Scholar 

  • Schruben LW ( 1982 ). Detecting initialization bias in simulation output . Opns Res 30 ( 3 ): 569 – 590 .

    Article  Google Scholar 

  • Schruben L, Singh H and Tierney L ( 1983 ). Optimal tests for initialization bias in simulation output . Opns Res 31 ( 6 ): 1167 – 1178 .

    Article  Google Scholar 

  • Sheth-Voss PA, Willemain TR and Haddock J ( 2005 ). Estimating the steady-state mean from short transient simulations . Eur J Opl Res 162 ( 2 ): 403 – 417 .

    Article  Google Scholar 

  • Spratt SC ( 1998 ). An evaluation of contemporary heuristics for the startup problem . MS thesis, Faculty of the School of Engineering and Applied Science, University of Virginia. .

  • Vassilacopoulos G ( 1989 ). Testing for initialization bias in simulation output . Simulation 52 ( 4 ): 151 – 153 .

    Article  Google Scholar 

  • White KP Jr ( 1997 ). An effective truncation heuristic for bias reduction in simulation output . Simulation 69 ( 6 ): 323 – 334 .

    Article  Google Scholar 

  • White KP Jr, Cobb MJ and Spratt SC ( 2000 ). A comparison of five steady-state truncation heuristics for simulation . In: Joines JA, Barton RR, Kang K and Fishwick PA (eds) . Proceedings of the 2000 Winter Simulation Conference . Society for Computer Simulation International: San Diego, CA, USA, pp . 755 – 760 .

    Chapter  Google Scholar 

  • Wilson JR and Pritsker AAB ( 1978a ). A survey of research on the simulation startup problem . Simulation 31 ( 2 ): 55 – 58 .

    Article  Google Scholar 

  • Wilson JR and Pritsker AAB ( 1978b ). Evaluation of startup policies in simulation experiments . Simulation 31 ( 3 ): 79 – 89 .

    Article  Google Scholar 

  • Yücesan E ( 1993 ). Randomization tests for initialization bias in simulation output . Nav Res Log 40 : 643 – 663 .

    Article  Google Scholar 

  • Zobel CW and White KP Jr ( 1999 ). Determining a warm-up period for a telephone network routing simulation . In: Farrington P et al (eds.) Proceedings of the 1999 Winter Simulation Conference, pp . 662 – 665 .

Download references

Acknowledgements

This work is part of the Automating Simulation Output Analysis (AutoSimOA) project (www.wbs.ac.uk/go/autosimoa) that is funded by the UK Engineering and Physical Sciences Research Council (EP/D033640/1). The work is being carried out in collaboration with SIMUL8 Corporation, who is also providing sponsorship for the project.

Author information

Authors and Affiliations

  1. Warwick Business School, The University of Warwick, Coventry, UK

    K Hoad, S Robinson & R Davies

Authors
  1. K Hoad
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. S Robinson
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. R Davies
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to S Robinson.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Hoad, K., Robinson, S. & Davies, R. Automating warm-up length estimation. J Oper Res Soc 61, 1389–1403 (2010). https://doi.org/10.1057/jors.2009.87

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/jors.2009.87

Keywords

Search

Navigation