Skip to main content
SpringerLink
Log in

A generic testing framework for agent-based simulation models

  • Article
  • Published:
Journal of Simulation

Abstract

Agent-based modelling and simulation (ABMS) had an increasing attention during the last decade. However, the weak validation and verification of agent-based simulation models makes ABMS hard to trust. There is no comprehensive tool set for verification and validation of agent-based simulation models, which demonstrates that inaccuracies exist and/or reveals the existing errors in the model. Moreover, on the practical side, many ABMS frameworks are in use. In this sense, we designed and developed a generic testing framework for agent-based simulation models to conduct validation and verification of models. This paper presents our testing framework in detail and demonstrates its effectiveness by showing its applicability on a realistic agent-based simulation case study.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Figure 1
Figure 2
Figure 3
Figure 4
Figure 5
Figure 6
Figure 7
Figure 8
Figure 9
Figure 10

Similar content being viewed by others

Notes

  1. The term level and the term scale are often used interchangeably. Here, the term level is chosen since it situates the described system considering its size, its characteristic evolution time or an inclusion relation with other systems or subsystems. However, the term scale refers to a dimension of analysis in which the system of interest can be measured.

  2. This algorithm arranges data points into clusters and it locates a centroid in each cluster This centroid is the point at which the distance from the rest of the points of the clusters is on average minimum.

  3. Law (2007) defines a simulation as a numerical technique that takes input data and creates output data based upon a model of a system.

  4. For example, Wolfram (1994) defines four classifications into which different Cellular Automata systems can be placed based on their long-term behaviours. The first one is evolving to a homogeneous state, which means that changes to the initial state have no impact on the final state.

  5. Burnstein (2003) refers data generators as load generators since load generators are aimed at being used in system-level tests. However, we use the term data generator since it can be used in all levels.

  6. The Unified Modeling Language (UML) is a standard that can be used ‘for analysis, design, and implementation of software-based systems as well as for modeling business and similar processes’. Thus, the UML is a visual language that can be used to create software architectures.

  7. We do not address implementation issues on how to apply these concepts in practice, as this is highly dependent upon the simulation framework used and the objective of the simulation study.

  8. Since many agent-based simulators use a global scheduler, such information can be retrieved from the scheduler of the simulation infrastructure.

  9. http://repast.sourceforge.net/, latest accessed 13 July 2012.

  10. http://cs.gmu.edu/eclab/projects/mason/, latest accessed 13 July 2012.

  11. JUnit, http://www.junit.org/

  12. A Dialog window (in Java) is an independent subwindow meant to carry temporary notice apart from the main Swing Application Window. Most Dialogs present an error message or warning to a user, but Dialogs can present images, directory trees, or just about anything compatible with the main Swing Application that manages them.

  13. Motor units are composed of one or more motoneurons and all of the muscle fibres they innervate.

  14. A reflex pathway is a type of neural pathway involved in the mediation of a reflex. Reflexes are involuntary reactions that occur in response to stimuli. They often bypass the brain altogether, allowing them to occur very quickly, although the brain receives information about the reflex as it happens.

  15. The soleus is a powerful muscle in the back part of the lower leg (the calf). It runs from just below the knee to the heel, and is involved in standing and walking.

  16. Ege University Center for Brain Research, http://www.eubam.ege.edu.tr/.

  17. The time for the end of the scenario execution may change at each execution, according to the values given by the developer for ScenarioExecuter. See Algorithm 8.

  18. In this article, testing methods and techniques with respect to the MAS properties they are able to address are extensively reviewed. But none of the reviewed studies focus on ABMS.

  19. The meso-level testing example given in this paper (Section 4.2) is a good example of white-box testing. In this test, the behaviour of a sub-society in order to handle a problem in certain conditions is tested by controlling its internal organization. The same sub-society, in another condition, may behave differently to handle the same problem. For example, rather than creating a new synapse, the sub-society may create a new neuron agent. This kind of solution may also satisfy this sub-society and make it produce the same output. However, the internal organization will be totally different. Therefore, only testing inputs and outputs in order to control uncertainty is not enough.

References

  • Balci O (1994). Validation, verification, and testing techniques throughout the life cycle of a simulation study. In: Proceedings of the 26th Conference on Winter simulation, WSC’94, Society for Computer Simulation International: San Diego, CA, pp 215–220.

  • Balci O (1995). Principles and techniques of simulation validation, verification, and testing. In: Proceedings of the 27th Conference on Winter simulation, WSC’ 95, IEEE Comp. Soc.: Arlington, VA, pp 147–154.

  • Bankes SC (2002). Agent-based modeling: A revolution? Proceedings of the National Academy of Sciences of the United States of America 99 (3): 7199–7200.

    Article  Google Scholar 

  • Beck K (2003). Test-driven Development: By Example. Addison-Wesley: Boston.

    Google Scholar 

  • Burnstein I (2003). Practical Software Testing. Springer: New York.

    Google Scholar 

  • Calvin W and Schwindt P (1972). Steps in production of motoneuron spikes during rhytmic firing. Journal of Neurophysiology 35 (3): 297–310.

    Google Scholar 

  • Capera D, Georgé J, Gleizes M and Glize P (2003). The AMAS theory for complex problem solving based on self-organizing cooperative agents. In: WETICE ’03: Proceedings of the Twelfth International Workshop on Enabling Technologies, IEEE Computer Society: Washington, DC, p 383.

  • Coelho R, Kulesza U, von Staa A and Lucena C (2006). Unit testing in multi-agent systems using mock agents and aspects. In: Proceedings of the 2006 Int. Workshop on Software eng. for large-scale multi-agent systems, SELMAS’06, ACM: New York, NY, pp 83–90.

  • Cooley P and Solano E (2011). Agent-based model (ABM) validation considerations. In: Proceedings of the Third International Conference on Advances in System Simulation (SIMUL 2011), IARIA, Barcelona, Spain, pp 134–139.

  • Dosi G, Fagiolo G and Roventini A (2006). An evolutionary model of endogenous business cycles. Computational Economics 27 (1): 3–34.

    Article  Google Scholar 

  • Epstein JM (2007). Agent-based computational models and generative social science. In: Generative Social Science Studies in Agent-based Computational Modeling, Introductory Chapters. Princeton University Press, Santa Fe, NM.

  • Feathers M (2004). Working Effectively with Legacy Code. Prentice Hall PTR: Upper Saddle River, NJ.

    Book  Google Scholar 

  • Ghosh S (1986). On the concept of dynamic multi-level simulation. In: Proceedings of the 19th annual symposium on Simulation, ANSS ‘86, IEEE Computer Society Press: Los Alamitos, CA, pp 201–205.

  • Grimm V et al (2005). Pattern-oriented modeling of agent-based complex systems: Lessons from ecology. Science 310 (5750): 987–991.

    Article  Google Scholar 

  • Gürcan Ö et al (2012). Simulating human single motor units using self-organizing agents. In: Sixth International IEEE Conference on Self-Adaptive and Self-Organizing Systems (SASO’2012), IEEE Computer Society: Lyon, France, pp 11–20.

  • Gürcan Ö, Dikenelli O and Türker KS (2010). Agent-based exploration of wiring of biological neural networks: Position paper. In: Trumph R (ed). 20th European Meeting on Cybernetics and Systems Research (EMCSR 2010. Austrian Society for Cybernetic Studies: Vienna, Austria, EU, pp 509–514.

    Google Scholar 

  • Helleboogh A, Vizzari G, Uhrmacher A and Michel F (2007). Modeling dynamic environments in multi-agent simulation. Autonomous Agents and Multi-agent Systems 14 (1): 87–116.

    Article  Google Scholar 

  • House R, Rousseau DM and Thomas-Hunt M (1995). The meso paradigm: A framework for the integration of micro and macro organizational behavior. Review of Organization Behavior 17: 71–114.

    Google Scholar 

  • Kaddoum E, Gleizes M-P, Georgé J-P and Picard G (2009). Characterizing and evaluating problem solving self-*systems (regular paper). (regular paper) In: The First Inter. Conference on Adaptive and Self-adaptive Systems and Applications (ADAPTIVE 2009), Athens, Greece, 15–20 November, CPS Production—IEEE Computer Society, page (electronic medium).

  • Klügl F (2008). A validation methodology for agent-based simulations. In: Proceedings of the 2008 ACM symposium on Applied computing, SAC¡¯08, ACM: New York, NY, pp 39–43.

  • Klügl F (2009). Multiagent simulation model design strategies. In: MAS&S @ MALLOW’09, Turin, Vol. 494, CEUR Workshop Proceedings, page (online).

  • Klügl F, Fehler M and Herrler R (2005). About the role of the environment in multi-agent simulations. In: Weyns D, Van Dyke Parunak H and Michel F (eds). Environments for Multi-agent Systems, Vol. 3374 of LNCS. Springer: Berlin/Heidelberg, pp 127–149.

    Chapter  Google Scholar 

  • Lamport L (1978). Time, clocks, and the ordering of events in a distributed system. Communications of the ACM 21 (7): 558–565.

    Article  Google Scholar 

  • Larman C (2004). Applying UML and Patterns: An Introduction to Object-oriented Analysis and Design and Iterative Development, 3rd edn. Prentice Hall PTR: Upper Saddle River, NJ.

    Google Scholar 

  • Law AM (2007). Simulation, Modeling and Analysis, 4th edn. McGraw Hill: New York.

    Google Scholar 

  • Luke S et al (2005). MASON: A multiagent simulation environment. Simulation 81 (7): 517–527.

    Article  Google Scholar 

  • MacQueen JB (1967). Some methods for classification and analysis of multivariate observations. In: Cam LML and Neyman J (eds). Proceedinges of the Fifth Berkeley Symposium on Mathematical Statistics and Probability, Vol 1. University of California Press, Berkeley, CA, pp 281–297.

    Google Scholar 

  • Malerba F, Nelson R, Orsenigo L and Winter S (1999). ‘History-friendly’ models of industry evolution: The computer industry. Industrial and Corporate Change 8 (1): 3–40.

    Article  Google Scholar 

  • Montanola-Sales C, Onggo BSS and Casanovas-Garcia J (2011). Agent-based simulation validation: A case study in demographic simulation. In: Proceedings of the Third International Conference on Advances in System Simulation (SIMUL 2011), Barcelona, Spain, pp 109–115.

  • Nguyen C et al (2011). Testing in multi-agent systems. In: Gleizes M-P and Gomez-Sanz J (eds). Agent-oriented Software Engineering X Vol. 6038 of Lecture Notes in Computer Science. Springer: Berlin/Heidelberg, pp 180–190.

    Chapter  Google Scholar 

  • Niazi MA, Hussain A and Kolberg M (2009). Verification and validation of agent-based simulation using the VOMAS approach. In: MAS&S @MALLOW’09, Turin, Vol. 494, CEUR Workshop Proceedings, page (on line).

  • Nikolai C and Madey G (2009). Tools of the trade: A survey of various agent based modeling platforms. Journal of Artificial Societies and Social Simulation 12 (2): 2.

    Google Scholar 

  • North M, Collier N and Vos J (2006). Experiences creating three implementations of the repast agent modeling toolkit. ACM Transactions on Modeling and Computer Simulation 16 (1): 1–25.

    Article  Google Scholar 

  • Pengfei X, Lees M Nan H and Viswanthatn TV (2011). Validation of agent-based simulation through human computation: An example of crowd simulation. In: Multi-agent-based Simulation XI pp 1–13.

  • Pidd M (2004). Computer Simulation in Management Science. John Wiles & Sons Inc, Indianapolis, IN.

    Google Scholar 

  • Polack FAC et al (2010). Reflections on the simulation of complex systems for science. In: Proceedings of International Conference on Computational Science (ICCS’2010), Oxford, UK, 22–26 March, pp 276–285 .

  • Raibulet C and Masciadri L (2009). Towards evaluation mechanisms for runtime adaptivity: From case studies to metrics. In: Proceedings of the 2009 Computation World: Future Computing, Service Computation, Cognitive, Adaptive, Content, Patterns, ComputationWorld’09, IEEE Comp. Soc: Washington, DC, pp 146–152.

  • Railsback SF and Grimm V (2011). Agent-based and Individual-based Modeling: A Practical Introduction. Princeton University Press, Princeton, NJ.

    Google Scholar 

  • Robinson S (1997). Simulation model verification and validation: Increasing the users’ confidence. In: Proceedings of the 29th Conference on Winter Simulation, WSC ‘97, IEEE Computer Society: Washington, DC, pp 53–59.

  • Sansores C and Pavon J (2005). Agent-based simulation replication: A model-driven architecture approach. In: 4th Mexican International Conference on Artificial Intelligence (MICAI’2005), Mexico, pp 244–253.

  • Sargent RG (2005). Verification and validation of simulation models. In: Proceedings of the 37th Conference on Winter simulation, WSC’05, Winter Simulation Conference, Orlando, FL, USA, 4–7 December, pp 130–143.

  • Schwindt P and Crill W (1982). Factors influencing motoneuron rhytmic firing: Results from a voltage-clamp study. Journal of Neurophysiology 48 (4): 875–890.

    Google Scholar 

  • Serrano E and Botia JA (2009). Programming multiagent systems. Chapter Infrastructure for Forensic Analysis of Multi-agent Systems, pp 168–183. Springer-Verlag: Berlin, Heidelberg.

  • Serrano E, Gómez-Sanz JJ, Botía JA and Pavón J (2009). Intelligent data analysis applied to debug complex software systems. Neurocomputing 72 (13–15): 2785–2795.

    Article  Google Scholar 

  • Sklar E (2007). Netlogo, a multi-agent simulation environment. Artificial Life 13 (3): 303–311.

    Article  Google Scholar 

  • Taillandier P, Drogoul A, Vo D and Amouroux E (2012). Gama: A simulation platform that integrates geographical information data, agent-based modeling and multi-scale control. In: Desai N, Liu A and Winikoff M (eds). The 13th International Conference on Principles and Practices of Multi-agent Systems (PRIMA). Kolkata, India, 12–15 November, Lecture Notes in Computer Science, Vol.7057, Springer: Berling Heidelberg, pp 242–258.

  • Terano T (2007). Exploring the vast parameter space of multi-agent based simulation. In: Antunes L and Takadama K (eds). Multi-agent Based Simulation VII. Lecture Notes in Computer Science, Vol. 4442, Springer: Berlin Heidelberg, pp 1–14.

  • Troitzsch KG (1996). Multilevel simulation. In: Klaus G et al (eds). Social Science Microsimulation: Berlin: Springer-Verlag, pp 107–122.

  • Türker KS and Miles TS (1991). Threshold depolarization measurements in resting human motoneurons. Journal of Neuroscience Methods 39 (1): 103–107.

    Article  Google Scholar 

  • Türker KS and Powers RK (2005). Black box revisited: A technique for estimating postsynaptic potentials in neurons. Trends in Neurosciences 28 (7): 379–386.

    Article  Google Scholar 

  • Uhrmacher A and Swartout W (2003). Agent-oriented Simulation. Kluwer Academic Publishers: Norwell, MA, pp 215–239.

    Google Scholar 

  • Utting M and Legeard B (2007). Practical Model-based Testing: A Tools Approach. Morgan Kaufmann Publishers Inc.: San Francisco, CA.

    Google Scholar 

  • Werker C and Brenner T (2004). Empirical calibration of simulation models. Papers on Economics and Evolution 2004-2010, Max Planck Institute of Economics, Evolutionary Economics Group.

  • Wilensky U and Rand W (2007). Making models match: Replicating an agent-based model. Journal of Artificial Societies and Social Simulation 10 (4): 2.

    Google Scholar 

  • Windrum P, Fagiolo G and Moneta A (2007). Empirical validation of agent-based models: Alternatives and prospects. Journal of Artificial Societies and Social Simulation 10 (2): 8.

    Google Scholar 

  • Wolfram S (1994). Cellular Automata and Complexity: Collected Papers Advanced Book Program Addison-Wesley Pub. Co: Indianapolis, IN.

    Google Scholar 

Download references

Acknowledgements

The authors thank Kemal S. Türker and S. Utku Yavuz from Ege University Center for Brain Research for supplying scientific data about the activity of motoneurons. The work described here was partially supported by Ege University under the BAP 10-MUH-004 project. Önder Gürcan is supported by the Turkish Scientific and Technical Research Council (TUBITAK) through a domestic PhD scholarship program (BAYG-2211) and by the French Government through the co-tutelle scholarship program.

Author information

Authors and Affiliations

  1. Ege University, Izmir, Turkey

    Ö Gürcan & O Dikenelli

  2. Toulouse III University, Toulouse, France

    Ö Gürcan & C Bernon

Authors
  1. Ö Gürcan
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. O Dikenelli
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. C Bernon
    View author publications

    You can also search for this author in PubMed Google Scholar

Rights and permissions

Reprints and permissions

About this article

Cite this article

Gürcan, Ö., Dikenelli, O. & Bernon, C. A generic testing framework for agent-based simulation models. J Simulation 7, 183–201 (2013). https://doi.org/10.1057/jos.2012.26

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1057/jos.2012.26

Keywords

Search

Navigation