1. Introduction

Distributed simulation is an application of distributed systems technology that enables models to be coupled over computer networks so that they interoperate during a simulation run. Initial research on distributed simulation has been conducted in the defence (Singhal and Zyda, 1999). It is a promising approach allowing for the interoperability between models and the reusability of them (Fujimoto, 2000). Besides interoperability and reusability Fujimoto mentions other benefits of distributed simulation, such as reduced execution time, geographical distribution, integrating simulation models from different vendors and fault tolerance. Other benefits are apparent as well, for instance the possibility to reuse existing components, support for information hiding and support for integrating heterogeneous models (Boer, 2005).

In order to carry out distributed simulation, simulation standards were initiated in the defence community, culminating in the high-level architecture (HLA), which is the most recent and most advanced approach for integrating simulation models. It is the official standard of all simulations of the Department of Defence (Kuhl et al, 1999) and it has been accepted as an IEEE standard (IEEE 1516, 2000).

Although the initial step in designing and developing the HLA standard was carried out by the defence community, this large effort was intended to support the industrial community as well (Dahmann et al, 1997). However, in the industrial domain application of distributed simulation is still in its infancy. The research community is aware of this phenomenon (Robinson, 2005). In order to find out the reasons behind it, during the last few years separate panel discussions were organized at the Winter Simulation Conference (Taylor et al, 2002, 2003). Some researchers have proposed methods for migrating the HLA concept into the industrial domain (McLean and Riddick, 2000; Rabe et al, 2001; Straburger, 2001, Revetria et al, 2003), which led to new insights regarding the applicability of HLA in industry. Furthermore, a forum, called the HLA commercial-off-the-shelf simulation package interoperability forum (HLA-CSPIF), was initiated that aims to create reference models for integration of distributed simulation models created in commercial-off-the-shelf (COTS) simulation packages (Taylor et al, 2006). This forum has recently become a COTS simulation package interoperability product development group (CSPI PDG) under the Simulation Interoperability Standards Organization (Taylor et al, 2007).

Most computer simulation models in industry are created using simulation packages (Nikoukaran et al, 1999). COTS simulation packages are the most advanced and widely used packages that are commercially available (Nikoukaran et al, 1999; Tewoldeberhan et al, 2002). Law and Kelton identify several advantages of COTS simulation packages over general purpose programming languages, explaining why simulation modelling has become more and more popular in recent years (Law and Kelton, 2000).

One reason why distributed simulation or HLA is not applied in industrial simulation projects might be that COTS simulation packages hardly support HLA or even distributed simulation. Consequently, simulation practitioners that apply COTS simulation packages for their daily use would not have a chance to apply distributed simulation. On the other hand, the simulation practitioners do not seem to request facilities for applying distributed simulation like transparent HLA interfaces. This might be caused by the fact that simulation practitioners cannot see the benefits because they do not have proper tools. So it seems we are stuck in a deadlock, caused by a chicken–egg scenario.

We have set up a research effort to investigate this scenario more deeply aiming at a solution to overcome the deadlock. Basically there are three research questions to be answered. First of all, the hypothesis that HLA is hardly applied in industry has to be validated. Second, we have to find reasons behind this phenomenon. If these obstacles can be overcome then, thirdly, we have to find ways to remedy the situation.

• RQ1: Is it true that the HLA standard is hardly applied in industry?
• RQ2: If so, why is HLA hardly applied in industry?
• RQ3: What is needed to overcome this situation?

In order to obtain answers to the above questions, in 2004 we conducted an extensive survey with experts in the field. The survey was divided into three segments. First of all we interrogated the COTS vendors, using a questionnaire, mainly aiming at an answer to research question RQ1 (although the other ones have been touched briefly). Secondly, we addressed a broader group of experts questioning them in much more detail on the other research questions as well. Finally, we analysed the answers and formulated at a list of requirements for adapting or redesigning existing distributed simulation architectures. This requirement list answers question RQ3. The complete research is elaborated in Boer (2005).

The next section presents HLA in a nutshell. Section 3 deals with the methodology we applied. Section 4 discusses the questionnaire and Section 5 reports on the interview. In order to apply these results and to work towards a solution of the problem that distributed simulation is hardly applied in industry, we need to perform an analysis on this data. This is described in Section 6. Finally, Section 7 concludes the paper. In this field, acronyms and abbreviations are heavily used. We include an appendix that which provides their expansion.

2. HLA in a nutshell

In order to carry out distributed simulation, different simulation standards were initiated in the military domain, such as SIMulator NETworking (SIMNET) (Pope, 1989), distributed interactive simulation (DIS) (DIS Steering Committee, 1994) and aggregate level simulation protocol (ALSP) (Weatherly et al, 1991). Most of these standards provided specific services for interoperability in a small niche of the simulation market, for example DIS for human-in-the-loop simulators or ALSP for war games (Straburger, 2001). The United States Department of Defence (DoD) believed that it would be more efficient to develop a more general solution to distributed simulation. Finally, in 1996 the defence modelling and simulation office (DMSO) presented the HLA, which is the most recent and most advanced approach for integrating simulation models (Kuhl et al, 1999). The roots of the HLA can be traced back to DIS and ALSP (Fujimoto, 2000). In September 1996, HLA was designated as the standard technical architecture for all DoD simulations that entailed that all simulations developed either by or for the US DoD had to be HLA compliant ('the DoD mandate').

One of the major notions in the HLA standard are the simulation models or, as HLA calls them, the federates. The HLA standard defines the notion of federate and federation, corresponding more or less to confederates and confederations from ALSP. A federate can be a computer simulation, a manned simulator, a supporting utility, such as a viewer or data collector or even an interface to a live player or instrumented facility. A federation is the alliance of one or more federates acting together in a distributed simulation to achieve a certain objective.

The Run Time Infrastructure (RTI) functional component behaves like a distributed operating system for a federation. It provides a set of general-purpose services that support federate to federate interactions and federation management. The interactions that take place between two federates are performed through the RTI, which offers six services: federation management, object management, time management, declaration management, ownership management and data distribution management (IEEE 1516, 2000).

Another main component within the HLA standard is the interface to the RTI. The HLA run time interface specification provides a standard way for federates to interact with the RTI, to invoke the above-mentioned RTI services, to support run time interactions among federates and to respond to requests from the RTI.

The HLA standard is formally defined by three components: object model template (OMT), interface specification and HLA Rules (IEEE 1516, 2000). An HLA object model provides a formal definition of the data that are transferred between federates. Although HLA applies an object-oriented worldview for representing the data that are transferred between federates, this view differs from the well-known object-oriented view defined in the software engineering domain in the sense that HLA object models do not specify methods for the objects. This object-oriented worldview does only define how federates have to represent themselves to other federates (Kuhl et al, 1999; (Straburger, 2001). During a simulation run the distributed models interoperate by means of exchanging simulated entities or interactions. Therefore, each distributed model should have a representation of the objects (entities and interactions) it provides and expects, respectively. For this reason, each individual federate provides the so-called simulation object model (SOM), which has to be specified in terms of the OMT (IEEE 1516, 2000). The SOM describes the simulation (federate) in terms of the types of objects (attributes) and interactions (parameters) that it can provide or will expect from other federates. Furthermore, each federation defines a so-called federation object model (FOM). The FOM combines the information from the individual SOMs of the federates, which contains all classes defined by the individual participants of the federation and gives a description of all shared information (IEEE 1516, 2000; Straburger, 2001).

The HLA interface specification describes the services provided to the federates by the RTI, and by the federates to the RTI (IEEE 1516, 2000). The RTI behaves like an operating system in which the different federates are embedded. It is a piece of software that provides all services that are needed for distributed execution. DMSO has produced an RTI that is known as DMSO RTI 1.3.

The third component of the HLA standard consists of the HLA rules. The HLA rules summarize the key principles behind the HLA. The rules are divided into two groups: there are five rules for federations and five rules for federates. Rules related to federations state, for example, that federations shall have an HLA FOM, specified in terms of HLA OMT, or, that during a federation execution, FOM data among federates shall be exchanged via the RTI. Similarly, a rule for federates specifies, for instance, that federates shall have an HLA SOM, documented in accordance with HLA OMT. The complete list of rules can be found in IEEE 1516 (2000).

HLA has gathered ground in defence and finally became the standard technical architecture for all Department of Defence simulations. This led to the fact that various departments started to develop their own version of the RTI that aimed to be more efficient than the DMSO-RTI. Lately, other more efficient commercially available RTIs appeared, such as the Pitch RTI (http://www.pitch.se/) and the MÄK RTI (http://www.mak.com/).

3. Applied methodology

A schematic overview of the set up of our survey is given in Figure 1. In the survey we followed the Delphi approach, the purpose of which is to elicit information and judgments from participants to facilitate problem solving, planning and decision making (Linstone and Turoff, 1975; Linstone, 1978). Delphi is a method that proves to be particularly useful when the individuals who need to interact cannot be brought together in a face-to-face exchange because of time or cost constraints (Kenis, 1995). The Delphi method is iterative. The results of an initial survey are summarized and then formed the basis of a second follow on survey. The results from the second survey are the basis of a third survey and so on (Linstone and Turoff, 1975). Accordingly, after getting the preliminary results, we maintain a continuous relation with the involved experts in follow-up sections in order to gather sufficient feedback for the validation of the end result.

Figure 1.

Survey on distributed simulation in industry (the abbreviations are expanded in the Appendix).

Full figure and legend (327K)

In order to obtain an answer to our first research question RQ1 'Is it true that the HLA standard is hardly applied in industry?', we tried to identify a community that has continuous and intensive contact with the industrial simulation area. The most appropriate group for this purpose would be the totality of software companies that carry out industrial simulation projects. However, it is impossible to identify all companies involved in these types of projects. If we interrogate only some companies, the results cannot be used to give a proper general answer to our questions because we do not know how many other projects are conducted out there. As a consequence, we needed to address another group. The community we have recognized as appropriate consists of the COTS simulation package vendors, who provide tools for their customers in industry. COTS vendors adapt their packages to the market needs they perceive. They have a continuous and intensive connection with industrial simulation practitioners who request new features in the package based on the projects they intend to carry out. In this way the COTS vendors have an opportunity to gain information about these projects, and thus about applying distributed simulation in industry. Therefore, our primary aim is to mobilize this group for providing an answer to our questions. Furthermore, COTS vendors can provide information regarding HLA support within their tool.

After selecting the participants the researcher must identify the collection procedures that he intends to apply. The literature mentions four basic ways of data collection: observation, questionnaire or interviews, documents and media (audio and visual) material (Creswell, 2003; Seale et al, 2004). Owing to the fact that there is a large number of COTS simulation package vendors active in the market, the type of the survey that we have chosen for our initial investigation is a structured questionnaire. This part of our survey corresponds to the leftmost column of Figure 1.

On the basis of the questionnaire results we performed an interview survey, depicted as the remainder of Figure 1. Besides some COTS vendors, we selected people whom we consider to be experts in distributed simulation and who have already had experience with distributed simulation either in industry or defence.

We thus applied two types of data collection: firstly, a structured questionnaire for COTS simulation package vendors, and secondly, an interview with open-ended questions based on our observations and the results of the questionnaire. The interview aims to go one step further and to focus more deeply on the research questions. As we have stated before, the interview is based to some extent on the questionnaire and accordingly, it can be considered as a second iteration in this research. For the interview we employed purposeful selection in order to determine the participants. The data in this case have been collected through a phone interview with open-ended questions.

In order to analyse the data collected from the interviews, we applied qualitative content analysis (Seale et al, 2004), following the generic steps presented in Creswell (2003, pp 191–195).

• Organize and prepare the data for analysis.
• Read through all data.
• Begin detailed analysis with a coding process.
• Use a coding process to generate a detailed description of topics and subtopics.
• The final step in data analysis involves making an interpretation or meaning of the data.

4. The questionnaire

In this section, we briefly describe the questionnaire for the COTS simulation package vendors. This is a condensed report, a fuller discussion can be found in Boer (2005) and Boer et al (2006a). The questionnaire consists of a quantitative part, implemented as six yes/no questions (cf. Table 2), which mainly deals with question RQ1 'Is it true that the HLA standard is hardly applied in industry?' In fact, we took a broader view by inquiring about all types of integration approaches, not only the ones through HLA. Besides the quantitative part, we added some open questions aiming at a provisional answer to questions RQ2 and RQ3.

4.1. The participants

In order to obtain an adequate assessment of the opinion of the COTS simulation package vendors, we tried to contact as many vendors as possible. To this end we have taken as a basis the survey conducted by James Swain (2007). The list contains 39 organizations, from which we have invited 35 organizations to participate. We have ignored four organizations, because the description of their activity does not fit in our target group. On the other hand, the list not being complete, we have added two additional COTS simulation package vendors, Wolverine Software and the Lanner Group.

We obtained filled-out questionnaires from 19 vendors (52%). They are presented in Table 1. In view of the evaluation of the survey it is relevant to observe that the main organizations are among these 19 ones.

Table 1 - Participating COTS simulation vendors.

Full table

4.2. Analysis of the collected data

In Table 2 we show the response on the quantitative questions from our questionnaire.

Table 2 - The quantitative results.

Full table

Before starting this survey our hypothesis was that distributed simulation (including the HLA standard) is rarely applied in industry. Looking superficially at the results in Table 2 it seems that our observation was contradicted. The answers given reflect that COTS simulation vendors recognize the success of their package in different industrial-oriented distributed simulation projects (questions 1–3), that there are quite some successful HLA-related projects (question 4) and there is a significant support for the HLA standard (question 5).

On the other hand, analysing the results in more detail there seems to be some discrepancy in the answers, especially when correlating the answers that vendors provided to questions 4 and 5. On the one hand, there are vendors who claim that they do not provide an HLA interface and they do not want to support it in a future version, whereas they or their customers have carried out HLA-related successful projects. On the other hand there are vendors who state that they provide an HLA interface and they support it in their package as an additional feature, although they never carried out successful HLA-related simulation projects. Furthermore, there are vendors who claim that they provide an HLA interface and they also applied it in successful HLA-related projects, but they do not want to support it as an additional feature in their package. Finally, there are vendors who affirmed that they do not provide an HLA interface and they never applied it in any successful project, although they support it as an additional feature in their package.

In order to get a clearer picture we sent out a second questionnaire, in line with the Delphi methodology. The results thereof indicate that the vendors interpret the notion 'supporting HLA' in different ways. Studying HLA, for example, or attending presentations regarding HLA, or providing low-level protocols, such as WinSock, is considered by them as an effort to support HLA as an additional feature, which to a certain extent might be true. However, this is far from what we would call de facto support and service for an efficient HLA interface for simulation practitioners.

So, although there seems to be active interest in HLA by about half of the COTS vendors, this interest is rather tentative and aiming at low-level solutions. This conclusion is strengthened by the answers that we obtained from the open questions. For instance, in relation to question 1–3 we asked the vendors to specify which protocols and middleware they have used to achieve interoperability. The most common answers were: COM (mentioned 13 times), WinSock (eight times) and HLA (seven times). Regarding question 6 we asked what other standards the vendors were considering. Most of them stated that they intended to stay with low-level technical protocols or middleware in their future versions like WinSock, COM, DCOM or .Net. Furthermore, we also asked for examples of projects (related to questions 1, 2 and 4) and we found only very few cases in which HLA was applied in industry, most of the HLA examples occurred in defence-oriented projects. Accommodating distributed simulation is mostly done by creating 'homespun' architectures based on low-level technical solutions (eg WinSock, CORBA, COM, etc) instead of HLA.

In conjunction with question 4 we asked the vendors who answered 'yes' about their experiences and the ones with a 'no' answer for their reasons. Here are some of the perceived drawbacks of HLA: data alignment (ie specification of the HLA FOM), translation of HLA concepts into COTS terms, exchange of heterogeneous data, verification and debugging of the distributed models, the heavy structure of the HLA RTI, performance, complex runtime management and a cost that is too high in relation to the benefits. One of the vendors explicitly stated 'I recall reviewing the HLA approach, and being impressed with its scope and ambition but sceptical about its practicability'.

5. The interview

The next step in our survey was to take the above-mentioned results as a starting point for a series of in-depth interviews involving unstructured and generally open-ended questions intending to elicit views and opinions of the participants. In the first part of this section we discuss the way we selected the participants and the groups in which they are classified as depicted in Figure 1. Then, in the second subsection we present the results of the interview.

5.1. The participants

As we mentioned before, through the interviews we aimed to collect data from experts. We tried to select purposefully those experts who, we believe, can support our research with their experience, view and opinions. We chose to select a heterogeneous group of experts, including people both from the industry and the defence simulation community, the latter especially because HLA was designed and developed by defence. The advantage over selecting a homogeneous group is that different groups can sense issues differently, they might consider the problem from a different angle (Schreiber et al, 2000). On the other hand, we intentionally tried to involve many practitioners who have applied HLA because we wanted to avoid bias against HLA.

We identified five groups, two from the industrial community and three from defence:

• COTS simulation package vendors (CV): First of all, we considered a subset of the COTS vendors; we chose them based on the result of the questionnaire. This group consisted of two persons.
• Industrial simulation practitioners (IP): The second group consisted of eight industrial-oriented simulation practitioners both from the scientific area and from practice. As a starting point for selecting these experts we considered the HLA-CSPIF group (http://www.cspif.com), which aims to stimulate the applicability of distributed simulation in the industrial domain. From this list five participants have been selected. We realize that we introduced some bias in using this list, because these people already have a need for or are involved in distributed simulation. On the other hand, we wanted to tap from the already existing knowledge on this topic as much as possible. In order to compensate for the bias, we selected three participants using pointers to interesting people as provided in the questionnaire by the COTS vendors.
• HLA designers and developers (HD): Experts who designed and developed HLA (two persons).
• HLA simulation practitioners in defence (HP): Simulation practitioners, who apply HLA in defence-oriented simulation projects (also two persons). For these two groups we took as a starting point DMSO https://www.dmso.mil/, MITRE (http://www.mitre.org/), TNO-FEL (http://www.tno.nl/instit/fel/) and SISO (http://www.sisostds.org/).
• Commercial HLA vendors (HV): Two HLA commercial vendors who were chosen from the DMSO vendor list published on the DMSO website.

We conducted a phone interview with all participants, except for one who validated the first version of the findings by means of a face-to-face interview. The average time for an interview was 58 min.

5.2. The results

We present the results in two ways. First we give a succinct list of all issues that were raised by the respondents. This is the first subsection. Then for each group of experts separately, we give a slightly more elaborate exposition, containing arguments and examples that have been omitted from the list.

Overview

For each remark in the list below, we indicate the group(s) of experts where these remarks originated from. We use the abbreviations from the list in Section 5.1. They constitute pointers to the section on COTS simulation package vendors to the section on Commercial HLA vendors that can be followed to obtain more elaborate information. The list is organized along the main questions that we prepared for our interview.

1. Differences between industrial and defence simulation.
   
   
   • Different business models (CV, IP and HD).
   • Small (IP) and cheap (CV, IP and HD) models in industry versus complex and expensive models in defence (HD and HP).
   • Throw away models in industry (CV) versus sophisticated, stable, robust (IP) and highly credible (HD) models in defence.
   • In defence, safety and secrecy of the models is an issue (IP).
   • Industry uses COTS packages (IP); defence uses programming languages (IP, HD and HP).
   • No maintenance of the models in industry (CV) versus requirement analysis (HD), verification and validation (IP) in defence.
     
   
   
2. Reusability of existing simulation models.
   
   
   • Hardly any reuse in industry (IP), one reason being that the models are too specific (CV).
   • Reuse in defence is feasible because the models have a high credibility and high building costs (HD).
   • There is some reuse of components in industry (CV).
   
   
3. Information hiding within the simulation model.
   
   
   • It is not an issue in (USA) defence, because the DoD owns everything (HD).
   • It is an issue once projects cross borders (HP).
   • It is an issue in simulation-based acquisition (HV).
   
   
4. Difficulties and benefits of applying the HLA standard.
   
   
   • Benefits:
     • Takes care of low-level complexity (IP and HV).
     • Generality (HV).
     • It is a standard (HP).
     
     
   • Difficulties both in defence and industry:
     • Semantic interoperability/object mapping (CV, IP, HP and HV).
     • Complexity (HP)/Steep learning curve (HV).
     • Needs high-level thinkers (HV).
     • Lack of transparency (HP and HV).
     • Bad performance of RTIs (IP, HP, but cf. HV).
     • Incompatibility of different RTIs (IP).
     
     
   • Difficulties (industry only):
     • Cost of the RTI (IP).
     • Too complex (CV and IP).
     • No experts in industry (CV and IP).
     • Too much functionality that is irrelevant for industry (IP and HD).
     
     
   • Desirable additions to HLA (HP and HV).
     
   
   
5. Integrating simulation models developed in COTS simulation packages using the HLA standard.
   
   
   • Obstacles:
     • Industry is less focused on distributed simulation (HD).
     • COTS vendors are reluctant to cooperate (HD and HV).
     • No proper tools (IP).
     • Ownership not implemented in COTS packages (IP).
       
       
   • Enablers:
     • Such projects already exist in industry (IP).
     • This will grow in importance, for instance due to globalization (IP).
       
   
   
6. Benefits and disadvantages of integrating simulation models designed and developed in COTS simulation packages (discussed with COTS vendors only).
   
   
   • In general no economic benefit observed; monolithic models are more adequate (CV).
   • Examples of application types where it could be useful (CV).
   
   
7. Support of commercial HLA tools for the industrial domain (discussed with HLA vendors only).
   
   
   • Vendors are open for collaboration with industry (HV).
   • Vendors target defence where HLA is the standard due to the DoD mandate (HV).
   
   
8. Visions on future 'idealized' distributed simulation architectures for industry.
   
   
   • A good architecture (painless, efficient with clear interfaces and communication) will generate a technology push (CV).
   • Simple and minimal functionality needed: entity passing, synchronization (CV, IP and HP), maybe federation join/resign (IP), publish/subscribe (IP).
   • Subsetting HLA (HD) versus a new industrial standard (IP).
   • Diverging opinions on interfacing to HLA using adaptors and middleware (CV and IP).
   • Protocol on the wire level instead of an interface to the RTI (HV).
   • 
     Internet ideas:
     • Repositories offering services (HD).
     • Grid computing (HV).
     • This entails a new business model (HD and HV).

Finally, we mention a remark that was made several times stating that issues related to HLA should be separated from issues of distributed simulation in general (IP and HP).

In the next subsections, we summarize the results of the interview organized for each selected group separately. For a more extensive exposition of this material we refer to Boer (2005).

COTS simulation package vendors

As it stands now, the COTS vendors do not see direct benefits in using distributed simulation to integrate models written using their packages. They see no economic benefit to cooperate with other vendors, and they argue that combining models written in one package can better be achieved in a monolithic way.

Applying distributed simulation is also hampered by the way models are built in industry. Industry 'wants results as fast and as cheaply as possible', most models are of a 'throw away' type, and they are hardly maintained during their life cycle. COTS vendors also mention the problems around mapping objects from different models and they state that most models are too specific to be reused. Applying HLA is perceived to only generate more problems, because there are no experienced users and HLA is too complex. According to these experts the adaptors available today do not offer a solution to this problem because still too much knowledge is called for to apply this technique. There exists some experience on distributed simulation in industry though, but this focuses on reuse at the component level instead of full model reuse.

On the other hand, they recognize that combining COTS models using distributed simulation might be fruitful in collaborative design and development, in emulation, in combining simulation with other applications, like ERP, with specialized simulation tools, or in trying to generate speedup through parallelization.

They believe that a good architecture might generate a technology push. The architecture should be simple, offering only minimal functionality, viz. data representation and exchange and time synchronization. Furthermore, integration should be relatively painless and efficient, leading to perspicuous models and clear communication between them.

Industrial simulation practitioners

A first relevant observation that was made by several industrial simulation practitioners is that there is a need to separate issues around HLA and COTS from problems that are actually rooted on a higher level, viz. distributed simulation in general.

The experts saw a clear difference in scope between the projects in defence and in industry. Industry is mainly interested in generating direct results; money is the big factor here. Projects are smaller than in defence and in general the models are not reused. COTS packages are designed for such applications. Defence has other objectives, safety and secrecy of the models are relevant, the models are sophisticated, stable and robust. Verification and validation is important, and mostly general-purpose programming languages are used. The industrial simulation practitioners mentioned several drawbacks of the HLA standard. One of these drawbacks is the performance of the current implementations. Experts relate this to the fact that HLA offers too much irrelevant functionality for industry. Minimal functionality for industry would be only time synchronization and entity passing, possibly augmented with the possibility to join and resign from a simulation run and to publish and subscribe to objects. Other problems that were mentioned by practitioners were incompatibility of RTIs and some ambiguities in the HLA specification. Furthermore, the fact that ownership is a notion that is not implemented in COTS packages causes additional problems when one intends to apply this HLA feature. In addition the mapping problem is mentioned by the industrial simulation practitioners as well. This might be solved by defining standard object descriptions. In this process, however, COTS vendors should be involved. Finally there is the issue of cost. There is a trend towards low-priced software packages. Adding the cost of integrating or interfacing to an RTI will increase cost and the customers, as it stands now, are not willing to pay more than 10% for this. On the other hand a remark was made that, given the prices of COTS packages and RTIs, this is attainable.

Several examples from industry are mentioned that have benefited or might benefit from distributed simulation, such as the virtual factory, material flow between two factories and combining models of production systems with a model of a supply chain. These are typical applications where cooperation is of benefit to all partners involved. A problem here is that currently there do not exist proper tools to solve the problems involved. It is expected that in the future these types of projects will gain importance, and additionally further globalization will generate a need for more cooperation. The industrial simulation practitioners perceive that distributed simulation has a place in industry; however, industry needs to be convinced on the benefits of distributed simulation. In order to realize this, time and realistic cases are needed.

A big hindrance is that distributed simulation and HLA are complex phenomena of a technical nature, whereas the practitioners in industry are generally not technically inclined. Although HLA takes care of quite some low level technicalities its interface is still too complex for industry. On the question whether adaptors and middleware solve this problem, the opinions diverge. Even if HLA is integrated into the modelling paradigm of COTS packages, the user is still exposed to the HLA functionality. Using stand-alone adaptors, that would also avoid being tied to a single RTI implementation, might give some relief here according to some experts. Others suggest to aim at an architecture that gives an interface much like a stand-alone COTS implementation using which one can specify at run time whether and how the model should be distributed.

HLA designers and developers

The experts provided insightful comments on the difference between simulation models built by defence and industry. In defence they observe a relatively small number of complex and expensive models. The complexity is related to the fact that there is much interaction between the constituting parts of the models, much more than in industrial models. Military models are often running on the edge of what is possible. This mandates full control over the environment explaining the choice of general-purpose languages. Owing to the complexity extensive requirement analysis is customary, which leads to high credibility of the models. This, and the high building costs involved make full model reuse feasible. Although the experts observe problems here, mainly related to semantic interoperability, reimplementation is not an issue. HLA is experienced as a good tool solving many low-level problems.

In industry HLA designers see many models being built of a similar kind. The most important issue here is cost that has to be kept low. These are the two main reasons why the standard approach in industry is to use COTS packages. They perceive industry as less focused on distributed simulation that explains their observation that COTS and HLA hardly go together. An additional argument is that HLA is developed for the military and that typical industry-related aspects have not been taken into account when designing this standard. Subsetting HLA towards the functionality needed by industry might be a good approach. In the opinion of the HLA developers, adopting HLA into COTS models is not a task for the model builder but for the package developers.

Another difference between the two application domains is that in defence there is one big player, the US DoD. This explains that in many cases, but not all, information hiding is not a big issue. In industry the experts see many players who are reluctant to cooperate out of fear to lose customers, which hampers development into the direction of coupling models written in different packages.

The experts involved in designing HLA envision a future in which ideas related to the use of Internet are implemented, repositories offering services that a customer may use, and lightweight clients. This entails a new business model in which clients are charged for services instead of packages.

Simulation practitioners in defence

The defence simulation practitioners mention that they see simulation modelling on a larger scale in defence than in industry, and that military simulation modelling is more research like, which entails that more control is needed over tools and the environment. This explains why defence uses general-purpose languages. They also observe more concern in the military domain on security aspects that leads to more emphasis on validation.

They agree that reuse is important, but again the semantic interoperability problem is emphasized. Submodels are often not designed with networking in mind with the effect that the data models do not fit together. This leads to the need to update the FOM of models in the case of applying HLA, and to change their internal workings. Sometimes one has to start anew from scratch, and it can even be the case that there is not enough time, budget or expertise to combine the submodels at all.

Concerning information hiding the defence practitioners state that it was a big issue in their international projects. The experts from defence perceive complexity, lack of transparency and insufficient performance as the most severe drawbacks of HLA. The complexity cannot be avoided because distributed simulation in itself is complex. They define transparency similar to the HLA vendors as the possibility to access the levels below the HLA RTI. Insufficient performance led one of the experts to implement a new RTI. Hierarchical federations and the option to specify more elaborate data models were mentioned as useful additions to the HLA standard. They perceive as the biggest advantage of HLA that it is a standard.

Experts from defence propose an idealized architecture that hides as much complexity as possible from the novice user offering only basic functionality, viz. data exchange, time management and execution control. Further, experienced users should be allowed more functionality and access to deeper layers enabling them to change default settings.

Commercial HLA vendors

Although the commercial HLA vendors are open for collaboration with industry, they are squarely targeting the defence domain. The main reason is that they hardly observe HLA projects in industry whereas, due to the US DoD mandate, defence has adopted HLA. The HLA vendors observe that COTS vendors are reluctant to cooperate with each other. They can understand this because they see no added value for them in agreeing on a common standard. Instead, they observe the tendency that COTS vendors try to lock their customers to their own package.

With respect to information hiding, a nice example is given by one of the experts, viz., simulation-based acquisition. Both experts stated that reuse is common in defence. They also mentioned the difficulty of semantically aligning the models to be combined. Not surprisingly, they see HLA as a good tool, that hides low-level technical details, is general and not locked to one specific area. As drawbacks of HLA they mention the steep learning curve and the need for users who are not only good programmers, but high-level thinkers as well. With regards to performance they admit that the DMSO RTI is not efficient enough, and that their products try to remedy this. Features that they would like to see added to the HLA standard are extendibility, fault tolerance, better security, the possibility to cooperate between different RTI implementations and better transparency in the sense that lower layers should remain visible through HLA.

In order to obtain greater acceptance of distributed simulation in industry HLA vendors mention middleware and the need for a protocol on the wire level that would enable package builders to write their own implementation rather than relying on RTI implementations built by others. Similar to the HLA developers, one expert envisages a new business model in which the modeller would combine parts obtained from the net that then would be put to work using grid computing.

6. Categorization of the collected data

This section organizes the data from Section 5.2 in a structured framework through coding. Coding involves extracting from the data obtained the relevant topics that were raised, and dividing these into subtopics. The resulting table forms the basis of a 'theory' that is presented in Boer (2005).

As a starting point for our categorization we considered the list on which our interview was based, cf. the section Overview. We used the items from this list as a preliminary list of topics. The answers that we obtained for these parts are combined, if feasible, and then recast into a list of subtopics. During this categorization process we observed that some subtopics are reconsidered in the answers in other parts as well.

The result is captured in Table 3. Notice the regrouping that has taken place. One new topic has been introduced in the list, 'characteristics of projects favouring a distributed simulation approach'. Under this new topic some old ones have been grouped, like 'reuse' and 'information hiding'. More information on this categorization process can be found in Boer (2005) and (Boer et al (2006b).

Table 3 - Topics derived from the survey.

Full table

It is illustrative to confront our findings with the ideas expressed in (Robinson, 2002, 2005), in which three modes of simulation practice are defined and four potential application areas of distributed simulation. The practice modes are the software engineering approach (complex, relatively problem independent models, to be reused during their lifespan of years), the organizational change mode (problem directed, relatively small models of a throw-away type) and the facilitation approach (models to be used as a heuristic in understanding and investigating a problem field). The paper indicates that in military the majority of the applications are of the software engineering type, whereas only few models of this type occur in industry, the majority there being of the organizational change type. Robinson also states that models of the third type are relatively rare, and in fact in our work we found no mention of it.

Robinson's list of application areas includes distributed simulation model execution (one model is distributed, or several models are linked), linking to other types of software (eg data bases) or real-time systems, experimentation (gaming, executing multiple replications or scenarios) and finally project processes (like sharing models). Clearly, we focused almost exclusively on the first area.

With the modes of practice on one axis and the application areas on the other one, Robinson constructs a matrix, and for each cell he indicates whether this combination can be characterized as pushed by technology or as pulled by market demands. For the sub-matrix covered by our survey, this reduces to two observations. First of all, for the software engineering mode the use of distributed simulation is pulled by the market. He explains this phenomenon by noting that in this case we deal with large-scale models and that a natural way to extend this scale is by applying distributiveness. Moreover, in military, both funding and expertise from the model builders is available for this purpose. On the other hand, for the organizational change mode distributiveness is (or can be) realized by a technology push, and he explains this by the fact that the characteristics of such models are the opposite ones as compared to software engineering type models.

The results of our survey clearly validate Robinson's observations, and can thus be interpreted as an experimental justification of his work. Our survey also indicates that there is an opportunity for distributed simulation in industry. Interestingly enough, as candidates for distributiveness both models of the software engineering type and of the organizational change type are suggested in our interview.

7. Conclusions and ensuing research

Our survey was based on the following three research questions: RQ1: Is it true that the HLA standard is hardly applied in industry?, RQ2: If so, why is HLA hardly applied in industry? and RQ3: What is needed to overcome this situation? It consisted of a questionnaire and a series of interviews.

The questionnaire aimed to validate our hypothesis that the HLA standard is rarely applied in industry. Looking superficially at the questionnaire results it seemed that our hypothesis was contradicted. Deeper analysis, however, unveiled several inconsistencies that seemed to favour our hypothesis again. Our conclusion is that, although the questionnaire could not support us to completely validate the hypothesis, from the analysis that we have carried out and a second round of questions we conclude that the hypothesis as it is still stands. As a matter of fact, in the next part of the overall survey it was firmly validated.

Concerning the second research question, there are several arguments we collected from the questionnaire, indicating why the HLA standard is hardly applied in industry. The heavy structure of the HLA RTI, performance issues, alignment of shared data, a complex runtime management, verification and debugging and translation of HLA concepts into COTS terms lay a burden on the developer, resulting in an unfavourable cost–benefit ratio.

Regarding the third research question we conclude that although some of the COTS simulation package vendors are planning to support distributed simulation or HLA in the future, currently there does not seem to be a big drive into this direction. Finally, we have observed that when an organization intends to accommodate distributed simulation this is mostly done by creating 'homespun' architectures based on low-level technical solutions (eg WinSock, CORBA, COM, etc). On the other hand, there are quite a few organizations that do not have any intention at all to support connection with other packages.

The next step in our survey was to take the above-mentioned results as a starting point for a series of interviews. Now that our hypothesis behind research question RQ1 is strengthened we were justified to delve more deeply into the other ones. We conducted interviews with people from a much bigger population than only the COTS vendors who naturally did cast light at these problems from their own point of view.

The results are summarized in the section Overview and can be organized along the lines of Table 3. We have concluded that the HLA standard is rarely used for integrating simulation models designed and developed in COTS simulation packages. Further, we highlighted several reasons for this, reasons relating to distributed simulation in general, to the HLA standard itself, and to the relation between the COTS simulation packages and the HLA standard. Interestingly all three categories connect to the very general economic reason of the cost–benefit ratio.

Based on these reasons, our next goal is to answer the following question: How can we make distributed simulation and existing distributed solutions, like HLA, more attractive to the industrial community?

One of the community activities that is taking place nowadays is the standardization effort of CSPI PDG, regarding the way in which COTS simulation packages interoperate via the HLA (Taylor et al, 2007). This standardization approach defines a set of interoperability reference models for COTS simulation packages that are being standardized by the CSPI PDG in their standard for commercial-off-the-shelf simulation package interoperability reference models (http://www.sisostds.org, accessed 3 March 2008). The introduction of this standard is one way to provide an answer to the above question.

Next to this option that incorporates HLA features in COTS packages, our survey indicates that there is another way to answer the above question, namely by introducing a light weight COTS-based distributed simulation architecture that might improve the unfavourable cost–benefit ratio. This idea has been worked out in Boer (2005) and will be discussed in a forthcoming paper (Boer et al, 2008), where a requirement list for an architecture for industrial distributed simulation is proposed based on the results of the survey described here. Taking this requirement list as a starting point, we have developed a lightweight distributed simulation architecture, called the FAMAS simulation backbone architecture, intended to couple port-related distributed simulation models. We presented this architecture in Boer (2005) where we evaluated the implemented architecture through two case study projects. We concluded that the backbone solution presented was appropriate for the purpose for which it was designed.

Notes

Appendix—Abbreviations

ALSP Aggregate-Level Simulation Protocol

COM Component Object Model

CORBA Common Object Request Broker Architecture

COTS Commercial-Off-The-Shelf

CSPI PDG COTS Simulation Package Interoperability Product Development Group

DCOM Distributed Component Object Model

DIS Distributed Interactive Simulation

DMSO Defence Modelling and Simulation Office

DoD Department of Defence

ESIW European Simulation Interoperability Workshop

ESM The European Simulation and Modelling Conference

FOM Federation Object Model

HLA High-Level Architecture

HLA-CSPIF The High-Level Architecture Commercial-Off-The-Shelf Simulation Package Interoperability Forum

OMT Object Model Template

PADS Parallel and Distributed Simulation

RTI Run Time Infrastructure

SIMNET SIMulator NETworking

SOM Simulation Object Model

SISO Simulation Interoperability Standards Organization

SIW Simulation Interoperability Workshop

TOMACS Transactions on Modelling and Computer Simulation

WSC Winter Simulation Conference

References

1. Boer CA (2005). Distributed simulation in industry. ERIM PhD Series Research in Management, Rotterdam, The Netherlands.
2. Boer CA, Bruin A de and Verbraeck A (2006a). Distributed simulation in industry—A survey, Part 1—The COTS vendors. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM and Fujimoto RM (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: Monterey, CA, USA, pp 1053–1060.
3. Boer CA, Bruin A de and Verbraeck A (2006b). Distributed simulation in industry—a survey, Part 2—Experts on distributed simulation. In: Perrone LF, Wieland FP, Liu J, Lawson BG, Nicol DM and Fujimoto RM (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: Monterey, CA, USA, pp 1061–1068.
4. Boer CA, Bruin A de and Verbraeck A (2008). Distributed simulation in industry: Overview and perspectives. In preparation.
5. Creswell JW (2003). Research Design. Qualitative, Quantitative, and Mixed Methods Approaches. Thousand Oaks, SAGE Publication: CA, USA.
6. Dahmann JS, Fujimoto RM and Weatherly RM (1997). The department of defense high level architecture. In: Andradóttir SK, Healy J, Withers DH and Nelson BL (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: Atlanta, GA, USA, pp 142–149.
7. DIS Steering Committee (1994). The DIS vision, a map to the future of distributed simulation. Technical Report IST-SP-94–01, Institute for Simulation and Training, Orlando, FL, USA.
8. Fujimoto RM (2000). Parallel and Distributed Simulation Systems. John Wiley and Sons, Inc.: New York, USA.
9. IEEE 1516 (2000). IEEE Standard for Modelling and Simulation (M&S) High Level Architecture (HLA). Institute of Electrical and Electronics Engineers: New York, USA.
10. Kenis D (1995). Improving group decisions. Designing and testing techniques for group decisions support systems applying Delphi principles. PhD Thesis, University of Utrecht, Faculty of Social Sciences, Utrecht, The Netherlands.
11. Kuhl F, Weatherly R and Dahmann J (1999). Creating Computer Simulation Systems: An Introduction to the High Level Architecture. Prentice-Hall: New Jersey, USA.
12. Law AM and Kelton WD (2000). Simulation Modeling and Analysis. McGraw-Hill: Boston, USA.
13. Linstone HA (1978). The Delphi technique. In: Fowles J (ed) Handbook of Future Research. Greenwood Press: London, pp 273–300.
14. Linstone HA and Turoff M (1975). The Delphi Method. Techniques and Applications. Addison-Wesley: London, UK.
15. McLean C and Riddick F (2000). The IMS MISSION architecture for distributed manufacturing simulation. In: Joines JA, Barton RR, Kang K and Fishwick PA (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: Orlando, FL, USA, pp 1539–1548.
16. Nikoukaran J, Hlupic V and Paul RJ (1999). A hierarchical framework for evaluating simulation software. J Simulat Pract Theory (SIMPRA) 7(3): 219–232. | Article |
17. Pope A (1989). The SIMNET network and protocols. Technical Report 7102, MA:BBN Systems and Technologies, Cambridge, MA, USA.
18. Rabe M, Jaekel FW and Gurtubai GG de (2001). Modelling and simulation for globally distributed enterprises. In: Heemink A, Dekker L, Swaan Arons H de, Smit I and Stijn T van (eds) The 4th International EUROSIM 2001 Congress. Elsevier: Delft, The Netherlands.
19. Revetria R, Blomjous PEJN and Van Houten SPA (2003). An HLA federation for evaluating multi-drop strategies in logistics. In: Verbraeck A and Hlupic V (eds) Proceedings of the 15th European Simulation Symposium. SCS European Publishing House: Delft, The Netherlands, pp 450–455.
20. Robinson S (2002). Modes of simulation practice: Approaches to business and military simulation. Simulat Pract Theory 10: 513–523. | Article |
21. Robinson S (2005). Distributed simulation and simulation practice. Simulation 81(5): 5–13. | Article |
22. Schreiber G et al (2000). Knowledge Engineering and Management: The CommonKADS Methodology. MIT Press: Cambridge, MA, USA.
23. Seale C, Gobo G, Gubrium JF and Silverman D (2004). Qualitative Research Practice. SAGE Publications Ltd: London, UK.
24. Singhal S and Zyda M (1999). Networked Virtual Environments. Design and Implementation. ACM Press, Addison-Wesley: New York, USA.
25. Straburger S (2001). Distributed simulation based on the high level architecture in civilian application domains. PhD thesis, University Otto-von-Guericke, Magdeburg, Germany.
26. Swain JJ (2007). INFORMS simulation software survey. OR/MS Today. Institute for Operations Research and the Management Sciences (INFORMS), USA. ( http://www.lionhrtpub.com/orms/surveys/Simulation/Simulation.html, accessed 2 March 2008).
27. Taylor SJE et al (2002). Distributed simulation and industry: Potentials and pitfalls. In: Yücesan E, Chen CH, Snowdon JL and Charnes JM (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: San Diego, CA, USA, pp 688–694.
28. Taylor SJE, Gan BP, Straburger S and Verbraeck A (2003). HLA-CSPIF panel on commercial off-the-shelf distributed simulation. In: Chick S, Sánchez PJ, Ferrin D and Morrice DJ (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: New Orleans, LA, USA, pp 881–887.
29. Taylor SJE et al (2007). The SISO CSPI PDG standard for commercial-off-the-shelf simulation package interoperability reference models. In: Henderson SG, Biller B, Hsieh MH, Shortle J, Tew JD and Barton RR (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: Washington, DC, USA, pp 594–602.
30. Taylor SJE, Wang XG, Turner SJ and Low MYH (2006). Integrating heterogeneous distributed COTS discrete-event simulation packages: An emerging standards-based approach. IEEE Trans Syst, Man Cybern 36(1): 109–122. | Article |
31. Tewoldeberhan TW, Verbraeck A, Valentin EC and Bardonnet G (2002). An evaluation and selection methodology for discrete-event simulation software. In: Yücesan E, Chen CH, Snowdon JL and Charnes JM (eds) Proceedings of the Winter Simulation Conference. Association for Computing Machinery Press: San Diego, CA, USA, pp 67–75.
32. Weatherly R, Seidel D and Weissman J (1991). Aggregate level simulation protocol. In: Proceedings of the 1991 Summer Computer Simulation Conference. Society for Computer Simulation: Baltimore, MD, USA, pp 953–958.

Acknowledgements

We thank the commercial-off-the-shelf simulation package vendors for participating in the questionnaire survey, the simulation experts for the useful discussions during the interview survey, and two anonymous referees for their constructive remarks.