1. Introduction

Conceptual modelling is a crucial stage of the simulation modelling process, and yet it is poorly understood. Robinson (2004) defined the conceptual model as a 'non-software specific description of the simulation model that is to be developed, describing the objectives, inputs, outputs, content, assumptions and simplifications of the model'. Conceptual modelling therefore involves deciding the way in which the virtual world of the simulation model should work. Law (1991) considered that for simulation projects 'the most difficult aspect of a study is that of determining the appropriate level of model detail'. However, little attention is devoted to conceptual modelling in most textbooks.

The advice that is provided often centres on the complexity or level of detail of the model. For example, Robinson (1994) proposed that the basic rule for what to include in a model is to use the minimum components required to achieve the project's objective. In fact, 'Model Simple—Think complicated' is one of Pidd's (2003) principles of modelling, and Ward (1989) and Salt (1993) also set out a number of advantages of a simple model. However, definitions of level of detail and complexity are not usually provided in the literature and there are no agreed ways of measuring them.

A particularly interesting study in this area is that of Willemain (1995) who carried out an experiment to investigate the initial stages of a modelling project. The experiment consisted of providing Operational Research (OR) experts with a description of an OR modelling problem, and asking the expert to speak aloud their thoughts on tackling the problem for a period of an hour, while recording this on tape. Transcripts of the recordings were then analysed by breaking them into 'chunks' (from a phrase to a couple of sentences) and categorizing each one by a topic in the modelling process. The five topics used by Willemain were context, structure, realization, assessment and implementation.

In Willemain's experiment, there were four different problems and four experts tackled all four problems. A further eight experts tackled one problem each giving a total of 24 sessions. The categorizations were analysed in various ways including a 'topic plot' showing which topic the expert was working on throughout the transcript, the number of transitions between each pair of topics, the proportion of lines of transcript devoted to each topic and a box plot of topic position. One of the main results was that even though the sessions only lasted an hour, the experts spent a considerable proportion of the time on all topics other than implementation, with a lot of alternation between the different topics. In particular, structure (essentially conceptual modelling) was often followed by assessment (essentially verification and validation) and assessment was often followed by structure. In other words, the experts would tend to develop an aspect of the conceptual model, then evaluate it and then often revise the conceptual model based on this evaluation. Recently, Willemain and Powell carried out a similar experiment using novice modellers (Powell and Willemain, 2007; Willemain and Powell, 2007). They identified five main ways in which the novices fell short of what they considered to be good modelling practice, which were: over-reliance on data, taking shortcuts, insufficient use of variables and relationships, ineffective self-regulation and overuse of brainstorming. In his earlier work, Willemain (1994) also carried out a survey of the 12 experts in his experiment, which provided revealing insights on their modelling styles and their views on the ideal qualities of modellers, models and clients.

Conceptual modelling is often thought of as a skill that improves with experience. One way for all modellers, but particularly novice modellers, to get better at conceptual modelling is therefore to draw on the experience of experts. Knowledge of what both expert and novice modellers actually do in practice is also an essential foundation for conceptual modelling research (Brooks, 2007). However, apart from the work just described, there is a lack of empirical studies or data in the literature on how modellers develop conceptual models and on how conceptual modelling relates to the other modelling topics. This paper describes a study to collect and analyse data on this process for an expert and several novice groups tackling real problems. The results are discussed and the lessons learnt and possible future work outlined.

2. Study objective

The objective of this study was to improve the understanding of the modelling process followed in practice by different modellers, focussing particularly on conceptual modelling. The general approach follows that of Willemain (1995) in collecting data on the topics worked on during the modelling process. However, here data were collected throughout a real project for an expert and nine groups of novices. The study therefore differs from Willemain's in four main ways: firstly, the projects are all simulation projects; secondly, they are real projects; thirdly, data were collected for the whole project rather than just the initial stage and fourthly, groups of novices as well as an expert were followed. In fact, moving to real-life projects, looking at novices and looking at groups of modellers were all future experiments suggested by Willemain.

3. Data collection

3.1. Expert project data collection

The first project used in the study was conducted by an expert. The expert holds a masters degree in OR and prior to the study had 4 years of modelling and simulation experience in a variety of application fields including manufacturing, military and healthcare. The project was carried out part-time by the expert over a period of 10 weeks, and involved modelling a call centre to improve the efficiency of staff usage. The simulation software used was Micro Saint Sharp (Alion MA&D Operation, CO, USA), which was selected by the client.

The expert was asked to record the total number of hours spent each week on different modelling topics. There was a desire to compare these results with those of Willemain (1995) and so Willemain's paper was used as a basis. The expert preferred to use one of the alternative list of topics (from Hillier and Lieberman, 1967) given in Willemain as follows (with the matching topic according to Willemain given in brackets): formulating the problem (context), constructing a mathematical model (structure), deriving a solution (realization), testing the model and solution (assessment), establishing controls over solution (implementation) and implementing the solution (implementation). Each week, the expert recorded the number of hours spent on each of these topics.

The Expert modeller was also interviewed each week and asked whether and how the conceptual model had changed during the week and, if there had been a change, about the process and reasons for changing the model. General issues, for instance, the main task of the week and whether working on one topic influenced the others were also discussed.

3.2. Novice projects data collection

Data were obtained for nine Lancaster University student group projects in two phases. Data for six projects were collected in phase 1 in 2005 and data for a further three projects were collected in phase 2 in 2006. All projects lasted for about 12 weeks. In phase 1, two of the groups were from the simulation module on the masters course in OR and the other four were from the undergraduate simulation course. In phase 2, all three projects were from the undergraduate simulation course. The undergraduate students were in their 2nd or 3rd years in various departments in the Management School. They had little programming and simulation modelling experience prior to the course. The backgrounds for the masters students varied depending on their first-degree subjects and previous work experience. Some had programming and modelling experience, but in general, their prior knowledge of simulation was limited. As the main assessment for both courses, the students were required to find a suitable project on a real system (typically from around the university campus) and carry out a complete simulation project. Therefore, although the projects are modelling a real problem there is no external client as such, although the projects are done with the cooperation of the external company if there is one. The masters groups had three students while the undergraduate groups had five students. The educational version of the simulation software package Witness (Lanner Group Ltd., Redditch, UK) was used for all projects. Table 1 shows the systems that were modelled.

Table 1 - Systems modelled in the novice projects.

Full table

In the phase 1 study, weekly questionnaires were handed out to each group before the project started. Each group was asked to record the total hours spent on the different topics every day during the week, as well as whether the conceptual model had changed during the week. In this case, a much more detailed list of topics was provided than the ones used by Willemain (1995) so as to obtain more detailed data and to reduce the amount of interpretation required by the students. In the subsequent analysis, the topics were combined into our own preferred list of simulation tasks. The topics were (with the topic from our list in brackets): identify alternative potential projects (problem structuring), contact/interview with the client (problem structuring), observe the system (problem structuring), discuss with experts (problem structuring), set project objectives (problem structuring), decide the model structure (conceptual modelling), model coding (model coding), collect data for the model (data collection and analysis), parameter estimation and distribution fitting (data collection and analysis), white-box validation (verification and validation), black-box validation (verification and validation), verification (verification and validation), experiment with the model and analyse the results (experimentation) and report writing (report writing).

The same data were collected in phase 2 but in an improved way. The limitation of the method used in phase 1 is that the reliability of the data depended on the accuracy of the students in recording the time spent and also on how well they were able to match their tasks against the categories provided. Also data were only recorded on a daily basis. To overcome these drawbacks, in the phase 2 studies, the researcher (Wang Wang) sat in on most of the student group meetings, observed their behaviour and recorded the relevant time herself in hourly intervals. Where group members conducted individual work outside the meetings, they reported to the researcher on what task they worked on and the time spent on that task. In addition, the updated computer model was saved at the end of each group meeting so that the changes to the model could be tracked. Collecting data in this way gives more confidence in the reliability of the data. In both studies, the hours were not adjusted for the number of people doing each task because of the difficulty in assessing the extra effort this represents. For example, two students working together on coding the model for 2 h was recorded as 2 h (rather than four).

4. Results

4.1. Results for expert

The analysis of the data follows some of Willemain's analysis by calculating the relative weights of the different topics, and showing a graphical representation of the topics over time. Figures 1 and 2 shows these results for the expert project, while Figure 3 shows the average weight given to each topic in the 24 sessions in Willemain's experiment measured in number of lines in the transcripts. As Figure 1 shows, the expert spent most time on modelling and testing the model.

Figure 1.

Proportion of time spent on each topic in the expert project. The topics are (with the matching Willemain topic in brackets): P (C)=formulating the problem (context), M (S)=constructing a mathematical model (structure), S (R)=deriving a solution (realization), T (A)=testing the model and solution (assessment), E (I)=establishing controls over solution (implementation) and I (I)=implementing the solution (implementation).

Full figure and legend (34K)

Figure 2.

Timeline plot for expert project. The topics are as in Figure 1. The data were collected weekly over 10 weeks, which are shown by the vertical dashed lines.

Full figure and legend (54K)

Figure 3.

Percentage of lines devoted to each topic in Willemain's 24 experiments (Willemain, 1995).

Full figure and legend (32K)

The timeline plot (Figure 2) shows the topics worked on during the project. The expert project data were obtained on a weekly basis over the 10 week period of the project. Only the total number of hours spent on the topics in each week were recorded. Since the precise timings during the week are not known the plot spreads the topics evenly during each week. If more than one topic was worked on during the week then this is shown by the bars not being full height in the plot (a full height bar would reach the horizontal line above on the plot). For example, in the second week the expert spent a total of 10 h working on the project, which consisted of 6 h on formulating the problem (P) and 4 h on constructing the model (M). This is shown in the plot by the heights of the bars for P and M being, respectively, 60% and 40% of a full height bar for each hour in a 10 h period (hours 4–13). This data collection was less detailed than Willemain's data obtained in a laboratory setting, where the protocol recorded what was happening all the time. One consequence is that where more than one topic took place during the week then the order and the interaction between the topics is not known. There could have been a lot of switching between the topics during the week or, on the other hand, the topics could have been worked on completely separately one after the other. This prevented a detailed analysis of the switching between topics as carried out by Willemain. Nevertheless, the topic plots still give useful information about the positions and sequence of the topics throughout the project. In particular, the extensive overlap between the topics does indicate a considerable amount of alternation between the topics rather than a linear process. In general, the topics were in the anticipated order with topics higher up on the y-axis expected to be later.

A comparison of Figures 1 and 3 shows a reasonably similar split between the topics. This perhaps indicates that the relative time spent by the expert on the different topics over the course of the whole project was similar to that spent by the experts in the initial hour of Willemain's experiment. However, this comparison should be treated with caution because it depends on how similar the allocation process was. In particular, an alternative list was used for the expert project and this may not match up perfectly with Willemain's categories. With hindsight, neither list gives a sufficiently detailed list of topics for a simulation project and the data collected for the novice projects are more informative in this respect.

4.2. Results for novices

The proportion of time spent on the topics for the novice projects in phase 1 and phase 2, respectively, are shown in Figures 4 and 5. In each case, the percentage of time on each topic was calculated for each project and then the project values were averaged. The pattern is reasonably similar for phase 1 and phase 2, which gives some additional confidence that the results for phase 1 are reliable even though they were recorded by the students themselves. A considerable amount of time was spent on data collection and report writing. Conceptual modelling received relatively little attention particularly in the phase 2 projects.

Figure 4.

Proportion of time spent on the topics in the six novice projects in phase 1. PS=problem structuring, CM=conceptual modelling, DC=data collection, MC=model coding, VV=verification and validation, EX=experimentation and RW=report writing.

Full figure and legend (35K)

Figure 5.

Proportion of time spent on the topics in the three novice projects in phase 2. The topics are as in Figure 4.

Full figure and legend (32K)

Observation by the researcher of the process for the phase 2 projects gave additional insight into the results. The high proportion of time spent on experimentation was partly due to the technical problems they experienced or mistakes they made. For example, one group did all the experimentation twice as they forgot to consider the warm-up period (the initial transient period before the model reaches the realistic conditions in the system) at the beginning. Another group had little understanding about warm-up and, as a result, they had to go through the lecture notes first before they could perform this task. Generally with these projects the groups have to collect their own data, which is the reason for the high proportion of time on data collection.

The timeline plot shown in Figure 6 is the same general format as Figure 2. As already explained, most of the data for the phase 2 novice projects were obtained by observation of the group by the researcher and was recorded on an hourly basis. The overlapping topics in hours 1, 13 and 15 are times when both topics were worked on during the hour. However, some of the work was done individually by the group members and the total time spent was just reported to the researcher. These data are therefore less detailed with the precise interaction between the topics not known. The period from hours 17–25 was not observed and instead the group members reported spending 3 h each on verification and validation, experimentation and report writing. As with the expert plot (Figure 2), such data are shown by spreading the topics evenly over the total period.

Figure 6.

Timeline plot for one of the phase 2 novice projects. Topics are as in Figure 4.

Full figure and legend (44K)

The pattern of the plot in Figure 6 is a fairly linear process. The novices tended to complete one topic then move onto the next with not much overlap and with very little returning to a previous topic. Most of the novice projects were similar in this respect. Although the topic categories are different, this is a quite different pattern to the expert (Figure 2) and also to the pattern in Willemain's (1995) experiments. Figure 2 shows much more overlap of topics although, as previously explained, the precise pattern within each week for the expert is not known. The overlap of the topics over several weeks shows that there was more switching between topics for the expert than the novices. Since there may also have been several iterations within each week for the expert, this difference may be even more marked than is shown on the graphs. Another comparison that can be made is that the expert started model testing (verification and validation) much earlier than the novice modellers who tended to leave it until after model coding was completed. As for the expert project, the average position of the topics for the novices was in the expected order with topics higher up the y-axis on Figure 6 expected to be later.

4.3. Further findings and analysis

Upon the completion of both the expert and novice studies, a further discussion took place with the expert to try and re-classify the topics to enable better comparison with the novice projects. Using our list of topics, the expert decomposed each original topic and provided its approximate weighting (eg establishing controls over solution=1/4 verification and validation+3/4 experimentation). This enabled a revised weight breakdown to be produced although not a revised timeline. However, discussion with the expert indicated that the general pattern of task overlapping would be similar. It should be noted that this re-allocation took place more than 12 months after the project finished and therefore, the values should be regarded as approximate. This revised weighting is shown in Figure 7. Again this shows that the data are very different to that of the novice projects (Figures 4 and 5). With the expert, the topic that received the most attention was conceptual modelling, and much more time was spent on verification and validation than experimentation. In general, it may be that experienced modellers have a greater appreciation than novices of the importance and benefits of both conceptual modelling and verification and validation, perhaps by learning from problems on previous projects. On the other hand, the lower attention on conceptual modelling and validation by the novices could be because the systems they modelled were so simple that these tasks were straightforward to perform. To investigate this further, the novice project reports were studied. This showed poor performance on validation and verification with this task receiving the lowest marks on average in the assessment of the reports. For example, two groups failed to distinguish between white-box validation and verification and out of nine projects investigated, only two groups performed black-box validation properly. Sometimes this was due to a lack of planning. For instance, two groups did not consider collecting data for validation when planning data collection.

Figure 7.

Proportion of time spent on each topic in the expert project using revised allocation to the same topics as the novices. Topics are as in Figure 4.

Full figure and legend (33K)

The conceptual modelling process can be considered in more detail based on the discussions with the expert and observations of the phase 2 novice groups. The expert developed the conceptual model at the beginning of the project and documented it in a system flow diagram, which guided the construction of the computer model. However, the novice groups devoted little time to understanding how the systems actually worked. They did discuss the process of the system, but rarely drew a diagram. After identifying the project, they tended to go straight to collecting data with little prior planning or consideration of the model structure. As a result, some of the data collected proved not to be useful. This is inefficient particularly as data collection is time consuming. Sometimes further discussions on the system process occurred at the model-coding stage with conceptual modelling and coding taking place together. Some groups only documented the conceptual model at the end in order to include a diagram in the report.

In the expert study and both novice studies, the subjects were asked to note any changes in the conceptual model each week and the reason. For the expert project, there was one significant conceptual model alteration towards the end of the project. This involved a scope reduction due to the fact that the collected data were not sufficient to support the model built. In one of the novice groups, one student left in the middle of the project causing a change of project application to make the problem easier to model. Another group attempted to increase the scope from modelling the busy hours of a grocery store to modelling the whole period of the business hours, but gave up after experiencing some difficulties in finding out the right distribution of data and transferring it into the computer model. The other novice groups did not adjust the conceptual models after the coding stage.

5. Discussion and conclusions

This study was designed to provide information to improve the understanding of conceptual modelling. The time spent on different topics had a quite different pattern between the expert project and the nine novice projects. The expert project had much more overlapping of the topics and had a higher proportion of time on conceptual modelling and verification and validation. The novice projects had much more time allocated to data collection because they had to collect the data themselves. One of the difficulties with research on real projects is that all the projects are different. Generally, the student projects are fairly simple problems and so the differences between the expert and novice data could be evidence of different working styles or it could be due to the greater complexity of the expert project.

Carrying out this type of research is difficult and a number of problems were encountered. Following (shadowing) experts in real projects has practical difficulties, such as project confidentiality, and finding the projects with the right size. Obtaining data in this way also requires a significant time commitment. The number of projects that could be followed was therefore limited. Following real projects also inevitably limits the analysis compared to an artificial laboratory experiment in that all the projects were different. For example, Willemain (1995) was able to compare different modellers tackling the same problem and some of the modellers tackling different problems, to try and identify any modeller and problem effects. With hindsight, using more topic categories for the expert project would have provided more detailed data. A possible ideal approach in the future would be to follow a real small-scale consultancy project, and then bring it to the classroom to be tackled by novices. However, finding an appropriate project is likely to be very difficult. A different approach would be to use a questionnaire to obtain data from a larger sample size of experts on their perceptions of their modelling projects.

Obtaining this sort of information about conceptual modelling and the modelling process is an important step towards a better understanding of the key aspects of successful modelling practice. In the long term, it is hoped that further research in this area will improve the success of simulation and OR projects, and that the information can help in the training of novice modellers.

References

1. Brooks RJ (2007). Conceptual modelling: Framework, principles, and future research Working paper no. 2007/011, Lancaster University Management School, Lancaster, UK.
2. Hillier F S and Lieberman G J (1967). Introduction to Operations Research. Holden-Day: San Francisco.
3. Law AM (1991). Simulation model's level of detail determines effectiveness. Indust Eng 23(10): 16–18. | ISI |
4. Pidd M (2003). Tools for Thinking, Modelling in Management Science, 2nd edn. John Wiley and Sons: Chichester.
5. Powell SG and Willemain TR (2007). How novices formulate models. Part I: Qualitative insights and implications for teaching. J Opl Res Soc 58(8): 983–995. | Article |
6. Robinson S (1994). Simulation projects—Building the right conceptual model. Indust Eng 26(9): 34–36. | ISI |
7. Robinson S (2004). Simulation: The Practice of Model Development and Use. Wiley: Chichester.
8. Salt JD (1993). Keynote address: Simulation should be easy and fun!. In: Evans GW: Mollaghasemi M, Russell EC and Biles WE (eds). Proceedings of the 1993 Winter Simulation Conference. New York: IEEE, pp 1–5.
9. Ward SC (1989). Arguments for constructively simple models. J Opl Res Soc 40(2): 141–153. | Article |
10. Willemain TR (1994). Insights on modeling from a dozen experts. Opns Res 42(2): 213–222.
11. Willemain TR (1995). Model formulation: What experts think about and when. Opns Res 43(6): 916–932.
12. Willemain TR and Powell SG (2007). How novices formulate models. Part II: A quantitative description of behaviour. J Opl Res Soc 58(10): 1271–1283.

Acknowledgements

We thank the anonymous referees for their helpful comments and suggestions, which improved the focus of the paper.