1. Introduction

Undertaking a review of modelling and simulation in health care is without doubt a Herculean task. This is a literature which, having carried out searches on consecutive days using the Web of Knowledge (WoK) bibliographic database (wok.mimas.ac.uk) and the search string ‘((healthcare or health care) and (modelling or modeling or simulation))’, was found to be expanding at the rate of about 30 articles a day. A search carried out on June 21, 2007 using the Ovid search engine (www.ovid.com) and the same search string resulted in 176 320 hits. It is hard to imagine how a single person, research group or academic department could begin to keep up with such a literature.

Nevertheless this is the task that the Research Into Global Healthcare Tool (RIGHT) project team set itself. RIGHT (www.right.org.uk) is a collaborative research venture between six UK universities, funded by the British Engineering and Physical Sciences Research Council. The aim of RIGHT is to assess the feasibility of applying to decision making in health care some of the best-practice modelling and simulation methods that are used to support decision making in other sectors, such as manufacturing industry and defence.

The first phase of the RIGHT project has involved eight extensive literature reviews, of which this is one. Nearly all of these involved massive literatures and therefore an innovative common methodology was devised and developed, in order to reduce the scope of the task to something achievable in the time available. The other review topics are: simulation and modelling in manufacturing industry, simulation and modelling in aerospace, simulation and modelling in defence, management methods (excluding simulation and modelling) in health care, management methods (excluding simulation and modelling) in manufacturing industry, stakeholder analysis and framework development.

The study was concerned only with modelling as understood by an operational researcher, namely a structured approach to understanding (and possibly, but not always, solving) a real-world problem through developing a simplified version of the real system. We were particularly, but not exclusively, interested in applications of simulation. This covered computer-based approaches such as discrete-event simulation, agent-based simulation and system dynamics, as well as role-playing or business-gaming simulations. In a medical context the word ‘model’ covers a wide range of meanings. Therefore, in order to avoid as far as possible clinical, biochemical, microbiological or pharmacological articles where the word model has a very technical and specialised meaning, we restricted the search criteria to the terms modelling and modeling.

The aim of undertaking this review, and indeed the other reviews in the RIGHT project, was not merely to produce an academic article. The overall aim of RIGHT is to produce a ‘toolkit’ of methods and an explanatory framework or user guide that will suggest, for a given type of health-care problem and a given set of available resources at the user's disposal, the most suitable method(s) to use. The RIGHT project is a feasibility study and the toolkit and user guide will be tested on a sample range of exemplar sites (Naseer et al, 2009).

2. The ancestry of health-care modelling reviews

Several review articles over the years have been written on health-care modelling. These have tended to focus either on a specific modelling methodology, such as discrete event simulation (for example, Fone et al, 2003) or on the use of modelling for a specific health-care setting, such as clinics (for example, Jun et al, 1999).

One of the earliest review articles in this field, and possibly the first comprehensive review of health-care modelling, was by Fries (1976), who compiled a list of 188 articles that the author grouped into 15 categories according to their area of application. These include forecasting demand, appointment systems, ambulance requirements and deployment, and health planning and programme evaluation. The articles were selected only if they used what Fries describes as ‘mathematical methods of modelling and solving decision problems that form the core of OR’. This bibliography was later supplemented with an additional 164 articles to make a total of 352 references (Fries, 1979). The review covers more than a dozen mainstream OR journals of that time, up to 1979, as well as referencing chasing as appropriate. The author does not provide details of the full list of journals searched nor the selection criteria, but one imagines that the 352 articles cited represent a large proportion of the body of health-care modelling literature at that time.

Two separate review articles on computer simulation projects were published 1 year later in 1980 by Tunnicliffe Wilson (1980). One article focused on applications to health-care population problems and the other on health-care facilities. Between them, they covered over 200 articles. A follow-up article by the same author (Wilson, 1981) focussed on implementation issues as the author reported that from the 200 reviewed articles, only 16 studies reported recommendations that had been acted upon.

Towards the end of the 1980s, Smith-Daniels et al (1988) reviewed the literature pertaining to acquisition decisions, for example sizing of facilities and facility location, and allocation decisions, for example inpatient admissions scheduling. They covered a number of techniques including simulation, queueing theory, Markov chains and heuristics. A few years later, Klein et al (1993) presented a bibliography that included medical decision making and simulation modelling with a focus on planning models.

Jun et al (1999) surveyed articles on the application of discrete event simulation modelling to health-care clinics and systems of clinics, for example hospitals, outpatient clinics and emergency departments. A taxonomy of the literature is presented covering published articles over the previous 20 years and categorised under two main themes: patient flow and allocation of resources. No discussion is made on the adopted review methodology and thus it is not possible to ascertain how systematic and wide-ranging this review is.

More recently, Fone et al (2003) produced a systematic review of computer simulation modelling in population health and health-care delivery. It is fair to say that this article is the first in health-care modelling to adopt a rigorous systematic review process that is described in detail in the article, and involved the screening of some 2729 references that eventually were reduced to 182 using inclusion criteria. The focus is entirely on discrete event simulation and articles are grouped into four application areas. The authors comment that although the number of modelling articles has grown substantially in recent years, very few report on outcomes of implementation of models and so the value of modelling requires further research. It is of interest to note that nothing appears to have changed over the years since Tunnicliffe Wilson made similar observations in 1981 (Wilson, 1981).

In summary, most of these previous reviews have focussed on simulation (and, in particular, discrete event simulation) or have included a broader range of OR and mathematical methodologies, but have focussed on specific application areas. Furthermore, most fail to describe the review process and presumably represent an exhaustive bibliography of articles from journals that happen to be searched by the review team. Certainly no systematic approach is reported, except that by Fone et al (2003). This article therefore fills a gap in the review literature by producing an up-to-date review unrestricted by methodology or application, and based on a systematic heuristic sampling review process covering a vast body of literature.

3. Review methodology and the RIGHT Information Template (RIT)

Within the 2-year timescale of the RIGHT project (of which the first 4 months was assigned to the literature reviews), it was clearly impossible to carry out anything approaching an exhaustive systematic review of any of these massive literatures. Therefore a heuristic, sampling-based approach was adopted across all eight reviews, using a variety of methods to identify the key articles and the emerging issues. This methodology has more in common with stratified experimental sampling than the kind of exhaustive survey typically attempted in a conventional literature review, for example a Cochrane systematic review (www.cochrane-handbook.org), where the aim is to ensure that all articles that meet a clearly defined set of inclusion criteria are read. In order to achieve a consistent approach across all eight reviews, a common template called the RIT was developed. The RIT contains the fields as shown in Table 1, which are recorded with fixed categories or free-text as appropriate. For this particular review, some of the fields of the standard RIT were modified slightly, as described at the end of this section. Some of the free-text fields in the RIT, in particular the ‘MethodName’ and the ‘FunctionalArea’ fields, were replaced by constrained lists in this study to facilitate quantitative analysis. The choice of specific methods was informed by the findings from the other RIGHT reviews.

Table 1 Information items included in the RIT

This study was far broader in scope than any of the previous health-care modelling reviews described above. The source literature was mainstream academic journal publications, accessible through three of the most widely used academic electronic databases: JSTOR (www.jstor.org), SCOPUS (www.scopus.com) and ISI WoK (www.wok.mimas.ac.uk). More general web searches using Google showed that the ‘grey’ literature in this area is equally massive and is worthy of further study in its own right. It will be the subject of a follow-up article later in the project, as arguably some of the most widely implemented work appears in the grey literature rather than the academic literature. SCOPUS covers journal publications from more diverse sources than JSTOR, but concentrates on more recent publications. Despite innovations in medical technology, the nature of the problems arising in health-care management has remained remarkably similar over the years. The SCOPUS search was limited to articles published after 1990, but the JSTOR search was unrestricted by date in order to capture the significant but older publications.

The literature review methodology consisted of three stages (Figure 1). In stage 1, a very broad set of search terms was used to produce an initial set of articles. The search string was ‘(health-care OR health care) AND (modelling OR modeling OR simulat*OR (system AND dynamic*) OR markov*)’, appearing in the title, abstract or keywords. In stage 2, a subset of these articles was selected for abstract review by a combination of ‘relevance rating’ and reference chasing as described below. Overall, 16% of the stage 1 articles were selected for abstract review, although this varied from 10% to 25% across the three different literature sources. In stage 3, the abstracts of all the stage 2 articles were read and a further down-selection made for inclusion in the final data set. The criteria used at this stage were that the article described a genuine application of modelling or simulation to a health-care problem. Any duplicates were removed at this stage, although there were surprisingly few of these between JSTOR and SCOPUS. The suitability of the stage 3 articles was then verified by full-text reading. In all 22% of the stage 2 articles were judged suitable for final inclusion, resulting in a total data set of 342 articles (119 from SCOPUS, 163 from JSTOR and 60 from WoK). A summary of the search results for stage 1 and sample sizes for stages 2 and 3 is shown in Table 2. The three stages required 3 months extensive work on searching, screening and recording required information, with approximately 20% of the time required for stage 1 and 40% each for stages 2 and 3.

Table 2 Publication counts for the three stages of literature review
Figure 1
figure 1

Stages of the review methodology.

JSTOR and SCOPUS both provide ‘relevance ratings’ and these were used in stage 2 to rank the first 500 articles in both databases for abstract scanning. It was not possible to discover the exact algorithm used to determine this relevance rating, but it was clearly based on the frequency of occurrence of the search terms. Many articles were eliminated at this stage, for example book reviews, abstracts of conference presentations or cost-effectiveness analyses of drug treatments (given we wanted to exclude the clinical, biochemical, microbiological and pharmacological literature). However, WoK does not provide such a ranking and therefore the innovative bibliometric visualisation tool CiteSpace Chen (2004, 2006) was used. Chen (2004, 2006) has demonstrated various uses of citation information and network analysis for the scientific literature. In particular, co-citation networks are a useful analytical method for the task of reference chasing. A co-citation network is a graphical representation of the references cited by a given set of publications enabling key articles that are widely referenced by later authors (ie, highly connected nodes of the network) to be identified. Using Citespace, a network was constructed using the cited references and citation count details from the stage 1 WoK articles, in order to down-select a set of relevant publications for stage 2 review. This set consisted of the 491 most cited references by more than 2500 publications in WoK selected with co-citation network; hence, it was representative of outcome from usual reference-chasing by researchers.

4. Data collection and recording

For each of the 342 articles in the final data set, the following information was recorded in an Excel worksheet:

  1. 1

    Methods

  2. 2

    Initiators

  3. 3

    Funding source

  4. 4

    Level of implementation

  5. 5

    Functional area

  6. 6

    Layer in the industry

  7. 7

    Country

  8. 8

    Databases and processes for literature review

  9. 9

    Year of publication

The ‘MethodName’ field from the standard RIT was expanded to allow up to three separate methods (primary, secondary and tertiary) to be recorded for each reviewed article, together with the software used, if stated. A two-level hierarchy was used to classify modelling and simulation methods in this review. For example, the high-level category ‘Simulation’ had eight sub-categories, including discrete-event, system dynamics, agent-based, distributed and Monte Carlo simulation. For each publication, a main method was assigned to the principal modelling approach employed in the study. A constrained set of method categories was used. Because many studies used more than one method, up to two subsidiary methods could be recorded. Thus, for instance, a study by Lehaney et al (1999) that used a Soft Systems (SSM) approach as a means to develop a discrete event simulation model would have two methods recorded, firstly its primary method, Simulation/Discrete Event Simulation, and secondly, Qualitative/SSM. Of the total 342 articles, 204 used only one method, 113 used two methods and 25 used three methods.

Similarly, data for the field ‘FunctionalArea’ were recorded at two levels. At the top level, four broad categories were used: stakeholder interest, clinical or organisational processes, patient care delivery planning and research/policy. A more detailed classification of health-care function used the following nine categories:

  1. 1

    Finance, Policy, Governance, Regulation

  2. 2

    Public Health, Community service planning

  3. 3

    Patient behaviour/characteristics

  4. 4

    Planning, System/resource utilisation

  5. 5

    Quality management, Performance monitoring or review

  6. 6

    Risk management, Forecasting

  7. 7

    Workforce/Staff management

  8. 8

    Research

  9. 9

    Other

Up to three of these categories could be recorded: a primary function and up to two other subsidiary functions. Of the total 342 articles, 102 were classified in one function only, 149 were classified in two categories and 91 were classified in three categories.

‘Layer’ (in the industry) was recorded at three levels: policy or regulation; facilitation or commissioning; and operation. Data for the field ‘ImplementationLevel’ were rated according to a three-level scale of implementation (see the Results section for further details).

5. Validation and verification

Systematic review approaches such as the Cochrane review methodology have a formalised structure in which the search strategy is highly prescriptive, and the inclusion and exclusion criteria for articles are precisely defined. A systematic review is (in theory at least) repeatable by other researchers, with identical results apart from the possible inclusion of articles that were unpublished at the time of the original review. The methodology described in this article can be similarly validated and repeated. Moreover, by way of ‘reality check’, the final list of 342 references was scanned by all four authors (who have, between them, over 50 years experience in the field of health-care modelling) to verify that certain well known, important articles from the literature had in fact been found and that no misclassified articles had been included. The full data set of references will be made available on the RIGHT website.

6. Results

6.1. Date of publication

The publication dates of the selected articles ranged from 1952–2007. However, the vast majority (82%) in our review was published after 1990. By decade, the percentages were: pre-1979: 7.0%; 1980–1989: 10.8%; 1990–1999: 36.3%; 2000–2007: 45.9%.

6.2. Country of origin

Each article was classified by the country in which the research study was carried out. When analysed by continent, the relative proportions were as follows: North America: 206 (60.2%); Europe: 84 (24.6%); Asia: 31 (9.1%); Africa: 10 (2.9%); Australasia: 6 (1.8%); South and Central America: 1 (0.3%). Four of the articles (1.2%) could not be classified by country. The vast majority of studies (85%), therefore, were undertaken in North America and Europe. Of the North American articles all but seven were conducted in the USA (the rest being Canada) and of the publications based in Europe, 55 of the 84 articles were from the UK. The preponderance of studies based in the US and UK is to a degree explained by the fact that the review was restricted to English language articles. However, it also almost certainly reflects the relatively high levels of health-care OR in these two countries.

6.3. Method

The majority of publications were found to fall into the categories of statistical analysis, statistical modelling, simulation and qualitative modelling. A smaller but significant number employ mathematical modelling, and very few fall into the remaining three categories, which are therefore aggregated and jointly classified as ‘Other’. Interestingly, where qualitative methods are used, they are very often a subsidiary method, whereas when mathematical modelling is used, it almost always forms the primary method. The primary method employed is shown in Figure 2 and Table 3.

Table 3 Table of association between primary and subsidiary methods
Figure 2
figure 2

Analysis of method by primary and subsidiary classifications.

When the more detailed second level of the modelling methodology tree was examined, a very wide range of methods was found in each of the major categories. Table 4 shows those methods which were used at least three times.

Table 4 Number of articles for each sub-category of method

Perhaps the most striking feature of this breakdown is the relatively low proportion of articles using these most common methods, with more than half of all articles having a primary method not shown in Table 4. In all 53% of articles have a primary method that is not observed in more than two articles. This gives an indication of the very wide variety of methods evident in the review. It can be seen that the most common primary method is some form of regression analysis (23% of all articles).

Interestingly, some techniques such as process mapping and Monte Carlo simulation were more commonly used as subsidiary methods. Typically, for instance, Monte Carlo simulation was used for testing or as a method of probabilistic sensitivity analysis for another form of model (eg, a Markov model). Qualitative approaches often formed a precursor to the development of a quantitative model such as a discrete event simulation.

The distribution of methods by year of publication, Figure 3, indicates that simulation and qualitative methods in particular are currently increasing in use. In contrast, other methods appear to have a similar uptake to the previous decade with mathematical modelling methods possibly in relative decline. The ‘Other’ category, for which the majority of articles are first observed post-2000, include spatial/GIS modelling, and system/software related methods such as UML (Unified Modeling Language) and IDEF (Integrated Definition Methods) for enterprise modelling and analysis.

Figure 3
figure 3

Analysis of method by year.

6.4. Funding

The primary source of funding, where reported, is shown in Figure 4. Funding sources were classified as commerce (such as consulting or commercial firms), academia (no formal Research Council grant/bursary provision), authorities (such as a Government organisation or agency), grants (funding bodies) or health services (such as direct funding from a hospital).

Figure 4
figure 4

Primary source of funding by method.

Overall, 60% of published work reported no formal funding, with only 4% funded directly by health services organisations. Notably, commercial funding has been mainly restricted to simulation studies with no examples of qualitative or mathematical modelling. However, in contrast, simulation fares less well with formal grant funding compared with other methods.

6.5. Functional area

The breakdown of publications by the top-level classification was as follows: stakeholder interest: 38 (11%); clinical and organisational processes and setup: 79 (23%); patient care requirement profiles and delivery planning: 117 (34%); research and policy: 108 (32%). The distribution of articles within the more detailed categories described above is shown in Figure 5, broken down by primary function and subsidiary functions. It demonstrates, for example, that planning and system/resource utilisation methods are predominant, and that unlike the other methods, quality management, performance monitoring and review methods are used more commonly as subsidiary methods.

Figure 5
figure 5

Distribution of function by primary and subsidiary classifications.

Figure 6 shows the relationships between function and method. Two particular features are that simulation methods are dominant in planning and system/resource utilisation, whereas statistical methods are dominant in finance, policy, governance and regulation. Further, more detailed analysis showed clear tendencies for certain functions to be associated with each other. For example, quality management and performance review was often coupled with planning system and resource allocation.

Figure 6
figure 6

Relationship between function and method.

6.6. Analysis by level of implementation

A key aspect of any study is the extent to which the model has actually been used in practice for its stated purpose. Each modelling study was rated according to a three-level scale of implementation: 1: Suggested (theoretically proposed by the authors); 2: Conceptualised (discussed with a client organisation); 3: Implemented (actually used in practice). The number of articles rated in each category was Suggested 171 (50%); Conceptualised 153 (44.7%); Implemented 18 (5.3%). Depressingly, these figures emphasise previous findings (Wilson, 1981; Fone et al, 2003) that levels of implementation for models in health-care OR are very small indeed and have not improved since the 1980s. A large proportion of modelling studies do, however, reach a conceptualised stage whereby a coherent approach is specified in a practical context with a health-care organisation.

Figure 7 shows the levels of implementation for each method. Statistical analysis was always either conceptualised or suggested with no instances of implementation. The proportion of conceptualised to suggested was higher for qualitative and statistical modelling, compared with mathematical modelling and simulation methods.

Figure 7
figure 7

Level of implementation by method.

7. Discussion

The aim of this review was to quantify and describe current levels of utilisation of modelling and simulation methods in health care, as reported in the mainstream academic literature. Ultimately, as part of the RIGHT project, this information will be used to develop an evidence-based ‘model selection toolkit’ to assist health-care professionals to choose an appropriate modelling approach to tackle a particular problem in a specific context. However, in a broader context, this study belongs to the family of health-care modelling reviews described earlier, and extends and develops some of this earlier work.

The findings on publication dates show a steadily increasing rate of publication in this field, with simulation and qualitative (soft) methods in particular rising in popularity. However, straightforward simulation studies are generally less successful in gaining Research Council funding, compared with other more complex methodologies. This is likely to be because Research Councils are generally looking for innovative experimental approaches, rather than standard methodologies with a proven track record. This is understandable, given that their role is to encourage new theoretical developments, but it does support the argument that the academic literature may not be the best place to look for practical applications of simulation. However, we have shown that simulation studies are generally more successful in attracting commercial funding.

In general, when considering funding sources, the academic literature shows a huge contrast with the ‘grey’ literature, as only 4% of studies were funded by a health service organisation. It is clear that the modelling work that is undoubtedly being undertaken within the health sector by business consultancies or by analysts employed within health-care organisations does not get written up for publication in academic journals.

The relationships between function and method suggest that certain business functions, such as finance, policy and regulation, are more likely to use statistical methods, arguably because these managers traditionally tend to have a more numerate background and are familiar with these approaches. On the other hand, simulation methods fare better in highly stochastic settings where the visual interface may be more important, such as resource utilisation and planning.

Overall levels of implementation are depressingly low and suggest that little has changed since previous review articles. Taylor et al (2009) report similar insights across the simulation modelling field, citing a lack of real-world involvement in published simulation modelling as a great, missed opportunity. Interestingly in our study, the implementation rates for statistical methods were particularly low. This may simply reflect the fact that such methods are very difficult for the lay person to understand, although they are of theoretical interest, so that a disproportionate number of statistical articles may get published in academic journals. This type of article often does not need a ‘client’ as such, as it may simply involve the application of some statistical method to secondary data derived from the literature. Conversely, qualitative approaches require a client as they cannot be used without interacting with human beings in some way. Therefore, it is less surprising that these methods report a comparatively high level of implementation.

When reflecting on the adopted methodology, a particular benefit of the approach was classifying the studies by more than one method that permitted co-associations to be explored. Likewise, there were benefits of allowing multiple functional areas that permitted examination of associations between functions and methods. A particular difficulty, however, was in constructing a viable taxonomy for all the methods. Eight categories were defined for this review but these are clearly open to debate. Having worked through the review process and resulting analyses, this is likely to assist in shaping future search criteria for bibliographic searches.

A key area of specific interest is the field of the so-called ‘grey literature’. It seems clear that many references to health-care modelling exist outside the domain of conventional journal publications. Commercial and promotional literature, website references and unpublished presentations, for instance, contain much of interest in this field. The challenge is to find a viable means of accessing and referencing these sources, which by definition are not recorded in conventional bibliographic databases. Despite this we believe that ‘grey literature’ may be centrally important in revealing lessons to be learned from the implementation of models in health care, an area that seems to be sorely absent in most of the research literature reviewed here.

In this review, the scope is limited to the specific area of OR type health-care modelling. This study begins to provide insights into the level of activity across many areas of application. It highlights important relationships and points to key areas of omission and neglect in the literature. Some of the key findings are summarised below:

  • The vast majority of studies were carried out in North America and Europe.

  • There is a preponderance of statistical approaches in the literature; however, simulation and qualitative modelling both currently appear to be enjoying a strong period of popularity, relative to earlier decades.

  • Qualitative methods are commonly used as a secondary method and often as a subsidiary to simulation.

  • Overall, an extraordinarily wide range of methods is revealed in the literature, and many of these methods are highly specific or bespoke to the project in question.

  • Simulation methods are prominent in planning and system/resource utilisation.

  • Statistical methods are prominent in the areas of finance, policy, governance and regulation.

  • In general there are few obvious strong associations and the data are highly varied.

  • Startlingly few studies report evidence of implementation, although a relatively large proportion do demonstrate a conceptualised model.

8. Conclusion

Clearly the literature in health-care simulation and modelling is vast and is expanding at a rapid rate. Moreover, this literature covers a very diverse range of applications with many interacting and overlapping areas. Added to this is the lack of standards and consistency in the use of key terms (for example, the use of the term ‘model’) between publications. The work of systematically reviewing and classifying the research literature in this area is therefore fraught with difficulties. Despite, and maybe because of this, there is great value in developing a viable taxonomy of the documented research. Such a framework provides a potential basis and structure for understanding the field as a whole.

The approach presented in this article provides a systematic and generic methodology that can be extended to review further areas of the literature as well as other types of information sources in health-care modelling and simulation. The field of Health Technology Assessment, for instance, is a fertile area of research in economic modeling, which could yield useful insights into the application of these techniques.

Given the multi-dimensional and relatively complex nature of this literature review, presentation is another important challenge. Here there is a role for visualisation tools (such as that presented by Citespace) to provide user-friendly, accessible means to graphically depict the key relationships in the analysis.