1. Introduction

The primary goal in multi-criteria decision making (MCDM) is to provide a set of criteria aggregation methodologies that enable the development of models considering the decision makers' (DMs') preferential system and judgment policy (Doumpos and Zopounidis, 2002). Achieving this goal requires the implementation of complex processes. While intuition and simple rules are still favourite decision-making methods, they may be dangerously inaccurate for complex decision problems. The model presented here can help DMs improve their decision quality when they are confronted with complex decision problems. Our model ensures consistency and completeness of the required information and synthesizes a vast amount of information using a manageable and easy to understand structure, while incorporating the use of human intuition and subjective analysis skills.

The analytical processes in our model help a DM decompose complex MCDM problems into manageable steps, making this model accessible to a wide variety of DMs and situations. We use analytic hierarchy process (AHP), the entropy method, and preference modelling to help DMs crystallize their thoughts and reduce the inconsistencies associated with MCDM. Although technical details of the model may be beyond the reach of some DMs, the basic concepts are not difficult to understand or implement. As such, the DMs can use available analytical tools and techniques with some assistance from the experts (Schoemaker and Russo, 1993).

The structured framework presented in this study has some obvious attractive features:

1. The generic nature of the model allows for the subjective evaluation of a finite number of decision alternatives on a finite number of performance criteria by a group of DMs. The mathematical and computational properties of the model are applicable to a wide range of real-world decision-making problems in MCDM.
2. The information requirements of the model are stratified into a hierarchy (criteria and their respective sub-criteria) that simplifies information input and allows the DMs to focus on a small area of the large problem. This process is also useful for seeking input from different experts or levels of management in the organization.
3. Inconsistencies are inevitable when dealing with subjective information from different DMs. The built-in inconsistency-checking mechanism of AHP helps to identify inconsistencies in judgments at very early stages of the computation process.

2. Problem description and background

The Integrated Human Exploration Mission Simulation Facility (INTEGRITY) project initiated at the Johnson Space Center (JSC) is expected to play an important role in increasing the success of analog missions. Analog missions are real-life, Earth-based science missions whose purpose is to help understand the operations, techniques, and technologies required to perform similar tasks during future human spaceflight missions. The goal of performing an analog mission is to prepare crewmembers and support teams and increase the productivity and scientific return during future space-based science missions in a low risk, low-cost environment. As nearly anyone with field experience can attest, even the most well-planned experiment usually does not go as expected—hardware does not work as desired, unforeseen logistical problems arise, planned procedures need to be modified to fit changing situations, etc. Theoretically, analog missions work many of these kinks out of the process before a similar mission is attempted in space. Although analog missions are able to perform these functions in a fashion that is considerably less costly than performing the missions in space, they still require budgets that are not insignificant.

Analog missions offer a unique opportunity to test hardware, operations, and tasks repeatability at a minimal cost and risk. Since these tests are performed in such a controllable environment, the opportunity arises to conduct numerous experiments that could help to answer several outstanding questions. For example, a time delay could be inserted easily into the communication systems that would help mission planners understand how to deal with this problem when actual crews are at a great distance from Earth. Another example of an item to be tested is the scheduling of a crew during an extended mission. Analog missions offer the opportunity to test various task schedules in order to help determine what reasonably can be expected of the crew over a given period of time. This information helps mission planners to set realistic goals and optimize the quality of the scientific return during actual missions. Analog missions also provide system designers with the opportunity to build, test, and evaluate their concepts. Once the system engineers and crews have been able to evaluate a given design, the feedback should flow back through the system engineer and their respective divisions. This information should then feedback into future design iterations made by the engineers, to create more robust systems and elements in subsequent designs.

The INTEGRITY Team (IT) is a 17-member team of experts and scientists from different divisions within the JSC. IT supports analog missions by providing human exploration mission vehicle, habitat, and planetary surface terrain simulation facilities and infrastructure. These simulation facilities are used to conduct integrated testing with human crews in support of advanced research and technology development efforts associated with future human exploration missions beyond low Earth orbit. Five simulators, capable of simulating the crew cabin architecture, integrated systems operations, and crew operations associated with a 1000-day class mission transit vehicle, will be housed within these simulation facilities. The IT is chartered with the task of assessing each of the following five simulators and developing an implementation plan that is in sync with JSC's mission, objectives and priorities.

Transit vehicle simulator (TVS): A transit vehicle for manned exploration missions would be used to transfer the mission crews between low Earth orbit, a gateway station such as the International Space Station, and the orbits for a mission destination such as an asteroid, the Moon or Mars. The TVS will be habitable and will attempt to accurately reflect appropriate mission scenarios and constraints by supporting a crew for 6–12 months.

Lander vehicle simulator (LVS): After the transit vehicle reaches orbit around a mission destination, the crew must descend to the surface by means of a landing vehicle. The lander vehicle must also provide ascent capability for rendezvous with and return to the transit vehicle in orbit. The lander vehicle simulator (LVS), a ground-based version of such a lander vehicle, will be habitable for short durations of several hours to several days.

Surface habitat simulator (SHS): A surface habitat on an extra-terrestrial surface must support and provide capabilities for a human crew to live and conduct an extra-terrestrial surface mission. It is likely that the duration of a surface mission will be in the range of 1 month to a year or more. The opportunities for Earth-based re-supply of a surface habitat will be limited, so the habitat must be largely self-supporting.

Roving vehicle simulator (RVS): Rovers offer several capabilities including surface translation for the crew, access to places considered too dangerous for manned exploration, maintenance of surface habitat, and scouting before surface operations by the crew. Exploration mission rovers consist of one or more surface-roving vehicles may be manned or unmanned and may have various levels of autonomy.

Surface terrain simulator (STS): Distinct from the other Integrity simulators, the STS does not involve designing and building a piece of technology or vehicle. Rather, it is a re-creation of an extra-terrestrial surface within the INTEGRITY facility. The surface simulation is expected to integrate and evaluate the technical abilities and operations of the lander (LVS), surface habitat (SHS), and rovers (RVS). It will consist of materials and features that are deliberately constructed to present mission simulations with realistic features of an extra-terrestrial surface.

Figure 1 shows how the analog mission vehicle, habitat, and surface terrain simulators will be utilized for a manned exploration mission to Mars. The diagram depicts outbound phases of the mission. Starting with mission phase 1, a crew would travel from Earth-orbit to a way station or Gateway in a Crew Transfer Vehicle. INTEGRITY would simulate this phase using the TVS. Continuing in a TVS, phase 2 would take the crew from the gateway station into orbit around Mars. From orbit, the crew would descend to the Martian surface in a lander vehicle (LVS) in phase 3. In phase 4, the crew would egress the lander vehicle and traverse the Martian surface (STS) to man the surface habitat (SHS) using a roving vehicle (RVS).

Figure 1.

Mars mission elements and phases.

Full figure and legend (92K)

3. Theoretical justifications

Roy (1990) argues that solving MCDM problems is not searching for some kind of optimal solution, but rather helping DMs master the (often complex) data involved in their problems and advance towards an acceptable solution. Multi-criteria analysis is not a ready-made recipe that can be applied to every situation. Methods should be chosen according to the value and quantity of information available, the nature of the problem, and the expectations of the DMs. As often happens in applied mathematics, the development of MCDM models is dictated by real-life problems. Therefore, it is not surprising that methods have appeared in a rather diffuse way, without any clear general methodology or basic theory (Vincke, 1992). The model presented here has evolved in an attempt to solve a complex MCDM problem at the JSC. It is a unique model with several features:

1. Traditionally, MCDM frameworks fall into three categories: the multi-objective value analysis (Keeney and Raiffa, 1976), the outranking method (Vincke, 1992), and the interactive methods (Vanderpooten and Vincke, 1989). The selection of a framework depends on the type of the problem, the type of the choices (continuous or discrete), the type of measurement scales, the type of importance weights, the type of dependency among the criteria, and the type of uncertainty (Vincke, 1992). We show how the integration of several mathematically sound techniques can reduce some of the difficulties in the selection of an appropriate framework. Rather than moulding the problem to fit into a framework, we integrate several techniques into a framework to address problem requirements.
2. Finding the 'best' MCDM framework is an elusive goal that may never be reached (Triantaphyllou, 2000). Pardalos and Hearn (2002) have argued that one of the major issues for future research in MCDM is to explore ways of combining criteria aggregation methodologies to enable the development of models that consider the DMs' preferential system in complex problems. Belton and Stewart (2002) also argue the need for integrating frameworks in MCDM. Our model has solved a complex and judgmental multi-criteria problem by suitably combining a set of well-known and proven techniques in MCDM. This integration allows for the objective data and subjective judgments to be used side-by-side in a mathematically sound decision model.
3. We have developed an alternative approach to traditional AHP problem structuring where hierarchies of decision criteria and alternatives are used to solve MCDM problems. Our framework integrates AHP preferences of decision criteria with the priority scores and entropy information. This structured framework aggregates the intuitive preferences of multiple DMs to assess the overall performance of alternative scenarios using a weighted sum model (Triantaphyllou, 2000).
4. The Entropy method is a commonly used method for calibrating the weights assigned to different decision criteria in MCDM (Hwang and Yoon, 1981; Zeleny, 1982). A criterion does not influence the final choice much when all the alternatives have a similar value for that criterion. The entropy concept suggests that if a criterion's values are the same, the criterion can be eliminated from further consideration. Alternately, the weight assigned to a criterion can be smaller if all the alternatives have similar values for a criterion. On the other hand, when the differences between a criterion's values across particular alternatives are greater, the criterion is viewed as more important. The entropy concept has been shown to be particularly useful to investigate contrasts between sets of data.

4. The procedure and the model

The MCDM model presented in this study was used by the IT to assess the importance of each INTEGRITY simulator. Schoemaker and Russo (1993) describe four general approaches to decision-making ranging from intuitive to highly analytical. These methods include intuitive judgments, rules and shortcuts, importance weighting, and value analysis. They argue that analytical methods such as importance weighting and value analysis are more complex but also more accurate than the intuitive approaches (Schoemaker and Russo, 1993). Our approach is a simple and yet sophisticated multi-objective value analysis model that attempts to uncover some of the complexities inherent in the evaluation. The proposed approach uses a series of intuitive and analytical methods to visually display an alternative's preference to the ideal and nadir alternatives as well as all other competing alternatives. The highest achievable scores with all currently considered criteria form a composite, an ideal alternative (Zeleny, 1982). Similarly, we form a nadir alternative with the lowest achievable scores. Using the preference scores and the theory of displaced ideal to grasp the extent of the emerging conflict between means and ends, the IT members explore the limits attainable with each criterion. An implementation plan was developed based on the group preference rankings provided by the model. A four-phase process as presented in Figure 2 was used by the IT for this prioritization:

Figure 2.

The overall procedure.

Full figure and legend (91K)

Phase 1. The relevant criteria and sub-criteria and their importance weights: Each IT member was asked to develop a list of all possible criteria relevant to the simulator evaluation. All the individual responses were collated by the facilitator into a comprehensive list which was shared with the IT at subsequent meetings. After several meetings and lengthy discussions, the list was revised by the IT into the hierarchical set of criteria and sub-criteria presented in Table 1.

Table 1 - Simulator ranking criteria and sub-criteria.

Full table

The discussions about criteria and sub-criteria raised some concerns about the natural sequencing that may or may not exist among the simulators. The facilitator developed a questionnaire for the IT to investigate this question further. The IT was asked in this questionnaire to offer their sequencing perception by providing a ranking between 1 and 5 to each simulator. They were instructed to give a ranking of 1 to the simulator that they believed had to be built first, 2 to the second simulator, etc. They were also instructed to use an average ranking for cases where they believed two or more simulators had to be built simultaneously. For example, a score of 2.5 was used for simulators B and C, if it was believed that both of them had to be built simultaneously after A. Table 2 presents the mean sequencing scores of the five simulators for the each IT member along with the median scores of the simulators. We performed Kruskal–Wallis test to see if there is a significant difference among the medians. The high P-value of 0.804 (relative to =0.1) shows that there is no statistical difference between the medians of the five simulators. In addition, because all z-values are considerably less than 1.96 (=0.05), there is no difference for each individual simulator from the overall median.

Table 2 - Simulator sequencing scores.

Full table

4.1. The analytic hierarchy process

AHP was used to develop a set of importance weights for the criteria and sub-criteria. The IT identified i criteria and j sub-criteria to be used as evaluation criteria in simulator evaluation. The importance weight of each criterion (w_i) and sub-criterion (w_ij) was captured and measured with AHP using the questionnaire presented in Appendix A. The IT members were asked to provide their subjective assessment of each pairwise comparison. Assuming that an IT member believes, c₁, c₂, ..., c_i are the i criteria that contribute to the importance of a simulator, the team member's goal is to assess the relative importance of these factors.

Saaty's AHP (Saaty and Vargas, 1998; Forman and Gass, 2001) is a method of deriving a set of weights to be associated with each of the i criteria or ij sub-criteria. Initially, the team member is asked to compare each possible pair c_j, c_k of criteria and provide judgments about which criteria are more important and by how much. AHP quantifies these judgments and represents them in an i i matrix:

• If c_j is judged to be of equal importance as c_k, then a_jk=1
• If c_j is judged to be more important than c_k, then a_jk>1
• If c_j is judged to be less important than c_k, then a_jk<1

Since the entry a_jk is the inverse of the entry a_kj, the matrix A is a reciprocal matrix. a_jk reflects the relative importance of c_j compared with criterion c_k. For example, a₁₂=1.25 indicates that c₁ is 1.25 times as important as c₂.

Then, the vector w representing the relative weights of each of the i criteria can be found by computing the normalized eigenvector corresponding to the maximum eigenvalue of matrix A. An eigenvalue of A is defined as which satisfies the following matrix equation:

where is a constant, called the eigenvalue, associated with the given eigenvector w. Saaty (1977, 1980, 1983, 1989, 1990 and 1994) has shown that the best estimate of w is the one associated with the maximum eigenvalue (_max) of the matrix A. Since the sum of the weights should be equal to 1.00, the normalized eigenvector is used. Saaty's algorithm for obtaining this w is incorporated in the software Expert Choice (Expert Choice, 2000).

One of the advantages of AHP is that it encourages team members to be consistent in their pairwise comparisons. Saaty suggests a measure of consistency for the pairwise comparisons. When the judgments are perfectly consistent, the maximum eigenvalue, _max, should equal i, the number of criteria that are compared. In general, the responses are not perfectly consistent, and _max is greater than n. The larger the _max, the greater is the degree of inconsistency. Saaty defines the consistency index (CI) as (_max-i)/(i-1), and provides the following random index (RI) table for matrices of order 3–10:

This RI is based on a simulation of a large number of randomly generated weights. Saaty recommends the calculation of a consistency ratio (CR), which is the ratio of CI to the RI for the same order matrix. A CR of 0.10 or less is considered acceptable. When the CR is unacceptable, the team member is made aware that his or her pairwise comparisons are logically inconsistent, and he or she is encouraged to revise them.

The responses were processed with Expert Choice (Expert Choice, 2000) and those with inconsistency ratios greater than 0.10 were asked to reconsider their judgments as suggested by Saaty. The mean importance weights were calculated for the IT after the necessary adjustments were made to inconsistent responses. Each IT member was presented with his/her individual score along with the group mean weights. IT members were given the opportunity to revisit their judgments and make revisions to their pairwise comparison scores based on this feedback. Some IT members took advantage of this opportunity and revised their judgments in the second round. The mean importance weights for the first and second round are presented in Table 3. While the second round results differ slightly from the first round results, the inconsistency was improved significantly. A similar approach was used to determine the relative importance of each sub-criterion.

Table 3 - Rounds 1 and 2 importance weights of the criteria and sub-criteria.

Full table

There has been some criticism of AHP in the operations research literature. Harker and Vargas (1987) show that AHP does have an axiomatic foundation, the cardinal measurement of preferences is fully represented by the eigenvector method, and the principles of hierarchical composition and rank reversal are valid. On the other hand, Dyer (1990a) has questioned the theoretical basis underlying AHP and argues that it can lead to preference reversals based on the alternative set being analysed. In response, Saaty (1990) explains how rank reversal is a positive feature when new reference points are introduced. We use the geometric aggregation rule to avoid the controversies associated with rank reversal (Dyer, 1990a, 1990b; Saaty, 1990; Harker and Vargas, 1990).

Phase 2. The relative priority of the simulators in terms of the sub-criteria: The priority scores of the five simulators for all sub-criteria were needed to develop the preference scores in our model. The Simulator Assessment Questionnaire presented in Appendix B was given to the IT to capture their perception of the relative importance of each simulator for each sub-criteria using a 10-point Lickert scale. The individual priority scores were averaged into the group priority scores given in Table 4.

Table 4 - The IT mean priority scores.

Full table

Next, we used the group priority scores presented in Table 4 to revise the initial weights of the sub-criteria (w_ij'—second-round weights in Table 3) using the entropy concept. The essential idea in entropy method is that the relative importance of a criterion is directly related to the information conveyed by the criterion relative to the set of alternatives under consideration. This means that the greater the dispersion in the evaluations of the alternatives for a given criterion, the more important the criterion. In other words, the most important criteria are those which have the greatest discriminating power between alternatives. In this method the importance weight of criteria could be determined without the direct involvement of the DM, in terms of the values derived from the evaluation of the alternatives. However, this is a complete contradiction to the notion that weights should represent the relative importance the DM attaches to the criteria. Therefore, we multiply the values of weights obtained by the entropy method (intrinsic weights) by the subjective weights representing the judgments of DMs obtained by the AHP. The final result, once normalized, is used in our model to represent the importance weight of the criteria and sub-criteria.

Each sub-criterion is an information source; therefore, the more information a sub-criterion reveals, the more relevant it is. Consequently, the more information the jth sub-criterion of the ith criterion reveals, the more relevant the sub-criterion is to the decision analysis. Zeleny (1982) argues that this intrinsic information must be used in parallel with the initial weights the DMs assigned to various sub-criteria. In other words, the overall importance weight of a criterion, w_ij, is directly related to the intrinsic weight, w"_ij, reflecting the average intrinsic information developed by the priority scores of the simulators, and the subjective weight, w_ij', reflecting the IT members' subjective assessment of its importance.

The more different the scores of a sub-criterion are for a set of simulators, the larger is the contrast intensity of the sub-criterion, and the greater is the amount of information transmitted by that sub-criterion. Assuming that the vector z_ij=(z_ij¹, ..., z_ij^q) characterizes the ith criterion, the jth sub-criterion, and the qth simulator; the entropy measure of the jth sub-criterion for the ith criterion is

where

and K>0, ln is the natural logarithm, 0z_ij^k1, and e(z_ij)0. When all z_ij^k are equal for a given i and j, then z_ij^k/Z_ij=1/q, and e(z_ij) assumes its maximum value, which is e_max=ln q. By setting K=1/e_max, we achieve 0e(z_ij)1. This normalization is necessary for meaningful comparisons. In addition, the total entropy is defined as

The smaller e(z_ij) is, the more information the jth sub-criterion transmits for the ith criterion, and the larger e(z_ij) is, the less information it transmits. When e(z_ij)=e_max=ln q, the jth sub-criterion for the ith criterion transmits no useful information. Next, the intrinsic weight is calculated as

where I is the total number of sub-criteria for a particular criteria under consideration.

Since w_ij" is inversely related to e(z_ij), 1-e(z_ij) is used instead and normalized to make sure 0w_ij"1 and .

The higher e(z_ij), the less information content is provided by the jth sub-criterion for the ith criterion. When the information content of the jth sub-criterion for the ith criterion is low, the corresponding intrinsic weight (w_ij") should be low. Thus, the intrinsic weight is assumed to be inversely related to the entropy and therefore, we use 1-e(z_ij) in the definition of the intrinsic weight.

The more different the priority scores z_ij^k are, the larger w"_ij is and the more important the jth sub-criterion for the ith criterion is. When all the priority scores z_ij^k are equal for the jth sub-criterion for the ith criterion, then w_ij"=0 for that sub-criterion. However, this is not true if the priority scores z_ij^k are equal for all the sub-criteria j. In that case, the weights are assumed to be equal or w_ij"=1/n, where n is the number of sub-criteria for a given criterion. Entropy multiplies the intrinsic weight w_ij" by the subjective weight w_ij' and normalizes the product to calculate the overall importance weight of the jth sub-criterion for the ith criterion w_ij,

When there is more than one priority score (n-ary sub-criteria), these priority scores are used to calculate the entropy within each simulator. These within-simulator intrinsic weights can influence the overall weight of the sub-criteria. In other words, the overall importance weight for an n-ary sub-criterion w_ij is related to its between-simulator intrinsic weight w_ij" the subjective weight w_ij' and the within-simulator intrinsic weight. Table 5 presents the initial weights, the intrinsic weights and the revised weights of each sub-criterion.

Table 5 - The initial, intrinsic, and revised importance weights of sub-criteria.

Full table

There are two other methods for calculating the intrinsic weights of criteria. Diakoulaki et al (2000) proposes a method based on the correlation between the columns of the decision matrix. Another method consists in measuring the importance of each criterion as a member of a coalition by means of the Shapley value (Grabisch and Roubens, 1999). We use the entropy method suggested by Zeleny (1982, Chapter 7) in this study because it is readily available in MCDM, provides consistent results, and easily accepted by DMs (Pomerol and Barba-Romero, 2000, Chapter 4).

Phase 3. The individual and group preference matrices: The next model is used to identify a preference matrix for each IT member and an overall preference matrix for the IT as a group. To formulate the model algebraically, let us assume, w_i is the weight of the ith criterion (i=1, 2, ..., m), w_ij the weight of the jth sub-criterion of the ith criterion (i=1, 2, ..., m and j=1, 2, ..., n) and z_ij^k the priority score of the kth alternative for the jth sub-criterion of the ith criterion (i=1, 2, ..., m; j=1, 2, ..., n; and k=1, 2, ..., q).

Assuming that a^k is the kth alternative and z_ij^k represents the priority score of the kth alternative for the jth sub-criterion of the ith criterion:

We identify the Ideal Alternative (a^*) whose priority scores are (z₁₁^*, ..., z_mn^*), where z_ij^*=max z_ij^k (k=1, 2, ..., q).

Similarly, we identify the Nadir Alternative (a) whose priority scores are (z₁₁, ..., z_mn), where z_ij=min z_ij^k (k=1, 2, ..., q).

Next, Z is revised by adding the Ideal and Nadir alternatives:

Then, the Preference Matrix showing the pairwise preference strength of one alternative compared with another P(a^c,a^d) is constructed using the following equation:

Note that P(a^*,a)=+1.000 is the maximum possible strength and the preference of a_c compared with a_d is a percentage of this maximum possible strength. In addition, the number of pairwise comparisons needed to complete the Preference Matrix in our model is (n+2)(n+1)/2 because P(a^c,a^d)=-P(a^d,a^c) and when comparing one element with itself, the comparison must give no preference (0).

We used pairwise comparisons to measure the relative importance of one alternative to another. Using the importance weights and the priority scores, a preference matrix similar to the one presented in Table 6 was developed for each IT member. The preference matrices of the individual IT members were averaged to obtain the final group preference matrix presented in Table 7. The ideal and nadir alternatives are added to the list of the alternatives to show the relative relationship of each alternative to the ideal and nadir. The formula for the preference between two alternatives is normalized so that the preference of the ideal alternative compared to the nadir is the greatest and equals 1.

Table 6 - Individual A's preference matrix.

Full table

Table 7 - Group preference matrix.

Full table

Phase 4. The final rankings of the simulators for the IT: The preference scores show the relative distance of one simulator from the ideal, nadir, and all other simulators. For example, in Table 7, LVS is more preferred to nadir with a relative distance of 0.622 while it is less preferred to ideal with a relative distance of 0.378 (given the 0–1 distance of nadir from ideal). Similarly, we can compare LVS with any other simulator such as RVS. LVS is preferred to RVS with a relative distance of 0.174. The pairwise preferences provided in Table 7 can be translated into the following rankings and relative distances:

Next, by reconstructing the above rankings and distances for all individual IT members, we performed a one-sample t-test to see if the differences of the means are equal to zero (see Table 8). The t-test, with =0.05, showed that Ideal–SHS, SHS–LVS, LVS–RVS, and TVS–Nadir were significantly different while there was no difference between RVS–STS and STS–TVS. We concluded that, with =0.05, the relative distance of 0.010 between RVS and STS and the relative distance of 0.079 between STS and TVS are not different. We shared this result with the IT. While the difference between STS and TVS was not statistically significant at the =0.05 confidence level, the team felt that there was a noticeable difference that was significant at =0.10. The IT finalized its recommendation and agreed on the following final consensus rankings of the simulators: SHS>LVS>RVS=STS>TVS. The decision was made to implement SHS first, LVS second, both RVS and STS together next, and TVS last.

Table 8 - Individual and group relative distances.

Full table

5. Evaluation

Prior to implementation of our model, the IT used simple voting to solve group decision problems. As a result of the lack of a systematic process and inadequate data gathering and analysis, the team was concerned about the potential for poor decision-making. Their desire to replace the voting method with a more complete, structured, and analytical process resulted in the creation of the model.

The proposed model was evaluated by the IT based on seven factors: decision quality, decision process quality, analytical capabilities, completeness, structured, flexibility, and simplicity. The questionnaire presented in Appendix C, was used to compare the POST-INTEGRITY framework with the PRE-INTEGRITY voting method. The results presented in Figure 3 shows that POST-INTEGRITY scores are higher on all of the assessment dimensions with the exception of simplicity. The PRE-INTEGRITY method was slightly preferred over the POST-INTEGRITY system in terms of simplicity.

Figure 3.

POST-INTEGRITY and PRE-INTEGRITY median scores.

Full figure and legend (32K)

Next, we performed Wilcoxon signed ranks test of medians. While computation of a median is easily justified for ordinal data, some statisticians have reservations about computing a mean for ordinal data. Therefore, we used median as the measure of central tendency. All the medians were statistically different from 0 for =0.05. In summary, the medians were statistically different from 0 for all evaluation criteria. In addition, the medians showed that POST-INTEGRITY was favoured over PRE-INTEGRITY for all evaluation criteria with the exception of simplicity. For simplicity, PRE-INTEGRITY was marginally favoured over POST-INTEGRITY (see Table 9).

Table 9 - POST-INTEGRITY test of medians.

Full table

6. Conclusion

Advances in computer technology and the availability of data have made MCDM more complex than ever. Schoemaker and Russo (1993) argue that as the significance and the complexity of a decision problem increases, so does the importance of the solution quality. While practicing mangers may favour simple approaches, they can be dangerously inaccurate for complex decision problems. Our model helps DMs (i) decompose a complex problem into manageable steps, (ii) ensure the consistency and completeness of the information, and (iii) synthesize the results through a series of logically sound techniques. This decomposition and synthesis is not intended to replace DMs; rather, it provides a systematic approach to support and supplement their judgments. Our model has several attractive features that address some of the limitations inherent in the previous voting method used by the IT:

1. Structured: The model stratifies the information requirements into a hierarchy that simplifies information input and help DMs focus on a small area of the large problem. This process can also help by dividing the problem into different levels of detail for purposes of seeking input from different experts in the organization.
2. Complete: The model processes a wide range of importance weights and preferences concerning multiple criteria, multiple alternatives, and multiple DMs.
3. Analytical: The value-analysis model utilized here is considered a MCDM approach.
4. Simple: The model has a simple to understand and yet theoretically sound scoring system.
5. Flexible: Our model does not limit the number of criteria, alternatives, or DMs.
6. Inconsistency: Inconsistencies are inevitable while dealing with subjective information from different DMs. The built-in inconsistency-checking mechanism of AHP helps in identifying inconsistencies in judgments at very early stages of the computation process.
7. Sensitivity analysis: The most important potential benefit of our model is its usefulness in examining how sensitive the results are to changes in the selected parameters.

It should be noted that using the step-by-step and structured approach presented here does not imply a deterministic approach to problem solving. While our model has enabled the IT members crystallize their thoughts and organize their judgments, it should be used very carefully. Team member judgment is an integral component of the model. While these judgments often mirror a team member's belief, they should be used with caution. As with any MCDM, the practicing managers must be aware of the limitations of subjective estimates. When empirical analysis is feasible and makes economic sense, it should be utilized to improve these estimates (Lodish, 1982).

References

1. Belton V and Stewart TJ (2002). Multiple Criteria Decision Analysis: An Integrated Approach. Kluwer Academic Publishers: Boston.
2. Diakoulaki D, Mavrotas G and Papayannakis L (2000). Objective weights of criteria for interfirm comparisons. Journées du groupe européen Aide Multicritére à la Décision 36^e, Luxembourg.
3. Doumpos M and Zopounidis C (2002). Multicriteria Decision Aid Classification Methods. Kluwer Academic Publishers: Boston.
4. Dyer JS (1990a). Remarks on the analytic hierarchy process. Mngt Sci 36: 249–258.
5. Dyer JS (1990b). A clarification of 'Remarks on the analytic hierarchy process'. Mngt Sci 36: 274–275.
6. Expert Choice (2000). Computer software. Decision Support Software: McLean.
7. Forman EH and Gass SI (2001). The analytic hierarchy process—an exposition. Opns Res 49: 469–486.
8. Grabisch M and Roubens M (1999). An axiomatic approach to the concept of interaction among players in cooperative games. Int J Game Theory 28: 547–565. | Article |
9. Harker PT and Vargas LG (1987). The theory of ratio scale estimation: Saaty's analytic hierarchy process. Mngt Sci 33: 1383–1403.
10. Harker PT and Vargas LG (1990). Reply to 'Remarks on the analytic hierarchy process' by JS Dyer. Mngt Sci 36: 269–273.
11. Hwang LC and Yoon K (1981). Multi Attribute Decision-Making: A Methods and Applications. Springer-Verlag: Berlin.
12. Keeney RL and Raiffa H (1976). Decisions with Multiple Objectives: Preference and Value Tradeoffs. Wiley: New York.
13. Lodish LM (1982). Experience with decision calculus models and decision support systems. In: Schulz R and Zoltners A (eds). Marketing Decision Models. North-Holland: New York.
14. Pardalos PM and Hearn D (2002). Multicriteria Decision Aid Classification Methods. Kluwer Academic Publishers: Boston.
15. Pomerol J-C and Barba-Romero S (2000). Multicriterion Decision in Management: Principles and Practice. Kluwer Academic Publishers: Boston.
16. Roy B (1990). Decision-aid and decision making. Eur J Opl Res 45: 324–331. | Article |
17. Saaty TL (1977). A scaling method for priorities in hierarchical structures. J Math Psychol 15: 234–281. | Article |
18. Saaty TL (1980). The Analytic Hierarchy Process. McGraw-Hill: New York.
19. Saaty TL (1983). Axiomatic foundations of the analytic hierarchy process. Mngt Sci 32: 841–855.
20. Saaty TL (1989). Decision making, scaling, and number crunching. Decision Sci 20: 404–409.
21. Saaty TL (1990). An exposition of the AHP in reply to the paper 'Remarks on the analytic hierarchy process'. Mngt Sci 36: 259–268.
22. Saaty TL (1994). How to make a decision: The analytic hierarchy process. Interfaces 24: 19–43.
23. Saaty TL and Vargas LG (1998). Diagnosis with dependent symptoms: Bayes theorem and the analytic hierarchy process. Opns Res 46: 491–502.
24. Schoemaker PJH and Russo JE (1993). A pyramid of decision approaches. California Mngt Rev 36: 9–31.
25. Triantaphyllou E (2000). Multi-criteria Decision Making Methods: A Comparative Study. Kluwer Academic Publishers: Boston.
26. Vanderpooten D and Vincke P (1989). Description and analysis of some representative interactive multicriteria procedures. Math Computer Modeling 12: 1221–1238. | Article |
27. Vincke P (1992). Multicriteria Decision Aid. Wiley: New York.
28. Zeleny M (1982). Multiple Criteria Decision Making. McGraw-Hill: New York.

Appendices

Appendix A. Simulator Criteria Weight Assessment Questionnaire

INSTRUCTIONS: Please respond by placing an 'X' on the appropriate location to indicate your perception of the relative importance of the INTEGRITY criteria for each of the following pairwise comparisons. Table A1

Table A1.

Full table (72K)

INSTRUCTIONS: Please respond by placing an 'X' on the appropriate location to indicate your perception of the relative importance of the COST Sub-criteria for each of the following pairwise comparisons. Table A2

Table A2.

Full table (103K)

INSTRUCTIONS: Please respond by placing an 'X' on the appropriate location to indicate your perception of the relative importance of the STRATEGIC PLAN RELEVANCE Sub-criteria for each of the following pairwise comparisons. Table A3

Table A3.

Full table (170K)

INSTRUCTIONS: Please respond by placing an 'X' on the appropriate location to indicate your perception of the relative importance of the COMPLEXITY Sub-criteria for each of the following pairwise comparisons. Table A4

Table A4.

Full table (87K)

INSTRUCTIONS: Please respond by placing an 'X' on the appropriate location to indicate your perception of the relative importance of the SCIENCE, OPERATIONS, AND TECHNOLOGY Sub-criteria for each of the following pairwise comparisons. Table A5

Table A5.

Full table (163K)

Appendix B. Simulator Assessment Questionnaire (Abbreviated Version)

INTEGRITY is designed to provide the building facilities and infrastructure required to house and operate all analog mission vehicle, habitat, and surface terrain simulators and to manage and coordinate all test mission operations associated with these simulators. INTEGRITY planners have identified five simulators, all capable of simulating the crew cabin architecture, integrated systems operations, and crew operations associated with a 1000-day class mission transit vehicle:

• Transit Vehicle Simulator (TVS)
• Lander Vehicle Simulator (LVS)
• Surface Habitat Simulator (SHS)
• Roving Vehicle Simulator (RVS)
• Surface Terrain Simulator (STS)

The purpose of this questionnaire is to capture your perception of the sub-criteria scores for the simulators using a 10-point Lickert Scale (1=Very Bad...10=Very Good). For example, when evaluating 'Cost to Design', note that lower cost is preferred to higher cost. If you think a simulator is very costly (bad), provide a low (bad) score and if you think the simulator is not too costly (good), provide a high (good) score. However, when evaluating 'Publicity Value', it should be noted that higher publicity value is preferred to lower value. Therefore, if you think a simulator has a great publicity value (good), provide a high (good) score and if you think it provides little value (bad), provide a low (bad) score. Table B1

Table B1.

Full table (72K)

Appendix C. INTEGRITY System Evaluation Questionnaire

INSTRUCTIONS: The following criteria are adopted by the INEGRITY Team to evaluate the simulator assessment system. Please respond by placing an 'X' on the appropriate location to indicate your preference for the POST-System or PRE-System based on Flexibility, Simplicity, Analytical Capabilities, Completeness, Structured Framework, Decision Process Quality, and Decision Quality. Table C1

Table C1.

Full table (43K)

Acknowledgements

This research was supported by NASA Grant number NAG 9-1526. I am grateful to Jack Knight, Steven Gonzales, Anthony Bruins, Terry Tri, Gretchen Thomas, John Park, Kevin Wells, Lac Nguyen, and the INTEGRITY Team, all from the Johnson Space Center, and my colleagues Dennis Kennedy and Jack Rappaport for their assistance with this research project.