Using multiattribute utility theory to avoid bad outcomes by focusing on the best systems in ranking and selection

Abstract

When making decisions under uncertainty, it seems natural to use constraints on performance to avoid the selection of a particularly bad system. However that intuition has been shown to impair good recommendations as demonstrated by some well-known results in the stochastic optimization literature. Our work on multiattribute ranking and selection procedures demonstrates that Pareto and constraint-based approaches could be used as part of a successful decision process; but a tradeoff-based approach, like multiattribute utility theory, is required to identify the true best system in all but a few special cases. We show that there is no guaranteed strategic equivalence between utility theory and constraint-based approaches when constraints on the means of the performance measures are used in the latter. Hence, a choice must be made as to which is appropriate. In this paper, we extend well-known results in the decision analysis literature to ranking and selection.

Appendix A

A.1. Examples of the issues associated with using constraints to capture preferences

Example of Blau’s dilemma: the negative value of perfect information

Suppose we are choosing between two configurations 1 and 2 based on two performance criteria X_m1 and X_m2 and we have the following information about the distributions of attribute performance.

illustration

Further assume our objective is to select the configuration based on the following rule:

In other words, if expectation of X_m2 for configuration m is not satisfied, E[X_m2]>70, the value to the decision maker for configuration m, f(x_m1, E[X_m2])=0; otherwise, f(x_m1, E[X_m2])=x_m1. As shown in Figure A1, the decision maker should choose Configuration 1: both configurations satisfy the constraint on the mean of X_m2 and so we choose the larger x_m1 associated with Configuration 1 and the decision maker receives a value of, f(x_m1, E[X_m2])=f(8, 70)=8. Note that although both options fail to satisfy the constraint on X₂ half of the time, there is no penalty because E[X_m2]⩽70 for both.

Figure A1

Best alternative when performance is defined based on f(x_m1, E[X_m2]).

Figure A2 shows the adjusted decision tree when the decision maker receives perfect information concerning the value of X_m2, for Configuration 1, x₁₂. Now, when she learns that x₁₂=90 she chooses Configuration 2 for a value of 3, but when she learns that x₁₂=50 she chooses Configuration 2 for a value of 8. The expected value is 0.5(3)+0.5(8)=5.5. Learning the precise value of x₁₂ results in a loss of 5.5−8=−2.5<0 in expected value when compared with the case with no information. This negative EVPI is commonly referred to as Blau’s Dilemma (Blau, 1974).

Figure A2

Best alternative when performance is defined based on f(x_m1, E[X_m2]) with Perfect Information about x₁₂.

The remedy for Blau’s dilemma is simple: evaluate the configurations based on the actual realization of X_m2, x_m2, rather than E[X_m2]. This change incorporates the fact that, in this example, half of the time each alternative fails to satisfy the constraint, effectively including the constraint as a penalty in the objective function as discussed in Section 3 and above. In other words, we introduce g(x_m1, x_m2)=x_m1 if x_m2⩽70; g(x_m1, x_m2)=0, otherwise. Decisions are then made based on E[g(x_m1, x_m2)] rather than f(x_m1, E[X_m2]); an expected utility function is one of many ways to implement the proper scoring procedure.

As shown in Figure A3, the decision maker whose preferences satisfy this specification still prefers Configuration 1 to Configuration 2, but the expected value of what is received is half of that in Figure EC1 (4 versus 8), due to the infeasibility of x₁₂ half of the time (note that the potential infeasibility of x₂₂ also reduces the value of Configuration 2 by half even though it is not selected).

Figure A3

Best alternative when performance is defined based on E[g(x_m1, x_m2)].

Now when the decision maker is presented with perfect information about the realization of x₁₂ in Figure A4, she chooses Configuration 2 when x₁₂=90 and Configuration 1 when x₁₂=50, for an expected value of 0.5(3)+0.5(8)=4.75. However, note that EVPI=4.75−4=0.75>0=EV and Blau’s dilemma has been resolved. The key feature of evaluations that feature non-negative EVPI is that they reflect the entire distribution of attribute performance and allow tradeoffs or penalties for outcomes that fail to achieve the desired performance levels. A utility function is just one way to capture the proper scoring of the alternatives.

Figure A4

Best alternative when performance is defined based on E[g(x_m1, x_m2)] with Perfect Information about x₁₂.

The previous example provides evidence of the problems associated with putting a constraint on the mean of a performance measure rather than allowing the distribution of performance to determine the value to the decision maker. Some approaches advocate a form of chance constrained programming (CCP) that puts a constraint on the probability that some condition occurs. This formulation also suffers from a negative value of perfect information. Suppose we are choosing between two alternative configurations, Configuration 1 and Configuration 2, based on two criteria that both follow normal distributions: Calls (X_m1) and Accidents (X_m2) for configuration m. Further we are interested in the following CCP formulation which maximizes calls as long as the probability of more than 2.00 accidents is less the 5%:

Further, assume Cfg 1 and Cfg 2 have the following characteristics.

illustration

Since Pr(X₁₂⩾2.00)⩽0.05 and Pr(X₂₂⩾2.00)⩽0.05, both Configuration 1 and Configuration 2 meet the constraint and we choose Configuration 1, E[X₁₁]= 20>E[X₂₁]= 18. We will refer to this as the expected value of the optimization problem without any additional information, or EV=20.

If we receive perfect information about X₁₂ then we would choose Configuration 1 when X₁₂⩽2.00, Pr(X₁₂⩾2.00)=0.00, which happens with probability 0.9772; with probability 0.0228 we learn that X₁₂>2.00, Pr(X₁₂⩾2.00)=1.00, and select Configuration 2. Hence the expected value with perfect information about X₁₂, EVPI(X₁₂)=0.0228 E[X₂₁]+0.9772 E[X₁₁]=0.0228(18)+0.9772(20)=19.9545. Now we have EVPI(X₁₂)−EV=19.9545−20=−0.0455<0 and the symptom of Blau’s dilemma surfaces again: we are worse off with the information.

In contrast, we could argue that when the constraint on X_m2 is satisfied (not satisfied), u(X_m2)=1 (0), or u(X_m2)=0 × Pr(X_m2>2.00)+1 × Pr(X_m2⩽2.00)=Pr(X_m2⩽2.00) and assume that u(X_m1)=E[X_m1]/20 without loss of generality. Further, let u(X_m1, X_m2)=0.5 u(X_m1)+0.5 u(X_m2).

If we maximize u(X_m1, X_m2) with no additional information

illustration

We would choose Configuration 1: u(X₁₁, X₁₂)=0.9886=EV>u(X₂₁, X₂₂)=0.9386

If we are able to gather perfect information about X₁₂, then (as shown in Figure A5)

Figure A5

Best alternative when performance is defined based on E[u(X_m1, X_m2)] with Perfect Information about X₁₂.

EVPI(X₁₂)=0.9986 and EVPI(X₁₂)−EV=0.9986−0.9886=0.0010>0. We are better off with the information and Blau’s dilemma has been resolved.

In both of these examples the root cause of the problem is that the constrained variable is not part of the objective function as we discuss in detail in Section 2. Featuring both criteria in the objective function allows for a tradeoff between them. For example, if one wants to focus exclusively on satisfying the constraint on X_m2 then w₂=1. In this extreme case, the alternatives would be sorted in increasing order of satisfying the constraint. The probability thresholds for the CCP constraints play an analogous role to the weights in an MAU model. These thresholds would have to be assessed and it is not clear how one would do that while there is a long literature on weight assessment in muliattribute utility theory.

Further, consider an extreme case such as Pr(X_m2⩾2.00)⩽0.000001. It is likely that there will no systems that satisfy this constraint and, similar to the setting in Section 2, the decision maker will have to increase the right hand side until one or more systems become feasible. Again, it is not clear that these are really hard constraints at all. In fact Hogan et al (1981) remind us statements like X_m2⩾2.00 are goals and Pr(X_m2⩾2.00)⩽0.000001 is a probability of attaining that goal. Further, note that when using multiattribute utility theory, even if no systems satisfy the desired target of X_m2⩽2.00 the decision maker gets feedback on the performance of every systems in addition to the other benefits discussed in Section 2.

Aside from being a reasonable property, positive EVPI is important when making decisions about what kind of information to gather. For example, suppose we are considering doing some additional testing on factors that could affect our estimates of the distribution of the number of accidents, X_m2, e.g. the effect of adding additional escorts for vessels. Or perhaps we are considering allocating scarce CPU cycles to generate additional replications to reduce the standard error of our performance estimates of X_m2. Given the CSO framework we could conclude that there is no need to gather any more information for X _m2 ; in fact we are better off without the additional information. Put another way, a negative EVPI is really a symptom of a more fundamental problem: a lack of tradeoffs in the objective function. Further, given that MAU is no more computationally expensive that other approaches when analyzing the same problem, it is unclear what benefits accrue from the use of other approaches. At first glance one might assume that there are more required assessments with MAU, however, as we argue here and in the paper, when you think carefully about other approaches there are other unknown parameters that must be estimated. For example, with CCP you must estimate the probability threshold for each goal and with constraint based optimization, what are the constraint levels in CSO? If we use a ‘soft’ constraint we’re back to assessing a utility function, etc.