Introduction

Physicians are constantly faced with interpreting the results of diagnostic tests. In a medical treatment decision, variations in patient physiology, heterogeneity of results across different clinical studies, imperfect expert knowledge of critical quantities and unknown or uncertain causal pathways and mechanisms all contribute to overall uncertainty. Bayesian modelling techniques are well suited for informing medical decisions under uncertainty (Dowie, 1996)

In this paper, we explore staging decisions for breast cancer patients presenting for radiation treatment (RT) after a surgical procedure to remove the tumour. Staging is the process of identifying characteristics of the cancer that are subsequently used to inform treatment decisions. Pathologic examinations of tissue from the patient for the presence of cancer cells (ie pathology reviews) are used to assess the stage and grade of cancer. Patients who have symptoms of metastasis (ie spread of cancer to distant tissues) are further assessed using appropriate imaging and biochemical tests. Uncertainty in staging can influence the particular RT technique employed.

We designed a Bayesian Monte Carlo simulation model reflecting the typical process of assessing patients for RT after surgery at Canadian cancer centres such as the Tom Baker Cancer Centre (TBCC) in Calgary, Alberta. Our simulation followed a cohort of hypothetical patients for the true stage and required RT technique were defined a priori (ie the reference data). The model then simulated the test and pathology results that would result in the 'real world', assuming evidence-based guidelines and given the uncertainties associated with the diagnostic tests, the pathology review process, and critical system variables. A comparison of the reference data with the simulated data allowed us to inform the question: 'What proportion of patients will be correctly staged and treated, given that the physicians employ evidence-based practices in their field?' The simulation model allowed us to imitate the real-world process, and provides better understanding of the process. To our knowledge, this is the first analysis of its type in the medical literature.

This paper is structured as follows. First, background information on breast cancer treatment and RT is presented. This includes details of the assessment process, particularly staging and the subsequent RT treatment technique. The next section provides details on the structure, data sources, assumptions and limitations of the model. The results of the model, including sensitivity and value of information analyses, are then presented. Conclusions and future work are presented in the concluding section.

Background

Breast cancer and RT

Breast cancer accounts for 30% of diagnosed cancer cases in Canadian women. The death rate is second only to lung cancer. According to the National Cancer Institute of Canada (www.ncic.cancer.ca; accessed 5 August 2005), the lifetime risk of developing breast cancer for women is estimated at 11.1%, or one in nine women, and the lifetime risk of dying from breast cancer is 3.7% or one in 27 women.

Surgery is the primary treatment for breast cancer. There are two forms of surgery for breast cancer patients: lumpectomy and mastectomy. Breast-conserving surgery, also known as lumpectomy, is surgery to remove the tumor while conserving, as much as possible, healthy breast tissue. Mastectomy is an operation aimed at removing the entire breast and lymph nodes, and in some cases, the underlying muscles. Surgery is often followed by RT. For patients undergoing breast-conserving therapy, studies show that adjuvant systemic therapy such as chemotherapy, hormonal and biological therapy cannot at present replace RT (Forrest et al, 1996; Fisher et al, 2002a, b). Thus, breast RT after lumpectomy is the standard of care in most facilities. For post-mastectomy patients, there has been considerable controversy regarding the degree of benefit derived from post-mastectomy RT (Donegan and Spratt, 2002).

RT is the treatment of cancer with targeted high-energy beams of ionizing radiation, usually X-rays. Such radiation injures destroys cancer cells in the area being treated, compromising the ability of these cells to continue to divide. Although radiation damages both cancer cells and normal cells, the goal of properly administered treatment is to keep normal tissue damage at an acceptable level while at the same time destroying the cancer cells. In breast cancer patients, RT is given adjuvantly to control recurrence. RT is most beneficial when the chosen technique is matched to the appropriate type, grade and stage of cancer. The stage of cancer, which indicates how far the cancer has progressed, is the most important factor in the choice of RT technique for breast cancer patients. Most breast cancer patients are treated with a two-field (two beams) technique, four-field (four beams) technique or with palliative intent (eg for pain management).

We have developed a system map for the RT process, which includes a domain we call Assessment (Lee et al, 2004) (Figure 1). A component of Assessment is staging, which is the process of identifying the stage of the cancer, using clinical and/or pathologic findings. The TNM (tumor size, lymph node involvement and degree of distant metastasis) system (Donegan and Spratt, 2002) classifies cancer patients into five main stages: 0, I, II, III and IV. Pathology reviews are used to determine Stages 0–III of breast cancer, while imaging and biochemical tests are used to assess the presence of metastatic disease, which defines Stage IV cancer. The correctness of staging is therefore, dependent on the adequacy of surgery, the accuracy of the pathology review and the accuracy of the imaging and biochemical tests employed. This in turn affects the accuracy of the RT treatment technique prescription.

Figure 1.

System map and domains of RT for cancer.

Full figure and legend (59K)

The Alberta Breast Cancer Program (ABCP) RT guide provides guidelines (www.albertabreast.com/patientmd/ccirtguide.php accessed 5 August 2005) for which RT technique to employ and when. Generally, the disease is characterized as invasive or non-invasive; cancer that has spread beyond the layer of tissue in which it developed and is growing into surrounding, healthy tissues and lymph nodes is said to be invasive. For non-invasive disease, the guide recommends two-field treatment. For invasive disease, the guide considers the number of nodes removed for evaluation during surgery and the number of positive nodes from those removed (for simplicity, we assume that at least 10 nodes are always removed for evaluation). Lymph nodes are determined to be positive for cancer if cancer cells can be found in the lymphatic system. A two-field technique is recommended for situations where about 1–3 of the removed nodes are positive, and otherwise a four-field treatment is recommended.

Methods

Model development

Information used in the model is typical for a Canadian cancer treatment facility, which is publicly funded and has a fixed annual budget (note that different types of tests may be used in a privately funded facility, as cost is less of a constraint). In most cases, we employed information from the peer-reviewed literature to define model inputs. In some cases where this information was not available or facility-specific information was used, we relied upon the expert opinions of two of the co-authors (KLK and PD).

In the simulation model, we followed a cohort of hypothetical patients through the system, allowing their simulated test results to be affected by parameters of the test (sensitivity, specificity and accuracy). These parameters were defined in the model as distributions to account for the uncertainty surrounding the true value of the tests. The results of these tests were used to assign the stage of cancer, which in turn influenced the prescribed RT technique. The results of the simulation (diagnosed cancer stage, prescribed RT technique) were compared to the reference data (true cancer stage, required RT technique), assigned a priori, to calculate the probability of incorrect staging diagnoses and RT treatment technique prescriptions.

The simulation model is composed of four distinct modules: the data generating module, the cancer stage identification module, the RT technique prescription module and the result analysis module. Figure 2 is a high-level representation of the model structure. The arrows indicate inputs and outputs to and from each computational module.

Figure 2.

High-level representation of the computational model.

Full figure and legend (30K)

Data generating module

The input for this module is a hypothetical cohort (assuming an average of 500 patients present for RT at the TBCC annually). Each hypothetical patient has already been assessed as having cancer. Occasionally, this diagnosis could be wrong. Based on expert opinion, 0.1% of the patients may actually have some condition other-than cancer.

For each hypothetical patient who has cancer, a stage is assigned based on expert estimation that approximately 10% of new patients present with Stage 0, 45%—Stage I, 30%—Stage II, 10%—Stage III and 5%—Stage IV cancer. The model uses this approximation as the true distribution of cancer stage for new patients presenting at a typical cancer treatment facility.

The prescribed RT technique depends on the cancer stage and the ratio of positive to negative lymph nodes removed during surgery. A review of the ABCP guidelines with an oncologist indicated that all diagnosed Stage 0 and I patients should be prescribed two-field treatments, and all Stage IV patients palliative treatment. For Stage II and III patients, the ratio of positive to negative lymph nodes becomes important in the assignment of an RT technique. Assuming that 10 or more nodes are removed during the surgery, patients are assigned a stage in keeping with guidelines. In the TBCC, approximately 80% of Stage II patients and 30% of Stage III patients are prescribed two-field treatments, while 20% of Stage II patients and 70% of Stage III patients are prescribed four-field treatments.

The diagnosed cancer stage is influenced by the true cancer stage (from the reference data set), the accuracy of pathology reviews and the accuracy of imaging and biochemical tests for discovery of metastases. All patients with Stage IV cancer were randomly assigned a primary site for metastases. The most common sites of metastases for breast cancer are the bone, liver and lung (Donegan and Spratt, 2002). Of the 5% of patients with actual Stage IV cancer, expert opinion suggests that 50% would have bone metastases, 30% liver metastases and 20% lung metastases. Based on the primary site assignment, different biochemical and imaging tests are employed to confirm metastases. A review of current literature along with experts on the research team identified the following tests: bone scans (S), carbohydrate antigen 15.3 (CA 15.3), carcinoembryonic antigen (CEA) and alkaline phosphate (AP) for detecting bone metastases; liver ultrasounds (LUS), aspartate aminotransferase/alanine aminotransferase (AST/ALT), CEA and gamma glutamyl transferase (GGT) tests for determining liver metastases; and chest X-rays (CXR) and CEA tests for determining chest metastases. In practice, most facilities use a subset of these tests and not all of them simultaneously. However, for purposes of model generality, we considered most recognized tests (Ravaioli et al, 2002).

Cancer staging module

This module used the information from the data-generating module to simulate the diagnosis of the cancer stage for each patient. Given the true cancer stage from the reference data, the results of the pathology review process were simulated to reflect the accuracy of the process. Experts predict that all Stage IV patients, approximately 30% of Stage III patients and 10% of Stage II patients will show symptoms for metastases (these include severe bone pain, bumps in other areas, shortness of breadth, ulcerations of skin, etc) and will subsequently undergo screening. The probability of metastases is calculated based on the prior probability of metastases to the site in question, and the simulated test results. A decision threshold probability of 80% was used in this model. If the calculated probability of metastases is above this threshold, the patient is assigned Stage IV; otherwise the patient retains the original pathology-assigned stage. The threshold is a subjective value that reflects the readiness of an oncologist to confirm a stage IV diagnosis. A high threshold value will decrease the number of false positives (patients in whom metastasis is absent but the test result is positive) but there is the risk of increasing the false negatives (patients in whom the condition is present but the test result is negative).

Determining the statistical accuracy of pathology examinations was difficult due to the lack of a 'gold standard' to which these reports may be compared. The most common approach to this problem is the review of pathology reports in prospective and retrospective studies. In a few studies specific to breast cancer, the rate of change of diagnosis was between 1.4 and 7.8% (Kronz et al, 1999; Staradub et al, 2002). Although these are not accuracy rates, they indicate the level of uncertainty associated with this process. Modelling the accuracy rate of pathology reviews as a uniform distribution between 92.2 and 98.6%, we generated pathology results such that mis-staging is more likely to occur between adjacent stages on the TNM scale. For simplicity, the assumption was made that the accuracy of pathology reviews in identifying cancer stage is the same for all stages of cancer. More information on how these results were simulated is available upon request from the authors.

We consider the patient to be in one of two health states with respect to metastases to a particular site: D (presence of distant metastasis) or (absence of distant metastasis). Assuming that T is a binary test that attempts to distinguish between D and . The value of T is characterized by four conditional probabilities: P(T+|D) (ie, the probability of a positive test result given that the patient is in state D), P(T+|), P(T-|D) and P(T-|). The first of these is commonly known as the sensitivity of the test, and the last as its specificity. Although sensitivity and specificity are important characteristics of a test, they are not the probabilities required in order to make a decision to treat or not treat a patient given test results. Of more importance are the predictive positive value P(D|T+) and the predictive negative value P(|T-) of the test. The predictive positive value is the probability of a condition (in this case distant metastasis), given a positive test result, while the predictive negative value is the probability of the absence of a condition (no distant metastases), given a negative test result. To estimate these values, an estimate of the prior probability of disease P(D) in the sample population from which the patient is drawn is required. The sensitivity and specificity of tests are standard values that can be retrieved from the literature and the results of randomized trials (Table 1). Using Bayes' rule (Russell and Norvig, 1995), the predictive positive value of test T is calculated as

Table 1 - Sensitivity and specificity of tests used for detection of metastases in breast cancer patients.

Full table

To our knowledge, there are no studies that have examined any potential dependency or correlation structure between the tests evaluated here. We therefore assumed independence. Given this assumption, when a battery of tests are used simultaneously (eg, T₁, T₂, T₃ and T₄) to diagnose a condition or state, the predictive value of the set of tests can be calculated on a case-by-case basis (depending on the individual test results). For example, assuming all positive results from the four tests mentioned earlier, the predictive positive value can be calculated as follows:

The calculated predictive value is a probabilistic value that lies between 0 and 1. The decision maker will often set a subjective threshold value, such that when the predictive value is above the set threshold, the condition is assumed to exist and the patient is treated for it, otherwise, either more testing is prescribed or the condition is assumed not to exist.

Staging is primarily determined based on pathology examination. The cancer stage is changed only if the test for metastases is positive, in which case the patient is assigned 'Stage IV'.

RT technique prescription module

The assigned RT prescription was dependent on the diagnosed cancer stage and the number of identified positive nodes. All diagnosed Stage 0 and I patients were prescribed two-field treatments, and all Stage IV patients palliative treatment in keeping with the ABCP recommendations. For patients diagnosed as Stage II and III patients, the prescribed treatment was influenced by the accuracy of pathology reviews in identifying positive lymph nodes from the set of removed nodes. The limited accuracy of the pathological process in identifying these nodes resulted in errors, which then affected the prescribed treatment for the patients.

Result analysis module

The result analysis module carries out one-on-one comparison of the simulated diagnosis and prescribed technique with the reference data for each hypothetical patient in the model. It provides the probability of a wrong stage diagnosis and/or prescribed RT technique. The results of this model are presented as averages of the results from 100 iterations with a patient population size of 500 (ie a two-dimensional simulation, Tables 2 and 3). This number of iterations achieves a reasonable degree of confidence in mean results (Morgan and Henrion, 1990).

Table 2 - Simulation results for staging of breast cancer patients.

Full table

Table 3 - Simulation results for RT treatment technique for breast cancer patients.

Full table

The simulation was run as a Latin Hypercube simulation (LHS), a variant of Monte Carlo simulation which focuses sampling on distribution tails (Helton and Davis, 2002). The modelling platform was Analytica (Lumina Decision Systems Inc.), which is a modular simulation program integrating risk and sensitivity analysis for models with uncertain inputs.

Uncertainty distributions and sources

Uncertain variables were defined as uniform distributions, which impose a minimal set of assumptions regarding parameters and shape (Lee and Wright, 1994). Information on the accuracy of pathology reviews, sensitivity and specificity of imaging and biochemical tests was defined from published data (Table 1). Other assumptions represent information from standard texts, practice guidelines and expert opinion. These are stated where applicable. No patient-specific information was employed in this analysis.

Results and discussion

The results of the simulations are presented in Tables 2 and 3. The 95% confidence interval for the mean of each result is shown. The analysis indicates that staging accuracy is highest for Stage I patients (98.38%) and lowest for Stage IV (82.83%) patients. This is in part due to the prior probability of patients in each of the stages: patients presenting for RT for the first time are more likely to be Stage I patients than Stage IV patients. The poor results for Stage IV patients may also be attributed to the less-than perfect sensitivity and specificity of the tests used for diagnosis.

The model assumed singular sites of metastases. The assumed low prior probability of patients who do not have cancer also explains the low accuracy of diagnosing such patients (more detailed facility specific information would allow refinement of this assumption). The results also reflect the fact that mis-staging is most likely to occur between adjacent stages on the severity scale. For example, Stage IV patients are more likely to be classified as Stage III (4.79%) patients than as Stage II (0.67%) patients; Stage I patients are more likely to be misclassified as Stage 0 or II patients than as Stage III or IV patients.

Mis-staging between Stage 0 and I patients is of relatively low consequence because they are prescribed the same type of treatment (two-field treatment). These patients (Stage 0 and I patients) also form a large proportion of incoming patients (54%). As a result of this, we found a high accuracy index (probability that required treatment is the two-field treatment given that prescribed treatment is also the two-field treatment) of over 99%. The errors in mis-staging Stage IV patients and misdiagnosing non-cancer patients (Stage X patients) were directly propagated to the treatment prescription, thus potentially subjecting patients to inappropriate treatments.

We conducted a detailed sensitivity analysis (Table 4) to determine which inputs were associated with the most variation in the outcome of the assessment for metastases. Spearman's rank correlation coefficient was used as a measure of the dependency between the uncertain model input values and the measured output (the probability of correct diagnosis of metastases). Sensitivity analysis was not performed for all stages because the pathology review is the only source of information available for determining the stage of non-metastatic patients, that is, patients in Stages 0–III of cancer.

Table 4 - Sensitivity analysis of the process of detection of distant metastases in the bone, lung and/or liver.

Full table

Table 4 presents details of the sensitivity analysis for all three possible sites of distant metastases: bone, liver and lung. The sensitivity analysis reveals that the assessment for bone metastases is very sensitive to the specificity of AP, bone scans (S), CA 15.3, and the sensitivity of CEA, and CA 15.3 (Table 1). The spread of possible values for the specificity of AP tests as reported in the literature (from 0.39 to 0.98) (Karmen et al, 1984; McGarrity et al, 1987; Reale et al, 1995; Ritzke et al, 1998; D'Alessandro et al, 2001; Guadagni et al, 2001; Ravaioli et al, 2002) introduces the largest amount of uncertainty in the expected outcome (Table 4) and subsequently has the highest rank correlation coefficient. Although the probability of correctly diagnosing bone metastases was consistently high for this battery of tests (between 0.985 and 0.992), further research and meta-analysis of the input variables may be required to narrow the range of their values.

Physicians often employ several information sources simultaneously with the intent of reducing uncertainty and compensating for the limitations of some of the information sources. Value of information analysis provides decision makers with a tool for evaluating the relative worth of information sources (Weinstein et al, 1980) and their effect on the outcome of a diagnosis. The quantitative value of an information source (prospectively evaluated) is referred to as the expected value of information. In our analysis, the expected value of information is a metric by which we measure the expected improvement in the correctness (or accuracy) of the staging process that may be directly attributed to the presence of improved information (Table 5).

Table 5 - Value of information analysis.

Full table

The analysis demonstrates that given a choice of two tests, a combination of CA 15.3 and S are of the greatest value, with an accuracy of 85.3% of correctly identifying bone metastases. The AP test appears to add little value to the S test. A panel of three tests reveals a greater expected value than a panel of two tests. From our analysis, the 'best' combination of three tests would be S, CEA and CA 15.3, with an accuracy of 91.6%. The added value of using all four tests is 0.5%.

The LUS test is the standard practice for the detection of liver metastases, thus it is included in all the test panels. Given the choice of only two tests in a panel, the best combination from our analysis would be LUS and CEA with an expected value of 88.1%. For liver metastases, using a panel of LUS, GGT and CEA is as valuable as having a panel of LUS, AT/ALT, GGT and CEA. Also using a panel of LUS and CEA is as valuable as using a panel of LUS, CEA and AST/ALT. This indicates that the AST/ALT test is of very little value even when used in combination with all the other tests. This may be due to its very low sensitivity and specificity (Table 1). The best battery of three tests would consist of LUS, GGT and CEA with an expected value of 89.3%.

Metastasis to the chest is mainly diagnosed via chest X-rays/radiographs (CXR). CEA is a tumour marker that is not very specific to the site of metastases or the type of cancer. However, it can alert physicians to the possibility of metastases to the chest, which can then be confirmed via CXR and/or biopsy. A combination of CXR and CEA yield an expected value of 85.0% as compared to 72.5% for CXR alone.

Conclusion and future work

The practical aim of our simulation study is to help quantify the degree of accuracy that can be expected in the process of staging and RT technique prescription for post-surgery breast cancer patients. To our knowledge, this is the first analysis of its type. We have demonstrated that the probability of errors in treatment, based on imperfect information, is small but non-trivial. Considering that RT is a procedure with potentially severe consequences (ie severe injury or death) if improperly administered, our analysis speaks to the necessity of careful assessment of patients.

We made some assumptions in this analysis. We assumed that clinicians follow best-evidence practice and use current evidence in this decision context. We also assumed that decisions were based on the results from a battery of tests and evaluated in a Bayesian framework, as is routinely assumed in the published literature (Weinstein et al, 1980; Hunink et al, 2001; Parmigiani, 2002). We did not explicitly address subjective judgement, which introduces variation across practitioners and thus may affect treatment decisions. Although most physicians are aware of evidence-based practices, usually published as practice guidelines on a provincial and/or national basis, the degree of adherence is unknown. Additionally, some physicians may not use Bayesian methods (implicitly or explicitly) in their decision-making process. Surveys to determine the degree of adherence to evidence-based guidelines and the use of Bayesian decision-making were not within the scope of our analysis, but would be important for further refining the model. Also, the question of consequence (eg adverse patient outcomes) is not addressed in this paper; we limited the analysis to evaluation of the probability of a wrong diagnosis and consequently a wrong treatment option. The question of consequence will be addressed in future studies.

We did not employ formal techniques to elicit expert opinion, nor did we attempt to refine input distributions beyond uniform distributions. We did not consider situations when test results are inconclusive and therefore could be repeated. However, the results of our sensitivity and value-of-information analyses could be used to identify variables for which refinement may be informative. We have assumed, for simplicity, only scenarios in which metastases occur in one site for each patient. In practice, it is possible to have simultaneous metastases to multiple sites. This would require much more complex modelling, but may be worthwhile as a refinement.

We suggest that use of models such as ours will inform patient safety and quality control programmes. A natural extension of this work would be to include consideration of costs and resource constraints in a decision analytic or optimization model.

References

1. Alderson PO et al (1983). Computed tomography, ultrasound, and scintigraphy of the liver in patients with colon or breast carcinoma: a prospective comparison. Radiology 149: 225–230. | PubMed | ChemPort |
2. Bruneton JN et al (1997). Etiologic diagnosis of hepatic lesions in cancer patients. Value of ultrasound and liver function tests. Clin Imaging 21: 366–371. | Article | PubMed | ChemPort |
3. Ciatto S et al (1985). Staging breast cancer–screening for occult metastases. Tumori 71(4): 339–344. | PubMed | ChemPort |
4. D'Alessandro R et al (2001). Serum tissue polypeptide specific antigen (TPS): a complementary tumor marker to CA 15-3 in the management of breast cancer. Breast Cancer Res Treat 68: 9–19. | Article | PubMed | ChemPort |
5. de Brauw LM et al (1991a). Hepatic metastases: comparative study of diagnostic ultrasound, CT, nuclear scintigraphy and laboratory tests. Neth J Surg 43: 92–94. | ChemPort |
6. de Brauw LM et al (1991b). Prospective comparative study of ultrasound, CT-scan, scintigraphy and laboratory tests to detect hepatic metastases. J Nucl Biol Med 35: 131–134. | ChemPort |
7. del Rocio Estrada-Sanchez G, Mireles Enriquez M and Valverde Castaneda N (2003). Reliability of CA 15-3 in the follow up of female patients with breast carcinoma and bone metastases. Rev Invest Clin 55: 412–418. | PubMed |
8. Donegan WL and Spratt JS (2002). Cancer of the Breast. (USA). Saunders–Elsevier Science: St Louis, MO.
9. Dowie J (1996). 'Evidence-based', 'cost-effective' and 'preference-driven' medicine: decision analysis based medical decision making is the pre-requisite. J Health Serv Res Policy 1: 104–113. | PubMed | ChemPort |
10. Engelhard K et al (2004). Comparison of whole-body MRI with automatic moving table technique and bone scintigraphy for screening for bone metastases in patients with breast cancer. Eur Radiol 14: 99–105. | Article | PubMed | ChemPort |
11. Fisher B et al (2002a). Twenty-year follow-up of a randomized trial comparing total mastectomy, lumpectomy, and lumpectomy plus irradiation for the treatment of invasive breast cancer. N Engl J Med 347(16): 1233–1241. | Article | PubMed | ISI |
12. Fisher B et al (2002b). Tamoxifen, radiation therapy or both for prevention of ipsilateral breast tumor recurrence after lumpectomy in women with invasive breast cancers of one centimeter or less. J Clin Oncol 20: 4141–4149. | Article | PubMed | ChemPort |
13. Forrest AP et al (1996). Randomised controlled trial of conservation therapy for breast cancer: 6-year analysis of the Scottish trial. Scottish Cancer Trials Breast Group. Lancet 348(9029): 708–713. | Article | PubMed | ISI | ChemPort |
14. Gallowitsch HJ et al (2003). F-18 fluorodeoxyglucose positron-emission tomography in the diagnosis of tumor recurrence and metastases in the follow-up of patients with breast carcinoma: a comparison to conventional imaging. Invest Radiol 38: 250–256. | Article | PubMed |
15. Guadagni F et al (2001). A re-evaluation of carcinoembryonic antigen (CEA) as a serum marker for breast cancer: a prospective longitudinal study. Clin Cancer Res 7: 2357–2362. | PubMed | ChemPort |
16. Helton JC and Davis FJ (2002). Illustration of sampling-based methods for uncertainty and sensitivity analysis. Risk Anal 22: 591–622. | Article | PubMed | ISI | ChemPort |
17. Hou MF et al (1999). Evaluation of serum CA27.29, CA15-3 and CEA in patients with breast cancer. Kaohsiung J Med Sci 15: 520–528. | PubMed | ChemPort |
18. Hunink M et al (2001). Deciding when to test. In: Hunink M. Decision Making in Health and Medicine, Integrating Evidence and Values. Cambridge University Press: Cambridge: UK. pp 157–176.
19. Karmen C et al (1984). Measurement of biliary alkaline phosphatase by mini-column chromatography and by electrophoresis and its application to the detection of liver metastases in patients with breast cancer. J Clin Pathol 37: 212–217. | PubMed | ChemPort |
20. Kim NK et al (1977). Value of alkaline phosphatase, 5'-nucleotidase, gamma-glutamyltransferase, and glutamate dehydrogenase activity measurements (single and combined) in serum in diagnosis of metastasis to the liver. Clin Chem 23: 2034–2038. | PubMed | ChemPort |
21. Kokko R, Hakama M and Holli K (2003). Role of chest X-ray in diagnosis of the first breast cancer relapse: a randomized trial. Breast Cancer Res Treat 81: 33–39. | Article | PubMed | ChemPort |
22. Kronz JD, Westra WH and Epstein JI (1999). Mandatory second opinion surgical pathology at a large referral hospital. Cancer 86(11): 2426–2435. | Article | PubMed | ChemPort |
23. Lacic M et al (1999). Bone marrow immunoscintigraphy for the detection of skeletal metastases in patients with breast cancer. Nucl Med Commun 20: 135–143. | PubMed | ChemPort |
24. Lauro S et al (1999). Comparison of CEA, MCA, CA 15-3 and CA 27-29 in follow-up and monitoring therapeutic response in breast cancer patients. Anticancer Res 19: 3511–3515. | PubMed | ChemPort |
25. Lee RC and Wright WE (1994). Development of human exposure factor distributions using maximum-entropy inference. J Expos Anal Environ Epi 4: 329–341.
26. McGarrity TJ, Samuels T and Wilson FA (1987). An analysis of imaging studies and liver function tests to detect hepatic neoplasia. Digest Dis Sci 32(10): 1113–1117. | Article | PubMed | ChemPort |
27. Morgan MG and Henrion M (1990). Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press: Cambridge.
28. Ohta M et al (2001). Whole body PET for the evaluation of bony metastases in patients with breast cancer: comparison with 99Tcm-MDP bone scintigraphy. Nucl Med Commun 22: 875–879. | Article | PubMed | ISI | ChemPort |
29. Parmigiani G (2002). Measuring uncertainty in complex decision analysis models. Stat Methods Med Res 11: 513–537. | Article | PubMed | ChemPort |
30. Prior JO et al (2003). The value of bone marrow scintigraphy using 99mTc monoclonal antigranulocyte antibodies in complement to bone scintigraphy in detecting bone metastases from primary breast cancer. Nucl Med Commun 24(1): 29–36. | Article | PubMed | ChemPort |
31. Ravaioli A et al (2002). Staging of breast cancer: new recommended standard procedure. Breast Cancer Res Treat 72: 53–60. | Article | PubMed |
32. Reale MG et al (1995). Skeletal alkaline phosphatase as a serum marker of bone metastases in the follow-up of patients with breast cancer. Int J Biol Markers 10: 42–46. | PubMed | ChemPort |
33. Ritzke C et al (1998). Alkaline phosphatase isoenzymes in detection and follow up of breast cancer metastases. Anticancer Res 18: 1243–1249. | PubMed | ChemPort |
34. Russell S and Norvig P (1995). Uncertainty. Artificial Intelligence: A Modern Approach. Prentice-Hall Inc.: Upper Saddle River NJ, pp 415–431.
35. Schattevoet WH, Corstens FH, Beex LA and Rosenbusch GJ (1986). Reliability of sonography in diagnosing liver metastases in breast and bronchial cancer. A retrospective comparison of physical examination, laboratory tests and liver scintigraphy. Ultraschall Med 7: 293–299. | PubMed | ChemPort |
36. Spanu A et al (2003). The role of (99m)Tc-tetrofosmin pinhole-SPECT in breast cancer non palpable axillary lymph node metastases detection. Q J Nucl Med 47: 116–128. | PubMed | ChemPort |
37. Staradub VL et al (2002). Changes in breast cancer therapy because of pathology second opinions. Ann Surgical Oncol 9(10): 982–987. | Article |
38. Tempero MA, Williams CA and Anderson JC (1986). The value of hepatic ultrasound and biochemical liver tests in screening for liver metastases. J Clin Oncol 4: 1074–1078. | PubMed | ChemPort |
39. Torres M et al (1995). CA 549 and SP2 in postoperative breast cancer patients. Comparison with CA 15.3, CEA and TPA. Int J Biol Markers 10: 94–99. | PubMed | ChemPort |
40. Vizcarra E et al (1996). Value of CA 15.3 in breast cancer and comparison with CEA and TPA: a study of specificity in disease-free follow-up patients and sensitivity in patients at diagnosis of the first metastasis. Breast Cancer Res Treat 37: 209–216. | Article | PubMed | ChemPort |
41. Waalkes TP et al (1982). Biological markers as an aid in the clinical management of patients with liver metastases. J Surg Oncol 20: 83–94. | PubMed | ChemPort |
42. Weinstein MC et al (1980). Clinical Decision Analysis. W.B. Saunders Company: London.
43. Wojtacki J et al (2001). Elevation of serum Ca 15-3 antigen: an early indicator of distant metastasis from breast cancer. Retrospective analysis of 733 cases. Przegl Lek 58: 498–503. | PubMed | ChemPort |