Background

A growing literature discusses the need to translate knowledge from epidemiological and other research into action, and the methods for achieving this transfer.1, 2, 3, 4 and 5 Although not without its critics,6 medical practice has seen a massive growth of evidence-based medicine.7, 8, 9 Now there are calls to use this approach in public health and health systems management.10 Some researchers present their findings to be accessible and to support informed decision making. Others rely on knowledge syntheses and systematic reviews to make their work useful for policy. More and more systematic reviews,11 have been generated, making available unprecedented quantities of evidence. With few unbiased filters to sift through it,12 policymakers and planners need to know more about how to use evidence that comes their way.

Part of the problem is the uneven mix of evidence from different sources. Much of the evidence does not come from systematic reviews of randomised controlled trials (RCTs) or even a single RCT. Protagonists of a particular viewpoint can present a single strand of evidence, ignoring other strands. Knowledgeable decision makers, with some ability to sift through highlights and to ask the right questions about evidence, would be an important safeguard. Many decision makers lack these skills.13, 14 A 2012 international forum on evidence-informed health policymaking in low- and middle-income countries called for building the capacity of potential research users to evaluate and use research evidence.15

Initiatives have attempted to build health policymakers’ and planners’ capacity to use evidence in both rich and resource-poor settings.16, 17 These initiatives mostly target government technical officers, senior civil servants, and other advisers, on the assumption that they are then better able to advise elected representatives who make decisions about health funding. Programmes to train elected representatives have focused primarily on debating, law making, ethics, and the like.18, 19, 20 and 21 We have not found any reports of training elected representatives in the interpretation and use of health, or other, evidence.

HIV prevalence in Botswana is among the highest in the world.22 Tackling the epidemic is a national public health priority. In 2011, after some ill-informed debate in Parliament, the ruling party withdrew a revised national AIDS policy for further revision. The minimum education required for members of Parliament in Botswana is 7 years of schooling; current members all have at least school leaving certificates (12 years), and many have university degrees. To support more informed parliamentary debate on HIV and other issues, the minister for Presidential Affairs and Public Administration arranged training for members of Parliament. It covered interpretation and use of evidence to support decision making, particularly in the health field. We describe this training in evidence-based policymaking.

Methods

Survey of elected representatives

In October 2011, the Office of the National Assembly invited all 57 elected national representatives (and four appointed members of Parliament) to attend training over 2 days just before the opening of the parliamentary session. This office also provided us with a list of representatives with their contact numbers. During the 2 weeks before the training, we attempted to contact and interview as many of them as possible. Contacting the elected representatives required repeated phone calls (often to contact numbers not originally listed). The interviewer arranged to conduct the short interview at a convenient time, by telephone or face-to-face. The short questionnaire included a mixture of closed and open questions about perceived needs for evidence to support parliamentary work, current access to evidence, and perceived needs for further training about types of evidence, its interpretation, and use. We conducted interviews mostly in Setswana with responses translated later into English. Interviewers entered responses to closed questions into an electronic spreadsheet and wrote out in full the responses to open questions.

Training sessions

The National AIDS Coordinating Agency (NACA) arranged and supported the training sessions. In November 2011, three of us (a professor of epidemiology who grew up in Botswana; a Canadian professor of biostatistics, originally from Lesotho; and a senior researcher leading a RCT of HIV prevention in Botswana, Namibia, and Swaziland) provided training for the representatives over one afternoon and the following morning. At the request of participants in 2011, the professor of epidemiology and the RCT researcher provided further training during two mornings in November 2012.

Table 1 summarises the training sessions, finalised in the light of responses to our survey of representatives. We emphasised the key role of counterfactual evidence and how to question biases in the generation of evidence. Participants learned about how policy relies on population parameters, like the number needed to treat (NNT) to prevent one adverse outcome, plus the relative irrelevance to policy of parameters of individual advantage. We also taught them about the odds ratio and its complement, ‘per cent protection’. Our teaching techniques included didactic lecture-style sessions, question and answer discussion sessions, and hands-on work in small groups. At the end of each training, participants completed a short anonymous evaluation questionnaire.

Table 1 Elements of the training sessions
Full size table

Results

Survey

None of the elected Representatives contacted in 2011 refused to be interviewed. Of those not interviewed, most were out of the country or not contactable; a few were unavailable for interview or could not attend the training because of pre-existing commitments. The interviewers conducted 17 interviews over the telephone and 10 face-to-face. They did not ask respondents about their age or educational level.

Nearly all respondents said they needed more evidence and more training about how to use evidence (Table 2). Asked about their sources of evidence, respondents mentioned evidence from committee papers, from the parliamentary research department, from lobbyists, from commissioned and non-commissioned advisers, from their constituencies, and from the press. For sources of evidence about HIV and AIDS, elected members considered the best to be: NACA, doctors and other health professionals, researchers, the Ministry of Health, the World Health Organisation, and civil society organisations.

Table 2 Perceived needs for evidence and knowledge of how to use evidence among 27 parliamentarians in 2011
Full size table

The respondents cited difficulties they faced using evidence in their work. They mentioned problems with jargon, technical language, and statistical terms that led to problems with communicating the evidence and perhaps to bad decisions.

“The use of jargon makes it difficult to comprehend the evidence”.

“Some of it is too complicated to understand, with difficult wording and statistical data. This makes your work difficult in trying to use the evidence to address issues”.

“The technical terms used in the evidence are a challenge; communicating the evidence to the public then becomes a problem”.

“When the reports are not very easy to understand, especially the terminology, you can make a decision based on something you don’t understand”.

“User-friendly information is needed for policy audit and advocacy”.

A common complaint was lack of research staff for Parliament, inexperienced research officers, and an overall lack of relevant evidence.

“Research officers are inexperienced because of high turnover”.

“There is a lack of researchers dedicated to working with parliament”.

“The parliamentary research office is under-staffed”.

“There is a paucity of evidence; the inadequacy of evidence leads to wrong decisions and conclusions”.

Some respondents complained about difficulties with access to evidence, for example, via the Internet.

“Accessing information on the internet is a problem because our computers don’t work”.

“I don’t have adequate facilities, such as a laptop and internet connection, when I am outside the office. Basic training in ICT is limited, IT resources are limited”.

Some mentioned concerns about out-dated information.

“The evidence from government agencies is out-dated”.

“It affects the decision-making process because the evidence is not up to date”.

“The government thinks what they have is enough; they are not receptive to new evidence”.

Nearly all the respondents (25/27) believed that if they were better equipped to use evidence, this could help their work. They described the ways. Some explained how it would help their personal effectiveness.

“It would boost my confidence”.

“It would improve the quality of my presentations, for example to committees”.

“I would make arguments with enlightenment and knowledge”.

Many thought it would improve the speed and effectiveness of decision making.

“It will enhance the speed at which we can address issues because we spend a lot of time trying to comprehend the evidence and we fail to meet our target”.

“It will help in effective policy making”.

“It will put us in a better position to plan and make sound decisions”.

“It could assist us as members of Parliament to bring about positive change”.

Some mentioned specifically that it could help them communicate with their constituents.

“We’d have information to pass to our people [in our constituencies], with actual facts and knowledge”.

“It would help us with communicating to the communities that are illiterate or semi-illiterate in our constituencies”.

The final section of our questionnaire offered possible elements of training about evidence and use of evidence. For each element, we asked if they would like to learn more. Interviewers administered this section to 24 of the responding members of Parliament. Respondents were keen to learn about all the listed elements: types of evidence and their advantages and limitations (22); the language of scientific evidence (what the terms really mean) (22); presenting evidence so it is useful for planning (24); and questions parliamentarians can ask about evidence to determine its importance for them (23). Some suggested other elements: where and how to use research evidence; research methodologies; interpretation and use of evidence.

The trainings and their evaluation

There were some logistic challenges: late or undelivered official invitations in 2011 and a tight schedule in 2012. Nevertheless, a total of 36 elected Representatives attended one or both of the training sessions, including 7 ministers, the Deputy Speaker, the leader of the opposition, and the chair of the parliamentary committee on health and HIV and AIDS. The Director of HIV and AIDS Prevention and Care in the Ministry of Health also attended, as did the National Coordinator and other NACA personnel. The educational background of the participants varied from a school leaving certificate to completion of several advanced degrees.

Attendees participated actively, requesting clarifications, and related the information to their own work and to future consideration of the national AIDS policy. A few participants made political points in the discussions that followed presentations, but most of the content was conspicuously non-partisan. The groups considering the evidence examples in the final section of the first training explained the limitations of the evidence for planning correctly. The groups in the final section of the second training proposed feasible actions to enhance HIV prevention efforts. Participants appreciated the need to measure the impact on new HIV infections.

Everyone rated all sessions positively. On a scale of 1–5 (least to most positive), in 2011 the mean score for relevance of the content was 4.25, for level of the content 4.15, and for presentation 4.20. A majority (12/19) thought the length of the training was ‘about right’ and 7/19 thought it was too short. Participants recommended that the training be repeated for those members of Parliament, including ministers, who were unable to attend, as well as for other stakeholders including local governments, traditional leaders, non-government organisations, and church leaders. In 2012, the mean score for relevance was 4.5, for level 4.3, and for presentation 4.0. In the days following each training session, the Speaker reported positive feedback from the participants.

Discussion

Elected national representatives play a key role in setting policies, including health policies and, crucially, agree on budgets to support policies. The results from our small survey indicate that elected Representatives are well aware of the need to use evidence in their work, and feel hampered by lack of good quality and relevant evidence. They face difficulties in interpreting the evidence as it is presented to them from different quarters and in evaluating its importance for policy. The decisions they have to make may have profound consequences. Is the evidence sufficiently compelling that it should lead to a new set of policies without delay, and with adequate budget allocation, even if that means shifting funds away from existing programmes? Or is it highly suggestive but needs confirmation before making a major policy shift? Or is it interesting but not convincing, either because of some methodological flaws, or because it comes from a quite different setting?

Many other considerations come into play when making policy decisions. Studies have documented hindrances to evidence-informed health policymaking in both low- and high-income countries13, 23 and two systematic reviews reported many barriers to health policymakers using evidence: decision makers’ perceptions about research evidence; lack of contact between researchers and policymakers; research that was not timely or relevant; mutual mistrust; competing influences; and power and budget struggles.24, 25

If elected representatives had a better understanding of the evidence they need to support rational decision making, instead of being passive recipients, they could start to demand different kinds of evidence. For example, they might start to demand evidence on population as well as individual benefit; also on the NNT (to prevent one adverse outcome) and the unit costs of different programmes. They might push for systematic reviews and research syntheses, or at least for studies with acceptable counterfactual evidence. They could become active and informed parties in setting the research agenda.

This might seem far from reality and indeed it is in most places. However, that will not change if we, as researchers and teachers, decide that members of Parliament have neither the background nor the interest to learn about types of evidence needed for decision making, and about how to interrogate and use evidence. The Botswana experience confirms that members of Parliament were aware of their need for training and interested in learning. The two training sessions in Botswana covered more than half the elected Representatives and seven ministers, despite logistic problems with late and undelivered official invitations in 2011 and a tight schedule in 2012, when the training in the mornings took place while Parliament was sitting in the afternoons.

A survey among members of Parliament in several countries found that some felt they did not need training for elements of their role and many were opposed to compulsory training.9 The training programmes on offer did not include interpretation and use of evidence.9 In our survey, nearly all the respondents said they could benefit from training about use of evidence; we did not ask them if they thought such training should be compulsory, as there is no plan to make it so.

A recent survey of elected representatives in the Pacific region found that few perceived that training had a significant impact on their individual performances or on the performance of the Parliament. Respondents mentioned that the training programmes failed to take account of differing educational levels, were short, irregular, ad hoc or duplicative, and did not include a needs assessment.26

We have no evidence so far about the impact of the training we provided. Nearly all of those who attended the training considered it useful and recommended that it be offered to the remaining members of Parliament, including ministers, as well as to other groups in the country. Participation was lively and feedback (formal and informal) was positive. It will surely be difficult to attribute positive changes to the training, as this work was not designed as a trial (randomised or not), rather as a feasibility exercise. We intended it to contribute to understanding about what sort of training elected representatives want and can manage.

Why was the approach apparently successful and to what extent could it be used elsewhere? Seeking the views of the elected representatives ahead of time, reflecting back their views to them – and using this as a guide to the level and content of the training set a positive tone and increased the relevance of the training for the participants. This approach could be used in other places.

We took care to make the content accessible to the audience, most of whom had no background in quantitative sciences. We drew on our extensive experience teaching about the use of epidemiology for planning in many resource-poor settings. To increase relevance of the learning materials, we used examples from Botswana and the southern Africa region. Different examples could be used in other countries.

Local factors were important. Botswana has a small, functional, and relatively peaceful Parliament. Major conflicts within Parliament, not uncommon in the region, would make multi-party training such as this very difficult to implement. Finally, the minister for Presidential Affairs and Public Administration is influential and strongly committed to the idea of training members of Parliament to interpret and use evidence; his championing our project was a key element.

We see this as the beginning of a long road. Next steps include: providing training for advisers and technical officers, so they can provide ongoing support for members of Parliament and respond positively to demands for more and better evidence; setting up a monitoring scheme, tracking, for example, how often research evidence is cited in parliamentary debates; and creating a tool kit for similar exercises in other parliaments in the region. Despite the likely challenges, if a similar training approach could be applied in other countries in sub-Saharan Africa, it could well contribute to formulating evidence-based policies to tackle the HIV epidemic and other pressing health problems.