The idea of a journal issue dedicated to exploring the impact that foresight has had on policymaking emerged out of a conference that SID organized in Nairobi in April 2014. The Rockefeller Foundation promoted, nurtured and funded a Searchlight Collective – a group of institutions spanning three continents that had over the past four years contributed insights on key trends and their potential impact on development policy and outcomes – which was meeting for the last time.

As part of this conference, SID convened a one-day public event to share the key emerging lessons from the trend-monitoring and horizon-scanning work of the collective and also to reflect on how exactly the insights emerging from this work were being put into practice. What emerged was a reflection within the collective, but also from other invited participants, largely around the question of whether the work we were doing had any significant impact on the policies and decisions taken or whether it was merely an intellectually self-indulgent set of exercises.

Following the Nairobi event SID interviewed members of the collective to understand what exactly they had learnt, and what they thought needed to be done differently in order to ‘make a difference’. The resulting reflections were captured in a mimeograph Dispatches from the Frontline – Using Pro-Poor Foresight to Influence Decision Making, the full text of which has been included in this issue. In this publication, three core observations were highlighted: the challenge of sustaining dialogue to enable learning to percolate through the system and hence ‘infiltrate’ decision-making models and processes; a need to focus on shifting behaviours and attitudes within a context of competing agendas; and that there is a wide panoply of tools, techniques and methodologies applied towards trying to embed foresight into institutional processes. This publication focused on and drew its insights from the work of the collective, but tapped into a very small sample of the world of foresight practitioners. Perhaps there was a need to cast our net wider and to explore other processes and lessons and to understand what was being observed and learnt. Hence this journal issue that has tried drawing upon the wisdom and counsel of a broader group of practitioners.

This journal brings together a wealth of experiences on lessons learnt from practicing foresight – lessons that can only come from those who have spent long hours in the trenches. We spoke at length with practitioners who shared valuable nuggets of wisdom that are not to be found in textbooks on the subject. The interviews presented here give us insights into their thoughts on the field and the challenges it needs to respond to. The contributing authors also make valuable contributions given the broad brief they received. We asked them to reflect on the utility of foresight and whether, in their view, it was leading to improved decisions and outcomes, particularly for the poor and vulnerable citizens of their communities. They did not disappoint.

So, does foresight improve decision-making? Theoretically, it is easy to respond in the affirmative. And it should – in principle being able to respond to changes as they emerge and to adapt institutions and systems to ensure that their purpose is not compromised or undermined by an evolving context and surprises springing out of the operational environment. The reality is more nuanced. Many of the articles here are cautionary tales. They suggest that a number of ingredients need to be in place to ensure that decision-makers take on board insights emerging from foresight processes. In her article, Helene Lavoix identifies the various challenges that need to be overcome in order to make foresight relevant and useful to policymaking. She suggests strongly that the foresight process be embedded within the organizational policy-making process with a view to reducing tensions between the foresight planners and the decision-makers, a relationship that inherently generates friction. Foresight work will inevitably challenge existing norms, mindsets and roles and this will create resistance to its messages within an institution. Therefore managing the ‘psychology’ of the process is as important as the products that emerge from it. In his interview, Andre Zaaiman reinforces many of these insights and exhorts practitioners to be humble and be aware that they will not have all the answers all of the time. He shares insights and anecdotes that shed light into the nature of the dilemmas that the foresight practitioner must confront. In particular, he suggests that the failure of many foresight processes to influence decision-makers might lie in the hubris of the practitioners who do not take enough time to know themselves and to understand how cognition and action interrelate. Zaaiman suggests that practitioners develop some competencies that will help them to succeed in their work – being able to understand how their minds process new information, and to be able to manage (and where necessary interrupt this) their own emotions and finally improve their ‘social intelligence’. In the final analysis, whether or not decision-makers are able to produce better decisions as a result of the insights that practitioners share with them is as much a function of the practitioner/advisor understanding the deeper psychosocial interactions at play as it is of being able to successfully facilitate learning within that group.

In her article, Barbara Heinzen starts off by arguing that futures work has largely failed – the inability to foresee and prepare for recent shocks (financial, economic and political) in spite of the growing ubiquity of futures practice has left many decision-makers upset that such things could disrupt their own finely woven plans. She refers to this as the ‘paradox of successful adoption but failed anticipation’. The article offers a rich collection of anecdotes and advice to futures practitioners to ensure that their work has a lasting impact on decision-makers. Continuing in the same vein as Zaaiman and Lavoix, there are strong overtures to understanding the organizational dynamics, appreciating complexity and its capacity to create surprises, and the need to invest in group learning to ensure that the emergent insights are properly embedded within the organizations in which they are being applied.

In many organizations and institutions, foresight work is still not taken seriously and is treated with the same respect reserved for arcane, magical rituals. As Claire Nelson recounts in her article, her attempts to introduce foresight practice into a large development bank were considered ‘outlandish at best, outlawish at worst’. The extent to which the development industry (development banks, large NGOs, governments and private sector stakeholders) are caught in the development aid trap is, in her view, a function of the inability of these actors to rethink their role and the outcomes they seek in a rapidly changing world whose complexity we are no longer able to grasp within the standard, simple models that have served us hitherto. Nelson suggests that the inability to create truly learning organizations and encourage cross-fertilization of ideas contributes immensely to this trap. She calls for the development industry to adopt foresight as a central plank of its modus operandi as this would allow it to have a greater appreciation of the challenges it faces, while at the same time providing greater space for sharing the knowledge and wisdom of all involved in the process – particularly as we now embark on the quest to attain the Sustainable Development Goals. For this to succeed, she astutely observes, it will perhaps be necessary to change the DNA of the industry and to dismantle the existing silos of experts that discourage insight and foresight – a difficult, but not impossible task.

Foresight studies have been applied to several global challenges with varying results. One sector where these have been applied extensively is in the field of science and technology. The natural ease with which researchers interact with complexity and data is perhaps one reason for its relative proliferation in this sector. Siwa Msangi, in his article, introduces us to a number of foresight studies around environment and agriculture and assesses their impact over time. Studies that churned out massive amounts of data have not always been successful – as is the case of the Millennium Ecosystem Assessment that proved to be ‘too technical’ for many decision-makers. Continuity is an important point that is also brought out – as exemplified by the Global Environmental Outlook and Intergovernmental Panel on Climate Change processes that are slowly gaining traction beyond the scientific community and are informing debates and decisions around the world. He argues for the need to ‘domesticate’ global studies and make them more accessible and relevant to local decision-makers and support these country studies to adapt them. He promotes the ‘Foresight Hub’, a network of practitioners and field experts, as a key support to this process, providing technical and contextual support as needed.

Mario Bazán and his colleagues give us an excellent story of the rise, fall and rise of foresight in Peru. Through their contribution, we are brought to understand that the creation of dedicated institutions to produce foresight is not good enough, particularly if these institutions lack the instruments to carry through on whatever insights they might deem relevant for the country. It is perhaps a lesson on the need to ensure buy-in at the appropriate political levels before engaging in this work. In spite of the challenges described in this article, it is also clear that there is a vibrant community interested in exploring futures and its relation to policy choices. Again, this is not an accident but the outcome of a deliberate effort to seed and nurture small groups over several decades spanning academia, the public and private sectors. Yet, and also very instructive, is the fact that a critical mass has not been reached to enable total institutional buy-in. The efforts described aimed a democratizing futures work, and its application in helping a marginalized community expand its social and economic opportunities are also instructive in underlining the value of foresight beyond the corridors of power. Francisco Mojica and his colleagues describe their efforts to retrofit development plans in a region of Colombia, updating them with state of the art information and technological advances. This work leads to the definition of a set of scenarios, one of which is chosen as a ‘guiding vision’ and for which a set of strategies and actions to see it through have been devised. While the fruits of their work have yet to mature, it is clear that where there is leadership that enables key stakeholders to talk across institutional and sectoral boundaries, interesting things can happen. We are, however, left wondering whether the transformation envisaged by this vision will really take place given the narrow time-span allocated for this purpose – six years. Perhaps we might still learn something out of this ambitious experiment.

Policymaking is inherently a political process and as such, is bound to be subject to political jostling of one form or another. Futures work cannot claim exception to this, particularly when the stakes are high. Even tackling poverty is not immune from attack by those who would argue for, or against certain interventions. As Leon Fuerth, a veteran of many policy battles aptly put it:

[T]he reason that foresight is so often unable to influence policy has relatively less to do with the presence or absence of superior information, than it does with the zeal and mendacity of those who do not scruple to smother the truth with data. That’s political battle, and it is what goes on at the ‘front line’. It is not fated to be a losing battle, but the outcome is never certain and issues of policy are never permanently settled.

When tackling issues such as climate change, water use or even inequality, all that contribute to poverty, there will always be contestation around what the ‘right thing’ to be done is. And perhaps the challenge here of the foresight practitioner or ‘advisor’ is that of trying to assist the contesting parties to appreciate the different mental models at play and to work towards strengthening a common ground between them.

In her article, Anumita Raj shares with us the experience of her institution in using foresight processes to help build bridges between governments with a history of frosty relationships. By involving and assisting the key political players in a foresight process around transboundary water resources, an important treaty on shared water resources has finally been signed and the new-found rapprochement is contributing to improved bilateral ties between India and Bangladesh. Her insights provide us with the value of such processes in helping strengthen common ground between competing and often divergent interest groups.

In their article, Catherine E. Cosgrove and William J. Cosgrove talk about ‘the tyranny of the immediate’ in which long-term goals and needs are sacrificed at the altar of the short-term, amply fed by short news cycles, political expediency and an impatient public. As such, politicians put off for as long as possible, making decisions today that could lead to a decline in their own popularity, even if doing so could lead to detrimental effects over the long-term. It is not immediately clear how to reverse this political inertia.

Indeed, the Cosgroves’ – reflecting on the World Water Vision process – conclude that we can only ‘hope in the assumption’ that foresight work that is interdisciplinary and robust with broad stakeholder participation will generate the desired outcomes. Leon Fuerth is more direct. He suggests that the poorest and wealthiest find common ground on which to approach the future, granted that we now ‘lack the self-restraint needed to control our destructive impulses, not just in the short term, but over very long time periods’.

A contrasting viewpoint to this emerges from Steve Kayizzi-Mugerwa, who examines the political economy of Africa’s transformation. African countries are attempting to catapult themselves into a brighter future, hoping to attain middle-income status in the coming generation. At face value, there is much to be sceptical about in this assertion given the depth of structural problems that the continent faces. Nevertheless, the optimistic language and imagery of the numerous ‘Vision’ documents that are being launched indicate the extent to which the transformation debate has captured the imagination of the continent’s political leaders. Africa’s transformation is undoubtedly inevitable although some might debate its trajectory. Much as there is clearly some top-level political buy-in for a transformation agenda, it is unclear the extent to which these ideas and agendas resonate with the broader public. Furthermore, the trade-offs that will undoubtedly have to be faced have not been negotiated with citizens. Public participation and ownership of these plans cannot be discounted. Citizens have a massive stake in the futures of their countries. They should be ‘counted in’ and involved in these processes as plans are made.

This point is echoed by Jacob Olugbenga Adesida in his reflection on his experience working with African governments under the umbrella of the now-defunct ‘Africa Futures’ programme established to implement futures work in African governments. Implementing foresight was not easy given the struggle to capture the attention of decision-makers and the perception that the concept was an external imposition. He suggests that African governments need to democratize decision-making and find ways of accelerating system-wide learning.

In their reflection, Alioune Sall and Geci Karuri-Sebina identify differences in expectations, language and perceptions of time between foresight practitioners and their policy-making interlocutors as obstacles to its deeper effectiveness in shaping policy in Africa. Unsurprisingly, the seriousness of individual policymakers in engaging with foresight and acting on its insights matters a great deal as demonstrated by examples of Mauritius and the city of Johannesburg. They argue that ultimately, regardless of whether every individual initiative is effective, doing foresight regularly is as hygienic, for society, as brushing one’s teeth is for individuals.

In the final analysis, the temptation to dismiss foresight processes as peripheral to decision-making might still remain strong in the mind of many readers. But like any methodological approach, the quality of the answers that emerge depends on the quality of the input into the process. The experiences shared by the practitioners in this journal suggest many of the vital ingredients for success. In our view, for foresight to work to be influential, it has to be practiced and implemented properly – like most things in life, there are no shortcuts. There is no single ‘right’ way of doing foresight work. A good combination of the various different tools and approaches can assist users to have a deeper and better understanding of the problems that they are called upon to address and the opportunities that beckon. Embedding learning and an appreciation of the insights emerging from any foresight process is a function of many elements. Above all, one cannot discount the time it takes for new ideas to be assimilated and settle into the system.

Foresight will not provide all the answers that are sought, but it can help to shape the agendas and ensuing conversations for leaders and decision-makers. It can help citizens understand the broad challenges they face as well as opportunities on the horizon.

Finally, and by way of ending this introduction, we would like to pay tribute to the Rockefeller Foundation – and its Managing Director for Research, Claudia Juech in particular – for a fabulous partnership that has enabled us to not only complete this enriching intellectual journey, but also for their stubborn conviction that there is a role for foresight in the public policy space. Their work in pushing ‘Pro-Poor’ foresight work has been, in our view, critical in bringing home to many policymakers (and consumers) that future research methods can successfully be used in the development sector and that in so doing, they provide unique opportunities for stakeholders to establish what roles they want to play in addressing the (growing) gaps between the wealthy and the poor around us. But when all is said and done, the lasting contribution perhaps will be that the Foundation dragged foresight from the shadows and increased its profile within the policy space and contributed immensely to beefing up capacity of various actors to use the tools of foresight with increased confidence, and to create a not-so-insignificant network for mutual support, experience sharing and professional growth.