Author: Konstantinos Vadratsikas

Published: August 26, 2022

“Evaluation is creation: hear it, you creators! Evaluating is itself the most valuable treasure of all that we value. It is only through evaluation that value exists: and without evaluation the nut of existence would be hollow.”

Friedrich Nietzsche

Every project, programme or intervention is designed with some objective(s) in mind. Workhours and resources are spent, tasks are completed and then it is time to answer the question “Have we achieved our objective?”. But the answer is rarely straightforward. Ambitious endeavours are usually too complex, to be able to understand at a glance whether their targets have been accomplished. Many factors may contribute to their success (or failure) and their outcomes may take time to materialize.

In fact, evaluating a project’s success is often an objective on its own, one of great importance too, as without proper evaluation methods we can never be sure of what was achieved and whether it was worth the effort. Therefore, it is no surprise that impact assessment is increasingly gaining importance, as policy-makers, investors, shareholders and all other kinds of stakeholders are becoming aware of its significance.

Impact assessment can be distinguished from other evaluation approaches by its focus on measuring change, rather than tasks and outputs. However, impact assessment is not a methodology or a tool. It can be accomplished through various methodologies, and the tools it employs are always adapted to the particular needs of the task at hand. In that respect, there can be no ready-made solutions for designing an impact assessment plan. Instead, each endeavour requires its own impact assessment approach, methods and tools, which are selected through meticulous analyses of the its specific requirements.

Nevertheless, although it is not possible to design one-size-fits-all impact assessment plans, there will always be rules of thumb for assessing the impact of projects with similar structure, scope or objectives. This article reviews four points that emerged while designing an impact assessment plan for the AfriConEU capacity development programme, and which may be applicable to other efforts with similar structures and objectives.

The AfriConEU project aims at empowering Digital Innovation Hubs (DIHs) by developing the first Trans-continental Networking Academy for DIHs. Two flagship programmes are implemented by the Academy, focusing on raising the capacity of DIHs to overcome challenges and establishing partnerships for joint action, respectively. The AfriConEU capacity building programme, which is the focus of this article, offers a series of free,

online and hybrid, training activities for DIHs with the purpose of teaching them innovative ways of conducting business, attracting financial resources and supporting their communities. The programme is currently underway and will be completed in July 2023, although training resources will be freely available on the AfriConEU website even after the end of the project in December 2023.

Tip #1: Clarify overall and specific objectives

Impact assessment is about measuring change, but not all change is relevant or coveted. Knowing what to expect, as well as how to identify unrelated changes and unforeseen effects of an intervention, is a crucial component for any attempt to identify change and understand its causes. In particular, impact assessment of capacity development programmes, which encompass a multiplicity of activities and objectives, feels like a shot in the dark unless we are aware of what each activity is expected to bring about or not.

But, how is this kind of understanding reached? The answer to that question lies within the project’s declared objectives themselves. People who design and implement capacity development programmes are usually experts in their fields. Every activity, task or assignment that is incorporated in the programme, is intended to have specific effects, all of which contribute to reaching the programme’s objectives. So, carefully study each objective stated in the programme’s description and try to identify how it’s addressed by the programme’s activities. But Beware! Overall objectives are often comprised by smaller targets. So, always keep this in mind, examine every task and understand how it contributes to the project’s purposes.

Tip #2: Focus on outcomes to measure actual change

Project objectives are often stated in an inclusive and abstract way. Especially in projects that involve training activities, such as capacity development programmes, much of the focus is often on what participants will learn to do, rather on what they are expected to achieve. That’s a real nuisance when it comes to measuring impact. It is not unusual that a project reaches its objectives but the impact is minimal.

Thus, pay close attention to every described target and try to envision how this will be reflected in the participants’ performance. Try to picture how it will affect their behaviour and then translate it into measurable targets. If you are uncertain or not very familiar with the programme’s study field, consult with the experts who designed the programme or ask them to clarify what they are trying to achieve in practice, with each of the targets they set. This way you will avoid designing improper tools and measuring irrelevant variables that reflect accomplished targets but null impact.

Tip #3: Get everyone on board

Assessing the impact of elaborate programmes, involving numerous activities and interrelated objectives, like capacity development programmes, is a quite laborious task and entails various risks. Collaboration is essential, as all involved parties are expected to contribute at some point of the procedure. Consulting with the experts who were in charge of designing the programme is only one aspect of the input that is required. The involvement of coordinators, trainers, facilitators as well as participants may also be required at certain points of the process (depending on the tools and methods applied in each case). In that respect, several things can go wrong (from failing to collect sufficient data, to collecting data that are completely irrelevant or skewed) and influence the outcome of impact assessment.

So, make sure you communicate the requirements of your impact assessment plan with all involved parties to ensure that the process is clear and always have a contingency plan. Ideally, prepare and present your methodology to everyone and run pilot assessment sessions, before the actual implementation phase begins.

Tip #4: Follow-up

Impact does not always become instantly visible. In fact, when dealing with capacity development endeavours, the programme’s effects may take months to materialize and begin to cause actual change and affect outcomes. Shifts of established work practices require time and learning to use new tools or skills needs practice to be perfected.

Be patient! Plan short-term, mid-term and (if possible) long-term impact assessment activities. Follow-up accordingly and utilize your assessment data to advice on support activities, in case your findings suggest so. Finally, keep everyone aware of what you discover each time you complete a round of your assessment. People will not intensify their efforts if they do not know they need to.

[NOTE: It should be noted that the points discussed in this article are also relevant for assessing the impact of AfriConEU’s partnership programme. Nevertheless, impact assessment of networking activities exhibits its own particularities, as it involves activities, which require different evaluation techniques that go beyond the scope of this article.]