ToR for External Evaluation of Ethiopia Joint Response (EJR)

Background of the Joint Response

This TOR is developed by SOS Children’s Villages for conducting an external evaluation of the Dutch Relief Alliance (DRA) – Ethiopia Joint Response (EJR) Program financed by the Netherlands Ministry of Foreign Affairs (MFA). The Ministry through Dutch Relief Alliance (DRA) granted €18.5 million to Ethiopia joint response to address protracted crisis with a duration of three years started on the January 2024 until the 31st of December 2026.

Led by SOS Children`s villages the Netherlands, Ethiopian Joint Response (EJR) program has been implementing a multi-sectorial and complementary humanitarian response. Since January 2024, EJR project has been responding to the need of conflict and drought affected peoples of Oromia (East Hararghe – Midhega Tola and Borana- Dubluk Woredas) and Amhara (North Wollo – Gidan and Wag Himra – Ziquala Woredas).

Now, Ethiopia Joint Response (EJR) project is under implementation, looking to harvest the fruits of its two years intervention, demonstrate results and incorporate finding into the upcoming new DRA strategy (beyond 2026). This necessitates well organized and sound evaluation to be conducted by experienced and independent consulting firms so that the effectiveness, efficiency, relevance, suitability and impact of our intervention will be therefore, easily measured and identified in terms of meeting its objectives. Moreover, the external evaluation will be conducted to assess the extent to which the DRA priorities: Accountability, Localization, Collaboration and Innovation principles applied during the project implementation and the barriers that partners were facing to meet the DRA priorities.

Purpose of the Evaluation

The purpose of the evaluation is to provide a comprehensive assessment of the results of the EJR, this is done in order to learn from the implementation and to ensure downward and upward accountability. Evaluation will cover the period of two years, January 1, 2024 to 31 December 2025 and expected to start by the beginning of 2026. Thus, the evaluations aims;

• Objective 1: Results achieved: The final evaluation will assess the overall performance of the EJR against selected OECD DAC criteria.
• Objective 2: To identify the contribution of the EJR to the strategic priorities of the Dutch Relief Alliance (Accountability, Localization, Collaboration and Innovation/learning).
• Objective 3: Assess the degree to which the EJR intervention complied with the Core Humanitarian Standards (CHS) and Sphere standards. In addition to this, the consultant need to review and combine the RTR findings.
• Objective 4: To identify and record lessons learned, best practices, strengths and provide recommendations for improvement.

 

Scope of the Evaluation

The final evaluation survey will be conducted with the objective of demonstrating results among the EJR partners operating in Gidan Woreda of North Wollo Zone, Ziquala Woreda of Wag Himra Zone of Amhara region, Midhega Tola Woreda of East Hararghe Zone and Dubluk Woreda of Borana zone Oromia region. The sectors addressed are Food Security & Livelihoods, MPC, Health, Protection and WASH. Protection is an important element, and partners are expected to mainstream protection throughout the activities. The project interventions target IDPs, returnees and vulnerable host communities. In addition, the response carries out activities in the areas of Localization, Collaboration, Innovation/learning and Accountability (these are strategic objectives defined by the DRA and be found in the log frame).

• The first objective refers to the success demonstrated against OECD DAC criteria focusing on Relevance, Coherence, Effectiveness and Efficiency.
• To identify the contribution of the EJR to the strategic priorities of the Dutch Relief Alliance (accountability, localization, collaboration and innovation/learning).
• The third objective measures the quality of EJR performance in terms of meeting the nine Core Humanitarian Standards focusing on Participation and inclusion, Effectiveness and timeliness, feedback and concerns and collaboration and coordination.
• To identify and record lessons learned, best practices, strengths and provide recommendations for improvement.

Objective 1 Results achieved: The final evaluation will assess the overall performance of the EJR against selected OECD DAC criteria and the performance indicators of the EJR projects.

Factors	Key Questions
Relevance/ appropriateness	To what extent are the objectives of the project valid? Was the design of the program most appropriate and relevant to the beneficiaries’ needs and priorities? To what extent was the project able to adapt and provide appropriate response to context changes and emerging local needs, and the priorities of beneficiaries?   In general, the consultant needs to address the question of “Was the overall project design relevant in the local context and for the intended beneficiaries (communities and local government)? If so, how? If not, why not? Were changes made during project implementation to increase the relevance of the activities? If so, how? If not, why not?
Coherence	How well does the intervention fit? Was the intervention consistent and synergistic with other interventions carried out by EJR Partners and local actors, as well as with relevant international norms and standards? If so, to what extent and how? If not, why not? To what extent has the project contributed to the overall goal? Was the project effective in delivering integrated live-saving assistance and responding to humanitarian needs? How effective was the approach used in the implementation of the project? Has the project achieved the intended outcomes in a timely manner? If so, to what extent and how? If not, why not. Were the resources allocated in the project appropriate to the scale of the project? Were project activities implemented in an efficient manner? Was the EJR approach focusing on limited number of Woredas/Kebeles and multiple service to the needs of the population efficient?
Effectiveness (including reach)
Efficiency (cost-effectiveness)

 

Objective 2: To identify the contribution of the EJR to the strategic priorities of the Dutch Relief Alliance (accountability, localization, collaboration and innovation/learning). This objective should address the CHS commitments (Commitment 1, 5 &6)

Objectives	Key Questions
Accountability to affected people (AAP)	Assess how the project was designed to maximize accountability toward the affected population?
Participation	How were the crisis-affected people (including vulnerable and marginalized groups, and people with disabilities) involved in the design and implementation of the project? How were their views used to guide decision-making? How are you involving community members in decision-making processes, and what mechanisms exist to ensure their voices are heard? Did the JR ensure that affected communities were able to exercise their rights and participate in decision-making processes? If so, to what extent and how? If not, why not?
Feedback and Complaint	What feedback and complaint mechanisms do you have in place, and how do you ensure that communities understand how to report concerns? How was the feedback collected, and what was done to ensure that the data collected was tracked, analyzed and incorporated and ensured that adjustments were made as a result of received feedback? Can you show us documented evidence of complaints collected so far, and how many of them were addressed? Which of these are sensitive complaints?
Inclusion	How are diversity, equity, and inclusion considerations integrated into the design phase as well is during implementation in your work with communities, especially for the most marginalized (e.g., PWDs, women, and children)?
Coordination and Collaboration	How did EJR partners coordinate their activities with other external actors, including government, other (I)NGOs, UN agencies? How has coordination across the different sectors (FSL, WASH, Health, Protection, MPC) been managed during the implementation of EJR? How are your activities coordinated with other EJR partners and government stakeholders? How do you ensure equitable decision-making and resource-sharing within partnerships? How do joint efforts, such as resource-sharing and joint field activities, vary between sectors? For instance, are specific sectors benefiting more from shared resources like vehicles or staff expertise?
Learning and Innovation	To what extent has the EJR facilitated peer-learning between EJR partners? What kind of learning activities have been most effective according to EJR partners? Is there any substantial anecdotal evidence on how activities to increase learning have affected the delivery of humanitarian aid by EJR partners? Did partners make any changes to their programming because of learnings?
Localisation	Did the capacity strengthening efforts of the EJR meet the needs of the local partners? Did the Crisis Modifier for local partners enable the local partners to design and manage their own quality response programs and respond rapidly to the identified crises? How did the EJR support local partners in raising their voice in relevant national and international fora? To what extent did the localization efforts of the EJR contribute the broader localization priorities of the humanitarian system in Ethiopia?

 

Proposed Methodology

As part of his/her assignment, the consultant will provide detailed planning on his/her proposed methodologies based on these Terms of Reference. SOS-CV (NL and Ethiopia) will review the planned methodologies proposed by the evaluator and provide feedback before the evaluation process begins.

The evaluation will be carried out in a transparent and participatory manner by involving relevant stakeholders (EJR members, local government and affected communities/program participants). A mixed-method approach is anticipated including, but not limited to, the following methods: –

1. Mixed qualitative (Desk study and review, Key informant interview, focus group discussions) and quantitative methods (structured questionnaire/household survey) are required. Methods can be determined together with an evaluator but finding stories of change as well as validation of outputs in the log frame is mandatory.
2. Field work (collecting data) in all project areas, if the situation getting improved in one of our project locations which is fully controlled by the UAGs
3. Links to resource portals to be included

The evaluation should be inclusive, taking into account gender, age, disability, and other vulnerability considerations.

Roles and responsibilities

JR Coordinator and Lead

• Lead on the overall evaluation process.
• Coordination of the whole Evaluation, including with JR partner organizations to schedule and organize data gathering visits.
• Lead on procurement process of consultancy services.
• Getting approval of the TOR from DRA
• Main focal point and contract holder for consultant with regards to consultancy services.
• Make key documentation, tools and background information available to the consultant.
• Provide feedback on (the inception and) the draft report to the consultant.
• Coordinate feedback from JR partners on the report.
• Writing the management response letter

Consultant:  

• Designing the Evaluation Framework: Developing the evaluation plan, including designing the methodology and the tools
• Data Collection: Conducting field visits, interviews, focus groups, and surveys as well as observations to gather qualitative and quantitative data from various stakeholders.
• Data Analysis: Analyzing the data collected
• Reporting: writing an evaluation report that summarizes findings, conclusions, and recommendations. This includes drafting, reviewing, and finalizing the report as well as providing the deliverables and outputs mentioned under 6.

JR Partners 

• Establish availability and identify tentative dates for the KII and field visits.
• Provide relevant documents to the Lead/review team/consultant.
• Attend validation and learning session/workshop organized by the Lead organization.
• Share the stock of best practices and learning from the organization that will be useful to share/discuss with the consultant.

1. Deliverables and Outputs

Inception report: The Inception report will highlight the methodology and the guiding principles of the evaluation and should reflect a good understanding of the set-up and structure of the EJR. The methodology and practical way of working will be mutually agreed upon between SOS CV and the consultant.

Data collection and data analysis tools: The evaluator(s) will develop the tools for data collection and data analysis in line with the structure of the tools in the inception report. There will be a briefing session on proposed methodologies and tools.

A debriefing and validation session to SOS CV NL and ETH other interested DRA partners to debrief on the major qualitative and quantitative findings before initiating the report.

Presentations and/or learning sessions with stakeholders (recipients, partners, and programme staff). Raw data of the research in MS Excel format.

Final Report Evaluation (maximum 30 pages, annexes excluded, in Microsoft Word format): including tables and graphs representing the data. AN evaluation report should contain action-oriented recommendations (max 2-3 per CHS commitment/Evaluation criteria)

For a detailed table of content,

• Cover page
• Table of Contents
• List of Acronyms
• List of Tables
• Executive Summary (should be a stand-alone document)
• Background of the Ethiopia Joint Response (EJR)
• Purpose of the Evaluation
• Scope of the Evaluation (Geographical, target group and thematic areas)
• Methodology (including sampling): (Mixed qualitative and quantitative methods)
• Main Findings and Learnings
• Conclusions and Recommendations for future programming (Max-10)
• Annexes
  • Project log frame
  • Evaluation ToR
  • Objectives and key questions
  • Methodology
  • Study schedule
  • List of people involved
  • Bibliography of consulted secondary sources
  • Finalized data collection tools
  • Photographs of the evaluation areas (separate file).
  • List of key informants.

Disclosure of information/ethics

It needs to be understood and agreed that the consultant shall, during and after the effective period of the contract, treat as confidential and not divulge, unless authorized in writing, any information obtained in the course of the performance of the contract. The ethics process and research need to comply with specified requirements (e.g. Code of Conduct, Research Policy and Standards).

Professional Skills and Qualifications and selection process

• Lead consultant with an academic degree in International Development, Development studies, Disaster Risk Management, Food Security, Humanitarian Action, or a related field.
• Strong experience in humanitarian response programming and knowledge of humanitarian standards (CHS, Sphere, CPMS, LEGS, SEADS, ADCAP, Code of Conduct).
• Demonstrated experience in leading evaluations of humanitarian response program.
• Experience in working different regions in Ethiopia context (desired)
• Experience with the DRA is an asset.
• Proven experience in Humanitarian program management, including large consortia programs
• Proven knowledge of humanitarian programmes and principles, including use of common standards, preferably in the sectors FSL, WASH, Health, Protection, and MPC.
• Experience in the use of participatory research methodology
• Excellent communication, listening and interpersonal skills.
• Demonstrated analytical and report-writing skills
• Fluent in English, understanding of local language will be an added value
• The consultant/team leader must have demonstrated consultancy track record and be recognized professionally in conducting evaluations/surveys with a high degree of proficiency