Call for Consultancy to Final Performance Evaluation

Project Title:  Emergency Multi-Sectoral Response to Northern Ethiopia Conflict-Affected  Communities.

Duty Station: Amhara (North Wollo Zone) & Tigray (North Western Zone); Ethiopia

Duration of Appointment: 60 days

Expected Start Date: September 2023 – December 2023

1.     Background

1.1.          Background to Samaritan’s Purse Program

Founded in 1970, Samaritan’s Purse (SP) is a non-profit, non-denominational, evangelical Christian organization, specializing in rapid emergency response programs to provide lifesaving assistance and develop the recovery and resilience of vulnerable communities around the world. Samaritan´s Purse has worked for over 40 years in more than 100 countries, providing rapid and protracted responses to manmade and natural disasters with extensive experience across key humanitarian sectors. Samaritan’s Purse work in Ethiopia began in the late 1980s and has continued in different forms for over 30 years. During that time, SP has worked across numerous geographic areas and sectors. Samaritan’s Purse partnered with the Office of U.S. Foreign Disaster Assistance (OFDA) in Ethiopia from 2008 – 2010, delivering critical services to over 25,000 program participants in Gambella Region. Samaritan’s Purse reengaged with direct programming in Ethiopia in 2018, and has active programs currently in Tigray and Amhara Region, including this grant from the Bureau for Humanitarian Assistance (BHA), which started April 2021, and is currently active within the health, nutrition, protection, WASH, and shelter sectors.

1.2.          Background to the Program/Project to be evaluated

On April 2021, SP started implementation of the presently entitled ‘Emergency Multi-Sectoral Response to Northern Ethiopia Conflict-Affected Communities Project’ to address the humanitarian crisis in the Tigray Region (which later expanded to include the Amhara Region). Initially designed as an emergency response, the activity was extended to a 33-month activity that includes emergency assistance as well as supports for post-conflict recovery. Samaritan’s Purse ensures that its services and support are inclusive and responsive to the needs of women, men, girls, and boys, especially the most vulnerable populations.

Theory of Change (ToC):

If SP provides rapid and efficient humanitarian relief to persons impacted by armed conflict using a multi-sectoral approach through the provision of essential medical services; nutrition assistance; protection services; emergency shelter and settlements support; and improve WASH access, then the target population will have transition from their current states of vulnerability to that of improved health and wellbeing, increased resilience, dignity and alleviated suffering.

Program Description:

The project, which is ending on December 2023, includes health system strengthening in primary health centers, provision of mobile medical and nutrition units, mental health and psychosocial support, prevention of gender-based violence (GBV), child protection, construction of emergency and transitional shelters, and comprehensive water, sanitation, and hygiene supports for internally displaced people, host, and returnee communities.

The project goal is to save lives, accelerate recovery, and increase resiliency by providing essential medical services, protection, nutrition, shelter, and water, sanitation and hygiene (WASH) support to vulnerable populations in the North Western Zone of Tigray and in the North Wollo Zone of Amhara. The five sub-purposes aligning with the five sectors include the following:-

1.     To save lives and bridge the gap to health service delivery in targeted areas of Tigray and Amhara;

2.     To protect the lives of the most vulnerable, reduce the risk of GBV, and provide psychosocial support for conflict-affected populations;

3.     To save lives and reduce the risk of malnutrition for children under five years old (U5);

4.     To increase access to emergency shelter for conflict-affected populations; and

5.     To reduce rates of waterborne diseases and increased resilience among conflict-affected populations.

2.     Purpose and Justification for the Evaluation

The evaluation aims to determine to what extent the project’s activity relevant and effective to the affected community. The results will inform BHA and SP senior decision makers, and will support future decision-making. Therefore, it requires information that meets scientific standards of credibility and reliability, which can support action and withstand criticism. This is a summative evaluation that will provide a conclusive judgment on the program.

The purpose of the evaluation is to assess the relevance, effectiveness, and identify key approaches that contribute to the outcomes as well as lessons learned and recommendations that can be taken forward to inform future emergency and recovery programming. The evaluation fulfils SP’s commitments to the United States Agency for International Development (USAID’s) BHA to assess the performance of its activities and achievements against the project’s objectives.

3.     Specific Objectives of the Evaluation and Evaluation Questions

The evaluation is a final/summative performance evaluation of the project’s implementation from April 2021 to December 2023. The evaluation will focus on the following criteria basis on the OECD-DAC criteria as a framework for achieving the intended objectives:

a)    Relevance:

• To what extent did the activity appropriately address the most urgent needs of the affected populations?

b)    Effectiveness:

• To what extent did the project achieve its stated outputs and outcomes?

c)     Lesson Learned

• To identify good practices, opportunities, and challenges that will provide inputs for strengthening this and future emergency and recovery responses.

4.     Scope of Evaluation

The activity evaluation will run for 60 days from the initial kickoff meeting to the submission of the final report, from September 15 to December 25, 2023. After the initial kickoff meeting, it is expected for the evaluator to start immediately with the desk review of project documents and planning, moving into the preparation of the questionnaires, selection of sites for evaluation, liaising with field teams to support in organization of primary data collection, and then moving into cleaning and analysis.

It will end with the production and submission of the final evaluation report. In the evaluation proposal defining the activities, output, and timeframe, the team leader will give a matrix for the evaluation process demonstrating the duration and break down of activities (expected timeline with key dates and deliverables are noted under section 9 below). It is important to note that final intervention activities will be ongoing while the evaluation takes place. The evaluation will be mainly focused on program beneficiaries in Amhara and Tigray region particularly in North Wollo and North western zone respectively. The end of project evaluation will provide a detailed analysis of achievements against the anticipated objectives (past baseline values, semi-annual reports) and measure up-to-date performance of the project. This aims to showcase the project’s relevance of the interventions, Effectiveness, and lesson learned of the project intervention towards future sustainability.

The team leader should also have a response plan that includes adhering to security and safety standards, protocols, and procedures as well as briefings on the road and security situation even though there are no overt, obvious concerns yet. Samaritan’s Purse is aware that survey responses can be arbitrary. With this information, SP will monitor how the team lead trains data collectors and employs the appropriate mitigating techniques to reduce the likelihood of response bias. Moreover, the team leader can fine-tune enumerator abilities during pilot test interviews, and end-of-day reviews will be used to gauge progress. In order to guarantee consistent data collection and uniform questionnaire understanding, team leader also regularly monitors the enumerator’s capacity. By doing this, data recording errors will reduced.

5.     Evaluation Methodology

The evaluation will use a non-experimental design because the base line data is not sufficient for a rigorous comparison of the outcomes before and after the intervention. At the time the baseline was conducted the project was envisioned to be completed less than a year, thus the baseline was conducted as per BHA’s recommendation for non-rigorous assessment as the life time of the project was assumed to be short. However, the project was extended more than twice because of the prolonged conflict in the region.

This evaluation will employ a mixed-methods approach that combines quantitative and qualitative data collection and analysis techniques. The quantitative data will consist of surveys, administrative records, and indicators of performance and outcome. The qualitative data will consist of interviews, focus groups, observations, and document review. The mixed-method approach will be employed to integrate quantitative and qualitative data to answer the evaluation questions. This will provide a more comprehensive and holistic picture of the evaluation context, process, and outcomes than either method alone. The evaluation will employ a careful planning and integration of the data collection, analysis, and interpretation stages and it will use sequential mixed methods design to ensure the method is aligned to the purpose, priority and timing of the data collection and analysis. To elaborate more about the analysis requirement, the Evaluator will make sure that data analyzed quantitatively from the household surveys using standard package software to generate descriptive statistics like frequencies, proportions, and means.

Thematic analysis, using key themes related to the evaluation objectives (by grouping responses by themes), will be employed for qualitative data analysis. Moreover, the evaluator will produce confidence intervals for all point estimates from samples and the expectation for advanced statistical analysis conducting significance testing, regression analysis, etc. where appropriate. When testing for differences between baseline and endline values, Annual/Semi-annual, etc, a two-sample t-test can be used for the type of data being analyzed. The data analysis includes triangulation of quantitative data with qualitative findings as well as data from primary sources with secondary data. Generally, the evaluator will analyze, interpret, and present the data in a way that is easily understandable for readers. Regarding to disaggregation requirement, the evaluator should apply in accordance with BHA indicator guide or Performance Indicator Reference Sheet/PIRS/.

After data analysis, the evaluator will prepare a draft evaluation report based on USAID Evaluation Report Template. The Evaluator is also expected to present findings using a variety of presentation methods and capture key discussion and use inputs generated to update the study report. Besides the final report, the team leader is required to prepare a summarized version of the report and a power point presentation with summary charts, tables and maps for dissemination of the findings of the evaluation to external stakeholders such as government, donors and others.

In addition to answering the evaluation questions above, SP will also be requesting the evaluator to include key results indicators from the indicator-tracking table (ITT) in their evaluation in accordance with the BHA PIRS for quantitative survey and the rest of the performance monitoring indicators that have been collected by the activity can be considered using qualitative methods to substantiate the analysis when needed.

The indicators requested to be included in the evaluation include the following:

Indicators to be analysed during Performance Evaluation		Target group
H15	Number and percent of community members who can recall target health education messages	Individual Health & Nutrition beneficiaries living in the intervention area
S05	Number and percent of beneficiaries reporting satisfaction with the quality of the NFIs received	Individual beneficiaries who benefited from BHA funded S&S project
S10	Percent of settlement beneficiaries who believe settlement interventions met or exceeded expectations
W09	Percent of households targeted by the hygiene promotion activity with no evidence of feces in the living area	WASH beneficiaries in the intervention area
W10	Percent of individuals targeted by the hygiene promotion activity who know at least three (3) of the five (5) critical times to wash hands
W11	Percent of households targeted by the hygiene promotion activity who store their drinking water safely in clean containers.
W13	Number of individuals directly utilizing improved sanitation services provided with BHA funding
W18	Percent of households targeted by latrine construction/promotion activity whose latrines are completed and clean
W19	Percent of latrines/ defecation sites in the target population with handwashing facilities that are functional and in use
W20	Average number of users per functioning toilet
W22	Percent of excreta disposal facilities built or rehabilitated in health facilities that are clean and functional	Direct observation of health facilities
W33	Percent of households targeted by the WASH activity that are collecting all water for drinking, cooking, and hygiene from improved water sources	WASH beneficiaries in the intervention area

Sampling strategies:

Regarding the sampling strategies, the evaluator will be expected to propose an appropriate and recommended sampling methodologies and techniques as per USAID-BHA Feed the Future/FtF/ sampling guide and Performance Indicator Reference Sheet/PIRS/ for each indicators. SPE will then further refine the proposed methods during the evaluation design phase. That will be mostly a representative target-based survey for Health & nutrition, Shelter & Settlement, and WASH sectors for quantitative survey and interviews; focus groups; observations; and document review for qualitative approaches.

6.     Stakeholder Involvement

Stakeholder participation is fundamental to SP evaluations. The consultant is expected to conduct a participatory evaluation providing for meaningful involvement by project partners, beneficiaries and other interested parties. Stakeholder participation is to be an integral component of evaluation design and planning; information collection; the development of findings; evaluation reporting; and results dissemination.

Key stakeholders in evaluation include: IHQ (Award & program management team), BHA headquarters and disaster assistance response team/DART/, implementing international and local partners, SP field and country office staff (Deputy Country Director program/DCD_Program/, Chief of Party/CoP/, PDO, M&E, program TAs), and local community members and others.

7.     Management of the Evaluation

The governance and management arrangements to carrying out this evaluation plan will be carried out by the evaluation mission team i.e. team lead, representatives from SP and if possible, from donor. When necessary, the DCD_Program, CoP, Area managers, Program Development Officer, and M&E will be on hand to provide advice, support, and to set up any decisions that may be required. Additional participants in validating the results include the Award and Program management team, beneficiary representatives, sector leads from government offices, and key project staff. The aforementioned team will review and approve the evaluation plan as well as the evaluation’s subsequent deliverables. Both parties will implement their portions in line with the contract agreement, as specified in the duties and responsibilities clause.

8.     Operating Principles and Values

The evaluator needs to apply standard ethical principles during the assignment. The evaluator will be responsible for ensuring that confidentiality is maintained and that a safe and respectful environment is provided during any interactions with service users/beneficiaries, interviewees, and staff. The evaluator must share with Samaritan’s Purse Ethiopia their confidentiality policy prior to engagement. The evaluator must also share how the data quality, data management, and data security will be adhered to during the evaluation to ensure collected, processed, and raw data deliverables. The incumbent shall not either during the term or after termination of the assignment, disclose any proprietary or confidential information related to the service without prior written consent of the contracting authority. Proprietary interests on all materials and documents prepared by the contract holder under this assignment shall become and remain properties of Samaritan’s Purse Ethiopia.

9.     Evaluation Timeline and Deliverables

The evaluation activities will run for an estimated 60 days and will evaluate the entire life of the project up until the period of evaluation. Required deliverables are the following:

a)     Secondary Data:Review of secondary data from varied and relevant sources that must be specified in the consultant’s proposal, including, but not limited to, relevant documents of Samaritan’s Purse Ethiopia. However, the evaluator should also provide other key secondary documents to be used relevant to the target localities. Activity documents available for evaluators include:

• Proposal documents (technical proposal narrative, budget, work plan, monitoring and evaluation plan, etc.
• Baseline reports
• Program reports to date
• Documents as requested by the evaluator

b)  The Inception Report: This will be required within eight workdays of the contract award; the consultant must submit a detailed inception report that shall detail the evaluation design and operational work plan, which must include the proposed data collection and analysis methods to address the key questions of the evaluation. The inception report shall also include questionnaires and interview protocols and must not exceed 15 pages.

c)    Evaluation Design: This will include the development of tools, data entry, and data analysis plan for designing the entire evaluation study.

d)     Tools and Methodologies: The evaluator will be expected to share the tools and methodology such as final surveys or data collection materials, codebooks, data cleaning guides, etc.) And the supporting documentation ensures a clear tracking system to confirm the recommended methods and approach that are followed to be employed in this evaluation with Samaritan’s Purse Ethiopia, and incorporate feedback provided if any.

e)     Data Quality: The evaluator will ensure that throughout the process of data collection, all necessary measures are undertaken to ensure that data quality management is adhered to. The proposal must clearly outline these quality control measures. The evaluator should also propose a digital data collection methodology that will be used. All datasets and syntax files used in software analysis packages will be submitted to the organization.

f)     Data Analysis and Interpretation: The evaluator will have to ensure proper analysis of all quantitative and qualitative data/information collected and ensure the feedback mechanism is defined. Data needs to be disaggregated by gender and any other social-economic categories as applicable. Moreover, the evaluator should prepare Indicator definition tables referencing which data from the collection tools will be used to calculate the selected indicator, how they will be calculated and identify the correct responses required to calculate the indicator, Codebooks and data cleaning guides, and Data entry templates (including validation rules and conditional formatting to reduce data entry errors)

g)       Regular Coordination: Throughout the study, the evaluator will maintain regular contact with the Samaritan’s Purse Ethiopia team contact person to ensure the organization is aware of progress throughout the evaluation period.

h)     Dissemination of results:The evaluator, with support from Samaritan’s Purse Ethiopia will conduct three dissemination meetings. The first meeting may be held with lead activity staff such as Country Director, Deputy Country Director – Programs, Chief of Party, and the M&E Manager. The second meeting will be virtually with key staff from across activity locations including Program Managers, Technical Leads, and Area Coordinators. The final meeting may include the donor, Samaritan’s Purse headquarters, and any other stakeholders, subject to change. Feedback from these meetings will be included in the final report that will be published and shared with all stakeholders. The report will be in English.

i)     Evaluation Report: The evaluator will submit detailed and accurate reports (draft and final) as per the evaluation objectives to Samaritan’s Purse Ethiopia within the agreed timeframe, in a specified format outlines below in section 14. Besides the final report, the team leader is required to prepare a summarized version of the report and a power point presentation with summary charts, tables and maps for dissemination of the findings of the evaluation to external stakeholders such as government, donors and others.

The evaluation is expected to begin no later than September 15, 2023. This date and those below for the key deliverables under this consultancy may be adjusted in the final agreement that will be developed later in consultation with the evaluator hired. The final evaluation report and summary and/or infographic brief(s) should be submitted to the M&E Manager no later than December 25, 2023.

Due dates for key deliverables:

–       Kickoff meeting: September 15, 2023

–       Draft inception report and plan: October 1, 2023

–       Final inception report and plan: October 15, 2023

–       Tool and site selection: November 1, 2023

–       Field work: November 1-30, 2023

–       Presentation of the draft evaluation findings and first meeting: December 1, 2023

–       Draft of the final evaluation report and infographic brief(s): December 8, 2023

–       Meeting two and three conducted with key stakeholders: December 8-15, 2023

–       Final evaluation report and infographic brief(s): December 22, 2023

Note: In the event of any delay, this must be communicated to and approved by the M&E Manager and a revised plan must be submitted to ensure final deadlines are met.

10.  Evaluator Qualifications and Competencies

Evaluation Team Leader Profile

Samaritan’s Purse Ethiopia seeks one high-level evaluator with vast experience in conducting performance evaluations for humanitarian activities especially in the specified projects above. Writing samples may be requested that will only be used to evaluate the applicant’s experience. Below is a key profile that Samaritan’s Purse Ethiopia seeks:

–       The evaluator should have a master’s degree in humanitarian/development studies, social sciences, or economics, or related field or a bachelor’s degree with relevant experience in place of a master’s degree.

–       The applicant must have minimum three years’ experience conducting evaluations in Ethiopia (desirable to have experience with USAID/BHA), using qualitative and quantitative approaches including but not limited to: surveys (population based or target group based), focus group discussions, key informant interviews, document observation, service delivery assessments, and perception surveys.

–       Excellent research skills, including the ability to collect, collate and analyze large amounts of qualitative data and identify critical aspects to succinctly communicate complex subject matter (in written and oral form) to make it accessible to wider audiences.

–       Proven experience in designing evaluation methodology/tools, data analysis like experience in statistical analysis of complex datasets using statistical software, report writing, etc.

–       Capacity to work collaboratively with multiple stakeholders.

–       Strong analytical, presentation, and writing skills in English; knowledge of Amharic and Tigrigna languages is a plus.

11.  Evaluation Budget and Payment Schedule

The project has set aside a budget and the payment schedule will be discussed with selected applicant and will be included in the contract.

12.  Agreement of Terms

Samaritan’s Purse Ethiopia reserves the right to accept or reject any proposal received without giving reasons and is not bound to accept the lowest, the highest or any bidder. Only the successful applicant will be contacted:

Additional Information:

• Selected applicant shall be required to sign and abide by the Samaritan’s Purse Safeguarding Policy, including prevention of sexual exploitation and abuse policy and Code of Conduct. Selected applicant shall abide by Samaritan’s Purse data privacy/management policies.
• Authorized to work in the country and has a renewed license
• Tax and insurance including country specific external consultant tax laws when needed.

13.  Data ownership and confidentiality

All datasets and documents passed to the consultants remain the property of SP and are required to be considered under the strictest confidence. The datasets and documents must not under any circumstance, be shared with a third party without prior SP IHQ approval.

14.  Application Process

Minimum Components and format of Final Report

The final report must include at minimum the following components, as outlined in USAID Evaluation Report Requirements and abbreviated below:

–       Title page, including identifying the evaluation as a performance evaluation

–       Abstract

–       Table of contents

–       Abbreviations/acronyms page

–       Executive summary

–       State the purpose of, the audience for, and anticipated use(s) of the evaluation

–       Describe the specific activity to be evaluated

–       Provide brief background information

–       State the evaluation questions

–       Describe evaluation methodology for data collection and analysis

–       Describe limitations of the evaluation methodology

–       Include evaluation findings and conclusions

–       Include actionable recommendations based on evidence

–       Ensure that all evaluation questions in the SOW are addressed

–       Include the annexes as described in USAID evaluation Report Requirements

–       Include translated tools or material used for evaluation

–       Annexes

Roles and Responsibilities

Samaritan’s Purse Ethiopia:

Samaritan’s Purse Ethiopia will be responsible for the following:

–       Technical review of methodologies, tools, and reports before these are implemented/shared.

–       Communication to field staff and relevant authorities.

–       Facilitating all logistics support in the field.

–       Costs: The organization will cover costs for enumerators as well as operational cost throughout the evaluation process including transport (air and vehicle support on the ground) and facilitating visits to activity locations. Along with this, SP will cover the costs of accommodation based on the accommodation rate at the SP compound in Shire and at the standard hotel SP uses in Amhara while in the field and provide working space and basic office/administrative support in Addis and field offices as needed.

Evaluator/Team Leader:

The evaluator will be responsible for the following:

–       The evaluation plan, the evaluation process and the dissemination of the evaluation findings are the main tasks that the evaluator should undertake and lead. The evaluator has a high degree of autonomy and initiative in designing and conducting the evaluation, as well as in communicating the results to relevant stakeholders.

–       To accomplish this task, the evaluator is responsible to define the criteria that will be used to assess the program performance along the dimensions that have been identified. The evaluator should reformulate the evaluation questions in a way that incorporates the performance standards that are agreed upon by the key stakeholders.

–       Further, the evaluator is responsible to design data collection tools and lead and manage data collection, cleaning, and analysis, write-up, and communication. It is expected that the applicant will employ various enumerators (costs covered by Samaritan’s Purse) to support data collection.

–       Costs: The evaluator is only responsible for his/her own professional feeds to accomplish the given tasks.

Proposal Components

All applicants must meet the minimum requirements described above. Each application should include a technical and financial proposal as described below.

General qualifications should include:

–       An application letter addressing the selection criteria including how the candidate’s previous experience matches the objectives. The application letter should be no longer than two pages.

–       Explanation of experience of conducting similar evaluations in Ethiopia especially thematic areas with funding from USAID/BHA, preferably within proposed geographical area.

–       CV of evaluator with details of qualifications and experience. The applicant should have the necessary competencies and experiences as described in the TOR to undertake this evaluation.

–       Contacts of three organizations that have recently contracted the evaluator to carry out a similar assignment.

–       Samples of work.

Technical proposal should include:

–       A detailed proposal that demonstrates a clear understanding of the TOR. Along with a written overview, this proposal should clearly detail the proposed methodology based on descriptions above especially the sampling method, scientifically accepted sampling computation, and the relevant methods of data collections. It is expected that the evaluation will take place in both project locations.

–       Draft work plan clearly indicating a proposed activity schedule.

Financial proposal should include:

–       Samaritan’s Purse Ethiopia expects a fixed price based on completion and acceptance of deliverables. It is preferred that costs are broken down by key deliverables noted above. It is expected that the evaluation will include two project locations. Note that all logistical costs and costs for enumerators will be borne by Samaritan’s Purse – as a result is expected that the financial proposal will only include the professional fees for the evaluator.

Criteria for Evaluating Proposal and Selection

The following criteria will be used in evaluating proposals.

Samaritan’s Purse Ethiopia will evaluate the responses to this RFP based on the applicants’ ability to provide a general overview of their qualifications and their proposed approach to the details contained in this SOW. The proposal will be evaluated based on the three criteria established above: qualifications, technical proposal, and financial proposal.

Moreover, the team leader should submit his/her past work, such as at least two written evaluation report in this subject matters and at least two written references but whenever SP need information and clarity, the evaluator will be available to respond timely as per the set deadlines for questions and/or the schedule for a bidders’ conference.