This document describes the purpose, key activities, roles and responsibilities, and envisioned outcomes of impact measurement work in FY10. Please send your comments and direct any questions to Velina Petrova or Maliha Khanof the Program Impact team of CARE USA.

Why Impact Measurement? Why now? – Rationale


CARE has identified global impact measurement, reporting and analysis an organizational priority, recognizing that it is becoming increasingly critical to strengthen the ability to assess and tell its global impact story. Internally, impact measurement ensures that we hold ourselves to the standards of program quality and moving towards achieving our mission: improving the lives of those we work for. Externally, impact measurement will help CARE’s executive, technical, policy advocacy, fundraising, and marketing and communications staff in their work with external stakeholders and constituencies. Impact measurement supports accountability: to those we work for, those we work with, and those who support and finance our efforts. Finally, impact data will provide the basis, to CARE-USA executive leadership and to other CI members, for more strategic decision-making and better investment strategies.

Formalizing this commitment, CARE has instituted Impact Measurement as one of its global priorities for FY10. To operationalize this priority, CARE has committed to pulling together existing disparate and new impact measurement processes into a coherent Impact Measurement System. This system would consist of various components with complementary functions: a Program & Project Directory; the Ubora Program Quality Assessment Tool (PQAT); regular meta-evaluations; procedures for technical portfolio reviews; proposal analyses; the CARE Women’s Empowerment Annual Impact Report; and strategic impact inquiries.

The notion of measuring impact is not new to CARE. We opened the door to impact measurement with the Strategic Impact Inquiry on women’s empowerment, which deepened our understanding of what genuine impact means and positioned CARE as a thought leader in the field – one willing to critically and transparently evaluate its work. Several sectoral teams have investigated their sectoral impact through in-depth reviews.The Latin America and Caribbean region has begun evaluating its contribution towards the achievement of the Millennium Development Goals, which has helped country offices form an evidence base to use in advocating for policy changes at the national level. Overall, CARE has a solid foundation to build upon as we begin to be more systematic about measuring impact.

Much of the impact measurement system is not new to CARE, but the various processes need to be streamlined and organized into a coherent system and integrated with Ubora. Implementing the system in its entirety is envisioned to be a 2-3 year process. The remainder of this document sets out a proposed process for work in FY10, coordinated by the Program Impact () team.

Back to Top

What is our roadmap?



The document below is a draft of CARE's Impact Measurement Road Map for FY11 - FY14. It gives a high-level view of the process timeline for impact measurement initiatives including: CARE's long-term program impact measurement, Millennium Development Indicator + measurement, Project and Program Directory, and an overall learning and reflection process.


How? – Key Activities for FY10


The following activities will be implemented in FY10, and they are not sequential but rather parallel and reiterative steps.

Finalize global impact and outcome indicatorsThroughout FY10
CARE’s approach to developing a set of global indicators, which has had a long history of discussion and debate, is referred to as an MDI+ approach. Indicators are based on the Millennium Development Goals (MDGs) and their related indicators, the Millennium Development Indicators (MDIs), which will enable CARE to communicate our impact in an internally consistent and externally communicable manner. Solely measuring our impact based on the MDGs, however, would not tell the full story of CARE’s work or priorities. A country can meet the MDGs nationally and still leave marginalized and vulnerable populations behind. The ‘plus’ in our MDI+ approach takes into account aspects of inequity and injustice that the original indicators do not pay sufficient attention to. Therefore, the proposed MDI+ process will adapt and augment the MDIs by applying them to the marginalized and excluded groups we work with. The process will also add the missing dimensions of the unifying framework (social positions and enabling environment) to the MDIs, which are very human-condition focused.
The Program Impact team has prepared a draft of proposed global impact and outcome indicators, relying on CARE’s experience and commonly accepted metrics in development (attached). In FY10, we propose to use this draft set of indicators as a starting point and work to revise, test, and refine them.We will coordinate a consultative process involving various groups throughout CARE-USA and other CI members. Our most important input will come from our Country Offices – Learning Labs and others – where the work of collecting data on these indicators will eventually take place. We will work together on issues of indicator selection, proxy indicators, operationalization, and measurement. The set of global impact and outcome indicators we arrive at, and the data collected on these indicators, will feed into the various impact assessment exercises and processes in CARE’s impact measurement strategy.

Formulate guidelines for and develop CO-level systems Throughout FY10
The most important input into the global indicators and guidelines will come from Country Office teams, who already have experience with measurement, who understand the local realities of data collection, and to whom we will look to make the guidance we develop realistic and credible. The Pi team is committed to building global impact measurement which is intellectually curious and rigorous, but also appropriate, practical and based on capacity and resources. Thus, we will work very closely with a few select Country Offices in fulfilling the global mandate for impact measurement.

We propose to work with a range of COs: some Learning Labs and some others; some with impact measurement work already initiated and some who are not there yet; some who have high capacity in this area, and others who do not. We will work with the DRDs-PQ and Technical Units in Atlanta, and of course with COs themselves, on a proposed list of COs to visit. We are also working with CARE-Australia to ensure that our visits are not limited to CARE-USA offices. We propose to do initial visits to learn from work already in place; working visits to test and revise indicators and measurement guidelines; and final visits to refine and finalize these. This work, as well as the actual trips, will be coordinated to the extent possible with Technical Units to ensure that impact measurement around programs and on the global level are coordinated, synchronized efforts. The goal of our Country Office visits will be to have a mutually beneficial result: support them in building CO-level impact measurement systems and using their experience to inform CARE’s global guidelines and processes.

Form a working group to vet and guide our workThroughout FY10
We propose to form a working group on global impact measurement composed of colleagues from Country Offices (Learning Labs and others); DRDs-PQ; Technical Units; OPM (Ubora team); GAER; and Finance from CARE-USA, as well as key CI member representatives. This working group would vet proposals and ensure that final decisions on global impact measurement accurately reflect our goals and experiences. The Pi team will set up and coordinate the working group.

Work with peer organizations to help identify lessons learned and situate CARE’s progress
In undertaking a task as large and ambitious as measuring impact globally, we need to learn from peer organizations and their experiences. Some of this work of connecting with and learning from others was already started by the old PIKL team and OPM. In FY10, we will continue our learning around impact measurement, now with more detail and clearer focus. In a process of shared learning for CARE, we will identify key questions to ask and challenges to understand; identify external organizations and contacts; and engage with them to learn from their experience. We will also document this process and make all materials and information available.

Gather Information for business process on impact measurement15 January 2010
We will work with select Learning Labs to document our process, progress and challenges and use this information for a business process describing the process of institutionalization and the resource needs of impact measurement. This will be done in collaboration with the Ubora team and will be developed by 15 January 2010 in time to inform allocation decisions for FY11. We will identify key information to keep track of and create a report template with categories such as: key resource issues/needs (human, technological, financial); certain pre-conditions to enable COs to collect impact data (e.g., programs, level of M&E systems, etc.); key achievements and key challenges/concerns. This documenting process, part of the Ubora performance analysis, will help us surface issues and needs to executive levels and inform their decisions regarding the investment that proper global impact measurement requires.

Develop CARE guidelines for impact measurement By end of FY10
The work we carry out in FY10 will accumulate into guidelines for impact measurement at CARE. If we imagine putting together a manual for impact measurement, our FY10 work should provide its content. We will aim to complete that by 1 May 2010; circulate it for input; revise and finalize it by mid-June 2010. These guidelines will cover how to operationalize indicators, how to collect data on them, and how to link CO-level systems to the global process.

Back to Top

Who? – Roles and Responsibilities


In describing the activities above, we have mentioned various groups of colleagues around CARE. Below are the roles and responsibilities we propose for each.

  • Program Impact (Pi) Team – The Program Impact team will coordinate these efforts; communicate clearly and frequently about progress, challenges, and next steps; serve as the bridge between different units and teams; and produce the written documents required for these processes. It will be our responsibility to organize an inclusive and consultative process; to make indicators and guidelines appropriate and ensure they reflect our Country Offices’ experience and capacities; and to surface and work on resource needs.

  • The Ubora team (OPM unit) – The Ubora team will work with Pi to integrate the global indicators into Ubora and link it to other key data collection and analysis processes, giving us an overall picture of CARE’s performance. They will also help surface key analysis emerging from Learning Labs and other COs about the business process requirements for impact measurement and help inform senior management decision-making and planning.

  • DRDs-Program Quality – We will look to the DRDs-PQ to help with refining draft indicators and guidelines; contribute to refining documentation; help identifying CO visits and joining visits where possible; and help supporting COs in testing data collection tools and linking the global reporting work to program impact measurement systems.

  • Learning Labs and other Country Offices – We will rely on the Country Offices engaged in this process to help make it realistic, credible, and practical for CARE. Their prior experience and contributions in FY10 will form the backbone of both indicator sets and guidelines. While our job on the global level will be to look across units and offices and consider the future of impact measurement and accountability at CARE, their job will be to ground our work in what is possible and what is still needed.

  • Technical Units – We will count on our Technical Unit colleagues to work with us in Learning Labs; keep program shift and impact measurement work linked in practice, not only in theory; actively help coordinate our efforts; and ensure that various work streams build on one another. Coordination will be required both in our visits to COs and in the work we do building measurement systems, operationalizing program designs, and providing support to our Country Offices.

The following picture attempts to depict what we envision this process to look like:
Impact_Measurement_Process.jpg

Back to Top

What? – End Products and Outcomes


As a result of the key activities outlined above, we hope to arrive at the following outcomes and products at the end of FY10:

  1. Validated and tested set of global outcomes and impact indicators, to be included in the Program & Project Directory and integrated into Ubora for wider roll-out to Country Offices

Beyond FY10, the set of global indicators tested and refined in FY10 will be embedded in the Directory and drawn on by Ubora, becoming the basis for the CARE-wide impact data collection.
  1. Guidance on impact measurement at CARE, with the following sections:

Operational links between:

global impact measurement
CO-level impact measurement
program and project data collection and analysis
Global outcome data collection
methods and tools
proxy indicators where relevant
sources of data (primary, secondary)
CO impact measurement system
essential components
processes for instituting a CO IM system
resources and capacity needed

Beyond FY10, the guidance we develop based on our work in FY10 will represent the content of an Impact Measurement Manual for CARE. The format of such a manual would be worked out in FY11, but the content will be produced based on our FY10 experience.

  1. Business plan for impact measurement at CARE, with the following sections, among others:

Business process description
Key resources required for impact measurement (human, time, financial, technological, etc.)
Resource development plan

Beyond FY10, the business plan will be used to inform decisions on impact assessment investments, work on resourcing impact measurement, and work on closing identified gaps between key resources and current realities.

We envision this FY10 work as the stepping stone toward global impact measurement that is appropriate to the organization’s current capacities, resourced, and strategically leveraged.

Please send your comments and direct all questions to the Program Impact team: Velina Petrova or Maliha Khan.

Back to Top


Resources




Please visit Care's Impact Measurement Strategy, which describes these different components in detail.



Back to Top




Impact Measurement Wiki | P-shift Wiki Home | How to Use a Wiki