51
Number of approaches developed, adapted, and/or adopted to facilitate adaptive management of a project, program, or initiative, by type
Refers to the number and variety of approaches that have been developed, adapted, and/or adopted to facilitate iterative approaches to learning and adapting while implementing a project, program, or initiative
This indicator refers to the number and type of approaches that have been developed, adapted, and/or adopted to facilitate iterative approaches to learning and adapting while implementing a project, program, or initiative to meet performance objectives. This may include the development of new and/or revision, adaptation, or adoption of existing approaches to help foster adaptive management.
Quantitative or qualitative data from programmatic records
Administrative/programmatic records
Annually, after work planning
This indicator reflects the planning a project, program, or initiative may take to prepare for and foster adaptive management. Using, adapting, or developing a wide range of approaches can signify a commitment by management to the importance of programmatic flexibility and change that best works for a specific team, project, program, or initiative. Projects, programs, and initiatives must determine the appropriate mix and number of approaches; more approaches is not necessarily better. Although this indicator measures the number of approaches, this should not suggest that it is better to use more approaches. The intent is to measure that there are intentionally selected approaches to facilitate adaptive management.
Although a project, program, or initiative may include adaptive management approaches in its work plan, it does not necessarily reflect the use of those sessions in decision making. Indicators in the Reflect subcategory address this common issue and challenge with apply adaptive practices.
The number of outputs updated or modified, by type (indicator 7) may be considered a behavior indicative of adaptive management. Number of KM approaches/methods/tools used, by type (indicator 13)
2017
Count
Wednesday, December 13, 2017
52
Leadership and staff support for adaptive practices
Refers to the extent to which leaders and staff demonstrate support for the adaptive management of a project, program, or initiative
This indicator refers to the extent to which leaders and staff of a project, program, or initiative support iterative approaches to learning and adapting. This may be self-reported or analyzed as a group or team, and may include awareness of the importance of adaptive practice.
Qualitative and quantitative data from responses to questionnaires (using Likert scales) regarding the degree of support from leaders and staff in the use of iterative and adaptive approaches.
Periodic surveys
Annually, or after specific activities
This indicator reflects the perception of staff and leadership in the utility of adaptive practices. Leadership that reinforces adaptive practices is a critical element of adaptive management of a project.
Self-reported data may be biased or may not empirically represent the context or practice.
On a scale of 1-5, 1 being not a lot, and 5 being a lot, how do you perceive leaders of this project, program, initiative to support the use of adaptive practices for managing the project, program, or initiative?
2017
Categorical scale, qualitative
Wednesday, December 13, 2017
53
Number of training sessions or activities focused on adaptive practices that were preplanned in a project, program, or initiative work plan
Refers to the number of training sessions or activities focused on iterative approaches to learning and adapting that were preplanned in a project, program, or initiative workplan
This indicator refers to the number of training sessions or activities focused on iterative approaches to learning and adapting that were preplanned in a project, program, or initiative work plan. This may include trainings, workshops, and learning events, including knowledge exchange events that are preplanned and budgeted to support the adaptive management of a project, program, or initiative.
Quantitative data from programmatic records
Administrative/programmatic records
Annually, after work planning
This indicator reflects the intentional use of adaptive management within a project, program, or initiative. Intentionally identifying adaptive management approaches in a work plan can signify a commitment by management to the importance of programmatic flexibility and change.
Although a project, program, or initiative may include adaptive management approaches in its work plan, it does not necessarily reflect the use of those sessions in decision making.
Pre-planning training sessions may be a separate indicator for new projects, programs, and initiatives. Ongoing projects, programs, and initiatives may only be able to report out on implemented training sessions (indicator 9 and 54). Indicator 54 is complimentary and recommended to be used in conjunction with indicator 53 as a way to measure intent and follow through.
2017
Count
Wednesday, December 13, 2017
54
Number of training sessions or activities focused on adaptive practices
Refers to the number of training sessions or activities delivered to increase awareness, understanding, or capacity in iterative approaches to learning and adapting
This indicator refers to the number of training sessions or activities delivered to increase awareness, understanding, or capacity in iterative approaches to learning and adapting among staff in a project, program, or initiative. Training sessions or activities may be delivered by internal or external experts on topics such as adaptive management; monitoring, evaluation, and learning; complexity-aware programming; facilitation techniques; and collaborating, learning, and adapting.
Quantitative data from programmatic records
Administrative/programmatic records
Annually, after work planning
This indicator reflects the need to both train staff in projects, programs, or initiatives in adaptive management and support the implementation of adaptive practices. Without awareness, understanding, capacity, and time, adaptive practices remain ad hoc. Carefully planning and monitoring training and activities can signify a commitment by management to the importance of programmatic flexibility and change.
Although a project, program, or initiative may include adaptive management approaches in its work plan, it does not necessarily reflect the use of those sessions in decision making. Although training sessions may be provided, it does not guarantee that the participants were able to internalize the learning and facilitate or use the training materials in the future.
The number of training sessions, workshops, or conferences conducted, by type (indicator 9) is specifically related to adaptive management. Indicator 53 is complimentary and recommended to be used in conjunction with indicator 54 as a way to measure intent and follow through.
2017
Count
Wednesday, December 13, 2017
55
Number of people trained in adaptive practicess
Refers to the number of staff trained in iterative approaches to learning and adapting
This indicator refers to the number of staff trained in iterative approaches to learning and adapting to improve projects, programs, or initiatives.
Quantitative data from administrative records or reports that provide the number of participants, characteristics of participants, gender, and other relevant information
Administrative records and reports
After specific activities
This indicator tracks the initial reach of adaptive practices. It is a simple way to establish a foundation of staff trained in programmatic flexibility and change.
Although a project, program, or initiative may include adaptive management approaches in its work plan, it does not necessarily reflect the use of those sessions in decision making. Although training sessions may be provided, it does not guarantee that the participants were able to internalize the learning and facilitate or use the training materials in the future.
Number of individuals served by a KM output, by type (indicator 14)
2017
Count
Wednesday, December 13, 2017
56
Percentage of target staff reporting an improvement in capacity to use adaptive practices
Refers to the percentage of target staff reporting an improvement in capacity (knowledge, skills, or abilities) to use adaptive practices
This indicator refers to the percentage of target staff reporting an improvement in capacity (knowledge, skills, or abilities) to use adaptive practices for the management of a project, program, or initiative as a results of participating in training or other activities aimed at building capacity in adaptive management. Target staff may include partners. Self-reporting, pre- and post-evaluations, or follow-up surveys should be conducted to determine the extent to which there was an improvement in awareness, understanding, or capacity in iterative approaches to learning and adapting.
Quantitative data from pre- and post-tests using survey questions and Likert scales to determine capacity to use adaptive practices and follow-up assessments at three and/or six months to determine knowledge retention; qualitative data can provide greater insight into target user capacity
Pre- and post-tests, follow-up surveys
Quarterly, semiannually, or after specific activities
This indicator can be used to monitor changes in capacity (awareness, knowledge, and skills) in adaptive practices over time (before training/activity and after training/activity). It is a simple way to establish a foundation of staff trained in programmatic flexibility and change.
Although a project, program, or initiative may include adaptive management approaches in their work plan, it does not necessarily reflect the use of those sessions in decision making. Self-reported data may be biased and may not empirically represent the context or practice.
The number of instances of staff reporting their KM capacities improved, by type (indicator 12) is useful, however, the proportion may be a more useful indicator for projects, programs, or initiatives with small staff.
2017
Proportion
Wednesday, December 13, 2017
57
Number of approaches, methods, tools, or events implemented for reflection and other adaptive practices
Refers to the number of adaptive practices (approaches, methods, tools, or events) used to facilitate the adaptive management of a project, program, or initiative
This indicator refers to the number of specific adaptive practices, including approaches, methods, tools, or events used to facilitate the adaptive management of a project, program, or initiative. This may include the number of in-person learning events, after-action reviews, lesson-learned workshops, communities of practice, new technologies that facilitate increased ease and frequency of interaction, or other related iterative approaches to learning and adapting. Although this indicator measures the number of approaches, this should not suggest that it is better to use more approaches. The intent is to measure the intentionally selected approaches used to facilitate adaptive management.
Quantitative or qualitative data from programmatic records, or self-report of number of adaptive practices conducted, by type; qualitative data to provide greater insight into actual use by staff
Administrative records and reports, self-report surveys
Quarterly, semiannually
This indicator reports the actual implementation of the planned adaptive practices that were identified to be used in the project, program, or initiative.
Although a project, program, or initiative has conducted adaptive management approaches, and followed through on their identified work plan activities, it does not necessarily mean that the sessions were of high quality or contributed to programmatic improvements. Projects, programs, and initiatives may find it more useful to measure the proportion of staff using adaptive approaches, however, because the number staff may change overtime (expand and contract) it may be difficult to measure over time.
The number of KM approaches/methods/tools used, by type (indicator 13) helps to ensure that a wide range of methods are used to meet diverse needs. While indicator 8 provides the number of KM coordinating/collaborating activities, by type, indicator 57 focuses on adaptive practices rather than general collaboration.
2017
Count
Wednesday, December 13, 2017
58
Number of sessions or activities that include analysis of and/or reflection on monitoring data
Refers to the number of sessions or activities focused on reflection and analysis of monitoring data from a project, program, or initiative to inform performance and adjustments
This indicator refers to the number of sessions or activities focused on reflection and analysis of monitoring data from a project, program, or initiative to inform performance and adjustments. This may include modifying results reviews, data-quality assessments, or other monitoring and evaluating activities implemented for accountability and to inform decision making to include more learning and reflection.
Quantitative data from administrative records or reports that provide the number of sessions or activities implemented with a focus on reflection and analysis of monitoring data
Administrative records and reports
Quarterly, semiannually, or after specific activities
This indicator reports on the implementation of planned adaptive practices identified to be used in the project, program, or initiative that specifically used routine or other monitoring data collected by the project, program, or initiative.
Although a project, program, or initiative has conducted adaptive management approaches, and followed through on their identified work plan activities, it does not necessarily mean that the sessions were of high quality or contributed to programmatic improvements. Quality can be assessed through user satisfaction surveys (see indicators 24 to 28); actions taken to make programmatic improvements can be assessed in an internal assessment.
The number of adaptive practices used (indicator 57) focuses on reflection and analysis of routine monitoring data, often by M&E staff. While indicator 8 provides the number of KM coordinating/collaborating activities, by type, indicator 58 focuses on adaptive practices rather than general collaboration. Indicator 58 is also linked to indicator 62, and can be used with indicator 62 to see how many of the sessions were useful in generating knowledge that was then applied.
2017
Count
Wednesday, December 13, 2017
59
Number of actionable recommendations identified or collected to inform project, program, or initiative performance
Refers to the number of actionable recommendations to inform project, program, or initiative performance or adjustments that were collected from the use of adaptive practices
This indicator refers to the number of actionable recommendations to inform project, program, or initiative performance or adjustments that were identified or collected from the use of adaptive practices or from sessions or activities focused on reflection and analysis of monitoring data. The number can be used to calculate a percentage of action taken.
Quantitative and qualitative data from the review and analysis of meeting minutes, reports, and other documentation to determine how many recommendations are actionable.
Administrative records and reports
Quarterly, semiannually, or after specific activities
This indicator reports the number of recommendations collected that are actionable. Recommendations should identify a point of contact or timeframe for its use, rather than be a general statement that provides no next steps. This indicator should help staff to reflect on whether the stated recommendations can be used, by whom, and by when. By collecting the number of actionable recommendations, a percentage of actions taken can be calculated during an internal assessment (see Act subcategory).
Although a project, program, or initiative has conducted adaptive management approaches, and followed through on their identified work plan activities, it does not necessarily mean that the sessions were of high quality or contributed to programmatic improvements. Review and analysis of documentation can help determine how many recommendations are actionable, such as actions taken within the scope of the project, program, or initiative; responsibility and next steps are clearly documented; and budget and time allocated, and so on. Teams can determine how best to define "actionable" to meet the project, program, and/or initiative needs.
The number of key actionable findings, experiences and lessons learned captured, evaluated,
synthesized, and packaged (indicator 5) is critical to collect in order to determine the number/proportion of actions taken.
2017
Count
Wednesday, December 13, 2017
60
Percentage of intended users who are satisfied with trainings, approaches, or events focused on adaptive practices
Refers to the percentage of intended users who are satisfied with trainings, approaches, or events that focus on or promote adaptive practices
This indicator measures the percentage of intended users who are satisfied with trainings, approaches, or events that focus on or promote the management of a project, program, or initiative through adaptive practices. A satisfied user indicates that intended needs and expectations were met. User feedback should inform future activities. Multiple data points will monitor change in user satisfaction over time.
Quantitative data from self-reported surveys or questionnaires using Likert scales to determine user satisfaction with trainings, approaches, or events focused on adaptive practices; qualitative data can provide greater insight into user experience, attitudes, and preferences
Surveys
Semiannually or after specific activities
The aim of this indicator is to gauge user satisfaction with the trainings, approaches, and events that were selected for adaptive practices purposes in the project, program, or initiative.
This indicator does not measure the quality of the trainings, approaches, or events, as it looks at self-reported satisfaction. It is possible for staff to be highly satisfied with a training or event, but for that training or event not have impact on the improvement of the project, program, initiative, or decision-making processes.
The number/percentage of intended users who are satisfied with a KM output (indicator 25), as satisfaction could be associated with knowledge retention and reuse.
2017
Categorical scale, proportion, qualitative
Wednesday, December 13, 2017
61
User rating of usefulness of content/outputs produced from the use of adaptive practices
Measures the perceived quality and/or relevance of content/outputs produced from the use of adaptive practices to inform the management of a project, program, and/or initiative
User rating of quality and/or relevance of content/outputs produced from the use of adaptive practices to inform the management of a project, program, and/or initiative. “Content” refers to the outputs, such as action steps, that emerge from a training, workshop, session, or event. "Relevance” refers to whether intended users find the information or knowledge applicable and important to their work. Multiple data points are used to monitor change in user reporting of the usefulness of adaptive practices over time.
Quantitative data from self-reported surveys or questionnaire using Likert scales to determine the quality or relevance of participating in activities intended to training in or use adaptive practices for program implementation
Surveys, after action reviews
Semiannually or after specific activities
This indicator measures the user rating of the quality of the outputs from adaptive practices used in the project, program, or initiative. It is a way to measure the usefulness or quality of the practices.
This indicator does not measure the quality of the trainings, approaches, or events, as it looks at self-reported usefulness. It is possible for staff to believe that the practices/outputs were of high quality or relevant, but for those practice/outputs not to have impact on the improvement of the project, program, initiative, or decision-making process. Qualitative data can provide greater insight into user experience, attitudes, and preferences and how quality can be improved.
User rating of usability of KM output (indicator 26); and
user rating of content of KM output and its relevance (indicator 27)
2017
Categorical scale, qualitative
Wednesday, December 13, 2017
62
Number of instances when target users report that their projects reused or adapted previously captured knowledge/resources to design or start a project, program, and/or initiative
Refers to the use of previously captured knowledge for decision making while designing or starting a project, program, or initiative
This indicator refers to the use of previously captured tacit and explicit knowledge for decision making while designing or starting an activity, such as a project, program, or initiative. This may include accessing previous after-action reviews, reviewing previous project documentation, conducting a peer assist with actors previously engaged in similar work, or conducting before-action reviews with the team. It can apply to work-related decisions and practices at project, program, initiative, and individual levels. Anecdotal evidence may be included regarding whether a consultation has occurred or resulted in actionable recommendations.
Quantitative and qualitative data from key informant interviews or focus group discussions, used to determine when previously captured knowledge or resources were use, reused, or adapted for a specific activity
Key informant interviews, focus group discussions, after-action reviews
Semiannually or after specific activities
This indicator measures actions taken to put reflection and knowledge into practice. By asking open-ended questions, the indicator captures not just the number of instances of the material being used, but also the way it was used, how it was adapted, and how it was useful in the design or start-up of a project, program, or initiative. This indicator addresses a common challenge in adaptive management—taking the time to use previously captured knowledge and resources. This number can be used to calculate a percentage.
Finding time for discussion and analysis can be challenging for a project, program, or initiative. This indicator is mostly used to evaluate the adaptive practices selected, not the program itself, which can result in a low prioritization of this indicator. Qualitative data collection can provide additional insights on the meaning of this indicator, and is strongly recommended because of potentially divergent points of view of what constitutes use, reuse, adaptation, and/or action.
Indicators related to action, such as decision making, policy, and practice (indicators 40 to 42), and users’ knowledge needs and feedback used to inform design and implementation of products and services (indicator 4). This indicator is also linked to indicator 58, in that indicator 58 counts the number of sessions, and indicator 62 counts the number of times someone reports that the knowledge gained has been used.
2017
Binary (y/n), count, qualitative
Wednesday, December 13, 2017
63
Number of instances when target users report the application of knowledge captured through reflection to inform decisions or to take corrective action
Refers to the use of knowledge in decision making while taking corrective action, or attempting to improve a project, program, or initiative
This indicator refers to the use of tacit and explicit knowledge in decision making while taking corrective action or attempting to improve an activity, such as a project, program, or initiative. It measures the use of actions or outputs from reflective learning sessions in decision making or practice, which may include work planning, requests for technical assistance, modifications in project implementation or management activities, data review meetings, or identification of additional project or program needs. It can apply to work-related decisions and practice at the project, program, initiative, or individual level.
Quantitative and qualitative data from key informant interviews or focus group discussions to determine when decisions were made based on knowledge, information, or insights captured during reflection
Key informant interviews, focus group discussions, after action reviews
Semiannually or after specific activities
The aim of this indicator is to measure actions taken to put reflection and knowledge into practice within the project, program, or initiative. By asking open-ended questions, the indicator captures not just the number of instances that knowledge was used to inform decisions or take corrective action, but also the way it was used, how it was adapted, and how it was useful.
Finding time for discussion and analysis can be challenging for a project, program, or initiative. This indicator is mostly used to evaluate the adaptive practices selected, not the program itself, which can result in a low prioritization of this indicator. Measuring this indicator should be carefully considered as it could require considerable level of effort. Whether projects will want/need to measure this is debatable, but it still is an important measure of the preparation and reflection that resulted from previous subcategories. Users may want to limit the actions taken from one specific adaptive practice, such as an after-action review, in order to make this indicator manageable. Qualitative data is strongly recommended.
Indicators related to action such as decision making, policy, and practice (indicators 40 to 42), and
users’ knowledge needs and feedback used to inform design and implementation of products
and services (indicator 4)
2017
Count, proportion
Wednesday, December 13, 2017
64
Degree of change in project norms or behaviors conducive to evidence-based decision making and action, as reported by target users
Refers to the extent to which target users report a change in project or program norms and behaviors toward those conducive to evidence-based decision making and action
This indicator refers to the extent to which target users report a change in project or program norms and behaviors toward those conducive to reflection and evidence-based action as an outcome of the use of adaptive practices. This may include changes in beliefs, opinions, and perceptions of both the project, program, or initiative and the individuals themselves regarding the value and benefit of the adaptive management of project, programs, or initiatives.
Quantitative data from surveys or questionnaires using Likert scales to determine degree of change in norms or behaviors, self and others; qualitative data from key information interviews and focus group discussions on specific norms and behaviors reported to have changed and why, and the implications of those changes
Surveys, key informant interviews, focus group discussions, after-action reviews
Annually
The aim of this indicator is to measure the usefulness of the adaptive practices in changing the culture of the project, program, or initiative to one that values evidence-based decision making and action.
It is difficult to measure norms; there are a lot of quality issues and little consistency in how norms are measured. Finding time for discussion and analysis can be challenging for a project, program, or initiative. This indicator is mostly used to evaluate the adaptive practices selected, not the program itself, which can result in a low prioritization of this indicator. It also is challenging, but not impossible, to get a good baseline of the project norms prior to the use of adaptive practices.
A similar indicator is the number/percentage of intended users who report that a KM output reinforced or validated existing knowledge (indicator 35). Indicator 64 identifies outputs specifically from adaptive practice sessions.
Evidence-base for Collaborating, Learning, and Adpting (eb4cla): https://usaidlearninglab.org/eb4cla
2017
Categorical scale, proportion, qualitative
Wednesday, December 13, 2017
65
Degree that adaptive practices have contributed to the objectives of a project, program, or initiative
Refers to the extent to which taking an adaptive approach has contributed to achievement of project, program, or initiative objectives
This indicator refers to the extent to which taking an adaptive approach has contributed to achievement of project, program, or initiative objectives. Specifically, that an adaptive practice has made an impact on a project, program, or initiative. Generally, projects, programs, or initiatives should not be required to show the unique contributions of adaptive practice on their objectives; instead, they should indicate the use of tools, approaches, and processes demonstrated to contribute to the more efficient and effective delivery of objectives.
Quantitative data from surveys or questionnaires using Likert scales to determine degree of change in norms or behaviors, self and others; qualitative data from key informant interviews and focus group discussions on specific norms and behaviors reported to have changed and why, and the implications of those changes
Surveys, key informant interviews, focus group discussions, after-action reviews, research studies
Once, at the conclusion of the project, program, or initiative
The aim of this indicator is to measure the extent to which adaptive practices improved and/or influenced project, program, or initiative outcomes.
Allocating project funds for this type of study, or building it into the evaluation of the project, program, or initiative, can be challenging due to competing priorities. It is likely that there would not be a comparable project, program, or initiative that could be evaluated as a control against the project, program, or initiative that did use adaptive practices, so the study of the difference would be minimal or extrapolated. However, one could compare similar projects, one that used adaptive practices, and one that did not, and compare outcomes. Ideally, the extent of this change would be shown through the collection of stories, cases, or examples about how adaptive practices contributed to a project, program, or initiative. It also is challenging, but not impossible, to get a good baseline of the project norms prior to the use of adaptive practices.
Evidence-base for Collaborating, Learning, and Adpting (eb4cla): https://usaidlearninglab.org/eb4cla
2017
Categorical scale, qualitative
Wednesday, December 13, 2017
