Capturing Change: Isn’t There A Better Way?

This challenge is the subject of a recent paper Capturing Change in Women’s Realities: A Critical Overview of Current Monitoring & Evaluation Frameworks and Approaches in which Srilatha Batliwala and I put forth a challenge and call to action to donors and civil society organizations focusing on women’s rights. It is a call to deeply question the culture of measurement and accountability systems that we have created and maintain.

Our works draws from research and strategizing with activists, donors, women’s rights organizations, and feminists in a variety of settings. We show how the dominant logical-frame and results based management type of evaluations and assessments that many donors rely on do not fully capture the complex changes related to women’s rights and feminist organizations’ efforts.

In one example from our paper, a landless agricultural worker who had participated in an empowerment program describes the impact the program made on her life: “Three years ago, when the landlord in whose fields I work addressed me, I would answer him looking down at his feet. Now, I answer with my eyes on his chest. Next year, I will be strong enough to look him right in the eyes when I speak to him” (p.19).

Take a moment to reflect on this change process. Would a quantitative indicator of self-confidence have captured this successful outcome in a similarly powerful way? Would we have even been able to capture this outcome using traditional logical framework approaches? What happens to the richness of this example when we begin to aggregate?

Women’s rights organizations have identified many challenges with the dominant frameworks preferred in development work to measure change. In many cases, the current fixation with accountability and measurement in development often leads us to highly technical, logical, and depoliticized assessments of the change efforts that women’s rights organizations seek. They are based on the assumption that we can simplistically measure whether program outputs, objectives, and goals were of value and effective based on inputs, activities, and resources.

These frameworks are often very inappropriate for measuring complex outcomes of most interest to women’s rights and feminist organizations (e.g., advocacy, contested law reforms, network building, reducing gender based violence). In addition, the rigid frameworks are not flexible to rapidly changing, repressive, or unstable economic, political, and social conditions under which many civil society organizations operate— where not implementing activities according to plan equals an implementation failure (pp.10, 12).

The development of self-confidence for the landless agricultural worker did not happen overnight. Using our example, this would likely mean the program would probably have been deemed a failure since change hadn’t happened by the end of the year. This highlights the reality of the profound mismatch between the long time frame needed for transformative changes in power to take place and the short time frame in which donors require evidence of change (p.14). The lived realities and experiences of women working tirelessly in patriarchal contexts to challenge family relations and limiting social norms, increase their earned income, go to school or work in safety, protect themselves from bodily harm, etc., show that change takes time.

Another significant problem with our frameworks is their ability to capture and measure the “two steps forward, one step back” phenomenon (p. 12). The work of women’s organizations takes on the incredibly difficult challenge of transforming power relations.  Progress towards this aim – and, at times, the very act of speaking out – often results in extreme backlashes and reversals of progress, such as those recently seen in violence against women human rights defenders.

Too often the emphasis on the need for hard evidence overtakes efficiency, ironically one of the hallmarks of the “value for money” approach. Too often evaluations end up being a “one-off” product of little productive use for donors and women’s organizations alike. In fact, in some forthcoming research by Srilatha and I, one feminist in a conflict zone noted that, in the first year of her organization’s major grant with a bilateral donor, she spent nearly 70% of her time trying to fulfill the technical and administrative requirements to the specifications of the company that had been outsourced to manage grant monitoring and assessment. This wasted precious time when she could have been strategizing, supporting fellow activists in resistance and challenging rights violations, repression, and violence. Indeed, she even thought about turning the grant down as she thought that her time would be better spent on the ground.

Such examples bring to life the common struggles of civil society organizations trying to make revolutionary changes in constraining circumstances. Isn’t there a better way?

In theory, we should be conducting monitoring and evaluation (M&E) to learn how change happens and identify our role in that process, to refine our interventions, and hold ourselves publicly accountable. Too often, we merely go about the M&E process to satisfy donor requirements and to help us increase our funding revenues (pp. 7-8). This is not particularly surprising given the state of the field today, but many are searching for more.

Increasingly, a wide range of development actors are actively seeking alternatives to the current trends in measurement and accountability, and asking how we can shift towards real learning. Indeed, a critical mass is forming: AWID has been engaging in extensive action-research with women’s organizations in addition to a collaborative learning project with donors-grantees around the MDG3 Fund over the past year; Rosalind Eyben (IDS) and Irene Guijt have been leading an innovative initiative called the Big Push Forward to counter the prevailing “audit culture” in development; Gender at Work and IDS are launching the Measuring Gender Equality Initiative this week; and the International Network of Women’s Funds (INWF) are involved in research on M&E models for women’s funds.

My next post will explore some alternatives, highlighting some promising practices and initiatives that AWID and others have been developing for measuring changes in gender equality work.

Alexandra Pittman is a Postdoctoral Fellow at the Hauser Center for Nonprofit Organizations and a Research Associate at the Association for Women’s Rights in Development (AWID). She has in-depth experience writing, conducting, and designing evaluations for NGOs and donors. She can be reached at Alexandra_Pittman@hks.harvard.edu