MEASURING RESULTS: HOW TO EVALUATE CO INITIATIVES

Funders of all persuasions - progressive, middle-of-the-road, conservative - can agree that a bottom line for funders is, or ought to be, getting results from their grantmaking. CO grantmaking is no exception to this rule.

Long-term funders of CO are convinced of its value and, for the most part, are more than satisfied with their funding results. Funders new to CO will need to be equally convinced that CO will produce outcomes of the type and scale they believe possible, necessary and/or desirable.

But how can funders classify and measure CO grantmaking results? What can be learned and how best to learn it? How soon can funders expect results?

This section of the Toolbox discusses the CO evaluation strategies of the Woods Fund. The Woods story, which is followed by tips for designing an evaluation system, includes informative pointers for funders who want to plan and implement a formal evaluation strategy. The Woods Fund evaluation is valuable, particularly for funders new to CO, because it documents the important achievements of CO and identifies current weaknesses and/or limitations that need attention if organizing practice is to improve and become an even stronger and more viable strategy for positive change. Other notable evaluations have been those conducted by the Boston Foundation, the Catholic Campaign for Human Development (CCHD) and other grantmakers. The complete evaluations of the Discount Foundation, CCHD and the Woods Foundation are available online at www.nfg.org.

Various funders have been and/or are incorporating mandates for evaluation in their grants to CO groups - often requiring the groups to contract for outside evaluation and to meet the funders' specifications. Some foundations examine CO groups and efforts as part of their own program reviews, to resolve questions about continuing support for CO or to expand support.

For more resources on developing and implementing evaluation systems, visit NFG's Web site at www.nfg.org.

How The Woods Fund of Chicago Approaches Evaluation. One of the most extensive evaluations of a foundation's CO grantmaking was carried out in the mid-1990s by the Woods Fund, a small foundation based in Chicago. Both the process and the results of the evaluation are noteworthy and offer considerable guidance for funders already involved with CO and those new to the field, as well as to CO groups.

The Woods Fund has long supported CO in the city through its grantmaking and other strategies. In 1995, the Fund engaged an outside evaluation team to examine its CO grantmaking, its major priority for over a decade. The evaluation team included seasoned community organizers and trained program evaluators.

The evaluation was extensive - the most substantial evaluation of CO ever undertaken by a foundation - and covered the Fund's CO grantmaking over a ten-year period, from the mid-1980s to the mid-1990s.

The team concluded that the Fund's $4.2 million investment had achieved significant results when judged by three broad criteria: community improvements, leadership development and democratic participation.

The evaluators stressed that CO's ability to achieve widespread community improvements was clear-cut and unambiguous. They reported that CO had successfully "brought millions of dollars into low-income communities for housing, job creation and other community improvements by challenging bank lending practices." Organizing also "trained and supported dozens of parent leaders in local schools, who have ousted non-performing principals and developed new local school programs and policies." And, finally, CO secured "significant public investments in neighborhoods...," and "won efforts to keep out resources and programs deemed inimical to the community's health (by successfully fighting) land fills and hazardous waste facilities."

The Woods Fund evaluation also found that "organizing has indeed been quite effective in promoting democratic participation in the wider community" and that it "developed dozens of leaders and involved thousands of citizens in securing these results."

Other findings candidly raised a number of critical issues and themes related to the constraints and limitations of CO as a strategy for change. Included were: 1) the precariousness of the organizing infrastructure itself, owing to the "weak and unstable funding base for organizing"; 2) the inattention given to "promoting democratic participation of individuals" within the community organizations studied by evaluators; 3) the limitations of CO in effectively addressing "fundamental urban problems," such as poverty, job and wage erosion, drugs and crime; 4) the lack of vision, or, conversely, parochialism that too often characterizes CO groups and activities; and, 5) the disconnection between CO and public policy work.

Following its review of the evaluation report and discussions with the evaluation team members, the Fund's trustees determined that the foundation would continue to place a high priority on funding CO. The Woods Fund reaffirmed its support for funding CO in its 1995 Annual Report. In part, the Fund's decision was responsive to another critical finding of its evaluation team with respect to the weakening funding base of CO groups in Chicago when the evaluation was conducted. The team found that:

At the same time that organizers have begun to face significant role strain, the funding infrastructure for organizing seems to have deteriorated. This declining external support for organizing has taken place in years when sources of support internal to the community have also eroded, thanks to growing class segregation, aging church facilities and declining middle class members, and the loss of business activity in our low income neighborhoods.⁶⁸

How the Catholic Campaign for Human Development (CCHD) Approaches Evaluation. CCHD has been one of the major funders of CO for over 30 years. During that period, CCHD has provided nearly $300 million to more than 3,500 projects. In 1994, the organization undertook a year-long study of its funding activities, carried out by John D. McCarthy of Catholic University of America. He examined 325 groups that received CCHD funding in 1991, 1992 or 1993. Below are some of the study's key findings.

Funding and Budgets

• The groups had a combined budget of $64,980,487 for the year for which they requested CCHD funding.
• The average budget for funded organizations was $213,050.
• Forty-five percent of the groups' income came from grants.
• Almost two-thirds of the groups' expenditures were for personnel.

Who They Are and Who They Serve

• One group in eight was at least 15 years old.
• Their work benefited an estimated 38.5 million people, of whom 18.2 million were poor. This represents half of the U.S. poverty population in 1994.
• The groups had an average of 16 board members and a median staff size of 3.1.
• The majority of those they served were minorities. The majority of members and half of the beneficiaries were poor. A majority of members, beneficiaries, staff and board members were women.

What They Do

• The most frequently addressed issues were housing, jobs, education and health.
• The most commonly used methods for reaching group goals were research (70.8 percent of groups) and membership development and training (69.5 percent). Six groups in 10 (59.1 percent) used protest, negotiation and other forms of direct action.
• Two-thirds of the groups used technical assistance for member, staff or board development.

One conclusion of the study was that CO works in low-income communities, and has significant impact at the local, state and national levels. The study found that the groups changed laws and policies and generated billions of dollars for low-income communities and their residents. Even the least successful groups had some victories.

The author concluded his report by stating:

The groups funded by the Catholic Campaign for Human Development are heavily minority and female in their composition. They address a broad range of issues with a broad range of methods and benefit large numbers of people. They tap outside resources for technical assistance and expertise and receive funding from major American institutions - religion, foundations, business, and government - and from a wide variety of grassroots sources. Many of the groups we have profiled have demonstrated staying power, with lifespans of at least 15 years.⁶⁹

Pointers for Designing a CO Evaluation System. Some funders are using innovative techniques to gain an accurate picture of and assess their CO grantmaking. For example, they are funding consultants to conduct periodic observations of grantee activities, prepare ongoing documentation of grantee work, and develop in-depth case studies. Others are underwriting retreats where varying questions and views are aired at length with grantee representatives and outsiders knowledgeable about the CO field.

Evaluating CO is not impossible, but it can be difficult. Using these and other methods singly or in combination may yield a useful and meaningful evaluation system. It is important to consider the cost of the evaluation, what can be gained from it to satisfy funders' needs and how it can contribute to strengthening grantees.

Funders new to CO will want to consult widely with other funders before embarking on the challenging work of designing and implementing an evaluation system. Some funders are developing or exploring evaluation designs that they hope can be useful to other funders in evaluating CO. Among them are FACT, the Public Welfare Foundation and the Unitarian Universalist Veatch Program at Shelter Rock.

The Woods Fund evaluation team made several recommendations for "increasing evaluations of organizing" because CO organizers, leaders and organizations can learn from evaluation and because too much that has passed for evaluation is too "quick and dirty" to generate significant learning. "The state of the art of outcome measurement in organizing is pretty crude."⁷⁰

They found three major problems to be addressed in designing a meaningful evaluation system.

1. The key to organizing success is its process, but valid benchmarks for assessing the success of this process have eluded us so far.
2. Numbers measures utterly fail to get at intensity, quality, the "spirit and the vision."...We need to find ways to supplement membership numbers with other measures that capture quality and intensity of participation. We need ways to supplement leadership numbers with other measures of leadership quality and sophistication.
3. Listing issues victories fails to isolate the role of CO in effecting the victory; assess the depth of challenge of the victory; or assess what impact the issue victory made on the community, the organization and the people involved.

The team of evaluators also felt that naturally occurring opportunities in CO for continuous evaluation are being missed. The heart of leadership and membership development - reflection-in-action - is an evaluative experience, they suggested. They asked, "How can organizing more systematically accumulate and distill the learnings from these separate reflections? And, is there a growing dichotomy between reflection and action?"⁷¹

For funders new to CO, it may be valuable to discuss the Woods Fund evaluation in some depth with representatives of the Fund, leaders of CO groups in Chicago who are grantees of the Fund, and members of the evaluation team.

In addition, two sociologists - Jacqueline B. Mondros and Scott M. Wilson72 - are tracking and writing about CO groups and doing useful groundbreaking work in developing methodology for evaluating CO. A number of academicians are studying and assessing faith-based CO networks as well, and others are examining CO's impact in various arenas such as health and education reform and environmental justice. Books and articles that may be helpful to funders interested in evaluating CO are referenced on NFG's Web site at www.nfg.org.

Another effort at evaluation has been developed by the Development Leadership Network (DLN). DLN is a network of hundreds of neighborhood-based community development practitioners who believe that CO should be integrated with bricks and mortar strategies, and that community development efforts must be accountable to the community members served. In partnership with the McAuley Institute, DLN has published a Success Measure Guidebook, developed by and for practitioners, to improve evaluation, to better manage programs, and to expand the ways in which practitioners are able to communicate to broader audiences about the benefits of community development programs and activities in low-income communities.

Winston Churchill once said, "Democracy is the worst form of government, except for all the rest." He could have been describing organizers' work to ensure a powerful role for the public in public life; to develop local leaders, to promote racial, ethnic and socioeconomic inclusion; and to demand fairness. This work is rarely tidy or quiet; it is lively and participatory. We believe it is more timely now than ever.