Evaluation as a Community Building Tool
By Robin Epstein

Evaluating nonprofits is a big business, and it is expensive for foundations and government to research and judge the organizations they fund. But are evaluators measuring the right things? This question has spurred debate about just what an evaluation is supposed to accomplish, and who it's supposed to serve.

When a community organization based in a New Orleans housing project launched a teen pregnancy prevention project, its leaders figured birth control - not abstinence - represented their best chance of reducing the number of kids having babies. The residents and service providers in the St. Thomas/Irish Channel Consortium (STICC) thought the local academic they'd hired to do the evaluation required by their funders shared that perspective. So they were taken aback when they learned midway through the evaluation that he was a fundamentalist Christian and his religious views had influenced his interpretation of the project's results. To him, the only outcome that spelled success was fewer teens having sex. On top of that, he over-reported drug abuse by asking a question that was so ambiguous kids thought he was asking if they'd ever taken Tylenol.

Evaluating nonprofits is a big business, and both foundations and the government pay good money for evaluators to research and judge the organizations they fund. Barbara Major, STICC's former chair, had discovered what many critics have been saying for years - all too often, evaluators aren't measuring the right things. Many aren't suited to do the judging in the first place - and some are arrogant or downright ignorant, generating reports jammed with useless or inaccurate data.

"There are a lot of people who feel like they've been burned by evaluators. [Evaluators] get parachuted in, don't have a sense for the context, and are supposed to determine whether some intervention 'worked,'" says Anne Kubisch, director of the Aspen Institute on Comprehensive Community Initiatives, a group that studies evaluation. For lots of nonprofits, she says, the attitude toward evaluations is "your friends don't need 'em, and your enemies will use 'em against you."

But while many housing developers, social service providers, and community organizers just hope to survive their evaluations with funding intact, others are trying to make the process more relevant and accountable. "Participatory evaluation" may sound like an oxymoron. But by encouraging evaluators to consult with local residents and nonprofit staff, this trend puts social programs in context. You could even call it a kind of grassroots evaluation, where community members have a voice in deciding whether or not community programs ultimately work.

It's at the center of a debate about just what an evaluation is supposed to accomplish, and who it's supposed to serve - and it's controversial. Evaluations have become an integral part of the grantmaking world. In fact, even smaller grants are now more likely to require an outside evaluation, once only a requirement for grants of hundreds of thousands of dollars. The field has the trappings of the multi-million-dollar industry it is: an American Evaluation Association, websites, lively arguments on an e-mail discussion group that boasts more than 1,200 subscribers, and a dozen campuses offering a Ph.D. in evaluation.

Evaluations generally require about 10 percent of a project's budget, though costs can swing from 2 percent to more than 15 percent. Cindy Guy, a senior research associate with the Annie E. Casey Foundation, estimates that Casey will spend between $3 and $5 million over eight years for the evaluation of its $30 million, six-city Jobs Initiative.

Evaluations are somewhat akin to audits that don't just look at the books. Funders "who are putting in a fortune have a right to know from a third party how it's going," says Anita Miller, who until recently was the executive director of the Comprehensive Community Revitalization Initiative in the South Bronx.

Some funders just want to know that their money was well-spent, while others are looking to see if they should proceed with a project. No matter why they've been authorized, evaluators often can't win for losing. They are accused both of being too powerful, destroying programs' reputations and devastating their fundraising, while at the same time being inconsequential, generating reports nobody sees that are released long after the program has ended.

In truth, the idea that a grant evaluation can cripple a program seems to be more fear than reality. While examples of inept evaluations aren't hard to find, dozens of interviews turned up no programs shut down because of one. Instead, the main danger is that an evaluation will produce little return for the hefty price tag.

Anti-poverty program evaluation was born in the Department of Defense (DOD), with President Lyndon Johnson as its midwife. Enamored of a strict DOD cost benefit analysis protocol, Johnson ordered every federal agency to adopt the same process. The military system didn't catch on - LBJ misjudged the amount of resources and data it required - but the issue brought evaluation to federally-funded social programs.

As the War on Poverty grew, so did governmental evaluations, and foundations followed suit. The late Sar Levitan, a George Washington University professor, estimated that $1.5 billion was spent on evaluations by government and foundation funders just between 1968 and 1975.

Today, evaluations are done by for-profit and nonprofit research firms, independent consultants, and university-based social scientists from disciplines like psychology, sociology, economics, anthropology, and education. Some are full-time evaluators, others do it as a sideline.

The evaluations themselves are just as diverse. A report can be so full of charts and graphs that it looks like a statistics textbook. Another might be purely descriptive, a collection of stories of neighborhood people.

Most evaluations now follow a basic formula. Evaluators create a neighborhood portrait with demographic data, extensive interviews with clients, or surveys with residents. The same variable are measured at the program's end. Variables may be as clear and tangible as crime rates or school graduations, or as amorphous as neighborhood pride or political will.

In the abstract, evaluations seem like a good thing. But people at every level of the system - local residents, program staff, funders, even the evaluators themselves - complain about what really goes on. Clients of social service and community development programs often feel alienated and judged by the evaluation process. Strangers come into the community, ask a lot of intimate questions, and often don't share their results. Social differences - class, education, race - between evaluator and evaluatee make the problem worse.

Program staff complain that evaluators' standards and measures aren't always appropriate or even predictable. For example, an evaluator trying to measure citizen participation might count heads at a community meeting. But a simple count misses many factors, like whether people were really involved in decision-making. And evaluators may disappoint funders as well. Some complain that too many evaluations tell them only whether an organization did exactly what it promised to do, providing few signposts for funders who want to replicate successful efforts. "Thumbs up or thumbs down doesn't work for complex problems," Kubisch says. "Our hope is that you learn both whether something works or not and why and how it works or not." Evaluating anything as unscientific as a community-based program is, by its nature, a subjective and messy business. One evaluator might say a workfare program that cuts welfare rolls is great. Another would say it's irresponsible, because participants are not gaining useful job skills. Neither could really be called impartial.

"People like me have come to believe that every social science study advocates some values and not others, serves somebody's interests and not others," says Jennifer Greene, a professor of policy analysis and management at Cornell University. Too often, she says, the interests of funders, policymakers and academics come first.

Nowadays, New Orleans' STICC has an evaluation committee that interrogates anyone who wants to do research at St.Thomas, extracting promises that might make traditional evaluators fear for their professional credibility. Researchers must agree to attend a two-and-a-half-day workshop on racism, share ownership of their data, and allow the group to review their evaluation design and determine how results are used.

Many of the hallmarks of the STICC plan - data sharing, community input, dialogue - resonate with the general idea of participatory evaluation, the notion that the people being studied should have a say. These themes are widespread in developing countries, but have started to catch on only more recently in the U.S. More ally than auditor, these new-style collaborative evaluators offer technical expertise but don't run the show. For funders who want to encourage civic engagement, participatory evaluation often makes more sense, since it gives more weight to the racial, economic, and social contexts in which programs operate. "For me, it's the only way to do it credibly and come up with recommendations that are useful," says David Fetterman, a past president of the American Evaluation Association. Participatory evaluation may also simply be a better tool to assess advocacy, public policy work, and organizing, says Anne Romasco, executive director of the James C. Penney Foundation. She points out that many foundations have shied away from funding these programs because their results are more difficult to identify.

Greene estimates that 10 to 15 percent of the evaluators in the U.S. are now trying participatory approaches. Fetterman says that about half tolerate the new approach and a quarter "really hate it." Certainly, it has its critics. "First, evaluation at its best is a highly technical business, and these are skills that are not easily found among lay persons," says Peter Rossi, professor emeritus of sociology at the University of Massachusetts at Amherst and a congenial skeptic of the trend. "You have to be skilled at handling numbers, at measuring social variables. The second objection is that the participants in a program are not necessarily the best persons to take an impartial and outside view of one's own interest in a program...The participatory evaluations that I've seen end up with everyone feeling very nice about each other but not necessarily having evaluated the program."

Defenders of the participatory approach counter that old-style evaluators don't like it because it gives community residents the power to define what they want to get out of the study. "Power issues and conflict issues really emerge quickly," says John Gaventa, one of the field's gurus and the former director of a participatory evaluation of rural Empowerment Zones. "That is different from asking people to fit their action into pre-defined definitions of success," says Gaventa.

The influence of participatory evaluation can be modest. But Cheryl Grills doesn't consider minor modifications to evaluator-formulated surveys to be sufficient. At her Los Angeles company, Imoyase, she practices what's known as "empowerment" evaluation. "To me, participation means that, out of the gate, you're involving the community in the design of the evaluation, the creation of instruments, the sampling procedure decisions, the collection of the data and the interpretation of the data," she says.

Grills admits that, on occasion, she's had to bend to views she didn't agree with. When residents on one team shrunk from including explicit questions about sexuality and drug use on a survey, she went along with them - even though her experience told her that most people will answer those questions. "I had to walk my talk," she says. "It takes you out of the realm of being in total control."

Grills' approach - putting the community in charge, listening rather than telling, using data to effect change - in a sense transforms the whole purpose of evaluation by turning evaluators into community-builders in their own right, and by using evaluation money to accomplish program goals. And that, some say, is just what the field needs. v

This story first appeared in the December 1998 issue of City Limits, New York's urban affairs magazine. For more information about City Limits, call 212-479-3344.

Robin Epstein writes about community organizations and is a consultant to the Ford Foundation. She is also the author of Citizen Power: Stories of America's New Civic Spirit, published this year by the Democracy Resource Center.

Resources:

The Institute of Development Studies' website www.ids.ac.uk/ida/particip has additional resources on participatory evaluation.

The Evaluation Exchange is based at the Harvard Family Research Center. Its website is www.hugsel.harvard.edu/~hrfp/eval/, or call 617-495-9108.

Back to NFG Home Page.

1301 Connecticut Ave NW, Suite 500 • Washington, DC 20036 • Phone: (202) 833-4690 • Fax: (202) 833-4694 • nfg@nfg.org
Copyright © 1998-2008 • Web Site Usage Policies