STATEWIDE EVALUATIONS SOME THOUGHTS by John A

This paper starts with a broad premise shared by many in our community about the growing importance of evaluation to effective legal services delivery. That premise is that good management involves regularly evaluating how effective we are in accomplishing our objectives. A more rigorous management expectation about self evaluations would be beneficial at all levels of our system. Individual programs will make better decisions regarding their operation, if those decisions are based on a thoughtful, well grounded appreciation of how effective current efforts to serve clients are. Innovative initiatives within a program will more quickly be adjusted to better serve clients –or be jettisoned if they don't, if there is informed knowledge about how effectively the initiative is working. As a national community, we have many questions to resolve regarding new delivery approaches, such as hotlines and web-based self-help, that cry out for, and now are receiving, thoughtful evaluations to help determine their value and effectiveness.

This paper considers the importance of the evaluation of statewide systems, and assumes that at that level too, regular evaluations would be beneficial. The framework for thinking about statewide evaluation is that it is an evaluation of the system, not just a single program, even in a state with a single statewide legal services program. Looking at the system means looking at how all the resources that are available to serve low income clients function together (or not) to respond to clients’ needs.

A great deal of attention has been focused in our community over the past few years on the functioning of legal services at a statewide level. State planning, statewide hotlines, statewide websites, configuration struggles and the introduction of new jargon (and values) about "state justice communities" and "100 percent access" have all focused a great deal of attention on the state level of legal services, and in some circumstances have generated controversy and tension. This paper is not directly about any of these statewide and state level issues, although a statewide evaluation might be relevant to any of them. Rather, it is about the value and challenge of looking at the system of legal services from a statewide perspective. Although we have focused significant time and resources on wrestling with state level questions, an actual evaluation of a statewide system is not that common.

I believe that an underlying objective of any evaluation should relate to the question of how effectively the system is responding to client needs. Within that larger question, however, there are a number of questions that a statewide evaluation might answer.

· What are the objectives the system is trying to meet and is it achieving them? This is certainly one of the more compelling purposes for undertaking a statewide evaluation. A number of states, for instance, have adopted a goal of 100 percent access to some form of service for all eligible clients. Such a goal will drive a number of strategic choices, and it would seem, therefore, to be extremely important to answer questions regarding how well the objective is being met now, to guide future decisions and as a baseline for the future. What percentage of clients have access? To what form of service do they have access? Is the service they have access to helping resolve the legal problem that they encounter? Are all populations equally able to access the system? To the degree that there are differences in access among various populations, what causes that difference? …geography? …language? …culture?

Periodic evaluations, over time, would be appropriate to examine if improvements have been made in access for clients. Are we closer to 100 percent then we started? Have we addressed the differences in access based on geography, language and culture? Have adjustments in the way that services have been provided addressed previously identified weaknesses? What further adjustments are appropriate?

Many states have, through state planning, identified other objectives for their statewide system. These are often couched in terms of core capacities or fundamental values. Whatever the nomenclature, such statements usually address issues related to access for clients, quality of service, the offering of a full range of service, integration of the system, and resource development and allocation. If such plans are real, they too should drive strategic choices about how the legal services system functions and how resources are allocated in the state. Those strategic choices should be based on an informed understanding of how well the core values are being served now and how well agreed objectives are being met. Do we, in fact, provide a full range of services? Are appropriate state-level elements in place to foster high quality throughout the state –elements such as, statewide training, statewide coordination of legal work, electronic access to legal research, and the like? Is the system generating new resources and expanding them effectively? Are resources allocated properly within the system? Are relationships among the system's key components positive, and do they support the system meeting its objectives?

· Is the system responding to the most important needs of clients? This is an important question, the evaluation of which is particularly germane to a statewide system. Different components of a statewide system will make choices about how they will serve clients and what their priorities are. Each decision of the various components that make up the statewide system may be appropriate for that institution. What may get lost, without conscious evaluation of how the components fit together, is how all the separate decisions add up in terms of whether all of the issues facing clients are being met. Are there legal issues of importance that nobody is addressing because they are new, or difficult to take on? Who addresses the needs of populations for whom LSC funds cannot be used, particularly the undocumented? Who deals with broad systemic issues? Who pursues remedies which are limited or prohibited for LSC grantees?

Several factors make this a particularly pertinent set of questions now. There are a number of significant changes afoot that make it likely that a statewide system may not be addressing all the needs that clients face. The first is a dramatic transition in the nature of poverty in this country. We are moving from a system in which poverty was tied to public entitlements to a system where at the core of poverty are low-wage jobs and marginal employment, and the safety net of public entitlements is limited or nonexistent. The great majority of poor persons continue to be women and children in single parent families. The personal and economic circumstances of poor families has changed dramatically as the household head is faced with acquiring necessary skills, finding employment, dealing with the lack of child care and transportation, and confronting the many issues of the workplace. The legal services community has not fully come to grips with the range of new legal problems which arise from such changed circumstances, and what we should be doing to respond.

Whether such issues have been identified and resources allocated to address them is one question which should be evaluated at a statewide level. What is, for instance, the responsibility of the legal services system for job creation, addressing systemic issues related to access to work, such as public transportation, child care, job training and the like? Who deals with issues related to state and regional economic development policies that impact on jobs, housing and the displacement of poor communities?

There are also significant demographic changes that raise statewide issues, particularly the dramatic increase in the number and diversity of immigrant communities in many states across the country. The populations are frequently isolated linguistically, culturally and politically, and often contain a number of undocumented persons. A statewide evaluation might focus on the question of whether the system overall is responding to such isolated clients. Who serves the undocumented? Who reaches out to communities that are subject to exploitation because of their isolation?

A third set of changes that suggest the importance of examining the system at a state level are the many new methodologies that are being instituted, such as hot lines and assisted self help. A statewide evaluation that examines whether the system is responsive to the needs of clients may also focus on an outcome question. Are the new methodologies providing meaningful assistance to clients? Does the work that the system produces solve the clients problem or meet clients' objectives? Is the work overall of adequate quality to get the job done?

· Are the necessary components of a statewide system in place? For many years, the legal services community has struggled with what is the proper focus for evaluation. The Standards for Providers of Civil Legal Services to the Poor adopted by the American Bar Association many years ago (1986) were focused on processes and functions that should be place in the program, with the presumption that their existence would lead to high quality and effective legal work on behalf of clients. Many evaluations, in turn, have traditionally focused on whether key components of the system are in place. Thus, for example, at a program level, an evaluation might focus on a number of functions, such as whether, among other things, there is a supervisory system and whether performance evaluations are conducted.

We have come less far as a community in agreeing about standards for statewide systems than we have in agreeing about local programs standards. Nevertheless, there are helpful writings that address the issue. Alan Houseman and De Miller, among others have struggled with performance measures at a state level and with identifying the essential components of a statewide system. State planning has certainly pushed us as a community to a greater, shared understanding of what the essential elements are of an effective statewide system, but those understandings are not fully codified. It may be time for a community-wide effort to agree on state-level standards to provide a better framework for evaluations of state systems.

This paper started with a premise that more commitment to self-evaluation is appropriate in the legal services community generally, and that this is true at a statewide level, as well. In truth, there are many challenges to such evaluations. First, they can be costly. Because an evaluation of a statewide system by definition involves multiple organizations, who commissions and pays for the evaluation can be problematic. State IOLTA funders are a logical choice, but the dynamic when a funder sponsors an evaluation can in some –though certainly not all– circumstances affect how it is treated and how it proceeds, precisely because it is by a funder.

Who calls for the evaluation can be problematic, as well. Statewide systems that are well organized are more apt to have a central decision-making core that can organize and fund a meaningful evaluation. Less well organized systems are less likely to be able to accomplish what needs to be done to pull it off. Sadly, they may be those which stand to gain the most from an objective look.

In any case, it is important that, to the greatest degree possible, a statewide evaluation be seen as being on behalf of the key stakeholders, responding to their interest in improving the capacity of the system and them to serve clients.

This leads to a second, underlying question: “What are the resources that are available as part of the system?" We have the tendency to look at what we can control, or at least significantly influence, as being the resources that are part of the statewide system. Most likely, however, there is a larger pool of resources that are being deployed, including the courts (who may sponsor pro se clinics), "unorganized" private attorneys (who may represent low income persons without charge, particularly in rural communities), other organizations that serve the low income community, and even clients themselves operating in a self-help manner. How far should an evaluation extend to cover such resources?

Broadening the focus to include the full range of resources, even those beyond our control, has two distinct advantages. First, if the evaluation seeks to answer the question, "Is the system effectively responding to clients' needs?" –the more that all potential resources are considered, the more the answer to such a question is likely to be accurate. Second, for resources that are not now in our control or zone of influence, the results of an evaluation might be to suggest strategies to build partnerships and alliances with them to increase coordination and minimize duplication.

On the other hand, the wider the net that is cast regarding what are deemed to be resources "available" to the system, the more complicated the evaluation may become. Resources that serve the same communities, but which we do not control or influence, may serve very disparate objectives, or indeed, may operate without any clear objective at all. An evaluation that examines whether the goals of the system are being met, therefore, becomes more complicated where some of the resources may not share, or even be aware of those goals.

Including “outlying” resources can also significantly increase costs. It is difficult to get the necessary data to examine how effectively, for instance, private lawyers are serving low income clients on a volunteer basis, when it is not as part of an organized program panel, or if a court sponsored pro se project is effectively helping users.[1] Nevertheless, these are important questions, and even if they are answered in only a few states, the value to the overall understanding of how clients are faring in statewide systems could be invaluable.

Expanding our engagement in evaluations at all levels of the system, including the state, will increase the effectiveness of legal services. It will not happen, however, without expense and, as with all things in resource poor legal services, priority choices will have to be made. Some statewide evaluations will answer questions that will only be relevant in the state where they occurred. Others, however, will provide insight with broader application. It seems clear that we will all profit from developing a greater capacity to share what has been tried, what has worked and what has been learned in the evaluation of statewide systems, as such evaluations are developed and implemented.

[1] Obtaining such data is not out of reach. An evaluation that I conducted of the statewide pro bono system in Maryland in 1992, included a component which evaluated the extent of the contribution of private attorneys who chose not to participate on organized panels. The results were instructive and to some surprising.