As a project manager, have you ever wondered how similar projects chose their indicators and data collection methods? Working for an urban planning department or citizen observatory, are you keen to know how other cities measure quality of life or sustainability? As evaluator, have you seen planning tools that are utterly incoherent, suffering from indicators that aren’t appropriate or just not measurable..?
Ever since I´ve first been confronted with the need to plug indicators into logframes (yet the standard planning tool for international development projects with my previous employers Caritas, CARE and Oxfam) I´ve longed for a library. To browse, to learn, to consult, to gain inspiration. It´s just not convincing that I´d have to keep on inventing wheels (that later turn out to be squar-cut instead of round..), without recurring to existing knowledge. How often have we seen indicators being made up the night before a project funding deadline that are later ignored throughout a project´s lifetime.
Many people working in the field of monitoring and evaluation share the impression that in results-based frameworks, the selection of appropriate indicators tends to be the trickiest and often poorly done part. On reviewing any given logframe one can generally spot indicators that are incoherent with objectives. In the case of community monitoring systems, the selection of suitable indicators is of course key to the whole endeavor. For both applications, there currently is a marked lack of information resources.
To improve indicator use a plausible first step is to increase the publicly available knowledge base. Several years ago some aid agencies started compiling “indicator menus”, e.g. CARE in its “proposed new menu of impact indicators” from 2003 and Save the Children in its 2008 document. Some NGOs had gone as far as to mandate the use of certain indicators which later turned out to be costly and not workable. Based on World Vision´s experience, Alex Jacob nicly concluded that mandatory indicators don´t work.
These are useful first resources but their scope is limited and the presentation as a document never as effective (user-friendly) as a web-based library.
In order to obtain feedback on my library idea I put it twice (2010 and 2013) to the discussion of the list serve MandEnews (with more than 3500 subscribers all over the world). On both occasions, there was a lively debate about risks and benefits of standardisation yet no objection against the idea of better information resources. To the contrary, many people were keen on this, writing e.g. to “Please keep us posted on how this goes as I imagine this is of great interest to a lot of people”.
The idea is that this indicator library will be and bring
- a resource for those searching ideas for an indicator for a particular objective or monitoring theme
- a means through which non-specialist audiences can start to understand indicators more, and fear them less
- a greater consensus as to what makes a ‘good’ indicator ‘good’
- convergence towards more uniform use of certain indicators, thus facilitating comparisons and cross-learning
- a means through which we promote higher standards and accuracy
- a contribution to a better design process, making monitoring and evaluation more efficient and ultimately increasing the quality of projects
- increased interest in and ultimately growth in local sustainability monitoring / community indicator systems
A search for related initiatives reveals a fair number of existing indicator compilations. Most are sector specific (e.g. on humanitarian aid, health, AIDS, environment, peacebuilding, municial management). A few are web-based, many others are presented as simple lists or Excel sheets. The existence of this prior work presents an opportunity as authors or publishers are unlikely to claim intellectual property rights but will presumably be interested in making a free contribution. Setting up a website used to be costly the relative costs for both creation and maintenance have decreased, as have costs for translating material into several languages.
What to include? Defining scope and characteristics
The challenge is making this manageable which requires defining its scope carefully. What to include? If one considers all topics (from archaeology to zoology) and types (outcome, output, process, etc) the number of possible indicators is infinite. There are commercial initiatives such as www.kpilibrary.com claiming on its website that “Over 445,000 members find & discuss over 6,500 key performance indicators”. I´d thus suggest to target
- the field of (community) development, sustainability, and quality of life
- the level of outcomes
Something similar had been tried in the past by NEF but isn´t extensive nor updated; the other most related initiative seems to the “impact builder” developed by BOND for aid and development projects. This database, however, is unfortunately not open to the public.
Considering all that´s around and missing this is what I believe would be ambitious but really adding value: an indicator library that is
- open access (possibly with wiki-style open contributions, though that may require extra maintenance and editing)
- multi-lingual (say English, French, Spanish to begin with)
- making indicators searchable per topic (e.g.: environmental monitoring), with observations and a discussion of pros and cons, as for any topic one generally has a choice of more sophisticated and expensive and more approximate and cheaper methods
- per indicator, specific advice on appropriate data collection methods
Ideally, to make this a really useful one-stop shop the emerging website could also include
- general support to relative newcomers about the whole area of logframes, indicators, data collection methods, etc
- a conceptual framework and practical advice on potential data sources that makes it easy for local sustainability initiatives to select indicatators (taking into account very different data availability contexts in industrial countries and poorer places)
So how to go forward..?
Who wants to take part? Based on the above “dream” I wrote a concept paper and have started discussing it with a few potential partners. IISD in Canada are quite interested (as this would be an excellent supplement to their compendium of indicator initiatives) and have relevant expertise, just need a bit of funding that needs to be found elsewhere. Any ideas?