Category Archives: Data quality

What can municipal sustainability reports achieve?

due to popular demand..: reposting from VNG International’s blog (28 May 2015)


 

Our struggle for global sustainability will be won or lost in cities” is a quote attributed to UN Secretary-General Ban Ki-moon and a sentiment shared by many people. Cities across the world take their role and responsibility very seriously and are keen on applying effective management and communication tools to promote sustainable development.

From a previous assignment (employed by GIZ to support local monitoring in Ecuador) I know that many Latin American municipalities have developed interesting practices themselves and search for good examples elsewhere. In this context many local decision makers in various countries wonder about municipal sustainability reports – What are their benefits, are they worth the cost and should we start publishing them?
Surprisingly there are hardly any publications dedicated to these questions. I thus started a search for practical answers and am glad to have teamed up with VNG International. The purpose is to

  • study current practice of municipal sustainability reports (in cooperation with the University of Twente and Erasmus University Rotterdam)
  • make practical use of research results and to inform the capacity building work of VNG International in various countries.

This blog post is to outline the project and share emergent findings. A first question is of course: What is a municipal sustainability report?

There is broad consensus about the meaning of sustainability (long-term viability) and of reports being a review of the current state (plus ideally trends and forecasts) and the municipality’s relevant actions and effects.

However, there is no universal definition of and even within countries a surprising variety of sustainability reporting forms and formats, making the research question trickier than expected. One key difference is frequency: Some cities write sustainability reports annually (e.g. Dublin) while others have issued reports in four-yearly intervals (e.g. Zurich, Nuremberg).

Further, some documents are called “sustainability reports” while others have similar content under a different title. Yet other cities have combined reports: In Utrecht, for example, the city issues a yearly statistics report (“Utrecht monitor”) with many sustainability-relevant indicators and a discussion of trends which is more elaborate than what some small municipalities may write in stand-alone sustainability reports. Moreover, some claim (e.g. the Integrated Reporting Council) that “integrated reporting” is superior to separate financial and sustainability reports because of better coupling of reporting with decision-making. This argument is especially made for commercial companies but also extended to the public sector.

The city of Melbourne has actually integrated some sustainability reporting guidelines (GRI 4) into its Annual Report and Basel is exploring something similar. However, Amsterdam recently opted for the opposite: it found that integrating broad sustainability indicators into the Annual Report was not effective in all regards and will restart the publication of separate sustainability reports to reach wide audiences.

What to conclude from this quick review? It appears that municipal sustainability reports continue to be a rarity. On the other hand, several cities that are known as sustainability “frontrunners” engage in sustainability reporting and do so in varying ways and for various purposes. In this situation, a case study analysing a set of different cities with different approaches seems most revealing. Six European cities (Amsterdam, Dublin, Nuremberg, Freiburg, Zurich and Basel) were thus chosen for a more detailed analysis of effectiveness. This study is underway with several interviews yet to be held and detailed results to be published by September.

   
Guy Morin, President of the Government Council of Basel-Stadt
Simone Pflaum, Head of Sustainability Management, City of Freiburg
Marijn Bosman, Municipal councillor, Amsterdam

Some interim conclusions are worth sharing already: Municipal sustainability reports are no “Swiss army knife” that can simultaneously improve internal management and external communication. If they are started with such expectations, they risk becoming “jack of all trades and master of none”. However, if purpose, design, and integration into institutional processes match well, they can be effective tools. I look forward to continuing the reporting on the relevance of such reporting.

How to find indicators? The case for an online library

As a project manager, have you ever wondered how similar projects chose their indicators and data collection methods? Working for an urban planning department or citizen observatory, are you keen to know how other cities measure quality of life or sustainability? As evaluator, have you seen planning tools that are utterly incoherent, suffering from indicators that aren’t appropriate or just not measurable..?

Ever since I´ve first been confronted with the need to plug indicators into logframes (yet the standard planning tool for international development projects with my previous employers Caritas, CARE and Oxfam) I´ve longed for a library. To browse, to learn, to consult, to gain inspiration. It´s just not convincing that I´d have to keep on inventing wheels (that later turn out to be squar-cut instead of round..), without recurring to existing knowledge. How often have we seen indicators being made up the night before a project funding deadline that are later ignored throughout a project´s lifetime.

Many people working in the field of monitoring and evaluation share the impression that in results-based frameworks, the selection of appropriate indicators tends to be the trickiest and often poorly done part. On reviewing any given logframe one can generally spot indicators that are incoherent with objectives. In the case of community monitoring systems, the selection of suitable indicators is of course key to the whole endeavor. For both applications, there currently is a marked lack of information resources.

20130913 logframe indicators

To improve indicator use a plausible first step is to increase the publicly available knowledge base. Several years ago some aid agencies started compiling “indicator menus”, e.g. CARE in its “proposed new menu of impact indicators” from 2003 and Save the Children in its 2008 document. Some NGOs had gone as far as to mandate the use of certain indicators which later turned out to be costly and not workable.  Based on World Vision´s experience, Alex Jacob nicly concluded that mandatory indicators don´t work.

These are useful first resources but their scope is limited and the presentation as a document never as effective (user-friendly) as a web-based library.

In order to obtain feedback on my library idea I put it twice (2010 and 2013) to the discussion of the list serve MandEnews (with more than 3500 subscribers all over the world). On both occasions, there was a lively debate about risks and benefits of standardisation yet no objection against the idea of better information resources. To the contrary, many people were keen on this, writing e.g. to “Please keep us posted on how this goes as I imagine this is of great interest to a lot of people”.

20130916 library_cartoon

Expected benefits

The idea is that this indicator library will be and bring

  • a resource for those searching ideas for an indicator for a particular objective or monitoring theme
  • a means through which non-specialist audiences can start to understand indicators more, and fear them less
  • a greater consensus as to what makes a ‘good’ indicator ‘good’
  • convergence towards more uniform use of certain indicators, thus facilitating comparisons and cross-learning
  • a means through which we promote higher standards and accuracy
  • a contribution to a better design process, making monitoring and evaluation more efficient and ultimately increasing the quality of projects
  • increased interest in and ultimately growth in local sustainability monitoring / community indicator systems

Opportunities

A search for related initiatives reveals a fair number of existing indicator compilations. Most are sector specific (e.g. on humanitarian aid, health, AIDS, environment, peacebuilding, municial management). A few are web-based, many others are presented as simple lists or Excel sheets. The existence of this prior work presents an opportunity as authors or publishers are unlikely to claim intellectual property rights but will presumably be interested in making a free contribution. Setting up a website used to be costly the relative costs for both creation and maintenance have decreased, as have costs for translating material into several languages.

What to include? Defining scope and characteristics

The challenge is making this manageable which requires defining its scope carefully. What to include? If one considers all topics (from archaeology to zoology) and types (outcome, output, process, etc) the number of possible indicators is infinite. There are commercial  initiatives such as www.kpilibrary.com claiming on its website that “Over 445,000 members find & discuss over 6,500 key performance indicators”. I´d thus suggest to target

  • the field of (community) development, sustainability, and quality of life
  • the level of outcomes

Something similar had been tried in the past by NEF but isn´t extensive nor updated; the other most related initiative seems to the “impact builder” developed by BOND for aid and development projects. This database, however, is unfortunately not open to the public.

Considering all that´s around and missing this is what I believe would be ambitious but really adding value: an indicator library that is

  • open access (possibly with wiki-style open contributions, though that may require extra maintenance and editing)
  • multi-lingual (say English, French, Spanish to begin with)
  • making indicators searchable per topic (e.g.: environmental monitoring), with observations and a discussion of pros and cons, as for any topic one generally has a choice of more sophisticated and expensive and more approximate and cheaper methods
  • per indicator, specific advice on appropriate data collection methods

Ideally, to make this a really useful one-stop shop the emerging website could also include

  • general  support to relative newcomers about the whole area of logframes, indicators, data collection methods, etc
  • a conceptual framework and practical advice on potential data sources that makes it easy for local sustainability initiatives to select indicatators (taking into account very different data availability contexts in industrial countries and poorer places)

So how to go forward..?

Who wants to take part? Based on the above “dream” I wrote a concept paper and have started discussing it with a few potential partners. IISD in Canada are quite interested (as this would be an excellent supplement to their compendium of indicator initiatives) and have relevant expertise, just need a bit of funding that needs to be found elsewhere. Any ideas?