Scoring companies’ sustainability performance is a hazardous mission

Albert Einstein is often quoted as having said: “Not everything that can be counted counts, and not everything that counts can be counted.”
 

This view is apparently not shared by the purveyors of corporate responsibility and sustainability indices and rankings. Their ratings seek to measure so many things, and come up with such complex systems for attributing scores to them, it would take the brain of Einstein to fully understand them.
 

According to consultancy SustainAbility, which has produced detailed research to “Rate the Raters”, there are 108 global rankings – and a plethora of national-level ones.
 

What most share is the view that getting companies to compare what they do against what everyone else is doing is the way to drive better practice.
 

One of the first such rankings was the UK-focused Business in the Environment Index, established back in 1996. It used a methodology based on the key components of an environmental management system (EMS) and produced a league table.
 

It was immensely powerful. Suddenly chief executives of top FTSE companies were seeing this list in the Financial Times and asking why they were scoring lower than their competitors.
 

In 1999, the SAM/Dow Jones Group Sustainability Indices (DJGSI) were created and, shortly after that, FTSE4good. These were explicitly investment vehicles, targeting the growing market of investors who saw sustainability issues as relevant to their investment decisions.
 

Each system had a number of things in common. They depended on companies providing them with detailed information in response to questionnaires. This created a heavy demand on companies to provide this information in the right form, and required considerable resources on the part of the ranking organisation to be able to process all the information they got.
 

These early systems evolved over time and are considered by some SRI analysts as among the most highly respected ratings tools in the marketplace, largely because they have had the time to grow and adapt in the face of experience.
 

The Business in the Environment Index became absorbed by a wider CR Index, operated by Business in the Community. But BITC continued to be hampered by a UK focus. The value for the large corporations in benchmarking was to be able to see how they measured up against their peers – and for many that peer group was a global one, not a UK one.
 

A bigger stage
 

So newer entrants began to grab the attention of commentators on the wider world stage. Some are sector-specific while some are universal. Some require companies to submit information while others use only publicly available data. Some focus on very specific issues, while others look at a broad range.
 

For Rory Sullivan, formerly head of responsible investment at Insight Investment and now an independent consultant, the ratings game has lost the value it may once have had.
 

“There are now too many benchmarks,” he says. “Some of the methodologies are very detailed, and often are not very transparent. Investors don’t have time to work through what’s useful information and what isn’t.”
 

The companies that do well in the lists do see value in them.
 

Home improvement giant Kingfisher, owner of B&Q in the UK, is currently a sector leader in the SAM/Dow Jones global index, and a “platinum” member of BITC’s CR Index. Director of corporate responsibility and government affairs Ray Baker says there have been real benefits to the company in achieving this status.
 

He says: “Once you start to establish yourself as a leader, there is no going back. It helps you to drive things forward internally because you’ve achieved this external recognition.”
 

The important thing, Baker contends, is that the bar is raised over time. “It is quite right that what was good enough two or three years ago isn’t good enough any more. We are now having to answer questions about parts of the business that previously we weren’t, such as brand management and innovation.”
 

Although Baker acknowledges that the resource burden of responding to such questionnaires is very high, he says the benefit has been that it “creates a discipline around the work we do”. But even a company such as Kingfisher finds itself having to prioritise those indices it takes part in, faced with the growth in number of requests.
 

ANZ, the Australian-based Dow Jones Sustainability Index sector leader for banking, finds itself in a similar position. Gerard Brown, the company’s corporate affairs general manager, values the work done by the key investment-focused ratings.
 

“The message it sends to our staff is very important,” Brown says. “Everyone wants to be proud of and feel good about what they do for a living and when an expert assessment says that how you go about your business and decision making is setting a high standard then that’s great recognition. We’ve found it also provides a positive platform to say ‘on the back of this we really need to get at issue XYZ’ which has obvious benefits.”
 

Does participation in the ratings help companies to make changes in what they do? Both Kingfisher and ANZ believe that the answer is yes, although they find it hard to identify specifics.
 

What gets measured gets criticised
 

Rory Sullivan believes that the impact can just as easily be a negative one. “Public indices produce a race among companies to score well,” he says. “Unfortunately, if the methodology of the index is not right it can push companies in the wrong direction in terms of what will actually improve their performance.
 

Michael Sadowski, the lead director for SustainAbility’s “Rate the Raters” research, agrees and notes particularly the worrying cases when companies tie the payment of bonuses to scores in the rankings.
 

“There is a bit of an emerging trend to tie compensation to performance on something like the DJGSI. Any company doing this is really putting themselves at risk, because the requirements of the system will then begin to become more important than things that actually lead to good performance.”
 

The issue was highlighted recently when Shell declared that it was ending its practice of tying pay to the DJGSI when it was dropped from the list on grounds that the company said were not based on its actual sustainability performance.
 

Certainly some of the rankings have their critics. One commentator, who asked not to be named, said the latest iteration of CR Magazine’s “100 Best Corporate Citizens” list was launched by Michael Porter who, in his speech, criticised “tick-box approaches” to corporate responsibility. “Some of us in the crowd looked at each other at that point, because the ‘best corporate citizens’ list could be typified as one of the worst examples of this.”
 

The Newsweek Green Rankings list has also attracted some criticism for its methodology. It includes an environmental impact score, based on “over 700 metrics” and a reputational questionnaire sent out to the users of corporateregister.com.
 

One corporate responsibility manager was dismissive of including opinion research in anything other than an openly declared opinion poll. “Do we think these individuals, knowledgeable as some of them may be, are going to have detailed, informed views about all the companies that could end up in the Green Rankings? You might as well toss a coin.”
 

Promises and achievements
 

One of the challenges facing the rankers is that it is almost impossible to genuinely compare performance in many issue areas – although some of the ratings claim to do exactly that. Best Corporate Citizens, Most Ethical Companies, Most Sustainable Companies – these are bold claims.
 

One commentator says: “Even within a sector, how am I supposed to compare an energy utility that inherited a lot of coal-fired power stations with one that has two nuclear power plants and owns prime land suitable for a windfarm?
 

The commentator argues that neither ‘absolute carbon emissions’, nor ‘rate of reduction of emissions’ gives a ranker sufficient information to make a comparison. “One company has a lower footprint. Another can achieve faster reductions. But neither can learn from the other how to play its hand better. Neither can be rated ‘good’ or ‘bad’ purely on the basis of this comparison.”
 

Another problem comes from the huge diversity of companies, how they are structured, and where their impacts come from. Indices that aim to compare companies across sectors have to come up with measures that make sense for everyone – and will often find that in so doing they create inflexible systems that cannot adapt to fit specific circumstances.
 

One company, which asked not to be named, says it has been penalised by inflexible scoring systems that were not able to adapt when their preconceptions don’t match reality.
 

In its case, it operates a number of temporary sites that have to be fuelled by oil, and it is much better for carbon emissions when it can take steps to get these sites connected to mains electricity. But the company got penalised by one ranking system that saw the electricity use go up and marked them down – even though carbon emissions had been reduced overall.
 

Another factor comes down to resources. Every ranking ends up pulling in large quantities of data, and needs to be able to process it in a reliable, knowledgeable way to generate a scoring. Some believe that, because the rankers often struggle to bring in income, they end up producing ill-considered results.
 

One company complained that it had been dropped from the DJGSI but when it approached SAM suggesting that errors had been made, they were brushed aside. “We were told that they didn’t have the resources to respond to possible errors. It didn’t leave us with the feeling that this was a very robust process, for all the ‘Dow Jones’ brand name suggests.”
 

Rodrigo Amandi, managing director of SAM, acknowledges that the question around the robustness was a valid one. SAM aims to ensure that its process is checked by a third party, and that the methodology is properly followed.
 

“Companies have to understand,” Amandi says, “that we only select the best of the pool for the indexes. There are a good number of other rules alongside the sustainability criteria – market cap, liquidity rules and so on. Being in the index is totally different to just running a sustainability assessment on a particular company.”

Michael Sadowski says that, in spite of the challenges, DJGSI is “one of the stronger ratings out there. It has evolved its approach over time, and asks questions around how sustainability is connected to the brand.”
 

But he adds that SustainAbility’s research has supported the belief that rating agencies’ resources are light in the face of the task they are trying to achieve.
 

“Across the board, research teams are very light. Very few have experience in the sector they’re dealing with. It’s a challenge for all in terms of their business model and funding. They tend to have junior teams because they simply can’t afford to pay high salaries.”
 

Conflict of interest?
 

One area that has received a lot of comment is the potential conflict of interest created by the fact that most of the rankings organisations offer some level of services to the companies being ranked. This often starts with the offer of more detailed feedback on their score. But it can extend into a full range of consultancy services. The confidential information the companies have provided them with gives the raters a thorough gap analysis of where the companies need to improve.
 

Some are very upfront about this. Goldman Sachs, for instance, has a prominent statement to investors that the company “does and seeks to do business with companies covered in its research reports. As a result, investors should be aware that the firm may have a conflict of interest that could affect the objectivity of this report.”
 

Others are less public about it. One company complained that when it received a feedback visit from BITC for the CR Index “every area where we achieved less than a perfect score was used to highlight something where we could spend additional money with BITC to address the problem. It wasn’t feedback, it was a sales call.”
 

Toby Shillito, director of the CR Index and advisory services at BITC, acknowledged it can be an issue although he pointed out that BITC is not-for-profit and support offered was connected to its charitable purpose in helping companies to improve.
 

“We have internal Chinese walls around consultancy and how the index is scored,” he adds. “Whether a company spends money with BITC or not cannot affect how they are assessed and scored within the index.”
 

Michael Sadowski sees it as a common dilemma. “Raters selling consultancy definitely creates a conflict of interest. You see plenty of examples of good ratings offering products as well. You don’t want to say absolutely you can’t sell anything, but there should be a lot more transparency over what is being done.”
 

Some of the key rankings
 

Aimed at investors:

Dow Jones Group Sustainability Indices

FTSE4Good

GS Sustain

Carbon Disclosure Project
 

Aimed at peer group/business audience:
 

CR Magazine Best Corporate Citizens

CSRHub

Ethisphere World’s Most Ethical Companies

Corporate Knights Global 100 Most Sustainable Companies

BITC CR Index
 

Aimed at general public/consumers:
 

GoodGuide

Newsweek Green Rankings
 

Case study: Ethisphere World’s Most Ethical Companies
 

Target audience
 

Companies
 

Objectives
 

Ethisphere’s World’s Most Ethical Companies was launched in 2007 “to identify those companies across 35 different industries that demonstrated leading ethical and compliance business practices”. Ethisphere did so because “we believed that honouring such companies would both educate and incentivise other organisations to strive for and achieve better ethical and compliance business practices”.
 

Key points
 

Ethisphere uses a number of measures across the areas of integrity, track record and reputation, compliance programme, industry leadership, innovation, corporate citizenship, corporate governance and, finally, tone from the top. Each of these is given a different weighting in the final score.
 

Following criticism of its lack of transparency over the methodology, and potential conflicts of interest, Ethisphere produced for its 2010 results a “media responsibility report” detailing key points about the methodology and highlighting the percentage of companies listed in the results that have a financial relationship with Ethisphere.
 

Why is it interesting?
 

Whereas most rankings measure, with varying degrees of credibility, how far companies go in managing their impacts on society, Ethisphere claims to rank companies on the degree to which they have embedded ethical values. Arguably, this is one of the trickiest claims there is in the rankings world.
 

To say that a company is compliant with the law, and has active corporate citizenship programmes, is hardly the same as saying that it is an ethical company. After all, Enron would have claimed as much before its fall.
 

One of the things the Ethisphere rating rewards is “innovation – does the company provide products, services and/or process innovation that positively contribute to public well-being?” It is not entirely clear why such criteria (with a 15% weighting for the scoring) is in this particular system. Arguably, it would discriminate against companies in sectors where there is no scope for such innovation that are, nevertheless perfectly ethical.
 

Of course, it could simply be a filter designed to keep the tobacco companies and others out of the final list. The presence of companies that are intuitively held by the public and others to be “bad by design” is often a controversial area for rankings when many such companies have become quite sophisticated on the systems that are measured and rewarded.
 

The danger of any “most ethical companies” list must surely be that it is more exposed than most when those firms that find themselves honoured get caught out doing something they shouldn’t. That danger is greater for the fact that the company entries are not independently verified.
 

Companies are quick to broadcast their presence on the list. For companies that may have had criticism from campaigners, being named one of the world’s “most ethical companies” is pretty heady stuff.
 

How does it identify leaders?
 

Companies submit applications to join the list. “Based on these applications, as well as information Ethisphere gathered throughout the year, a list of semi-finalists is created.” Exactly how this works is not clear.
 

The semi-finalists are then given an “in-depth survey questionnaire” to complete on their ethics and compliance programme, governance and corporate responsibility.
 

Ethisphere analyses these responses, along with supporting information, and allots an “EQ score”.
 

Which companies are highlighted as leaders?
 

Overwhelmingly, the “World’s Most Ethical Companies” seem to be US companies. Mind you, Ethisphere declares that Novo Nordisk is a US company, so all might not be as it seems.
 

Among the long list you will find Nike, HP, GE, PepsiCo, Google and Gap. The UK contributes Standard Chartered, Vodafone and National Grid. L’Oréal, Ikea and Westpac are among non-US companies making a showing.
 

Case study: GS Sustain
 

Target audience
 

Investors
 

Objectives
 

“To provide an objective measure of the effectiveness with which companies address the environmental, social and governance issues facing their industry.” The “GS” stands for Goldman Sachs – the owner of the rating.
 

Key points
 

GS Sustain uses about 20 sector-specific indicators to identify its high-scoring companies. It makes full details of the scoring available to the companies that have been rated, and the investors that become its customers – but not the general public.
 

Goldman Sachs uses Bloomberg information to spot check some of its data.
 

When it was launched in 2007, the company said that the framework provided “a methodology to evaluate a company’s interaction with the economy in general, its industry, society and the environment. We believe that it is a good overall proxy for the management quality of companies relative to their peers and, as such, gives insight as to their ability to succeed on a sustainable basis.”
 

Why is it interesting?
 

Most other indices score companies in absolute terms – based on whether they have in place measures and practices of which the rankers approve. Their reference point is purely one of the company, its impact, and how it compares to its peers.
 

GS Sustain, however, aims to put its measures in the context of a forward-looking vision of how the business environment is due to change. In so doing, it wants to reverse the trend towards short-term investment thinking.
 

It sees, for instance, that we are approaching a tipping point on climate change, brought about by global population growth and growing will for political action. This will increase the rate of change that will be forced upon industries, and growth in the cost of carbon emissions will be one of the change agents.
 

In that scenario, some industries will be more affected than others. In the view of Goldman Sachs, the key areas where major change will be required are oil and gas, airlines and other transport, chemicals, mining, steel and aluminium, power and non-power utilities.
 

So it focuses on companies that are well set up to succeed in the face of this likely scenario. Specifically, it looks for:
 

• companies in high carbon use industries that are leading in reducing their own consumption;
   
• companies in less exposed industries that are adjusting in the light of likely future changes; and
   
• solution providers to climate problems or their consequences – often companies in fast-growing new markets.
   

According to Michael Sadowski of SustainAbility, GS Sustain is “trail-blazing, both because of the forward-looking nature of the exercise, which is pretty unusual among ratings, and their focus on keeping it from becoming too complicated. You can see exactly how they evaluate you.”
 

How does it identify leaders?
 

GS Sustain looks at a company’s strategic positioning within its sector – some straightforward questions about how well positioned the firm is to grow in the future, based on its products, the strength of its brand and so on. This is then put alongside the company’s approach to environmental, social and governance issues. This part is used as a “proxy” for the quality of its overall management.
 

This leaves the ranking based on three parts: return on capital, industry positioning and quality of management.
 

Which companies are highlighted as leaders?
 

Exelon, Centrica, Fortum, Vale, BHP Billiton, BG Group and Monsanto – among others.