Some thoughts about league tables and public service organisations

In public service organisations excellence is more important than choice.
Aaron Sloman
Last updated: 26 May 2007

NOTE ON FORMATTING:
Adjust the width of your browser window to make the lines the length you prefer.
This web site does not attempt to impose restrictions on line length or font size.

---

DISCLAIMER
THIS IS NOT AN OFFICIAL DOCUMENT OF THE UNIVERSITY OF BIRMINGHAM OR THE SCHOOL OF COMPUTER SCIENCE.
NEITHER THE UNIVERSITY NOR THE SCHOOL HAS ENDORSED THE OPINIONS EXPRESSED HERE.

Background

This document arose in the context of discussion of how academic league tables should be used in decision making and marketing in a university department. I shall probably return to this to add more content, clarify the arguments, etc.

Should we trumpet our successes in league tables?

Many university departments, and even whole universities, that manage to obtain high rankings in some league table or other (usually concerned with either research or teaching) now advertise their ratings on web pages and marketing literature.

The implication is that the majority of universities (i.e. the ones lower in the tables) are inferior. A consequence of this is that whether such pronouncements are accurate or not they will influence decision-making in various quarters in such a way as to attract resources (through good student applications, good job applicants, funding allocations) towards a small subset of the organisations, thereby amplifying differences that already exist, or, in some cases introducing real differences in quality where previously the alleged differences were spurious.

A major flaw in this mode of management is that instead of organisations like universities, schools, hospitals, transport services, etc. cooperating to meet a spread of needs by efficient sharing of information and other resources they will be motivated to try to ensure that they remain above their peers in the league tables, in order to attract more rewards.

This will obviously degrade the overall quality of service in the majority of organisations. The published league tables give individuals the spurious belief that they are able to make well informed choices. Nobody seems to notice that an effect can be to reduce the quality for the majority, who for various reasons have to go to places that come below the top-ranked ones. The organisations that are not in the touted high quality top N percent (where N may be 1, 4, 10, 15, 20, but usually not 50 or 75) will by definition be in the majority.

If they really are less good should they not be given help and resources to enable them to improve, instead of punishment in the form of bad publicity and reduced resources? That would require a system of professional monitoring and professional mutual help, such as were provided for many years through external examiner schemes, research reviews done by high quality researchers invited from other organisations, and by internal reviews. Such mechanisms were not always used to maximal effect, in part because the people attempting to administer them were not well educated in management and administration. But replacing them with numerical targets, numerical grades and the associated league tables has, I believe, thrown out the baby with the bath-water and replaced it with a viper.

The philosophy behind government manipulation methods

That strategy is based on a model of human beings as highly efficient utility-maximising (especially self-utility maximising) machines, a view exposed with devastating effect (and a bit too much gimmickry) in the recent (March 2007) BBC TV trilogy: 'The trap' (directed by Adam Curtis) currently available online here http://www.disseminate.com/2007/04/bbcs-trap.html

Also directly accessible on google video (search on google video for 'sitruc'):

Episode 1: http://video.google.co.uk/videoplay?docid=8372545413887273321
Episode 2: http://video.google.co.uk/videoplay?docid=-7849982478877371384
Episode 3 part 1 http://video.google.co.uk/videoplay?docid=3291992041130722257
Episode 3 part 2 http://video.google.co.uk/videoplay?docid=-121006630030775636
Episode 3 part 3 http://video.google.co.uk/videoplay?docid=4729189253956590972

On that model if you allow everyone to maximise their own rewards by working out how to achieve goals set by governments for the national good, then all those individual self-optimising decisions (e.g. doing what leads to larger grants from governments) will serve the common good. (As I and others have pointed out it often simply motivates managers and workers to optimise their ratings according to the target criteria, at the cost of providing good services.)

That model of how to govern by using humans as local optimisers, was (according to evidence presented in 'The Trap') mainly inspired by the theory of games and decisions, including the work of Nobel Prize winner John Nash, which he now says (in an interview in 'The Trap') is based on an incorrect model of human thought and decision making, The over-simple mechanist model of mind seems to have influenced not only the Thatcher government but also the so-called 'New Labour' (really 'New Tory') government, with disastrous consequences (and a few small successes).

The model assumes that individuals will always do what is in their own long term best interest, and that they never really care about anything else to the same extent, e.g. helping other people who are not doing so well. The 'clever' idea was thought to be that governments could set up structures that cause all the selfish energy to have effects that are for the common good, or the good of the nation.

Some of us know that humans are capable of rising to a higher level than the model presupposes, and have seen many examples who do, though it is less likely to happen where the whole social fabric punishes those who try, for instance because really working for the common good can sometimes help other groups (e.g. departments in other universities) to improve their teaching and research so that their ratings go up relative to ours.

It should be obvious that helping another 'competing' department do better work is a worthwhile activity: and that is what many individual academics do. But the current funding model implies that that sort of activity should be punished.

Towards a better model of the problem

Here's a comparison:

A great bridge across a chasm (e.g. one dividing ignorance and knowledge, bad health from good health) rests on a collection of pillars. Some of the pillars are in excellent condition, others beginning to crumble and weaken.

The wrong way to fix the situation is to find all the best pillars and invest in making them better, while further diverting resources from the weaker pillars.

But that's exactly the policy that started in the Thatcher government and has been called 'modernisation' by the Blair and Brown twins.

It may be a good model for many commercial developments, but not for national services.

One reason it is wrong is that it completely ignores the deep logic of the notion of 'better', which I analysed in a paper based in part on a study of the structure of reports in Which? the consumer magazine in the 1960s, online here:

"How to derive 'better' from 'is'"
American Philosophical Quarterly,
Vol 6, Number 1, Jan 1969, pp 43--52

A colleague, for whom I have considerable respect, used this argument to defend the publication of league tables:

> It is much better for the facts to be known to people and for us to
> educate them about how to interpret the facts.

Exactly. And one of the deep facts is that 'better' is a multi-argument predicate yielding different orderings (often only partial orderings) as some of its arguments vary while others remain fixed. In particular, the current mechanisms only expose a very narrow, superficial and misleading subset of facts, and exposes them for bad purposes: i.e. not to help the nation improve the departments that perform less well, but in order to punish them until they improve themselves -- using dwindling resources!

One reason why league tables that try to produce a unitary ordering of quality are seriously misleading is that they are based on ignorance of the logic of 'better', in politicians, newspaper editors and the general public.

But one department, and one university, cannot change the system.

If more vice chancellors understood the issues, and understood how they have been manipulated by politicians in the last twenty years and decided to unite to resist that tyranny, then universities might become much healthier places and the nation would benefit.

But they probably will not because too many of them have absorbed the ideology promoting the model of you and me as utility-maximising engines.

Many of us are something better than that.

NOTE

I believe that the decision to introduce so-called 'top-up fees' for university students in most of the UK (a decision rejected by Scotland) was a part of this view of humans as self-optimisers, and fatally ignored many other aspects of humanity and our social and educational systems, including leaving unchanged many damaging features of our post-school educational system.

Some of these points are discussed in my letter to my MP Lynne Jones about top up fees here, and in my notes on elitism.

Maintained by Aaron Sloman
School of Computer Science
The University of Birmingham