I have just discovered that the UK Tory party has been reported as having recognized the importance of the idea proposed below and also in my discussion of the flaws in the national health service IT project and how to avoid them, posted on this web site in 2006.Their policy is reported here:
http://www.channelregister.co.uk/2009/01/27/tory_linux_push/Did they read my notes, or independently reach the same conclusion?
Posted in Enterprise, 27th January 2009 15:46 GMT
The Tory party will if elected end government over-spending on IT projects by simply choosing open source alternatives and splitting projects up, it believes.
Hmmm. Does that make the Tories the only intelligent party in the UK???
Amazing thought!
Some time ago I reacted to news reports and email discussions among Heads and Professors of Computer Science concerning the disastrous national IT project for the National Health service, whose problems some computer scientists thought (and probably still think) can be significantly reduced by using more powerful tools, including rigorous testing methodologies.Without disputing the claim that some improvement could be produced by using better tools and engineering design methodologies, I argued that the problems were much deeper and were concerned with the near impossibility of determining requirements in advance for such a large and complex project.
I produced a large web site producing a variety of theoretical and empirical arguments, emphasising the difficulty of finding out requirements for complex systems prior to building them. That is not a problem that can be solved by starting with formal methods and tools.
A related discussion occurred on a computing mailing list early in 2009, and I tried to make the point again, but much more briefly (though it is still too long). I present the summary argument below. Later I'll add some qualifications and clarifications.
Microsoft has improved a lot in recent years
As someone who has always disliked MSWindows (I'll refrain from
listing my gripes here but see this) but who has to interact with it
because my wife uses a wonderful orienteering map-making package (OCAD)
that runs only on windows, I have noticed a huge improvement as
regards stability/reliability between the years she was using
Windows 2000 and the use of XP in the last couple of years. So
whoever brought about those improvements has to be congratulated.
Microsoft certainly took note when customers complained about
security problems as the number of PCs connected to the Internet
grew.
Actually some critics predicted the problems in advance, but at that
stage it seems Microsoft paid no attention. They have now fortunately
changed significantly for the better.
E.g. when my wife's machine became infected with some nasty malware
a few weeks ago I was able to use a free microsoft service here
http://onecare.live.com/
as part of the disinfection process. (I think it required the use of
two other tools as well, though I was struggling with no real
knowledge of how MSwindows works so I don't really know what finally
removed all the symptoms.)
She certainly finds that XP is much more robust than the Windows
2000 system she had previously used.
Technical quality is not enough
But it's one thing to improve the technical quality of software,
including security and dependability: the question of 'fitness for
purpose' is a separate issue.
This is terribly important because fitness has many levels and many
aspects and some of the most important aspects cannot be formalised,
and many of those that can are simply unknown at the beginning of a
big project because of the complexity of the problems, the novelty
of what is being attempted, the variety and number of users, the
features of the social systems, equipment and institutional cultures
that already exist, and the diversity of opinions and preferences.
So even if a government agency came up with a very detailed and
rigorously defined set of requirements and even if a large computing
company produced a system that provably conformed to that
requirements specification, that would not establish fitness for
purpose because the requirements specification failed to meet the
intended more abstract requirements (which may even have been
inconsistent, or unattainable for other reasons).
This is a terribly important point that I think academic computer
scientists prefer not to have think about.
The point can be made in many ways, the starkest being that
rigorously proving that some system meets its requirements
specification, says absolutely nothing about the quality of the
requirements specification (apart from the laudable precision
that makes a formal proof possible).
The specification may be seriously unfit for the ultimate purpose,
or collection of purposes of the project.
I suspect that that is the main flaw in the UK NHS IT project, and
the tools offered by computer scientists will not, as far as I know,
help to remedy that particular flaw. They may help to remove other
flaws, e.g. frequency of crashing, or failure to do what a
government agency specified. But that does not mean the system will
therefore serve the interests of all the relevant subsections of the
community: patients, doctors, nurses, patients' relatives, ambulance
drivers, paramedics, hospital managers, etc.
When computing systems have to interact in depth with other bits of
the universe, whether they are chemical plants, weather systems,
complex flying machines, a national air traffic system, a hospital,
a school, a social service, an epidemic, a railway system, or human
brains, it is totally impossible to come up with any demonstrably
correct requirements analysis for a system that may only come into
operation several years later and is expected to go on being used
for years after that.
The only way to deal with the unattainable requirements specification
issue for large projects, especially projects of national
importance, is to accept from the beginning that
the process of design and implementation is a significant part of
the process of finding out what the requirements are.
The implications of that are very deep.
The need for open, collaborative, research and problem-solving
In general, that kind of research can no more be done effectively by a
single research team than research in hard problems in the sciences
and social sciences.
That requires not one large monolithic project with a strict
specification worked out in advance, but a lot of experiments done
in parallel, to find out what the needs are, what the unexpected
behaviours of various bits of the universe are like, and which sorts
of designs (if any) actually work well in which sorts of contexts,
where working well may itself be something for which standards
change as users learn about what is possible, and discover through
experience what they like and dislike, and as highly creative
criminals and mischief-makers discover new opportunities for their
activities.
The development of the internet (warts and all) is an existence
proof that this sort of anarchic process can produce an amazingly
complex, powerful, useful, albeit flawed, system. Many of the flaws
are not flaws in the system, but in a subset of humans who use it.
People don't know in advance what they will want
When the early processes of development of the technology that made
the internet possible started nearly 40 years ago, it would have
been impossible, to devise a set of requirements for the internet as
it actually developed. People don't know in advance what they will
want when it becomes available.
As someone who has developed user software on a small scale I
sometimes found that asking people in advance what they would like
produced incorrect information, about what they really would like
when faced with it. Sometimes my guesses about their preferences
proved better than theirs! Mockups can help, but static mockups are
no substitute for experience with a prototype.
Knowledge distribution and intellectual property
If the recommendation to replace large monolithic product
development processes wherever possible with a lot of smaller scale
exploratory problem-solving processes going on in parallel, is
accepted, that raises another problem: knowledge distribution.
Things discovered in one experiment need to be made available for
use by other experimenters -- and their end users, to maximise the
benefits of new knowledge for everyone.
This development process requires mechanisms for knowledge transfer
so that what is learnt in different places can spread to where it is
needed, including lessons about what doesn't work, and what some of
the consequences of failure can be.
The rapid growth of the internet after the basic technology had been
developed would have been impossible but for very public sharing of
ideas and solutions and rapid testing by people who were not the
original developers, including testing of modifications of
modifications, etc., and spinning off rival systems that may be far
superior to heroically produced early versions. (Remember the first
browsers?)
Compare the growth and spread of the programming language php.
That means, I think, that, whatever corporations do in their own internally
funded projects, all public funding for systems development, as for
scientific research, must impose an obligation to make the results
freely available to all, including other developers, and including testers
in non-computing disciplines who are interested in the applications -- not
just testers recruited and paid by the original developers or the original
funders.
Every system will have some flaws.
Among the many unforeseen problems of development of the internet
are the growth of spam, and the new opportunities for criminal
activities of many kinds.
Dealing with the problems that arise from deployment of a new system
need not always involve changing the system.
It is not primarily the fault of a river if someone uses it to poison
the fish caught downstream by a rival. But that only means that the
design of the system needs to be linked to adjunct services as new
problems, starting outside the service, turn up (e.g. river patrols,
and water testing may have to be added).
NOTE:
The BBC Radio 4 Analysis Programme on Inspiring Green Innovation
On Monday 6th July and Sunday 12 Jul 2009, the BBC Radio 4 Analysis
programme broadcast an investigation into alternative ways of
simulating new development in energy producing and using
technologies required to avert the looming environmental crisis.
http://www.bbc.co.uk/programmes/b00lg8hg
"Inspiring Green Innovation"
I was interested to notice that they reached conclusions very
similar to the conclusions I had reached regarding major IT
developments, namely that there should not be monolithic centrally
funded projects but many different shorter projects run in parallel,
with the possibility of learning from them and terminating the
unsuccessful ones.
They did not mention two points I had stressed:
(a) The need to ensure that contracts do not allow the companies
employed to retain intellectual property developed with government
funding: the results of both good and bad experiments must be
available to all, in order that maximum benefit can be extracted from
them.
(b) The development of the internet between the early 1970s and the
end of the century illustrates all my points.
This may mean taxpayers paying more for what is actually developed in order to make the intellectual property public, but paying for it in much smaller chunks, so that early results are freely available for others to try to use and improve on, including others who may wish to invest in developing improved versions without public funding, in order to provide commercial products or services on the basis of those improvements. (The form of licence should permit this.) So the wide availability of early and intermediate results will be of enormous public value (assuming this is not a project that absolutely has to be kept secret, e.g. because of national security issues). The value gained will include: (a) other developers not having to reinvent the good ideas in order to provide a useful starting point for some new good ideas that did not occur to the original contractor; (b) errors and failures resulting from mistaken assumptions made by the original requirements specification and other errors and failures can feed valuable information into future investigations, so that they avoid those mistakes (both design mistakes and adopting mistaken requirements/targets); (c) perhaps most importantly, if the contracts are relatively short term and results are open this gives governments the option to switch future contracts to developers with excellent new ideas, instead of being stuck with the original developer whose impoverished ideas have not been adequately exposed. In short, the higher expenditure on early prototypes, in order to keep intellectual property in the public domain may be more than offset by both lower costs in later developments (because errors are not repeated) and much higher benefits achieved because of the regular opportunities to switch attention, and funding, to new developers with new promising ideas. Someone may like to prove a conjectured theorem: the costs of particular pieces of publicly-funded technology developed on this model will usually be higher than the cost of the same technology developed in a conventional monolithic project (because the IP will need to be paid for), but the total value for money cost will be much higher in the long run because there will be far less expenditure on very large and inadequate systems, and the social benefit of the good small open products will far greater for the whole community than the benefit of similarly good items forming part of a closed monolithic product. Moreover, if production of usable freely available documentation is part of the contract, suppliers will not be able to save money by skimping on documentation in a way that may go undetected internally, but can lead to serious problems later on, e.g. difficulties of maintenance.
If the IT companies are bidding for large long term contracts they
are tempted to make promises that nobody could possibly keep because
nobody knows enough about the problems and the requirements at that
early stage.
If all they are contracted to do during early phases of a project is
do exploratory work and produce public code and documentation, and
reports on testing, they will not need to raise false long term
hopes and any promises they fail to keep will become visible at an
early stage.
Computer scientists don't like to think about all these issues: it
is much more intellectually exciting to be able to represent a
problem formally, solve it, and prove formally that it has been
solved. But acknowledging, as some experienced software engineers
do, that that is not a complete fix, and in some cases may be only a
small part of the real problem, requires changes in the way claims
are presented by the research community about how to deal with the
problems of public procurement.
One of the causes of the high quality of the Ocad package mentioned
above, was that its main developer (recently deceased alas) was also
a user of all the different services it provided: help with doing
surveys to get map data, reading in and aligning/undistorting
sketches and aerial photographs as bitmaps to provide background to
the map under development, creating and editing maps of various
sorts, producing different competitive courses based on the same
map, and printing out the information to be given to planners,
controllers, and competitors for each course, printing the maps in
different ways for different sorts of orienteering events, etc. and
finally using a map in running on a course. The main designer had
deep 'user' knowledge of all the different uses: I believe he was a
map surveyor, map maker, course planner, orienteering competitor,
etc., as well as being a software engineer.
It is rare that users have the skills and knowledge to be
developers, so alternative ways of incorporating user expertise need
to be developed, and that requires computer scientists interested in
developing complex applied systems to acquire deep knowledge of and
work closely with experts in other fields -- physics, chemical
engineering, mechanical engineering, biology, aerodynamics,
meteorology, various kinds of manufacturing process, medicine, human
psychology, hospital management, primary school teaching, or
whatever. (Even philosophers in some cases, e.g. where ethical
issues are involved or where new uses require old concepts to be
clarified and modified.)
Governments and procurement agencies have to change.
More importantly, it requires major changes in government policies,
the ways publicly funded developers operate and their expectations,
the scale of projects funded at national level, and especially
replacement of an ethos of commercial competition with one of open,
cooperative (while competitive), problem solving for the general
benefit, possibly without ever selecting a single global solution
for any of the major problems, since that can stifle further
learning.
[Although standards and deep integration can be a good thing,
diversity and the freedom to assemble components from different
sources can also make things harder for hackers and criminals, as
well as providing more seeds for future development. Has anyone
analysed the tradeoffs between the benefits of uniformity and the
costs of vulnerability and rigidity? Compare diversity in a gene
pool.]
Occasionally there are public administrators who understand the need
for flexibility. Around 2000 our department had a large grant to
enable us to acquire a multi-cpu linux computer grid to support our
research. The procedures stipulated by EPSRC for selecting a
supplier required us to specify what we wanted and then find out who
could supply it at the cheapest price. We attempted to do this and
then found that the tenders offered were not comparable because the
different suppliers made different guesses as to how much money we
had available, and how we wanted to divide it between different
pieces of technology available, and also tried to second-guess some
of the things that would impress us about their products. I realised
that the procedure was badly broken, and asked EPSRC for permission
to change the process, by telling all bidders exactly how much
money we had available, and the sorts of things we wanted to do, and
then inviting them to specify what they could provide for that sum
of money.
At first there was strong opposition to 'breaking the rules' but
fortunately there was an intelligent person at EPSRC who decided to
take the risk of giving us permission to go ahead. The result was
that we had much clearer offerings from the suppliers, including
suggestions for different ways of spending that some of money on
different combinations of their products and services. It was then
much easier for us to take a sensible decision about how to proceed,
and we ended up with a grid that provided an excellent service for
several years.
I hope similar intelligence and flexibility exists in other parts of
the civil service and government. Such flexibility could lead to
major, highly beneficial, changes in large scale procurement
procedures.
Even Microsoft ??
I have the impression that even Microsoft is beginning to understand
the importance of open, shared, problem-solving and learning, and is
gingerly(?) moving in that direction:
E.g. see:
http://www.microsoft.com/opensource/
http://www.phpclasses.org/blog/post/85-What-is-Microsoft-up-to-with-PHP.html
http://port25.technet.com/archive/2008/12/17/how-microsoft-will-support-odf.aspx
This seems to me (as an outsider with little specialist knowledge)
to indicate a seismic shift in culture at Microsoft.
[Note Added 23 Jul 2009
See http://resources.zdnet.co.uk/articles/comment/0,1000002985,39689353,00.htm
Microsoft's magnificent seven open-source options
Rupert Goodwins ZDNet.co.uk
Published: 22 Jul 2009 15:00 BST
Microsoft's magnificent seven open-source options
"Now Microsoft has officially decided that the GPL is a good thing
and is using it to release code for Linux, it's time for the
software company to take advantage of the many good things that
being a member of the open-source club brings. It's not quite the
Berlin Wall coming down - not yet - but reunification may be on the
cards....."
]
It may be too little too late, but the long term benefits, if
continued, could ultimately exceed all the good done by the Gates
Foundation! It will lead to the intellectual fragmentation of
Microsoft, and that can only be a good thing, if the fragmentation
produces multiple sub-communities within Microsoft engaging
fruitfully with multiple external communities, for mutual benefit.
No doubt other big companies that have taken steps in this direction
will continue to do so.
Perhaps the process will be facilitated by shareholders having
learnt the hard way recently that trying always to maximise short
term profits may not be the best long term policy, as that causes
the system to create imaginary profits whose unreality will sooner
or later be exposed.
Can governments and the civil service be educated too?
Some time ago Apple realised that going on extending their own operating system was not sensible because it was never initially designed for multi-user computers linked in networks where security and other concerns had to be built in to the system from the bottom levels. So they gave up their own operating system, took on an open source version of Unix (BSD, very like linux), gave it a new name, and tried to help their customers through the painful process of conversion, in part by emulation. Microsoft has employed large numbers of highly intelligent people. They are no longer encumbered by the high priest who would resist major changes analogous to the Apple strategy. Perhaps they too have learnt the great benefits of open source, namely allowing multiple experiments to be pursued in parallel with results being publicly comparable so that rapid learning is possible, and have also learnt the benefits of an operating system that from the start was designed for multiple users (we used Unix as a multi-user operating system on a DEC PDP11/40 from about 1975 at Sussex university -- though in some respects it was not then and still is not as good as some older multi-user operating systems designed for mainframed, e.g. ICL's George 4 and DEC's VMS). If they are moving towards a new more functional, more maintainable, more extendable design with major differences from their legacy windows operating systems, then all the above experiments engaging with open source development communities could be part of a strategic plan to develop a much better new operating system. How will they then deal with users who don't want to give up their own software or wait for new implementations? Simple: let them do what many linux users do now: they run linux on a powerful but inexpensive PC with a lot of core memory and disc space, and they run windows software in a virtual machine that provides the functionality required. This could be wine, or vmware or some other system. If linux users can do that now, and Microsoft know it, then I am sure they are intelligent enough to see the huge potential strategic developments in the long term if they follow the same strategy. Will they too develop a variant of Unix? Maybe they would prefer to go back to one of the more sophisticated multi-user operating systems, e.g. supporting multiple privileges (unlike Linux/Unix systems). Perhaps VMS, which is now OpenVMS? But there is not so much know how among world-wide software development communities that they could build on if they took that route -- because it is not nearly as widely used as linux/unix and their variants.
Partial index of my 'misc' pages.
Maintained by
Aaron Sloman
School of Computer Science
The University of Birmingham