1. INTRODUCTION

In I.T. evaluation we are faced with the concepts of - using stark names - Good and Evil. But what are Good and Evil? Let us use the terms in the following ways: Good means positive quality related to the information system, Evil means some definite harm, and is not just the lack of goodness. But how do we understand them? And how can we 'measure' them, or at least reveal them, in the real life situation of information systems?

1.1 Approaches to I.T. Evaluation

In what has been called the technicist approach Good and Evil refer to the technical quality of the artifact we are using - whether it is accurate, keeps crashing, interfaces with other software, is easy to use, has good documentation, uses too many resources, etc. An advantage of this approach is that we can evaluate the artifact independently of application. But this approach is very limited, and the technical quality of the artifact seldom bears any relation to its real quality (Landauer, 1996).

The user-centred approach takes account of the interaction between user and artifact. Good means easy to use, which can be differentiated further into ease of learning, ease of finding one's way around the artifact, cognitive load required when using it, and attractiveness of the user interface. Evil means it is not just not easy to use but, for example, it is actually annoying and generates negative emotions in the user. The problem with this approach is that the artifact can be easy to use and still detrimental.

In the cost-benefit approach, Good is when financial profits increase as a result of using the artifact, Evil is when they decrease. The problems of this perspective are well known: not everything can be given a financial cost, short term profits might be obtained at expense of the long term, profits might be made by the organization at the expense of others on whom it depends, and so on.

In the organization-centred approach, Good is when the processes or roles in the organization are enhanced, Evil is when they are harmed. This goes beyond the technology, its immediate users and financial measures. But it is still problematic, in that important aspects of the life of employees, such as health, morale, self-worth, are too easy to overlook in the drive for every greater competitive advantage. This can rebound on the organization later, to its detriment.

Other approaches could be, and have been, adopted, each of which seeks to remedy problems in the previous one. Mumford and Weir's Ethics (1979), for example, gives priority to job satisfaction. But how do we know that, with each such approach, we are not just making the same mistake, of elevating one aspect while ignoring another, to our eventual detriment. Also, how do we tackle situations in which the evaluation for each stakeholder is different? Sauer (1993) suggests an inclusive approach, in which if anyone uses the system then it is a success (Good) while Lyytinen and Hirscheim (1987) take the opposite approach, that if anyone dislikes it then it is a failure (Evil). But these can be rather blunt instruments, and we need something a little more precise.

1.2 Sustainability

In this paper we try to cut across this never-ending dialectical process, to seek aspects that we should consider in evaluating information systems and which transcend the interests of individual stakeholders. It is based on a pluralistic philosophy that has been of value in a number of fields, not least the assessment of urban sustainability. Lombardi (2001) proposed that sustainability, which is multi-person, multi-period, multi-disciplinary in nature (Lombardi and Brandon, 1997), can be usefully defined in terms of our functioning in a number of aspects, such that if we function well in each of the aspects then sustainability is more likely ensure than if we function poorly in any.

Use of information systems is like sustainability. Many people are involved, as developers, users, managers, suppliers, maintainers, customers, etc. and many who are indirectly affected because of changes in working practices etc. It is multi-period: short, medium and long term. It is multi-disciplinary, including technical disciplines, many that relate to the application of use, accountancy, management, marketing, production, etc. We therefore follow Lombardi's lead and consider the aspects that gave her such wide coverage in handling sustainability.

2. ASPECTS OF USE

2.1 Overview of Aspectual Functioning

Lombardi used aspects proposed by the Dutch philosopher, the late Herman Dooyeweerd. Though she adapted them for sustainability, Dooyeweerd (1955) proposed they were of general application, modes of being, aspects of reality in which we, and indeed everything in the Cosmos, functions. He identified fifteen distinct aspects, each having its own kernel meaning:

• 1. Quantitative aspect, of amount
• 2. Spatial aspect, of continuous extension
• 3. Kinematic aspect, of flowing movement
• 4. Physical aspect, of energy and mass
• 5. Biotic aspect, of life functions
• 6. Sensitive aspect, of sense, feeling and emotion
• 7. Analytical aspect, of distinction, clarity and logic
• 8. Formative aspect, of history, culture, creativity, achievement and technology
• 9. Lingual aspect, of symbolic meaning and communication
• 10. Social aspect, of social interaction, relationships and institutions
• 11. Economic aspect, of frugality, skilled use of limited resources
• 12. Aesthetic aspect, of harmony, surprise and fun
• 13. Juridical aspect, of 'what is due', rights, responsibilities
• 14. Ethical aspect, of self-giving love, generosity
• 15. Pistic aspect, of faith, commitment and vision.

The kernel meaning of each aspect is fundamentally irreducible to that of other aspects, but together they form an ordered spectrum of Meaning. The meaning of a single aspect is quite broad; for example the formative aspect covers culture, history, technology, creativity, achievement of goals, planning, formulation of artifacts, formulation of ideas, methodology, technique, and so on - everything in which human formation is central. (The reader does not need to understand all aspects; the text below will explain what is needed.)

When an individual carries out any task s/he is functioning in a number of aspects (often all of them). For example, as I write a letter, I am primarily functioning in the lingual aspect (in which meaning is conveyed by symbols). But I am also functioning in earlier aspects in order to do so (e.g. the biotic, in that I undertake life functions like breathing, digesting food, etc.), and in later aspects, which give a wider meaning or 'flavour' to my lingual functioning (e.g. the social aspect, being polite rather than rude in my letter, the juridical, in trying to give the recipient and others what is their due).

Dooyeweerd elaborated in terms of two types of relationship amongst aspects: dependency and analogy, respectively. Our functioning in dependency-related aspects is normally taken for granted and we only become aware of them in breakdown situations (as discussed for example by Heidegger). But our functioning in analogy-related aspects often guides and gives purpose to our functioning in the primary aspects. Most human activity is multi-aspectual functioning - and this includes all the activities around use of information systems.

2.2 Example of Multi-Aspectual Use of Information System

Mitev's (2001) account of the failure of the SNCF Socrate rail ticketing system provides an example of multiple aspects of failure of an I.T. system. In the quotation that follows we indicate deficient aspectual functioning by the aspect's numerical order in the list above in square brackets, and to detrimental aspectual repercussions it leads to by a similar aspect number followed by 'r':

"Technical malfunctions [8], political pressure [15], poor management [11], unions and user resistance [15] led to an inadequate [13r] and to some extent chaotic [12r] implementation. Staff training [9] was inadequate and did not prepare [13r] salespeople to face tariff inconsistencies and ticketing problems. The user interface was designed using the airlines logic and was not user-friendly [6r]. The new ticket proved unacceptable [6r] to customers. Public relations [9] failed to prepare the public to such a dramatic change [12]. The inadequate database information [7] on timetable and routes of trains, inaccurate fare information [1], and unavailability [11] of ticket exchange capabilities caused major problems for the SNCF sales force and customers alike. Impossible reservations [8] on some trains, inappropriate prices [13] and wrong train connections [3] led to large [1r] queues [2r] of irate [6r] customers in all [1r] major stations. Booked [13r] tickets were for non-existent trains [11r] whilst other trains ran empty [11r], railway unions went on strike [11r], and passengers' associations sued SNCF [13r]." [Mitev's referencing removed]

What can be seen from this brief analysis is, firstly, what a wide range of aspects contributed to the overall failure of the information system, as indicated by even this brief excerpt. In particular, it was not just an economic, managerial or technical matter. Secondly, the above example illustrates how an aspectual analysis might be started. We now discuss aspectual analysis in more detail.

2.3 Multi-aspectual Assessment of Good and Evil: Shalom

The philosophical justification for why an aspectual analysis is valid and useful for evaluating an information system is given later. Here we will briefly outline the overall approach. Each aspect has its own distinct set of laws, ranging from largely determinative laws in the earlier aspects (e.g. laws of arithmetic, laws of physics) to normative laws in the later aspects (such as laws of syntax, semantics and pragmatics in the lingual aspect, laws concerning social interaction and social institutions in the social aspect, laws of harmony, surprise and fun in the aesthetic aspect).

All our activity has repercussions, which derive from each aspect in which we function in carrying out that activity. If we function in line with the laws of an aspect then there will be positive repercussions from that functioning, while if we function against the laws of an aspect then there will be negative repercussions. For example, in I am rude in my letter then the recipient might be less disposed to grant my request. Since our activity is multi-aspectual functioning, we can expect distinct types of repercussion: social repercussions, economic repercussions, juridical repercussions, and so on. So, for any activity, there will be a profile of repercussions, positive, negative, weak or strong, which can be depicted as in Fig. 1, where bars to the right and left express positive and negative repercussions of the author's use of Deluxe Paint for creating animations.

In fact, such aspectual analysis is better if two such 'Christmas Trees' are compiled, one showing the quality of functioning around the design, development, use of the system, and the other of the repercussions. The two trees for the SNCF failure, created from the above short transcript by simply counting the positive and negative mentions of each aspect therein, would be as follows (note the one-sidedness of both, indicating a major failure):

This is the basic a means by which we can differentiate benefit from detriment, Good from Evil. We now discuss how it can be extended to address a number of normally tricky issues in I.S. use.

2.4 Unintended, Indirect, Long-Term, and Multiple Impact

Because of an ontological claim Dooyeweerd made for them (see below), the aspects transcend all situations and all stakeholders, so we can use the above type of analysis to address unintended impact, indirect impact, long-term impact and multiple stakeholders.

• First, aspectual analysis can help identify stakeholders that tend to be forgotten, if we ask ourselves, for each aspect, who might be active or who might feel repercussions in that aspect (Basden, Bergvall-Kåreborn and Mirijamdotter, in prep).

• According to this theory, unintended impacts can arise not only from lack of specialist knowledge but even more from aspects being overlooked during design, development or use. In addition to the two types of aspectual analysis mentioned above, we can analyse how much attention has been given during these processes to issues in each aspect, and plot this likewise on a tree. Aspects whose bars are short or missing are ones that have been overlooked and hence are the ones in which unintended impact is most likely to occur. Knowing the laws of these aspects, one can often gain an overview of what the unintended impacts are likely to be, and over what timescale (see below). Conversely, by carefully considering of all aspects during design and development, the incidence of unintended impact can be reduced.

• Aspectual analysis can be carried out, not only for the whole system as above, but also for each stakeholder, charting the functioning of and repercussions on each, as deemed necessary. By identifying the aspects that are most important to the various stakeholders, as requirements, hopes, expectations, perspectives and value systems, disagreements can be clarified and discussed without suppressing the valid concerns of any stakeholder.

• Indirect impacts can be analysed by chains of aspectual repercussions. As a stakeholder receives some repercussion of aspectual functioning, they respond by changing their own functioning, and so, in turn, the latter functioning generates yet more repercussions. In this way, expanding chains of aspectual repercussion are generated.

• Both long and short term repercussions can be analysed, once it is recognised that response time increases along the aspectual sequence. The repercussions of pre-formative aspects tend to be immediate, those up to the social tend to be short term, while the timescale of repercussion in post-social aspects lengthens, with up to a century for the pistic in some cases. This is because they depend on a social element for their full outworking.

• The middle aspects tend to be about information and reasoning, and thus are likely to be most intimately affected by deployment of information technology.

2.5 Use of the Aspectual Analysis

The 'Christmas tree' of aspectual functioning described above is just a start. It is a multi-purpose tool, but it presents a danger of too quantitative an approach in which we give too much attention to the pseudo-quantities designated by the lengths of the bars. Instead, we should look for patterns, such as clusters of either positive or negative functioning, and use these to tell us where further and deeper analysis is needed. We can see some examples of such clusters in Fig. 1.

• Where there is a bunch of aspects showing a cluster of benefit, then:
  • Make a more detailed analysis to ensure that the indicated benefits will actually materialize.
  • Assess critically just how important the benefits are, when set in the context of human roles. Early aspects often yield only peripheral benefit.

• When there is a bunch of aspects showing a cluster of detriment, then two things should be analysed:
  • How important is the detriment?
  • What can be done to reduce the detriment?

• Such clusters can indicate which experts to engage in the subsequent analyses, since many expert disciplines tend to centre on one aspect.

The analyses that should be stimulated by the above considerations can be carried out using Winfield's (2000) MAKE methodology. This is a method for analysing a domain of ill-structured knowledge in a manner that stimulates consideration of every aspect by those involved in the situation. It has proved to be very usable and easy to learn, and highly effective in obtaining wide aspectual coverage.

3. PHILOSOPHICAL UNDERPINNINGS

Having seen something of the utility of an aspectual approach, we can of course adopt it uncritically. But at some stage we must ask ask ourselves why this suite of aspects is any better than others, and whether it has a good foundation. One possible answer lies in the background to Dooyeweerd's proposal. We have painted a picture of an analytical tool, but if the philosophical underpinning is sound then it can developed further to become a whole integrated approach.

3.1 Escaping Plato

"All else", said A N Whitehead (1937) about what has happened in Western thinking over the last 2500 years, "is a mere footnote to Plato". The aspects are part of a much wider philosophical framework that arose from Dooyeweerd's (1955) concern that this has been so, and that it has led inexorably to fundamental problems that divorce theory from practice and rob us of philosophical foundations for meaning and purpose.

As Habermas (1972) has noted, the concept of theory "presupposed a demarcation between Being and Time". Dooyeweerd believed this presupposition to be untenable because it led to separating theory from practice. He undertook a thorough analysis of theoretical thought over the past 3,000 years, with more detailed examination of the key thinkers like Kant, and showed how it has been unable to bring together coherence and diversity, or meaning, being and time, or law and entity, or theory and practice. He showed that while proposals might enjoy acclaim over a few decades or even centuries, most eventually proved unsustainable because of the deep underlying presupposition that we can seek the basic ('divine', Clouser (1991)) Principle on which all else depends within temporal reality or experience itself. This presupposition normally leads eventually to hypostatic reductions that damage diversity or destroy coherence.

Rationalism and postivism are rightly criticised today, and Dooyeweerd makes the criticism explicit: they hypostatize (i.e. absolutize) the analytic aspect and reduce all else to it. That is, the analytic is the foundation on which all else rests, and all else shall be sacrificed to it or on its behalf, whether this be feeling, interpretation, ethics, faith, or whatever else has been ignored, suppressed or explained away down through the history of rationalism and positivism. But the answer is not interpretivism, nor even critical theory. To Dooyeweerd these two just hypostatize one of the other aspects, usually the lingual aspect of interpretation or sometimes the formative aspect of will and achievement (leading to historicism).

To Dooyeweerd, no aspect is designed to withstand the pressure of being absolutized. All are relative, referring not only to each other through the analogical and dependency relationships, but ultimately to their Creator. All depends on the Creator; nothing in temporal reality or imagination is self-dependent (Clouser, 1991).

3.2 A New Framework

However, Dooyeweerd did not only demolish; he accepted the challenge of constructing something to replace what he wished to remove. He accepted that all theoretical thought, including his own, rests on presupposition (as Habermas agrees) so instead of vainly seeking to escape presupposition (as phenomenology tried to) he sought and adopted a different one. In place of Being as the fundamental property of all we experience in temporal reality, he presupposed Meaning. This led him to conceive ontology, epistemology, theory, practice, the subject-object relationship, and particularly Good and Evil, in novel ways. This in turn allowed him to postulate a pluralistic ontology, which we have met as his suite of irreducible yet related aspects above. What is given is not Entity, but Law - laws of aspects - and entities come into existence therefrom.

This 'intertwined irreducibility' of aspects is Dooyeweerd's starting point for addressing the philosophic issue of unity and diversity and explaining both the diversity and coherence that we experience in life. This gives us hope that an aspectual analysis of information systems of information systems can be more than a device to dissect reality, and instead lead to an holistic understanding of its diversity. This is why the emphasis in aspectual analysis is not on the detail but on the patterns.

Dooyeweerd made an ontological claim for his notion of aspectuality: that aspectual laws pertain whether or not people have discovered them, are aware of them or obey or transgress them. This is the philosophic basis for there being repercussions of all activity that transcend the stakeholders' knowledge and interests, and thus the framework's ability to handle unintended consquences and multiple stakeholders. However, while the repercussions transcend the stakeholders' interests, to what extent each is relevant will depend on those very interests, and on context. This is why an aspectual analysis of stakeholders is important: to indicate where we need to make decisions of such relevance.

The philosohical underpinning for being able to differentiate Good from Evil, benefit from detriment, comes from Dooyeweerd's notion of Law. Western liberalism sees Law as constraints on freedom, because it presupposes self-dependent Being. In contrast, Dooyeweerd sees Law as enabling, a gift of the Creator that enables all in the Cosmos to be meaningful and to participate in meaningful activity that leads to a joyful, responsive, full shalom for all. Law defines the contribution to this shalom a particular aspect can make. It is because of this that Dooyeweerd maintains that if we operate in line with the laws of all aspects then things will go well, but if we transgress any then things will suffer. Kant drove a wedge between Ought and Is, and generations of technologists since then have focused on one to the detriment of the other, on technology without ethics. Dooyeweerd reconnects them because both Ought and Is are founded in Meaning, the former in the normativity of the aspects and the latter in subjection to laws of aspects.

We can see now the effect of absolutizing an aspect: other aspects become denatured, suppressed and ignored, so that we increasingly transgress their laws, to our eventual detriment. This is why we have claimed that multi-aspectual analysis is vital to information technology evualation. It is why we have been motivated to find methods for it. Since it seems to newcomers that fifteen aspects is onerous, we have developed two 'gentle' ways of approaching such analysis. One is the visual tool of the 'Christmas tree'. The other is Winfield's Multi-Aspectual Knowledge Elicitation (2000), mentioned above, which allows the analyst and client to focus on a couple of key aspects with which they are familiar, before gradually extending to all the others.

3.3 The Validity of the Aspects

Though Dooyeweerd made an ontological claim for aspects as such, he recognised that both his own proposal for what they, and also our collective theoretical knowledge of their laws gleaned through science, will always be partial and flawed. This is because both the distinguishing of aspects and the activity of science and theory-making, are functioning in the analytic aspect - and all aspectual functioning is, as we said above, relative. Dooyeweerd's view of science is discussed in detail in Clouser (1991); briefly it is that science is the isolation of an aspect in order to study its laws without interference from those of other aspects. Each aspect defines a scientific area (e.g. physical sciences, social sciences) and has a distinct epistemology, which means that each science has a different proper scientific method which should not be forced upon other scientific areas (a mistake that some positivists made). Clouser distinguishes theory making from everyday thinking, and shows how a Dooyeweerdian approach restores dignity to the latter on account of its being multi-aspectual in contrast to science's uni-aspectuality, but we do not discuss that further here.

What is of relevance here is that this should lead to humility on behalf of scientists and those who make use of specialist knowledge - and also all those who employ a suite of aspects for tasks like I.T. evaluation. While we might adopt them in day to day practice, we should always be open to the possibility that our understanding of them stands in need of refinement.

However, does this mean that we must be dominated by skepticism? Dooyeweerd would answer "No!" for two reasons. One is that, though we can never fully understand the kernels of the aspects by means of theoretical thinking, but that they may by intuitively grasped. We can never explain fully what justice is, for example, but we intuitively know, understand and recognise it, even if our intuition sometimes requires to mature. Winfield's (2000) findings support this: after a suitable short period of learning, the aspects were understood by lay clients sufficiently well as to make significant progress in MAKE.

The other is that, though Kant claimed we can never know the 'Ding an Sich' so that thinkers since Kant have presupposed what Tarnas (1991) calls the radical illegibility of the world, Dooyeweerd claimed that Created Reality tends to reveal itself to us rather than hide itself. That is, our (analytical etc.) functioning tends to be in line with the aspects as they really pertain even though flawed - so long as we do not try to absolutize any.

However we still have to justify why Dooyeweerd's suite of aspects should be better than anybody else's. Some (Hart, 1984, de Raadt, 1997) question the precise number of aspects though they do not propose any major modifications to Dooyeweerd's suite. Others, coming from very different backgrounds, have proposed suites of aspects without any reference to Dooyeweerd. Checkland suggested Five E's, Maslow gave us his famous triangle of needs. Dooyeweerd's suite seems superior to most proposals for the following reasons:

• Coverage. Dooyeweerd's has wider coverage.
  • Most other suites are found to be subsets of Dooyeweerd's.
  • Similarly, most things we experience seem to fit - and fit naturally - into Dooyeweerd's aspects, once understood (though it could be argued that this could be subject to forcing).
• Dooyeweerd's theory of modal aspects is part of a larger, coherent philosophy, not just a suite that has been offered.
  • Dooyeweerd tackled the problem of how diversity is to remain coherent rather than ending up as mere fragmentation, by positing not only irreducibility between the aspects, but also relatedness of specific kinds.
  • Dooyeweerd has made philosophical proposals about what constitutes a distinct aspect. For example, that it lacks antinomies, which arise from conflation of two aspects.
  • Being based on Meaning rather than Being, his suite is particularly amenable to handling the purpose and meaning we encounter in everyday use of information technology.
• Scrutiny. While all suites must emerge from empirical experience (Dooyeweerd argues for the importance of the latter), Dooyeweerd's suite has also been subjected to philosphical, teleological and historical scrutiny.
  • The philosophical scrutiny has involved seeking antinomies within the aspects, and setting the aspects within a framework of Meaning.
  • The teleological scrutiny has involved discussing the role of each aspect in the total spectrum of Meaning.
  • The historical scrutiny has involved a survey of 2,500 years of Western thinking to detect the aspects that thinkers have believed to be important, and how they treated them.
• Because of the latter, Dooyeweerd's suite is likely to have a cross-cultural and trans-contextual applicability that most other suites lack.
• Personal Qualities.
  • Dooyeweerd spent a life's work thinking about the aspects, with no intellectual axe to grind within the conventional views.
  • Dooyeweerd's was self-critical in the Habermasian sense, aware that presuppositions underlie all theoretical thinking, including his own. It is not clear to what extent the other proposers of suites were.

Therefore we are justified in adopting his suite as a starting point for I.T. evaluation, even though we may refine it sensitively in the process.

4. CONCLUSIONS

We have presented an approach to evaluating information technology within its context of use that is based on the notion of a suite of irreducible yet related aspects, that provides a means by which we can differentiate benefit from detriment, Good from Evil, with wide coverage. The suite of aspects discussed here has an advantage over others because it is grounded in philosophy as well as experience, and has undergone scrutiny. It came from a stream of philosophy that breaks with conventional presuppositions and provides a strong ontology that combines determinative and normative aspects of a situation, and yet avoids sinking into either positivism or interpretivism. The tenor of the whole philosophy is to bring theory and practice together, restoring the dignity of the latter.

We have illustrated how this approach might be developed into a usable methodology that can address the problems that often bedevil evaluation: multiple stakeholders, multiple timescales, indirect repercussions, etc. This is because the aspects transcend not only the individual stakeholders but even inter-cultural differences. We commend the aspectual approach to the I.T. evaluation community for examination, testing and refinement.

REFERENCES