Blackwell's home page.*Oracle Corporation (UK)
Empirical studies of programmers have rarely considered the cognitive effects of the group environment in which professional programmers work. This paper describes an exercise that realistically simulates group programming activity. The relevance of this type of exercise is justified by reference to current theories of social context in HCI, and a complete set of the experimental materials used in the exercise are provided in an appendix, in order to facilitate future investigations of this type.
What kind of people do we study in Psychology of Programming, and what relevance do our studies have to the lives of those people outside the experimental setting? A typical answer would be that there are two kinds of people. When we study 'novices', we are interested in how people learn programming skills. The kind of skills that we observe them acquiring during our experiment will later be useful to them in the same way as something that they would learn from a book or a lecture. When we study 'experts', we are interested in the skills that (typically) professional programmers apply in their work. We hope that we are employing and assessing those same skills in the context of an experimental task that simulates some aspect of their real work.
Both of these assumptions have been criticised. The educational assumption, that we can study a skill being developed in an experimental or educational setting that will later be 'transferred' to the real world, has been strongly criticised by Lave (1988). She describes several examples in the domain of mathematics, where impressive competence in practical situations appears unrelated to performance in a classroom or experimental setting. Suchman (1987) applied the same criticism to the study of human-computer interaction. A study of a real software design context by Bødker (1991) similarly found evidence of skills that could be observed only in the professional context, not in a laboratory situation.
Studies such as these are generally linked to a broader criticism of experimental cognitive psychology -; that laboratory experiments are irrelevant to human performance in real situations, and that symbolic cognitive models do not resemble the behaviour of an agent operating in the world. This debate is fought vigorously by Vera & Simon (1993), Agre (1993), Clancey (1993), Greeno & Moore (1993) and Suchman (1993). A defense of HCI research in the face of this debate has already been made by Green, Davies & Gilmore (1996), and we have nothing to add to it.
Nevertheless, this debate should alert us to the dangers of studying experimental tasks that are too far removed from real programming situations. Curtis (1988) described five experimental paradigms that could be used to investigate different aspects of programming situations, and two of those were singled out as having been under-represented in psychology of programming research. These were the study of group behaviour, including factors such as roles within a programming team, and organizational behaviour, investigating the effects of different structures, organisation sizes, management techniques and so on.
Ironically, these aspects of programming that we psychologists generally fail to study are precisely those aspects that take novice professional programmers by surprise when they graduate with their computer science degrees. Brooks (1975) famously observed the unexpected consequences of group interaction and management structure in professional programming contexts. An attempt is often made to teach these 'engineering' issues as a vocational component of computer science courses (for an example of British teaching, see Bott et. al. 1991), but this is one aspect of novice performance that psychologists of programming have seldom observed.
At the first PPIG student workshop, held in September 1996, the authors were invited to conduct a tutorial session aimed at communicating the nature of a professional programming environment, as distinguished from the educational programming situations that are more familiar to Psychology of Programming researchers. The tutorial was designed in the form of a role-playing exercise rather than a lecture and discussion format; it was described in the workshop programme as "Experience a typical hour in the life of a professional programmer". This format allowed a surprisingly realistic simulation of the psychological factors that are involved in medium-size (defined roughly as less than 50 people working over a period of less than two years) programming projects.
The simulated task used for this exercise was the implementation of an automated conference management system. The fact that the tutorial took place in the context of a conference meant that participants were already familiar with the domain. A specification for the system is included in the appendix.
The programming language used in the tutorial was a paper-based language, informally defined as follows:
Two hours before the start of the tutorial session, five "coordinators" were appointed. Each was given responsibility for a specific technical area of the project, and was given a copy of the system specification as well as material relevant to their own technical responsibilities. The briefing material is reproduced in full in an appendix to this paper. At the start of the session, each coordinator was given a plastic hat to wear as a sign of technical prestige. We (the two authors) started the session by introducing ourselves as the project manager and client representative for the project. The programming language was explained to the 16 participants (using an enlarged display example of three completed modules) in about 5 minutes. Participants were then told that the client would be arriving after 20 minutes to see a "proof of concept" test of a single system function. Finally, a productivity chart was unveiled. Participants were told that their performance in the tutorial would be assessed by the number of modules that they managed to implement in the course of the session. They were given a productivity goal, and told that there would be a prize for the most productive engineer.
The specification materials that were provided to coordinators were made intentionally incomplete, inconsistent and ambiguous, in order to simulate real conditions in a programming project. Coordinators were intentionally given no training or advice in how to manage teams, and were placed in the position of having to compete for resources with other coordinators (because of the fact that they did not have teams assigned to them, but had to recruit people according to their needs). The project manager corrected resource imbalances by arbitrarily removing staff from some teams and reassigning them to others, thereby reducing productivity in both teams (as in a real programming project).
|
Appoint coordinators, giving them the specification and their instructions |
start - 2 hours
|
|
Distribute work materials and copies of specification in meeting room |
start - 10 minutes |
|
Introduce session format, explain programming language, give productivity goal |
start |
|
Construction phase 1 |
start + 10 minutes |
|
'Proof of concept test' of a single registration, and productivity review |
start + 30 minutes |
|
Construction phase 2 |
start + 35 minutes |
|
Productivity review |
start + 55 minutes |
|
Construction phase 3 |
start + 60 minutes |
|
Tutorial discussion period |
start + 80 minutes |
The presentations of the system specification, coordinator roles and programming language syntax were all understood by participants without any further questions. Participants then formed very quickly into teams around coordinators near them. Some coordinators had taken the precaution of recruiting team members before the start of the session. Interestingly, the most successul recruiting was carried out by the system test coordinator, whose responsibilities were in fact the least immediate.
Most teams started work using a fairly strict approach to design. The database coordinator and timetable/budgetting coordinators in particular concentrated at first on their own design problems without interacting with members of other teams. A certain amount of interaction took place between the net transaction coordinator and the coordinator for interfaces to other systems - the system functionality was defined in terms of online net transactions, and all external data was available through external interfaces, so there was no perceived need for any internal structure to the system. This lack of internal strucucture was later regretted.
At the first 20 minute review, the client made an inspection, and expressed extreme disappointment on learning that the proof-of-concept demonstration was not complete. The project manager then performed typical management activities - asked the coordinators for estimates of how long they would take to complete their share ot the proof-of-concept functionality, chastised the engineers for their low productivity (no modules at all had yet been produced), and announced that all long-term design work had to be set aside in order to meet the project milestone on which a large progress payment depended.
After this "pep-talk", interaction between groups increased greatly. Coordinators started to enlist assistance from each other with prioritised areas of functionality and interfaces between modules. The members of the system test group, who were pleased with their state of readiness to start testing, were chastised by the project manager for their lack of productivity (they had, of course, not even started to implement any modules). It was suggested that they should do some real work - couldn't they see how busy everyone else was? They immediately joined other groups, thereby reducing the productivity of the groups they joined, as other engineers and coordinators took time out to explain their design approach to the new arrivals.
It was at this stage that the activity of the tutorial started very closely to resemble a real programming project. Coordinators spent too much time in discussion with each other, while the engineers on their teams continued with work that was about to be redesigned. Engineers tried to raise their own productivity by speaking directly to engineers in other groups, making requests for modules that they needed. Partially functional "stub" modules were implemented to support such requests, only to require later redefinition. Meanwhile, one or two enterprising engineers realised that they would never be assessed on the basis of anything other than a productivity measure that was unrelated to system functionality. They therefore created empty or meaningless modules, un-noticed by other engineers on the project (the eventual winner of the productivity prize was one of these). Most engineers in this exercise were actually quite altruistic by comparison with real programmers. Where most professional programmers are concerned with their own reward levels before system functionality, the opposite was generally true here.
During the tutorial discussion session, many of the above points were also noted by participants. They observed that the greatest problems in system implementation lay in communication between groups, and the ability to persuade other people to cooperate with one's own priorities. These skills were far more important to engineers than analysis, problem solving ability or understanding of the programming language. At an individual level, there were conflicting time demands that each engineer had to resolve for him or herself (one of those being the productivity measure). The context in which these decisions were taken included a degree of time pressure that was thought to be realistic in terms of professional programming, but far greater than the time pressures that are experienced in experimental programming tasks.
Participants wih commercial experience agreed that this exercise involved a range of cognitive demands typical of real programming projects. Despite that fact, it is very difficult to characterise those demands in terms of existing literature in psychology of programming, or even in general problem solving. It is immediately apparent that the problem statement (the material in appendix A), although easily interpreted by all participants, leaves the problem immensely under-constrained. The majority of constraints are introduced by other actors through the creation of the shared artefact - system structure and module interfaces.
The goal structure of the problem also resides in the shared artefact. Many sub-goals can be deferred to the degree that they become what is known amongst professional programmers as an "S.E.P." - somebody else's problem. This has profound effects on the goal hierarchy, because sub-trees of personal goals can effectively be pruned (transferred to other people) by negotiation or procrastination. Even top-level goals (statements in the specification) can be eliminated by negotiation with the project manager or the client.
Some of these aspects of problem solving activity in groups have been a focus of research in CSCW (computer supported cooperative work). Ironically, CSCW software tools obscure the structure of the shared artefact by removing it from the physical world. In this exercise, the perceptual salience of constraints was far greater than can be achieved on a computer screen - pieces of strings crossing the room, and large fluorescent stars were clear signals of the emerging constraint structure. Goal priorities were also reinforced by the presence of coordinators or other engineers crossing the room to shout at each other from closer quarters.
Despite the reduced perceptual range of interaction with CSCW tools, there has in the past been some hope that such tools might facilitate development of a shared artefact by enforcing a formally structured model for goal-directed discourse. It is now clear, however, that "any attempt to confine human conduct within prescriptive formal models is not only impracticable but also impossible in principle" (Mantovani, p.259). Mantovani's work in computer mediated communication (CMC) and Carroll's analysis of scenarios of use both provide frameworks in which problem solving can be studied with an awareness of social context.
Mantovani has developed a three-layer model combining social context (including an historic repertory of cultural models and social norms), the interpretation of opportunities and goals in everyday situations, and interaction with specific tools in order to perform specific tasks. Within this model, artefacts simply mediate social and cultural games - the cognitive "transparency" of the artefacts in the exercise reported above had the effect of focusing attention on these games. Carroll (1996) extends his rationale for individual scenarios of artefact use, in which he asks what are the user's goals, how is the artefact interpreted, and the task planned and executed. He studies organisational rationale in precisely analogous terms - how are the organisation's goals executed by use of the artefact in a group context? In terms of Carroll's theory, this simulation exercise successfully focused attention on managing the relationship between group and individual goals rather than on interaction with the artefact itself.
Although there have been theoretical attempts to place design and problem solving activity within a context of use, there are no well-established experimental paradigms for studying the construction of shared artefacts. The exercise reported here, although not organised for experimental data collection, did in fact provide a reasonable simulation of social factors within a controlled context that would have been suitable for data recording. Most work products were committed to paper, and verbal negotiations could also have been preserved by providing coordinators with 'minute books' in which to record the topic and conclusion of their meetings, or by analysing frequency of meetings from a video protocol (the photographs in Appendix B give an impression of the contents of such a protocol). On the basis of this material, it would be possible to compare the effects of manipulations such as the ones we carried out - separating responsibility for data storage functions from external interfaces, insisting on partial implementations for demonstration purposes, or adding people to design teams.
These manipulations are typical of real practice in commercial programming, and would also provide an empirical basis for testing theories of situated and social cognition. Although the 80 minutes of our exercise might seem ridiculously short by comparison to the one or two years of a genuine medium-size software project, we have reason to believe that the simulation was a faithful one - an experimental version lasting three hours would allow a complete implementation and test cycle to be observed.
This paper has described a reasonably successful attempt to simulate the social environment of a medium-size software project within a very short timescale. The theoretical justification for this exercise comes from current theories of social context in HCI that have seldom been applied to psychology of programming. There is little empirical evidence to support these theories in any case, so we propose that a simulation of the type reported here would be a reasonable approach to experimental manipulation and investigation of contextual effects in group problem solving.
Empirical studies based on small-scale simulations of this type would be timely not just in their contribution to theoretical HCI, but by application to the commercial practice of project management. During this exercise we were able to observe (and potentially measure) anecdotally familiar consequences of common software project management practices. An experimental investigation of this anecdotal wisdom would be valuable to project managers as well as to researchers investigating the ways that groups build and use shared artefacts.
Alan Blackwell's research is supported by the Medical Research Council and the Advanced Software Centre of Hitachi Europe Limited. The authors are grateful to Jeff James of Interval Research for reviewing the tutorial materials and to the delegates at the PPIG student workshop - especially "coordinators" David Bennett, Malcolm Bradley, Andreas Harnack, Chuck Kann and Bunny Tjaden. We apologise for spoiling an otherwise relaxed weekend!
Agre, P.E. (1993). The symbolic worldview: reply to Vera and Simon. Cognitive Science, 17(1), 61-70.
Bannon, L.J. & Bødker, S. (1991). Beyond the interface: Encountering artifacts in use. In J.M. Carroll (Ed.) Designing Interaction: Psychology at the Human-Computer Interface. Cambridge University Press.
Bott, F., Coleman, A., Eaton, J. & Rowland, D. (1991). Professional issues in software engineering. London: Pitman.
Brooks, F.P. Jr. (1975). The mythical man-month: Essays on software engineering. Reading: MA: Addison-Wesley.
Bødker, S. (1989). A human activity approach to user interfaces. Human-Computer Interaction, 4(3), 171-195.
Bødker, S. (1991). Through the interface: A human activity approach to user interface design. Hillsdale: Lawrence Erlbaum Associates.
Carroll, J.M. (1996). Becoming social: expanding scenario-based approaches in HCI. Behaviour & Information Technology, 15(4), 266-275.
Clancey, W.J. (1993). Situated action: a neuropsychological interpretation response to Vera and Simon. Cognitive Science, 17(1), 87-116.
Curtis, B. (1988). Five paradigms in the psychology of programming. In M. Helander (Ed.), Handbook of Human-Computer Interaction. Elsevier.
Green, T.R.G, Davies, S.P. & Gilmore, D.J. (1996). Delivering cognitive psychology to HCI: The problems of common language and of knowledge transfer. Interacting with Computers, 8(1), 89-111.
Greeno, J.G. & Moore, J.L. (1993). Situativity and symbols: response to Vera and Simon. Cognitive Science, 17(1), 49-60.
Greeno, J.G. (1989). Situations, mental models, and generative knowledge. In D. Klahr & K. Kotovsky (Eds). Complex information processing: The impact of Herbert A. Simon. Hillsdale, NJ: Lawrence Erlbaum Associates.
Lave, J. (1988). Cognition in practice: Mind, mathematics and culture in everyday life. Cambridge, UK: Cambridge University Press.
Mantovani, G. (1996). Social context in HCI: A new framework for mental models, cooperation and communication. Cognitive Science, 20, 237-269.
Nardi, B.A. (1992). Studying context: a comparison of activity theory, situated action models, and distributed cognition. In Proceedings of St. Petersburg HCI Conference.
Suchman, L. (1987). Plans and situated actions: The problem of human-machine communication. Cambridge University Press.
Suchman, L. (1993). Response to Vera and Simon's situated action: A symbolic interpretation. Cognitive Science, 17(1), 71-76.
Vera, A.H & Simon, H.A. (1993). Situated action: A symbolic interpretation. Cognitive Science, 17(1), 7-48.
The materials used in the course of the tutorial were published with this paper. They are available on a separate page.
Some photographs taken during the course of the tutorial session are also available.