In P. Vanneste, K. Bertels, B. De Decker & J.-M. Jaques (Eds.),
Proceedings of the 8th Annual Workshop of the
Psychology of Programming Interest Group, pp. 105-113.
Metaphor and analogy have always been important issues in the design of graphical user interfaces. They have also provided an important perspective in the study of how we acquire mental models of programming languages. They should therefore be central to the analysis of cognitive ergonomics in visual programming languages. This paper proposes a framework in which metaphor and analogy are the basis for studying abstract problem solving in the context of visual programming.
The relationship between metaphor and analogy is not always clear in the HCI literature. The desktop metaphor has clearly been valuable as an exemplar of direct manipulation, but it is hard to say how that relates to the Oxford English Dictionary's definition of metaphor as a figure of speech. The OED mentions that metaphor involves an analagous correspondance, but a figure of speech is a long way from the detailed mental models (reviewed by Wozny 1989) that led Halasz and Moran to declare analogy harmful. This paper suggests that the roles of analogy and metaphor in the psychology of programming can be separated, and the separation can be derived from cognitive theories of their function.
There have been a series of relatively reductionist models describing analogy in terms of correspondances between propositional networks that represent the base and target domains of the analogy. Gentner's (1983) structure mapping theory has been particularly influential, and has been used as a basis for experimenting with the use of analogy in programming (Chee 1993). This approach has also been developed in terms of more general analyses of problem solving goals (Holland et. al. 1986). These studies assumed that either the structure (Gentner) or semantic content (Gick and Holyoak 1983) of an analogy is more important than literal similarity, but Keane (1988) showed experimentally that problem-solvers are unlikely to identify an appropriate analogy without some surface resemblance clues.
If we follow these theorists, and divorce the problem of retrieving a suitable analogy from the question of forming the mapping, we have something very like a model of metaphor (Verbrugge & McCarrell 1977). Both the base and target domains are presented in a metaphor, so the only remaining operation is creating the mapping. Gentner has extended the structure-mapping model to analysis of relationships between domains in a metaphor (Gentner et. al. 1988), but experimental investigation of metaphor by Tourangeau and Sternberg (1982) suggests that the relationship between the base and target domains is asymmetrical - the experience of creating new structures in the target domain while interpreting the metaphor is what makes it apt.
Retrieval is a far more central concern in studies of creative analogy, which form an independent topic of research. Both Dreistadt (1968) and Shepard (1978) have reviewed the experience of creativity as reported by scientists and mathematicians. These reports tend to centre on the experience of forming mental images as a way of relating a new problem in an unstructured domain to existing experience. Lewis (1991) offers a more theoretical account of extending visual attunements to refer to other than their native situations. Kaufmann (1979) carried out experimental work to test whether individual differences in imagery were correlated with ability to interpret metaphor, and found evidence for a visual link between metaphor and analogy. He made an explicit contrast between the role of language in manipulating familiar concepts, and that of images in manipulating unfamiliar ones.
These studies lend support to the use of metaphor for analysing the terms in which a graphical user interface is presented, while analogical processes are more relevant to the problem-solving task itself. If vision also plays a special part in the retrieval process that distinguishes analogy from metaphor in a problem solving context, this forms the basis for an analysis of visual programming. Metaphor and analogy are both ways of describing abstractions in concrete terms, and the abstractions of the programming process can benefit from both techniques.
Describing the gulf between the device domain and problem domain (Norman 1991) has become a familiar paradigm in HCI theory. Payne's Yoked State Space hypothesis (Payne et. al. 1990) is one way of describing how the user manages cognitive activity in both domains. Programming tasks exacerbate this situation, because the problem domain itself consists of multiple domains, including both the application and the design.
The activity of programming involves planning the behaviour of a virtual machine that is defined by the programming language. Much of the programming task consists of maintaining a mental model of that machine. In a visual programming environment, the device domain consists of some metaphorical (in the HCI sense) presentation of the virtual machine. In LabView, for example, the program is represented as a collection of electronic devices connected together with pieces of wire.
This metaphor must be kept separate from the problem domain. Consider what would happen in LabView, if the programmer were developing a circuit simulation program. Confusion between the wires and components in their problem domain and the wires and components in the LabView device domain would make the task impossible. Nevertheless, we have observed the value of analogical processes in problem solving. This is a good reason for trying to keep analogy separate from metaphor when analysing visual programming.
Would it be possible to define a completely abstract virtual machine, so that there is no need for a device domain metaphor? This has been the goal of many programming language designers, who have developed languages such as Scheme or SML, based on pure mathematical abstractions. Students learning these languages, however, form their own concrete metaphors. This has been found in both Scheme (Eisenberg et. al. 1987) and SML (Segal & Schuman 1992), and is not surprising in view of Payne's observation of the formation of analogical models for everyday encounters with computers (Payne 1991). The advantage of explicitly presenting a device metaphor in a visual programming language is that the user can be provided with a carefully chosen basis for their mental model when planning the behaviour of the device.
Nardi and Zarmer (1993) have suggested that the most important role of a visual programming environment is not in any device metaphor, but in providing a visual formalism for the programmer to see the problem structure. Since then there has been theoretical support for the cognitive benefits of visual representations in formal reasoning tasks (Stenning and Oblerlander 1995) and also experimental evidence for the effect that this can have on mental models of programmers (Green and Navarro 1995), but the use of visual formalisms in the problem domain does not preclude the description of the device domain in metaphorical terms.
The evidence from the studies cited here supports my contention: that use of metaphor should be restricted to presentation of the task domain, and that analogy is best suited to describing operations in the task domain, where it becomes a technique for solving problems involving different levels of abstraction.
A programming project starts with specific requirements of behaviour for the program. In LabView, the programmer might understand the devices to be controlled and the required user interface, but not have the abstract components that will form the structure of the program. Analysis and design consists of creating a new set of abstractions as the conceptual tools for program construction. This derivation of abstractions from particulars lies partly in arranging data according to diagrammatic conventions that allow the designer to directly perceive abstractions (as in Bertin 1981), but diagrams also provide the opportunity for visual analogies to be formed between specific problem contexts.
The abstractions that are developed by the expert analyst can be compared to the software artefacts described by B┐dker (1989), although her analysis does not treat the software design activity itself in the same detail as the use of the system being designed. The activity of building a complex program is mediated not only by the tools provided in the programming environment, but by the conceptual tools created as part of the activity. Presenting these tools as visual components in the manner of LabView allows them to be treated as specific representatives of an abstraction (Arnheim 1970).
The programming process, when analysed in terms of the separate representations of the device and problem domains, can be seen as an iterative process. The first part of the cycle is abstraction from the task, as when a new LabView component (or `virtual instrument') is created. The second phase of the cycle is generation of new structure from those abstractions, but in terms of the device metaphor; wiring virtual instruments together in LabView, for example. In a conventional (textual) language, abstraction activities include defining new classes, types or functions - tools that will structure the generation process. Generation activites include declaration of data and production of procedural code - direct control of the virtual machine.
The iteration between abstraction and generation is not strictly ordered in most programming (Guindon 1990). It can be regarded as analogous to the parsing-gnisrap cycle (Green et. al. 1987). Both kinds of cycle alternate between decoding of information in the environment and production of an artefact, but abstraction-generation involves analogy and metaphor, where parsing-gnisrap involves literal linguistic constructs.
Alternatively, the programming process can be considered in terms of a dialogue. In conversation, when one participant makes a creative analogy that structures the topic in a new way, that analogy is presented to the other participant in the form of a metaphor. It can then become a new referent in the conversation. In the human computer dialogue of programming, the abstractions that are created in the course of analogical problem solving (new components in LabView, or classes in C++) can be integrated into the device metaphor for later manipulation in a similar way.
There are a number of experimental paradigms that may form a basis for investigating hypotheses regarding the cognitive processes involved in visual programming. In the device domain, schematic representations of pulley systems have been widely analysed. Larkin and Simon (1987) modelled the way in which visual representations can provide a basis for more efficient cognitive processes, including localised search and recognition of structure without labelling.
Hegarty and Just (1993) used a gaze-tracking method to analyse the role that a diagram of a pulley system plays in building a mental model from a description of that system. Fallside and Just (1994) carried out a similar analysis of subjects observing an animation of the schematic device alone, confirming the role that a diagram plays in focus of attention on specific components. These results provided experimental data to support a rule-based model like that used by Larkin and Simon to analyse the use of diagrams.
If these experiments with diagrams of physical mechanisms were adapted to analysis of the metaphorical device diagrams in visual programming languages, the hypothesis that metaphor is used to manipulate abstractions during problem solving could be tested in terms of the programmer's attention.
In the task domain, there have been few studies of the generation of abstraction during problem-solving, either by analogical or other means. Kaufmann (1980) conducted experiments that showed a relationship between analogical solution of novel physical problems and mental imagery. He extended this theory to the case of extremely unfamiliar problems, where physical manipulation of components became necessary. The mnemonic role of spatial context and maps in retrieval of abstract information has also been demonstrated (McNamara 1986), but this has not been related to schematic diagrams.
The use of diagrams as a means of analysing a problem domain was considered by Larkin and Simon in the context of geometric and physical problems. The role of mental images in more abstract problems has long been applied by Johnson-Laird to formal logical problems, and diagrammatic reasoning in this type of problem is now being investigated by researchers such as Stenning and Oberlander (1995).
These methodological approaches to problem-solving, spatial cognition and reasoning with diagrams provide some possible starting points for investigation of visual analogy as a means of developing abstract design concepts. If this can be combined with evidence for the utility of visual metaphors in manipulating abstract devices, the function of visual programming in the cycle of abstraction and generation can be used as a basis for separating the roles of analogy and metaphor in programming activity.
The structure of this investigation framework is summarised in the following table:
|.||Metaphor in device domain||Analogy in problem domain|
|Representation of abstraction||Metaphor constrains display of abstraction||Analogy structures problem abstraction|
|Mediation of abstract artefacts||Abstractions manipulated within the metaphor||Abstractions created from visual formalisms|
|Parsing - gnisrap with abstractions||Device structures generated from abstraction||Diagram tools support visual analysis|
The framework proposed in this paper would be usable as a basis for analysis of visual programming environments. In addition to the factors considered in earlier analyses of interface metaphors (does the metaphor allow accurate prediction of functionality, are the manipulations consistent with the metaphor, and so on), we can analyse the role that a device metaphor takes in the abstraction-generation cycle:
We can also ask whether the problem domain is represented in a way that supports analogical and abstract reasoning:
As an example of such an analysis, we could suggest ways in which these questions might be answered in the case of the LabView environment. The fundamental metaphor is one of laboratory instruments, electronic components and front panel controls, all connected together with wires. The virtual machine has a dataflow model, which might be metaphorically interpreted by the intended user base (laborarory technicians) as current flow in the connecting wires. The creation of new abstractions is described as building a new "virtual instrument" that can be connected with wires in the same way as the many instruments that are supplied with the product. The abstract content of a new virtual instrument can be expressed with a new icon to represent that instrument, and relations between a range of abstractions can be directly manipulated by connecting them via wires.
In terms of support for abstract reasoning, the LabView diagram notation has fewer characteristics of an ideal visual formalism.
The notation of LabView strongly resembles the box and wire notation of the flowchart, which is likely to be the most familiar software design notation to a user base of laboratory technicians. In fact, the flowchart is based on a control flow abstraction, whereas LabView diagrams express dataflow. This could lead to trouble in interpreting the relationships between abstract components.
Arnheim, R. (1970). Visual Thinking. London: Faber and Faber.
Bertin, J. (1981). Graphics and Graphic Information Processing. (Tr. W.G. Berg & P. Scott) Berlin: Walter de Gruyter.
Burstein, M.H. (1988). Combining analogies in mental models. In Analogical Reasoning: Perspectives of Artificial Intelligence, Cognitive Science and Philosophy, D.H. Helma (Ed), Kluwer.
B┐dker, S. (1989). A human activity approach to user interfaces. Human-Computer Interaction, 4(3), 171-195.
B┐dker, S. (1991). Through the Interface: A Human Activity Approach to User Interface Design. Hillsdale: Lawrence Erlbaum Associates.
Carroll, J.M. & Mack, R.L. (1985). Metaphor, computing systems and active learning. International Journal of Man-Machine Studies. 22(1), 39-57.
Chee, Yam San. (1993). Applying Gentner's theory of analogy to the teaching of computer programming. International Journal of Man-Machine Studies, 38(3).
Dreistadt, R. (1968). An analysis of the use of analogies and metaphors in science. The Journal of Psychology. 68, 97-116.
Eisenberg, M., Mitchell R. & Franklyn T. (1987). Understanding procedures as objects. In Empirical Studies of Programmers: Second Workshop. G.M. Olson, S. Sheppard & E. Soloway (Eds.). Norwood, NJ: Ablex.
Fallside, D.C. & Just, M.A. (1994). Understanding the kinematics of a simple machine. Visual Cognition, 1(4), 401-432.
Gentner, D. (1983). Structure-mapping: a theoretical framework for analogy. Cognitive Science, 7(2), 155-170.
Gentner, D., Falkenhainer, B., & Skorstad, J. (1988). Viewing metaphor as analogy. In Analogical Reasoning: Perspectives of Artificial Intelligence, Cognitive Science and Philosophy, D.H. Helma (Ed), Kluwer.
Gick, M.L. & Holyoak, K.J. (1983). Schema induction and analogical transfer. Cognitive Psychology, 15(1), 1-38.
Green, T.R.G., Bellamy, R.K.E. & Parker, J.M. (1987). Parsing and Gnisrap: A Model of Device Use. In Empirical Studies of Programmers: Second Workshop. G.M. Olson, S. Sheppard & E. Soloway (Eds.) Norwood, NJ: Ablex.
Green, T.R.G., Petre, M. & Bellamy, R.K.E. (1991). Comprehensibility of visual and textual programs: A test of superlativism against the `match-mismatch' conjecture. In Empirical Studies of Programmers: Fourth Workshop, J. Koenemann-Belliveau, T.G. Moher, S.P. Robertson (Eds.) Norwood, NJ: Ablex.
Green, T.R.G. & Navarro, R. (1995). Programming plans, imagery and visual programming. In Interact `95..
Guindon, R. (1990). Designing the design process: exploiting opportunistic thoughts. Human Computer Interaction 5, 305-344.
Hegarty, M. (1992). Mental animation: Inferring motion from static displays of mechanical systems. Journal of Experimental Psychology: Learning, Memory and Cognition., 18(5), 1084-1102.
Hegarty, M. & Just, M.A. (1993). Constructing mental models of machines from text and diagrams. Journal of Memory and Language, 32, 717-742.
Holland, J.H., K.J. Holyoak, R.E. Nisbett & Thagard, P.R. (1986). Induction: Processes of Inference, Learning, and Discovery. MIT Press.
Hutchins, E.L., Hollan, J.D. & Norman, D.A. (1986). Direct manipulation interfaces. In User-Centered System Design, D.A. Norman & S.W. Draper (Eds), Hillsdale, NJ: Erlbaum.
Kaufmann, G. (1979). Visual Imagery and its Relation to Problem Solving. Oslo, Norway: Universitetsforlaget.
Kaufmann, G. (1980). Imagery, Language and Cognition. Oslo, Norway: Universitetsforlaget.
Keane, M.T. (1988). Analogical Problem Solving. Ellis Horwood Ltd.
Larkin, J.H. & Simon, H.A. (1987). Why a diagram is (sometimes) worth ten thousand words. Cognitive Science, 11, 65-99
Lewis, C.M. (1991). Visualization and situations. In Situation Theory and Its Applications. J.Barwise, J.M.Gawron, G.Plotkin & S.Tutiya (Eds), Stanford University: CSLI.
McNamara, T.P. (1986). Mental representations of spatial relations. Cognitive Psychology, 18, 87-121.
Nardi, B.A. & Zarmer, C.L. (1993). Beyond models and metaphors: visual formalisms in user interface design. Journal of Visual Languages and Computing, 4(1), 5-33.
Norman, D.A. (1991). Cognitive artifacts. In Designing Interaction: Psychology at the Human-Computer Interface, J. M. Carroll (Ed), Cambridge: Cambridge University Press.
Paivio, A. (1978). On exploring visual knowledge. In Visual Learning, Thinking, and Communication, B.S. Randhawa & W.E. Coffman (Eds.) Academic Press.
Payne, S.J. (1988). Metaphorical instruction and the early learning of an abbreviated-command computer system. Acta Psychologica. 69, 207-230.
Payne, S.J. (1991). A descriptive study of mental models. Behaviour and Information Technology. 10(1), 3-21.
Payne, S.J., Squibb, H.R. & Howes, A. (1990). The nature of device models: the yoked state space hypothesis and some experiments with text editors. Human-Computer Interaction, 5(4), 415-444.
Petre, M. (1995). Why looking isn't always seeing: readership skills and graphical programming. Communications of the ACM, 38(6).
Rieman, J., Clayton L., Young, R.M. & Polson, P. (1994). Why is a raven like a writing desk? Lessons in interface consistency and analogical reasoning from two cognitive architectures. In Human Factors in Computing Systems CHI 94. 438-444.
Rumelhart, D.E. & Norman, D.A. (1981). Analogical processes in learning. In Cognitive Skills and their Acquisition. J. R. Anderson (Ed): Hillsdale, NJ: Lawrence Erlbaum Associates.
Segal, J. & Schuman, S. (1992). Empirical Studies of Learners of Functional Programming, The 5th Workshop of the Psychology of Programming Interest Group. Paris: INRIA.
Shepard, R.N. (1978). Externalization of mental images and the act of creation. In Visual Learning, Thinking and Communication, Randhawa, B.S. & Coffman, W.E. (Eds): New York, Academic Press.
Shin, S.-J. (1991). A situation-theoretic account of valid reasoning with Venn diagrams. In Situation Theory and Its Applications. J. Barwise, J.M. Gawron, G. Plotkin & S. Tutiya (Eds.) Stanford University: CSLI.
Smyth, M., Anderson, B. & Alty, J.L. (1995). Metaphor reflections and a tool for thought. People and Computers X (Proc HCI `95) M.A.R. Kirby, A.J. Dix & J.E. Finlay (Eds). Cambridge University Press.
Spiro, R.J., Feltovich, P.J., Coulson, R.L. & Anderson, D.K. (1989). Multiple analogies for complex concepts: antidotes for analogy-induced misconception in advanced knowledge acquisition. In Similarity and Analogical Reasoning, S. Vosniadou and A. Ortony (Eds.) Cambridge University Press.
Stenning, K. & Oberlander, J. (1991). Reasoning with words, pictures and calculi: computation versus justification. In Situation Theory and Its Applications. J. Barwise, J.M. Gawron, G. Plotkin & S. Tutiya (Eds): Stanford University: CSLI.
Stenning, K. & Oberlander, J. (1995). A cognitive theory of graphical and linguistic reasoning: Logic and implementation. Cognitive Science, 19(1), 97-140.
Tourangeau, R. & Sternberg, R.J. (1982). Understanding and appreciating metaphors. Cognition, 11(3), 203-244.
Verbrugge, R.R. & McCarrell, N.S. (1977). Metaphoric comprehension: studies in reminding and resembling. Cognitive Psychology, 9, 494-533.
Wozny, L.A. (1989). The application of metaphor, analogy and conceptual models in computer systems. Interacting with Computers,. 1(3), 273-283.
Click to return to Alan Blackwell's home page.