September 1996
In Proceedings IEEE Symposium on Visual Languages, pp. 240-246.
This research involved a qualitative and quantitative study of statements made by computer scientists about the ways in which they think that visual programming assists the thought processes of the programmer. This type of metacognitive knowledge has been shown in psychological experiments to have significant effects on performance in cognitive tasks. It is particularly important in the design of programming environments, where HCI factors of the environment constrain the programmer's design behaviour according to the beliefs of the environment designer. The metacognitive knowledge expressed in the visual programming literature is categorised in this paper into a range of micro-theories, the frequency of statements found in each category are compared, and the theoretical assumptions are evaluated in terms of relevant research in cognitive psychology.
Computer scientists generally apply their own metacognitive beliefs when designing new software. These beliefs may come from introspection, from discussions with colleagues, from technical literature, or from user studies. The metacognitive beliefs of computer scientists have been particularly important in the development of visual programming (VP) languages, which are a concerted attempt to facilitate the mental processes involved in programming by exploiting advanced user interface technology.
This study investigated the metacognitive knowledge that has been applied to the development of VP environments by surveying the cognitive theories reported in the VP literature. Approximately 140 papers were analysed, of which 41 made substantial statements expressing cognitive theories. These theories have been gathered into 12 categories, and the quantitative level of support for these categories is presented below.
The original sources of this metacognitive knowledge are cited only rarely. Nevertheless, many of the theories discovered do appear to have originated from research in experimental psychology and cognitive science. The apparent origins, where I was able to determine them, are also presented in the discussion of each category.
The passages that were selected from these publications were broken into individual phrases, and the content of these phrases was then grouped hierarchically. Synonymous statements from multiple sources were first grouped together. This resulted in 94 unique semantic elements, out of a total of 212 sometimes synonymous assertions. The 94 elements were then grouped by their cognitive referents into 12 topic categories. Each topic category encompassed all statements on a particular cognitive theme. The ability to group the assertions this closely shows a remarkable consistency of metacognitive concerns amongst VP researchers.
The number of semantic elements grouped within each topic provides an indication of the breadth of opinion expressed on that theme, while the number of authors addressing the topic provide an indication of the level of interest in the theme. These figures are summarised in the following table:
Topic Category | Breadth | Interest |
Straightforward advantages | 18 | 16 |
Improved productivity | 8 | 16 |
Abstraction versus concreteness | 9 | 11 |
Expression of structure | 7 | 8 |
Comparison to natural language | 7 | 10 |
Expressivity of pictures | 5 | 9 |
Mental imagery | 8 | 10 |
Resemblance to the real world | 4 | 4 |
Cognitive resources | 14 | 14 |
Naturalness of pictures | 12 | 19 |
Syntax versus semantics | 7 | 12 |
Direct icon manipulation | 5 | 5 |
We can see from this table that some topics attracted wide interest with a relatively small range of opinions (the productivity of VP languages, for example), while others (such as the cognitive resources relevant to VP) reflect different views from almost every writer. The following sections discuss the range of assertions that were found for each topic, and present some of the psychological research relevant to that topic.
These statements are best described as examples of `superlativism' - superiority in every respect. They might be natural expressions of enthusiasm from the developers of any new technology, but these attributes are not in fact, recognisably specific to programming.
Researchers including Green, Petre and Bellamy [29] have addressed some of the claims of superlativism, with simple experiments demonstrating that there are at least some common programming activites where visual languages are inferior to text languages. A more restrained point of view would be that different representations are good for different things; overstated superlativist beliefs tend to obscure the real advantages of VP.
There is little research that has investigated relative speed of learning to program, although a claim of competence without training seems unlikely. The relative ease of use of different environments can however be compared using the cognitive dimensions formulated by Green and Petre[27,28].
Despite agreement that abstraction is an important cognitive issue for VP, there is no consensus on how vision and abstraction are related. Some state that pictures are good at showing abstraction[4], or communicate a higher level of abstraction[50], while others say that abstract data is challenging for VP precisely because it is not inherently visual[10].
One interesting suggestion is that abstract cognitive models are more easily processed in well structured problems, but that concrete models are better for semi-structured problems[30]. Recent research in situated logic has suggested that some diagram formats increase the specificity in a problem representation, thereby making the solution more tractable than with formal notations [57].
The restrictions of the plane do mean that any 2-D representation can only express a certain degree of the structure in a very complex system. Application of the cognitive dimension of hidden dependencies [27,28] suggests that very complex systems are unlikely to be trivially easy to comprehend when presented visually[34]. Once a system is sufficiently complex that encapsulation becomes necessary, hidden dependencies are just as problematic in text as in visual languages.
It is also possible that a visual notation can better express the structure of the computer itself, so that we can understand its internal state[18]. This is not supported by literature on mental models of machines, where text alone is found to be just as effective as pictures alone in understanding a system [31], but this research should be extended to models of programming languages, which are usually verbal in research to date [41].
Some writers suggest that people have always found it natural to communicate with images[38], that pictures are a universal form of communication[54], and that VPs will transcend language barriers and allow exchange of programs between different nations[58] (although the vocabulary must first be designed not to be culture-specific[52]).
The grammars of programming languages are actually quite distinct from any natural language, while non-English speakers seem to have little difficulty learning the few dozen keywords in programming languages. This is not surprising, considering the fundamentally arbitrary nature of lexical assignment in natural language. In fact, English speakers may be handicapped in programming because they borrow inappropriate vocabulary from English [2].
Some cultures do have logographic rather than phonetic scripts, providing an interesting basis for speculation about VPs. Perhaps natural language imposes constraints that are removed by visual communication; if that were true, Chinese should be superior to the linear syntax of Indo-European languages[13]. This position is not universally accepted - another author observes that written languages have evolved from hieroglyphics to more advanced modern forms[36].
Some less controversial claims include the observation that images can supplement other forms of communication[1], or that pictures can be an efficient means of communication for those with language handicaps[58]. In the latter case, a parallel is sometimes drawn with the way that pictures can convey a message to the illiterate. It is suggested that VP can therefore provide a means of communication for the computer illiterate[61], or that pre-literate children can use VP systems[58].
The superior information content of pictures is often explained in terms of cognitive theories. An example is the observation that people communicate and acquire information more quickly through pictures than through text because an entire image as treated as a "single unit" by the brain, thereby providing a high communication bandwidth when compared to words, which must be recognised one at a time[23]. It is certainly true that experts in a notation remember `chunks' rather than individual elements [19], but there are strong objections to the idea that the visual sensory modality is some kind of high speed data network[37].
The spirit of the newspaper motto may be better captured by statements of attitude than of information content. These include, for example, the beliefs that people generally prefer pictures to words[54], motion pictures to books[58], and graphs to tables[23].
This is described in terms of a "semantic gap" between the programmer's conceptual model of what their program should do, and the computational model of the program itself[49]. The same effect has been described by Norman as gulfs of execution and evaluation between the user's goals and the system [44].
The claim is that this gap can be "bridged" by visual representations, allowing the user to see concepts directly[6]. This assumption of the image-like nature of thought has been a controversial one [40]. Many psychologists have argued for the existence of a mental `spatial array' in which visual percepts are stored and manipulated, and this is reflected in the VP literature by claims of structural identity between percepts and concepts [55]. On the other hand, many supposed effects of mental imagery have been attributed to propositional representations [48], in which case reading from equivalent text or graphics should result in identical concepts.
One conclusion that seems clear from psychological research is that spatial problems are best presented in spatial form. A typical spatial problem in programming is layout of a user interface, in which case environments such as Visual Basic encourage a mental model that is a close match to the goal domain (as recommended by Gutfreund[30]).
When choosing aspects of common human experience that exhibit the naturalness of VP languages, one writer identifies a number of common graphical representations that can be used directly as metaphors for programming. These include musical scores, recipe books and construction schedules[58]. Not many of us have regular experience of construction schedules, but both of the other examples are perhaps better regarded as special forms of linear text. Choosing a good metaphor for any computer application is an enterprise that certainly deserves analysis [56].
Paivio goes on to say that speech is actually interpreted in parallel, while decoding an image involves a sequence of attention, but this is not often noted in VP literature. Computer scientists are familiar with the fact that simultaneously processing two-dimensional images usually employs advanced parallel computers, and conclude that vision makes the fullest use of the human brain[42,23,38]. It is hardly necessary to point out that the brain always works in a highly parallel way, not just when processing visual data - this area of metacognitive knowledge is obviously greatly influenced by the adoption of computational metaphors for thought.
The distinction between left and right brain functions is mentioned only occasionally[53], but it seems to underlie many of the assumptions about the cognitive differences between visual and verbal processing. Interestingly, if programming tasks do involve visual cognitive systems as suggested by many of these writers, a non-visual representation would be better for programming, because it would interfere less with the visual processing of the conceptual task [3].
As evidence for this view, the fact is presented that technical specialists in many fields choose to draw diagrams in order to help them understand complex situations[15]. Reference is made to the standard diagram forms used by software engineers as being "familiar"[49], or even "traditional"[34].
Despite the weight of tradition, it is precisely these familiar graphical activities that led Petre and Green to conclude that interpretation of diagrammatic notations does not come naturally [46]. In a study of electronic circuit designers, they found that learning to interpret a diagrammatic notation involves complex skills. People who are expert at using a notation such as circuit schematics actually see different things when compared to novices viewing the same diagram.
If VP did only require skills that come naturally[14], it would no longer require training[35] or be time consuming, frustrating and labor-intensive[53]. It could be done not only by programming novices[58], but by "end-users"[34], "common users"[9], those "not familiar with computers"[33] or even "computer illiterates"[61]. This ideal of programming by intuition is not supported by psychological research.
The way in which VP can achieve this goal is often expressed as a difference between syntax and semantics; the programmer need not be concerned with the syntax of a program, which is of interest only to the computer. It is the semantics of the program, what it is supposed to do, that is important.
Psycholinguists as well as HCI specialists [43] know very well that all language has syntax, otherwise it would not be language. VP environments can remove the need for some mundane tasks, such as making sure that parentheses match or that lines are terminated by a semicolon[14], but the changes in abstraction level that hide these "low-level" syntactic constructs[34,49] are the real advances in each generation of `automatic' programming. The distinction between syntax and semantics is therefore something of a red herring. VPs cannot express semantics without syntax[50], but neither can any notation system.
Some discussions suggest that design notations contain purely semantic information, which is `translated' into syntax when the program is written. This is taken to mean either that the translation process is unnecessary when a VP is used[5] or that semantics and syntax can be written down side-by-side in a visual environment[35]. Once again, all notations have syntax, so the real advance in VP is achieved by assisting the programmer to translate a set of design semantics from one syntax to another.
It is unlikely that icons are more "intuitively meaningful"[9] than words for many programming constructs, but the effect of location as a coding mechanism[12] has more psychological support. Green and Barnard found that representational icons are easier to remember than abstract ones during initial encounters with a new environment, but that with increased familiarity, spatial location is more effective in helping to find particular functions[26].
Most of the metacognitive beliefs found in this survey have a logical basis in introspection, cognitive theory, or folk psychology. In some cases, they are well-founded, but in others there is either logical or empirical evidence making them inappropriate.
Expectations of superlativism and natural linguistic expertise are certainly unjustified, as are appeals to intuition, the capacity of the human brain (in either hemisphere) or the information content of pictures. The use of the visual system to deal with abstraction and structure through presentation of formal notations and direct manipulation metaphors may bear more fruit under the guidance of frameworks such as cognitive dimensions[27,28]. If these factors can be understood, it is more likely that we will then find the expected empirical evidence for productivity increases.
These factors should be taken into account when planning VP research or designing VP environments. This survey was an early stage of a research project that aims to define a more rigorous scientific basis for discussing the cognitive processes involved in visual programming.
1. E. Baroth & C. Hartsough, "Visual Programming in the Real World". In Visual Object-Oriented Programming Concepts and Environments, Edd. M. Burnett, A. Goldberg & T. Lewis, Manning, p.22, 1995
2. J. Bonar & E. Soloway, "Preprogramming knowledge: A major source of misconceptions in novice programmers", Human-Computer Interaction, 1(2), pp. 133-161, 1985
3. L.R. Brooks, The suppression of visualisation in reading. Quarterly Journal of Experimental Psychology, 19, pp. 289-299, 1967
4. M.H. Brown & R. Sedgewick, "A system for algorithm animation". Proc SIGGRAPH `84, p.178, 1984
5. M.M. Burnett & A.L. Ambler, "Interactive visual data abstraction in a declarative visual programming language". Journal of Visual Languages and Computing, 5(1), 1994
6. M.M. Burnett, M. Baker, C. Bohus, P. Carlson, S. Yang & P. van Zee, "Scaling up visual programming languages". IEEE Computer, March 1995
7. B.-W. Chang, D. Ungar & R.B. Smith, "Getting Close to Objects". In Visual Object-Oriented Programming Concepts and Environments, Edd. M. Burnett, A. Goldberg & T. Lewis, Manning, p.186, 1995
8. S.-K. Chang, "Visual Languages: a tutorial and survey". IEEE Software, January 1987
9. S.-K. Chang, G. Costagliola, G. Pacini, M. Tucci, G. Tortora, B. Yu & J.S. Yu, "Visual-Language System for User Interfaces". IEEE Software, March 1995
10. S.-K. Chang, "Principles of Visual Languages". In Principles of Visual Programming Systems, Ed. S-K. Chang, Prentice-Hall, p.2, 1990
11. G. Costagliola, A. De Lucia, S. Orefice & G. Tortora, "Automatic generation of visual programming environments". IEEE Computer, March 1995
12. B.J. Cox, "Object Oriented Programming: An Evolutionary Approach". Addison Wesley, p.158, 1986
13. P.T. Cox & T. Pietrzykowski, "Using a pictorial representation to combine dataflow and object-orientation in a language independent programming mechanism". Proc. Int. Comp. Sci. Conf., IEEE, 1988
14. J.L. Diaz-Herrera & R.C. Flude, "PASCAL/HSD: a graphical programming system". Proc. COMPSAC, IEEE, 1980
15. L.K. Dillon, G. Kutty, P.M. Melliar-Smith, L.E. Moser & Y.S. Ramakrishna, "Visual specifications for temporal reasoning". Journal of Visual Languages and Computing, 5(1), 1994
16. R. Dreistadt, "An analysis of the use of analogies and metaphors in science" The Journal of Psychology, 68, pp. 97-116, 1968
17. R.A. Duisberg, "Animation using temporal constraints: an overview of the animus system". Human Computer Interaction, 3(3), 1988
18. M. Edel, "The Tinkertoy graphical programming environment". Proc. COMPSAC, IEEE, 1986
19. D.E. Egan & B.J. Schwartz, "Chunking in recall of symbolic drawings", Memory and Cognition, 7(2), pp. 149-158, 1979
20. J.H. Flavell, "Metacognition and cognitive monitoring," American Psychologist 34(10), pp. 906-911, 1979.
21. L. Ford & D. Tallis, "Interacting Visual Abstractions of Programs". University of Exeter Department of Computer Science Report, RN 273, 1994
22. E.P. Glinert & J. Gonczarowski, "A (Formal) Model for (Iconic) Programming Environments". Proc. Interact `87
23. E.P. Glinert & S.L. Tanimoto, "Pict: an interactive graphical programming environment". IEEE Computer, November 1984
24. E.P. Glinert, "Nontextual Programming Environments". In Principles of Visual Programming Systems, Ed. S-K. Chang, Prentice-Hall, pp.145,148,170, 1990
25. A. Goldberg, M. Burnett & T. Lewis, "What Is Visual Object-Oriented Programming?". In Visual Object-Oriented Programming Concepts and Environments, Edd. M. Burnett, A. Goldberg & T. Lewis, Manning, p.11, 1995
26. A.J.K. Green & P.J. Barnard, "Iconic interfacing: the role of icon distinctiveness and fixed or variable screen locations", In Human-Computer Interaction - Interact `90, Edd. D. Diaper, D. Gilmore, G. Cockton, B. Shackel. Amsterdam: Elsevier, 1990
27. T.R.G. Green, Keynote address at VL `96.
28. T.R.G. Green & M. Petre "Usability analysis of visual programming environments: a 'cognitive dimensions' approach" Journal of Visual Languages and Computing (in press).
29. T.R.G. Green, M. Petre & R.K.E. Bellamy, "Comprehensibility of visual and textual programs: A test of superlativism against the `match-mismatch' conjecture" In Empirical Studies of Programmers: Fourth Workshop, J. Koenemann-Belliveau, T.G. Moher, S.P. Robertson (Eds): Norwood, NJ: Ablex, 1991
30. S.H. Gutfreund, "Maniplicons in ThinkerToy". Proc. OOPSLA 87, ACM, 1987
31. M. Hegarty & M.A. Just, "Constructing mental models of machines from text and diagrams", Journal of Memory and Language 32, pp. 717-742, 1993
32. K.-T. Huang, "Visual Interface Design Systems". In Principles of Visual Programming Systems, Ed. S-K. Chang, Prentice-Hall, p.68, 1990
33. T. Ichikawa & M. Hirakawa, "Visual programming - toward realization of user-friendly programming environments". Proc 2nd Fall Joint Computer Conference, IEEE, p.129, 1987
34. G. Karsai, "A configurable visual programming environment: a tool for domain-specific programming". IEEE Computer, March 1995
35. T.D. Kimura, A. Apte, S. Sengupta & J.W. Chan, "Form/Formula: A visual programming paradigm for user-definable user interfaces". IEEE Computer, March 1995
36. M.E. Kopache, & E.P. Glinert, "C2: A mixed textual/graphical environment for C". Proc. IEEE Workshop on Visual Languages, 1988
37. J.H. Larkin & H.A. Simon, "Why a diagram is (sometimes) worth ten thousand words", Cognitive Science 11, pp. 65-99, 1987
37. Lewis, C.M., "Visualization and situations", In Situation Theory and Its Applications, J.Barwise, J.M.Gawron, G.Plotkin & S.Tutiya (Eds):, Stanford University: CSLI, 1991.
38. K.N. Lodding, "Iconic Interfacing". IEEE Computer Graphics and Applications, 3(2), 1983
39. H.D. Lord, "Visual programming for visual applications: a new look for computing". Object Magazine, 4(4), 1994
40. C. McGuinness, "Visual imagery: The question of representation", The Irish Journal of Psychology, 10(2), pp. 188-200, 1989
41. R.E. Mayer, "From Novice to Expert", In Handbook of Human-Computer Interaction, ed. M. Helander: Elsevier, 1988
42. B.A. Myers, "Visual Programming, Programming by Example, and Program Visualization: a taxonomy, Proc. CHI 86, pp.59-66, 1986
43. B.A. Nardi, "A Small Matter of Programming: Perspectives on end-user computing", MIT Press, 1993
44. D.A. Norman, "Cognitive artifacts", In Designing Interaction: Psychology at the Human-Computer Interface, Ed. J.M. Carroll, Cambridge: Cambridge University Press, 1991
45. A. Paivio "Imagery and Verbal Processes", New York: Holt, Rinehart and Winston, 1971
46. M. Petre & T.R.G. Green, "Learning to read graphics: some evidence that `seeing' an information display is an acquired skill", Journal of Visual Languages and Computing, 4(1), pp. 5-33, 1993
47. M.C. Pong & N. Ng, "PIGS - A System for programming with interactive graphical support". Software - Practice and Experience, 13(9), 1983
48. Z.W. Pylyshyn, "The imagery debate: analogue media versus tacit knowledge", Psychological Review 88(1), pp. 16-45, 1981
49. A. Repenning & T. Sumner, "Agentsheets: a medium for creating domain-oriented visual languages". IEEE Computer, March 1995
50. S. Schiffer & J.H. Fršhlich, "Visual Programming and Software Engineering with Vista". In Visual Object-Oriented Programming Concepts and Environments, Edd. M. Burnett, A. Goldberg & T. Lewis, Manning, p.201, 1995
51. B. Shneiderman, "The future of interactive systems and the emergence of direct manipulation", Behaviour and Information Technology, 1, pp. 237-256, 1982
52. N.C. Shu, "A visual programming environment for automatic programming". Proc 21st Hawaii Int. Conf on System Sciences, IEEE, p.662, 1988
53. N.C. Shu, "Visual Programming". Van Nostrand Reinhold Company Ltd., pp.1,6, 1988
54. N.C. Shu, "Visual Programming Languages: A perspective and a dimensional analysis". In Visual Languages Edd. S.K. Chang, T. Ichikawa & P.A. Ligomenides, 1986
55. D.C. Smith, "PYGMALION: A Computer Program to Model and Stimulate Creative Thinking". Birkhaeuser, 1977
56. M. Smyth, B. Anderson, & J.L. Alty, "Metaphor reflections and a tool for thought", In People and Computers X (Proc HCI `95) Edd. M.A.R. Kirby, A.J.Dix & J.E.Finlay: Cambridge University Press, 1995
57. K. Stenning & J. Oberlander, "A Cognitive Theory of Graphical and Linguistic Reasoning: Logic and Implementation" Cognitive Science, 19(1), pp. 97-140, 1995
58. S.L. Tanimoto & E.P. Glinert, "Designing iconic programming systems: representation and learnability". Proc IEEE Workshop on Visual Languages, p.54, 1986
59. L.L. Tripp, "A survey of graphical notations for program design - an update". Software Engineering Notes, 13(4), 1988
60. E. Winograd & R.M. Soloway, "On forgetting the location of things stored in special places", Journal of Experimental Psychology: General, 115(4), 366-372 , 1986
61. W.T. Wood & S.K. Wood, "Icons in everyday life". Proc. INTERACT `87
62. R. Yeung, "MPL - A graphical programming environment for matrix processing based on constraints". Proc. IEEE Workshop on Visual Languages, 1988
63. R. Yeung, "The design and implementation of MPL: a matrix-oriented programming environment based on logic and constraints". In Visual Languages and Visual Programming Ed. S-K Chang, p. 214, 1990