This paper is also available as PDF (suitable for printing).

Extract from Blackwell, A.F. (1998). Metaphor in Diagrams
Unpublished PhD Thesis, University of Cambridge.

Chapter 3 - Metacognition among Diagram Users

These paradigms change the very way you think. They lead to new habits and models of behaviour that are more powerful and productive. They can lead to a human-machine synergism.
Smith, Irby, Kimbal, Verplank & Harslem
(1982), p. 272.

As described in chapter 2, a class of diagrammatic tools which support particularly complex problem solving is the class of visual programming languages (VPLs). VPLs are also unusual in that they have a relatively recent history compared to other types of diagram, and it is therefore possible to investigate the reasons why they have been developed and promoted. Green, with various collaborators, has made many empirical studies of VPL use. This empirical approach to comparing the relative advantages of different programming languages can be contrasted with superlativist claims for the superiority of VPLs (Green, Petre & Bellamy 1991). Several studies have supported Green's contention that superlativism is unjustified - the empirical evidence is reviewed by Whitley (1997), and dates back to comparisons finding no difference in performance between flowchart users and those writing plans in English (Soloway, Bonar & Ehrlich 1983).

Despite the paucity of empirical evidence, and the availability of analytic tools such as Green's Cognitive Dimensions, superlativism has been widespread in VPL research. At one level, superlativist claims might appear ridiculous, as though someone were advocating the replacement of written language with images (Dondis 1973, Barker & Manji 1989). In fact, some researchers candidly acknowledge the lack of empirical evidence for the benefits of their work, and suggest that novelty is sufficient justification for new notations to be developed (e.g. Hyrskykari 1993). Other developers of VPLs report negative findings after empirical evaluations of their own projects, but this is seldom regarded as a reason to abandon the project (e.g. Ladret & Rueher 1991, Grant in press). It is far more common, however, to expend considerable effort on designing graphical alternatives to existing textual notations without ever questioning why the textual notations are inadequate (e.g. Missikoff & Pizzicanella 1996).

This chapter does not evaluate the empirical evidence for and against visual languages, nor the theoretical considerations that cause an information structure to be well suited to a particular task. Instead, it investigates the metacognitive assumptions that underlie the superlativist claims made in VPL research. Those assumptions are generally based on the beliefs that VPL researchers hold about the nature of problem-solving and programming, and about the nature and uses of diagrams. These metacognitive beliefs - beliefs about the nature of our own thought processes (Flavell 1979) - are important for several reasons. Firstly, metacognitive beliefs can have significant effects on the way that people choose cognitive tools. For example, many people believe that hiding an object in an unusual place will make it easier to remember. This particular metacognitive belief is unfounded (Winograd & Soloway 1986), a fact which is useful to know when planning one's household affairs.

Secondly, the metacognitive beliefs of VPL designers influence the choices that they make when designing new languages, and hence affect the characteristics of the languages themselves. These beliefs may not be questioned where they are in accordance with popular introspection, as observed by Meyer (1997) in his reanalysis of Washburne's classic study of graphs. Thirdly, metacognitive beliefs influence the strategies that people use when solving problems using diagrams (Schoenfeld 1983). Users of a VPL will similarly use the tool in ways determined by their own beliefs about the value of diagrams. Unfortunately, metacognition is not always an accurate guide for performance in comprehension and problem-solving: Glenberg, Wilkinson and Epstein (1982) found that 92% of experimental subjects were certain they had understood a passage that was actually self-contradictory, while Metcalfe and Wiebe (1987) found that success in solving insight problems was unrelated to subjects' expectations of their performance. This chapter presents three studies of metacognitive beliefs. All three survey the opinions of experts regarding the cognitive advantages of VPLs, but the three survey populations have very different experiences of visual programming. The first survey investigates the metacognitive beliefs of VPL researchers, as published in the visual programming research literature. The second compares the opinions of experienced programmers who had not used a visual programming language. The third investigates the opinions of programmers who are experienced users of a commercially available VPL.

Survey 1: Metacognitive statements in the computer science literature

This first study investigates the systematic nature of metacognitive beliefs among VPL researchers. Other fields of computer science exaggerate the intuitiveness of the innovations they introduce, but the underlying assumption is often a straightforward simplification (such as the contention that an "object" in object-oriented programming is as easy to identify and manipulate as an object in the real world - Blackwell 1993). A corresponding simplification in VPL research (as in research into diagrams and illustration) might be an appeal to the 1920s marketing slogan, now would-be Chinese proverb "a picture is worth a thousand words" (Mieder 1990, Blackwell 1997a). The justification for VPL research is seldom so simple, however. This study investigates the range of statements that have been made by VPL researchers.

The results of this study have previously been published in Blackwell (1996b).

Population

The main source of data for this survey was a two-volume collection of well-known papers in visual language research, published by the IEEE (Glinert, Ed. 1990a, 1990b). This was supplemented by several textbooks on visual programming languages (Shu 1988b, Chang, Ed. 1990a, 1990b, Burnett, Goldberg & Lewis, Eds. 1995), by two volumes of the Journal of Visual Languages and Computing, and by a search for visual programming articles in two popular computer science journals (Communications of the ACM and IEEE Software) over the period after the publication of the Glinert collection. Approximately 140 publications were considered in all. The original publication dates ranged from 1977 to 1995, with a median year of 1988.

Method

The first stage in data collection involved the identification of passages in these papers where the research was justified, or the significance of the results discussed, in terms not directly derived from the technology described. These passages generally appeared in either the introduction or the concluding sections of the paper. A statement such as "visual programming languages will result in improved productivity because the human mind is optimised for vision" (not an actual quote from the study) would be a typical target for analysis.

Once these passages had been collected, they were divided into individual phrases, and each phrase was classified according to the theme that it addressed. This segmentation and classification was repeated at two different stages of the research. The initial classification (described in Blackwell 1996b) considered only the material collected in the current survey (i.e. it did not include distinctions discovered during the analysis of surveys 2 and 3), and was not based on any external reference point. The second classification considered material collected in all three of the surveys that are described in this chapter, and provided a common coding framework for all three sets of data. It is the later of the two classifications that is described here. An analysis based on this second classification has been published previously by Whitley and Blackwell (1997).

The classification framework was based on an initial sample (20%) of the material from all three surveys, which was considered by two coders - coding decisions regarding this sample were discussed at length, in order to clarify the interpretation of each theme. A separate sample was used to assess inter-rater reliability at the end of the third survey. The theoretical motivation for the framework was mixed, addressing relatively independent research questions arising from the respective projects of Whitley and myself. Nevertheless, each phrase in the survey material was allocated uniquely to one theme in the classification framework. The allocation of phrases to independent research questions is most relevant in survey 3, and is discussed further there.

Sample responses

40 passages describing metacognitive beliefs were identified in the corpus of 140 research publications surveyed. In the following summary of the themes that were addressed, publications are identified by an index number. The actual publications are listed in table 3.1, with page numbers identifying the locations of those passages which were found within longer texts - full citations can be found in the bibliography.

Table 3.1. Survey sources in visual programming literature

Themes

The following discussion describes the themes that were used in the final classification of phrases from all three surveys. An overall hierarchical arrangement of the themes is illustrated in figure 3.1. Where superscript index numbers in brackets appear in the theme descriptions (e.g. "[31]") these refer to specific sources cited in table 3.1.

A) Contextual significance

This category includes several themes describing the benefits that might be found when VPLs were applied to an actual task, possibly in a commercial programming environment. It is broken down into three themes: the first describes the general impact of VPLs, the second specifically considers the benefits of VPLs in learning to use a new language, and the third considers the productivity that can be achieved using a VPL.

A.1) General impact

Many statements found in this survey described visual languages as being easy to use (user friendly[8], helpful[19], straightforward[9], reliable[4] etc.) without mentioning any specific benefits or justifications. Often these statements were extended to claims about the particular relevance of VP to classes of users who may find textual programming to be too difficult (e.g. students[23,39] "end-users"[24], "common users"[7], or pre-literate children[37]). As no reasons are given for these statements, they can be regarded as direct expressions of the superlativism described by Green, Petre and Bellamy (1991). These statements provide little further ground for analysis, although it is interesting to note that it is apparently easy to publish unsupported statements such as this in well-regarded scientific forums. Furthermore, the positive attributes described are not recognisably specific to programming.

A.2) Learnability

A very common statement of the main advantage of VPLs is that they will no longer require effortful learning or training[25,39], because they use skills that come naturally[12] use native intelligence[18] or are intuitively meaningful[7,9] even to people who are not familiar with computers[23]or computer illiterates[39]. Software developers often claim the virtue of intuitiveness for their interface designs, while being unclear of how this has been achieved. In the statements here, the main justification seems to be that the VPL can be understood immediately by analogy from previous experience. This belief is considered as the main concern of theme D.1, but it should be noted at this point that new notations are far more likely to require lengthy acquisition of expertise (Petre & Green 1993), rather than being accessible, immediate and obvious[37].

A.3) Productivity

Professional programmers are highly concerned with their productivity - how fast they can complete a project. For the professional, this is even more important than the ease of learning a new language, because learning a language is a worthwhile investment if it results in increased productivity. This issue is regularly mentioned in the two surveys of professional programmers, but also receives some attention in the research literature[19,32]. These are generally simple statements of opinion or anecdotal reports of widespread success[17], rather than citing any empirical evidence, although one of the papers covered in the survey does report an empirical comparison of projects using a VPL and a text language[1]. The reasons why a language might increase productivity include ease of use[4], of writing[29], and of modification[20]. Readability (theme B.1) possibly has even more impact on productivity, but it was the professional programmers in the second two surveys who were more likely to consider software development tasks other than coding.

B) Notational characteristics

This category includes those statements that concentrate on the characteristics of the notation itself, rather than the context in which it is used, or the cognitive processes involved in interpretation and creation. These other issues are always implicit, of course, so this is a matter of emphasis rather than discontinuity. The category is divided into six themes, most of which can be related to one or more of Green's cognitive dimensions (Green 1989, Green & Petre 1996): readability (hidden dependencies, visibility, diffuseness) is the most common concern when evaluating a notation; documentation (secondary notation, role expressiveness) describes the value of a notation for communicating with other people; and syntax reduction (repetition viscosity, premature commitment) describes the ways in which a notation can obstruct manipulation. Modularity (abstraction gradient, knock-on viscosity) and power (diffuseness, abstraction gradient) consider whether the notation supports standard software engineering practice, while real projects also require programs that mix notations, because most software must process text, whether or not the program is created diagrammatically.

B.1) Readability

Respondents in the surveys of professional programmers (surveys 2 and 3) paid great attention to the value of a VPL in reading programs, rather than in creating them. They also tended to consider specific syntactic constructs or overall views of program function. Those responses are discussed in more detail later - the research literature sampled in this survey emphasised more general questions. Graphical notations were described as being generally easier to comprehend than text[41,37], largely because they explicitly represent meaningful relationships between elements[4,32]. This is in accordance with Larkin and Simon's (1987) model of locality and connectivity in diagram use. There is an element of overstatement, however - complex systems are unlikely to be trivially easy to comprehend[24], even when presented visually, because it is seldom possible to represent all possible relationships in two dimensions (the cognitive dimension of hidden dependencies). It is also suggested that a visual notation can better express the structure of the computer itself, so that we can understand its internal state[15] - this is a proposal which is investigated in experiment 1 in the next chapter. Readability is also compromised by the cognitive dimension of diffuseness - but researchers tended to claim that visual notations are less diffuse than text i.e. "a picture is worth ten thousand words", but with a scientific gloss "Pictures are more powerful than words as a means of communication. They can convey more meaning in a more concise unit of expression."[33]

B.2) Documentation

Programmers generally object to the drudgery of documenting the behaviour of code they have written. If a visual notation successfully communicates the intention of the designer[32,25], then additional documentation may be unnecessary. This question was of far more concern to professional programmers than to researchers, who were more likely to describe the same notational attributes in terms of the way that they directly express semantics during programme construction (B.3) or facilitate communication between members of a programming team or between designers and users of a system[1].

B.3) Syntax reduction

In the ideal intuitive programming language, no work would be needed to write a program. The user can express what they want naturally, and the programming work would be "automatic"[24,33]. New programming languages have been described as automatic ever since the development of FORTRAN. The way in which VPLs achieve this goal is often expressed in a comparison between syntax and semantics; the programmer need not be concerned with the syntax of a program[32], which is of interest only to the computer. It is the semantics of the program, what it is supposed to do, that is important. These views perhaps reflect some confusion about the nature of linguistic syntax - some discussions suggest that design notations contain purely semantic information, which is 'translated' into syntax when the program is written (this is addressed further under theme C.2). This is taken to mean either that the translation process is unnecessary when a VPL is used[3] or that semantics and syntax can be divorced and even written down side-by-side in a visual environment[25]. The real concern underlying these statements is likely to come from programming languages that require syntactic elements purely for the convenience of the compiler: making sure that keywords are used, variable types defined or lines terminated by a semicolon[12]. It is as these "low-level" syntactic constructs are obviated by more intelligent compilers[24,31] that the main advances are achieved in each generation of 'automatic' programming.

B.4) Modularity

Support for modularity is an important attribute of any programming language. Surveys 2 and 3 were conducted during a period when modularity was being given paramount emphasis via the promotion of "object-oriented" programming languages. Discussions of the advantages of VPLs sometimes attribute those advantages to the ways that VPLs encourage modularity or object-orientation: by presenting modules as icons on the screen or by physically enclosing modules within a diagrammatic boundary, for example. Research publications tended to make a clear distinction between the advantages of visual programming and the advantages of modularity, however.

B.5) Power and scalability

Respondents in survey 2 often commented that VPLs were unlikely to be as powerful as the languages they already used. The definition of "power" in this case is open to debate. Some programmers consider the most powerful languages to be those which encourage the highest level of abstraction. Research publications do claim that VPLs are good at showing abstraction[2], or communicate a higher level of abstraction[32], while others note that abstract data is challenging for VP precisely because it is not inherently visual[8]. As will be discussed in survey 2, these respondents were more likely to be concerned about reduced access to lower levels of abstraction within the computer.

B.6) Retention of text

Respondents in surveys 2 and 3 often noted situations in which text would continue to be necessary, even when using a VPL. Research publications on visual programming were unlikely to make this criticism of VPLs, however. It is an observation that tends to be assumed, rather than being an interesting subject for comment.

C) Cognitive processes

This category includes three themes that describe cognitive aspects of the programming task, rather than notational elements. Classification decisions for these themes were conservative: a statement saying that visual languages were easier to read, for example, is classified as a statement about the notation rather than about the cognitive processes involved in reading. The category is divided into three themes: the first deals with perception of visual languages, and short term visual memory. The second includes descriptions of mental models in visual terms, and the third includes questions of affect.

C.1) Perception and memory

The statements central to this theme claim that VPLs take better advantage of the visual sensory modality than do textual languages. It is observed that the human mind is "optimised" for vision[29,14], making shapes easier to process than words[22,16]. Many of these statements refer to mental processes in specifically computational terms. In computer science research, image processing algorithms are often implemented using parallel computer architectures, whereas language parsing algorithms seldom are. Some computer scientists appear to draw a direct analogy to human cognition, saying that vision makes the fullest use of the parallel computational architecture of the human brain[29,18,27]. Similar analogies are drawn in descriptions of visual memory. Visual percepts are considered to provide a superior basis for mnemonic chunking, because an entire image as treated as a "single unit" by the brain, thereby providing a high communication bandwidth when compared to words, which must be recognised one at a time[18]. Although it must be a tempting speculation, only a few writers suggest that VPLs employ right hemisphere resources while textual programming languages employ left hemisphere linguistic resources[34]. Perhaps increased popular awareness of functional localisation in neuroscience may cause such speculation to be more common in the VPL literature in future.

C.2) Mental models

Statements were assigned to theme C.1 when they referred to cognitive processes involved in perception of visual languages. A number of statements considered the process of generating new programs, starting from the premise that the mental model of the program might already be in a visual rather than a linguistic form[5] - an introspection that is also reported by expert programmers (Petre & Blackwell 1997). A VPL might therefore represent this visual mental model better than text can[28]. If there is a "semantic gap" between the programmer's conceptual model of what their program should do, and the computational model of the program itself[31], this gap can be "bridged" because the VPL depicts concepts directly[4,36]. This discussion is obviously related to theme B.3 (syntax reduction), but statements were assigned to the current theme where they expressly described the nature of the mental model.

C.3) Preference and affect

Many of the respondents in survey 3 simply stated that the VPL they used was more fun than textual equivalents. This is a perfectly good reason to choose a programming language, and may directly facilitate improved performance (Petre, Blackwell & Green 1998). Publications in the visual programming literature note that visual programming is more satisfying[17], appealing[24] or alluring[27] than text, which is time consuming, frustrating and labour-intensive[34]. The spirit of the advertising slogan "a picture is worth a thousand words" may be better captured by statements of attitude than of information content - these research publications observe the visual bias of modern society, in which people generally prefer pictures to words[35], motion pictures to books[37], and graphs to tables[18].

D) Metaphorical comparisons

This category contains themes with a direct bearing on the discussion of chapter 1. These authors are of course familiar with the desktop metaphor, and with the school of HCI research that teaches the importance of metaphorical interfaces. It is divided into three themes: the first refers to the analogical processes through which we can understand computers by relating them to our experience of the real world. The second refers to the claims of conceptual metaphor, that we understand abstractions in terms of concrete physical experience. The third refers to the metaphors of communication by which we compare human-computer interaction to natural language and conversation.

D.1) Applying real world experience

As in other HCI contexts, direct manipulation interfaces are a valuable aspect of a VPL. The reasons why this might be so are described in different ways, however. It is unlikely that icons are more "intuitively meaningful"[7] than words for many programming constructs, but Larkin and Simon's (1987) description of location as a coding mechanism (one which avoids hidden dependencies in reasonably small, planar diagrams) is echoed by these responses[10]. Direct manipulation employs manipulation skills based on ordinary actions in the physical world[10,27], but VPLs also use metaphors based on more specialised experience. Technical specialists in many fields choose to draw diagrams in order to help them understand complex situations[13]. Reference is made to the standard diagram forms used by software engineers as being "familiar"[31], or even "traditional"[24]. Perhaps other graphical representations such as musical scores, recipe books and construction schedules[37] could also be used directly as metaphors for programming.

D.2) Making the abstract concrete

Several of these extracts stated that people find it easier to deal with the concrete than the abstract[5], and that solutions are easier to perceive if abstract information is converted to a concrete (i.e. visual) form[13]. This may simply be reflecting an extension of the principle of direct manipulation, but more detailed cognitive theories are advanced, for example that abstract mental models are more easily processed in well structured problems, but that concrete models are better for semi-structured problems[21]. This sense of abstraction as distinct from concrete representations has been separated in the theme classification from level of abstraction, which is usually used in a more specialised sense by computer scientists (and is included in B.5).

D.3) Comparisons to natural language

The high-level contextual metaphor of user interface as a dialogue between the user and the machine is even more seductive when programming - where the form of interface is described as a "language". Programming languages are often compared to human language, as are the visual languages considered in this survey[19]. Visual languages are, of course, far less like human language than earlier computer languages such as COBOL, but this has not prevented attempts to extend the metaphor. Some passages found in this survey suggest that people have always found it natural to communicate with images[27], that pictures are a universal form of communication[35], and that VPLs will transcend language barriers and allow exchange of programs between different nations[37] (although the vocabulary must first be designed not to be culture-specific[33]). Support for these arguments comes from some interesting sources - although some writers observe that written languages have evolved from hieroglyphics to more advanced modern forms[26], others suggest that logographic scripts such as Chinese are superior to the scripts of Indo-European languages which are constrained by the need to follow phonetic structures[11].

E) Miscellaneous observations

A few statements in surveys 2 and 3 could not be assigned to any of the themes described above, either because they could not be interpreted, or because they expressed a unique opinion that did not touch on any of the defined themes. These are not analysed in further detail.

Results

Each statement was allocated to one of the themes, and classified as to whether it drew a negative or positive conclusion about VPLs within the terms of the theme. (Statements could also be classified as equivocal or ambiguous). Note that the tallies reported below count the number of respondents addressing each theme, not the total number of statements. If one respondent made multiple statements on the same theme, this contributed a count of one to the tally. If a respondent made both positive and negative statements on the same theme however, this would contribute to both tallies. Figure 3.2 illustrates the distribution of statements among the themes in survey 1.

This distribution is compared to surveys 2 and 3 below. The proportion of statements found in each theme was very different in each survey, as was the proportion of positive and negative statements; figure 3.6 will show that the statements found in research publications included very few negative statements regarding any aspect of VPL use.

The classification category containing the largest total number of statements across several themes was category A (contextual significance) - that VPLs would have a positive general impact, producing improvements in productivity while also being easy to learn.

The theme receiving most attention overall was that of readability - this is obviously seen as a fundamental reason for introducing visual programming. Other notational properties (category B) are not emphasised to the same extent, especially where they describe features that VPLs share with other programming languages (modularity, for example). Notational characteristics that affect program writing, such as syntax reduction, also receive less attention.

Statements regarding category C (cognitive effects of VPL use) include substantial numbers addressing all three themes. Mental models and perception/memory are particularly frequent, with levels of attention similar to those devoted to learnability and general impact. The three themes within category D, metaphorical correspondences between VPLs and previous experience, abstraction, and natural language receive approximately equal amounts of attention.

These themes are considered in more detail when contrasted with surveys 2 and 3 later in this chapter.

Discussion

The number of non-specific statements that are made about the general advantages of visual programming can be seen as a quantitative estimate of the predominance in the VPL literature of superlativism, as defined by Green, Petre and Bellamy (1991). This type of statement may also occur in other scientific disciplines - wherever it is necessary to describe the significance of research in general terms, even if the research itself is not applied. It is a matter for concern, however, where claims are made with no empirical support, when they could have been tested empirically. This is particularly true of the claims for learnability of VPLs - learning effects could be evaluated experimentally, but this has not been done in the publications surveyed.

Across the three surveys, a relatively wide range of themes addressed the notational properties of VPLs when compared to textual languages. This is an important comparison to make - Green's work on cognitive dimensions (Green 1989, Green & Petre 1996) has demonstrated the profound influence that notational structure can have on the ability to perform different cognitive tasks. In this survey, however, one type of task is addressed more often than all others. Despite the fact that reading a computer program is rather less demanding than writing or modifying one, it is the readability of VPLs that attracts most attention here. The reasons for that imbalance will not be addressed in the rest of this thesis, but the possibilities are worth considering. It is not that researchers are unaware of the need to write programs as well as read them: the themes in categories C and D address issues related to program writing. It is possible that improving readability is considered to be a more fundamental or tractable research issue than writability - either because writability is little affected by the language used, or because writing a program requires that the author continually read the code being generated (Green's parsing-gnisrap model - Green, Bellamy & Parker 1987). Alternatively, researchers may suspect that VPLs are less writable than textual languages, but choose not to comment on this in their publications. There is some evidence for this latter possibility in the statements made by professional users of a VPL in survey 3.

The themes in category C include most of the more explicitly metacognitive statements found in this survey about mental processing of visual languages. These statements are very much statements of psychological theory, but psychological research is almost never cited in their support. What is their status, in that case? They include a consistent range of arguments, and could be described as a theory of naive psychology (Hayes 1985) analogous to McCloskey's (1983) observations of naive models of physical phenomena. The population under consideration in this survey have a rather unusual basis for their psychological theories, however - as computer scientists, their simulation of mental processes (Gordon 1986) may be influenced by their experience of computers, just as Watt (1997) suggests that non-programmers understand computers on the basis of anthropocentric psychological models. Computational metaphors of mind do occasionally appear in the literature on reasoning with diagrams (e.g. Waisel, Wallace and Willemain's (1997) description of Johnson-Laird's mental models as "analogous to a Structured Query Language database"), but the statements found in this survey tend to be less precise. This is particularly notable in the statements describing human cognition in terms of "bandwidth", "efficiency" or "parallel processing". When computer scientists speak of cognition, they may be unduly influenced by the nature of their work, as claimed by Roszak:

[The computer] brings with it certain deep assumptions about the nature of mentality. Embodied in the machine there is an idea of what the mind is and how it works. The idea is there because scientists who purport to understand cognition and intelligence have put it there ... The subliminal lesson that is being taught whenever the computer is used (unless a careful effort is made to offset that effect) is the data processing model of the mind.
Roszak (1986), pp. 245,246

The metacognitive models found in this survey are not derived solely from observations of computers, however. The terminology used in some of the publications does indicate origins in psychological research - use of the term "chunking", for example[18]. It is possible that many of these statements can be traced to the earliest research into visual programming, which was directly motivated by the same cognitive theories that have been influential in HCI. Chapter 3 considered the influence of Smith's theory of metaphor as a tool for manipulating abstraction in graphical user interfaces (Smith 1996). The PYGMALION system that he developed to express those theories (Smith 1977) was not only one of the earliest graphical user interfaces, however, but one of the earliest VPLs. Popular descriptions of his work on the Xerox Star (Smith, Irby, Kimball & Harslem 1982, Smith, Irby, Kimball, Verplank & Harslem 1982, Johnson, Roberts et. al. 1989) may be partly responsible for broad dissemination of his theories of mental representation, visual short-term memory, creative use of mental images, and other related topics - his book cites the sources of these psychological theories, but few of those that follow him do so.

Survey 2: Professional users of conventional programming languages

This second survey was designed to answer the main question raised by the first. Are the metacognitive beliefs expressed by VPL researchers commonplace among computer programmers, or do they simply reflect the research traditions of the VPL academic community? In order to answer this, I selected a survey population who were unlikely to have been exposed to the research literature on VPLs, but who were both familiar with computer programming and accustomed to comparisons of different programming languages and techniques.

The results of this study have previously been published in Blackwell (1996c).

Population

This survey was conducted at a trade show organised by a computer magazine - the EXE Developer's Show - held in London on June 15-16, 1995. The nature of the magazine gives some insight into the survey population. EXE Magazine is a popular British programming magazine, similar to (but with a smaller circulation than) the American publications American Programmer or Dr. Dobb's Journal - it contains a mixture of opinion columns and educational articles and is widely read by professional programmers, as well as by knowledgeable hobbyist programmers. It does not publish academic papers, but would discuss topics such as visual programming in the context of a product review. Such reviews would normally be based on manufacturer's press releases rather than academic analyses, however. Readers of the magazine might therefore have some awareness of new commercial trends in programming languages, but would probably not have encountered the cognitive theories motivating VPL research. Although there was a small conference associated with the show, the speakers addressed commercial issues rather than scientific ones - a programmer with a strong interest in programming language research would have chosen to attend one of the many more technical conferences held in London in 1995, rather than this one.

Method

The survey was conducted using a written questionnaire, reproduced in Appendix A. It started with two qualifying questions. The first question tested whether the respondents were professional programmers, as expected by the trade show organisers. The title and the second introductory question were worded in order to avoid confusion between VPLs and the Microsoft Visual Basic and Visual C++ products (which would have been very familiar to most respondents). The body of the questionnaire consistently used the term 'graphical programming' rather than 'visual programming', and the second introductory question explained this as follows:

Do you have any experience of a graphical programming language, where the programmer does almost all programming by manipulating diagrams instead of typing text? Please note that this definition does not include tools like Visual Basic, where the program logic is created using a text language.

In order to ensure that the point about Visual Basic had been assimilated, respondents were asked to name a VPL that they had used, seen or read about.

The third question asked respondents to compare a graphical and a textual programming language along five different dimensions: ease of use, power, enjoyability, readability and changeability. They were asked to name the text language they were comparing, and to either name a graphical language or write "guess" if they did not know the name of one. Two Likert scales were provided for each dimension, so that the respondent made separate assessments of graphical and textual languages on each dimension. The extremes of the scales were labelled separately, as follows: "hard to write/easy to write", "weak/powerful", "irritating/enjoyable", "unreadable/readable", "hard to change/easy to change". A further three unlabelled scales were provided, so that respondents could make comparisons on other dimensions that they might consider to be important. A differential response was encouraged by using six point scales, with no mid-point.

The final question was designed to elicit statements about the cognitive nature of the programming task, as investigated in survey 1. It read:

Please explain in a few sentences how you think a graphical programming language might make a difference to the "brain-work" involved in programming.

The responses to this final question were analysed using the same coding frame that was described under the heading of survey 1. As for survey 1, an initial analysis of the results was published before survey 3 was carried out (Blackwell 1996c). The results reported here are based on the single coding frame that was formulated later to encompass surveys 1, 2 and 3 (Whitley & Blackwell 1997). The results that are presented in the following section are therefore organised according to the same structure of themes and categories that was described earlier.

Results

The number of people attending the EXE Developer's Show was somewhat less than 1000 people (the organisers did not release exact figures). I distributed 506 questionnaires to visitors as they arrived at the show, and 88 of these were returned, either to a collection point on one of the exhibition stands, or by post afterwards.

According to their answers to Question 1, all but two of the respondents (98%) were professional programmers. This is as expected from the target readership of EXE Magazine.

Despite the instructions given in Question 2, 25% of the respondents did name Visual Basic (or a similar product) as the VPL that they were familiar with. This is a severe difficulty when surveying opinions about novel programming languages. Although the questionnaire never used the word "visual" and explicitly excluded Visual Basic, respondents who were unfamiliar with visual programming tended to respond confidently, but by reference to languages that they knew. The responses were divided into three groups according to reported experience with VPLs. (Seven respondents did not complete this question).

• Group I had seen, or were familiar with, a VPL (N=23).
• Group II had no prior experience of VPLs (N=37).
• Group III named Visual Basic or a similar product as a VPL (N=21).

All groups of respondents considered VPLs to be less powerful than text (the average rating for graphical languages was lower by 1.48 on the scale of 6), but easier to use (the average rating for graphical languages was higher by 0.77). The only scale for which there was a significant difference between the groups was the rating of whether graphical languages would be enjoyable to use. Group I gave graphical languages a slightly higher rating on this scale (by 0.35 out of 6), while groups II and III gave graphical languages lower ratings: by 1.14 and 0.33 respectively, F(2,78)=3.51, p<.05. When the independence of the different rating scales was compared, the strongest correlations (both positive) were found between the ratings of power and enjoyability (r=.55, p<.001), and between readability and changeability (r=.53, p<.001).

Although a significant difference was observed between some of the ratings given by different groups within the sample, a quantitative comparison of the number of positive and negative statements made in response to the open question found no significant difference between the three groups, c2(2, N=6)=0.69, p=.71. The statements made by respondents to this survey are therefore considered as a single group when comparing them to those made by VPL researchers (survey 1) and professional VPL users (survey 3). The following discussion describes the differences in emphasis that were found within each coding theme for this survey. The variation in distribution of statements between surveys 1 and 2 is shown in figure 3.4.

A) Contextual significance

Respondents in this survey were more likely to describe VPLs as having limited applicability, or only being appropriate for use in certain contexts.

A.1) General impact

Where VPL researchers often made superlativist claims regarding the benefits of VPLs, programmers in this survey were more cautious about the advantages. They were also readier to impute practical disadvantages to VPLs, including increased effort, reduced quality, or difficulty of maintenance. The motivation for this hostility might be partly explained by the concern expressed by some respondents that the ultimate objective of VPLs was to make professional programmers redundant by allowing other employees of a company to do programming work.

A.2) Learnability

Whereas VPL researchers often claimed that VPLs are naturally intuitive, the only professional programmer who mentioned intuition stated that VPLs were less intuitive than text.

A.3) Productivity

Professional programmers generally accord greater importance to productivity than to ease of learning. Responses to this survey considered that VPLs were intended for casual users, with a shallow learning curve that limits productivity.

B) Notational characteristics

Some respondents, having used many different programming languages, considered that their work is little affected by the language that they use. They observed that all languages are equivalent in their computational potential, and apparently have no regard for the cognitive ergonomics of the notation they used. Nevertheless, many respondents did make statements on notational issues.

B.1) Readability

Some respondents to this survey did agree with VPL researchers that VPLs can clarify relationships within a program. Many also expressed reservations about whether visual programs would scale up to large programs, or could contain more complex information.

B.2) Documentation

Where documentation was mentioned, respondents considered that a VPL would be beneficial, because easier to read.

B.3) Syntax reduction

While researchers proposed that VPLs allow direct manipulation of semantic information without syntax, programmers did not describe them in these terms. They did say that the programmer may be better able to concentrate on design and functionality rather than coding. This view was also commonly found in survey 3, and it interacts with theme C.2, which proposes that the suitability of the VPL for design tasks arises because of a resemblance to the mental model of the programmer.

B.4) Modularity

Many respondents were particularly concerned with the principles of structured design, stating that VPLs might encourage modularisation, decomposition, assembly of software components or object-orientation. All of these are topics that were of common concern in the industry at the time the survey was conducted, and might have been attributed to any new product.

B.5) Power and scalability

The tone of statements addressing this theme was divided: 33 respondents made positive comments regarding the power of VPLs, while 40 were sceptical or completely negative. Many limitations were mentioned, including that VPLs did not allow direct access to the machine, that large programs exceeded or cluttered the view area, or that program functionality is fragmented and obscured by event-driven code.

B.6) Retention of text

Several respondents pointed out that VPLs still require typing; for names, expressions, or component customisation.

C) Cognitive processes

Question 4 in the survey explicitly asked respondents to comment on the brain work involved in programming. This should have produced a strong bias toward the type of statements found in survey 1 on the themes related to cognitive processes. Despite this encouragement, respondents in survey 2 made proportionally fewer statements on these themes than were found in survey 1.

C.1) Perception and memory

Some programmers proposed that VPLs might reduce cognitive load, freeing the programmer to think about other problems. No statements were made about perceptual resources or working memory.

C.2) Mental models

Several programmers did describe VPLs as corresponding to their mental models. As in surveys 1 and 3, some respondents said that VPLs could directly express ideas - a property described in one case as 'direct stimulation of the brain'.

C.3) Preference and affect

As noted earlier, groups II and III (who had no experience of VPLs) were more likely than group I to give a poor rating to the enjoyability of visual programming. This is borne out in responses to this open question. Where researchers stated that a major benefit of VPLs would be programmer's enjoyment in using them, a number of respondents in survey 2 objected that VPLs would reduce the quality of their work, make maintenance more difficult, waste their time, and reduce the intellectu al challenge in their work.

D) Metaphorical comparisons

Do the statements made by VPL researchers about the value of metaphor reflect a general belief among professional computer users, or are they specific to the research community, perhaps due to the influence of only a few publications, as suggested in the analysis of survey 1?

D.1) Applying real world experience

Respondents in this survey never observed that VPLs might resemble the real world. They did make some comments about the use of direct manipulation interfaces, but these were often negative - objecting that icons were hard to remember and manipulate. Group III respondents were positive about the general benefits of graphical user interfaces, but this is unsurprising, given that for Visual Basic/C++ programmers, the creation of graphical user interfaces is their livelihood.

D.2) Making the abstract concrete

Where VPL researchers stated that people find it easier to deal with the concrete than the abstract, and that solutions are easier to perceive if abstractions are converted to a visual form, some programmers mentioned that the use of diagrams can restrict the development of abstractions.

D.3) Comparisons to natural language

One respondent in group III did echo the ambitions of VPL researchers, that VPLs might help to cross language barriers by being less dependent on English. As a Visual Basic/C++ user, this concern was probably derived from the emphasis on internationalisation in those tools.

E) Miscellaneous observations

A few respondents made comments relevant to only one product. These were not analysed any further.

Discussion

The assessments made by respondents in this survey can be reasonably summarised by the relative ratings they made of VPLs and textual programming languages on Likert scales. The attitude towards VPLs was generally sceptical, and even hostile in some respects. This was true for each of the three groups identified in the sample, although respondents who had at least seen a VPL were more likely to be charitable, particularly in perceiving VPLs as being enjoyable to use - the same attitude is observed on a larger scale in survey 3. It is notable that Group II, with no experience at all of VPLs, were completely convinced that they would be unpleasant to use. This type of response is consistent with other studies of professional computer users, which find that their attitudes to their tools are often a matter of vigorous allegiance - this effect is strongest when the new tool is substantially different to the old, as in the case of the Macintosh (Jones 1990).

This attitude appears to be linked to the niche that VPLs are considered to fill in the spectrum of software tools. Respondents expressed the main advantage of VPLs as being that they were easy to use. This ease of use is thought to be achieved to the detriment of the tool's power, however. For these professional tool users, it is the power of the tool that is most strongly correlated with their assessment of whether they would enjoy using it - this correlation will be seen again in survey 3.

The responses to the open question follow the same broad pattern of opinion. VPLs are described as easy to use in some respects, but deficient in many aspects important to professional programmers. The general impact of VPLs (category A) is defined by the fact that they are designed for casual users, but at the expense of reduced productivity. They even threaten the livelihood of professional programmers, by encouraging non-professionals to meddle with programming.

Statements made about the notational advantages of VPLs (category B) were generally realistic, indicating that programmers have some appreciation of the characteristics described by Green's Cognitive Dimensions. Rather than appealing to intuition, it is more reasonable to attribute the advantages of a visual representation to the fact that it clarifies relationships, and expresses functionality in a way less obscured by typographical syntax. This advantage is offset by the fact that the visual representation may allow a more limited repertoire of expression, or may not scale up to the description of larger systems.

Despite the fact that the open question specifically asked respondents to speculate about the cognitive processes involved in programming, most of them restricted their comments to aspects of programming with which they were familiar - the notation itself, and the context in which it is used. The proportion of statements describing cognitive processes (category C) and the use of metaphor (category D) was far lower than had been found in survey 1. The few respondents who did mention these issues seemed to do so on the basis of some familiarity with HCI research (for example, in referring to cognitive load). The one exception in these categories was theme C.2 - mental models. Of those respondents who made any statement about cognitive issues, the great majority described visual representations as corresponding to the mental model they had of the design. This is in accordance with the introspective statements found by Petre and Blackwell (1997) in their analysis of expert programmers' descriptions of mental imagery. The respondents in this survey did not elaborate their descriptions, so it is not possible to tell whether they are referring to the complex images described by Petre and Blackwell, or simpler representations such as flowcharts - further evidence on this question was found in survey 3.

Survey 3: Users of a visual programming language

A large difference was found between the opinions expressed in surveys 1 & 2. This result does not give any support to the hypothesis proposed in the analysis of survey 1 - that there are widespread and consistent metacognitive theories of naive psychology regarding the way that programmers use visual representations. The responses in survey 2 appeared to be heavily biased, however, by the hostility that the respondents expressed toward VPLs. It may be the case that survey 2 respondents have similar metacognitive theories to VPL researchers, but they chose not to express them, concentrating instead on negative aspects of VPLs. In order to test this, a further survey was made of experienced users of visual languages

This study was used to collect data for two relatively independent projects - the second project being an investigation of the computational advantages of LabVIEW, conducted by Vanderbilt University PhD student Kirsten Whitley. It has previously been published by Whitley and Blackwell (1997).

The population of survey 3 were users of LabVIEW (Laboratory Virtual Instrument Engineering Workbench), a programming environment marketed by National Instruments for development of data acquisition, analysis, display and control applications. LabVIEW features a dataflow-based VPL, called G, which is promoted as being usable by scientists and engineers having limited programming experience, but who need to write software to interact with laboratory equipment. LabVIEW has been commercially available for 10 years, and has enjoyed relatively wide success compared to other VPLs, the majority of which are research prototypes. It is therefore an ideal candidate for studying a VPL with a sizable population of experienced users.

There have also been previous empirical studies of LabVIEW. Baroth and Hartsough (1995) describe the experience of their company using VPLs. In one case study they compared the progress of two development teams, one using LabVIEW and the other using the textual language C. The teams developed systems concurrently, to the same specification. After three months, the C team had not yet achieved the specification, while the LabVIEW team had exceeded it. On the basis of this and other studies, Baroth and Hartsough report that LabVIEW projects are 4 to 10 times faster than those using a textual programming language. They attribute this superiority to the fact that VPLs are easier to read, and that the style of LabVIEW makes it familiar to the electronics engineers in their company. In contrast, an experimental study by Green, Petre and Bellamy (1991) revealed no performance benefits resulting from LabVIEW's visual notations for conditional logic. In fact, their sample of experienced LabVIEW users and electronic designers were uniformly faster at interpreting conditional logic written in a textual language than in LabVIEW.

The clarification of these apparently contradictory findings was one of the main concerns in the design of survey 3. It is possible that Baroth and Hartsough were incorrect in attributing performance improvements to LabVIEW's visual notation. It may be that LabVIEW does improve programming but that the cause of the improvement stems from language features other than the visual syntax. These issues are the concern of Whitley's project (Whitley 1997), and are described by Whitley and Blackwell (1997, 1998). Those parts of survey 3 which relate to Whitley's project are not described in any further detail in the current discussion.

Population

The respondents in this survey were recruited from two sources of LabVIEW users. The main source was info-labview, a mailing list for LabVIEW programmers. As of January, 1997, info-labview had approximately 2,300 subscribers. The second source was an email directory, compiled by National Instruments, of academic users of LabVIEW. We sent invitations to participate to 104 addresses from this directory. The third source was readers of the Internet newsgroup comp.lang.visual. National Instruments also created a link to our survey from their home page on the World Wide Web.

Method

This survey was administered electronically in two versions: an HTML form accessible via the World Wide Web, and an e-mail form for which responses could be inserted between question texts. The majority of responses used the HTML version, which can be seen in Appendix A. When a response was submitted from the Web page, a program on our Web server (a cgi-script implemented in Perl) analysed the response, checking for complete answers. If the script detected missing information, it asked the respondent to fill in the missing information. This precaution resulted in a substantial increase in the number of complete responses.

The questionnaire was divided into four parts. The first requested information about the respondent's programming background. Question 1 asked for an email address (this was optional - in case clarification was needed - but was eventually used only to award incentive prizes supplied by National Instruments). Question 2, like the first question in survey 2, asked whether the respondents were professional programmers. Question 3 asked respondents to make a qualitative estimate of programming experience, in terms of the number and size of projects completed in LabVIEW and in other languages. Question 4 asked respondents to name the programming language that they had had the most experience using.

The second part of the questionnaire elicited opinions about LabVIEW as a whole, without emphasizing its visual aspects. This part was exploratory, and included several open questions designed to invite comments about non-visual aspects of LabVIEW, if respondents considered those to be important. Questions 5, 7 and 8 asked respondents for their overall opinion of LabVIEW, examples of how LabVIEW makes programming easier, and examples of how LabVIEW makes programming difficult. Question 6 directly addressed Whitley's hypothesis regarding Baroth and Hartsough's observed productivity increases from LabVIEW use. It asked respondents to assess the relative importance of various LabVIEW features using a 6-point Likert scale. The visual language G was only one of these features - others included reusable libraries of LabVIEW code, data acquisition hardware, and user interface development facilities.

The third part of the questionnaire asked explicit questions about the visual aspects of LabVIEW (i.e. the VPL G), and was very similar to the questionnaire used in survey 2. In Question 9, respondents are asked to compare G along several dimensions against a textual programming language of their choice. The first five of the dimensions are identical to the dimensions used in survey 2: power, ease of use, readability, changeability and enjoyability. We also added two further dimensions specifically to address the findings of Green, Petre and Bellamy (1991) - these asked respondents to compare support in G for conditional logic (as tested by Green et. al) and repetitive logic. Finally, question 10 is identical to the final question in survey 2; it asks respondents to explain how and why the graphical nature of G affects the "brain-work" required in programming.

The coding frame used for open questions was described above in the discussion of survey 1, although the five theme categories described earlier only constitute half of the themes to which statements could be assigned in survey 3. A second hierarchy of themes dealt with non-visual aspects of LabVIEW, as described by Whitley and Blackwell (1998). Those themes are not discussed here. Although respondents were not asked specifically about the effect of using the graphical language until the end of the questionnaire, some respondents did choose to describe cognitive effects when answering the earlier questions. We therefore aggregated all responses to open questions, and coded them as referring to visual or non-visual aspects of LabVIEW according to the content of the response, rather than the context of the question being answered. This is consistent with the approach taken in survey 1, where metacognitive statements were found in contexts ostensibly describing questions of computer science rather than questions of psychology.

Results

The survey was advertised on 11 March 1997, and we received 227 complete responses by 1 April. The majority of respondents (132) were professional programmers, as in survey 2. Of the remainder, 75 said that programming was only part of their job, 14 were academics and 6 gave 'other' responses. No significant differences were found between these groups in their responses to any of the questions discussed below.

As expected, LabVIEW users held far more positive opinions about VPLs than the programmers questioned in survey 2. When the relative differences between the ratings given to text languages and LabVIEW were compared to the relative differences in survey 2, multivariate analysis of variance (MANOVA) indicates a significant difference across all five rating scales, F(5,297)=0.317, p<.001. The rating differences between the two surveys are illustrated in figure 3.5. LabVIEW programmers rated LabVIEW as superior to text in all respects, including those where survey 2 respondents thought that VPLs would be worse than text: power, readability and enjoyability, t(305)=8.53, p<.001, t(306)=5.80, p<.001 and t(302)=8.18, p<.001 respectively. Although survey 2 respondents considered VPLs less powerful than text, they did accept that VPLs might make programs easier to write. Survey 3 respondents also considered that LabVIEW improved ease of writing more than power, t(225)=7.70, p<.001. The relative size of this difference is greater in survey 2 than in survey 3 however, t(305)=3.51, p<.001. Both survey 2 and survey 3 respondents considered that VPLs facilitated writing programs more than reading them: t(80)=3.95, p<.001 and t(226)=2.90, p<.005 respectively.

In survey 2, the sub-group who had never encountered a VPL were found to be more negative in their opinions. A complementary effect was observed in this survey. Respondents who were not familiar with any textual programming language were instructed to enter "guess" when naming a textual language to compare to LabVIEW. The 21 respondents who did so gave significantly poorer ratings to textual languages than other survey 3 respondents, especially regarding the power of text relative to LabVIEW, t(224)=4.68, p<.001). As with survey 2, programmers who know little about a language are likely to have a poor opinion of how powerful it is. This hypothesis is supported by correlations between the amount of experience reported by respondents and the total rating marks given. The total rating given to LabVIEW was positively correlated with amount of LabVIEW experience, but there was no correlation with general programming experience, r=.35, p<.001. The reverse was true of total rating given to text which was only correlated with general experience, r=.34, p<.001. This suggests that programmers do not normally assess programming languages by critical generalisation from their experience of other languages, but rather by a switch of opinion from scepticism to comfort and familiarity.

One of the objectives of this survey was to find out whether LabVIEW programmers were aware of its weakness in expressing conditional logic, as observed by Green et al. (Green, Petre & Bellamy 1991; Green & Petre 1992). If so, they should give a lower rating to LabVIEW when comparing its conditional logic to text languages than they do when comparing repetition constructs. In fact, respondents' awareness of this issue depended on the amount of familiarity they had with textual languages. Those who entered "guess" in question 9 were significantly more positive than other respondents in their assessment of conditional logic in LabVIEW, t(224)=2.76, p<.01, but not in their assessment of repetition constructs. Similarly, respondents who reported having completed more projects with other languages than with LabVIEW generally gave a negative assessment of LabVIEW's conditional logic (poorer than text by 0.36), while the rest of the sample gave it a positive assessment (better than text by 0.39), t(224)=3.13, p<.005. These textually-experienced respondents gave a neutral assessment of LabVIEW's repetition constructs (the mean assessment differed only by 0.008 from that given to text languages).

The coding framework for responses to open format questions has been discussed in the presentation of surveys 1 and 2. The version of the framework described in this thesis was created to encompass all three surveys, however. In the case of survey 3, the sample was sufficiently large to allow a simple assessment of coding reliability between the two raters. One rater coded all responses, then the second rater coded a randomly selected 20% sample, excluding those responses that had been used when creating the coding framework. The codes assigned by the second rater agreed with the first in 92% of the statements. The total proportion of negative and positive opinions found in survey 3 was intermediate between surveys 1 and 2, as shown in figure 3.6. Survey 3 respondents were more likely to be positive about VPLs than in survey 2, but less than in survey 1. They were more likely to be negative than survey 1, but less than survey 2. This is consistent with the assessments made on the rating scales of surveys 2 and 3. The following paragraphs compare the level of interest in each of the themes across the three surveys.

A) Contextual significance

Figure 3.7 summarises the relative proportion of statements made on contextual significance themes in surveys 1, 2 and 3.

A.1) General impact

As with statements found in survey 1, LabVIEW programmers often mentioned general ease of use as an advantage of VPLs. Survey 2 respondents more often stated that this goal would not be achieved - that VPLs would make no difference to users, or even be more difficult to use.

A.2) Learnability

Both VPL researchers and LabVIEW programmers cited ease of learning as a benefit of VPLs, where survey 2 respondents were more likely to criticise the time required to learn simplistic languages. LabVIEW programmers did express reservations about the difficulty of learning the dataflow paradigm of LabVIEW, which was very different to languages that they had used previously.

A.3) Productivity

LabVIEW programmers reported productivity benefits not only in coding, but in the design, debugging and maintenance phases of software development. The survey 2 programmers believed the reverse - that the advantages of visual programming would be found only during coding and that these other phases would see no change or even decreased productivity.

B) Notational characteristics

Figure 3.8. summarises the relative proportion of statements made on notational themes in surveys 1, 2 and 3.

B.1) Readability

Respondents in all three surveys regularly noted that VPLs clarified the structure of a program. LabVIEW programmers were more likely than survey 1 or 2 respondents to discuss "Gestalt" views or the "big picture" of a program. They were also more likely to describe the negative aspects of LabVIEW's readability; complaining about messy, cluttered code - literally resembling "spaghetti".

B.2) Documentation

Only a few respondents in each survey referred specifically to the role that visual representations can play as a documentation and communication medium. Baroth and Hartsough's (1995) opinion that LabVIEW facilitates communication between programmers and their clients is not widely supported.

B.3) Syntax reduction

VPLs do mitigate the problem of minor syntax errors in programming, This aspect of VPLs was not often mentioned by VPL researchers, however. The professional programmers in survey 2 and 3 were concerned with minimising syntax. In fact, two LabVIEW programmers complained about the novel syntactic demands in wiring up large programs.

B.4) Modularity

Survey 2 respondents mentioned modular programming more often than survey 3. This is not a major emphasis in LabVIEW, whereas survey 2 was conducted at a time when object-orientation was the main technical trend being adopted in commercial programming.

B.5) Power and scalability

Survey 2 respondents appeared concerned that high-level VPLs might deny them access to the low-level facilities of the machine that are so important in PC programming. This has often been a matter of concern with new generations of programming language, and it was most uniformly seen as a disadvantage by survey 2 respondents. Neither researchers nor LabVIEW programmers appeared as concerned.

B.6) Retention of text

A few survey 2 respondents seemed to find it hard to believe that an arbitrary computational operation could be represented graphically; stating that there would always be some level where the visual representation would be inadequate. This is not the case in LabVIEW.

C) Cognitive processes

Figure 3.9. summarises the relative proportion of statements made on cognitive process themes in surveys 1, 2 and 3.

C.1) Perception and memory

Survey 3 respondents seldom mentioned perceptual processes. One respondent explicitly discounted the type of statement made in survey 1: "the human brain is massively parallel but basically operates in a linear fashion. G is parallel, but not in the least bit linear."

C.2) Mental models

As in surveys 1 and 2, LabVIEW programmers also stated that the designs they construct in their minds are in some sense pictorial or that VPLs are closer in nature to their thoughts than are textual languages. These opinions may be influenced by the predominance of diagrammatic notations in software design, but they are supported by the introspective evidence reported by Petre and Blackwell (1997).

C.3) Preference and affect

Survey 1 researchers stated that VPLs would be popular simply because people would enjoy using them. Several LabVIEW programmers said that they think using LabVIEW is fun, but the conservative coding policy followed in this survey meant that those statements were not interpreted as relating to the G language unless respondents explicitly said so - it may be the case that respondents consider the editing environment or the front panel assembly in LabView to be fun, rather than the G language.

D) Metaphorical comparisons

Figure 3.10. summarises the relative proportion of statements made on metaphorical comparison themes in surveys 1, 2 and 3.

D.1) Applying real world experience

LabVIEW exemplifies the principle of presenting abstraction in terms of real-world concepts that are already familiar to the programmer. Several LabVIEW users agreed that their previous experience in engineering or electronic circuit-building had been very important to them in learning the wiring metaphor that is central to LabVIEW.

D.2) Making the abstract concrete

The claim made by VPL researchers, that VPLs make abstract concepts easier to understand by presenting them in the form of concrete images, was hardly mentioned by respondents in either survey 2 or 3.

D.3) Comparisons to natural language

Survey 3 respondents seldom compared VPLs to human languages; a comparison that was found in survey 1.

E) Miscellaneous observations

In survey 3, some issues specific to LabVIEW were not compared directly to other surveys, despite the fact that they can be seen as general problems in VPLs - for example, the fact that it can be difficult to produce a paper print-out of a visual program. This type of issue may be a significant irritation to professional users of a VPL, but neither researchers nor non-VPL users would be likely to consider it.

Discussion

The results of this survey have emphasised the conservatism of professional programmers' opinions about their tools. In both surveys 2 and 3, respondents were most positive about the tools that they had most experience using. Their opinion of new techniques or unfamiliar languages was generally sceptical and even hostile. These opinions were often expressed confidently, but appear to be associated with lack of experience and understanding of the tools being discussed. The reasons for this attitude have not been investigated here, but there are several obvious candidates: the highly competitive market for software tools, the existence of support groups where programmers meet to reinforce their allegiance to specific languages, and the desire to protect one's professional specialist knowledge of existing tools, among other factors. In the context of these attitudes, it is not surprising that so many respondents in survey 2, rather than echoing the opinions of VPL researchers about the benefits of visual programming, denied that VPLs would bring any advantages at all. The balance of opinion was restored in survey 3, where, with much the same motivation, LabVIEW programmers were highly positive about VPLs and derided textual programming languages (especially if they had not used them).

In cases where the survey respondents analysed language facilities on a more critical basis, it seems that the area of improvement most interesting to professional programmers is facilities that might improve their productivity during mundane tasks. These improvements can be analysed in terms of relatively well-understood usability measures, especially Green's Cognitive Dimensions. Survey 3 respondents with experience of both VPL and text languages confirmed the finding of Green, Petre and Bellamy (1991) that experimental subjects are slower to interpret conditional logic in LabVIEW than in BASIC.

There are further observations made by respondents that have not yet been investigated empirically, and may be worthy of research. The VPL researchers in survey 1 appear to consider that the main advantage of visual representations is that they are easier to read. Respondents in both surveys 2 and 3, however, reported that the main contribution is that they are easier to write, with readability being relatively poor. A similar distinction between the appearance of comprehension and actual lack of understanding has also been found in studies of English text reading (Glenberg, Wilkinson & Epstein 1982). Another theme expressed in both surveys 2 and 3 is the perceived trade-off between power and usability in programming languages. There is no obvious theoretical reason why a powerful language should be difficult to use. It would be interesting to investigate whether this perception comes from the dichotomous marketing of programming tools in "student" or "end-user" versus "professional" categories, or whether the perceived trade-off arises from cognitive dimensions such as diffuseness and abstraction gradient.

Professional programmers, while aware of potential improvements in productivity from new languages, give little attention to the theories of cognition that are mentioned by so many VPL researchers as motivating the development of VPLs. The one topic in this area that is addressed regularly in all three surveys is the belief that programming involves the creation of image-like mental representations. This intuition is supported by in-depth interviews with expert programmers, as reported by Petre and Blackwell (1997). Despite this common intuition, it is not a necessary fact that the formation of image-like mental models will be facilitated by the use a diagrammatic external representation - what Scaife and Rogers (1996) call the resemblance fallacy. The relationship between mental imagery and diagrams motivates the first experiment to be discussed in chapter 3, and is investigated in considerable detail in chapter 4.

The hypothesis that VPLs are essentially metaphorical is seldom raised by the professional programmers of surveys 2 and 3. LabVIEW includes a quite specific metaphor - of programming as the creation of an electronic circuit (the cursor, when connecting components, resembles a spool of wire). Despite this emphasis, relatively few LabVIEW programmers mentioned it as an important benefit. If the idea of programming languages as metaphors is not widespread amongst programmers, why is it so often mentioned by VPL researchers? The researchers do not cite specific sources, so it would appear that this metacognitive theory has been transmitted through the VPL research community by informal means, perhaps originating from Smith's (1977) description of visual programming as a "metaphor for thought". After completing the formal analysis of survey 1, and throughout this research project, I continued to find statements in the research literature making even more explicit claims about the relationship between abstraction, mental imagery and metaphor. These might easily have appeared in Smith's original publication. A typical example is:

Rather than forcing the programmer to make his concepts of how a system should work conform to a given programming language and environment, the environment and language should conform to the programmer's concepts. […] Attempting to have programmers directly [sic] with their conceptualizations has several interesting implications: The environment has to be graphical. People often think using pictures. Many of the conceptual views of programming are two-dimensional, graphical ones. The programmer must be able to work using the pictures that compose his conceptualizations and to assign meaning to these pictures.
Reiss (1987), p. 178

The remainder of the current thesis principally investigates this premise of VPL research. If the intuitive claims of VPL research are justified, and if Smith's original insights are supported by empirical evidence, then the use of metaphor must be treated far more seriously as a fundamental design principle for all types of diagram. The experiments in the next chapter and in chapter 5 undertake this investigation.

Continue reading chapter 4 , or return to table of contents and download information .