Course pages 2012–13

Usability of Programming Languages

The principal means of assessment is three written assignments and a final presentation. Participation in seminars is also required.

The goal of the course is that the final presentation should be of a standard suitable for presentation as a student paper at a research venue such as PPIG.

All written assignments should be submitted to the Graduate Education office.

Assignment A guidance

The requirement for assignment A is to describe the background to your proposed study. This should include three aspects:

1. A description of the experimental environment. This may be a programming language, a programming paradigm, a specific tool, or a programmable product.
2. A brief review of literature in psychology of programming and related fields that is relevant to your particular research concern.
3. A summary of one or more previous empirical studies that have used a similar experimental method, or evaluated the same kind of environment, as a starting point for designing your own study.

The overall length of the assignment will be 2-3 pages (around 1000 words), and should refer to at least 4 or 5 previous publications, possibly more. These guidelines are for estimation purposes only, not a requirement. I will be happy to discuss them further in the next lecture.

Assignment B outline

The outline version should have the same overall structure as Assignment B itself. However, it is not expected that all details will be finalised, or that narrative text is written in full publication-style. The outline will be discussed in class, with the expectation that the final version of assignment B will be modified in various ways.

Assignment B guidance

The requirement for assignment B is to describe the design of your proposed study. This should address:

1. The research question(s) that you are addressing.
2. The structure of the study. If the study is a controlled experiment, this should include numbers of tasks, conditions and measures, as well as general structure: are conditions within or between subjects, and are groups, conditions, and task order balanced?
3. Participants. How many, how will they be recruited, are they divided into experimental groups?
4. Tasks. What will you be asking participants to do (including training exercises)? How long will each task take? Include any pre- or post-experiment questionnaires or interviews.
5. Data. What data will you collect? Will any transcription or coding be required? Will it be necessary to recruit further participants (for example to make independent ratings during coding)?
6. Analysis. How will you address the original research questions, based on the data that has been collected?
7. Ethics. Will your experiment raise any ethical issues? Before proceeding to a pilot study, you will need to make an ethics application. This should mention your research question, details of participants and tasks, and any potential ethical concerns you have identified and are addressing.

If your planned study is not a controlled experiment, then there may be various other questions that should be discussed. These will depend on the nature of the study.

The assignment should be no longer than necessary to address these questions. It should be written as far as possible in the style of a research publication - though as a proposal, it will be in the future tense. It is not necessary to repeat the introductory motivation and background literature review from assignment A - but sufficient detail should be provided that the nature of the tasks, conditions, participants and so on is clearly explained in context. Where direct references are made to other research (for example, to replicate tasks, or apply standardised measures), citations should be used.

Assignment C outline

The outline version of assignment C should focus on the experimental results, structured in a way that makes it clear how these are derived from research questions, and support analytic findings. Some members of the class may still be recruiting participants, or completing their experiments. This is taken into account when marking.

1. Review the research questions being addressed
2. Summarise the structure of the study design (groups, treatments, measures etc)
3. Report current status of the study, and summarise the data collected
4. Describe the analysis work that has been completed, and if sufficient data is available, discuss the findings

Assignment C - final report

This should be a complete and self-contained report, presented in the style of submissions to a major research venue in the field. It will incorporate or extend material from the earlier assignments where appropriate. It should present motivation, background literature, and theoretical concerns of the work. It should describe the study that has been conducted at a level of detail that would allow scientific replication. The structure of the empirical investigation should be reported in a standard format for this kind of study. Results, analysis and interpretation should lead to scientifically defensible conclusions.